4.4 Building the Dataset: The Stage That Decides Everything

Everything in chapter 4.5 is mechanical. The dataset is not. A training run with mediocre parameters on an excellent dataset produces a usable identity. A training run with perfect parameters on a narrow dataset produces a LoRA that can only make one photograph of one person.

Here is the reframing that matters for a synthetic persona, because it inverts the standard advice: you do not collect this dataset, you design it. Tutorials for real people treat acquisition as the hard part — twenty-five good photographs of somebody who is not a model. You have an unlimited supply of candidates, because you can generate them. Your hard part is selection: producing a set that is diverse enough to teach the model what varies, and consistent enough to be the same person throughout.

How many, really

The field converges on 20 to 50 images, and the framing is important: that is a range, not a target. Fewer than about 15 and the model has not seen enough of the face. More than roughly 50 and you are mostly adding redundancy, which teaches the model your redundancy rather than your range.

The workflow that produces the right number:

1. Generate a large pool — hundreds of candidates, from many prompt templates, using the adapter or reference path from 4.3.
2. Score for identity against the reference plate, discarding anything that does not look like the persona.
3. Score the survivors for coverage against the checklist below.
4. Keep 20 to 30 and stop. Resist the urge to keep 80 because they were good.

The two scores are different questions. A hundred images that all pass the identity test and are all frontal portraits with the same smile is a worse dataset than twenty-five that span the checklist. Coverage is the variable, not count.

The coverage checklist

Every dimension below should appear at least twice, and the marked ones are mandatory.

The 3/4 angles are the most common cause of production drift and the most common omission in first datasets. Frontal images are the easiest to generate and the hardest to generalize from. If your dataset is all frontal, the model has no information about how this face behaves when it turns, so every non-frontal generation is a fresh sample — which is exactly the drift 4.1 described.

One full-body shot is the minimum for proportions. Without it the model has seen your persona's face but not their body, and head-to-body ratio becomes a random variable across the feed.

The hard rules of selection

No occluded faces. No hands on cheeks, no hair over an eye, no sunglasses in the core set, no scarves at the jawline. The model cannot learn what it cannot see, so occluded regions get filled from the base model's general priors — and that is precisely where the face changes. If the persona wears glasses, run a separate accessory subset, or accept that they will always be wearing them.

Consistent skin rendering, not consistent lighting. Varied lighting is desirable; wildly varied color temperature is not. The LoRA learns a skin tone, and if half the dataset is warm indoor tungsten and half is blue overcast, that tone becomes scene-dependent. Keep white balance consistent and use soft, diffuse sources for most of the set, with one or two hard-light frames for coverage.

Native resolution, never upscaled. An upscaler invents high-frequency texture — pores, hair strands, fabric weave — that was not in the source. A LoRA trained on upscaled inputs learns the upscaler's idea of skin as part of the identity, and the result has a characteristic hardness that no amount of prompt work removes. [UNVERIFIED] Worse, the invented detail is inconsistent between images, so the model sees a face that changes texture from frame to frame. Generate at the training resolution instead of upscaling to it.

Sharp, not processed. No aggressive denoising, no beauty filtering, no heavy sharpening. Skips and over-processing are both artifacts the LoRA will faithfully memorize.

The clothing trap

This is the trap that costs people a weekend, and it is invisible until after training.

If the same jacket appears in twenty of twenty-five images, the model has no way to know that the jacket is not part of the person. The association gets baked into the trigger. The result: your persona's face changes when you change their clothes, which is exactly backwards, and it is maddening to diagnose because the LoRA looks excellent in testing — on the same jacket.

Three defenses, use all three:

• Wardrobe variety in the dataset. Four or five distinct outfits, including at least a couple of plain, unbranded ones.
• Caption the clothing. Chapter 4.5 covers captioning; the relevant rule here is that what you name in the caption is what the model can separate from the trigger.
• A neutral baseline. Some images with plain backgrounds and simple clothing, so the face has a canonical representation independent of scene.

Preparation

The last stage before training, and the one where a small mistake wastes a multi-hour run.

Crop deliberately. Square or the trainer's expected aspect ratio, framing the face consistently — roughly the same head-to-frame ratio across the set. Wildly varying crop scales teach the model that scale is part of the identity.

Match the training resolution. Whatever the trainer expects. Feeding 4K images to a trainer that works at 1024 wastes time and risks resampling artifacts.

Consistent color treatment across the set. One color pipeline, applied to all of them. Mixed grades become part of the face.

Simple, sequential filenames. Several trainers expect a numbered sequence in a folder named for the concept. Nothing about the filename needs to be clever; it needs to sort.

Separate the concepts if the trainer supports it. Many trainers let you place body shots in their own subfolder with a repeat count. This lets proportions train distinctly from the face, and it is the correct structure for a persona, whose body and face are both identity. [UNTESTED]

Hold out five images. Set aside five of your best that will never be used in training. This is your test set for 4.6, and it is the difference between measuring generalization and measuring memorization. A LoRA tested on its own training images will score well and tell you nothing.

Where the candidates come from

Three sources, and a good dataset draws on all three.

The adapter's output. Once you have a reference plate, the training-free adapter from 4.3 can generate candidates in volume. This is the fastest source and the one with a known ceiling: every candidate inherits the adapter's characteristic softness and its expression bias. Select against that, and do not let the set become a monoculture of one adapter's look.

Variations on the plate. Take the reference plate itself through an instruction-based edit model and vary one dimension at a time — turn the head, change the light, change the outfit. This produces the most on-identity images and the least natural variation, because every frame traces back to the same source pixels.

Production picks. The strongest source, and the slowest. As you publish, keep the frames that looked most like the persona and add them to a growing reserve. Over a few months this gives you a set from your real pipeline, which is the distribution you actually care about. Chapter 4.6's register is where these get flagged.

The reserve is the reason the dataset is never finished. A persona in month six has a better dataset available than it had in month one, and retraining is a normal maintenance operation rather than a one-off.

Consent, provenance, and the model release you may not have

One administrative step that belongs with the dataset rather than after it.

If your persona is entirely generated — the face came from a model and was never a specific real person — record that fact and the generation history. That is a provenance record, and chapter 11.5 wants it for disclosure purposes.

If you seeded the persona from a real person's photograph, a stock image, or a hired model, the situation is different and chapter 11.4 governs it. The consent documentation travels with the dataset, in the same folder, in the same version control. Two reasons: a training run is irreversible, so the record of what it was trained on must exist before the run and not after it; and a dataset without provenance is a dataset you cannot use commercially with confidence, no matter how good the LoRA turns out.

Store it alongside the images. Chapter 3.6's folder tree already has the place for it.

Where this goes wrong

Generating the whole set from three prompt templates. You have built 25 near-duplicates and will train a LoRA that can make exactly one picture.

No 3/4 angles. The single largest cause of production drift, and the easiest omission to make.

Occluded faces, sunglasses, hands on cheeks. The model fills the gap with someone else's face.

Upscaled inputs. You have taught the LoRA your upscaler's hallucinated skin.

One outfit. The clothing trap, and it will present as a mysterious identity failure.

Testing on training images. Memorization is not identity.

Keeping everything that passed the identity filter. Coverage is the criterion, and redundancy is a cost.

Chasing 100 images. Past roughly 50, you are adding work, not information.

The gate

Fill the coverage table with counts from your own dataset. Every mandatory row needs at least two images, the 3/4 rows need four, and there must be at least one full-body frame.

Then answer:

1. How many candidates did you generate, and how many survived?
2. Which mandatory coverage rows are still empty?
3. How many distinct outfits are represented?
4. Are there any occluded faces left in the set?
5. Which five images are held out and never trained on?

What you leave with

The reframing that a synthetic persona's dataset is designed rather than collected, the honest quantity range and the two-score selection workflow, a coverage checklist with the 3/4 angles flagged as mandatory, four hard selection rules, the clothing trap and its three defenses, a preparation routine, and a held-out set that makes chapter 4.6's quality test mean something.