3.2 Concepts You Actually Need
You do not need to understand diffusion mathematics to run a persona business. You do need enough vocabulary to stop guessing, because almost every consistency problem looks like a mystery until you can name the variable that caused it.
This chapter is the minimum set. Everything here is used in chapter 4, where the face is actually solved. Nothing here is theory for its own sake — each concept comes with the decision it changes.
The four sampling settings
These four appear in every interface, and they are the four variables that quietly break reproducibility.
The random starting point; the same seed with the same settings reproduces the same image.. The random starting point. Same seed plus the same prompt, model, and settings produces the same image. That property is what makes it useful: to test one change at a time, lock the seed and vary only the thing you are testing. Without that discipline you cannot tell whether a better image came from your prompt edit or from the dice.
Two cautions. A seed does not transfer between models, and often not between samplers — treating a seed as a portable identity is a category error. And a locked seed is a debugging tool, not a production setting; a content calendar of subtly identical images is its own failure.
Steps. How many denoising iterations the The algorithm walking from noise to image, such as Euler or a DPM++ variant. runs. More is not better indefinitely. A typical modern The trained weights a persona's look comes from; its license governs any adapter trained on it. converges somewhere around 20 to 30 steps, and past that you are paying render time for imperceptible change. Distilled models — schneller variants, Lightning, Turbo, and Klein-class weights — are trained to converge in roughly 4 to 8 steps, and running them at 30 produces nothing but a longer wait.
Classifier-free guidance: how hard the model is pushed toward the prompt; high values over-bake an identity LoRA.. Classifier-free guidance: how hard the model is pushed toward the prompt. Low values give it freedom; high values obey the text and, past a point, produce burned contrast, halos, and the over-saturated look that reads as cheap. Typical working ranges are roughly 4 to 7 for full-quality bases, and distilled models often run at 1.0 because guidance is baked into training. [AS OF 2026-10]
The reason CFG matters for this book specifically: a A small low-rank adapter, tens to a couple hundred megabytes, that nudges a base toward one face. applied at high CFG over-bakes. Identity LoRAs tend to produce a harder, more plastic face as guidance rises. If your persona looks waxy, the fix is frequently to lower CFG before you retrain anything.
Sampler and The noise schedule the sampler follows, such as normal or karras.. The sampler is the algorithm that walks from noise to image (Euler, DPM++ variants, and others). The scheduler is the noise schedule it follows (normal, karras, and others). They change texture, convergence speed, and fine detail.
The practical rule that follows from all four: freeze them. A persona's look is partly an artifact of its sampling configuration. Pick a preset, write it into the persona's specification, and vary only the prompt and the composition. Chapter 3.5 benchmarks that preset so you know what it costs in time.
Latent, VAE, and text encoders
Three components you will meet in error messages before you meet them in documentation.
The compressed representation the model denoises in, which is why img2img is approximate and inpainting is soft-edged. is the compressed representation the model works in. A 1024-pixel image becomes a much smaller grid of numbers, the model denoises in that space, and the result is decoded back to pixels. Everything about why Denoising a whole image from your input at a strength value, restyling a shot while keeping its framing. is approximate, why Regenerating only a masked region; the surgical tool for a hand, an eye, or one bad detail. has soft edges, and why some operations cannot be undone follows from that one fact: you are editing a compression, not an image.
The The encoder-decoder pair between pixels and latent space; a mismatched one shows as color shift and washed shadows. is the encoder-decoder pair that moves between pixels and latent space. It matters for one practical reason: a mismatched or swapped VAE produces color shift, washed-out shadows, or blown highlights. If your output is inexplicably desaturated and you changed nothing else, suspect the VAE before you suspect the prompt.
Text encoders turn your prompt into something the model can condition on. Older families use A CLIP-style text encoder that responds best to comma-separated fragments; used by older model families.-style encoders, which respond best to comma-separated fragments. Newer families use a A long-context text encoder that handles full sentences and word relationships; write prose, not tag soup. that handles full sentences and preserves relationships between words.
The decision this changes: write in the dialect your family expects. Tag soup in a long-context encoder wastes its strength. Full prose in a CLIP encoder gets truncated and diluted. And because a LoRA trains against one encoder, prompting style is a property of the family you committed to in 3.1 — one more cost of switching.
Three ways to teach a model a face
These are the training options, ordered by weight.
LoRA (low-rank adapter). A small set of additional weights — typically tens to a couple hundred megabytes — that nudges the base model without changing it. Training on a rented GPU costs roughly $2 at the base price of a hosted trainer and takes roughly two to seven hours. [AS OF 2026-10] You need on the order of 15 to 30 well-chosen reference images of a person. [UNVERIFIED]
Its properties are why chapter 4.5 is a full chapter about it: the file is small and portable, it does not modify the base model, it can be combined with others, and it is the identity artifact you keep if a vendor disappears.
Fine-tuning more of the model on a handful of images: higher fidelity, gigabyte outputs, more overfitting.. Fine-tuning a larger portion of the model on a handful of images. Higher fidelity to the subject, output files measured in gigabytes, slower to train, and considerably more prone to overfitting the reference set — including its lighting and background. For persona work it has largely been displaced by LoRA, which reaches comparable identity fidelity at a fraction of the cost. [UNVERIFIED]
Full fine-tune. Retraining the model. Hundreds to thousands of images, serious compute, gigabyte checkpoints. This is a studio budget, not a solo operator's, and nothing in this book requires it.
The decision rule: LoRA is the default, and it is the artifact that matters. Chapters 4.4 and 4.5 are about doing it well.
Four ways to use a face without training
Identity adapters inject a reference face into the generation process rather than teaching the model anything. They are fast, they need no dataset, and they are the correct starting point — chapter 4.2 puts them first on the ladder for exactly that reason.
Conditions generation on reference images, transferring style and composition as well as face; unreliable as identity alone.. Conditions the diffusion process on one or more reference images. It transfers what the encoder sees, which includes style and composition as well as face. Useful for look and feel; unreliable as an identity mechanism on its own.
IPAdapter FaceID. Adds an identity embedding extracted by a face recognition model, which sharpens identity retention considerably compared with plain IPAdapter.
A face embedding plus a landmark path, so one reference holds identity as pose and scene change.. Uses a face embedding plus a landmark or keypoint The numbered folder convention such as 00-bible/ and 40-output/ that keeps every asset addressable., so a single reference image can hold identity while pose and scene change. Strong single-image identity retention; requires the InstantID stack installed alongside your base.
Injects an identity embedding with contrastive alignment designed to leave the base aesthetic untouched; no training.. Injects an identity embedding using a contrastive alignment designed to leave the base model's aesthetic untouched. Practical advantages: no training, fast, works with FLUX-class models. Practical limits: less stable across extreme poses, ages, and heavy stylization than a well-trained LoRA.
The shared shape of all four: something extracts a face embedding, and that embedding is injected into the conditioning. None of them are as controllable as a LoRA trained on your own dataset, and all of them get you a first result today. That trade is the ladder in 4.2.
Three ways to change an image
Different operations, different jobs, and they are routinely confused.
img2img. Denoises the whole image starting from your input, controlled by a strength value. Low strength preserves composition and changes little; high strength invents. This is the tool for restyling a shot while keeping the framing.
Inpainting. Regenerates only a masked region. This is the surgical tool: a hand, a background, a piece of jewelry, a single bad eye. It is also how you repair a face in an otherwise good composition, by masking the face and regenerating it with Instruction-based edit models and adapters that hold the subject while the scene changes..
Reference conditioning. Instruction-based edit models and adapters that hold the subject while the scene changes. This is the identity-preserving operation, and the one a persona pipeline should be built around.
The order they belong in: generate → reference-condition for variation → inpaint for defects. Running inpainting first and reference work second re-rolls the thing you repaired.
Why upscaling and skin detail go last
This is the sequencing rule people break most often, and the reason is structural.
Upscalers and face detailers operate on pixels, not on the latent representation. If you upscale and detail a face, then go back and regenerate the outfit with reference conditioning, the model re-renders the whole region at working resolution and the detail you added is discarded. You paid for it twice and kept it once.
The order that respects this:
- Composition and pose.
- Identity — reference conditioning, adapter, or LoRA.
- Defects — hands, eyes, jewelry, background artifacts, via inpainting.
- Upscale.
- Face detail, at low denoise strength.
- Color and grade.
Two cautions about step 5. Face detailers run at a denoise strength that re-renders part of the face; keep it low — roughly 0.2 to 0.35 — or you will replace the identity you have established. [UNTESTED] And over-detailing produces the smooth, poreless look that reads as synthetic at a glance, which costs you on the realism tiers described in 11.2. Judge face detail at 100% zoom, not in a thumbnail.
Where this goes wrong
Chasing settings instead of freezing them. Consistency is partly a configuration artifact. Lock the preset, then debug the prompt.
Varying two variables at once. Without a locked seed you cannot attribute the change, and you will conclude the wrong thing about your workflow for months.
Using a seed as an identity. It is a starting point, not a face.
Running a distilled model at 30 steps. Slower, identical output, and it makes benchmarking meaningless.
Blaming the prompt for a VAE mismatch. Desaturation with no prompt change is a component problem.
Detailing before you finish composing. Detail work is the last operation or it is wasted.
The gate
- Can you state your persona's sampler, scheduler, steps, and CFG from memory?
- Do you know which encoder family your base model uses, and is your prompt style matched to it?
- Which identity method are you using — adapter or LoRA — and why that one at your current volume?
- In what order does your pipeline run composition, identity, defects, upscale, and detail?
- At what zoom level do you approve a face?
What you leave with
Four sampling settings you will freeze, three components you will recognize in error messages, the training options ordered by cost, four adapters that need no training, three edit operations with distinct jobs, and one sequencing rule: detail last, always.