3.5 Benchmarking Your Own Workshop
One afternoon of measurement gives you three things you will use for the life of the project: a realistic publishing cadence, a defensible cost per post, and the ability to tell a regression from a preference.
Without a baseline, every slowdown is either invisible or alarming, and you cannot tell which. Ten minutes per test, four tests, recorded in one table.
The protocol, which matters more than the tests
Every test below is worthless if the conditions drift. Fix them:
- Same prompt, same The random starting point; the same seed with the same settings reproduces the same image., same settings across every run you intend to compare.
- Same resolution and same How many denoising iterations run; full-quality bases converge near 20–30, distilled models near 4–8..
- Record the peak VRAM, not the average. Peaks are what crash a run forty minutes in.
- Record the environment — driver version, PyTorch version, model file and precision.
- Run each test twice and keep the second number. The first run of a model spends time loading weights into memory; the first number is a measurement of your disk.
The rule for reading your table: a change in one field with everything else fixed is information. Anything else is noise.
Test 1: the baseline image
One 1024-pixel image, 20 steps, one generation, fresh process.
Record the wall-clock seconds and the peak VRAM. This is the number everything else is measured against, and it converts directly into capacity: if the render takes 40 seconds and you need 20 published images a month with an average of five attempts each, you are looking at roughly an hour of pure render time — plus the The largest total line in the pipeline cost table, and the line operators omit., which is the larger term.
Run it once more on a distilled model at six steps. The difference tells you what your drafting tier buys you.
Test 2: the drift test, and the only test that changes a decision
This is the most valuable measurement in the chapter, and almost nobody runs it.
Take one identity reference — your persona's face, at the quality you intend to publish. Run the same edit instruction against it on every candidate model in your chosen family:
same person, red jacket, studio light, facing camera, neutral expression
Three models minimum. Same reference image, same instruction, same output resolution.
Now score the identity hold. Use a simple scale, and be honest about the middle:
| Score | Meaning |
|---|---|
| 5 | Indistinguishable from the reference to someone who knows the face |
| 4 | Same person, minor variation in jawline or eye shape |
| 3 | Could be a sibling; a follower would question it in a feed |
| 2 | Same demographic, different person |
| 1 | Unrelated |
The model that scores highest is your production model for anything involving the face, whatever the general rankings say. This test costs twenty minutes and prevents months of fighting an identity that a different One saved state of a model or adapter, identified by a hash and recorded as provenance. would have held.
Re-run it whenever you change the The trained weights a persona's look comes from; its license governs any adapter trained on it., the adapter version, or the identity artifact. Drift does not announce itself; it accumulates across a content calendar.
Test 3: video throughput
A 6-second image-to-video render from a single still.
The reference point: a 4090-class card with 24 GB of VRAM and about 90 GB of system RAM completes this in one to two minutes. [UNVERIFIED] Treat it as the benchmark to beat or to accept, not as a guarantee — your number depends on resolution, motion complexity, and the model.
Two things to record beyond the time:
- VRAM peak. Video models spike harder than image models, and a peak that fits at one resolution will not fit at the next one up.
- The failed-attempt rate. If three of five renders come back unusable, your effective time per usable A CLIP-style text encoder that responds best to comma-separated fragments; used by older model families. is triple the wall-clock, and that number belongs in chapter 0.4's cost formula.
Test 4: the mini training run
A A small low-rank adapter, tens to a couple hundred megabytes, that nudges a base toward one face. training run on ten images, at the smallest sensible configuration. The goal is not a good LoRA. The goal is to prove the pipeline works end to end before you invest a curated dataset and seven hours.
Record: did it complete, how long did it take, what was the peak VRAM, and what did the output look like. A run that completes and produces a weak LoRA is a success. A run that throws an error at step zero is a much better discovery now than after you have curated thirty images.
If your plan is hosted training, run this test there instead. The point is to learn the shape of the run, and the shape is the same everywhere.
The results table
Fill this in once, date it, and keep it. This is the zero point.
| Test | Machine / model | Time | VRAM peak | Notes |
|---|---|---|---|---|
| Baseline image, 20 steps, 1024px | ||||
| Baseline image, distilled, 6 steps | ||||
| Drift: model A | identity score | |||
| Drift: model B | identity score | |||
| Drift: model C | identity score | |||
| 6 s image-to-video | failed-attempt rate | |||
| Mini LoRA, 10 images | completed? |
One caveat about the drift scores: they are yours, not universal. A model that scores 4 in your test may score 2 in someone else's, because identity hold depends on the reference quality, the prompt, and the adapter version as much as on the checkpoint. That is the argument for running the test rather than reading a comparison — and for re-running it when any of those three change.
Turning the numbers into money
Two derivations, both feeding chapter 0.4:
Time per published asset = (render time × average attempts) + (human time). The human term is almost always larger, and it is the one people leave out of their cost accounting.
Money cost per published asset = (render time × hourly machine cost) + (human time × your hourly value) + any API charges. For a local render, the machine cost is electricity and depreciation; for a rented card, it is the hourly rate; for an API, it is per call.
Re-run the whole suite when any of these happen: new hardware, a major driver or framework update, a new base model, or a change of host. Otherwise, once a quarter is enough to catch slow drift from environment changes.
Where this goes wrong
Benchmarking once and never again. The table's value is in the comparison, which requires a second measurement.
Using a different prompt per run. You have measured your own inconsistency.
Recording the average VRAM instead of the peak. The peak is what fails.
Not recording the environment version. A result without its configuration cannot be reproduced or explained.
Skipping the drift test. It is the only one that changes which model you produce with.
Treating the reference numbers in this chapter as your numbers. They are a sanity check. Yours is the one that goes in the cost formula.
Benchmarking at a resolution you do not publish at. Measure the workflow you run.
The gate
- Do you have a dated results table with all four tests complete?
- Which model won the drift test, and is it the one you are producing with?
- What is your time per published asset, attempts included?
- What is your money cost per published asset?
- Where is the environment configuration recorded alongside the numbers?
What you leave with
A fixed protocol, four tests, an identity-drift score that decides your production model, a zero-point table to compare against in six months, and the two derivations that convert render times into the only two numbers that matter: time per published asset, and money cost per published asset.