LLM Output Variance: When a Single Targeted Retry Pays Off

Retrying an LLM call that already returned a valid answer sounds wasteful. Most of the time it is. But we measured something in a production translation pipeline that changed our mind: when we regenerated 55 accepted outputs with the same model, the same prompt, and the same input, 25 of them came back different. That is a 45% variance rate on supposedly settled answers. This article is about the narrow case where that variance becomes a resource: a one-shot, condition-gated retry that recovers real quality, and the two failure modes (result clobbering and budget exhaustion) you must design around before turning it on.

The measurement: 45% of outputs changed on identical input

The pipeline in question fills a multilingual terminology glossary. A batch generator asks two LLMs for translations of medical-aesthetics terms, a consensus step compares them, and a back-translation check verifies meaning. Outputs that pass all gates get stored as confirmed entries.

During an audit we deleted 55 confirmed Chinese entries and ran the exact same generation again. No prompt change, no model change, no temperature change. The second run agreed with the first on only 30 entries. The other 25 came back with a different surface form.

The differences were not random noise. They split into three groups:

• Formatting drift: capitalization, spacing, and synonym-level swaps that any normalizer would treat as equal.
• Real improvements: the second run produced the established local market name for a device where the first run had left the English brand name untouched.
• Real regressions: the second run dropped a correct local name and fell back to the English brand name.

The second and third groups are the interesting ones. They are the same phenomenon pointing in opposite directions. The model does not reliably know the local market name for long-tail brand terms. It sometimes produces it and sometimes does not, with meaningful probability on both sides. A single sample from that distribution is not the model's best answer. It is just one draw.

Once you see generation as a draw from a distribution, the retry question changes shape. The question is no longer "should I trust the model" but "for which outputs is a second draw likely to beat the first, and how do I recognize them cheaply?"

Retry only on a checkable property violation

Retrying everything doubles your cost for near-zero gain: for the 30 stable entries a second draw returns the same thing, and for the drifting entries you have no way to know which draw was better. A blind retry converts variance into churn.

The escape is to find a property that is:

1. Deterministically checkable without another LLM call.
2. Strongly correlated with "the second draw is likely better."
3. Cheap enough to test on every output.

In our case that property was writing-system mismatch. Chinese glossary entries should normally contain Han characters. The Chinese aesthetics market coins local names even for imported brand devices, so a Chinese slot holding a Latin-only string usually means "the model did not retrieve the local name on this draw," not "no local name exists." A pure function can test this: iterate the code points, count Han versus Latin letters, no network involved.

So the retry gate became: after the full validation chain passes, if the accepted output for a Chinese slot is Latin-only, regenerate that one slot once, with one added instruction: "prefer the established local-market Han name if one exists; keep the original name for brands without one."

Two details matter here. First, the retry reuses the same generation, consensus, and verification pipeline as the original attempt. A retry that skips validation is not a retry, it is a bypass. Second, the added instruction is appended to the base prompt rather than replacing it, so the retry differs from the first draw in exactly one way. You want the delta to be attributable.

The results on live data: a filler-brand slot that first came back as the raw English brand name returned with the correct Han market name on retry. Slots that stayed Latin after the retry were overwhelmingly minor brands where no established local name exists, which is exactly the population you want left over. Post-run, about 11% of newly filled Chinese slots remained Latin-only, and those carry a review flag instead of silently passing.

Never overwrite a passing result with a failure

The first failure mode of any retry design is clobbering. Your original output passed every gate. The retry might not. If you write the retry result unconditionally, a passing entry can be replaced by a rejection, and a slot that was filled becomes empty. That is strictly worse than not retrying.

The contract we ended up with:

snapshot = copy(result)          // everything the pipeline may mutate
reset(result)                    // back to undecided state
regenerate with reinforced prompt
run consensus + verification     // same gates as the original
adopt only if:
    verdict == pass
    AND output now satisfies the property (contains Han)
    AND output does not collide with an existing confirmed entry
otherwise:
    restore(snapshot)            // the passing original wins

Each condition earns its place. The pass check keeps validation authority where it belongs. The property check rejects a retry that returned Latin again, because adopting it would spend the budget without fixing anything. The collision check matters because a new Han name can duplicate a different concept's confirmed translation; adopting it would trip the pipeline's duplicate gate later and take the whole slot down with it. Checking before adoption means a colliding retry quietly restores the original instead of destroying a valid entry.

The snapshot has to cover every field the pipeline mutates, not just the output text. Ours carries seven fields including the verdict, the back-translation, and the similarity distance. Missing one means the restored entry is a chimera of two runs.

The hidden cost: retry budgets and permanent exhaustion

The second failure mode is subtler and nearly shipped in our first implementation. Batch pipelines usually cap attempts per slot. Ours allows three: after three failed judgments, a slot leaves the population permanently so a poisoned input cannot burn LLM spend every night.

The attempt counter is derived from audit-log rows. Our first design logged the retry as its own row, which reads as one extra attempt per night. A slot that gets retried nightly would hit the three-attempt ceiling in two nights and drop out of the fill population forever. The feature meant to fill more slots would have been quietly deleting them.

The fix was to make the retry invisible to the budget. The retry writes no separate audit row. Instead it appends a marker to the final row's summary field, recording the original output alongside. That one marker does double duty:

• Budget neutrality: attempt counting sees one row per night, the same as before.
• Once-only enforcement: before retrying, the pipeline reads the latest row for the slot. Marker present means this slot already had its retry in the current judgment cycle, so skip.

We scoped once-only to the judgment cycle rather than forever. If a reviewer later rejects the entry and the slot re-enters the population, it gets a fresh retry along with its fresh attempts. Permanent once-only would mean a slot judged years ago could never benefit again, which serves nobody.

If you take one thing from this section: before adding any retry to a batch system, find every consumer of your attempt or cost accounting and check what your retry looks like to each of them. Retries interact with budgets, dedup checks, exhaustion predicates, and monitoring. None of those consumers know your extra call was "just a retry" unless you make it so explicitly.

Where this pattern applies, and where it does not

The pattern generalizes to any generation task with a cheap, decisive property check:

• Structured output that must parse (JSON schema, date formats): parse failure is the property.
• Translations that must be in the target script: writing-system check, as here.
• Code generation that must compile: the compiler is the property check.
• Constrained rewriting (length limits, banned terms): a scanner is the property check.

It does not apply when quality is a matter of degree with no deterministic test. If you cannot decide "violated or not" with a pure function, you are back to LLM-judges-LLM, which is a different tool with different costs. It also loses value when variance is low: we saw 45% on long-tail terminology, but on well-known terms the same pipeline was nearly deterministic, and a retry there is pure waste. Measure your variance on a deleted-and-regenerated sample before assuming a retry will find anything new.

One more honest limit: a single retry samples the distribution twice. For our workload that recovered most of the recoverable cases, because the property check told us exactly which slots were worth a second draw. If your recoverable set needs five draws, the economics change, and you should probably fix the prompt instead of sampling harder.

FAQ

Why not raise temperature or use a different model for the retry instead? Changing two variables at once makes the outcome unattributable. Our retry changes only one thing, an appended instruction targeting the known failure, so an improvement can be credited and kept. Swapping models also breaks the guarantee that retry outputs face the same validation the original faced.

Why retry only once instead of until the property holds? Because "no established local name exists" is a legitimate terminal state. For minor brands the correct answer is the Latin name, and a loop would either spin forever or need a separate stop condition that is harder to reason about than "one shot, then flag for review."

Is 45% variance normal for LLMs? It depends heavily on how close the task is to the edge of the model's knowledge. Our well-known terms were stable across runs; the variance concentrated in long-tail entries where the model is genuinely uncertain. That concentration is exactly why a targeted retry works: the property check finds the uncertain slots.

Does this replace human review? No. It shrinks the review queue. Outputs that stay in violation after the retry get flagged, and the flag carries the original output so a reviewer sees both draws. The pattern moves effort from "review everything" to "review what the machine could not fix with one more try."