A Non-Answer Is Not a Disagreement in LLM Consensus Gates

A two-model consensus gate is a simple idea: ask two independent LLMs the same question, act only when they agree, and route everything else to a human. It works, and I have written before about why consensus beats a single model for decisions that must be right. This post is about a failure mode inside the pattern itself. In a pipeline that classifies scraped terminology candidates, our disagreement rate climbed from 18 percent to 68 percent over a few weeks, and a growing pile of items got permanently kicked out of automation as "the models cannot agree." A large share of those items had never been disagreed about at all. One of the models simply had not answered, and the code counted silence as dissent.

A consensus gate has four outcomes, not two

The naive mental model of a consensus gate has two results: the models agree, or they do not. The real state space is larger, and the extra states are exactly where reliability leaks out:

1. Both models answered, same verdict. Safe to act.
2. Both models answered, different verdicts. A genuine disagreement, worth a retry or a human look.
3. One model answered, the other did not. You have one opinion, not two. Nothing was compared.
4. Neither model answered. A full outage for this batch.

States 2 and 3 feel similar when you are writing the code, because in both cases you cannot proceed. But they mean opposite things. A disagreement is a signal about the data: the item is genuinely ambiguous, and repeating the question is unlikely to converge. A non-answer is a signal about the infrastructure: a timeout, a rate limit, a content filter, a partial parse. The item itself might be trivially easy.

Most implementations encode state 4, because a total outage is loud and obvious. State 3 is the one that gets silently folded into state 2, and that fold is the bug.

How a non-answer becomes a disagreement in code

The collapse usually happens at a map lookup. Each model returns a batch of judgments, you index them by item, and then you compare:

gv, gok := geminiVerdicts[item]
ov, ook := openaiVerdicts[item]
if !gok || !ook {
    return Judgment{Agreed: false} // missing answer, "not agreed"
}
if gv != ov {
    return Judgment{Agreed: false} // real disagreement
}

Both paths produce the same struct. Downstream, Agreed == false is recorded as drop-disagree, a retry counter goes up, and after three failed attempts the item is marked exhausted and removed from the automated population. The system cannot tell you, even in its logs, whether an exhausted item was three times ambiguous or three times unlucky.

Two details make this worse than it looks. First, partial parse loss is common. A model asked to judge a batch of 15 items sometimes returns 14, with no error anywhere. The missing item lands in state 3 through no fault of its own. Second, one-sided failure is correlated in time. When a provider has a bad night, an entire batch falls into state 3 at once, and every item in it burns a retry.

The retry budget is the real casualty

Our gate gave each item three attempts. Three genuine disagreements is a reasonable definition of "the models will not converge on this, a human should look." The retry budget is a filter for ambiguity.

Counting non-answers against that budget quietly changes what the filter selects for. An item that hit two flaky nights and one real disagreement is exhausted with the same standing as an item that was ambiguous three times. Worse, a sustained one-sided outage converts infrastructure failure directly into data verdicts: every item processed during the outage marches one step toward exhaustion, and after three such nights the pipeline has "concluded" something about hundreds of items it never actually evaluated.

That is what our numbers showed once we could finally see them. Retrying items labeled as disagreements changed the outcome about a third of the time (of items retried after a first "disagreement", 181 later resolved to an agreed rejection and 139 to an agreed acceptance, while 577 stayed in disagreement). Some of that instability was genuine model wobble on borderline inputs. But an unknown share of it was state 3 wearing state 2's name, and before the fix the log format made the two indistinguishable: the code stored a fixed explanatory string, not what each model had said.

The fix: record the non-answer, do not count it

The change is small and has three parts.

First, widen the judgment struct so the truth survives past the comparison:

type Judgment struct {
    Agreed    bool
    Oneside   bool   // at least one model gave no verdict
    GeminiOut string // raw verdict, "" means no answer
    OpenAIOut string
}

Populate the per-model fields before any early return. The natural place to short-circuit on a missing answer is also the place where that information is about to be lost, so the ordering matters: fill first, then return.

Second, give the non-answer its own recorded verdict, and keep it out of the retry count. Our exhaustion rule was max(attempt) >= 3 over the item's log rows. Writing the non-answer row with attempt = 0 makes it visible to every audit query while contributing nothing to exhaustion. The next real attempt still computes its number as max(attempt) + 1, so the sequence of genuine attempts is unbroken.

Third, define one-sided broadly. Oneside should be true when either model is missing, including when both are, as long as the batch as a whole came back. A batch where both models dropped the same item is still a non-comparison, not a disagreement. Reserve the total-outage path for the case where neither model returned anything, and skip logging entirely there, because a dead night should not leave per-item traces at all.

The poison pill, and a three-part escalation guard

Excluding non-answers from exhaustion creates a new hazard. If some specific input reliably makes one provider go silent (a content filter is the classic cause), that item now retries forever. It re-enters the population every night, burns two LLM calls, and never resolves. The queue has grown a poison pill.

The fix for the fix is to escalate persistent one-sided failures back into the counted path, with guards so that escalation cannot fire during an ordinary outage:

• Count non-answer rows inside a sliding window, not over all time. We used 14 days. An item that collected one flaky night per month for a year is not a poison pill, and a lifetime count would eventually escalate it anyway.
• Before escalating, check the batch. If most of the batch is one-sided tonight, that is a provider incident, not an item property. Suppress escalation for the whole batch and record plain non-answers. Suppression errs in the safe direction: the worst case is one more night of retries.
• When you do escalate, keep the recorded verdict identical to a genuine disagreement so every downstream consumer (exhaustion queries, dashboards, drain estimates) behaves exactly as before, and put a fixed marker string in a separate reason column. A human reading the item later must be able to see "this left automation because a provider kept failing on it," because the correct response to that is different from the correct response to real ambiguity.

With a three-strike escalation on top of a three-attempt budget, the worst-case lifetime of a poison pill is bounded at six nights, and none of that bound is spent lying about what happened.

What changes once you can see the difference

The observable win is immediate: the nightly stats split into "disagreed" and "one-sided," and for the first time the pipeline can answer whether a bad week was a model-quality problem or an infrastructure problem. Before the change, that question was not merely hard, it was unanswerable from the data, because the log rows stored a constant string where the evidence should have been. Recording each model's raw verdict costs two columns and removes an entire class of guessing.

There is also a subtle accounting rule worth stating: report statistics by what was actually recorded, not by what the classifier saw. An item that arrived as one-sided but was escalated into the counted path belongs in the "disagreement" column of tonight's report, because that is what the database now says about it. If the tally and the stored rows drift apart, every future debugging session starts with reconciling them.

Checklist for your own gate

• Enumerate the four outcomes explicitly in code: agree, differ, one-sided, outage. If two of them share a struct value, you have already merged them.
• Populate per-model raw verdicts before any early return, and store them. "" for a missing answer is fine; a constant explanatory string is not.
• Keep non-answers out of the exhaustion rule. An attempt = 0 row under a max(attempt) rule is a cheap way to record without counting.
• Treat "both models missed this item inside an otherwise healthy batch" as one-sided, not as disagreement and not as outage.
• Bound the poison pill: escalate after N one-sided failures within a sliding window, suppress escalation when the whole batch is failing, and mark escalated rows with a distinct reason.
• Make the tally follow the record. Statistics should be computed from what was written, not from the branch that was taken.

None of this is specific to LLMs. Any voting system with unreliable voters has the same three-way distinction between dissent, absence, and blackout. LLM pipelines just make it easy to miss, because both a timeout and a disagreement arrive as the same thing: a comparison you cannot complete tonight.