Reusing a Validator Flips What Failure Means in Your Pipeline

A validator that has run in production for months is the most tempting code to reuse. It is tested, it is tuned, and it already knows the shape of your data. So when you need to audit records that were written before the validator existed, pointing it at the old rows looks like a one-line job: same function, new input set. That instinct is right about the logic and wrong about the consequences. A check that returns "not valid" means "do not write this" on the generation path and "delete what is already there" on the audit path, and the code cannot tell the difference. This post is about the specific failure that produces, and the three-way judgment that fixes it.

The asymmetry hiding in a shared checker

Consider a pipeline that generates translated terms and validates each one before storing it. The validator runs three stages: an exact-match shortcut, a cheap embedding-distance test, and for the ambiguous middle band, a language model that answers yes or no. Anything that fails, at any stage, is simply not written. The caller moves on and retries tomorrow.

Now look at the error handling inside that validator:

for _, r := range gray {
    ok, err := e.verifyGray(ctx, r.ko, r.backKo)
    if err != nil {
        r.verdict = DropBacktranslate  // treat an API error as a failed check
        continue
    }
    if ok {
        r.verdict = VerdictPass
    } else {
        r.verdict = DropBacktranslate
    }
}

On the generation path this is correct and even elegant. A timeout, a rate limit, a malformed response: all of them collapse into "we did not confirm this, so we will not write it." Fail-closed. The cost of a false negative is one skipped row and one retry tomorrow.

Point the same function at existing confirmed records and the cost inverts. A single API timeout now marks a correct, human-approved row for deletion. The validator did not change. The blast radius did, because the caller turned a decision not to write into a decision to destroy.

The trap is that this asymmetry is invisible at the call site. The audit code reads as innocently as the generation code:

verdicts, err := engine.VerifyPairs(ctx, pairs)
for _, v := range verdicts {
    if !v.Pass {
        rejectRecord(ctx, v.ID)  // looks symmetric, is not
    }
}

What the return value cannot tell you

The deeper problem is that a boolean pass flag compresses two different worlds into one. "The model examined this pair and said they do not match" and "we never got an answer" both arrive as Pass: false. On the generation path that compression is harmless, since both outcomes lead to the same safe action. On the audit path you desperately need to separate them, and the type signature will not let you.

The fix is not to make the validator smarter. It is to stop asking the validator for a verdict and start asking it for evidence. Most validators already return more than the boolean: a distance, a confidence, an intermediate artifact. Route your destructive decision through those instead.

In our case the return type carried a distance alongside the flag:

type PairVerdict struct {
    Pass   bool
    BackKo string   // the round-trip translation, empty if we never got one
    Dist   float64  // embedding distance, 0 when the shortcut matched
}

That distance is measured, not judged. An API error cannot fabricate a large distance, because a failed call never produces one at all. So a threshold on distance is a claim about the data, while the boolean is a claim about the process.

The three-way judgment

Auditing needs three outcomes, not two, and the third one is the whole point:

• Delete: hard evidence of a mismatch. In our pipeline that is a distance above the upper band, well past the region where legitimate synonyms live.
• Keep: the validator affirmatively passed the record, including cases where the language model looked at an ambiguous pair and approved it.
• Hold: everything else. The check could not complete, the round trip came back empty, or the model said no inside the ambiguous band. Record the finding, leave the data alone, and put it in a human queue.

The hold bucket is what makes reuse safe. It converts "I am not sure" from a destructive action into a work item. It also converts your audit from a cleanup job into a triage job, which is what an audit should have been all along.

There is a subtle ordering bug waiting in this design, and it is worth spelling out because we hit it. Our distance field is zero in two completely different situations: when the exact-match shortcut fires (a perfect pass) and when the round trip came back empty (a total failure to evaluate). A two-branch rule written on distance alone would file the empty round trip as a pass, write a pass record, and thereby exclude that row from every future audit through the idempotency filter. The row would be marked verified forever without ever having been verified once. Checking the emptiness of the round-trip artifact before touching the distance costs one line and closes that hole.

Numbers from a dry run

We built the audit with the three-way rule and ran it in dry-run mode over 300 legacy records before touching anything. The distance histogram was the interesting part. Every record fell below the upper threshold. Nothing qualified for deletion.

More telling was the middle band. A term meaning "dermal filler injection" had translations in four languages that all round-tripped to "hyaluronic acid injection," landing at a distance of 0.36. Those are correct translations. The round trip described the substance instead of the procedure, which is exactly the kind of drift a language model marks as "not the same." Under a two-way rule those four rows would have been deleted, in four languages, for being right.

That single observation justified the whole design. It also produced a second finding we did not expect: our zero-deletion result was not proof that the data is clean. It was partly an artifact of the population filter. The audit only looked at records whose parent entry was in a confirmed state, and our most famous misalignment example, a term whose translations had drifted into a paragraph from a doctor's biography, lived under an unconfirmed parent. The audit could not have found it. A validator can only be as good as the rows you hand it.

Rules for moving a validator to a destructive path

• Ask what a failure costs on each path. If one path pays a retry and the other pays deleted data, you are not reusing a validator, you are giving it a new job.
• Route destruction through measurements, not verdicts. Distances, counts, and diffs cannot be fabricated by an infrastructure failure. Booleans can.
• Add a hold bucket before you add the delete. If your audit has no third outcome, it will express uncertainty as damage.
• Check the artifact, not just the score. An empty intermediate result often shares a numeric value with a perfect one. Test that case explicitly.
• Watch for idempotency traps. If you record "checked" to avoid re-processing, a misfiled pass is permanent. The row is marked verified and will never be looked at again.
• Dry-run over a real slice first, and look at the distribution. A zero-deletion result is information: either the data is clean, your threshold is wrong, or your population filter is excluding the exact rows you were hunting.
• Preserve the original before overwriting it. Ours went into an existing unused column on the log table. Rollback insurance is usually cheaper than it looks.

The general shape

This is not really about validators. It is about the fact that a function's safety properties are not intrinsic to the function. They live in the pair of the function and its caller. Fail-closed is a property of "what happens when this check cannot complete," and what happens is decided entirely by the code on the other side of the return statement.

The next time you point mature validation logic at a new dataset, the question to ask is not whether the logic still applies. It is whether the meaning of "no" still applies. If saying no used to mean "wait," and now it means "destroy," you need a third word.