Scraped Translations vs LLM Consensus, a Blind Benchmark

Our domain glossary had two possible sources for non-native terminology: scrape the official multilingual pages that domain owners publish themselves, or generate translations with a two-model LLM consensus pipeline. Intuition said the official pages must be the ground truth. We measured it, and the intuition was wrong: in a blind judgment of every disagreement, the LLM output won 9 cases to 1. This post walks through the experiment design, the actual numbers, why scraped "official" translations lose, and how we retired the crawler without breaking a live service.

Two ways to source a multilingual glossary

A domain glossary maps a source-language concept (a procedure name, a product, a symptom) to its display form in each target language. Recall matters: when a term is missing, the downstream translator falls back to generic wording and loses the domain register.

The first pipeline crawled paired pages. For each site we registered the source-language URL and each translated URL, crawled both sides with the same budget, aligned blocks with embedding similarity, and promoted high-confidence anchor pairs directly into the glossary. This is attractive on paper because a site owner presumably reviewed their own translations.

The second pipeline never looks at the translated pages. It takes the source-language term, asks two different LLM providers to translate it independently (neither sees the other's output), and accepts the result only when the two answers agree after normalization, or sit within a small embedding distance. Survivors then pass a back-translation gate: translate the candidate back to the source language with a plain NMT service, and require either an exact normalized match or a cosine distance under a per-language threshold, with a gray zone that a third judgment call resolves.

The experiment: blind judges and identical conditions

Comparing pipelines honestly means removing every variable except the source. We sampled 136 glossary entries that the crawler had promoted as primary display forms, across the three languages where the crawl actually worked (60, 50, and 26 entries respectively). For each entry we regenerated a translation with the consensus pipeline under identical conditions: the same two generator models it uses in production, the exact production prompt, and the same batch size cap, so the comparison measured the source, not the setup.

Then we scored two things.

First, agreement rate: how often did the two-model consensus, on its own, reproduce the scraped form exactly after normalization (case folding, width normalization, whitespace and punctuation cleanup, script-level unification)?

Second, for every case where consensus produced something different from the scraped form, we ran a blind judgment. Each conflict became an A/B pair with the source hidden and the order shuffled deterministically, so a judge could not learn that "A means crawler". Two stronger models, neither of which took part in generation, judged independently with a rubric aimed at glossary fitness: is this the term a professional would actually use, free of marketing noise? We only counted a winner when both judges agreed.

What the numbers said

The headline results, in order of how much they surprised us:

Three quarters of the time, two independent models with no access to the official page converged on exactly the string the site owner had published. That alone says the consensus mechanism recovers "official" terminology without ever seeing it.

The disagreements are where the story flips. Of 19 conflicts, the blind judges unanimously preferred the crawled form exactly once. Nine times they preferred the LLM form, and the remaining nine were split or judged equally acceptable. The scraped side did not merely tie; it lost the contested ground.

Coverage settled the decision. The crawl pipeline only functions where a site actually publishes a translated page, which in our domain meant three languages with real volume and eight languages with almost nothing. The consensus pipeline generates for every language on the list. At the moment of measurement it had produced 16.6 times more accepted primary entries than years of crawling had.

Why official pages lose: the noise is structural

Reading the 19 conflicts one by one explained the loss. The crawled forms were not wrong translations. They were not translations at all, in many cases:

• A language code captured as a term: the glossary entry for "body shape" had the literal string "en" as its display form, scraped from a language switcher.
• Menu concatenation: two adjacent navigation items fused into one string, producing a plausible-looking compound term that does not exist.
• Promotion arithmetic: "300 shots" style quantity prefixes glued onto procedure names, because pricing tables sit next to headings.
• Parenthetical asides and ingredient suffixes that belong to marketing copy, not terminology.

The important part: this survived cleanup. We had already run a dedicated noise-purge pass over the crawled data months earlier and removed hundreds of contaminated rows. The examples above are what remained after that effort. Crawl noise is structural, because page layout keeps generating it, while a generation pipeline never emits a nav bar in the first place.

There is one honest caveat. The single case the crawl won was a locally preferred transliteration of a branded device, the kind of thing a marketing team decides and a model cannot deduce. If your glossary is dominated by brand transliterations with official local spellings, keep a manual override tier for exactly those entries. Ours has one, and the migration preserved it.

The consensus gate that replaced the crawler

The replacement pipeline is cheap to describe and cheap to run:

term -> generator A (model 1)   -> agree after normalization? -> accept
     -> generator B (model 2)   -> else: embedding distance <= 0.20? -> accept
accepted -> back-translate -> exact match OR cosine <= threshold(lang)
         -> gray zone (<= 0.50) -> single judge yes/no -> insert or drop

Design choices that mattered in practice:

• The two generators run independently and never see each other's output. Consensus by copying is not consensus.
• Normalization before comparison absorbs meaningless variance: width forms, case, whitespace, script-level equivalences. Without it, agreement rate drops and you pay for false disagreements.
• The back-translation gate catches confident nonsense. Two models can agree on a fluent but wrong rendering; a round trip through a plain NMT service exposes it, because the wrong rendering does not come back to the source term.
• Drops are recorded with an attempt counter. A slot that fails three times leaves the population, so the nightly batch does not burn budget re-arguing the same hard case forever. When we later upgraded the generator models, we selectively reset only the slots whose last failure was a disagreement or a back-translation miss, and left the "both models call it noise" and "duplicate" drops closed. The upgraded models recovered 297 additional entries from that reset on the first pass.

Retiring a scraper under a live service

The measurement gave us the decision; the migration still had to avoid breaking a service that reads the glossary in real time. The order of operations ended up mattering more than any single step:

1. Ship and deploy the code that removes the crawler from the nightly schedule first. If you delete data while the old crawler is still scheduled, the next run happily re-scrapes and re-promotes everything you just removed.
2. Point the model-name environment variables at the new generators on the batch workers before regenerating, or the nightly job will quietly refill missing slots with the old models. Silent fallback to a default model is the failure mode to fear here, so we verified the model tag recorded in the audit log after the first small batch.
3. Take a database-level backup of every row you are about to delete, including the rows that foreign-key cascades will take with them. A CSV of the primary table is not a rollback plan.
4. Delete in batches with a lock timeout, then regenerate, then re-run your regression benchmark. Ours held exactly at its pre-migration value (a pair-hit rate of 0.88), which is the number that let us call the migration done.

FAQ

Why not just use one strong model instead of two? A single model has no internal signal for "I am guessing". Two independent models disagree precisely on the entries where guessing happens, and that disagreement is the filter. In our data, dropping disagreements and gating the rest cut contamination to effectively zero at a cost of skipping roughly 40 percent of candidates per pass, most of which succeed on a later attempt.

Does this generalize beyond glossaries? The pattern (independent double generation, normalized agreement, round-trip verification, bounded retries) fits any task where outputs are short, checkable strings: entity names, code identifiers, category labels. It fits poorly where outputs are long free text, because agreement after normalization stops being meaningful.

Is scraping ever still worth it? Yes, for the source-language side. Official pages remain a good source of which concepts exist in a domain, and our replacement still crawls source-language pages to discover new terms. What we retired is trusting the translated side of those pages as ground truth.

What about the languages the site never published? That was the deciding argument. A scraper cannot produce what a site does not publish. Eight of our eleven target languages had near-zero crawlable coverage, so the choice was not "scraped quality vs generated quality" but "generated coverage vs nothing".