Frequency Is Not Importance: A Ranking Axis That Failed Its Gate

We needed to rank a domain vocabulary so that the most important terms would fit into a tightly budgeted prompt. The obvious signal was frequency: count how often each term appears in a trusted corpus, rank by count, done. We built it, measured it, and the top 20 terms came back as "consultation", "treatment", "face", "reservation", and sixteen other words a general model already knows perfectly well. Not one brand name made the list. The axis we had built to promote distinctive vocabulary was promoting exactly the vocabulary that needed no promotion.

This post is about why frequency failed as an importance signal, the two traps we dodged before that failure (usage-derived frequency and raw counts), and the part that actually saved us: designing the rollout so the axis could fail cheaply, with a distribution check standing between the code and the behavior change.

The setup: 30 slots, thousands of candidates

The consumer of this ranking was a keyword biasing prompt for speech recognition. The prompt has a hard token budget, and in practice about 30 terms fit. The vocabulary database holds several thousand confirmed terms. Whatever ranking function we choose decides which 1 percent of the vocabulary gets to influence transcription; everything else might as well not exist.

Our existing ranking used source trust tiers (hand-entered terms above site-mined terms above machine-generated ones) with token cost as a tiebreaker. The problem was the middle of the distribution: thousands of terms share the same tier, and within a tier the tiebreaker favored short strings. Short strings in a human vocabulary skew generic. Two-character everyday words were floating to the top of the prompt while the brand names the whole system existed to transcribe sat below the cutoff.

Frequency looked like the fix. Important terms should appear often in domain text; obscure noise should not. That intuition is not wrong, but it is incomplete in a way that matters.

Trap one: usage frequency amplifies your own errors

The first candidate corpus was our own usage logs: transcripts of what users actually said. This is the most tempting corpus because it reflects real demand. It is also poisoned in a subtle way when the pipeline that produces it is the pipeline you are trying to fix.

Our transcription engine was misrecognizing a certain brand name as a similar-sounding competitor term. In the usage corpus, the wrong term appeared twelve times for every one appearance of the right term, because the corpus records what the engine wrote, not what the speaker said. Rank by that corpus and the misrecognition gets promoted into the biasing prompt, which makes the engine even more confident in the wrong term, which further inflates its frequency. The error funds its own growth.

The rule we took away: never derive a ranking signal from the output of the system the ranking is supposed to correct. Feedback loops of this shape do not announce themselves; they just quietly converge on the failure mode.

Trap two: raw counts reward repetition, not consensus

The second candidate was crawl frequency from trusted domain websites, which sidesteps the feedback loop because the text never passes through the recognition engine. But raw occurrence counts have their own bias: a single site that repeats a term on every page (navigation, footers, promotional banners) can outvote ten sites that each mention a different, genuinely important term once.

We switched the statistic from raw count to document frequency at the source level: in how many distinct sites does this term appear at all? A term that shows up across seven independent clinics is common ground for the whole domain; a term that appears 400 times on one clinic's site is that clinic's marketing. We also capped the axis at a small maximum so that it could break ties between trust tiers without overpowering them.

This was the right refinement of the wrong idea. It fixed manipulation and skew, and it still could not fix the core problem.

The measurement: generic words win anyway

With counting, deduplication, and source attribution built and tested, we ran the counter across 28 sites and looked at the top of the distribution before wiring the axis into the live ranking. The top 20 by document frequency:

consultation, procedure, face, contour, treatment, reservation, pain, inflammation, forehead, tissue, body shape, chest, elasticity, fat, lifting, filler, and a few more of the same character. Every single one is a word any speech model transcribes correctly out of the box. Zero brand names.

In hindsight the result is obvious. Document frequency measures agreement across sources, and what independent sources agree on most is ordinary language. Every clinic writes "consultation" and "reservation"; only some carry any given device or product. In a specialist corpus, breadth of appearance correlates with genericness at the top of the distribution. The distinctive vocabulary we wanted lives in the middle: present in several sources, absent from the extremes.

Frequency, in any flavor, answers "how common is this term?" Importance, for our purpose, was "how much does the model need help with this term?" Those are close to opposite questions. The terms a model needs help with are precisely the ones underrepresented in general text.

The part that worked: a gate between code and behavior

Here is what kept this from being an expensive mistake. During planning review, one reviewer flagged exactly this risk: crawl frequency might correlate with genericness and promote the junk instead of demoting it. We could not settle the question by argument, so we restructured the rollout around a measurement:

• The axis was implemented with an invariant, verified by a unit test: if every term's frequency is zero, the ranking is byte-for-byte identical to the old ranking. Empty data means the axis does not exist.
• The frequency table was populated by a separate batch step that was deliberately not enabled in the scheduled pipeline yet.
• Between population and adoption stood a written gate: inspect the top 20 of the distribution. If it is dominated by distinctive domain terms, adopt the axis. If it is dominated by generic words, truncate the table and skip adoption.

The gate failed, and failing was cheap. We truncated the table, left the axis code dormant behind its zero-data invariant, and shipped the fallback we had specified in advance: take the generic words the measurement had surfaced and add them to the stopword list that excludes terms from the prompt entirely. The measurement was not wasted; it produced the best stopword candidates we could have asked for, backed by data instead of intuition.

The general pattern: when a ranking signal is plausible but unproven, ship the mechanism and the data separately, put a distribution check between them, and pre-commit to what "failure" looks like and what the fallback is. The alternative, wiring the axis straight into production and evaluating it by watching quality metrics drift, costs weeks and erodes trust in the whole system.

What we would use instead of frequency

The gate left us with a sharper question: if not frequency, what does predict "the model needs help with this term"? Three signals survived the postmortem discussion:

• Tokenizer fragmentation. Terms that a model's tokenizer shatters into many pieces are terms the model has rarely seen. This is computable offline from the vocabulary alone, with no corpus at all, and in our earlier measurements it separated brand names from everyday words cleanly.
• Observed confusion. A dictionary of actual misrecognitions, collected from production incidents, names the exact terms that need boosting. It is small, but every entry is ground truth.
• Mid-band document frequency. If frequency is used at all, the useful band is the middle: terms present in a few independent sources but not all of them. The top of the distribution is generic; the bottom is noise.

None of these is as easy as a count. That is rather the point.

Checklist for shipping a corpus-derived ranking signal

• Never rank by a corpus produced by the system you are trying to correct; closed loops amplify their own errors.
• Prefer document frequency across independent sources over raw counts, and cap the axis so it cannot dominate stronger signals.
• Before adoption, look at the actual top of the distribution, not the metric's description. Decide the pass criterion before you look.
• Implement the axis with a zero-data invariant so that "no data" provably equals "old behavior", and keep population and adoption as separate switches.
• Pre-commit to a fallback. Ours was a data-driven stopword expansion, and it turned a failed axis into a useful artifact.
• Remember what frequency measures. Commonness and importance point in different directions exactly when your vocabulary is specialized, which is the only time you need the ranking at all.

The axis is still in the codebase, dormant. If we ever find a signal that passes the gate, adoption is one batch job away. Until then, the gate did its job: it let a plausible idea fail in a spreadsheet instead of in production.