Whisper's Prompt Silently Truncates at 224 Tokens, From the Front

We shipped a keyword biasing prompt for Whisper transcription and it made our transcriptions worse. Not slightly worse: the prompt actively pushed the model toward the wrong brand names. The root cause was a unit mismatch that the API never reports. Whisper's prompt parameter has an effective limit of 224 tokens, anything beyond that is dropped silently, and the part that gets dropped is the front. This post walks through how we proved that behavior with a black-box A/B test, and the budget arithmetic that fixed it.

What the prompt parameter actually does

Whisper accepts an optional prompt with each transcription request. The official guidance says it can be used to "guide the model's style or specify how to spell unfamiliar words." For domain-heavy audio, such as medical consultations, legal dictation, or product support calls, this is the main lever you have with a hosted API: list your domain vocabulary in the prompt and the model becomes far more likely to spell those terms correctly.

Under the hood the prompt is not free text the model reasons about. It is encoded and placed in the decoder's context window as if it were the transcript of the preceding audio segment. The model continues from it. That framing explains two properties that surprised us:

• The prompt competes with the output for context space. Whisper's text context is 448 tokens, and the implementation reserves at most half of it, minus one, for the prompt: 223 to 224 tokens.
• When the prompt is too long, the reference implementation keeps the tail, not the head. In the original source this is a one-line slice: prompt_tokens[-(n_ctx // 2 - 1):]. Everything before that boundary vanishes before the model sees it.

Neither property is surfaced by hosted APIs. The provider we used validates a separate limit, 896 characters, and returns an error when you exceed it. Nothing warns you about the token limit, because the truncation happens inside the model runtime, not in request validation.

How a helpful prompt becomes a harmful one

Our prompt was assembled from a glossary database: a hand-written seed list of the most important brand names first, then database terms appended until a byte budget was full. The budget was 850 bytes, chosen against the "896" limit in the provider's error message, which we had assumed was bytes.

Three problems stacked on top of each other:

1. The 896 limit is characters, not bytes. Korean text is 3 bytes per character in UTF-8, so an 850-byte prompt is only around 280 characters. We were using less than a third of the character budget while believing we were near it.
2. Those 280 characters were about 380 tokens. CJK scripts tokenize expensively in Whisper's BPE vocabulary, roughly 1.2 to 1.4 tokens per character in our measurements. The 224-token ceiling was exceeded by 70 percent on every single request.
3. The overflow was cut from the front, which is exactly where we had put the most important terms.

The failure mode was vicious. Our seed list started with the brand names users say most often. Those died first. What survived, purely by accident of database row order, was a cluster of similar-sounding but wrong terms near the end of the prompt. So when a user said something like "Sonalift" (all brand names in this post are stand-ins), the model, primed with the surviving lookalike "Sonaline", transcribed the wrong brand with confidence. A prompt with no biasing at all performed better than ours, because ours was biasing toward the mistakes.

There was one more twist worth flagging: the assembly query had no ORDER BY clause. Which 40 of the 1,200 candidate terms made it into the prompt depended on physical row order in the database, which shifts after vacuums and bulk writes. The same code, the same audio, produced different transcriptions on different days. If your biasing behaves nondeterministically, check whether your term selection is actually deterministic before blaming the model.

Proving front-truncation with a positional A/B test

You do not need access to model internals to verify the truncation direction. You need one over-limit prompt and two arrangements of it.

We built a prompt of about 460 characters, roughly 400 tokens, so comfortably over the 224-token ceiling but under the provider's 896-character validation. Then we ran the same audio file through it twice:

• Arrangement A: the three critical terms placed at the front, filler vocabulary after them.
• Arrangement B: identical filler first, the same three terms at the very end.

The result was unambiguous. Arrangement A transcribed all three terms wrong, reproducing our production bug exactly. Arrangement B transcribed all three correctly. Same content, same token count, same audio; only the position changed. That is front-truncation observed from the outside, no source access required.

The practical rule falls out immediately: in a Whisper prompt, the end is prime real estate. If truncation ever happens, whatever you placed last survives. Put your must-have terms at the tail, and treat the head as the sacrificial zone.

Budgeting tokens without shipping a tokenizer

The durable fix is to never exceed the limit in the first place, which means counting tokens before sending. We did not want to embed a BPE tokenizer in a Go service for this one purpose, so we built an estimator with per-script upper-bound coefficients, measured against the real tokenizer on our production vocabulary:

func estimateTokens(s string) int {
    sum := 0.0
    for _, r := range s {
        switch {
        case r > 0xFFFF: // beyond BMP: surrogate pairs, emoji
            sum += 2.0
        case unicode.Is(unicode.Hangul, r) || unicode.Is(unicode.Han, r) ||
            unicode.Is(unicode.Hiragana, r) || unicode.Is(unicode.Katakana, r):
            sum += 1.4
        case unicode.Is(unicode.Thai, r):
            sum += 1.0
        case unicode.Is(unicode.Cyrillic, r) || unicode.Is(unicode.Arabic, r):
            sum += 0.55
        case (unicode.IsLetter(r) && unicode.Is(unicode.Latin, r)) ||
            unicode.IsDigit(r) || r == ' ':
            sum += 0.45
        default: // punctuation and symbols
            sum += 1.0
        }
    }
    return int(math.Ceil(sum))
}

Three design decisions matter more than the exact numbers:

• Sweep runes, do not classify the whole string. Real vocabulary mixes scripts inside a single term ("V-line", "3D lift", brand names with digits). A single per-language coefficient misestimates exactly the terms you care about.
• Coefficients must be upper bounds, not averages. An average estimator that undershoots by 15 percent quietly reintroduces the truncation you are trying to prevent. We set our internal budget to 200 tokens against the 224 ceiling, so the estimator may overshoot but must never undershoot.
• Punctuation is not "other latin". Our first draft used 0.45 for everything non-CJK. Strings like "A/B/C-123" and trademark symbols tokenize at close to one token per character, so symbols got their own 1.0 lane.

The test suite pinned fixture strings with token counts measured by the actual Whisper tokenizer offline, then asserted estimate >= actual for every fixture. That assertion caught a real bug before deployment: our Thai coefficient of 0.7 came from averaging ordinary dictionary words, but the transliterated brand names that actually populate a biasing prompt tokenize at about 1.0 tokens per character. The upper-bound test failed, we raised the coefficient, and a whole class of silent truncation for Thai users never shipped. Write the assertion as a hard inequality; an "within 15 percent either way" tolerance would have let that bug through.

Reserve the tail budget before filling the rest

Once you can count tokens, assembly order still matters. Two rules made the final prompt robust:

First, reserve the budget for your critical terms before admitting anything else. We compute the cost of the tail block, the seed examples plus the pinned must-have terms, subtract it from the 200-token budget, and only then admit ranked glossary terms into the remainder. The naive order, fill greedily and append pins last, has a failure mode where low-value terms consume the budget and the pins get dropped by your own guard.

Second, place terms in ascending order of importance, so the string ends with the most critical ones. Inside the budget this changes nothing. But if any component of the pipeline ever miscounts, front-truncation deletes your least important terms first. It converts a worst-case catastrophic failure into a graceful one.

The final assembled prompt in our system: 25 ranked glossary terms, then the example seed, then three pinned brand names at the absolute end, totaling around 170 estimated tokens. The two production misrecognitions that started this investigation both disappeared, verified against the original audio clips before and after.

Checklist for prompt biasing against a hosted Whisper API

• The effective prompt limit is 224 tokens. Provider-side character validation (896 characters on the API we used) is a separate, much looser constraint. Budget against tokens.
• Overflow is truncated from the front, silently. Place critical vocabulary at the end of the prompt, never the beginning.
• Verify truncation behavior yourself with a positional A/B test: one over-limit prompt, critical terms at front versus back, same audio.
• If you estimate tokens without a tokenizer, sweep per rune with per-script coefficients, and make them upper bounds enforced by tests against real tokenizer output. Expect transliterated foreign brand names to tokenize far more expensively than dictionary text.
• Reserve budget for must-have terms first, fill the remainder by rank, and keep selection deterministic: stable sort keys all the way down to a unique id.
• Log estimated tokens per assembled prompt and alert when the estimate approaches the ceiling. The API will never tell you.

The uncomfortable summary is that a biasing prompt is a budgeted data structure, not a string. Treat it with the same care you would give a fixed-size buffer, because that is what the model turns it into.