A Feature That Never Ran, Because One Service Lacked a Key

A feature shipped, and every request to it failed with a 503 from that day forward. Nobody noticed for four days. The cause was not a bug in the feature. One of four deploy targets that share the same container image was missing an API key, and the local development environment always supplied that key, so no amount of local testing could have caught it. This is what that failure looks like, why the response time alone identified the layer, and the checklist that would have prevented it.

What a 1.4 millisecond failure tells you

The access log for the broken endpoint looked like this:

14:20:07  503  /api/v1/extract/preview  0.001846s
14:20:06  503  /api/v1/extract/preview  0.001463s
14:20:06  503  /api/v1/extract/preview  0.001414s

Nineteen requests in one day, all 503, all under two milliseconds. That number is the whole diagnosis.

The endpoint parses two uploaded files, runs them through a language model twice, and takes 30 to 60 seconds on success. A failure that arrives in 1.4 milliseconds did not attempt any of that. It did not read the request body. It was rejected at the door.

Response time sorts failures into layers before you read a single line of code:

A 503 that takes as long as a success is a dependency going down mid-flight. A 503 that returns instantly is a guard clause. Those are different bugs with different fixes, and the timestamp column tells you which one you have before you open an editor.

In this case the guard clause was a capability check at the top of the handler:

if s.embedder == nil || s.primaryLLM == nil || s.secondaryLLM == nil {
    writeError(w, http.StatusServiceUnavailable, "models are not available")
    return
}

One of those three was nil on every instance, on every request, since the day the feature went out.

Why local development hides a missing credential

The nil one was the second language model client, which is built only when an API key is present:

if cfg.OpenAIKey != "" {
    s.secondaryLLM = newClient(cfg.OpenAIKey)
} else {
    log.Warn("secondary model disabled: API key not set")
}

That warning was printed on every instance start for four days. Nobody was reading startup logs for a service that appeared to be running fine, because it was running fine. Every other page in the application worked.

The service is one of four deploy targets built from the same container image: a public API, a staging copy, a nightly batch job, and an internal review tool. Three of them had the key. The internal tool did not.

The reason that one target was different is worth naming, because it generalizes. The internal tool had spent its whole life needing nothing but a database connection. It was a table viewer with edit buttons. Its deploy configuration listed a database URL and a cache password, and that list had been correct for months. Then a feature landed in that tool that called a language model, and the assumption underneath the config list quietly stopped being true. The code changed. The deploy spec did not.

Local development made this invisible. On a developer machine the key comes from a .env file or a shared config loader, so the client is always constructed and the guard clause never fires. Every local run of that feature succeeded. Integration tests against a local server succeeded. The failure existed only in the gap between what the code now needed and what one deployment actually provided, and nothing that runs on a laptop can see that gap.

This is the property that makes credential drift nastier than an ordinary bug. Most bugs fail in at least one environment you look at. This one succeeds everywhere except production, and production fails silently unless someone opens that specific screen.

The error that lost its own message

The server was not being cryptic. It answered with a specific sentence:

{"detail": "models are not available"}

The browser showed Request failed (503).

The client helper that turned a failed response into a message read three keys, and the one the server actually used was not among them:

async errText(r) {
  const d = await r.json();
  return d.error || d.message || `Request failed (${r.status})`;
}

The API standardized on detail. The frontend helper was written against a different convention and never revisited. Every error body from every route was being fetched, parsed, and thrown away, leaving only a status code. The fix is one identifier:

return d.detail || d.error || d.message || `Request failed (${r.status})`;

This kind of mismatch is invisible in the happy path and invisible in tests that assert on status codes. It only shows up when someone is trying to debug from a screenshot, which is exactly when you need it. A contract mismatch on the error path costs nothing until it costs you days.

Two smaller changes made the remaining diagnosis self service. The guard clause now names which capability is missing instead of grouping three of them behind one sentence:

func (s *Server) missingCapabilities() []string {
    var miss []string
    if s.embedder == nil       { miss = append(miss, "embeddings") }
    if s.primaryLLM == nil     { miss = append(miss, "primary model") }
    if s.secondaryLLM == nil   { miss = append(miss, "secondary model (API key)") }
    return miss
}

And the rejection is logged with the same list, so the answer is in the response body and in the log line, not only in a startup warning that scrolled past four days ago.

A checklist for adding an external dependency

The repair itself was one command. Injecting the missing secret took seconds, and the endpoint went from 503 in 1.4ms to 200 in 30.5 seconds on the first try. The interesting part is the five checks that would have made the four days unnecessary.

1. List every deploy target that runs this image. Not the one you are testing. Services, staging copies, batch jobs, internal tools. Same binary means same new requirement.
2. Update the creation script, not just the live service. A one-off injection into a running service is invisible to the next person who recreates it from the script. If your redeploys only swap the image, the injection survives, and that is exactly why nobody notices the script is now wrong.
3. Make the startup warning fail loudly, or make the health check know. A log.Warn on a service that otherwise starts fine is not a signal. Either refuse to start when a declared feature cannot work, or expose capability state where someone looks.
4. Return the name of what is missing. Grouping several capabilities behind one message costs you the entire diagnosis when one of them is absent.
5. Assume local testing cannot verify steps 1 and 2. This is the uncomfortable one. Your machine has the credential. No test you can run there proves the deployment has it.

Steps 1 and 2 are the ones that actually prevent this class of failure, and they are the two that no test suite will do for you. They are a habit, not a tool.

Common questions

Should the service refuse to start when a credential is missing?

It depends on whether the dependency is core or optional. If the whole service is useless without it, fail fast at startup so the deploy visibly fails and rolls back. If it powers one feature among many, starting is correct, but then the capability state has to be visible somewhere other than a startup log. A degraded service that looks healthy is the worst of both.

Would a smoke test after deploy have caught it?

Only if the smoke test exercised that specific feature, which for a 30 second language model call is an expensive thing to run on every deploy. A cheaper version is a readiness endpoint that reports which optional capabilities are live, and a deploy step that compares that report against what the release expects. You are testing configuration, not behavior, so it can be fast.

Is this just a case for infrastructure as code?

Declared infrastructure helps, because the deploy spec lives next to the code that needs it and reviewers see both in the same change. It does not remove the problem. Somebody still has to add the new secret to the right target, and a declarative file with the wrong content deploys wrong content reliably. What it buys you is that the mistake is visible in a diff instead of living only in a console.

How do you find existing instances of this drift?

Enumerate the deploy targets that share an image and diff their environment and secret lists against each other. Differences are not automatically wrong, since a batch job legitimately needs less than a public API. But every difference should have a reason someone can state. The one in this incident had no reason. It was simply older than the feature that needed it.