The Completion Event That Never Arrived: Two Silent Drops

A user reported that one download button never finished. The spinner ran forever. The obvious guess was that the render had failed, so I went looking for the error. There was no error. The file existed in object storage, the database row pointed at it, and a local replay of the same input produced a byte-identical artifact. The work had succeeded three times. Only the notification was lost, and it was lost in two independent ways.

What the logs actually said

The pipeline was ordinary: the client posts a job, the server returns 202 and renders in the background, then publishes a completion event on a pub/sub channel. A server-sent events endpoint fans that event out to whichever browser tabs are subscribed.

I lined up the publish timestamps against the SSE connection windows. Each connection logs its duration on close, so the start time is the close time minus the duration.

SSE window (server log)   completion published   received
13:36:04 - 13:36:33   ->  13:36:44               no subscriber
13:37:14 - 13:41:25   ->  13:41:32               no subscriber
13:45:59 - 13:49:41   ->  13:50:07               no subscriber

Three for three, the event fired into an empty channel. Redis pub/sub does not queue for absent subscribers, so the message was gone the moment it was published. That much was clear within an hour. What took longer was accepting that two separate bugs were producing the same symptom.

Another detail in the same logs turned out to matter more than the misses themselves: the connection durations were wildly uneven. 2.58s, 28.59s, 5.79s, 35.2s. A long-lived stream with a 20 second heartbeat and a 30 minute session cap should not close after 2.58 seconds.

Drop one: a per-item done closed the whole stream

The status codes in this system are integers where the last three digits encode a phase. A helper decided whether a status was terminal:

func isDone(s Status) bool {
	r := s % 1000
	return r == 500 || r == 600 || r == 800
}

The SSE loop used that helper to decide when to stop streaming and return. Some time earlier, a new status had been added for "one track finished rendering", and it was numbered 5600. Nobody noticed that 5600 % 1000 is 600.

So every successful per-item completion was classified as terminal, and the handler returned. The stream closed. A test made the asymmetry stark:

status=5600 (item done)     isDone=true   terminal=true    stream closes
status=5910 (item failed)   isDone=false  terminal=false   stream stays

Failures kept the connection alive. Successes killed it. If a user queued nine items, the first success closed the stream, the browser waited its reconnect delay of one second, and every completion published inside that window vanished. The uneven connection durations were this bug, visible in production for weeks, looking like nothing more than flaky networking.

The fix is small, and it is the second half that matters:

func isSSETerminal(s Status) bool {
	if s == StatusItemDone {
		return false
	}
	return (isDone(s) && s >= StatusFirstPhaseDone) ||
		s == StatusError || s == StatusFailed
}

I kept isDone untouched because other callers depend on it, and I wrote a test that walks every integer from 0 to 10000 and asserts the new predicate matches the old expression except at that one value. Extracting the decision into a named function is what made that test possible at all. The original was an inline boolean inside a 230 line handler that also owned an HTTP response writer, database reads, and a rate limiter.

The general lesson is not about one bad constant. Deriving control flow from arithmetic on an identifier means every new identifier is a chance to silently change behavior somewhere far away. A terminal bool field on the status definition would have made the mistake impossible to write.

Drop two: subscribing after you start the work

The second drop was in the client. The order was: post the job, wait for the 202, then open the SSE connection and register interest.

That works when rendering takes a while. It fails when the job is fast. One item in this batch had a single segment, and its render finished in 1.5 seconds, well before the browser had a subscription registered. That item failed every single time, while items with 30 or 200 segments almost always worked. The bug looked data-dependent, which is why it survived so long: it reproduced only on the one input nobody thought was special.

The obvious repair is to subscribe first. That is what I did, but the ordering alone was not enough, and the reason is worth stating because it bit me during implementation. The client closed the connection as soon as its pending set emptied, and that check ran on every message. The server sends an initial status frame the moment a connection opens. So opening the connection early, on its own, produced: connect, receive initial frame, pending set is empty, close. The race came right back. Subscribing early and keeping the subscription alive are one change, not two.

Why the obvious fallback made it worse

Since the completion event can be missed, the instinct is to add a recovery path. I designed one twice and threw both away.

The first was client polling: after posting, poll a result endpoint every few seconds until the answer appears. It looks robust until you notice where the poll timer and the job id live. They live in browser memory. The original bug report said the item would not download "even after refreshing several times". A refresh destroys exactly the state the fallback depends on. The fallback covered every case except the one that was reported.

The second was server-side replay: store each result in a hash keyed by job, and when a client connects, read the hash and replay any results it finds. This survives a refresh in principle. In practice it created a worse failure than the one it fixed. Because the client now subscribes before posting, the item is always in the pending set at connect time. A stale result from an earlier run would be replayed, matched against that pending entry, and treated as the answer to the request that had not even been sent yet. The user would silently receive an old artifact and believe it was the new one. An endless spinner is a bad outcome that the user can see. A wrong file that looks right is worse.

There was a third problem: the SSE handler was shared with other consumers, and one of them classified any status ending in failed as its own failure. Replaying stored per-item failures would have made unrelated operations show error banners.

So the fallback was dropped entirely, and the effort went into not losing the event in the first place. Subscribe before starting the work, keep the subscription open while the panel is open, and stop closing the stream on per-item completions. A recovery path that is weaker than the primary path is not redundancy. It is a second thing to maintain that fails in the same conditions, plus a new class of bug.

What is still not covered

Removing the fallback means accepting a smaller guarantee, and it is worth naming it rather than pretending otherwise. This is live delivery reliability, not durable completion. If the server restarts mid-render, or the pub/sub link drops for a moment, or the browser is frozen in a background tab long enough to miss the frame, the notification is still gone.

What replaced the fallback is a client-side timeout. After 30 minutes the item flips to an error state and the user can retry. That is a worse experience than automatic recovery, and it is honest about what the system actually promises. For the single-segment item that started all this, retrying costs 1.5 seconds.

Two smaller adjustments came out of the same review. The session cap on the stream had to exceed the client timeout, otherwise the server hangs up first and manufactures the very reconnect gap that drops events. And the connection limiter keyed per user had a shorter TTL than the new session length, which meant a live connection would age out of its own counter and let extra connections past the limit. Neither was in the original plan. Both were found by asking what else the timeout change touched.

FAQ

Is this an argument against pub/sub for completion notifications? No. Pub/sub is a reasonable transport for "tell the browser now". The mistake is treating it as storage. If you need the answer to survive a refresh, the answer has to live somewhere durable and be fetchable by identity, which means a real endpoint with an id you can reconstruct, not a replay keyed by whatever the client happens to be waiting for.

Why not just fix the client and skip the server change? Because the two bugs are independent. Subscribing early fixes fast jobs. It does nothing about the stream closing on the first success, which is what breaks batches. Each fix alone leaves a reproducible failure.

How do you find a bug like the terminal status one? Look for durations that do not match the design. The stream was configured to live for 30 minutes and was closing after 3 seconds. That gap was in the logs the whole time and read as network flakiness. Log the reason a long-lived connection closed, not just that it closed.

Would a job queue with a status table have avoided all of this? Probably, and that is the right long-term shape: a job id returned on submit, a status readable at any time, and the push channel as an optimization rather than the only way to learn the outcome. That is a larger change than a fix for a live bug, and it is worth being explicit about deferring it rather than half-building it under pressure.