Auto-Restarting Hung Workers Without Causing a Restart Storm

Two workers in a GPU fleet hung on the same evening, six minutes apart. The monitoring stack noticed within two minutes and posted a detailed alert: server name, verdict, the job that killed it. Then nothing happened for six hours, until a human woke up and typed docker restart.

The alert was perfect. The outage was long anyway. This post is about the gap between those two facts: why detection systems so often have eyes but no hands, and what it takes to wire a restart into the detection path without creating a worse problem, a restart storm that kills healthy machines.

Detection and remediation grow up separately

Nobody designs this gap on purpose. It accumulates.

Our system had two independent safety mechanisms, built months apart for different incidents. The first was a hang detector: if a worker reports processing but its heartbeat goes stale for more than 120 seconds, declare it unresponsive and page the channel. The second was an auto-restarter: if a worker sits in an explicit error state for 30 minutes, collect its logs, post them, and restart the container over SSH, with a daily circuit breaker capping restarts at three per server.

Each worked as designed. The trap is in the join: the restarter was keyed to a different predicate than the detector. Restart required state == "error". The hang we actually got was a heap corruption that froze the process mid-job, so the state field stayed processing forever while the heartbeat flatlined. The detector fired. The restarter's condition never became true. Worse, the restarter had a politeness rule: if the hang detector has confirmed a hang, step aside and let it handle things. The component it deferred to had no hands.

When your remediation is keyed to a different predicate than your detection, every failure mode that only one of them recognizes becomes an alert with no follow-up. Audit the pair as a unit: for each verdict the detector can produce, name the component that acts on it. Any verdict whose answer is "a person, eventually" is a gap you have chosen, and it should at least be a documented choice.

A restart is not a safe action by default

The fix sounds trivial: call the restart function from the hang path. The reason it took a design review instead of a one-line patch is that an automated restart is a loaded gun. The failure mode of a bad restart policy is not one dead worker, it is a fleet-wide storm where automation kills machines faster than they can boot.

The safeguards that made it shippable, in the order they run:

Persistence before action. A confirmed hang must persist for a fixed delay (we chose 3 minutes) before restart fires. Detection already required two consecutive bad observations; the delay adds tolerance for GC pauses, transient load spikes, and monitoring blips. The timer lives in Redis via SETNX, not in process memory, for reasons covered below.

A per-server execution lock. Two backend instances run the same monitoring loops. Both will reach the same conclusion at roughly the same time. A SETNX lock with a short TTL ensures exactly one of them executes the restart; the loser simply skips. Deduplication of alerts and serialization of actions are different concerns, so we key them separately: a dedup key suppresses duplicate notifications per incident, the lock suppresses duplicate commands per server.

Revalidation at the last moment. Between the decision and the SSH command sit log collection and alert posting, up to a minute of latency. The worker may have recovered by itself in that window. Re-read the status immediately before executing; if the heartbeat is fresh, cancel and clean up. Restarting a healthy machine because it was sick a minute ago is exactly the kind of harm automation must not do.

A shared circuit breaker. All restart paths increment one counter per server per day, and past three, automation stops and only alerts. Crucially, the counter increments just before the command actually executes, not when the pipeline starts. Early versions counted every attempt, so a run that failed at log collection still burned budget, and a noisy detector could exhaust the allowance without a single restart happening.

Escalation when the medicine fails. A restart that does not cure the hang must not end the story. We record the restart time; if the same server is still hung ten minutes later, a one-time "restart did not recover" page goes out with a mention. Without this, the system sends "restart triggered" and goes silent, and a reader assumes the problem is handled. That silent-after-action state is how the original six-hour outage happened, and automation can reproduce it perfectly.

The restart you just did will look like a hang

The subtlest failure we found in review, at roughly coin-flip probability, was self-inflicted: the double restart.

After docker restart, the container spends a minute or more booting and loading models. During that window the old status record is still sitting in Redis saying processing with a stale heartbeat, because the dying process never got to update it. To the recovery loop, that is indistinguishable from a fresh hang. It re-arms its timer, waits out the delay, and restarts the machine again mid-boot. Each round burns circuit breaker budget, so one real hang can exhaust the daily allowance killing a booting container repeatedly, and then automation gives up exactly when it was needed.

The fix is a cooldown key: after a restart, suppress re-arming for a fixed window (we use 10 minutes) and clear the persistence timer whenever the cooldown is present. The general rule: any automated action that temporarily makes the system look broken must leave a marker that tells the detector "this is me, not a new incident."

Two adjacent traps from the same review are worth naming:

Measure incident duration in shared storage, not process memory. Our first draft read the hang's start time from an in-memory struct. That field was refreshed on every re-confirmation cycle, so the measured duration hovered near zero forever and the restart threshold was unreachable. The working design anchors the start time in Redis with SETNX: first observer wins, every instance sees the same clock, and a multi-instance deployment cannot reset each other's timers.

Derive the trigger condition from the data, not from local conclusions. With two monitor instances, each has its own view of "confirmed." If instance A arms the shared timer and instance B, one observation behind, still considers the server healthy, B would clear A's timer, and the two would fight indefinitely. Deriving the predicate directly from the shared status record makes every instance compute the same answer.

Some failures should not trigger a restart

Wiring one verdict to remediation does not mean wiring all of them. We deliberately left two alone.

A missing status record means the agent is unreachable: the server may be rebooting, network-partitioned, or intentionally powered off, and one of our machines is off for cost reasons right now. SSH-restarting a machine in an unknown power state is somewhere between useless and harmful, so that verdict stays alert-only. And a job whose progress markers stall while its heartbeat stays healthy gets no restart either, because long-running jobs legitimately look like that, and killing a live worker over a slow task trades a false positive for real damage.

The boundary rule that fell out of this: automate remediation only for failure modes where the action is idempotent-ish, the blast radius is one already-dead worker, and a wrong guess is cheap. Everything else escalates to a person, quickly and loudly.

Common questions

Why not just use container health checks and let the orchestrator restart? If you are on Kubernetes with proper liveness probes, do that first. This pattern applies when the "hang" is visible only at the application level (heartbeats in Redis, job progress) while the process still accepts TCP connections, which passes most port-level health checks. Our zombie answered connections for six hours.

Is a 3-minute delay too slow? Add up the real timeline: stale threshold, two confirmation cycles, the delay, then restart and model loading. Ours lands around ten minutes from failure to recovered, versus six hours with a human in the loop. Tightening the delay below the boot time of the service buys little and raises the false-positive cost.

Why share one circuit breaker across restart paths instead of one each? The breaker protects the physical machine, and the machine does not care which code path restarted it. Separate budgets multiply the worst case. If you split them, cap the sum.

What if the restart command itself fails? Treat it as a first-class outcome, not a log line. Release the dedup key so the next cycle can retry, and page a human immediately, because a machine that cannot even be restarted remotely is past what automation should be deciding about.