Graceful Shutdown and Request Draining on SIGTERM in Go

A container that exits the instant it receives a termination signal cuts every request it was still serving. The client sees a connection reset or a 5xx, an open database transaction rolls back mid-write, a buffered batch of logs never flushes, and a lock the process was holding stays held until it times out somewhere else. The fix is not complicated, but it has to happen in a specific order: stop accepting new work, let the in-flight work finish under a deadline, then release resources. This post walks through that sequence in Go, with the signal handling, the server drain, and the cleanup hooks, plus where the deadlines have to line up so the platform does not kill you halfway through.

Termination is a normal event, not an exception

The first mental shift is to stop treating shutdown as a rare failure. In a container platform it is one of the most frequent things that happens to your process. An autoscaler adds an instance under load and removes it when load drops. A rolling deploy replaces every instance one by one. A node is drained for maintenance and every container on it is asked to leave. A scale-to-zero service is torn down after an idle window. Each of these ends with the same message to your process: a termination signal, usually SIGTERM, followed by a hard SIGKILL after a grace period if you have not exited on your own.

If you only think about shutdown when a machine crashes, you will under-build it, because a crash is rare and you can tolerate the occasional dropped request. But a deploy is not rare. If every deploy drops the requests that were in flight on the instances being replaced, then every deploy shows some users an error. On a service that deploys several times a day and scales up and down all day, "handle SIGTERM correctly" is not a nice-to-have. It is the difference between a deploy being invisible and a deploy being a small outage.

By default many programs do nothing special with SIGTERM. The signal's default action is to terminate the process immediately, and that is exactly what you do not want. Here is what that instant exit costs, concretely:

• In-flight HTTP requests are cut. The client gets a reset connection or an incomplete response and, depending on the method, may or may not be safe to retry.
• An open transaction is aborted. If the request was halfway through a multi-statement write, the database rolls it back, which is at least consistent, but the user's action silently did not happen.
• Buffered writes are lost. Anything you batched in memory to flush later, log lines, metrics, an outbound event queue, dies with the process.
• Held resources are not released cleanly. A distributed lock, a lease, an in-use connection from a pool on the other side of the wire, all of these now depend on a timeout to be reclaimed instead of being handed back.

None of this shows up in local testing, because locally you stop the server with Ctrl-C when it is idle. It shows up in production, spread thin across every deploy, as a low background rate of errors that is hard to attribute because it correlates with your own deploys rather than with traffic.

The shutdown sequence, in order

The whole technique is three phases in a fixed order. Getting the order wrong is the most common mistake, so it is worth stating plainly before any code.

The reason intake has to stop first is mechanical. If you begin waiting for in-flight requests while new ones are still arriving, the set of in-flight requests never shrinks to zero, and your drain runs until the deadline every time. You stop the inflow, the existing requests complete one by one, and the count reaches zero on its own. Only then is it safe to close database connections and flush buffers, because doing that while a handler is still running would pull the floor out from under it.

Catching the signal in Go

Go makes the signal part clean. Since the standard library added signal.NotifyContext, you can turn SIGTERM and SIGINT into a context that cancels when either arrives, and hang the whole shutdown off that cancellation.

func main() {
	ctx, stop := signal.NotifyContext(context.Background(),
		syscall.SIGTERM, syscall.SIGINT)
	defer stop()

	srv := &http.Server{Addr: ":8080", Handler: newRouter()}

	// Serve in a goroutine so main can wait on the signal.
	go func() {
		if err := srv.ListenAndServe(); err != nil &&
			!errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("listen: %v", err)
		}
	}()

	<-ctx.Done() // blocks until SIGTERM or SIGINT
	log.Println("shutdown signal received, draining")

	if err := drain(srv); err != nil {
		log.Printf("graceful shutdown incomplete: %v", err)
	}
	log.Println("shutdown complete")
}

The important detail is that ListenAndServe returns http.ErrServerClosed when you shut it down deliberately, and that is a normal, expected return, not an error to fail on. Treating it as fatal is a common bug that makes clean shutdowns log noisy errors.

Draining in-flight requests with Shutdown

The server's own Shutdown method is the drain. It stops the listeners so no new connections are accepted, then blocks until every active request has returned, or until the context you pass it is cancelled. That context is where your deadline lives.

func drain(srv *http.Server) error {
	// Stop routing to this instance first.
	health.SetNotReady()

	// Give in-flight requests a bounded window to finish.
	ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
	defer cancel()

	// Shutdown stops new connections, then waits for active ones.
	if err := srv.Shutdown(ctx); err != nil {
		// Deadline hit: some requests were still running. Give up on
		// them but record it, do not hang waiting for a SIGKILL.
		return fmt.Errorf("drain deadline exceeded: %w", err)
	}

	// Safe now: nothing is still serving. Release resources.
	return cleanup()
}

Shutdown returning nil means the drain finished cleanly and every request completed. It returning the context's deadline error means time ran out with requests still active. That is not a crash. It is you deciding that waiting longer is worse than proceeding, which is the right call when the platform is about to send SIGKILL anyway. Log the fact so a rising rate of deadline hits is visible, because that usually means requests are taking longer than the window allows and the window or the requests need attention.

One caveat: Shutdown does not close hijacked connections such as WebSockets or other long-lived streams, and it will wait on them. If you have those, you need to signal them to close separately, for example by cancelling a context they select on, so the drain can actually complete.

Fail the health check before you drain

Notice that drain calls health.SetNotReady() before it calls Shutdown. This is the piece people miss, and without it the drain is racy.

In front of your instance sits a load balancer or service mesh that decides where to route traffic based on a readiness or health check. If you go straight to Shutdown, there is a window where the load balancer still believes this instance is healthy and keeps sending it new requests, which now hit a server that is refusing connections. The result is exactly the dropped requests you were trying to avoid, just moved to a different spot.

The fix is to flip your readiness endpoint to failing first, then wait long enough for the load balancer to notice and remove you from its pool, and only then begin the real drain. In practice that looks like a short sleep tuned to the health check interval.

func (h *Health) SetNotReady() { h.ready.Store(false) }

func (h *Health) ReadyHandler(w http.ResponseWriter, r *http.Request) {
	if h.ready.Load() {
		w.WriteHeader(http.StatusOK)
		return
	}
	w.WriteHeader(http.StatusServiceUnavailable)
}

After SetNotReady, a brief pause (a few times the health check interval) lets the load balancer stop routing before Shutdown closes the door. The exact pause depends on how the platform probes you, but the principle holds everywhere: stop being advertised as available before you stop being available.

Cleanup hooks: flush, release, close

Once Shutdown returns cleanly, no handler is running, so it is finally safe to tear down the things handlers depend on. This is where you flush buffers, return leases and locks, and close connection pools, in that order.

func cleanup() error {
	var errs []error

	// Flush anything batched in memory before connections close.
	if err := logBuffer.Flush(context.Background()); err != nil {
		errs = append(errs, fmt.Errorf("flush logs: %w", err))
	}
	if err := metrics.Flush(context.Background()); err != nil {
		errs = append(errs, fmt.Errorf("flush metrics: %w", err))
	}

	// Release distributed locks and leases so peers can proceed now,
	// instead of waiting for the lease to time out.
	if err := locks.ReleaseAll(context.Background()); err != nil {
		errs = append(errs, fmt.Errorf("release locks: %w", err))
	}

	// Close pools last, after flush and release have used them.
	db.Close()
	cache.Close()

	return errors.Join(errs...)
}

Order matters inside cleanup too. Flush before you close the connections the flush needs. Release locks before you close the client that talks to the lock store. Close pools last. If you close the database pool first and then try to flush a batch that writes to the database, the flush fails and you lose the batch, which is the loss you were trying to prevent.

Releasing locks explicitly is worth calling out. If you rely only on lease expiry, a clean shutdown still leaves the lock held for the full lease duration, during which no peer can take it. Handing it back on the way out lets the next instance proceed immediately, and the expiry stays as a safety net for the unclean case where the process was killed outright.

Keep the drain deadline under the platform's grace period

Your drain deadline has to be strictly shorter than the grace period the platform gives you between SIGTERM and SIGKILL. This is the constraint that ties the whole thing together, and it is easy to get wrong because the numbers live in two different places.

Say the platform sends SIGTERM and will send SIGKILL 30 seconds later if you have not exited. Your total shutdown, the readiness pause plus the drain plus cleanup, must fit inside that 30 seconds with margin. If your drain alone is set to 30 seconds, then the readiness pause and cleanup push you past the grace period, SIGKILL arrives mid-cleanup, and you lose the buffers you were flushing. A safe split for a 30 second grace might be a few seconds of readiness pause, around 20 to 25 seconds of drain, and a couple of seconds reserved for cleanup.

Two practical notes. First, if your platform's grace period is configurable, set it explicitly rather than trusting a default, because defaults are often short (10 seconds is common) and a 25 second drain against a 10 second grace means SIGKILL always wins. Second, the deadline is a ceiling, not a target. Most shutdowns drain in well under the limit because most requests are short. The deadline only bites on the slow tail, and hitting it should be rare enough that each occurrence is worth a log line.

Idempotency, retries, and the cost of draining

Even a correct drain drops something eventually. A request that is genuinely slower than your entire grace period, a connection that dies during the readiness window before the load balancer reacts, an unclean SIGKILL because the process wedged. Graceful shutdown shrinks the dropped set to nearly nothing, but nearly nothing is not nothing, so the last layer is to make a dropped request survivable.

Two properties do most of the work. Make write operations idempotent, so that a client or upstream that retries a request it is unsure about does not double-apply it. An idempotency key on the request, checked against recent operations, turns a retried payment or a retried create into a safe no-op. And let clients retry idempotent requests on a connection reset, because during a rolling deploy a retry usually lands on a healthy instance and succeeds. Together these mean the rare request that slips through the drain is retried and completes, rather than being lost.

This is also why you should not chase a perfect drain. Pushing the deadline higher to catch the last slow request has a real cost, and idempotency plus retry covers that tail far more cheaply than a longer deadline does.

Draining is not free. Every deploy now waits for the slowest in-flight request on each instance before that instance exits, so a rolling deploy takes longer, bounded by your drain deadline times the rollout shape. Scale-down is slower for the same reason, since a removed instance lingers until it drains. If you set a generous 25 second deadline and your rollout replaces instances in sequence, the added minutes are real.

The tuning knob is the drain deadline, and it is a genuine tradeoff rather than a value to maximize. Too short and you cut requests that would have finished a second later, which defeats the point. Too long and every deploy and every scale-in event drags, and you are mostly waiting on a long tail that idempotency and retry would have handled anyway. Pick a deadline that covers the bulk of your request latency distribution, the high percentile of normal requests rather than the absolute worst case, and let retries mop up beyond it. The goal is not to guarantee that no request is ever dropped. It is to make shutdown ordinary, so that the autoscaler and the deploy pipeline can move instances around all day without any of it reaching a user.