Backpressure for Worker Queues That Fill Too Fast to Drain

A queue between a producer and a worker feels like a shock absorber. Work arrives in bursts, the queue holds it, and workers drain it at their own pace. That intuition holds only while the average arrival rate stays below the average drain rate. The moment producers outrun workers for long enough, an unbounded queue stops absorbing anything. It grows without limit, memory or the queue store runs out, and the whole system falls over at once. The fix is not a bigger queue. It is backpressure: a bounded queue that pushes the overload back onto the producer instead of hiding it until the crash. This post shows why unbounded queues fail, what backpressure looks like in Go, and the tradeoffs that come with learning to say no.

Why an unbounded queue postpones the crash instead of absorbing it

Think of the queue as a bucket with water pouring in and draining out. If the drain is at least as fast as the inflow on average, the level stays bounded and bursts smooth out. If the inflow is even slightly faster on average, the level rises forever. There is no size of bucket that fixes a permanent imbalance, only a size that changes how long you wait before it overflows.

An unbounded queue removes the overflow point, which sounds like a feature and is actually the failure. The imbalance is still there, so the backlog grows until it consumes all available memory or fills the queue store to its hard limit. Two things break before that, and both are worse than a clean rejection.

The first is latency. A job that enters a queue with a million items ahead of it waits for a million items to drain. By the time a worker reaches it, the request that created it has often timed out, the user has left, or the result is stale. You are now spending your scarce worker capacity computing answers nobody is waiting for, which slows the drain rate further and deepens the backlog. This is a feedback loop that gets worse on its own.

The second is the cascade. When the queue store finally runs out of memory, it does not fail politely. It can take down the process that hosts it, block every producer trying to enqueue, and knock over unrelated services that shared the same machine or the same connection pool. A single slow worker turns into a system-wide outage. The unbounded queue did not prevent the collapse, it delayed it and made it bigger.

Backpressure is a bounded queue that pushes back

Backpressure is the property that when a downstream stage cannot keep up, the pressure travels upstream to the producer instead of piling up invisibly in the middle. The simplest way to build it is a queue with a hard capacity. When the queue is full, the producer cannot enqueue, and it has to make a decision right there. That decision is the whole point. It is where the system chooses how to shed load on purpose rather than having the load shed it.

In Go the primitive is already backpressure aware. A buffered channel is a bounded queue, and sending to a full channel blocks the sender until a worker makes room. That block is backpressure in its most direct form.

// A buffered channel of capacity N is a bounded queue. When it holds N jobs,
// the next send blocks until a worker receives one. The block is the
// backpressure: the producer cannot outrun the workers.
jobs := make(chan Job, 256)

// Producer. This send parks the goroutine if the buffer is full, so the
// producer's own rate is capped by how fast workers drain.
jobs <- job

A blocking send is the right behavior when the producer can afford to slow down, for example an internal batch loader that has no user waiting on the other end. Slowing the producer to match the workers is exactly what you want, because the two rates are now coupled and the backlog cannot grow past the buffer.

But a blocking send is the wrong behavior when there is a user or an upstream caller with a deadline. Making an HTTP handler block indefinitely because the workers are behind just moves the unbounded wait from the queue into your connection pool, and you run out of connections instead of memory. When someone is waiting, you want to reject fast rather than block, so the caller learns immediately that the system is saturated and can back off or fail over.

// Non-blocking enqueue for request paths. If the buffer is full, do not wait.
// Reject now so the caller gets a fast, honest "try later" instead of a hang.
select {
case jobs <- job:
    // accepted
default:
    return errQueueFull // surface as HTTP 429 or 503 with Retry-After
}

The two options, block the producer or reject the job, are the two faces of backpressure. Blocking couples the rates. Rejecting sheds the excess. Which one you pick per producer depends on whether anything is waiting, and a real system usually uses both in different places.

Cap the concurrency so workers cannot flood what is downstream

Bounding the queue protects you from the producer. You also have to bound the workers, because the workers are producers for whatever sits below them, usually a database, a GPU, or another service. A common mistake is to spawn a goroutine per job. It looks clean and it removes the queue entirely, but it just relocates the unbounded growth. Ten thousand incoming jobs become ten thousand goroutines all hammering the database at once, and the database is now the thing that falls over.

A fixed worker pool is the cap. You start a known number of workers, each pulls from the bounded queue, and that number is the maximum concurrency that will ever hit the resource below. The pool size becomes a deliberate limit you set from the capacity of the downstream, not an accident of how much traffic arrived.

// Fixed worker pool. poolSize is the hard ceiling on concurrent work hitting
// the downstream resource. It does not grow with load, which is the point.
func startPool(poolSize int, jobs <-chan Job) {
    var wg sync.WaitGroup
    for i := 0; i < poolSize; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for job := range jobs {
                process(job) // the only place downstream load is generated
            }
        }()
    }
    wg.Wait()
}

Now the two bounds work together. The buffered channel caps how much work can wait, and the pool caps how much work can run. Total load on the downstream is at most poolSize concurrent operations plus a queue of at most the buffer capacity, and neither number moves when a traffic spike arrives. The spike hits the enqueue path, gets rejected or slowed there, and never reaches the database as a flood. Pick the pool size from the downstream capacity, for example the connection pool limit of your database or the number of GPUs you own, not from a round number that feels generous.

Drop work that is already too old to matter

Bounding the queue and the pool keeps the system standing, but under sustained overload the jobs that do get in can still be stale by the time a worker reaches them. If a request has a five second budget and a job has been waiting eight seconds, running it is pure waste. You spend worker time producing a result the caller has already given up on, which steals capacity from fresh jobs that still have a chance.

A deadline on each job fixes this. Stamp the job with the time it must be done by, and have the worker check that deadline before starting the expensive part. If the deadline has passed, discard the job and move on. In Go the idiomatic carrier is a context with a deadline, which also lets you cancel work already in flight.

type Job struct {
    Payload  []byte
    Deadline time.Time
}

func process(job Job) {
    // Skip work that can no longer be useful. Under overload this is what
    // keeps workers spending their time on jobs someone still wants.
    if time.Now().After(job.Deadline) {
        metrics.Expired.Inc()
        return
    }
    ctx, cancel := context.WithDeadline(context.Background(), job.Deadline)
    defer cancel()
    doExpensiveWork(ctx, job.Payload)
}

Dropping stale work is a form of backpressure aimed at time rather than volume. It admits that some jobs are already lost and refuses to let them consume resources on the way out. During a spike, this is what lets a saturated system keep serving the requests that are still live instead of grinding through a backlog of corpses.

Unbounded queue versus backpressure, side by side

The difference between the two designs is easiest to see under the case that matters, sustained overload, where arrivals stay above drain for a meaningful stretch of time.

The unbounded column is not a system that handles more load. It is a system that handles the same load and hides the failure until it is catastrophic. The bounded column fails earlier, smaller, and legibly, which is what you want from a failure.

Priority, isolation, and retries that do not amplify the problem

Two more pieces make backpressure practical rather than blunt. The first is that not all work is equal. If a health check and a bulk import share one queue, a flood of imports will starve the health checks, and your monitoring goes dark exactly when you need it. Separate queues with separate pools isolate the important work so that overload in one lane cannot drown another. A small dedicated pool for critical jobs keeps them flowing even while the bulk lane is rejecting everything.

The second is retries. Retrying a rejected job is reasonable, but a naive retry loop is how backpressure turns into a retry storm. When the system rejects a job because it is saturated, retrying immediately adds load to an already saturated system, and if every client does the same the retries alone can keep the system down long after the original spike passed. Retries need a backoff that grows on each attempt and a hard cap on the number of attempts, so a rejection leads to a slower next try rather than an instant second hit. Backpressure and backoff are the same idea applied at two ends. The server pushes back by rejecting, and the client respects the push by slowing down instead of pounding harder.

Observability ties it together. Queue depth and wait time are your early warning. A queue that is trending toward full, or a wait time creeping toward the deadline, tells you the imbalance is starting well before anything is rejected. Alert on those two numbers and you get to add workers or slow producers on your own schedule, instead of finding out from an outage.

The tradeoff: partial failure is the price of staying up

Backpressure is not free, and it is honest to name the cost. A bounded queue rejects jobs. A capped pool serves fewer concurrent requests. A deadline throws away work that took real effort to produce. Every one of these is a partial failure, and someone experiences it as a request that did not go through. If you measure success as never rejecting anything, backpressure looks like a regression.

The comparison that matters is not backpressure against a perfect system that accepts everything. No such system exists once producers can outrun workers. The comparison is partial failure against total failure. An unbounded queue accepts every job right up to the moment it accepts none of them and takes the process down with it. A bounded queue rejects a fraction of jobs continuously and keeps serving the rest. This is graceful degradation, and it is the property that separates systems that wobble under load from systems that fall over.

The instinct to accept all work and sort it out later is fine when load is low, and low load is where most systems live most of the time. Stability comes from the behavior at the edge, when load is high and rising, and there the useful skill is knowing how to refuse. A queue that can say no is protecting itself. Backpressure is not the system failing. It is the system choosing which small failures to take so it never has to take the large one.