How Cloud Run CPU Throttling Breaks Background Goroutines

A Go service that ran fine on a VM started losing its lease locks the day it moved to Cloud Run. The lease heartbeat, a plain time.Ticker firing every thirty seconds, would go quiet for minutes at a time. Nothing in the code changed. The cause is a Cloud Run default that most guides mention in one line and then move past: by default your container gets CPU only while it is serving a request. Between requests the CPU is throttled to almost nothing, and any goroutine you expected to keep running in the background stops running with it.

If you have background loops on Cloud Run, tickers, cache refreshers, heartbeats, queue consumers, this is the single behavior most likely to break them in a way that never shows up locally. This post explains the mechanism, shows the failure with Go code, and walks through four ways to fix it with the tradeoffs of each.

How Cloud Run allocates CPU

Cloud Run has two CPU allocation modes, and the default is the one that surprises people.

The default is "CPU allocated during requests" (sometimes called request-based billing or CPU throttling). Your container is given full CPU while it is handling at least one request. The moment the last in-flight request finishes, Cloud Run throttles the CPU to a tiny fraction, close to zero. The container is still alive and its memory is intact, but almost no CPU cycles are scheduled. You are billed only for the CPU used during requests, which is why this mode is cheaper.

The other mode is "CPU always allocated" (instance-based billing). The container keeps full CPU for its entire lifetime, whether or not a request is in flight. You pay for the whole time the instance exists, and in exchange background work runs normally.

Locally and on a plain VM, your process always has CPU, so background goroutines just work. The default Cloud Run mode removes that assumption without changing a line of your code, which is exactly why the failure is confusing. The program is correct. The scheduling environment underneath it changed.

What actually breaks between requests

Anything that assumes steady background progress is at risk. The common cases:

• A time.Ticker that fires on a fixed interval to refresh an in-memory cache. The cache goes stale because the refresh goroutine is starved of CPU between requests.
• A lease or lock heartbeat that renews "every N seconds". If the renewal goroutine does not get scheduled, the lease expires and another worker steals it.
• A metrics or log flusher that batches in memory and flushes on a timer. The flush is delayed until the next request happens to wake the instance.
• A background queue or Pub/Sub pull consumer that loops waiting for messages. It processes nothing while no request is inbound.

The important detail about time.Ticker is that the timer itself is not the problem. The tick is delivered based on wall-clock time, and the value can even sit buffered in the channel. The problem is that the goroutine waiting on that channel needs CPU to wake up and run the work, and that CPU is exactly what throttling takes away. So the tick "fires" on schedule, but the handler runs late, or in bursts, or only when a request happens to bring the instance back to full CPU.

The heartbeat case is the sharpest because it is silent and it corrupts shared state. You are not just late. You lose a lock you thought you held.

A heartbeat that expires its own lease

Here is the shape of the code that broke. A worker acquires a lease, then starts a goroutine to renew it on a timer while it does a long job.

func (w *Worker) run(ctx context.Context) error {
	if err := w.store.AcquireLease(ctx, w.id, 90*time.Second); err != nil {
		return err
	}
	go w.heartbeat(ctx) // renew the lease in the background
	return w.processLongJob(ctx)
}

func (w *Worker) heartbeat(ctx context.Context) {
	t := time.NewTicker(30 * time.Second)
	defer t.Stop()
	for {
		select {
		case <-ctx.Done():
			return
		case <-t.C:
			// Needs CPU to run. Under request-based throttling,
			// this may not be scheduled until the next request.
			_ = w.store.RenewLease(ctx, w.id, 90*time.Second)
		}
	}
}

On a VM this is fine. On Cloud Run with the default allocation, if processLongJob is itself waiting on I/O and no new requests arrive, the instance is throttled, the heartbeat goroutine does not get scheduled, and after ninety seconds the lease expires. Another instance sees an expired lease and picks up the same job. Now two workers run the same task, which is the exact failure the lease was meant to prevent.

The root confusion is that the code looks concurrent and correct, and it is, under an environment that gives goroutines CPU. Cloud Run's default does not promise that between requests. There are two ways out: change the environment so background goroutines get CPU, or change the design so you do not depend on a background goroutine at all. The next four sections cover both.

Option 1: CPU always allocated with a minimum instance

The most direct fix is to stop throttling. Set the instance to keep CPU at all times and keep at least one instance warm.

gcloud run services update my-service \
  --no-cpu-throttling \
  --min-instances=1

--no-cpu-throttling switches the service to instance-based CPU, so goroutines run between requests. --min-instances=1 keeps at least one instance alive so there is always a container for the background loop to run in. You need both. Always-allocated CPU on an instance that scaled to zero still has no loop running, because the instance no longer exists.

The tradeoff is cost. You now pay for CPU and memory for the whole time the instance is up, not just during requests, and a min instance means you are paying twenty-four hours a day even at zero traffic. For a service that genuinely needs a resident background loop, this is the honest price and it is often small. For a service that added a background goroutine out of habit, it is paying continuously to keep alive a thing that did not need to be resident. Before reaching for this, ask whether the work really needs to run outside of requests at all. If it does, this is the clean answer.

Option 2: Move background work into the request path

If the background job exists only to keep something fresh for the next request, you may not need a background loop at all. Do the work when a request arrives, guarded so it runs at most once per interval.

type cache struct {
	mu        sync.Mutex
	data      []Item
	refreshed time.Time
}

func (c *cache) get(ctx context.Context, refresh func(context.Context) ([]Item, error)) ([]Item, error) {
	c.mu.Lock()
	defer c.mu.Unlock()
	if time.Since(c.refreshed) < 60*time.Second {
		return c.data, nil // still fresh, no work
	}
	data, err := refresh(ctx)
	if err != nil {
		if c.data != nil {
			return c.data, nil // serve stale on refresh failure
		}
		return nil, err
	}
	c.data, c.refreshed = data, time.Now()
	return c.data, nil
}

Now the refresh is triggered by traffic, which is exactly when CPU is guaranteed. The timer becomes a staleness check instead of a scheduler. This suits caches, small periodic recomputations, and anything whose only consumer is the request itself.

It does not suit work that must happen on a schedule regardless of traffic. If no request arrives for an hour, nothing refreshes for an hour. For a cache that is acceptable, because the first request after the gap pays a one-time refresh. For a task that must fire at a specific time, it is not acceptable, and you want one of the next two options.

Option 3: Split periodic work into a Cloud Run Job and Cloud Scheduler

For anything that is genuinely cron-shaped, run on a schedule, do a unit of work, exit, the correct model is not a long-lived loop inside a request-serving service. It is a Cloud Run Job triggered by Cloud Scheduler.

A Cloud Run Job runs a container to completion rather than serving requests, and it always has CPU for its whole run. Cloud Scheduler invokes it on a cron expression. Your service stays lean and stateless, and the batch work lives in its own place with its own timeout and retry policy.

# Deploy the periodic work as a Job (runs to completion, always has CPU)
gcloud run jobs create refresh-job \
  --image=REGION-docker.pkg.dev/PROJECT/repo/refresh:latest \
  --task-timeout=600 --max-retries=1

# Fire it every 10 minutes with Cloud Scheduler
gcloud scheduler jobs create http refresh-schedule \
  --schedule="*/10 * * * *" \
  --uri="https://REGION-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT/jobs/refresh-job:run" \
  --http-method=POST \
  --oauth-service-account-email=[email protected]

This separates the request-response model from batch work cleanly. The Job gets CPU because it is not subject to request-based throttling at all. It scales to zero when idle and costs nothing between runs. Scheduler gives you real cron semantics, retries, and a run history you can inspect.

The tradeoff is granularity and coupling. Cloud Scheduler's finest interval is one minute, so this is not the tool for something that must happen every few seconds. And the Job runs in a separate process, so any state it needs must live in a database or cache both sides can reach, not in the service's memory. For most genuinely periodic maintenance, refreshing a materialized view, expiring old records, sending a digest, this is the model that matches the shape of the work.

Option 4: Event-driven with Pub/Sub push

If the background loop was really consuming a queue, invert it. Instead of a goroutine that pulls messages in a loop, let Pub/Sub push each message to an HTTP endpoint on your service. Each delivery is a normal request, so it gets CPU, and there is no resident loop to starve.

// Pub/Sub push delivers one message per HTTP POST.
// This runs as a request, so it always has CPU.
func handlePush(w http.ResponseWriter, r *http.Request) {
	var env struct {
		Message struct {
			Data []byte `json:"data"`
		} `json:"message"`
	}
	if err := json.NewDecoder(r.Body).Decode(&env); err != nil {
		http.Error(w, "bad envelope", http.StatusBadRequest)
		return
	}
	if err := processMessage(r.Context(), env.Message.Data); err != nil {
		http.Error(w, "retry", http.StatusInternalServerError) // Pub/Sub redelivers
		return
	}
	w.WriteHeader(http.StatusNoContent) // ack
}

Return a 2xx to acknowledge, or a 5xx to make Pub/Sub redeliver later. The subscription's retry and dead-letter policy handles failures, so you delete a whole class of hand-rolled retry loop. The service can scale to zero and wake on the next message.

The cost is that push delivery is per message over HTTP, which is different from a pull consumer that batches. If you need high-throughput batching or ordered pull processing, a pull model on a component with always-allocated CPU fits better. But for reacting to events one at a time, push turns background work into ordinary request handling, which is the model Cloud Run's default is built for.

Which option fits, and when always-on CPU is right

The deciding question is not "how do I keep my goroutine alive". It is "does this really need a resident background loop, or can it be request-driven or event-driven instead". Most background loops that break on Cloud Run were never required to be resident. They were a convenient way to do work that fits better as a request, a scheduled job, or an event handler.

Reach for always-allocated CPU (Option 1) when the work genuinely cannot be expressed as a request or an event and must run continuously inside the serving process. A sub-second internal tick, a live in-memory aggregation, a streaming pull consumer that must keep a connection open, a WebSocket fan-out. These need CPU between requests by their nature, and paying for a warm instance is the correct call. It is a decision with a bill attached, not a default to flip because a ticker misbehaved.

For everything else, the throttling is not a bug to work around. It is a signal that the work belongs outside the request-serving path. A lease heartbeat is a good final example. Even with always-on CPU, the safest version does not depend on the heartbeat firing at all. Make the lease expiry-based with a compare-and-swap on the deadline, so a stale lease is reclaimable even if a renewal is missed. Push the correctness into the data, so a process that briefly loses CPU cannot corrupt shared state. Cloud Run's default forces that discipline earlier than a VM ever would, and the systems that come out of it tend to be the sturdier for it.