Lease Locks with CAS and Heartbeat for Single-Runner Jobs

Some jobs must run one at a time. An ingest that rebuilds an index, a nightly batch that charges accounts, a migration that mutates rows in place: run two copies concurrently and you get double writes, corrupted state, or a race no test will reproduce. The naive fix is a lock, and the naive lock has a fatal flaw. If the process holding it dies, the lock is held forever and the job never runs again. A lease fixes that by expiring. This post builds a lease lock with compare-and-swap acquisition and a heartbeat, and walks the failure modes that force each part of the design.

Why a plain lock is not enough

The instinct is to set a flag: write locked = true before the job, delete it after. It works right up until the process crashes in the middle. Now the flag is stuck at true, no other worker will start, and the job is dead until a human notices and clears it by hand. You have traded a concurrency bug for an availability bug, and the availability bug wakes you up at night.

The core problem is that a plain lock has no concept of the holder still being alive. It records that someone took the lock, not that someone is still working. Any lock you cannot recover from a dead holder is a deadlock waiting for the wrong moment. What you actually want is a lock that a healthy worker keeps, but that a crashed worker automatically gives up. That is a lease.

A lease is a lock with a deadline

A lease is a lock that expires unless it is renewed. Instead of "worker A holds this," it records "worker A holds this until time T." If A is alive, it keeps pushing T into the future. If A dies, T arrives, the lease is considered free, and another worker can take it. The lock recovers itself with no human in the loop.

Three operations make up the whole mechanism:

• Acquire. Take the lease only if it is free or expired, and record who took it and when it expires.
• Heartbeat. While working, periodically extend the expiry so a live holder never loses it.
• Release. On clean finish, give it up immediately so the next run does not have to wait for expiry.

The subtle part is that acquire and heartbeat both have to be atomic against every other worker trying the same thing at the same instant. That is where compare-and-swap comes in.

Acquiring with compare-and-swap

The danger during acquisition is the gap between checking and taking. If a worker reads "the lease is free" and then writes "I hold it" as two separate steps, a second worker can slip in between the read and the write, and now both believe they hold the lease. The check and the take have to be one indivisible operation.

Compare-and-swap gives you exactly that: set the value only if it still matches what you expected. On a store with an atomic conditional write, acquisition becomes a single guarded operation. Here it is against a document store, taking the lease only when there is no current holder or the current holder's lease has already expired.

// Acquire only if nobody holds the lease or the holder's lease has expired.
// The filter and the update are one atomic operation, so two workers cannot
// both win.
now := time.Now()
res, err := coll.UpdateOne(ctx,
	bson.M{
		"_id": lockKey,
		"$or": []bson.M{
			{"holder": ""},
			{"expires_at": bson.M{"$lt": now}},
		},
	},
	bson.M{"$set": bson.M{
		"holder":     workerID,
		"expires_at": now.Add(leaseTTL),
	}},
	options.Update().SetUpsert(true),
)
acquired := err == nil && (res.ModifiedCount == 1 || res.UpsertedCount == 1)

Because the condition and the write are atomic, exactly one worker can win the race for a free lease. Everyone else sees their ModifiedCount come back zero and knows to back off. The same shape works on Redis with a conditional SET key value NX PX ttl, where NX is the compare (only if absent) and PX sets the expiry in one command.

Keeping the lease alive with a heartbeat

A long job outlives any reasonable lease TTL, so the holder has to renew. The heartbeat is a timer that fires well inside the TTL and pushes the expiry forward, but only if this worker is still the recorded holder. That last condition matters: a heartbeat must never resurrect a lease this worker has already lost.

// Renew only if we are still the holder. If we lost the lease, this writes
// nothing and the ok flag tells us to stop working.
func heartbeat(ctx context.Context, coll *mongo.Collection, lockKey, workerID string, ttl time.Duration) (bool, error) {
	res, err := coll.UpdateOne(ctx,
		bson.M{"_id": lockKey, "holder": workerID},
		bson.M{"$set": bson.M{"expires_at": time.Now().Add(ttl)}},
	)
	if err != nil {
		return false, err
	}
	return res.ModifiedCount == 1, nil
}

The ratio between heartbeat interval and TTL is the tuning knob, and it is a direct tradeoff. Set the interval to roughly a third of the TTL. That leaves room for a heartbeat or two to fail transiently before the lease actually expires, so a brief network blip does not hand your lease to another worker. A short TTL recovers a crashed holder quickly but is less tolerant of blips. A long TTL is forgiving of blips but leaves the lease stuck longer after a real crash. Pick the TTL from how fast you need to recover, then set the interval to a third of it.

The failure mode heartbeats do not fix

Here is the uncomfortable truth about lease locks, and the reason they guarantee less than they appear to. Suppose the holder does not crash but simply stalls: a long garbage-collection pause, a scheduler starvation, a frozen network call. The lease expires because no heartbeat arrived. A second worker legitimately acquires it and starts running. Then the first worker wakes up, still believing it holds the lease, and does one more write before its next heartbeat discovers the truth. For that instant, two workers are live.

No timeout tuning removes this. It is fundamental: a distributed lock based on time can always be violated by a holder that pauses longer than the timeout, because the holder cannot tell the difference between "I paused" and "no time passed." What you can do is make the lease advisory for correctness rather than the only line of defense. The lease keeps the common case single-runner and cheap. The writes underneath it are made safe on their own, so that even the rare double-runner instant cannot corrupt anything.

The standard tool for that is a fencing token: a number that increases every time the lease is granted. The worker attaches its token to every write, and the resource rejects any write carrying a token older than the highest it has seen. A paused worker that wakes up holds a stale token, so its late write is refused. Combined with idempotent writes, the lease handles liveness and performance while the fencing token and idempotency handle correctness. Do not ask the lease alone to guarantee mutual exclusion, because over a network it cannot.

Releasing cleanly, and only your own lease

On a clean finish, release immediately instead of waiting for the TTL to lapse, so the next scheduled run does not sit idle waiting for an expiry that has nothing to do with it. The release has the same guard as the heartbeat: delete the lease only if you are still the holder, so a worker that already lost its lease does not delete the lease a different worker now legitimately holds.

// Release only our own lease. Never delete a lease we no longer hold.
_, err := coll.DeleteOne(ctx, bson.M{"_id": lockKey, "holder": workerID})

That holder: workerID condition is not decoration. Without it, a slow worker that lost its lease could delete the current holder's lease on its way out, and you are back to two runners. Every mutation of the lease, acquire, heartbeat, release, is conditioned on the identity of the worker performing it. That single discipline is what keeps the mechanism honest.

When you do not need any of this

A lease lock is machinery, and machinery you do not need is a liability. If your scheduler already guarantees a single instance, some managed job runners do, with a concurrency limit of one and no overlap, then you may already have exclusion and adding a lease is redundant. If the job is naturally idempotent and safe to run concurrently, running two copies is merely wasteful rather than dangerous, and a lease is optimization, not correctness.

Reach for a lease lock when three conditions hold together: the job must not run concurrently, your scheduler cannot guarantee that on its own (because retries, redeploys, or multiple triggers can overlap), and the work is expensive or mutating enough that a double run does real harm. When those line up, compare-and-swap acquisition plus a heartbeat gives you a lock that survives crashes without a human, and pairing it with fencing tokens and idempotent writes gives you the correctness the lease alone cannot promise.