Idempotent Credit Ledgers for Refund-Then-Recover Retries

A usage-metered system refunds credits when a job fails. The uncomfortable case is when that same job later recovers on a retry and succeeds. Now the refund was wrong, and you have to reclaim the credits without charging twice. The fix is an append-only ledger of reverse-operation pairs, where every refund and every reclaim is a single conditional write guarded on the current state. This post shows the ledger shape, the atomic guard that blocks both double refunds and double charges, and how a retry queue feeds it without looping forever.

The refund-then-recover problem

In a metered system, a user action costs credits: rendering a video, running a transcription, generating an image. The common design charges up front, runs the job, and refunds on failure so the user does not pay for work they never received. That refund is where the trouble starts.

Distributed jobs do not fail cleanly. A worker times out at 30 seconds, marks the job failed, and refunds 40 credits. But the GPU that was doing the work actually finished at 32 seconds, and the result lands in storage two seconds after the timeout fired. The job is not failed. It recovered. The ledger now shows a refund for work that exists and will be delivered.

The naive implementation makes this worse. If failure handling and success handling are two independent code paths with no shared state, you get one of two bugs. Two failure signals for the same job (a timeout plus a later error callback) each refund, so the user is credited 80 for a 40-credit job. Or a refund followed by a recovery followed by a second recovery signal each reclaims, so the user is charged twice for one job. Both are correctness failures in a domain where correctness is the whole point.

The root cause is the same one behind most distributed-systems bugs: an operation that reads state and then writes state as two separate steps, with a gap in between where another worker can act on the same stale read.

Model every change as a reverse-operation pair

The first decision is to never mutate or delete a prior ledger entry. The ledger is append-only. A job that is charged, refunded, then reclaimed produces three immutable rows:

• charge: -40
• refund: +40
• reclaim: -40

The user's balance is the sum of the rows, not a mutable counter you overwrite. Each entry carries the job_id, a type, an amount, and a created_at, so the full history of every credit is auditable. When support asks why a user has the balance they have, you replay the rows.

Refund and reclaim are inverses of each other, which is what makes recovery expressible. A reclaim is only meaningful if a refund happened first and has not already been reclaimed. That condition is not something you check in application code with an if. It is the guard on the write itself.

The atomic guard that stops double refunds

The double-refund bug is a classic time-of-check-to-time-of-use race. Worker A reads is_refunded = false and decides to refund. Before it writes, worker B reads the same is_refunded = false and also decides to refund. Both write. The user is credited twice.

Closing the gap means the check and the write have to be one indivisible operation. On a document store with conditional updates, that is a single UpdateOne whose filter is the check and whose update is the write. The flag lives on the job document, and only the worker that flips it from false to true is allowed to append the ledger row.

// Refund only if this job has not already been refunded.
// The filter and the $set are one atomic operation, so two concurrent
// failure signals cannot both win.
now := time.Now()
res, err := jobs.UpdateOne(ctx,
	bson.M{"_id": jobID, "is_refunded": false},
	bson.M{"$set": bson.M{
		"is_refunded": true,
		"refunded_at": now,
	}},
)
if err != nil {
	return err
}
if res.ModifiedCount == 1 {
	// We won the guard. Append the ledger row exactly once.
	appendLedger(ctx, jobID, "refund", cost, now)
}

The worker whose ModifiedCount comes back 1 won the guard and appends the refund. Every other worker sees ModifiedCount come back 0, because the filter no longer matches once the flag is set, and does nothing. The database serializes the conditional writes for you, so no amount of concurrency produces two refunds.

Reclaiming credits when the job recovers

Recovery uses the mirror guard. When a job that was already refunded later succeeds, you reclaim the credits. The condition is that the job was refunded (is_refunded = true) and has not yet been reclaimed (is_reclaimed = false), and again the check and the write are one operation.

// Reclaim only if the job was refunded and not yet reclaimed.
now := time.Now()
res, err := jobs.UpdateOne(ctx,
	bson.M{"_id": jobID, "is_refunded": true, "is_reclaimed": false},
	bson.M{"$set": bson.M{
		"is_reclaimed": true,
		"reclaimed_at": now,
	}},
)
if err != nil {
	return err
}
if res.ModifiedCount == 1 {
	appendLedger(ctx, jobID, "reclaim", cost, now)
}

The elegant part is what happens on the ordinary success path. A job that succeeded on its first attempt was never refunded, so is_refunded is still false, the filter fails to match, ModifiedCount is 0, and no reclaim row is written. That is exactly right: there is nothing to reclaim because nothing was refunded. The success handler runs the same reclaim call unconditionally, and the guard decides whether it applies. The success path is idempotent whether or not a refund ever happened, and whether it runs once or five times from duplicate delivery.

Note that the reclaim amount comes from the recorded cost on the job, not from anything the retry reports. Pinning the amount to the original charge means a recovered job reclaims exactly what it refunded, never a different number.

State transitions of a metered job

Two boolean flags, is_refunded and is_reclaimed, define a small state machine. Every valid path through it keeps the ledger balanced, and every guard above is one edge of this machine.

The two terminal states both settle at a net effect of -cost, which is correct: the user paid for the work exactly once. The refunded-but-not-reclaimed state settles at 0, correct for work that genuinely failed and was never delivered. There is no path that reaches -2 times cost or 0 after delivery, because each edge is gated by a flag that only one writer can flip.

Re-queueing failed work with backoff

A refunded job is not always dead. If the failure was a transient dependency (a GPU node that dropped, a storage call that timed out), the job goes back on a retry queue rather than to a graveyard. Pushing to the tail of a Redis list with RPush gives a simple FIFO retry channel, and each re-queue carries an incremented attempt count and a not-before timestamp so a broken dependency does not get hammered on a tight loop.

// Re-queue a failed job with exponential backoff, capped.
attempt := job.Attempts + 1
backoff := time.Duration(
	math.Min(
		float64(baseDelay)*math.Pow(2, float64(attempt)),
		float64(maxDelay),
	),
)
job.Attempts = attempt
job.NotBefore = time.Now().Add(backoff)
payload, _ := json.Marshal(job)
rdb.RPush(ctx, "jobs:retry", payload)

The consumer skips any job whose NotBefore is in the future, re-queuing it without work, so the delay is honored without a separate delayed-queue mechanism. Exponential growth capped at maxDelay means the first retry is quick, but a persistently failing dependency backs off to a slow poll instead of a busy loop that burns the very resource it is waiting on.

A dead-letter guard stops infinite retries

Backoff alone does not stop a job that will never succeed. Without a ceiling, a permanently broken job cycles through refund, re-queue, fail, refund forever, and because the refund guard is idempotent, it does no financial damage, but it wastes workers and clutters the queue. The attempt count is also the ceiling.

// Terminal guard: past the cap, dead-letter instead of retrying.
if job.Attempts >= maxAttempts {
	rdb.RPush(ctx, "jobs:dead-letter", payload)
	return
}

The dead-letter list is a holding area for human inspection, not an automatic retry channel. Nothing consumes it on a timer. Just as important, the same is_reclaimed and is_refunded flags act as idempotency keys against duplicate delivery. If a job is delivered twice from the retry queue (at-least-once delivery guarantees this will happen), the second delivery hits a job whose flags already reflect its terminal state, and the guarded writes match nothing. Reprocessing a job that already settled cannot move the balance. The queue can deliver a message any number of times, and the ledger stays correct because correctness lives in the conditional writes, not in the delivery guarantee.

The tradeoff: ledger complexity against correctness

The simple model is one line of logic: failure equals refund, done. Fewer states, less code, nothing to reason about. It is also wrong the moment a failure is not final, which in a distributed system is often. Every worker timeout that races a slow success, every duplicate callback, every retry that recovers, breaks the simple model quietly, and quiet billing bugs are the expensive kind.

The ledger model costs you two flags, a pair of reverse operations, a retry queue with backoff, and a dead-letter path. That is real complexity, and it is not free to hold in your head. The question is whether the domain earns it. For a free-tier feature where miscounting a handful of credits harms no one, skip all of it and eat the occasional drift. For paid credits, where a balance is money-shaped and a user noticing a double charge is a support ticket and a trust problem, the append-only audit trail and the atomic guards pay for themselves the first time a job you already refunded comes back to life. The rule that makes the whole thing work is small: make every ledger operation idempotent, and gate every state transition on an atomic guard that only one writer can pass.