Why Atomic Counters Drift and How Reconciliation Fixes It

A derived counter maintained with atomic increments will drift from the source of truth. Not because the increment is wrong, but because atomicity guarantees one operation, not agreement with a separate record. Crashes, retries, and partial failures each leave the number a little off, and the errors accumulate. The fix is not a bigger lock. It is to treat the counter as a fast cache and run a periodic job that recomputes the true value from the source and writes it back, so the number is eventually correct even when it is briefly wrong.

What drift is, and why atomicity does not prevent it

Say you store comments as rows, and you want to show a comment count on every post without scanning the comment collection on each page load. The obvious optimization is a derived counter: keep a comment_count field on the post and bump it whenever a comment is created.

// Fast path: bump the derived counter whenever a comment is created.
// This single update is atomic for this one document. Nothing here ties
// it to the actual number of comment rows that exist.
_, err := counters.UpdateOne(ctx,
    bson.M{"_id": postID},
    bson.M{"$inc": bson.M{"comment_count": 1}},
    options.Update().SetUpsert(true),
)

The $inc is atomic. Two concurrent increments will not lose an update the way a read-modify-write would. So it is tempting to conclude the counter is correct. It is not, and the reason is a category error about what atomicity buys you.

Atomicity is a property of a single operation. It says this one increment happens completely or not at all, with no interleaving that corrupts the value. It says nothing about whether the sequence of increments matches the number of rows in another collection. The counter and the comment rows are two separate pieces of state, updated by two separate operations. Consistency between them is not a property either operation has. Any time those two writes are not part of one atomic unit, and across a document store and separate collections they usually are not, the door is open for them to disagree. That disagreement is drift.

Where the drift comes from

Drift is not one bug. It is a family of small gaps between the row write and the counter write, and each one pushes the count in a predictable direction. Knowing the sources tells you what the reconciliation job has to correct and what the metric should look like when a real bug appears.

Two things stand out. First, the errors do not cancel. A system that both drops some increments and double-applies others does not average out to correct. It lands somewhere wrong. Second, the direction of the error is a diagnostic. A counter that only ever runs high points at duplicate processing or a missing decrement. One that only runs low points at a write path that fails after the row commits. The reconciliation job that fixes the number can also tell you which of these you have, if you record the gap before you close it.

Treat the counter as a cache, not the source of truth

The mental model that makes all of this manageable is to stop thinking of the counter as a fact. It is a cache of a fact. The fact is the set of comment rows. The counter is a precomputed answer to a question you did not want to run on every page load.

Once the counter is a cache, two rules follow. A cache is allowed to be stale, so a briefly wrong count is acceptable rather than a crisis. And a cache must have a way to be rebuilt from the thing it caches, because a cache you cannot regenerate is just unreliable primary data. The source of truth stays authoritative. It is the collection of rows, queried directly, ignoring the counter entirely. Any time you need a number you can stake a decision on, you count the rows. The counter is for cheap reads where a small transient error costs nothing.

This reframing is what makes the whole approach honest. You are not promising an always-correct counter and quietly failing to deliver it. You are promising a fast approximate counter plus a source of truth you can always fall back to, and a job that keeps the two close.

The reconciliation job that recomputes and overwrites

Reconciliation is a scheduled job, off the hot path, that recomputes the true count from the rows and writes it into the counter. Because it runs on a timer rather than on every request, it is allowed to do the expensive thing the hot path was avoiding, which is actually counting.

The naive version is three lines: count the rows, write the number, done.

// Naive reconciliation: recompute from the source of truth and overwrite.
// Correct in isolation, but see the race in the next section.
trueCount, err := comments.CountDocuments(ctx, bson.M{
    "post_id":    postID,
    "deleted_at": bson.M{"$exists": false},
})
if err != nil {
    return err
}
_, err = counters.UpdateOne(ctx,
    bson.M{"_id": postID},
    bson.M{"$set": bson.M{
        "comment_count": trueCount,
        "reconciled_at": time.Now(),
    }},
)

This corrects every kind of drift at once, over and under, because it does not trust the old value at all. It rederives the number from the rows. Whatever the old counter said, right or wrong, is discarded.

There is one hazard, and it is the reason the naive version is not the final version. Between the moment CountDocuments returns and the moment the $set lands, new comments can arrive. The hot path increments the counter for them. Then the $set overwrites the counter with a number that was computed before those increments existed, and they are lost. The reconciliation, meant to fix drift, has just created some.

Reconcile against a watermark so in-flight writes survive

The clean way to avoid clobbering concurrent writes is to reconcile only a settled prefix of the data and leave recent writes alone. This needs a small change to how the counter is stored. Split it into two fields: a base_count that reconciliation owns, and a live_delta that the hot path owns. The displayed number is their sum.

The hot path stops touching the reconciled value. It only ever increments the live delta.

// Hot path now only touches live_delta. Reconciliation never overwrites
// this field, so a concurrent increment can never be clobbered.
_, err := counters.UpdateOne(ctx,
    bson.M{"_id": postID},
    bson.M{"$inc": bson.M{"live_delta": 1}},
    options.Update().SetUpsert(true),
)

Reads add the two fields:

displayed := doc.BaseCount + doc.LiveDelta

Reconciliation picks a watermark, a timestamp far enough in the past that no new row will ever land with an older creation time. In practice that means older than your write-visibility lag and older than the oldest open transaction, so a few seconds is usually plenty. It counts the truth up to that watermark, installs it as the new base, and removes from the live delta exactly the increments the base now accounts for. All of that write happens in one atomic update so the fields never disagree.

// Settled boundary: no new row can appear with a timestamp older than this.
watermark := time.Now().Add(-30 * time.Second)

baseTrue, err := comments.CountDocuments(ctx, bson.M{
    "post_id":    postID,
    "deleted_at": bson.M{"$exists": false},
    "created_at": bson.M{"$lte": watermark},
})
if err != nil {
    return err
}

// Atomic pipeline update. Install the recomputed base, and subtract from
// live_delta the number of rows the base has just absorbed (baseTrue minus
// the old base_count). Increments for rows after the watermark stay in
// live_delta untouched.
_, err = counters.UpdateOne(ctx,
    bson.M{"_id": postID},
    bson.A{
        bson.M{"$set": bson.M{
            "live_delta": bson.M{"$subtract": bson.A{
                "$live_delta",
                bson.M{"$subtract": bson.A{baseTrue, "$base_count"}},
            }},
            "base_count":    baseTrue,
            "reconciled_at": "$$NOW",
        }},
    },
)

The reason this is safe is that reconciliation and the hot path now write disjoint fields. The base is rederived from the rows, so any drift that accumulated in the old base is wiped. The live delta holds only the very recent increments, which the next cycle will fold into the base once they too are past the watermark. Concurrent writes during reconciliation land in the live delta and are never in the path of the overwrite. The counter converges without a lock and without pausing writes.

If you want the simplest possible version and can tolerate a slightly larger transient error, keep the single-field overwrite and just accept that a write racing the reconcile might be briefly lost and then corrected on the next run. That is a legitimate choice for very low-write counters. The watermark and the split fields are what you reach for when the write rate is high enough that clobbering matters.

Measure the drift so it surfaces real bugs

The number reconciliation throws away is more valuable than the number it writes. Before you install the new base, you know the old value and the true value. The difference is the drift, and it is direct evidence about the health of your write path.

drift := baseTrue - oldBaseCount
metrics.Observe("counter_drift", float64(drift),
    "collection", "comments")

Emit that as a gauge or a histogram, tagged by counter type. Now you have a signal, and it tells a story. Drift that hovers near zero and stays there means the hot path is basically correct and reconciliation is just cleaning up rare crashes. Drift that grows between runs means increments are being lost or double-applied faster than you thought, and the sign tells you which. A sudden spike lines up with a deploy or an incident and points straight at the change that broke the write path.

Without this, reconciliation hides your bugs. It quietly papers over a broken increment path every night, and you never learn the path is broken because the displayed number always looks fine by morning. The metric turns a silent correction into an alert. The job still fixes the symptom, but now it also reports the disease. If drift ever crosses a threshold you care about, page someone, because at that point the counter is drifting faster than a nightly job can safely mask.

When an approximate counter is not enough

This entire approach trades instantaneous accuracy for cheap reads and self-healing. That trade is right for a large class of counters and wrong for a specific one, and the line between them is money.

For statistics, rankings, and display counts, view totals, comment counts, like tallies, follower numbers, a count that is off by a few for a few minutes is invisible to users and costs nothing. Forcing those to be exact and real time in a distributed system means a transaction or a lock on the hot path of your busiest writes, and the cost of that grows with traffic. A fast approximate counter with periodic reconciliation is the pragmatic answer, and eventually correct is a strong enough guarantee.

For anything where the count gates a decision with real consequences, the calculus flips. A wallet balance, a paid quota you enforce, inventory you can oversell, a credit ledger that must not go negative: these cannot be a lazily reconciled cache. For those, the source of truth belongs on the read path. You count or you read the authoritative balance at decision time, or you make the decrement itself transactional against the constraint. A counter that is briefly wrong is fine for a like tally and unacceptable for an amount of money.

There is also an operational cost to weigh. Counting rows is not free, and a reconciliation job that recomputes every counter over a huge collection can become its own load problem. The usual answers are to reconcile only documents that changed since the last run, to window the scan by time, to stagger it so you are not counting everything at once, and to run it often enough that drift stays small but rarely enough that the scan stays cheap. That tuning is the real work of running reconciled counters. The design is simple. Keeping the job affordable as the data grows is where the engineering goes.