Deterministic ULIDs for Idempotent Upserts

When a database forbids automatic write retries, every write you send has to be idempotent on its own, because nothing else is going to dedupe it for you. The technique this post is about is a deterministic ULID: a document id derived from the content that identifies the document, rather than from a random source. It costs almost nothing to adopt, it makes retries safe by construction, and it comes with exactly one rule you must never break. This is the full derivation, the failure it prevents, the invariant that keeps it correct, and the cases where it does not apply.

Why random ids are dangerous under manual retries

A ULID is a 128-bit identifier: a timestamp in the high bits, then randomness in the low bits. The timestamp prefix means ULIDs sort roughly by creation time, which is genuinely useful for a primary key. The random suffix is what makes each one unique, and it is also what makes them dangerous the moment you retry a write by hand.

Picture an insert that uses a fresh random id. The write leaves your service, the database applies it, and then the acknowledgement is lost on the way back. Your code, correctly, does not know whether the write landed, so it retries. The retry carries a new random id, because you generated a new ULID. Now there are two documents describing the same logical thing, with two different ids, and nothing in the system can tell they are duplicates. You did everything right and still corrupted your data.

In real MongoDB, retryable writes paper over this. The driver attaches a transaction number, the server deduplicates, and your retry is safe. But some engines do not support that mechanism and require retryWrites=false, at which point the safety net is gone and the responsibility is yours. The random id, which was fine when the server was deduping for you, becomes a liability the instant you are the one retrying.

Deriving the id removes the ambiguity entirely

The fix is to stop generating the id from randomness and start deriving it from the fields that identify the document. If the same logical document always produces the same id, then a retried write targets the same id, and an upsert turns it into a harmless overwrite instead of a second row.

For a multilingual blog, a post is uniquely identified by its slug, its language, and its publication instant. So we build the ULID from exactly those:

// PostID derives a stable ULID from the fields that identify a post.
// The same (slug, lang, publishedAt) always yields the same id, so an
// upsert is a no-op on every retry instead of a duplicate insert.
func PostID(slug, lang string, publishedAt int64) string {
	sum := sha256.Sum256([]byte(slug + ":" + lang))
	return buildULID(uint64(publishedAt), sum[:10])
}

Two design choices are load-bearing here. The timestamp portion of the ULID comes from publishedAt, not from the wall clock at write time, so the id does not change if you re-ingest the same post tomorrow. The random portion is replaced by the first ten bytes of a SHA-256 hash of the identifying fields, so it is deterministic but still spreads ids across the keyspace to avoid hot-spotting on a single index range. The result is an id that still looks and sorts like a normal ULID, still carries a meaningful timestamp prefix, but is reproducible from the content alone.

With that id in hand, the write becomes an upsert keyed by it:

// First call inserts, every retry overwrites identical data.
// Row count never grows, no server-side dedup required.
_, err := coll.UpdateByID(ctx, PostID(slug, lang, publishedAt),
	bson.M{"$set": doc},
	options.Update().SetUpsert(true),
)

Run this once and it inserts. Run it a hundred times and the database holds exactly one document, because every call after the first is writing the same fields to the same id. The operation is idempotent with no coordination, no transaction, and no dedupe table.

Idempotency is a property of the whole pipeline, not one call

It is tempting to stop at the upsert and declare victory, but idempotency has to hold end to end or it does not hold at all. If the id is stable but the document body includes a freshly generated timestamp or a random field, then two runs produce two different bodies for the same id, and while you will not get a duplicate row, you will get needless writes and a document that changes every time the pipeline runs. That churns your change feed, invalidates caches, and makes it impossible to tell a real edit from a no-op.

So the discipline extends to the payload. Anything in the document that is derived from the identifying content should itself be deterministic. A content hash of the source is a good example: compute it from the input, store it on the document, and skip the write entirely when the stored hash matches the incoming one. Now a re-run of an unchanged post is not just a safe overwrite, it is no write at all.

// Skip the write when nothing changed. Idempotent and cheap.
if existing != nil && existing.SourceHash == incoming.SourceHash {
	return nil // unchanged, nothing to do
}

The mental model that keeps this correct: same input, same id, same body, and ideally no write when the input has not changed. Each layer of that reinforces the one above it.

The one invariant you must hold

The id is derived from publishedAt, which means publishedAt must be immutable for the life of the document. This is the single rule the whole scheme rests on, and breaking it is subtle enough to deserve its own warning.

Suppose a post was published, its id was computed from its publication date, and it is sitting in the database. Later, someone edits the publication date in the source. The next ingest computes a new id, because the input to the derivation changed. The upsert with the new id inserts a brand new document, and the old one, under the old id, is now an orphan that no code will ever update or delete again. You did not overwrite the post. You forked it.

Guard against this explicitly rather than trusting authors to remember. On ingest, look up the existing document by its identifying fields, recompute the id, and if the stored id and the freshly computed id disagree, stop with a loud error instead of writing.

// If publishedAt changed, the derived id changed, and a blind upsert would
// orphan the old document. Refuse to proceed.
if existing != nil && existing.ID != PostID(slug, lang, publishedAt) {
	return fmt.Errorf("publishedAt is immutable for %s/%s: changing it orphans the old document", slug, lang)
}

Editing the body is always fine, because the body is not part of the id. Only the fields that feed the derivation are frozen. Make that boundary explicit in your authoring docs so the rule is a known constraint rather than a trap people find by accident.

When a deterministic id is the wrong choice

This technique is not universal, and using it where it does not fit creates its own problems. It works when a document has a natural identity, a set of fields that genuinely define which document it is. A blog post has that: slug, language, date. A user account has it: an email or an external id.

It does not work when documents have no natural key and each one is a distinct event. An append-only log of independent occurrences, a stream of clicks, a queue of jobs where two identical-looking entries are legitimately two different things, all of these want unique random ids precisely because you do not want a second identical event to overwrite the first. For those, reach for a normal random ULID and, if you need retry safety, get it from an idempotency key carried in the request rather than baked into the primary id.

The test is a single question: if two writes carry identical content, should the second replace the first or sit beside it? If replace, derive the id. If sit beside, keep it random. Answer that per collection, not once for the whole system.

What you get for the effort

The effort is close to zero, a hash and a helper function, and the payoff is that a whole class of distributed-systems bug simply cannot occur. Retries are safe because they converge on the same id. Re-ingesting unchanged content is a cheap no-op because the body and hash are stable. And the database forbidding automatic write retries stops being a limitation you fight and becomes a constraint you have already satisfied, because your writes were idempotent to begin with. The database's strictness and your id scheme end up pointing in the same direction.