Content-Hash Keys for Media, Immutable and Deduplicated

When you upload an image to object storage, the obvious key is its filename: img/hero.png. It reads well and it is easy to reason about, but it quietly signs you up for cache invalidation, duplicate objects, and URLs baked into your database. There is a different key that avoids all three. Instead of naming the object after the file, name it after the bytes. Compute the sha256 of the file content and store it at img/{sha256}.{ext}. The content decides the key, not the name. This post is the full case for that choice, the Go that implements it, and the one case where a filename key is still the right call.

How a content-hash key works

The rule is one line: the storage key is a hash of the file content. Read the bytes, run sha256 over them, hex-encode the digest, and put the object at img/{digest}.{ext}. A file whose contents hash to 9f86d0... lands at img/9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08.png.

Two properties fall out of this immediately, and everything else in this post is a consequence of them.

• The same bytes always produce the same key. Upload the identical file twice, from two different sources, with two different original names, and both land on the exact same object.
• Different bytes produce a different key. Change a single pixel and the digest changes completely, so the edited file is a new object at a new key, and the old key still points at the old bytes.

This is content addressing. The address of the data is derived from the data. Git does the same thing with blobs, and for the same reasons.

Immutability: cache forever, invalidate for free

Because a key can only ever refer to one exact sequence of bytes, the object at that key never changes. That single fact unlocks the strongest caching header the web has:

Cache-Control: public, immutable, max-age=31536000

max-age=31536000 is one year. immutable tells the browser it does not even need to revalidate with an If-None-Match when the user reloads. Both are safe here because the promise they encode is actually true: the bytes behind this URL will not change for as long as the object exists. A CDN in front of R2 or S3 caches it once and serves it from the edge until eviction, and the origin sees almost no repeat traffic for that file.

Now compare the invalidation story. With a filename key, updating hero.png means overwriting the object, and every cache that already holds the old version is now wrong. You have to purge the CDN, and you have to hope no browser is holding a stale copy under that same URL. With a content-hash key you never overwrite anything. The edited image is a new key, so it is a new URL, and the new URL was never cached because it did not exist a moment ago. The old URL keeps serving the old bytes to anyone who still references it. Cache invalidation stops being an operation you perform. It becomes something that cannot happen, because URLs never change meaning.

Deduplication: HeadObject, then maybe PutObject

If identical bytes always map to the same key, then storing the same file twice is storing it once. The upload path checks for the object before writing it:

1. Compute the content-hash key.
2. HeadObject on that key.
3. If it exists, do nothing. The bytes are already there.
4. If it does not exist, PutObject.

This makes the upload idempotent. Running it again is cheap and safe, which matters when an ingest job retries or when the same asset appears in three different posts. A shared logo referenced across forty articles is one object, not forty. You pay for the HeadObject, which is a metadata call and far cheaper than a redundant upload of the full body.

// putImmutable stores b under a content-hash key, skipping the upload
// if an object with that key already exists.
func putImmutable(ctx context.Context, cli *s3.Client, bucket string, b []byte, ext string) (string, error) {
	sum := sha256.Sum256(b)
	key := fmt.Sprintf("img/%s.%s", hex.EncodeToString(sum[:]), ext)

	_, err := cli.HeadObject(ctx, &s3.HeadObjectInput{
		Bucket: &bucket,
		Key:    &key,
	})
	if err == nil {
		return key, nil // already present, dedup hit
	}
	var nf *types.NotFound
	if !errors.As(err, &nf) {
		return "", fmt.Errorf("head %s: %w", key, err)
	}

	_, err = cli.PutObject(ctx, &s3.PutObjectInput{
		Bucket:       &bucket,
		Key:          &key,
		Body:         bytes.NewReader(b),
		ContentType:  aws.String(contentType(ext)),
		CacheControl: aws.String("public, immutable, max-age=31536000"),
	})
	if err != nil {
		return "", fmt.Errorf("put %s: %w", key, err)
	}
	return key, nil
}

Note that HeadObject returns a typed NotFound error for a missing key, and everything else is a real failure you should not swallow. Treating any error as absence would turn a transient network fault into a duplicate upload, which defeats the point.

Separate the storage from the database

The third payoff has nothing to do with caching. It is about where the URL lives.

A common mistake is to store the full media URL in the database: https://media.example.com/img/9f86d0....png sitting in a column next to the post. It works until the day you change hosts, rename a bucket, or move a domain. Then every one of those absolute URLs is wrong, and fixing them means a migration over your content.

The content-hash key gives you a clean split. The database stores only the key, img/9f86d0....png, and never the host. The absolute URL is assembled at render time from an environment variable:

func mediaURL(base, key string) string {
	return strings.TrimRight(base, "/") + "/" + key
}

// base comes from MEDIA_BASE_URL, e.g. https://media.example.com
url := mediaURL(cfg.MediaBaseURL, post.HeaderKey)

Moving off R2 onto a different provider, or putting a new CDN in front, is now a two-step operation that touches no rows and no article bodies. Copy the objects to the new location, then change MEDIA_BASE_URL. The keys are identical on both sides because they are derived from content, so the same key resolves to the same bytes wherever it is hosted. Your database does not know or care which bucket is serving the files today.

The upload pipeline, end to end

Here is how the pieces fit in an ingest step that processes a Markdown post. The author writes relative references like ![diagram](images/flow.svg), and the pipeline rewrites them into content-hash keys as it uploads.

1. Parse the post and collect every local image reference.
2. For each referenced file, read the bytes and compute the content-hash key.
3. HeadObject, then PutObject only on a miss, with the immutable cache header.
4. Replace the relative reference in the body with the stored key.
5. Persist the post. The body and the database now carry keys, never absolute URLs.

The rewrite step is what keeps the author experience simple. Writers deal in relative paths that make sense in a repo, and the pipeline is the only place that knows about hashing and buckets.

func processBody(ctx context.Context, up *Uploader, body string, refs []ImageRef) (string, error) {
	for _, ref := range refs {
		b, err := os.ReadFile(ref.LocalPath)
		if err != nil {
			return "", fmt.Errorf("read %s: %w", ref.LocalPath, err)
		}
		key, err := up.PutImmutable(ctx, b, ref.Ext)
		if err != nil {
			return "", err
		}
		body = strings.ReplaceAll(body, ref.Original, key)
	}
	return body, nil
}

The tradeoff: orphans and no in-place update

Content addressing is not free, and the same property that makes it good makes it awkward in one specific way. Because any change to the bytes produces a new key, you cannot update a file in place. The concept of overwrite does not exist. Edit an image and you get a new object at a new key. The old object stays exactly where it was.

If nothing references the old key anymore, that object is an orphan. It costs storage and nobody reads it. Over time, edited and replaced assets accumulate as dead weight. There is no automatic cleanup, because storage cannot tell whether some other post, or an old cached page, still points at that key. You need a garbage collector: periodically list the objects in the bucket, list the keys actually referenced across your live content, and delete the objects that appear in the first set but not the second. Run it on a schedule, and give orphans a grace period before deletion so that a caller mid-flight does not get a 404.

The table below lays out the two models side by side.

The last two rows are the cost. Everything above them is the benefit. Whether the trade is worth it depends on how often your files actually change and how much you value the caching and dedup you get in return.

When a filename key is still the better choice

Content addressing wins for media that is written once and read many times, which describes almost all images, video, fonts, and static downloads on a content site. It is the wrong tool when the identity of a slot matters more than the identity of the bytes.

Reach for a stable filename key when:

• You want one canonical URL that always serves the latest version, and you would rather purge a CDN than mint a new URL. A logo/current.svg that marketing overwrites in place, with callers who must not update their references, is a filename job.
• The path itself carries meaning your system depends on, like users/{id}/avatar.png, where the location is a lookup key and there is exactly one live object per slot. Content addressing would scatter each avatar edit across new keys and leave you tracking which one is current anyway.
• Objects are large, mutable, and edited constantly, so the orphan pile would grow faster than any reasonable GC could justify.

For those cases, accept the invalidation work and keep the name. For everything that is effectively write-once, let the content pick the key. You trade an in-place update you rarely needed for immutability, dedup, and a database that survives a storage move without a single row changing.