Reading MP4 Metadata From a 2GB File Without Downloading It

We had a worker that needed the duration, resolution, and codec of thousands of uploaded videos, and nothing else. The naive version downloaded each file in full, read a few hundred bytes of header, and threw the rest away. For a 2GB video that is 2GB of transfer to learn four numbers. This post shows how to read that header directly over HTTP, transferring a few kilobytes instead of the whole file, by understanding how an MP4 is laid out and using Range requests to fetch only the parts that matter.

The problem: you need 200 bytes out of 2 gigabytes

Metadata like duration, width, height, and codec sits in a small structured header. The actual audio and video samples are the rest, and for a long video the rest is almost everything. A 2GB file might carry a few kilobytes of metadata and 1.999GB of media samples.

If your service processes one video occasionally, downloading the whole thing to read the header is wasteful but survivable. If a worker fleet is classifying thousands of uploads, the full download dominates everything. It burns egress from your object store, it holds a connection open for the length of a large transfer, and it puts real latency in front of a job that should finish in well under a second. The transfer time is not proportional to the work; it is proportional to the file size, which is the wrong thing to pay for.

The goal is to make the cost proportional to the metadata, not the media. To do that you need two things: a map of where the metadata lives inside the file, and a way to fetch only that region without pulling the bytes around it.

How an MP4 is laid out as boxes

An MP4 file is a flat sequence of boxes, sometimes called atoms. A box is the simplest possible container. It starts with an 8-byte header: a 4-byte big-endian size, then a 4-byte type such as ftyp, moov, or mdat. The size counts the header itself, so a box of size 8 is an empty box with only a header. After the header come the box contents, which run until the size is exhausted, and then the next box begins.

At the top level of a typical file you see a small number of boxes:

• ftyp: the file type and compatible brands. Tiny, always near the front.
• moov: the movie box. This is the metadata container. Duration, track list, resolution, codec parameters, and sample tables all live inside it. It is small relative to the media, usually kilobytes to low megabytes.
• mdat: the media data. This is the raw sample bytes for audio and video. For any real video this is the overwhelming majority of the file.

There can be others, and the order is not fixed by the format. The two facts that make cheap parsing possible are these. First, every box announces its own size in its header, so you can find where the next box starts without reading the current box's contents. Second, the metadata you want is entirely inside moov, and mdat contains nothing you need for basic metadata. So the strategy writes itself: read box headers, skip mdat by arithmetic, and only read the bytes inside moov.

Reading one box header with a Range request

Every step depends on being able to read a small byte range at a known offset. Over HTTP that is a Range request. You ask for bytes N through M, and a server that supports ranges answers with 206 Partial Content and just those bytes. Servers advertise this with an Accept-Ranges: bytes header, and object stores like S3-compatible storage support it by default.

Here is a helper that fetches an exact byte range, and one that reads and decodes a single box header from a given offset:

package mp4

import (
	"encoding/binary"
	"fmt"
	"io"
	"net/http"
)

// fetchRange returns bytes [start, start+length) from url using an HTTP Range request.
func fetchRange(client *http.Client, url string, start, length int64) ([]byte, error) {
	req, err := http.NewRequest(http.MethodGet, url, nil)
	if err != nil {
		return nil, err
	}
	end := start + length - 1
	req.Header.Set("Range", fmt.Sprintf("bytes=%d-%d", start, end))

	resp, err := client.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusPartialContent {
		return nil, fmt.Errorf("range not honored, got status %d", resp.StatusCode)
	}
	return io.ReadAll(io.LimitReader(resp.Body, length))
}

type boxHeader struct {
	size       int64 // total box size including the header
	boxType    string
	headerSize int64 // 8 for 32-bit size, 16 for 64-bit size
}

// readBoxHeader reads the 8 byte header at offset, expanding to 16 bytes for a 64-bit size.
func readBoxHeader(client *http.Client, url string, offset int64) (boxHeader, error) {
	buf, err := fetchRange(client, url, offset, 8)
	if err != nil {
		return boxHeader{}, err
	}
	size := int64(binary.BigEndian.Uint32(buf[0:4]))
	boxType := string(buf[4:8])
	return boxHeader{size: size, boxType: boxType, headerSize: 8}, nil
}

One header fetch is 8 bytes on the wire plus HTTP overhead. The overhead of the request and response headers dwarfs the 8-byte body, so in practice you pay a few hundred bytes per box header. A file with a handful of top-level boxes costs a handful of these requests. That is the whole point: the transfer is now proportional to the number of boxes, not the size of the file.

Walking the boxes and skipping mdat by arithmetic

With a header reader in hand, the top-level walk is a loop. Read the header at the current offset, look at the type, and decide what to do. If it is moov, you have found the metadata and can fetch that whole box to parse it. If it is anything else, including the huge mdat, you do not read its contents at all. You add its size to the offset and move to the next box.

// findMoov walks top-level boxes and returns the byte range of the moov box.
// It never reads the contents of mdat or any other box it skips.
func findMoov(client *http.Client, url string, fileSize int64) (start, length int64, err error) {
	var offset int64
	for offset < fileSize {
		h, err := readBoxHeader(client, url, offset)
		if err != nil {
			return 0, 0, err
		}
		if h.size < h.headerSize {
			return 0, 0, fmt.Errorf("invalid box size %d at offset %d", h.size, offset)
		}
		if h.boxType == "moov" {
			return offset, h.size, nil
		}
		// Skip this box entirely. This is the key move: for mdat we jump
		// past gigabytes of media without transferring any of it.
		offset += h.size
	}
	return 0, 0, fmt.Errorf("moov box not found")
}

The line offset += h.size is where the savings come from. When the walk reaches a 1.9GB mdat, it does not stream it, buffer it, or touch it. It reads the 8-byte header, learns the size, and adds that size to the offset. The next request lands at the box that follows mdat. The media data is skipped as pure arithmetic on offsets.

Once findMoov returns a range, one more Range request pulls the entire moov box into memory, where you parse the nested boxes to extract duration and track info. Because moov is small, reading it in full is cheap and simpler than sub-ranging inside it.

The difference in bytes transferred is stark:

The transfer for the Range walk is the size of moov plus a few box headers, regardless of how large the media is. A 200MB video and a 20GB video cost roughly the same to inspect.

The catch: moov placement decides your latency

The format does not fix where moov goes, and its placement changes everything about how fast the walk finishes.

Some files are written faststart, meaning moov is moved to the front, right after ftyp and before mdat. These files are made for streaming, because a player can read the metadata immediately without seeking to the end. For a faststart file the walk finds moov in its first or second header read, and you are done.

Other files put moov at the end, after mdat. This is common for recordings and for anything written in a single sequential pass, because the writer does not know the final size of the movie box until it has finished writing all the samples, so it appends moov last. For these files the top-level walk has to step over mdat to reach moov. The good news is that stepping over mdat is still just offset += h.size, so it costs one extra header read, not a download of the media.

There is an optimization worth knowing. If your first forward step lands on mdat, you can guess that moov is at the tail and fetch the last part of the file directly with a Range request for the final chunk, then scan backward or parse that tail region. A common heuristic is to request the last 256KB or so and look for the moov type there. This turns a two-step walk into a single tail fetch. It is an optimization, not a requirement; the plain forward walk already avoids downloading the media either way.

The one case that genuinely costs more is a mdat whose declared size is zero or extends to the end of the file, which some formats use to mean run to end. When you see that, treat the rest of the file as media and switch to the tail strategy to locate moov.

Handling 64-bit sizes and nested boxes

Two details separate a toy parser from one that survives real files.

The first is large sizes. The 32-bit size field maxes out below 4GB, which is not enough for a large mdat. The format handles this with an escape value. If the 32-bit size equals 1, the real size is a 64-bit value stored in the 8 bytes immediately after the type, making the header 16 bytes instead of 8. A size of 0 means the box runs to the end of the file. Your header reader has to check for these before trusting the 32-bit number:

func readBoxHeaderFull(client *http.Client, url string, offset, fileSize int64) (boxHeader, error) {
	buf, err := fetchRange(client, url, offset, 16) // read enough for a 64-bit header
	if err != nil {
		return boxHeader{}, err
	}
	size := int64(binary.BigEndian.Uint32(buf[0:4]))
	boxType := string(buf[4:8])

	switch size {
	case 1:
		// 64-bit largesize follows the type field.
		size = int64(binary.BigEndian.Uint64(buf[8:16]))
		return boxHeader{size: size, boxType: boxType, headerSize: 16}, nil
	case 0:
		// Box extends to the end of the file.
		size = fileSize - offset
		return boxHeader{size: size, boxType: boxType, headerSize: 8}, nil
	default:
		return boxHeader{size: size, boxType: boxType, headerSize: 8}, nil
	}
}

The second detail is nesting. moov is not a leaf. Inside it sit mvhd for the movie header with the overall duration and timescale, and one trak per track. Inside each trak is a chain of mdia, minf, stbl, and finally stsd, the sample description that names the codec. Parsing nesting uses the same header logic recursively, but on the in-memory moov bytes rather than over HTTP, because you already fetched the whole box. You walk the children the same way you walked the top level: read a header, act on the type, advance by the size. The only difference is that a container box like trak means you descend into its contents instead of skipping them.

You do not need to Range-request inside moov. The whole point of pulling moov in one fetch is that everything you want is now local, and local parsing is trivial byte-slice arithmetic.

When downloading the whole file is the better call

Range parsing is the right tool when the file is remote, large, and you only need a small header. Push any of those the other way and the calculus changes.

• The file is local, or already in memory. There is no transfer to save. Open it and read, or hand it to a library. Range logic only adds complexity.
• The server does not support Range. Check for Accept-Ranges: bytes, or send a range and confirm a 206 response. If you get 200 with the whole body, the server ignored the range and you are downloading everything anyway. Some CDNs and misconfigured origins do this.
• You are about to read most of the file regardless. If the next step is transcoding, thumbnailing every scene, or hashing the whole file, you will pull all the bytes soon. Reading the header separately just adds a round trip before the real work.
• The metadata you need is not in moov. Fragmented MP4, the kind used for adaptive streaming, spreads timing across moof boxes throughout the file. If you need per-fragment detail rather than the top-level summary, a single header fetch will not give it to you, and the access pattern is different.

For the common worker case, remote object storage, large media files, and a need for only duration and codec, the Range walk turns a job whose cost scaled with file size into one whose cost scales with metadata size. That is the difference between a fleet that can inspect thousands of videos cheaply and one that spends most of its time and egress budget moving bytes it immediately discards.