Skip to content
Blog

A fragmented MP4 muxer and a sturdier H.264 parser for joy4

Florian Mathieu
, 3 min read, filed under go, video, mp4, h264, fuzzing, open-source

Tech stack
  • Go
  • MP4
  • H.264
  • FFmpeg
Repository on GitHubRealKeyboardWarrior/joy4

joy4 is a Go library for audio and video: it reads and writes RTSP, RTMP, FLV and MP4, and comes with codec parsers for H.264 and AAC. It’s a good fit for things like camera recorders, where you pull a stream from an RTSP camera and write it to disk. I maintain a fork with two pieces of work in it: a fragmented MP4 muxer, and a more robust H.264 parser.

Why fragmented MP4

A regular MP4 file keeps its index, the moov box, at the end or the start, and that index describes every sample in the file. To write it, you need to have seen every sample. That’s fine for a file that’s finished. For a recording that’s still running, it’s a problem: until the muxer closes the file, there’s no valid index, and if the process crashes or the power goes out, the whole recording is unreadable.

Fragmented MP4 solves this. The moov at the start only describes the tracks, and the samples follow in self-contained fragments: a moof box that describes a run of samples, then an mdat box with the samples themselves. Every fragment is playable as soon as it’s written. It’s also the format browsers expect for streaming through Media Source Extensions, and the one HLS and DASH use for their segments.

The muxer

The fragmented MP4 muxer (+907/−279) plugs into joy4’s existing muxer interface, so converting a file is the same avutil.CopyFile(muxer, demuxer) call as for any other format. A few design decisions shape it:

  • It starts at a keyframe. Every other video frame is a delta against the last keyframe, so a file that starts mid-GOP can’t be decoded until the next one. The muxer drops packets until it sees the first video keyframe, and it does that for all streams, not just video, so audio from before that point doesn’t end up out of sync.
  • Every keyframe starts a new fragment. When the next keyframe arrives, the muxer flushes what it has as a moof and mdat. That makes every fragment start with a decodable frame, which is what players need to seek or start mid-stream.
  • The header announces fragments. The moov box contains a mvex box, with a trex entry per track, which tells players to expect movie fragments.
  • Every sample is described explicitly. Each track’s fragment has a tfdt box with its decode time, and a trun box with the duration and size of each sample, plus a composition time offset when the stream has B-frames.
  • Offsets are fixed after the fact. The trun boxes point into the mdat by byte offset, and that offset depends on the size of the moof that contains them. The muxer builds the moof, measures it, then rewrites the data offsets before writing anything.

Writing the muxer also meant extending joy4’s MP4 box code, mp4io, with the fragment boxes it didn’t fully support yet.

Hardening the H.264 parser

The other half of the work, in 2023, went into the H.264 parser.

A raw H.264 stream comes in one of two framings. Annex B separates NAL units with start codes (00 00 01), and is what you typically get from RTSP. AVCC prefixes every NAL unit with its length, and is what MP4 stores. joy4’s SplitNALUs guesses which one it’s looking at, and splits the data into NAL units.

That guess could go wrong in a dangerous way. Some Annex B frames happen to start with bytes that also read as a plausible AVCC length, and the parser then sliced past the end of the buffer and panicked. For a recorder, that’s a crash on a single unlucky frame from a camera. I fixed the overflow, with the frame that triggered it as a regression test, then split the Annex B and AVCC code apart and added fuzz tests, so the next input like that gets found by the fuzzer and not by a camera.

Alongside that, I added the encoding side: an AVCC encoder and an Annex B to AVCC converter, which you need to take a stream from an RTSP camera and put it in an MP4. I also brought the FFmpeg bindings up to FFmpeg 4, added CI, and made the muxer return an error on an invalid stream index instead of misbehaving.

Result

  • A fragmented MP4 muxer for joy4, so recordings are playable while they’re written and survive a crash.
  • An H.264 parser that no longer panics on ambiguous input, with fuzz tests to keep it that way.
  • Annex B to AVCC conversion, FFmpeg 4 support and CI.