Transcoding as a DAG

Encoding a two-hour 4K master on one machine is slow, and a single failure wastes the whole run. So the encode is split into a graph of small tasks. Each task handles one chunk at one quality. Thousands of tasks can run at once, and a failed task retries alone.

The big idea

Picture a bakery making one batter in several cake sizes. The batter is poured into pans, each pan is baked at each size in parallel, and the finished cakes are boxed once all are done.

The picture breaks where the dependencies are real. Packaging must wait for every chunk, so the graph is parallel only where tasks are independent.

Walking through it

The pipeline has four stages:

inspect source → split into keyframe-aligned chunks
              → encode each chunk × each rendition, in parallel
              → package segments and write manifests
              → mark status = ready (thumbnails and checks run alongside)

A concrete example: a 10-minute video split into 4-second chunks gives 600 ÷ 4 = 150 chunks. With five renditions, that is 150 × 5 = 750 encode tasks.

Each task is keyed by (video_id, chunk, rendition). If a worker is preempted on chunk 42 at 720p, only that one task runs again. The other 749 are untouched. That only works if the task is idempotent, meaning that running it twice gives the same output.

Codecs are a separate choice. H.264 is the most compatible. Newer codecs such as VP9, HEVC and AV1 compress better but need more encoding work and have uneven device support. Many platforms keep an H.264 ladder for compatibility and add a newer codec for capable devices. Each extra codec adds storage.

Common mistakes

• Splitting on arbitrary byte offsets. A chunk that does not start on a keyframe cannot be decoded on its own.
• Non-idempotent tasks. A retry then writes duplicate output, or a half-written file.
• Marking the title ready too early. Every rendition and sidecar the player may request must exist first.
• Adding codecs without counting storage. Every codec is another full set of files.

Check yourself

1. A worker dies while encoding chunk 42 at 720p. What runs again?

Answer: Only that task. Tasks are keyed by video, chunk and rendition, and running them again is safe.

2. Why does a 10-minute video with 4-second chunks produce 750 tasks?

Answer: 600 ÷ 4 = 150 chunks, and 150 chunks × 5 renditions = 750 tasks.

Key takeaways

• Split the master into keyframe-aligned chunks and encode each chunk for each rendition.
• Independent tasks run in parallel and retry on their own.
• Make tasks idempotent, so a retry never duplicates output.
• Mark the title ready only after every needed file exists.

Further learning

Next, see how the player chooses among these renditions in adaptive bitrate streaming.