Case studies
Case study: video streaming
YouTube and Netflix-style delivery, with resumable uploads, parallel transcoding into a bitrate ladder, HLS and DASH adaptive streaming, and CDN economics.
A video product is two systems that share an object store. The write path turns one uploaded file into many renditions and short segments. The read path barely touches your data center. The player fetches a small manifest, then a steady stream of segments from a CDN edge near the viewer.
The asymmetry is extreme. A popular title may be watched around 100,000 times for every upload. You pay once at upload, in CPU and storage for every rendition, so each view becomes a cache hit with a fast start. This page covers on-demand video. Live streaming uses a different ingest path.
Requirements
- Upload videos lasting minutes to hours, and survive flaky mobile networks.
- Encode a bitrate ladder, from 240p up to 4K.
- Start playback in under about 2 seconds, worldwide.
- Serve millions of concurrent viewers. Egress cost dominates the bill.
Concepts in this topic
| Step | Concept | What it answers |
|---|---|---|
| 1 | Video segments and manifests | How is one video split into files a player can fetch? |
| 2 | Transcoding as a DAG | How do you encode many renditions in parallel? |
| 3 | Adaptive bitrate | How does the player pick quality as the network changes? |
Upload path
Do not stream gigabytes through your application servers.
- The client asks the upload API for a video ID. The metadata row starts as
uploading. - The API returns presigned multipart URLs for object storage.
- The client uploads parts directly and retries only the parts that failed.
- On completion, the API checks size and checksum, marks the video
processing, and enqueues a job on a queue.
Resumable uploads matter more than fancy encoders at this stage. A creator on hotel Wi-Fi needs the upload to pick up where it stopped.
Playback path
The playback API checks authorization and returns a signed manifest URL. The player does the rest: it downloads the master manifest, picks a starting rendition, fetches segments in order, and adjusts quality at each boundary.
CDN strategy
Video bytes are the textbook CDN workload: immutable, cacheable and heavily shared.
- Pre-position hot titles at the edge. Netflix goes further and places caches inside ISP networks, through its Open Connect appliances.
- Pull long-tail content from origin on first request, behind an origin shield so a miss storm does not reach the object store.
- Sign URLs so manifests cannot be shared indefinitely.
A startup usually buys a CDN and designs for a high hit ratio. See CDN hit ratio and cost.
Cost
Storage is a one-time cost per rendition. Egress is continuous, and for popular content it dominates.
1 GB streamed to 1,000,000 unique viewers ≈ 1 PB of egressEncode spend is front-loaded, but egress keeps growing as long as people watch. A codec that cuts bitrate by 30% can pay for its slower encode. Move rarely watched titles to cheaper storage.
Metadata and views
- The metadata database holds title, owner, status and manifest URLs. Owners need strongly consistent reads. Public watch pages can use a cache.
- View counts are approximate and batched. Players send heartbeats to a stream, and a job aggregates them. Never run one
UPDATEper play. - Recommendations consume watch events asynchronously. Playback never waits for them.
DRM
Platforms usually encrypt segments with a DRM system such as Widevine, FairPlay or PlayReady, applied during packaging. The license server joins the startup path, so it must be highly available.
Failure modes
| Failure | Mitigation |
|---|---|
| Upload fails at 90% | Multipart resume; completed parts are not resent |
| Worker preempted mid-encode | Idempotent tasks keyed by video, chunk and rendition |
| Bad encode | Hold ready, alert, and allow re-queueing |
| New episode floods origin | Pre-warm edges, use a shield, rate-limit miss fills |
| Manifest leaked | Short-lived signatures, revoke on takedown |
A ready status should mean the player can fetch every file it may request. Sidecars such as thumbnails, subtitles and alternate audio are outputs of the same pipeline, and the manifest should leave out anything that is missing.
Metrics
Track time to first frame, rebuffer ratio, and CDN hit ratio by region. A spike in rebuffering on one network often means a bad peering path, not an app bug.
Common mistakes
- Proxying uploads through the app. It ties up servers with bytes they never need to read.
- Starting playback at the top rung. A 4K first segment can miss the 2-second start.
- Counting views with one database write per play. At scale, that write rate is unmanageable.
- Forgetting the storage cost of extra codecs. Each codec adds a full set of files.
Key takeaways
- Uploads go straight to object storage. Apps enqueue work and do not proxy bytes.
- Transcoding is a parallel graph over chunks and renditions, with aligned keyframes.
- HLS or DASH manifests with short segments let ordinary CDNs deliver video.
- Egress dominates cost, and view counts should be batched.
Further learning
A Netflix engineering talk from re:Invent. It covers how playback, personalization and delivery fit together at scale.
Related
Object storage
Store files and media in S3-style object stores. Keep bytes out of your app servers and databases, and design uploads, metadata and costs around that split.
CDNs and the edge
Serve content from points of presence near users. A CDN cuts latency, absorbs attacks and cuts bandwidth costs, and its cache keys, headers and purge rules decide how well it works.
Message queues
A durable buffer between the code that creates work and the code that does it. Start here for the map of queue concepts.
Video segments and manifests
A video is stored as many short, immutable segments, listed in a manifest. The player downloads the small manifest first, then fetches segments in order from the nearest cache.
Transcoding as a DAG
Turn one uploaded master file into many renditions by splitting it into chunks and encoding each chunk in parallel, as a directed acyclic graph of small, retryable tasks.
Adaptive bitrate streaming
The player measures its bandwidth and buffer after each segment, then picks the quality of the next segment, so playback keeps going on a slow network.
CDN hit ratio and cost
CDN economics come down to egress price and hit ratio. Each point of hit ratio you gain cuts origin traffic, and a jump from 80% to 95% cuts origin load by a factor of four.