Data structures and storage engines
Serialization, schemas and compression
How data becomes bytes for the wire and for disk, how formats like JSON, Protobuf and Avro differ, how to change a schema safely, and when compression pays. Start here for data formats.
Every time a service calls another, a browser talks to an API, or Kafka stores an event, in-memory objects become bytes, and the bytes become objects again. That translation is serialization and deserialization. A verbose format costs bandwidth, CPU and storage on every message. A compact binary format with no plan for change can break every consumer when one field is renamed.
This topic covers the choice of format, how schemas let binary data be compact and still readable, how to change a schema without breaking anyone, when to compress, and the bugs that show up in production.
The big idea
A format decides three things: how big each message is, how fast it is to encode and decode, and how it changes over time. A schema is the contract that ties a writer's bytes to a reader's fields. Text formats often carry their names in every message. Binary formats keep the contract in a schema, which is why they are small.
Concepts in this section
Read them in this order.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Text vs binary formats | Why is one record 54 bytes in JSON and 21 bytes in Protobuf? |
| 2 | Binary schemas, Protobuf and Avro | How do field numbers and writer and reader schemas make bytes readable? |
| 3 | Schema evolution | How do I change a field without breaking old or new readers? |
| 4 | Compression tradeoffs | When does compressing a payload save more CPU than it costs? |
| 5 | Serialization pitfalls | Which bugs hide in the encoding, such as big integers in JSON? |
Choosing a format
Pick by the job, not by habit.
| Job | Usual choice | Why |
|---|---|---|
| Public HTTP API | JSON | Anyone can read it with curl |
| Internal RPC between services | Protobuf with gRPC | Generated code and explicit field numbers |
| Events on Kafka or a queue | Avro or Protobuf, with a schema registry | Many teams evolve their schemas independently |
| Analytics files | Parquet | Column scans read only what they need |
Common mistakes
- Choosing a format without measuring. Measure size and CPU on real payloads.
- Reusing a field number or name. Old data decodes as the wrong field.
- Publishing JSON in a byte array without a contract. Once several teams read the topic, the contract is what keeps it working.
- Compressing tiny messages. The CPU often costs more than the bytes saved.
Key takeaways
- Serialization sets the size, speed and evolution cost of every message.
- JSON is universal and verbose. Protobuf and Avro are compact and need a schema.
- Changes are safe when old and new readers can both read the data. Add optional fields with defaults, and never reuse a field number.
- Compression trades CPU for bytes. Measure the ratio and the cost on your own data.
- Watch for big integers in JSON, mixed timestamp units, enums that grow, and checksums mistaken for security.
Further learning
Jordan's systems-design walkthrough covers Protobuf, Thrift and Avro encoding and evolution, and pairs well with Kleppmann's writing.
For event streams, see event streaming. For HTTP versioning, see API design.
Related
Numbers, text and encodings
How integers, floats, text, timestamps and Base64 are really stored as bytes, and the production bugs each one causes. Start here for the data-representation map.
Serialization pitfalls
The production bugs that come from encoded data. Large integers in JSON, mixed timestamp units, enums, null versus absent, money as floats, and checksums that are not security.
API design
An API is a contract that outlives the code behind it. Design the styles, methods, pagination, errors and async flows so changes stay additive.
Event streaming and Kafka
How a durable log lets many consumers read the same events at their own pace, and how Kafka builds that log from partitions, replicas and consumer groups.