Text vs binary formats

Every time one service calls another, a browser talks to an API, or Kafka stores an event, objects in memory must become bytes. Those bytes become objects again on the other side. That round trip is serialization (encoding to bytes) and deserialization (decoding back). The format you choose sets how big each message is and how much CPU both sides spend.

The big idea

A text format like JSON repeats each field name in every record. A binary format writes the field identity once, in a schema, or as a short number. Multiply that difference by billions of messages and it shows up in bandwidth, storage and CPU.

JSON wins on universality. Every browser, tool and language reads it. It loses on size, parse cost and types. For example, JSON numbers can collide with the 64-bit floats in JavaScript, which is covered in the pitfalls page.

Walking through one record

Take this record: user_id = 42, email = "ada@example.com", active = true.

import json

record = {"user_id": 42, "email": "ada@example.com", "active": True}
text = json.dumps(record, separators=(",", ":"))
print(text, len(text.encode()))

Output:

{"user_id":42,"email":"ada@example.com","active":true} 54

Now the same record in Protobuf. Each field is a 1-byte tag (field number plus type), then its value. Field names never appear in the bytes.

field 1 (user_id):  tag 0x08 , varint 42         →  2 bytes
field 2 (email):    tag 0x12 , length 15, text   → 17 bytes
field 3 (active):   tag 0x18 , value 1           →  2 bytes
total                                             = 21 bytes

The JSON is 54 bytes and the Protobuf is 21. At 100,000 messages per second, that is about 5.4 MB/s of JSON against about 2.1 MB/s of Protobuf. The gap grows with more fields and longer field names.

Exact sizes depend on the encoder. The order of magnitude is what matters.

Parquet is a different job

Parquet is not a message format. It stores data in columns, compresses each column and keeps the schema in the file footer. A scan that reads three columns from a wide table reads only those three, so it is far cheaper than scanning rows of JSON. Service calls want one record at a time, so they use row formats. Analytics wants columns. See query execution for why the layouts differ.

Common mistakes

• Choosing a format by habit. Measure the size and CPU on your real payloads.
• Using a binary format with no schema story. Without a schema, nobody can read the bytes safely later.
• Ignoring the cost of repeated field names. A fleet that sends the same record a billion times pays for the names a billion times.

Check yourself

1. Why is the Protobuf version smaller than the JSON version of the same record? Protobuf writes a small field number instead of the field name, and it stores integers in compact varint form.
2. You publish an API to third-party developers who use curl. Which format is the better choice? JSON. Anyone can read it with no tooling.

Key takeaways

• Serialization turns objects into bytes and back. The format decides the size and speed.
• JSON is readable and universal, but it repeats field names and is large.
• Protobuf and Avro use compact binary, with the schema carrying the field identity.
• Parquet is a columnar file format for analytics, not a message format.

Further learning

Next, read the hub, serialization, schemas and compression, to see how these formats change over time. Then read binary schemas for how Protobuf and Avro use the schema to read bytes.