Binary schemas, Protobuf and Avro

A binary format is only compact because it leaves things out. Protobuf leaves out field names and writes field numbers. Avro leaves out per-field markers and writes values in the order the schema lists them. In both cases, the reader needs the schema to make sense of the bytes. This page shows how each format uses the schema, so you know what has to be kept in step.

Protocol Buffers: field numbers are identity

You write a .proto file. Each field gets a number that never changes.

syntax = "proto3";

message User {
  int64  user_id = 1;
  string email   = 2;
  bool   active  = 3;
}

Each present field is written as a tag, which is its number and wire type, followed by the value. The tag for user_id is 0x08, because field 1 with a varint value gives (1 << 3) | 0. A code generator turns the .proto into typed classes in Go, Java, TypeScript and other languages.

The number is the field's identity. A reader that sees tag 2 knows it is the email, even if the field was renamed in the source code. Renaming is safe on the wire. Reusing a number for a different meaning is not, because old data would decode as the wrong field.

Avro: the writer's schema and the reader's schema

Avro schemas are usually JSON documents. The binary encoding writes each value in the order the schema lists the fields, with no tags. Without the schema, the bytes are only numbers and strings with no labels.

To decode, the library needs two schemas.

• The writer's schema, which describes the bytes that were written.
• The reader's schema, which describes what the code wants.

The library matches fields by name, using aliases where a field was renamed, and fills in defaults for fields the writer did not have.

Suppose the writer used version 1, with user_id and email. The reader expects version 2, which adds phone with a default of null.

writer v1 bytes:   user_id = 42, email = "ada@example.com"
reader v2 expects: user_id, email, phone (default null)
result:            user_id = 42, email = "ada@example.com", phone = null

If the writer had phone and the old reader does not know it, the library drops the field. Avro keeps both directions working when new fields have defaults.

For Kafka, each message usually carries a small schema ID. The full schema lives in a schema registry. That keeps every message small, because the schema is sent once, not with every record.

Choosing between them

Thrift is close to Protobuf in spirit, with field IDs and its own interface language. You will still find it in older infrastructure. For new RPC work, Protobuf is the usual choice.

Common mistakes

• Reusing a Protobuf field number. Old data decodes as the wrong field. Mark deleted numbers as reserved.
• Renaming an Avro field without an alias. The reader cannot match it, and the field is lost or fails.
• Sending Avro bytes with no schema ID. Consumers then need to guess which schema to use.
• Changing a field's type in place. This is usually a breaking change in both formats.

Check yourself

1. Why does renaming a Protobuf field not change the bytes on the wire? The wire carries the field number, not the name. The number is the identity.
2. An Avro reader expects a field the writer never wrote. What happens? The reader uses the field's default value. This is why new fields need defaults.

Key takeaways

• Protobuf identifies fields by permanent numbers written in each message.
• Avro writes values in schema order with no tags, so decoding needs the writer's and reader's schemas.
• New fields need defaults in Avro and new numbers in Protobuf. Reserve deleted numbers.
• A schema registry keeps Avro messages small by storing the schema once.

Further learning

Next, read schema evolution for how to change a schema without breaking the systems that read it.