Serialization pitfalls

Most serialization bugs are not crashes. The data arrives, parses and looks right, but a number is off, a time is shifted or a field means something different to the reader. These bugs live in the gap between what the writer meant and what the reader decoded. This page lists the common ones and how to prevent each.

Big integers in JSON and JavaScript

JavaScript stores numbers as 64-bit floats. Integers are exact only up to Number.MAX_SAFE_INTEGER, which is 2^53 − 1. Above that, a value is rounded silently.

sent in JSON:      {"id": 9007199254740993}
read in JavaScript: 9007199254740992

Snowflake IDs and 64-bit database keys often exceed this limit. Send them as strings in JSON, or use a binary format with a true 64-bit integer type. The details are in binary, integers and overflow.

Timestamp units and zones

A timestamp can be seconds or milliseconds, and it can be a Unix number or a local string. One consumer reading seconds and another reading milliseconds shifts every event by a factor of 1,000. Make the unit and the zone explicit in the schema. Use a typed timestamp when the format has one, such as Protobuf's google.protobuf.Timestamp.

Enums that grow

Adding a new enum value is a compatibility hazard. An old reader with a switch that has no default branch may crash on the new value. Plan for an unknown value: give the switch a default branch that logs and keeps going. In Protobuf, an enum value the reader does not know should be kept, not treated as an error.

Null versus absent

These three JSON payloads mean different things.

{"phone": null}     phone was cleared on purpose
{}                  phone was not sent, so it is unknown
{"phone": "555-0100"}   phone has a value

Avro unions such as ["null", "string"] make null explicit. Protobuf tracks whether a field is set, and the rules differ across editions. Write down what each state means: "unknown" or "cleared by the user". Otherwise, one service reads a missing field as an empty value, and another treats it as a deletion.

Money as a float

Encode currency as integer minor units, such as total_cents, or as a decimal string. Do not use a binary float or double, in any format. The rounding errors are the same whether the value travels as JSON or Protobuf. See floating point and money.

Checksums are not security

Storage formats and network frames often carry a checksum, so bit flips can be caught.

Common mistakes

• Trusting the parse. Valid JSON can still hold a rounded integer or a wrong unit.
• Leaving the unit implicit. Put seconds or milliseconds in the field name, or use a typed timestamp.
• Using a switch with no default on an enum. New values crash old readers.
• Treating a checksum as a signature. It detects damage, not tampering.

Check yourself

1. Why does 9007199254740993 come back as 9007199254740992 in JavaScript? The value is above 2^53 − 1, so the 64-bit float cannot hold it exactly, and it rounds.
2. A message has no phone field. Why must the producer and consumer agree on what that means? Absent can mean unknown, or it can mean the user cleared the value. The two readings lead to different actions.

Key takeaways

• Integers above 2^53 − 1 need strings or a true 64-bit type in JSON.
• Make timestamp units and zones explicit, and use typed timestamps where available.
• Plan for enum values that a reader does not yet know.
• Define null, absent and cleared as separate states.
• Store money as integers or decimal strings.
• A CRC shows that bits were not damaged, not that the data is authentic.

Further learning

Kleppmann's book covers encoding, schema evolution and the failure modes of data formats in depth.