JSON, binary JSON, and Protocol Buffers

Text format pitfalls

JSON, XML, and CSV work in any language, but numbers are vague. JSON does not split integers from decimals, so parsers often use 64-bit floats. Integers above 2^53 then come back slightly wrong, which breaks 64-bit IDs. There is no type for raw bytes, so binary data is sent as Base64 text, about a third bigger. CSV has no schema and fuzzy escaping rules. Still, when two companies must agree on a format, these are often good enough.

JSON Schema

JSON Schema adds types and value rules, like "age is an integer from 0 to 150". Its default open content model allows fields not in the schema. A closed model allows only listed fields. So a typical schema says what is not allowed. Conditions and remote references make big schemas hard to evolve.

Binary JSON

MessagePack, CBOR, and BSON write JSON as bytes. With no schema, every record still carries every field name. The saving is small, and you lose readability.

Protobuf field tags

Protocol Buffers needs a schema. Each field has a type, a name, and a number called its tag. The encoded bytes carry tags, not names. Integers use a variable length: 0 to 127 fit in one byte, 300 needs two. A list is just the same tag repeated.

Protobuf schema evolution

Names are not in the data, so renaming is safe. You can never change a tag: old data would be read as the wrong field. Add fields with new tags: old code skips unknown tags and can keep them. New code reading old data gets a default, like 0 or "". When you remove a field, reserve its tag so no one reuses it. Changing int32 to int64 is risky: old code cuts big values to fit 32 bits.

Remember

• JSON numbers above 2^53 lose precision in many languages.
• Open JSON Schemas allow extra fields by default.
• Binary JSON still stores field names.
• Protobuf stores tags, so tags are forever.