Guide

Avro and the value of schemas

Writer's and reader's schema

Avro has a schema, but no tag numbers. It is written in Avro IDL (for people) or JSON (for programs). The bytes hold only values, in schema order, with nothing marking fields or types. To decode, you need the writer's schema: the exact one used to encode. The app also has a reader's schema: the shape it expects. Avro decodes with the writer's schema, then matches fields to the reader's by name. Writer-only fields are dropped. Reader-only fields get the reader's default.

Avro evolution rules

Backward: the writer's schema is older than the reader's. Forward: the writer's schema is newer. Only add or remove fields that have a default. Adding one without a default breaks backward compatibility. Removing one without a default breaks forward compatibility. A type change needs a conversion Avro knows, like int to long. Null needs a union, like null or long, and null can be the default only if it comes first. Renames (aliases in the reader) and new union branches are backward compatible only.

Finding the writer's schema

The schema is too big to send with each record. Instead:

  • A big file: the schema once at the start.
  • A database: a version number per record, plus a store of all versions.
  • A network link: agree on the version when the connection opens.

Generated schemas

To export a database, a program makes one Avro field per column. When columns change, the next export builds a new schema. Readers match by name, so they keep working. With protobuf, someone must assign tags by hand and never reuse old ones.

Merits of schemas

Schema formats skip field names, so they are small. The schema is documentation that stays correct, since decoding needs it. A store of versions lets you check compatibility before deploy. Generated code gets compile-time type checks. The cost: the bytes are unreadable without the schema.

Remember

  • Avro needs the writer's exact schema, then matches by name.
  • Add or remove only fields with defaults.
  • No tags makes Avro good for generated schemas.