Apache Avro
Apache Avro is a data serialization framework used to store and exchange data efficiently, especially in big data systems like Apache Hadoop and Apache Kafka.
Avro provides:
• Rich data structures.
• A compact, fast, binary data format.
• A container file, to store persistent data.
• Remote procedure call (RPC).
• Simple integration with dynamic languages.
Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.
Why we use Apache Avro?
• Efficient data storage (smaller than JSON/XML)
• Fast serialization/deserialization
• Strong schema support
• Works well in distributed systems
• Enables schema evolution (very important)
Core concept of Avro: Schema + Data
Avro always uses a schema (JSON format) to describe data structure.
Example schema:
{
"type": "record",
"name": "User",
"fields": [
{ "name": "name", "type": "string" },
{ "name": "age", "type": "int" }
]
}
Example data:
{
"name": "Alice",
"age": 30
}
Avro stores this in binary format, not plain JSON.
Key features of Avro
1. Compact binary format
• Smaller size → saves storage & bandwidth
2. Schema evolution
• You can change schema over time:
• Add/remove fields
• Maintain backward/forward compatibility
3. No per-record schema overhead
• Schema is stored once (not repeated per record)
4. Language-agnostic
• Works with Java, C#, Python, etc.
5. Fast processing
• Optimized for big data pipelines
When to use Avro?
Use it when:
• You need efficient data serialization
• Working with big data systems
• Sending data through Kafka or messaging systems
• Need schema evolution
• Storing data in Hadoop/HDFS
When NOT to use Avro?
• Human-readable format is required (use JSON)
• Simple apps with small data
• Debugging-heavy workflows (binary is harder to inspect)
How Avro works (simplified flow)?
• Define schema
• Serialize data → binary format
• Store or transmit
• Deserialize using schema
Alternative Technologies
1. Protocol Buffers
• Very fast and compact
• Requires .proto files
• Strong typing
2. Apache Parquet
• Optimized for analytics (column-based)
• Better for querying than Avro
3. JSON
• Human-readable
• Larger and slower
4. Thrift
• Similar to Protobuf
• Includes RPC support
Advantages of Avro over alternatives
• Compact and fast
• Strong schema management
• Backward/forward compatibility
• Great for distributed systems
• Widely supported in big data tools
Disadvantages of Avro
• Not human-readable
• Requires schema management
• Debugging is harder than JSON/XML
• Slight setup complexity
Avro Schemas
Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.
When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present.
When Avro is used in RPC, the client and server exchange schemas in the connection handshake. (This can be optimized so that, for most calls, no schemas are actually transmitted.) Since both client and server both have the other's full schema, correspondence between same named fields, missing fields, extra fields, etc. can all be easily resolved.
Avro schemas are defined with JSON . This facilitates implementation in languages that already have JSON libraries.
Comparison with other systems
Avro provides functionality similar to systems such as Thrift, Protocol Buffers, etc. Avro differs from these systems in the following fundamental aspects.
• Dynamic typing: Avro does not require that code be generated. Data is always accompanied by a schema that permits full processing of that data without code generation, static datatypes, etc. This facilitates construction of generic data-processing systems and languages.
• Untagged data: Since the schema is present when data is read, considerably less type information need be encoded with data, resulting in smaller serialization size.
• No manually-assigned field IDs: When a schema changes, both the old and new schema are always present when processing data, so differences may be resolved symbolically, using field names.
Apache Avro, Avro, Apache, and the Avro and Apache logos are trademarks of The Apache Software Foundation.
