Background

File Formats Evolution

Why not use CSV/XML/JSON? 

1. Repeated or no meta information.
2. Files are not splittable, so cannot be used in a map-reduce environment.
3. Missing/ Limited schema definition and evolution support.
   1. Can leverage “JsonSchema” to maintain schema separately for JSON.
   2. It may still require transformation based on a schema, so why not consider Avro/Proto?
4. No native compression of repeating values and indexing capabilities. Binary JSON supports indexing. Link
5. High consumption of bandwidth.
6. Bad Space Utilisation.
7. Poor performance.

Important Terminologies

• Serialisation → Process of converting objects such as arrays and dictionaries into byte streams that can be efficiently stored and transferred elsewhere.
• Deserialisation → Using byte stream to get the original objects back
• Backward Compatibility → New version of software can run code written in old version.

• Forward Compatibility → Older version of software can run code written in new version.

• Schema Evolution – Schema evolution allows you to update the schema used to write new data, while maintaining backwards compatibility with the schema(s) of your old data. Then you can read it all together, as if all of the data has one schema. Of course there are precise rules governing the changes allowed, to maintain compatibility. Those rules are listed under Schema Resolution.
• Registering and using Schema :

• Schema Registry tools (More details → Coming Soon) : 
  • Confluent Schema Registry
  • Apicurio Registry
  • AWS Glue Schema Registry

Avro and ProtoBuf

Both are Language-neutral data serialisation system, which relies on a schema-based system.

Avro

Apache Avro was released by the Hadoop working group in 2009. It is a row-based format that has a high degree of splitting. It is also described as a data serialization system similar to Java Serialization. The schema is stored in JSON format, while the data is stored in binary format, which minimizes file size and maximizes efficiency.

• Schema is stored along with the Avro data in a file for any further processing.
• We can also maintain separate schema registry :
  1. In RPC, the client and the server exchange schemas during the connection.
  2. Schema-less Avro is used heavily in HDFS network communication. (serialiser and deserialiser knows the data schema in advance)
• Avro supports both dynamic and static types as per the requirement.
  1. without the static compilation step and greater interoperability with dynamic languages
  2. Code generation is nonetheless still available in Avro for statically typed languages as an alternative optimisation
• Avro has support for primitive ( int, boolean, string , float etc.) and complex ( enums, arrays, maps, unions etc.) types. Link
• Avro creates binary structured format that is both compressible and splittable. Hence it can be efficiently used as the input to Hadoop MapReduce jobs and that’s the reason it is built in the Hadoop ecosystem.
• Schema Definition Example:

{
    "type": "record",
    "name": "Person",
    "fields": [
        {"name": "userName",        "type": "string"},
        {"name": "favouriteNumber", "type": ["null", "long"]},
        {"name": "interests",       "type": {"type": "array", "items": "string"}}
    ]
}

…or in an IDL:

record Person {
    string               userName;
    union { null, long } favouriteNumber;
    array<string>        interests;
}

• Strings are just a length prefix followed by UTF-8 bytes

• Avro WorkFlow
  • Serialisation
    • Create Avro schema → .avsc file
    • Read this schema in your code using SchemaParser
      • Write avro data file
  • Deserialisation
    • Without External schema
      • Read file directly.
    • With External schema
      • Read schema ← Using schema parser in Spark
      • Open avro file using schema definition of step a. above.

Protobuf

Protocol buffers, usually referred as Protobuf, is a protocol developed by Google to allow serialization and deserialization of structured data. Google developed it with the goal to provide a better way, compared to XML, to make systems communicate.

• Developed by Google and was open sourced in 2008.
• Protobuf is easy to use in microservices, especially where performance and interoperability is important.
• Schema is to be maintained separately.
• It only supports static types. 
• Support more complex data types as compared to Avro. Link
• Schema Definition Example : 

message Person {
    required string user_name        = 1;
    optional int64  favourite_number = 2;
    repeated string interests        = 3;
}

• Each field starts with a byte that indicates its tag number (the numbers 1, 2, 3 from the schema), and the type of the field.
  • This lead to larger file size as compared to Avro (when data size is more as compared to schema stored).

• Protobuf workflow :
  • Serialisation 
    • Create proto schema → .proto file
    • Compile proto schema using “protoc” compiler for target language.
    • Read schema definition by importing compiled class generated in above step.
      • Write proto binary data file.
  • Deserialisation
    • Without External Schema ← Not Supported
    • With External Schema
      • Import compiled class generated by compiler → To get Schema definition
      • Open binary file using schema definition of step 1 above.

Avro vs ProtoBuf Comparison Summary

Conclusion

Avro seems a better fit for BigData use cases as it is widely used in multiple frameworks. Splittable, schema along with data and native compression techniques are major advantages over Protocol Buffer.

Protobuf is easy to use in microservices, especially where performance and interoperability are important and is superior to Avro in this area.

Reference Links