跳到主要内容

3 篇博文 含有标签「avro」

查看所有标签

Avro to Parquet Pipeline in Big Data Applications

· 阅读需 5 分钟

Introduction​

In modern data architectures, Avro data is often used in messaging systems like Apache Kafka due to its compact binary format and schema evolution support. However, in data lakes or data warehouses, Parquet is preferred for storage due to its columnar format and efficient read performance. This article covers the architecture, pipeline design, and real-life sample code for implementing a pipeline that ingests Avro data from a messaging system and saves it in Parquet format in a big data storage solution like Amazon S3.


Architecture Overview​

This section provides an overview of the components in a typical architecture that ingests Avro from a messaging system and saves it as Parquet in big data storage.

Diagram: Avro to Parquet Pipeline Architecture

Messaging System (Kafka) ----> Stream Processing (Spark) ----> S3 Storage (Parquet Format)

Avro and Parquet Formats​

  • Avro: A row-based binary format, commonly used for serializing data in messaging systems due to its compact size and schema evolution capabilities.
  • Parquet: A columnar storage format optimized for reading specific columns, making it highly efficient for analytics and big data storage solutions.

Role of Messaging Systems​

Messaging systems like Apache Kafka are widely used to transport streaming data across systems in real time. Kafka provides fault-tolerance and durability, ensuring data is safely delivered to downstream applications for processing.

Storage in Big Data Storage Solutions​

Big data storage solutions, such as Amazon S3, offer durable and cost-effective storage for large-scale data. Storing data in Parquet format on S3 enables easy integration with data analytics frameworks like Spark, Presto, and Hive.


Data Pipeline Design​

The pipeline includes the following key stages:

  1. Data Ingestion: Consuming Avro-encoded messages from Kafka.
  2. Processing and Transformation: Converting Avro data to Parquet format.
  3. Storage: Saving the transformed Parquet files to Amazon S3.

Ingesting Data from Messaging System​

The first step in the pipeline is ingesting data from a messaging system like Kafka. Spark Structured Streaming can be used to consume Avro-encoded messages from Kafka topics in real-time.

Diagram: Ingesting Avro Data from Kafka

Kafka Topic (Avro Messages) ---> Spark Structured Streaming ---> Data Transformation

Processing and Converting Avro to Parquet​

Spark Structured Streaming allows us to process and transform the incoming Avro data. To convert Avro to Parquet, the schema needs to be mapped, and the data is then serialized to the Parquet format.

Diagram: Converting Avro to Parquet in Spark

Avro Data (Spark DataFrame) ---> Schema Mapping ---> Parquet Data

Saving to Big Data Storage (S3)​

Once converted to Parquet, the data can be written to Amazon S3 in a structured format. Parquet files in S3 are organized by partitions (e.g., by date or another logical grouping) to optimize retrieval and querying performance.

Diagram: Saving to S3

Parquet Data (Partitioned by Date) ---> Amazon S3 Bucket ---> Analytics-ready Storage

Real-Life Production Sample Code in Spark​

The following sample code demonstrates how to implement this pipeline in Apache Spark with Structured Streaming.

Reading from Kafka​

Use Spark to read Avro data from a Kafka topic in real-time.

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.avro.functions.from_avro;

public class AvroToParquetPipeline {

public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.appName("Avro to Parquet Pipeline")
.getOrCreate();

String kafkaBootstrapServers = "kafka-broker1:9092,kafka-broker2:9092";
String kafkaTopic = "avro-topic";

Dataset<Row> avroDF = spark
.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootstrapServers)
.option("subscribe", kafkaTopic)
.load()
.selectExpr("CAST(value AS BINARY) as avroData");

// Define the Avro schema (can be fetched from Schema Registry if available)
String avroSchema = "{ \"type\": \"record\", \"name\": \"User\", \"fields\": [ {\"name\": \"id\", \"type\": \"int\"}, {\"name\": \"name\", \"type\": \"string\"}, {\"name\": \"age\", \"type\": \"int\"} ] }";

Dataset<Row> parsedDF = avroDF
.select(from_avro(avroDF.col("avroData"), avroSchema).as("data"))
.select("data.*");

parsedDF.printSchema(); // Debugging schema after parsing Avro
}
}

Converting Avro to Parquet​

Once we have parsed the Avro data into a DataFrame, we can convert it into Parquet format.

Dataset<Row> parquetDF = parsedDF.repartition(1); // Optional: Customize partitioning as needed

Writing to S3​

Writing the data to Amazon S3 requires specifying the destination path and configuring partitioning if necessary.

import org.apache.spark.sql.streaming.Trigger;

String s3Path = "s3a://your-bucket/path/to/destination/";

parquetDF.writeStream()
.format("parquet")
.option("path", s3Path)
.option("checkpointLocation", "s3a://your-bucket/path/to/checkpoint/")
.partitionBy("date") // Optional: Partitioning by a specific column (e.g., date)
.trigger(Trigger.ProcessingTime("5 minutes")) // Controls batch frequency
.start()
.awaitTermination();

Conclusion​

This article covered an architecture for ingesting Avro data from a messaging system, converting it to Parquet, and saving it in Amazon S3. Using Spark Structured Streaming and Amazon S3 for storage, this pipeline is a robust solution for managing Avro-to-Parquet transformations in big data applications.

Deep Dive into Avro and Parquet Formats

· 阅读需 16 分钟

Introduction​

As data systems grow in complexity and scale, choosing efficient data serialization formats becomes crucial. This article provides a deep dive into Apache Avro and Apache Parquet, two popular formats in the Big Data ecosystem. Both formats have distinct characteristics, making them suitable for different types of data and use cases.


Understanding Avro Format​

Apache Avro is a data serialization format that facilitates the encoding of complex data structures into a compact, binary format, making it highly efficient for streaming and storage.

Avro Schema​

The Avro schema describes the structure of data stored in Avro format and is defined using JSON. Avro schemas allow the specification of nested fields, and each schema comprises two parts:

  • Primitive types: null, boolean, int, long, float, double, bytes, and string.
  • Complex types: record, enum, array, map, union, and fixed.

Example Avro Schema:

{
"type": "record",
"name": "Employee",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": null},
{"name": "salary", "type": "double"}
]
}

In this schema:

  • The Employee record has fields id, name, email, and salary.
  • The email field is optional, allowing null values.

Avro Schema in Depth​

Apache Avro is a data serialization framework with a focus on compact storage, schema evolution, and interoperability, especially popular in data pipelines with tools like Apache Kafka, Hadoop, and Spark. Avro schema is a JSON-based definition that describes the structure, types, and constraints of the data. This schema allows for efficient serialization and deserialization, making it versatile for data storage and streaming applications.

Avro’s schema format is highly adaptable, supporting both simple and complex data structures and allowing schema evolution without disrupting data compatibility. This makes Avro particularly suited for big data pipelines and environments where schemas may need frequent updates. Its JSON-based structure is easy to read and interpret while enabling efficient storage and retrieval in binary format, ideal for high-performance data processing.

Here’s an in-depth look at Avro schemas:

1. Schema Basics​

Avro schemas are written in JSON and define the data types and structure. A schema specifies:

  • Type: The data type, like string, int, float, or complex types like record and array.
  • Name: The name of the field (only required for named types like record).
  • Namespace: Optional, used to provide a unique context for named types, helpful in managing type names in larger systems.
  • Fields: Defines each field within complex types, like record.

Example of a simple Avro schema:

{
"type": "record",
"name": "User",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": "string"}
]
}

This schema defines a User record with three fields: id, name, and email.

2. Primitive Data Types​

Avro supports several primitive data types:

  • Null: Represents a null value.
  • Boolean: Represents true or false.
  • Int: 32-bit signed integer.
  • Long: 64-bit signed integer.
  • Float: 32-bit IEEE floating-point number.
  • Double: 64-bit IEEE floating-point number.
  • Bytes: Sequence of bytes, useful for binary data.
  • String: Unicode character sequence.

3. Complex Data Types​

Avro supports complex data types to handle more advanced data structures:

  • Record: Similar to a structured data object or table row; contains fields, each with its own name and type.
  • Enum: Defines a list of allowed values.
  • Array: An ordered collection of values of a specified type.
  • Map: A collection of key-value pairs with string keys and values of a specified type.
  • Union: Allows fields to have multiple types, which supports optional fields and complex structures.
  • Fixed: Represents a fixed-size binary value, useful for things like fixed-length IDs.

Example of complex types:

{
"type": "record",
"name": "Person",
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": ["null", "int"], "default": null},
{"name": "emails", "type": {"type": "array", "items": "string"}},
{"name": "address", "type": {
"type": "record",
"name": "Address",
"fields": [
{"name": "street", "type": "string"},
{"name": "city", "type": "string"}
]
}}
]
}

This schema defines a Person record with an optional age, a list of emails, and a nested Address record.

4. Union Types​

Unions in Avro schemas allow a field to hold one of multiple types, helping to represent optional fields:

  • A union is represented by an array of types, such as ["null", "int"] for a nullable integer.
  • The first type in the union is considered the default.

Example:

{"name": "birthdate", "type": ["null", "string"], "default": null}

Here, birthdate is either null or a string, allowing it to be optional.

5. Default Values​

  • Fields can have default values if they are missing in the data, which is particularly useful when evolving schemas.
  • Default values must match the field type and are defined within the schema.

Example:

{"name": "isActive", "type": "boolean", "default": true}

6. Schema Evolution​

Avro is designed to handle schema evolution, meaning new schemas can be applied to existing data files without breaking compatibility:

  • Backward Compatibility: New schema can read old data (e.g., adding new fields with default values).
  • Forward Compatibility: Old schema can read new data (e.g., adding optional fields).
  • Full Compatibility: Both backward and forward compatible.

Schema evolution rules include:

  • Adding a new field with a default value is backward-compatible.
  • Removing a field is backward-compatible if it is not required by consumers.
  • Changing a field type requires careful consideration to ensure compatibility.

7. Named Types and Namespaces​

In Avro, named types like record and enum require unique names, often scoped by namespaces:

  • Name: The unique identifier for a type.
  • Namespace: Provides context to prevent naming conflicts, similar to a package in Java or a module in Python.

8. Schema Example with Namespaces​

Example using namespace for managing complex data:

{
"namespace": "com.example",
"type": "record",
"name": "Customer",
"fields": [
{"name": "id", "type": "int"},
{"name": "contact", "type": {
"type": "record",
"name": "ContactInfo",
"fields": [
{"name": "email", "type": "string"},
{"name": "phone", "type": ["null", "string"], "default": null}
]
}}
]
}

Schema Registry​

Schema Registry is a crucial component that allows the storage and retrieval of schemas used by producers and consumers of data. It ensures that producers and consumers agree on the data structure, supporting schema evolution and compatibility checks.

Diagram: Schema Registry Workflow

+-----------+ +--------------------+
| Producer | Write | Schema Registry |
| | ----------> | |
+-----------+ +--------------------+
|
|
| Write
v
+-----------+ +-----------------+
| Data | Read/Write | Consumer |
| Storage | <--------------> | |
+-----------+ +-----------------+
  • Producers register the schema before sending data to storage.
  • Consumers fetch the schema from the registry to deserialize and understand the data.

Storage Architecture for Avro in HDFS/S3​

When storing Avro data in HDFS or S3, Avro files contain both the schema and data, allowing each file to be fully self-contained and readable independently.

Diagram: Avro Storage in HDFS/S3

+------------------------+
| HDFS or S3 Storage |
+------------------------+
|
+------------------------+
| Avro File |
| + Schema |
+------------------------+
|
+------------------------+
| Blocks |
+------------------------+
  • Each Avro file contains schema and data blocks, making it easily portable across storage platforms.
  • Data Access: Avro files can be read without external dependencies, as each file includes the required schema.

Sample Code in Spark​

Below is a Spark example that reads data from an Avro file, processes it, and writes it back to HDFS as Avro format.

import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.avro._

// Initialize Spark session
val spark = SparkSession.builder
.appName("AvroExample")
.getOrCreate()

// Define the Avro schema as JSON
val avroSchema = """
{
"type": "record",
"name": "Employee",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": null},
{"name": "salary", "type": "double"}
]
}
"""

// Read Avro file from HDFS
val df: DataFrame = spark.read
.format("avro")
.load("hdfs://path/to/input/avro/file")

// Process the DataFrame (example: filter employees with salary > 5000)
val filteredDf = df.filter("salary > 5000")

// Write back to HDFS in Avro format
filteredDf.write
.format("avro")
.option("avroSchema", avroSchema)
.save("hdfs://path/to/output/avro/file")

Exploring Parquet Format​

Apache Parquet is a columnar storage format optimized for complex nested data structures, making it an ideal choice for big data processing.

Parquet Schema and Compression​

Parquet organizes data in columns, allowing for efficient compression and encoding. This format is particularly beneficial for queries that scan specific columns.

Compression Techniques:

  • Dictionary Encoding: Reduces data size by replacing duplicate values with codes.
  • Bit Packing: Optimizes storage by packing multiple values into smaller data units.

Diagram: Parquet File Structure

+------------------------------+
| Parquet |
+------------------------------+
| Row Group 1 |
| +--------------------+ |
| | Column 1 | |
| | Column 2 | |
| | ... | |
| +--------------------+ |
| |
| Row Group 2 |
| +--------------------+ |
| | Column 1 | |
| | Column 2 | |
| | ... | |
| +--------------------+ |
+------------------------------+
  • Each Row Group contains columns stored together, optimizing compression and query performance.
  • Columnar storage allows quick access to specific fields, reducing read and write times.

Parquet Schema​

Parquet’s schema defines the structure, types, and layout of the data stored in the file, making it highly efficient for analytics and queries. This schema is stored in the file’s metadata and allows query engines to interpret the data without needing to read the entire file, which speeds up operations significantly.

The Parquet schema plays a crucial role in its columnar storage efficiency, managing data sparsity, and enabling rich, nested data structures with low storage overhead. It facilitates schema evolution, where additional columns or fields can be added without rewriting the entire file, and supports advanced filtering and optimizations for large data analytics tasks.

Here’s an overview of the main components of the Parquet schema:

1. Schema Hierarchy and Structure​

Parquet schema is organized hierarchically and supports complex nested structures:

  • Message Type: This is the root element in a Parquet schema, defining the data structure at the highest level.
  • Fields: Each field in the schema has a name, data type, and potentially nested subfields if the data structure is complex. Parquet supports:
    • Primitive Types: such as int32, int64, boolean, float, double, and binary.
    • Nested Types: Parquet can store complex structures like lists, maps, and structs, which are stored as nested types in the schema.

For example, a Parquet schema could look like:

message schema {
required int32 id;
optional group address {
required binary street (UTF8);
optional int32 zip_code;
}
}

This schema describes a record with an id and an optional nested address group, which contains a street and an optional zip_code.

2. Column Definition: Required, Optional, Repeated​

Each field in the Parquet schema has a repetition level:

  • Required: Every row must contain a value for this field.
  • Optional: A field that can be present or absent in each row.
  • Repeated: Used to model lists or repeated elements, allowing multiple values for the field in a row.

3. Definition and Repetition Levels​

Parquet’s schema model is hierarchical and optimized for efficient storage and retrieval in columnar format. It uses Definition Levels and Repetition Levels to handle nullability, optional fields, and nested data structures. Let’s walk through an example to show how these levels work.

To manage complex data structures and sparsity, Parquet uses definition levels and repetition levels:

  • Definition Levels track whether each level in the hierarchy has data or is null, it tells us how deep in the structure a field is defined.
  • Repetition Levels track whether an element is the first instance or a repeated instance in nested lists or repeated fields, it tells us if the field is part of a repeated structure and, if so, how deep the repeat goes.

For instance, if a row has a list of addresses, the repetition levels can indicate the position of each address within the list, while definition levels show whether each element is present or missing.

Example Parquet Schema​

Consider the following JSON structure, which could be a nested data schema for a Parquet file:

{
"name": "Alice",
"contacts": [
{
"type": "email",
"address": "alice@example.com"
},
{
"type": "phone",
"number": null
}
],
"addresses": [
{
"street": "123 Main St",
"city": "Wonderland"
}
]
}

Here’s the corresponding Parquet schema:

message Person {
required binary name (UTF8);
repeated group contacts {
required binary type (UTF8);
optional binary address (UTF8);
optional binary number (UTF8);
}
repeated group addresses {
required binary street (UTF8);
required binary city (UTF8);
}
}
Parquet Structure: Definition and Repetition Levels​
  1. Definition Level: Indicates the presence or absence of values at each level in a schema. This level is used to define nullability and optional fields.
  2. Repetition Level: Indicates if a field is part of a repeated structure (e.g., a list or an array). This is useful for nested or repeated structures, telling the reader if a given level repeats within the hierarchy.

Let’s break down how these levels work for each field in the example above:

1. Field: name​
  • Definition Level: 1 (required field, so it's always defined)
  • Repetition Level: 0 (not repeated)
2. Field: contacts​
  • Definition Level: 1 to 2, depending on the presence of address and number fields.
    • If both address and number are null, the definition level is lower.
  • Repetition Level: 1 (this field is repeated since contacts is a list)
3. Fields Inside contacts: type, address, number​
  • type
    • Definition Level: 2 (required field within the repeated group)
    • Repetition Level: 1
  • address and number
    • Definition Level: 2 or 3 (optional fields)
      • A definition level of 2 means the contacts object is present but address or number is null.
      • A definition level of 3 indicates both fields are filled.
    • Repetition Level: 1 (repeated inside contacts)
4. Field: addresses​
  • Definition Level: 1 to 2
  • Repetition Level: 1 (repeated field)
5. Fields Inside addresses: street, city​
  • Both street and city are required within the repeated addresses group.
  • Definition Level: 2
  • Repetition Level: 1 (both are repeated with each addresses group entry)
Example Encoding with Levels​

Assuming data with these values:

{
"name": "Alice",
"contacts": [
{
"type": "email",
"address": "alice@example.com",
"number": null
},
{
"type": "phone",
"number": "123-456-7890"
}
],
"addresses": [
{
"street": "123 Main St",
"city": "Wonderland"
}
]
}

The encoded levels might look like this:

FieldValueDefinition LevelRepetition Level
name"Alice"10
contacts[0].type"email"21
contacts[0].address"alice@example.com"31
contacts[0].numbernull21
contacts[1].type"phone"21
contacts[1].addressnull21
contacts[1].number"123-456-7890"31
addresses[0].street"123 Main St"21
addresses[0].city"Wonderland"21

These levels help Parquet efficiently handle complex nested and repeated data structures, enabling better data compression and faster querying by only storing relevant levels and values for each entry.

4. Logical Types and Annotations​

Parquet supports logical types that provide additional context for primitive types:

  • Logical Types: These are type annotations like UTF8 for text strings, DATE for dates, and DECIMAL for high-precision decimals.
  • Logical types help map data from systems that use richer data types (like SQL databases) to Parquet’s primitive types.

Example:

optional int32 birthdate (DATE);

5. Column Index and Row Group Metadata​

  • Column Index: Parquet files include metadata about each column, such as min/max values, null counts, and distinct counts. This allows query engines to skip entire blocks if they don't match the filter criteria.
  • Row Group Metadata: Parquet stores data in row groups (blocks of rows), each containing column data for that group. Each row group stores metadata like the number of rows and data size, which helps with efficient reads.

Example of a Parquet Schema in JSON Representation​

A Parquet schema can also be represented in JSON format to show the hierarchy and structure clearly. For instance:

{
"name": "schema",
"type": "message",
"fields": [
{
"name": "id",
"type": "INT32",
"repetition_type": "REQUIRED"
},
{
"name": "address",
"type": "group",
"repetition_type": "OPTIONAL",
"fields": [
{
"name": "street",
"type": "BINARY",
"logicalType": "UTF8",
"repetition_type": "REQUIRED"
},
{
"name": "zip_code",
"type": "INT32",
"repetition_type": "OPTIONAL"
}
]
}
]
}

In this JSON schema:

  • The id field is required.
  • The address field is optional and has two nested fields: a required street and an optional zip_code.

Storage Architecture for Parquet in HDFS/S3​

In HDFS/S3 storage, Parquet files are organized in row groups with each column stored independently within those groups.

Diagram: Parquet Storage in HDFS/S3

+------------------------+
| HDFS or S3 Storage |
+------------------------+
|
+------------------------+
| Parquet File |
+------------------------+
|
+------------------------+
| Row Groups |
+------------------------+
|
+------------------------+
| Columns within Groups |
+------------------------+
  • Row Groups: Partitioned sections that allow for distributed reads.
  • Columns: Stored separately within each row group for efficient access.

Sample Code in Spark​

Below is a Spark example that reads data from a Parquet file, performs transformations, and writes it back to HDFS as Parquet format.

import org.apache.spark.sql.{DataFrame, SparkSession}

// Initialize Spark session
val spark = SparkSession.builder
.appName("ParquetExample")
.getOrCreate()

// Read Parquet file from HDFS
val df: DataFrame = spark.read
.parquet("hdfs://path/to/input/parquet/file")

// Process the DataFrame (example: select employees with salary > 5000)
val filteredDf = df.filter("salary > 5000")

// Write back to HDFS in Parquet format
filteredDf.write
.parquet("hdfs://path/to/output/parquet/file")

Conclusion​

This article provided an in-depth look at Avro and Parquet formats, exploring their unique characteristics, schema management, and storage architectures. With examples in Spark, we demonstrated practical implementations in a real-world production environment, highlighting the versatility of both formats for big data solutions.

Deep Dive into Avro and Parquet Together

· 阅读需 14 分钟

Introduction​

In modern data architectures, Avro data is often used in messaging systems like Apache Kafka due to its compact binary format and schema evolution support. However, in data lakes or data warehouses, Parquet is preferred for storage due to its columnar format and efficient read performance. This article covers the architecture, pipeline design, and real-life sample code for implementing a pipeline that ingests Avro data from a messaging system and saves it in Parquet format in a big data storage solution like Amazon S3.


Architecture Overview​

This section provides an overview of the components in a typical architecture that ingests Avro from a messaging system and saves it as Parquet in big data storage.

Diagram: Avro to Parquet Pipeline Architecture

Messaging System (Kafka) ----> Stream Processing (Spark) ----> S3 Storage (Parquet Format)

### Avro and Parquet Formats

  • Avro: A row-based binary format, commonly used for serializing data in messaging systems due to its compact size and schema evolution capabilities.
  • Parquet: A columnar storage format optimized for reading specific columns, making it highly efficient for analytics and big data storage solutions.

### Role of Messaging Systems

Messaging systems like Apache Kafka are widely used to transport streaming data across systems in real time. Kafka provides fault-tolerance and durability, ensuring data is safely delivered to downstream applications for processing.

### Storage in Big Data Storage Solutions

Big data storage solutions, such as Amazon S3, offer durable and cost-effective storage for large-scale data. Storing data in Parquet format on S3 enables easy integration with data analytics frameworks like Spark, Presto, and Hive.


Data Pipeline Design​

The pipeline includes the following key stages:

  1. Data Ingestion: Consuming Avro-encoded messages from Kafka.
  2. Processing and Transformation: Converting Avro data to Parquet format.
  3. Storage: Saving the transformed Parquet files to Amazon S3.

Ingesting Data from Messaging System​

The first step in the pipeline is ingesting data from a messaging system like Kafka. Spark Structured Streaming can be used to consume Avro-encoded messages from Kafka topics in real-time.

Diagram: Ingesting Avro Data from Kafka

Kafka Topic (Avro Messages) ---> Spark Structured Streaming ---> Data Transformation

Processing and Converting Avro to Parquet​

Spark Structured Streaming allows us to process and transform the incoming Avro data. To convert Avro to Parquet, the schema needs to be mapped, and the data is then serialized to the Parquet format.

Diagram: Converting Avro to Parquet in Spark

Avro Data (Spark DataFrame) ---> Schema Mapping ---> Parquet Data

Saving to Big Data Storage (S3)​

Once converted to Parquet, the data can be written to Amazon S3 in a structured format. Parquet files in S3 are organized by partitions (e.g., by date or another logical grouping) to optimize retrieval and querying performance.

Diagram: Saving to S3

Parquet Data (Partitioned by Date) ---> Amazon S3 Bucket ---> Analytics-ready Storage

Understanding the Structural Differences​

In Avro and Parquet formats, the data organization is fundamentally different:

  • Avro: A row-based format where each record is serialized as a single entity, storing all fields of each row together.
  • Parquet: A columnar storage format where data is grouped by columns rather than rows, optimizing for efficient reads on specific columns (ideal for analytical workloads).

To illustrate, let’s consider an Avro record with two attributes:

{
"name": "John Doe",
"age": 30
}

In Avro, the above record would be stored as a single contiguous block of binary data, with fields appearing in the order defined by the schema.

When converting this record to Parquet, the storage layout changes to a column-oriented format. Parquet organizes data by columns:

  • Each column (e.g., "name" and "age") is stored separately in column chunks.
  • Parquet organizes records into row groups, where each row group has column chunks for each column defined in the schema.

Diagram: Avro to Parquet Conversion for Record { "name": "John Doe", "age": 30 }

Parquet Column ChunkData Stored
Name Column ChunkJohn Doe
Age Column Chunk30

In a more extensive dataset with thousands of records, Parquet’s layout significantly optimizes read performance because each column chunk can be accessed independently. For instance, if only the "age" column is required for analysis, Parquet reads only the relevant chunk, skipping over unrelated columns like "name."


Avro to Parquet Conversion​

The steps for converting Avro to Parquet in a system like Apache Spark generally follow this sequence:

  1. Avro Data Loading:

    • Avro-encoded data is parsed and loaded into an intermediate in-memory structure, such as a DataFrame in Spark.
    • Avro schema is either read from the Avro file (or topic in case of Kafka) or supplied separately.
  2. Schema Mapping:

    • Spark maps the Avro schema to an equivalent internal schema (e.g., Spark schema), which is essential to support Parquet’s columnar structure.
    • Fields in the Avro schema (like "name" and "age") are mapped to equivalent columns.
  3. Data Conversion:

    • Each Avro record is then serialized into a format compatible with Parquet’s column-oriented layout.
    • Each attribute is transformed to be written in column chunks rather than row-by-row.
  4. Data Writing (Parquet):

    • Parquet organizes data into row groups, and within each row group, column data is stored in contiguous chunks.
    • Additional metadata (e.g., min, max, dictionary encoding) is recorded for each column chunk, which is useful for optimizations in querying.

Example: In Spark, a sample code to convert Avro to Parquet may look like this:

// Load Avro data into a DataFrame
Dataset<Row> avroDF = spark.read().format("avro").load("path/to/avro/file");

// Write data to Parquet format
avroDF.write().parquet("path/to/output/parquet");

This process reorganizes the data from a row-based layout (Avro) to a columnar layout (Parquet).


Reading from Parquet and Converting Back to Avro​

When reading Parquet data and converting it back to Avro, the system reverses the process, transforming columnar data back into row-based data.

Steps:

  1. Load Parquet Data:

    • Parquet files are read into memory. In a Spark-based setup, this would involve loading Parquet files into a DataFrame.
  2. Column Extraction:

    • Parquet retrieves data by accessing each relevant column chunk. Parquet's metadata allows efficient navigation to the specific data of interest.
  3. Row Assembly:

    • As each column’s data is accessed, the system assembles rows by gathering data from each column chunk.
    • The row-based structure compatible with Avro is recreated by combining values from each column for every row.
  4. Schema Mapping:

    • Parquet columns are mapped to Avro fields based on a schema. This schema alignment ensures that columns match Avro’s row-oriented requirements.
    • The DataFrame (or equivalent data structure) is serialized back into the Avro format.
  5. Write to Avro:

    • The row-based structure is serialized in Avro’s binary format, and the schema is either embedded or referenced as needed.

Example Code: Converting Parquet to Avro in Spark

// Load Parquet data into a DataFrame
Dataset<Row> parquetDF = spark.read().parquet("path/to/parquet/file");

// Write data back to Avro format
parquetDF.write().format("avro").save("path/to/output/avro");

This code converts the columnar Parquet format back into Avro’s row-based format.


Parquet’s Row Groups and Column Chunks​

In Parquet, data is stored in row groups. Each row group contains data for a set number of rows, and within each row group, data is stored in column chunks (one for each column). The row group is Parquet’s key structural element that ensures the association between column data across multiple rows:

  • Row Group: A set of rows is organized as a contiguous data block. Each row group in Parquet represents a batch of rows from the original dataset.
  • Column Chunk: Within each row group, each column’s data is stored in a separate column chunk.

For example, if we have a dataset with 10,000 records, it might be divided into several row groups (say 5,000 rows per row group). Each row group then contains the data for the columns name and age in its respective column chunks.

Ensuring Data Belongs to the Same Record​

To ensure data from each column belongs to the same record, Parquet relies on row ordering within each row group. The ordering within each column chunk of a row group matches across all columns:

  • Ordering: Each row in a row group corresponds to the same index across all column chunks within that row group. For instance, the first value in the name column chunk of a row group corresponds to the first value in the age column chunk for the same row group.
  • Offset Metadata: Parquet uses metadata about the offsets within each column chunk to ensure each row can be reconstructed correctly when reading.

So, in your example, the row group might look like this:

Row IndexName Column ChunkAge Column Chunk
Row 1"John Doe"30
Row 2"Jane Smith"25
Row 3"Alice Brown"28

Each entry in the name and age column chunks corresponds by index. Parquet uses metadata within the row group and column chunks to indicate where each row begins and ends, making sure that during reads, each row’s data across columns is aligned.

Reading from Parquet and Converting Back to Avro​

When reading Parquet data back into a row-based format (like Avro), the system iterates over each row group:

  • Row Assembly: For each row in a row group, Parquet retrieves the respective values from each column chunk based on their row index. This reassembles each row’s fields together to form the original record.
  • Schema Mapping: The column metadata stored in Parquet includes the schema, allowing the correct assignment of values back to each field in the record.

What if Null Values in Parquet​

In Parquet, each column chunk within a row group holds data in columnar format, and Parquet efficiently handles cases where some rows may lack a value for a given column by using specific encoding and metadata, rather than empty placeholders. Here’s how it manages the scenario:

1. Null Values in Columns​

If a column lacks a value for a particular row, Parquet does not insert an empty placeholder. Instead:

  • Definition Levels are used to indicate whether a value is present or null. Each level specifies if a field has a value or is missing at that row.
  • Repetition Levels track nested structures within rows, so nested nulls and repeated fields can be handled in one scan.

2. Encodings for Efficient Storage​

Parquet uses encodings like Run-Length Encoding (RLE) and Dictionary Encoding to represent repeated or missing values compactly:

  • Run-Length Encoding groups consecutive null values (or any repeating values) into a single representation, saving space when large chunks are missing in a column.
  • Dictionary Encoding can efficiently map common values (like repeated nulls or default values) using dictionary lookup tables.

3. Metadata Tracking and Optional Fields​

  • If a field is defined as optional in the schema, it allows a row to be saved without a value for that field. Parquet only includes these rows in the row group metadata, which helps query engines like Spark identify missing values.

Example Scenario​

Let’s say we have a Parquet file with three rows and three columns, where one column lacks values for some rows.

Row GroupColumn AColumn BColumn C
Row 1Value 1NullValue 3
Row 2Value 4Value 5Null
Row 3NullValue 6Value 7

In this case:

  • Definition Levels for Column B and Column C would show which rows contain nulls.
  • The data structure itself would store only the non-null values, reducing storage without explicit placeholders.

This approach is particularly useful when handling sparse data, as it minimizes the storage footprint and increases reading efficiency.

Example in Spark​

Here’s how Spark handles this under the hood when reading Parquet data and converting it back to Avro:

// Reading from Parquet
Dataset<Row> parquetDF = spark.read().parquet("path/to/parquet/file");

// Writing back to Avro, reassembling rows
parquetDF.write().format("avro").save("path/to/output/avro");

During this process, Spark ensures that each row’s values across columns are correctly aligned by leveraging Parquet’s row group and column chunk metadata. This alignment allows it to convert back to Avro’s row-based format without losing the association between name and age in each record.


Real-Life Production Sample Code in Spark​

The following sample code demonstrates how to implement this pipeline in Apache Spark with Structured Streaming.

Reading from Kafka​

Use Spark to read Avro data from a Kafka topic in real-time.

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.avro.functions.from_avro;

public class AvroToParquetPipeline {

public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.appName("Avro to Parquet Pipeline")
.getOrCreate();

String kafkaBootstrapServers = "kafka-broker1:9092,kafka-broker2:9092";
String kafkaTopic = "avro-topic";

Dataset<Row> avroDF = spark
.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootstrapServers)
.option("subscribe", kafkaTopic)
.load()
.selectExpr("CAST(value AS BINARY) as avroData");

// Define the Avro schema (can be fetched from Schema Registry if available)
String avroSchema = "{ \"type\": \"record\", \"name\": \"User\", \"fields\": [ {\"name\": \"id\", \"type\": \"int\"}, {\"name\": \"name\", \"type\": \"string\"}, {\"name\": \"age\", \"type\": \"int\"} ] }";

Dataset<Row> parsedDF = avroDF
.select(from_avro(avroDF.col("avroData"), avroSchema).as("data"))
.select("data.*");

parsedDF.printSchema(); // Debugging schema after parsing Avro
}
}

Converting Avro to Parquet​

Once we have parsed the Avro data into a DataFrame, we can convert it into Parquet format.

Dataset<Row> parquetDF = parsedDF.repartition(1); // Optional: Customize partitioning as needed

Writing to S3​

Writing the data to Amazon S3 requires specifying the destination path and configuring partitioning if necessary.

import org.apache.spark.sql.streaming.Trigger;

String s3Path = "s3a://your-bucket/path/to/destination/";

parquetDF.writeStream()
.format("parquet")
.option("path", s3Path)
.option("checkpointLocation", "s3a://your-bucket/path/to/checkpoint/")
.partitionBy("date") // Optional: Partitioning by a specific column (e.g., date)
.trigger(Trigger.ProcessingTime("5 minutes")) // Controls batch frequency
.start()
.awaitTermination();

Conclusion​

This article covered an architecture for ingesting Avro data from a messaging system, converting it to Parquet, and saving it in Amazon S3. Using Spark Structured Streaming and Amazon S3 for storage, this pipeline is a robust solution for managing Avro-to-Parquet transformations in big data applications. Understanding the structural differences between Avro and Parquet, as well as the conversion process, enables data engineers to design efficient and scalable data pipelines.