跳到主要内容

Deep Dive into Avro and Parquet Formats

· 阅读需 16 分钟

Introduction​

As data systems grow in complexity and scale, choosing efficient data serialization formats becomes crucial. This article provides a deep dive into Apache Avro and Apache Parquet, two popular formats in the Big Data ecosystem. Both formats have distinct characteristics, making them suitable for different types of data and use cases.


Understanding Avro Format​

Apache Avro is a data serialization format that facilitates the encoding of complex data structures into a compact, binary format, making it highly efficient for streaming and storage.

Avro Schema​

The Avro schema describes the structure of data stored in Avro format and is defined using JSON. Avro schemas allow the specification of nested fields, and each schema comprises two parts:

  • Primitive types: null, boolean, int, long, float, double, bytes, and string.
  • Complex types: record, enum, array, map, union, and fixed.

Example Avro Schema:

{
"type": "record",
"name": "Employee",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": null},
{"name": "salary", "type": "double"}
]
}

In this schema:

  • The Employee record has fields id, name, email, and salary.
  • The email field is optional, allowing null values.

Avro Schema in Depth​

Apache Avro is a data serialization framework with a focus on compact storage, schema evolution, and interoperability, especially popular in data pipelines with tools like Apache Kafka, Hadoop, and Spark. Avro schema is a JSON-based definition that describes the structure, types, and constraints of the data. This schema allows for efficient serialization and deserialization, making it versatile for data storage and streaming applications.

Avro’s schema format is highly adaptable, supporting both simple and complex data structures and allowing schema evolution without disrupting data compatibility. This makes Avro particularly suited for big data pipelines and environments where schemas may need frequent updates. Its JSON-based structure is easy to read and interpret while enabling efficient storage and retrieval in binary format, ideal for high-performance data processing.

Here’s an in-depth look at Avro schemas:

1. Schema Basics​

Avro schemas are written in JSON and define the data types and structure. A schema specifies:

  • Type: The data type, like string, int, float, or complex types like record and array.
  • Name: The name of the field (only required for named types like record).
  • Namespace: Optional, used to provide a unique context for named types, helpful in managing type names in larger systems.
  • Fields: Defines each field within complex types, like record.

Example of a simple Avro schema:

{
"type": "record",
"name": "User",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": "string"}
]
}

This schema defines a User record with three fields: id, name, and email.

2. Primitive Data Types​

Avro supports several primitive data types:

  • Null: Represents a null value.
  • Boolean: Represents true or false.
  • Int: 32-bit signed integer.
  • Long: 64-bit signed integer.
  • Float: 32-bit IEEE floating-point number.
  • Double: 64-bit IEEE floating-point number.
  • Bytes: Sequence of bytes, useful for binary data.
  • String: Unicode character sequence.

3. Complex Data Types​

Avro supports complex data types to handle more advanced data structures:

  • Record: Similar to a structured data object or table row; contains fields, each with its own name and type.
  • Enum: Defines a list of allowed values.
  • Array: An ordered collection of values of a specified type.
  • Map: A collection of key-value pairs with string keys and values of a specified type.
  • Union: Allows fields to have multiple types, which supports optional fields and complex structures.
  • Fixed: Represents a fixed-size binary value, useful for things like fixed-length IDs.

Example of complex types:

{
"type": "record",
"name": "Person",
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": ["null", "int"], "default": null},
{"name": "emails", "type": {"type": "array", "items": "string"}},
{"name": "address", "type": {
"type": "record",
"name": "Address",
"fields": [
{"name": "street", "type": "string"},
{"name": "city", "type": "string"}
]
}}
]
}

This schema defines a Person record with an optional age, a list of emails, and a nested Address record.

4. Union Types​

Unions in Avro schemas allow a field to hold one of multiple types, helping to represent optional fields:

  • A union is represented by an array of types, such as ["null", "int"] for a nullable integer.
  • The first type in the union is considered the default.

Example:

{"name": "birthdate", "type": ["null", "string"], "default": null}

Here, birthdate is either null or a string, allowing it to be optional.

5. Default Values​

  • Fields can have default values if they are missing in the data, which is particularly useful when evolving schemas.
  • Default values must match the field type and are defined within the schema.

Example:

{"name": "isActive", "type": "boolean", "default": true}

6. Schema Evolution​

Avro is designed to handle schema evolution, meaning new schemas can be applied to existing data files without breaking compatibility:

  • Backward Compatibility: New schema can read old data (e.g., adding new fields with default values).
  • Forward Compatibility: Old schema can read new data (e.g., adding optional fields).
  • Full Compatibility: Both backward and forward compatible.

Schema evolution rules include:

  • Adding a new field with a default value is backward-compatible.
  • Removing a field is backward-compatible if it is not required by consumers.
  • Changing a field type requires careful consideration to ensure compatibility.

7. Named Types and Namespaces​

In Avro, named types like record and enum require unique names, often scoped by namespaces:

  • Name: The unique identifier for a type.
  • Namespace: Provides context to prevent naming conflicts, similar to a package in Java or a module in Python.

8. Schema Example with Namespaces​

Example using namespace for managing complex data:

{
"namespace": "com.example",
"type": "record",
"name": "Customer",
"fields": [
{"name": "id", "type": "int"},
{"name": "contact", "type": {
"type": "record",
"name": "ContactInfo",
"fields": [
{"name": "email", "type": "string"},
{"name": "phone", "type": ["null", "string"], "default": null}
]
}}
]
}

Schema Registry​

Schema Registry is a crucial component that allows the storage and retrieval of schemas used by producers and consumers of data. It ensures that producers and consumers agree on the data structure, supporting schema evolution and compatibility checks.

Diagram: Schema Registry Workflow

+-----------+ +--------------------+
| Producer | Write | Schema Registry |
| | ----------> | |
+-----------+ +--------------------+
|
|
| Write
v
+-----------+ +-----------------+
| Data | Read/Write | Consumer |
| Storage | <--------------> | |
+-----------+ +-----------------+
  • Producers register the schema before sending data to storage.
  • Consumers fetch the schema from the registry to deserialize and understand the data.

Storage Architecture for Avro in HDFS/S3​

When storing Avro data in HDFS or S3, Avro files contain both the schema and data, allowing each file to be fully self-contained and readable independently.

Diagram: Avro Storage in HDFS/S3

+------------------------+
| HDFS or S3 Storage |
+------------------------+
|
+------------------------+
| Avro File |
| + Schema |
+------------------------+
|
+------------------------+
| Blocks |
+------------------------+
  • Each Avro file contains schema and data blocks, making it easily portable across storage platforms.
  • Data Access: Avro files can be read without external dependencies, as each file includes the required schema.

Sample Code in Spark​

Below is a Spark example that reads data from an Avro file, processes it, and writes it back to HDFS as Avro format.

import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.avro._

// Initialize Spark session
val spark = SparkSession.builder
.appName("AvroExample")
.getOrCreate()

// Define the Avro schema as JSON
val avroSchema = """
{
"type": "record",
"name": "Employee",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": null},
{"name": "salary", "type": "double"}
]
}
"""

// Read Avro file from HDFS
val df: DataFrame = spark.read
.format("avro")
.load("hdfs://path/to/input/avro/file")

// Process the DataFrame (example: filter employees with salary > 5000)
val filteredDf = df.filter("salary > 5000")

// Write back to HDFS in Avro format
filteredDf.write
.format("avro")
.option("avroSchema", avroSchema)
.save("hdfs://path/to/output/avro/file")

Exploring Parquet Format​

Apache Parquet is a columnar storage format optimized for complex nested data structures, making it an ideal choice for big data processing.

Parquet Schema and Compression​

Parquet organizes data in columns, allowing for efficient compression and encoding. This format is particularly beneficial for queries that scan specific columns.

Compression Techniques:

  • Dictionary Encoding: Reduces data size by replacing duplicate values with codes.
  • Bit Packing: Optimizes storage by packing multiple values into smaller data units.

Diagram: Parquet File Structure

+------------------------------+
| Parquet |
+------------------------------+
| Row Group 1 |
| +--------------------+ |
| | Column 1 | |
| | Column 2 | |
| | ... | |
| +--------------------+ |
| |
| Row Group 2 |
| +--------------------+ |
| | Column 1 | |
| | Column 2 | |
| | ... | |
| +--------------------+ |
+------------------------------+
  • Each Row Group contains columns stored together, optimizing compression and query performance.
  • Columnar storage allows quick access to specific fields, reducing read and write times.

Parquet Schema​

Parquet’s schema defines the structure, types, and layout of the data stored in the file, making it highly efficient for analytics and queries. This schema is stored in the file’s metadata and allows query engines to interpret the data without needing to read the entire file, which speeds up operations significantly.

The Parquet schema plays a crucial role in its columnar storage efficiency, managing data sparsity, and enabling rich, nested data structures with low storage overhead. It facilitates schema evolution, where additional columns or fields can be added without rewriting the entire file, and supports advanced filtering and optimizations for large data analytics tasks.

Here’s an overview of the main components of the Parquet schema:

1. Schema Hierarchy and Structure​

Parquet schema is organized hierarchically and supports complex nested structures:

  • Message Type: This is the root element in a Parquet schema, defining the data structure at the highest level.
  • Fields: Each field in the schema has a name, data type, and potentially nested subfields if the data structure is complex. Parquet supports:
    • Primitive Types: such as int32, int64, boolean, float, double, and binary.
    • Nested Types: Parquet can store complex structures like lists, maps, and structs, which are stored as nested types in the schema.

For example, a Parquet schema could look like:

message schema {
required int32 id;
optional group address {
required binary street (UTF8);
optional int32 zip_code;
}
}

This schema describes a record with an id and an optional nested address group, which contains a street and an optional zip_code.

2. Column Definition: Required, Optional, Repeated​

Each field in the Parquet schema has a repetition level:

  • Required: Every row must contain a value for this field.
  • Optional: A field that can be present or absent in each row.
  • Repeated: Used to model lists or repeated elements, allowing multiple values for the field in a row.

3. Definition and Repetition Levels​

Parquet’s schema model is hierarchical and optimized for efficient storage and retrieval in columnar format. It uses Definition Levels and Repetition Levels to handle nullability, optional fields, and nested data structures. Let’s walk through an example to show how these levels work.

To manage complex data structures and sparsity, Parquet uses definition levels and repetition levels:

  • Definition Levels track whether each level in the hierarchy has data or is null, it tells us how deep in the structure a field is defined.
  • Repetition Levels track whether an element is the first instance or a repeated instance in nested lists or repeated fields, it tells us if the field is part of a repeated structure and, if so, how deep the repeat goes.

For instance, if a row has a list of addresses, the repetition levels can indicate the position of each address within the list, while definition levels show whether each element is present or missing.

Example Parquet Schema​

Consider the following JSON structure, which could be a nested data schema for a Parquet file:

{
"name": "Alice",
"contacts": [
{
"type": "email",
"address": "alice@example.com"
},
{
"type": "phone",
"number": null
}
],
"addresses": [
{
"street": "123 Main St",
"city": "Wonderland"
}
]
}

Here’s the corresponding Parquet schema:

message Person {
required binary name (UTF8);
repeated group contacts {
required binary type (UTF8);
optional binary address (UTF8);
optional binary number (UTF8);
}
repeated group addresses {
required binary street (UTF8);
required binary city (UTF8);
}
}
Parquet Structure: Definition and Repetition Levels​
  1. Definition Level: Indicates the presence or absence of values at each level in a schema. This level is used to define nullability and optional fields.
  2. Repetition Level: Indicates if a field is part of a repeated structure (e.g., a list or an array). This is useful for nested or repeated structures, telling the reader if a given level repeats within the hierarchy.

Let’s break down how these levels work for each field in the example above:

1. Field: name​
  • Definition Level: 1 (required field, so it's always defined)
  • Repetition Level: 0 (not repeated)
2. Field: contacts​
  • Definition Level: 1 to 2, depending on the presence of address and number fields.
    • If both address and number are null, the definition level is lower.
  • Repetition Level: 1 (this field is repeated since contacts is a list)
3. Fields Inside contacts: type, address, number​
  • type
    • Definition Level: 2 (required field within the repeated group)
    • Repetition Level: 1
  • address and number
    • Definition Level: 2 or 3 (optional fields)
      • A definition level of 2 means the contacts object is present but address or number is null.
      • A definition level of 3 indicates both fields are filled.
    • Repetition Level: 1 (repeated inside contacts)
4. Field: addresses​
  • Definition Level: 1 to 2
  • Repetition Level: 1 (repeated field)
5. Fields Inside addresses: street, city​
  • Both street and city are required within the repeated addresses group.
  • Definition Level: 2
  • Repetition Level: 1 (both are repeated with each addresses group entry)
Example Encoding with Levels​

Assuming data with these values:

{
"name": "Alice",
"contacts": [
{
"type": "email",
"address": "alice@example.com",
"number": null
},
{
"type": "phone",
"number": "123-456-7890"
}
],
"addresses": [
{
"street": "123 Main St",
"city": "Wonderland"
}
]
}

The encoded levels might look like this:

FieldValueDefinition LevelRepetition Level
name"Alice"10
contacts[0].type"email"21
contacts[0].address"alice@example.com"31
contacts[0].numbernull21
contacts[1].type"phone"21
contacts[1].addressnull21
contacts[1].number"123-456-7890"31
addresses[0].street"123 Main St"21
addresses[0].city"Wonderland"21

These levels help Parquet efficiently handle complex nested and repeated data structures, enabling better data compression and faster querying by only storing relevant levels and values for each entry.

4. Logical Types and Annotations​

Parquet supports logical types that provide additional context for primitive types:

  • Logical Types: These are type annotations like UTF8 for text strings, DATE for dates, and DECIMAL for high-precision decimals.
  • Logical types help map data from systems that use richer data types (like SQL databases) to Parquet’s primitive types.

Example:

optional int32 birthdate (DATE);

5. Column Index and Row Group Metadata​

  • Column Index: Parquet files include metadata about each column, such as min/max values, null counts, and distinct counts. This allows query engines to skip entire blocks if they don't match the filter criteria.
  • Row Group Metadata: Parquet stores data in row groups (blocks of rows), each containing column data for that group. Each row group stores metadata like the number of rows and data size, which helps with efficient reads.

Example of a Parquet Schema in JSON Representation​

A Parquet schema can also be represented in JSON format to show the hierarchy and structure clearly. For instance:

{
"name": "schema",
"type": "message",
"fields": [
{
"name": "id",
"type": "INT32",
"repetition_type": "REQUIRED"
},
{
"name": "address",
"type": "group",
"repetition_type": "OPTIONAL",
"fields": [
{
"name": "street",
"type": "BINARY",
"logicalType": "UTF8",
"repetition_type": "REQUIRED"
},
{
"name": "zip_code",
"type": "INT32",
"repetition_type": "OPTIONAL"
}
]
}
]
}

In this JSON schema:

  • The id field is required.
  • The address field is optional and has two nested fields: a required street and an optional zip_code.

Storage Architecture for Parquet in HDFS/S3​

In HDFS/S3 storage, Parquet files are organized in row groups with each column stored independently within those groups.

Diagram: Parquet Storage in HDFS/S3

+------------------------+
| HDFS or S3 Storage |
+------------------------+
|
+------------------------+
| Parquet File |
+------------------------+
|
+------------------------+
| Row Groups |
+------------------------+
|
+------------------------+
| Columns within Groups |
+------------------------+
  • Row Groups: Partitioned sections that allow for distributed reads.
  • Columns: Stored separately within each row group for efficient access.

Sample Code in Spark​

Below is a Spark example that reads data from a Parquet file, performs transformations, and writes it back to HDFS as Parquet format.

import org.apache.spark.sql.{DataFrame, SparkSession}

// Initialize Spark session
val spark = SparkSession.builder
.appName("ParquetExample")
.getOrCreate()

// Read Parquet file from HDFS
val df: DataFrame = spark.read
.parquet("hdfs://path/to/input/parquet/file")

// Process the DataFrame (example: select employees with salary > 5000)
val filteredDf = df.filter("salary > 5000")

// Write back to HDFS in Parquet format
filteredDf.write
.parquet("hdfs://path/to/output/parquet/file")

Conclusion​

This article provided an in-depth look at Avro and Parquet formats, exploring their unique characteristics, schema management, and storage architectures. With examples in Spark, we demonstrated practical implementations in a real-world production environment, highlighting the versatility of both formats for big data solutions.

Deep Dive into Avro and Parquet Together

· 阅读需 14 分钟

Introduction​

In modern data architectures, Avro data is often used in messaging systems like Apache Kafka due to its compact binary format and schema evolution support. However, in data lakes or data warehouses, Parquet is preferred for storage due to its columnar format and efficient read performance. This article covers the architecture, pipeline design, and real-life sample code for implementing a pipeline that ingests Avro data from a messaging system and saves it in Parquet format in a big data storage solution like Amazon S3.


Architecture Overview​

This section provides an overview of the components in a typical architecture that ingests Avro from a messaging system and saves it as Parquet in big data storage.

Diagram: Avro to Parquet Pipeline Architecture

Messaging System (Kafka) ----> Stream Processing (Spark) ----> S3 Storage (Parquet Format)

### Avro and Parquet Formats

  • Avro: A row-based binary format, commonly used for serializing data in messaging systems due to its compact size and schema evolution capabilities.
  • Parquet: A columnar storage format optimized for reading specific columns, making it highly efficient for analytics and big data storage solutions.

### Role of Messaging Systems

Messaging systems like Apache Kafka are widely used to transport streaming data across systems in real time. Kafka provides fault-tolerance and durability, ensuring data is safely delivered to downstream applications for processing.

### Storage in Big Data Storage Solutions

Big data storage solutions, such as Amazon S3, offer durable and cost-effective storage for large-scale data. Storing data in Parquet format on S3 enables easy integration with data analytics frameworks like Spark, Presto, and Hive.


Data Pipeline Design​

The pipeline includes the following key stages:

  1. Data Ingestion: Consuming Avro-encoded messages from Kafka.
  2. Processing and Transformation: Converting Avro data to Parquet format.
  3. Storage: Saving the transformed Parquet files to Amazon S3.

Ingesting Data from Messaging System​

The first step in the pipeline is ingesting data from a messaging system like Kafka. Spark Structured Streaming can be used to consume Avro-encoded messages from Kafka topics in real-time.

Diagram: Ingesting Avro Data from Kafka

Kafka Topic (Avro Messages) ---> Spark Structured Streaming ---> Data Transformation

Processing and Converting Avro to Parquet​

Spark Structured Streaming allows us to process and transform the incoming Avro data. To convert Avro to Parquet, the schema needs to be mapped, and the data is then serialized to the Parquet format.

Diagram: Converting Avro to Parquet in Spark

Avro Data (Spark DataFrame) ---> Schema Mapping ---> Parquet Data

Saving to Big Data Storage (S3)​

Once converted to Parquet, the data can be written to Amazon S3 in a structured format. Parquet files in S3 are organized by partitions (e.g., by date or another logical grouping) to optimize retrieval and querying performance.

Diagram: Saving to S3

Parquet Data (Partitioned by Date) ---> Amazon S3 Bucket ---> Analytics-ready Storage

Understanding the Structural Differences​

In Avro and Parquet formats, the data organization is fundamentally different:

  • Avro: A row-based format where each record is serialized as a single entity, storing all fields of each row together.
  • Parquet: A columnar storage format where data is grouped by columns rather than rows, optimizing for efficient reads on specific columns (ideal for analytical workloads).

To illustrate, let’s consider an Avro record with two attributes:

{
"name": "John Doe",
"age": 30
}

In Avro, the above record would be stored as a single contiguous block of binary data, with fields appearing in the order defined by the schema.

When converting this record to Parquet, the storage layout changes to a column-oriented format. Parquet organizes data by columns:

  • Each column (e.g., "name" and "age") is stored separately in column chunks.
  • Parquet organizes records into row groups, where each row group has column chunks for each column defined in the schema.

Diagram: Avro to Parquet Conversion for Record { "name": "John Doe", "age": 30 }

Parquet Column ChunkData Stored
Name Column ChunkJohn Doe
Age Column Chunk30

In a more extensive dataset with thousands of records, Parquet’s layout significantly optimizes read performance because each column chunk can be accessed independently. For instance, if only the "age" column is required for analysis, Parquet reads only the relevant chunk, skipping over unrelated columns like "name."


Avro to Parquet Conversion​

The steps for converting Avro to Parquet in a system like Apache Spark generally follow this sequence:

  1. Avro Data Loading:

    • Avro-encoded data is parsed and loaded into an intermediate in-memory structure, such as a DataFrame in Spark.
    • Avro schema is either read from the Avro file (or topic in case of Kafka) or supplied separately.
  2. Schema Mapping:

    • Spark maps the Avro schema to an equivalent internal schema (e.g., Spark schema), which is essential to support Parquet’s columnar structure.
    • Fields in the Avro schema (like "name" and "age") are mapped to equivalent columns.
  3. Data Conversion:

    • Each Avro record is then serialized into a format compatible with Parquet’s column-oriented layout.
    • Each attribute is transformed to be written in column chunks rather than row-by-row.
  4. Data Writing (Parquet):

    • Parquet organizes data into row groups, and within each row group, column data is stored in contiguous chunks.
    • Additional metadata (e.g., min, max, dictionary encoding) is recorded for each column chunk, which is useful for optimizations in querying.

Example: In Spark, a sample code to convert Avro to Parquet may look like this:

// Load Avro data into a DataFrame
Dataset<Row> avroDF = spark.read().format("avro").load("path/to/avro/file");

// Write data to Parquet format
avroDF.write().parquet("path/to/output/parquet");

This process reorganizes the data from a row-based layout (Avro) to a columnar layout (Parquet).


Reading from Parquet and Converting Back to Avro​

When reading Parquet data and converting it back to Avro, the system reverses the process, transforming columnar data back into row-based data.

Steps:

  1. Load Parquet Data:

    • Parquet files are read into memory. In a Spark-based setup, this would involve loading Parquet files into a DataFrame.
  2. Column Extraction:

    • Parquet retrieves data by accessing each relevant column chunk. Parquet's metadata allows efficient navigation to the specific data of interest.
  3. Row Assembly:

    • As each column’s data is accessed, the system assembles rows by gathering data from each column chunk.
    • The row-based structure compatible with Avro is recreated by combining values from each column for every row.
  4. Schema Mapping:

    • Parquet columns are mapped to Avro fields based on a schema. This schema alignment ensures that columns match Avro’s row-oriented requirements.
    • The DataFrame (or equivalent data structure) is serialized back into the Avro format.
  5. Write to Avro:

    • The row-based structure is serialized in Avro’s binary format, and the schema is either embedded or referenced as needed.

Example Code: Converting Parquet to Avro in Spark

// Load Parquet data into a DataFrame
Dataset<Row> parquetDF = spark.read().parquet("path/to/parquet/file");

// Write data back to Avro format
parquetDF.write().format("avro").save("path/to/output/avro");

This code converts the columnar Parquet format back into Avro’s row-based format.


Parquet’s Row Groups and Column Chunks​

In Parquet, data is stored in row groups. Each row group contains data for a set number of rows, and within each row group, data is stored in column chunks (one for each column). The row group is Parquet’s key structural element that ensures the association between column data across multiple rows:

  • Row Group: A set of rows is organized as a contiguous data block. Each row group in Parquet represents a batch of rows from the original dataset.
  • Column Chunk: Within each row group, each column’s data is stored in a separate column chunk.

For example, if we have a dataset with 10,000 records, it might be divided into several row groups (say 5,000 rows per row group). Each row group then contains the data for the columns name and age in its respective column chunks.

Ensuring Data Belongs to the Same Record​

To ensure data from each column belongs to the same record, Parquet relies on row ordering within each row group. The ordering within each column chunk of a row group matches across all columns:

  • Ordering: Each row in a row group corresponds to the same index across all column chunks within that row group. For instance, the first value in the name column chunk of a row group corresponds to the first value in the age column chunk for the same row group.
  • Offset Metadata: Parquet uses metadata about the offsets within each column chunk to ensure each row can be reconstructed correctly when reading.

So, in your example, the row group might look like this:

Row IndexName Column ChunkAge Column Chunk
Row 1"John Doe"30
Row 2"Jane Smith"25
Row 3"Alice Brown"28

Each entry in the name and age column chunks corresponds by index. Parquet uses metadata within the row group and column chunks to indicate where each row begins and ends, making sure that during reads, each row’s data across columns is aligned.

Reading from Parquet and Converting Back to Avro​

When reading Parquet data back into a row-based format (like Avro), the system iterates over each row group:

  • Row Assembly: For each row in a row group, Parquet retrieves the respective values from each column chunk based on their row index. This reassembles each row’s fields together to form the original record.
  • Schema Mapping: The column metadata stored in Parquet includes the schema, allowing the correct assignment of values back to each field in the record.

What if Null Values in Parquet​

In Parquet, each column chunk within a row group holds data in columnar format, and Parquet efficiently handles cases where some rows may lack a value for a given column by using specific encoding and metadata, rather than empty placeholders. Here’s how it manages the scenario:

1. Null Values in Columns​

If a column lacks a value for a particular row, Parquet does not insert an empty placeholder. Instead:

  • Definition Levels are used to indicate whether a value is present or null. Each level specifies if a field has a value or is missing at that row.
  • Repetition Levels track nested structures within rows, so nested nulls and repeated fields can be handled in one scan.

2. Encodings for Efficient Storage​

Parquet uses encodings like Run-Length Encoding (RLE) and Dictionary Encoding to represent repeated or missing values compactly:

  • Run-Length Encoding groups consecutive null values (or any repeating values) into a single representation, saving space when large chunks are missing in a column.
  • Dictionary Encoding can efficiently map common values (like repeated nulls or default values) using dictionary lookup tables.

3. Metadata Tracking and Optional Fields​

  • If a field is defined as optional in the schema, it allows a row to be saved without a value for that field. Parquet only includes these rows in the row group metadata, which helps query engines like Spark identify missing values.

Example Scenario​

Let’s say we have a Parquet file with three rows and three columns, where one column lacks values for some rows.

Row GroupColumn AColumn BColumn C
Row 1Value 1NullValue 3
Row 2Value 4Value 5Null
Row 3NullValue 6Value 7

In this case:

  • Definition Levels for Column B and Column C would show which rows contain nulls.
  • The data structure itself would store only the non-null values, reducing storage without explicit placeholders.

This approach is particularly useful when handling sparse data, as it minimizes the storage footprint and increases reading efficiency.

Example in Spark​

Here’s how Spark handles this under the hood when reading Parquet data and converting it back to Avro:

// Reading from Parquet
Dataset<Row> parquetDF = spark.read().parquet("path/to/parquet/file");

// Writing back to Avro, reassembling rows
parquetDF.write().format("avro").save("path/to/output/avro");

During this process, Spark ensures that each row’s values across columns are correctly aligned by leveraging Parquet’s row group and column chunk metadata. This alignment allows it to convert back to Avro’s row-based format without losing the association between name and age in each record.


Real-Life Production Sample Code in Spark​

The following sample code demonstrates how to implement this pipeline in Apache Spark with Structured Streaming.

Reading from Kafka​

Use Spark to read Avro data from a Kafka topic in real-time.

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.avro.functions.from_avro;

public class AvroToParquetPipeline {

public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.appName("Avro to Parquet Pipeline")
.getOrCreate();

String kafkaBootstrapServers = "kafka-broker1:9092,kafka-broker2:9092";
String kafkaTopic = "avro-topic";

Dataset<Row> avroDF = spark
.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootstrapServers)
.option("subscribe", kafkaTopic)
.load()
.selectExpr("CAST(value AS BINARY) as avroData");

// Define the Avro schema (can be fetched from Schema Registry if available)
String avroSchema = "{ \"type\": \"record\", \"name\": \"User\", \"fields\": [ {\"name\": \"id\", \"type\": \"int\"}, {\"name\": \"name\", \"type\": \"string\"}, {\"name\": \"age\", \"type\": \"int\"} ] }";

Dataset<Row> parsedDF = avroDF
.select(from_avro(avroDF.col("avroData"), avroSchema).as("data"))
.select("data.*");

parsedDF.printSchema(); // Debugging schema after parsing Avro
}
}

Converting Avro to Parquet​

Once we have parsed the Avro data into a DataFrame, we can convert it into Parquet format.

Dataset<Row> parquetDF = parsedDF.repartition(1); // Optional: Customize partitioning as needed

Writing to S3​

Writing the data to Amazon S3 requires specifying the destination path and configuring partitioning if necessary.

import org.apache.spark.sql.streaming.Trigger;

String s3Path = "s3a://your-bucket/path/to/destination/";

parquetDF.writeStream()
.format("parquet")
.option("path", s3Path)
.option("checkpointLocation", "s3a://your-bucket/path/to/checkpoint/")
.partitionBy("date") // Optional: Partitioning by a specific column (e.g., date)
.trigger(Trigger.ProcessingTime("5 minutes")) // Controls batch frequency
.start()
.awaitTermination();

Conclusion​

This article covered an architecture for ingesting Avro data from a messaging system, converting it to Parquet, and saving it in Amazon S3. Using Spark Structured Streaming and Amazon S3 for storage, this pipeline is a robust solution for managing Avro-to-Parquet transformations in big data applications. Understanding the structural differences between Avro and Parquet, as well as the conversion process, enables data engineers to design efficient and scalable data pipelines.

Deep Dive into Java Core Multi-Threading Concepts

· 阅读需 5 分钟

Java multithreading is a fundamental part of the language that allows concurrent execution of multiple parts of a program to achieve greater efficiency and responsiveness. This article explores the latest Java version's core multithreading concepts, the Java Memory Model (JMM), thread lifecycle, key classes, and real-life examples.

1. Java Thread Model​

Java's thread model is based on the OS thread model, providing abstraction to manage and create threads directly. Java threads allow programs to perform multiple tasks concurrently, improving resource utilization. Thread priorities help in determining the order of thread execution, though scheduling ultimately depends on the underlying OS.

2. Java Memory Model (JMM)​

Java Thread Memory Model Diagram​

+-------------------+ +-------------------+
| | | |
| Main Memory |<----->| Thread 1 |
| | | |
+-------------------+ +-------------------+
| |
| |
v v
+-------------------+ +-------------------+
| | | |
| Thread 2 |<----->| Thread 3 |
| | | |
+-------------------+ +-------------------+

The Java Memory Model (JMM) defines how threads interact through memory and what values can be read or written by threads. The JMM provides:

  1. Visibility: Changes made by one thread are visible to other threads.
  2. Atomicity: Certain actions are indivisible or atomic.
  3. Ordering: The sequence in which instructions are executed.

JMM Key Concepts:​

  • Happens-Before Relationship: Ensures that memory writes by one specific action are visible to another specific action.
  • Volatile Keyword: Variables declared as volatile ensure visibility of changes across threads.
  • Synchronized Blocks: Using synchronized enforces an order of access to variables between threads.

3. Thread Lifecycle and State Machine​

Java threads go through several states:

StateDescription
NewThe thread is created but not yet started.
RunnableThe thread is eligible to run but may be waiting for a CPU cycle.
BlockedThe thread is waiting to acquire a lock to enter a synchronized block.
WaitingThe thread is waiting indefinitely for another thread to perform an action.
Timed WaitingThe thread is waiting for another thread for a specified period.
TerminatedThe thread has completed execution.

Thread State Machine Diagram​

NEW -> Runnable -> BLOCKED -> WAITING <-> TIMED_WAITING -> TERMINATED
+-------------------+ +-------------------+
| | | |
| NEW |------>| RUNNABLE |
| | | |
+-------------------+ +-------------------+
| |
| |
v v
+-------------------+ +-------------------+
| | | |
| BLOCKED |<----->| WAITING |
| | | |
+-------------------+ +-------------------+
| |
| |
v v
+-------------------+ +-------------------+
| | | |
| TIMED_WAITING |<----->| TERMINATED |
| | | |
+-------------------+ +-------------------+

4. Key Threading Classes in Java​

4.1 Thread Class​

The Thread class provides methods to create, start, and manage threads. You can define a custom thread by extending the Thread class and overriding its run() method.

Example:

public class MyThread extends Thread {
@Override
public void run() {
System.out.println("Thread is running");
}
}

MyThread thread = new MyThread();
thread.start();

4.2 Runnable Interface​

The Runnable interface is a functional interface representing a task to be executed on a thread. It separates the task from the thread itself, allowing reusability and flexibility.

Example:

public class MyRunnable implements Runnable {
@Override
public void run() {
System.out.println("Runnable is running");
}
}

Thread thread = new Thread(new MyRunnable());
thread.start();

4.3 Callable and Future​

Callable allows tasks to return results or throw checked exceptions, while Future represents the result of an asynchronous computation.

Example:

Callable<Integer> task = () -> {
Thread.sleep(1000);
return 123;
};

ExecutorService executor = Executors.newFixedThreadPool(1);
Future<Integer> future = executor.submit(task);

Integer result = future.get(); // blocks until the result is available
executor.shutdown();

4.4 Executors Framework​

The Executors framework provides a higher-level API for thread management, thread pooling, and scheduling, abstracting the creation and management of threads.

Example:

ExecutorService executor = Executors.newFixedThreadPool(5);
for (int i = 0; i < 10; i++) {
executor.submit(() -> {
System.out.println("Task running");
});
}
executor.shutdown();

5. Synchronization and Locks​

Java provides several ways to control access to shared resources, including synchronized methods and blocks, the ReentrantLock class, and the ReadWriteLock.

Synchronized​

The synchronized keyword ensures that only one thread can access a method or block at a time.

public synchronized void increment() {
count++;
}

ReentrantLock​

ReentrantLock provides explicit lock mechanisms, giving more control over thread coordination.

Lock lock = new ReentrantLock();
lock.lock();
try {
// critical section
} finally {
lock.unlock();
}

6. Real-Life Production Example​

Consider a production scenario where we have multiple tasks needing parallel execution but require control over access to shared resources. Here’s a multithreaded solution using a ScheduledExecutorService in a banking transaction processing system.

import java.util.concurrent.*;

public class TransactionProcessor {
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(3);

public void startProcessing() {
Runnable task = () -> {
// Simulate fetching and processing a transaction
System.out.println("Processing transaction: " + Thread.currentThread().getName());
try {
Thread.sleep(2000); // Simulating a transaction processing delay
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
};

// Schedule at fixed rate
scheduler.scheduleAtFixedRate(task, 0, 5, TimeUnit.SECONDS);
}

public static void main(String[] args) {
TransactionProcessor processor = new TransactionProcessor();
processor.startProcessing();
}
}

Explanation​

  • We use ScheduledExecutorService for periodic transaction processing.
  • The thread pool’s scheduleAtFixedRate method ensures that the task runs every 5 seconds, enabling controlled execution with set intervals.
  • This design is beneficial in systems where tasks need continuous or scheduled processing but must prevent overloading resources.

This article provides an overview of multithreading concepts, memory models, core classes, and real-world examples. Java’s concurrency mechanisms offer flexibility and control, making it essential to use these constructs carefully to manage resources efficiently and avoid pitfalls such as deadlocks and race conditions.

Deep Dive into Java Core Multi-Threading

· 阅读需 6 分钟

This article explores Java’s thread pool concept, detailing each component and state involved. We’ll cover tasks, threads, workers, thread pools, and the state transitions that occur during task execution. Detailed diagrams and annotated code examples demonstrate how each part interacts, making this a practical reference for developing multithreaded Java applications.


1. Introduction​

Java’s ThreadPoolExecutor provides a robust framework for managing a pool of worker threads that can efficiently process a series of asynchronous tasks. It’s designed to handle tasks without creating new threads for every request, conserving resources and reducing overhead. This article breaks down each term and component involved in a Java thread pool to provide a clear, practical understanding.


2. Java Thread Pool: Overview and Components​

A thread pool is a collection of threads that are managed to execute tasks concurrently. Rather than creating a new thread for every task, a pool of threads is reused, allowing efficient execution without constant thread creation and destruction.

Here’s a textual diagram illustrating a typical Java thread pool architecture:

+-------------------------------------------+
| Thread Pool |
|-------------------------------------------|
| |
| Task Queue --> [Task1] [Task2] [TaskN] |
| |
| +-------------------------------+ |
| | Worker Thread | |
| |-------------------------------| |
| | |--Thread-1--|--Running--| | |
| | |--Thread-2--|--Runnable-| | |
| | |--Thread-N--|--Waiting--| | |
| +-------------------------------+ |
+-------------------------------------------+

3. Core Terms Explained​

Task​

  • Definition: A Runnable or Callable object representing a unit of work.
  • Purpose: The task contains logic that needs to be executed by a thread in the pool.

Thread​

  • Definition: An individual worker capable of executing a task.
  • States: Threads can be in various states, including NEW, RUNNABLE, WAITING, TIMED_WAITING, BLOCKED, and TERMINATED.

Worker​

  • Definition: An entity within the thread pool that wraps a thread, providing task management.
  • Role: Handles the execution of tasks by submitting them to a thread in the pool.

Thread Pool​

  • Definition: A collection of threads managed by an executor that oversees thread lifecycle and task allocation.
  • Purpose: To improve performance by reusing threads for multiple tasks, avoiding the cost of creating and destroying threads frequently.

4. Thread Lifecycle in a Thread Pool​

A thread in a pool goes through multiple states throughout its lifecycle. Let’s illustrate these states and transitions with a state diagram.

+-----------+
| NEW | -- Thread created
+-----------+
|
v
+-----------+
| RUNNABLE | -- Thread ready to execute
+-----------+
|
v
+-----------+
| RUNNING | -- Thread actively executing a task
+-----------+
|
v
+-----------+
| BLOCKED | -- Thread blocked, waiting for a resource
+-----------+
|
v
+-----------+
| WAITING | -- Thread waiting indefinitely for a condition
+-----------+
|
v
+-----------+
| TERMINATED| -- Thread completed execution or terminated
+-----------+

5. Thread State Transitions​

Each thread’s state changes as it progresses through its lifecycle:

  1. NEW: The thread is created but has not yet started.
  2. RUNNABLE: The thread is ready to run but is waiting for CPU availability.
  3. RUNNING: The thread is actively running its assigned task.
  4. BLOCKED: The thread is waiting to acquire a lock to access a synchronized resource.
  5. WAITING/TIMED_WAITING: The thread is waiting indefinitely or for a specified time, typically for task completion or resource availability.
  6. TERMINATED: The thread has completed its task and will either be recycled for a new task or discarded if no longer needed.

6. Sample Code and Explanation​

Below is a sample code demonstrating a thread pool with comments explaining each part of the process.

import java.util.concurrent.*;

public class ThreadPoolExample {
public static void main(String[] args) {
// Create a thread pool with a fixed number of threads
ExecutorService threadPool = Executors.newFixedThreadPool(3);

// Submit multiple tasks to the thread pool
for (int i = 1; i <= 5; i++) {
threadPool.submit(new Task(i));
}

// Shutdown the pool after tasks are completed
threadPool.shutdown();
}
}

// Task class implementing Runnable, representing a unit of work for the pool
class Task implements Runnable {
private final int taskId;

public Task(int taskId) {
this.taskId = taskId;
}

@Override
public void run() {
System.out.println("Executing Task " + taskId + " with " + Thread.currentThread().getName());
try {
Thread.sleep(2000); // Simulate task duration
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Task " + taskId + " completed.");
}
}

In this example:

  • ThreadPoolExecutor: Manages a pool of three threads.
  • Task: Each task simulates a job that runs for two seconds, allowing us to observe thread reuse.
  • Thread: Each task is assigned a thread by the thread pool, which either executes it immediately or queues it until a thread becomes available.

7. Handling TERMINATED Threads​

What Happens When a Thread Reaches TERMINATED?​

Once a thread in the pool completes its task and reaches the TERMINATED state, it does not automatically return to RUNNABLE. In a fixed pool, the terminated thread is replaced by a new one, ensuring the pool maintains its core size. In cached pools, the thread may be recycled if the task load increases, maintaining efficiency.

Reuse in Fixed Pools​

In fixed-size pools, a thread that terminates due to completion is usually retained within the pool as a worker for future tasks.

State Machine Transition​

The Java thread pool manages thread lifecycles by detecting TERMINATED threads and maintaining the required pool size by creating new threads as needed. This automatic management ensures the pool maintains availability and task-handling capacity.


8. Conclusion​

Java’s thread pool provides an efficient way to handle concurrent tasks by reusing threads and managing their lifecycles. Key concepts like tasks, threads, and workers work together to ensure a scalable, resource-efficient application structure. By understanding thread pool states, configurations, and transition mechanisms, developers can design more resilient and optimized concurrent applications.

Deep Dive into Spring Scheduler

· 阅读需 9 分钟

Introduction​

Spring Scheduler provides a simple way to schedule tasks within a Spring application. It offers flexibility in managing time-based events and task execution, allowing developers to configure tasks in a variety of ways, including sequential, parallel, and conditional execution.


Understanding Spring Scheduler Basics​

Spring Scheduler uses annotations to simplify the scheduling of methods within beans. The primary annotation used is @Scheduled, which can define various timing configurations, such as fixed rate, fixed delay, and cron expressions.

Enabling Scheduling​

Before defining tasks, enable scheduling in your Spring application by adding the @EnableScheduling annotation.

import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableScheduling;

@Configuration
@EnableScheduling
public class SchedulingConfig {
// Any other scheduling-related configuration if needed
}

Simple Scheduling with @Scheduled​

The @Scheduled annotation can be applied to any method to enable scheduling. Below is a simple example:

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
public class SimpleScheduler {

@Scheduled(fixedRate = 5000)
public void printMessage() {
System.out.println("This message prints every 5 seconds.");
}
}

In this example, printMessage() runs every 5 seconds.


Scheduling Scenarios​

Spring Scheduler supports various scenarios for task scheduling, which can be useful for specific use cases in a production environment.

Sequential Job Execution​

In sequential execution, jobs run in a specific order, with each job waiting for the previous one to complete before starting.

Example: Sequential Scheduling Using Fixed Delay

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
public class SequentialScheduler {

@Scheduled(fixedDelay = 10000)
public void job1() {
System.out.println("Job 1 started.");
// simulate a delay
try { Thread.sleep(2000); } catch (InterruptedException ignored) {}
System.out.println("Job 1 completed.");
}

@Scheduled(fixedDelay = 10000)
public void job2() {
System.out.println("Job 2 started after Job 1.");
// simulate a delay
try { Thread.sleep(3000); } catch (InterruptedException ignored) {}
System.out.println("Job 2 completed.");
}
}

With fixedDelay, Job 2 only begins after Job 1 completes.

Parallel Job Execution​

To execute jobs in parallel, we configure a task executor that runs multiple tasks concurrently.

Example: Parallel Scheduling Using a Custom Task Executor

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

@Configuration
@EnableAsync
public class ParallelSchedulingConfig {

@Bean
public ThreadPoolTaskExecutor taskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(2); // Number of concurrent tasks allowed
executor.setMaxPoolSize(5);
executor.initialize();
return executor;
}
}

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Component;

@Component
public class ParallelScheduler {

@Async
@Scheduled(fixedRate = 10000)
public void jobA() {
System.out.println("Parallel Job A started.");
}

@Async
@Scheduled(fixedRate = 10000)
public void jobB() {
System.out.println("Parallel Job B started concurrently with Job A.");
}
}

With @Async, jobA and jobB run concurrently, allowing for parallel execution.

Job Waiting Until Previous Job Completes​

To configure jobs to wait for previous jobs to finish, use fixedDelay with asynchronous methods to ensure that each task waits until its predecessor is completed.

Diagram: Job Waiting until Previous Job Completes

Job 1 ---> Wait ---> Job 2 ---> Wait ---> Job 3
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
public class WaitingScheduler {

@Scheduled(fixedDelay = 12000)
public void job() {
System.out.println("Job started and waits until the previous instance completes.");
// Simulate processing time
try { Thread.sleep(5000); } catch (InterruptedException ignored) {}
System.out.println("Job completed.");
}
}

In this scenario, each new execution waits until the previous one finishes.


Cron Expressions and Fixed Rates​

Spring Scheduler allows defining precise timing patterns using fixed rate, fixed delay, and cron expressions.

Fixed Rate vs Fixed Delay​

  • Fixed Rate: Executes a task at a specified interval, regardless of task completion time.
  • Fixed Delay: Executes a task with a delay between the end of the previous execution and the start of the next.
@Scheduled(fixedRate = 60000) // Executes every 60 seconds
public void fixedRateTask() {
System.out.println("Fixed Rate Task");
}

@Scheduled(fixedDelay = 60000) // Executes with a 60-second delay after completion
public void fixedDelayTask() {
System.out.println("Fixed Delay Task");
}

Using Cron Expressions​

Cron expressions allow precise scheduling based on time. Format: second minute hour day month weekday.

Example: Scheduling a Task for 5:00 AM Every Day

@Scheduled(cron = "0 0 5 * * ?")
public void dailyTask() {
System.out.println("This task runs at 5:00 AM every day.");
}

Conditional Scheduling with Annotations​

Spring allows scheduling tasks based on conditions. Combining @ConditionalOnProperty with @Scheduled, we can configure tasks to run only when certain conditions are met.

Conditional Scheduling Example​

This example schedules a job only if a specific property is set to true.

import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
@ConditionalOnProperty(name = "scheduler.enabled", havingValue = "true")
public class ConditionalScheduler {

@Scheduled(fixedRate = 10000)
public void conditionalTask() {
System.out.println("This task runs only if 'scheduler.enabled=true'.");
}
}

To enable or disable this task, add scheduler.enabled=true or false in the configuration.


All-In-One Sample​

import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

@Configuration
@EnableScheduling
public class ProductionSchedulerConfig {

@Bean
public ThreadPoolTaskExecutor taskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(3);
executor.setMaxPoolSize(5);
executor.initialize();
return executor;
}
}

import org.springframework.scheduling.annotation.Async;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
@ConditionalOnProperty(name = "scheduler.enabled", havingValue = "true", matchIfMissing = true)
public class ProductionScheduler {

@Async
@Scheduled(cron = "0 0 12 * * ?") // Executes at noon daily
public void dailySummaryJob() {


System.out.println("Executing daily summary job...");
}

@Async
@Scheduled(fixedDelay = 15000) // Executes every 15 seconds after the previous execution ends
public void periodicCleanupJob() {
System.out.println("Running periodic cleanup job...");
}
}

ConditionalOnProperty with Scheduled Job in Depth​

The @ConditionalOnProperty annotation in Spring is typically used to conditionally enable or disable beans based on specific configuration properties. In this case:

@ConditionalOnProperty(name = "scheduler.enabled", havingValue = "true")

This will activate the annotated bean only if the scheduler.enabled property is set to true in your configuration (like in application.properties or environment variables).

Scenario: Running a Scheduled Job in Only One Pod in a Kubernetes Cluster​

When deploying a Spring application in a Kubernetes cluster across multiple containers (pods), ensuring that a scheduled job only runs in one specific pod can be tricky because Spring's @Scheduled jobs will run in every instance where the job bean is active. Here’s how to solve this problem by using the @ConditionalOnProperty annotation in combination with Kubernetes configurations.

Solution Outline​

  1. Set a Leader Pod: Configure only one of the pods to have scheduler.enabled=true using Kubernetes annotations and environment variables, making it the leader pod responsible for running the scheduled job.
  2. Configure Pods with Unique Environment Variables: Use Kubernetes configMaps or Secrets to assign the scheduler.enabled property as true only in the leader pod’s environment. Other pods should not have this property set to true, so they won’t run the job.

Steps to Implement​

  1. Create a ConfigMap or Secret for the Scheduler Property:

    • Define a ConfigMap that holds the scheduler.enabled setting for the lead pod.
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: scheduler-config
    data:
    scheduler.enabled: "true"
  2. Use Pod Annotations to Select the Leader Pod:

    • When deploying multiple replicas, use a label selector or custom script to ensure only one pod is given the scheduler.enabled=true setting.
  3. Configure the Deployment to Pass the scheduler.enabled Environment Variable:

    • In your Kubernetes Deployment file, you can use envFrom to load the environment variables from the ConfigMap, but with a selector so only one pod uses this ConfigMap.
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: scheduler-job
    spec:
    replicas: 3 # Running multiple instances
    selector:
    matchLabels:
    app: scheduler-app
    template:
    metadata:
    labels:
    app: scheduler-app
    spec:
    containers:
    - name: scheduler-container
    image: your-image
    envFrom:
    - configMapRef:
    name: scheduler-config
    env:
    - name: POD_NAME
    valueFrom:
    fieldRef:
    fieldPath: metadata.name
  4. Conditional Logic in Spring Boot:

    • With @ConditionalOnProperty, the @Scheduled method will only activate if the scheduler.enabled property is true. So, only the pod with the scheduler.enabled=true environment variable will run the job.
    @Component
    @ConditionalOnProperty(name = "scheduler.enabled", havingValue = "true")
    public class ScheduledJob {

    @Scheduled(fixedRate = 5000)
    public void performTask() {
    System.out.println("Scheduled job is running on the leader pod!");
    }
    }

Additional Consideration: Use Leader Election​

For robust setups, you could implement leader election to dynamically select the leader pod. This can be achieved using tools like Spring Cloud Kubernetes Leader Election or implementing a custom leader election mechanism based on a shared lock in an external data store, such as Redis or ZooKeeper.

By setting scheduler.enabled=true conditionally based on which pod is the elected leader, you can ensure the scheduled job runs in only one container at a time, with the ability to switch seamlessly if the leader pod goes down.

Conclusion​

Spring Scheduler provides flexibility and control over task scheduling. By using @Scheduled with different configurations, we can create complex scheduling workflows that address specific business needs.

Spark in Kubernetes

· 阅读需 7 分钟

This guide covers the steps to deploy an Apache Spark cluster with a master, two workers, and a Spark History Server on Kubernetes. It configures a GCS bucket as the default file system and installs Hive with PostgreSQL as its backend. An Ingress is set up for accessing the Spark UI and History Server.

Architecture Overview​

The setup includes the following components:

  • Spark Master and Workers: Deployed using StatefulSets for stable identity and persistence.
  • History Server: Deployed using a Deployment and configured to store Spark event logs in GCS.
  • Hive and PostgreSQL: PostgreSQL is used as the backend for Hive metadata, and Hive is deployed to store Spark tables.
  • Ingress: Provides external access to Spark Master UI and History Server UI.

Diagram: Architecture Overview​

+---------------------------+
| |
| Kubernetes |
| |
+---------------------------+
| |
| +---------+ +---------+|
| | Spark | | Spark ||
| | Master | | Workers ||
| +---------+ +---------+|
| |
| +-----------+ |
| | History | |
| | Server | |
| +-----------+ |
| | |
| Ingress |
+---------------------------+
|
External Access

Prerequisites​

  • Kubernetes Cluster with access to deploy resources.
  • kubectl command-line tool configured to access the cluster.
  • Helm for deploying Hive and PostgreSQL.
  • Google Cloud Storage (GCS) bucket created for Spark event logs.
  • Google Cloud SDK installed and configured for authentication to GCS.

1. Configuring Google Cloud Storage (GCS) Bucket​

  1. Create a GCS bucket if not already created:

    gsutil mb gs://your-spark-event-logs
  2. Set up a GCS service account key and save it as a Kubernetes secret. This will allow Spark to access GCS:

    kubectl create secret generic gcs-key \
    --from-file=key.json=/path/to/gcs-service-account-key.json

2. Deploying PostgreSQL for Hive Metadata​

PostgreSQL will act as the relational database for Hive’s metadata.

PostgreSQL YAML​

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgresql-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgresql
spec:
replicas: 1
selector:
matchLabels:
app: postgresql
template:
metadata:
labels:
app: postgresql
spec:
containers:
- name: postgresql
image: postgres:13
env:
- name: POSTGRES_DB
value: "hive_metastore"
- name: POSTGRES_USER
value: "hiveuser"
- name: POSTGRES_PASSWORD
value: "hivepassword"
volumeMounts:
- mountPath: "/var/lib/postgresql/data"
name: postgresql-data
volumes:
- name: postgresql-data
persistentVolumeClaim:
claimName: postgresql-pvc

3. Installing Hive Metastore​

Hive Metastore will manage metadata for Spark.

Hive Configuration​

  1. Download and install Hive Helm chart:

    helm repo add bitnami https://charts.bitnami.com/bitnami
    helm install hive bitnami/hive --set postgresql.enabled=false
  2. Update Hive configuration to use PostgreSQL.

# Values.yaml for Hive
hive:
metastore:
databaseType: postgres
jdbcUrl: jdbc:postgresql://postgresql-service/hive_metastore
user: hiveuser
password: hivepassword

4. Deploying Spark Master and Workers​

Spark Master and Worker StatefulSet YAML​

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: spark-master
spec:
serviceName: spark-master
replicas: 1
selector:
matchLabels:
app: spark
template:
metadata:
labels:
app: spark
role: master
spec:
containers:
- name: spark-master
image: spark:latest
ports:
- containerPort: 7077
- containerPort: 8080
env:
- name: SPARK_MASTER_HOST
value: "spark-master"

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: spark-worker
spec:
replicas: 2
selector:
matchLabels:
app: spark
role: worker
template:
metadata:
labels:
app: spark
role: worker
spec:
containers:
- name: spark-worker
image: spark:latest
ports:
- containerPort: 8081
env:
- name: SPARK_WORKER_CORES
value: "1"
- name: SPARK_WORKER_MEMORY
value: "1G"
- name: SPARK_MASTER
value: "spark://spark-master:7077"

5. Setting Up the Spark History Server​

The History Server will read event logs from the GCS bucket for job history.

Spark History Server YAML​

apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-history-server
spec:
replicas: 1
selector:
matchLabels:
app: spark-history
template:
metadata:
labels:
app: spark-history
spec:
containers:
- name: spark-history-server
image: spark:latest
env:
- name: SPARK_HISTORY_OPTS
value: "-Dspark.history.fs.logDirectory=gs://your-spark-event-logs"
volumeMounts:
- mountPath: /opt/spark/work-dir
name: gcs-key
subPath: key.json
volumes:
- name: gcs-key
secret:
secretName: gcs-key

6. Configuring Ingress​

Ingress will expose the Spark Master UI and History Server UI.

Ingress YAML​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: spark-ingress
spec:
rules:
- host: spark.example.com
http:
paths:
- path: /master
pathType: Prefix
backend:
service:
name: spark-master
port:
number: 8080
- path: /history
pathType: Prefix
backend:
service:
name: spark-history-server
port:
number: 18080

7. Testing the Deployment with Spark Job​

To verify the installation, use spark-submit to run a simple calculation for PI.

  1. Submit a PI calculation job using spark-submit.

    kubectl exec -it spark-master-0 -- spark-submit \
    --class org.apache.spark.examples.SparkPi \
    --master spark://spark-master:7077 \
    /opt/spark/examples/jars/spark-examples_2.12-3.0.1.jar 1000
  2. Check the Spark Master and History Server UIs

to confirm job execution and view logs.

Furthermore​

StatefulSet vs Deployment​

In the official Helm chart for Apache Spark, the choice of StatefulSet for Spark Master and Worker, and Deployment for the Spark History Server, aligns with the specific requirements and behaviors of each component. Here’s a breakdown of the reasons for these design choices:

1. Spark Master and Worker as StatefulSets​

  • Persistence and Stability: The Spark Master and Worker nodes need stable, unique identities to communicate reliably. Using StatefulSet ensures that each replica (instance) has a stable network identity (hostname) and, if configured, persistent storage.
  • Leader Election and Failover: Spark Master in standalone mode has a primary leader and backup node architecture, where the leader is responsible for the cluster state. StatefulSets allow these nodes to maintain consistent, unique identifiers, which is crucial for leader election and failover without disrupting the Spark cluster.
  • Worker Registration: Spark Workers register with the Master, and having a unique, stable network identity ensures workers don’t inadvertently re-register or conflict with each other after restarts or failures. StatefulSets are the preferred way to handle this kind of behavior.

2. Spark History Server as a Deployment​

  • Stateless Operation: The Spark History Server is fundamentally a stateless application. It reads from a pre-configured storage (such as HDFS or S3) where application logs are stored but doesn’t require persistent storage for its own operation. Using a Deployment is appropriate here, as the history server can be easily replicated or scaled without maintaining unique identities.
  • Easier Scaling and Update Management: Deployments provide automatic scaling and rolling updates, which are more suitable for stateless services. If you need to increase the availability of the history server, you can simply increase the replica count in the Deployment, and Kubernetes will handle the scaling automatically.
  • Failure Recovery: Since there’s no dependency on unique network identities or persistent state, if the History Server pod fails, it can be easily recreated without any additional orchestration, which is exactly what Deployments are optimized for.

Summary Table​

ComponentKubernetes KindPurposeReasoning
Spark MasterStatefulSetPrimary and backup master for Spark clusterNeeds stable identity for cluster coordination and leader election
Spark WorkerStatefulSetWorker nodes that register with Spark MasterStable network identity ensures reliable registration and communication with Master
Spark History ServerDeploymentReads logs and provides historical application dataStateless and doesn’t need unique identities; easier to scale and restart as a Deployment

This setup optimizes the deployment structure for each Spark component's unique requirements, leveraging Kubernetes' StatefulSet and Deployment resources for their intended purposes.

Troubleshooting Kubernetes: Pod Stuck in Terminating State After Deletion

· 阅读需 6 分钟

Introduction​

This document provides a detailed troubleshooting guide for a common Kubernetes issue where a Pod remains in the Terminating state after being deleted using the kubectl delete command. The analysis includes examining logs, identifying suspicious points, and understanding the underlying causes.


Problem Background​

After deleting a business Pod using the kubectl delete command, the Pod remained in the Terminating state.


Cause Analysis​

Initial Investigation​

To begin with, the kubectl describe command was used to inspect the Pod:

[root@node1 ~]# kubectl describe pod -n xxx cam1-78b6fc6bc8-cjsw5

No obvious anomalies were found in the output.

Kubelet Logs Analysis​

Next, the kubelet component logs were examined for any anomalies during the Pod deletion process. The following logs were filtered for relevance:

I0728 16:24:57.339295 9744 kubelet.go:1904] SyncLoop (DELETE, "api"): "cam1-78b6fc6bc8-cjsw5_cam(5c948341-c030-4996-b888-f032577d97b0)"
I0728 16:24:57.339720 9744 kuberuntime_container.go:581] Killing container "docker://a73082a4a9a4cec174bb0d1c256cc11d804d93137551b9bfd3e6fa1522e98589" with 60 second grace period
I0728 16:25:18.259418 9744 kubelet.go:1904] SyncLoop (DELETE, "api"): "cam1-78b6fc6bc8-cjsw5_cam(5c948341-c030-4996-b888-f032577d97b0)"
2021-07-28 16:25:19.247 [INFO][394011] ipam.go 1173: Releasing all IPs with handle 'cam.cam1-78b6fc6bc8-cjsw5'
2021-07-28 16:25:19.254 [INFO][393585] k8s.go 498: Teardown processing complete.

Suspicious Point 1: Pod IP Retrieval Failure​

W0728 16:25:19.303513 9744 docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "cam1-78b6fc6bc8-cjsw5_cam": Unexpected command output Device "eth0" does not exist.
with error: exit status 1

Suspicious Point 2: Unmount Failure​

E0728 16:25:20.939400 9744 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g podName:5c948341-c030-4996-b888-f032577d97b0 nodeName:}" failed. No retries permitted until 2021-07-28 16:25:21.439325811 +0800 CST m=+199182.605079651 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"diag-log\" (UniqueName: \"kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g\") pod \"5c948341-c030-4996-b888-f032577d97b0\" (UID: \"5c948341-c030-4996-b888-f032577d97b0\") : Unmount failed: exit status 32\nUnmounting arguments: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g\nOutput: umount: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g:Target is busy.\n (In some case using lsof(8) or fuser(1) can\n find useful information about the process using the device)\n\n"

Suspicious Points​

Suspicious Point 1: Pod IP Retrieval Failure​

The log indicates a failure to retrieve the Pod IP:

W0728 16:25:19.303513 9744 docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "cam1-78b6fc6bc8-cjsw5_cam": Unexpected command output Device "eth0" does not exist.
with error: exit status 1

This error was traced to the getIP method in the docker_sandbox.go file:

func (ds *dockerService) getIP(podSandboxID string, sandbox *dockertypes.ContainerJSON) string {
if sandbox.NetworkSettings == nil {
return ""
}
if networkNamespaceMode(sandbox) == runtimeapi.NamespaceMode_NODE {
// For sandboxes using host network, the shim is not responsible for
// reporting the IP.
return ""
}

// Don't bother getting IP if the pod is known and networking isn't ready
ready, ok := ds.getNetworkReady(podSandboxID)
if ok && !ready {
return ""
}

ip, err := ds.getIPFromPlugin(sandbox)
if err == nil {
return ip
}

if sandbox.NetworkSettings.IPAddress != "" {
return sandbox.NetworkSettings.IPAddress
}
if sandbox.NetworkSettings.GlobalIPv6Address != "" {
return sandbox.NetworkSettings.GlobalIPv6Address
}

// Error log here
klog.Warningf("failed to read pod IP from plugin/docker: %v", err)
return ""
}

Suspicious Point 2: Unmount Failure​

The unmount failure was logged as an ERROR:

E0728 16:25:20.939400 9744 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g podName:5c948341-c030-4996-b888-f032577d97b0 nodeName:}" failed. No retries permitted until 2021-07-28 16:25:21.439325811 +0800 CST m=+199182.605079651 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"diag-log\" (UniqueName: \"kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g\") pod \"5c948341-c030-4996-b888-f032577d97b0\" (UID: \"5c948341-c030-4996-b888-f032577d97b0\") : Unmount failed: exit status 32\nUnmounting arguments: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g\nOutput: umount: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g:Target is busy.\n (In some case using lsof(8) or fuser(1) can\n find useful information about the process using the device)\n\n"

Pod Deletion Process​

The deletion of a Pod in Kubernetes follows these steps:

  1. API Server DELETE Call: The kubectl delete command triggers a DELETE request to the Kubernetes API server with a default grace period of 30 seconds.
  2. Update Pod Metadata: The Pod's metadata is updated with DeletionTimestamp and DeletionGracePeriodSeconds fields, but the Pod is not yet deleted from etcd.
  3. Kubelet Event Handling: The kubelet component listens for Pod updates and initiates the killPod() method.
  4. Second DELETE Call: The kubelet makes a second DELETE request to the API server with a grace period of 0 seconds.
  5. Pod Deletion from etcd: The API server deletes the Pod from etcd.

Verification and Testing​

To verify the deletion process, a test was conducted in a test environment:

[root@node2 ~]# kubectl delete pod -n xxx testpodrc2-7b749f6c9c-qh68l
pod "testpodrc2-7b749f6c9c-qh68l" deleted

Filtered kubelet logs:

I0730 13:27:31.854178 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:31.854511 24588 kuberuntime_container.go:581] Killing container "docker://e2a1cd5f2165e12cf0b46e12f9cd4d656d593f75e85c0de058e0a2f376a5557e" with 30 second grace period
I0730 13:27:32.203167 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:32.993294 24588 kubelet.go:1933] SyncLoop (PLEG): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)", event: &pleg.PodLifecycleEvent{ID:"85ee282f-a843-4f10-a99c-79d447f83f2a", Type:"ContainerDied", Data:"e2a1cd5f2165e12cf0b46e12f9cd4d656d593f75e85c0de058e0a2f376a5557e"}
I0730 13:27:32.993428 24588 kubelet.go:1933] SyncLoop (PLEG): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)", event: &pleg.PodLifecycleEvent{ID:"85ee282f-a843-4f10-a99c-79d447f83f2a", Type:"ContainerDied", Data:"c6a587614976beed0cbb6e5fabf70a2d039eec6c160154fce007fe2bb1ba3b4f"}
I0730 13:27:34.072494 24588 kubelet_pods.go:1090] Killing unwanted pod "testpodrc2-7b749f6c9c-qh68l"
I0730 13:27:40.084182 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:40.085735 24588 kubelet.go:1898] SyncLoop (REMOVE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"

Conclusion​

The analysis revealed that the Pod remained in the Terminating state due to a failure in unmounting the GlusterFS volume. The unmount failure was caused by the volume being busy, which prevented the second DELETE

Using GlusterFS with Kubernetes

· 阅读需 6 分钟

Introduction​

This article provides a comprehensive guide on using GlusterFS with Kubernetes. It covers the basics of GlusterFS, its architecture, and how to mount GlusterFS as the file system for a Pod in Kubernetes. Additionally, it compares GlusterFS with other main alternatives in the storage landscape.


GlusterFS Overview​

GlusterFS (Gluster File System) is an open-source, distributed file system designed to scale out in building-block fashion to store petabytes of data. It aggregates various storage servers over Ethernet or InfiniBand RDMA interconnects into one large parallel network file system. GlusterFS is part of the Gluster project, an effort by Red Hat to provide a unified distributed storage solution.

Key Features​

  1. Scalability: GlusterFS can scale to several petabytes and handle thousands of clients.
  2. Flexibility: It uses a stackable user space design, allowing for flexible topologies for data storage and retrieval.
  3. High Availability: GlusterFS provides features like replication and failover to ensure data availability and reliability.
  4. Performance: It uses various data access protocols like NFS, SMB, and native GlusterFS protocol to optimize performance.
  5. Elasticity: GlusterFS can dynamically add or remove storage resources without disrupting services.

Architecture​

GlusterFS architecture is based on a client-server model, where the storage servers (bricks) are the servers, and the clients access the data through the GlusterFS client. The architecture is modular, allowing for various configurations and optimizations.

  1. Bricks: Basic units of storage, represented as directories on servers.
  2. Volumes: Logical collections of bricks. Data is striped, replicated, or distributed across these bricks.
  3. Translators: Modules that perform various functions like I/O, network, and protocol handling.

Use Cases​

  • Big Data and Analytics: Suitable for storing and processing large datasets.
  • Media Streaming: Efficiently handles large media files.
  • Backup and Archiving: Provides reliable storage for backups and archives.
  • Cloud Storage: Acts as a scalable storage backend for cloud services.

Mounting GlusterFS with Kubernetes​

To use GlusterFS as the file system for a Pod in Kubernetes, you need to create a Persistent Volume (PV) and a Persistent Volume Claim (PVC). Below are the steps to achieve this:

Step 1: Install GlusterFS Client​

Ensure that the GlusterFS client (glusterfs-client) is installed on all Kubernetes nodes. This client is necessary for mounting GlusterFS volumes.

sudo apt-get install glusterfs-client

Step 2: Create a GlusterFS Volume​

Create a GlusterFS volume on your GlusterFS cluster. For example:

gluster volume create k8s-volume replica 3 server1:/brick1 server2:/brick2 server3:/brick3 force
gluster volume start k8s-volume

Step 3: Create a Persistent Volume (PV)​

Create a PV that references the GlusterFS volume.

apiVersion: v1
kind: PersistentVolume
metadata:
name: gluster-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
glusterfs:
endpoints: "glusterfs-cluster"
path: "k8s-volume"
readOnly: false

Step 4: Create a Persistent Volume Claim (PVC)​

Create a PVC that binds to the PV.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gluster-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi

Step 5: Use the PVC in a Pod​

Mount the PVC in a Pod.

apiVersion: v1
kind: Pod
metadata:
name: gluster-pod
spec:
containers:
- name: app
image: nginx
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: gluster-volume
volumes:
- name: gluster-volume
persistentVolumeClaim:
claimName: gluster-pvc

Textual Diagram of GlusterFS Architecture​

+-------------------+ +-------------------+ +-------------------+
| | | | | |
| GlusterFS Client |<----->| GlusterFS Server |<----->| GlusterFS Server |
| | | | | |
+-------------------+ +-------------------+ +-------------------+
| | |
| | |
v v v
+-------------------+ +-------------------+ +-------------------+
| | | | | |
| Brick 1 | | Brick 2 | | Brick 3 |
| | | | | |
+-------------------+ +-------------------+ +-------------------+

Comparison with Other Main Alternatives​

NFS (Network File System)​

  • Pros:
    • Simple to set up and manage.
    • Widely supported and well-documented.
  • Cons:
    • Single point of failure.
    • Limited scalability compared to GlusterFS.
    • Performance can degrade with high latency.

Ceph​

  • Pros:
    • Highly scalable and fault-tolerant.
    • Supports object, block, and file storage.
    • Active community and strong support.
  • Cons:
    • Complex to set up and manage.
    • Requires significant resources for optimal performance.

Amazon EFS (Elastic File System)​

  • Pros:
    • Fully managed service.
    • Highly available and durable.
    • Scales automatically with no need to provision storage.
  • Cons:
    • Cloud-specific, not suitable for on-premises deployments.
    • Higher cost compared to self-managed solutions.

OpenEBS​

  • Pros:
    • Kubernetes-native storage solution.
    • Easy to deploy and manage.
    • Supports various storage engines.
  • Cons:
    • Younger project with a smaller community.
    • May require more tuning for performance.

Conclusion​

GlusterFS is a powerful, scalable, and flexible distributed file system that can be effectively integrated with Kubernetes for persistent storage solutions. By following the steps outlined in this guide, you can mount GlusterFS volumes in your Kubernetes Pods. Additionally, understanding the architecture and comparing GlusterFS with other storage alternatives helps in making informed decisions for your storage needs.


References​

  1. GlusterFS Official Documentation
  2. Red Hat Gluster Storage
  3. GlusterFS GitHub Repository
  4. GlusterFS Wiki
  5. Kubernetes Documentation - Persistent Volumes
  6. NFS Documentation
  7. Ceph Documentation
  8. Amazon EFS Documentation
  9. OpenEBS Documentation

By following these steps and understanding the architecture and comparisons, you can effectively use GlusterFS with Kubernetes and make informed decisions about storage solutions for your Kubernetes clusters.

Analyzing High Memory Usage in Kubernetes

· 阅读需 4 分钟

Introduction​

In Kubernetes environments, monitoring resource usage is crucial for maintaining optimal performance and resource allocation. This article delves into a specific case where a user observed high memory usage by a Harbor pod on one of the nodes, leading to a discrepancy between kubectl top and docker stats results. We will analyze the cause, verify the memory calculation methods, and provide a solution to accurately monitor memory usage.


Problem Background​

A user observed that one of the nodes in their Kubernetes cluster showed high memory usage by a Harbor pod, approximately 3.7G, which seemed unusually high compared to other pods.

[root@node02 ~]# kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP
node01 Ready master 10d v1.15.12 100.1.0.10 <none>
node02 Ready master 12d v1.15.12 100.1.0.11 <none>
node03 Ready master 10d v1.15.12 100.1.0.12 <none>

[root@node02 ~]# kubectl top pod -A | grep harbor
kube-system harbor-master1-sxg2l 15m 150Mi
kube-system harbor-master2-ncvb8 8m 3781Mi
kube-system harbor-master3-2gdsn 14m 227Mi

Cause Analysis​

To understand the memory usage, the user compared the results from kubectl top and docker stats. Theoretically, docker stats should provide a more accurate result.

Using docker stats​

[root@node02 ~]# docker stats | grep harbor
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM %
10a230bee3c7 k8s_nginx_harbor-master2-xxx 0.02% 14.15MiB / 94.26GiB 0.01%
6ba14a04fd77 k8s_harbor-portal_harbor-master2-xxx 0.01% 13.73MiB / 94.26GiB 0.01%
324413da20a9 k8s_harbor-jobservice_harbor-master2-xxx 0.11% 21.54MiB / 94.26GiB 0.02%
d880b61cf4cb k8s_harbor-core_harbor-master2-xxx 0.12% 33.2MiB / 94.26GiB 0.03%
186c064d0930 k8s_harbor-registryctl_harbor-master2-xxx 0.01% 8.34MiB / 94.26GiB 0.01%
52a50204a962 k8s_harbor-registry_harbor-master2-xxx 0.06% 29.99MiB / 94.26GiB 0.03%
86031ddd0314 k8s_harbor-redis_harbor-master2-xxx 0.14% 11.51MiB / 94.26GiB 0.01%
6366207680f2 k8s_harbor-database_harbor-master2-xxx 0.45% 8.859MiB / 94.26GiB 0.01%

The total memory usage from docker stats was around 140M, significantly lower than the 3.7G reported by kubectl top.


Verification of Memory Calculation​

The discrepancy arises from the different methods used by kubectl top and docker stats to calculate memory usage.

  • kubectl top Calculation: memory.usage_in_bytes - inactive_file
  • docker stats Calculation: memory.usage_in_bytes - cache

If the cache is large, kubectl top will report a higher memory usage.

Verifying with memory_stats​

To verify, the user fetched the memory_stats data using Docker's API:

curl -s --unix-socket /var/run/docker.sock http:/v1.24/containers/xxx/stats | jq ."memory_stats"

Sample memory_stats data:

"memory_stats": {
"usage": 14913536,
"max_usage": 15183872,
"stats": {
"active_anon": 14835712,
"active_file": 0,
"cache": 77824,
"dirty": 0,
"hierarchical_memory_limit": 101205622784,
"hierarchical_memsw_limit": 9223372036854772000,
"inactive_anon": 4096,
"inactive_file": 73728,
...
}

By applying the formulas, the user confirmed that the results from kubectl top and docker stats were consistent with the expected calculations.


Understanding Cache Usage​

The high cache usage in Harbor was primarily due to the registry component, which handles Docker image storage and involves extensive read/write operations on image layers.

"memory_stats": {
"usage": 5845127168,
"max_usage": 22050988032,
"stats": {
"active_anon": 31576064,
"active_file": 3778052096,
"cache": 5813551104,
"dirty": 0,
"hierarchical_memory_limit": 101205622784,
"hierarchical_memsw_limit": 9223372036854772000,
"inactive_anon": 0,
"inactive_file": 2035499008,
...
}

Solution​

To address the discrepancy, the user was informed that kubectl top includes cache usage, which can be misleading. The cache, being non-critical, can be reclaimed by the system during memory pressure or manually released. The user was advised to use docker stats for a more accurate representation of actual memory usage.


Conclusion​

This case study highlights the importance of understanding the different methods used by Kubernetes and Docker to calculate memory usage. By verifying the calculations and understanding the role of cache, users can make more informed decisions about resource management in their Kubernetes clusters. Using docker stats provides a more accurate view of memory usage, especially in scenarios where cache usage is significant.

Deep Dive Redis Integration With Spring Boot Application

· 阅读需 12 分钟

Table of Contents​


Introduction​

In distributed systems, caching is a critical mechanism to improve performance by reducing the load on backend services and databases. This article delves into the design and implementation of cache mechanisms for distributed systems, covering high-level architecture, detailed designs, and sample code for various scenarios.


High-Level Architecture Design​

Diagram​

graph TD
A[Client] --> B[Load Balancer]
B --> C[Service Layer]
C --> D[Cache Layer]
C --> E[Data Storage Layer]
D --> E
F[Monitoring and Metrics] --> C
F --> D
F --> E

Explanation​

  1. Client: The client sends requests to the distributed system.
  2. Load Balancer: Distributes incoming requests across multiple instances of the service layer.
  3. Service Layer: Handles business logic and interacts with the cache and data storage layers.
  4. Cache Layer: Stores frequently accessed data to reduce the load on the data storage layer.
  5. Data Storage Layer: The primary data store (e.g., database) where data is persisted.
  6. Monitoring and Metrics: Collects and monitors performance metrics for the entire system.

Detail Designs​

Cache Layer​

The cache layer is responsible for storing frequently accessed data to reduce the load on the data storage layer. It can be implemented using distributed caching solutions like Redis or Hazelcast.

Redis Setup and Integration​

Redis is an in-memory data structure store that can be used as a distributed cache. It supports various data structures like strings, hashes, lists, sets, and sorted sets.

Step 1: Install Redis​

You can install Redis on your local machine or use a cloud-based Redis service like AWS ElastiCache or Redis Labs.

For local installation on Linux:

sudo apt-get update
sudo apt-get install redis-server

For local installation on macOS:

brew install redis
Step 2: Start Redis Server​

Start the Redis server:

redis-server
Step 3: Verify Redis Installation​

Verify that Redis is running:

redis-cli ping

If Redis is running correctly, you should see PONG as the response.

Redis Configuration​

Redis configuration can be managed through the redis.conf file. Common configurations include setting the maximum memory limit, configuring persistence, and setting up replication.

Example redis.conf snippet:

maxmemory 256mb
maxmemory-policy allkeys-lru

Redis Integration with Spring Boot​

To integrate Redis with a Spring Boot application, you need to add the necessary dependencies and configure the Redis connection.

Step 1: Add Dependencies​

Add the following dependencies to your pom.xml:

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-cache</artifactId>
</dependency>
Step 2: Configure Redis Connection​

Configure the Redis connection in your application.yml:

spring:
redis:
host: localhost
port: 6379
cache:
type: redis
Step 3: Enable Caching​

Enable caching in your Spring Boot application by adding the @EnableCaching annotation to your main application class:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cache.annotation.EnableCaching;

@SpringBootApplication
@EnableCaching
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
Step 4: Use Redis Cache​

Use the @Cacheable annotation to cache method results in Redis:

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;

@Service
public class CacheService {

@Cacheable(value = "userCache", key = "#userId")
public User getUserById(Long userId) {
// Fetch user from data storage layer
return fetchUserFromDatabase(userId);
}

private User fetchUserFromDatabase(Long userId) {
// Database query logic
return new User();
}
}

Best Practices for Redis in Kubernetes​

When deploying Redis in Kubernetes, it's important to follow best practices to ensure high availability, scalability, and security.

Step 1: Deploy Redis in Kubernetes​

You can deploy Redis in Kubernetes using a StatefulSet for stateful applications or a Deployment for stateless applications. Here's an example of deploying Redis using a StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
serviceName: redis
replicas: 3
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:latest
ports:
- containerPort: 6379
volumeMounts:
- name: redis-data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
Step 2: Create a Headless Service​

Create a headless service to manage the Redis StatefulSet:

apiVersion: v1
kind: Service
metadata:
name: redis
spec:
clusterIP: None
selector:
app: redis
ports:
- port: 6379
targetPort: 6379
Step 3: Configure Redis Access in Spring Boot​

When deploying your Spring Boot application in Kubernetes, you should use the service name to access Redis. Update your application.yml to use the service name:

spring:
redis:
host: redis
port: 6379
cache:
type: redis
Step 4: Enable Redis Persistence​

To enable Redis persistence, configure the redis.conf file to use either RDB (Redis Database Backup) or AOF (Append-Only File) persistence:

save 900 1
save 300 10
save 60 10000
appendonly yes
Step 5: Secure Redis​

Secure Redis by setting a password and enabling TLS:

requirepass yourpassword
tls-port 6379
tls-cert-file /path/to/redis.crt
tls-key-file /path/to/redis.key
tls-ca-cert-file /path/to/ca.crt
Step 6: Monitor Redis​

Use Prometheus and Grafana to monitor Redis performance and health. Deploy a Redis exporter to expose Redis metrics:

apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-exporter
spec:
replicas: 1
selector:
matchLabels:
app: redis-exporter
template:
metadata:
labels:
app: redis-exporter
spec:
containers:
- name: redis-exporter
image: oliver006/redis_exporter:latest
args:
- --redis.addr=redis:6379
- --redis.password=yourpassword
ports:
- containerPort: 9121

Create a service to expose the Redis exporter:

apiVersion: v1
kind: Service
metadata:
name: redis-exporter
spec:
selector:
app: redis-exporter
ports:
- port: 9121
targetPort: 9121

Data Storage Layer​

The data storage layer is the primary data store where data is persisted. It can be a relational database, NoSQL database, or any other storage solution.

Diagram​

graph TD
A[Service Layer] --> B[Data Storage Layer]
B --> C[Relational Database]
B --> D[NoSQL Database]

Sample Code​

import org.springframework.stereotype.Repository;

@Repository
public class UserRepository {

public User findById(Long userId) {
// Database query logic
return new User();
}

public void save(User user) {
// Database save logic
}
}

Service Layer​

The service layer handles business logic and interacts with the cache and data storage layers. It is responsible for managing the flow of data between the client, cache, and data storage.

Diagram​

graph TD
A[Client] --> B[Service Layer]
B --> C[Cache Layer]
B --> D[Data Storage Layer]

Sample Code​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class UserService {

@Autowired
private CacheService cacheService;

@Autowired
private UserRepository userRepository;

public User getUserById(Long userId) {
User user = cacheService.getUserById(userId);
if (user == null) {
user = userRepository.findById(userId);
cacheService.saveUser(user);
}
return user;
}

public void saveUser(User user) {
userRepository.save(user);
cacheService.saveUser(user);
}
}

Load Balancer​

The load balancer distributes incoming requests across multiple instances of the service layer to ensure high availability and scalability.

Diagram​

graph TD
A[Client] --> B[Load Balancer]
B --> C[Service Instance 1]
B --> D[Service Instance 2]
B --> E[Service Instance 3]

Sample Code​

Load balancers are typically implemented using infrastructure components like NGINX, HAProxy, or cloud-provided load balancers (e.g., AWS ELB, Google Cloud Load Balancer).

Monitoring and Metrics​

Monitoring and metrics collection are essential for tracking the performance and health of the distributed system. Tools like Prometheus, Grafana, and ELK stack can be used for this purpose.

Diagram​

graph TD
A[Service Layer] --> B[Monitoring and Metrics]
B --> C[Prometheus]
B --> D[Grafana]
B --> E[ELK Stack]

Sample Code​

import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;

public class MetricsCollector {

private static final Counter cacheHits = Counter.build()
.name("cache_hits_total")
.help("Total number of cache hits.")
.register();

private static final Counter cacheMisses = Counter.build()
.name("cache_misses_total")
.help("Total number of cache misses.")
.register();

private static final Gauge cacheSize = Gauge.build()
.name("cache_size")
.help("Current size of the cache.")
.register();

public static void recordCacheHit() {
cacheHits.inc();
}

public static void recordCacheMiss() {
cacheMisses.inc();
}

public static void setCacheSize(long size) {
cacheSize.set(size);
}
}

Complex Scenario 1: Flow Control and Backpressure Handling​

Problem Statement​

In high-traffic scenarios, the cache might become a bottleneck if not properly managed. Flow control and backpressure handling are essential to prevent the cache from being overwhelmed.

Solution Architecture​

  1. Rate Limiting: Implement rate limiting to control the number of requests hitting the cache.
  2. Circuit Breaker: Use a circuit breaker to handle failures gracefully and prevent cascading failures.
  3. Backpressure Handling: Implement backpressure mechanisms to manage the flow of data between services.

Flow Chart​

graph TD
A[Client Request] --> B{Rate Limiter}
B -->|Allowed| C[Cache Lookup]
B -->|Denied| D[Fallback to Database]
C --> E{Cache Hit?}
E -->|Yes| F[Return Cached Data]
E -->|No| G[Fetch from Database]
G --> H[Update Cache]
H --> F
D --> G

Sample Code​

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;

@Service
public class UserService {

@Cacheable(value = "userCache", key = "#userId")
public User getUserById(Long userId) {
// Fetch user from database
return fetchUserFromDatabase(userId);
}

private User fetchUserFromDatabase(Long userId) {
// Database query logic
return new User();
}
}

Complex Scenario 2: Cache Invalidation and Data Synchronization​

Problem Statement​

Ensuring that the cache remains consistent with the underlying data store is challenging, especially in distributed systems.

Solution Architecture​

  1. Event-Driven Invalidation: Use event-driven mechanisms to invalidate the cache when data changes.
  2. Distributed Locking: Use distributed locking to ensure that only one service updates the cache at a time.
  3. TTL-Based Invalidation: Use Time-to-Live (TTL) to automatically invalidate cache entries after a certain period.

Sequence Diagram​

sequenceDiagram
participant Client
participant Service
participant Cache
participant Database

Client->>Service: Update User
Service->>Database: Update User in Database
Database-->>Service: User Updated
Service->>Cache: Invalidate User Cache
Cache-->>Service: Cache Invalidated
Service-->>Client: User Updated

Sample Code​

import org.springframework.cache.annotation.CacheEvict;
import org.springframework.stereotype.Service;

@Service
public class UserService {

@CacheEvict(value = "userCache", key = "#userId")
public void updateUser(Long userId, User user) {
// Update user in database
updateUserInDatabase(userId, user);
}

private void updateUserInDatabase(Long userId, User user) {
// Database update logic
}
}

Complex Scenario 3: Cache Stampede Prevention​

Problem Statement​

A cache stampede occurs when multiple requests for the same data hit the cache simultaneously, leading to a spike in database load.

Solution Architecture​

  1. Probabilistic Early Expiration: Use probabilistic early expiration to spread out cache expiration times.
  2. Locking Mechanism: Implement a locking mechanism to ensure that only one request populates the cache at a time.
  3. Background Refresh: Use background refresh to keep the cache up-to-date without impacting the user experience.

Flow Chart​

graph TD
A[Client Request] --> B{Cache Hit?}
B -->|Yes| C[Return Cached Data]
B -->|No| D{Lock Acquired?}
D -->|Yes| E[Fetch from Database]
D -->|No| F[Wait for Lock]
E --> G[Update Cache]
G --> C
F --> B

Sample Code​

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;

import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

@Service
public class UserService {

private final Lock lock = new ReentrantLock();

@Cacheable(value = "userCache", key = "#userId")
public User getUserById(Long userId) {
try {
lock.lock();
// Double-check to ensure the cache is still empty
User cachedUser = fetchUserFromCache(userId);
if (cachedUser != null) {
return cachedUser;
}
// Fetch user from database
User user = fetchUserFromDatabase(userId);
// Update cache
updateCache(userId, user);
return user;
} finally {
lock.unlock();
}
}

private User fetchUserFromCache(Long userId) {
// Cache lookup logic
return null;
}

private User fetchUserFromDatabase(Long userId) {
// Database query logic
return new User();
}

private void updateCache(Long userId, User user) {
// Cache update logic
}
}

Conclusion​

Caching is a powerful mechanism for improving the performance and scalability of distributed systems. By understanding the high-level architecture and detailed designs, you can effectively implement cache mechanisms that handle various complex scenarios like flow control, cache invalidation, and cache stampede prevention. Use the provided sample code and diagrams to implement efficient caching in your distributed system.