Skip to main content

3 posts tagged with "kafka"

View All Tags

Spark in Spring Boot Application

· 11 min read

Introduction​

This article provides a comprehensive guide to developing a Spark application with Spring Boot 3.3.5 to communicate with Kafka and Elasticsearch. The application will handle Avro records in Kafka, JSON data in Elasticsearch, and Parquet files in MinIO using Spark DataFrame and Spark SQL. The article covers the setup, integration, and advanced scenarios for handling large-scale data processing, including the use of Avro Schema Registry when communicating with Kafka.


Prerequisites​

  • Java 17 or later
  • Apache Spark 3.5.4
  • Spring Boot 3.3.5
  • Kafka 3.x
  • Elasticsearch 8.x
  • MinIO
  • Maven or Gradle
  • Confluent Schema Registry

Project Folder Structure​

spark-spring-boot-app/
├── src/
│ ├── main/
│ │ ├── java/
│ │ │ ├── com/
│ │ │ │ ├── example/
│ │ │ │ │ ├── config/
│ │ │ │ │ │ ├── SparkConfig.java
│ │ │ │ │ ├── service/
│ │ │ │ │ │ ├── KafkaService.java
│ │ │ │ │ │ ├── ElasticsearchService.java
│ │ │ │ │ │ ├── MinIOService.java
│ │ │ │ │ │ ├── DataFrameService.java
│ │ │ │ │ ├── SparkSpringBootConsoleApplication.java
│ │ ├── resources/
│ │ │ ├── application.properties
│ ├── test/
│ │ ├── java/
│ │ │ ├── com/
│ │ │ │ ├── example/
│ │ │ │ │ ├── service/
│ │ │ │ │ │ ├── KafkaServiceTest.java
│ │ │ │ │ │ ├── ElasticsearchServiceTest.java
│ │ │ │ │ │ ├── MinIOServiceTest.java
│ │ │ │ │ │ ├── DataFrameServiceTest.java
├── pom.xml (or build.gradle)

Setting Up the Spring Boot Application​

Step 1: Create a Spring Boot Project​

Create a new Spring Boot project using Spring Initializr or your preferred IDE. Include the following dependencies:

  • Spring for Apache Kafka
  • Spring Data Elasticsearch
  • Spring for Apache Spark

Step 2: Configure Application Properties​

Configure the application properties for Kafka, Elasticsearch, and MinIO:

# Kafka Configuration
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.ByteArraySerializer

# Schema Registry Configuration
schema.registry.url=http://localhost:8081

# Elasticsearch Configuration
spring.elasticsearch.rest.uris=http://localhost:9200

# MinIO Configuration
minio.endpoint=http://localhost:9000
minio.accessKey=your-access-key
minio.secretKey=your-secret-key
minio.bucket=your-bucket

Integrating Spark with Spring Boot​

Step 1: Add Spark Dependencies​

Add the following dependencies to your pom.xml or build.gradle file:

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>7.0.1</version>
</dependency>

Step 2: Configure Spark​

Configure Spark in your Spring Boot application using annotation-style configuration:

import org.apache.spark.sql.SparkSession;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class SparkConfig {

@Bean
public SparkSession sparkSession() {
return SparkSession.builder()
.appName("SparkSpringBootApp")
.master("local[*]")
.config("spark.sql.shuffle.partitions", "10")
.getOrCreate();
}
}

Communicating with Kafka​

Reading Avro Records from Kafka​

To read Avro records from Kafka, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readAvroFromKafka(String topic) {
return sparkSession.read()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.load()
.selectExpr("CAST(value AS BINARY)")
.select(org.apache.spark.sql.avro.functions.from_avro(col("value"), "your_avro_schema"));
}
}

Reading Avro Records from Kafka with Schema Registry​

To read Avro records from Kafka using the Schema Registry, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

@Value("${schema.registry.url}")
private String schemaRegistryUrl;

public Dataset<Row> readAvroFromKafka(String topic) {
return sparkSession.read()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.load()
.selectExpr("CAST(value AS BINARY)")
.select(org.apache.spark.sql.avro.functions.from_avro(col("value"), schemaRegistryUrl + "/subjects/" + topic + "-value/versions/latest"));
}
}

Writing Data to Kafka​

To write data to Kafka, use the spark-sql-kafka-0-10 library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

public void writeToKafka(Dataset<Row> dataset, String topic) {
dataset.selectExpr("to_json(struct(*)) AS value")
.write()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", topic)
.save();
}
}

Writing Avro Records to Kafka with Schema Registry​

To write Avro records to Kafka using the Schema Registry, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

@Value("${schema.registry.url}")
private String schemaRegistryUrl;

public void writeAvroToKafka(Dataset<Row> dataset, String topic) {
dataset.select(org.apache.spark.sql.avro.functions.to_avro(struct("*")).as("value"))
.write()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", topic)
.option("schema.registry.url", schemaRegistryUrl)
.save();
}
}

Communicating with Elasticsearch​

Reading JSON Data from Elasticsearch​

To read JSON data from Elasticsearch, use the elasticsearch-hadoop library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ElasticsearchService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readFromElasticsearch(String index) {
return sparkSession.read()
.format("org.elasticsearch.spark.sql")
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.resource", index)
.load();
}
}

Writing Data to Elasticsearch​

To write data to Elasticsearch, use the elasticsearch-hadoop library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ElasticsearchService {

@Autowired
private SparkSession sparkSession;

public void writeToElasticsearch(Dataset<Row> dataset, String index) {
dataset.write()
.format("org.elasticsearch.spark.sql")
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.resource", index)
.save();
}
}

Handling Parquet Files in MinIO​

Reading Parquet Files from MinIO​

To read Parquet files from MinIO, use the spark-hadoop-cloud library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class MinIOService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readParquetFromMinIO(String bucket, String path) {
return sparkSession.read()
.format("parquet")
.option("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000")
.option("spark.hadoop.fs.s3a.access.key", "your-access-key")
.option("spark.hadoop.fs.s3a.secret.key", "your-secret-key")
.option("spark.hadoop.fs.s3a.path.style.access", "true")
.load("s3a://" + bucket + "/" + path);
}
}

Writing Parquet Files to MinIO​

To write Parquet files to MinIO, use the spark-hadoop-cloud library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class MinIOService {

@Autowired
private SparkSession sparkSession;

public void writeParquetToMinIO(Dataset<Row> dataset, String bucket, String path) {
dataset.write()
.format("parquet")
.option("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000")
.option("spark.hadoop.fs.s3a.access.key", "your-access-key")
.option("spark.hadoop.fs.s3a.secret.key", "your-secret-key")
.option("spark.hadoop.fs.s3a.path.style.access", "true")
.save("s3a://" + bucket + "/" + path);
}
}

Using Spark DataFrame and Spark SQL​

Creating DataFrames​

To create DataFrames, use the SparkSession object:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class DataFrameService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> createDataFrame() {
return sparkSession.read()
.format("csv")
.option("header", "true")
.load("path/to/csv/file");
}
}

Querying DataFrames with Spark SQL​

To query DataFrames using Spark SQL, register the DataFrame as a temporary view:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class DataFrameService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> queryDataFrame(Dataset<Row> df) {
df.createOrReplaceTempView("my_table");
return sparkSession.sql("SELECT * FROM my_table WHERE column = 'value'");
}
}

Unit Test Cases​

KafkaServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class KafkaServiceTest {

@Autowired
private KafkaService kafkaService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadAvroFromKafka() {
Dataset<Row> dataset = kafkaService.readAvroFromKafka("test-topic");
assertNotNull(dataset);
}

@Test
public void testWriteAvroToKafka() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
kafkaService.writeAvroToKafka(dataset, "test-topic");
}
}

ElasticsearchServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class ElasticsearchServiceTest {

@Autowired
private ElasticsearchService elasticsearchService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadFromElasticsearch() {
Dataset<Row> dataset = elasticsearchService.readFromElasticsearch("test-index");
assertNotNull(dataset);
}

@Test
public void testWriteToElasticsearch() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
elasticsearchService.writeToElasticsearch(dataset, "test-index");
}
}

MinIOServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class MinIOServiceTest {

@Autowired
private MinIOService minIOService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadParquetFromMinIO() {
Dataset<Row> dataset = minIOService.readParquetFromMinIO("test-bucket", "test-path");
assertNotNull(dataset);
}

@Test
public void testWriteParquetToMinIO() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
minIOService.writeParquetToMinIO(dataset, "test-bucket", "test-path");
}
}

DataFrameServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class DataFrameServiceTest {

@Autowired
private DataFrameService dataFrameService;

@Autowired
private SparkSession sparkSession;

@Test
public void testCreateDataFrame() {
Dataset<Row> dataset = dataFrameService.createDataFrame();
assertNotNull(dataset);
}

@Test
public void testQueryDataFrame() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
Dataset<Row> result = dataFrameService.queryDataFrame(dataset);
assertNotNull(result);
}
}

Design Diagrams​

Component Diagram​

+-------------------+ +-------------------+ +-------------------+
| | | | | |
| Spring Boot | | Apache Spark | | Kafka |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| | |
v v v
+-------------------+ +-------------------+ +-------------------+
| | | | | |
| KafkaService | | Elasticsearch | | MinIOService |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| | |
v v v
+--------------------+ +-------------------+ +---------------------------------------+
| | | | | |
| DataFrameService | | SparkConfig | | SparkSpringBootConsoleApplication |
| | | | | |
+--------------------+ +-------------------+ +---------------------------------------+

Sequence Diagram​

+---------+ +---------+ +---------+ +---------+
| | | | | | | |
| Client | | Spring | | Spark | | Kafka |
| | | Boot | | | | |
+---------+ +---------+ +---------+ +---------+
| | | |
| Request | | |
|----------------->| | |
| | | |
| | Read Avro | |
| | from Kafka | |
| |-------------->| |
| | | |
| | | Send Avro |
| | |<------------->|
| | | |
| | Process | |
| | Data | |
| |-------------->| |
| | | |
| | Write to | |
| | Elasticsearch| |
| |-------------->| |
| | | |
| | Write to | |
| | MinIO | |
| |-------------->| |
| | | |
| Response | | |
|<-----------------| | |
| | | |

Advanced Scenarios​

Complex Scenario 1: Flow Control and Backpressure Handling​

In scenarios where your application needs to handle flow control and backpressure, you can use Spark's built-in mechanisms for handling large-scale data processing.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class FlowControlService {

@Autowired
private SparkSession sparkSession;

public void handleFlowControl(Dataset<Row> dataset) {
dataset.writeStream()
.format("console")
.option("checkpointLocation", "path/to/checkpoint")
.start()
.awaitTermination();
}
}

Complex Scenario 2: Handling Large-Scale Data Processing​

For large-scale data processing, use Spark's distributed processing capabilities and optimize resource usage.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class LargeScaleProcessingService {

@Autowired
private SparkSession sparkSession;

public void processLargeScaleData(Dataset<Row> dataset) {
dataset.repartition(100)
.write()
.format("parquet")
.save("path/to/output");
}
}

Conclusion​

Developing a Spark application with Spring Boot 3.3.5 to communicate with Kafka and Elasticsearch involves setting up the Spring Boot application, integrating Spark, and handling data using Spark DataFrame and Spark SQL. By following the best practices outlined in this article, you can create efficient and scalable data processing applications.


References​

  1. Spring Boot Documentation
  2. Apache Spark Documentation
  3. Apache Kafka Documentation
  4. Elasticsearch Documentation
  5. MinIO Documentation
  6. Confluent Schema Registry Documentation

By following these best practices, you can leverage Spark and Spring Boot to build powerful and scalable data processing applications.

Kafka Producer Tuning Guide

· 9 min read

Kafka Producer Tuning Guide


1. Introduction​

Apache Kafka is a distributed streaming platform often used for building real-time data pipelines and streaming applications. Kafka producers are responsible for sending records to Kafka brokers, and tuning their performance is essential to achieving high throughput, reliability, and proper data ordering. In this article, we will cover the various aspects of Kafka producer tuning, including optimizing throughput, ensuring reliability, avoiding data duplication, and maintaining data order.


2. Optimizing Producer Throughput​

Throughput optimization focuses on increasing the number of messages the producer can send per second. Kafka producers offer multiple configuration options to improve throughput:

2.1 Batching​

Kafka producers can batch multiple records before sending them to the broker, reducing the number of network requests.

  • Batch size (batch.size): Determines the maximum number of bytes in a batch. Increasing this value allows the producer to group more records in one request, increasing throughput.
props.put("batch.size", 16384); // 16KB per batch

2.2 Compression​

Enabling compression can reduce the size of data being sent, decreasing network traffic and increasing throughput.

  • Compression type (compression.type): Supported types are gzip, snappy, lz4, and zstd.
props.put("compression.type", "snappy");

2.3 Asynchronous Send​

Producers can send data asynchronously to minimize the time spent waiting for broker acknowledgments.

  • Linger time (linger.ms): The producer will wait for up to this time before sending a batch of messages, allowing more records to be added to the batch.
props.put("linger.ms", 10); // Wait 10 ms before sending a batch

2.4 Producer Configuration for High Throughput​

Below is a configuration snippet optimized for throughput:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "1"); // Leader acknowledgment for faster send
props.put("compression.type", "lz4");
props.put("batch.size", 32768); // 32KB batch size
props.put("linger.ms", 5); // Wait for 5ms before sending
props.put("max.in.flight.requests.per.connection", "5"); // Max concurrent requests

3. Ensuring Data Reliability​

Reliability involves ensuring that messages are delivered and persisted correctly by the Kafka broker. To achieve this, several configurations control how Kafka handles acknowledgment and retries.

3.1 Acknowledgment (acks)​

  • acks configuration: Controls how many replicas must acknowledge the record. The safest setting is acks=all, ensuring the data is replicated to all in-sync replicas (ISR).
props.put("acks", "all");

3.2 Retries and Idempotence​

  • retries: Controls the number of retry attempts if a request fails.
  • enable.idempotence: Ensures that retries don't result in duplicated messages.
props.put("retries", 3); // Retry 3 times on failure
props.put("enable.idempotence", "true"); // Avoid duplicate messages

3.3 Durability with Replication​

Kafka ensures durability with data replication across multiple brokers. Setting the acks=all guarantees that the data is fully replicated before considering the record as successfully sent.

props.put("acks", "all"); // Wait for acknowledgment from all replicas
props.put("min.insync.replicas", 2); // At least 2 replicas must acknowledge

4. Guaranteeing Exactly Once Semantics​

Kafka provides exactly-once semantics (EOS), ensuring that data is neither lost nor duplicated. This is critical for financial systems and applications that cannot tolerate duplicates or lost data.

4.1 Idempotent Producer​

An idempotent producer ensures that duplicate messages are not written to the Kafka log, even in case of retries. This is achieved by enabling the idempotence feature.

props.put("enable.idempotence", "true");

4.2 Transactions​

Transactions allow multiple operations (e.g., sending messages to different partitions) to be treated as a single unit, ensuring that either all messages are written successfully or none are written.

// Configuring a transactional producer
props.put("transactional.id", "my-transaction-id");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();
producer.beginTransaction();

try {
producer.send(new ProducerRecord<>("topic1", "key", "value"));
producer.send(new ProducerRecord<>("topic2", "key", "value"));
producer.commitTransaction();
} catch (Exception e) {
producer.abortTransaction();
}

5. Maintaining Data Order​

By default, Kafka guarantees that records are written in the order they are sent, but only within a partition. You can control data ordering by correctly configuring partitioning strategies.

5.1 Partitioning​

Kafka producers can partition records using a key. Messages with the same key will always go to the same partition, ensuring their order is maintained.

producer.send(new ProducerRecord<>("topic", "key", "value"));

5.2 Producer Configuration for Ordering​

To maintain message order, especially during retries, the max.in.flight.requests.per.connection setting is crucial. Setting it to 1 ensures that only one request is sent to the broker at a time.

props.put("max.in.flight.requests.per.connection", 1); // Guarantees ordering
props.put("enable.idempotence", "true"); // Ensures no duplicates

6. Conclusion​

Kafka producer tuning involves balancing between throughput, reliability, and data ordering based on the specific requirements of the application. By configuring batching, compression, retries, and idempotence, Kafka producers can achieve high performance while maintaining the necessary level of data reliability and ordering.


Diagram: Kafka Producer Configuration for Throughput and Reliability​

+----------------------+ +----------------------+ +---------------------+
| Producer Properties | -----> | Batch, Compression, | -----> | High Throughput and |
| Configuration | | Asynchronous Send | | Reliable Delivery |
+----------------------+ +----------------------+ +---------------------+
|
v
+----------------------+ +---------------------+ +---------------------+
| Idempotence, | -----> | Exactly-Once | -----> | Guaranteed Message |
| Transactions Enabled | | Semantics | | Ordering and No Dups|
+----------------------+ +---------------------+ +---------------------+

Configuration Parameters​

max.in.flight.requests.per.connection​

The max.in.flight.requests.per.connection configuration in Kafka producers controls how many requests can be sent concurrently to the Kafka broker over a single connection. This setting plays a crucial role in determining how Kafka handles the trade-off between throughput, message ordering, and data reliability.

How it Works:​

  • When a producer sends messages to Kafka, each message is sent in a request to the broker.
  • If the producer has multiple requests to send at the same time (due to batching or high message rates), these requests can either be sent sequentially or in parallel.
  • The max.in.flight.requests.per.connection setting defines how many such requests can be "in-flight" — meaning waiting for a response from the broker — at any given time.
Default Value:​

  • By default, this value is set to 5. This allows the producer to send up to five requests at once before receiving responses for any of them.
props.put("max.in.flight.requests.per.connection", 5);
Impact on Performance and Ordering:​

1. Throughput:​

  • Higher Values: Setting max.in.flight.requests.per.connection to a higher number increases throughput, as the producer can send multiple requests simultaneously without waiting for each one to be acknowledged. This reduces the time spent waiting and maximizes network bandwidth usage.

  • Lower Values: If you lower the value, for example, to 1, the producer will wait for a response from the broker for each request before sending the next one. While this ensures ordering, it can limit throughput because the producer spends more time waiting for acknowledgments.

2. Message Ordering:​

  • Higher Values: When max.in.flight.requests.per.connection is greater than 1, messages can potentially be delivered out of order if a failure occurs. For example, if two requests (Request A and Request B) are sent in parallel, and Request A fails while Request B succeeds, Kafka may retry Request A, leading to message reordering.

  • Lower Values: Setting this value to 1 ensures strict message ordering. This is because the producer waits for each request to be acknowledged before sending the next one, preventing any reordering even in the case of retries.

props.put("max.in.flight.requests.per.connection", 1); // Guarantees strict message order
3. Reliability:​

  • Higher Values: While more concurrent requests can improve throughput, they can lead to situations where retries (in case of failure) result in duplicate messages or out-of-order delivery.

  • Lower Values: With a value of 1, Kafka guarantees that even if a retry occurs (due to network failure, for example), messages will be delivered in the correct order. Coupled with enable.idempotence=true, this ensures that no duplicate messages are sent, and they arrive in the correct sequence.

  • For High Throughput:

    • Use higher values (such as 5 or more). This allows Kafka to send multiple messages simultaneously, maximizing throughput at the potential cost of strict ordering if failures occur.
  • For Exactly Once Semantics (EOS):

    • Use 1 in conjunction with enable.idempotence=true and retries to ensure no duplicates and guarantee strict message ordering.
// High throughput with risk of out-of-order messages on failure
props.put("max.in.flight.requests.per.connection", 5);
props.put("enable.idempotence", "true");

// Strict ordering and exactly-once delivery
props.put("max.in.flight.requests.per.connection", 1);
props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);

Summary of Key Points:​

  • Higher max.in.flight.requests.per.connection increases throughput but may lead to message reordering on failure.
  • Lower max.in.flight.requests.per.connection ensures strict ordering but may decrease throughput as the producer waits for each acknowledgment.
  • For exactly-once semantics (EOS) and strict ordering, set it to 1 in conjunction with enable.idempotence=true.

Diagram: Effect of max.in.flight.requests.per.connection on Kafka Producer Behavior​

+------------------------+ +----------------------+ +-----------------------+
| max.in.flight = 5 | ------> | Higher Throughput | -----> | Possible Reordering |
| Multiple concurrent | | Faster message send | | Retries can cause out-|
| requests allowed | | with reduced latency | | of-order delivery |
+------------------------+ +----------------------+ +-----------------------+

+------------------------+ +----------------------+ +-----------------------+
| max.in.flight = 1 | ------> | Strict Ordering | -----> | Lower Throughput |
| One request at a time | | Each message waits | | Wait time for each |
| Strict sequence | | for ack before next | | acknowledgment |
+------------------------+ +----------------------+ +-----------------------+

Kafka Transaction

· 6 min read

Kafka Transaction Workflow


1. Introduction​

Let’s first recall the concept of a transaction: either all operations succeed, or all of them fail. Kafka transactions follow the same rule.

Starting from Kafka 0.11.0.0, two major features were introduced: idempotence and transactions. Understanding transactions requires understanding idempotence because transactions are built on top of idempotence. This article explains how Kafka transactions work in simple terms, without diving too deep into technical details.

The key points we will cover:

  • What is idempotence? How to enable it?
  • How does idempotence work?
  • What are Kafka transactions? How to enable transactions?
  • How do Kafka transactions work?

2. Idempotence in Kafka​

Idempotence ensures that no matter how many times the producer sends the same message, the Kafka broker will only persist it once, ensuring no data duplication. Idempotence is enabled by default and can be configured using the enable.idempotence setting.

2.1 How Idempotence Works​

The idempotence mechanism in Kafka is simple. Each message has a unique key composed of <PID, Partition, SeqNumber>:

  • PID (Producer ID): A unique identifier assigned to each producer when it starts.
  • Partition: The target partition for the message.
  • SeqNumber: An incrementing ID assigned to each message by the producer, ensuring that each message has a unique sequence number.

Kafka prevents duplicate message persistence for messages with the same key. However, idempotence is limited to single partition and single session guarantees. If Kafka restarts and a new PID is assigned, duplicates could occur. This is where Kafka transactions come into play.


3. Kafka Transactions​

Kafka transactions allow for atomic writes across multiple topics and partitions. All messages in the same transaction are either fully committed or fully aborted.

Kafka transactions are primarily concerned with producer transactions, though consumer transactions are also possible. However, consumers rely on the producer’s transaction mechanism, so we will focus on producer transactions here.

3.1 How to Enable Kafka Transactions​

Before delving into the internals of Kafka transactions, let’s start with a simple example.

Example: Creating a Kafka Producer with Transactions​

Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

// Set the transactional ID, this is required
properties.setProperty(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "transactional_id_1");

KafkaProducer<String, String> producer = new KafkaProducer<>(properties);

// Initialize transaction
producer.initTransactions();

// Start transaction
producer.beginTransaction();

try {
// Send 10 messages to Kafka, if an error occurs, all messages will be aborted
for (int i = 0; i < 10; i++) {
producer.send(new ProducerRecord<>("topic-test", "a message " + i));
}
// Commit the transaction
producer.commitTransaction();
} catch (Exception e) {
// Abort the transaction if any exception occurs
producer.abortTransaction();
} finally {
producer.close();
}

In this code:

  • The producer is configured with a transactional ID, which is essential for Kafka transactions.
  • The producer starts a transaction with beginTransaction().
  • After sending messages, it commits the transaction using commitTransaction(). If any error occurs, the transaction is aborted with abortTransaction().

3.2 How Kafka Transactions Work​

1) Starting the Producer and Assigning a Transaction Coordinator​

When using transactions, the producer must be assigned a transactional ID. Upon startup, Kafka assigns a Transaction Coordinator based on this ID. Each Kafka broker has a transaction coordinator responsible for assigning Producer IDs (PIDs) and managing transactions.

The allocation of a transaction coordinator involves a special Kafka topic __transaction_state, which by default has 50 partitions. Each partition is responsible for a portion of the transactions. Kafka calculates the hash of the transactional ID and assigns the producer to a specific partition, whose leader broker will become the transaction coordinator.

After assigning a coordinator, the coordinator assigns a PID to the producer, allowing it to start sending messages.

2) Sending Messages​

After receiving its PID, the producer informs the coordinator which partitions the transaction will target. Then, it starts sending messages. These messages are flagged as part of a transaction.

Once the producer has sent all its transactional messages, it sends either a Commit or Abort request to the transaction coordinator, signaling the end of its task.

3) Committing or Aborting the Transaction​

When the producer starts sending messages, the coordinator marks the beginning of the transaction and records it in the __transaction_state topic.

Once all messages are sent, or in case of failure, the coordinator will receive a Commit or Abort request. The coordinator then communicates with all the topics involved in the transaction. If the transaction is successful, the topics confirm receipt of the messages, and the coordinator records the successful transaction.

If the transaction fails, all messages related to the transaction are discarded, and the coordinator records the failure in __transaction_state.


4. Diagram: Kafka Transaction Flow​

Below is a diagram representing the Kafka transaction process:

+---------------------------+ +----------------------+ +---------------------+
| Producer Starts | -----> | Assign Transaction | -----> | Send Messages to |
| (with transactional ID) | | Coordinator & Assign | | Partitions |
+---------------------------+ | PID | +---------------------+
+----------------------+

|
|
v
+------------------------+ +------------------------+ +----------------------+
| Transaction Begins | -----> | Send Commit/Abort | -----> | Coordinator Confirms |
+------------------------+ | Request to Coordinator | | Success or Failure |
+------------------------+ +----------------------+
  1. Producer Starts: The producer is assigned a transactional ID and a transaction coordinator.
  2. Sending Messages: Messages are sent as part of the transaction.
  3. Commit/Abort: The producer sends a Commit or Abort request to the coordinator.
  4. Confirmation: The coordinator confirms the success or failure of the transaction.

5. Conclusion​

Kafka transactions are a powerful feature that enables atomic writes across multiple topics and partitions, ensuring either all messages succeed or all fail. Built on top of Kafka's idempotence mechanism, Kafka transactions provide robust guarantees for message processing, making it a reliable tool for distributed systems.

Key Takeaways:​

  • Idempotence: Ensures that messages are not duplicated, even with retries.
  • Transactions: Allow atomic writes across multiple topics and partitions.
  • Transaction Coordinator: Manages the lifecycle of transactions, including assigning PIDs and handling commit/abort requests.
  • __transaction_state: Special Kafka topic that stores transaction metadata.

By mastering these concepts, you can build robust and reliable Kafka-based applications that handle complex message processing scenarios with ease.