Spark in Spring Boot Application
- Introduction
- Prerequisites
- Project Folder Structure
- Setting Up the Spring Boot Application
- Integrating Spark with Spring Boot
- Communicating with Kafka
- Communicating with Elasticsearch
- Handling Parquet Files in MinIO
- Using Spark DataFrame and Spark SQL
- Unit Test Cases
- Design Diagrams
- Advanced Scenarios
- Conclusion
- References
Introduction
This article provides a comprehensive guide to developing a Spark application with Spring Boot 3.3.5 to communicate with Kafka and Elasticsearch. The application will handle Avro records in Kafka, JSON data in Elasticsearch, and Parquet files in MinIO using Spark DataFrame and Spark SQL. The article covers the setup, integration, and advanced scenarios for handling large-scale data processing, including the use of Avro Schema Registry when communicating with Kafka.
Prerequisites
- Java 17 or later
- Apache Spark 3.5.4
- Spring Boot 3.3.5
- Kafka 3.x
- Elasticsearch 8.x
- MinIO
- Maven or Gradle
- Confluent Schema Registry
Project Folder Structure
spark-spring-boot-app/
├── src/
│ ├── main/
│ │ ├── java/
│ │ │ ├── com/
│ │ │ │ ├── example/
│ │ │ │ │ ├── config/
│ │ │ │ │ │ ├── SparkConfig.java
│ │ │ │ │ ├── service/
│ │ │ │ │ │ ├── KafkaService.java
│ │ │ │ │ │ ├── ElasticsearchService.java
│ │ │ │ │ │ ├── MinIOService.java
│ │ │ │ │ │ ├── DataFrameService.java
│ │ │ │ │ ├── SparkSpringBootConsoleApplication.java
│ │ ├── resources/
│ │ │ ├── application.properties
│ ├── test/
│ │ ├── java/
│ │ │ ├── com/
│ │ │ │ ├── example/
│ │ │ │ │ ├── service/
│ │ │ │ │ │ ├── KafkaServiceTest.java
│ │ │ │ │ │ ├── ElasticsearchServiceTest.java
│ │ │ │ │ │ ├── MinIOServiceTest.java
│ │ │ │ │ │ ├── DataFrameServiceTest.java
├── pom.xml (or build.gradle)
Setting Up the Spring Boot Application
Step 1: Create a Spring Boot Project
Create a new Spring Boot project using Spring Initializr or your preferred IDE. Include the following dependencies:
- Spring for Apache Kafka
- Spring Data Elasticsearch
- Spring for Apache Spark
Step 2: Configure Application Properties
Configure the application properties for Kafka, Elasticsearch, and MinIO:
# Kafka Configuration
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.ByteArraySerializer
# Schema Registry Configuration
schema.registry.url=http://localhost:8081
# Elasticsearch Configuration
spring.elasticsearch.rest.uris=http://localhost:9200
# MinIO Configuration
minio.endpoint=http://localhost:9000
minio.accessKey=your-access-key
minio.secretKey=your-secret-key
minio.bucket=your-bucket
Integrating Spark with Spring Boot
Step 1: Add Spark Dependencies
Add the following dependencies to your pom.xml or build.gradle file:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>7.0.1</version>
</dependency>
Step 2: Configure Spark
Configure Spark in your Spring Boot application using annotation-style configuration:
import org.apache.spark.sql.SparkSession;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class SparkConfig {
@Bean
public SparkSession sparkSession() {
return SparkSession.builder()
.appName("SparkSpringBootApp")
.master("local[*]")
.config("spark.sql.shuffle.partitions", "10")
.getOrCreate();
}
}
Communicating with Kafka
Reading Avro Records from Kafka
To read Avro records from Kafka, use the spark-avro and spark-sql-kafka-0-10 libraries:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class KafkaService {
@Autowired
private SparkSession sparkSession;
public Dataset<Row> readAvroFromKafka(String topic) {
return sparkSession.read()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.load()
.selectExpr("CAST(value AS BINARY)")
.select(org.apache.spark.sql.avro.functions.from_avro(col("value"), "your_avro_schema"));
}
}
Reading Avro Records from Kafka with Schema Registry
To read Avro records from Kafka using the Schema Registry, use the spark-avro and spark-sql-kafka-0-10 libraries:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
@Service
public class KafkaService {
@Autowired
private SparkSession sparkSession;
@Value("${schema.registry.url}")
private String schemaRegistryUrl;
public Dataset<Row> readAvroFromKafka(String topic) {
return sparkSession.read()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.load()
.selectExpr("CAST(value AS BINARY)")
.select(org.apache.spark.sql.avro.functions.from_avro(col("value"), schemaRegistryUrl + "/subjects/" + topic + "-value/versions/latest"));
}
}
Writing Data to Kafka
To write data to Kafka, use the spark-sql-kafka-0-10 library:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class KafkaService {
@Autowired
private SparkSession sparkSession;
public void writeToKafka(Dataset<Row> dataset, String topic) {
dataset.selectExpr("to_json(struct(*)) AS value")
.write()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", topic)
.save();
}
}
Writing Avro Records to Kafka with Schema Registry
To write Avro records to Kafka using the Schema Registry, use the spark-avro and spark-sql-kafka-0-10 libraries:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
@Service
public class KafkaService {
@Autowired
private SparkSession sparkSession;
@Value("${schema.registry.url}")
private String schemaRegistryUrl;
public void writeAvroToKafka(Dataset<Row> dataset, String topic) {
dataset.select(org.apache.spark.sql.avro.functions.to_avro(struct("*")).as("value"))
.write()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", topic)
.option("schema.registry.url", schemaRegistryUrl)
.save();
}
}
Communicating with Elasticsearch
Reading JSON Data from Elasticsearch
To read JSON data from Elasticsearch, use the elasticsearch-hadoop library:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class ElasticsearchService {
@Autowired
private SparkSession sparkSession;
public Dataset<Row> readFromElasticsearch(String index) {
return sparkSession.read()
.format("org.elasticsearch.spark.sql")
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.resource", index)
.load();
}
}
Writing Data to Elasticsearch
To write data to Elasticsearch, use the elasticsearch-hadoop library:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class ElasticsearchService {
@Autowired
private SparkSession sparkSession;
public void writeToElasticsearch(Dataset<Row> dataset, String index) {
dataset.write()
.format("org.elasticsearch.spark.sql")
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.resource", index)
.save();
}
}
Handling Parquet Files in MinIO
Reading Parquet Files from MinIO
To read Parquet files from MinIO, use the spark-hadoop-cloud library:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class MinIOService {
@Autowired
private SparkSession sparkSession;
public Dataset<Row> readParquetFromMinIO(String bucket, String path) {
return sparkSession.read()
.format("parquet")
.option("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000")
.option("spark.hadoop.fs.s3a.access.key", "your-access-key")
.option("spark.hadoop.fs.s3a.secret.key", "your-secret-key")
.option("spark.hadoop.fs.s3a.path.style.access", "true")
.load("s3a://" + bucket + "/" + path);
}
}
Writing Parquet Files to MinIO
To write Parquet files to MinIO, use the spark-hadoop-cloud library:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class MinIOService {
@Autowired
private SparkSession sparkSession;
public void writeParquetToMinIO(Dataset<Row> dataset, String bucket, String path) {
dataset.write()
.format("parquet")
.option("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000")
.option("spark.hadoop.fs.s3a.access.key", "your-access-key")
.option("spark.hadoop.fs.s3a.secret.key", "your-secret-key")
.option("spark.hadoop.fs.s3a.path.style.access", "true")
.save("s3a://" + bucket + "/" + path);
}
}
Using Spark DataFrame and Spark SQL
Creating DataFrames
To create DataFrames, use the SparkSession object:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class DataFrameService {
@Autowired
private SparkSession sparkSession;
public Dataset<Row> createDataFrame() {
return sparkSession.read()
.format("csv")
.option("header", "true")
.load("path/to/csv/file");
}
}
Querying DataFrames with Spark SQL
To query DataFrames using Spark SQL, register the DataFrame as a temporary view:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class DataFrameService {
@Autowired
private SparkSession sparkSession;
public Dataset<Row> queryDataFrame(Dataset<Row> df) {
df.createOrReplaceTempView("my_table");
return sparkSession.sql("SELECT * FROM my_table WHERE column = 'value'");
}
}
Unit Test Cases
KafkaServiceTest.java
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@SpringBootTest
public class KafkaServiceTest {
@Autowired
private KafkaService kafkaService;
@Autowired
private SparkSession sparkSession;
@Test
public void testReadAvroFromKafka() {
Dataset<Row> dataset = kafkaService.readAvroFromKafka("test-topic");
assertNotNull(dataset);
}
@Test
public void testWriteAvroToKafka() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
kafkaService.writeAvroToKafka(dataset, "test-topic");
}
}
ElasticsearchServiceTest.java
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@SpringBootTest
public class ElasticsearchServiceTest {
@Autowired
private ElasticsearchService elasticsearchService;
@Autowired
private SparkSession sparkSession;
@Test
public void testReadFromElasticsearch() {
Dataset<Row> dataset = elasticsearchService.readFromElasticsearch("test-index");
assertNotNull(dataset);
}
@Test
public void testWriteToElasticsearch() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
elasticsearchService.writeToElasticsearch(dataset, "test-index");
}
}
MinIOServiceTest.java
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@SpringBootTest
public class MinIOServiceTest {
@Autowired
private MinIOService minIOService;
@Autowired
private SparkSession sparkSession;
@Test
public void testReadParquetFromMinIO() {
Dataset<Row> dataset = minIOService.readParquetFromMinIO("test-bucket", "test-path");
assertNotNull(dataset);
}
@Test
public void testWriteParquetToMinIO() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
minIOService.writeParquetToMinIO(dataset, "test-bucket", "test-path");
}
}
DataFrameServiceTest.java
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@SpringBootTest
public class DataFrameServiceTest {
@Autowired
private DataFrameService dataFrameService;
@Autowired
private SparkSession sparkSession;
@Test
public void testCreateDataFrame() {
Dataset<Row> dataset = dataFrameService.createDataFrame();
assertNotNull(dataset);
}
@Test
public void testQueryDataFrame() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
Dataset<Row> result = dataFrameService.queryDataFrame(dataset);
assertNotNull(result);
}
}
Design Diagrams
Component Diagram
+-------------------+ +-------------------+ +-------------------+
| | | | | |
| Spring Boot | | Apache Spark | | Kafka |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| | |
v v v
+-------------------+ +-------------------+ +-------------------+
| | | | | |
| KafkaService | | Elasticsearch | | MinIOService |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| | |
v v v
+--------------------+ +-------------------+ +---------------------------------------+
| | | | | |
| DataFrameService | | SparkConfig | | SparkSpringBootConsoleApplication |
| | | | | |
+--------------------+ +-------------------+ +---------------------------------------+
Sequence Diagram
+---------+ +---------+ +---------+ +---------+
| | | | | | | |
| Client | | Spring | | Spark | | Kafka |
| | | Boot | | | | |
+---------+ +---------+ +---------+ +---------+
| | | |
| Request | | |
|----------------->| | |
| | | |
| | Read Avro | |
| | from Kafka | |
| |-------------->| |
| | | |
| | | Send Avro |
| | |<------------->|
| | | |
| | Process | |
| | Data | |
| |-------------->| |
| | | |
| | Write to | |
| | Elasticsearch| |
| |-------------->| |
| | | |
| | Write to | |
| | MinIO | |
| |-------------->| |
| | | |
| Response | | |
|<-----------------| | |
| | | |
Advanced Scenarios
Complex Scenario 1: Flow Control and Backpressure Handling
In scenarios where your application needs to handle flow control and backpressure, you can use Spark's built-in mechanisms for handling large-scale data processing.
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class FlowControlService {
@Autowired
private SparkSession sparkSession;
public void handleFlowControl(Dataset<Row> dataset) {
dataset.writeStream()
.format("console")
.option("checkpointLocation", "path/to/checkpoint")
.start()
.awaitTermination();
}
}
Complex Scenario 2: Handling Large-Scale Data Processing
For large-scale data processing, use Spark's distributed processing capabilities and optimize resource usage.
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class LargeScaleProcessingService {
@Autowired
private SparkSession sparkSession;
public void processLargeScaleData(Dataset<Row> dataset) {
dataset.repartition(100)
.write()
.format("parquet")
.save("path/to/output");
}
}
Conclusion
Developing a Spark application with Spring Boot 3.3.5 to communicate with Kafka and Elasticsearch involves setting up the Spring Boot application, integrating Spark, and handling data using Spark DataFrame and Spark SQL. By following the best practices outlined in this article, you can create efficient and scalable data processing applications.
References
- Spring Boot Documentation
- Apache Spark Documentation
- Apache Kafka Documentation
- Elasticsearch Documentation
- MinIO Documentation
- Confluent Schema Registry Documentation
By following these best practices, you can leverage Spark and Spring Boot to build powerful and scalable data processing applications.