跳到主要内容

1 篇博文 含有标签「big-data」

查看所有标签

Spark in Spring Boot Application

· 阅读需 11 分钟

Introduction​

This article provides a comprehensive guide to developing a Spark application with Spring Boot 3.3.5 to communicate with Kafka and Elasticsearch. The application will handle Avro records in Kafka, JSON data in Elasticsearch, and Parquet files in MinIO using Spark DataFrame and Spark SQL. The article covers the setup, integration, and advanced scenarios for handling large-scale data processing, including the use of Avro Schema Registry when communicating with Kafka.


Prerequisites​

  • Java 17 or later
  • Apache Spark 3.5.4
  • Spring Boot 3.3.5
  • Kafka 3.x
  • Elasticsearch 8.x
  • MinIO
  • Maven or Gradle
  • Confluent Schema Registry

Project Folder Structure​

spark-spring-boot-app/
├── src/
│ ├── main/
│ │ ├── java/
│ │ │ ├── com/
│ │ │ │ ├── example/
│ │ │ │ │ ├── config/
│ │ │ │ │ │ ├── SparkConfig.java
│ │ │ │ │ ├── service/
│ │ │ │ │ │ ├── KafkaService.java
│ │ │ │ │ │ ├── ElasticsearchService.java
│ │ │ │ │ │ ├── MinIOService.java
│ │ │ │ │ │ ├── DataFrameService.java
│ │ │ │ │ ├── SparkSpringBootConsoleApplication.java
│ │ ├── resources/
│ │ │ ├── application.properties
│ ├── test/
│ │ ├── java/
│ │ │ ├── com/
│ │ │ │ ├── example/
│ │ │ │ │ ├── service/
│ │ │ │ │ │ ├── KafkaServiceTest.java
│ │ │ │ │ │ ├── ElasticsearchServiceTest.java
│ │ │ │ │ │ ├── MinIOServiceTest.java
│ │ │ │ │ │ ├── DataFrameServiceTest.java
├── pom.xml (or build.gradle)

Setting Up the Spring Boot Application​

Step 1: Create a Spring Boot Project​

Create a new Spring Boot project using Spring Initializr or your preferred IDE. Include the following dependencies:

  • Spring for Apache Kafka
  • Spring Data Elasticsearch
  • Spring for Apache Spark

Step 2: Configure Application Properties​

Configure the application properties for Kafka, Elasticsearch, and MinIO:

# Kafka Configuration
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.ByteArraySerializer

# Schema Registry Configuration
schema.registry.url=http://localhost:8081

# Elasticsearch Configuration
spring.elasticsearch.rest.uris=http://localhost:9200

# MinIO Configuration
minio.endpoint=http://localhost:9000
minio.accessKey=your-access-key
minio.secretKey=your-secret-key
minio.bucket=your-bucket

Integrating Spark with Spring Boot​

Step 1: Add Spark Dependencies​

Add the following dependencies to your pom.xml or build.gradle file:

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>7.0.1</version>
</dependency>

Step 2: Configure Spark​

Configure Spark in your Spring Boot application using annotation-style configuration:

import org.apache.spark.sql.SparkSession;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class SparkConfig {

@Bean
public SparkSession sparkSession() {
return SparkSession.builder()
.appName("SparkSpringBootApp")
.master("local[*]")
.config("spark.sql.shuffle.partitions", "10")
.getOrCreate();
}
}

Communicating with Kafka​

Reading Avro Records from Kafka​

To read Avro records from Kafka, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readAvroFromKafka(String topic) {
return sparkSession.read()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.load()
.selectExpr("CAST(value AS BINARY)")
.select(org.apache.spark.sql.avro.functions.from_avro(col("value"), "your_avro_schema"));
}
}

Reading Avro Records from Kafka with Schema Registry​

To read Avro records from Kafka using the Schema Registry, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

@Value("${schema.registry.url}")
private String schemaRegistryUrl;

public Dataset<Row> readAvroFromKafka(String topic) {
return sparkSession.read()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.load()
.selectExpr("CAST(value AS BINARY)")
.select(org.apache.spark.sql.avro.functions.from_avro(col("value"), schemaRegistryUrl + "/subjects/" + topic + "-value/versions/latest"));
}
}

Writing Data to Kafka​

To write data to Kafka, use the spark-sql-kafka-0-10 library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

public void writeToKafka(Dataset<Row> dataset, String topic) {
dataset.selectExpr("to_json(struct(*)) AS value")
.write()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", topic)
.save();
}
}

Writing Avro Records to Kafka with Schema Registry​

To write Avro records to Kafka using the Schema Registry, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

@Value("${schema.registry.url}")
private String schemaRegistryUrl;

public void writeAvroToKafka(Dataset<Row> dataset, String topic) {
dataset.select(org.apache.spark.sql.avro.functions.to_avro(struct("*")).as("value"))
.write()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", topic)
.option("schema.registry.url", schemaRegistryUrl)
.save();
}
}

Communicating with Elasticsearch​

Reading JSON Data from Elasticsearch​

To read JSON data from Elasticsearch, use the elasticsearch-hadoop library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ElasticsearchService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readFromElasticsearch(String index) {
return sparkSession.read()
.format("org.elasticsearch.spark.sql")
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.resource", index)
.load();
}
}

Writing Data to Elasticsearch​

To write data to Elasticsearch, use the elasticsearch-hadoop library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ElasticsearchService {

@Autowired
private SparkSession sparkSession;

public void writeToElasticsearch(Dataset<Row> dataset, String index) {
dataset.write()
.format("org.elasticsearch.spark.sql")
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.resource", index)
.save();
}
}

Handling Parquet Files in MinIO​

Reading Parquet Files from MinIO​

To read Parquet files from MinIO, use the spark-hadoop-cloud library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class MinIOService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readParquetFromMinIO(String bucket, String path) {
return sparkSession.read()
.format("parquet")
.option("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000")
.option("spark.hadoop.fs.s3a.access.key", "your-access-key")
.option("spark.hadoop.fs.s3a.secret.key", "your-secret-key")
.option("spark.hadoop.fs.s3a.path.style.access", "true")
.load("s3a://" + bucket + "/" + path);
}
}

Writing Parquet Files to MinIO​

To write Parquet files to MinIO, use the spark-hadoop-cloud library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class MinIOService {

@Autowired
private SparkSession sparkSession;

public void writeParquetToMinIO(Dataset<Row> dataset, String bucket, String path) {
dataset.write()
.format("parquet")
.option("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000")
.option("spark.hadoop.fs.s3a.access.key", "your-access-key")
.option("spark.hadoop.fs.s3a.secret.key", "your-secret-key")
.option("spark.hadoop.fs.s3a.path.style.access", "true")
.save("s3a://" + bucket + "/" + path);
}
}

Using Spark DataFrame and Spark SQL​

Creating DataFrames​

To create DataFrames, use the SparkSession object:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class DataFrameService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> createDataFrame() {
return sparkSession.read()
.format("csv")
.option("header", "true")
.load("path/to/csv/file");
}
}

Querying DataFrames with Spark SQL​

To query DataFrames using Spark SQL, register the DataFrame as a temporary view:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class DataFrameService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> queryDataFrame(Dataset<Row> df) {
df.createOrReplaceTempView("my_table");
return sparkSession.sql("SELECT * FROM my_table WHERE column = 'value'");
}
}

Unit Test Cases​

KafkaServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class KafkaServiceTest {

@Autowired
private KafkaService kafkaService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadAvroFromKafka() {
Dataset<Row> dataset = kafkaService.readAvroFromKafka("test-topic");
assertNotNull(dataset);
}

@Test
public void testWriteAvroToKafka() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
kafkaService.writeAvroToKafka(dataset, "test-topic");
}
}

ElasticsearchServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class ElasticsearchServiceTest {

@Autowired
private ElasticsearchService elasticsearchService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadFromElasticsearch() {
Dataset<Row> dataset = elasticsearchService.readFromElasticsearch("test-index");
assertNotNull(dataset);
}

@Test
public void testWriteToElasticsearch() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
elasticsearchService.writeToElasticsearch(dataset, "test-index");
}
}

MinIOServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class MinIOServiceTest {

@Autowired
private MinIOService minIOService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadParquetFromMinIO() {
Dataset<Row> dataset = minIOService.readParquetFromMinIO("test-bucket", "test-path");
assertNotNull(dataset);
}

@Test
public void testWriteParquetToMinIO() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
minIOService.writeParquetToMinIO(dataset, "test-bucket", "test-path");
}
}

DataFrameServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class DataFrameServiceTest {

@Autowired
private DataFrameService dataFrameService;

@Autowired
private SparkSession sparkSession;

@Test
public void testCreateDataFrame() {
Dataset<Row> dataset = dataFrameService.createDataFrame();
assertNotNull(dataset);
}

@Test
public void testQueryDataFrame() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
Dataset<Row> result = dataFrameService.queryDataFrame(dataset);
assertNotNull(result);
}
}

Design Diagrams​

Component Diagram​

+-------------------+ +-------------------+ +-------------------+
| | | | | |
| Spring Boot | | Apache Spark | | Kafka |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| | |
v v v
+-------------------+ +-------------------+ +-------------------+
| | | | | |
| KafkaService | | Elasticsearch | | MinIOService |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| | |
v v v
+--------------------+ +-------------------+ +---------------------------------------+
| | | | | |
| DataFrameService | | SparkConfig | | SparkSpringBootConsoleApplication |
| | | | | |
+--------------------+ +-------------------+ +---------------------------------------+

Sequence Diagram​

+---------+ +---------+ +---------+ +---------+
| | | | | | | |
| Client | | Spring | | Spark | | Kafka |
| | | Boot | | | | |
+---------+ +---------+ +---------+ +---------+
| | | |
| Request | | |
|----------------->| | |
| | | |
| | Read Avro | |
| | from Kafka | |
| |-------------->| |
| | | |
| | | Send Avro |
| | |<------------->|
| | | |
| | Process | |
| | Data | |
| |-------------->| |
| | | |
| | Write to | |
| | Elasticsearch| |
| |-------------->| |
| | | |
| | Write to | |
| | MinIO | |
| |-------------->| |
| | | |
| Response | | |
|<-----------------| | |
| | | |

Advanced Scenarios​

Complex Scenario 1: Flow Control and Backpressure Handling​

In scenarios where your application needs to handle flow control and backpressure, you can use Spark's built-in mechanisms for handling large-scale data processing.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class FlowControlService {

@Autowired
private SparkSession sparkSession;

public void handleFlowControl(Dataset<Row> dataset) {
dataset.writeStream()
.format("console")
.option("checkpointLocation", "path/to/checkpoint")
.start()
.awaitTermination();
}
}

Complex Scenario 2: Handling Large-Scale Data Processing​

For large-scale data processing, use Spark's distributed processing capabilities and optimize resource usage.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class LargeScaleProcessingService {

@Autowired
private SparkSession sparkSession;

public void processLargeScaleData(Dataset<Row> dataset) {
dataset.repartition(100)
.write()
.format("parquet")
.save("path/to/output");
}
}

Conclusion​

Developing a Spark application with Spring Boot 3.3.5 to communicate with Kafka and Elasticsearch involves setting up the Spring Boot application, integrating Spark, and handling data using Spark DataFrame and Spark SQL. By following the best practices outlined in this article, you can create efficient and scalable data processing applications.


References​

  1. Spring Boot Documentation
  2. Apache Spark Documentation
  3. Apache Kafka Documentation
  4. Elasticsearch Documentation
  5. MinIO Documentation
  6. Confluent Schema Registry Documentation

By following these best practices, you can leverage Spark and Spring Boot to build powerful and scalable data processing applications.