跳到主要内容

3 篇博文 含有标签「spark」

查看所有标签

Spark as a Service with Spring Boot

· 阅读需 13 分钟
  1. Tech Stack and Target Environment
  2. Architecture Design
  3. Flow Descriptions
  4. Sequence Diagrams
  5. Components Design
  6. Configuration and Code
  7. Job Queue Management
  8. Spark Job Submission
  9. Test Cases
  10. Global Exception Handler
  11. Project Structure

In this article, we will explore the development of a stateless Spring Boot application using Spring WebFlux 3.3.5 and Apache Spark 3.5.4. This application will serve as a Spark-as-a-Service, designed to run in a Kubernetes (K8s) environment with multiple replicas. The application will use a centralized queue in Redis to manage Spark job requests, with a copy of the job queue also maintained in PostgreSQL for job history and tracking. The application will ensure that each job is executed only once, manage job states and statuses correctly, and provide RESTful endpoints for client interaction and monitoring.

Tech Stack and Target Environment​

Spark Cluster​

  • Version: 3.5.4
  • Deployment: Kubernetes namespace whereq-saas-spark
  • Configuration: Default Bitnami Helm chart (1 master, 2 workers)

Spring Boot WebFlux Application​

  • Version: 3.3.5
  • Deployment: Kubernetes namespace whereq-saas-web
  • Replicas: 2
  • Ingress: Exposed for frontend access

Queue and Database​

  • Queue: Redis (for job queue management)
  • Database: PostgreSQL with R2DBC driver
  • Container Orchestration: Kubernetes

Architecture Design​

The application is structured to be stateless and scalable. It integrates with Redis to queue job submissions, PostgreSQL for persistent storage, and the Spark REST API for job submission and tracking.

High-Level Architecture​

graph TD
A[Client] -->|Submit Task| B[Spring Boot Application]
B -->|Store Task Request| C[Redis Queue]
C -->|Sync Job Request| D[PostgreSQL]
D -->|Fetch Job| B
B -->|Check Cluster Status| E[Spark REST API]
E -->|Determine Resources| B
B -->|Submit Job| F[Spark Master]
F -->|Execute Job| G[Spark Worker]
G -->|Store Result| D
B -->|Query Job History| D
B -->|Monitor Job Queue| C

Components​

  • Job Controller: Accepts job submissions via HTTP POST and returns a task ID.
  • Job Queue Service: Handles job insertion into Redis and PostgreSQL.
  • Job Monitor Service: Monitors Redis, submits jobs to Spark, and tracks job status.
  • Spark Service: Interfaces with the Spark REST API for job submission and status updates.
  • State Machine: Manages job state transitions (e.g., ready, running, completed) and status updates (success, failed).

Architecture Diagram​

+---------------+ +---------------------+ +----------------+
| Job Controller| ----> | Job Queue Service | -----> | Redis Queue |
+---------------+ +---------------------+ +----------------+
|
+--------------------------------------------------------+
|
+-------------------+ +------------------+ +------------------+
| Job Monitor | -----> | Spark Service | <-----> | Spark Master Pod |
| (Background Task) | +------------------+ +------------------+
+-------------------+ |
|
+-------------------------------+
| PostgreSQL (Sync Job State) |
+-------------------------------+

Detailed Architecture​

  1. Client: Submits Spark job requests via RESTful endpoints.
  2. Spring Boot Application:
    • Receives job requests.
    • Stores job requests in the Redis queue.
    • Syncs job requests with PostgreSQL.
    • Checks cluster status via Spark REST API.
    • Submits jobs to Spark Master based on available resources.
    • Manages job states and statuses.
    • Provides RESTful endpoints for job history queries and monitoring.
  3. Redis Queue: Stores job requests.
  4. PostgreSQL: Stores job requests and history.
  5. Spark REST API: Provides cluster status information.
  6. Spark Master: Receives job submissions.
  7. Spark Worker: Executes Spark jobs.
  8. PostgreSQL: Stores job results and history.

Flow Descriptions​

Task Submission Flow​

  1. Task Controller receives task requests via HTTP POST.
  2. Task Queue Service validates the request, assigns a task ID, stores it in Redis with a READY state, and syncs it to PostgreSQL.
  3. The service returns a task ID to the client.

Spark Job Submission Flow​

  1. Job Monitor Service picks up jobs in the READY state from the Redis queue using FIFO.
  2. The job is submitted to the Spark master via the Spark Service, and the job state is updated to SUBMITTED.
  3. The state is synced to PostgreSQL.
  4. The Job Monitor Service starts tracking the job status until completion or failure.
  5. The final state (COMPLETED or FAILED) is updated in both Redis and PostgreSQL.

sequenceDiagram
participant Client
participant SpringBootApp
participant RedisQueue
participant PostgreSQL
participant SparkRESTAPI
participant SparkMaster
participant SparkWorker

Client ->> SpringBootApp: Submit Job Request
SpringBootApp ->> RedisQueue: Store Job Request
RedisQueue ->> PostgreSQL: Sync Job Request
SpringBootApp ->> SparkRESTAPI: Check Cluster Status
SparkRESTAPI -->> SpringBootApp: Cluster Status
SpringBootApp ->> SparkMaster: Submit Job
SparkMaster ->> SparkWorker: Execute Job
SparkWorker -->> PostgreSQL: Store Job Result
SpringBootApp -->> Client: Job Status

Job History Query​

sequenceDiagram
participant Client
participant SpringBootApp
participant PostgreSQL

Client ->> SpringBootApp: Query Job History
SpringBootApp ->> PostgreSQL: Fetch Job History
PostgreSQL -->> SpringBootApp: Job History
SpringBootApp -->> Client: Job History

​

Components Design​

1. Job Request Model​

public class JobRequest {
private String id;
private String jobName;
private String jobConfig;
private JobState state;
private JobStatus status;
// Getters and Setters
}

2. Job State and Status Enums​

public enum JobState {
READY, PENDING, RUNNING, COMPLETED
}

public enum JobStatus {
SUCCESS, FAILED
}

3. Redis Queue Service​

@Service
public class RedisQueueService {
private final ReactiveRedisTemplate<String, JobRequest> redisTemplate;

public RedisQueueService(ReactiveRedisTemplate<String, JobRequest> redisTemplate) {
this.redisTemplate = redisTemplate;
}

public Mono<Boolean> enqueueJob(JobRequest jobRequest) {
return redisTemplate.opsForList().rightPush("jobQueue", jobRequest);
}

public Mono<JobRequest> dequeueJob() {
return redisTemplate.opsForList().leftPop("jobQueue");
}

public Mono<Boolean> updateJobStatusInQueue(String jobId, JobState state, JobStatus status) {
return redisTemplate.opsForList().indexOf("jobQueue", jobId)
.flatMap(index -> redisTemplate.opsForList().set(index, new JobRequest(jobId, state, status)));
}
}

4. PostgreSQL Service​

@Service
public class PostgresService {
private final R2dbcEntityTemplate r2dbcEntityTemplate;

public PostgresService(R2dbcEntityTemplate r2dbcEntityTemplate) {
this.r2dbcEntityTemplate = r2dbcEntityTemplate;
}

public Mono<JobRequest> saveJobRequest(JobRequest jobRequest) {
return r2dbcEntityTemplate.insert(jobRequest);
}

public Mono<JobRequest> getJobById(String jobId) {
return r2dbcEntityTemplate.selectOne(Query.query(Criteria.where("id").is(jobId)), JobRequest.class);
}

public Mono<Integer> updateJobStatus(String jobId, JobState state, JobStatus status) {
return r2dbcEntityTemplate.update(JobRequest.class)
.matching(Query.query(Criteria.where("id").is(jobId)))
.apply(Update.update("state", state).set("status", status));
}

public Flux<JobRequest> getJobHistory(Map<String, String> queryParams) {
// Fetch job history from PostgreSQL
return r2dbcEntityTemplate.select(JobRequest.class).all();
}
}

5. Spark REST API Service​

@Service
public class SparkRestApiService {
private final WebClient webClient;

public SparkRestApiService(WebClient.Builder webClientBuilder) {
this.webClient = webClientBuilder.baseUrl("http://spark-master:8080").build();
}

public Mono<ClusterStatus> getClusterStatus() {
return webClient.get()
.uri("/api/v1/applications")
.retrieve()
.bodyToMono(String.class)
.map(response -> {
// Parse response and return ClusterStatus
return new ClusterStatus();
});
}

public Mono<JobStatus> monitorJobStatus(String jobId) {
return webClient.get()
.uri("/api/v1/submissions/status/{jobId}", jobId)
.retrieve()
.bodyToMono(String.class)
.map(response -> {
// Parse response and return JobStatus
return JobStatus.SUCCESS; // Example
});
}
}

6. Job Executor Service​

@Service
public class JobExecutorService {
private final RedisQueueService redisQueueService;
private final PostgresService postgresService;
private final SparkRestApiService sparkRestApiService;
private final WebClient webClient;

public JobExecutorService(RedisQueueService redisQueueService, PostgresService postgresService, SparkRestApiService sparkRestApiService, WebClient.Builder webClientBuilder) {
this.redisQueueService = redisQueueService;
this.postgresService = postgresService;
this.sparkRestApiService = sparkRestApiService;
this.webClient = webClientBuilder.baseUrl("http://spark-master:8080").build();
}

public Mono<Void> executeNextJob() {
return redisQueueService.dequeueJob()
.flatMap(jobRequest -> {
jobRequest.setState(JobState.PENDING);
return postgresService.updateJobStatus(jobRequest.getId(), JobState.PENDING, JobStatus.SUCCESS)
.then(sparkRestApiService.getClusterStatus())
.flatMap(clusterStatus -> {
if (clusterStatus.hasAvailableResources()) {
return submitJobToSpark(jobRequest);
} else {
return redisQueueService.enqueueJob(jobRequest).then();
}
});
});
}

private Mono<Void> submitJobToSpark(JobRequest jobRequest) {
return webClient.post()
.uri("/v1/submissions/create")
.bodyValue(jobRequest)
.retrieve()
.toBodilessEntity()
.flatMap(response -> {
if (response.getStatusCode().is2xxSuccessful()) {
jobRequest.setState(JobState.RUNNING);
return postgresService.updateJobStatus(jobRequest.getId(), JobState.RUNNING, JobStatus.SUCCESS)
.then(monitorJobResult(jobRequest));
} else {
jobRequest.setState(JobState.COMPLETED);
jobRequest.setStatus(JobStatus.FAILED);
return updateJobStatus(jobRequest);
}
});
}

private Mono<Void> monitorJobResult(JobRequest jobRequest) {
return sparkRestApiService.monitorJobStatus(jobRequest.getId())
.flatMap(jobStatus -> {
jobRequest.setState(JobState.COMPLETED);
jobRequest.setStatus(jobStatus);
return updateJobStatus(jobRequest);
});
}

private Mono<Void> updateJobStatus(JobRequest jobRequest) {
return redisQueueService.updateJobStatusInQueue(jobRequest.getId(), jobRequest.getState(), jobRequest.getStatus())
.then(postgresService.updateJobStatus(jobRequest.getId(), jobRequest.getState(), jobRequest.getStatus()))
.then();
}
}

7. RESTful Endpoints​

@RestController
@RequestMapping("/api/jobs")
public class JobController {
private final JobExecutorService jobExecutorService;
private final PostgresService postgresService;

public JobController(JobExecutorService jobExecutorService, PostgresService postgresService) {
this.jobExecutorService = jobExecutorService;
this.postgresService = postgresService;
}

@PostMapping
public Mono<ResponseEntity<String>> submitJob(@RequestBody JobRequest jobRequest) {
return jobExecutorService.submitJob(jobRequest)
.thenReturn(ResponseEntity.ok("Job submitted successfully"));
}

@GetMapping("/history")
public Flux<JobRequest> getJobHistory(@RequestParam Map<String, String> queryParams) {
return postgresService.getJobHistory(queryParams);
}

@GetMapping("/status")
public Mono<ResponseEntity<String>> getJobStatus(@RequestParam String jobId) {
return postgresService.getJobById(jobId)
.map(jobRequest -> ResponseEntity.ok(jobRequest.getStatus().name()));
}
}

​

Configuration and Code​

1. Spring Boot Application Configuration​

spring:
redis:
host: localhost
port: 6379
r2dbc:
url: r2dbc:postgresql://localhost:5432/spark_jobs
username: postgres
password: postgres
webclient:
baseUrl: http://spark-master:8080

2. Spark REST API Configuration​

@Configuration
public class SparkRestApiConfig {
@Bean
public WebClient.Builder webClientBuilder() {
return WebClient.builder();
}
}

3. Redis Configuration​

@Configuration
public class RedisConfig {
@Bean
public ReactiveRedisTemplate<String, JobRequest> reactiveRedisTemplate(ReactiveRedisConnectionFactory factory) {
RedisSerializationContext<String, JobRequest> serializationContext = RedisSerializationContext
.newSerializationContext(new StringRedisSerializer())
.value(new Jackson2JsonRedisSerializer<>(JobRequest.class))
.build();
return new ReactiveRedisTemplate<>(factory, serializationContext);
}
}

4. R2DBC Configuration​

@Configuration
public class R2dbcConfig {
@Bean
public ConnectionFactory connectionFactory(R2dbcProperties properties) {
return ConnectionFactories.get(properties.getUrl());
}

@Bean
public R2dbcEntityTemplate r2dbcEntityTemplate(ConnectionFactory connectionFactory) {
DatabaseClient databaseClient = DatabaseClient.create(connectionFactory);
return new R2dbcEntityTemplate(databaseClient);
}
}

5. Swagger Configuration​

@Configuration
@EnableSwagger2WebFlux
public class SwaggerConfig {
@Bean
public Docket api() {
return new Docket(DocumentationType.SWAGGER_2)
.select()
.apis(RequestHandlerSelectors.basePackage("com.example.sparkasaservice.controller"))
.paths(PathSelectors.any())
.build();
}
}

​

Job Queue Management​

1. Job Queue Service​

The RedisQueueService class is responsible for managing the job queue in Redis.

@Service
public class RedisQueueService {
private final ReactiveRedisTemplate<String, JobRequest> redisTemplate;

public RedisQueueService(ReactiveRedisTemplate<String, JobRequest> redisTemplate) {
this.redisTemplate = redisTemplate;
}

public Mono<Boolean> enqueueJob(JobRequest jobRequest) {
return redisTemplate.opsForList().rightPush("jobQueue", jobRequest);
}

public Mono<JobRequest> dequeueJob() {
return redisTemplate.opsForList().leftPop("jobQueue");
}

public Mono<Boolean> updateJobStatusInQueue(String jobId, JobState state, JobStatus status) {
return redisTemplate.opsForList().indexOf("jobQueue", jobId)
.flatMap(index -> redisTemplate.opsForList().set(index, new JobRequest(jobId, state, status)));
}
}

2. Job State Management​

The JobRequest class includes fields for JobState and JobStatus to manage the state and status of each job.

public class JobRequest {
private String id;
private String jobName;
private String jobConfig;
private JobState state;
private JobStatus status;
// Getters and Setters
}

3. Job Execution​

The JobExecutorService class is responsible for executing jobs from the queue. It ensures that each job is executed only once and updates the job state and status accordingly.

@Service
public class JobExecutorService {
private final RedisQueueService redisQueueService;
private final PostgresService postgresService;
private final SparkRestApiService sparkRestApiService;
private final WebClient webClient;

public JobExecutorService(RedisQueueService redisQueueService, PostgresService postgresService, SparkRestApiService sparkRestApiService, WebClient.Builder webClientBuilder) {
this.redisQueueService = redisQueueService;
this.postgresService = postgresService;
this.sparkRestApiService = sparkRestApiService;
this.webClient = webClientBuilder.baseUrl("http://spark-master:8080").build();
}

public Mono<Void> executeNextJob() {
return redisQueueService.dequeueJob()
.flatMap(jobRequest -> {
jobRequest.setState(JobState.PENDING);
return postgresService.updateJobStatus(jobRequest.getId(), JobState.PENDING, JobStatus.SUCCESS)
.then(sparkRestApiService.getClusterStatus())
.flatMap(clusterStatus -> {
if (clusterStatus.hasAvailableResources()) {
return submitJobToSpark(jobRequest);
} else {
return redisQueueService.enqueueJob(jobRequest).then();
}
});
});
}

private Mono<Void> submitJobToSpark(JobRequest jobRequest) {
return webClient.post()
.uri("/v1/submissions/create")
.bodyValue(jobRequest)
.retrieve()
.toBodilessEntity()
.flatMap(response -> {
if (response.getStatusCode().is2xxSuccessful()) {
jobRequest.setState(JobState.RUNNING);
return postgresService.updateJobStatus(jobRequest.getId(), JobState.RUNNING, JobStatus.SUCCESS)
.then(monitorJobResult(jobRequest));
} else {
jobRequest.setState(JobState.COMPLETED);
jobRequest.setStatus(JobStatus.FAILED);
return updateJobStatus(jobRequest);
}
});
}

private Mono<Void> monitorJobResult(JobRequest jobRequest) {
return sparkRestApiService.monitorJobStatus(jobRequest.getId())
.flatMap(jobStatus -> {
jobRequest.setState(JobState.COMPLETED);
jobRequest.setStatus(jobStatus);
return updateJobStatus(jobRequest);
});
}

private Mono<Void> updateJobStatus(JobRequest jobRequest) {
return redisQueueService.updateJobStatusInQueue(jobRequest.getId(), jobRequest.getState(), jobRequest.getStatus())
.then(postgresService.updateJobStatus(jobRequest.getId(), jobRequest.getState(), jobRequest.getStatus()))
.then();
}
}

​

Spark Job Submission​

1. Job Submission Endpoint​

The JobController class provides an endpoint for submitting jobs. It uses the JobExecutorService to handle job submission.

@RestController
@RequestMapping("/api/jobs")
public class JobController {
private final JobExecutorService jobExecutorService;
private final PostgresService postgresService;

public JobController(JobExecutorService jobExecutorService, PostgresService postgresService) {
this.jobExecutorService = jobExecutorService;
this.postgresService = postgresService;
}

@PostMapping
public Mono<ResponseEntity<String>> submitJob(@RequestBody JobRequest jobRequest) {
return jobExecutorService.submitJob(jobRequest)
.thenReturn(ResponseEntity.ok("Job submitted successfully"));
}

@GetMapping("/history")
public Flux<JobRequest> getJobHistory(@RequestParam Map<String, String> queryParams) {
return postgresService.getJobHistory(queryParams);
}

@GetMapping("/status")
public Mono<ResponseEntity<String>> getJobStatus(@RequestParam String jobId) {
return postgresService.getJobById(jobId)
.map(jobRequest -> ResponseEntity.ok(jobRequest.getStatus().name()));
}
}

2. Job Execution​

The JobExecutorService class is responsible for executing jobs from the queue. It ensures that each job is executed only once and updates the job state and status accordingly.

@Service
public class JobExecutorService {
private final RedisQueueService redisQueueService;
private final PostgresService postgresService;
private final SparkRestApiService sparkRestApiService;
private final WebClient webClient;

public JobExecutorService(RedisQueueService redisQueueService, PostgresService postgresService, SparkRestApiService sparkRestApiService, WebClient.Builder webClientBuilder) {
this.redisQueueService = redisQueueService;
this.postgresService = postgresService;
this.sparkRestApiService = sparkRestApiService;
this.webClient = webClientBuilder.baseUrl("http://spark-master:8080").build();
}

public Mono<Void> executeNextJob() {
return redisQueueService.dequeueJob()
.flatMap(jobRequest -> {
jobRequest.setState(JobState.PENDING);
return postgresService.updateJobStatus(jobRequest.getId(), JobState.PENDING, JobStatus.SUCCESS)
.then(sparkRestApiService.getClusterStatus())
.flatMap(clusterStatus -> {
if (clusterStatus.hasAvailableResources()) {
return submitJobToSpark(jobRequest);
} else {
return redisQueueService.enqueueJob(jobRequest).then();
}
});
});
}

private Mono<Void> submitJobToSpark(JobRequest jobRequest) {
return webClient.post()
.uri("/v1/submissions/create")
.bodyValue(jobRequest)
.retrieve()
.toBodilessEntity()
.flatMap(response -> {
if (response.getStatusCode().is2xxSuccessful()) {
jobRequest.setState(JobState.RUNNING);
return postgresService.updateJobStatus(jobRequest.getId(), JobState.RUNNING, JobStatus.SUCCESS)
.then(monitorJobResult(jobRequest));
} else {
jobRequest.setState(JobState.COMPLETED);
jobRequest.setStatus(JobStatus.FAILED);
return updateJobStatus(jobRequest);
}
});
}

private Mono<Void> monitorJobResult(JobRequest jobRequest) {
return sparkRestApiService.monitorJobStatus(jobRequest.getId())
.flatMap(jobStatus -> {
jobRequest.setState(JobState.COMPLETED);
jobRequest.setStatus(jobStatus);
return updateJobStatus(jobRequest);
});
}

private Mono<Void> updateJobStatus(JobRequest jobRequest) {
return redisQueueService.updateJobStatusInQueue(jobRequest.getId(), jobRequest.getState(), jobRequest.getStatus())
.then(postgresService.updateJobStatus(jobRequest.getId(), jobRequest.getState(), jobRequest.getStatus()))
.then();
}
}

​

Test Cases​

1. Job Submission Test​

@SpringBootTest
@AutoConfigureWebTestClient
public class JobControllerTest {

@Autowired
private WebTestClient webTestClient;

@Test
public void testJobSubmission() {
JobRequest jobRequest = new JobRequest();
jobRequest.setId("1");
jobRequest.setJobName("TestJob");
jobRequest.setJobConfig("SELECT * FROM test_table");

webTestClient.post().uri("/api/jobs")
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(jobRequest)
.exchange()
.expectStatus().isOk()
.expectBody(String.class).isEqualTo("Job submitted successfully");
}
}

2. Job Execution Test​

@SpringBootTest
public class JobExecutorServiceTest {

@Autowired
private JobExecutorService jobExecutorService;

@Autowired
private RedisQueueService redisQueueService;

@Autowired
private PostgresService postgresService;

@Test
public void testJobExecution() {
JobRequest jobRequest = new JobRequest();
jobRequest.setId("1");
jobRequest.setJobName("TestJob");
jobRequest.setJobConfig("SELECT * FROM test_table");

StepVerifier.create(redisQueueService.enqueueJob(jobRequest)
.then(jobExecutorService.executeNextJob())
.then(postgresService.getJobById("1")))
.expectNextMatches(executedJob -> {
assertNotNull(executedJob);
assertEquals(JobState.COMPLETED, executedJob.getState());
assertEquals(JobStatus.SUCCESS, executedJob.getStatus());
return true;
})
.verifyComplete();
}
}

3. Job History Query Test​

@SpringBootTest
@AutoConfigureWebTestClient
public class JobControllerTest {

@Autowired
private WebTestClient webTestClient;

@Test
public void testJobHistoryQuery() {
webTestClient.get().uri(uriBuilder -> uriBuilder
.path("/api/jobs/history")
.queryParam("jobName", "TestJob")
.build())
.exchange()
.expectStatus().isOk()
.expectBodyList(JobRequest.class)
.value(jobRequests -> {
assertNotNull(jobRequests);
assertTrue(jobRequests.stream().anyMatch(jobRequest -> "TestJob".equals(jobRequest.getJobName())));
});
}
}

4. Job Status Query Test​

@SpringBootTest
@AutoConfigureWebTestClient
public class JobControllerTest {

@Autowired
private WebTestClient webTestClient;

@Test
public void testJobStatusQuery() {
webTestClient.get().uri("/api/jobs/status?jobId=1")
.exchange()
.expectStatus().isOk()
.expectBody(String.class).isEqualTo("SUCCESS");
}
}

​

Global Exception Handler​

1. Global Exception Handler Configuration​

@ControllerAdvice
public class GlobalExceptionHandler {

@ExceptionHandler(Exception.class)
public ResponseEntity<String> handleException(Exception ex) {
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("An error occurred: " + ex.getMessage());
}

@ExceptionHandler(WebClientResponseException.class)
public ResponseEntity<String> handleWebClientResponseException(WebClientResponseException ex) {
return ResponseEntity.status(ex.getStatusCode()).body("WebClient error: " + ex.getResponseBodyAsString());
}

@ExceptionHandler(R2dbcDataIntegrityViolationException.class)
public ResponseEntity<String> handleDataIntegrityViolationException(R2dbcDataIntegrityViolationException ex) {
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body("Data integrity violation: " + ex.getMessage());
}
}

​

Project Structure​

Project Structure Diagram​

spark-as-a-service/
├── src/
│ ├── main/
│ │ ├── java/
│ │ │ └── com/
│ │ │ └── example/
│ │ │ └── sparkasaservice/
│ │ │ ├── config/
│ │ │ │ ├── RedisConfig.java
│ │ │ │ ├── SparkRestApiConfig.java
│ │ │ │ ├── R2dbcConfig.java
│ │ │ │ └── SwaggerConfig.java
│ │ │ ├── controller/
│ │ │ │ └── JobController.java
│ │ │ ├── exception/
│ │ │ │ └── GlobalExceptionHandler.java
│ │ │ ├── model/
│ │ │ │ ├── JobRequest.java
│ │ │ │ ├── JobState.java
│ │ │ │ └── JobStatus.java
│ │ │ ├── repository/
│ │ │ │ └── JobRepository.java
│ │ │ ├── service/
│ │ │ │ ├── JobExecutorService.java
│ │ │ │ ├── PostgresService.java
│ │ │ │ ├── RedisQueueService.java
│ │ │ │ └── SparkRestApiService.java
│ │ │ └── SparkAsAServiceApplication.java
│ │ └── resources/
│ │ ├── application.yml
│ │ └── schema.sql
│ └── test/
│ ├── java/
│ │ └── com/
│ │ └── example/
│ │ └── sparkasaservice/
│ │ ├── controller/
│ │ │ └── JobControllerTest.java
│ │ ├── service/
│ │ │ └── JobExecutorServiceTest.java
│ │ └── SparkAsAServiceApplicationTests.java
│ └── resources/
│ └── application-test.yml
└── pom.xml

Listening to Redis Queue​

1. Redis Queue Listener Configuration​

To listen to the Redis queue and trigger the executeNextJob method, we can use Spring Data Redis's reactive support with Lettuce as the Redis client.

2. Redis Queue Listener Service​

@Service
public class RedisQueueListenerService {
private final JobExecutorService jobExecutorService;
private final ReactiveRedisTemplate<String, JobRequest> redisTemplate;

public RedisQueueListenerService(JobExecutorService jobExecutorService, ReactiveRedisTemplate<String, JobRequest> redisTemplate) {
this.jobExecutorService = jobExecutorService;
this.redisTemplate = redisTemplate;
}

@PostConstruct
public void startListening() {
redisTemplate.listenToPattern("__keyevent@*__:rpush")
.doOnNext(message -> {
if (message.getChannel().equals("__keyevent@0__:rpush") && message.getMessage().equals("jobQueue")) {
jobExecutorService.executeNextJob().subscribe();
}
})
.subscribe();
}
}

3. Redis Configuration for Pub/Sub​

@Configuration
public class RedisConfig {
@Bean
public ReactiveRedisTemplate<String, JobRequest> reactiveRedisTemplate(ReactiveRedisConnectionFactory factory) {
RedisSerializationContext<String, JobRequest> serializationContext = RedisSerializationContext
.newSerializationContext(new StringRedisSerializer())
.value(new Jackson2JsonRedisSerializer<>(JobRequest.class))
.build();
return new ReactiveRedisTemplate<>(factory, serializationContext);
}

@Bean
public ReactiveRedisConnectionFactory reactiveRedisConnectionFactory(RedisProperties redisProperties) {
return new LettuceConnectionFactory(redisProperties.getHost(), redisProperties.getPort());
}
}

This article provides a comprehensive guide to developing a stateless Spring Boot application for Spark as a Service, along with detailed explanations of job queue management, Spark job submission, and test cases. Each section is designed to be detailed and actionable, ensuring that readers can implement the concepts effectively.

Spark in Spring Boot Application

· 阅读需 11 分钟

Introduction​

This article provides a comprehensive guide to developing a Spark application with Spring Boot 3.3.5 to communicate with Kafka and Elasticsearch. The application will handle Avro records in Kafka, JSON data in Elasticsearch, and Parquet files in MinIO using Spark DataFrame and Spark SQL. The article covers the setup, integration, and advanced scenarios for handling large-scale data processing, including the use of Avro Schema Registry when communicating with Kafka.


Prerequisites​

  • Java 17 or later
  • Apache Spark 3.5.4
  • Spring Boot 3.3.5
  • Kafka 3.x
  • Elasticsearch 8.x
  • MinIO
  • Maven or Gradle
  • Confluent Schema Registry

Project Folder Structure​

spark-spring-boot-app/
├── src/
│ ├── main/
│ │ ├── java/
│ │ │ ├── com/
│ │ │ │ ├── example/
│ │ │ │ │ ├── config/
│ │ │ │ │ │ ├── SparkConfig.java
│ │ │ │ │ ├── service/
│ │ │ │ │ │ ├── KafkaService.java
│ │ │ │ │ │ ├── ElasticsearchService.java
│ │ │ │ │ │ ├── MinIOService.java
│ │ │ │ │ │ ├── DataFrameService.java
│ │ │ │ │ ├── SparkSpringBootConsoleApplication.java
│ │ ├── resources/
│ │ │ ├── application.properties
│ ├── test/
│ │ ├── java/
│ │ │ ├── com/
│ │ │ │ ├── example/
│ │ │ │ │ ├── service/
│ │ │ │ │ │ ├── KafkaServiceTest.java
│ │ │ │ │ │ ├── ElasticsearchServiceTest.java
│ │ │ │ │ │ ├── MinIOServiceTest.java
│ │ │ │ │ │ ├── DataFrameServiceTest.java
├── pom.xml (or build.gradle)

Setting Up the Spring Boot Application​

Step 1: Create a Spring Boot Project​

Create a new Spring Boot project using Spring Initializr or your preferred IDE. Include the following dependencies:

  • Spring for Apache Kafka
  • Spring Data Elasticsearch
  • Spring for Apache Spark

Step 2: Configure Application Properties​

Configure the application properties for Kafka, Elasticsearch, and MinIO:

# Kafka Configuration
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.ByteArraySerializer

# Schema Registry Configuration
schema.registry.url=http://localhost:8081

# Elasticsearch Configuration
spring.elasticsearch.rest.uris=http://localhost:9200

# MinIO Configuration
minio.endpoint=http://localhost:9000
minio.accessKey=your-access-key
minio.secretKey=your-secret-key
minio.bucket=your-bucket

Integrating Spark with Spring Boot​

Step 1: Add Spark Dependencies​

Add the following dependencies to your pom.xml or build.gradle file:

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.12</artifactId>
<version>3.5.4</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>7.0.1</version>
</dependency>

Step 2: Configure Spark​

Configure Spark in your Spring Boot application using annotation-style configuration:

import org.apache.spark.sql.SparkSession;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class SparkConfig {

@Bean
public SparkSession sparkSession() {
return SparkSession.builder()
.appName("SparkSpringBootApp")
.master("local[*]")
.config("spark.sql.shuffle.partitions", "10")
.getOrCreate();
}
}

Communicating with Kafka​

Reading Avro Records from Kafka​

To read Avro records from Kafka, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readAvroFromKafka(String topic) {
return sparkSession.read()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.load()
.selectExpr("CAST(value AS BINARY)")
.select(org.apache.spark.sql.avro.functions.from_avro(col("value"), "your_avro_schema"));
}
}

Reading Avro Records from Kafka with Schema Registry​

To read Avro records from Kafka using the Schema Registry, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

@Value("${schema.registry.url}")
private String schemaRegistryUrl;

public Dataset<Row> readAvroFromKafka(String topic) {
return sparkSession.read()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", topic)
.load()
.selectExpr("CAST(value AS BINARY)")
.select(org.apache.spark.sql.avro.functions.from_avro(col("value"), schemaRegistryUrl + "/subjects/" + topic + "-value/versions/latest"));
}
}

Writing Data to Kafka​

To write data to Kafka, use the spark-sql-kafka-0-10 library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

public void writeToKafka(Dataset<Row> dataset, String topic) {
dataset.selectExpr("to_json(struct(*)) AS value")
.write()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", topic)
.save();
}
}

Writing Avro Records to Kafka with Schema Registry​

To write Avro records to Kafka using the Schema Registry, use the spark-avro and spark-sql-kafka-0-10 libraries:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class KafkaService {

@Autowired
private SparkSession sparkSession;

@Value("${schema.registry.url}")
private String schemaRegistryUrl;

public void writeAvroToKafka(Dataset<Row> dataset, String topic) {
dataset.select(org.apache.spark.sql.avro.functions.to_avro(struct("*")).as("value"))
.write()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", topic)
.option("schema.registry.url", schemaRegistryUrl)
.save();
}
}

Communicating with Elasticsearch​

Reading JSON Data from Elasticsearch​

To read JSON data from Elasticsearch, use the elasticsearch-hadoop library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ElasticsearchService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readFromElasticsearch(String index) {
return sparkSession.read()
.format("org.elasticsearch.spark.sql")
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.resource", index)
.load();
}
}

Writing Data to Elasticsearch​

To write data to Elasticsearch, use the elasticsearch-hadoop library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ElasticsearchService {

@Autowired
private SparkSession sparkSession;

public void writeToElasticsearch(Dataset<Row> dataset, String index) {
dataset.write()
.format("org.elasticsearch.spark.sql")
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.resource", index)
.save();
}
}

Handling Parquet Files in MinIO​

Reading Parquet Files from MinIO​

To read Parquet files from MinIO, use the spark-hadoop-cloud library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class MinIOService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> readParquetFromMinIO(String bucket, String path) {
return sparkSession.read()
.format("parquet")
.option("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000")
.option("spark.hadoop.fs.s3a.access.key", "your-access-key")
.option("spark.hadoop.fs.s3a.secret.key", "your-secret-key")
.option("spark.hadoop.fs.s3a.path.style.access", "true")
.load("s3a://" + bucket + "/" + path);
}
}

Writing Parquet Files to MinIO​

To write Parquet files to MinIO, use the spark-hadoop-cloud library:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class MinIOService {

@Autowired
private SparkSession sparkSession;

public void writeParquetToMinIO(Dataset<Row> dataset, String bucket, String path) {
dataset.write()
.format("parquet")
.option("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000")
.option("spark.hadoop.fs.s3a.access.key", "your-access-key")
.option("spark.hadoop.fs.s3a.secret.key", "your-secret-key")
.option("spark.hadoop.fs.s3a.path.style.access", "true")
.save("s3a://" + bucket + "/" + path);
}
}

Using Spark DataFrame and Spark SQL​

Creating DataFrames​

To create DataFrames, use the SparkSession object:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class DataFrameService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> createDataFrame() {
return sparkSession.read()
.format("csv")
.option("header", "true")
.load("path/to/csv/file");
}
}

Querying DataFrames with Spark SQL​

To query DataFrames using Spark SQL, register the DataFrame as a temporary view:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class DataFrameService {

@Autowired
private SparkSession sparkSession;

public Dataset<Row> queryDataFrame(Dataset<Row> df) {
df.createOrReplaceTempView("my_table");
return sparkSession.sql("SELECT * FROM my_table WHERE column = 'value'");
}
}

Unit Test Cases​

KafkaServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class KafkaServiceTest {

@Autowired
private KafkaService kafkaService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadAvroFromKafka() {
Dataset<Row> dataset = kafkaService.readAvroFromKafka("test-topic");
assertNotNull(dataset);
}

@Test
public void testWriteAvroToKafka() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
kafkaService.writeAvroToKafka(dataset, "test-topic");
}
}

ElasticsearchServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class ElasticsearchServiceTest {

@Autowired
private ElasticsearchService elasticsearchService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadFromElasticsearch() {
Dataset<Row> dataset = elasticsearchService.readFromElasticsearch("test-index");
assertNotNull(dataset);
}

@Test
public void testWriteToElasticsearch() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
elasticsearchService.writeToElasticsearch(dataset, "test-index");
}
}

MinIOServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class MinIOServiceTest {

@Autowired
private MinIOService minIOService;

@Autowired
private SparkSession sparkSession;

@Test
public void testReadParquetFromMinIO() {
Dataset<Row> dataset = minIOService.readParquetFromMinIO("test-bucket", "test-path");
assertNotNull(dataset);
}

@Test
public void testWriteParquetToMinIO() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
minIOService.writeParquetToMinIO(dataset, "test-bucket", "test-path");
}
}

DataFrameServiceTest.java​

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

@SpringBootTest
public class DataFrameServiceTest {

@Autowired
private DataFrameService dataFrameService;

@Autowired
private SparkSession sparkSession;

@Test
public void testCreateDataFrame() {
Dataset<Row> dataset = dataFrameService.createDataFrame();
assertNotNull(dataset);
}

@Test
public void testQueryDataFrame() {
Dataset<Row> dataset = sparkSession.read().json("path/to/json/file");
Dataset<Row> result = dataFrameService.queryDataFrame(dataset);
assertNotNull(result);
}
}

Design Diagrams​

Component Diagram​

+-------------------+ +-------------------+ +-------------------+
| | | | | |
| Spring Boot | | Apache Spark | | Kafka |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| | |
v v v
+-------------------+ +-------------------+ +-------------------+
| | | | | |
| KafkaService | | Elasticsearch | | MinIOService |
| | | | | |
+--------+----------+ +--------+----------+ +--------+----------+
| | |
| | |
v v v
+--------------------+ +-------------------+ +---------------------------------------+
| | | | | |
| DataFrameService | | SparkConfig | | SparkSpringBootConsoleApplication |
| | | | | |
+--------------------+ +-------------------+ +---------------------------------------+

Sequence Diagram​

+---------+ +---------+ +---------+ +---------+
| | | | | | | |
| Client | | Spring | | Spark | | Kafka |
| | | Boot | | | | |
+---------+ +---------+ +---------+ +---------+
| | | |
| Request | | |
|----------------->| | |
| | | |
| | Read Avro | |
| | from Kafka | |
| |-------------->| |
| | | |
| | | Send Avro |
| | |<------------->|
| | | |
| | Process | |
| | Data | |
| |-------------->| |
| | | |
| | Write to | |
| | Elasticsearch| |
| |-------------->| |
| | | |
| | Write to | |
| | MinIO | |
| |-------------->| |
| | | |
| Response | | |
|<-----------------| | |
| | | |

Advanced Scenarios​

Complex Scenario 1: Flow Control and Backpressure Handling​

In scenarios where your application needs to handle flow control and backpressure, you can use Spark's built-in mechanisms for handling large-scale data processing.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class FlowControlService {

@Autowired
private SparkSession sparkSession;

public void handleFlowControl(Dataset<Row> dataset) {
dataset.writeStream()
.format("console")
.option("checkpointLocation", "path/to/checkpoint")
.start()
.awaitTermination();
}
}

Complex Scenario 2: Handling Large-Scale Data Processing​

For large-scale data processing, use Spark's distributed processing capabilities and optimize resource usage.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class LargeScaleProcessingService {

@Autowired
private SparkSession sparkSession;

public void processLargeScaleData(Dataset<Row> dataset) {
dataset.repartition(100)
.write()
.format("parquet")
.save("path/to/output");
}
}

Conclusion​

Developing a Spark application with Spring Boot 3.3.5 to communicate with Kafka and Elasticsearch involves setting up the Spring Boot application, integrating Spark, and handling data using Spark DataFrame and Spark SQL. By following the best practices outlined in this article, you can create efficient and scalable data processing applications.


References​

  1. Spring Boot Documentation
  2. Apache Spark Documentation
  3. Apache Kafka Documentation
  4. Elasticsearch Documentation
  5. MinIO Documentation
  6. Confluent Schema Registry Documentation

By following these best practices, you can leverage Spark and Spring Boot to build powerful and scalable data processing applications.

Spark in Kubernetes

· 阅读需 7 分钟

This guide covers the steps to deploy an Apache Spark cluster with a master, two workers, and a Spark History Server on Kubernetes. It configures a GCS bucket as the default file system and installs Hive with PostgreSQL as its backend. An Ingress is set up for accessing the Spark UI and History Server.

Architecture Overview​

The setup includes the following components:

  • Spark Master and Workers: Deployed using StatefulSets for stable identity and persistence.
  • History Server: Deployed using a Deployment and configured to store Spark event logs in GCS.
  • Hive and PostgreSQL: PostgreSQL is used as the backend for Hive metadata, and Hive is deployed to store Spark tables.
  • Ingress: Provides external access to Spark Master UI and History Server UI.

Diagram: Architecture Overview​

+---------------------------+
| |
| Kubernetes |
| |
+---------------------------+
| |
| +---------+ +---------+|
| | Spark | | Spark ||
| | Master | | Workers ||
| +---------+ +---------+|
| |
| +-----------+ |
| | History | |
| | Server | |
| +-----------+ |
| | |
| Ingress |
+---------------------------+
|
External Access

Prerequisites​

  • Kubernetes Cluster with access to deploy resources.
  • kubectl command-line tool configured to access the cluster.
  • Helm for deploying Hive and PostgreSQL.
  • Google Cloud Storage (GCS) bucket created for Spark event logs.
  • Google Cloud SDK installed and configured for authentication to GCS.

1. Configuring Google Cloud Storage (GCS) Bucket​

  1. Create a GCS bucket if not already created:

    gsutil mb gs://your-spark-event-logs
  2. Set up a GCS service account key and save it as a Kubernetes secret. This will allow Spark to access GCS:

    kubectl create secret generic gcs-key \
    --from-file=key.json=/path/to/gcs-service-account-key.json

2. Deploying PostgreSQL for Hive Metadata​

PostgreSQL will act as the relational database for Hive’s metadata.

PostgreSQL YAML​

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgresql-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgresql
spec:
replicas: 1
selector:
matchLabels:
app: postgresql
template:
metadata:
labels:
app: postgresql
spec:
containers:
- name: postgresql
image: postgres:13
env:
- name: POSTGRES_DB
value: "hive_metastore"
- name: POSTGRES_USER
value: "hiveuser"
- name: POSTGRES_PASSWORD
value: "hivepassword"
volumeMounts:
- mountPath: "/var/lib/postgresql/data"
name: postgresql-data
volumes:
- name: postgresql-data
persistentVolumeClaim:
claimName: postgresql-pvc

3. Installing Hive Metastore​

Hive Metastore will manage metadata for Spark.

Hive Configuration​

  1. Download and install Hive Helm chart:

    helm repo add bitnami https://charts.bitnami.com/bitnami
    helm install hive bitnami/hive --set postgresql.enabled=false
  2. Update Hive configuration to use PostgreSQL.

# Values.yaml for Hive
hive:
metastore:
databaseType: postgres
jdbcUrl: jdbc:postgresql://postgresql-service/hive_metastore
user: hiveuser
password: hivepassword

4. Deploying Spark Master and Workers​

Spark Master and Worker StatefulSet YAML​

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: spark-master
spec:
serviceName: spark-master
replicas: 1
selector:
matchLabels:
app: spark
template:
metadata:
labels:
app: spark
role: master
spec:
containers:
- name: spark-master
image: spark:latest
ports:
- containerPort: 7077
- containerPort: 8080
env:
- name: SPARK_MASTER_HOST
value: "spark-master"

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: spark-worker
spec:
replicas: 2
selector:
matchLabels:
app: spark
role: worker
template:
metadata:
labels:
app: spark
role: worker
spec:
containers:
- name: spark-worker
image: spark:latest
ports:
- containerPort: 8081
env:
- name: SPARK_WORKER_CORES
value: "1"
- name: SPARK_WORKER_MEMORY
value: "1G"
- name: SPARK_MASTER
value: "spark://spark-master:7077"

5. Setting Up the Spark History Server​

The History Server will read event logs from the GCS bucket for job history.

Spark History Server YAML​

apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-history-server
spec:
replicas: 1
selector:
matchLabels:
app: spark-history
template:
metadata:
labels:
app: spark-history
spec:
containers:
- name: spark-history-server
image: spark:latest
env:
- name: SPARK_HISTORY_OPTS
value: "-Dspark.history.fs.logDirectory=gs://your-spark-event-logs"
volumeMounts:
- mountPath: /opt/spark/work-dir
name: gcs-key
subPath: key.json
volumes:
- name: gcs-key
secret:
secretName: gcs-key

6. Configuring Ingress​

Ingress will expose the Spark Master UI and History Server UI.

Ingress YAML​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: spark-ingress
spec:
rules:
- host: spark.example.com
http:
paths:
- path: /master
pathType: Prefix
backend:
service:
name: spark-master
port:
number: 8080
- path: /history
pathType: Prefix
backend:
service:
name: spark-history-server
port:
number: 18080

7. Testing the Deployment with Spark Job​

To verify the installation, use spark-submit to run a simple calculation for PI.

  1. Submit a PI calculation job using spark-submit.

    kubectl exec -it spark-master-0 -- spark-submit \
    --class org.apache.spark.examples.SparkPi \
    --master spark://spark-master:7077 \
    /opt/spark/examples/jars/spark-examples_2.12-3.0.1.jar 1000
  2. Check the Spark Master and History Server UIs

to confirm job execution and view logs.

Furthermore​

StatefulSet vs Deployment​

In the official Helm chart for Apache Spark, the choice of StatefulSet for Spark Master and Worker, and Deployment for the Spark History Server, aligns with the specific requirements and behaviors of each component. Here’s a breakdown of the reasons for these design choices:

1. Spark Master and Worker as StatefulSets​

  • Persistence and Stability: The Spark Master and Worker nodes need stable, unique identities to communicate reliably. Using StatefulSet ensures that each replica (instance) has a stable network identity (hostname) and, if configured, persistent storage.
  • Leader Election and Failover: Spark Master in standalone mode has a primary leader and backup node architecture, where the leader is responsible for the cluster state. StatefulSets allow these nodes to maintain consistent, unique identifiers, which is crucial for leader election and failover without disrupting the Spark cluster.
  • Worker Registration: Spark Workers register with the Master, and having a unique, stable network identity ensures workers don’t inadvertently re-register or conflict with each other after restarts or failures. StatefulSets are the preferred way to handle this kind of behavior.

2. Spark History Server as a Deployment​

  • Stateless Operation: The Spark History Server is fundamentally a stateless application. It reads from a pre-configured storage (such as HDFS or S3) where application logs are stored but doesn’t require persistent storage for its own operation. Using a Deployment is appropriate here, as the history server can be easily replicated or scaled without maintaining unique identities.
  • Easier Scaling and Update Management: Deployments provide automatic scaling and rolling updates, which are more suitable for stateless services. If you need to increase the availability of the history server, you can simply increase the replica count in the Deployment, and Kubernetes will handle the scaling automatically.
  • Failure Recovery: Since there’s no dependency on unique network identities or persistent state, if the History Server pod fails, it can be easily recreated without any additional orchestration, which is exactly what Deployments are optimized for.

Summary Table​

ComponentKubernetes KindPurposeReasoning
Spark MasterStatefulSetPrimary and backup master for Spark clusterNeeds stable identity for cluster coordination and leader election
Spark WorkerStatefulSetWorker nodes that register with Spark MasterStable network identity ensures reliable registration and communication with Master
Spark History ServerDeploymentReads logs and provides historical application dataStateless and doesn’t need unique identities; easier to scale and restart as a Deployment

This setup optimizes the deployment structure for each Spark component's unique requirements, leveraging Kubernetes' StatefulSet and Deployment resources for their intended purposes.