Skip to main content

35 posts tagged with "deep-dive"

View All Tags

Deep Dive into Core Kubernetes Components

· 3 min read

This article dives deep into the core components of Kubernetes, explaining their roles, functions, and interconnections. Textual diagrams illustrate the architecture to offer a clearer understanding of each component's interactions.


API Server​

The API Server is the core entry point for all Kubernetes interactions. It exposes the Kubernetes API, validates requests, and interacts with other components. Key functionalities include:

  • Authentication: Verifies the identity of incoming requests.
  • Authorization: Manages RBAC (Role-Based Access Control) policies.
  • Admission Control: Applies constraints and transformations to objects at request time.
  • API Requests: Handles CRUD (Create, Read, Update, Delete) requests on Kubernetes objects.

Diagram:

+----------------+
| |
| API Server |
| |
+----------------+
|
v
+----------------+
| |
| etcd |
| |
+----------------+

etcd​

etcd is a highly available key-value store used by Kubernetes to persist the cluster state. It maintains the configurations and states of each component and is essential for recovering cluster state in case of a restart. Each component retrieves cluster state through the API Server, which interfaces with etcd.

Diagram:

+----------------+ +----------------+
| | | |
| API Server | <-------> | etcd |
| | | |
+----------------+ +----------------+

Controller Manager​

The Controller Manager includes various controllers (e.g., Node, Replication, Endpoint) that regulate the state of cluster resources. Each controller reconciles the cluster's actual state with the desired state. When discrepancies arise, controllers initiate adjustments (e.g., scaling replicas).

Diagram:

+----------------+
| |
| Controller |
| Manager |
| |
+----------------+
|
v
+----------------+
| |
| API Server |
| |
+----------------+

Scheduler​

The Scheduler assigns Pods to Nodes based on resource needs and constraints. The Scheduler evaluates factors like CPU, memory, and affinity rules before assigning Pods, ensuring optimal workload distribution.

Diagram:

+----------------+
| |
| Scheduler |
| |
+----------------+
|
v
+----------------+
| |
| API Server |
| |
+----------------+

Kubelet​

The Kubelet runs on each Node and is responsible for managing container instances. It communicates with the API Server to receive Pod specifications, ensuring they’re running as defined. It monitors containers, collects resource usage, and sends reports back to the API Server.

Diagram:

+----------------+ +----------------+
| | | |
| Kubelet | <-> | API Server |
| | | |
+----------------+ +----------------+

Kube Proxy​

Kube Proxy manages network communication across Nodes in the cluster, implementing Kubernetes networking policies. It facilitates communication within Pods by maintaining network rules and forwarding requests as needed.

Diagram:

+----------------+ +----------------+
| | | |
| Kube Proxy | <-> | Kubelet |
| | | |
+----------------+ +----------------+

Communication Flow​

In Kubernetes, components communicate through a centralized and secure API Server, with etcd acting as the persistent store. The workflow for scheduling, updating, and scaling applications includes:

  1. User Command: Users issue requests to the API Server.
  2. Scheduler: Schedules new Pods.
  3. Controller Manager: Enforces and monitors the cluster state.
  4. Kubelet and Proxy: Enforce changes and maintain container and network policies.

Overall Diagram:

+----------------+ +----------------+
| | | |
| API Server | <--------> | etcd |
| | | |
+----------------+ +----------------+
| |
| |
v v
+----------------+ +----------------+
| | | |
| Controller | | Scheduler |
| Manager | | |
+----------------+ +----------------+
| |
v v
+----------------+ +----------------+
| | | |
| Kubelet | | Kube Proxy |
| | | |
+----------------+ +----------------+

Each Kubernetes component plays a specific role in cluster operations, and their communication ensures the cluster maintains its desired state and remains responsive to changing demands.

Deep Dive into Kubernetes Ingress

· 3 min read

In Kubernetes, Ingress is a crucial component that manages external access to services within a cluster, typically via HTTP or HTTPS. It provides a centralized entry point for external traffic, applying routing rules to direct requests to appropriate services. This article explores the architecture and responsibilities of Kubernetes Ingress, how it interacts with other components, and provides visual architecture diagrams.


What is Kubernetes Ingress?​

Ingress in Kubernetes is an API object that manages external access to services, usually HTTP. Instead of exposing each service individually through a Service object of type LoadBalancer or NodePort, Ingress offers a central route, managing traffic with a single IP address or DNS name for the entire cluster. Key benefits include:

  • Centralized Traffic Management: Ingress consolidates traffic through a single point of entry, allowing easy route management.
  • Path-Based Routing: Requests can be directed to specific services based on request paths or hostnames.
  • SSL Termination: Ingress can terminate SSL/TLS connections, ensuring secure HTTPS access.

Ingress Types​

Kubernetes supports two main Ingress types:

  • Simple Ingress: For straightforward routing to backend services based on path or hostname.
  • Advanced Ingress: With more complex routing rules, integrating SSL termination, URL rewrites, and custom filtering.

Ingress Controllers​

An Ingress Controller is the implementation that enforces Ingress rules, usually deployed as a Pod within the cluster. Popular Ingress controllers include NGINX, HAProxy, Traefik, and Istio. The Ingress Controller is responsible for:

  1. Monitoring Ingress Resources: Watching for any changes in Ingress resources and updating routing rules accordingly.
  2. Load Balancing and SSL Termination: Providing efficient load balancing for HTTP/HTTPS requests.
  3. Proxying and Forwarding: Accepting requests on behalf of the Ingress and forwarding them to the correct Service and Pod.

The Ingress Controller must be installed in the cluster to interpret and apply Ingress resources, as Kubernetes does not ship with a default controller.


Ingress Rules and Annotations​

Ingress rules define the routing policy for external traffic. These rules can direct traffic based on URL paths or hostnames, and each rule specifies the Service and Port the request should route to. Annotations add configurations like timeouts, request limits, and additional routing options.

Example of an Ingress Rule​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: example.com
http:
paths:
- path: /app
pathType: Prefix
backend:
service:
name: app-service
port:
number: 80

In this example, requests to example.com/app are routed to the app-service on port 80.


Ingress Architecture Diagram​

Ingress with Path-Based Routing​

+-----------------------------+
| External User |
+-----------------------------+
|
v
+-----------------------------+
| Ingress Controller | <-- Manages Ingress Rules
+-----------------------------+
|
+-------+------+--------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Service A | | Service B | | Service C |
+-----------+ +-----------+ +-----------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Pod A1 | | Pod B1 | | Pod C1 |
+-----------+ +-----------+ +-----------+

In this diagram:

  • Ingress Controller routes based on rules that direct traffic to Services.
  • Services are abstractions that select Pods through labels.
  • Pods run the actual applications and receive traffic forwarded by the Ingress Controller.

This setup allows centralized and managed external access to cluster resources.


By centralizing access with Ingress, Kubernetes can handle complex routing configurations, ensuring that traffic flows efficiently and securely across the cluster.

Deep Dive into Kubernetes Pod and Service Interactions

· 3 min read

In Kubernetes, Pods and Services work together to ensure reliable deployment, scaling, and network communication for containerized applications. This article provides a deep dive into their responsibilities, roles, and interaction processes within the Kubernetes architecture.

Understanding Kubernetes Pods​

A Pod is the smallest deployable unit in Kubernetes. Each Pod represents one or more containers and is typically designed to hold a single application instance. Key responsibilities and characteristics of Pods include:

  • Containerized Workloads: A Pod holds one or more containers that share storage, network, and specifications.
  • Networking: Each Pod gets its own IP address, allowing containers within it to communicate over localhost.
  • Ephemeral Nature: Pods are disposable. When they fail or are deleted, a new Pod with a different IP is often created to replace them.
  • Scalability: Kubernetes replicates Pods based on defined scaling requirements to handle application load.

Pod Lifecycle:

  1. Scheduled: The Scheduler places the Pod on an appropriate Node.
  2. Running: Containers within the Pod are executing.
  3. Terminated: When the Pod is completed or deleted.

Understanding Kubernetes Services​

A Service in Kubernetes is a stable networking endpoint that exposes a set of Pods as a network-accessible service, typically grouped by labels. Services abstract the dynamic nature of Pods and allow continuous network access. The main responsibilities of Services include:

  • Stable Networking: Services maintain a consistent IP address and DNS name despite the ephemeral nature of Pods.
  • Load Balancing: Services automatically distribute incoming traffic across available Pods, balancing load.
  • Discovery: Pods can be dynamically assigned to a Service based on matching labels, enabling flexible scaling and updates.

Types of Services:

  1. ClusterIP (default): Exposes a service within the cluster.
  2. NodePort: Exposes a service on each Node’s IP at a static port.
  3. LoadBalancer: Creates an external load balancer to expose the service externally.
  4. ExternalName: Maps the Service to a DNS name outside of the cluster.

Pod-Service Communication​

Kubernetes leverages label selectors and IP allocation to connect Pods with Services. Here’s how communication works:

  1. Label Selection: Services use labels to select Pods for traffic routing.
  2. Endpoints Object: Kubernetes creates an Endpoint resource to track the IP addresses of Pods associated with the Service.
  3. Service Proxying: kube-proxy intercepts traffic to a Service IP and forwards it to one of the Pod IPs listed in the Endpoints.
  4. Service IP and DNS: Each Service has a unique IP and is discoverable via a DNS name, facilitating easy communication.

Architecture Diagrams​

Pod Structure​

+----------------------------------+
| Pod |
| +-----------+ +-----------+ |
| | Container | | Container | |
| | App | | Sidecar | |
| +-----------+ +-----------+ |
| Network (localhost) |
| Shared Storage |
+----------------------------------+

Service Structure and Flow​

+-----------------+ +------------------+
| Service IP | <------------>| kube-proxy |
+-----------------+ +------------------+
| |
v v
+-----------------+ +------------------+ +------------------+
| Pod 1 | <-----> | Pod 2 | <- | Pod 3 |
+-----------------+ +------------------+ +------------------+

In summary, Kubernetes Pods run containerized workloads, while Services provide stable network access to those workloads, regardless of individual Pod lifetimes. Together, they enable dynamic scaling, fault tolerance, and efficient networking within Kubernetes clusters.

Microservice - Spring Cloud Gateway

· 16 min read
  1. Introduction
  2. Why We Need API Gateway
  3. Build API Gateway
  4. Setting Up API Gateway
  5. Route Configuration
  6. Security and JWT Authentication
  7. Dockerfile Configuration
  8. Helm Chart Configuration
  9. Project Structure
  10. API Gateway Architecture
  11. Sequence Diagrams
  12. HTTPS Configuration for API Gateway
  13. Difference Between lb:// and http:// Protocols
  14. Conclusion

Introduction​

In a microservices architecture, an API Gateway acts as a single entry point for client requests, routing them to the appropriate microservices within the Kubernetes cluster. This article will delve into the setup and configuration of a Spring Cloud API Gateway, focusing on JWT authentication validation, route management, and deployment using Docker and Helm.


Why We Need API Gateway​

Single Entry Point​

An API Gateway provides a single entry point for all client requests, simplifying client-side code and reducing the number of endpoints clients need to interact with.

Routing and Load Balancing​

The gateway routes requests to the appropriate microservices and handles load balancing across multiple instances of each service.

Security​

It centralizes security concerns such as authentication and authorization, reducing the need for each microservice to implement these features independently.

Protocol Translation​

The gateway can handle protocol translation, allowing clients to use a single protocol (e.g., HTTP/2) while communicating with microservices using different protocols.


Build API Gateway​

Is there an official API Gateway image?

  • Currently, Spring Cloud Gateway does not offer an official pre-built Docker image, developers build a Spring Boot application with spring-cloud-starter-gateway and custom configurations (routes, JWT filter, etc.).

  • As gateway services are often customized for specific routing, load balancing, security and potentially other cross-cutting concerns and needs. It acts as the main entry point for all external requests to your microservices. The gateway can either pull routing configurations from a service discovery mechanism(like Spring Cloud Kubernetes Discovery) or have the configurations set explicitly.

Steps to Build a Production-Ready API Gateway

  1. Route Configuration: Define routing rules either in application.yml or using Java annotations.
  2. Security Setup: Add JWT validation (e.g., JwtAuthenticationFilter) and configure security policies in the gateway.
  3. Service Discovery: Use Spring Cloud Kubernetes for service discovery, allowing the gateway to dynamically discover services.
  4. Containerization and Deployment: Build a Docker image and deploy it to Kubernetes with a Helm chart for scalability and configuration management.

Setting Up API Gateway​

Step 1: Add Dependencies​

Add the necessary dependencies to your pom.xml file.

<dependencies>
<!-- Spring Boot Starter -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>

<!-- Spring Cloud Gateway -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>

<!-- Spring Security -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>

<!-- JWT -->
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt</artifactId>
<version>0.9.1</version>
</dependency>
</dependencies>

Step 2: Create the Main Application Class​

Create the main application class for the Spring Cloud Gateway.

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ApiGatewayApplication {
public static void main(String[] args) {
SpringApplication.run(ApiGatewayApplication.class, args);
}
}

Service Discovery and Routing Configuration​

YAML-based Configuration​

Configure routes in the application.yml file.

spring:
cloud:
gateway:
discovery:
locator:
enabled: true
lower-case-service-id: true # Ensures lowercase naming for services
routes:
- id: profile_service
uri: lb://profile-svc
predicates:
- Path=/profile/**
- id: inventory_service
uri: lb://inventory-svc
predicates:
- Path=/inventory/**
- id: order_service
uri: lb://order-svc
predicates:
- Path=/order/**
- id: shipment_service
uri: lb://shipment-svc
predicates:
- Path=/shipment/**

kubernetes:
discovery:
all-namespaces: true # Discover services across all namespaces
service-labels: # Optional: filter based on labels
environment: production
  • lb:// */ scheme: Enables load balancing across multiple instances of each microservice.
  • Service Discovery: The gateway will detect microservices like inventory-service or shipment-service using Kubernetes service discovery, eliminating the need for hardcoding service addresses.

Annotation-based Configuration​

Alternatively, configure routes programmatically using annotations.

import org.springframework.cloud.gateway.route.RouteLocator;
import org.springframework.cloud.gateway.route.builder.RouteLocatorBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class GatewayRoutesConfig {

@Bean
public RouteLocator gatewayRoutes(RouteLocatorBuilder builder) {
return builder.routes()
.route("profile_service_route", r -> r.path("/profile/**")
.uri("lb://profile-svc"))
.route("inventory_service_route", r -> r.path("/inventory/**")
.uri("lb://inventory-svc"))
.route("order_service_route", r -> r.path("/order/**")
.uri("lb://order-svc"))
.route("shipment_service_route", r -> r.path("/shipment/**")
.uri("lb://shipment-svc"))
.build();
}
}

Enable Service Discovery To enable service discovery, you can use the @EnableDiscoveryClient annotation. This annotation is typically placed in a configuration class.

import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableDiscoveryClient
public class DiscoveryConfig {
// Additional configuration for service discovery can be added here if needed
}

application.yml Configuration for Service Discovery​

Ensure that the following configuration is present in your application.yml or application.properties file to enable service discovery and other settings:

spring:
cloud:
gateway:
discovery:
locator:
enabled: true
lower-case-service-id: true
kubernetes:
discovery:
all-namespaces: true
service-labels:
environment: production

Summary​

  • Annotation-based Configuration: The GatewayRoutesConfig class defines the routes using the RouteLocatorBuilder.
  • Service Discovery: The DiscoveryConfig class enables service discovery using the @EnableDiscoveryClient annotation.
  • YAML Configuration: The application.yml file contains the necessary settings for service discovery and gateway configuration.

Note: The discovery settings, such as enabling discovery locator and configuring Kubernetes discovery, cannot be directly configured using annotations in the configuration class. These settings are typically configured in the application.yml or application.properties file.


Security and JWT Authentication​

Add Dependencies Include spring-boot-starter-oauth2-resource-server and spring-cloud-starter-gateway in your pom.xml:

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
</dependencies>

Gateway Application Properties Configuration: Configure application.yml for JWT validation. This includes specifying the JWT issuer URI and enabling authorization.

spring:
cloud:
gateway:
discovery:
locator:
enabled: true
routes:
- id: inventory-service
uri: lb://inventory-service
predicates:
- Path=/inventory/**
filters:
- TokenRelay # Forward the token to the downstream services
security:
oauth2:
resourceserver:
jwt:
issuer-uri: https://<your-auth-provider>/realms/your-realm
  • TokenRelay Filter: Passes the JWT token along with requests to the backend services.
  • Issuer URI: This points to your authentication provider, e.g., an OpenID Connect provider like Keycloak or Auth0.

Enable JWT Authentication: Set up SecurityConfig to enable JWT authentication and configure routes for authorized access.

@Configuration
@EnableWebFluxSecurity
public class SecurityConfig {

@Bean
public SecurityWebFilterChain securityWebFilterChain(ServerHttpSecurity http) {
http
.authorizeExchange()
.pathMatchers("/inventory/**", "/shipment/**").authenticated() // Require auth for specified paths
.anyExchange().permitAll()
.and()
.oauth2ResourceServer()
.jwt(); // Enable JWT authentication

return http.build();
}
}
  • This configuration restricts access to /inventory/** and /shipment/** endpoints to authenticated users, while allowing others as specified.

Annotation-based Configuration for Routes and Filters

import org.springframework.cloud.gateway.route.RouteLocator;
import org.springframework.cloud.gateway.route.builder.RouteLocatorBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.web.server.ServerHttpSecurity;
import org.springframework.security.web.server.SecurityWebFilterChain;

@Configuration
public class GatewayRoutesConfig {

@Bean
public RouteLocator gatewayRoutes(RouteLocatorBuilder builder) {
return builder.routes()
.route("inventory-service", r -> r.path("/inventory/**")
.filters(f -> f.filter(new JwtAuthenticationFilter("ROLE_USER,ROLE_ADMIN")))
.uri("lb://inventory-service"))
.build();
}

@Bean
public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
http
.authorizeExchange()
.anyExchange().authenticated()
.and()
.oauth2ResourceServer()
.jwt()
.jwkSetUri("https://example-issuer.com/.well-known/jwks.json")
.and()
.and()
.csrf().disable();
return http.build();
}
}

>**Note**: **Annotation-based Configuration**: The `GatewayRoutesConfig` class defines the routes and filters using the `RouteLocatorBuilder`. It also configures security settings using Spring Security annotations.

JWT Authentication Filter​

Create a JWT authentication filter to verify incoming requests.

import org.springframework.security.authentication.AuthenticationManager;
import org.springframework.security.web.server.authentication.AuthenticationWebFilter;
import org.springframework.security.oauth2.jwt.JwtDecoder;
import org.springframework.security.oauth2.jwt.ReactiveJwtDecoder;

@Configuration
public class SecurityConfig {

@Autowired
private JwtDecoder jwtDecoder;

@Bean
public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
AuthenticationWebFilter jwtAuthFilter = new AuthenticationWebFilter(authenticationManager());
http
.authorizeExchange()
.pathMatchers("/oauth2/**").permitAll()
.pathMatchers("/inventory/**", "/shipment/**").authenticated().permitAll()
.anyExchange().authenticated()
.and().addFilterAt(jwtAuthFilter, SecurityWebFiltersOrder.AUTHENTICATION);

return http.build();
}

@Bean
public AuthenticationManager authenticationManager() {
return new JwtAuthenticationManager(jwtDecoder);
}
}

JWT Decoder Configuration​

Configure the JWT decoder to validate tokens.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.oauth2.jwt.JwtDecoder;
import org.springframework.security.oauth2.jwt.NimbusJwtDecoder;

@Configuration
public class JwtConfig {

@Bean
public JwtDecoder jwtDecoder() {
return NimbusJwtDecoder.withPublicKey(publicKey).build();
}
}

Service-to-Service Communication via Gateway​

To communicate between services, such as shipment-service needing to communicate with inventory-service, microservices can send requests through the API Gateway.

This method simplifies routing, as services only need to know the gateway address and the service paths (e.g., /inventory/**). This approach enhances security and centralizes cross-cutting concerns like authentication, logging, and rate limiting.

Benefits of Gateway-based Communication​

  1. Centralized Security: JWT and other auth mechanisms can be handled at the gateway.
  2. Simplified Routing: Each service interacts with others via a single entry point.
  3. Enhanced Scalability: Kubernetes manages load balancing for each service, and the gateway balances requests across pods.

Service Discovery and Routing Workflow​

[External Client] --> [API Gateway]
|
| (Discovers routes from Kubernetes Discovery)
|
[inventory-service] <--+-- "lb://inventory-service"
[shipment-service] <--+-- "lb://shipment-service"
[other-service] <--+-- Discovered automatically

Sequence Diagram: Inventory and Shipment Communication via Gateway

  1. Shipment Service sends a request to API Gateway at /inventory/items.
  2. API Gateway validates the JWT token, looks up inventory-service route, and forwards the request.
  3. Inventory Service responds with the requested data.
  4. API Gateway relays the response to Shipment Service.
Shipment Service --> API Gateway: Request /inventory/items with JWT
API Gateway --> inventory-service: Validate JWT and route to inventory
inventory-service --> API Gateway: Return data
API Gateway --> Shipment Service: Relay data response

Example: Sending an Authenticated Request to Another Microservice via the Gateway

In shipment-service, use Spring's WebClient to call the inventory-service through the API Gateway.

  1. Define WebClient Bean for Consistency Across Calls

    @Bean
    public WebClient.Builder webClientBuilder() {
    return WebClient.builder();
    }
  2. Send Authenticated Request through Gateway

    @Autowired
    private WebClient.Builder webClientBuilder;

    public InventoryResponse fetchInventoryItems(String token) {
    return webClientBuilder
    .build()
    .get()
    .uri("http://api-gateway/inventory/items")
    .header(HttpHeaders.AUTHORIZATION, "Bearer " + token)
    .retrieve()
    .bodyToMono(InventoryResponse.class)
    .block();
    }
    • Token Forwarding: The token is forwarded with the request, allowing the gateway to validate it before routing to the target service.
    • Gateway URI: Requests go through http://api-gateway/inventory/items, routing based on service discovery.

In this example:

  • The gateway is deployed separately in Kubernetes and handles dynamic service discovery and routing via lb:// URIs.
  • Service-to-service calls leverage the API Gateway for ease of scaling, simplified routing, and centralized security. Each service only needs to know the API Gateway’s address and service paths.

Comparison: Gateway vs. Direct Service-to-Service Calls​

AspectAPI Gateway RoutingDirect Service-to-Service Calls
RoutingCentralized, via API GatewayDirect, each service must know other services
JWT ValidationCentralized in GatewayEach service needs to validate tokens
ScalingGateway handles scaling and balancingEach service scales independently
SecurityJWT validated once at the gatewayJWT validation needed in each service
Use CaseBest for external and internal callsIdeal for internal-only, tightly integrated

Dockerfile Configuration​

Create a Dockerfile for building the API Gateway image.

# Start with the base image for Java
FROM openjdk:17-jdk-slim

# Set working directory
WORKDIR /app

# Copy the API Gateway JAR to the image
COPY target/api-gateway.jar api-gateway.jar

# Expose port 8443 for the API Gateway
EXPOSE 8443

# Run the API Gateway application
ENTRYPOINT ["java", "-jar", "api-gateway.jar"]

Build and Push the Docker Image​

docker build -t myregistry/api-gateway:latest .
docker push myregistry/api-gateway:latest

Helm Chart Configuration​

Helm Chart Structure​

api-gateway/
├── Chart.yaml
├── values.yaml
└── templates/
├── deployment.yaml
├── service.yaml
└── ingress.yaml

Chart.yaml​

apiVersion: v2
name: api-gateway
version: 0.1.0
description: API Gateway for Microservices

values.yaml​

image:
repository: myregistry/api-gateway
tag: latest
pullPolicy: IfNotPresent

replicaCount: 2

service:
type: ClusterIP
port: 8443

ingress:
enabled: true
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
hosts:
- host: api.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: api-gateway-tls
hosts:
- api.example.com

templates/deployment.yaml​

apiVersion: apps/v1
kind: Deployment
metadata:
name: api-gateway
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: api-gateway
template:
metadata:
labels:
app: api-gateway
spec:
containers:
- name: api-gateway
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports:
- containerPort: 8443

templates/service.yaml​

apiVersion: v1
kind: Service
metadata:
name: api-gateway
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: 8443
selector:
app: api-gateway

templates/ingress.yaml​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-gateway-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- api.example.com
secretName: api-gateway-tls
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-gateway
port:
number: 8443

Project Structure​

api-gateway/
├── src/
│ ├── main/
│ │ ├── java/
│ │ │ └── com/
│ │ │ └── example/
│ │ │ └── apigateway/
│ │ │ ├── ApiGatewayApplication.java
│ │ │ ├── config/
│ │ │ │ ├── GatewayRoutesConfig.java
│ │ │ │ ├── SecurityConfig.java
│ │ │ │ └── JwtConfig.java
│ │ └── resources/
│ │ └── application.yml
├── Dockerfile
├── pom.xml
└── helm/
└── api-gateway/
├── Chart.yaml
├── values.yaml
└── templates/
├── deployment.yaml
├── service.yaml
└── ingress.yaml

API Gateway Architecture​

graph TD
A[Client] -->|HTTPS| B[API Gateway]
B -->|lb://| C[Profile Service]
B -->|lb://| D[Inventory Service]
B -->|lb://| E[Order Service]
B -->|lb://| F[Shipment Service]
B -->|lb://| G[OAuth2 Service]

Explanation​

  1. Client: Sends requests to the API Gateway over HTTPS.
  2. API Gateway: Routes requests to the appropriate microservice using lb:// protocol.
  3. Microservices: Each service handles requests and returns responses to the API Gateway.

Sequence Diagrams​

Sequence Diagram for Client Request​

sequenceDiagram
participant Client
participant API Gateway
participant Profile Service
participant Inventory Service
participant Order Service
participant Shipment Service

Client->>API Gateway: HTTPS Request
API Gateway->>Profile Service: lb://profile-svc
API Gateway->>Inventory Service: lb://inventory-svc
API Gateway->>Order Service: lb://order-svc
API Gateway->>Shipment Service: lb://shipment-svc
Profile Service-->>API Gateway: Response
Inventory Service-->>API Gateway: Response
Order Service-->>API Gateway: Response
Shipment Service-->>API Gateway: Response
API Gateway-->>Client: HTTPS Response

Sequence Diagram for JWT Authentication​

sequenceDiagram
participant Client
participant API Gateway
participant OAuth2 Service

Client->>API Gateway: HTTPS Request with JWT
API Gateway->>OAuth2 Service: Validate JWT
OAuth2 Service-->>API Gateway: JWT Valid
API Gateway-->>Client: HTTPS Response

HTTPS Configuration for API Gateway​

Step 1: Generate SSL Certificate​

Generate a self-signed SSL certificate or obtain one from a Certificate Authority (CA).

# Generate a self-signed certificate (for testing purposes)
keytool -genkeypair -alias api-gateway -keyalg RSA -keysize 2048 -storetype PKCS12 -keystore api-gateway-keystore.p12 -validity 365

Step 2: Configure HTTPS in application.yml​

Add the keystore settings to application.yml.

server:
port: 8443
ssl:
enabled: true
key-store: classpath:api-gateway-keystore.p12
key-store-password: your_password_here
key-store-type: PKCS12

Step 3: Update Routes to Use lb://​

Ensure that the routes use lb:// to leverage service discovery and load balancing.

spring:
cloud:
gateway:
routes:
- id: profile_service
uri: lb://profile-svc
predicates:
- Path=/profile/**
- id: inventory_service
uri: lb://inventory-svc
predicates:
- Path=/inventory/**
- id: order_service
uri: lb://order-svc
predicates:
- Path=/order/**
- id: shipment_service
uri: lb://shipment-svc
predicates:
- Path=/shipment/**

Step 4: Update Kubernetes Ingress for HTTPS​

Configure the Kubernetes Ingress to redirect traffic to the API Gateway’s HTTPS endpoint.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-gateway-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- api.example.com
secretName: api-gateway-tls
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-gateway
port:
number: 8443

Difference Between lb:// and http:// Protocols​

lb:// Protocol​

  • Service Discovery: Uses Kubernetes service discovery to find the appropriate service instance.
  • Load Balancing: Automatically load balances requests across multiple instances of the service.
  • Dynamic Routing: Routes can be dynamically updated based on service availability.

Example:

spring:
cloud:
gateway:
routes:
- id: profile_service
uri: lb://profile-svc
predicates:
- Path=/profile/**

http:// Protocol​

  • Direct Routing: Routes requests directly to a specific URL.
  • No Load Balancing: Does not provide load balancing or service discovery.
  • Static Routing: Routes are static and do not change based on service availability.

Example:

spring:
cloud:
gateway:
routes:
- id: profile_service
uri: http://profile-svc:8080
predicates:
- Path=/profile/**

When to Use Each Protocol​

  • Use lb:// for routes within Kubernetes service discovery, allowing the gateway to leverage load balancing.
  • Use http:// for direct routes to a specific URL or when no service discovery is available.

The other option

Here’s a step-by-step setup for building the API Gateway with JWT authentication and routing configurations:

Step 1: Configure API Gateway with JWT Authentication and Routing​

# application.yml
spring:
cloud:
gateway:
routes:
- id: inventory-service
uri: lb://inventory-service
predicates:
- Path=/inventory/**
filters:
- name: JwtAuthenticationFilter
args:
roles: "ROLE_USER,ROLE_ADMIN" # Add roles if role-based access is needed
discovery:
locator:
enabled: true # Enables dynamic route lookup from discovery service
default-filters:
- StripPrefix=1 # Adjusts the path for downstream services

security:
oauth2:
resourceserver:
jwt:
issuer-uri: https://example-issuer.com # Point to the JWT issuer
jwk-set-uri: https://example-issuer.com/.well-known/jwks.json # Set up for validation

Step 2: Implement JWT Authentication Filter for Gateway​

The custom JWT filter ensures each request includes a valid JWT token.

// JwtAuthenticationFilter.java
@Component
public class JwtAuthenticationFilter implements GatewayFilter {

private final JwtDecoder jwtDecoder;

public JwtAuthenticationFilter(JwtDecoder jwtDecoder) {
this.jwtDecoder = jwtDecoder;
}

@Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
String authHeader = exchange.getRequest().getHeaders().getFirst(HttpHeaders.AUTHORIZATION);
if (authHeader != null && authHeader.startsWith("Bearer ")) {
String token = authHeader.substring(7);
try {
Jwt jwt = jwtDecoder.decode(token); // Validates the JWT
return chain.filter(exchange); // If valid, proceed with the request
} catch (JwtException e) {
exchange.getResponse().setStatusCode(HttpStatus.UNAUTHORIZED);
return exchange.getResponse().setComplete();
}
}
exchange.getResponse().setStatusCode(HttpStatus.UNAUTHORIZED);
return exchange.getResponse().setComplete();
}
}

Add the filter in application.yml for it to be applied to specific routes (like inventory-service in this case).

Step 3: Building and Deploying the API Gateway as a Docker Image​

  1. Dockerfile for the API Gateway:

    # Dockerfile for api-gateway
    FROM openjdk:17-jdk-slim
    ARG JAR_FILE=target/api-gateway.jar
    COPY ${JAR_FILE} api-gateway.jar
    ENTRYPOINT ["java", "-jar", "/api-gateway.jar"]
  2. Build the Docker Image:

    docker build -t myregistry/api-gateway:latest .
    docker push myregistry/api-gateway:latest # Push to your registry
  3. Kubernetes Deployment Configuration:

    # api-gateway-deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: api-gateway
    labels:
    app: api-gateway
    spec:
    replicas: 2
    selector:
    matchLabels:
    app: api-gateway
    template:
    metadata:
    labels:
    app: api-gateway
    spec:
    containers:
    - name: api-gateway
    image: myregistry/api-gateway:latest
    ports:
    - containerPort: 8080
    env:
    - name: SPRING_PROFILES_ACTIVE
    value: k8s

Step 4: Service-to-Service Communication via Gateway​

Now, other services can call the gateway (using api-gateway service DNS) and append the target service’s route (/inventory/*) to the request path.

Sample Code for a Downstream Service Request Using WebClient with JWT​

// ShipmentService.java
@Service
public class ShipmentService {

private final WebClient webClient;

public ShipmentService(WebClient.Builder webClientBuilder) {
this.webClient = webClientBuilder
.baseUrl("http://api-gateway") // Gateway URL in Kubernetes DNS
.defaultHeader(HttpHeaders.AUTHORIZATION, "Bearer " + jwtToken)
.build();
}

public Inventory getInventory(String id) {
return webClient.get()
.uri("/inventory/" + id)
.retrieve()
.bodyToMono(Inventory.class)
.block();
}
}

Comparison: Gateway vs. Interceptor-Based JWT Validation​

AspectGateway with JWT FilterService-Level JWT Interceptors
Centralized ControlAll JWT validation centralized at gatewayEach service has independent JWT validation
Configuration ComplexityEasier for microservices, managed in gatewayMore complex across multiple services
Performance OverheadCan cause bottleneck at gateway if overloadedDistributed load, but duplicate processing
ResilienceSingle point of failure without redundancyMore resilient, distributed across services
Authorization FlexibilityOffers consistent authentication policyEach service can implement role-based restrictions

Deep Dive into Executors and ThreadPoolExecutor

· 15 min read

Scaling a thread pool efficiently is essential to balance resource usage with workload demands, ensuring performance without over-provisioning or exhausting system resources. Below are common scaling strategies, including scenarios for using each and detailed production code examples with comments.

1. Overview of Thread Pool Scaling Strategies​

In thread pool management, the primary goal is to allocate just enough threads to handle tasks effectively without overloading the CPU or running into resource constraints. The Executor framework in Java provides tools for dynamic scaling via multiple built-in pool types and strategies, such as FixedThreadPool for predictable workloads and CachedThreadPool for on-demand scaling.


2. FixedThreadPool - Predictable Task Loads​

For workloads where the number of concurrent tasks is consistent or predictable, using a FixedThreadPool is efficient. The pool is created with a predefined number of threads, ensuring no new threads are added or removed during runtime. This strategy is suitable for batch processing or tasks with steady arrival rates.

Sample Code:

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class FixedThreadPoolExample {
// Define a pool with a fixed number of threads for predictable task load
private static final int THREAD_POOL_SIZE = 5;
private static final ExecutorService executorService = Executors.newFixedThreadPool(THREAD_POOL_SIZE);

public static void main(String[] args) {
// Simulate task submission to the fixed thread pool
for (int i = 0; i < 10; i++) {
final int taskId = i;
executorService.submit(() -> processTask(taskId));
}

// Shutdown the executor service to release resources after processing
executorService.shutdown();
}

private static void processTask(int taskId) {
System.out.println("Processing Task ID: " + taskId + " with thread " + Thread.currentThread().getName());
try {
// Simulating task processing duration
Thread.sleep(1000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}

Output:

Processing Task ID: 0 with thread pool-1-thread-1
Processing Task ID: 1 with thread pool-1-thread-2
Processing Task ID: 2 with thread pool-1-thread-3
Processing Task ID: 3 with thread pool-1-thread-4
Processing Task ID: 4 with thread pool-1-thread-5
Processing Task ID: 5 with thread pool-1-thread-1
Processing Task ID: 6 with thread pool-1-thread-2
Processing Task ID: 7 with thread pool-1-thread-3
Processing Task ID: 8 with thread pool-1-thread-4
Processing Task ID: 9 with thread pool-1-thread-5

Explanation:

  1. Thread Pool Initialization:

    • A fixed number of threads (5) handle the tasks sequentially.
    • The thread pool is initialized with 5 threads, named pool-1-thread-1 to pool-1-thread-5.
  2. Task Submission:

    • The for loop submits 10 tasks to the thread pool.
    • Each task is assigned a unique taskId from 0 to 9.
  3. Task Execution:

    • The tasks are executed by the available threads in the thread pool.
    • Since there are 5 threads, the first 5 tasks (0 to 4) are executed concurrently by the 5 threads.
    • After the first 5 tasks are completed, the next 5 tasks (5 to 9) are executed by the same threads, but the order of execution may vary.
  4. Thread Reuse:

    • The executor will reuse threads, limiting thread creation overhead.
    • The threads in the thread pool are reused to execute subsequent tasks. For example, pool-1-thread-1 might execute task 0 and then task 5.
  5. The shutdown method releases resources once all tasks complete.

Advantages: Predictable memory usage and better performance control for tasks with a steady rate.


3. CachedThreadPool - Unpredictable, Short-Lived Tasks​

A CachedThreadPool dynamically adjusts the pool size depending on task demand. This pool is ideal for handling bursty or sporadic loads. Threads are created as needed and, if idle, removed after a timeout period, allowing the system to scale back.

Sample Code:

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class CachedThreadPoolExample {
// Using CachedThreadPool for sporadic and short-lived tasks
private static final ExecutorService executorService = Executors.newCachedThreadPool();

public static void main(String[] args) {
// Simulate submitting bursty tasks
for (int i = 0; i < 10; i++) {
final int taskId = i;
executorService.submit(() -> handleBurstTask(taskId));
}

// Shutdown executor once all tasks are completed
executorService.shutdown();
}

private static void handleBurstTask(int taskId) {
System.out.println("Handling Burst Task ID: " + taskId + " with thread " + Thread.currentThread().getName());
try {
// Simulating brief processing
Thread.sleep(500);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}

Output:

Handling Burst Task ID: 0 with thread pool-1-thread-1
Handling Burst Task ID: 1 with thread pool-1-thread-2
Handling Burst Task ID: 2 with thread pool-1-thread-3
Handling Burst Task ID: 3 with thread pool-1-thread-4
Handling Burst Task ID: 4 with thread pool-1-thread-5
Handling Burst Task ID: 5 with thread pool-1-thread-6
Handling Burst Task ID: 6 with thread pool-1-thread-7
Handling Burst Task ID: 7 with thread pool-1-thread-8
Handling Burst Task ID: 8 with thread pool-1-thread-9
Handling Burst Task ID: 9 with thread pool-1-thread-10

Explanation:

  1. Thread Pool Initialization:

    • The CachedThreadPool adds threads on demand up to Integer.MAX_VALUE, suitable for short-lived tasks.
    • The CachedThreadPool dynamically creates new threads as needed and reuses previously constructed threads when they are available.
  2. Task Submission:

    • The for loop submits 10 tasks to the thread pool.
    • Each task is assigned a unique taskId from 0 to 9.
  3. Task Execution:

    • The tasks are executed by the available threads in the thread pool.
    • Since the CachedThreadPool creates new threads as needed, it will create up to 10 threads to handle the 10 tasks.
  4. Thread Creation and Reuse:

    • Threads that are idle for 60 seconds are removed, preventing memory buildup.
    • Ideal for systems that handle irregular spikes without knowing task demand in advance.
    • The CachedThreadPool creates new threads for each task initially because no threads are available.
    • The threads are named pool-1-thread-1 to pool-1-thread-10.

Advantages: Scalability and flexibility for dynamic task arrival rates.


4. Dynamic Scaling with Flexible Configuration​

In a production environment, scaling thread pools dynamically may involve using the ThreadPoolExecutor with customized parameters, such as adjustable core pool size and task queue types. This allows for on-the-fly adjustments based on real-time performance metrics.

Sample Code:

import java.util.concurrent.*;

public class DynamicThreadPoolScaling {
public static void main(String[] args) {
// Custom ThreadPoolExecutor with flexible parameters for dynamic scaling
ThreadPoolExecutor executor = new ThreadPoolExecutor(
2, // corePoolSize
10, // maximumPoolSize
60L, // keepAliveTime
TimeUnit.SECONDS,
new LinkedBlockingQueue<>(5), // workQueue
Executors.defaultThreadFactory(),
new ThreadPoolExecutor.CallerRunsPolicy() // Rejection policy
);

for (int i = 0; i < 20; i++) {
final int taskId = i;
executor.submit(() -> performDynamicTask(taskId));
}

// Shut down the executor after tasks completion
executor.shutdown();
}

private static void performDynamicTask(int taskId) {
System.out.println("Performing Dynamic Task ID: " + taskId + " with thread " + Thread.currentThread().getName());
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}

Explanation of Parameters:

  • corePoolSize: Defines the minimum number of threads.
  • maximumPoolSize: Sets a scalable limit for threads.
  • LinkedBlockingQueue: Limits task queue to prevent memory overflows.
  • CallerRunsPolicy: Ensures tasks run in the caller's thread if limits are reached, providing backpressure control.

Output:

Performing Dynamic Task ID: 0 with thread pool-1-thread-1
Performing Dynamic Task ID: 1 with thread pool-1-thread-2
Performing Dynamic Task ID: 2 with thread pool-1-thread-1
Performing Dynamic Task ID: 3 with thread pool-1-thread-2
Performing Dynamic Task ID: 4 with thread pool-1-thread-1
Performing Dynamic Task ID: 5 with thread pool-1-thread-2
Performing Dynamic Task ID: 6 with thread pool-1-thread-1
Performing Dynamic Task ID: 7 with thread pool-1-thread-2
Performing Dynamic Task ID: 8 with thread pool-1-thread-1
Performing Dynamic Task ID: 9 with thread pool-1-thread-2
Performing Dynamic Task ID: 10 with thread pool-1-thread-3
Performing Dynamic Task ID: 11 with thread pool-1-thread-4
Performing Dynamic Task ID: 12 with thread pool-1-thread-5
Performing Dynamic Task ID: 13 with thread pool-1-thread-6
Performing Dynamic Task ID: 14 with thread pool-1-thread-7
Performing Dynamic Task ID: 15 with thread pool-1-thread-8
Performing Dynamic Task ID: 16 with thread pool-1-thread-9
Performing Dynamic Task ID: 17 with thread pool-1-thread-10
Performing Dynamic Task ID: 18 with thread main
Performing Dynamic Task ID: 19 with thread main

Explaination of Output:

  1. ThreadPoolExecutor Initialization:

    • The ThreadPoolExecutor is initialized with:
      • corePoolSize: 2
      • maximumPoolSize: 10
      • keepAliveTime: 60 seconds
      • workQueue: LinkedBlockingQueue with a capacity of 5
      • threadFactory: Default thread factory
      • rejectionPolicy: CallerRunsPolicy
  2. Task Submission:

    • The for loop submits 20 tasks to the thread pool.
    • Each task is assigned a unique taskId from 0 to 19.
  3. Task Execution:

    • Initially, the core threads (pool-1-thread-1 and pool-1-thread-2) handle the first 2 tasks.
    • The next 5 tasks are queued in the LinkedBlockingQueue.
    • As the core threads finish their tasks, they pick up tasks from the queue.
    • When the queue is full (5 tasks), new threads are created up to the maximumPoolSize (10 threads) to handle the remaining tasks.
    • When all 10 threads are busy and the queue is full, the CallerRunsPolicy kicks in, and the remaining tasks are executed by the main thread (main).

Detailed Steps​

  1. Task 0 to 1 Execution:

    • Task 0 is executed by pool-1-thread-1.
    • Task 1 is executed by pool-1-thread-2.
  2. Task 2 to 6 Queued:

    • Tasks 2 to 6 are queued in the LinkedBlockingQueue.
  3. Task 2 to 6 Execution:

    • Tasks 2 to 6 are executed by pool-1-thread-1 and pool-1-thread-2 as they finish their initial tasks.
  4. Task 7 to 11 Queued:

    • Tasks 7 to 11 are queued in the LinkedBlockingQueue.
  5. Task 7 to 11 Execution:

    • Tasks 7 to 11 are executed by pool-1-thread-1 and pool-1-thread-2 as they finish their previous tasks.
  6. Task 12 to 16 Execution:

    • New threads (pool-1-thread-3 to pool-1-thread-7) are created to handle tasks 12 to 16.
  7. Task 17 to 19 Execution:

    • New threads (pool-1-thread-8 to pool-1-thread-10) are created to handle tasks 17 to 19.
  8. Task 18 and 19 Execution by Main Thread:

    • Since all threads are busy and the queue is full, the CallerRunsPolicy causes the main thread to execute tasks 18 and 19.

5. Thread Pool Strategy Decision​

+-------------------------+
| Task Load Predictable? |
+-----------+-------------+
|
|
Yes | No
+----------+------------+
| |
+-------v----------+ +------v------------+
| FixedThreadPool | | CachedThreadPool |
+-------+----------+ +-----+-------------+
| |
| |
Use if tasks are steady Use if tasks are bursty or
and concurrent unpredictable

Using Executors to create thread pools (e.g., via Executors.newFixedThreadPool() or Executors.newCachedThreadPool()) is convenient but introduces hidden risks and drawbacks in certain production scenarios. Let's look at the specific drawbacks and explore best practices for mitigating these issues.


6. Drawbacks of Using Executors to Create Thread Pools​

  1. Unbounded Queue with newFixedThreadPool:

    • The newFixedThreadPool(int nThreads) method uses a LinkedBlockingQueue without a maximum size limit for its task queue. This unbounded queue can lead to memory exhaustion if tasks are submitted faster than they can be processed.
    • When a large number of tasks are added, the queue will keep growing, which can lead to OutOfMemoryError and system instability.
  2. Unbounded Thread Creation in newCachedThreadPool:

    • The newCachedThreadPool() uses a SynchronousQueue that directly hands off tasks to threads. If no threads are idle to handle new tasks, it creates new threads up to Integer.MAX_VALUE.
    • With a high load, this pool may create excessive threads, leading to CPU saturation and potential memory exhaustion, especially in long-running applications.
  3. Lack of Control Over Rejection Policy:

    • The methods in Executors do not allow explicit configuration of rejection policies for when tasks exceed pool capacity. Default policies can fail to handle high-load scenarios gracefully, often leading to performance degradation and unhandled task rejections.
  4. Harder to Manage Idle Threads:

    • For pools like newCachedThreadPool, idle threads remain for a default 60 seconds before being terminated. While this may be appropriate for some workloads, it can lead to inefficiencies in scenarios where shorter idle times would conserve resources.

7. Best Practices Solution: Using ThreadPoolExecutor Directly​

To avoid these pitfalls, it's best to directly instantiate a ThreadPoolExecutor where you can explicitly configure the task queue size, rejection policies, and timeout values. Here’s how to address each issue with recommended configurations:

A. Configure Bounded Queues to Prevent Memory Leaks:

  • Use a bounded task queue, like ArrayBlockingQueue or a sized LinkedBlockingQueue, to control the number of waiting tasks and prevent out-of-memory errors.
  • This limits the number of tasks waiting in the queue, ensuring system stability by capping memory use.
new ThreadPoolExecutor(
corePoolSize,
maxPoolSize,
keepAliveTime,
TimeUnit.SECONDS,
new ArrayBlockingQueue<>(queueSize)
);

B. Limit Thread Growth to Prevent Resource Exhaustion:

  • Use a capped maximumPoolSize when tasks can surge unexpectedly. Setting a limit, e.g., based on the number of available CPU cores, helps prevent CPU saturation.

    When tasks can surge unexpectedly, it's crucial to set a capped maximumPoolSize to prevent the thread pool from creating an excessive number of threads. An unbounded maximumPoolSize can lead to CPU saturation, excessive memory usage, and degraded performance due to excessive context switching.

    CPU Saturation CPU saturation occurs when the CPU is overwhelmed with too many threads, leading to:

    • High CPU Utilization: The CPU spends most of its time executing threads, leaving little time for other system processes.
    • Context Switching Overhead: Frequent context switching between threads consumes CPU cycles, reducing overall efficiency.
    • Memory Pressure: Each thread consumes memory, and a large number of threads can lead to memory exhaustion.

    To prevent CPU saturation, it's recommended to set a maximumPoolSize based on the number of available CPU cores. This ensures that the thread pool can handle a reasonable number of concurrent tasks without overwhelming the CPU.

    A common approach is to set the maximumPoolSize to a value slightly higher than the number of CPU cores to account for both CPU-bound and I/O-bound tasks. A simple formula is maximumPoolSize = N * U

    Where:

    • N: Number of available CPU cores.
    • U: Utilization factor (typically between 1.0 and 2.0).
      • U = 1.0: For CPU-bound tasks where each thread is expected to fully utilize a CPU core.
      • U = 1.5 to 2.0: For mixed CPU-bound and I/O-bound tasks, where some threads may be waiting on I/O operations.

    Example Calculation Suppose you have a system with 8 CPU cores and you expect a mix of CPU-bound and I/O-bound tasks. You can set the maximumPoolSize as follows maximumPoolSize = 8 * 1.5 = 12

  • For highly concurrent applications, you can use dynamic sizing strategies with monitoring metrics (like CPU and memory usage) to inform adjustments to pool parameters.

C. Explicitly Set Rejection Policies:

  • When tasks exceed the pool’s capacity, control the pool’s behavior with rejection policies. ThreadPoolExecutor provides options like CallerRunsPolicy, DiscardPolicy, and DiscardOldestPolicy:
    • CallerRunsPolicy: Executes tasks in the caller thread, which helps slow down task submissions under heavy load.
    • DiscardOldestPolicy: Removes the oldest queued task, making room for newer, high-priority tasks.
  • This choice depends on your application’s tolerance for unprocessed tasks.
new ThreadPoolExecutor(
corePoolSize,
maxPoolSize,
keepAliveTime,
TimeUnit.SECONDS,
new LinkedBlockingQueue<>(queueSize),
new ThreadPoolExecutor.CallerRunsPolicy()
);

Adjust Idle Timeout for Cached Threads:

  • For pools similar to CachedThreadPool, customize the idle timeout to conserve resources by releasing threads sooner when they are not needed. ThreadPoolExecutor allows control over idle time with setKeepAliveTime.
ThreadPoolExecutor cachedPool = new ThreadPoolExecutor(
0, Integer.MAX_VALUE,
30L, TimeUnit.SECONDS, // Reduced keep-alive time
new SynchronousQueue<>(),
Executors.defaultThreadFactory()
);

8. ThreadPoolExecutor with Best Practice Configurations​

Below is a best practice setup using ThreadPoolExecutor that incorporates bounded queues, a limited thread count, a suitable rejection policy, and a controlled idle timeout:

import java.util.concurrent.*;

public class BestPracticeThreadPool {

public static ExecutorService createThreadPool() {
int corePoolSize = 4;
int maxPoolSize = 10;
long keepAliveTime = 30L;
int queueCapacity = 50;

return new ThreadPoolExecutor(
corePoolSize,
maxPoolSize,
keepAliveTime,
TimeUnit.SECONDS,
new ArrayBlockingQueue<>(queueCapacity),
Executors.defaultThreadFactory(),
new ThreadPoolExecutor.CallerRunsPolicy() // Rejection policy to handle overloads
);
}

public static void main(String[] args) {
ExecutorService threadPool = createThreadPool();

// Submitting example tasks
for (int i = 0; i < 100; i++) {
final int taskId = i;
threadPool.submit(() -> processTask(taskId));
}

// Properly shutdown to free resources
threadPool.shutdown();
}

private static void processTask(int taskId) {
System.out.println("Processing Task ID: " + taskId + " with thread " + Thread.currentThread().getName());
try {
Thread.sleep(500); // Simulate work
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}

Explanation of Configurations:

  • ArrayBlockingQueue limits queue size to 50, managing memory consumption.
  • CallerRunsPolicy applies backpressure to slow down submissions.
  • corePoolSize and maxPoolSize control CPU utilization.
  • keepAliveTime of 30 seconds clears idle threads efficiently.

9. Summary​

Using ThreadPoolExecutor directly provides complete control over pool configurations:

  • Bounded queues prevent memory overflows.
  • Limited threads avoid CPU exhaustion.
  • Rejection policies ensure controlled behavior under load.
  • Idle timeouts conserve system resources.

These adjustments create a resilient and scalable thread pool setup suitable for high-performance, resource-intensive applications.

Deep Dive into Spring Boot Multi-Threading

· 7 min read

Introduction to Spring Boot Multi-Threading

Spring Boot provides robust support for multi-threading, enabling developers to build highly concurrent and scalable applications. Leveraging multi-threading in Spring Boot can significantly improve the performance and responsiveness of your application. This article delves into best practices, annotation-based configuration, and real-life production scenarios to help you master multi-threading in Spring Boot.


Best Practices for Spring Boot Multi-Threading

  1. Use @Async for Asynchronous Processing: The @Async annotation allows methods to run in a separate thread, making it easy to implement asynchronous processing.
  2. Configure Thread Pools: Use TaskExecutor to configure thread pools with specific characteristics like core pool size, max pool size, and queue capacity.
  3. Handle Exceptions Gracefully: Implement exception handling mechanisms to manage exceptions that occur in asynchronous tasks.
  4. Monitor Thread Pools: Use Spring Boot Actuator to monitor thread pool metrics and ensure optimal performance.
  5. Avoid Blocking Operations: Ensure that asynchronous methods do not perform blocking operations, which can degrade performance.

Annotation-Based Configuration

Spring Boot provides several annotations to configure and manage multi-threading:

  • @EnableAsync: Enables asynchronous processing in the application.
  • @Async: Marks a method as asynchronous, meaning it will run in a separate thread.
  • @Configuration: Indicates that a class declares one or more @Bean methods and may be processed by the Spring container to generate bean definitions.
  • @Bean: Indicates that a method produces a bean to be managed by the Spring container.

Diagram of Multi-Threading in Spring Boot

+-------------------------+
| Spring Boot Application |
| - @EnableAsync |
| - @Configuration |
| - @Bean |
+-------------------------+
|
v
+-------------------+
| TaskExecutor |
| - corePoolSize |
| - maxPoolSize |
| - queueCapacity |
+-------------------+
|
v
+-------------------+
| @Async Methods |
| - Method1 |
| - Method2 |
+-------------------+
|
v
+-------------------+
| Thread Pool |
| - Thread 1 |
| - Thread 2 |
| - Thread 3 |
+-------------------+
|
v
+-------------------+
| Task Execution |
| - Task 1 |
| - Task 2 |
| - Task 3 |
+-------------------+

Sample Code with Detailed Comments

Configuration Class​

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;

@Configuration
@EnableAsync // Enable asynchronous processing
public class AsyncConfig {

@Bean(name = "taskExecutor")
public Executor taskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(10); // Core pool size
executor.setMaxPoolSize(20); // Maximum pool size
executor.setQueueCapacity(100); // Queue capacity
executor.setThreadNamePrefix("AsyncThread-"); // Thread name prefix
executor.initialize();
return executor;
}
}

Service Class​

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

@Service
public class AsyncService {

@Async("taskExecutor") // Use the taskExecutor bean for this method
public void performAsyncTask(String taskName) {
System.out.println("Starting task: " + taskName + " on thread: " + Thread.currentThread().getName());
try {
// Simulate task execution time
Thread.sleep(2000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Completed task: " + taskName + " on thread: " + Thread.currentThread().getName());
}
}

Controller Class​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class AsyncController {

@Autowired
private AsyncService asyncService;

@GetMapping("/performTask")
public String performTask(@RequestParam String taskName) {
asyncService.performAsyncTask(taskName);
return "Task " + taskName + " has been submitted for asynchronous processing.";
}
}

Application Class​

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableAsync;

@SpringBootApplication
@EnableAsync // Enable asynchronous processing
public class AsyncApplication {

public static void main(String[] args) {
SpringApplication.run(AsyncApplication.class, args);
}
}

Complex Scenario 1: Flow Control and Backpressure Handling

In high-throughput systems, it's crucial to manage the flow of tasks to avoid overwhelming the system. Backpressure handling ensures that the system can gracefully handle a large number of incoming tasks.

Configuration Class​

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;

@Configuration
@EnableAsync
public class BackpressureConfig {

@Bean(name = "backpressureExecutor")
public Executor backpressureExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5); // Core pool size
executor.setMaxPoolSize(10); // Maximum pool size
executor.setQueueCapacity(50); // Queue capacity
executor.setThreadNamePrefix("BackpressureThread-"); // Thread name prefix
executor.initialize();
return executor;
}
}

Service Class​

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

@Service
public class BackpressureService {

@Async("backpressureExecutor")
public void handleTask(String taskName) {
System.out.println("Handling task: " + taskName + " on thread: " + Thread.currentThread().getName());
try {
// Simulate task execution time
Thread.sleep(3000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Completed handling task: " + taskName + " on thread: " + Thread.currentThread().getName());
}
}

Controller Class​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class BackpressureController {

@Autowired
private BackpressureService backpressureService;

@GetMapping("/handleTask")
public String handleTask(@RequestParam String taskName) {
backpressureService.handleTask(taskName);
return "Task " + taskName + " has been submitted for backpressure handling.";
}
}

Complex Scenario 2: Task Prioritization

In some scenarios, certain tasks may need to be prioritized over others. Spring Boot allows you to configure task executors with priority queues to handle such cases.

Configuration Class​

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;
import java.util.concurrent.PriorityBlockingQueue;

@Configuration
@EnableAsync
public class PriorityConfig {

@Bean(name = "priorityExecutor")
public Executor priorityExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5); // Core pool size
executor.setMaxPoolSize(10); // Maximum pool size
executor.setQueueCapacity(50); // Queue capacity
executor.setThreadNamePrefix("PriorityThread-"); // Thread name prefix
executor.setTaskDecorator(new PriorityTaskDecorator()); // Set task decorator for priority
executor.initialize();
return executor;
}
}

Task Decorator​

import org.springframework.core.task.TaskDecorator;

public class PriorityTaskDecorator implements TaskDecorator {

@Override
public Runnable decorate(Runnable runnable) {
return () -> {
// Set priority logic here
System.out.println("Setting priority for task: " + runnable);
runnable.run();
};
}
}

Service Class​

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

@Service
public class PriorityService {

@Async("priorityExecutor")
public void prioritizeTask(String taskName) {
System.out.println("Prioritizing task: " + taskName + " on thread: " + Thread.currentThread().getName());
try {
// Simulate task execution time
Thread.sleep(3000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Completed prioritizing task: " + taskName + " on thread: " + Thread.currentThread().getName());
}
}

Controller Class​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class PriorityController {

@Autowired
private PriorityService priorityService;

@GetMapping("/prioritizeTask")
public String prioritizeTask(@RequestParam String taskName) {
priorityService.prioritizeTask(taskName);
return "Task " + taskName + " has been submitted for prioritization.";
}
}

Complex Scenario 3: Handling Long-Running Tasks

Long-running tasks can block threads and degrade system performance. Spring Boot provides mechanisms to handle such tasks efficiently.

Configuration Class​

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;

@Configuration
@EnableAsync
public class LongRunningConfig {

@Bean(name = "longRunningExecutor")
public Executor longRunningExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5); // Core pool size
executor.setMaxPoolSize(10); // Maximum pool size
executor.setQueueCapacity(50); // Queue capacity
executor.setThreadNamePrefix("LongRunningThread-"); // Thread name prefix
executor.initialize();
return executor;
}
}

Service Class​

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

@Service
public class LongRunningService {

@Async("longRunningExecutor")
public void handleLongRunningTask(String taskName) {
System.out.println("Handling long-running task: " + taskName + " on thread: " + Thread.currentThread().getName());
try {
// Simulate long-running task execution time
Thread.sleep(10000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Completed handling long-running task: " + taskName + " on thread: " + Thread.currentThread().getName());
}
}

Controller Class​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class LongRunningController {

@Autowired
private LongRunningService longRunningService;

@GetMapping("/handleLongRunningTask")
public String handleLongRunningTask(@RequestParam String taskName) {
longRunningService.handleLongRunningTask(taskName);
return "Long-running task " + taskName + " has been submitted for processing.";
}
}

Conclusion

Spring Boot's support for multi-threading allows developers to build highly concurrent and scalable applications. By following best practices and leveraging annotation-based configuration, you can efficiently manage asynchronous tasks, handle backpressure, prioritize tasks, and manage long-running tasks. This article provides a comprehensive guide to mastering multi-threading in Spring Boot, complete with detailed sample code and real-life production scenarios.

Microservice meet with Spring Cloud and Kubernetes

· 13 min read

Introduction​

The objective is to design and deploy a group of microservices: Profile Service, Inventory Service, Order Service, Shipment Service, and an OAuth2 Service for centralized authentication. Key requirements include robust service discovery, API Gateway setup, secure communication, and efficient load balancing for high availability and fault tolerance in Kubernetes.

High-Level Architecture​

Each microservice will be deployed in a Kubernetes cluster with multiple replicas, leveraging Spring Cloud Kubernetes Discovery for service registration and Spring Cloud LoadBalancer for client-side load balancing. An API Gateway will handle external requests, providing routing, load balancing, and JWT-based authentication. Communication between services is secured with JWT tokens generated by the OAuth2 Service.

Service Design​

Each microservice is structured independently, deployed as a standalone Spring Boot application with individual responsibilities and access rules. Here’s an overview of the key services:

1. Profile Service​

Manages customer profiles, providing endpoints for CRUD operations on profile data. Accessible to services like Order and Shipment for customer information retrieval.

2. Inventory Service​

Tracks inventory levels and provides endpoints to update, retrieve, and manage inventory data. Called by the Order Service for real-time inventory status updates.

3. Order Service​

Processes customer orders, calculates totals, and interfaces with Inventory and Profile Services. This service is central to the architecture, relying on other services for a smooth customer experience.

4. Shipment Service​

Manages shipment tracking and logistics for orders, called by the Order Service once an order is processed and confirmed.

5. OAuth2 Service​

Acts as the authentication provider, issuing and validating JWT tokens for secure communication. The API Gateway uses these tokens to authenticate requests.

Order-Service Auth-Service Inventory-Service
| | |
|--- Request JWT ------------->| |
| | |
|<-- Receive JWT --------------| |
| | |
|--- Call Inventory ---------->| |
| (with JWT) | |
| |--- Validate JWT------------>|
| | |
| |<-- JWT Valid/Invalid--------|
| | |
|<-- Receive Inventory Data -- | |

Kubernetes Deployment and Service Discovery​

Each service will be deployed to Kubernetes as a Deployment with multiple replicas and exposed via a Service to enable discovery. Spring Cloud Kubernetes is used for service registration and discovery.

Configuring Kubernetes Service Discovery for Each Microservice​

To set up service discovery in Kubernetes, add the following dependencies to your pom.xml:

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-discovery</artifactId>
</dependency>

Configure the application.yml in each service to enable discovery:

spring:
cloud:
kubernetes:
discovery:
enabled: true #Enable service discovery
config:
enabled: true
loadbalancer:
enabled: true #Enable loadbalancer

This configuration registers each service with Kubernetes' internal DNS, allowing them to be discoverable by service name within the namespace. This way, each microservice is registered under its own Kubernetes service name, and can resolve other services (e.g., http://inventory-service:8080) when communicating with them. This configuration, combined with Kubernetes DNS, helps maintain an efficient, loosely coupled architecture.

Note: The Kubernetes Service Discovery configuration should indeed be included in each individual microservice’s configuration to ensure they can all register themselves with the Kubernetes DNS and be discoverable by other services in the namespace. This setup allows each microservice to dynamically locate and communicate with other services via Kubernetes' internal DNS naming conventions.

Why Configure Service Discovery in Each Microservice?

Each microservice is an independently deployed unit, so configuring Spring Cloud Kubernetes Discovery in each service's application.yml allows it to:

  1. Register with Kubernetes DNS under a unique service name, enabling internal routing across services.
  2. Access other services by simply using their service names, as Kubernetes DNS provides a consistent way to locate services within the cluster.

Note Here you might be noticed that both the discovery and load balancer components are configured within each microservice. This design allows each microservice to dynamically discover the current instances of other services (using the spring-cloud-kubernetes-discovery integration) and balance requests among those instances using the spring-cloud-loadbalancer component.

Service Registration​

Each microservice is deployed as a separate Deployment and Service in Kubernetes. Here’s an example of the inventory-service Kubernetes configuration:

To deploy and register microservices like an Inventory Service in Kubernetes, we use Spring Cloud’s Kubernetes Discovery capabilities, allowing services to register with Kubernetes’ DNS-based service registry automatically.

Step 1: Create a Kubernetes Service and Deployment for each microservice:

YAML files for defining the service and deployment:

# inventory-service-deployment.yml
apiVersion: v1
kind: Service
metadata:
name: inventory-service
spec:
selector:
app: inventory
ports:
- protocol: TCP
port: 8080
targetPort: 8080
type: ClusterIP # Internal access within the cluster

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: inventory-deployment
spec:
replicas: 2 # Number of instances (pods) of the Inventory Service
selector:
matchLabels:
app: inventory
template:
metadata:
labels:
app: inventory
spec:
containers:
- name: inventory
image: myregistry/inventory-service:latest
ports:
- containerPort: 8080

This configuration deploys inventory-service with two replicas, registered under the service name inventory-service. Other services can communicate with it via http://inventory-service:8080.

More comprehensive explainations When deploying an inventory-deployment with two replicas, you will indeed see two pods running in parallel in the specified namespace. Kubernetes names these pods based on the deployment name, followed by a unique suffix (not just sequential numbers), so you may see pod names like inventory-deployment-abc12 and inventory-deployment-def34.

In Kubernetes, these two pods are managed by a single inventory-service which acts as a layer of abstraction. When the client load balancer targets inventory-service, the service uses Kubernetes' built-in load balancing to forward requests to one of the available inventory-deployment pods.

How Kubernetes Service Load Balancing Works:

  1. Service Abstraction: The inventory-service has a Cluster IP that exposes the service within the cluster.
  2. Endpoints Selection: The service maintains an updated list of active pod IPs (called "endpoints") that match the deployment selector.
  3. Round-Robin Load Balancing: Kubernetes uses a round-robin approach by default. When a request reaches inventory-service, it selects one of the pod IPs associated with the deployment, balancing requests across pods automatically. The service doesn't keep state; each request is independently forwarded to one of the pod replicas.

Example Flow:

  1. Client Request: A client load balancer sends a request to the inventory-service endpoint.
  2. Service to Pod Routing: Kubernetes service routing picks an endpoint (pod IP) from the inventory-deployment pods and forwards the request to it.

For better control and observability, custom load balancer configurations can also be implemented to fine-tune how the requests are distributed, especially in setups requiring sticky sessions or advanced routing.

Step 2: Register Service with Kubernetes Discovery:

Spring Cloud Kubernetes automatically registers inventory-service based on the deployment labels. Enable Spring Cloud Discovery in your Spring Boot application via @EnableDiscoveryClient:

@EnableDiscoveryClient
@SpringBootApplication
public class InventoryServiceApplication {
public static void main(String[] args) {
SpringApplication.run(InventoryServiceApplication.class, args);
}
}

Step 3: Configure Discovery in application.yml:

spring:
cloud:
kubernetes:
discovery:
enabled: true
all-namespaces: true # Optional: enables discovery across namespaces

Load Balancing with Spring Cloud LoadBalancer​

To efficiently distribute incoming requests across available instances, Spring Cloud LoadBalancer provides a client-side load-balancing solution, automatically distributing traffic among instances of a service.

Enable Spring Cloud LoadBalancer by adding Spring Cloud LoadBalancer dependency:

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>

Configure a ServiceInstanceListSupplier bean for custom load balancing behavior if required:

import org.springframework.cloud.client.loadbalancer.ServiceInstanceListSupplier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class LoadBalancerConfig {

@Bean
public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier() {
return ServiceInstanceListSupplier.builder()
.withDiscoveryClient()
.withCaching()
.build();
}
}

Set Up Load Balancer with WebClient:

  • Configure a WebClient bean with load balancing to distribute requests among inventory-service instances.
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.context.annotation.Bean;
import org.springframework.web.reactive.function.client.WebClient;

@Configuration
public class WebClientConfig {

@Bean
@LoadBalanced
public WebClient.Builder loadBalancedWebClientBuilder() {
return WebClient.builder();
}
}

Use Load-Balanced WebClient in the calling service:

@Service
public class InventoryClient {

private final WebClient.Builder webClientBuilder;

public InventoryClient(WebClient.Builder webClientBuilder) {
this.webClientBuilder = webClientBuilder;
}

public Mono<Product> getProductById(String productId) {
return webClientBuilder.build()
.get()
.uri("http://inventory-service/products/{id}", productId)
.retrieve()
.bodyToMono(Product.class);
}
}

How Load Balancing Works:

  • Spring Cloud LoadBalancer, by default, uses a round-robin strategy to distribute requests.
  • It balances requests between the two inventory-service pods and automatically updates when new instances are added or removed.

  1. By default, Spring Cloud LoadBalancer will round-robin requests to different instances, or customize by implementing a custom LoadBalancerRequestTransformer.

API Gateway Setup​

The Spring Cloud API Gateway is deployed as a separate microservice, managing traffic and providing JWT-based authentication. The gateway routes requests to the appropriate services and load-balances among available replicas.

Routing and Load Balancing​

[External Client] --> [API Gateway]
|
| (Discovers routes from Kubernetes Discovery)
|
[inventory-service] <--+-- "lb://inventory-service"
[shipment-service] <--+-- "lb://shipment-service"
[order-service] <--+-- "lb://order-service"
[profile-service] <--+-- "lb://profile-service"
[other-service] <--+-- Discovered automatically

Configure the gateway’s application.yml to define routes to each service:

spring:
cloud:
gateway:
routes:
- id: profile
uri: http://profile-service
predicates:
- Path=/profile/**
- id: inventory
uri: http://inventory-service
predicates:
- Path=/inventory/**
default-filters:
- TokenRelay

The TokenRelay filter enables the gateway to forward JWT tokens to downstream services for authenticated requests.

Configuring JWT Authentication​

For security, JWT validation can be implemented in the gateway or within each service. Here’s how to configure JWT validation in the gateway:

  1. Add OAuth2 dependencies to the gateway:

    <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
    </dependency>
  2. Configure JWT validation in the gateway’s application.yml:

    spring:
    security:
    oauth2:
    resourceserver:
    jwt:
    issuer-uri: https://auth-server/oauth2/default

Microservice Communication​

Microservices communicate through Spring Cloud LoadBalancer and WebClient.

Using WebClient with Load Balancer​

To call the inventory-service from the order-service, configure WebClient with load balancing:

@Configuration
public class WebClientConfig {

@LoadBalanced
@Bean
public WebClient.Builder webClientBuilder() {
return WebClient.builder();
}
}

Example: Order Service Calls Profile Service​

Here’s how order-service might call profile-service:

@RestController
public class OrderController {

private final WebClient.Builder webClientBuilder;

@Autowired
public OrderController(WebClient.Builder webClientBuilder) {
this.webClientBuilder = webClientBuilder;
}

@GetMapping("/order/{orderId}")
public Mono<OrderDetails> getOrderDetails(@PathVariable String orderId) {
return webClientBuilder.build()
.get()
.uri("http://profile-service/profile/{orderId}", orderId) // The uri can be either http or lb(load balancer)
.retrieve()
.bodyToMono(OrderDetails.class);
}
}

Securing Microservices​

Secure Communication with mTLS and JWT

mTLS (mutual TLS) and JWT (JSON Web Token) are two primary ways to secure communication between microservices.

  • mTLS: Ensures that both the client and server authenticate each other, useful in a zero-trust network.
  • JWT: Adds stateless authentication, where each request includes a signed token with user and session data.

Each microservice validates JWT tokens received from the gateway, ensuring secure and authenticated communication.

JWT Authentication with API Gateway​

JWT Authentication is often managed at the API Gateway, validating tokens and authorizing requests before forwarding them to services.

In this approach, JWT validation is centralized in the gateway. Tokens are verified at the gateway and forwarded to services, reducing the validation load on individual services.

  1. Add Spring Security Dependency for JWT support:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
  1. Configure JWT Verification in Gateway:

Validate JWTs at the gateway level, allowing only authorized requests to reach downstream services.

@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter {

@Override
protected void configure(HttpSecurity http) throws Exception {
http
.authorizeRequests()
.antMatchers("/api/**").authenticated()
.and()
.oauth2ResourceServer().jwt();
}
}

Alternative: JWT Validation in Interceptors​

Alternatively, JWT validation can be handled within each microservice using a custom Interceptor, within services, add an Interceptor to validate JWTs before processing requests.

import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;

@Component
public class JwtInterceptor implements HandlerInterceptor {

@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
String token = request.getHeader("Authorization");
// Verify the JWT token
// Decode and validate claims
return isValidToken(token);
}
}

Pros and Cons of JWT and mTLS​

ApproachProsCons
mTLSHigh security with mutual authentication.Requires certificate management, adds complexity.
JWTStateless, easily scalable, widely supported.Token expiration handling; replay attacks if not carefully implemented.

<a name="kubernetes-deployment-and-auto-scaling"></a>
## Kubernetes Deployment and Auto-scaling

Scaling services dynamically is fundamental in Kubernetes. Kubernetes provides an effective way to handle **auto-scaling** of microservices based on resource utilization, e.g. CPU, memory, or custom metrics.

Each microservice is balanced using **Spring Cloud LoadBalancer**, ensuring even distribution across replicas.

> **Step 1**: Configuring Horizontal Pod Autoscaler (HPA)

1. Enable auto-scaling in Kubernetes based on CPU usage by defining an HPA resource.

**inventory-hpa.yaml**
```yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: inventory-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: inventory-deployment
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 75
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50

Step 2: Implementing Cluster Scaling

Use Kubernetes’ Cluster Autoscaler for scaling nodes as workload demand grows, ensuring optimal resource use without manual intervention.


Ensuring Secure Microservice Communication​

Securing communication between services is critical. Use mTLS (mutual TLS) and JWT tokens for service-to-service authentication within Kubernetes clusters.

Step 1: Configure Mutual TLS (mTLS)​

Istio or Linkerd can help enforce mTLS between services:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: your-namespace
spec:
mtls:
mode: STRICT

Summary​

  1. Use ConfigMaps and Secrets for configuration.
  2. Implement autoscaling with HPA.
  3. Secure communication with mTLS and JWT.
  4. Monitor with readiness/liveness probes and Prometheus/Grafana.

This guide provides an architecture blueprint for deploying secure, scalable, and efficient microservices in Kubernetes using Spring Cloud. The API Gateway simplifies external access, service discovery, and JWT-based authentication, while Kubernetes ensures high availability and resilience across services.

Create a Java Thread Pool From Scratch

· 17 min read

Custom Java Thread Pool: Proof of Concept

In this article, we will create a simple but complete custom thread pool. We’ll build a SimpleThreadPool class that includes key features such as managing worker threads, a task queue, and shutdown capabilities. We'll break down the code into logical components, explain the design concepts, and provide sample usage.


Implementation of SimpleThreadPool​

The core classes in our implementation are:

  • SimpleThreadPool: Manages the thread pool and task queue.
  • Worker: Represents the worker threads that execute tasks.
import java.util.LinkedList;
import java.util.Queue;

public class SimpleThreadPool {
private final int poolSize;
private final PoolWorker[] workers;
private final Queue<Runnable> taskQueue;
private boolean isShutdown = false;

public SimpleThreadPool(int poolSize) {
this.poolSize = poolSize;
taskQueue = new LinkedList<>();
workers = new PoolWorker[poolSize];

// Start worker threads
for (int i = 0; i < poolSize; i++) {
workers[i] = new PoolWorker();
workers[i].start();
}
}

// Submit a task to the thread pool
public synchronized void submit(Runnable task) {
if (!isShutdown) {
taskQueue.add(task);
notify(); // Notify workers that a task is available
} else {
throw new IllegalStateException("ThreadPool is shut down");
}
}

// Initiates an orderly shutdown
public synchronized void shutdown() {
isShutdown = true;
for (PoolWorker worker : workers) {
worker.interrupt(); // Interrupt each worker to stop waiting
}
}

// Worker class represents threads in the pool
private class PoolWorker extends Thread {
public void run() {
Runnable task;

while (true) {
synchronized (SimpleThreadPool.this) {
while (taskQueue.isEmpty() && !isShutdown) {
try {
SimpleThreadPool.this.wait(); // Wait for a task
} catch (InterruptedException ignored) {
return; // Exit if interrupted during shutdown
}
}
// If shutdown and no tasks, end thread
if (isShutdown && taskQueue.isEmpty()) {
return;
}
task = taskQueue.poll(); // Take task from queue
}
try {
task.run(); // Execute task outside synchronized block
} catch (RuntimeException e) {
System.err.println("Task execution failed: " + e.getMessage());
}
}
}
}
}

Explanation of Key Concepts​

  1. Task Queue: A LinkedList serves as the task queue, holding Runnable tasks for worker threads.
  2. Worker Threads: PoolWorker threads execute tasks. Each worker loops to fetch a task from the queue and processes it if available.
  3. Synchronization with wait and notify:
    • wait(): Called when no tasks are available, putting the thread in a waiting state to save CPU resources.
    • notify(): Called when a new task is added, waking up a worker thread to execute the task.
  4. Graceful Shutdown:
    • shutdown(): Sets the isShutdown flag to stop accepting tasks and interrupts all worker threads, allowing them to exit if they’re waiting for a task.

Sample Code to Use the SimpleThreadPool​

This example demonstrates how to use the custom SimpleThreadPool to run tasks, including graceful shutdown.

public class SimpleThreadPoolExample {
public static void main(String[] args) {
// Create a thread pool with 3 threads
SimpleThreadPool threadPool = new SimpleThreadPool(3);

// Submit tasks to the thread pool
for (int i = 1; i <= 5; i++) {
int taskId = i;
threadPool.submit(() -> {
System.out.println("Task " + taskId + " is starting on thread " + Thread.currentThread().getName());
try {
Thread.sleep(1000); // Simulate task execution time
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Task " + taskId + " completed.");
});
}

// Shutdown the thread pool after tasks complete
threadPool.shutdown();
}
}

Explanation of Sample Usage Code​

  1. Creating a Thread Pool: A SimpleThreadPool instance with 3 threads is created.
  2. Submitting Tasks: Five tasks are submitted to the pool, each of which simulates a brief work period.
  3. Shutting Down the Pool: After submitting the tasks, the pool is gracefully shut down using shutdown().

Expected Output​

Task 1 is starting on thread Thread-0
Task 2 is starting on thread Thread-1
Task 3 is starting on thread Thread-2
Task 1 completed.
Task 2 completed.
Task 3 completed.
Task 4 is starting on thread Thread-0
Task 5 is starting on thread Thread-1
Task 4 completed.
Task 5 completed.

Conclusion​

This SimpleThreadPool is a demonstration of how a basic thread pool is structured and works:

  • Worker Threads repeatedly pick up and execute tasks from a shared queue.
  • Synchronization between task producers and worker threads prevents busy-waiting.
  • Graceful Shutdown lets worker threads exit safely after processing all queued tasks.

This POC lays the foundation for understanding and extending Java thread pool management by illustrating how resources are managed and synchronized in a concurrent environment.


Custom Java Thread Pool with Separate Task Class

Separating the Task as an individual class will enhance the code's modularity and flexibility, making it easier to reuse or extend for specific task-related features. Let's modify the SimpleThreadPool implementation to include an external Task class.

Here’s the updated implementation:

  1. SimpleThreadPool: Manages worker threads and the task queue.
  2. Task: Represents individual tasks, implementing Runnable.
  3. PoolWorker: Represents the worker threads that pick up tasks from the queue and execute them.

Task Class Implementation​

We’ll define Task as a standalone class that implements Runnable. Each task can have custom logic.

public class Task implements Runnable {
private final int taskId;

public Task(int taskId) {
this.taskId = taskId;
}

@Override
public void run() {
System.out.println("Task " + taskId + " is starting on thread " + Thread.currentThread().getName());
try {
Thread.sleep(1000); // Simulate task execution time
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // Handle interruption
}
System.out.println("Task " + taskId + " completed.");
}
}
  • taskId: Each task is given an ID to identify it during execution.
  • run() Method: Defines the task's actions, including a simulated delay.

SimpleThreadPool Implementation​

The SimpleThreadPool class manages the task queue and distributes tasks among worker threads.

import java.util.LinkedList;
import java.util.Queue;

public class SimpleThreadPool {
private final int poolSize;
private final PoolWorker[] workers;
private final Queue<Runnable> taskQueue;
private boolean isShutdown = false;

public SimpleThreadPool(int poolSize) {
this.poolSize = poolSize;
taskQueue = new LinkedList<>();
workers = new PoolWorker[poolSize];

// Initialize and start each worker thread
for (int i = 0; i < poolSize; i++) {
workers[i] = new PoolWorker();
workers[i].start();
}
}

// Method to submit a task to the thread pool
public synchronized void submit(Runnable task) {
if (!isShutdown) {
taskQueue.add(task);
notify(); // Notify a waiting worker thread about the new task
} else {
throw new IllegalStateException("ThreadPool is shut down and cannot accept tasks");
}
}

// Initiates shutdown and interrupts all waiting worker threads
public synchronized void shutdown() {
isShutdown = true;
for (PoolWorker worker : workers) {
worker.interrupt(); // Interrupts workers waiting for tasks
}
}

// Worker class that executes tasks from the queue
private class PoolWorker extends Thread {
public void run() {
while (true) {
Runnable task;
synchronized (SimpleThreadPool.this) {
while (taskQueue.isEmpty() && !isShutdown) {
try {
SimpleThreadPool.this.wait(); // Wait for a new task
} catch (InterruptedException e) {
return; // Exit if interrupted during shutdown
}
}
if (isShutdown && taskQueue.isEmpty()) {
return; // Exit if shutdown and no tasks are left
}
task = taskQueue.poll(); // Fetch a task
}
try {
task.run(); // Execute the task outside the synchronized block
} catch (RuntimeException e) {
System.err.println("Task execution failed: " + e.getMessage());
}
}
}
}
}

Sample Code to Use the Custom Thread Pool with Task​

This example demonstrates submitting Task instances to the SimpleThreadPool.

public class SimpleThreadPoolExample {
public static void main(String[] args) {
// Create a thread pool with 3 threads
SimpleThreadPool threadPool = new SimpleThreadPool(3);

// Submit tasks to the thread pool
for (int i = 1; i <= 5; i++) {
Task task = new Task(i); // Create a new Task with a unique ID
threadPool.submit(task); // Submit the Task to the thread pool
}

// Shutdown the thread pool after tasks are complete
threadPool.shutdown();
}
}

Explanation of Sample Usage​

  1. Task Creation: Each Task object has a unique ID, making it easy to track execution progress.
  2. Task Submission: Tasks are added to the SimpleThreadPool’s task queue.
  3. Shutdown: Once all tasks are submitted, shutdown() stops the pool after finishing queued tasks.

Expected Output​

Task 1 is starting on thread Thread-0
Task 2 is starting on thread Thread-1
Task 3 is starting on thread Thread-2
Task 1 completed.
Task 2 completed.
Task 3 completed.
Task 4 is starting on thread Thread-0
Task 5 is starting on thread Thread-1
Task 4 completed.
Task 5 completed.

This implementation improves modularity by allowing different Task implementations and extends the reusability of SimpleThreadPool. Each component (Task, PoolWorker, and SimpleThreadPool) is designed for flexible use in varied production scenarios.


Wait but what happened to the terminated thread?

To demonstrate the behavior where a terminated thread in a thread pool is repurposed and moves back to the runnable state when a new task is added, we need to tweak the SimpleThreadPool to simulate this behavior.

Typically, when threads in a pool complete their tasks, they wait for new tasks rather than fully terminating (they don't reach the "TERMINATED" state in Java's built-in ThreadPoolExecutor). However, for demonstration, I’ll create logic that represents threads reaching a "stopped" state and being reactivated for new tasks.

Let's add a mechanism to track the states and illustrate the thread reusability behavior.


Enhanced SimpleThreadPool Implementation​

Below is an enhanced version of SimpleThreadPool with thread state tracking. We’ll add logging and state simulation to show when threads are "terminated" and later reactivated when new tasks arrive.

import java.util.LinkedList;
import java.util.Queue;

public class SimpleThreadPool {
private final int poolSize;
private final PoolWorker[] workers;
private final Queue<Runnable> taskQueue;
private boolean isShutdown = false;

public SimpleThreadPool(int poolSize) {
this.poolSize = poolSize;
taskQueue = new LinkedList<>();
workers = new PoolWorker[poolSize];

// Initialize and start each worker thread
for (int i = 0; i < poolSize; i++) {
workers[i] = new PoolWorker(i + 1); // Assign a unique ID to each worker
workers[i].start();
}
}

// Method to submit a task to the thread pool
public synchronized void submit(Runnable task) {
if (!isShutdown) {
taskQueue.add(task);
notify(); // Notify a waiting worker thread about the new task
} else {
throw new IllegalStateException("ThreadPool is shut down and cannot accept tasks");
}
}

// Initiates shutdown and interrupts all waiting worker threads
public synchronized void shutdown() {
isShutdown = true;
for (PoolWorker worker : workers) {
worker.interrupt(); // Interrupts workers waiting for tasks
}
}

// Worker class that executes tasks from the queue
private class PoolWorker extends Thread {
private final int workerId;
private boolean isTerminated = false;

public PoolWorker(int id) {
this.workerId = id;
}

public void run() {
while (true) {
Runnable task;
synchronized (SimpleThreadPool.this) {
while (taskQueue.isEmpty() && !isShutdown) {
System.out.println("Worker-" + workerId + " is waiting for a task...");
try {
SimpleThreadPool.this.wait(); // Wait for a new task
} catch (InterruptedException e) {
System.out.println("Worker-" + workerId + " is terminated.");
isTerminated = true;
return; // Exit if interrupted during shutdown
}
}
if (isShutdown && taskQueue.isEmpty()) {
System.out.println("Worker-" + workerId + " is terminated as pool shuts down.");
isTerminated = true;
return; // Exit if shutdown and no tasks are left
}
task = taskQueue.poll(); // Fetch a task
}

try {
isTerminated = false; // Worker is active
System.out.println("Worker-" + workerId + " is executing task.");
task.run(); // Execute the task
System.out.println("Worker-" + workerId + " completed task and goes back to waiting.");
} catch (RuntimeException e) {
System.err.println("Task execution failed in Worker-" + workerId + ": " + e.getMessage());
}
}
}
}
}

Enhanced Task Class Implementation​

The Task class remains mostly the same, but we can add unique identifiers to differentiate tasks.

public class Task implements Runnable {
private final int taskId;

public Task(int taskId) {
this.taskId = taskId;
}

@Override
public void run() {
System.out.println("Task " + taskId + " is starting on thread " + Thread.currentThread().getName());
try {
Thread.sleep(1000); // Simulate task execution time
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // Handle interruption
}
System.out.println("Task " + taskId + " completed.");
}
}

Usage and State Transition Simulation​

Let's add a usage example showing how to submit tasks, observe workers entering a “terminated” state, and then reactivating to process new tasks.

public class ThreadPoolExample {
public static void main(String[] args) throws InterruptedException {
// Create a thread pool with 2 workers
SimpleThreadPool threadPool = new SimpleThreadPool(2);

// Submit initial tasks to the thread pool
for (int i = 1; i <= 3; i++) {
threadPool.submit(new Task(i));
}

// Wait to let initial tasks complete
Thread.sleep(3000);

// Submit additional tasks to show reactivation
System.out.println("Adding more tasks to test thread reactivation...");
for (int i = 4; i <= 5; i++) {
threadPool.submit(new Task(i));
}

// Give some time for tasks to be picked up
Thread.sleep(3000);

// Shutdown the thread pool
threadPool.shutdown();
}
}

Explanation of the Workflow and State Transitions​

  1. Initial Task Submission: Three tasks are submitted, and two workers pick them up immediately since the pool size is 2.
  2. Worker Reactivation: After the initial tasks complete, we add two more tasks, which are picked up by waiting workers.
  3. Shutdown Process: Finally, we shut down the thread pool, and the workers gracefully terminate after completing their tasks.

Expected Output (Sample)​

Worker-1 is waiting for a task...
Worker-2 is waiting for a task...
Worker-1 is executing task.
Task 1 is starting on thread Worker-1
Worker-2 is executing task.
Task 2 is starting on thread Worker-2
Task 1 completed.
Worker-1 completed task and goes back to waiting.
Task 2 completed.
Worker-2 completed task and goes back to waiting.
Worker-1 is executing task.
Task 3 is starting on thread Worker-1
Task 3 completed.
Worker-1 completed task and goes back to waiting.
Worker-1 is waiting for a task...
Worker-2 is waiting for a task...

Adding more tasks to test thread reactivation...
Worker-1 is executing task.
Task 4 is starting on thread Worker-1
Worker-2 is executing task.
Task 5 is starting on thread Worker-2
Task 4 completed.
Worker-1 completed task and goes back to waiting.
Task 5 completed.
Worker-2 completed task and goes back to waiting.
Worker-1 is terminated as pool shuts down.
Worker-2 is terminated as pool shuts down.

This output demonstrates that the workers remain active, go back to a waiting state after completing tasks, and terminate only after shutdown() is called.

How thread being actually reused?

In a typical thread pool implementation, once a thread completes its task, it does not transition to the "TERMINATED" state and then back to the "RUNNABLE" state. Instead, the thread remains in a waiting state (often referred to as "idle" or "waiting for work") until a new task is available. This waiting state is different from the "TERMINATED" state, which indicates that the thread has finished execution and cannot be reused.

Here’s a more detailed explanation:

Thread States in a Thread Pool​

  1. NEW: The thread has been created but not yet started.
  2. RUNNABLE: The thread is executing in the JVM.
  3. BLOCKED: The thread is blocked waiting for a monitor lock.
  4. WAITING: The thread is waiting indefinitely for another thread to perform a particular action.
  5. TIMED_WAITING: The thread is waiting for another thread to perform an action for up to a specified waiting time.
  6. TERMINATED: The thread has completed execution.

In a thread pool, threads typically cycle between the RUNNABLE and WAITING states:

  • RUNNABLE: When a thread is actively executing a task.
  • WAITING: When a thread is idle, waiting for a new task to be added to the queue.

How Threads Are Reused​

When a thread completes its task, it checks the task queue for new tasks. If no tasks are available, the thread waits (using wait() or a similar mechanism) until a new task is submitted. This waiting state is not the same as the "TERMINATED" state. A thread in the "TERMINATED" state cannot be reused; it must be recreated, which is inefficient.

Example of Thread Reuse in SimpleThreadPool​

In the SimpleThreadPool example, threads are reused as follows:

  1. Thread Creation: When the thread pool is created, a fixed number of worker threads are started.
  2. Task Execution: Each worker thread fetches a task from the queue and executes it.
  3. Waiting for New Tasks: After completing a task, the worker thread checks the queue for new tasks. If no tasks are available, it waits (wait()) until a new task is submitted.
  4. Reactivation: When a new task is submitted, the waiting threads are notified (notify()), and one of them wakes up to execute the new task.

Code Example​

Here’s a simplified version of the SimpleThreadPool to illustrate this:

import java.util.LinkedList;
import java.util.Queue;

public class SimpleThreadPool {
private final int poolSize;
private final PoolWorker[] workers;
private final Queue<Runnable> taskQueue;
private boolean isShutdown = false;

public SimpleThreadPool(int poolSize) {
this.poolSize = poolSize;
taskQueue = new LinkedList<>();
workers = new PoolWorker[poolSize];

// Start worker threads
for (int i = 0; i < poolSize; i++) {
workers[i] = new PoolWorker();
workers[i].start();
}
}

public synchronized void submit(Runnable task) {
if (!isShutdown) {
taskQueue.add(task);
notify(); // Notify a waiting worker thread
} else {
throw new IllegalStateException("ThreadPool is shut down");
}
}

public synchronized void shutdown() {
isShutdown = true;
for (PoolWorker worker : workers) {
worker.interrupt(); // Interrupt each worker to stop waiting
}
}

private class PoolWorker extends Thread {
public void run() {
while (true) {
Runnable task;
synchronized (SimpleThreadPool.this) {
while (taskQueue.isEmpty() && !isShutdown) {
try {
SimpleThreadPool.this.wait(); // Wait for a task
} catch (InterruptedException ignored) {
return; // Exit if interrupted during shutdown
}
}
if (isShutdown && taskQueue.isEmpty()) {
return; // Exit if shutdown and no tasks are left
}
task = taskQueue.poll(); // Take task from queue
}
try {
task.run(); // Execute task
} catch (RuntimeException e) {
System.err.println("Task execution failed: " + e.getMessage());
}
}
}
}
}

Key Points​

  • Thread Reuse: Threads do not transition to the "TERMINATED" state after completing a task. Instead, they wait for new tasks.
  • Task Queue: The task queue holds tasks that are waiting to be executed. When a new task is submitted, a waiting thread is notified and picks up the task.
  • Graceful Shutdown: The shutdown() method sets the isShutdown flag to stop accepting new tasks and interrupts all worker threads to ensure they exit if they are waiting.

Summary​

In Java’s ThreadPoolExecutor (or our custom pool example), the worker threads do not actually terminate after completing a task. Instead, they enter a waiting or idle state and wait to pick up the next task. The main goal of a thread pool is to reuse threads efficiently, keeping them alive for as long as the pool is running and ready to transition back to the runnable state whenever a new task becomes available.

Here's how it works:

  1. Runnable → Running: A thread in the pool becomes running when it starts executing a task.
  2. Running → Waiting/Idle: Once it finishes the task, it doesn't terminate. Instead, it returns to an idle state within the pool, waiting for more tasks to be added to the queue.
  3. Waiting → Runnable: When a new task is submitted, an idle worker transitions to runnable and executes the task.

The threads only reach a terminated state when the pool itself is shut down (e.g., calling shutdown() in ThreadPoolExecutor). Only then will the threads stop, exit the idle/waiting state, and go to terminated. This approach keeps the pool flexible and efficient, reducing the overhead of creating and destroying threads for every task.

So in short: threads in a thread pool do not reach terminated after completing tasks. They go back to idle/waiting and can accept new tasks. And threads in a thread pool do not go back to the "RUNNABLE" state from the "TERMINATED" state neither. Instead, they remain in a waiting state until new tasks are available, ensuring efficient reuse of thread resources.

Deep Dive into Avro and Parquet Formats

· 16 min read

Introduction​

As data systems grow in complexity and scale, choosing efficient data serialization formats becomes crucial. This article provides a deep dive into Apache Avro and Apache Parquet, two popular formats in the Big Data ecosystem. Both formats have distinct characteristics, making them suitable for different types of data and use cases.


Understanding Avro Format​

Apache Avro is a data serialization format that facilitates the encoding of complex data structures into a compact, binary format, making it highly efficient for streaming and storage.

Avro Schema​

The Avro schema describes the structure of data stored in Avro format and is defined using JSON. Avro schemas allow the specification of nested fields, and each schema comprises two parts:

  • Primitive types: null, boolean, int, long, float, double, bytes, and string.
  • Complex types: record, enum, array, map, union, and fixed.

Example Avro Schema:

{
"type": "record",
"name": "Employee",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": null},
{"name": "salary", "type": "double"}
]
}

In this schema:

  • The Employee record has fields id, name, email, and salary.
  • The email field is optional, allowing null values.

Avro Schema in Depth​

Apache Avro is a data serialization framework with a focus on compact storage, schema evolution, and interoperability, especially popular in data pipelines with tools like Apache Kafka, Hadoop, and Spark. Avro schema is a JSON-based definition that describes the structure, types, and constraints of the data. This schema allows for efficient serialization and deserialization, making it versatile for data storage and streaming applications.

Avro’s schema format is highly adaptable, supporting both simple and complex data structures and allowing schema evolution without disrupting data compatibility. This makes Avro particularly suited for big data pipelines and environments where schemas may need frequent updates. Its JSON-based structure is easy to read and interpret while enabling efficient storage and retrieval in binary format, ideal for high-performance data processing.

Here’s an in-depth look at Avro schemas:

1. Schema Basics​

Avro schemas are written in JSON and define the data types and structure. A schema specifies:

  • Type: The data type, like string, int, float, or complex types like record and array.
  • Name: The name of the field (only required for named types like record).
  • Namespace: Optional, used to provide a unique context for named types, helpful in managing type names in larger systems.
  • Fields: Defines each field within complex types, like record.

Example of a simple Avro schema:

{
"type": "record",
"name": "User",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": "string"}
]
}

This schema defines a User record with three fields: id, name, and email.

2. Primitive Data Types​

Avro supports several primitive data types:

  • Null: Represents a null value.
  • Boolean: Represents true or false.
  • Int: 32-bit signed integer.
  • Long: 64-bit signed integer.
  • Float: 32-bit IEEE floating-point number.
  • Double: 64-bit IEEE floating-point number.
  • Bytes: Sequence of bytes, useful for binary data.
  • String: Unicode character sequence.

3. Complex Data Types​

Avro supports complex data types to handle more advanced data structures:

  • Record: Similar to a structured data object or table row; contains fields, each with its own name and type.
  • Enum: Defines a list of allowed values.
  • Array: An ordered collection of values of a specified type.
  • Map: A collection of key-value pairs with string keys and values of a specified type.
  • Union: Allows fields to have multiple types, which supports optional fields and complex structures.
  • Fixed: Represents a fixed-size binary value, useful for things like fixed-length IDs.

Example of complex types:

{
"type": "record",
"name": "Person",
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": ["null", "int"], "default": null},
{"name": "emails", "type": {"type": "array", "items": "string"}},
{"name": "address", "type": {
"type": "record",
"name": "Address",
"fields": [
{"name": "street", "type": "string"},
{"name": "city", "type": "string"}
]
}}
]
}

This schema defines a Person record with an optional age, a list of emails, and a nested Address record.

4. Union Types​

Unions in Avro schemas allow a field to hold one of multiple types, helping to represent optional fields:

  • A union is represented by an array of types, such as ["null", "int"] for a nullable integer.
  • The first type in the union is considered the default.

Example:

{"name": "birthdate", "type": ["null", "string"], "default": null}

Here, birthdate is either null or a string, allowing it to be optional.

5. Default Values​

  • Fields can have default values if they are missing in the data, which is particularly useful when evolving schemas.
  • Default values must match the field type and are defined within the schema.

Example:

{"name": "isActive", "type": "boolean", "default": true}

6. Schema Evolution​

Avro is designed to handle schema evolution, meaning new schemas can be applied to existing data files without breaking compatibility:

  • Backward Compatibility: New schema can read old data (e.g., adding new fields with default values).
  • Forward Compatibility: Old schema can read new data (e.g., adding optional fields).
  • Full Compatibility: Both backward and forward compatible.

Schema evolution rules include:

  • Adding a new field with a default value is backward-compatible.
  • Removing a field is backward-compatible if it is not required by consumers.
  • Changing a field type requires careful consideration to ensure compatibility.

7. Named Types and Namespaces​

In Avro, named types like record and enum require unique names, often scoped by namespaces:

  • Name: The unique identifier for a type.
  • Namespace: Provides context to prevent naming conflicts, similar to a package in Java or a module in Python.

8. Schema Example with Namespaces​

Example using namespace for managing complex data:

{
"namespace": "com.example",
"type": "record",
"name": "Customer",
"fields": [
{"name": "id", "type": "int"},
{"name": "contact", "type": {
"type": "record",
"name": "ContactInfo",
"fields": [
{"name": "email", "type": "string"},
{"name": "phone", "type": ["null", "string"], "default": null}
]
}}
]
}

Schema Registry​

Schema Registry is a crucial component that allows the storage and retrieval of schemas used by producers and consumers of data. It ensures that producers and consumers agree on the data structure, supporting schema evolution and compatibility checks.

Diagram: Schema Registry Workflow

+-----------+ +--------------------+
| Producer | Write | Schema Registry |
| | ----------> | |
+-----------+ +--------------------+
|
|
| Write
v
+-----------+ +-----------------+
| Data | Read/Write | Consumer |
| Storage | <--------------> | |
+-----------+ +-----------------+
  • Producers register the schema before sending data to storage.
  • Consumers fetch the schema from the registry to deserialize and understand the data.

Storage Architecture for Avro in HDFS/S3​

When storing Avro data in HDFS or S3, Avro files contain both the schema and data, allowing each file to be fully self-contained and readable independently.

Diagram: Avro Storage in HDFS/S3

+------------------------+
| HDFS or S3 Storage |
+------------------------+
|
+------------------------+
| Avro File |
| + Schema |
+------------------------+
|
+------------------------+
| Blocks |
+------------------------+
  • Each Avro file contains schema and data blocks, making it easily portable across storage platforms.
  • Data Access: Avro files can be read without external dependencies, as each file includes the required schema.

Sample Code in Spark​

Below is a Spark example that reads data from an Avro file, processes it, and writes it back to HDFS as Avro format.

import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.avro._

// Initialize Spark session
val spark = SparkSession.builder
.appName("AvroExample")
.getOrCreate()

// Define the Avro schema as JSON
val avroSchema = """
{
"type": "record",
"name": "Employee",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": null},
{"name": "salary", "type": "double"}
]
}
"""

// Read Avro file from HDFS
val df: DataFrame = spark.read
.format("avro")
.load("hdfs://path/to/input/avro/file")

// Process the DataFrame (example: filter employees with salary > 5000)
val filteredDf = df.filter("salary > 5000")

// Write back to HDFS in Avro format
filteredDf.write
.format("avro")
.option("avroSchema", avroSchema)
.save("hdfs://path/to/output/avro/file")

Exploring Parquet Format​

Apache Parquet is a columnar storage format optimized for complex nested data structures, making it an ideal choice for big data processing.

Parquet Schema and Compression​

Parquet organizes data in columns, allowing for efficient compression and encoding. This format is particularly beneficial for queries that scan specific columns.

Compression Techniques:

  • Dictionary Encoding: Reduces data size by replacing duplicate values with codes.
  • Bit Packing: Optimizes storage by packing multiple values into smaller data units.

Diagram: Parquet File Structure

+------------------------------+
| Parquet |
+------------------------------+
| Row Group 1 |
| +--------------------+ |
| | Column 1 | |
| | Column 2 | |
| | ... | |
| +--------------------+ |
| |
| Row Group 2 |
| +--------------------+ |
| | Column 1 | |
| | Column 2 | |
| | ... | |
| +--------------------+ |
+------------------------------+
  • Each Row Group contains columns stored together, optimizing compression and query performance.
  • Columnar storage allows quick access to specific fields, reducing read and write times.

Parquet Schema​

Parquet’s schema defines the structure, types, and layout of the data stored in the file, making it highly efficient for analytics and queries. This schema is stored in the file’s metadata and allows query engines to interpret the data without needing to read the entire file, which speeds up operations significantly.

The Parquet schema plays a crucial role in its columnar storage efficiency, managing data sparsity, and enabling rich, nested data structures with low storage overhead. It facilitates schema evolution, where additional columns or fields can be added without rewriting the entire file, and supports advanced filtering and optimizations for large data analytics tasks.

Here’s an overview of the main components of the Parquet schema:

1. Schema Hierarchy and Structure​

Parquet schema is organized hierarchically and supports complex nested structures:

  • Message Type: This is the root element in a Parquet schema, defining the data structure at the highest level.
  • Fields: Each field in the schema has a name, data type, and potentially nested subfields if the data structure is complex. Parquet supports:
    • Primitive Types: such as int32, int64, boolean, float, double, and binary.
    • Nested Types: Parquet can store complex structures like lists, maps, and structs, which are stored as nested types in the schema.

For example, a Parquet schema could look like:

message schema {
required int32 id;
optional group address {
required binary street (UTF8);
optional int32 zip_code;
}
}

This schema describes a record with an id and an optional nested address group, which contains a street and an optional zip_code.

2. Column Definition: Required, Optional, Repeated​

Each field in the Parquet schema has a repetition level:

  • Required: Every row must contain a value for this field.
  • Optional: A field that can be present or absent in each row.
  • Repeated: Used to model lists or repeated elements, allowing multiple values for the field in a row.

3. Definition and Repetition Levels​

Parquet’s schema model is hierarchical and optimized for efficient storage and retrieval in columnar format. It uses Definition Levels and Repetition Levels to handle nullability, optional fields, and nested data structures. Let’s walk through an example to show how these levels work.

To manage complex data structures and sparsity, Parquet uses definition levels and repetition levels:

  • Definition Levels track whether each level in the hierarchy has data or is null, it tells us how deep in the structure a field is defined.
  • Repetition Levels track whether an element is the first instance or a repeated instance in nested lists or repeated fields, it tells us if the field is part of a repeated structure and, if so, how deep the repeat goes.

For instance, if a row has a list of addresses, the repetition levels can indicate the position of each address within the list, while definition levels show whether each element is present or missing.

Example Parquet Schema​

Consider the following JSON structure, which could be a nested data schema for a Parquet file:

{
"name": "Alice",
"contacts": [
{
"type": "email",
"address": "alice@example.com"
},
{
"type": "phone",
"number": null
}
],
"addresses": [
{
"street": "123 Main St",
"city": "Wonderland"
}
]
}

Here’s the corresponding Parquet schema:

message Person {
required binary name (UTF8);
repeated group contacts {
required binary type (UTF8);
optional binary address (UTF8);
optional binary number (UTF8);
}
repeated group addresses {
required binary street (UTF8);
required binary city (UTF8);
}
}
Parquet Structure: Definition and Repetition Levels​
  1. Definition Level: Indicates the presence or absence of values at each level in a schema. This level is used to define nullability and optional fields.
  2. Repetition Level: Indicates if a field is part of a repeated structure (e.g., a list or an array). This is useful for nested or repeated structures, telling the reader if a given level repeats within the hierarchy.

Let’s break down how these levels work for each field in the example above:

1. Field: name​
  • Definition Level: 1 (required field, so it's always defined)
  • Repetition Level: 0 (not repeated)
2. Field: contacts​
  • Definition Level: 1 to 2, depending on the presence of address and number fields.
    • If both address and number are null, the definition level is lower.
  • Repetition Level: 1 (this field is repeated since contacts is a list)
3. Fields Inside contacts: type, address, number​
  • type
    • Definition Level: 2 (required field within the repeated group)
    • Repetition Level: 1
  • address and number
    • Definition Level: 2 or 3 (optional fields)
      • A definition level of 2 means the contacts object is present but address or number is null.
      • A definition level of 3 indicates both fields are filled.
    • Repetition Level: 1 (repeated inside contacts)
4. Field: addresses​
  • Definition Level: 1 to 2
  • Repetition Level: 1 (repeated field)
5. Fields Inside addresses: street, city​
  • Both street and city are required within the repeated addresses group.
  • Definition Level: 2
  • Repetition Level: 1 (both are repeated with each addresses group entry)
Example Encoding with Levels​

Assuming data with these values:

{
"name": "Alice",
"contacts": [
{
"type": "email",
"address": "alice@example.com",
"number": null
},
{
"type": "phone",
"number": "123-456-7890"
}
],
"addresses": [
{
"street": "123 Main St",
"city": "Wonderland"
}
]
}

The encoded levels might look like this:

FieldValueDefinition LevelRepetition Level
name"Alice"10
contacts[0].type"email"21
contacts[0].address"alice@example.com"31
contacts[0].numbernull21
contacts[1].type"phone"21
contacts[1].addressnull21
contacts[1].number"123-456-7890"31
addresses[0].street"123 Main St"21
addresses[0].city"Wonderland"21

These levels help Parquet efficiently handle complex nested and repeated data structures, enabling better data compression and faster querying by only storing relevant levels and values for each entry.

4. Logical Types and Annotations​

Parquet supports logical types that provide additional context for primitive types:

  • Logical Types: These are type annotations like UTF8 for text strings, DATE for dates, and DECIMAL for high-precision decimals.
  • Logical types help map data from systems that use richer data types (like SQL databases) to Parquet’s primitive types.

Example:

optional int32 birthdate (DATE);

5. Column Index and Row Group Metadata​

  • Column Index: Parquet files include metadata about each column, such as min/max values, null counts, and distinct counts. This allows query engines to skip entire blocks if they don't match the filter criteria.
  • Row Group Metadata: Parquet stores data in row groups (blocks of rows), each containing column data for that group. Each row group stores metadata like the number of rows and data size, which helps with efficient reads.

Example of a Parquet Schema in JSON Representation​

A Parquet schema can also be represented in JSON format to show the hierarchy and structure clearly. For instance:

{
"name": "schema",
"type": "message",
"fields": [
{
"name": "id",
"type": "INT32",
"repetition_type": "REQUIRED"
},
{
"name": "address",
"type": "group",
"repetition_type": "OPTIONAL",
"fields": [
{
"name": "street",
"type": "BINARY",
"logicalType": "UTF8",
"repetition_type": "REQUIRED"
},
{
"name": "zip_code",
"type": "INT32",
"repetition_type": "OPTIONAL"
}
]
}
]
}

In this JSON schema:

  • The id field is required.
  • The address field is optional and has two nested fields: a required street and an optional zip_code.

Storage Architecture for Parquet in HDFS/S3​

In HDFS/S3 storage, Parquet files are organized in row groups with each column stored independently within those groups.

Diagram: Parquet Storage in HDFS/S3

+------------------------+
| HDFS or S3 Storage |
+------------------------+
|
+------------------------+
| Parquet File |
+------------------------+
|
+------------------------+
| Row Groups |
+------------------------+
|
+------------------------+
| Columns within Groups |
+------------------------+
  • Row Groups: Partitioned sections that allow for distributed reads.
  • Columns: Stored separately within each row group for efficient access.

Sample Code in Spark​

Below is a Spark example that reads data from a Parquet file, performs transformations, and writes it back to HDFS as Parquet format.

import org.apache.spark.sql.{DataFrame, SparkSession}

// Initialize Spark session
val spark = SparkSession.builder
.appName("ParquetExample")
.getOrCreate()

// Read Parquet file from HDFS
val df: DataFrame = spark.read
.parquet("hdfs://path/to/input/parquet/file")

// Process the DataFrame (example: select employees with salary > 5000)
val filteredDf = df.filter("salary > 5000")

// Write back to HDFS in Parquet format
filteredDf.write
.parquet("hdfs://path/to/output/parquet/file")

Conclusion​

This article provided an in-depth look at Avro and Parquet formats, exploring their unique characteristics, schema management, and storage architectures. With examples in Spark, we demonstrated practical implementations in a real-world production environment, highlighting the versatility of both formats for big data solutions.

Deep Dive into Avro and Parquet Together

· 14 min read

Introduction​

In modern data architectures, Avro data is often used in messaging systems like Apache Kafka due to its compact binary format and schema evolution support. However, in data lakes or data warehouses, Parquet is preferred for storage due to its columnar format and efficient read performance. This article covers the architecture, pipeline design, and real-life sample code for implementing a pipeline that ingests Avro data from a messaging system and saves it in Parquet format in a big data storage solution like Amazon S3.


Architecture Overview​

This section provides an overview of the components in a typical architecture that ingests Avro from a messaging system and saves it as Parquet in big data storage.

Diagram: Avro to Parquet Pipeline Architecture

Messaging System (Kafka) ----> Stream Processing (Spark) ----> S3 Storage (Parquet Format)

### Avro and Parquet Formats

  • Avro: A row-based binary format, commonly used for serializing data in messaging systems due to its compact size and schema evolution capabilities.
  • Parquet: A columnar storage format optimized for reading specific columns, making it highly efficient for analytics and big data storage solutions.

### Role of Messaging Systems

Messaging systems like Apache Kafka are widely used to transport streaming data across systems in real time. Kafka provides fault-tolerance and durability, ensuring data is safely delivered to downstream applications for processing.

### Storage in Big Data Storage Solutions

Big data storage solutions, such as Amazon S3, offer durable and cost-effective storage for large-scale data. Storing data in Parquet format on S3 enables easy integration with data analytics frameworks like Spark, Presto, and Hive.


Data Pipeline Design​

The pipeline includes the following key stages:

  1. Data Ingestion: Consuming Avro-encoded messages from Kafka.
  2. Processing and Transformation: Converting Avro data to Parquet format.
  3. Storage: Saving the transformed Parquet files to Amazon S3.

Ingesting Data from Messaging System​

The first step in the pipeline is ingesting data from a messaging system like Kafka. Spark Structured Streaming can be used to consume Avro-encoded messages from Kafka topics in real-time.

Diagram: Ingesting Avro Data from Kafka

Kafka Topic (Avro Messages) ---> Spark Structured Streaming ---> Data Transformation

Processing and Converting Avro to Parquet​

Spark Structured Streaming allows us to process and transform the incoming Avro data. To convert Avro to Parquet, the schema needs to be mapped, and the data is then serialized to the Parquet format.

Diagram: Converting Avro to Parquet in Spark

Avro Data (Spark DataFrame) ---> Schema Mapping ---> Parquet Data

Saving to Big Data Storage (S3)​

Once converted to Parquet, the data can be written to Amazon S3 in a structured format. Parquet files in S3 are organized by partitions (e.g., by date or another logical grouping) to optimize retrieval and querying performance.

Diagram: Saving to S3

Parquet Data (Partitioned by Date) ---> Amazon S3 Bucket ---> Analytics-ready Storage

Understanding the Structural Differences​

In Avro and Parquet formats, the data organization is fundamentally different:

  • Avro: A row-based format where each record is serialized as a single entity, storing all fields of each row together.
  • Parquet: A columnar storage format where data is grouped by columns rather than rows, optimizing for efficient reads on specific columns (ideal for analytical workloads).

To illustrate, let’s consider an Avro record with two attributes:

{
"name": "John Doe",
"age": 30
}

In Avro, the above record would be stored as a single contiguous block of binary data, with fields appearing in the order defined by the schema.

When converting this record to Parquet, the storage layout changes to a column-oriented format. Parquet organizes data by columns:

  • Each column (e.g., "name" and "age") is stored separately in column chunks.
  • Parquet organizes records into row groups, where each row group has column chunks for each column defined in the schema.

Diagram: Avro to Parquet Conversion for Record { "name": "John Doe", "age": 30 }

Parquet Column ChunkData Stored
Name Column ChunkJohn Doe
Age Column Chunk30

In a more extensive dataset with thousands of records, Parquet’s layout significantly optimizes read performance because each column chunk can be accessed independently. For instance, if only the "age" column is required for analysis, Parquet reads only the relevant chunk, skipping over unrelated columns like "name."


Avro to Parquet Conversion​

The steps for converting Avro to Parquet in a system like Apache Spark generally follow this sequence:

  1. Avro Data Loading:

    • Avro-encoded data is parsed and loaded into an intermediate in-memory structure, such as a DataFrame in Spark.
    • Avro schema is either read from the Avro file (or topic in case of Kafka) or supplied separately.
  2. Schema Mapping:

    • Spark maps the Avro schema to an equivalent internal schema (e.g., Spark schema), which is essential to support Parquet’s columnar structure.
    • Fields in the Avro schema (like "name" and "age") are mapped to equivalent columns.
  3. Data Conversion:

    • Each Avro record is then serialized into a format compatible with Parquet’s column-oriented layout.
    • Each attribute is transformed to be written in column chunks rather than row-by-row.
  4. Data Writing (Parquet):

    • Parquet organizes data into row groups, and within each row group, column data is stored in contiguous chunks.
    • Additional metadata (e.g., min, max, dictionary encoding) is recorded for each column chunk, which is useful for optimizations in querying.

Example: In Spark, a sample code to convert Avro to Parquet may look like this:

// Load Avro data into a DataFrame
Dataset<Row> avroDF = spark.read().format("avro").load("path/to/avro/file");

// Write data to Parquet format
avroDF.write().parquet("path/to/output/parquet");

This process reorganizes the data from a row-based layout (Avro) to a columnar layout (Parquet).


Reading from Parquet and Converting Back to Avro​

When reading Parquet data and converting it back to Avro, the system reverses the process, transforming columnar data back into row-based data.

Steps:

  1. Load Parquet Data:

    • Parquet files are read into memory. In a Spark-based setup, this would involve loading Parquet files into a DataFrame.
  2. Column Extraction:

    • Parquet retrieves data by accessing each relevant column chunk. Parquet's metadata allows efficient navigation to the specific data of interest.
  3. Row Assembly:

    • As each column’s data is accessed, the system assembles rows by gathering data from each column chunk.
    • The row-based structure compatible with Avro is recreated by combining values from each column for every row.
  4. Schema Mapping:

    • Parquet columns are mapped to Avro fields based on a schema. This schema alignment ensures that columns match Avro’s row-oriented requirements.
    • The DataFrame (or equivalent data structure) is serialized back into the Avro format.
  5. Write to Avro:

    • The row-based structure is serialized in Avro’s binary format, and the schema is either embedded or referenced as needed.

Example Code: Converting Parquet to Avro in Spark

// Load Parquet data into a DataFrame
Dataset<Row> parquetDF = spark.read().parquet("path/to/parquet/file");

// Write data back to Avro format
parquetDF.write().format("avro").save("path/to/output/avro");

This code converts the columnar Parquet format back into Avro’s row-based format.


Parquet’s Row Groups and Column Chunks​

In Parquet, data is stored in row groups. Each row group contains data for a set number of rows, and within each row group, data is stored in column chunks (one for each column). The row group is Parquet’s key structural element that ensures the association between column data across multiple rows:

  • Row Group: A set of rows is organized as a contiguous data block. Each row group in Parquet represents a batch of rows from the original dataset.
  • Column Chunk: Within each row group, each column’s data is stored in a separate column chunk.

For example, if we have a dataset with 10,000 records, it might be divided into several row groups (say 5,000 rows per row group). Each row group then contains the data for the columns name and age in its respective column chunks.

Ensuring Data Belongs to the Same Record​

To ensure data from each column belongs to the same record, Parquet relies on row ordering within each row group. The ordering within each column chunk of a row group matches across all columns:

  • Ordering: Each row in a row group corresponds to the same index across all column chunks within that row group. For instance, the first value in the name column chunk of a row group corresponds to the first value in the age column chunk for the same row group.
  • Offset Metadata: Parquet uses metadata about the offsets within each column chunk to ensure each row can be reconstructed correctly when reading.

So, in your example, the row group might look like this:

Row IndexName Column ChunkAge Column Chunk
Row 1"John Doe"30
Row 2"Jane Smith"25
Row 3"Alice Brown"28

Each entry in the name and age column chunks corresponds by index. Parquet uses metadata within the row group and column chunks to indicate where each row begins and ends, making sure that during reads, each row’s data across columns is aligned.

Reading from Parquet and Converting Back to Avro​

When reading Parquet data back into a row-based format (like Avro), the system iterates over each row group:

  • Row Assembly: For each row in a row group, Parquet retrieves the respective values from each column chunk based on their row index. This reassembles each row’s fields together to form the original record.
  • Schema Mapping: The column metadata stored in Parquet includes the schema, allowing the correct assignment of values back to each field in the record.

What if Null Values in Parquet​

In Parquet, each column chunk within a row group holds data in columnar format, and Parquet efficiently handles cases where some rows may lack a value for a given column by using specific encoding and metadata, rather than empty placeholders. Here’s how it manages the scenario:

1. Null Values in Columns​

If a column lacks a value for a particular row, Parquet does not insert an empty placeholder. Instead:

  • Definition Levels are used to indicate whether a value is present or null. Each level specifies if a field has a value or is missing at that row.
  • Repetition Levels track nested structures within rows, so nested nulls and repeated fields can be handled in one scan.

2. Encodings for Efficient Storage​

Parquet uses encodings like Run-Length Encoding (RLE) and Dictionary Encoding to represent repeated or missing values compactly:

  • Run-Length Encoding groups consecutive null values (or any repeating values) into a single representation, saving space when large chunks are missing in a column.
  • Dictionary Encoding can efficiently map common values (like repeated nulls or default values) using dictionary lookup tables.

3. Metadata Tracking and Optional Fields​

  • If a field is defined as optional in the schema, it allows a row to be saved without a value for that field. Parquet only includes these rows in the row group metadata, which helps query engines like Spark identify missing values.

Example Scenario​

Let’s say we have a Parquet file with three rows and three columns, where one column lacks values for some rows.

Row GroupColumn AColumn BColumn C
Row 1Value 1NullValue 3
Row 2Value 4Value 5Null
Row 3NullValue 6Value 7

In this case:

  • Definition Levels for Column B and Column C would show which rows contain nulls.
  • The data structure itself would store only the non-null values, reducing storage without explicit placeholders.

This approach is particularly useful when handling sparse data, as it minimizes the storage footprint and increases reading efficiency.

Example in Spark​

Here’s how Spark handles this under the hood when reading Parquet data and converting it back to Avro:

// Reading from Parquet
Dataset<Row> parquetDF = spark.read().parquet("path/to/parquet/file");

// Writing back to Avro, reassembling rows
parquetDF.write().format("avro").save("path/to/output/avro");

During this process, Spark ensures that each row’s values across columns are correctly aligned by leveraging Parquet’s row group and column chunk metadata. This alignment allows it to convert back to Avro’s row-based format without losing the association between name and age in each record.


Real-Life Production Sample Code in Spark​

The following sample code demonstrates how to implement this pipeline in Apache Spark with Structured Streaming.

Reading from Kafka​

Use Spark to read Avro data from a Kafka topic in real-time.

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.avro.functions.from_avro;

public class AvroToParquetPipeline {

public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.appName("Avro to Parquet Pipeline")
.getOrCreate();

String kafkaBootstrapServers = "kafka-broker1:9092,kafka-broker2:9092";
String kafkaTopic = "avro-topic";

Dataset<Row> avroDF = spark
.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootstrapServers)
.option("subscribe", kafkaTopic)
.load()
.selectExpr("CAST(value AS BINARY) as avroData");

// Define the Avro schema (can be fetched from Schema Registry if available)
String avroSchema = "{ \"type\": \"record\", \"name\": \"User\", \"fields\": [ {\"name\": \"id\", \"type\": \"int\"}, {\"name\": \"name\", \"type\": \"string\"}, {\"name\": \"age\", \"type\": \"int\"} ] }";

Dataset<Row> parsedDF = avroDF
.select(from_avro(avroDF.col("avroData"), avroSchema).as("data"))
.select("data.*");

parsedDF.printSchema(); // Debugging schema after parsing Avro
}
}

Converting Avro to Parquet​

Once we have parsed the Avro data into a DataFrame, we can convert it into Parquet format.

Dataset<Row> parquetDF = parsedDF.repartition(1); // Optional: Customize partitioning as needed

Writing to S3​

Writing the data to Amazon S3 requires specifying the destination path and configuring partitioning if necessary.

import org.apache.spark.sql.streaming.Trigger;

String s3Path = "s3a://your-bucket/path/to/destination/";

parquetDF.writeStream()
.format("parquet")
.option("path", s3Path)
.option("checkpointLocation", "s3a://your-bucket/path/to/checkpoint/")
.partitionBy("date") // Optional: Partitioning by a specific column (e.g., date)
.trigger(Trigger.ProcessingTime("5 minutes")) // Controls batch frequency
.start()
.awaitTermination();

Conclusion​

This article covered an architecture for ingesting Avro data from a messaging system, converting it to Parquet, and saving it in Amazon S3. Using Spark Structured Streaming and Amazon S3 for storage, this pipeline is a robust solution for managing Avro-to-Parquet transformations in big data applications. Understanding the structural differences between Avro and Parquet, as well as the conversion process, enables data engineers to design efficient and scalable data pipelines.