Skip to main content

Deep Dive into Spring Boot Multi-Threading

· 7 min read

Introduction to Spring Boot Multi-Threading

Spring Boot provides robust support for multi-threading, enabling developers to build highly concurrent and scalable applications. Leveraging multi-threading in Spring Boot can significantly improve the performance and responsiveness of your application. This article delves into best practices, annotation-based configuration, and real-life production scenarios to help you master multi-threading in Spring Boot.


Best Practices for Spring Boot Multi-Threading

  1. Use @Async for Asynchronous Processing: The @Async annotation allows methods to run in a separate thread, making it easy to implement asynchronous processing.
  2. Configure Thread Pools: Use TaskExecutor to configure thread pools with specific characteristics like core pool size, max pool size, and queue capacity.
  3. Handle Exceptions Gracefully: Implement exception handling mechanisms to manage exceptions that occur in asynchronous tasks.
  4. Monitor Thread Pools: Use Spring Boot Actuator to monitor thread pool metrics and ensure optimal performance.
  5. Avoid Blocking Operations: Ensure that asynchronous methods do not perform blocking operations, which can degrade performance.

Annotation-Based Configuration

Spring Boot provides several annotations to configure and manage multi-threading:

  • @EnableAsync: Enables asynchronous processing in the application.
  • @Async: Marks a method as asynchronous, meaning it will run in a separate thread.
  • @Configuration: Indicates that a class declares one or more @Bean methods and may be processed by the Spring container to generate bean definitions.
  • @Bean: Indicates that a method produces a bean to be managed by the Spring container.

Diagram of Multi-Threading in Spring Boot

+-------------------------+
| Spring Boot Application |
| - @EnableAsync |
| - @Configuration |
| - @Bean |
+-------------------------+
|
v
+-------------------+
| TaskExecutor |
| - corePoolSize |
| - maxPoolSize |
| - queueCapacity |
+-------------------+
|
v
+-------------------+
| @Async Methods |
| - Method1 |
| - Method2 |
+-------------------+
|
v
+-------------------+
| Thread Pool |
| - Thread 1 |
| - Thread 2 |
| - Thread 3 |
+-------------------+
|
v
+-------------------+
| Task Execution |
| - Task 1 |
| - Task 2 |
| - Task 3 |
+-------------------+

Sample Code with Detailed Comments

Configuration Class​

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;

@Configuration
@EnableAsync // Enable asynchronous processing
public class AsyncConfig {

@Bean(name = "taskExecutor")
public Executor taskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(10); // Core pool size
executor.setMaxPoolSize(20); // Maximum pool size
executor.setQueueCapacity(100); // Queue capacity
executor.setThreadNamePrefix("AsyncThread-"); // Thread name prefix
executor.initialize();
return executor;
}
}

Service Class​

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

@Service
public class AsyncService {

@Async("taskExecutor") // Use the taskExecutor bean for this method
public void performAsyncTask(String taskName) {
System.out.println("Starting task: " + taskName + " on thread: " + Thread.currentThread().getName());
try {
// Simulate task execution time
Thread.sleep(2000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Completed task: " + taskName + " on thread: " + Thread.currentThread().getName());
}
}

Controller Class​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class AsyncController {

@Autowired
private AsyncService asyncService;

@GetMapping("/performTask")
public String performTask(@RequestParam String taskName) {
asyncService.performAsyncTask(taskName);
return "Task " + taskName + " has been submitted for asynchronous processing.";
}
}

Application Class​

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableAsync;

@SpringBootApplication
@EnableAsync // Enable asynchronous processing
public class AsyncApplication {

public static void main(String[] args) {
SpringApplication.run(AsyncApplication.class, args);
}
}

Complex Scenario 1: Flow Control and Backpressure Handling

In high-throughput systems, it's crucial to manage the flow of tasks to avoid overwhelming the system. Backpressure handling ensures that the system can gracefully handle a large number of incoming tasks.

Configuration Class​

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;

@Configuration
@EnableAsync
public class BackpressureConfig {

@Bean(name = "backpressureExecutor")
public Executor backpressureExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5); // Core pool size
executor.setMaxPoolSize(10); // Maximum pool size
executor.setQueueCapacity(50); // Queue capacity
executor.setThreadNamePrefix("BackpressureThread-"); // Thread name prefix
executor.initialize();
return executor;
}
}

Service Class​

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

@Service
public class BackpressureService {

@Async("backpressureExecutor")
public void handleTask(String taskName) {
System.out.println("Handling task: " + taskName + " on thread: " + Thread.currentThread().getName());
try {
// Simulate task execution time
Thread.sleep(3000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Completed handling task: " + taskName + " on thread: " + Thread.currentThread().getName());
}
}

Controller Class​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class BackpressureController {

@Autowired
private BackpressureService backpressureService;

@GetMapping("/handleTask")
public String handleTask(@RequestParam String taskName) {
backpressureService.handleTask(taskName);
return "Task " + taskName + " has been submitted for backpressure handling.";
}
}

Complex Scenario 2: Task Prioritization

In some scenarios, certain tasks may need to be prioritized over others. Spring Boot allows you to configure task executors with priority queues to handle such cases.

Configuration Class​

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;
import java.util.concurrent.PriorityBlockingQueue;

@Configuration
@EnableAsync
public class PriorityConfig {

@Bean(name = "priorityExecutor")
public Executor priorityExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5); // Core pool size
executor.setMaxPoolSize(10); // Maximum pool size
executor.setQueueCapacity(50); // Queue capacity
executor.setThreadNamePrefix("PriorityThread-"); // Thread name prefix
executor.setTaskDecorator(new PriorityTaskDecorator()); // Set task decorator for priority
executor.initialize();
return executor;
}
}

Task Decorator​

import org.springframework.core.task.TaskDecorator;

public class PriorityTaskDecorator implements TaskDecorator {

@Override
public Runnable decorate(Runnable runnable) {
return () -> {
// Set priority logic here
System.out.println("Setting priority for task: " + runnable);
runnable.run();
};
}
}

Service Class​

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

@Service
public class PriorityService {

@Async("priorityExecutor")
public void prioritizeTask(String taskName) {
System.out.println("Prioritizing task: " + taskName + " on thread: " + Thread.currentThread().getName());
try {
// Simulate task execution time
Thread.sleep(3000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Completed prioritizing task: " + taskName + " on thread: " + Thread.currentThread().getName());
}
}

Controller Class​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class PriorityController {

@Autowired
private PriorityService priorityService;

@GetMapping("/prioritizeTask")
public String prioritizeTask(@RequestParam String taskName) {
priorityService.prioritizeTask(taskName);
return "Task " + taskName + " has been submitted for prioritization.";
}
}

Complex Scenario 3: Handling Long-Running Tasks

Long-running tasks can block threads and degrade system performance. Spring Boot provides mechanisms to handle such tasks efficiently.

Configuration Class​

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;

@Configuration
@EnableAsync
public class LongRunningConfig {

@Bean(name = "longRunningExecutor")
public Executor longRunningExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5); // Core pool size
executor.setMaxPoolSize(10); // Maximum pool size
executor.setQueueCapacity(50); // Queue capacity
executor.setThreadNamePrefix("LongRunningThread-"); // Thread name prefix
executor.initialize();
return executor;
}
}

Service Class​

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

@Service
public class LongRunningService {

@Async("longRunningExecutor")
public void handleLongRunningTask(String taskName) {
System.out.println("Handling long-running task: " + taskName + " on thread: " + Thread.currentThread().getName());
try {
// Simulate long-running task execution time
Thread.sleep(10000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Completed handling long-running task: " + taskName + " on thread: " + Thread.currentThread().getName());
}
}

Controller Class​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class LongRunningController {

@Autowired
private LongRunningService longRunningService;

@GetMapping("/handleLongRunningTask")
public String handleLongRunningTask(@RequestParam String taskName) {
longRunningService.handleLongRunningTask(taskName);
return "Long-running task " + taskName + " has been submitted for processing.";
}
}

Conclusion

Spring Boot's support for multi-threading allows developers to build highly concurrent and scalable applications. By following best practices and leveraging annotation-based configuration, you can efficiently manage asynchronous tasks, handle backpressure, prioritize tasks, and manage long-running tasks. This article provides a comprehensive guide to mastering multi-threading in Spring Boot, complete with detailed sample code and real-life production scenarios.

Microservice meet with Spring Cloud and Kubernetes

· 13 min read

Introduction​

The objective is to design and deploy a group of microservices: Profile Service, Inventory Service, Order Service, Shipment Service, and an OAuth2 Service for centralized authentication. Key requirements include robust service discovery, API Gateway setup, secure communication, and efficient load balancing for high availability and fault tolerance in Kubernetes.

High-Level Architecture​

Each microservice will be deployed in a Kubernetes cluster with multiple replicas, leveraging Spring Cloud Kubernetes Discovery for service registration and Spring Cloud LoadBalancer for client-side load balancing. An API Gateway will handle external requests, providing routing, load balancing, and JWT-based authentication. Communication between services is secured with JWT tokens generated by the OAuth2 Service.

Service Design​

Each microservice is structured independently, deployed as a standalone Spring Boot application with individual responsibilities and access rules. Here’s an overview of the key services:

1. Profile Service​

Manages customer profiles, providing endpoints for CRUD operations on profile data. Accessible to services like Order and Shipment for customer information retrieval.

2. Inventory Service​

Tracks inventory levels and provides endpoints to update, retrieve, and manage inventory data. Called by the Order Service for real-time inventory status updates.

3. Order Service​

Processes customer orders, calculates totals, and interfaces with Inventory and Profile Services. This service is central to the architecture, relying on other services for a smooth customer experience.

4. Shipment Service​

Manages shipment tracking and logistics for orders, called by the Order Service once an order is processed and confirmed.

5. OAuth2 Service​

Acts as the authentication provider, issuing and validating JWT tokens for secure communication. The API Gateway uses these tokens to authenticate requests.

Order-Service Auth-Service Inventory-Service
| | |
|--- Request JWT ------------->| |
| | |
|<-- Receive JWT --------------| |
| | |
|--- Call Inventory ---------->| |
| (with JWT) | |
| |--- Validate JWT------------>|
| | |
| |<-- JWT Valid/Invalid--------|
| | |
|<-- Receive Inventory Data -- | |

Kubernetes Deployment and Service Discovery​

Each service will be deployed to Kubernetes as a Deployment with multiple replicas and exposed via a Service to enable discovery. Spring Cloud Kubernetes is used for service registration and discovery.

Configuring Kubernetes Service Discovery for Each Microservice​

To set up service discovery in Kubernetes, add the following dependencies to your pom.xml:

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-discovery</artifactId>
</dependency>

Configure the application.yml in each service to enable discovery:

spring:
cloud:
kubernetes:
discovery:
enabled: true #Enable service discovery
config:
enabled: true
loadbalancer:
enabled: true #Enable loadbalancer

This configuration registers each service with Kubernetes' internal DNS, allowing them to be discoverable by service name within the namespace. This way, each microservice is registered under its own Kubernetes service name, and can resolve other services (e.g., http://inventory-service:8080) when communicating with them. This configuration, combined with Kubernetes DNS, helps maintain an efficient, loosely coupled architecture.

Note: The Kubernetes Service Discovery configuration should indeed be included in each individual microservice’s configuration to ensure they can all register themselves with the Kubernetes DNS and be discoverable by other services in the namespace. This setup allows each microservice to dynamically locate and communicate with other services via Kubernetes' internal DNS naming conventions.

Why Configure Service Discovery in Each Microservice?

Each microservice is an independently deployed unit, so configuring Spring Cloud Kubernetes Discovery in each service's application.yml allows it to:

  1. Register with Kubernetes DNS under a unique service name, enabling internal routing across services.
  2. Access other services by simply using their service names, as Kubernetes DNS provides a consistent way to locate services within the cluster.

Note Here you might be noticed that both the discovery and load balancer components are configured within each microservice. This design allows each microservice to dynamically discover the current instances of other services (using the spring-cloud-kubernetes-discovery integration) and balance requests among those instances using the spring-cloud-loadbalancer component.

Service Registration​

Each microservice is deployed as a separate Deployment and Service in Kubernetes. Here’s an example of the inventory-service Kubernetes configuration:

To deploy and register microservices like an Inventory Service in Kubernetes, we use Spring Cloud’s Kubernetes Discovery capabilities, allowing services to register with Kubernetes’ DNS-based service registry automatically.

Step 1: Create a Kubernetes Service and Deployment for each microservice:

YAML files for defining the service and deployment:

# inventory-service-deployment.yml
apiVersion: v1
kind: Service
metadata:
name: inventory-service
spec:
selector:
app: inventory
ports:
- protocol: TCP
port: 8080
targetPort: 8080
type: ClusterIP # Internal access within the cluster

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: inventory-deployment
spec:
replicas: 2 # Number of instances (pods) of the Inventory Service
selector:
matchLabels:
app: inventory
template:
metadata:
labels:
app: inventory
spec:
containers:
- name: inventory
image: myregistry/inventory-service:latest
ports:
- containerPort: 8080

This configuration deploys inventory-service with two replicas, registered under the service name inventory-service. Other services can communicate with it via http://inventory-service:8080.

More comprehensive explainations When deploying an inventory-deployment with two replicas, you will indeed see two pods running in parallel in the specified namespace. Kubernetes names these pods based on the deployment name, followed by a unique suffix (not just sequential numbers), so you may see pod names like inventory-deployment-abc12 and inventory-deployment-def34.

In Kubernetes, these two pods are managed by a single inventory-service which acts as a layer of abstraction. When the client load balancer targets inventory-service, the service uses Kubernetes' built-in load balancing to forward requests to one of the available inventory-deployment pods.

How Kubernetes Service Load Balancing Works:

  1. Service Abstraction: The inventory-service has a Cluster IP that exposes the service within the cluster.
  2. Endpoints Selection: The service maintains an updated list of active pod IPs (called "endpoints") that match the deployment selector.
  3. Round-Robin Load Balancing: Kubernetes uses a round-robin approach by default. When a request reaches inventory-service, it selects one of the pod IPs associated with the deployment, balancing requests across pods automatically. The service doesn't keep state; each request is independently forwarded to one of the pod replicas.

Example Flow:

  1. Client Request: A client load balancer sends a request to the inventory-service endpoint.
  2. Service to Pod Routing: Kubernetes service routing picks an endpoint (pod IP) from the inventory-deployment pods and forwards the request to it.

For better control and observability, custom load balancer configurations can also be implemented to fine-tune how the requests are distributed, especially in setups requiring sticky sessions or advanced routing.

Step 2: Register Service with Kubernetes Discovery:

Spring Cloud Kubernetes automatically registers inventory-service based on the deployment labels. Enable Spring Cloud Discovery in your Spring Boot application via @EnableDiscoveryClient:

@EnableDiscoveryClient
@SpringBootApplication
public class InventoryServiceApplication {
public static void main(String[] args) {
SpringApplication.run(InventoryServiceApplication.class, args);
}
}

Step 3: Configure Discovery in application.yml:

spring:
cloud:
kubernetes:
discovery:
enabled: true
all-namespaces: true # Optional: enables discovery across namespaces

Load Balancing with Spring Cloud LoadBalancer​

To efficiently distribute incoming requests across available instances, Spring Cloud LoadBalancer provides a client-side load-balancing solution, automatically distributing traffic among instances of a service.

Enable Spring Cloud LoadBalancer by adding Spring Cloud LoadBalancer dependency:

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>

Configure a ServiceInstanceListSupplier bean for custom load balancing behavior if required:

import org.springframework.cloud.client.loadbalancer.ServiceInstanceListSupplier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class LoadBalancerConfig {

@Bean
public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier() {
return ServiceInstanceListSupplier.builder()
.withDiscoveryClient()
.withCaching()
.build();
}
}

Set Up Load Balancer with WebClient:

  • Configure a WebClient bean with load balancing to distribute requests among inventory-service instances.
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.context.annotation.Bean;
import org.springframework.web.reactive.function.client.WebClient;

@Configuration
public class WebClientConfig {

@Bean
@LoadBalanced
public WebClient.Builder loadBalancedWebClientBuilder() {
return WebClient.builder();
}
}

Use Load-Balanced WebClient in the calling service:

@Service
public class InventoryClient {

private final WebClient.Builder webClientBuilder;

public InventoryClient(WebClient.Builder webClientBuilder) {
this.webClientBuilder = webClientBuilder;
}

public Mono<Product> getProductById(String productId) {
return webClientBuilder.build()
.get()
.uri("http://inventory-service/products/{id}", productId)
.retrieve()
.bodyToMono(Product.class);
}
}

How Load Balancing Works:

  • Spring Cloud LoadBalancer, by default, uses a round-robin strategy to distribute requests.
  • It balances requests between the two inventory-service pods and automatically updates when new instances are added or removed.

  1. By default, Spring Cloud LoadBalancer will round-robin requests to different instances, or customize by implementing a custom LoadBalancerRequestTransformer.

API Gateway Setup​

The Spring Cloud API Gateway is deployed as a separate microservice, managing traffic and providing JWT-based authentication. The gateway routes requests to the appropriate services and load-balances among available replicas.

Routing and Load Balancing​

[External Client] --> [API Gateway]
|
| (Discovers routes from Kubernetes Discovery)
|
[inventory-service] <--+-- "lb://inventory-service"
[shipment-service] <--+-- "lb://shipment-service"
[order-service] <--+-- "lb://order-service"
[profile-service] <--+-- "lb://profile-service"
[other-service] <--+-- Discovered automatically

Configure the gateway’s application.yml to define routes to each service:

spring:
cloud:
gateway:
routes:
- id: profile
uri: http://profile-service
predicates:
- Path=/profile/**
- id: inventory
uri: http://inventory-service
predicates:
- Path=/inventory/**
default-filters:
- TokenRelay

The TokenRelay filter enables the gateway to forward JWT tokens to downstream services for authenticated requests.

Configuring JWT Authentication​

For security, JWT validation can be implemented in the gateway or within each service. Here’s how to configure JWT validation in the gateway:

  1. Add OAuth2 dependencies to the gateway:

    <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
    </dependency>
  2. Configure JWT validation in the gateway’s application.yml:

    spring:
    security:
    oauth2:
    resourceserver:
    jwt:
    issuer-uri: https://auth-server/oauth2/default

Microservice Communication​

Microservices communicate through Spring Cloud LoadBalancer and WebClient.

Using WebClient with Load Balancer​

To call the inventory-service from the order-service, configure WebClient with load balancing:

@Configuration
public class WebClientConfig {

@LoadBalanced
@Bean
public WebClient.Builder webClientBuilder() {
return WebClient.builder();
}
}

Example: Order Service Calls Profile Service​

Here’s how order-service might call profile-service:

@RestController
public class OrderController {

private final WebClient.Builder webClientBuilder;

@Autowired
public OrderController(WebClient.Builder webClientBuilder) {
this.webClientBuilder = webClientBuilder;
}

@GetMapping("/order/{orderId}")
public Mono<OrderDetails> getOrderDetails(@PathVariable String orderId) {
return webClientBuilder.build()
.get()
.uri("http://profile-service/profile/{orderId}", orderId) // The uri can be either http or lb(load balancer)
.retrieve()
.bodyToMono(OrderDetails.class);
}
}

Securing Microservices​

Secure Communication with mTLS and JWT

mTLS (mutual TLS) and JWT (JSON Web Token) are two primary ways to secure communication between microservices.

  • mTLS: Ensures that both the client and server authenticate each other, useful in a zero-trust network.
  • JWT: Adds stateless authentication, where each request includes a signed token with user and session data.

Each microservice validates JWT tokens received from the gateway, ensuring secure and authenticated communication.

JWT Authentication with API Gateway​

JWT Authentication is often managed at the API Gateway, validating tokens and authorizing requests before forwarding them to services.

In this approach, JWT validation is centralized in the gateway. Tokens are verified at the gateway and forwarded to services, reducing the validation load on individual services.

  1. Add Spring Security Dependency for JWT support:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
  1. Configure JWT Verification in Gateway:

Validate JWTs at the gateway level, allowing only authorized requests to reach downstream services.

@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter {

@Override
protected void configure(HttpSecurity http) throws Exception {
http
.authorizeRequests()
.antMatchers("/api/**").authenticated()
.and()
.oauth2ResourceServer().jwt();
}
}

Alternative: JWT Validation in Interceptors​

Alternatively, JWT validation can be handled within each microservice using a custom Interceptor, within services, add an Interceptor to validate JWTs before processing requests.

import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;

@Component
public class JwtInterceptor implements HandlerInterceptor {

@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
String token = request.getHeader("Authorization");
// Verify the JWT token
// Decode and validate claims
return isValidToken(token);
}
}

Pros and Cons of JWT and mTLS​

ApproachProsCons
mTLSHigh security with mutual authentication.Requires certificate management, adds complexity.
JWTStateless, easily scalable, widely supported.Token expiration handling; replay attacks if not carefully implemented.

<a name="kubernetes-deployment-and-auto-scaling"></a>
## Kubernetes Deployment and Auto-scaling

Scaling services dynamically is fundamental in Kubernetes. Kubernetes provides an effective way to handle **auto-scaling** of microservices based on resource utilization, e.g. CPU, memory, or custom metrics.

Each microservice is balanced using **Spring Cloud LoadBalancer**, ensuring even distribution across replicas.

> **Step 1**: Configuring Horizontal Pod Autoscaler (HPA)

1. Enable auto-scaling in Kubernetes based on CPU usage by defining an HPA resource.

**inventory-hpa.yaml**
```yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: inventory-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: inventory-deployment
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 75
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50

Step 2: Implementing Cluster Scaling

Use Kubernetes’ Cluster Autoscaler for scaling nodes as workload demand grows, ensuring optimal resource use without manual intervention.


Ensuring Secure Microservice Communication​

Securing communication between services is critical. Use mTLS (mutual TLS) and JWT tokens for service-to-service authentication within Kubernetes clusters.

Step 1: Configure Mutual TLS (mTLS)​

Istio or Linkerd can help enforce mTLS between services:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: your-namespace
spec:
mtls:
mode: STRICT

Summary​

  1. Use ConfigMaps and Secrets for configuration.
  2. Implement autoscaling with HPA.
  3. Secure communication with mTLS and JWT.
  4. Monitor with readiness/liveness probes and Prometheus/Grafana.

This guide provides an architecture blueprint for deploying secure, scalable, and efficient microservices in Kubernetes using Spring Cloud. The API Gateway simplifies external access, service discovery, and JWT-based authentication, while Kubernetes ensures high availability and resilience across services.

Programming Standards of Java Concurrency

· 5 min read

Thread safety is a critical aspect of building robust and scalable Java applications. Ensuring that your application handles concurrent operations correctly can prevent many common issues such as race conditions, deadlocks, and data inconsistencies. This article provides best practices and detailed guidelines for achieving thread safety in Java applications.


1. Thread Safety in Singleton Objects

  • Requirement: Ensure that singleton objects and their methods are thread-safe.
  • Explanation: Resource-driven classes, utility classes, and singleton factory classes must be thread-safe to prevent race conditions.

2. Meaningful Thread Naming

  • Requirement: Assign meaningful names to threads and thread pools for easier debugging.
  • Example: Custom thread factory with grouping based on external features (e.g., datacenter ID).
public class UserThreadFactory implements ThreadFactory {
private final String namePrefix;
private final AtomicInteger nextId = new AtomicInteger(1);

UserThreadFactory(String whatFeaturOfGroup) {
namePrefix = "From UserThreadFactory's " + whatFeaturOfGroup + "-Worker-";
}

@Override
public Thread newThread(Runnable task) {
String name = namePrefix + nextId.getAndIncrement();
Thread thread = new Thread(null, task, name, 0, false);
System.out.println(thread.getName());
return thread;
}
}

3. Thread Pool Usage

  • Requirement: Use thread pools instead of creating threads manually.
  • Explanation: Thread pools reduce the overhead of thread creation and destruction, preventing resource exhaustion and excessive context switching.

4. Explicit ThreadPoolExecutor Configuration

  • Requirement: Use ThreadPoolExecutor instead of Executors for creating thread pools.
  • Explanation: Executors can lead to resource exhaustion due to unbounded queues and thread creation.
ThreadPoolExecutor executor = new ThreadPoolExecutor(
corePoolSize, maxPoolSize, keepAliveTime, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(queueCapacity),
new UserThreadFactory("Datacenter-1")
);

5. Thread-Safe Date Formatting

  • Requirement: Use thread-safe date formatting.
  • Example: Use ThreadLocal for SimpleDateFormat or switch to DateTimeFormatter in JDK 8.
private static final ThreadLocal<DateFormat> df = ThreadLocal.withInitial(() -> new SimpleDateFormat("yyyy-MM-dd"));

6. ThreadLocal Variable Cleanup

  • Requirement: Clean up ThreadLocal variables, especially in thread pools.
  • Example: Use try-finally blocks to ensure cleanup.
objectThreadLocal.set(userInfo);
try {
// ...
} finally {
objectThreadLocal.remove();
}

7. Locking Performance Considerations

  • Requirement: Minimize the performance impact of locks.
  • Explanation: Prefer lock-free data structures, lock only necessary code blocks, and use object-level locks instead of class-level locks.

8. Consistent Locking Order

  • Requirement: Maintain consistent locking order to prevent deadlocks.
  • Explanation: Ensure that all threads acquire locks in the same order to avoid circular wait conditions.

9. Exception Handling in Locking

  • Requirement: Handle exceptions properly to avoid lock leaks.
  • Example: Ensure that locks are released even if exceptions occur.
Lock lock = new ReentrantLock();
boolean isLocked = lock.tryLock();
if (isLocked) {
try {
doSomething();
doOthers();
} finally {
lock.unlock();
}
}

10. Optimistic and Pessimistic Locking

  • Requirement: Use appropriate locking strategies based on conflict probability.
  • Explanation: Use optimistic locking for low conflict scenarios and pessimistic locking for high conflict scenarios.

11. ScheduledExecutorService for Timed Tasks

  • Requirement: Use ScheduledExecutorService instead of Timer for concurrent timed tasks.
  • Explanation: Timer can terminate all tasks if one throws an uncaught exception.

12. CountDownLatch for Synchronization

  • Requirement: Use CountDownLatch for synchronization in multi-threaded tasks.
  • Explanation: Ensure that all threads call countDown and handle exceptions to avoid blocking the main thread.

13. Random Instance Management

  • Requirement: Avoid sharing Random instances across threads.
  • Example: Use ThreadLocalRandom in JDK 7+ or ensure each thread has its own Random instance.

14. Double-Checked Locking

  • Requirement: Use volatile for double-checked locking to avoid initialization issues.
  • Example: Declare the target variable as volatile.
private volatile Helper helper = null;

15. Volatile and Atomic Operations

  • Requirement: Use volatile for one-write-many-read scenarios and Atomic classes for multi-write scenarios.
  • Example: Use AtomicInteger or LongAdder for atomic operations.

16. HashMap Resize Risks

  • Requirement: Be cautious of HashMap resizing in high-concurrency scenarios.
  • Explanation: Resizing can lead to deadlocks and high CPU usage.

17. ThreadLocal Usage

  • Requirement: Use ThreadLocal with static variables carefully.
  • Explanation: ThreadLocal variables are shared within a thread but can cause issues if not managed properly.

By following these best practices, you can ensure that your Java application is thread-safe, scalable, and robust. These guidelines cover a wide range of scenarios, from basic thread safety to advanced concurrency patterns, helping you build high-performance applications.

Avro to Parquet Pipeline in Big Data Applications

· 5 min read

Introduction​

In modern data architectures, Avro data is often used in messaging systems like Apache Kafka due to its compact binary format and schema evolution support. However, in data lakes or data warehouses, Parquet is preferred for storage due to its columnar format and efficient read performance. This article covers the architecture, pipeline design, and real-life sample code for implementing a pipeline that ingests Avro data from a messaging system and saves it in Parquet format in a big data storage solution like Amazon S3.


Architecture Overview​

This section provides an overview of the components in a typical architecture that ingests Avro from a messaging system and saves it as Parquet in big data storage.

Diagram: Avro to Parquet Pipeline Architecture

Messaging System (Kafka) ----> Stream Processing (Spark) ----> S3 Storage (Parquet Format)

Avro and Parquet Formats​

  • Avro: A row-based binary format, commonly used for serializing data in messaging systems due to its compact size and schema evolution capabilities.
  • Parquet: A columnar storage format optimized for reading specific columns, making it highly efficient for analytics and big data storage solutions.

Role of Messaging Systems​

Messaging systems like Apache Kafka are widely used to transport streaming data across systems in real time. Kafka provides fault-tolerance and durability, ensuring data is safely delivered to downstream applications for processing.

Storage in Big Data Storage Solutions​

Big data storage solutions, such as Amazon S3, offer durable and cost-effective storage for large-scale data. Storing data in Parquet format on S3 enables easy integration with data analytics frameworks like Spark, Presto, and Hive.


Data Pipeline Design​

The pipeline includes the following key stages:

  1. Data Ingestion: Consuming Avro-encoded messages from Kafka.
  2. Processing and Transformation: Converting Avro data to Parquet format.
  3. Storage: Saving the transformed Parquet files to Amazon S3.

Ingesting Data from Messaging System​

The first step in the pipeline is ingesting data from a messaging system like Kafka. Spark Structured Streaming can be used to consume Avro-encoded messages from Kafka topics in real-time.

Diagram: Ingesting Avro Data from Kafka

Kafka Topic (Avro Messages) ---> Spark Structured Streaming ---> Data Transformation

Processing and Converting Avro to Parquet​

Spark Structured Streaming allows us to process and transform the incoming Avro data. To convert Avro to Parquet, the schema needs to be mapped, and the data is then serialized to the Parquet format.

Diagram: Converting Avro to Parquet in Spark

Avro Data (Spark DataFrame) ---> Schema Mapping ---> Parquet Data

Saving to Big Data Storage (S3)​

Once converted to Parquet, the data can be written to Amazon S3 in a structured format. Parquet files in S3 are organized by partitions (e.g., by date or another logical grouping) to optimize retrieval and querying performance.

Diagram: Saving to S3

Parquet Data (Partitioned by Date) ---> Amazon S3 Bucket ---> Analytics-ready Storage

Real-Life Production Sample Code in Spark​

The following sample code demonstrates how to implement this pipeline in Apache Spark with Structured Streaming.

Reading from Kafka​

Use Spark to read Avro data from a Kafka topic in real-time.

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.avro.functions.from_avro;

public class AvroToParquetPipeline {

public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.appName("Avro to Parquet Pipeline")
.getOrCreate();

String kafkaBootstrapServers = "kafka-broker1:9092,kafka-broker2:9092";
String kafkaTopic = "avro-topic";

Dataset<Row> avroDF = spark
.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootstrapServers)
.option("subscribe", kafkaTopic)
.load()
.selectExpr("CAST(value AS BINARY) as avroData");

// Define the Avro schema (can be fetched from Schema Registry if available)
String avroSchema = "{ \"type\": \"record\", \"name\": \"User\", \"fields\": [ {\"name\": \"id\", \"type\": \"int\"}, {\"name\": \"name\", \"type\": \"string\"}, {\"name\": \"age\", \"type\": \"int\"} ] }";

Dataset<Row> parsedDF = avroDF
.select(from_avro(avroDF.col("avroData"), avroSchema).as("data"))
.select("data.*");

parsedDF.printSchema(); // Debugging schema after parsing Avro
}
}

Converting Avro to Parquet​

Once we have parsed the Avro data into a DataFrame, we can convert it into Parquet format.

Dataset<Row> parquetDF = parsedDF.repartition(1); // Optional: Customize partitioning as needed

Writing to S3​

Writing the data to Amazon S3 requires specifying the destination path and configuring partitioning if necessary.

import org.apache.spark.sql.streaming.Trigger;

String s3Path = "s3a://your-bucket/path/to/destination/";

parquetDF.writeStream()
.format("parquet")
.option("path", s3Path)
.option("checkpointLocation", "s3a://your-bucket/path/to/checkpoint/")
.partitionBy("date") // Optional: Partitioning by a specific column (e.g., date)
.trigger(Trigger.ProcessingTime("5 minutes")) // Controls batch frequency
.start()
.awaitTermination();

Conclusion​

This article covered an architecture for ingesting Avro data from a messaging system, converting it to Parquet, and saving it in Amazon S3. Using Spark Structured Streaming and Amazon S3 for storage, this pipeline is a robust solution for managing Avro-to-Parquet transformations in big data applications.

Best Practices of Java Core Multi-Threading Programming

· 9 min read

Introduction​

Java's multi-threading capabilities are powerful, but they require careful handling to avoid common pitfalls such as race conditions, deadlocks, and excessive synchronization. This article provides a deep dive into best practices for Java core multi-threading programming, including detailed textual diagrams, explanations, and real-life production sample code.


Thread Safety​

Use Synchronization Wisely​

Synchronization ensures that only one thread can execute a synchronized block or method at a time. However, excessive synchronization can lead to performance bottlenecks.

+-------------------+ +-------------------+
| | | |
| Thread 1 |<----->| Shared Resource|
| | | |
+-------------------+ +-------------------+
| |
| |
v v
+-------------------+ +-------------------+
| | | |
| Thread 2 |<----->| Shared Resource|
| | | |
+-------------------+ +-------------------+
class Counter {
private int count = 0;

public synchronized void increment() {
count++;
}

public synchronized int getCount() {
return count;
}
}

public class Main {
public static void main(String[] args) throws InterruptedException {
Counter counter = new Counter();

Thread t1 = new Thread(() -> {
for (int i = 0; i < 1000; i++) {
counter.increment();
}
});

Thread t2 = new Thread(() -> {
for (int i = 0; i < 1000; i++) {
counter.increment();
}
});

t1.start();
t2.start();

t1.join();
t2.join();

System.out.println("Count: " + counter.getCount()); // Count: 2000
}
}

Avoid Excessive Synchronization​

Excessive synchronization can lead to performance issues. Use finer-grained locks or non-blocking algorithms where possible.

class FineGrainedCounter {
private int count = 0;
private final Object lock = new Object();

public void increment() {
synchronized (lock) {
count++;
}
}

public int getCount() {
synchronized (lock) {
return count;
}
}
}

Use Atomic Classes​

Atomic classes provide thread-safe operations on single variables without the need for explicit synchronization.

import java.util.concurrent.atomic.AtomicInteger;

class AtomicCounter {
private AtomicInteger count = new AtomicInteger(0);

public void increment() {
count.incrementAndGet();
}

public int getCount() {
return count.get();
}
}

public class Main {
public static void main(String[] args) throws InterruptedException {
AtomicCounter counter = new AtomicCounter();

Thread t1 = new Thread(() -> {
for (int i = 0; i < 1000; i++) {
counter.increment();
}
});

Thread t2 = new Thread(() -> {
for (int i = 0; i < 1000; i++) {
counter.increment();
}
});

t1.start();
t2.start();

t1.join();
t2.join();

System.out.println("Count: " + counter.getCount()); // Count: 2000
}
}

Thread Communication​

Use wait() and notify()​

Threads can communicate using wait(), notify(), and notifyAll() methods.

+-------------------+ +-------------------+
| | | |
| Producer |<----->| Consumer |
| | | |
+-------------------+ +-------------------+
| |
| |
v v
+-------------------+ +-------------------+
| | | |
| Shared Buffer |<----->| Shared Buffer |
| | | |
+-------------------+ +-------------------+
class SharedBuffer {
private int[] buffer = new int[10];
private int count = 0;

public synchronized void produce(int value) throws InterruptedException {
while (count == buffer.length) {
wait();
}
buffer[count++] = value;
notifyAll();
}

public synchronized int consume() throws InterruptedException {
while (count == 0) {
wait();
}
int value = buffer[--count];
notifyAll();
return value;
}
}

class Producer implements Runnable {
private SharedBuffer buffer;

public Producer(SharedBuffer buffer) {
this.buffer = buffer;
}

@Override
public void run() {
for (int i = 0; i < 100; i++) {
try {
buffer.produce(i);
System.out.println("Produced: " + i);
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}

class Consumer implements Runnable {
private SharedBuffer buffer;

public Consumer(SharedBuffer buffer) {
this.buffer = buffer;
}

@Override
public void run() {
for (int i = 0; i < 100; i++) {
try {
int value = buffer.consume();
System.out.println("Consumed: " + value);
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}

public class Main {
public static void main(String[] args) {
SharedBuffer buffer = new SharedBuffer();
new Thread(new Producer(buffer)).start();
new Thread(new Consumer(buffer)).start();
}
}

Avoid Busy Waiting​

Busy waiting is inefficient and should be avoided. Use wait() and notify() instead.

class BusyWaitExample {
private boolean flag = false;

public synchronized void setFlag(boolean flag) {
this.flag = flag;
notifyAll();
}

public synchronized void waitForFlag() throws InterruptedException {
while (!flag) {
wait();
}
}
}

Thread Pooling​

Use ExecutorService​

Thread pooling manages a pool of worker threads to execute tasks concurrently.

+-------------------+ +-------------------+
| | | |
| Task 1 |<----->| Thread Pool |
| | | |
+-------------------+ +-------------------+
| |
| |
v v
+-------------------+ +-------------------+
| | | |
| Task 2 |<----->| Worker Threads |
| | | |
+-------------------+ +-------------------+
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Main {
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(5);

for (int i = 0; i < 10; i++) {
executor.submit(() -> {
System.out.println("Thread " + Thread.currentThread().getName() + " is running");
});
}

executor.shutdown();
}
}

Custom Thread Pools​

Custom thread pools can be created using ThreadPoolExecutor.

import java.util.concurrent.Executors;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

public class Main {
public static void main(String[] args) {
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(5);

for (int i = 0; i < 10; i++) {
executor.submit(() -> {
System.out.println("Thread " + Thread.currentThread().getName() + " is running");
});
}

executor.shutdown();
try {
executor.awaitTermination(1, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}

Deadlock Prevention​

Lock Ordering​

Ensure that locks are acquired in a consistent order to prevent deadlocks.

+-------------------+ +-------------------+
| | | |
| Thread 1 |<----->| Resource A |
| | | |
+-------------------+ +-------------------+
| |
| |
v v
+-------------------+ +-------------------+
| | | |
| Thread 2 |<----->| Resource B |
| | | |
+-------------------+ +-------------------+
public class Main {
public static void main(String[] args) {
Object lock1 = new Object();
Object lock2 = new Object();

Thread t1 = new Thread(() -> {
synchronized (lock1) {
System.out.println("Thread 1: Holding lock 1...");
try { Thread.sleep(10); } catch (InterruptedException e) {}
System.out.println("Thread 1: Waiting for lock 2...");
synchronized (lock2) {
System.out.println("Thread 1: Holding lock 1 & 2...");
}
}
});

Thread t2 = new Thread(() -> {
synchronized (lock1) { // Acquire locks in the same order
System.out.println("Thread 2: Holding lock 1...");
try { Thread.sleep(10); } catch (InterruptedException e) {}
System.out.println("Thread 2: Waiting for lock 2...");
synchronized (lock2) {
System.out.println("Thread 2: Holding lock 1 & 2...");
}
}
});

t1.start();
t2.start();
}
}

Timeout Mechanisms​

Use timeouts to avoid waiting indefinitely for a lock.

import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

public class Main {
public static void main(String[] args) {
Lock lock1 = new ReentrantLock();
Lock lock2 = new ReentrantLock();

Thread t1 = new Thread(() -> {
try {
if (lock1.tryLock(1, TimeUnit.SECONDS)) {
System.out.println("Thread 1: Holding lock 1...");
try { Thread.sleep(10); } catch (InterruptedException e) {}
if (lock2.tryLock(1, TimeUnit.SECONDS)) {
System.out.println("Thread 1: Holding lock 1 & 2...");
lock2.unlock();
}
lock1.unlock();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
});

Thread t2 = new Thread(() -> {
try {
if (lock1.tryLock(1, TimeUnit.SECONDS)) {
System.out.println("Thread 2: Holding lock 1...");
try { Thread.sleep(10); } catch (InterruptedException e) {}
if (lock2.tryLock(1, TimeUnit.SECONDS)) {
System.out.println("Thread 2: Holding lock 1 & 2...");
lock2.unlock();
}
lock1.unlock();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
});

t1.start();
t2.start();
}
}

Concurrency Utilities​

Use Concurrent Collections​

Concurrent collections provide thread-safe data structures.

import java.util.concurrent.ConcurrentHashMap;

public class Main {
public static void main(String[] args) {
ConcurrentHashMap<String, Integer> map = new ConcurrentHashMap<>();

Thread t1 = new Thread(() -> {
for (int i = 0; i < 1000; i++) {
map.put("Key" + i, i);
}
});

Thread t2 = new Thread(() -> {
for (int i = 0; i < 1000; i++) {
map.put("Key" + i, i);
}
});

t1.start();
t2.start();

try {
t1.join();
t2.join();
} catch (InterruptedException e) {
e.printStackTrace();
}

System.out.println("Map size: " + map.size()); // Map size: 1000
}
}

Fork/Join Framework​

The Fork/Join framework is designed for tasks that can be recursively split into smaller tasks.

import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.RecursiveTask;

class SumTask extends RecursiveTask<Integer> {
private static final int THRESHOLD = 10;
private int[] array;
private int start;
private int end;

public SumTask(int[] array, int start, int end) {
this.array = array;
this.start = start;
this.end = end;
}

@Override
protected Integer compute() {
if (end - start <= THRESHOLD) {
int sum = 0;
for (int i = start; i < end; i++) {
sum += array[i];
}
return sum;
} else {
int mid = (start + end) / 2;
SumTask leftTask = new SumTask(array, start, mid);
SumTask rightTask = new SumTask(array, mid, end);
leftTask.fork();
int rightResult = rightTask.compute();
int leftResult = leftTask.join();
return leftResult + rightResult;
}
}
}

public class Main {
public static void main(String[] args) {
int[] array = new int[100];
for (int i = 0; i < array.length; i++) {
array[i] = i + 1;
}

ForkJoinPool pool = new ForkJoinPool();
int sum = pool.invoke(new SumTask(array, 0, array.length));
System.out.println("Sum: " + sum); // Sum: 5050
}
}

Thread Local Variables​

ThreadLocal variables allow each thread to have its own copy of a variable.

public class Main {
private static ThreadLocal<Integer> threadLocal = ThreadLocal.withInitial(() -> 0);

public static void main(String[] args) {
Thread t1 = new Thread(() -> {
threadLocal.set(1);
System.out.println("Thread 1: " + threadLocal.get());
});

Thread t2 = new Thread(() -> {
threadLocal.set(2);
System.out.println("Thread 2: " + threadLocal.get());
});

t1.start();
t2.start();
}
}

Example: Concurrent Web Server​

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

class RequestHandler implements Runnable {
private Socket clientSocket;

public RequestHandler(Socket clientSocket) {
this.clientSocket = clientSocket;
}

@Override
public void run() {
try {
// Handle the client request
System.out.println("Handling request from " + clientSocket.getInetAddress());
// Simulate processing time
Thread.sleep(1000);
clientSocket.getOutputStream().write("HTTP/1.1 200 OK\r\n\r\nHello, World!".getBytes());
clientSocket.close();
} catch (IOException | InterruptedException e) {
e.printStackTrace();
}
}
}

public class WebServer {
public static void main(String[] args) throws IOException {
ServerSocket serverSocket = new ServerSocket(8080);
ExecutorService executor = Executors.newFixedThreadPool(10);

while (true) {
Socket clientSocket = serverSocket.accept();
executor.submit(new RequestHandler(clientSocket));
}
}
}

Conclusion​

Java's multi-threading capabilities are powerful, but they require careful handling to avoid common pitfalls such as race conditions, deadlocks, and excessive synchronization. By following the best practices outlined in this article, you can write robust, high-performance Java applications that leverage the full potential of multi-threading. The provided real-life production sample code illustrates how these concepts can be applied in practical scenarios.


References​

  1. Java Concurrency in Practice
  2. Oracle Java Documentation
  3. Java Memory Model
  4. Fork/Join Framework
  5. Concurrent Collections

By mastering these concepts and practices, you can build robust, high-performance Java applications that leverage the full potential of multi-threading.

Best Practices of Java Core ThreadPoolExecutor

· 8 min read

1. Why Executors is Discouraged​

Java’s Executors class provides several factory methods to create various types of thread pools:

public static ExecutorService newSingleThreadExecutor()
public static ExecutorService newFixedThreadPool(int nThreads)
public static ExecutorService newCachedThreadPool()
public static ScheduledExecutorService newSingleThreadScheduledExecutor()
public static ScheduledExecutorService newScheduledThreadPool(int corePoolSize)

These convenience methods simplify thread pool creation, but they do not allow fine-tuning. Using Executors to create thread pools is generally discouraged due to potential risks, such as resource exhaustion. The official guidance is to use ThreadPoolExecutor directly, enabling full control over pool configurations, ensuring stability under varying load conditions.

public class ThreadDemo {
public static void main(String[] args) {
ExecutorService es = Executors.newSingleThreadExecutor();
}
}

Why it's discouraged: Creating a thread pool this way can lead to issues like uncontrolled resource usage. Best practice: use ThreadPoolExecutor instead.

2. ThreadPoolExecutor Introduction​

Java's ThreadPoolExecutor is a cornerstone of concurrent programming. It provides a high-performance, versatile API for managing threads and handling tasks, but it requires careful configuration to prevent resource wastage, improve system responsiveness, and enhance maintainability.


3. Overview of ThreadPoolExecutor​

The ThreadPoolExecutor manages thread allocation, scaling, and task queueing while optimizing thread lifecycle to minimize system overhead. Here is a basic textual model of a ThreadPoolExecutor lifecycle and memory model:

+-----------------------------+
| Task Queue |
| (BlockingQueue<Runnable>) |
+------------+----------------+
|
+-----------------v-----------------+
| |
| ThreadPoolExecutor |
| |
+----------------+------------------+
|
+----------------------+--------------------+
| Worker Threads |
| +-----------------------------------+ |
| | Worker Thread 1 | |
| |-----------------------------------| |
| | Worker Thread 2 | |
| |-----------------------------------| |
| | Worker Thread N | |
+-------------------------------------------+
  • Worker Threads: Execute tasks within a controlled environment.
  • Task Queue: Holds tasks awaiting execution, preventing unnecessary thread creation.
  • ThreadPoolExecutor: Controls the execution strategy, pool scaling, and task rejection policies.

To avoid risks, use ThreadPoolExecutor directly for creating thread pools:

public class ThreadDemo {
public static void main(String[] args) {
ExecutorService es = new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(10), Executors.defaultThreadFactory(),
new ThreadPoolExecutor.DiscardPolicy());
}
}

5. Core Configuration Parameters​

The ThreadPoolExecutor relies on several key configuration parameters:

  • corePoolSize: Minimum number of threads in the pool.
  • maximumPoolSize: Maximum threads allowed in the pool.
  • keepAliveTime: The time a thread can remain idle before termination if beyond corePoolSize.
  • workQueue: Queue holding tasks before they are assigned to threads.

Best Practice: Choose configuration values based on workload type and system capacity.​


6. Analyzing Executors Source Code​

The methods newSingleThreadExecutor(), newFixedThreadPool(int nThreads), and newCachedThreadPool() all internally use ThreadPoolExecutor. Below are the source code implementations from JDK 8:

public static ExecutorService newSingleThreadExecutor() {
return new FinalizableDelegatedExecutorService(
new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>())
);
}

public static ExecutorService newFixedThreadPool(int nThreads) {
return new ThreadPoolExecutor(nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>());
}

public static ExecutorService newCachedThreadPool() {
return new ThreadPoolExecutor(0, Integer.MAX_VALUE, 60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>());
}

All three methods use ThreadPoolExecutor but provide different default configurations. These static methods abstract parameters, which can lead to undesired behaviors under high loads.

7. ThreadPoolExecutor Constructors and Parameters​

When creating thread pools with ThreadPoolExecutor, initializing parameters must be specified. One of its constructors takes the following parameters:

public ThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit,
BlockingQueue<Runnable> workQueue, ThreadFactory threadFactory,
RejectedExecutionHandler handler)

Parameter Breakdown​

  • corePoolSize: Number of threads to keep in the pool.
  • maximumPoolSize: Maximum number of threads allowed.
  • keepAliveTime: Time limit for idle threads over corePoolSize before termination.
  • unit: Unit of time for keepAliveTime.
  • workQueue: Queue to hold tasks before execution.
  • threadFactory: Factory for creating threads, usually Executors.defaultThreadFactory().
  • handler: Policy for handling tasks when the pool reaches its limit.

8. Advanced Queue Management​

Choosing the Right Queue:

  1. SynchronousQueue: No internal capacity, ideal for transient spikes.
  2. LinkedBlockingQueue: Unbounded, suitable for long-running processes.
  3. ArrayBlockingQueue: Fixed size, optimal for bounded workloads.

Note: Select a queue type aligned with task load, duration, and resource availability.

Relationships between corePoolSize and maximumPoolSize​

  • If thread count < corePoolSize, a new thread is created for each task.
  • If thread count >= corePoolSize, tasks are added to the task queue.
  • If the queue is full and thread count < maximumPoolSize, a new thread is created.
  • If thread count >= maximumPoolSize, the handler policy determines the behavior.

Note: These relationships vary with the type of queue (bounded or unbounded). Unbounded queues will prevent maximum thread creation from being reached.

Understanding workQueue Types​

The workQueue holds pending tasks that await execution. Different queue types handle tasks differently:

  • SynchronousQueue: A direct hand-off mechanism with no capacity. If no thread is immediately available, a new one is created or the task is rejected based on maximumPoolSize.
  • LinkedBlockingQueue: An unbounded queue. When tasks exceed corePoolSize, they are added to the queue instead of creating new threads. Warning: Unbounded growth may exhaust memory if tasks are generated faster than processed.

9. Thread Pool Scaling Strategies​

Efficient scaling balances resource use against workload demands. Consider using dynamic scaling mechanisms for responsive systems.

  • FixedThreadPool: When task load and timing are predictable.
  • CachedThreadPool: For unpredictable or short-lived task loads.

10. RejectedExecutionHandler: Strategies for Handling Excess Tasks​

When the pool and queue are full, the handler parameter specifies a strategy to handle tasks. Java provides four built-in policies:

  • DiscardOldestPolicy: Removes the oldest task in the queue to accommodate the new task.
  • CallerRunsPolicy: Executes the task in the caller thread if the pool is busy, providing a backpressure mechanism.
  • DiscardPolicy: Silently discards the task without further action.
  • AbortPolicy: Throws an exception to signal that the pool is at capacity.

Note: Customize RejectedExecutionHandler based on workload characteristics. Expecially for applications where resilience is critical, consider implementing a custom rejection handler or using CallerRunsPolicy to limit load. Proper configuration can prevent issues like out-of-memory errors.

RejectedExecutionHandler customHandler = new RejectedExecutionHandler() {
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
System.out.println("Task rejected: " + r.toString());
// custom logic for rejected task
}
};

11. Enhanced Monitoring Techniques​

Monitoring thread and queue activity is essential for real-time scaling and troubleshooting. Use the following tools and approaches:

  • JMX (Java Management Extensions): For real-time metrics on threads, tasks, and system load.
  • Thread Pool Executor's getPoolSize and getQueue().size() Methods: To track runtime capacity and pending task counts.

12. Backpressure Handling in Production​

Backpressure handling in high-demand environments prevents overloads by controlling the flow of tasks:

  • Rejection Policies: Define fallback strategies for rejected tasks.
  • Queue Sizing: Set boundaries on task input rate.

Diagram: Dynamic Task Flow and Backpressure​

Client Requests
|
+----v----+ Backpressure Control
| Task |--------------------+
| Queue | |
+---------+ |
| |
+------v-------+ |
| Thread | |
| Pool | |
+--------------+----------------+

13. Example​

import java.util.concurrent.*;

public class OptimizedThreadPoolExecutor {
public static void main(String[] args) {
// 1. Configure core parameters with scaling strategy
ThreadPoolExecutor executor = new ThreadPoolExecutor(
4, // core pool size
10, // maximum pool size
30, // keep-alive time
TimeUnit.SECONDS, // keep-alive time unit
new ArrayBlockingQueue<>(100), // task queue type
new ThreadPoolExecutor.CallerRunsPolicy() // rejection policy
);

// 2. Simulate task submission
for (int i = 0; i < 200; i++) {
executor.submit(new Task(i));
}

executor.shutdown();
}

static class Task implements Runnable {
private final int taskId;

Task(int taskId) {
this.taskId = taskId;
}

@Override
public void run() {
System.out.println("Executing Task " + taskId);
try {
Thread.sleep(500); // Simulate work
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}

14. Conclusion​

Summary: Use ThreadPoolExecutor directly with appropriately sized pools, tailored queue types, and effective rejection policies to ensure optimal, safe performance. Mastering ThreadPoolExecutor for complex, production-grade applications requires an understanding of core parameters, workload patterns, and monitoring needs. These best practices are intended to improve performance, efficiency, and system resilience in multithreaded environments.

Building a ChatGPT-like System - II

· 5 min read

This guide provides an in-depth exploration of building a ChatGPT-style conversational AI system. We cover everything from model setup and inference engine design to system architecture, incorporating caching, scalability, and production-ready features.

System Overview​

A conversational AI system has several core components responsible for processing, generating, and managing dialogue flow with users. This overview explains the minimal structure for a functional, extensible ChatGPT-style system.

1. Core Dialogue Architecture​

The architecture focuses on the interaction between the user, the web interface, and the inference engine that generates responses.

Diagram: Core Dialogue Flow​

+---------------+ +-------------------+ +-----------------------+
| Web UI | <-- HTTP --> | Server/API | <-- REST --> | Inference Runtime |
| (User Access) | | (Request Routing) | | (LLM Model Handling) |
+---------------+ +-------------------+ +-----------------------+
^ |
|<--------------------------------------------------------+
  1. Web UI: Handles user interaction and displays responses.
  2. Server/API: Receives requests and forwards them to the inference runtime.
  3. Inference Runtime: Loads the large language model (LLM), processes the input, and generates a response.

2. Session and History Management​

Problem of Stateless Inference​

Standard inference runtimes are typically stateless, meaning they cannot recall previous messages in a conversation. To handle multi-turn dialogue, we integrate a database to persist session data and user message history.

Diagram: Enhanced System with Session and History Management​

+---------------+ +-------------------+ +-----------------------+
| Web UI | <-- HTTP --> | Server/API | <-- REST --> | Inference Runtime |
| | | | | |
+---------------+ +-------------------+ +-----------------------+
|
+---------------- Database ----------------+
| User Session, ChatID, and Messages |
+------------------------------------------+
  1. Database: Stores user sessions, chat histories, and other metadata.
  2. Server/API: Uses REST endpoints to interact with the database, retrieving history when needed.

3. REST API Design​

The Server/API manages communication with the Web UI via REST APIs, handling user requests, responses, and session tracking.

REST Endpoints:

  • POST /chat: Starts a new conversation.
  • POST /chat/:chatID/completion: Continues a conversation in an existing session.
  • GET /chats: Retrieves the list of active sessions.
  • DELETE /chat/:chatID: Deletes a specified session.

Sample API Call to Start a Conversation

curl -X POST http://yourserver.com/chat \
-H "Content-Type: application/json" \
-d '{"userID": "12345", "message": "Hello, ChatGPT!"}'

4. Message Structure and Prompt Format​

The prompt format structures conversation history and provides context to the model.

Example Prompt Format in JSON

[
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Hello!" },
{ "role": "assistant", "content": "Hello there, how may I assist you today?" },
{ "role": "user", "content": "How are you?" }
]
  1. Role: Indicates the message source (system, user, or assistant).
  2. Content: The message text.

5. Managing History and Context​

Techniques for History Management​

  • Direct Prompt Fill: Append all previous messages to the prompt, suitable for short interactions.
  • Context Truncation: Omit older messages when context length exceeds the model’s token limit.
  • History Summarization: Generate a summary of prior messages to condense the context, enabling longer conversations within token limits.

6. System Extensions and Scalability​

In a high-traffic environment, it is essential to scale the system horizontally and incorporate caching for repeated requests.

7. Caching Strategies​

Implementing caching reduces redundant processing for similar queries. This approach involves caching semantic meanings of requests rather than exact matches.

Diagram: Cache with Vector Embeddings for Similar Query Detection​

+---------------+ +------------+ +----------------+
| User Query | -> Embedding -> | Cache | Cache Hit -> | Response |
+---------------+ +------------+ +----------------+
| No Hit
v
+---------------------------+ +--------------------+
| Text Embedding Model | --> | Vector Storage and |
| (e.g., BERT/Word2Vec) | | Search Database |
+---------------------------+ +--------------------+
  1. Embedding Runtime: Converts queries into vector embeddings.
  2. Vector Storage: Stores embeddings to detect semantically similar queries.

Cache Scope Considerations​

  • Single User Cache: Stores responses per user session.
  • Global Cache: Shared across all users, introducing potential privacy considerations.

8. Elastic Scaling and Load Balancing​

Scaling is crucial for supporting high traffic. Stateless components like the server and inference runtime can be scaled horizontally using Kubernetes.

Diagram: Load Balanced System Architecture with Elastic Scaling​

+--------------------------+ +---------------+ +--------------------------+
| Gateway | ---> | Server | ---> | Inference |
| | | | | Runtime (Auto-scalable) |
| Load Balancer | | (Stateless) | | |
+--------------------------+ +---------------+ +--------------------------+
  • Gateway: Distributes traffic across multiple servers.
  • Message Queue (MQ): Handles requests asynchronously for stable inference.

9. Production Readiness​

To make the system production-ready, additional steps are essential.

Checklist for Production​

  • Technology Selection:
    • Database: PostgreSQL or MongoDB for session persistence.
    • Inference Engine: Use efficient models like llama.cpp, HuggingFace Transformers, or vLLM.
  • Observability:
    • Implement logging, tracing, and metrics.
  • CI/CD and Deployment:
    • Use Kubernetes or cloud-based deployment environments.

Conclusion​

This guide provides a foundational approach to building a ChatGPT-like system, managing sessions, scaling the architecture, and ensuring production reliability. From basic dialogue to advanced caching and scalability, these steps can help create a highly responsive and robust conversational AI.

Building a ChatGPT-like System

· 5 min read

Introduction​

ChatGPT is a conversational system based on Large Language Models (LLMs). This article introduces the process of building a system similar to ChatGPT, covering everything from the model, inference engine, to the overall architecture.


System Overview​

Our primary focus will be on the core of the conversation system.

System Diagram:

+--------------+
| Web |
| Interface |
+--------------+
|
|
v
+--------------+
| Server |
+--------------+
|
|
v
+--------------------------+
| Inference Runtime |
| (LLM Inference Engine) |
+--------------------------+
|
|
v
+--------------+
| Response |
+--------------+

Core Dialogue Flow

In this basic system framework:

  • The web interface interacts with users,
  • The server receives user requests and forwards them to the inference runtime (inference engine),
  • The inference runtime loads the LLM to generate a response, returning it to the user.

Session and Historical Message Management​

However, the above system has a critical flaw: it lacks session and historical message management for users. Common inference runtimes are typically stateless and do not directly support multi-turn conversations with historical context storage, meaning the system "forgets" previous exchanges within a single session.

To address this, we need some essential modifications.

Enhanced System Diagram:

+--------------+
| Web |
| Interface |
+--------------+
|
|
v
+--------------+
| Server |
+--------------+
|
|
v
+--------------------------+
| Inference Runtime |
| (LLM Inference Engine) |
+--------------------------+
|
|
v
+--------------+
| Database |
+--------------+

Enhanced System with Session ahdn History Management

In this setup, we introduce a database component to store user sessions and historical messages.

Communication Protocols and APIs​

For communication, the web interface and server can use HTTP protocols. Basic REST APIs might include:

  • POST /chat: Initiate a new session.
  • POST /chat/:chatID/completion: Continue a conversation within an existing session.
  • GET /chats: Retrieve a list of sessions.
  • DELETE /chat/:chatID: Delete a specific session.

The server persists data in the database, structuring conversation messages with essential fields like userID, chatID, userMessage, and assistantMessage.

Standardized Prompt Format​

When sending data to the inference runtime, the server uses a standardized prompt format:

[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Hello!"
},
{
"role": "assistant",
"content": "Hello there, how may I assist you today?"
},
{
"role": "user",
"content": "How are you?"
}
]

The role field represents different participants: system defines the conversation context, user represents user input, and assistant represents the model’s output.

Managing Historical Messages​

There are several ways to handle historical messages:

  • Direct Prompt Filling: Add historical messages in the prompt as user and assistant entries. This works well for short conversations.
  • Dynamic Context Adjustment: Discard earlier messages to fit within the LLM’s token limit.
  • Message Summarization: Use the inference engine to summarize conversation history, compressing information before filling the prompt.

With session and historical message management, we establish a basic system framework.


System Expansion​

As the user base grows, the system will require further scalability.

Adding Cache Mechanism​

One approach to scalability is implementing a cache to avoid repeated inference processes. The cache key-value pair would be the user's question and the AI's response. A cache hit would depend on whether the semantic content of the question is similar. For example, if two questions have different wording but the same meaning, the system should return the same response.

Enhanced Caching System Diagram:

+--------------+
| Web |
| Interface |
+--------------+
|
|
v
+--------------+
| Server |
+--------------+
|
|
v
+--------------------------+
| Inference Runtime |
| (LLM Inference Engine) |
+--------------------------+
|
|
v
+--------------------------+
| Embedding |
| Runtime & Model |
+--------------------------+
|
|
v
+--------------------------+
| Vector Storage & Search |
+--------------------------+

Cache with Vector Embeddings for Similar Query Detection

Apart from the cache module, we add embedding runtime and a text embedding model to convert text into vectors. If two vectors are close, their text content is semantically similar. Vector storage and search modules are used to store and retrieve vectors.

Scope of Caching​

Caching can be scoped to:

  • Single-user cache: Caching value is low as users rarely repeat the same question.
  • Global cache: Allows reuse across users, but risks exposing sensitive data.

After data analysis and validation, decide if cache implementation is beneficial.

Elastic Scalability​

Elastic scaling is another key to handling high concurrency. The server is stateless, which allows for easy horizontal scaling.

Scalability System Diagram:

+--------------+
| Web |
| Interface |
+--------------+
|
|
v
+--------------+
| Gateway |
| (Load Bal.) |
+--------------+
|
|
v
+--------------------------+
| Server Instances |
+--------------------------+
|
|
v
+--------------------------+
| Inference Runtimes |
+--------------------------+

Load Balanced System Architecture with Elastic Scaling

Here, we introduce a gateway for load balancing. The inference runtime is also stateless and supports elastic scaling, but it requires more hardware resources and may respond slower than the server. To maintain stability under peak load, add a message queue (MQ) and change request handling to asynchronous, improving system resilience.


Production Readiness​

The system architecture covers the basic logic, but to achieve production-readiness, further steps are necessary:

  • Technology Selection: Choose a database (e.g., PostgreSQL or MongoDB) and inference engine (e.g., llama.cpp, HuggingFace Transformers, vLLM) and map logical components to physical components.
  • Observability: Integrate logging, tracing, metrics, and set up monitoring and alerting.
  • CI/CD and Deployment Environment: Configure automation pipelines for continuous integration and deployment, and select the appropriate environment (e.g., cloud platforms or Kubernetes).

Conclusion​

This article provides an overview of constructing a ChatGPT-like system, covering the core conversation flow and expanding to include session management and historical message storage. We discussed strategies for system expansion and laid out steps toward production readiness.


References​


This enhanced Markdown document covers each requested update, including detailed technical sections, diagrams, keywords, and sections for expansion and production readiness.

Comprehensive Guide to Kubernetes

· 7 min read

This guide provides a detailed overview of Kubernetes concepts, examples, use cases, installation YAML examples, and visual representations of the architecture.

Cluster​

A Kubernetes Cluster is a group of nodes (machines) that host and manage containerized applications. Each cluster has at least one master node, which controls the state of the cluster, and multiple worker nodes.

Example​

A cloud-based cluster with one master node and multiple worker nodes in Google Kubernetes Engine (GKE).

Use Case​

An ideal setup for running a production microservices application with high availability.

YAML Structure for Cluster Creation​

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker

Node​

A Node is a single machine within a Kubernetes cluster, capable of running one or more pods.

Example​

In a cloud environment, each node could be a virtual machine instance.

Use Case​

Nodes distribute workloads across the infrastructure.

YAML Snippet​

The node configuration is usually specified by the cluster management tool (e.g., GKE or EKS) rather than in YAML, as it pertains to the infrastructure layer.


Pod​

A Pod is the smallest deployable unit in Kubernetes, which can contain one or more containers.

Example​

A pod running a web server container and a logging container.

Use Case​

Running tightly coupled applications together within the same lifecycle.

YAML Structure for Pod​

apiVersion: v1
kind: Pod
metadata:
name: web-server-pod
spec:
containers:
- name: web-server
image: nginx
- name: logger
image: fluentd

Deployment​

A Deployment is a Kubernetes object for managing a group of identical pods. It provides capabilities like rolling updates and rollback.

Example​

A deployment with three replicas of a web server pod.

Use Case​

Maintaining a stable version of an application by managing pod replicas.

YAML Structure for Deployment​

apiVersion: apps/v1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: nginx:1.17.4

Service​

A Service provides a stable endpoint for accessing a set of pods.

Example​

A service that exposes an application to the public internet.

Use Case​

Load balancing traffic to an application across multiple pods.

YAML Structure for Service​

apiVersion: v1
kind: Service
metadata:
name: web-service
spec:
type: LoadBalancer
selector:
app: web
ports:
- protocol: TCP
port: 80
targetPort: 80

Ingress​

Ingress manages external access to services, typically using HTTP.

Example​

An ingress with rules to route traffic to different services based on URL paths.

Use Case​

Routing /api to backend services and /static to frontend services on the same domain.

YAML Structure for Ingress​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
- host: example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
- path: /static
pathType: Prefix
backend:
service:
name: static-service
port:
number: 80

ConfigMap​

A ConfigMap stores non-sensitive configuration data.

Example​

A ConfigMap for environment-specific variables.

Use Case​

Keeping configuration separate from the application code.

YAML Structure for ConfigMap​

apiVersion: v1
kind: ConfigMap
metadata:
name: config-demo
data:
DATABASE_URL: "jdbc:mysql://dbhost:3306/mydatabase"
DEBUG_LEVEL: "3"

Secret​

A Secret stores sensitive information, like passwords and tokens.

Example​

A secret containing API keys for external services.

Use Case​

Securing sensitive data without exposing it in the code.

YAML Structure for Secret​

apiVersion: v1
kind: Secret
metadata:
name: api-key-secret
type: Opaque
data:
api-key: c29tZS1yYW5kb20ta2V5Cg==

Persistent Volume (PV) and Persistent Volume Claim (PVC)​

A Persistent Volume provides storage, while a Persistent Volume Claim is a request for that storage.

Example​

A PV representing cloud disk storage and a PVC used by a database pod.

Use Case​

Ensuring data persistence for applications that require durable storage.

YAML Structure for PV and PVC​

apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-storage
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi

Namespace​

A Namespace is a way to divide cluster resources among multiple users.

Example​

Separate namespaces for development, staging, and production environments.

Use Case​

Organizing resources for different environments or teams.

YAML Structure for Namespace​

apiVersion: v1
kind: Namespace
metadata:
name: dev-environment

DaemonSet​

A DaemonSet ensures that specific pods run on each node.

Example​

Running a logging agent on every node in the cluster.

Use Case​

Managing node-wide services like logging or monitoring.

YAML Structure for DaemonSet​

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: logger
spec:
selector:
matchLabels:
name: logger
template:
metadata:
labels:
name: logger
spec:
containers:
- name: fluentd
image: fluent/fluentd

Job and CronJob​

Job runs a one-time task, while CronJob schedules a recurring task.

Example​

A Job to back up a database once and a CronJob to clean up logs daily.

Use Case​

Automating scheduled tasks or batch processing.

YAML Structure for Job and CronJob​

apiVersion: batch/v1
kind: Job
metadata:
name: backup-job
spec:
template:
spec:
containers:
- name: backup
image: backup-tool
restartPolicy: OnFailure

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: daily-cleanup
spec:
schedule: "0 0 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: cleanup
image: cleanup-tool
restartPolicy: OnFailure

Horizontal Pod Autoscaler (HPA)​

The HPA scales pod replicas automatically based on resource utilization.

Example​

Scaling web server pods based on CPU usage.

Use Case​

Scaling applications to handle variable workloads.

YAML Structure for HPA​

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: web-hpa
spec:
scaleTarget

Ref:
apiVersion: apps/v1
kind: Deployment
name: web-deployment
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 70

StatefulSet​

A StatefulSet manages stateful applications with unique and persistent storage.

Example​

A StatefulSet for a MongoDB replica set.

Use Case​

Managing stateful applications where each instance requires its own storage.

YAML Structure for StatefulSet​

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
replicas: 3
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
containers:
- name: mongo
image: mongo
ports:
- containerPort: 27017
name: mongo

Volume and StorageClass​

Volumes provide storage to pods, while StorageClass defines types of storage.

Example​

SSD-based storage for a database application.

Use Case​

Choosing storage for applications based on performance requirements.

YAML Structure for StorageClass and Volume​

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: fast-pv
spec:
storageClassName: fast
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce

NetworkPolicy​

NetworkPolicy defines network access rules between pods.

Example​

Restricting access to specific pods.

Use Case​

Securing inter-service communication.

YAML Structure for NetworkPolicy​

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress: []

Custom Resource Definitions (CRDs)​

CRDs extend Kubernetes by allowing users to create custom resources.

Example​

A custom resource to manage database schema migrations.

Use Case​

Adding domain-specific resources for applications.

YAML Structure for CRD​

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: migrations.mycompany.com
spec:
group: mycompany.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: migrations
singular: migration
kind: Migration

Helm Charts​

Helm is a package manager for Kubernetes, and Helm Charts are pre-configured packages.

Example​

A Helm chart for deploying a WordPress site.

Use Case​

Simplifying application deployment and management.

Example Helm Command​

helm install my-wordpress bitnami/wordpress

These concepts are essential for understanding and operating Kubernetes in production, allowing you to deploy, manage, and scale applications effectively. Each example and YAML configuration highlights the power and flexibility Kubernetes offers in a variety of scenarios.

Create a Java Thread Pool From Scratch

· 17 min read

Custom Java Thread Pool: Proof of Concept

In this article, we will create a simple but complete custom thread pool. We’ll build a SimpleThreadPool class that includes key features such as managing worker threads, a task queue, and shutdown capabilities. We'll break down the code into logical components, explain the design concepts, and provide sample usage.


Implementation of SimpleThreadPool​

The core classes in our implementation are:

  • SimpleThreadPool: Manages the thread pool and task queue.
  • Worker: Represents the worker threads that execute tasks.
import java.util.LinkedList;
import java.util.Queue;

public class SimpleThreadPool {
private final int poolSize;
private final PoolWorker[] workers;
private final Queue<Runnable> taskQueue;
private boolean isShutdown = false;

public SimpleThreadPool(int poolSize) {
this.poolSize = poolSize;
taskQueue = new LinkedList<>();
workers = new PoolWorker[poolSize];

// Start worker threads
for (int i = 0; i < poolSize; i++) {
workers[i] = new PoolWorker();
workers[i].start();
}
}

// Submit a task to the thread pool
public synchronized void submit(Runnable task) {
if (!isShutdown) {
taskQueue.add(task);
notify(); // Notify workers that a task is available
} else {
throw new IllegalStateException("ThreadPool is shut down");
}
}

// Initiates an orderly shutdown
public synchronized void shutdown() {
isShutdown = true;
for (PoolWorker worker : workers) {
worker.interrupt(); // Interrupt each worker to stop waiting
}
}

// Worker class represents threads in the pool
private class PoolWorker extends Thread {
public void run() {
Runnable task;

while (true) {
synchronized (SimpleThreadPool.this) {
while (taskQueue.isEmpty() && !isShutdown) {
try {
SimpleThreadPool.this.wait(); // Wait for a task
} catch (InterruptedException ignored) {
return; // Exit if interrupted during shutdown
}
}
// If shutdown and no tasks, end thread
if (isShutdown && taskQueue.isEmpty()) {
return;
}
task = taskQueue.poll(); // Take task from queue
}
try {
task.run(); // Execute task outside synchronized block
} catch (RuntimeException e) {
System.err.println("Task execution failed: " + e.getMessage());
}
}
}
}
}

Explanation of Key Concepts​

  1. Task Queue: A LinkedList serves as the task queue, holding Runnable tasks for worker threads.
  2. Worker Threads: PoolWorker threads execute tasks. Each worker loops to fetch a task from the queue and processes it if available.
  3. Synchronization with wait and notify:
    • wait(): Called when no tasks are available, putting the thread in a waiting state to save CPU resources.
    • notify(): Called when a new task is added, waking up a worker thread to execute the task.
  4. Graceful Shutdown:
    • shutdown(): Sets the isShutdown flag to stop accepting tasks and interrupts all worker threads, allowing them to exit if they’re waiting for a task.

Sample Code to Use the SimpleThreadPool​

This example demonstrates how to use the custom SimpleThreadPool to run tasks, including graceful shutdown.

public class SimpleThreadPoolExample {
public static void main(String[] args) {
// Create a thread pool with 3 threads
SimpleThreadPool threadPool = new SimpleThreadPool(3);

// Submit tasks to the thread pool
for (int i = 1; i <= 5; i++) {
int taskId = i;
threadPool.submit(() -> {
System.out.println("Task " + taskId + " is starting on thread " + Thread.currentThread().getName());
try {
Thread.sleep(1000); // Simulate task execution time
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
System.out.println("Task " + taskId + " completed.");
});
}

// Shutdown the thread pool after tasks complete
threadPool.shutdown();
}
}

Explanation of Sample Usage Code​

  1. Creating a Thread Pool: A SimpleThreadPool instance with 3 threads is created.
  2. Submitting Tasks: Five tasks are submitted to the pool, each of which simulates a brief work period.
  3. Shutting Down the Pool: After submitting the tasks, the pool is gracefully shut down using shutdown().

Expected Output​

Task 1 is starting on thread Thread-0
Task 2 is starting on thread Thread-1
Task 3 is starting on thread Thread-2
Task 1 completed.
Task 2 completed.
Task 3 completed.
Task 4 is starting on thread Thread-0
Task 5 is starting on thread Thread-1
Task 4 completed.
Task 5 completed.

Conclusion​

This SimpleThreadPool is a demonstration of how a basic thread pool is structured and works:

  • Worker Threads repeatedly pick up and execute tasks from a shared queue.
  • Synchronization between task producers and worker threads prevents busy-waiting.
  • Graceful Shutdown lets worker threads exit safely after processing all queued tasks.

This POC lays the foundation for understanding and extending Java thread pool management by illustrating how resources are managed and synchronized in a concurrent environment.


Custom Java Thread Pool with Separate Task Class

Separating the Task as an individual class will enhance the code's modularity and flexibility, making it easier to reuse or extend for specific task-related features. Let's modify the SimpleThreadPool implementation to include an external Task class.

Here’s the updated implementation:

  1. SimpleThreadPool: Manages worker threads and the task queue.
  2. Task: Represents individual tasks, implementing Runnable.
  3. PoolWorker: Represents the worker threads that pick up tasks from the queue and execute them.

Task Class Implementation​

We’ll define Task as a standalone class that implements Runnable. Each task can have custom logic.

public class Task implements Runnable {
private final int taskId;

public Task(int taskId) {
this.taskId = taskId;
}

@Override
public void run() {
System.out.println("Task " + taskId + " is starting on thread " + Thread.currentThread().getName());
try {
Thread.sleep(1000); // Simulate task execution time
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // Handle interruption
}
System.out.println("Task " + taskId + " completed.");
}
}
  • taskId: Each task is given an ID to identify it during execution.
  • run() Method: Defines the task's actions, including a simulated delay.

SimpleThreadPool Implementation​

The SimpleThreadPool class manages the task queue and distributes tasks among worker threads.

import java.util.LinkedList;
import java.util.Queue;

public class SimpleThreadPool {
private final int poolSize;
private final PoolWorker[] workers;
private final Queue<Runnable> taskQueue;
private boolean isShutdown = false;

public SimpleThreadPool(int poolSize) {
this.poolSize = poolSize;
taskQueue = new LinkedList<>();
workers = new PoolWorker[poolSize];

// Initialize and start each worker thread
for (int i = 0; i < poolSize; i++) {
workers[i] = new PoolWorker();
workers[i].start();
}
}

// Method to submit a task to the thread pool
public synchronized void submit(Runnable task) {
if (!isShutdown) {
taskQueue.add(task);
notify(); // Notify a waiting worker thread about the new task
} else {
throw new IllegalStateException("ThreadPool is shut down and cannot accept tasks");
}
}

// Initiates shutdown and interrupts all waiting worker threads
public synchronized void shutdown() {
isShutdown = true;
for (PoolWorker worker : workers) {
worker.interrupt(); // Interrupts workers waiting for tasks
}
}

// Worker class that executes tasks from the queue
private class PoolWorker extends Thread {
public void run() {
while (true) {
Runnable task;
synchronized (SimpleThreadPool.this) {
while (taskQueue.isEmpty() && !isShutdown) {
try {
SimpleThreadPool.this.wait(); // Wait for a new task
} catch (InterruptedException e) {
return; // Exit if interrupted during shutdown
}
}
if (isShutdown && taskQueue.isEmpty()) {
return; // Exit if shutdown and no tasks are left
}
task = taskQueue.poll(); // Fetch a task
}
try {
task.run(); // Execute the task outside the synchronized block
} catch (RuntimeException e) {
System.err.println("Task execution failed: " + e.getMessage());
}
}
}
}
}

Sample Code to Use the Custom Thread Pool with Task​

This example demonstrates submitting Task instances to the SimpleThreadPool.

public class SimpleThreadPoolExample {
public static void main(String[] args) {
// Create a thread pool with 3 threads
SimpleThreadPool threadPool = new SimpleThreadPool(3);

// Submit tasks to the thread pool
for (int i = 1; i <= 5; i++) {
Task task = new Task(i); // Create a new Task with a unique ID
threadPool.submit(task); // Submit the Task to the thread pool
}

// Shutdown the thread pool after tasks are complete
threadPool.shutdown();
}
}

Explanation of Sample Usage​

  1. Task Creation: Each Task object has a unique ID, making it easy to track execution progress.
  2. Task Submission: Tasks are added to the SimpleThreadPool’s task queue.
  3. Shutdown: Once all tasks are submitted, shutdown() stops the pool after finishing queued tasks.

Expected Output​

Task 1 is starting on thread Thread-0
Task 2 is starting on thread Thread-1
Task 3 is starting on thread Thread-2
Task 1 completed.
Task 2 completed.
Task 3 completed.
Task 4 is starting on thread Thread-0
Task 5 is starting on thread Thread-1
Task 4 completed.
Task 5 completed.

This implementation improves modularity by allowing different Task implementations and extends the reusability of SimpleThreadPool. Each component (Task, PoolWorker, and SimpleThreadPool) is designed for flexible use in varied production scenarios.


Wait but what happened to the terminated thread?

To demonstrate the behavior where a terminated thread in a thread pool is repurposed and moves back to the runnable state when a new task is added, we need to tweak the SimpleThreadPool to simulate this behavior.

Typically, when threads in a pool complete their tasks, they wait for new tasks rather than fully terminating (they don't reach the "TERMINATED" state in Java's built-in ThreadPoolExecutor). However, for demonstration, I’ll create logic that represents threads reaching a "stopped" state and being reactivated for new tasks.

Let's add a mechanism to track the states and illustrate the thread reusability behavior.


Enhanced SimpleThreadPool Implementation​

Below is an enhanced version of SimpleThreadPool with thread state tracking. We’ll add logging and state simulation to show when threads are "terminated" and later reactivated when new tasks arrive.

import java.util.LinkedList;
import java.util.Queue;

public class SimpleThreadPool {
private final int poolSize;
private final PoolWorker[] workers;
private final Queue<Runnable> taskQueue;
private boolean isShutdown = false;

public SimpleThreadPool(int poolSize) {
this.poolSize = poolSize;
taskQueue = new LinkedList<>();
workers = new PoolWorker[poolSize];

// Initialize and start each worker thread
for (int i = 0; i < poolSize; i++) {
workers[i] = new PoolWorker(i + 1); // Assign a unique ID to each worker
workers[i].start();
}
}

// Method to submit a task to the thread pool
public synchronized void submit(Runnable task) {
if (!isShutdown) {
taskQueue.add(task);
notify(); // Notify a waiting worker thread about the new task
} else {
throw new IllegalStateException("ThreadPool is shut down and cannot accept tasks");
}
}

// Initiates shutdown and interrupts all waiting worker threads
public synchronized void shutdown() {
isShutdown = true;
for (PoolWorker worker : workers) {
worker.interrupt(); // Interrupts workers waiting for tasks
}
}

// Worker class that executes tasks from the queue
private class PoolWorker extends Thread {
private final int workerId;
private boolean isTerminated = false;

public PoolWorker(int id) {
this.workerId = id;
}

public void run() {
while (true) {
Runnable task;
synchronized (SimpleThreadPool.this) {
while (taskQueue.isEmpty() && !isShutdown) {
System.out.println("Worker-" + workerId + " is waiting for a task...");
try {
SimpleThreadPool.this.wait(); // Wait for a new task
} catch (InterruptedException e) {
System.out.println("Worker-" + workerId + " is terminated.");
isTerminated = true;
return; // Exit if interrupted during shutdown
}
}
if (isShutdown && taskQueue.isEmpty()) {
System.out.println("Worker-" + workerId + " is terminated as pool shuts down.");
isTerminated = true;
return; // Exit if shutdown and no tasks are left
}
task = taskQueue.poll(); // Fetch a task
}

try {
isTerminated = false; // Worker is active
System.out.println("Worker-" + workerId + " is executing task.");
task.run(); // Execute the task
System.out.println("Worker-" + workerId + " completed task and goes back to waiting.");
} catch (RuntimeException e) {
System.err.println("Task execution failed in Worker-" + workerId + ": " + e.getMessage());
}
}
}
}
}

Enhanced Task Class Implementation​

The Task class remains mostly the same, but we can add unique identifiers to differentiate tasks.

public class Task implements Runnable {
private final int taskId;

public Task(int taskId) {
this.taskId = taskId;
}

@Override
public void run() {
System.out.println("Task " + taskId + " is starting on thread " + Thread.currentThread().getName());
try {
Thread.sleep(1000); // Simulate task execution time
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // Handle interruption
}
System.out.println("Task " + taskId + " completed.");
}
}

Usage and State Transition Simulation​

Let's add a usage example showing how to submit tasks, observe workers entering a “terminated” state, and then reactivating to process new tasks.

public class ThreadPoolExample {
public static void main(String[] args) throws InterruptedException {
// Create a thread pool with 2 workers
SimpleThreadPool threadPool = new SimpleThreadPool(2);

// Submit initial tasks to the thread pool
for (int i = 1; i <= 3; i++) {
threadPool.submit(new Task(i));
}

// Wait to let initial tasks complete
Thread.sleep(3000);

// Submit additional tasks to show reactivation
System.out.println("Adding more tasks to test thread reactivation...");
for (int i = 4; i <= 5; i++) {
threadPool.submit(new Task(i));
}

// Give some time for tasks to be picked up
Thread.sleep(3000);

// Shutdown the thread pool
threadPool.shutdown();
}
}

Explanation of the Workflow and State Transitions​

  1. Initial Task Submission: Three tasks are submitted, and two workers pick them up immediately since the pool size is 2.
  2. Worker Reactivation: After the initial tasks complete, we add two more tasks, which are picked up by waiting workers.
  3. Shutdown Process: Finally, we shut down the thread pool, and the workers gracefully terminate after completing their tasks.

Expected Output (Sample)​

Worker-1 is waiting for a task...
Worker-2 is waiting for a task...
Worker-1 is executing task.
Task 1 is starting on thread Worker-1
Worker-2 is executing task.
Task 2 is starting on thread Worker-2
Task 1 completed.
Worker-1 completed task and goes back to waiting.
Task 2 completed.
Worker-2 completed task and goes back to waiting.
Worker-1 is executing task.
Task 3 is starting on thread Worker-1
Task 3 completed.
Worker-1 completed task and goes back to waiting.
Worker-1 is waiting for a task...
Worker-2 is waiting for a task...

Adding more tasks to test thread reactivation...
Worker-1 is executing task.
Task 4 is starting on thread Worker-1
Worker-2 is executing task.
Task 5 is starting on thread Worker-2
Task 4 completed.
Worker-1 completed task and goes back to waiting.
Task 5 completed.
Worker-2 completed task and goes back to waiting.
Worker-1 is terminated as pool shuts down.
Worker-2 is terminated as pool shuts down.

This output demonstrates that the workers remain active, go back to a waiting state after completing tasks, and terminate only after shutdown() is called.

How thread being actually reused?

In a typical thread pool implementation, once a thread completes its task, it does not transition to the "TERMINATED" state and then back to the "RUNNABLE" state. Instead, the thread remains in a waiting state (often referred to as "idle" or "waiting for work") until a new task is available. This waiting state is different from the "TERMINATED" state, which indicates that the thread has finished execution and cannot be reused.

Here’s a more detailed explanation:

Thread States in a Thread Pool​

  1. NEW: The thread has been created but not yet started.
  2. RUNNABLE: The thread is executing in the JVM.
  3. BLOCKED: The thread is blocked waiting for a monitor lock.
  4. WAITING: The thread is waiting indefinitely for another thread to perform a particular action.
  5. TIMED_WAITING: The thread is waiting for another thread to perform an action for up to a specified waiting time.
  6. TERMINATED: The thread has completed execution.

In a thread pool, threads typically cycle between the RUNNABLE and WAITING states:

  • RUNNABLE: When a thread is actively executing a task.
  • WAITING: When a thread is idle, waiting for a new task to be added to the queue.

How Threads Are Reused​

When a thread completes its task, it checks the task queue for new tasks. If no tasks are available, the thread waits (using wait() or a similar mechanism) until a new task is submitted. This waiting state is not the same as the "TERMINATED" state. A thread in the "TERMINATED" state cannot be reused; it must be recreated, which is inefficient.

Example of Thread Reuse in SimpleThreadPool​

In the SimpleThreadPool example, threads are reused as follows:

  1. Thread Creation: When the thread pool is created, a fixed number of worker threads are started.
  2. Task Execution: Each worker thread fetches a task from the queue and executes it.
  3. Waiting for New Tasks: After completing a task, the worker thread checks the queue for new tasks. If no tasks are available, it waits (wait()) until a new task is submitted.
  4. Reactivation: When a new task is submitted, the waiting threads are notified (notify()), and one of them wakes up to execute the new task.

Code Example​

Here’s a simplified version of the SimpleThreadPool to illustrate this:

import java.util.LinkedList;
import java.util.Queue;

public class SimpleThreadPool {
private final int poolSize;
private final PoolWorker[] workers;
private final Queue<Runnable> taskQueue;
private boolean isShutdown = false;

public SimpleThreadPool(int poolSize) {
this.poolSize = poolSize;
taskQueue = new LinkedList<>();
workers = new PoolWorker[poolSize];

// Start worker threads
for (int i = 0; i < poolSize; i++) {
workers[i] = new PoolWorker();
workers[i].start();
}
}

public synchronized void submit(Runnable task) {
if (!isShutdown) {
taskQueue.add(task);
notify(); // Notify a waiting worker thread
} else {
throw new IllegalStateException("ThreadPool is shut down");
}
}

public synchronized void shutdown() {
isShutdown = true;
for (PoolWorker worker : workers) {
worker.interrupt(); // Interrupt each worker to stop waiting
}
}

private class PoolWorker extends Thread {
public void run() {
while (true) {
Runnable task;
synchronized (SimpleThreadPool.this) {
while (taskQueue.isEmpty() && !isShutdown) {
try {
SimpleThreadPool.this.wait(); // Wait for a task
} catch (InterruptedException ignored) {
return; // Exit if interrupted during shutdown
}
}
if (isShutdown && taskQueue.isEmpty()) {
return; // Exit if shutdown and no tasks are left
}
task = taskQueue.poll(); // Take task from queue
}
try {
task.run(); // Execute task
} catch (RuntimeException e) {
System.err.println("Task execution failed: " + e.getMessage());
}
}
}
}
}

Key Points​

  • Thread Reuse: Threads do not transition to the "TERMINATED" state after completing a task. Instead, they wait for new tasks.
  • Task Queue: The task queue holds tasks that are waiting to be executed. When a new task is submitted, a waiting thread is notified and picks up the task.
  • Graceful Shutdown: The shutdown() method sets the isShutdown flag to stop accepting new tasks and interrupts all worker threads to ensure they exit if they are waiting.

Summary​

In Java’s ThreadPoolExecutor (or our custom pool example), the worker threads do not actually terminate after completing a task. Instead, they enter a waiting or idle state and wait to pick up the next task. The main goal of a thread pool is to reuse threads efficiently, keeping them alive for as long as the pool is running and ready to transition back to the runnable state whenever a new task becomes available.

Here's how it works:

  1. Runnable → Running: A thread in the pool becomes running when it starts executing a task.
  2. Running → Waiting/Idle: Once it finishes the task, it doesn't terminate. Instead, it returns to an idle state within the pool, waiting for more tasks to be added to the queue.
  3. Waiting → Runnable: When a new task is submitted, an idle worker transitions to runnable and executes the task.

The threads only reach a terminated state when the pool itself is shut down (e.g., calling shutdown() in ThreadPoolExecutor). Only then will the threads stop, exit the idle/waiting state, and go to terminated. This approach keeps the pool flexible and efficient, reducing the overhead of creating and destroying threads for every task.

So in short: threads in a thread pool do not reach terminated after completing tasks. They go back to idle/waiting and can accept new tasks. And threads in a thread pool do not go back to the "RUNNABLE" state from the "TERMINATED" state neither. Instead, they remain in a waiting state until new tasks are available, ensuring efficient reuse of thread resources.