Skip to main content

20 posts tagged with "kubernetes"

View All Tags

Understanding Kubernetes Pods and Services

· 5 min read

In Kubernetes, Pods and Services are fundamental components, each with distinct responsibilities that work together to provide scalable, resilient applications. Here’s an in-depth look into how these units interact, their roles, and how they communicate.

1. Kubernetes Pod: The Basic Unit of Deployment​

A Pod is the smallest deployable unit in Kubernetes, representing one or more containers. Containers within the same Pod:

  • Share resources: Same IP address, storage volumes, and network namespace.
  • Enable inter-container communication: Containers in a Pod can access each other on localhost.
  • Are ephemeral: When a Pod fails, it is replaced by a new Pod instance, often with a new IP address.

Pod Architecture​

Each Pod contains one or more containers with shared resources. Below is a textual representation of a typical Pod structure:

+-----------------------+
| Pod |
| IP: 10.244.1.5 |
| |
| +------------------+ |
| | Container A | |
| | - Application | |
| +------------------+ |
| |
| +------------------+ |
| | Container B | |
| | - Sidecar Proxy | |
| +------------------+ |
+-----------------------+

In this structure, Container A might run the main application, while Container B is a helper (e.g., proxy or logging container). These containers share an IP address and can communicate directly.

2. Real Scenario Explaination​

In this setup, the sidecar container collects logs from the PostgreSQL container by sharing a directory between them. Here’s a breakdown of how it works:

  1. Shared Volume: A shared emptyDir volume, postgres-logs, is mounted in both the PostgreSQL and sidecar containers. PostgreSQL is configured to output logs to /var/log/postgresql within its filesystem, which maps to the shared volume.

  2. Log Collector Configuration: The sidecar container, running Fluent Bit, is configured to read from /var/log/postgresql (also mapped to the shared volume). Fluent Bit’s configuration specifies the tail input plugin to read new entries as they are appended.

  3. Log Processing and Output: Fluent Bit processes the logs, then sends them to Elasticsearch through its output configuration, allowing centralized log analysis and monitoring.

This method ensures that logs are constantly accessible to the sidecar without modifying the PostgreSQL container directly.

Example Fluent Bit Configuration in configmap.yaml​

apiVersion: v1
kind: ConfigMap
metadata:
name: postgresql-log-config
data:
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info

[INPUT]
Name tail
Path /var/log/postgresql/postgresql.log
Tag postgresql

[OUTPUT]
Name es
Match *
Host {{ .Values.elasticsearch.host }}
Port 9200
Index {{ .Values.elasticsearch.index }}

This setup allows the sidecar to monitor log updates efficiently, pushing each entry to Elasticsearch as it appears in the log file.

2. Kubernetes Service: Stable Networking for Dynamic Pods​

A Service is an abstraction that provides a stable, permanent network interface for a set of Pods. Since Pod IP addresses are temporary, a Service routes requests to Pods and handles the load balancing across them.

Key Service Types​

  • ClusterIP: Exposes the Service only within the cluster.
  • NodePort: Exposes the Service on each node’s IP at a specific port.
  • LoadBalancer: Exposes the Service externally, often through a cloud provider’s load balancer.
  • Headless Service: Exposes each Pod individually, without load balancing, often used for direct client-to-Pod connections.

Example Service Architecture​

The Service selects Pods with specific labels and routes traffic to them dynamically. Here’s a simplified text diagram for a Service:

+-------------------+ +------------------+
| Client Request |------------>| Service |
+-------------------+ | (ClusterIP) |
+--------+---------+
|
|
+------------+-----------+
| |
| |
+-------v------+ +---------v------+
| Pod | | Pod |
| IP: Dynamic | | IP: Dynamic |
+--------------+ +----------------+

The Service maintains a list of Pods based on matching labels and routes incoming requests to any healthy Pod in the group, providing an abstraction layer for easy access.

3. Communication Between Pods and Services​

Kubernetes manages Pod-to-Service and Service-to-Pod communication using:

  1. Labels and Selectors: Services target Pods by their labels.
  2. Cluster DNS: Each Service receives a unique hostname, which can be used within the cluster to resolve the Service IP.
  3. Load Balancing: Services distribute requests across Pods, using internal load balancing mechanisms (IP tables, kube-proxy).

When a client inside the cluster calls a Service, Kubernetes DNS resolves the Service name to a virtual IP, and the Service routes the request to one of the associated Pods.

Dynamic Pod Communication Example​

The following diagram illustrates how a Service enables stable access to multiple Pods, even if the Pods are added or removed over time:

+-------------------+ +-----------------+
| Client |----------------->| Service: my-app |
+-------------------+ | IP: 10.96.0.1 |
+-------+---------+
|
+-------------------+-------------------+
| |
+--------v-----------+ +--------v------------+
| Pod | | Pod |
| Label: app=my-app | | Label: app=my-app |
+--------------------+ +---------------------+

Here, the client accesses my-app, and the Service dynamically routes the request to an available Pod with the matching label.

4. Summary​

  • Pods: Contain containers, share IP addresses, ephemeral in nature.
  • Services: Provide stable, permanent access to a group of Pods with specific labels, enable load balancing, and ensure that applications can consistently reach the backend Pods, even with dynamic scaling.

By decoupling Pod IP addresses from the client-facing endpoint, Services ensure reliable, scalable, and high-availability communication within Kubernetes applications.

Example Fluent Bit Configuration in configmap.yaml​

apiVersion: v1
kind: ConfigMap
metadata:
name: postgresql-log-config
data:
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info

[INPUT]
Name tail
Path /var/log/postgresql/postgresql.log
Tag postgresql

[OUTPUT]
Name es
Match *
Host {{ .Values.elasticsearch.host }}
Port 9200
Index {{ .Values.elasticsearch.index }}

This setup allows the sidecar to monitor log updates efficiently, pushing each entry to Elasticsearch as it appears in the log file.

Microservice meet with Spring Cloud and Kubernetes

· 13 min read

Introduction​

The objective is to design and deploy a group of microservices: Profile Service, Inventory Service, Order Service, Shipment Service, and an OAuth2 Service for centralized authentication. Key requirements include robust service discovery, API Gateway setup, secure communication, and efficient load balancing for high availability and fault tolerance in Kubernetes.

High-Level Architecture​

Each microservice will be deployed in a Kubernetes cluster with multiple replicas, leveraging Spring Cloud Kubernetes Discovery for service registration and Spring Cloud LoadBalancer for client-side load balancing. An API Gateway will handle external requests, providing routing, load balancing, and JWT-based authentication. Communication between services is secured with JWT tokens generated by the OAuth2 Service.

Service Design​

Each microservice is structured independently, deployed as a standalone Spring Boot application with individual responsibilities and access rules. Here’s an overview of the key services:

1. Profile Service​

Manages customer profiles, providing endpoints for CRUD operations on profile data. Accessible to services like Order and Shipment for customer information retrieval.

2. Inventory Service​

Tracks inventory levels and provides endpoints to update, retrieve, and manage inventory data. Called by the Order Service for real-time inventory status updates.

3. Order Service​

Processes customer orders, calculates totals, and interfaces with Inventory and Profile Services. This service is central to the architecture, relying on other services for a smooth customer experience.

4. Shipment Service​

Manages shipment tracking and logistics for orders, called by the Order Service once an order is processed and confirmed.

5. OAuth2 Service​

Acts as the authentication provider, issuing and validating JWT tokens for secure communication. The API Gateway uses these tokens to authenticate requests.

Order-Service Auth-Service Inventory-Service
| | |
|--- Request JWT ------------->| |
| | |
|<-- Receive JWT --------------| |
| | |
|--- Call Inventory ---------->| |
| (with JWT) | |
| |--- Validate JWT------------>|
| | |
| |<-- JWT Valid/Invalid--------|
| | |
|<-- Receive Inventory Data -- | |

Kubernetes Deployment and Service Discovery​

Each service will be deployed to Kubernetes as a Deployment with multiple replicas and exposed via a Service to enable discovery. Spring Cloud Kubernetes is used for service registration and discovery.

Configuring Kubernetes Service Discovery for Each Microservice​

To set up service discovery in Kubernetes, add the following dependencies to your pom.xml:

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-discovery</artifactId>
</dependency>

Configure the application.yml in each service to enable discovery:

spring:
cloud:
kubernetes:
discovery:
enabled: true #Enable service discovery
config:
enabled: true
loadbalancer:
enabled: true #Enable loadbalancer

This configuration registers each service with Kubernetes' internal DNS, allowing them to be discoverable by service name within the namespace. This way, each microservice is registered under its own Kubernetes service name, and can resolve other services (e.g., http://inventory-service:8080) when communicating with them. This configuration, combined with Kubernetes DNS, helps maintain an efficient, loosely coupled architecture.

Note: The Kubernetes Service Discovery configuration should indeed be included in each individual microservice’s configuration to ensure they can all register themselves with the Kubernetes DNS and be discoverable by other services in the namespace. This setup allows each microservice to dynamically locate and communicate with other services via Kubernetes' internal DNS naming conventions.

Why Configure Service Discovery in Each Microservice?

Each microservice is an independently deployed unit, so configuring Spring Cloud Kubernetes Discovery in each service's application.yml allows it to:

  1. Register with Kubernetes DNS under a unique service name, enabling internal routing across services.
  2. Access other services by simply using their service names, as Kubernetes DNS provides a consistent way to locate services within the cluster.

Note Here you might be noticed that both the discovery and load balancer components are configured within each microservice. This design allows each microservice to dynamically discover the current instances of other services (using the spring-cloud-kubernetes-discovery integration) and balance requests among those instances using the spring-cloud-loadbalancer component.

Service Registration​

Each microservice is deployed as a separate Deployment and Service in Kubernetes. Here’s an example of the inventory-service Kubernetes configuration:

To deploy and register microservices like an Inventory Service in Kubernetes, we use Spring Cloud’s Kubernetes Discovery capabilities, allowing services to register with Kubernetes’ DNS-based service registry automatically.

Step 1: Create a Kubernetes Service and Deployment for each microservice:

YAML files for defining the service and deployment:

# inventory-service-deployment.yml
apiVersion: v1
kind: Service
metadata:
name: inventory-service
spec:
selector:
app: inventory
ports:
- protocol: TCP
port: 8080
targetPort: 8080
type: ClusterIP # Internal access within the cluster

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: inventory-deployment
spec:
replicas: 2 # Number of instances (pods) of the Inventory Service
selector:
matchLabels:
app: inventory
template:
metadata:
labels:
app: inventory
spec:
containers:
- name: inventory
image: myregistry/inventory-service:latest
ports:
- containerPort: 8080

This configuration deploys inventory-service with two replicas, registered under the service name inventory-service. Other services can communicate with it via http://inventory-service:8080.

More comprehensive explainations When deploying an inventory-deployment with two replicas, you will indeed see two pods running in parallel in the specified namespace. Kubernetes names these pods based on the deployment name, followed by a unique suffix (not just sequential numbers), so you may see pod names like inventory-deployment-abc12 and inventory-deployment-def34.

In Kubernetes, these two pods are managed by a single inventory-service which acts as a layer of abstraction. When the client load balancer targets inventory-service, the service uses Kubernetes' built-in load balancing to forward requests to one of the available inventory-deployment pods.

How Kubernetes Service Load Balancing Works:

  1. Service Abstraction: The inventory-service has a Cluster IP that exposes the service within the cluster.
  2. Endpoints Selection: The service maintains an updated list of active pod IPs (called "endpoints") that match the deployment selector.
  3. Round-Robin Load Balancing: Kubernetes uses a round-robin approach by default. When a request reaches inventory-service, it selects one of the pod IPs associated with the deployment, balancing requests across pods automatically. The service doesn't keep state; each request is independently forwarded to one of the pod replicas.

Example Flow:

  1. Client Request: A client load balancer sends a request to the inventory-service endpoint.
  2. Service to Pod Routing: Kubernetes service routing picks an endpoint (pod IP) from the inventory-deployment pods and forwards the request to it.

For better control and observability, custom load balancer configurations can also be implemented to fine-tune how the requests are distributed, especially in setups requiring sticky sessions or advanced routing.

Step 2: Register Service with Kubernetes Discovery:

Spring Cloud Kubernetes automatically registers inventory-service based on the deployment labels. Enable Spring Cloud Discovery in your Spring Boot application via @EnableDiscoveryClient:

@EnableDiscoveryClient
@SpringBootApplication
public class InventoryServiceApplication {
public static void main(String[] args) {
SpringApplication.run(InventoryServiceApplication.class, args);
}
}

Step 3: Configure Discovery in application.yml:

spring:
cloud:
kubernetes:
discovery:
enabled: true
all-namespaces: true # Optional: enables discovery across namespaces

Load Balancing with Spring Cloud LoadBalancer​

To efficiently distribute incoming requests across available instances, Spring Cloud LoadBalancer provides a client-side load-balancing solution, automatically distributing traffic among instances of a service.

Enable Spring Cloud LoadBalancer by adding Spring Cloud LoadBalancer dependency:

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>

Configure a ServiceInstanceListSupplier bean for custom load balancing behavior if required:

import org.springframework.cloud.client.loadbalancer.ServiceInstanceListSupplier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class LoadBalancerConfig {

@Bean
public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier() {
return ServiceInstanceListSupplier.builder()
.withDiscoveryClient()
.withCaching()
.build();
}
}

Set Up Load Balancer with WebClient:

  • Configure a WebClient bean with load balancing to distribute requests among inventory-service instances.
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.context.annotation.Bean;
import org.springframework.web.reactive.function.client.WebClient;

@Configuration
public class WebClientConfig {

@Bean
@LoadBalanced
public WebClient.Builder loadBalancedWebClientBuilder() {
return WebClient.builder();
}
}

Use Load-Balanced WebClient in the calling service:

@Service
public class InventoryClient {

private final WebClient.Builder webClientBuilder;

public InventoryClient(WebClient.Builder webClientBuilder) {
this.webClientBuilder = webClientBuilder;
}

public Mono<Product> getProductById(String productId) {
return webClientBuilder.build()
.get()
.uri("http://inventory-service/products/{id}", productId)
.retrieve()
.bodyToMono(Product.class);
}
}

How Load Balancing Works:

  • Spring Cloud LoadBalancer, by default, uses a round-robin strategy to distribute requests.
  • It balances requests between the two inventory-service pods and automatically updates when new instances are added or removed.

  1. By default, Spring Cloud LoadBalancer will round-robin requests to different instances, or customize by implementing a custom LoadBalancerRequestTransformer.

API Gateway Setup​

The Spring Cloud API Gateway is deployed as a separate microservice, managing traffic and providing JWT-based authentication. The gateway routes requests to the appropriate services and load-balances among available replicas.

Routing and Load Balancing​

[External Client] --> [API Gateway]
|
| (Discovers routes from Kubernetes Discovery)
|
[inventory-service] <--+-- "lb://inventory-service"
[shipment-service] <--+-- "lb://shipment-service"
[order-service] <--+-- "lb://order-service"
[profile-service] <--+-- "lb://profile-service"
[other-service] <--+-- Discovered automatically

Configure the gateway’s application.yml to define routes to each service:

spring:
cloud:
gateway:
routes:
- id: profile
uri: http://profile-service
predicates:
- Path=/profile/**
- id: inventory
uri: http://inventory-service
predicates:
- Path=/inventory/**
default-filters:
- TokenRelay

The TokenRelay filter enables the gateway to forward JWT tokens to downstream services for authenticated requests.

Configuring JWT Authentication​

For security, JWT validation can be implemented in the gateway or within each service. Here’s how to configure JWT validation in the gateway:

  1. Add OAuth2 dependencies to the gateway:

    <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
    </dependency>
  2. Configure JWT validation in the gateway’s application.yml:

    spring:
    security:
    oauth2:
    resourceserver:
    jwt:
    issuer-uri: https://auth-server/oauth2/default

Microservice Communication​

Microservices communicate through Spring Cloud LoadBalancer and WebClient.

Using WebClient with Load Balancer​

To call the inventory-service from the order-service, configure WebClient with load balancing:

@Configuration
public class WebClientConfig {

@LoadBalanced
@Bean
public WebClient.Builder webClientBuilder() {
return WebClient.builder();
}
}

Example: Order Service Calls Profile Service​

Here’s how order-service might call profile-service:

@RestController
public class OrderController {

private final WebClient.Builder webClientBuilder;

@Autowired
public OrderController(WebClient.Builder webClientBuilder) {
this.webClientBuilder = webClientBuilder;
}

@GetMapping("/order/{orderId}")
public Mono<OrderDetails> getOrderDetails(@PathVariable String orderId) {
return webClientBuilder.build()
.get()
.uri("http://profile-service/profile/{orderId}", orderId) // The uri can be either http or lb(load balancer)
.retrieve()
.bodyToMono(OrderDetails.class);
}
}

Securing Microservices​

Secure Communication with mTLS and JWT

mTLS (mutual TLS) and JWT (JSON Web Token) are two primary ways to secure communication between microservices.

  • mTLS: Ensures that both the client and server authenticate each other, useful in a zero-trust network.
  • JWT: Adds stateless authentication, where each request includes a signed token with user and session data.

Each microservice validates JWT tokens received from the gateway, ensuring secure and authenticated communication.

JWT Authentication with API Gateway​

JWT Authentication is often managed at the API Gateway, validating tokens and authorizing requests before forwarding them to services.

In this approach, JWT validation is centralized in the gateway. Tokens are verified at the gateway and forwarded to services, reducing the validation load on individual services.

  1. Add Spring Security Dependency for JWT support:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
  1. Configure JWT Verification in Gateway:

Validate JWTs at the gateway level, allowing only authorized requests to reach downstream services.

@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter {

@Override
protected void configure(HttpSecurity http) throws Exception {
http
.authorizeRequests()
.antMatchers("/api/**").authenticated()
.and()
.oauth2ResourceServer().jwt();
}
}

Alternative: JWT Validation in Interceptors​

Alternatively, JWT validation can be handled within each microservice using a custom Interceptor, within services, add an Interceptor to validate JWTs before processing requests.

import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;

@Component
public class JwtInterceptor implements HandlerInterceptor {

@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
String token = request.getHeader("Authorization");
// Verify the JWT token
// Decode and validate claims
return isValidToken(token);
}
}

Pros and Cons of JWT and mTLS​

ApproachProsCons
mTLSHigh security with mutual authentication.Requires certificate management, adds complexity.
JWTStateless, easily scalable, widely supported.Token expiration handling; replay attacks if not carefully implemented.

<a name="kubernetes-deployment-and-auto-scaling"></a>
## Kubernetes Deployment and Auto-scaling

Scaling services dynamically is fundamental in Kubernetes. Kubernetes provides an effective way to handle **auto-scaling** of microservices based on resource utilization, e.g. CPU, memory, or custom metrics.

Each microservice is balanced using **Spring Cloud LoadBalancer**, ensuring even distribution across replicas.

> **Step 1**: Configuring Horizontal Pod Autoscaler (HPA)

1. Enable auto-scaling in Kubernetes based on CPU usage by defining an HPA resource.

**inventory-hpa.yaml**
```yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: inventory-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: inventory-deployment
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 75
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50

Step 2: Implementing Cluster Scaling

Use Kubernetes’ Cluster Autoscaler for scaling nodes as workload demand grows, ensuring optimal resource use without manual intervention.


Ensuring Secure Microservice Communication​

Securing communication between services is critical. Use mTLS (mutual TLS) and JWT tokens for service-to-service authentication within Kubernetes clusters.

Step 1: Configure Mutual TLS (mTLS)​

Istio or Linkerd can help enforce mTLS between services:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: your-namespace
spec:
mtls:
mode: STRICT

Summary​

  1. Use ConfigMaps and Secrets for configuration.
  2. Implement autoscaling with HPA.
  3. Secure communication with mTLS and JWT.
  4. Monitor with readiness/liveness probes and Prometheus/Grafana.

This guide provides an architecture blueprint for deploying secure, scalable, and efficient microservices in Kubernetes using Spring Cloud. The API Gateway simplifies external access, service discovery, and JWT-based authentication, while Kubernetes ensures high availability and resilience across services.

Comprehensive Guide to Kubernetes

· 7 min read

This guide provides a detailed overview of Kubernetes concepts, examples, use cases, installation YAML examples, and visual representations of the architecture.

Cluster​

A Kubernetes Cluster is a group of nodes (machines) that host and manage containerized applications. Each cluster has at least one master node, which controls the state of the cluster, and multiple worker nodes.

Example​

A cloud-based cluster with one master node and multiple worker nodes in Google Kubernetes Engine (GKE).

Use Case​

An ideal setup for running a production microservices application with high availability.

YAML Structure for Cluster Creation​

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker

Node​

A Node is a single machine within a Kubernetes cluster, capable of running one or more pods.

Example​

In a cloud environment, each node could be a virtual machine instance.

Use Case​

Nodes distribute workloads across the infrastructure.

YAML Snippet​

The node configuration is usually specified by the cluster management tool (e.g., GKE or EKS) rather than in YAML, as it pertains to the infrastructure layer.


Pod​

A Pod is the smallest deployable unit in Kubernetes, which can contain one or more containers.

Example​

A pod running a web server container and a logging container.

Use Case​

Running tightly coupled applications together within the same lifecycle.

YAML Structure for Pod​

apiVersion: v1
kind: Pod
metadata:
name: web-server-pod
spec:
containers:
- name: web-server
image: nginx
- name: logger
image: fluentd

Deployment​

A Deployment is a Kubernetes object for managing a group of identical pods. It provides capabilities like rolling updates and rollback.

Example​

A deployment with three replicas of a web server pod.

Use Case​

Maintaining a stable version of an application by managing pod replicas.

YAML Structure for Deployment​

apiVersion: apps/v1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: nginx:1.17.4

Service​

A Service provides a stable endpoint for accessing a set of pods.

Example​

A service that exposes an application to the public internet.

Use Case​

Load balancing traffic to an application across multiple pods.

YAML Structure for Service​

apiVersion: v1
kind: Service
metadata:
name: web-service
spec:
type: LoadBalancer
selector:
app: web
ports:
- protocol: TCP
port: 80
targetPort: 80

Ingress​

Ingress manages external access to services, typically using HTTP.

Example​

An ingress with rules to route traffic to different services based on URL paths.

Use Case​

Routing /api to backend services and /static to frontend services on the same domain.

YAML Structure for Ingress​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
- host: example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
- path: /static
pathType: Prefix
backend:
service:
name: static-service
port:
number: 80

ConfigMap​

A ConfigMap stores non-sensitive configuration data.

Example​

A ConfigMap for environment-specific variables.

Use Case​

Keeping configuration separate from the application code.

YAML Structure for ConfigMap​

apiVersion: v1
kind: ConfigMap
metadata:
name: config-demo
data:
DATABASE_URL: "jdbc:mysql://dbhost:3306/mydatabase"
DEBUG_LEVEL: "3"

Secret​

A Secret stores sensitive information, like passwords and tokens.

Example​

A secret containing API keys for external services.

Use Case​

Securing sensitive data without exposing it in the code.

YAML Structure for Secret​

apiVersion: v1
kind: Secret
metadata:
name: api-key-secret
type: Opaque
data:
api-key: c29tZS1yYW5kb20ta2V5Cg==

Persistent Volume (PV) and Persistent Volume Claim (PVC)​

A Persistent Volume provides storage, while a Persistent Volume Claim is a request for that storage.

Example​

A PV representing cloud disk storage and a PVC used by a database pod.

Use Case​

Ensuring data persistence for applications that require durable storage.

YAML Structure for PV and PVC​

apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-storage
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi

Namespace​

A Namespace is a way to divide cluster resources among multiple users.

Example​

Separate namespaces for development, staging, and production environments.

Use Case​

Organizing resources for different environments or teams.

YAML Structure for Namespace​

apiVersion: v1
kind: Namespace
metadata:
name: dev-environment

DaemonSet​

A DaemonSet ensures that specific pods run on each node.

Example​

Running a logging agent on every node in the cluster.

Use Case​

Managing node-wide services like logging or monitoring.

YAML Structure for DaemonSet​

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: logger
spec:
selector:
matchLabels:
name: logger
template:
metadata:
labels:
name: logger
spec:
containers:
- name: fluentd
image: fluent/fluentd

Job and CronJob​

Job runs a one-time task, while CronJob schedules a recurring task.

Example​

A Job to back up a database once and a CronJob to clean up logs daily.

Use Case​

Automating scheduled tasks or batch processing.

YAML Structure for Job and CronJob​

apiVersion: batch/v1
kind: Job
metadata:
name: backup-job
spec:
template:
spec:
containers:
- name: backup
image: backup-tool
restartPolicy: OnFailure

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: daily-cleanup
spec:
schedule: "0 0 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: cleanup
image: cleanup-tool
restartPolicy: OnFailure

Horizontal Pod Autoscaler (HPA)​

The HPA scales pod replicas automatically based on resource utilization.

Example​

Scaling web server pods based on CPU usage.

Use Case​

Scaling applications to handle variable workloads.

YAML Structure for HPA​

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: web-hpa
spec:
scaleTarget

Ref:
apiVersion: apps/v1
kind: Deployment
name: web-deployment
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 70

StatefulSet​

A StatefulSet manages stateful applications with unique and persistent storage.

Example​

A StatefulSet for a MongoDB replica set.

Use Case​

Managing stateful applications where each instance requires its own storage.

YAML Structure for StatefulSet​

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
replicas: 3
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
containers:
- name: mongo
image: mongo
ports:
- containerPort: 27017
name: mongo

Volume and StorageClass​

Volumes provide storage to pods, while StorageClass defines types of storage.

Example​

SSD-based storage for a database application.

Use Case​

Choosing storage for applications based on performance requirements.

YAML Structure for StorageClass and Volume​

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: fast-pv
spec:
storageClassName: fast
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce

NetworkPolicy​

NetworkPolicy defines network access rules between pods.

Example​

Restricting access to specific pods.

Use Case​

Securing inter-service communication.

YAML Structure for NetworkPolicy​

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress: []

Custom Resource Definitions (CRDs)​

CRDs extend Kubernetes by allowing users to create custom resources.

Example​

A custom resource to manage database schema migrations.

Use Case​

Adding domain-specific resources for applications.

YAML Structure for CRD​

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: migrations.mycompany.com
spec:
group: mycompany.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: migrations
singular: migration
kind: Migration

Helm Charts​

Helm is a package manager for Kubernetes, and Helm Charts are pre-configured packages.

Example​

A Helm chart for deploying a WordPress site.

Use Case​

Simplifying application deployment and management.

Example Helm Command​

helm install my-wordpress bitnami/wordpress

These concepts are essential for understanding and operating Kubernetes in production, allowing you to deploy, manage, and scale applications effectively. Each example and YAML configuration highlights the power and flexibility Kubernetes offers in a variety of scenarios.

Spark in Kubernetes

· 7 min read

This guide covers the steps to deploy an Apache Spark cluster with a master, two workers, and a Spark History Server on Kubernetes. It configures a GCS bucket as the default file system and installs Hive with PostgreSQL as its backend. An Ingress is set up for accessing the Spark UI and History Server.

Architecture Overview​

The setup includes the following components:

  • Spark Master and Workers: Deployed using StatefulSets for stable identity and persistence.
  • History Server: Deployed using a Deployment and configured to store Spark event logs in GCS.
  • Hive and PostgreSQL: PostgreSQL is used as the backend for Hive metadata, and Hive is deployed to store Spark tables.
  • Ingress: Provides external access to Spark Master UI and History Server UI.

Diagram: Architecture Overview​

+---------------------------+
| |
| Kubernetes |
| |
+---------------------------+
| |
| +---------+ +---------+|
| | Spark | | Spark ||
| | Master | | Workers ||
| +---------+ +---------+|
| |
| +-----------+ |
| | History | |
| | Server | |
| +-----------+ |
| | |
| Ingress |
+---------------------------+
|
External Access

Prerequisites​

  • Kubernetes Cluster with access to deploy resources.
  • kubectl command-line tool configured to access the cluster.
  • Helm for deploying Hive and PostgreSQL.
  • Google Cloud Storage (GCS) bucket created for Spark event logs.
  • Google Cloud SDK installed and configured for authentication to GCS.

1. Configuring Google Cloud Storage (GCS) Bucket​

  1. Create a GCS bucket if not already created:

    gsutil mb gs://your-spark-event-logs
  2. Set up a GCS service account key and save it as a Kubernetes secret. This will allow Spark to access GCS:

    kubectl create secret generic gcs-key \
    --from-file=key.json=/path/to/gcs-service-account-key.json

2. Deploying PostgreSQL for Hive Metadata​

PostgreSQL will act as the relational database for Hive’s metadata.

PostgreSQL YAML​

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgresql-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgresql
spec:
replicas: 1
selector:
matchLabels:
app: postgresql
template:
metadata:
labels:
app: postgresql
spec:
containers:
- name: postgresql
image: postgres:13
env:
- name: POSTGRES_DB
value: "hive_metastore"
- name: POSTGRES_USER
value: "hiveuser"
- name: POSTGRES_PASSWORD
value: "hivepassword"
volumeMounts:
- mountPath: "/var/lib/postgresql/data"
name: postgresql-data
volumes:
- name: postgresql-data
persistentVolumeClaim:
claimName: postgresql-pvc

3. Installing Hive Metastore​

Hive Metastore will manage metadata for Spark.

Hive Configuration​

  1. Download and install Hive Helm chart:

    helm repo add bitnami https://charts.bitnami.com/bitnami
    helm install hive bitnami/hive --set postgresql.enabled=false
  2. Update Hive configuration to use PostgreSQL.

# Values.yaml for Hive
hive:
metastore:
databaseType: postgres
jdbcUrl: jdbc:postgresql://postgresql-service/hive_metastore
user: hiveuser
password: hivepassword

4. Deploying Spark Master and Workers​

Spark Master and Worker StatefulSet YAML​

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: spark-master
spec:
serviceName: spark-master
replicas: 1
selector:
matchLabels:
app: spark
template:
metadata:
labels:
app: spark
role: master
spec:
containers:
- name: spark-master
image: spark:latest
ports:
- containerPort: 7077
- containerPort: 8080
env:
- name: SPARK_MASTER_HOST
value: "spark-master"

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: spark-worker
spec:
replicas: 2
selector:
matchLabels:
app: spark
role: worker
template:
metadata:
labels:
app: spark
role: worker
spec:
containers:
- name: spark-worker
image: spark:latest
ports:
- containerPort: 8081
env:
- name: SPARK_WORKER_CORES
value: "1"
- name: SPARK_WORKER_MEMORY
value: "1G"
- name: SPARK_MASTER
value: "spark://spark-master:7077"

5. Setting Up the Spark History Server​

The History Server will read event logs from the GCS bucket for job history.

Spark History Server YAML​

apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-history-server
spec:
replicas: 1
selector:
matchLabels:
app: spark-history
template:
metadata:
labels:
app: spark-history
spec:
containers:
- name: spark-history-server
image: spark:latest
env:
- name: SPARK_HISTORY_OPTS
value: "-Dspark.history.fs.logDirectory=gs://your-spark-event-logs"
volumeMounts:
- mountPath: /opt/spark/work-dir
name: gcs-key
subPath: key.json
volumes:
- name: gcs-key
secret:
secretName: gcs-key

6. Configuring Ingress​

Ingress will expose the Spark Master UI and History Server UI.

Ingress YAML​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: spark-ingress
spec:
rules:
- host: spark.example.com
http:
paths:
- path: /master
pathType: Prefix
backend:
service:
name: spark-master
port:
number: 8080
- path: /history
pathType: Prefix
backend:
service:
name: spark-history-server
port:
number: 18080

7. Testing the Deployment with Spark Job​

To verify the installation, use spark-submit to run a simple calculation for PI.

  1. Submit a PI calculation job using spark-submit.

    kubectl exec -it spark-master-0 -- spark-submit \
    --class org.apache.spark.examples.SparkPi \
    --master spark://spark-master:7077 \
    /opt/spark/examples/jars/spark-examples_2.12-3.0.1.jar 1000
  2. Check the Spark Master and History Server UIs

to confirm job execution and view logs.

Furthermore​

StatefulSet vs Deployment​

In the official Helm chart for Apache Spark, the choice of StatefulSet for Spark Master and Worker, and Deployment for the Spark History Server, aligns with the specific requirements and behaviors of each component. Here’s a breakdown of the reasons for these design choices:

1. Spark Master and Worker as StatefulSets​

  • Persistence and Stability: The Spark Master and Worker nodes need stable, unique identities to communicate reliably. Using StatefulSet ensures that each replica (instance) has a stable network identity (hostname) and, if configured, persistent storage.
  • Leader Election and Failover: Spark Master in standalone mode has a primary leader and backup node architecture, where the leader is responsible for the cluster state. StatefulSets allow these nodes to maintain consistent, unique identifiers, which is crucial for leader election and failover without disrupting the Spark cluster.
  • Worker Registration: Spark Workers register with the Master, and having a unique, stable network identity ensures workers don’t inadvertently re-register or conflict with each other after restarts or failures. StatefulSets are the preferred way to handle this kind of behavior.

2. Spark History Server as a Deployment​

  • Stateless Operation: The Spark History Server is fundamentally a stateless application. It reads from a pre-configured storage (such as HDFS or S3) where application logs are stored but doesn’t require persistent storage for its own operation. Using a Deployment is appropriate here, as the history server can be easily replicated or scaled without maintaining unique identities.
  • Easier Scaling and Update Management: Deployments provide automatic scaling and rolling updates, which are more suitable for stateless services. If you need to increase the availability of the history server, you can simply increase the replica count in the Deployment, and Kubernetes will handle the scaling automatically.
  • Failure Recovery: Since there’s no dependency on unique network identities or persistent state, if the History Server pod fails, it can be easily recreated without any additional orchestration, which is exactly what Deployments are optimized for.

Summary Table​

ComponentKubernetes KindPurposeReasoning
Spark MasterStatefulSetPrimary and backup master for Spark clusterNeeds stable identity for cluster coordination and leader election
Spark WorkerStatefulSetWorker nodes that register with Spark MasterStable network identity ensures reliable registration and communication with Master
Spark History ServerDeploymentReads logs and provides historical application dataStateless and doesn’t need unique identities; easier to scale and restart as a Deployment

This setup optimizes the deployment structure for each Spark component's unique requirements, leveraging Kubernetes' StatefulSet and Deployment resources for their intended purposes.

Troubleshooting Kubernetes: Pod Stuck in Terminating State After Deletion

· 6 min read

Introduction​

This document provides a detailed troubleshooting guide for a common Kubernetes issue where a Pod remains in the Terminating state after being deleted using the kubectl delete command. The analysis includes examining logs, identifying suspicious points, and understanding the underlying causes.


Problem Background​

After deleting a business Pod using the kubectl delete command, the Pod remained in the Terminating state.


Cause Analysis​

Initial Investigation​

To begin with, the kubectl describe command was used to inspect the Pod:

[root@node1 ~]# kubectl describe pod -n xxx cam1-78b6fc6bc8-cjsw5

No obvious anomalies were found in the output.

Kubelet Logs Analysis​

Next, the kubelet component logs were examined for any anomalies during the Pod deletion process. The following logs were filtered for relevance:

I0728 16:24:57.339295 9744 kubelet.go:1904] SyncLoop (DELETE, "api"): "cam1-78b6fc6bc8-cjsw5_cam(5c948341-c030-4996-b888-f032577d97b0)"
I0728 16:24:57.339720 9744 kuberuntime_container.go:581] Killing container "docker://a73082a4a9a4cec174bb0d1c256cc11d804d93137551b9bfd3e6fa1522e98589" with 60 second grace period
I0728 16:25:18.259418 9744 kubelet.go:1904] SyncLoop (DELETE, "api"): "cam1-78b6fc6bc8-cjsw5_cam(5c948341-c030-4996-b888-f032577d97b0)"
2021-07-28 16:25:19.247 [INFO][394011] ipam.go 1173: Releasing all IPs with handle 'cam.cam1-78b6fc6bc8-cjsw5'
2021-07-28 16:25:19.254 [INFO][393585] k8s.go 498: Teardown processing complete.

Suspicious Point 1: Pod IP Retrieval Failure​

W0728 16:25:19.303513 9744 docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "cam1-78b6fc6bc8-cjsw5_cam": Unexpected command output Device "eth0" does not exist.
with error: exit status 1

Suspicious Point 2: Unmount Failure​

E0728 16:25:20.939400 9744 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g podName:5c948341-c030-4996-b888-f032577d97b0 nodeName:}" failed. No retries permitted until 2021-07-28 16:25:21.439325811 +0800 CST m=+199182.605079651 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"diag-log\" (UniqueName: \"kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g\") pod \"5c948341-c030-4996-b888-f032577d97b0\" (UID: \"5c948341-c030-4996-b888-f032577d97b0\") : Unmount failed: exit status 32\nUnmounting arguments: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g\nOutput: umount: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g:Target is busy.\n (In some case using lsof(8) or fuser(1) can\n find useful information about the process using the device)\n\n"

Suspicious Points​

Suspicious Point 1: Pod IP Retrieval Failure​

The log indicates a failure to retrieve the Pod IP:

W0728 16:25:19.303513 9744 docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "cam1-78b6fc6bc8-cjsw5_cam": Unexpected command output Device "eth0" does not exist.
with error: exit status 1

This error was traced to the getIP method in the docker_sandbox.go file:

func (ds *dockerService) getIP(podSandboxID string, sandbox *dockertypes.ContainerJSON) string {
if sandbox.NetworkSettings == nil {
return ""
}
if networkNamespaceMode(sandbox) == runtimeapi.NamespaceMode_NODE {
// For sandboxes using host network, the shim is not responsible for
// reporting the IP.
return ""
}

// Don't bother getting IP if the pod is known and networking isn't ready
ready, ok := ds.getNetworkReady(podSandboxID)
if ok && !ready {
return ""
}

ip, err := ds.getIPFromPlugin(sandbox)
if err == nil {
return ip
}

if sandbox.NetworkSettings.IPAddress != "" {
return sandbox.NetworkSettings.IPAddress
}
if sandbox.NetworkSettings.GlobalIPv6Address != "" {
return sandbox.NetworkSettings.GlobalIPv6Address
}

// Error log here
klog.Warningf("failed to read pod IP from plugin/docker: %v", err)
return ""
}

Suspicious Point 2: Unmount Failure​

The unmount failure was logged as an ERROR:

E0728 16:25:20.939400 9744 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g podName:5c948341-c030-4996-b888-f032577d97b0 nodeName:}" failed. No retries permitted until 2021-07-28 16:25:21.439325811 +0800 CST m=+199182.605079651 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"diag-log\" (UniqueName: \"kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g\") pod \"5c948341-c030-4996-b888-f032577d97b0\" (UID: \"5c948341-c030-4996-b888-f032577d97b0\") : Unmount failed: exit status 32\nUnmounting arguments: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g\nOutput: umount: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g:Target is busy.\n (In some case using lsof(8) or fuser(1) can\n find useful information about the process using the device)\n\n"

Pod Deletion Process​

The deletion of a Pod in Kubernetes follows these steps:

  1. API Server DELETE Call: The kubectl delete command triggers a DELETE request to the Kubernetes API server with a default grace period of 30 seconds.
  2. Update Pod Metadata: The Pod's metadata is updated with DeletionTimestamp and DeletionGracePeriodSeconds fields, but the Pod is not yet deleted from etcd.
  3. Kubelet Event Handling: The kubelet component listens for Pod updates and initiates the killPod() method.
  4. Second DELETE Call: The kubelet makes a second DELETE request to the API server with a grace period of 0 seconds.
  5. Pod Deletion from etcd: The API server deletes the Pod from etcd.

Verification and Testing​

To verify the deletion process, a test was conducted in a test environment:

[root@node2 ~]# kubectl delete pod -n xxx testpodrc2-7b749f6c9c-qh68l
pod "testpodrc2-7b749f6c9c-qh68l" deleted

Filtered kubelet logs:

I0730 13:27:31.854178 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:31.854511 24588 kuberuntime_container.go:581] Killing container "docker://e2a1cd5f2165e12cf0b46e12f9cd4d656d593f75e85c0de058e0a2f376a5557e" with 30 second grace period
I0730 13:27:32.203167 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:32.993294 24588 kubelet.go:1933] SyncLoop (PLEG): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)", event: &pleg.PodLifecycleEvent{ID:"85ee282f-a843-4f10-a99c-79d447f83f2a", Type:"ContainerDied", Data:"e2a1cd5f2165e12cf0b46e12f9cd4d656d593f75e85c0de058e0a2f376a5557e"}
I0730 13:27:32.993428 24588 kubelet.go:1933] SyncLoop (PLEG): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)", event: &pleg.PodLifecycleEvent{ID:"85ee282f-a843-4f10-a99c-79d447f83f2a", Type:"ContainerDied", Data:"c6a587614976beed0cbb6e5fabf70a2d039eec6c160154fce007fe2bb1ba3b4f"}
I0730 13:27:34.072494 24588 kubelet_pods.go:1090] Killing unwanted pod "testpodrc2-7b749f6c9c-qh68l"
I0730 13:27:40.084182 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:40.085735 24588 kubelet.go:1898] SyncLoop (REMOVE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"

Conclusion​

The analysis revealed that the Pod remained in the Terminating state due to a failure in unmounting the GlusterFS volume. The unmount failure was caused by the volume being busy, which prevented the second DELETE

Using GlusterFS with Kubernetes

· 6 min read

Introduction​

This article provides a comprehensive guide on using GlusterFS with Kubernetes. It covers the basics of GlusterFS, its architecture, and how to mount GlusterFS as the file system for a Pod in Kubernetes. Additionally, it compares GlusterFS with other main alternatives in the storage landscape.


GlusterFS Overview​

GlusterFS (Gluster File System) is an open-source, distributed file system designed to scale out in building-block fashion to store petabytes of data. It aggregates various storage servers over Ethernet or InfiniBand RDMA interconnects into one large parallel network file system. GlusterFS is part of the Gluster project, an effort by Red Hat to provide a unified distributed storage solution.

Key Features​

  1. Scalability: GlusterFS can scale to several petabytes and handle thousands of clients.
  2. Flexibility: It uses a stackable user space design, allowing for flexible topologies for data storage and retrieval.
  3. High Availability: GlusterFS provides features like replication and failover to ensure data availability and reliability.
  4. Performance: It uses various data access protocols like NFS, SMB, and native GlusterFS protocol to optimize performance.
  5. Elasticity: GlusterFS can dynamically add or remove storage resources without disrupting services.

Architecture​

GlusterFS architecture is based on a client-server model, where the storage servers (bricks) are the servers, and the clients access the data through the GlusterFS client. The architecture is modular, allowing for various configurations and optimizations.

  1. Bricks: Basic units of storage, represented as directories on servers.
  2. Volumes: Logical collections of bricks. Data is striped, replicated, or distributed across these bricks.
  3. Translators: Modules that perform various functions like I/O, network, and protocol handling.

Use Cases​

  • Big Data and Analytics: Suitable for storing and processing large datasets.
  • Media Streaming: Efficiently handles large media files.
  • Backup and Archiving: Provides reliable storage for backups and archives.
  • Cloud Storage: Acts as a scalable storage backend for cloud services.

Mounting GlusterFS with Kubernetes​

To use GlusterFS as the file system for a Pod in Kubernetes, you need to create a Persistent Volume (PV) and a Persistent Volume Claim (PVC). Below are the steps to achieve this:

Step 1: Install GlusterFS Client​

Ensure that the GlusterFS client (glusterfs-client) is installed on all Kubernetes nodes. This client is necessary for mounting GlusterFS volumes.

sudo apt-get install glusterfs-client

Step 2: Create a GlusterFS Volume​

Create a GlusterFS volume on your GlusterFS cluster. For example:

gluster volume create k8s-volume replica 3 server1:/brick1 server2:/brick2 server3:/brick3 force
gluster volume start k8s-volume

Step 3: Create a Persistent Volume (PV)​

Create a PV that references the GlusterFS volume.

apiVersion: v1
kind: PersistentVolume
metadata:
name: gluster-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
glusterfs:
endpoints: "glusterfs-cluster"
path: "k8s-volume"
readOnly: false

Step 4: Create a Persistent Volume Claim (PVC)​

Create a PVC that binds to the PV.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gluster-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi

Step 5: Use the PVC in a Pod​

Mount the PVC in a Pod.

apiVersion: v1
kind: Pod
metadata:
name: gluster-pod
spec:
containers:
- name: app
image: nginx
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: gluster-volume
volumes:
- name: gluster-volume
persistentVolumeClaim:
claimName: gluster-pvc

Textual Diagram of GlusterFS Architecture​

+-------------------+ +-------------------+ +-------------------+
| | | | | |
| GlusterFS Client |<----->| GlusterFS Server |<----->| GlusterFS Server |
| | | | | |
+-------------------+ +-------------------+ +-------------------+
| | |
| | |
v v v
+-------------------+ +-------------------+ +-------------------+
| | | | | |
| Brick 1 | | Brick 2 | | Brick 3 |
| | | | | |
+-------------------+ +-------------------+ +-------------------+

Comparison with Other Main Alternatives​

NFS (Network File System)​

  • Pros:
    • Simple to set up and manage.
    • Widely supported and well-documented.
  • Cons:
    • Single point of failure.
    • Limited scalability compared to GlusterFS.
    • Performance can degrade with high latency.

Ceph​

  • Pros:
    • Highly scalable and fault-tolerant.
    • Supports object, block, and file storage.
    • Active community and strong support.
  • Cons:
    • Complex to set up and manage.
    • Requires significant resources for optimal performance.

Amazon EFS (Elastic File System)​

  • Pros:
    • Fully managed service.
    • Highly available and durable.
    • Scales automatically with no need to provision storage.
  • Cons:
    • Cloud-specific, not suitable for on-premises deployments.
    • Higher cost compared to self-managed solutions.

OpenEBS​

  • Pros:
    • Kubernetes-native storage solution.
    • Easy to deploy and manage.
    • Supports various storage engines.
  • Cons:
    • Younger project with a smaller community.
    • May require more tuning for performance.

Conclusion​

GlusterFS is a powerful, scalable, and flexible distributed file system that can be effectively integrated with Kubernetes for persistent storage solutions. By following the steps outlined in this guide, you can mount GlusterFS volumes in your Kubernetes Pods. Additionally, understanding the architecture and comparing GlusterFS with other storage alternatives helps in making informed decisions for your storage needs.


References​

  1. GlusterFS Official Documentation
  2. Red Hat Gluster Storage
  3. GlusterFS GitHub Repository
  4. GlusterFS Wiki
  5. Kubernetes Documentation - Persistent Volumes
  6. NFS Documentation
  7. Ceph Documentation
  8. Amazon EFS Documentation
  9. OpenEBS Documentation

By following these steps and understanding the architecture and comparisons, you can effectively use GlusterFS with Kubernetes and make informed decisions about storage solutions for your Kubernetes clusters.

Analyzing High Memory Usage in Kubernetes

· 4 min read

Introduction​

In Kubernetes environments, monitoring resource usage is crucial for maintaining optimal performance and resource allocation. This article delves into a specific case where a user observed high memory usage by a Harbor pod on one of the nodes, leading to a discrepancy between kubectl top and docker stats results. We will analyze the cause, verify the memory calculation methods, and provide a solution to accurately monitor memory usage.


Problem Background​

A user observed that one of the nodes in their Kubernetes cluster showed high memory usage by a Harbor pod, approximately 3.7G, which seemed unusually high compared to other pods.

[root@node02 ~]# kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP
node01 Ready master 10d v1.15.12 100.1.0.10 <none>
node02 Ready master 12d v1.15.12 100.1.0.11 <none>
node03 Ready master 10d v1.15.12 100.1.0.12 <none>

[root@node02 ~]# kubectl top pod -A | grep harbor
kube-system harbor-master1-sxg2l 15m 150Mi
kube-system harbor-master2-ncvb8 8m 3781Mi
kube-system harbor-master3-2gdsn 14m 227Mi

Cause Analysis​

To understand the memory usage, the user compared the results from kubectl top and docker stats. Theoretically, docker stats should provide a more accurate result.

Using docker stats​

[root@node02 ~]# docker stats | grep harbor
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM %
10a230bee3c7 k8s_nginx_harbor-master2-xxx 0.02% 14.15MiB / 94.26GiB 0.01%
6ba14a04fd77 k8s_harbor-portal_harbor-master2-xxx 0.01% 13.73MiB / 94.26GiB 0.01%
324413da20a9 k8s_harbor-jobservice_harbor-master2-xxx 0.11% 21.54MiB / 94.26GiB 0.02%
d880b61cf4cb k8s_harbor-core_harbor-master2-xxx 0.12% 33.2MiB / 94.26GiB 0.03%
186c064d0930 k8s_harbor-registryctl_harbor-master2-xxx 0.01% 8.34MiB / 94.26GiB 0.01%
52a50204a962 k8s_harbor-registry_harbor-master2-xxx 0.06% 29.99MiB / 94.26GiB 0.03%
86031ddd0314 k8s_harbor-redis_harbor-master2-xxx 0.14% 11.51MiB / 94.26GiB 0.01%
6366207680f2 k8s_harbor-database_harbor-master2-xxx 0.45% 8.859MiB / 94.26GiB 0.01%

The total memory usage from docker stats was around 140M, significantly lower than the 3.7G reported by kubectl top.


Verification of Memory Calculation​

The discrepancy arises from the different methods used by kubectl top and docker stats to calculate memory usage.

  • kubectl top Calculation: memory.usage_in_bytes - inactive_file
  • docker stats Calculation: memory.usage_in_bytes - cache

If the cache is large, kubectl top will report a higher memory usage.

Verifying with memory_stats​

To verify, the user fetched the memory_stats data using Docker's API:

curl -s --unix-socket /var/run/docker.sock http:/v1.24/containers/xxx/stats | jq ."memory_stats"

Sample memory_stats data:

"memory_stats": {
"usage": 14913536,
"max_usage": 15183872,
"stats": {
"active_anon": 14835712,
"active_file": 0,
"cache": 77824,
"dirty": 0,
"hierarchical_memory_limit": 101205622784,
"hierarchical_memsw_limit": 9223372036854772000,
"inactive_anon": 4096,
"inactive_file": 73728,
...
}

By applying the formulas, the user confirmed that the results from kubectl top and docker stats were consistent with the expected calculations.


Understanding Cache Usage​

The high cache usage in Harbor was primarily due to the registry component, which handles Docker image storage and involves extensive read/write operations on image layers.

"memory_stats": {
"usage": 5845127168,
"max_usage": 22050988032,
"stats": {
"active_anon": 31576064,
"active_file": 3778052096,
"cache": 5813551104,
"dirty": 0,
"hierarchical_memory_limit": 101205622784,
"hierarchical_memsw_limit": 9223372036854772000,
"inactive_anon": 0,
"inactive_file": 2035499008,
...
}

Solution​

To address the discrepancy, the user was informed that kubectl top includes cache usage, which can be misleading. The cache, being non-critical, can be reclaimed by the system during memory pressure or manually released. The user was advised to use docker stats for a more accurate representation of actual memory usage.


Conclusion​

This case study highlights the importance of understanding the different methods used by Kubernetes and Docker to calculate memory usage. By verifying the calculations and understanding the role of cache, users can make more informed decisions about resource management in their Kubernetes clusters. Using docker stats provides a more accurate view of memory usage, especially in scenarios where cache usage is significant.

Kubernetes StatefulSet and Deployment

· 4 min read

In Kubernetes, StatefulSet and Deployment are two important concepts for managing workloads, each tailored to specific types of applications.


Overview: StatefulSet vs. Deployment​

1. Deployment​

  • A Deployment manages stateless applications, where each instance (Pod) is identical and can be easily replaced without any dependency on the previous instance’s state.
  • It allows for easy scaling up or down, rolling updates, and rollback.
  • Deployments are ideal for applications that don’t need unique identities or persistent storage tied to each Pod.

2. StatefulSet​

  • A StatefulSet is used to manage stateful applications where each Pod maintains a unique identity and persistent storage.
  • Pods created by StatefulSets are given fixed identities, which are essential for applications that depend on stable network IDs or persistent data volumes.
  • It ensures an ordered and predictable sequence for Pod creation, deletion, and scaling operations, making it suitable for stateful applications.

Key Differences Between Deployment and StatefulSet​

FeatureDeploymentStatefulSet
Use CaseStateless applicationsStateful applications
Pod IdentityAll Pods are identicalEach Pod has a unique, stable identity
Persistent StorageNot tied to specific Pods; ephemeral by defaultPersistent Volume Claims (PVC) linked to each Pod
Pod ManagementUnordered Pod creation, deletion, or scalingOrdered Pod creation, deletion, and scaling
ScalingIndependent scaling; any Pod can be replacedScaling maintains Pod identity; Pods are replaced in order
NetworkingPods are accessed via a common service nameEach Pod gets a unique DNS endpoint

When to Use Deployment vs. StatefulSet​

Deployment Use Cases:​

  • Stateless Applications: For applications that don’t require persistent storage or a unique identity, like web servers, RESTful APIs, front-end services, and microservices.
  • Scale-Out Workloads: When you need to scale the application horizontally with many replicas, such as with compute-intensive tasks (e.g., image processing).
  • CI/CD Pipelines: Deployment objects are suitable for managing frequent updates and rollbacks in stateless applications.
  • Load-Balanced Services: For applications where all instances serve the same purpose and can respond to requests in a round-robin or load-balanced fashion.

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
replicas: 3
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: frontend
image: nginx:latest

StatefulSet Use Cases:​

  • Databases: For databases like MySQL, PostgreSQL, or Cassandra, where each instance needs persistent storage and must be able to rejoin the cluster with the same identity after a restart.
  • Distributed Applications: For clustered or distributed applications that need stable network identities, such as Apache Kafka, Zookeeper, or Redis in a master-slave configuration.
  • Ordered Processing: For workloads that require ordered startup and shutdown, maintaining consistency across the replicas.
  • Persistent Storage Requirements: For applications that need durable storage across restarts, such as analytics applications or logging systems that rely on data persistence.

Example:

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: database
spec:
serviceName: "database-service"
replicas: 3
selector:
matchLabels:
app: database
template:
metadata:
labels:
app: database
spec:
containers:
- name: db
image: mysql:5.7
volumeMounts:
- name: data
mountPath: /var/lib/mysql
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi

Real-World Scenarios​

  1. Use StatefulSet for a Database Cluster:

    • Applications like Cassandra or MongoDB require that each node (or replica) has a unique identifier, stable storage, and maintains a specific order in the cluster.
    • StatefulSet will ensure each database Pod keeps its persistent storage (e.g., transaction logs) and its network identity, allowing for seamless cluster membership and fault recovery.
  2. Use Deployment for a Web Application Front-End:

    • Stateless applications like front-end services, web servers, and microservices, which don’t require persistent storage, are best managed by Deployments.
    • Deployments allow fast horizontal scaling to handle fluctuating traffic, automated rolling updates to support frequent releases, and easy recovery from failure.
  3. Use StatefulSet for Messaging Queues:

    • Applications like Kafka, RabbitMQ, and Redis often require ordered replica management, durable storage, and unique identifiers for cluster consistency.
    • StatefulSet will ensure that the pods are started in sequence and keep their storage intact across restarts.
  4. Use Deployment for API Gateways:

    • API Gateways, which act as proxies or load balancers for backend services, are stateless by design. Deployments allow for fast failover and load distribution while handling updates without data dependency concerns.

In summary, choose StatefulSet for workloads needing state persistence, stable identities, and ordered management (typically databases and clustered applications). Opt for Deployment for stateless, load-balanced, and horizontally scalable applications that need easy updates and high availability.

Troubleshooting High Currency Issues in Kubernetes - Continued

· 4 min read

Pod Restart CrashLoopBackOff Loop Due to High Concurrency (Continued)


Introduction​

Following the previous issue, the environment experienced another instance of Harbor and Calico pods restarting due to failed health checks. Additionally, using kubectl commands to enter the pods was extremely slow or timed out. This article continues the troubleshooting process to identify and resolve the root cause.


Problem Background​

After a period, the environment again faced issues with Harbor and Calico pods restarting due to failed health checks. Using kubectl commands to enter the pods was slow or timed out.

[root@node01 ~]# kubectl exec -it -n system node1-59c9475bc6-zkhq5 bash
^

Cause Analysis​

The root cause of the repeated restarts was previously identified as health check timeouts, with TCP connections stuck in the SYN_SENT phase.

[root@node01 ~]# netstat -anp | grep 23380
tcp 0 0 127.0.0.1:23380 0.0.0.0:* LISTEN 38914/kubelet
tcp 0 0 127.0.0.1:38983 127.0.0.1:23380 SYN_SENT -

Verification of Previous Fixes​

First, check if the previous kernel parameter adjustments are still in place.

[root@node01 ~]# cat /etc/sysctl.conf
net.ipv4.tcp_max_syn_backlog = 32768
net.core.somaxconn = 32768

Verify if the changes have taken effect.

[root@node01 ~]# ss -lnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 32768 127.0.0.1:23380 *:*

Despite the parameters being correctly set, the issue persisted.


Exploring Additional Causes​

Given the global impact of the issue, it was likely due to a system-level performance bottleneck. High business load affects CPU, memory, disk, and connection counts. Since the connections were long-lived, the file descriptor limit could be a factor.


Identifying File Descriptor Limits​

Check the number of open file descriptors.

[root@node01 ~]# lsof -p 45775 | wc -l
17974

[root@node01 ~]# lsof -p 45775 | grep "sock" | wc -l
12051

The process had over 10,000 open sockets, exceeding the default limit of 1024.

[root@node01 ~]# ulimit -n
1024

Despite exceeding the limit, the business pod did not report "too many open files" errors.

[root@node01 ~]# kubectl logs -n system node1-59c9475bc6-zkhq5
start config
...

Solution​

Temporarily increase the file descriptor limit.

[root@node01 ~]# ulimit -n 65535
[root@node01 ~]# ulimit -n
65535

After the adjustment, kubectl commands responded normally, and the SYN_SENT phase issue was resolved.

[root@node01 ~]# kubectl exec -it -n system node1-59c9475bc6-zkhq5 bash
[root@node1-59c9475bc6-zkhq5]# exit
[root@node01 ~]# kubectl exec -it -n system node1-59c9475bc6-zkhq5 bash
[root@node1-59c9475bc6-zkhq5]# exit
[root@node01 ~]# kubectl exec -it -n system node1-59c9475bc6-zkhq5 bash
[root@node1-59c9475bc6-zkhq5]# exit

[root@node01 ~]# netstat -anp | grep 23380
tcp 0 0 127.0.0.1:23380 0.0.0.0:* LISTEN 38914/kubelet
tcp 0 0 127.0.0.1:56369 127.0.0.1:23380 TIME_WAIT -
tcp 0 0 127.0.0.1:23380 127.0.0.1:57601 TIME_WAIT -
tcp 0 0 127.0.0.1:23380 127.0.0.1:57479 TIME_WAIT -

Conclusion​

This case study highlights the importance of considering system-level parameters, such as file descriptor limits, in high-concurrency scenarios. Adjusting these parameters can resolve performance bottlenecks and ensure stable operation of Kubernetes environments. For environments with high business load, it is strongly recommended to perform comprehensive operating system-level performance optimizations.

Troubleshooting High Currency Issues in Kubernetes

· 5 min read

Pod Restart CrashLoopBackOff Loop Due to High Concurrency


Introduction​

In a Kubernetes cluster environment, a business operation involving extensive configuration distribution (lasting several hours or more) led to repeated restarts of Calico pods. This article details the troubleshooting process, focusing on identifying the root cause and implementing a solution.


Problem Background​

During a high-concurrency operation, Calico pods in the Kubernetes cluster began to restart repeatedly.

[root@node02 ~]# kubectl get pod -n kube-system -owide | grep node01
calico-kube-controllers-6f59b8cdd8-8v2qw 1/1 Running 0 4h45m 10.10.119.238 node01 <none> <none>
calico-node-b8w2b 1/1 CrashLoopBackOff 43 3d19h 10.10.119.238 node01 <none> <none>
coredns-795cc9c45c-k7qpb 1/1 Running 0 4h45m 177.177.237.42 node01 <none> <none>
...

Analysis Process​

Upon observing the CrashLoopBackOff state of the pods, the first step was to use kubectl describe to gather more information.

[root@node02 ~]# kubectl describe pod -n kube-system calico-node-b8w2b
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 58m (x111 over 3h12m) kubelet, node01 (combined from similar events): Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Normal Pulled 43m (x36 over 3d19h) kubelet, node01 Container image "calico/node:v3.15.1" already present on machine
Warning Unhealthy 8m16s (x499 over 3h43m) kubelet, node01 Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning BackOff 3m31s (x437 over 3h3m) kubelet, node01 Back-off restarting failed container

The events indicated that the liveness probe was failing due to a connection timeout.


Investigating the Issue​

Manual execution of the health check command revealed slow response times, which were significantly higher than the usual millisecond-level response.

[root@node01 ~]# time curl -i http://localhost:9099/liveness
HTTP/1.1 204 No Content
Date: Tue, 15 Jun 2021 06:24:35 GMT
real 0m1.012s
user 0m0.003s
sys 0m0.005s

Logs from Calico's bird, confd, and felix components did not show any apparent errors. Checking the port status indicated that it was listening correctly.

[root@node02 ~]# netstat -anp | grep 9099
tcp 0 0 127.0.0.1:9099 0.0.0.0:* LISTEN 1202/calico-node
tcp 0 0 127.0.0.1:9099 127.0.0.1:56728 TIME_WAIT -
tcp 0 0 127.0.0.1:56546 127.0.0.1:9099 TIME_WAIT -

TCP Connection Timeout​

Given the connection timeout error, the TCP connection state was observed.

[root@node01 ~]# netstat -na | awk '/^tcp/{s[$6]++}END{for(key in s) print key,s[key]}'
LISTEN 49
ESTABLISHED 284
SYN_SENT 4
TIME_WAIT 176

The top command showed that the business Java process was consuming high CPU, peaking at over 2000%.

[root@node01 ~]# top
top - 14:28:57 up 13 days, 27 min, 2 users, load average: 9.55, 9.93, 9.91
Tasks: 1149 total, 1 running, 1146 sleeping, 0 stopped, 2 zombie
%Cpu(s): 16.0 us, 2.9 sy, 0.0 ni, 80.9 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 15249982+total, 21419184 free, 55542588 used, 75538048 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 94226176 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6754 root 20 0 66.8g 25.1g 290100 S 700.0 17.3 2971:49 java
25214 root 20 0 6309076 179992 37016 S 36.8 0.1 439:06.29 kubelet
20331 root 20 0 3196660 172364 24908 S 21.1 0.1 349:56.64 dockerd

Understanding CPU Load​

Checking the total number of CPU cores and comparing it with the load average and CPU usage indicated that the load was not excessively high.

[root@node01 ~]# cat /proc/cpuinfo | grep "physical id" | sort | uniq | wc -l
48
[root@node01 ~]# cat /proc/cpuinfo | grep "cpu cores" | uniq
cpu cores: 1

Given the high concurrency, the issue was likely related to the TCP connection setup. Understanding the TCP connection phases helped pinpoint the problem.


Kernel Parameter Tuning​

The issue was traced to the SYN_SENT phase of TCP connections. Adjusting kernel parameters related to TCP connections resolved the problem.

TCP State Transition Diagram​

graph TD
CLOSED -->|Client sends SYN| SYN_SENT
SYN_SENT -->|Server responds with SYN+ACK| SYN_RECV
SYN_RECV -->|Client sends ACK| ESTABLISHED
ESTABLISHED -->|Connection closed by either party| FIN_WAIT_1
FIN_WAIT_1 -->|ACK received| FIN_WAIT_2
FIN_WAIT_2 -->|FIN received| TIME_WAIT
TIME_WAIT -->|Timeout| CLOSED
FIN_WAIT_1 -->|FIN received| CLOSING
CLOSING -->|ACK received| TIME_WAIT
SYN_RECV -->|Server sends FIN| LAST_ACK
LAST_ACK -->|Client sends ACK| CLOSED

Adjusting Kernel Parameters​

[root@node01 ~]# cat /etc/sysctl.conf
...
net.ipv4.tcp_max_syn_backlog = 32768
net.core.somaxconn = 32768

[root@node01 ~]# sysctl -p
...
net.ipv4.tcp_max_syn_backlog = 32768
net.core.somaxconn = 32768

After applying these changes, the health check commands executed without delay, and the pods returned to normal.

[root@node01 ~]# time curl -i http://localhost:9099/liveness
HTTP/1.1 204 No Content
Date: Tue, 15 Jun 2021 14:48:38 GMT
real 0m0.011s
user 0m0.004s
sys 0m0.004s

Conclusion​

This case study highlights the importance of understanding and tuning kernel parameters in high-concurrency scenarios. By adjusting parameters related to TCP connection handling, the issue of repeated pod restarts due to high CPU load and connection timeouts was resolved. This approach can be applied to similar issues in Kubernetes environments to ensure stable and efficient operation.