跳到主要内容

20 篇博文 含有标签「kubernetes」

查看所有标签

Why Kubernetes Chose etcd

· 阅读需 5 分钟

A Deep Dive into Kubernetes’ Control Plane Backbone​


1. One-Sentence Answer (Executive Summary)​

Kubernetes chose etcd because it provides strong consistency, fault tolerance, watchability, and simplicity—exactly what a distributed control plane needs.


2. What Kubernetes Needs at Its Core​

Before understanding why etcd, we must understand what Kubernetes fundamentally is.

Kubernetes Is NOT:​

  • A container runtime
  • A scheduler only
  • A deployment tool

Kubernetes IS:​

A distributed state machine that continuously reconciles desired state with actual state

That sentence alone explains why etcd matters.


3. Kubernetes Mental Model​

User declares desired state
↓
Kubernetes stores desired state
↓
Controllers compare desired vs actual
↓
Actions taken to reconcile
↓
State updated

Key Insight​

👉 Everything depends on a reliable, consistent source of truth


4. The Role of the Kubernetes Data Store​

The data store must hold:

  • Desired state (YAML manifests)
  • Cluster metadata
  • Node status
  • Pod status
  • ConfigMaps
  • Secrets
  • Leases
  • Locks
  • Leader elections

This data store is the single source of truth.


5. Why This Data Store Is So Critical​

If the data store is wrong, Kubernetes is wrong.

Wrong state → Wrong scheduling
Wrong locks → Split brain
Wrong config → Cluster outage

So Kubernetes needs a store that is:

✔ Always correct
✔ Always consistent
✔ Always available (with quorum)
✔ Observable (watchable)
✔ Deterministic

6. Why NOT Redis?​

Let’s eliminate obvious candidates first.

Redis Characteristics​

Fast
In-memory
Eventual consistency
No consensus protocol
Best-effort durability

Why Redis Fails Kubernetes’ Needs​

Redis write → Master
Replica lag possible
Failover may lose writes

Diagram​

APIServer → Redis
↓
Possible stale read

🚫 Unacceptable for Kubernetes Kubernetes requires linearizable writes, not eventual correctness.


7. Why NOT ZooKeeper?​

ZooKeeper was a strong candidate.

ZooKeeper Strengths​

Strong consistency
Leader election
Distributed locks
Proven correctness

Why Kubernetes Did NOT Choose ZooKeeper​

IssueExplanation
Operational complexityHard to operate at scale
Watch semanticsOne-time triggers
API ergonomicsNot cloud-native
Data modelHierarchical, not flexible
ToolingHeavy JVM ecosystem

ZooKeeper was correct but heavy.


8. Enter etcd​

etcd Core Identity​

Distributed
Strongly consistent
Key-value store
Raft consensus
Watch-first design

9. etcd at a Glance​

┌───────────────┐
│ Kubernetes │
│ Control Plane │
└───────┬───────┘
│
┌───────▼───────┐
│ etcd │
│ (Source of │
│ Truth) │
└───────────────┘

10. Strong Consistency: The #1 Reason​

What etcd Guarantees​

Write acknowledged
↓
Majority of nodes committed
↓
All future reads see it

Diagram (Raft)​

Client
↓
Leader
↓
Follower A
↓
Follower B
(majority ACK)

✔ No stale reads ✔ No lost writes ✔ No split brain


11. Kubernetes Control Plane Is a Distributed State Machine​

State S₀
↓
Event E₁
↓
State S₁
↓
Event E₂
↓
State S₂

etcd guarantees:

  • Ordered events
  • Durable transitions
  • Deterministic replay

12. Watch Is a First-Class Primitive in etcd​

This is hugely important.

etcd Watch Model​

Client sets watch
↓
Key changes
↓
Guaranteed event stream

Kubernetes Depends on This​

etcd change
↓
APIServer
↓
Controller
↓
Action

Controllers are event-driven, not polling-based.


13. Compare Watch Semantics​

SystemWatch Reliability
Redis Pub/SubBest effort
ZooKeeperOne-shot
etcdContinuous, ordered, resumable

Only etcd matches Kubernetes needs.


14. Simple Data Model = Huge Advantage​

etcd Uses Flat Key-Value Space​

/registry/pods/ns1/podA
/registry/services/ns1/svcA
/registry/nodes/node1

Benefits​

✔ Easy versioning
✔ Easy compaction
✔ Easy snapshot
✔ Easy migration

ZooKeeper’s tree model complicates this.


15. Transactions & Compare-And-Swap​

etcd supports:

IF version == X
THEN update
ELSE fail

Kubernetes uses this for:

  • Leader election
  • Resource locking
  • Optimistic concurrency control

16. Leader Election (Kubernetes Everywhere)​

Controller A
Controller B
Controller C

Only ONE should be active.

etcd Makes This Safe​

Lease → TTL → automatic expiry

Diagram:

Controller A holds lease
↓
Crash
↓
Lease expires
↓
Controller B elected

17. etcd Is Disk-Backed (Not Memory-Only)​

Why This Matters​

Crash
↓
Restart
↓
State recovered

etcd durability is mandatory for cluster recovery.


18. Snapshot + WAL Model​

Writes → WAL
↓
Periodic snapshot

Benefits​

  • Fast recovery
  • Small snapshots
  • Replayable history

Perfect for Kubernetes.


19. Kubernetes Can Rebuild the Cluster from etcd Alone​

This is HUGE.

Nodes die
Pods die
APIServer restarts

As long as etcd survives, Kubernetes reconstructs everything.


20. etcd Is Cloud-Native Friendly​

FeatureWhy It Matters
gRPC APIPerformance & tooling
TLS everywhereSecurity
Small binaryEasy deployment
RaftSimple mental model

21. Scaling Expectations Match Perfectly​

Kubernetes Data Profile​

Low write volume
Small objects
High read consistency

etcd Designed Exactly for This​

etcd is not designed for:

  • High QPS counters
  • Large blobs
  • Hot data paths

Which aligns perfectly.


22. Operational Model Alignment​

3–5 nodes
Quorum based
Simple backups
Predictable behavior

Kubernetes admins can reason about failures.


23. What etcd Is NOT Used For in Kubernetes​

❌ Pod logs
❌ Metrics
❌ Application data
❌ High-frequency events

etcd is control plane only.


24. Why etcd Won Over ZooKeeper (Final Comparison)​

CriteriaZooKeeperetcd
ConsistencyStrongStrong
Watch modelWeakExcellent
APIHeavyClean
Cloud-nativeNoYes
Operational burdenHighMedium
Fit for Kubernetes❌✅

25. Final Mental Model​

etcd = Brain
Kubernetes = Nervous System
Nodes = Muscles

If the brain is wrong, the body fails.


26. Final Takeaway​

Kubernetes chose etcd because:

✔ Strong consistency
✔ Event-driven watches
✔ Simple, correct semantics
✔ Cloud-native design
✔ Perfect data profile match

etcd is not fast—but Kubernetes does not need fast. Kubernetes needs correct.


Kubernetes Node Lifecycle Overview

· 阅读需 2 分钟

A Kubernetes Node is a core component with a lifecycle that includes registration, operation, and decommissioning. This article briefly explains key events in the Node lifecycle.

Kubernetes Node Lifecycle Overview

Node Registration​

Each node must run kubelet, which initiates a registration request to kube-apiserver, creating a node resource object. registerNode (default: true) in the kubelet configuration controls automatic registration. If set to false, nodes must be registered manually.

The nodename is determined by:

  • The cloud provider, if configured.
  • Otherwise, the local hostname, which --hostname-override can modify.

Node Heartbeat Mechanism​

The heartbeat includes updating both .status and a corresponding lease object for the node. nodeStatusUpdateFrequency (default: 10 seconds) triggers kubelet to update .status on changes or every 10 seconds. Additionally, each node maintains a lease object in kube-node-lease with an update frequency set by nodeLeaseDurationSeconds (default: 40 seconds) multiplied by 0.25, or 10 seconds.

Node Health Monitoring​

The node-controller in controller-manager monitors node health. If no issues arise, the node remains active. However, if a node loses connectivity or crashes, node-controller will detect this after a period specified by --node-monitor-grace-period (default: 40 seconds, increasing to 50 in v1.32). After this period, the node status changes to Unknown, and a Taint is applied to prevent new pod scheduling. If heartbeat remains absent for 5 more minutes, the node-controller starts eviction of pods and other resources from the node.

A graceful node shutdown follows a similar procedure: marking taints, rescheduling pods, and decommissioning the node.

Best Practices of kubectl

· 阅读需 6 分钟

Best Practices of Using kubectl to Analyze Kubernetes Cluster Status

Introduction​

Monitoring and analyzing the status of a Kubernetes (K8s) cluster is crucial for maintaining the health and efficiency of your applications. The kubectl command-line tool provides powerful capabilities for interacting with and analyzing your Kubernetes cluster. This article provides a comprehensive guide to the best practices of using kubectl to analyze cluster status, covering various scenarios such as checking cluster information, monitoring resources, troubleshooting issues, and performing advanced operations.


Basic Cluster Information​

Get Cluster Info​

To get basic information about the cluster, use the kubectl cluster-info command:

kubectl cluster-info

This command displays the Kubernetes master and DNS endpoints.

List Nodes​

To list all nodes in the cluster, use the kubectl get nodes command:

kubectl get nodes

This command displays the status, roles, age, and version of each node.

List Namespaces​

To list all namespaces in the cluster, use the kubectl get namespaces command:

kubectl get namespaces

This command displays the name and status of each namespace.


Resource Monitoring​

List Pods​

To list all pods in a namespace, use the kubectl get pods command:

kubectl get pods -n <namespace>

This command displays the name, ready status, status, restarts, and age of each pod.

List Deployments​

To list all deployments in a namespace, use the kubectl get deployments command:

kubectl get deployments -n <namespace>

This command displays the name, desired replicas, current replicas, up-to-date replicas, and available replicas of each deployment.

List Services​

To list all services in a namespace, use the kubectl get services command:

kubectl get services -n <namespace>

This command displays the name, type, cluster IP, external IP, and ports of each service.

List ConfigMaps​

To list all ConfigMaps in a namespace, use the kubectl get configmaps command:

kubectl get configmaps -n <namespace>

This command displays the name and age of each ConfigMap.

List Secrets​

To list all Secrets in a namespace, use the kubectl get secrets command:

kubectl get secrets -n <namespace>

This command displays the name and age of each Secret.


Advanced Resource Analysis​

Describe Resources​

To get detailed information about a specific resource, use the kubectl describe command:

kubectl describe <resource> <name> -n <namespace>

For example, to describe a pod:

kubectl describe pod <pod-name> -n <namespace>

This command provides detailed information about the resource, including events and status.

Logs​

To view the logs of a pod, use the kubectl logs command:

kubectl logs <pod-name> -n <namespace>

To follow the logs in real-time, use the -f flag:

kubectl logs -f <pod-name> -n <namespace>

Exec into Pod​

To execute a command inside a running pod, use the kubectl exec command:

kubectl exec -it <pod-name> -n <namespace> -- /bin/bash

This command opens an interactive shell inside the pod.

Port Forwarding​

To forward a local port to a port on a pod, use the kubectl port-forward command:

kubectl port-forward <pod-name> <local-port>:<pod-port> -n <namespace>

This command allows you to access the pod's service locally.


Troubleshooting​

Check Pod Status​

To check the status of a specific pod, use the kubectl get pods command:

kubectl get pods <pod-name> -n <namespace>

This command displays the status and other details of the pod.

Check Node Status​

To check the status of a specific node, use the kubectl get nodes command:

kubectl get nodes <node-name>

This command displays the status and other details of the node.

Check Events​

To check recent events in the cluster, use the kubectl get events command:

kubectl get events -n <namespace>

This command displays recent events, which can help diagnose issues.

Check Resource Limits​

To check resource limits and requests for a pod, use the kubectl describe command:

kubectl describe pod <pod-name> -n <namespace>

This command provides detailed information about resource limits and requests.


Advanced Scenarios​

Scaling Deployments​

To scale a deployment, use the kubectl scale command:

kubectl scale deployment <deployment-name> --replicas=<number> -n <namespace>

This command scales the deployment to the specified number of replicas.

Rolling Updates​

To perform a rolling update on a deployment, use the kubectl rollout command:

kubectl rollout restart deployment <deployment-name> -n <namespace>

This command restarts the deployment, performing a rolling update.

Debugging with kubectl​

To debug a pod, use the kubectl debug command:

kubectl debug <pod-name> -n <namespace>

This command creates a debugging container inside the pod.


Conclusion​

Monitoring and analyzing the status of a Kubernetes cluster is essential for maintaining the health and efficiency of your applications. By mastering the various options and scenarios covered in this article, you can effectively use kubectl to interact with and analyze your Kubernetes cluster.


References​

  1. kubectl Cheat Sheet
  2. kubectl Command Reference
  3. Kubernetes Documentation
  4. Kubernetes Best Practices

By following these best practices, you can leverage kubectl to its full potential and efficiently manage your Kubernetes cluster monitoring and troubleshooting tasks.

Deep Dive into Kubectl port-forward

· 阅读需 11 分钟

Introduction​

The kubectl port-forward command is a powerful tool in Kubernetes that allows you to forward local ports to ports on a pod or service. This is particularly useful for accessing services running inside a Kubernetes cluster from your local machine, debugging applications, and accessing internal services without exposing them to the public internet. This article provides a deep dive into the kubectl port-forward command, covering various scenarios, best practices, and advanced usage, with a detailed example of accessing a PostgreSQL database deployed in a Kubernetes cluster.


How kubectl port-forward Works​

Overview​

The kubectl port-forward command establishes a TCP connection between your local machine and a specific pod or service in the Kubernetes cluster. This connection is bidirectional, meaning that traffic sent to the local port is forwarded to the pod or service, and vice versa.

Steps​

  1. Identify the Pod or Service: The command specifies the pod or service to which the port forwarding will be established.
  2. Establish a Connection: kubectl establishes a TCP connection to the specified pod or service in the Kubernetes cluster.
  3. Forward Traffic: kubectl listens on the specified local port and forwards incoming traffic to the corresponding port on the pod or service.

Example​

To forward local port 5432 to a PostgreSQL pod running on port 5432:

kubectl port-forward postgresql-0 5432:5432 -n database

In this example:

  • postgresql-0 is the name of the PostgreSQL pod.
  • 5432:5432 specifies that local port 5432 will be forwarded to port 5432 on the pod.
  • -n database specifies the namespace where the pod is located.

Basic Usage​

Forwarding a Single Port​

To forward a local port to a port on a pod, use the following command:

kubectl port-forward <pod-name> <local-port>:<pod-port> -n <namespace>

For example, to forward local port 8080 to port 80 on a pod named my-pod in the default namespace:

kubectl port-forward my-pod 8080:80

Forwarding a Single Port to a Service​

To forward a local port to a port on a service, use the following command:

kubectl port-forward service/<service-name> <local-port>:<service-port> -n <namespace>

For example, to forward local port 8080 to port 80 on a service named my-service in the default namespace:

kubectl port-forward service/my-service 8080:80

Advanced Usage​

Forwarding Multiple Ports​

To forward multiple ports, specify multiple <local-port>:<pod-port> pairs:

kubectl port-forward <pod-name> <local-port1>:<pod-port1> <local-port2>:<pod-port2> -n <namespace>

For example, to forward local ports 8080 and 9090 to ports 80 and 90 on a pod named my-pod:

kubectl port-forward my-pod 8080:80 9090:90

Forwarding to a Specific Pod​

If you have multiple pods with the same label, you can forward to a specific pod by specifying the pod name:

kubectl port-forward <pod-name> <local-port>:<pod-port> -n <namespace>

For example, to forward local port 8080 to port 80 on a specific pod named my-pod-1:

kubectl port-forward my-pod-1 8080:80

Forwarding to a Service​

To forward to a service, use the service/ prefix:

kubectl port-forward service/<service-name> <local-port>:<service-port> -n <namespace>

For example, to forward local port 8080 to port 80 on a service named my-service:

kubectl port-forward service/my-service 8080:80

Common Scenarios​

Accessing a PostgreSQL Database​

To access a PostgreSQL database running inside a Kubernetes cluster, follow these steps:

  1. Identify the PostgreSQL Pod: First, identify the name of the PostgreSQL pod. You can list all pods in the namespace where PostgreSQL is deployed:

    kubectl get pods -n <namespace>

    For example, if the PostgreSQL pod is named postgresql-0 in the database namespace:

    kubectl get pods -n database
  2. Port Forward to the PostgreSQL Pod: Forward the PostgreSQL port (default is 5432) to a local port:

    kubectl port-forward <pod-name> <local-port>:5432 -n <namespace>

    For example, to forward local port 5432 to the PostgreSQL pod:

    kubectl port-forward postgresql-0 5432:5432 -n database
  3. Connect to the PostgreSQL Database: Use a PostgreSQL client to connect to the database using the local port. The JDBC URL to connect to the PostgreSQL database from your local machine would be:

    jdbc:postgresql://localhost:<local-port>/<database-name>

    For example, if the database name is mydb, the JDBC URL would be:

    jdbc:postgresql://localhost:5432/mydb

    You can also specify the username and password in the JDBC URL:

    jdbc:postgresql://localhost:5432/mydb?user=<username>&password=<password>

    Alternatively, you can use a PostgreSQL client like psql to connect:

    psql -h localhost -p 5432 -U <username> -d <database-name>

    For example:

    psql -h localhost -p 5432 -U postgres -d mydb
  4. Underlying Mechanism

The kubectl port-forward command sets up a proxy that listens on the specified local port (e.g., 5432) and forwards incoming traffic to the corresponding port on the PostgreSQL pod in the Kubernetes cluster. This proxy effectively creates a tunnel between your local machine and the Kubernetes cluster, allowing you to use localhost to connect to the database.

Example

When you run the following command:

kubectl port-forward postgresql-0 5432:5432 -n database

kubectl sets up a proxy that listens on localhost:5432 and forwards incoming traffic to postgresql-0:5432 in the database namespace.


Best Practices​

Security Considerations​

  • Limit Access: Only forward ports when necessary and limit access to the forwarded ports.
  • Use Namespaces: Use namespaces to isolate resources and limit the scope of port forwarding.
  • Monitor Access: Monitor access to forwarded ports and review logs for suspicious activity.

Resource Management​

  • Avoid Overloading: Avoid forwarding too many ports simultaneously to prevent overloading the local machine.
  • Use Ephemeral Ports: Use ephemeral ports (e.g., 49152-65535) for local forwarding to avoid conflicts with other services.

Monitoring and Logging​

  • Enable Logging: Enable logging for port forwarding to track access and diagnose issues.
  • Monitor Performance: Monitor the performance of the local machine and the Kubernetes cluster to ensure port forwarding does not impact performance.

Lifecycle of kubectl port-forward​

Running as a Process​

When you run the kubectl port-forward command, kubectl starts a new process on your local machine. This process listens on the specified local port and forwards traffic to the corresponding port on the pod or service in the Kubernetes cluster.

Example​

When you run the following command:

kubectl port-forward postgresql-0 5432:5432 -n database

kubectl starts a new process that listens on localhost:5432 and forwards incoming traffic to postgresql-0:5432 in the database namespace.

Termination​

The kubectl port-forward process runs until you manually terminate it. You can terminate the process by pressing Ctrl+C in the terminal where the command is running.


Temporary Nature of Port Forwarding​

Temporary Connection​

The port forwarding established by kubectl port-forward is temporary. It only lasts as long as the kubectl port-forward process is running. Once you terminate the process (e.g., by pressing Ctrl+C), the port forwarding stops, and you can no longer access the pod or service via the forwarded port.

Impact of Reboot​

If you reboot your local machine, the kubectl port-forward process will be terminated, and the port forwarding will be lost. You will need to rerun the kubectl port-forward command after rebooting to reestablish the connection.


Persisting Port Forwarding​

Using Systemd Service​

To persist port forwarding across reboots, you can create a systemd service that runs the kubectl port-forward command on startup.

Step-by-Step Guide​

  1. Create a Script: Create a script that runs the kubectl port-forward command.

    #!/bin/bash
    kubectl port-forward postgresql-0 5432:5432 -n database

    Save this script as port-forward.sh and make it executable:

    chmod +x port-forward.sh
  2. Create a Systemd Service: Create a systemd service file to run the script on startup.

    [Unit]
    Description=Kubernetes Port Forwarding for PostgreSQL

    [Service]
    ExecStart=/path/to/port-forward.sh
    Restart=always

    [Install]
    WantedBy=multi-user.target

    Save this file as /etc/systemd/system/kubectl-port-forward.service.

  3. Enable and Start the Service: Enable and start the systemd service.

    sudo systemctl enable kubectl-port-forward.service
    sudo systemctl start kubectl-port-forward.service

Using Cron Job​

Another approach is to use a cron job to run the kubectl port-forward command on startup.

Step-by-Step Guide​

  1. Create a Script: Create a script that runs the kubectl port-forward command.

    #!/bin/bash
    kubectl port-forward postgresql-0 5432:5432 -n database

    Save this script as port-forward.sh and make it executable:

    chmod +x port-forward.sh
  2. Create a Cron Job: Create a cron job to run the script on startup.

    @reboot /path/to/port-forward.sh

    Add this line to your crontab by running:

    crontab -e

Security Considerations​

Limiting Access​

  • Only Forward Ports When Necessary: Only forward ports when necessary and limit access to the forwarded ports.
  • Use Namespaces: Use namespaces to isolate resources and limit the scope of port forwarding.
  • Monitor Access: Monitor access to forwarded ports and review logs for suspicious activity.

Resource Management​

  • Avoid Overloading: Avoid forwarding too many ports simultaneously to prevent overloading the local machine.
  • Use Ephemeral Ports: Use ephemeral ports (e.g., 49152-65535) for local forwarding to avoid conflicts with other services.

Monitoring and Logging​

  • Enable Logging: Enable logging for port forwarding to track access and diagnose issues.
  • Monitor Performance: Monitor the performance of the local machine and the Kubernetes cluster to ensure port forwarding does not impact performance.

Conclusion​

The kubectl port-forward command is a versatile and powerful tool for accessing services running inside a Kubernetes cluster from your local machine. By mastering the various options and scenarios covered in this article, you can effectively use kubectl port-forward for debugging applications, accessing databases, and accessing internal services. The detailed example of accessing a PostgreSQL database deployed in a Kubernetes cluster demonstrates how to use kubectl port-forward to connect to a database from your local machine.

The port forwarding established by kubectl port-forward is temporary and only lasts as long as the kubectl port-forward process is running. If you reboot your local machine, the port forwarding will be lost, and you will need to rerun the command to reestablish the connection.

To persist port forwarding across reboots, you can create a systemd service or a cron job to run the kubectl port-forward command on startup.


References​

  1. kubectl port-forward Documentation
  2. Kubernetes Best Practices
  3. Kubernetes Security Best Practices
  4. PostgreSQL JDBC Driver Documentation
  5. Systemd Documentation
  6. Cron Documentation

By following these best practices, you can leverage kubectl port-forward to its full potential and efficiently manage your Kubernetes cluster access and debugging tasks.

Troubleshooting Pod-to-Pod Communication in Kubernetes

· 阅读需 5 分钟

Introduction​

In a K8S cluster, PodA is experiencing issues when using a service name to access PodB. PodA is on node1, and PodB is on node2. This article will walk through the troubleshooting steps to identify and resolve the issue.

Problem Description​

PodA is unable to access PodB using the service name, and the request is failing. PodA is on node1, and PodB is on node2.

Troubleshooting Steps​

Step 1: Capture Network Traffic​

First, use tcpdump to capture network traffic and observe if there are any anomalies:

[root@node1 ~]# tcpdump -n -i ens192 port 50300
...
13:48:17.630335 IP 177.177.176.150.distinct -> 10.96.22.136.50300: UDP, length 214
13:48:17.630407 IP 192.168.7.21.distinct -> 10.96.22.136.50300: UDP, length 214
...

From the captured packets, the source address and port are 177.177.176.150:50901, and the destination address and port are 10.96.22.136:50300. The destination IP 10.96.22.136 is the IP address obtained by PodA when using the server-svc service name.

Step 2: Verify Service IP​

Check if the IP address 10.96.22.136 is correct:

[root@node1 ~]# kubectl get pod -A -owide|grep server
ss server-xxx-xxx 1/1 Running 0 20h 177.177.176.150 node1
ss server-xxx-xxx 1/1 Running 0 20h 177.177.254.245 node2
ss server-xxx-xxx 1/1 Running 0 20h 177.177.18.152 node3
[root@node1 ~]# kubectl get svc -A -owide|grep server
ss server-svc ClusterIP 10.96.182.195 <none> 50300/UDP

The source IP is correct, but the destination IP does not match the expected server-svc IP 10.96.182.195.

Step 3: Check DNS Resolution​

Since K8S uses CoreDNS for service discovery starting from v1.13, verify if the DNS resolution is correct inside PodA:

[root@node1 ~]# kubectl exec -it -n ss server-xxx-xxx -- cat /etc/resolve.conf
nameserver 10.96.0.10
search ss.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

[root@node1 ~]# kubectl exec -it -n ss server-xxx-xxx -- nslookup server-svc
Server: 10.96.0.10

Name: ss
Address: 10.96.182.195

The DNS resolution is correct, and PodA can resolve server-svc to 10.96.182.195.

Step 4: Investigate Service IP​

Check if 10.96.22.136 is associated with any other service, iptables rules, or routes:

[root@node1 ~]# kubectl get svc -A -owide|grep 10.96.22.136

[root@node1 ~]# iptables-save|grep 10.96.22.136

[root@node1 ~]# ip route|grep 10.96.22.136

The IP 10.96.22.136 does not exist in the cluster.

Step 5: Verify Outgoing Traffic from Pod​

Check the destination IP when the traffic leaves PodA:

[root@node1 ~]# ip route|grep 177.177.176.150
177.177.176.150 dev cali9afa4438787 scope link

[root@node1 ~]# tcpdump -n -i cali9afa4438787 port 50300
...
14:16:40.821511 IP 177.177.176.150.50902 -> 10.96.22.136.50300: UDP, length 214
...

The destination IP is incorrect when leaving PodA.

Step 6: Simulate Traffic from Pod​

Simulate a UDP packet from PodA using nc to verify if the destination IP is correct:

[root@node1 ~]# kubectl exec -it -n ss server-xxx-xxx -- echo “test” | nc -u server-svc 50300 -p 9999

[root@node1 ~]# tcpdump -n -i cali9afa4438787 port 50300
...
15:46:45.871580 IP 177.177.176.150.50902 -> 10.96.182.195.50300: UDP, length 54
...

The simulated traffic has the correct destination IP.

Step 7: Restart Pod​

Restart PodA to see if the issue is resolved:

[root@node1 ~]# docker ps |grep server-xxx-xxx | grep -v POD |awk '{print $1}' |xargs docker restart

[root@node1 ~]# tcpdump -n -i ens192 port 50300
...
15:58:17.150535 IP 177.177.176.150.distinct -> 10.96.182.195.50300: UDP, length 214
15:58:17.150607 IP 192.168.7.21.distinct -> 10.96.182.195.50300: UDP, length 214
...

The issue is resolved after restarting PodA.

Step 8: Identify Java DNS Caching​

Investigate if Java DNS caching is causing the issue:

In Java, DNS caching is used to improve performance by storing DNS lookup results. When InetAddress is used to resolve a domain name, the result is cached by the JVM. This cache can cause stale DNS entries to be used.

To resolve this, adjust the DNS caching settings:

  1. Set the networkaddress.cache.ttl property in the Java security settings.
  2. Modify the java.security file to set the networkaddress.cache.negative.ttl property.

Solution​

Adjust the DNS caching settings in the business application based on the business scenario.

Conclusion​

This article detailed the troubleshooting steps to identify and resolve an issue where PodA was unable to access PodB using a service name in a K8S cluster. The root cause was identified as Java DNS caching, and the issue was resolved by adjusting the DNS caching settings.

Deep Dive into Core Kubernetes Components

· 阅读需 3 分钟

This article dives deep into the core components of Kubernetes, explaining their roles, functions, and interconnections. Textual diagrams illustrate the architecture to offer a clearer understanding of each component's interactions.


API Server​

The API Server is the core entry point for all Kubernetes interactions. It exposes the Kubernetes API, validates requests, and interacts with other components. Key functionalities include:

  • Authentication: Verifies the identity of incoming requests.
  • Authorization: Manages RBAC (Role-Based Access Control) policies.
  • Admission Control: Applies constraints and transformations to objects at request time.
  • API Requests: Handles CRUD (Create, Read, Update, Delete) requests on Kubernetes objects.

Diagram:

+----------------+
| |
| API Server |
| |
+----------------+
|
v
+----------------+
| |
| etcd |
| |
+----------------+

etcd​

etcd is a highly available key-value store used by Kubernetes to persist the cluster state. It maintains the configurations and states of each component and is essential for recovering cluster state in case of a restart. Each component retrieves cluster state through the API Server, which interfaces with etcd.

Diagram:

+----------------+ +----------------+
| | | |
| API Server | <-------> | etcd |
| | | |
+----------------+ +----------------+

Controller Manager​

The Controller Manager includes various controllers (e.g., Node, Replication, Endpoint) that regulate the state of cluster resources. Each controller reconciles the cluster's actual state with the desired state. When discrepancies arise, controllers initiate adjustments (e.g., scaling replicas).

Diagram:

+----------------+
| |
| Controller |
| Manager |
| |
+----------------+
|
v
+----------------+
| |
| API Server |
| |
+----------------+

Scheduler​

The Scheduler assigns Pods to Nodes based on resource needs and constraints. The Scheduler evaluates factors like CPU, memory, and affinity rules before assigning Pods, ensuring optimal workload distribution.

Diagram:

+----------------+
| |
| Scheduler |
| |
+----------------+
|
v
+----------------+
| |
| API Server |
| |
+----------------+

Kubelet​

The Kubelet runs on each Node and is responsible for managing container instances. It communicates with the API Server to receive Pod specifications, ensuring they’re running as defined. It monitors containers, collects resource usage, and sends reports back to the API Server.

Diagram:

+----------------+ +----------------+
| | | |
| Kubelet | <-> | API Server |
| | | |
+----------------+ +----------------+

Kube Proxy​

Kube Proxy manages network communication across Nodes in the cluster, implementing Kubernetes networking policies. It facilitates communication within Pods by maintaining network rules and forwarding requests as needed.

Diagram:

+----------------+ +----------------+
| | | |
| Kube Proxy | <-> | Kubelet |
| | | |
+----------------+ +----------------+

Communication Flow​

In Kubernetes, components communicate through a centralized and secure API Server, with etcd acting as the persistent store. The workflow for scheduling, updating, and scaling applications includes:

  1. User Command: Users issue requests to the API Server.
  2. Scheduler: Schedules new Pods.
  3. Controller Manager: Enforces and monitors the cluster state.
  4. Kubelet and Proxy: Enforce changes and maintain container and network policies.

Overall Diagram:

+----------------+ +----------------+
| | | |
| API Server | <--------> | etcd |
| | | |
+----------------+ +----------------+
| |
| |
v v
+----------------+ +----------------+
| | | |
| Controller | | Scheduler |
| Manager | | |
+----------------+ +----------------+
| |
v v
+----------------+ +----------------+
| | | |
| Kubelet | | Kube Proxy |
| | | |
+----------------+ +----------------+

Each Kubernetes component plays a specific role in cluster operations, and their communication ensures the cluster maintains its desired state and remains responsive to changing demands.

Deep Dive into Kubernetes Ingress

· 阅读需 3 分钟

In Kubernetes, Ingress is a crucial component that manages external access to services within a cluster, typically via HTTP or HTTPS. It provides a centralized entry point for external traffic, applying routing rules to direct requests to appropriate services. This article explores the architecture and responsibilities of Kubernetes Ingress, how it interacts with other components, and provides visual architecture diagrams.


What is Kubernetes Ingress?​

Ingress in Kubernetes is an API object that manages external access to services, usually HTTP. Instead of exposing each service individually through a Service object of type LoadBalancer or NodePort, Ingress offers a central route, managing traffic with a single IP address or DNS name for the entire cluster. Key benefits include:

  • Centralized Traffic Management: Ingress consolidates traffic through a single point of entry, allowing easy route management.
  • Path-Based Routing: Requests can be directed to specific services based on request paths or hostnames.
  • SSL Termination: Ingress can terminate SSL/TLS connections, ensuring secure HTTPS access.

Ingress Types​

Kubernetes supports two main Ingress types:

  • Simple Ingress: For straightforward routing to backend services based on path or hostname.
  • Advanced Ingress: With more complex routing rules, integrating SSL termination, URL rewrites, and custom filtering.

Ingress Controllers​

An Ingress Controller is the implementation that enforces Ingress rules, usually deployed as a Pod within the cluster. Popular Ingress controllers include NGINX, HAProxy, Traefik, and Istio. The Ingress Controller is responsible for:

  1. Monitoring Ingress Resources: Watching for any changes in Ingress resources and updating routing rules accordingly.
  2. Load Balancing and SSL Termination: Providing efficient load balancing for HTTP/HTTPS requests.
  3. Proxying and Forwarding: Accepting requests on behalf of the Ingress and forwarding them to the correct Service and Pod.

The Ingress Controller must be installed in the cluster to interpret and apply Ingress resources, as Kubernetes does not ship with a default controller.


Ingress Rules and Annotations​

Ingress rules define the routing policy for external traffic. These rules can direct traffic based on URL paths or hostnames, and each rule specifies the Service and Port the request should route to. Annotations add configurations like timeouts, request limits, and additional routing options.

Example of an Ingress Rule​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: example.com
http:
paths:
- path: /app
pathType: Prefix
backend:
service:
name: app-service
port:
number: 80

In this example, requests to example.com/app are routed to the app-service on port 80.


Ingress Architecture Diagram​

Ingress with Path-Based Routing​

+-----------------------------+
| External User |
+-----------------------------+
|
v
+-----------------------------+
| Ingress Controller | <-- Manages Ingress Rules
+-----------------------------+
|
+-------+------+--------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Service A | | Service B | | Service C |
+-----------+ +-----------+ +-----------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Pod A1 | | Pod B1 | | Pod C1 |
+-----------+ +-----------+ +-----------+

In this diagram:

  • Ingress Controller routes based on rules that direct traffic to Services.
  • Services are abstractions that select Pods through labels.
  • Pods run the actual applications and receive traffic forwarded by the Ingress Controller.

This setup allows centralized and managed external access to cluster resources.


By centralizing access with Ingress, Kubernetes can handle complex routing configurations, ensuring that traffic flows efficiently and securely across the cluster.

Deep Dive into Kubernetes Pod and Service Interactions

· 阅读需 3 分钟

In Kubernetes, Pods and Services work together to ensure reliable deployment, scaling, and network communication for containerized applications. This article provides a deep dive into their responsibilities, roles, and interaction processes within the Kubernetes architecture.

Understanding Kubernetes Pods​

A Pod is the smallest deployable unit in Kubernetes. Each Pod represents one or more containers and is typically designed to hold a single application instance. Key responsibilities and characteristics of Pods include:

  • Containerized Workloads: A Pod holds one or more containers that share storage, network, and specifications.
  • Networking: Each Pod gets its own IP address, allowing containers within it to communicate over localhost.
  • Ephemeral Nature: Pods are disposable. When they fail or are deleted, a new Pod with a different IP is often created to replace them.
  • Scalability: Kubernetes replicates Pods based on defined scaling requirements to handle application load.

Pod Lifecycle:

  1. Scheduled: The Scheduler places the Pod on an appropriate Node.
  2. Running: Containers within the Pod are executing.
  3. Terminated: When the Pod is completed or deleted.

Understanding Kubernetes Services​

A Service in Kubernetes is a stable networking endpoint that exposes a set of Pods as a network-accessible service, typically grouped by labels. Services abstract the dynamic nature of Pods and allow continuous network access. The main responsibilities of Services include:

  • Stable Networking: Services maintain a consistent IP address and DNS name despite the ephemeral nature of Pods.
  • Load Balancing: Services automatically distribute incoming traffic across available Pods, balancing load.
  • Discovery: Pods can be dynamically assigned to a Service based on matching labels, enabling flexible scaling and updates.

Types of Services:

  1. ClusterIP (default): Exposes a service within the cluster.
  2. NodePort: Exposes a service on each Node’s IP at a static port.
  3. LoadBalancer: Creates an external load balancer to expose the service externally.
  4. ExternalName: Maps the Service to a DNS name outside of the cluster.

Pod-Service Communication​

Kubernetes leverages label selectors and IP allocation to connect Pods with Services. Here’s how communication works:

  1. Label Selection: Services use labels to select Pods for traffic routing.
  2. Endpoints Object: Kubernetes creates an Endpoint resource to track the IP addresses of Pods associated with the Service.
  3. Service Proxying: kube-proxy intercepts traffic to a Service IP and forwards it to one of the Pod IPs listed in the Endpoints.
  4. Service IP and DNS: Each Service has a unique IP and is discoverable via a DNS name, facilitating easy communication.

Architecture Diagrams​

Pod Structure​

+----------------------------------+
| Pod |
| +-----------+ +-----------+ |
| | Container | | Container | |
| | App | | Sidecar | |
| +-----------+ +-----------+ |
| Network (localhost) |
| Shared Storage |
+----------------------------------+

Service Structure and Flow​

+-----------------+ +------------------+
| Service IP | <------------>| kube-proxy |
+-----------------+ +------------------+
| |
v v
+-----------------+ +------------------+ +------------------+
| Pod 1 | <-----> | Pod 2 | <- | Pod 3 |
+-----------------+ +------------------+ +------------------+

In summary, Kubernetes Pods run containerized workloads, while Services provide stable network access to those workloads, regardless of individual Pod lifetimes. Together, they enable dynamic scaling, fault tolerance, and efficient networking within Kubernetes clusters.

Best Practices of Deploying PostgreSQL in Kubernetes

· 阅读需 3 分钟

This guide walks through deploying PostgreSQL in a Kubernetes Pod with a sidecar container that collects PostgreSQL logs and forwards them to Elasticsearch. We’ll use Helm to manage deployments, including setting up necessary ConfigMaps and Secrets for configuration.

Project Structure​

postgresql-logging/
├── charts/
│ └── postgresql-log-sidecar/
│ ├── Chart.yaml
│ ├── templates/
│ │ ├── configmap.yaml
│ │ ├── deployment.yaml
│ │ ├── secret.yaml
│ │ ├── service.yaml
│ ├── values.yaml
├── README.md

1. Architecture Overview​

The following diagram shows how the PostgreSQL Pod and the sidecar interact for log collection:

+-----------------------------------+
| PostgreSQL Pod |
| |
| +-----------------------------+ |
| | PostgreSQL Container | |
| | - Exposes Port 5432 | |
| +-----------------------------+ |
| |
| +-----------------------------+ |
| | Log Collector | |
| | - Collects Logs | |
| | - Pushes to Elasticsearch | |
| +-----------------------------+ |
+-----------------------------------+

2. Helm Chart Configuration​

Chart.yaml​

apiVersion: v2
name: postgresql-log-sidecar
description: A Helm chart for PostgreSQL with log sidecar
version: 0.1.0
appVersion: "13"

values.yaml​

Define values for PostgreSQL configuration and the sidecar container.

postgresql:
username: admin
password: password123
database: appdb

elasticsearch:
host: "elasticsearch:9200"
index: "postgresql-logs"

sidecar:
image: fluent/fluent-bit
tag: "1.8"
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"

3. ConfigMap and Secret​

configmap.yaml​

This ConfigMap includes log configuration for the sidecar container.

apiVersion: v1
kind: ConfigMap
metadata:
name: postgresql-log-config
data:
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info

[INPUT]
Name tail
Path /var/log/postgresql/postgresql.log
Tag postgresql

[OUTPUT]
Name es
Match *
Host {{ .Values.elasticsearch.host }}
Port 9200
Index {{ .Values.elasticsearch.index }}

secret.yaml​

Create a Secret for PostgreSQL credentials.

apiVersion: v1
kind: Secret
metadata:
name: postgresql-secret
type: Opaque
data:
postgresql-username: {{ .Values.postgresql.username | b64enc }}
postgresql-password: {{ .Values.postgresql.password | b64enc }}

4. Deployment Configuration​

deployment.yaml​

The main deployment file sets up PostgreSQL and the Fluent Bit sidecar container.

apiVersion: apps/v1
kind: Deployment
metadata:
name: postgresql-log-sidecar
spec:
replicas: 1
selector:
matchLabels:
app: postgresql
template:
metadata:
labels:
app: postgresql
spec:
containers:
- name: postgresql
image: postgres:13
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: {{ .Values.postgresql.database }}
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgresql-secret
key: postgresql-username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgresql-secret
key: postgresql-password
volumeMounts:
- name: postgres-logs
mountPath: /var/log/postgresql
resources:
requests:
memory: "500Mi"
limits:
memory: "1Gi"

- name: log-collector
image: {{ .Values.sidecar.image }}:{{ .Values.sidecar.tag }}
args: ["-c", "/fluent-bit/etc/fluent-bit.conf"]
volumeMounts:
- name: postgres-logs
mountPath: /var/log/postgresql
- name: config
mountPath: /fluent-bit/etc
subPath: fluent-bit.conf
resources:
limits:
memory: {{ .Values.sidecar.resources.limits.memory }}
requests:
memory: {{ .Values.sidecar.resources.requests.memory }}
volumes:
- name: postgres-logs
emptyDir: {}
- name: config
configMap:
name: postgresql-log-config

service.yaml​

Create a Service to expose PostgreSQL internally within the cluster.

apiVersion: v1
kind: Service
metadata:
name: postgresql
spec:
type: ClusterIP
ports:
- port: 5432
targetPort: 5432
selector:
app: postgresql

5. Deploying the Helm Chart​

Install the chart with:

helm install postgresql-log-sidecar ./charts/postgresql-log-sidecar

Microservice - Spring Cloud Gateway

· 阅读需 16 分钟
  1. Introduction
  2. Why We Need API Gateway
  3. Build API Gateway
  4. Setting Up API Gateway
  5. Route Configuration
  6. Security and JWT Authentication
  7. Dockerfile Configuration
  8. Helm Chart Configuration
  9. Project Structure
  10. API Gateway Architecture
  11. Sequence Diagrams
  12. HTTPS Configuration for API Gateway
  13. Difference Between lb:// and http:// Protocols
  14. Conclusion

Introduction​

In a microservices architecture, an API Gateway acts as a single entry point for client requests, routing them to the appropriate microservices within the Kubernetes cluster. This article will delve into the setup and configuration of a Spring Cloud API Gateway, focusing on JWT authentication validation, route management, and deployment using Docker and Helm.


Why We Need API Gateway​

Single Entry Point​

An API Gateway provides a single entry point for all client requests, simplifying client-side code and reducing the number of endpoints clients need to interact with.

Routing and Load Balancing​

The gateway routes requests to the appropriate microservices and handles load balancing across multiple instances of each service.

Security​

It centralizes security concerns such as authentication and authorization, reducing the need for each microservice to implement these features independently.

Protocol Translation​

The gateway can handle protocol translation, allowing clients to use a single protocol (e.g., HTTP/2) while communicating with microservices using different protocols.


Build API Gateway​

Is there an official API Gateway image?

  • Currently, Spring Cloud Gateway does not offer an official pre-built Docker image, developers build a Spring Boot application with spring-cloud-starter-gateway and custom configurations (routes, JWT filter, etc.).

  • As gateway services are often customized for specific routing, load balancing, security and potentially other cross-cutting concerns and needs. It acts as the main entry point for all external requests to your microservices. The gateway can either pull routing configurations from a service discovery mechanism(like Spring Cloud Kubernetes Discovery) or have the configurations set explicitly.

Steps to Build a Production-Ready API Gateway

  1. Route Configuration: Define routing rules either in application.yml or using Java annotations.
  2. Security Setup: Add JWT validation (e.g., JwtAuthenticationFilter) and configure security policies in the gateway.
  3. Service Discovery: Use Spring Cloud Kubernetes for service discovery, allowing the gateway to dynamically discover services.
  4. Containerization and Deployment: Build a Docker image and deploy it to Kubernetes with a Helm chart for scalability and configuration management.

Setting Up API Gateway​

Step 1: Add Dependencies​

Add the necessary dependencies to your pom.xml file.

<dependencies>
<!-- Spring Boot Starter -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>

<!-- Spring Cloud Gateway -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>

<!-- Spring Security -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>

<!-- JWT -->
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt</artifactId>
<version>0.9.1</version>
</dependency>
</dependencies>

Step 2: Create the Main Application Class​

Create the main application class for the Spring Cloud Gateway.

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ApiGatewayApplication {
public static void main(String[] args) {
SpringApplication.run(ApiGatewayApplication.class, args);
}
}

Service Discovery and Routing Configuration​

YAML-based Configuration​

Configure routes in the application.yml file.

spring:
cloud:
gateway:
discovery:
locator:
enabled: true
lower-case-service-id: true # Ensures lowercase naming for services
routes:
- id: profile_service
uri: lb://profile-svc
predicates:
- Path=/profile/**
- id: inventory_service
uri: lb://inventory-svc
predicates:
- Path=/inventory/**
- id: order_service
uri: lb://order-svc
predicates:
- Path=/order/**
- id: shipment_service
uri: lb://shipment-svc
predicates:
- Path=/shipment/**

kubernetes:
discovery:
all-namespaces: true # Discover services across all namespaces
service-labels: # Optional: filter based on labels
environment: production
  • lb:// */ scheme: Enables load balancing across multiple instances of each microservice.
  • Service Discovery: The gateway will detect microservices like inventory-service or shipment-service using Kubernetes service discovery, eliminating the need for hardcoding service addresses.

Annotation-based Configuration​

Alternatively, configure routes programmatically using annotations.

import org.springframework.cloud.gateway.route.RouteLocator;
import org.springframework.cloud.gateway.route.builder.RouteLocatorBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class GatewayRoutesConfig {

@Bean
public RouteLocator gatewayRoutes(RouteLocatorBuilder builder) {
return builder.routes()
.route("profile_service_route", r -> r.path("/profile/**")
.uri("lb://profile-svc"))
.route("inventory_service_route", r -> r.path("/inventory/**")
.uri("lb://inventory-svc"))
.route("order_service_route", r -> r.path("/order/**")
.uri("lb://order-svc"))
.route("shipment_service_route", r -> r.path("/shipment/**")
.uri("lb://shipment-svc"))
.build();
}
}

Enable Service Discovery To enable service discovery, you can use the @EnableDiscoveryClient annotation. This annotation is typically placed in a configuration class.

import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableDiscoveryClient
public class DiscoveryConfig {
// Additional configuration for service discovery can be added here if needed
}

application.yml Configuration for Service Discovery​

Ensure that the following configuration is present in your application.yml or application.properties file to enable service discovery and other settings:

spring:
cloud:
gateway:
discovery:
locator:
enabled: true
lower-case-service-id: true
kubernetes:
discovery:
all-namespaces: true
service-labels:
environment: production

Summary​

  • Annotation-based Configuration: The GatewayRoutesConfig class defines the routes using the RouteLocatorBuilder.
  • Service Discovery: The DiscoveryConfig class enables service discovery using the @EnableDiscoveryClient annotation.
  • YAML Configuration: The application.yml file contains the necessary settings for service discovery and gateway configuration.

Note: The discovery settings, such as enabling discovery locator and configuring Kubernetes discovery, cannot be directly configured using annotations in the configuration class. These settings are typically configured in the application.yml or application.properties file.


Security and JWT Authentication​

Add Dependencies Include spring-boot-starter-oauth2-resource-server and spring-cloud-starter-gateway in your pom.xml:

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
</dependencies>

Gateway Application Properties Configuration: Configure application.yml for JWT validation. This includes specifying the JWT issuer URI and enabling authorization.

spring:
cloud:
gateway:
discovery:
locator:
enabled: true
routes:
- id: inventory-service
uri: lb://inventory-service
predicates:
- Path=/inventory/**
filters:
- TokenRelay # Forward the token to the downstream services
security:
oauth2:
resourceserver:
jwt:
issuer-uri: https://<your-auth-provider>/realms/your-realm
  • TokenRelay Filter: Passes the JWT token along with requests to the backend services.
  • Issuer URI: This points to your authentication provider, e.g., an OpenID Connect provider like Keycloak or Auth0.

Enable JWT Authentication: Set up SecurityConfig to enable JWT authentication and configure routes for authorized access.

@Configuration
@EnableWebFluxSecurity
public class SecurityConfig {

@Bean
public SecurityWebFilterChain securityWebFilterChain(ServerHttpSecurity http) {
http
.authorizeExchange()
.pathMatchers("/inventory/**", "/shipment/**").authenticated() // Require auth for specified paths
.anyExchange().permitAll()
.and()
.oauth2ResourceServer()
.jwt(); // Enable JWT authentication

return http.build();
}
}
  • This configuration restricts access to /inventory/** and /shipment/** endpoints to authenticated users, while allowing others as specified.

Annotation-based Configuration for Routes and Filters

import org.springframework.cloud.gateway.route.RouteLocator;
import org.springframework.cloud.gateway.route.builder.RouteLocatorBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.web.server.ServerHttpSecurity;
import org.springframework.security.web.server.SecurityWebFilterChain;

@Configuration
public class GatewayRoutesConfig {

@Bean
public RouteLocator gatewayRoutes(RouteLocatorBuilder builder) {
return builder.routes()
.route("inventory-service", r -> r.path("/inventory/**")
.filters(f -> f.filter(new JwtAuthenticationFilter("ROLE_USER,ROLE_ADMIN")))
.uri("lb://inventory-service"))
.build();
}

@Bean
public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
http
.authorizeExchange()
.anyExchange().authenticated()
.and()
.oauth2ResourceServer()
.jwt()
.jwkSetUri("https://example-issuer.com/.well-known/jwks.json")
.and()
.and()
.csrf().disable();
return http.build();
}
}

>**Note**: **Annotation-based Configuration**: The `GatewayRoutesConfig` class defines the routes and filters using the `RouteLocatorBuilder`. It also configures security settings using Spring Security annotations.

JWT Authentication Filter​

Create a JWT authentication filter to verify incoming requests.

import org.springframework.security.authentication.AuthenticationManager;
import org.springframework.security.web.server.authentication.AuthenticationWebFilter;
import org.springframework.security.oauth2.jwt.JwtDecoder;
import org.springframework.security.oauth2.jwt.ReactiveJwtDecoder;

@Configuration
public class SecurityConfig {

@Autowired
private JwtDecoder jwtDecoder;

@Bean
public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
AuthenticationWebFilter jwtAuthFilter = new AuthenticationWebFilter(authenticationManager());
http
.authorizeExchange()
.pathMatchers("/oauth2/**").permitAll()
.pathMatchers("/inventory/**", "/shipment/**").authenticated().permitAll()
.anyExchange().authenticated()
.and().addFilterAt(jwtAuthFilter, SecurityWebFiltersOrder.AUTHENTICATION);

return http.build();
}

@Bean
public AuthenticationManager authenticationManager() {
return new JwtAuthenticationManager(jwtDecoder);
}
}

JWT Decoder Configuration​

Configure the JWT decoder to validate tokens.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.oauth2.jwt.JwtDecoder;
import org.springframework.security.oauth2.jwt.NimbusJwtDecoder;

@Configuration
public class JwtConfig {

@Bean
public JwtDecoder jwtDecoder() {
return NimbusJwtDecoder.withPublicKey(publicKey).build();
}
}

Service-to-Service Communication via Gateway​

To communicate between services, such as shipment-service needing to communicate with inventory-service, microservices can send requests through the API Gateway.

This method simplifies routing, as services only need to know the gateway address and the service paths (e.g., /inventory/**). This approach enhances security and centralizes cross-cutting concerns like authentication, logging, and rate limiting.

Benefits of Gateway-based Communication​

  1. Centralized Security: JWT and other auth mechanisms can be handled at the gateway.
  2. Simplified Routing: Each service interacts with others via a single entry point.
  3. Enhanced Scalability: Kubernetes manages load balancing for each service, and the gateway balances requests across pods.

Service Discovery and Routing Workflow​

[External Client] --> [API Gateway]
|
| (Discovers routes from Kubernetes Discovery)
|
[inventory-service] <--+-- "lb://inventory-service"
[shipment-service] <--+-- "lb://shipment-service"
[other-service] <--+-- Discovered automatically

Sequence Diagram: Inventory and Shipment Communication via Gateway

  1. Shipment Service sends a request to API Gateway at /inventory/items.
  2. API Gateway validates the JWT token, looks up inventory-service route, and forwards the request.
  3. Inventory Service responds with the requested data.
  4. API Gateway relays the response to Shipment Service.
Shipment Service --> API Gateway: Request /inventory/items with JWT
API Gateway --> inventory-service: Validate JWT and route to inventory
inventory-service --> API Gateway: Return data
API Gateway --> Shipment Service: Relay data response

Example: Sending an Authenticated Request to Another Microservice via the Gateway

In shipment-service, use Spring's WebClient to call the inventory-service through the API Gateway.

  1. Define WebClient Bean for Consistency Across Calls

    @Bean
    public WebClient.Builder webClientBuilder() {
    return WebClient.builder();
    }
  2. Send Authenticated Request through Gateway

    @Autowired
    private WebClient.Builder webClientBuilder;

    public InventoryResponse fetchInventoryItems(String token) {
    return webClientBuilder
    .build()
    .get()
    .uri("http://api-gateway/inventory/items")
    .header(HttpHeaders.AUTHORIZATION, "Bearer " + token)
    .retrieve()
    .bodyToMono(InventoryResponse.class)
    .block();
    }
    • Token Forwarding: The token is forwarded with the request, allowing the gateway to validate it before routing to the target service.
    • Gateway URI: Requests go through http://api-gateway/inventory/items, routing based on service discovery.

In this example:

  • The gateway is deployed separately in Kubernetes and handles dynamic service discovery and routing via lb:// URIs.
  • Service-to-service calls leverage the API Gateway for ease of scaling, simplified routing, and centralized security. Each service only needs to know the API Gateway’s address and service paths.

Comparison: Gateway vs. Direct Service-to-Service Calls​

AspectAPI Gateway RoutingDirect Service-to-Service Calls
RoutingCentralized, via API GatewayDirect, each service must know other services
JWT ValidationCentralized in GatewayEach service needs to validate tokens
ScalingGateway handles scaling and balancingEach service scales independently
SecurityJWT validated once at the gatewayJWT validation needed in each service
Use CaseBest for external and internal callsIdeal for internal-only, tightly integrated

Dockerfile Configuration​

Create a Dockerfile for building the API Gateway image.

# Start with the base image for Java
FROM openjdk:17-jdk-slim

# Set working directory
WORKDIR /app

# Copy the API Gateway JAR to the image
COPY target/api-gateway.jar api-gateway.jar

# Expose port 8443 for the API Gateway
EXPOSE 8443

# Run the API Gateway application
ENTRYPOINT ["java", "-jar", "api-gateway.jar"]

Build and Push the Docker Image​

docker build -t myregistry/api-gateway:latest .
docker push myregistry/api-gateway:latest

Helm Chart Configuration​

Helm Chart Structure​

api-gateway/
├── Chart.yaml
├── values.yaml
└── templates/
├── deployment.yaml
├── service.yaml
└── ingress.yaml

Chart.yaml​

apiVersion: v2
name: api-gateway
version: 0.1.0
description: API Gateway for Microservices

values.yaml​

image:
repository: myregistry/api-gateway
tag: latest
pullPolicy: IfNotPresent

replicaCount: 2

service:
type: ClusterIP
port: 8443

ingress:
enabled: true
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
hosts:
- host: api.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: api-gateway-tls
hosts:
- api.example.com

templates/deployment.yaml​

apiVersion: apps/v1
kind: Deployment
metadata:
name: api-gateway
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: api-gateway
template:
metadata:
labels:
app: api-gateway
spec:
containers:
- name: api-gateway
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports:
- containerPort: 8443

templates/service.yaml​

apiVersion: v1
kind: Service
metadata:
name: api-gateway
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: 8443
selector:
app: api-gateway

templates/ingress.yaml​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-gateway-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- api.example.com
secretName: api-gateway-tls
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-gateway
port:
number: 8443

Project Structure​

api-gateway/
├── src/
│ ├── main/
│ │ ├── java/
│ │ │ └── com/
│ │ │ └── example/
│ │ │ └── apigateway/
│ │ │ ├── ApiGatewayApplication.java
│ │ │ ├── config/
│ │ │ │ ├── GatewayRoutesConfig.java
│ │ │ │ ├── SecurityConfig.java
│ │ │ │ └── JwtConfig.java
│ │ └── resources/
│ │ └── application.yml
├── Dockerfile
├── pom.xml
└── helm/
└── api-gateway/
├── Chart.yaml
├── values.yaml
└── templates/
├── deployment.yaml
├── service.yaml
└── ingress.yaml

API Gateway Architecture​

graph TD
A[Client] -->|HTTPS| B[API Gateway]
B -->|lb://| C[Profile Service]
B -->|lb://| D[Inventory Service]
B -->|lb://| E[Order Service]
B -->|lb://| F[Shipment Service]
B -->|lb://| G[OAuth2 Service]

Explanation​

  1. Client: Sends requests to the API Gateway over HTTPS.
  2. API Gateway: Routes requests to the appropriate microservice using lb:// protocol.
  3. Microservices: Each service handles requests and returns responses to the API Gateway.

Sequence Diagrams​

Sequence Diagram for Client Request​

sequenceDiagram
participant Client
participant API Gateway
participant Profile Service
participant Inventory Service
participant Order Service
participant Shipment Service

Client->>API Gateway: HTTPS Request
API Gateway->>Profile Service: lb://profile-svc
API Gateway->>Inventory Service: lb://inventory-svc
API Gateway->>Order Service: lb://order-svc
API Gateway->>Shipment Service: lb://shipment-svc
Profile Service-->>API Gateway: Response
Inventory Service-->>API Gateway: Response
Order Service-->>API Gateway: Response
Shipment Service-->>API Gateway: Response
API Gateway-->>Client: HTTPS Response

Sequence Diagram for JWT Authentication​

sequenceDiagram
participant Client
participant API Gateway
participant OAuth2 Service

Client->>API Gateway: HTTPS Request with JWT
API Gateway->>OAuth2 Service: Validate JWT
OAuth2 Service-->>API Gateway: JWT Valid
API Gateway-->>Client: HTTPS Response

HTTPS Configuration for API Gateway​

Step 1: Generate SSL Certificate​

Generate a self-signed SSL certificate or obtain one from a Certificate Authority (CA).

# Generate a self-signed certificate (for testing purposes)
keytool -genkeypair -alias api-gateway -keyalg RSA -keysize 2048 -storetype PKCS12 -keystore api-gateway-keystore.p12 -validity 365

Step 2: Configure HTTPS in application.yml​

Add the keystore settings to application.yml.

server:
port: 8443
ssl:
enabled: true
key-store: classpath:api-gateway-keystore.p12
key-store-password: your_password_here
key-store-type: PKCS12

Step 3: Update Routes to Use lb://​

Ensure that the routes use lb:// to leverage service discovery and load balancing.

spring:
cloud:
gateway:
routes:
- id: profile_service
uri: lb://profile-svc
predicates:
- Path=/profile/**
- id: inventory_service
uri: lb://inventory-svc
predicates:
- Path=/inventory/**
- id: order_service
uri: lb://order-svc
predicates:
- Path=/order/**
- id: shipment_service
uri: lb://shipment-svc
predicates:
- Path=/shipment/**

Step 4: Update Kubernetes Ingress for HTTPS​

Configure the Kubernetes Ingress to redirect traffic to the API Gateway’s HTTPS endpoint.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-gateway-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- api.example.com
secretName: api-gateway-tls
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-gateway
port:
number: 8443

Difference Between lb:// and http:// Protocols​

lb:// Protocol​

  • Service Discovery: Uses Kubernetes service discovery to find the appropriate service instance.
  • Load Balancing: Automatically load balances requests across multiple instances of the service.
  • Dynamic Routing: Routes can be dynamically updated based on service availability.

Example:

spring:
cloud:
gateway:
routes:
- id: profile_service
uri: lb://profile-svc
predicates:
- Path=/profile/**

http:// Protocol​

  • Direct Routing: Routes requests directly to a specific URL.
  • No Load Balancing: Does not provide load balancing or service discovery.
  • Static Routing: Routes are static and do not change based on service availability.

Example:

spring:
cloud:
gateway:
routes:
- id: profile_service
uri: http://profile-svc:8080
predicates:
- Path=/profile/**

When to Use Each Protocol​

  • Use lb:// for routes within Kubernetes service discovery, allowing the gateway to leverage load balancing.
  • Use http:// for direct routes to a specific URL or when no service discovery is available.

The other option

Here’s a step-by-step setup for building the API Gateway with JWT authentication and routing configurations:

Step 1: Configure API Gateway with JWT Authentication and Routing​

# application.yml
spring:
cloud:
gateway:
routes:
- id: inventory-service
uri: lb://inventory-service
predicates:
- Path=/inventory/**
filters:
- name: JwtAuthenticationFilter
args:
roles: "ROLE_USER,ROLE_ADMIN" # Add roles if role-based access is needed
discovery:
locator:
enabled: true # Enables dynamic route lookup from discovery service
default-filters:
- StripPrefix=1 # Adjusts the path for downstream services

security:
oauth2:
resourceserver:
jwt:
issuer-uri: https://example-issuer.com # Point to the JWT issuer
jwk-set-uri: https://example-issuer.com/.well-known/jwks.json # Set up for validation

Step 2: Implement JWT Authentication Filter for Gateway​

The custom JWT filter ensures each request includes a valid JWT token.

// JwtAuthenticationFilter.java
@Component
public class JwtAuthenticationFilter implements GatewayFilter {

private final JwtDecoder jwtDecoder;

public JwtAuthenticationFilter(JwtDecoder jwtDecoder) {
this.jwtDecoder = jwtDecoder;
}

@Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
String authHeader = exchange.getRequest().getHeaders().getFirst(HttpHeaders.AUTHORIZATION);
if (authHeader != null && authHeader.startsWith("Bearer ")) {
String token = authHeader.substring(7);
try {
Jwt jwt = jwtDecoder.decode(token); // Validates the JWT
return chain.filter(exchange); // If valid, proceed with the request
} catch (JwtException e) {
exchange.getResponse().setStatusCode(HttpStatus.UNAUTHORIZED);
return exchange.getResponse().setComplete();
}
}
exchange.getResponse().setStatusCode(HttpStatus.UNAUTHORIZED);
return exchange.getResponse().setComplete();
}
}

Add the filter in application.yml for it to be applied to specific routes (like inventory-service in this case).

Step 3: Building and Deploying the API Gateway as a Docker Image​

  1. Dockerfile for the API Gateway:

    # Dockerfile for api-gateway
    FROM openjdk:17-jdk-slim
    ARG JAR_FILE=target/api-gateway.jar
    COPY ${JAR_FILE} api-gateway.jar
    ENTRYPOINT ["java", "-jar", "/api-gateway.jar"]
  2. Build the Docker Image:

    docker build -t myregistry/api-gateway:latest .
    docker push myregistry/api-gateway:latest # Push to your registry
  3. Kubernetes Deployment Configuration:

    # api-gateway-deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: api-gateway
    labels:
    app: api-gateway
    spec:
    replicas: 2
    selector:
    matchLabels:
    app: api-gateway
    template:
    metadata:
    labels:
    app: api-gateway
    spec:
    containers:
    - name: api-gateway
    image: myregistry/api-gateway:latest
    ports:
    - containerPort: 8080
    env:
    - name: SPRING_PROFILES_ACTIVE
    value: k8s

Step 4: Service-to-Service Communication via Gateway​

Now, other services can call the gateway (using api-gateway service DNS) and append the target service’s route (/inventory/*) to the request path.

Sample Code for a Downstream Service Request Using WebClient with JWT​

// ShipmentService.java
@Service
public class ShipmentService {

private final WebClient webClient;

public ShipmentService(WebClient.Builder webClientBuilder) {
this.webClient = webClientBuilder
.baseUrl("http://api-gateway") // Gateway URL in Kubernetes DNS
.defaultHeader(HttpHeaders.AUTHORIZATION, "Bearer " + jwtToken)
.build();
}

public Inventory getInventory(String id) {
return webClient.get()
.uri("/inventory/" + id)
.retrieve()
.bodyToMono(Inventory.class)
.block();
}
}

Comparison: Gateway vs. Interceptor-Based JWT Validation​

AspectGateway with JWT FilterService-Level JWT Interceptors
Centralized ControlAll JWT validation centralized at gatewayEach service has independent JWT validation
Configuration ComplexityEasier for microservices, managed in gatewayMore complex across multiple services
Performance OverheadCan cause bottleneck at gateway if overloadedDistributed load, but duplicate processing
ResilienceSingle point of failure without redundancyMore resilient, distributed across services
Authorization FlexibilityOffers consistent authentication policyEach service can implement role-based restrictions