Skip to main content

6 posts tagged with "troubleshooting"

View All Tags

Best Practices of Linux System Performance Analysis and Tuning with CLI

· 5 min read

Introduction​

Monitoring and tuning Linux system performance is crucial for maintaining the health and efficiency of your infrastructure. Command line tools like top, htop, iotop, sar, ps, free, vmstat, and netstat provide powerful capabilities for analyzing and optimizing system performance. This article provides a comprehensive guide to the best practices of using these tools, covering various scenarios such as checking CPU and memory usage, monitoring processes, and troubleshooting real production issues.


Basic System Monitoring​

Check CPU Usage​

To check CPU usage, use the top command:

top

This command displays real-time CPU and memory usage, along with a list of running processes.

Check Memory Usage​

To check memory usage, use the free command:

free -h

This command displays the total, used, and free memory in a human-readable format.

Check Disk I/O​

To check disk I/O, use the iostat command:

iostat -x 1

This command displays detailed I/O statistics, including read and write rates.

Check Network Traffic​

To check network traffic, use the iftop command:

iftop -i eth0

This command displays real-time network traffic on the eth0 interface.


Process Monitoring​

List Running Processes​

To list all running processes, use the ps command:

ps aux

This command displays a detailed list of all running processes.

Check Process Status​

To check the status of a specific process, use the ps command with the process ID (PID):

ps -p <PID> -o pid,ppid,cmd,%cpu,%mem,state

This command displays the status of the process with the specified PID.

Kill a Process​

To kill a process, use the kill command with the process ID (PID):

kill <PID>

This command sends a termination signal to the process with the specified PID.


Advanced Performance Tuning​

Top​

To monitor system performance in real-time, use the top command:

top

This command provides a dynamic view of system resources and running processes.

Htop​

To monitor system performance with a more user-friendly interface, use htop:

htop

This command provides a colorful and interactive view of system resources and running processes.

Iotop​

To monitor disk I/O in real-time, use the iotop command:

iotop

This command displays real-time disk I/O statistics for running processes.

Sar​

To collect and report system activity, use the sar command:

sar -u 1 5

This command collects CPU usage statistics every second for 5 seconds.


Troubleshooting Real Production Issues​

High CPU Usage​

To identify processes causing high CPU usage, use top or htop:

top

Sort processes by CPU usage by pressing P.

High Memory Usage​

To identify processes causing high memory usage, use top or htop:

top

Sort processes by memory usage by pressing M.

High Disk I/O​

To identify processes causing high disk I/O, use iotop:

iotop

This command displays real-time disk I/O statistics for running processes.

Network Latency​

To diagnose network latency, use ping and traceroute:

ping google.com
traceroute google.com

These commands help identify network latency and routing issues.


Conclusion​

Monitoring and tuning Linux system performance is essential for maintaining the health and efficiency of your infrastructure. By mastering the various options and scenarios covered in this article, you can effectively use command line tools like top, htop, iotop, sar, ps, free, vmstat, and netstat to analyze and optimize system performance.


References​

  1. top Man Page
  2. htop Man Page
  3. iotop Man Page
  4. sar Man Page
  5. ps Man Page
  6. free Man Page
  7. iostat Man Page
  8. iftop Man Page
  9. ping Man Page
  10. traceroute Man Page

By following these best practices, you can leverage these system performance tools to their full potential and efficiently manage your monitoring and tuning tasks.

Troubleshooting Pod-to-Pod Communication in Kubernetes

· 5 min read

Introduction​

In a K8S cluster, PodA is experiencing issues when using a service name to access PodB. PodA is on node1, and PodB is on node2. This article will walk through the troubleshooting steps to identify and resolve the issue.

Problem Description​

PodA is unable to access PodB using the service name, and the request is failing. PodA is on node1, and PodB is on node2.

Troubleshooting Steps​

Step 1: Capture Network Traffic​

First, use tcpdump to capture network traffic and observe if there are any anomalies:

[root@node1 ~]# tcpdump -n -i ens192 port 50300
...
13:48:17.630335 IP 177.177.176.150.distinct -> 10.96.22.136.50300: UDP, length 214
13:48:17.630407 IP 192.168.7.21.distinct -> 10.96.22.136.50300: UDP, length 214
...

From the captured packets, the source address and port are 177.177.176.150:50901, and the destination address and port are 10.96.22.136:50300. The destination IP 10.96.22.136 is the IP address obtained by PodA when using the server-svc service name.

Step 2: Verify Service IP​

Check if the IP address 10.96.22.136 is correct:

[root@node1 ~]# kubectl get pod -A -owide|grep server
ss server-xxx-xxx 1/1 Running 0 20h 177.177.176.150 node1
ss server-xxx-xxx 1/1 Running 0 20h 177.177.254.245 node2
ss server-xxx-xxx 1/1 Running 0 20h 177.177.18.152 node3
[root@node1 ~]# kubectl get svc -A -owide|grep server
ss server-svc ClusterIP 10.96.182.195 <none> 50300/UDP

The source IP is correct, but the destination IP does not match the expected server-svc IP 10.96.182.195.

Step 3: Check DNS Resolution​

Since K8S uses CoreDNS for service discovery starting from v1.13, verify if the DNS resolution is correct inside PodA:

[root@node1 ~]# kubectl exec -it -n ss server-xxx-xxx -- cat /etc/resolve.conf
nameserver 10.96.0.10
search ss.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

[root@node1 ~]# kubectl exec -it -n ss server-xxx-xxx -- nslookup server-svc
Server: 10.96.0.10

Name: ss
Address: 10.96.182.195

The DNS resolution is correct, and PodA can resolve server-svc to 10.96.182.195.

Step 4: Investigate Service IP​

Check if 10.96.22.136 is associated with any other service, iptables rules, or routes:

[root@node1 ~]# kubectl get svc -A -owide|grep 10.96.22.136

[root@node1 ~]# iptables-save|grep 10.96.22.136

[root@node1 ~]# ip route|grep 10.96.22.136

The IP 10.96.22.136 does not exist in the cluster.

Step 5: Verify Outgoing Traffic from Pod​

Check the destination IP when the traffic leaves PodA:

[root@node1 ~]# ip route|grep 177.177.176.150
177.177.176.150 dev cali9afa4438787 scope link

[root@node1 ~]# tcpdump -n -i cali9afa4438787 port 50300
...
14:16:40.821511 IP 177.177.176.150.50902 -> 10.96.22.136.50300: UDP, length 214
...

The destination IP is incorrect when leaving PodA.

Step 6: Simulate Traffic from Pod​

Simulate a UDP packet from PodA using nc to verify if the destination IP is correct:

[root@node1 ~]# kubectl exec -it -n ss server-xxx-xxx -- echo “test” | nc -u server-svc 50300 -p 9999

[root@node1 ~]# tcpdump -n -i cali9afa4438787 port 50300
...
15:46:45.871580 IP 177.177.176.150.50902 -> 10.96.182.195.50300: UDP, length 54
...

The simulated traffic has the correct destination IP.

Step 7: Restart Pod​

Restart PodA to see if the issue is resolved:

[root@node1 ~]# docker ps |grep server-xxx-xxx | grep -v POD |awk '{print $1}' |xargs docker restart

[root@node1 ~]# tcpdump -n -i ens192 port 50300
...
15:58:17.150535 IP 177.177.176.150.distinct -> 10.96.182.195.50300: UDP, length 214
15:58:17.150607 IP 192.168.7.21.distinct -> 10.96.182.195.50300: UDP, length 214
...

The issue is resolved after restarting PodA.

Step 8: Identify Java DNS Caching​

Investigate if Java DNS caching is causing the issue:

In Java, DNS caching is used to improve performance by storing DNS lookup results. When InetAddress is used to resolve a domain name, the result is cached by the JVM. This cache can cause stale DNS entries to be used.

To resolve this, adjust the DNS caching settings:

  1. Set the networkaddress.cache.ttl property in the Java security settings.
  2. Modify the java.security file to set the networkaddress.cache.negative.ttl property.

Solution​

Adjust the DNS caching settings in the business application based on the business scenario.

Conclusion​

This article detailed the troubleshooting steps to identify and resolve an issue where PodA was unable to access PodB using a service name in a K8S cluster. The root cause was identified as Java DNS caching, and the issue was resolved by adjusting the DNS caching settings.

Troubleshooting Kubernetes: Pod Stuck in Terminating State After Deletion

· 6 min read

Introduction​

This document provides a detailed troubleshooting guide for a common Kubernetes issue where a Pod remains in the Terminating state after being deleted using the kubectl delete command. The analysis includes examining logs, identifying suspicious points, and understanding the underlying causes.


Problem Background​

After deleting a business Pod using the kubectl delete command, the Pod remained in the Terminating state.


Cause Analysis​

Initial Investigation​

To begin with, the kubectl describe command was used to inspect the Pod:

[root@node1 ~]# kubectl describe pod -n xxx cam1-78b6fc6bc8-cjsw5

No obvious anomalies were found in the output.

Kubelet Logs Analysis​

Next, the kubelet component logs were examined for any anomalies during the Pod deletion process. The following logs were filtered for relevance:

I0728 16:24:57.339295 9744 kubelet.go:1904] SyncLoop (DELETE, "api"): "cam1-78b6fc6bc8-cjsw5_cam(5c948341-c030-4996-b888-f032577d97b0)"
I0728 16:24:57.339720 9744 kuberuntime_container.go:581] Killing container "docker://a73082a4a9a4cec174bb0d1c256cc11d804d93137551b9bfd3e6fa1522e98589" with 60 second grace period
I0728 16:25:18.259418 9744 kubelet.go:1904] SyncLoop (DELETE, "api"): "cam1-78b6fc6bc8-cjsw5_cam(5c948341-c030-4996-b888-f032577d97b0)"
2021-07-28 16:25:19.247 [INFO][394011] ipam.go 1173: Releasing all IPs with handle 'cam.cam1-78b6fc6bc8-cjsw5'
2021-07-28 16:25:19.254 [INFO][393585] k8s.go 498: Teardown processing complete.

Suspicious Point 1: Pod IP Retrieval Failure​

W0728 16:25:19.303513 9744 docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "cam1-78b6fc6bc8-cjsw5_cam": Unexpected command output Device "eth0" does not exist.
with error: exit status 1

Suspicious Point 2: Unmount Failure​

E0728 16:25:20.939400 9744 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g podName:5c948341-c030-4996-b888-f032577d97b0 nodeName:}" failed. No retries permitted until 2021-07-28 16:25:21.439325811 +0800 CST m=+199182.605079651 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"diag-log\" (UniqueName: \"kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g\") pod \"5c948341-c030-4996-b888-f032577d97b0\" (UID: \"5c948341-c030-4996-b888-f032577d97b0\") : Unmount failed: exit status 32\nUnmounting arguments: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g\nOutput: umount: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g:Target is busy.\n (In some case using lsof(8) or fuser(1) can\n find useful information about the process using the device)\n\n"

Suspicious Points​

Suspicious Point 1: Pod IP Retrieval Failure​

The log indicates a failure to retrieve the Pod IP:

W0728 16:25:19.303513 9744 docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "cam1-78b6fc6bc8-cjsw5_cam": Unexpected command output Device "eth0" does not exist.
with error: exit status 1

This error was traced to the getIP method in the docker_sandbox.go file:

func (ds *dockerService) getIP(podSandboxID string, sandbox *dockertypes.ContainerJSON) string {
if sandbox.NetworkSettings == nil {
return ""
}
if networkNamespaceMode(sandbox) == runtimeapi.NamespaceMode_NODE {
// For sandboxes using host network, the shim is not responsible for
// reporting the IP.
return ""
}

// Don't bother getting IP if the pod is known and networking isn't ready
ready, ok := ds.getNetworkReady(podSandboxID)
if ok && !ready {
return ""
}

ip, err := ds.getIPFromPlugin(sandbox)
if err == nil {
return ip
}

if sandbox.NetworkSettings.IPAddress != "" {
return sandbox.NetworkSettings.IPAddress
}
if sandbox.NetworkSettings.GlobalIPv6Address != "" {
return sandbox.NetworkSettings.GlobalIPv6Address
}

// Error log here
klog.Warningf("failed to read pod IP from plugin/docker: %v", err)
return ""
}

Suspicious Point 2: Unmount Failure​

The unmount failure was logged as an ERROR:

E0728 16:25:20.939400 9744 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g podName:5c948341-c030-4996-b888-f032577d97b0 nodeName:}" failed. No retries permitted until 2021-07-28 16:25:21.439325811 +0800 CST m=+199182.605079651 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"diag-log\" (UniqueName: \"kubernetes.io/glusterfs/5c948341-c030-4996-b888-f032577d97b0-cam-pv-50g\") pod \"5c948341-c030-4996-b888-f032577d97b0\" (UID: \"5c948341-c030-4996-b888-f032577d97b0\") : Unmount failed: exit status 32\nUnmounting arguments: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g\nOutput: umount: /var/lib/kubelet/pods/5c948341-c030-4996-b888-f032577d97b0/volumes/kubernetes.io~glusterfs/cam-pv-50g:Target is busy.\n (In some case using lsof(8) or fuser(1) can\n find useful information about the process using the device)\n\n"

Pod Deletion Process​

The deletion of a Pod in Kubernetes follows these steps:

  1. API Server DELETE Call: The kubectl delete command triggers a DELETE request to the Kubernetes API server with a default grace period of 30 seconds.
  2. Update Pod Metadata: The Pod's metadata is updated with DeletionTimestamp and DeletionGracePeriodSeconds fields, but the Pod is not yet deleted from etcd.
  3. Kubelet Event Handling: The kubelet component listens for Pod updates and initiates the killPod() method.
  4. Second DELETE Call: The kubelet makes a second DELETE request to the API server with a grace period of 0 seconds.
  5. Pod Deletion from etcd: The API server deletes the Pod from etcd.

Verification and Testing​

To verify the deletion process, a test was conducted in a test environment:

[root@node2 ~]# kubectl delete pod -n xxx testpodrc2-7b749f6c9c-qh68l
pod "testpodrc2-7b749f6c9c-qh68l" deleted

Filtered kubelet logs:

I0730 13:27:31.854178 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:31.854511 24588 kuberuntime_container.go:581] Killing container "docker://e2a1cd5f2165e12cf0b46e12f9cd4d656d593f75e85c0de058e0a2f376a5557e" with 30 second grace period
I0730 13:27:32.203167 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:32.993294 24588 kubelet.go:1933] SyncLoop (PLEG): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)", event: &pleg.PodLifecycleEvent{ID:"85ee282f-a843-4f10-a99c-79d447f83f2a", Type:"ContainerDied", Data:"e2a1cd5f2165e12cf0b46e12f9cd4d656d593f75e85c0de058e0a2f376a5557e"}
I0730 13:27:32.993428 24588 kubelet.go:1933] SyncLoop (PLEG): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)", event: &pleg.PodLifecycleEvent{ID:"85ee282f-a843-4f10-a99c-79d447f83f2a", Type:"ContainerDied", Data:"c6a587614976beed0cbb6e5fabf70a2d039eec6c160154fce007fe2bb1ba3b4f"}
I0730 13:27:34.072494 24588 kubelet_pods.go:1090] Killing unwanted pod "testpodrc2-7b749f6c9c-qh68l"
I0730 13:27:40.084182 24588 kubelet.go:1904] SyncLoop (DELETE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"
I0730 13:27:40.085735 24588 kubelet.go:1898] SyncLoop (REMOVE, "api"): "testpodrc2-7b749f6c9c-qh68l_testpod(85ee282f-a843-4f10-a99c-79d447f83f2a)"

Conclusion​

The analysis revealed that the Pod remained in the Terminating state due to a failure in unmounting the GlusterFS volume. The unmount failure was caused by the volume being busy, which prevented the second DELETE

Analyzing High Memory Usage in Kubernetes

· 4 min read

Introduction​

In Kubernetes environments, monitoring resource usage is crucial for maintaining optimal performance and resource allocation. This article delves into a specific case where a user observed high memory usage by a Harbor pod on one of the nodes, leading to a discrepancy between kubectl top and docker stats results. We will analyze the cause, verify the memory calculation methods, and provide a solution to accurately monitor memory usage.


Problem Background​

A user observed that one of the nodes in their Kubernetes cluster showed high memory usage by a Harbor pod, approximately 3.7G, which seemed unusually high compared to other pods.

[root@node02 ~]# kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP
node01 Ready master 10d v1.15.12 100.1.0.10 <none>
node02 Ready master 12d v1.15.12 100.1.0.11 <none>
node03 Ready master 10d v1.15.12 100.1.0.12 <none>

[root@node02 ~]# kubectl top pod -A | grep harbor
kube-system harbor-master1-sxg2l 15m 150Mi
kube-system harbor-master2-ncvb8 8m 3781Mi
kube-system harbor-master3-2gdsn 14m 227Mi

Cause Analysis​

To understand the memory usage, the user compared the results from kubectl top and docker stats. Theoretically, docker stats should provide a more accurate result.

Using docker stats​

[root@node02 ~]# docker stats | grep harbor
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM %
10a230bee3c7 k8s_nginx_harbor-master2-xxx 0.02% 14.15MiB / 94.26GiB 0.01%
6ba14a04fd77 k8s_harbor-portal_harbor-master2-xxx 0.01% 13.73MiB / 94.26GiB 0.01%
324413da20a9 k8s_harbor-jobservice_harbor-master2-xxx 0.11% 21.54MiB / 94.26GiB 0.02%
d880b61cf4cb k8s_harbor-core_harbor-master2-xxx 0.12% 33.2MiB / 94.26GiB 0.03%
186c064d0930 k8s_harbor-registryctl_harbor-master2-xxx 0.01% 8.34MiB / 94.26GiB 0.01%
52a50204a962 k8s_harbor-registry_harbor-master2-xxx 0.06% 29.99MiB / 94.26GiB 0.03%
86031ddd0314 k8s_harbor-redis_harbor-master2-xxx 0.14% 11.51MiB / 94.26GiB 0.01%
6366207680f2 k8s_harbor-database_harbor-master2-xxx 0.45% 8.859MiB / 94.26GiB 0.01%

The total memory usage from docker stats was around 140M, significantly lower than the 3.7G reported by kubectl top.


Verification of Memory Calculation​

The discrepancy arises from the different methods used by kubectl top and docker stats to calculate memory usage.

  • kubectl top Calculation: memory.usage_in_bytes - inactive_file
  • docker stats Calculation: memory.usage_in_bytes - cache

If the cache is large, kubectl top will report a higher memory usage.

Verifying with memory_stats​

To verify, the user fetched the memory_stats data using Docker's API:

curl -s --unix-socket /var/run/docker.sock http:/v1.24/containers/xxx/stats | jq ."memory_stats"

Sample memory_stats data:

"memory_stats": {
"usage": 14913536,
"max_usage": 15183872,
"stats": {
"active_anon": 14835712,
"active_file": 0,
"cache": 77824,
"dirty": 0,
"hierarchical_memory_limit": 101205622784,
"hierarchical_memsw_limit": 9223372036854772000,
"inactive_anon": 4096,
"inactive_file": 73728,
...
}

By applying the formulas, the user confirmed that the results from kubectl top and docker stats were consistent with the expected calculations.


Understanding Cache Usage​

The high cache usage in Harbor was primarily due to the registry component, which handles Docker image storage and involves extensive read/write operations on image layers.

"memory_stats": {
"usage": 5845127168,
"max_usage": 22050988032,
"stats": {
"active_anon": 31576064,
"active_file": 3778052096,
"cache": 5813551104,
"dirty": 0,
"hierarchical_memory_limit": 101205622784,
"hierarchical_memsw_limit": 9223372036854772000,
"inactive_anon": 0,
"inactive_file": 2035499008,
...
}

Solution​

To address the discrepancy, the user was informed that kubectl top includes cache usage, which can be misleading. The cache, being non-critical, can be reclaimed by the system during memory pressure or manually released. The user was advised to use docker stats for a more accurate representation of actual memory usage.


Conclusion​

This case study highlights the importance of understanding the different methods used by Kubernetes and Docker to calculate memory usage. By verifying the calculations and understanding the role of cache, users can make more informed decisions about resource management in their Kubernetes clusters. Using docker stats provides a more accurate view of memory usage, especially in scenarios where cache usage is significant.

Troubleshooting High Currency Issues in Kubernetes - Continued

· 4 min read

Pod Restart CrashLoopBackOff Loop Due to High Concurrency (Continued)


Introduction​

Following the previous issue, the environment experienced another instance of Harbor and Calico pods restarting due to failed health checks. Additionally, using kubectl commands to enter the pods was extremely slow or timed out. This article continues the troubleshooting process to identify and resolve the root cause.


Problem Background​

After a period, the environment again faced issues with Harbor and Calico pods restarting due to failed health checks. Using kubectl commands to enter the pods was slow or timed out.

[root@node01 ~]# kubectl exec -it -n system node1-59c9475bc6-zkhq5 bash
^

Cause Analysis​

The root cause of the repeated restarts was previously identified as health check timeouts, with TCP connections stuck in the SYN_SENT phase.

[root@node01 ~]# netstat -anp | grep 23380
tcp 0 0 127.0.0.1:23380 0.0.0.0:* LISTEN 38914/kubelet
tcp 0 0 127.0.0.1:38983 127.0.0.1:23380 SYN_SENT -

Verification of Previous Fixes​

First, check if the previous kernel parameter adjustments are still in place.

[root@node01 ~]# cat /etc/sysctl.conf
net.ipv4.tcp_max_syn_backlog = 32768
net.core.somaxconn = 32768

Verify if the changes have taken effect.

[root@node01 ~]# ss -lnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 32768 127.0.0.1:23380 *:*

Despite the parameters being correctly set, the issue persisted.


Exploring Additional Causes​

Given the global impact of the issue, it was likely due to a system-level performance bottleneck. High business load affects CPU, memory, disk, and connection counts. Since the connections were long-lived, the file descriptor limit could be a factor.


Identifying File Descriptor Limits​

Check the number of open file descriptors.

[root@node01 ~]# lsof -p 45775 | wc -l
17974

[root@node01 ~]# lsof -p 45775 | grep "sock" | wc -l
12051

The process had over 10,000 open sockets, exceeding the default limit of 1024.

[root@node01 ~]# ulimit -n
1024

Despite exceeding the limit, the business pod did not report "too many open files" errors.

[root@node01 ~]# kubectl logs -n system node1-59c9475bc6-zkhq5
start config
...

Solution​

Temporarily increase the file descriptor limit.

[root@node01 ~]# ulimit -n 65535
[root@node01 ~]# ulimit -n
65535

After the adjustment, kubectl commands responded normally, and the SYN_SENT phase issue was resolved.

[root@node01 ~]# kubectl exec -it -n system node1-59c9475bc6-zkhq5 bash
[root@node1-59c9475bc6-zkhq5]# exit
[root@node01 ~]# kubectl exec -it -n system node1-59c9475bc6-zkhq5 bash
[root@node1-59c9475bc6-zkhq5]# exit
[root@node01 ~]# kubectl exec -it -n system node1-59c9475bc6-zkhq5 bash
[root@node1-59c9475bc6-zkhq5]# exit

[root@node01 ~]# netstat -anp | grep 23380
tcp 0 0 127.0.0.1:23380 0.0.0.0:* LISTEN 38914/kubelet
tcp 0 0 127.0.0.1:56369 127.0.0.1:23380 TIME_WAIT -
tcp 0 0 127.0.0.1:23380 127.0.0.1:57601 TIME_WAIT -
tcp 0 0 127.0.0.1:23380 127.0.0.1:57479 TIME_WAIT -

Conclusion​

This case study highlights the importance of considering system-level parameters, such as file descriptor limits, in high-concurrency scenarios. Adjusting these parameters can resolve performance bottlenecks and ensure stable operation of Kubernetes environments. For environments with high business load, it is strongly recommended to perform comprehensive operating system-level performance optimizations.

Troubleshooting High Currency Issues in Kubernetes

· 5 min read

Pod Restart CrashLoopBackOff Loop Due to High Concurrency


Introduction​

In a Kubernetes cluster environment, a business operation involving extensive configuration distribution (lasting several hours or more) led to repeated restarts of Calico pods. This article details the troubleshooting process, focusing on identifying the root cause and implementing a solution.


Problem Background​

During a high-concurrency operation, Calico pods in the Kubernetes cluster began to restart repeatedly.

[root@node02 ~]# kubectl get pod -n kube-system -owide | grep node01
calico-kube-controllers-6f59b8cdd8-8v2qw 1/1 Running 0 4h45m 10.10.119.238 node01 <none> <none>
calico-node-b8w2b 1/1 CrashLoopBackOff 43 3d19h 10.10.119.238 node01 <none> <none>
coredns-795cc9c45c-k7qpb 1/1 Running 0 4h45m 177.177.237.42 node01 <none> <none>
...

Analysis Process​

Upon observing the CrashLoopBackOff state of the pods, the first step was to use kubectl describe to gather more information.

[root@node02 ~]# kubectl describe pod -n kube-system calico-node-b8w2b
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 58m (x111 over 3h12m) kubelet, node01 (combined from similar events): Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Normal Pulled 43m (x36 over 3d19h) kubelet, node01 Container image "calico/node:v3.15.1" already present on machine
Warning Unhealthy 8m16s (x499 over 3h43m) kubelet, node01 Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning BackOff 3m31s (x437 over 3h3m) kubelet, node01 Back-off restarting failed container

The events indicated that the liveness probe was failing due to a connection timeout.


Investigating the Issue​

Manual execution of the health check command revealed slow response times, which were significantly higher than the usual millisecond-level response.

[root@node01 ~]# time curl -i http://localhost:9099/liveness
HTTP/1.1 204 No Content
Date: Tue, 15 Jun 2021 06:24:35 GMT
real 0m1.012s
user 0m0.003s
sys 0m0.005s

Logs from Calico's bird, confd, and felix components did not show any apparent errors. Checking the port status indicated that it was listening correctly.

[root@node02 ~]# netstat -anp | grep 9099
tcp 0 0 127.0.0.1:9099 0.0.0.0:* LISTEN 1202/calico-node
tcp 0 0 127.0.0.1:9099 127.0.0.1:56728 TIME_WAIT -
tcp 0 0 127.0.0.1:56546 127.0.0.1:9099 TIME_WAIT -

TCP Connection Timeout​

Given the connection timeout error, the TCP connection state was observed.

[root@node01 ~]# netstat -na | awk '/^tcp/{s[$6]++}END{for(key in s) print key,s[key]}'
LISTEN 49
ESTABLISHED 284
SYN_SENT 4
TIME_WAIT 176

The top command showed that the business Java process was consuming high CPU, peaking at over 2000%.

[root@node01 ~]# top
top - 14:28:57 up 13 days, 27 min, 2 users, load average: 9.55, 9.93, 9.91
Tasks: 1149 total, 1 running, 1146 sleeping, 0 stopped, 2 zombie
%Cpu(s): 16.0 us, 2.9 sy, 0.0 ni, 80.9 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 15249982+total, 21419184 free, 55542588 used, 75538048 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 94226176 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6754 root 20 0 66.8g 25.1g 290100 S 700.0 17.3 2971:49 java
25214 root 20 0 6309076 179992 37016 S 36.8 0.1 439:06.29 kubelet
20331 root 20 0 3196660 172364 24908 S 21.1 0.1 349:56.64 dockerd

Understanding CPU Load​

Checking the total number of CPU cores and comparing it with the load average and CPU usage indicated that the load was not excessively high.

[root@node01 ~]# cat /proc/cpuinfo | grep "physical id" | sort | uniq | wc -l
48
[root@node01 ~]# cat /proc/cpuinfo | grep "cpu cores" | uniq
cpu cores: 1

Given the high concurrency, the issue was likely related to the TCP connection setup. Understanding the TCP connection phases helped pinpoint the problem.


Kernel Parameter Tuning​

The issue was traced to the SYN_SENT phase of TCP connections. Adjusting kernel parameters related to TCP connections resolved the problem.

TCP State Transition Diagram​

graph TD
CLOSED -->|Client sends SYN| SYN_SENT
SYN_SENT -->|Server responds with SYN+ACK| SYN_RECV
SYN_RECV -->|Client sends ACK| ESTABLISHED
ESTABLISHED -->|Connection closed by either party| FIN_WAIT_1
FIN_WAIT_1 -->|ACK received| FIN_WAIT_2
FIN_WAIT_2 -->|FIN received| TIME_WAIT
TIME_WAIT -->|Timeout| CLOSED
FIN_WAIT_1 -->|FIN received| CLOSING
CLOSING -->|ACK received| TIME_WAIT
SYN_RECV -->|Server sends FIN| LAST_ACK
LAST_ACK -->|Client sends ACK| CLOSED

Adjusting Kernel Parameters​

[root@node01 ~]# cat /etc/sysctl.conf
...
net.ipv4.tcp_max_syn_backlog = 32768
net.core.somaxconn = 32768

[root@node01 ~]# sysctl -p
...
net.ipv4.tcp_max_syn_backlog = 32768
net.core.somaxconn = 32768

After applying these changes, the health check commands executed without delay, and the pods returned to normal.

[root@node01 ~]# time curl -i http://localhost:9099/liveness
HTTP/1.1 204 No Content
Date: Tue, 15 Jun 2021 14:48:38 GMT
real 0m0.011s
user 0m0.004s
sys 0m0.004s

Conclusion​

This case study highlights the importance of understanding and tuning kernel parameters in high-concurrency scenarios. By adjusting parameters related to TCP connection handling, the issue of repeated pod restarts due to high CPU load and connection timeouts was resolved. This approach can be applied to similar issues in Kubernetes environments to ensure stable and efficient operation.