跳到主要内容

2 篇博文 含有标签「etcd」

查看所有标签

Why Kubernetes Chose etcd

· 阅读需 5 分钟

A Deep Dive into Kubernetes’ Control Plane Backbone​


1. One-Sentence Answer (Executive Summary)​

Kubernetes chose etcd because it provides strong consistency, fault tolerance, watchability, and simplicity—exactly what a distributed control plane needs.


2. What Kubernetes Needs at Its Core​

Before understanding why etcd, we must understand what Kubernetes fundamentally is.

Kubernetes Is NOT:​

  • A container runtime
  • A scheduler only
  • A deployment tool

Kubernetes IS:​

A distributed state machine that continuously reconciles desired state with actual state

That sentence alone explains why etcd matters.


3. Kubernetes Mental Model​

User declares desired state
↓
Kubernetes stores desired state
↓
Controllers compare desired vs actual
↓
Actions taken to reconcile
↓
State updated

Key Insight​

👉 Everything depends on a reliable, consistent source of truth


4. The Role of the Kubernetes Data Store​

The data store must hold:

  • Desired state (YAML manifests)
  • Cluster metadata
  • Node status
  • Pod status
  • ConfigMaps
  • Secrets
  • Leases
  • Locks
  • Leader elections

This data store is the single source of truth.


5. Why This Data Store Is So Critical​

If the data store is wrong, Kubernetes is wrong.

Wrong state → Wrong scheduling
Wrong locks → Split brain
Wrong config → Cluster outage

So Kubernetes needs a store that is:

✔ Always correct
✔ Always consistent
✔ Always available (with quorum)
✔ Observable (watchable)
✔ Deterministic

6. Why NOT Redis?​

Let’s eliminate obvious candidates first.

Redis Characteristics​

Fast
In-memory
Eventual consistency
No consensus protocol
Best-effort durability

Why Redis Fails Kubernetes’ Needs​

Redis write → Master
Replica lag possible
Failover may lose writes

Diagram​

APIServer → Redis
↓
Possible stale read

🚫 Unacceptable for Kubernetes Kubernetes requires linearizable writes, not eventual correctness.


7. Why NOT ZooKeeper?​

ZooKeeper was a strong candidate.

ZooKeeper Strengths​

Strong consistency
Leader election
Distributed locks
Proven correctness

Why Kubernetes Did NOT Choose ZooKeeper​

IssueExplanation
Operational complexityHard to operate at scale
Watch semanticsOne-time triggers
API ergonomicsNot cloud-native
Data modelHierarchical, not flexible
ToolingHeavy JVM ecosystem

ZooKeeper was correct but heavy.


8. Enter etcd​

etcd Core Identity​

Distributed
Strongly consistent
Key-value store
Raft consensus
Watch-first design

9. etcd at a Glance​

┌───────────────┐
│ Kubernetes │
│ Control Plane │
└───────┬───────┘
│
┌───────▼───────┐
│ etcd │
│ (Source of │
│ Truth) │
└───────────────┘

10. Strong Consistency: The #1 Reason​

What etcd Guarantees​

Write acknowledged
↓
Majority of nodes committed
↓
All future reads see it

Diagram (Raft)​

Client
↓
Leader
↓
Follower A
↓
Follower B
(majority ACK)

✔ No stale reads ✔ No lost writes ✔ No split brain


11. Kubernetes Control Plane Is a Distributed State Machine​

State S₀
↓
Event E₁
↓
State S₁
↓
Event E₂
↓
State S₂

etcd guarantees:

  • Ordered events
  • Durable transitions
  • Deterministic replay

12. Watch Is a First-Class Primitive in etcd​

This is hugely important.

etcd Watch Model​

Client sets watch
↓
Key changes
↓
Guaranteed event stream

Kubernetes Depends on This​

etcd change
↓
APIServer
↓
Controller
↓
Action

Controllers are event-driven, not polling-based.


13. Compare Watch Semantics​

SystemWatch Reliability
Redis Pub/SubBest effort
ZooKeeperOne-shot
etcdContinuous, ordered, resumable

Only etcd matches Kubernetes needs.


14. Simple Data Model = Huge Advantage​

etcd Uses Flat Key-Value Space​

/registry/pods/ns1/podA
/registry/services/ns1/svcA
/registry/nodes/node1

Benefits​

✔ Easy versioning
✔ Easy compaction
✔ Easy snapshot
✔ Easy migration

ZooKeeper’s tree model complicates this.


15. Transactions & Compare-And-Swap​

etcd supports:

IF version == X
THEN update
ELSE fail

Kubernetes uses this for:

  • Leader election
  • Resource locking
  • Optimistic concurrency control

16. Leader Election (Kubernetes Everywhere)​

Controller A
Controller B
Controller C

Only ONE should be active.

etcd Makes This Safe​

Lease → TTL → automatic expiry

Diagram:

Controller A holds lease
↓
Crash
↓
Lease expires
↓
Controller B elected

17. etcd Is Disk-Backed (Not Memory-Only)​

Why This Matters​

Crash
↓
Restart
↓
State recovered

etcd durability is mandatory for cluster recovery.


18. Snapshot + WAL Model​

Writes → WAL
↓
Periodic snapshot

Benefits​

  • Fast recovery
  • Small snapshots
  • Replayable history

Perfect for Kubernetes.


19. Kubernetes Can Rebuild the Cluster from etcd Alone​

This is HUGE.

Nodes die
Pods die
APIServer restarts

As long as etcd survives, Kubernetes reconstructs everything.


20. etcd Is Cloud-Native Friendly​

FeatureWhy It Matters
gRPC APIPerformance & tooling
TLS everywhereSecurity
Small binaryEasy deployment
RaftSimple mental model

21. Scaling Expectations Match Perfectly​

Kubernetes Data Profile​

Low write volume
Small objects
High read consistency

etcd Designed Exactly for This​

etcd is not designed for:

  • High QPS counters
  • Large blobs
  • Hot data paths

Which aligns perfectly.


22. Operational Model Alignment​

3–5 nodes
Quorum based
Simple backups
Predictable behavior

Kubernetes admins can reason about failures.


23. What etcd Is NOT Used For in Kubernetes​

❌ Pod logs
❌ Metrics
❌ Application data
❌ High-frequency events

etcd is control plane only.


24. Why etcd Won Over ZooKeeper (Final Comparison)​

CriteriaZooKeeperetcd
ConsistencyStrongStrong
Watch modelWeakExcellent
APIHeavyClean
Cloud-nativeNoYes
Operational burdenHighMedium
Fit for Kubernetes❌✅

25. Final Mental Model​

etcd = Brain
Kubernetes = Nervous System
Nodes = Muscles

If the brain is wrong, the body fails.


26. Final Takeaway​

Kubernetes chose etcd because:

✔ Strong consistency
✔ Event-driven watches
✔ Simple, correct semantics
✔ Cloud-native design
✔ Perfect data profile match

etcd is not fast—but Kubernetes does not need fast. Kubernetes needs correct.


ZooKeeper vs etcd

· 阅读需 4 分钟

Short answer up front:

ZooKeeper is chosen over etcd when absolute correctness, strict ordering, and coordination semantics matter more than cloud-native ergonomics and operational simplicity.

This goes straight to the philosophy and historical trade-offs of distributed systems.


1. “Correctness” Means Different Things in Different Systems​

When people say “ZooKeeper is more correct”, they usually mean coordination correctness, not storage correctness.

Two kinds of correctness​

TypeMeaning
Data correctnessReads return the latest committed value
Coordination correctnessOrdering, fencing, session semantics are strictly enforced

Both etcd and ZooKeeper are data-correct. ZooKeeper is exceptionally strong at coordination correctness.


2. ZooKeeper’s Original Purpose (This Matters a LOT)​

ZooKeeper was not designed as a KV store.

It was designed to solve distributed coordination problems:

  • Leader election
  • Distributed locks
  • Membership tracking
  • Configuration management
  • Barrier synchronization

ZooKeeper’s design assumption​

“If coordination is wrong, the entire distributed system collapses.”

This assumption drove extremely conservative design choices.


3. ZooKeeper’s Strongest Weapon: Session Semantics​

ZooKeeper has sessions, etcd does not (in the same way).

ZooKeeper session model​

Client connects
↓
Session established
↓
Ephemeral nodes tied to session
↓
Client dies → session expires → nodes deleted

Why this is powerful​

Lock correctness
Leader correctness
Membership correctness

Diagram​

Client A holds lock
↓
Client A crashes
↓
ZooKeeper deletes ephemeral node
↓
Lock immediately released

This is automatic fencing, not optional logic.


4. ZooKeeper’s Ordering Guarantees Are Extremely Strict​

ZooKeeper guarantees:

  • Total order of writes
  • Monotonic reads per client
  • No reordering under failure
  • Linearizable operations

But more importantly:

ZooKeeper exposes ordering as a first-class primitive


5. Sequential znodes (A Big Deal)​

ZooKeeper can create nodes like:

/lock/node-00000001
/lock/node-00000002
/lock/node-00000003

This enables:

  • Fair locks
  • FIFO queues
  • Barrier algorithms

etcd cannot do this natively.


6. ZooKeeper Watches vs etcd Watches (Subtle Difference)​

You might hear:

“etcd has better watches”

This is true for data propagation, but false for coordination logic.

ZooKeeper watches are intentionally weaker​

  • One-shot
  • Explicit re-registration
  • Tied to session lifecycle

Why?

Because ZooKeeper forces you to reason about race conditions explicitly.

This reduces false assumptions in coordination logic.


7. ZooKeeper Is Extremely Conservative by Design​

ZooKeeper assumes:

Network partitions WILL happen
Clients WILL crash
GC pauses WILL happen

So it prefers:

  • Blocking over guessing
  • Failure over inconsistency
  • Simplicity over convenience

etcd is conservative too — but ZooKeeper is paranoid.


8. Why Large-Scale Distributed Systems Still Choose ZooKeeper​

Typical ZooKeeper users​

  • Apache Kafka
  • Apache HBase
  • Hadoop YARN
  • Apache Storm
  • Apache Solr
  • Large financial systems
  • Telco control planes

These systems care about:

✔ Strict leader fencing
✔ Lock correctness
✔ No split-brain EVER
✔ Predictable failure modes

9. ZooKeeper vs etcd: Philosophy Comparison​

DimensionZooKeeperetcd
Primary roleCoordination serviceConsistent KV store
Session semanticsFirst-classLease-based
Lock safetyExtremely strongStrong but simpler
Ordering primitivesBuilt-inLimited
API philosophyExplicit, low-levelSimple, declarative
Cloud-native❌✅
Operational simplicity❌✅

10. Why Kubernetes Did NOT Choose ZooKeeper​

Now the flip side.

Kubernetes does not need:

  • Sequential znodes
  • Complex lock trees
  • Session-heavy logic

Kubernetes needs:

✔ State storage
✔ Event watching
✔ Simple transactions
✔ Declarative reconciliation

ZooKeeper would have been overkill and harder to operate.


11. Why ZooKeeper Is Still the “Primary Option” in Some Domains​

ZooKeeper wins when:

You need coordination correctness > everything else​

Examples:

  • Financial trading leader election
  • Distributed job schedulers
  • Metadata services with strict fencing
  • Multi-writer consensus with fairness guarantees

In these systems:

A single incorrect lock is worse than total outage

ZooKeeper optimizes for never being wrong, even if that means being unavailable.


12. Failure Mode Philosophy (Very Important)​

ZooKeeper failure philosophy​

If unsure → stop

etcd failure philosophy​

If quorum exists → continue

Both are correct — but ZooKeeper is stricter.


13. Mental Model Summary​

ZooKeeper = Distributed coordination kernel
etcd = Distributed state store

ZooKeeper acts like a kernel enforcing rules. etcd acts like a database holding truth.


14. Final Answer (Concise)​

ZooKeeper is chosen over etcd when:

  • Coordination correctness is paramount
  • Leader fencing must be absolute
  • Fair ordering matters
  • Failures must be explicit, not hidden
  • Operational complexity is acceptable

etcd is chosen when:

  • State correctness matters
  • Watchability matters
  • Cloud-native ops matter
  • Simplicity matters

15. One-Line Rule of Thumb​

If you fear split-brain more than downtime → ZooKeeper If you fear operational complexity more than coordination edge cases → etcd