Skip to main content

35 posts tagged with "deep-dive"

View All Tags

Deep Dive into Redis I

· 37 min read

Table of Contents​

  1. Redis Architecture: The Complete Picture
  2. Fundamentals
  3. Data Structures and Operations
  4. Persistence Mechanisms
  5. RDB and AOF File Formats Deep Dive
  6. Replication and High Availability
  7. Redis Cluster and Sharding
  8. Hash Slots Deep Dive
  9. Operational Considerations
  10. Mental Model Summary
  11. Quick Reference Guide

Redis Architecture: The Complete Picture​

Before diving into details, here's the complete Redis ecosystem at a glance:

graph TB
subgraph "Application Layer"
App[Application]
end

subgraph "Redis Layer"
subgraph "Single Instance"
Master1[Master]
Replica1[Replica]
Master1 --> Replica1
end

subgraph "Redis Cluster (Sharded)"
M1[Master A<br/>Slots 0-5460]
M2[Master B<br/>Slots 5461-10922]
M3[Master C<br/>Slots 10923-16383]
R1[Replica A]
R2[Replica B]
R3[Replica C]
M1 --> R1
M2 --> R2
M3 --> R3
end

subgraph "High Availability"
S1[Sentinel 1]
S2[Sentinel 2]
S3[Sentinel 3]
end
end

subgraph "Persistence Layer"
RDB[(RDB Snapshot)]
AOF[(AOF Log)]
end

App --> Master1
App --> M1
App --> M2
App --> M3

S1 -.Monitor.-> Master1
S2 -.Monitor.-> Master1
S3 -.Monitor.-> Master1

Master1 --> RDB
Master1 --> AOF
M1 --> RDB
M2 --> RDB
M3 --> RDB

Key Components:

ComponentPurposeWhen to Use
Single MasterBasic Redis instanceSmall datasets, simple caching
Master + ReplicaRead scaling + failoverRead-heavy workloads
SentinelAutomatic failoverHigh availability without sharding
Redis ClusterHorizontal scaling (sharding)Large datasets exceeding single node
RDBPoint-in-time snapshotsFast restarts, backups
AOFWrite operation logMaximum data durability

Decision Tree:

flowchart TD
Start{Data fits in<br/>single instance?}
Start -->|Yes| HA{Need high<br/>availability?}
Start -->|No| UseCluster[Use Redis Cluster]

HA -->|No| SingleMaster[Single Master<br/>+ optional replicas]
HA -->|Yes| UseSentinel[Master + Replicas<br/>+ Sentinel]

UseCluster --> Persist{Need persistence?}
UseSentinel --> Persist
SingleMaster --> Persist

Persist -->|Fast restart| UseRDB[Enable RDB]
Persist -->|Max durability| UseAOF[Enable AOF]
Persist -->|Best of both| UseHybrid[Enable Both<br/>Recommended]

Fundamentals​

What Is Redis?​

Redis stands for REmote DIctionary Server.

At its heart, Redis is:

  • An in-memory data store
  • Extremely fast (microsecond latency)
  • Used as:
    • Cache
    • Database
    • Message broker
    • Counter system
    • Coordination system

Redis in One Sentence​

Redis is a single-threaded, in-memory data structure server optimized for speed, simplicity, and predictable behavior.

Where Redis Fits in a System​

flowchart LR
Client --> Redis
Client --> DB[(Database)]
Redis --> DB

Redis often sits between your application and your database to:

  • Reduce load
  • Improve latency
  • Handle real-time data

Core Redis Philosophy​

Redis makes several strong design choices:

PrincipleMeaning
In-memoryData lives in RAM for speed
Single-threadedNo locks, no race conditions
Simple data structuresPredictable performance
Explicit durabilityYou choose persistence level

Why Single-Threaded Is a Feature (Not a Bug)​

sequenceDiagram
participant Client1
participant Client2
participant Redis

Client1->>Redis: INCR counter
Client2->>Redis: INCR counter
Redis-->>Client1: OK
Redis-->>Client2: OK

Because Redis processes one command at a time:

  • No locks
  • No inconsistent states
  • Operations like INCR are naturally atomic

Redis Consistency Model​

Redis favors:

Consistency per key, not global consistency

Key Guarantees:

  • Single command = atomic
  • No partial writes
  • Replica lag possible
  • Cluster operations limited across shards

Redis Performance Model​

Why Redis is fast:

graph TD
RAM --> NoDiskIO
SingleThread --> NoLocks
SimpleCommands --> PredictableLatency

Typical Latency:

OperationLatency
GET~1µs
INCR~1–2µs
Network RTTDominant cost

Redis Is Not a Silver Bullet​

Redis trades features for speed.

What Redis Is Great At:

  • Counters
  • Caching
  • Rate limiting
  • Queues
  • Leaderboards
  • Distributed locks

What Redis Is Not Good At:

  • Complex queries
  • Large cold datasets
  • Long-term storage without planning

Data Structures and Operations​

Redis Data Model (Mental Model)​

Redis is not a table store.

Think of Redis as a huge hash map in memory:

graph TD
Redis --> Key1["user:1"]
Redis --> Key2["order:991"]
Redis --> Key3["counter:login"]

Each key:

  • Is a string
  • Maps to a data structure, not just a value

Redis Data Structures (Conceptual View)​

Redis supports rich native data types:

TypeThink of it as
StringBasic value / counter
HashObject
ListQueue / Stack
SetUnique collection
Sorted SetRanking system
StreamAppend-only log

Example Mental Mapping:

graph LR
AppObject --> RedisHash["user:42 (hash)"]
Queue --> RedisList["jobs:list"]
Ranking --> ZSet["leaderboard"]

Redis as a Counter System (INCR)​

Counters are one of Redis's killer features.

Why INCR Is Special:

  • Atomic
  • Lock-free
  • Extremely fast
  • Works under massive concurrency

How INCR Works Internally:

sequenceDiagram
participant App1
participant App2
participant Redis

App1->>Redis: INCR page_views
App2->>Redis: INCR page_views
Redis->>Redis: page_views = page_views + 1
Redis-->>App1: 101
Redis-->>App2: 102

Because Redis is single-threaded:

  • No race conditions
  • No lost updates

How Redis Stores Data​

Important Clarification:

Redis does NOT understand { "user": "whereq" } as a data structure by default. Redis only sees bytes.

The application must decide how to store data:

Option 1: Simple key-value

SET user whereq

Stored as:

"user" -> "whereq"

Option 2: Namespaced key

SET user:whereq ""

Stored as:

"user:whereq" -> ""

Option 3: JSON serialized

SET user:whereq '{"user":"whereq"}'

Stored as:

"user:whereq" -> '{"user":"whereq"}'

Option 4: Redis Hash (logical structure)

HSET user:whereq user whereq

Stored internally as:

"user:whereq" -> hash(user=whereq)

Persistence Mechanisms​

Overview​

Redis is fundamentally an in-memory database, but it provides optional durability through two complementary persistence mechanisms:

graph LR
Memory["In-Memory Data<br/>(Source of Truth)"] --> RDB["RDB<br/>(Point-in-Time Snapshot)"]
Memory --> AOF["AOF<br/>(Write Command Log)"]

Key Principle:

Memory is always the source of truth. RDB and AOF are derived artifacts for recovery.

AspectRDBAOF
What it storesBinary snapshot of all dataSequential log of write commands
When it writesPeriodic (configurable intervals)After every write (configurable)
Recovery speedFast (binary load)Slower (replay commands)
Data loss riskHigher (between snapshots)Lower (per-write durability)
File sizeCompactLarger (needs rewriting)
Best forBackups, replica syncMaximum durability

Data Flow Through Persistence:

sequenceDiagram
participant Client
participant Redis Memory
participant AOF
participant RDB

Client->>Redis Memory: SET key value
Redis Memory->>Client: OK
Redis Memory->>AOF: Append command (immediate)

Note over Redis Memory,RDB: Later (based on save intervals)
Redis Memory->>RDB: Fork & snapshot all data

Note over Redis Memory: On restart:
alt AOF exists
AOF->>Redis Memory: Replay commands
else RDB only
RDB->>Redis Memory: Load snapshot
end

Relationship Between Components:

┌───────────────────────────────────────────────┐
│ │
│ WRITES FLOW (Write Command) │
│ │
│ Client → Memory → AOF (append) │
│ ↓ │
│ (periodic) │
│ ↓ │
│ RDB (snapshot) │
│ │
└───────────────────────────────────────────────┘

┌───────────────────────────────────────────────┐
│ │
│ RECOVERY FLOW (Redis Restart) │
│ │
│ 1. Check if AOF exists? │
│ ├─ Yes → Load AOF (replay all commands) │
│ └─ No → Load RDB (restore snapshot) │
│ │
│ 2. Memory rebuilt │
│ 3. Ready to serve │
│ │
└───────────────────────────────────────────────┘

RDB (Snapshotting)​

What RDB Really Is:

RDB = Redis DataBase file

It is:

  • A binary file on disk
  • Representing a point-in-time snapshot of Redis memory
  • Created by Redis itself, not by the OS

Think of RDB as:

📸 "A photograph of all Redis data at a specific moment in time"

What is inside an RDB file?

  • Every key
  • Every value
  • The data type of each value
  • Expiration metadata (TTL)
graph TD
RAM["Redis In-Memory Data"] --> Snapshot["RDB File on Disk"]

Is RDB a runtime state?

No. RDB exists outside Redis memory.

Redis runtime state is:

  • Live data in RAM
  • Active connections
  • Client buffers

RDB is only used for:

  • Restart recovery
  • Backup
  • Replication bootstrap

How RDB is created (conceptually):

Redis:

  1. Forks a child process
  2. Child process reads memory
  3. Writes a compact binary file
  4. Parent keeps serving requests
sequenceDiagram
Redis->>OS: fork()
Child->>Disk: write snapshot
Parent->>Clients: continue serving

✅ No blocking of the main Redis thread ❗ Extra memory during fork (copy-on-write)

When Redis uses RDB:

SituationUse RDB?
Redis restart✅
Full data reload✅
Replica sync (initial)✅
Incremental updates❌

Characteristics:

  • Periodic full snapshot
  • Fast restart
  • Risk of data loss between snapshots
timeline
t1 : Snapshot
t2 : Snapshot
t3 : Crash (data lost between t2 and t3)

AOF (Append Only File)​

What AOF Is:

  • Logs every write
  • More durable
  • Slightly slower
flowchart LR
Write --> AppendLog --> Disk

How Redis Manages Both RDB and AOF Together​

Key clarification:

There is NO extra daemon, NO sync process, NO background service.

Everything is handled by the Redis server process itself.

RDB vs AOF – different purposes:

RDBAOF
SnapshotCommand log
PeriodicContinuous
Fast restoreDurable
CompactLarger

They do not sync with each other.

How Redis actually works with both enabled:

graph TD
Write --> Memory
Write --> AOF
Timer --> RDB

Important insight:

  • Memory is the source of truth
  • RDB and AOF are derived artifacts

Restart logic (very important):

When Redis restarts:

  1. If AOF exists → AOF is used
  2. Else → RDB is used
flowchart TD
Start --> AOFExists{AOF exists?}
AOFExists -->|Yes| LoadAOF
AOFExists -->|No| LoadRDB

Why no syncing is needed:

Because:

  • Redis never writes from RDB to AOF
  • Redis never writes from AOF to RDB
  • Both are written from memory

Memory → Persistence Not persistence → memory → persistence

Production Reality:

Most systems use:

graph TD
Redis --> RDB
Redis --> AOF

Snapshots for speed + AOF for safety.

Persistence and Migration​

When slot migration occurs:

  • Migration moves keys in RAM
  • Does NOT force persistence
  • Persistence happens later

If a master crashes after migration but before snapshot:

Redis handles this safely:

  1. Master restarts and loads entire RDB into RAM
  2. Cluster metadata (slot ownership) overrides RDB contents
  3. Keys that no longer belong to this master are marked invalid
  4. They will NOT be served and will eventually be deleted

Key Principle:

Cluster metadata (slot ownership) overrides RDB contents

Will snapshot happen automatically after migration?

No. Slot migration does NOT automatically trigger an RDB snapshot.

Redis deliberately decouples:

  • data movement
  • persistence timing

Otherwise, resharding would cause massive disk IO spikes and latency stalls.

Important:

  • RDB = local historical photo
  • RAM = source of truth
  • Slots = routing metadata
  • Migration = RAM → RAM transfer
  • Persistence = future snapshots reflect reality

RDB and AOF File Formats Deep Dive​

This section explores the internal structure of Redis persistence files, how they are organized on disk, and the engineering trade-offs made in their design.

RDB File Format and Structure​

What's Inside an RDB File?

An RDB file is a binary serialization of Redis's entire in-memory dataset at a specific point in time.

graph TD
RDB[RDB File] --> Header[Header: Magic + Version]
RDB --> DB0[Database 0]
RDB --> DB1[Database 1]
RDB --> DBN[Database N]
RDB --> EOF[EOF Marker + Checksum]

DB0 --> Meta0[Metadata: DB selector]
DB0 --> Keys0[Key-Value Pairs]
DB0 --> Expire0[Expiry Information]

RDB File Structure (Sequential Layout):

┌─────────────────────────────────────────────────────────┐
│ MAGIC NUMBER: "REDIS" (5 bytes) │
├─────────────────────────────────────────────────────────┤
│ VERSION: "0009" (4 bytes, current version) │
├─────────────────────────────────────────────────────────┤
│ AUXILIARY FIELDS (optional metadata) │
│ - Redis version │
│ - Creation timestamp │
│ - Used memory │
│ - Replication info │
├─────────────────────────────────────────────────────────┤
│ DATABASE SELECTOR (e.g., DB 0) │
├─────────────────────────────────────────────────────────┤
│ RESIZE DB (hash table size hints) │
├─────────────────────────────────────────────────────────┤
│ KEY-VALUE PAIRS │
│ ┌──────────────────────────────────────┐ │
│ │ Type byte (string/list/hash/etc.) │ │
│ │ Key (encoded string) │ │
│ │ Value (type-specific encoding) │ │
│ │ Expiry (optional, millisecond TTL) │ │
│ └──────────────────────────────────────┘ │
│ ... (repeat for all keys in DB) │
├─────────────────────────────────────────────────────────┤
│ DATABASE SELECTOR (e.g., DB 1) │
│ ... (more key-value pairs) │
├─────────────────────────────────────────────────────────┤
│ EOF MARKER (0xFF) │
├─────────────────────────────────────────────────────────┤
│ CRC64 CHECKSUM (8 bytes) │
└─────────────────────────────────────────────────────────┘

Key-Value Encoding:

Each data type has optimized encoding:

Data TypeEncoding StrategyExample
StringRaw bytes, integer encoding for numbers"123" → 3-byte integer vs 3-byte string
ListZiplist (compact) or LinkedListSmall lists: compressed format
HashZiplist or Hash TableDepends on size thresholds
SetIntSet or Hash TableInteger-only sets: compact array
Sorted SetZiplist or Skip ListHybrid encoding based on size
StreamRadix tree nodesSpecialized compact format

Example: String Encoding

Small integers (0-9999):
Type: 1 byte (REDIS_ENCODING_INT)
Value: variable length integer encoding

Short strings (<= 20 bytes):
Type: 1 byte
Length: 1 byte
Data: N bytes

Long strings:
Type: 1 byte
Length: 4 bytes (encoded)
Data: N bytes

LZF Compressed strings:
Type: 1 byte + compression flag
Compressed length: variable
Uncompressed length: variable
Compressed data: N bytes

Does RDB Have Indexes?

❌ No — RDB has NO internal indexes.

RDB is designed for:

  • Sequential write (fast snapshot creation)
  • Sequential read (fast full restore)
  • Minimal file size (no index overhead)

How RDB Restore Works:

sequenceDiagram
participant Redis
participant RDB File
participant Memory

Redis->>RDB File: Open and validate checksum
RDB File->>Redis: Read header
Redis->>RDB File: Read DB selector
loop For each key-value pair
RDB File->>Redis: Read type + key + value
Redis->>Memory: Insert into hash table
Redis->>Memory: Set expiry if present
end
RDB File->>Redis: EOF marker
Redis->>Redis: Validate CRC64
Redis->>Memory: Rebuild indexes (if needed)

Key Point:

RDB is a dump, not a database. It's optimized for bulk I/O, not random access.

RDB Trade-offs:

Advantage ✅Disadvantage ❌
Compact binary formatNo indexes = can't query RDB directly
Fast to write (sequential I/O)Must load entire file to access any data
Fast to read (sequential load)Memory spike during fork (copy-on-write)
Built-in compression (LZF)Periodic snapshots = potential data loss
CRC64 integrity checkLarge datasets = long snapshot time
Portable across architecturesNot human-readable

RDB Creation Process (Technical Details):

flowchart TD
Start[Write command triggers save] --> Fork[fork() creates child process]
Fork --> COW[Parent uses Copy-on-Write]
Fork --> Child[Child has memory snapshot]

COW --> Parent[Parent continues serving requests]
Child --> Iterate[Iterate all databases]
Iterate --> Encode[Encode each key-value]
Encode --> Write[Sequential write to temp file]
Write --> Rename[Atomic rename to dump.rdb]
Rename --> Exit[Child exits]

Parent --> Continue[Continues normally]
Exit --> Continue

Why RDB Uses fork():

  • Parent process never blocks on disk I/O
  • Child gets consistent point-in-time snapshot
  • Copy-on-write = minimal memory overhead (only modified pages are copied)

Storage Organization:

Redis Data Directory:
/var/lib/redis/
├── dump.rdb ← Current RDB snapshot
├── temp-<pid>.rdb ← Temporary file during save (deleted after rename)
└── appendonly.aof ← AOF file (if enabled)

RDB Configuration:

# Automatic snapshots
save 900 1 # After 900 sec (15 min) if at least 1 key changed
save 300 10 # After 300 sec (5 min) if at least 10 keys changed
save 60 10000 # After 60 sec if at least 10000 keys changed

# Compression
rdbcompression yes # Use LZF compression (CPU vs size trade-off)

# Checksum
rdbchecksum yes # CRC64 integrity check

AOF File Format and Structure​

What's Inside an AOF File?

AOF (Append-Only File) is a text-based log of every write operation, stored in RESP protocol format (Redis Serialization Protocol).

graph TD
AOF[AOF File] --> Cmd1[Command 1: SET key1 value1]
AOF --> Cmd2[Command 2: INCR counter]
AOF --> Cmd3[Command 3: LPUSH list item]
AOF --> CmdN[Command N: ...]

Cmd1 --> RESP1[RESP Format]
Cmd2 --> RESP2[RESP Format]

AOF File Structure (Sequential Log):

┌─────────────────────────────────────────────────────────┐
│ *2 ← Array with 2 elements │
│ $6 ← Bulk string, 6 bytes │
│ SELECT ← Command name │
│ $1 ← Bulk string, 1 byte │
│ 0 ← Database number │
├─────────────────────────────────────────────────────────┤
│ *3 ← Array with 3 elements │
│ $3 ← Bulk string, 3 bytes │
│ SET ← Command name │
│ $4 ← Bulk string, 4 bytes │
│ key1 ← Key │
│ $6 ← Bulk string, 6 bytes │
│ value1 ← Value │
├─────────────────────────────────────────────────────────┤
│ *2 │
│ $4 │
│ INCR │
│ $7 │
│ counter │
├─────────────────────────────────────────────────────────┤
│ ... (commands continue chronologically) │
└─────────────────────────────────────────────────────────┘

RESP Protocol Format:

RESP Types:
*N = Array with N elements
$N = Bulk string with N bytes
+ = Simple string
- = Error
: = Integer

Example: Storing a Hash

HSET user:1 name "Alice" age 30

Becomes in AOF:

*5
$4
HSET
$6
user:1
$4
name
$5
Alice
$3
age
$2
30

Does AOF Have Indexes?

❌ No — AOF also has NO indexes.

AOF is:

  • Append-only (writes only at the end)
  • Sequential read (replay from start)
  • No random access needed

How AOF Restore Works:

sequenceDiagram
participant Redis
participant AOF File
participant Memory

Redis->>AOF File: Open file
loop For each command
AOF File->>Redis: Read RESP command
Redis->>Redis: Parse command
Redis->>Memory: Execute command
Redis->>Memory: Update data structures
end
AOF File->>Redis: EOF reached
Redis->>Memory: Ready to serve

AOF Rewrite Process:

Over time, AOF grows as it logs every write. Redis periodically rewrites AOF to compact it.

flowchart TD
Start[AOF grows too large] --> Trigger[Auto-rewrite triggered]
Trigger --> Fork[fork() child process]

Fork --> Parent[Parent: Continue logging to old AOF]
Fork --> Child[Child: Iterate current memory]

Child --> Snapshot[Read current state from memory]
Snapshot --> Generate[Generate minimal commands]
Generate --> NewAOF[Write to temp AOF file]

Parent --> Buffer[Buffer new writes in memory]
NewAOF --> Append[Append buffered writes]
Append --> Rename[Atomic rename to appendonly.aof]
Rename --> Replace[Old AOF replaced]

Before Rewrite:

SET counter 1
INCR counter
INCR counter
INCR counter
DEL temp_key
SET temp_key value
DEL temp_key
SET user:1 Alice
SET user:1 Bob
... (thousands of operations)

After Rewrite:

SET counter 4
SET user:1 Bob
... (only final state)

AOF Rewrite = Converting history into current state

AOF Fsync Strategies:

AOF offers three durability levels via appendfsync:

StrategyBehaviorDurabilityPerformanceData Loss Risk
alwaysfsync() after every writeMaximumSlowest~0 (only crash during write)
everysecfsync() once per secondGoodFast~1 second of data
noLet OS decide when to flushMinimalFastestPotentially 30+ seconds

Technical Detail:

// Simplified AOF write flow
write_command_to_aof_buffer(command);

if (appendfsync == ALWAYS) {
fsync(aof_fd); // Force write to disk NOW
} else if (appendfsync == EVERYSEC) {
// Background thread calls fsync() every second
schedule_fsync_background();
} else {
// OS will flush when it wants
}

Storage Organization:

Redis Data Directory:
/var/lib/redis/
├── appendonly.aof ← Main AOF file
├── appendonly.aof.manifest ← Manifest for multi-part AOF (Redis 7+)
├── appendonly.aof.1.base.rdb ← Base RDB (Redis 7+)
├── appendonly.aof.1.incr.aof ← Incremental AOF (Redis 7+)
└── temp-rewriteaof-<pid>.aof ← Temporary during rewrite

Redis 7+ Multi-Part AOF:

Redis 7 introduced AOF 2.0 with multiple files:

graph LR
Base[Base RDB Snapshot] --> Incr1[Incremental AOF 1]
Incr1 --> Incr2[Incremental AOF 2]
Incr2 --> IncrN[Incremental AOF N]

Benefits:

  • Faster rewrite (doesn't block as long)
  • Better crash recovery
  • Lower memory overhead

AOF Trade-offs:

Advantage ✅Disadvantage ❌
Much better durability (per-write)Larger file size than RDB
Human-readable (RESP text format)Slower recovery (must replay all commands)
Automatic rewrite compactionRewrite can spike CPU/memory
Granular fsync controlNot portable across Redis versions (sometimes)
Append-only = crash-safe writesCan grow very large without rewrite
Preserves command historyNo random access to specific keys

AOF Configuration:

# Enable AOF
appendonly yes

# Fsync strategy
appendfsync everysec # Recommended balance

# Auto-rewrite trigger
auto-aof-rewrite-percentage 100 # Rewrite when 100% larger
auto-aof-rewrite-min-size 64mb # Minimum size to trigger

# AOF file name
appendfilename "appendonly.aof"

# Use RDB preamble (hybrid mode)
aof-use-rdb-preamble yes # Store base snapshot as RDB + incremental AOF

Hybrid Persistence: RDB + AOF Together​

How They Work Together:

graph TD
Write[Write Command] --> Memory[Update Memory]
Memory --> AOF[Append to AOF]

Timer[Periodic Timer] --> RDB[Create RDB Snapshot]
RDB --> Disk1[dump.rdb]
AOF --> Disk2[appendonly.aof]

Memory -.Source of Truth.-> Both[Both derive from memory]

Restart Priority:

flowchart TD
Start[Redis Starts] --> Check{AOF enabled?}
Check -->|Yes| AOFExists{AOF file exists?}
Check -->|No| UseRDB[Load dump.rdb]

AOFExists -->|Yes| LoadAOF[Load AOF<br/>Higher priority]
AOFExists -->|No| LoadRDB[Load dump.rdb]

LoadAOF --> Validate[Validate data]
LoadRDB --> Validate
UseRDB --> Validate

Why AOF Has Priority?

AOF is more up-to-date (potentially has writes after last RDB snapshot)

Hybrid Persistence Strategy (Best Practice):

# Enable both
save 900 1
appendonly yes
appendfsync everysec

# Use RDB-format base in AOF rewrites
aof-use-rdb-preamble yes

Result:

  • RDB provides fast restart baseline
  • AOF provides durability since last snapshot
  • Hybrid mode = best of both worlds

File Format Evolution and Compatibility​

RDB Versions:

VersionRedis VersionKey Changes
11.0Initial format
52.6Millisecond expiry
62.8LZF compression improvements
73.2Quick list encoding
95.0Stream data type, 64-bit lengths

Compatibility:

  • Redis can read older RDB formats
  • Redis cannot read newer RDB formats
  • Always test before downgrading

AOF Format:

  • RESP2 (traditional)
  • RESP3 (Redis 6+, backward compatible)
  • Multi-part AOF (Redis 7+)

Performance Characteristics​

RDB Performance:

graph LR
Fork[fork] -->|Instant| Child[Child Process]
Child -->|Sequential Write| Disk[Disk I/O]
Disk -->|~1-10 seconds| Complete[Complete]

Fork -.Copy-on-Write.-> Memory[Memory Overhead]

Typical RDB Save Time:

Dataset SizeApproximate TimeMemory Overhead
1 GB1-2 seconds< 50 MB (COW)
10 GB10-20 seconds< 500 MB (COW)
100 GB100-200 seconds< 5 GB (COW)

AOF Performance:

  • Write latency: +0.1-1 ms (depends on fsync strategy)
  • Rewrite time: Similar to RDB (background fork)
  • Recovery time: Slower than RDB (replay all commands)

Comparison:

Recovery Time:
RDB: 10 GB → ~30 seconds (sequential read + hash table rebuild)
AOF: 10 GB → ~5 minutes (parse + execute millions of commands)

Durability:
RDB: Last snapshot (potentially minutes/hours old)
AOF: Last fsync (potentially 0-1 seconds old)

Internal Storage and I/O Optimization​

Linux Kernel Optimizations Redis Uses:

  1. fork() + Copy-on-Write

    • Shared memory pages until modification
    • Minimal memory duplication
  2. mmap() for file I/O

    • Memory-mapped files for fast access
  3. fallocate() for AOF

    • Pre-allocate disk space for better performance
  4. fdatasync() vs fsync()

    • fdatasync() = faster, skips metadata updates

Write Amplification:

AOF Write Amplification = How many disk writes per logical write?

appendfsync always: 1 fsync per write = High durability, high cost
appendfsync everysec: 1 fsync per second = Good balance
appendfsync no: OS buffering = Low cost, low durability

Trade-offs Summary​

RDB:

✅ Fast restart
✅ Compact file
✅ No runtime overhead
✅ Perfect for backups

❌ Data loss risk (between snapshots)
❌ Memory spike during fork
❌ Long save time for huge datasets
❌ No incremental updates

AOF:

✅ High durability (per-write)
✅ Human-readable
✅ Continuous updates
✅ Automatic compaction (rewrite)

❌ Larger files
❌ Slower recovery
❌ Slight write latency overhead
❌ Rewrite can be expensive

Hybrid (Recommended):

✅ Best of both worlds
✅ Fast recovery (RDB base)
✅ Minimal data loss (AOF incremental)

⚠️ Slightly more complex configuration
⚠️ Uses more disk space

Key Takeaways​

  1. RDB = Binary snapshot, NO indexes, sequential I/O optimized
  2. AOF = Command log, NO indexes, append-only optimized
  3. Both use fork() to avoid blocking main thread
  4. Neither has random access capability - they're recovery mechanisms, not databases
  5. Hybrid mode (RDB + AOF) is the recommended production setup
  6. File formats evolve with Redis versions - test compatibility when upgrading

Mental Model:

┌─────────────────────────────────────────────┐
│ In-Memory Data Structures │
│ (Source of Truth) │
└─────────┬─────────────────────┬─────────────┘
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ RDB │ │ AOF │
│ Snapshot │ │ Command │
│ File │ │ Log │
└──────────┘ └──────────┘

No Indexes No Indexes
Sequential I/O Append-Only
Fast Recovery High Durability

Replication and High Availability​

Master–Replica Model​

Replication means copying data from one Redis to others.

graph TD
Master --> Replica1
Master --> Replica2
  • Writes go to Master
  • Reads can go to Replicas
  • Replicas are eventually consistent

Why Replicas Matter:

  • Scale reads
  • Protect against data loss
  • Enable failover

How Master–Replica Works Internally​

Key correction:

❗ Master is NOT always the only interface to the application

That's a design choice, not a Redis rule.

Logical model:

graph TD
App --> Master
Master --> Replica1
Master --> Replica2
  • Writes → Master
  • Reads → Master or Replicas (optional)

How data flows:

There is no separate sync daemon.

Replication is done by:

  • Redis master
  • Redis replica
  • Using a replication protocol

Replication phases:

Phase 1: Full sync (initial)

sequenceDiagram
Replica->>Master: SYNC
Master->>Replica: RDB snapshot
Master->>Replica: buffered writes

Phase 2: Continuous replication

sequenceDiagram
Client->>Master: Write
Master->>Replica: Propagate command
  • Replicas replay commands
  • Replication is asynchronous

Can applications talk to replicas?

Yes — but carefully.

OperationSafe on Replica?
Reads✅
Writes❌
Strong consistency❌

Master and Replica Relationship:

  • Master and replica are the same Redis server binary
  • Same persistence mechanisms (RDB/AOF)
  • Same data model
  • Same networking behavior
  • Roles are logical, not physical

Replacement happens via failover, not magically.

graph LR
Master --> Replica
Replica -->|Failover| NewMaster

Sentinel (High Availability Brain)​

Redis Sentinel watches Redis instances.

What Sentinel Does:

  • Monitors Redis health
  • Detects failures
  • Promotes replicas
  • Reconfigures clients

Sentinel Architecture:

graph TD
Sentinel1 --> Master
Sentinel2 --> Master
Sentinel3 --> Master
Master --> Replica

Sentinels vote to avoid false failover.

Why 3 Sentinels?

Quorum and fault tolerance

Sentinel is NOT Redis:

  • Sentinel is a separate process
  • It monitors Redis nodes
  • It talks to other Sentinels

Why multiple Sentinels?

If only 1 Sentinel:

  • Network glitch = false failover

If 2 Sentinels:

  • Split brain possible

If 3 Sentinels:

  • Majority vote works

Odd number rule:

graph TD
Sentinel1 --> Vote
Sentinel2 --> Vote
Sentinel3 --> Vote

Majority = 2 out of 3 Failover only happens if quorum is reached

Sentinel failure tolerance:

SentinelsCan lose
31
52

Automatic Failover Flow:

sequenceDiagram
Sentinel->>Master: Ping
Master--x Sentinel: No response
Sentinel->>Replica: Promote
Replica->>Sentinel: I am master

Redis Cluster and Sharding​

Understanding Nodes, Instances, and Masters​

Terminology:

TermMeaning
NodeA machine / VM / container
Redis instanceOne Redis process
MasterA Redis instance that owns slots
ReplicaA Redis instance replicating a master

Important:

🔴 One Redis instance = one role (master OR replica) 🔴 One node can host multiple Redis instances (rare in prod)

Correct architecture:

graph TD
Node1 --> MasterA
Node2 --> ReplicaA1
Node3 --> ReplicaA2

Best practice:

One Redis instance per node for:

  • Simpler failure domain
  • Clear CPU / memory isolation
  • Predictable persistence behavior

What Is Redis Cluster?​

When a single Redis node is not enough, you use Redis Cluster.

What Is Sharding?

Splitting data across multiple Redis nodes.

graph TD
App --> Slot1[Redis Node A]
App --> Slot2[Redis Node B]
App --> Slot3[Redis Node C]

Correct Redis Cluster Layout:

A Redis Cluster has multiple masters, each responsible for a subset of slots

graph TD
Master1 --> Replica1
Master2 --> Replica2
Master3 --> Replica3

Each master:

  • Owns a different shard (slot range)
  • Has 0+ replicas

Why multiple masters are required:

ReasonExplanation
ScalabilityOne master cannot hold entire keyspace
ThroughputWrites distributed across masters
AvailabilityFailure isolates only part of data

Understanding Sharding​

Correct definition:

Sharding partitions the KEYSPACE, not snapshots

Sharding:

  • Happens at runtime
  • Happens at key routing time
  • Is independent of persistence

Correct sharding model:

graph TD
Keyspace --> Slot0-5000 --> MasterA
Keyspace --> Slot5001-10000 --> MasterB
Keyspace --> Slot10001-16383 --> MasterC

Replica relationship:

graph TD
MasterA --> ReplicaA1
MasterB --> ReplicaB1

❗ Replicas do NOT participate in sharding ❗ Replicas mirror exactly one master's shard

Snapshot relationship:

graph TD
MasterA --> RDB_A
MasterB --> RDB_B
ReplicaA1 --> RDB_A_replica

📌 Each instance snapshots only its own shard

Important Clarifications:

  • Sharding = logical keyspace partitioning concept
  • Realized by assigning slot ranges to Redis master instances
  • NOT a file, NOT a database, NOT a persistence mechanism

Shard Definition:

Shard = Master + its assigned slots (+ replicas)

One master owns one or more slot ranges, which together form a shard.

A master can own multiple non-contiguous slot ranges, but conceptually they are treated as one shard.

graph TD
MasterA --> Slot1_100
MasterA --> Slot300_500

Redis Cluster Architecture​

Critical Understanding:

Redis has NO "cluster main controller" Redis Cluster is fully decentralized

Every Redis node:

  • Knows the slot map
  • Can redirect clients
  • Acts independently

What Redis Cluster Solves:

ProblemSolved?
Horizontal scaling✅
Node failure✅
Cross-slot transactions❌ (limited)

Big Picture:

graph TD
App --> RedisCluster
RedisCluster --> Master1
RedisCluster --> Master2
Master1 --> Replica1
Master2 --> Replica2
Sentinel --> RedisCluster
RedisCluster --> Disk

Hash Slots Deep Dive​

Why Redis Has Hash Slots (Not Just Hash % N)​

Before Redis Cluster (pre-3.0), people did:

node = hash(key) % number_of_nodes

Problems with that approach:

  • Adding/removing a node reshuffles almost all keys
  • Massive cache invalidation
  • Impossible to do online resharding

Redis Cluster Solution: Hash Slots

Redis introduces an indirection layer:

Key → Hash → Slot → Master Node
flowchart LR
Key --> Hash
Hash --> Slot[0..16383]
Slot --> Master

Fixed number of slots:

  • 16384 slots (0–16383)
  • Slots never change
  • Only slot → node mapping changes

This is the foundation of Redis Cluster scalability.

Exact Hash / Slot Algorithm​

Step 1: Extract hash tag (if any)

key = user:{42}:profile
hash_input = 42

If {} exists → only content inside is hashed If not → entire key is hashed

Step 2: CRC16

slot = CRC16(hash_input) % 16384

Why CRC16?

  • Fast
  • Good distribution
  • Deterministic
  • Same across all clients

Step 3: Slot → Master lookup

flowchart LR
Slot -->|Slot table| MasterA

Important:

🔹 Only the key string matters for slot calculation 🔹 The value is irrelevant for routing

Slot Ranges and Assignment​

Is the Redis slot just an integer calculated by CRC16?

✅ Yes — in modern Redis Cluster:

slot = CRC16(key) % 16384
  • Slot range is fixed: 0 – 16383
  • This has never changed since Redis Cluster was introduced
  • Slots are integers only, not ranges, not objects

Important:

A key always hashes to the same slot forever. Slot migration only changes which node owns that slot, not how the slot is computed.

Example: 3-Master Cluster

Total slots = 16384

MasterSlots
Master A0 – 5460
Master B5461 – 10922
Master C10923 – 16383

This is:

  • Balanced
  • Explicit
  • Deterministic

📌 These ranges are not required to be contiguous.

Can slots in the same shard be non-contiguous?

✅ Yes — absolutely allowed

Redis Cluster rule:

A master can own any set of slots, contiguous or not

Example:

graph TD
Slot1_100 --> MasterA
Slot201_300 --> MasterA
Slot101_200 --> MasterB

✔ Redis does not require continuity ✔ Continuity is usually chosen for operational simplicity

Why Redis allows non-contiguous slots:

  • Online resharding
  • Gradual migration
  • Failure recovery
  • Load balancing

Why operators prefer contiguous slots:

  • Easier to reason about
  • Faster debugging
  • Cleaner monitoring

Slot Assignment​

Who assigns and manages slot ranges?

❌ Not applications ❌ Not hashing logic ❌ Not modulo arithmetic

✅ Redis Cluster itself (with admin tools)

Slot assignment happens via:

  • redis-cli --cluster create
  • redis-cli --cluster reshard
  • Redis operators / orchestration systems
  • Cloud Redis services

How slot assignment works internally:

Each master maintains:

slots[0..16383] → owned / not owned

Cluster-wide metadata looks like:

flowchart TB
SlotMap["Cluster Slot Map"]

SlotMap -->|0-5460| MasterA
SlotMap -->|5461-10922| MasterB
SlotMap -->|10923-16383| MasterC

Clients cache this map.

Where Is Slot Ownership Stored?

Each Redis node keeps:

  • A full copy of the cluster topology
  • Slot → node mapping
  • Node health state

Stored in:

nodes.conf

This file is:

  • Managed automatically
  • Updated by Redis itself
  • Never edited manually

Slot Migration​

Are Slots Physical Things?

❌ No — slots are NOT physical objects

A slot is:

  • A logical label (0–16383)
  • Used only for routing and ownership
  • Stored as cluster metadata in each Redis node

There is:

  • ❌ No slot file
  • ❌ No slot directory
  • ❌ No slot process
  • ❌ No slot memory region

What is physical?

PhysicalDescription
Redis instanceOS process
MemoryActual key-value storage
KeysStored in RAM
RDB / AOFPersistence files
TCP connectionsClient & cluster bus

Slots are just numbers used to decide which keys belong where.

What Does "Slot Migration" Actually Mean?

Slot migration is a two-part operation:

1. Move ownership of a slot
2. Move keys belonging to that slot

Step-by-Step Slot Migration:

Let's migrate slot 8000 from Master A → Master B

Step 1: Mark slot as "migrating" and "importing"

sequenceDiagram
participant A as Master A
participant B as Master B
participant C as Clients

A->>A: slot 8000 = MIGRATING to B
B->>B: slot 8000 = IMPORTING from A
  • Slot still logically exists
  • Both nodes know migration is in progress

Step 2: Move keys belonging to the slot

flowchart LR
Key1 -->|DUMP| A
A -->|RESTORE| B

Redis internally:

  • Scans keys in slot 8000 on A
  • Serializes each key
  • Sends to B
  • Deletes from A

This is key-level movement, not memory blocks.

What actually moves:

KEY + VALUE + TTL + TYPE

Internally via:

MIGRATE host port key db timeout

Step 3: Finalize slot ownership

sequenceDiagram
A->>A: remove slot 8000
B->>B: own slot 8000
A->>Cluster: update slot map
B->>Cluster: update slot map

Now:

slot 8000 → Master B

Important Corrections:

  • Slots are never transmitted
  • Only key-value pairs move
  • Slot metadata is updated separately

There is no structure like:

{
"slot": 5793,
"data": {...}
}

Who Coordinates Slot Migration?

❌ Not Sentinel ❌ Not a central controller ❌ Not ZooKeeper ❌ Not etcd

✅ Redis nodes coordinate themselves

Redis Cluster is fully decentralized.

Coordination happens via:

  • Cluster bus (TCP port: client_port + 10000)
  • Gossip protocol
  • Cluster state machine inside each Redis process

Who initiates migration?

Usually:

  • redis-cli --cluster reshard
  • Or orchestration tools (K8s operator, Ansible, etc.)

But once initiated: ➡ Redis nodes do everything themselves

How Do Clients Survive During Migration?

This is where ASK / MOVED comes in.

During migration:

sequenceDiagram
Client->>MasterA: GET key_in_8000
MasterA-->>Client: ASK MasterB
Client->>MasterB: ASKING + GET key
  • Temporary redirect
  • Client retries once
  • No downtime

After migration:

sequenceDiagram
Client->>AnyNode: GET key
Node-->>Client: MOVED slot 8000 MasterB
  • Permanent redirect
  • Client updates slot cache

Hash Tags: Forcing Co-Location​

Problem:

Multi-key operations must be on same slot

MGET user:1 user:2 ❌ (likely different slots)

Solution: Hash tags

user:{42}:name
user:{42}:email

Both hash on 42

flowchart LR
user42name --> SlotX
user42email --> SlotX
SlotX --> MasterA

✔ Multi-key ops work ✔ Transactions work ✔ Lua scripts work

Hash Tag Rule:

Only substring inside { } is hashed

Best practice:

  • Use hash tags sparingly
  • Avoid hotspotting
  • Never hash on low-cardinality values (e.g. {US})

Can You Customize Hash / Slot Logic?​

What you CANNOT customize:

ItemCustomizable
Number of slots❌ Fixed at 16384
Hash algorithm❌ CRC16 is fixed
Slot computation❌ Fixed
Slot redirection behavior❌ Fixed

Redis must behave identically across all clients.

What you CAN influence:

1. Key design

This is your only real control

user:{id}:*
order:{id}:*
session:{id}

2. Slot distribution

Via:

  • CLUSTER ADDSLOTS
  • CLUSTER DELSLOTS
  • redis-cli --cluster reshard

3. Read routing

  • Read from master only
  • Or allow replica reads (client-level)

Complete Data Flow Example​

Example: Writing and Reading Data

Data: {user: "whereq"}

Write Flow:

  1. Application stores:

    SET user:whereq '{"user":"whereq"}'
  2. Redis sees:

    KEY = "user:whereq"
    VALUE = '{"user":"whereq"}'
  3. Slot calculation:

    slot = CRC16("user:whereq") % 16384 = 5793
  4. Cluster metadata contains:

    MasterA owns slots: 4096–8191 (example)
    → slot 5793 ∈ MasterA
  5. Data stored in MasterA's RAM (not directly in RDB)

Read Flow:

  1. Application sends:

    GET user:whereq
  2. Client hashes:

    CRC16("user:whereq") → slot 5793
  3. Client (or Redis redirect) routes to MasterA

  4. MasterA returns:

    {"user":"whereq"}

After Migration:

If slot 5793 migrates to MasterB:

  1. MasterA sends entire key-value pair to MasterB
  2. Cluster metadata updated: slot 5793 now belongs to MasterB
  3. Application reads: same hash → slot 5793 → now routes to MasterB
  4. Application receives data transparently

Key Principle:

Slots route data; nodes store data


Operational Considerations​

Adding a New Master to the Cluster​

What happens when you add a new master node?

Step 1: New master joins the cluster

redis-cli --cluster add-node NewMaster ExistingMaster

Result:

  • New master joins with ZERO slots
  • It holds no data
  • It serves no traffic
flowchart LR
A[Master A<br/>Slots 0–5460]
B[Master B<br/>Slots 5461–10922]
C[Master C<br/>Slots 10923–16383]
D[New Master D<br/>Slots: none]

A --- B --- C --- D

📌 Redis will never auto-assign slots. This is intentional.

Why Redis does NOT auto-rebalance slots:

Auto-rebalance would be dangerous:

RiskExplanation
Massive data movementMillions/billions of keys
Latency spikesMIGRATE blocks CPU
Unpredictable IODisk + network storms
App impactCascading failures

Redis follows the principle:

Data movement must be explicit, observable, and reversible

Step 2: Resharding (manual or scripted)

You explicitly run:

redis-cli --cluster reshard

Or specify:

redis-cli --cluster reshard \
--cluster-from <old-master-id> \
--cluster-to <new-master-id> \
--cluster-slots 4096

This:

  • Moves slots, not files
  • Migrates key/value pairs
  • Updates cluster metadata
  • Happens online

Example: 3 Masters → 4 Masters

Before:

MasterSlots
A0–5460
B5461–10922
C10923–16383

Target distribution (example):

MasterSlots
A0–4095
B4096–8191
C8192–12287
D12288–16383

During migration:

sequenceDiagram
participant Client
participant A as MasterA
participant D as MasterD

Client->>A: GET key(slot=13000)
A-->>Client: ASK 13000 D
Client->>D: GET key
D-->>Client: value

Redis uses ASK redirects during migration.

What happens to RDB/AOF after migration?

Persistence follows memory ownership:

  1. Key moves from A → D (RAM)
  2. Slot ownership updated
  3. Next RDB snapshot:
    • A's RDB ❌ no longer has the key
    • D's RDB ✅ contains the key

No syncing between RDBs ever happens.

Adding replicas:

You normally add replicas AFTER slot resharding:

redis-cli --cluster add-node ReplicaOfD NewMasterD --cluster-slave

Replicas automatically sync data.

Does application need to change anything?

❌ No changes required

Clients:

  • Cache slot → master map
  • Receive MOVED / ASK
  • Refresh metadata automatically

This is why slots exist.

Memory Management​

Redis masters must keep data in RAM:

✅ Yes — Redis is fundamentally an in-memory database

Redis guarantees:

  • All active data lives in RAM
  • Persistence is for recovery only

What if data exceeds RAM?

Redis gives you explicit strategies, not silent paging.

Option 1: Eviction (most common)

maxmemory 32gb
maxmemory-policy allkeys-lru

Policies include:

  • allkeys-lru
  • volatile-lru
  • allkeys-random
  • noeviction (write fails)

📌 Redis will never swap to disk automatically.

Option 2: Redis Cluster (scale horizontally)

Instead of bigger RAM:

Add masters
Reshard slots
Distribute memory

Option 3: Redis on Flash (specialized)

Enterprise / Redis Stack only:

  • SSD-backed
  • Still RAM-first
  • Different architecture

Best Practices​

Slot layout:

  • Prefer contiguous slot ranges
  • Evenly distribute across masters

Key naming:

  • Predictable
  • Stable
  • Avoid random prefixes

Bad:

abc123:user:1
xyz999:user:2

Good:

user:1
user:2

Hash tags:

Use only when required:

✔ Sessions ✔ User-scoped objects ❌ Global counters ❌ Time series

Avoid hotspots:

Bad:

counter:{global}

Better:

counter:{region}:{id}

When Adding Masters:

✔ Add master first (empty) ✔ Reshard gradually ✔ Move slots from multiple masters ✔ Monitor latency during migration ❌ Don't auto-rebalance ❌ Don't migrate too many slots at once


Mental Model Summary​

Final Mental Model (The One to Keep)​

KEY --(CRC16 % 16384)--> SLOT --(cluster map)--> MASTER --> REPLICAS
  • Slot ≠ shard
  • Slot ≠ data
  • Slot = routing label
  • Shard = ownership of many slots

Complete Architecture​

KEY
↓
CRC16 hash
↓
SLOT (0–16383)
↓
Cluster Metadata
↓
Owning Master
↓
RAM (source of truth)
↓
RDB / AOF (recovery only)

Key Truths​

TopicCore Truth
RDBDisk snapshot of memory
AOFWrite command log
Both togetherMemory is source of truth
ReplicationMaster streams commands
SentinelExternal HA coordinator
ShardingData split by slots
SlotsLogical routing layer

One-Sentence Summary​

Redis Cluster shards the keyspace using hash slots, assigns slots to masters, replicates data to replicas, and persists each instance independently using RDB/AOF — all without a central controller.

Redis Mental Checklist (Hero Level)​

When designing with Redis, always ask:

  1. Is this hot data?
  2. Do I need atomic operations?
  3. What happens if Redis restarts?
  4. Can I tolerate eventual consistency?
  5. Do I need sharding or just replicas?

Final Thought​

Redis is powerful because it is opinionated.

If you:

  • Respect its strengths
  • Understand its limits
  • Design intentionally

Redis becomes one of the most reliable and elegant tools in distributed systems.


Quick Reference Guide​

Common Redis Patterns​

1. Cache Pattern:

Application → Check Redis → Hit? Return : Fetch from DB → Store in Redis → Return

2. Counter Pattern:

INCR page_views
INCRBY user:123:credits 50

3. Distributed Lock:

SET lock:resource "token" NX PX 30000 # Atomic lock with 30s TTL

4. Rate Limiting:

INCR rate:user:123:20260108 # Count requests per day
EXPIRE rate:user:123:20260108 86400

5. Session Store:

SETEX session:abc123 3600 '{"user_id":123,"role":"admin"}'

Common Commands Cheat Sheet​

OperationCommandExample
Set valueSET key value [EX seconds]SET user:1 "Alice" EX 3600
Get valueGET keyGET user:1
IncrementINCR keyINCR counter
Hash setHSET key field valueHSET user:1 name "Alice"
Hash getHGET key fieldHGET user:1 name
List pushLPUSH key valueLPUSH queue:jobs task1
List popRPOP keyRPOP queue:jobs
Set addSADD key memberSADD tags:post:1 "redis"
Sorted set addZADD key score memberZADD leaderboard 100 "player1"
Check existenceEXISTS keyEXISTS user:1
DeleteDEL keyDEL user:1
Set expiryEXPIRE key secondsEXPIRE session:abc 3600
Get TTLTTL keyTTL session:abc

Configuration Quick Reference​

Persistence:

# RDB Snapshots
save 900 1 # Save if 1+ keys changed in 900s
save 300 10 # Save if 10+ keys changed in 300s
save 60 10000 # Save if 10000+ keys changed in 60s
rdbcompression yes
rdbchecksum yes

# AOF
appendonly yes
appendfsync everysec # Options: always | everysec | no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-use-rdb-preamble yes # Hybrid mode

# Hybrid (Recommended)
save 900 1
appendonly yes
appendfsync everysec
aof-use-rdb-preamble yes

Memory Management:

maxmemory 2gb
maxmemory-policy allkeys-lru # Options:
# allkeys-lru - Evict any key, LRU
# volatile-lru - Evict keys with TTL, LRU
# allkeys-random - Evict any key, random
# volatile-random - Evict keys with TTL, random
# volatile-ttl - Evict keys with shortest TTL
# noeviction - Return errors when memory limit reached

Replication:

# On replica
replicaof <master-ip> <master-port>
replica-read-only yes

Cluster:

cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000

Troubleshooting Guide​

ProblemPossible CausesSolutions
Slow queriesLarge keys, blocking commandsUse SCAN instead of KEYS, avoid SMEMBERS on large sets
Memory spikesRDB fork during saveIncrease memory, schedule saves during low traffic
Data lossNo persistence enabledEnable AOF with appendfsync everysec
Connection timeoutsNetwork issues, maxclients limitCheck maxclients, network latency
Replication lagMaster overloaded, slow networkOptimize write load, check network bandwidth
Cluster MOVED errorsSlot migration in progressNormal during resharding, wait or retry
OOM errorsMemory limit reachedCheck maxmemory-policy, add more nodes

Performance Tips​

DO:

  • ✅ Use pipelining for bulk operations
  • ✅ Use Redis transactions (MULTI/EXEC) for atomicity
  • ✅ Set appropriate TTLs to expire unused data
  • ✅ Use hash tags for multi-key operations in cluster
  • ✅ Monitor with INFO command and Redis monitoring tools
  • ✅ Use connection pooling in applications
  • ✅ Batch operations when possible

DON'T:

  • ❌ Use KEYS * in production (use SCAN instead)
  • ❌ Store huge values (> 100KB) in single keys
  • ❌ Use Redis as primary database without persistence
  • ❌ Run blocking commands (BLPOP with long timeout) without care
  • ❌ Ignore memory limits and eviction policies
  • ❌ Use transactions across multiple hash slots in cluster
  • ❌ Store sensitive data without encryption layer

Redis Versions Features​

VersionKey Features
3.0Redis Cluster introduced
3.2GEO commands, Quick List
4.0Modules system, lazy freeing
5.0Streams data type, new sorted set commands
6.0SSL/TLS, ACLs, RESP3, threaded I/O
6.2Tracking, client-side caching
7.0Functions, sharded pub/sub, ACL improvements, Multi-part AOF

Monitoring Metrics​

Key Metrics to Watch:

# Memory usage
INFO memory
- used_memory
- used_memory_rss
- mem_fragmentation_ratio

# Performance
INFO stats
- instantaneous_ops_per_sec
- total_commands_processed
- keyspace_hits / keyspace_misses (hit ratio)

# Persistence
INFO persistence
- rdb_last_save_time
- rdb_changes_since_last_save
- aof_current_size

# Replication
INFO replication
- role (master/replica)
- connected_slaves
- master_repl_offset

# Cluster
CLUSTER INFO
- cluster_state
- cluster_slots_assigned
- cluster_slots_ok

Resource Planning​

Sizing Guidelines:

Use CaseRecommended SetupExample
Small cacheSingle master1-4 GB RAM
Production cacheMaster + 1-2 replicas + Sentinel8-32 GB RAM per node
Large dataset3+ master cluster, 1 replica each32-256 GB RAM per node
High availabilitySentinel (3+ instances) or ClusterOdd number of sentinels

Network Requirements:

  • Client ↔ Redis: Low latency (< 1ms ideal)
  • Cluster nodes: Low latency (< 1ms), high bandwidth
  • Replication: Sustained bandwidth for writes

Key Corrections and Clarifications​

Slots and Data Storage​

Important:

  • Slots are just labels
  • Nodes own labels
  • Keys follow labels
  • Redis nodes gossip label ownership
  • Shard = naming convenience

Correct Understanding:

  • Slot is a pure logic term
  • Slot represents the keyspace
  • Slot is basically in Redis metadata
  • Slot is a logic label
  • Data is NOT stored "in slots"
  • Data is stored in Master's memory
  • Slot routes request to the correct master

Migration and Persistence​

Important:

  • Slots are NOT stored in RDB
  • Slots are NOT persisted with data
  • Slots live in cluster metadata
RDB = raw key/value memory dump
Cluster metadata = routing truth

Summary Answers:

QuestionCorrect Answer
Snapshot after migration?❌ No
Temporary RDB inconsistency?✅ Yes, expected
Crash before snapshot safe?✅ Yes
Slot logic during RDB load?Metadata-driven
Hash during load?Only for validation
Data always in RAM?✅ Yes
Exceed RAM?Evict or scale

Application Interaction​

Does the application care which master or replica it talks to?

✅ Conceptually correct

Writes:

  • Application does not choose the master
  • Client library computes hash slot
  • Client routes to correct master
  • If wrong node → MOVED redirect

✔ Application logic remains unaware

Reads:

  • Reads go to masters by default
  • Guarantees strong consistency
  • Clients may read from replicas
  • Requires explicit client configuration

⚠️ Replica reads may be eventually consistent

Adding a New Master​

When a new master is added later, Redis does NOT automatically rebalance slots. Slot redistribution (resharding) is an explicit, controlled operation.

Key Principle:

Adding a master does NOT change hashing. It only changes slot ownership — explicitly.

KEY → SLOT (fixed) → MASTER (mutable)

Deep Dive into Redis II

· 5 min read

1. Big Picture: What Redis Actually Is​

At its core, Redis is:

An in-memory data store with optional persistence and optional clustering (sharding + replication)

Everything else is layered on top of this idea.


2. Node, Master, Replica — The Physical Roles​

Node​

A node is a single Redis process:

  • One Redis server
  • One TCP port
  • One memory space
  • One persistence configuration (RDB/AOF)
[ Redis Node ]
├── RAM
├── RDB file (optional)
├── AOF file (optional)
└── Network endpoint

Master​

A master node:

  • Accepts writes
  • Owns slots (in cluster mode)
  • Replicates data to replicas
[ Master Node ]
├── Slot ownership
├── Handles writes
├── RAM = source of truth
└── Replication stream

Replica (Slave)​

A replica:

  • Receives data from a master
  • Is read-only
  • Can be promoted if master fails
Master ---> Replica
RAM RAM
RDB RDB

⚠️ Replicas do not own slots


3. Snapshot vs RDB vs AOF (Critical Distinction)​

Snapshot (Concept)​

A snapshot is a conceptual idea:

“Redis memory state at a specific moment in time”

It is not a file.


RDB (Redis Database File)​

RDB is the physical result of a snapshot

Time T
┌───────────────┐
│ Redis RAM │ ---- SNAPSHOT ----> dump.rdb
└───────────────┘

Key points:

  • Full memory dump

  • Binary file

  • Point-in-time

  • Created via:

    • SAVE
    • BGSAVE
    • Auto rules (e.g., every N seconds if M writes)

AOF (Append Only File)​

AOF records operations, not memory state.

SET a 1
INCR a
INCR a
DEL b

Key points:

  • Append-only
  • Grows over time
  • Can be rewritten (AOF rewrite)
  • Replays commands to rebuild RAM

RDB vs AOF (Mental Model)​

RDB: "What memory looks like"
AOF: "How memory was built"

4. Does RDB File Size Change?​

YES — absolutely

RDB characteristics:​

  • Each dump is independent

  • File size can:

    • Increase
    • Decrease
  • Depends on:

    • Key count
    • Value sizes
    • Data types
    • Compression efficiency
RDB at T1: 120 MB
RDB at T2: 95 MB (keys deleted / smaller values)

❌ There is NO incremental RDB writing

Each RDB is:

A full rewrite, from scratch


AOF File Size Behavior​

  • Always appends
  • Grows monotonically
  • Periodically compacted by rewrite

5. Shards vs Sharding vs Slots​

Sharding (Concept)​

Sharding is a logical data partitioning strategy

“Split the total keyspace across multiple masters”

Sharding itself:

  • Is NOT a file
  • Is NOT persistence
  • Is NOT memory layout

Shard (In Redis Cluster)​

A shard usually means:

One master + its replicas owning a subset of hash slots

Shard 1: MasterA + Replicas
Shard 2: MasterB + Replicas
Shard 3: MasterC + Replicas

6. Slots — The Core of Redis Cluster​

Slot Basics​

Redis Cluster defines:

  • 16384 hash slots
  • Slots are integers: 0 → 16383

Every key maps to exactly one slot


Slot Calculation​

In modern Redis:

slot = CRC16(key) % 16384

Optionally using hash tags:

{user:123}:name
{user:123}:email

→ Same slot


Slots Are LOGICAL​

Slots:

  • Are NOT stored in RDB

  • Are NOT stored in AOF

  • Exist only in:

    • Cluster metadata
    • Node configuration
    • Client routing logic

7. Slot → Master Mapping (Routing)​

Key: user:whereq
|
CRC16
|
Slot 5793
|
Cluster Metadata
|
MasterA
Client
|
|-- key --> slot
|
|-- slot --> master

Clients either:

  • Cache slot tables
  • Or ask Redis (MOVED / ASK)

8. How Slot Ranges Are Assigned​

Initial Cluster Creation​

Slots are manually assigned (by admin tools):

3 Masters
16384 / 3 ≈ 5461 slots each

MasterA: 0 - 5460
MasterB: 5461 - 10922
MasterC: 10923 - 16383

⚠️ Redis does NOT do:

slot % number_of_masters

That would break consistency.


9. Adding a New Master (Re-sharding)​

When a new master joins:

  1. Admin decides how many slots to move
  2. Slots are migrated
  3. Keys are moved key-by-key
Before:
MasterA: 0-5460
MasterB: 5461-10922
MasterC: 10923-16383

After adding MasterD:
MasterA: 0-4095
MasterB: 4096-8191
MasterC: 8192-12287
MasterD: 12288-16383

10. Slot Migration — What Really Moves?​

Key Truth​

Slot migration moves KEY-VALUE PAIRS in RAM NOT RDB files


Migration Flow​

Slot 5793 moves from A → B

MasterA RAM:
user:whereq -> {"user":"whereq"}

MasterB RAM:
(empty)
1) Cluster metadata updated (MIGRATING / IMPORTING)
2) Key-value pairs copied
3) Ownership finalized
MasterA --> send key + value --> MasterB

Slot Structure Clarification​

❌ Slots do NOT store values like:

5793 -> {...}

✅ Actual storage:

"user:whereq" -> '{"user":"whereq"}'

Slot is only used for routing


11. RDB + Slot Migration Interaction​

Scenario Timeline​

Step 1: RDB Snapshot​

MasterA RAM:
user:whereq

dump.rdb:
user:whereq

✔ Correct


Step 2: Slot Migration Happens​

MasterA --> sends key+value --> MasterB

Now:

MasterA RAM: (no user:whereq)
MasterB RAM: user:whereq

Step 3: No Immediate Snapshot​

Redis does NOT auto-trigger RDB after migration.

This is OK because:

RDB is not authoritative for cluster topology


Crash Safety​

If MasterA crashes:

  1. It reloads RDB

  2. It rejoins cluster

  3. Cluster metadata says:

    • Slot 5793 belongs to MasterB
  4. MasterA:

    • Deletes or ignores that key
    • Syncs from correct owner if replica

✔ No double ownership


12. Does Redis Re-hash During RDB Load?​

YES — but conceptually.

When loading RDB:

  • Redis loads keys
  • Slot is recalculated
  • If slot not owned → key discarded or redirected
RDB load
|
|-- key --> slot
|-- slot --> ownership check

13. Routing Summary Diagram​

Client
|
|-- key
|-- CRC16
|-- slot
|
+--> Master (via cluster map)

14. Memory Is Always the Source of Truth​

Redis Fundamental Rule​

All Redis data must fit in RAM

Persistence is:

  • Recovery mechanism
  • Replication mechanism
  • NOT primary storage

What If Data Exceeds RAM?​

Redis will:

  • Fail writes
  • Or evict keys (if configured)

Eviction policies:

noeviction
allkeys-lru
volatile-lru
allkeys-random
...

Redis is NOT a disk-first database.


15. Final Concept Relationship Map​

┌────────────┐
│ Client │
└─────┬──────┘
│
Routing
│
┌──────▼───────┐
│ Hash Slot │ (CRC16 % 16384)
└──────┬───────┘
│
┌────────▼────────┐
│ Master Node │
│ (RAM) │
└──────┬─────────┘
│
┌────────▼────────┐
│ Replica Nodes │
└────────────────┘

Persistence:
RAM --snapshot--> RDB
RAM --commands--> AOF

16. One-Sentence Mental Models​

  • Snapshot → a moment in memory
  • RDB → binary memory dump
  • AOF → command history
  • Slot → routing number
  • Shard → slot-owning master group
  • Sharding → data partition strategy
  • Master → writes + slot owner
  • Replica → copy + failover
  • Node → Redis process
  • Routing → key → slot → master

Deep Dive into Redis III

· 4 min read

Great question — cluster metadata is the last missing puzzle piece that makes everything you learned click. I’ll explain what it is, where it lives, what’s inside it, how it’s updated, and who trusts it, again with diagrams and zero code.


Redis Cluster Metadata — Deep Dive

1. What Is Redis Cluster Metadata?​

Redis cluster metadata is the shared logical map that tells every node:

“Who owns which slots, who is master/replica of whom, and how to route keys.”

It is:

  • NOT user data
  • NOT stored in RDB
  • NOT stored in AOF
  • NOT centralized

It is distributed and eventually consistent.


2. What Information Is in Cluster Metadata?​

Each node stores the same logical structure (with slight timing differences):

Core Contents​

Cluster State:
- cluster_id
- epoch (version)
- nodes:
- node_id
- ip:port
- role (master / replica)
- master_of (if replica)
- slots owned (if master)
- flags (fail, pfail, handshake, etc.)

Conceptual Structure (Simplified)​

CLUSTER METADATA
│
├── Cluster ID
├── Epoch (version)
│
├── Node A
│ ├── id: A1
│ ├── role: master
│ ├── slots: 0-5460
│ └── replicas: B1, B2
│
├── Node B
│ ├── id: C1
│ ├── role: master
│ ├── slots: 5461-10922
│
└── Node C
├── id: D1
├── role: master
├── slots: 10923-16383

3. Where Is Cluster Metadata Stored?​

On Every Node — Locally​

Each Redis node stores cluster metadata in:

nodes.conf

Important properties:

  • Written automatically by Redis
  • Updated dynamically
  • Not meant for manual editing
  • Survives restart
Redis Node
├── RAM
├── dump.rdb
├── appendonly.aof
└── nodes.conf <-- cluster metadata

Key Point​

nodes.conf is NOT a configuration file you manage

It is Redis’s internal cluster ledger


4. nodes.conf — What Does It Look Like Conceptually?​

Conceptually (not exact syntax):

node A1 10.0.0.1:6379 master - 0-5460
node B1 10.0.0.2:6379 slave A1
node C1 10.0.0.3:6379 master - 5461-10922
node D1 10.0.0.4:6379 master - 10923-16383

Each node has:

  • A unique node ID
  • Role
  • Slot ownership (masters only)
  • Relationship to others

5. How Cluster Metadata Is Distributed​

Gossip Protocol​

Redis Cluster uses a gossip-style protocol:

Node A <-- gossip --> Node B <-- gossip --> Node C

Metadata exchange happens:

  • Periodically
  • Piggybacked on heartbeat packets
  • Incrementally

This ensures:

  • No central coordinator
  • No single source of truth
  • High availability

6. Cluster Epoch — Versioning the Metadata​

Why Epoch Exists​

When multiple changes happen:

  • Failover
  • Slot migration
  • Master promotion

Redis needs to decide:

“Which view of the cluster is newer?”

Epoch Rule​

Higher epoch = newer truth

Every metadata-changing event:

  • Increments epoch
  • Spreads via gossip

7. Slot Ownership in Metadata​

Slots are represented as ranges, not individual numbers.

Node A:
slots: 0-4095, 8192-12287

This allows:

  • Non-contiguous ownership
  • Partial migration

During Migration (Transitional States)​

Node A:
MIGRATING slot 5793 -> Node B

Node B:
IMPORTING slot 5793 <- Node A

Clients may see:

  • ASK
  • MOVED

8. Who Updates Cluster Metadata?​

Different Scenarios​

EventWho TriggersHow
Cluster creationAdmin toolAssign slots
Add masterAdmin toolRebalance slots
Slot migrationSource & target mastersSlot states
Master failureRedis nodesVoting + epoch
Replica promotionRedis nodesAutomatic

Important Clarification​

Redis does NOT have a leader node for metadata

All nodes:

  • Observe
  • Vote
  • Converge

9. Client’s View of Cluster Metadata​

Clients:

  • Cache slot → node map
  • Learn from Redis replies
Client --> Node
<-- MOVED 5793 10.0.0.2:6379

Client updates local cache.


Client Metadata vs Server Metadata​

Client cache = hint
Cluster metadata = authority

10. Metadata Is Separate from Data​

This separation is intentional and powerful:

DATA PATH:
Key -> RAM -> RDB/AOF

CONTROL PATH:
Slot -> Metadata -> Routing

This is why:

  • Slot migration doesn’t move files
  • RDB doesn’t store slot info
  • Restarting nodes doesn’t break routing

11. Failure Scenario: Why Metadata Saves You​

Master Crash After Migration​

MasterA crashes
MasterB owns slot 5793

On restart:

  1. MasterA loads RDB
  2. Reads nodes.conf
  3. Sees slot 5793 not owned
  4. Discards or syncs properly

✔ No split-brain


12. Mental Model Diagram (Complete)​

┌────────────┐
│ Client │
└─────┬──────┘
│
Slot Calculation
│
┌──────▼───────┐
│ Slot (0-16383)│
└──────┬───────┘
│
┌─────────────▼─────────────┐
│ Cluster Metadata (nodes) │
│ - slot ranges │
│ - roles │
│ - epochs │
└─────────────┬─────────────┘
│
┌────────▼────────┐
│ Master Node │
│ RAM │
└────────┬────────┘
│
┌─────────▼─────────┐
│ Replica Nodes │
└───────────────────┘

13. One-Sentence Summary​

Redis cluster metadata is a distributed, versioned routing ledger that maps hash slots to master nodes, stored in nodes.conf, propagated by gossip, and completely independent from user data and persistence.

Deep Dive into Keycloak II

· 5 min read

Answer:​

OAuth2 is an authorization framework — it gives applications access to protected resources via access tokens. OIDC is an authentication layer built on top of OAuth2 — it provides identity information by adding ID tokens, user info endpoint, and standardized flows.

You use OAuth2 when an app needs permissions You use OIDC when an app needs login + identity.


2. How does Keycloak handle session management compared to stateless JWT systems?

Answer:​

Keycloak maintains a server-side session for each authenticated user. Even though it issues JWTs, it “peeks” into the token’s session ID to enforce logout and token invalidation. This means:

  • Keycloak can force logout of any user
  • Admin can revoke access tokens instantly
  • Tokens are short-lived; refresh tokens renew them

Stateless JWT-only systems cannot revoke tokens without additional infrastructure.


3. What’s the purpose of a Keycloak Realm?

Answer:​

A Realm is an isolated authentication domain. Each realm has:

  • its own clients
  • identity providers
  • users
  • roles
  • token settings

Realms do not share users. This isolates tenants for multi-tenant apps.


4. Explain the OAuth2 Authorization Code Flow with PKCE.

Answer:​

Used by public clients (SPA, mobile). Steps:

  1. Client generates a code verifier and challenge
  2. User is redirected to Keycloak login
  3. After successful login, Keycloak returns an authorization code
  4. Client exchanges the code for tokens including the code verifier
  5. Keycloak recomputes the challenge to verify request integrity

PKCE prevents code interception attacks.


5. What Keycloak feature would you use for Single Sign-On across multiple apps?

Answer:​

SSO session + OIDC Authorization Code Flow. Apps use the same Keycloak realm; once the user is logged into one app, Keycloak issues tokens to others without re-authentication.


6. How do you configure role-based access control in Keycloak?

Answer:​

  • Create realm roles or client roles
  • Map roles to users or groups
  • Configure clients to include roles in tokens (scope + mappers)
  • Your service enforces authorization based on the roles claim in JWT

7. What is the difference between Realm Roles and Client Roles?

Answer:​

  • Realm roles are global across the entire realm
  • Client roles belong to a specific client (service)

Client roles prevent namespace collisions and support service-specific authorization.


8. How do you secure a Spring Boot REST API using Keycloak?

Answer:​

For Spring Security 6 / Boot 3:

  1. Use the OAuth2 Resource Server starter
  2. Configure issuer-uri pointed to Keycloak realm
  3. Enable JWT validation
  4. Add authorization rules based on roles in JWT

Resource server verifies tokens by fetching Keycloak’s public keys (JWKS).


9. What are Keycloak Identity Providers?

Answer:​

Identity Providers allow Keycloak to delegate login to external systems like:

  • Google
  • GitHub
  • Azure AD
  • Another Keycloak realm

Keycloak becomes an identity broker.


10. How does token revocation work in Keycloak?

Answer:​

Keycloak stores sessions server-side. When an admin revokes tokens or logs a user out:

  • Active tokens become invalid
  • Refresh token rotation prevents re-use
  • Next request with old token fails with 401

This is a major advantage over purely stateless JWT systems.


11. What is Offline Session and Offline Token?

Answer:​

Offline tokens are long-lived refresh tokens not tied to active user sessions. Useful for background jobs or CLI tools. Users do not need to stay logged in.


12. How do you integrate Keycloak with an API Gateway (e.g., Kong, Nginx, Spring Cloud Gateway)?

Answer:​

Use the gateway as an OAuth2 resource server. It:

  • validates JWTs from Keycloak
  • checks scopes/roles
  • forwards authorized requests to microservices
  • optionally strips sensitive headers

Microservices behind the gateway become stateless and do not validate tokens directly.


13. What is the Keycloak Admin API used for?

Answer:​

Automating user and client management:

  • create/update users
  • reset passwords
  • assign roles
  • manage groups
  • manage clients
  • trigger impersonation

Often used in CI/CD or multi-tenant systems.


14. How would you scale Keycloak for production?

Answer:​

  • Run multiple Keycloak nodes behind a load balancer
  • Use a shared database (PostgreSQL recommended)
  • Use sticky sessions only if needed (usually not needed post-Keycloak 17+)
  • Enable cross-node session cache with Infinispan
  • Offload static content via reverse proxy

Keycloak 21+ with Quarkus has much lower memory footprint and better clustering.


15. What are the common pitfalls when integrating SPAs (React) with Keycloak?

Answer:​

  1. Storing tokens insecurely (never use localStorage → use memory with silent refresh).
  2. Using Implicit Flow (deprecated).
  3. Not handling token refresh properly.
  4. Not using PKCE.
  5. Not validating roles on backend.

Best practice: React = public client using Authorization Code + PKCE.


16. How do service-to-service calls work with Keycloak?

Answer:​

Use Client Credentials Flow:

  • No user involved
  • The client authenticates with client secret
  • Keycloak returns an access token
  • Service uses token to call downstream service

This is machine-to-machine authentication.


17. What’s the difference between Access Token and ID Token?

Answer:​

  • Access token = permissions for APIs (authorization)
  • ID token = identity info about the user (authentication)

APIs should ignore ID tokens.


18. How do you restrict which scopes a client may request?

Answer:​

Use Client Scopes:

  • Default client scopes
  • Optional client scopes
  • Protocol mappers to include additional claims
  • Scope parameter to control token content

This prevents over-privileged tokens.


19. What is the Keycloak Token Exchange?

Answer:​

Allows exchanging one token for another. Use cases:

  • service-to-service delegation
  • switching realms
  • getting a token for another client
  • impersonation workflows

Powerful in microservice architectures.


20. How do you debug token validation issues?

Steps to answer:​

  1. Decode token using JWT.io
  2. Verify issuer (iss) matches realm
  3. Confirm audience (aud) includes client ID
  4. Check signature using JWKS
  5. Check token expiration
  6. Check scope/roles included

Most validation failures come from misconfigured issuer-uri or missing mappers.

Keyclock Revocation Mechanism

· 4 min read

⭐ Senior-Level Answer​

Keycloak maintains a server-side Session Store (in-memory or distributed depending on the cluster). This enables Keycloak to centrally track active user sessions including:

  • Each login session
  • Access tokens issued
  • Refresh tokens issued
  • Client sessions
  • Offline sessions
  • Device-specific sessions

When a user logs out or an admin revokes access, Keycloak performs:​

1. Session invalidation in the Session Store​

Keycloak marks the user session as invalid inside its central session map:

session_id -> state = INVALID

This invalidation immediately affects:

  • All clients (web apps, mobile apps, services)
  • All sessions (browser, mobile, REST clients)
  • All tokens linked to the session

This is because all tokens in Keycloak are tied to a Keycloak session ID (sid claim).


2. Refresh Token Blacklist / revocation​

Keycloak stores refresh tokens in a server-side token store (unlike fully stateless JWT systems).

When the user logs out:

  • All refresh tokens belonging to the session are invalidated.
  • Any refresh request with those tokens is rejected.

This blocks all new access tokens across all devices.


3. Token Introspection (for OAuth2 Introspection clients)​

If a client uses token_introspection:

/protocol/openid-connect/token/introspect

Keycloak will return:

"active": false

as soon as the session is invalidated.

This enables immediate revocation for “online” services.


4. Realm-wide or client-side Revocation Events​

Keycloak issues revocation events when:

  • Admin revokes all sessions
  • Client is removed
  • Roles or permissions change
  • User is disabled

Clients using Keycloak Adapters (Tomcat, Spring, Node, Quarkus) receive the event and invalidate local caches of tokens.


⭐ Why Keycloak Can Do This (Technical Reason)

Because Keycloak is NOT fully stateless. It manages a stateful authentication session, and tokens reference that session.

Therefore, revocation propagates instantly and globally.


🔥 Q2 — Why can’t typical stateless JWT systems revoke access as effectively?

⭐ Senior-Level Answer​

A typical stateless JWT architecture uses self-contained JWTs:

  • Token verification is done by services locally via public key.
  • Services do not contact the authorization server again after token issuance.

This means:

❌ Once a JWT is issued, it cannot be revoked until it expires​

Unless you add non-standard extensions.

❌ No central session store​

There is nowhere to mark a session as invalid.

❌ No refresh token blacklist​

Unless implemented manually with a database or Redis.

❌ No push revocation events to API clients​

(Standard OAuth2 libraries don’t have this built in.)


💥 Final summary:​

Keycloak (stateful + centralized session model)​

✔ Server-side session invalidation ✔ Refresh token revocation ✔ Token introspection ✔ Push revocation events to adapters ✔ Offline session management ✔ Works across multiple devices, browsers, apps ✔ Suitable for enterprise SSO

Common JWT-only stateless systems​

❌ No central session ❌ Cannot revoke tokens already issued ❌ Must wait until token expiration ❌ Must manually implement token blacklist ❌ Cannot force logout across multiple clients

This difference is fundamental to why enterprises choose Keycloak.


🔥 Q3 — How does Keycloak ensure cluster-wide revocation in HA mode?

⭐ Senior-Level Answer​

Keycloak in cluster mode (Infinispan or newer methods) shares:

  • User sessions
  • Client sessions
  • Offline sessions
  • Login failure counters
  • Revocation timestamps

All nodes listen to Infinispan cache events.

So when one node invalidates a session:

cluster-broadcast:
event = session-revocation
sid = 7ab3...

Every node immediately removes the session from its local cache.

This ensures revocation is cluster-wide and instantaneous.


🔥 Q4 — How does Keycloak reduce the JWT statelessness problem?

⭐ Keycloak issues stateless JWT access tokens but:​

1. They have short TTL (5–15 minutes recommended) So compromise window is small.

2. They are always tied to a session (sid claim).

3. They can be introspected by resource servers if required.

Keycloak combines:

  • Stateless JWT access tokens (performance)
  • Stateful refresh token & session management (security)

This hybrid model is why Keycloak works well for enterprise.


🔥 Q5 — How to architect revocation-sensitive microservices with Keycloak?

⭐ Best practice​

Token TypeRecommended Use
Access Token (JWT)Short TTL (2–10 min), stateless verification
Refresh TokenManaged by Keycloak; revocation effective
Token IntrospectionOnly for zero-trust or sensitive APIs
Keycloak AdaptersTo receive push revocation events

Your services should not store long-lived sessions themselves. Let Keycloak handle that.

Deep Dive into Local Storage Usage in React with TypeScript

· 4 min read

Local storage is a simple yet powerful tool for persisting data in web applications. In this article, we'll explore every detail of local storage usage in React with TypeScript, including best practices, sample code, and test cases. The illustrations are based on the latest versions of React (18.x) and TypeScript (5.x).


What is Local Storage?​

Local storage is a web API that allows developers to store key-value pairs in a web browser. The data persists even after the browser is closed, making it ideal for saving user preferences, authentication tokens, and other non-sensitive data.

Key Characteristics of Local Storage:​

  • Synchronous API: Operations are blocking.
  • Storage Limit: Approximately 5MB per domain.
  • Data Lifetime: Persists until explicitly deleted.
  • Data Scope: Accessible only within the same origin.

Using Local Storage in React with TypeScript​

1. Basic CRUD Operations​

Local storage provides three core methods:

  • localStorage.setItem(key, value)
  • localStorage.getItem(key)
  • localStorage.removeItem(key)

Example: Wrapper Utility for Local Storage​

We can create a type-safe utility to handle common operations:

// utils/localStorage.ts
export class LocalStorageHelper {
static setItem<T>(key: string, value: T): void {
localStorage.setItem(key, JSON.stringify(value));
}

static getItem<T>(key: string): T | null {
const value = localStorage.getItem(key);
return value ? JSON.parse(value) as T : null;
}

static removeItem(key: string): void {
localStorage.removeItem(key);
}
}

Test Cases​

Using Jest, we can write unit tests to validate the utility:

// utils/localStorage.test.ts
import { LocalStorageHelper } from "./localStorage";

describe("LocalStorageHelper", () => {
beforeEach(() => {
localStorage.clear();
});

it("should set and retrieve an item", () => {
LocalStorageHelper.setItem("key", { name: "John" });
const item = LocalStorageHelper.getItem<{ name: string }>("key");
expect(item).toEqual({ name: "John" });
});

it("should return null for non-existent keys", () => {
const item = LocalStorageHelper.getItem("nonexistent");
expect(item).toBeNull();
});

it("should remove an item", () => {
LocalStorageHelper.setItem("key", "value");
LocalStorageHelper.removeItem("key");
const item = LocalStorageHelper.getItem("key");
expect(item).toBeNull();
});
});

2. Integrating Local Storage in React Components​

Example: Using Local Storage with useState​

// components/Counter.tsx
import React, { useState, useEffect } from "react";
import { LocalStorageHelper } from "../utils/localStorage";

const Counter: React.FC = () => {
const [count, setCount] = useState<number>(() => {
return LocalStorageHelper.getItem<number>("count") || 0;
});

useEffect(() => {
LocalStorageHelper.setItem("count", count);
}, [count]);

return (
<div>
<h1>Counter: {count}</h1>
<button onClick={() => setCount((prev) => prev + 1)}>Increment</button>
<button onClick={() => setCount((prev) => prev - 1)}>Decrement</button>
</div>
);
};

export default Counter;

Test Cases​

// components/Counter.test.tsx
import React from "react";
import { render, screen, fireEvent } from "@testing-library/react";
import Counter from "./Counter";

describe("Counter Component", () => {
it("should initialize with value from local storage", () => {
localStorage.setItem("count", "5");
render(<Counter />);
expect(screen.getByText(/Counter: 5/i)).toBeInTheDocument();
});

it("should increment the counter", () => {
render(<Counter />);
fireEvent.click(screen.getByText(/Increment/i));
expect(screen.getByText(/Counter: 1/i)).toBeInTheDocument();
});

it("should decrement the counter", () => {
render(<Counter />);
fireEvent.click(screen.getByText(/Decrement/i));
expect(screen.getByText(/Counter: -1/i)).toBeInTheDocument();
});
});

3. React Hooks for Local Storage​

We can create a custom hook to encapsulate the logic for interacting with local storage.

Example: Custom Hook​

// hooks/useLocalStorage.ts
import { useState } from "react";

function useLocalStorage<T>(key: string, initialValue: T) {
const [storedValue, setStoredValue] = useState<T>(() => {
try {
const item = localStorage.getItem(key);
return item ? JSON.parse(item) : initialValue;
} catch (error) {
console.error("Error reading localStorage key", key, error);
return initialValue;
}
});

const setValue = (value: T) => {
try {
setStoredValue(value);
localStorage.setItem(key, JSON.stringify(value));
} catch (error) {
console.error("Error setting localStorage key", key, error);
}
};

return [storedValue, setValue] as const;
}

export default useLocalStorage;

Usage Example​

// components/ThemeSwitcher.tsx
import React from "react";
import useLocalStorage from "../hooks/useLocalStorage";

const ThemeSwitcher: React.FC = () => {
const [theme, setTheme] = useLocalStorage<string>("theme", "light");

return (
<div className={`theme-${theme}`}>
<h1>Current Theme: {theme}</h1>
<button onClick={() => setTheme("light")}>Light</button>
<button onClick={() => setTheme("dark")}>Dark</button>
</div>
);
};

export default ThemeSwitcher;

4. Best Practices for Local Storage in React​

  1. Avoid Storing Sensitive Data: Local storage is not secure; avoid storing sensitive information like passwords or tokens.
  2. Validate Stored Data: Always validate and sanitize data retrieved from local storage.
  3. Error Handling: Wrap local storage operations in try-catch blocks to handle potential errors.
  4. Abstract Logic: Use utilities or hooks to encapsulate local storage interactions for reusability and type safety.
  5. Synchronize Across Tabs: Use the storage event to synchronize data across multiple tabs.

Example: Tab Synchronization​

// hooks/useSyncLocalStorage.ts
import { useEffect } from "react";

export function useSyncLocalStorage(key: string, callback: (newValue: string | null) => void) {
useEffect(() => {
const handler = (event: StorageEvent) => {
if (event.key === key) {
callback(event.newValue);
}
};

window.addEventListener("storage", handler);
return () => window.removeEventListener("storage", handler);
}, [key, callback]);
}

Deep Dive into ReactJS Hooks

· 5 min read

React Hooks, introduced in React 16.8, revolutionized functional components by adding state and lifecycle capabilities, enabling a simpler, more powerful way to manage logic within components. This guide will provide a deep dive into React’s core hooks, detailing how they work, the scenarios in which they shine, and best practices for leveraging them effectively.


Introduction to Hooks​

Hooks allow you to use state, lifecycle methods, and other React features without writing a class. They provide a way to reuse logic across components in a more composable and understandable way, helping developers avoid the complex nesting and lifecycle clutter often encountered with higher-order components and render props.

Why Hooks?​

  • Simplify state and side effect management in functional components.
  • Enhance code reuse with custom hooks.
  • Make components cleaner and more readable.

Basic Hooks​

React offers several core hooks that cover common needs such as state, effects, and references.

useState: Managing State in Functional Components​

useState is used to declare state within functional components. It returns an array with two values: the current state and a function to update it.

Syntax:​

const [state, setState] = useState(initialValue);

Practical Scenario: Toggling a Modal​

import React, { useState } from 'react';

function ModalToggle() {
const [isOpen, setIsOpen] = useState(false);

const toggleModal = () => setIsOpen(!isOpen);

return (
<div>
<button onClick={toggleModal}>Toggle Modal</button>
{isOpen && <Modal />}
</div>
);
}

useEffect: Side Effects and Lifecycle Events​

useEffect manages side effects (such as data fetching, subscriptions, or DOM manipulation). It runs after the component renders and accepts a dependency array that controls when it should re-run.

Syntax:​

useEffect(() => {
// effect code here
return () => {
// cleanup code here
};
}, [dependencies]);

Practical Scenario: Fetching Data on Component Load​

import React, { useState, useEffect } from 'react';

function UserList() {
const [users, setUsers] = useState([]);

useEffect(() => {
async function fetchUsers() {
const response = await fetch('/api/users');
const data = await response.json();
setUsers(data);
}

fetchUsers();
}, []); // Empty array means this runs only once, similar to componentDidMount

return (
<ul>
{users.map(user => <li key={user.id}>{user.name}</li>)}
</ul>
);
}

Tips:​

  • Cleanup: Return a cleanup function to avoid memory leaks (e.g., unsubscribing from a WebSocket).
  • Dependencies: Use the dependency array to specify when the effect should re-run.

Advanced Hooks​

Beyond the basic hooks, React provides additional hooks for managing more specialized situations.

useContext: Accessing Context Values​

useContext allows components to access and consume context values directly without needing to wrap them in a Consumer.

Syntax:​

const contextValue = useContext(MyContext);

Practical Scenario: Global Theme Management​

import React, { useContext } from 'react';
import { ThemeContext } from './ThemeContext';

function ThemedButton() {
const theme = useContext(ThemeContext);

return (
<button style={{ background: theme.background, color: theme.foreground }}>
I am styled by theme context!
</button>
);
}

Additional Hooks​

useReducer: Managing Complex State​

useReducer is an alternative to useState for managing complex state transitions, typically in situations where you would use Redux or other state management libraries.

Syntax:​

const [state, dispatch] = useReducer(reducer, initialState);

Practical Scenario: A Counter with Increment/Decrement​

import React, { useReducer } from 'react';

function reducer(state, action) {
switch (action.type) {
case 'increment':
return { count: state.count + 1 };
case 'decrement':
return { count: state.count - 1 };
default:
throw new Error();
}
}

function Counter() {
const [state, dispatch] = useReducer(reducer, { count: 0 });

return (
<div>
<button onClick={() => dispatch({ type: 'decrement' })}>-</button>
<span>{state.count}</span>
<button onClick={() => dispatch({ type: 'increment' })}>+</button>
</div>
);
}

useRef: Persistent Mutable References​

useRef creates a persistent reference that doesn’t cause re-renders, making it ideal for storing mutable values like DOM elements.

Syntax:​

const ref = useRef(initialValue);

Practical Scenario: Focusing an Input Element​

import React, { useRef } from 'react';

function TextInputWithFocus() {
const inputRef = useRef(null);

const handleClick = () => {
inputRef.current.focus();
};

return (
<div>
<input ref={inputRef} type="text" />
<button onClick={handleClick}>Focus the input</button>
</div>
);
}

useMemo: Memoizing Expensive Calculations​

useMemo optimizes performance by memoizing expensive calculations, only recomputing them when dependencies change.

Syntax:​

const memoizedValue = useMemo(() => computeExpensiveValue(a, b), [a, b]);

Practical Scenario: Filtering a Large List​

import React, { useState, useMemo } from 'react';

function FilterableList({ items }) {
const [query, setQuery] = useState('');

const filteredItems = useMemo(() => {
return items.filter(item => item.toLowerCase().includes(query.toLowerCase()));
}, [items, query]);

return (
<div>
<input value={query} onChange={e => setQuery(e.target.value)} placeholder="Filter items" />
<ul>
{filteredItems.map(item => <li key={item}>{item}</li>)}
</ul>
</div>
);
}

Custom Hooks​

Custom hooks are functions that allow you to extract and reuse stateful logic across components. They enable you to abstract complex logic in a cleaner, more maintainable way.

Practical Scenario: A useFetch Hook for Data Fetching​

import { useState, useEffect } from 'react';

function useFetch(url) {
const [data, setData] = useState(null);
const [loading, setLoading] = useState(true);

useEffect(() => {
async function fetchData() {
setLoading(true);
const response = await fetch(url);
const result = await response.json();
setData(result);
setLoading(false);
}

fetchData();
}, [url]);

return { data, loading };
}

export default useFetch;

Using useFetch in a Component​

import React from 'react';
import useFetch from './useFetch';

function Users() {
const { data, loading } = useFetch('/api/users');

if (loading) return <p>Loading...</p>;
return (
<ul>
{data.map(user => <li key={user.id}>{user.name}</li>)}
</ul>
);
}

Error Handling and Best Practices​

Error Handling in Hooks​

  1. Wrap async operations with a try-catch to handle errors gracefully.
  2. Global error boundaries for handling errors in effects or custom hooks (using libraries like react-error-boundary).

Best Practices​

  • Dependency Arrays: Always add dependencies in useEffect and useMemo dependency arrays to avoid stale closures.
  • Avoid Side Effects in Render: Avoid triggering side effects inside render methods to prevent unexpected behavior.

Conclusion​

React Hooks provide powerful, flexible solutions for managing state, effects, and other logic in functional components. Mastering these core hooks, along with custom hooks, enables you to write cleaner, more maintainable code.

With practical examples, error handling tips, and performance optimizations, you’re now equipped to build complex React applications that harness the full power of hooks.

Best Practices for Local Storage Handling in ReactJS

· 11 min read

Introduction​

Local storage is a powerful tool for persisting data in web applications. In ReactJS, local storage can be used to store user preferences, shopping cart items, and other data that needs to persist across sessions. This article will deep dive into best practices for handling local storage in ReactJS with TypeScript, focusing on functional components and the latest version of the tech stack. We will provide real production scenarios with TypeScript code samples, design diagrams, project folder structures, unit test cases, and commands to initialize and add dependencies using Yarn.

1. Local Storage Basics​

1.1 Project Setup​

To get started, we need to set up a React project with TypeScript and add the necessary dependencies using Yarn.

Yarn Commands​

# Create a React project with TypeScript template
yarn create react-app local-storage-app --template typescript

# Navigate into the project directory
cd local-storage-app

# Add any additional dependencies if needed (e.g., testing libraries)
yarn add @testing-library/react @testing-library/jest-dom

1.2 Project Folder Structure​

project-root/
├── src/
│ ├── components/
│ │ ├── UserPreferencesComponent.tsx
│ │ ├── ShoppingCartComponent.tsx
│ ├── hooks/
│ │ └── useLocalStorage.ts
│ ├── utils/
│ │ ├── localStorageUtils.ts
│ ├── App.tsx
│ ├── index.tsx
├── tests/
│ ├── components/
│ │ ├── UserPreferencesComponent.test.tsx
│ │ ├── ShoppingCartComponent.test.tsx
│ ├── utils/
│ │ ├── localStorageUtils.test.ts
├── package.json
├── tsconfig.json
├── yarn.lock

2.1 What is Local Storage?​

Local storage is a web storage API that allows you to store key-value pairs in a web browser. It is a synchronous, persistent storage that allows developers to store data client-side. The data stored in local storage persists even after the browser is closed and reopened. Leveraging local storage in React applications with TypeScript can help maintain user state, save preferences, and reduce API calls.

2.2 Local Storage vs Session Storage​

  • Local Storage: Data persists across browser sessions.
  • Session Storage: Data is cleared when the browser session ends (e.g., when the browser is closed).

3. Local Storage Basics​

3.1 Basic API​

Local storage provides a simple API: setItem(), getItem(), and removeItem(). Data stored in local storage is accessible across browser sessions.

// Setting an item
localStorage.setItem('key', 'value');

// Getting an item
const value = localStorage.getItem('key');

// Removing an item
localStorage.removeItem('key');

3.2 Type Safety​

Use TypeScript to ensure type safety when interacting with local storage. Create utility functions or custom hooks in TypeScript to handle type-safe operations.

interface UserPreferences {
theme: string;
notifications: boolean;
}

const setUserPreferences = (preferences: UserPreferences): void => {
setLocalStorageItem('userPreferences', preferences);
};

const getUserPreferences = (): UserPreferences | null => {
return getLocalStorageItem<UserPreferences>('userPreferences');
};

3.3 Error Handling​

Handle errors gracefully when interacting with local storage. Wrap local storage operations in try-catch blocks to handle any potential errors gracefully.

export const setLocalStorageItem = <T>(key: string, value: T): void => {
try {
localStorage.setItem(key, JSON.stringify(value));
} catch (error) {
console.error(`Error setting local storage item: ${error}`);
}
};

export const getLocalStorageItem = <T>(key: string): T | null => {
try {
const item = localStorage.getItem(key);
return item ? JSON.parse(item) : null;
} catch (error) {
console.error(`Error getting local storage item: ${error}`);
return null;
}
};

3.4 Data Serialization​

Ensure that data is properly serialized and deserialized when storing and retrieving from local storage. Always serialize complex data structures as strings using JSON.stringify().

export const setLocalStorageItem = <T>(key: string, value: T): void => {
localStorage.setItem(key, JSON.stringify(value));
};

export const getLocalStorageItem = <T>(key: string): T | null => {
const item = localStorage.getItem(key);
return item ? JSON.parse(item) : null;
};

3.5 Expiration Handling​

Implement expiration handling for local storage items to ensure that stale data is not used.

interface ExpirableItem<T> {
value: T;
expiration: number;
}

export const setExpirableLocalStorageItem = <T>(key: string, value: T, expirationInMinutes: number): void => {
const expiration = new Date().getTime() + expirationInMinutes * 60 * 1000;
const item: ExpirableItem<T> = { value, expiration };
setLocalStorageItem(key, item);
};

export const getExpirableLocalStorageItem = <T>(key: string): T | null => {
const item = getLocalStorageItem<ExpirableItem<T>>(key);
if (item && item.expiration > new Date().getTime()) {
return item.value;
}
removeLocalStorageItem(key);
return null;
};

4. Best Practices​

4.1 Encapsulating Local Storage Access​

Encapsulate local storage logic inside React hooks or service functions to keep components clean. Encapsulate local storage access within utility functions to avoid code duplication and improve maintainability.

export const setLocalStorageItem = <T>(key: string, value: T): void => {
localStorage.setItem(key, JSON.stringify(value));
};

export const getLocalStorageItem = <T>(key: string): T | null => {
const item = localStorage.getItem(key);
return item ? JSON.parse(item) : null;
};

export const removeLocalStorageItem = (key: string): void => {
localStorage.removeItem(key);
};

5 More Scenarios​

5.1 Scenario 1: User Preferences​

import React, { useState, useEffect } from 'react';
import { setUserPreferences, getUserPreferences } from './localStorageUtils';

interface UserPreferences {
theme: string;
notifications: boolean;
}

const UserPreferencesComponent: React.FC = () => {
const [preferences, setPreferences] = useState<UserPreferences | null>(null);

useEffect(() => {
const storedPreferences = getUserPreferences();
if (storedPreferences) {
setPreferences(storedPreferences);
}
}, []);

const handleThemeChange = (theme: string) => {
const updatedPreferences = { ...preferences, theme } as UserPreferences;
setUserPreferences(updatedPreferences);
setPreferences(updatedPreferences);
};

const handleNotificationsChange = (notifications: boolean) => {
const updatedPreferences = { ...preferences, notifications } as UserPreferences;
setUserPreferences(updatedPreferences);
setPreferences(updatedPreferences);
};

return (
<div>
<h1>User Preferences</h1>
<div>
<label>Theme:</label>
<select value={preferences?.theme} onChange={(e) => handleThemeChange(e.target.value)}>
<option value="light">Light</option>
<option value="dark">Dark</option>
</select>
</div>
<div>
<label>Notifications:</label>
<input
type="checkbox"
checked={preferences?.notifications}
onChange={(e) => handleNotificationsChange(e.target.checked)}
/>
</div>
</div>
);
};

export default UserPreferencesComponent;

5.2 Scenario 2: Shopping Cart​

import React, { useState, useEffect } from 'react';
import { setLocalStorageItem, getLocalStorageItem, removeLocalStorageItem } from './localStorageUtils';

interface CartItem {
id: number;
name: string;
price: number;
}

const ShoppingCartComponent: React.FC = () => {
const [cartItems, setCartItems] = useState<CartItem[]>([]);

useEffect(() => {
const storedCartItems = getLocalStorageItem<CartItem[]>('cartItems');
if (storedCartItems) {
setCartItems(storedCartItems);
}
}, []);

const addToCart = (item: CartItem) => {
const updatedCartItems = [...cartItems, item];
setLocalStorageItem('cartItems', updatedCartItems);
setCartItems(updatedCartItems);
};

const removeFromCart = (id: number) => {
const updatedCartItems = cartItems.filter((item) => item.id !== id);
setLocalStorageItem('cartItems', updatedCartItems);
setCartItems(updatedCartItems);
};

return (
<div>
<h1>Shopping Cart</h1>
<ul>
{cartItems.map((item) => (
<li key={item.id}>
{item.name} - ${item.price}
<button onClick={() => removeFromCart(item.id)}>Remove</button>
</li>
))}
</ul>
<button onClick={() => addToCart({ id: Date.now(), name: 'New Item', price: 10 })}>Add Item</button>
</div>
);
};

export default ShoppingCartComponent;

6-Design Diagrams​

6.1 Component Diagram​

+-------------------+ +-------------------+ +-------------------+
| App | | UserPreferences | | ShoppingCart |
+-------------------+ +-------------------+ +-------------------+
| | |
| (1) User Preferences | |
|-------------------------->| |
| | (2) Shopping Cart |
| |-------------------------->|
| | |
| (3) Local Storage | |
|<--------------------------| |
| | |
+-------------------+ +-------------------+ +-------------------+
| App | | UserPreferences | | ShoppingCart |
+-------------------+ +-------------------+ +-------------------+

6.2 Sequence Diagram​

+-------------------+ +-------------------+ +-------------------+
| User | | UserPreferences | | Local Storage |
+-------------------+ +-------------------+ +-------------------+
| | |
| (1) Change Theme | |
|-------------------------->| |
| | (2) Update Preferences |
| |-------------------------->|
| | |
| (3) Store Preferences | |
|<--------------------------| |
| | |
+-------------------+ +-------------------+ +-------------------+
| User | | UserPreferences | | Local Storage |
+-------------------+ +-------------------+ +-------------------+

7. Unit Test Cases​

7.1 UserPreferencesComponent.test.tsx​

import React from 'react';
import { render, fireEvent } from '@testing-library/react';
import UserPreferencesComponent from '../../src/components/UserPreferencesComponent';
import { setUserPreferences, getUserPreferences } from '../../src/utils/localStorageUtils';

jest.mock('../../src/utils/localStorageUtils', () => ({
setUserPreferences: jest.fn(),
getUserPreferences: jest.fn(),
}));

test('UserPreferencesComponent updates and retrieves preferences', () => {
(getUserPreferences as jest.Mock).mockReturnValue({ theme: 'light', notifications: true });

const { getByLabelText, getByText } = render(<UserPreferencesComponent />);

fireEvent.change(getByLabelText('Theme:'), { target: { value: 'dark' } });
expect(setUserPreferences).toHaveBeenCalledWith({ theme: 'dark', notifications: true });

fireEvent.click(getByLabelText('Notifications:'));
expect(setUserPreferences).toHaveBeenCalledWith({ theme: 'dark', notifications: false });
});

7.2 ShoppingCartComponent.test.tsx​

import React from 'react';
import { render, fireEvent } from '@testing-library/react';
import ShoppingCartComponent from '../../src/components/ShoppingCartComponent';
import { setLocalStorageItem, getLocalStorageItem, removeLocalStorageItem } from '../../src/utils/localStorageUtils';

jest.mock('../../src/utils/localStorageUtils', () => ({
setLocalStorageItem: jest.fn(),
getLocalStorageItem: jest.fn(),
removeLocalStorageItem: jest.fn(),
}));

test('ShoppingCartComponent adds and removes items', () => {
(getLocalStorageItem as jest.Mock).mockReturnValue([{ id: 1, name: 'Item 1', price: 10 }]);

const { getByText } = render(<ShoppingCartComponent />);

fireEvent.click(getByText('Add Item'));
expect(setLocalStorageItem).toHaveBeenCalled();

fireEvent.click(getByText('Remove'));
expect(removeLocalStorageItem).toHaveBeenCalled();
});

8. Commands to Init and Add Dependencies​

# Initialize a new React project with TypeScript
yarn create react-app my-app --template typescript

# Navigate to the project directory
cd my-app

# Add Testing Library
yarn add --dev @testing-library/react @testing-library/jest-dom

9. Conclusion​

Local storage is a powerful tool for persisting data in web applications. This article provided a deep dive into best practices for handling local storage in ReactJS with TypeScript, focusing on functional components and the latest version of the tech stack. We explored real production scenarios with TypeScript code samples, design diagrams, project folder structures, unit test cases, and commands to initialize and add dependencies using Yarn.

By understanding and applying these best practices, developers can effectively manage local storage in their ReactJS applications, ensuring they remain scalable, maintainable, and performant.

10. More Samples​

10.1 Project Setup​

To get started, we need to set up a React project with TypeScript and add the necessary dependencies using Yarn.

Yarn Commands

# Create a React project with TypeScript template
yarn create react-app local-storage-app --template typescript

# Navigate into the project directory
cd local-storage-app

# Add any additional dependencies if needed (e.g., testing libraries)
yarn add @testing-library/react @testing-library/jest-dom

Project Folder Structure

local-storage-app/
├── src/
│ ├── components/
│ │ ├── LocalStorageHandler.tsx
│ │ └── UserPreferences.tsx
│ ├── hooks/
│ │ └── useLocalStorage.ts
│ ├── tests/
│ │ └── LocalStorageHandler.test.tsx
│ ├── App.tsx
│ └── index.tsx
├── public/
└── package.json

useLocalStorage.ts

// src/hooks/useLocalStorage.ts
import { useState } from 'react';

/**
* Custom hook for local storage handling
* @param key - The key under which data is stored in local storage
* @param initialValue - The initial value to use if no value is stored
* @returns A tuple containing the stored value and a function to set the value
*/
function useLocalStorage<T>(key: string, initialValue: T) {
const [storedValue, setStoredValue] = useState<T>(() => {
try {
const item = localStorage.getItem(key);
return item ? (JSON.parse(item) as T) : initialValue;
} catch (error) {
console.error('Error reading from local storage', error);
return initialValue;
}
});

/**
* Sets a new value in local storage
* @param value - The new value to store
*/
const setValue = (value: T) => {
try {
setStoredValue(value);
localStorage.setItem(key, JSON.stringify(value));
} catch (error) {
console.error('Error saving to local storage', error);
}
};

return [storedValue, setValue] as const;
}

export default useLocalStorage;

UserPreferences: Detailed Code for Using useLocalStorage in a Component

// src/components/UserPreferences.tsx
import React from 'react';
import useLocalStorage from '../hooks/useLocalStorage';

const UserPreferences: React.FC = () => {
// Using the custom hook to manage theme preference
const [theme, setTheme] = useLocalStorage<string>('theme', 'light');

// Toggles between 'light' and 'dark' themes
const toggleTheme = () => {
setTheme(theme === 'light' ? 'dark' : 'light');
};

return (
<div>
<h1>User Preferences</h1>
<p>Current theme: {theme}</p>
<button onClick={toggleTheme}>
Toggle Theme
</button>
</div>
);
};

export default UserPreferences;

Type-Safe Local Storage Utility Functions For larger applications, you might prefer having utility functions to handle local storage logic:

// src/utils/localStorageUtils.ts
export const getLocalStorageItem = <T>(key: string, defaultValue: T): T => {
try {
const storedItem = localStorage.getItem(key);
return storedItem ? (JSON.parse(storedItem) as T) : defaultValue;
} catch (error) {
console.error(`Error reading key "${key}" from local storage`, error);
return defaultValue;
}
};

export const setLocalStorageItem = <T>(key: string, value: T): void => {
try {
localStorage.setItem(key, JSON.stringify(value));
} catch (error) {
console.error(`Error setting key "${key}" in local storage`, error);
}
};

Detailed Unit Test Cases

// src/tests/useLocalStorage.test.tsx
import { renderHook, act } from '@testing-library/react';
import useLocalStorage from '../hooks/useLocalStorage';

describe('useLocalStorage hook', () => {
beforeEach(() => {
localStorage.clear();
});

it('should return initial value if no value is stored', () => {
const { result } = renderHook(() => useLocalStorage('testKey', 'initialValue'));
expect(result.current[0]).toBe('initialValue');
});

it('should save and retrieve a value', () => {
const { result } = renderHook(() => useLocalStorage('testKey', 'initialValue'));

act(() => {
result.current[1]('newValue');
});

expect(result.current[0]).toBe('newValue');
expect(localStorage.getItem('testKey')).toBe(JSON.stringify('newValue'));
});

it('should handle JSON parsing errors gracefully', () => {
localStorage.setItem('testKey', '{invalid JSON');
const { result } = renderHook(() => useLocalStorage('testKey', 'defaultValue'));
expect(result.current[0]).toBe('defaultValue');
});
});

Commands to Run Unit Tests

# Run unit tests with Yarn
yarn test

References​


This article provides a comprehensive guide to best practices for local storage handling in ReactJS with TypeScript, including detailed code samples, design diagrams, project folder structures, unit test cases, and commands to initialize and add dependencies using Yarn. By understanding and applying these practices, developers can leverage the full potential of local storage in their ReactJS projects.

Deep Dive into State Management in ReactJS

· 12 min read

Introduction​

State management is a critical aspect of building complex ReactJS applications. Proper state management ensures that your application remains scalable, maintainable, and performant. This article will deep dive into state management in ReactJS, focusing on functional components and the latest version of the tech stack. We will explore two main state management options: React Context API and Redux. Additionally, we will provide real production scenarios with TypeScript code samples, design diagrams, project folder structures, unit test cases, and commands to initialize and add dependencies using Yarn.

1. State Management Basics​

1.1 What is State?​

State in ReactJS is an object that represents the dynamic data of a component. It determines how the component renders and behaves. State can be local to a single component or shared across multiple components.

State management refers to managing data that changes over time, ensuring that the user interface updates appropriately. Choosing the right state management solution is crucial for developing scalable and maintainable applications. We'll explore useState and useReducer for local state management and Redux and Zustand for global state management, with detailed TypeScript examples.

1.2 Local State vs Global State​

  • Local State: State that is confined to a single component. It is managed using the useState hook.
  • Global State: State that is shared across multiple components. It is managed using state management libraries like React Context API or Redux.

2. Setting Up the Project​

Initialize the React Project with TypeScript​

Run the following commands to create a new React project with TypeScript and install the necessary dependencies:

# Create a new React app with TypeScript
yarn create react-app react-state-management --template typescript

# Navigate to the project folder
cd react-state-management

# Install Redux and Zustand for state management
yarn add redux react-redux @reduxjs/toolkit zustand

# Install TypeScript types for React and Redux
yarn add -D @types/react-redux

3. State Management Options​

3.1 React Context API​

The React Context API provides a way to pass data through the component tree without having to pass props down manually at every level. It is ideal for small to medium-sized applications.

import React, { createContext, useContext, useState } from 'react';

interface AuthContextType {
isAuthenticated: boolean;
login: () => void;
logout: () => void;
}

const AuthContext = createContext<AuthContextType>({
isAuthenticated: false,
login: () => {},
logout: () => {},
});

export const AuthProvider: React.FC = ({ children }) => {
const [isAuthenticated, setIsAuthenticated] = useState(false);

const login = () => setIsAuthenticated(true);
const logout = () => setIsAuthenticated(false);

return (
<AuthContext.Provider value={{ isAuthenticated, login, logout }}>
{children}
</AuthContext.Provider>
);
};

export const useAuth = () => useContext(AuthContext);

4. Local State Management with useState and useReducer​

4.1 Using useState​

useState is simple to use and ideal for managing local state.

TypeScript Code Sample:

import React, { useState } from 'react';

const Counter: React.FC = () => {
const [count, setCount] = useState<number>(0);

const increment = (): void => {
setCount(prevCount => prevCount + 1);
};

return (
<div>
<p>Current Count: {count}</p>
<button onClick={increment}>Increment</button>
</div>
);
};

export default Counter;

Best Practices:

  • Use useState for simple and straightforward state updates.
  • Type your state variables for better type safety.

4.2 Using useReducer​

useReducer is suitable for managing complex state logic.

TypeScript Code Sample:

import React, { useReducer } from 'react';

interface State {
count: number;
}

interface Action {
type: 'increment' | 'decrement';
}

const initialState: State = { count: 0 };

function reducer(state: State, action: Action): State {
switch (action.type) {
case 'increment':
return { count: state.count + 1 };
case 'decrement':
return { count: state.count - 1 };
default:
throw new Error('Unknown action type');
}
}

const Counter: React.FC = () => {
const [state, dispatch] = useReducer(reducer, initialState);

return (
<div>
<p>Current Count: {state.count}</p>
<button onClick={() => dispatch({ type: 'increment' })}>Increment</button>
<button onClick={() => dispatch({ type: 'decrement' })}>Decrement</button>
</div>
);
};

export default Counter;

Best Practices:

  • Use useReducer when state logic is complex or involves multiple sub-values.
  • Type both state and actions for enhanced type safety.

5. Global State Management with Redux​

Redux is a predictable state container for JavaScript apps. It is widely used for managing global state in large-scale applications.

5.1 Architecture of Redux​

Redux Architecture:

+------------+ +---------------+ +-------------+
| Action | --> | Reducer | --> | Store |
+------------+ +---------------+ +-------------+
^ |
| |
+------------------------------------------+

5.2 Redux state machine​

Here's a textual representation of a state machine diagram for a Redux state management flow:

+--------------------+ Action +----------------------+
| | (dispatch) | |
| React Component | -----------------> | Action |
| | | (type & payload) |
+--------------------+ +----------------------+
|
|
Dispatches action
|
v
+----------------+
| |
| Reducer |
| (pure function)|
+----------------+
|
Updates state |
based on action |
v
+----------------+
| |
| Store |
| (holds state) |
+----------------+
|
State subscription |
triggers re-render |
of component |
v
+--------------------+ Updated +----------------------+
| | State | |
| React Component | <----------------- | Store |
| | | |
+--------------------+ +----------------------+

Explanation:​

  • React Component: The UI component where user interactions or events trigger actions.
  • Action: An object describing what type of event occurred (e.g., { type: 'INCREMENT' }).
  • Reducer: A pure function that takes the current state and an action, then returns a new state.
  • Store: The central place where the state is kept, updated, and managed.
  • State Flow: Components subscribe to the store to get updates when the state changes, causing a re-render to reflect new data.

5.2 Redux Samples​

The Simple One

import { createStore } from 'redux';

interface State {
count: number;
}

interface Action {
type: string;
}

const initialState: State = { count: 0 };

const reducer = (state = initialState, action: Action) => {
switch (action.type) {
case 'INCREMENT':
return { count: state.count + 1 };
case 'DECREMENT':
return { count: state.count - 1 };
default:
return state;
}
};

const store = createStore(reducer);

export default store;

The Sample of Details actions.ts:

export const increment = () => ({ type: 'INCREMENT' } as const);
export const decrement = () => ({ type: 'DECREMENT' } as const);

export type ActionType = ReturnType<typeof increment | typeof decrement>;

reducers.ts:

import { ActionType } from './actions';

interface State {
count: number;
}

const initialState: State = { count: 0 };

export const counterReducer = (state: State = initialState, action: ActionType): State => {
switch (action.type) {
case 'INCREMENT':
return { count: state.count + 1 };
case 'DECREMENT':
return { count: state.count - 1 };
default:
return state;
}
};

store.ts:

import { createStore } from 'redux';
import { counterReducer } from './reducers';

const store = createStore(counterReducer);

export default store;

App.tsx:

import React from 'react';
import { Provider, useDispatch, useSelector } from 'react-redux';
import store from './state/redux/store';
import { increment, decrement } from './state/redux/actions';

interface RootState {
count: number;
}

const App: React.FC = () => {
const count = useSelector((state: RootState) => state.count);
const dispatch = useDispatch();

return (
<Provider store={store}>
<div>
<p>Current Count: {count}</p>
<button onClick={() => dispatch(increment())}>Increment</button>
<button onClick={() => dispatch(decrement())}>Decrement</button>
</div>
</Provider>
);
};

export default App;

6. Alternative Global State Management with Zustand​

Zustand offers a simple and flexible state management solution without the boilerplate of Redux.

6.1 Architecture of Zustand​

Simple Architecture:

Component --> useStore --> State & Actions

6.2 TypeScript Code Example​

useStore.ts:

import create from 'zustand';

interface StoreState {
count: number;
increment: () => void;
decrement: () => void;
}

const useStore = create<StoreState>(set => ({
count: 0,
increment: () => set(state => ({ count: state.count + 1 })),
decrement: () => set(state => ({ count: state.count - 1 }))
}));

export default useStore;

App.tsx:

import React from 'react';
import useStore from './state/zustand/useStore';

const App

: React.FC = () => {
const { count, increment, decrement } = useStore();

return (
<div>
<p>Current Count: {count}</p>
<button onClick={increment}>Increment</button>
<button onClick={decrement}>Decrement</button>
</div>
);
};

export default App;

More Scenarios​

7.1 Scenario 1: User Authentication​

React Context API​

import React from 'react';
import { AuthProvider, useAuth } from './AuthContext';

const LoginPage: React.FC = () => {
const { login } = useAuth();

return (
<div>
<h1>Login Page</h1>
<button onClick={login}>Login</button>
</div>
);
};

const Dashboard: React.FC = () => {
const { isAuthenticated, logout } = useAuth();

return (
<div>
<h1>Dashboard</h1>
{isAuthenticated ? (
<button onClick={logout}>Logout</button>
) : (
<p>Please login to access the dashboard.</p>
)}
</div>
);
};

const App: React.FC = () => (
<AuthProvider>
<LoginPage />
<Dashboard />
</AuthProvider>
);

export default App;

Redux​

import React from 'react';
import { Provider, useSelector, useDispatch } from 'react-redux';
import store from './store';

const LoginPage: React.FC = () => {
const dispatch = useDispatch();

const login = () => dispatch({ type: 'LOGIN' });

return (
<div>
<h1>Login Page</h1>
<button onClick={login}>Login</button>
</div>
);
};

const Dashboard: React.FC = () => {
const isAuthenticated = useSelector((state: any) => state.isAuthenticated);
const dispatch = useDispatch();

const logout = () => dispatch({ type: 'LOGOUT' });

return (
<div>
<h1>Dashboard</h1>
{isAuthenticated ? (
<button onClick={logout}>Logout</button>
) : (
<p>Please login to access the dashboard.</p>
)}
</div>
);
};

const App: React.FC = () => (
<Provider store={store}>
<LoginPage />
<Dashboard />
</Provider>
);

export default App;

7.2 Scenario 2: Shopping Cart​

React Context API​

import React, { createContext, useContext, useState } from 'react';

interface CartItem {
id: number;
name: string;
price: number;
}

interface CartContextType {
cartItems: CartItem[];
addToCart: (item: CartItem) => void;
removeFromCart: (id: number) => void;
}

const CartContext = createContext<CartContextType>({
cartItems: [],
addToCart: () => {},
removeFromCart: () => {},
});

export const CartProvider: React.FC = ({ children }) => {
const [cartItems, setCartItems] = useState<CartItem[]>([]);

const addToCart = (item: CartItem) => setCartItems([...cartItems, item]);
const removeFromCart = (id: number) =>
setCartItems(cartItems.filter((item) => item.id !== id));

return (
<CartContext.Provider value={{ cartItems, addToCart, removeFromCart }}>
{children}
</CartContext.Provider>
);
};

export const useCart = () => useContext(CartContext);

Redux​

import { createStore } from 'redux';

interface CartItem {
id: number;
name: string;
price: number;
}

interface State {
cartItems: CartItem[];
}

interface Action {
type: string;
payload?: any;
}

const initialState: State = { cartItems: [] };

const reducer = (state = initialState, action: Action) => {
switch (action.type) {
case 'ADD_TO_CART':
return { cartItems: [...state.cartItems, action.payload] };
case 'REMOVE_FROM_CART':
return {
cartItems: state.cartItems.filter((item) => item.id !== action.payload),
};
default:
return state;
}
};

const store = createStore(reducer);

export default store;

8. Some Diagrams​

+-------------------+ +-------------------+ +-------------------+
| App | | AuthProvider | | CartProvider |
+-------------------+ +-------------------+ +-------------------+
| | |
| (1) Auth Context | |
|--------------------------->| |
| | (2) Cart Context |
| |-------------------------->|
| | |
| (3) LoginPage | |
|<------------ --------------| |
| | |
| (4) Dashboard | |
|<------------ --------------| |
| | |
+-------------------+ +-------------------+ +-------------------+
| App | | AuthProvider | | CartProvider |
+-------------------+ +-------------------+ +-------------------+
+-------------------+ +-------------------+ +-------------------+
| User | | LoginPage | | AuthProvider |
+-------------------+ +-------------------+ +-------------------+
| | |
| (1) Click Login Button | |
|-------------------------->| |
| | (2) Dispatch Login Action |
| |-------------------------->|
| | |
| (3) Update Auth State | |
|<--------------------------| |
| | |
+-------------------+ +-------------------+ +-------------------+
| User | | LoginPage | | AuthProvider |
+-------------------+ +-------------------+ +-------------------+

Project Folder Structure​

project-root/
├── src/
│ ├── components/
│ │ ├── LoginPage.tsx
│ │ ├── Dashboard.tsx
│ ├── contexts/
│ │ ├── AuthContext.tsx
│ │ ├── CartContext.tsx
│ ├── redux/
│ │ ├── store.ts
│ ├── App.tsx
│ ├── index.tsx
├── tests/
│ ├── components/
│ │ ├── LoginPage.test.tsx
│ │ ├── Dashboard.test.tsx
│ ├── contexts/
│ │ ├── AuthContext.test.tsx
│ │ ├── CartContext.test.tsx
│ ├── redux/
│ │ ├── store.test.ts
├── package.json
├── tsconfig.json
├── yarn.lock

Unit Test Cases​

LoginPage.test.tsx​

import React from 'react';
import { render, fireEvent } from '@testing-library/react';
import { AuthProvider, useAuth } from '../../src/contexts/AuthContext';
import LoginPage from '../../src/components/LoginPage';

const TestComponent: React.FC = () => {
const { isAuthenticated } = useAuth();
return <div>{isAuthenticated ? 'Logged In' : 'Logged Out'}</div>;
};

test('LoginPage logs in user', () => {
const { getByText } = render(
<AuthProvider>
<LoginPage />
<TestComponent />
</AuthProvider>
);

fireEvent.click(getByText('Login'));
expect(getByText('Logged In')).toBeInTheDocument();
});

Dashboard.test.tsx​

import React from 'react';
import { render, fireEvent } from '@testing-library/react';
import { AuthProvider, useAuth } from '../../src/contexts/AuthContext';
import Dashboard from '../../src/components/Dashboard';

const TestComponent: React.FC = () => {
const { isAuthenticated } = useAuth();
return <div>{isAuthenticated ? 'Logged In' : 'Logged Out'}</div>;
};

test('Dashboard logs out user', () => {
const { getByText } = render(
<AuthProvider>
<Dashboard />
<TestComponent />
</AuthProvider>
);

fireEvent.click(getByText('Logout'));
expect(getByText('Logged Out')).toBeInTheDocument();
});

Commands to Init and Add Dependencies​

# Initialize a new React project with TypeScript
yarn create react-app my-app --template typescript

# Navigate to the project directory
cd my-app

# Add Redux and React-Redux
yarn add redux react-redux

# Add React Context API (no additional packages needed)

# Add Testing Library
yarn add --dev @testing-library/react @testing-library/jest-dom

Conclusion​

State management is a crucial aspect of building scalable and maintainable ReactJS applications. This article provided a deep dive into state management in ReactJS, focusing on functional components and the latest version of the tech stack. We explored two main state management options: React Context API and Redux. Additionally, we provided real production scenarios with TypeScript code samples, design diagrams, project folder structures, unit test cases, and commands to initialize and add dependencies using Yarn.

By understanding and applying these concepts, developers can effectively manage state in their ReactJS applications, ensuring they remain scalable, maintainable, and performant.

References​


This article provides a comprehensive guide to state management in ReactJS, including detailed code samples, design diagrams, project folder structures, unit test cases, and commands to initialize and add dependencies using Yarn. By understanding and applying these practices, developers can leverage the full potential of state management in their ReactJS projects.

Deep Dive into Kubectl port-forward

· 11 min read

Introduction​

The kubectl port-forward command is a powerful tool in Kubernetes that allows you to forward local ports to ports on a pod or service. This is particularly useful for accessing services running inside a Kubernetes cluster from your local machine, debugging applications, and accessing internal services without exposing them to the public internet. This article provides a deep dive into the kubectl port-forward command, covering various scenarios, best practices, and advanced usage, with a detailed example of accessing a PostgreSQL database deployed in a Kubernetes cluster.


How kubectl port-forward Works​

Overview​

The kubectl port-forward command establishes a TCP connection between your local machine and a specific pod or service in the Kubernetes cluster. This connection is bidirectional, meaning that traffic sent to the local port is forwarded to the pod or service, and vice versa.

Steps​

  1. Identify the Pod or Service: The command specifies the pod or service to which the port forwarding will be established.
  2. Establish a Connection: kubectl establishes a TCP connection to the specified pod or service in the Kubernetes cluster.
  3. Forward Traffic: kubectl listens on the specified local port and forwards incoming traffic to the corresponding port on the pod or service.

Example​

To forward local port 5432 to a PostgreSQL pod running on port 5432:

kubectl port-forward postgresql-0 5432:5432 -n database

In this example:

  • postgresql-0 is the name of the PostgreSQL pod.
  • 5432:5432 specifies that local port 5432 will be forwarded to port 5432 on the pod.
  • -n database specifies the namespace where the pod is located.

Basic Usage​

Forwarding a Single Port​

To forward a local port to a port on a pod, use the following command:

kubectl port-forward <pod-name> <local-port>:<pod-port> -n <namespace>

For example, to forward local port 8080 to port 80 on a pod named my-pod in the default namespace:

kubectl port-forward my-pod 8080:80

Forwarding a Single Port to a Service​

To forward a local port to a port on a service, use the following command:

kubectl port-forward service/<service-name> <local-port>:<service-port> -n <namespace>

For example, to forward local port 8080 to port 80 on a service named my-service in the default namespace:

kubectl port-forward service/my-service 8080:80

Advanced Usage​

Forwarding Multiple Ports​

To forward multiple ports, specify multiple <local-port>:<pod-port> pairs:

kubectl port-forward <pod-name> <local-port1>:<pod-port1> <local-port2>:<pod-port2> -n <namespace>

For example, to forward local ports 8080 and 9090 to ports 80 and 90 on a pod named my-pod:

kubectl port-forward my-pod 8080:80 9090:90

Forwarding to a Specific Pod​

If you have multiple pods with the same label, you can forward to a specific pod by specifying the pod name:

kubectl port-forward <pod-name> <local-port>:<pod-port> -n <namespace>

For example, to forward local port 8080 to port 80 on a specific pod named my-pod-1:

kubectl port-forward my-pod-1 8080:80

Forwarding to a Service​

To forward to a service, use the service/ prefix:

kubectl port-forward service/<service-name> <local-port>:<service-port> -n <namespace>

For example, to forward local port 8080 to port 80 on a service named my-service:

kubectl port-forward service/my-service 8080:80

Common Scenarios​

Accessing a PostgreSQL Database​

To access a PostgreSQL database running inside a Kubernetes cluster, follow these steps:

  1. Identify the PostgreSQL Pod: First, identify the name of the PostgreSQL pod. You can list all pods in the namespace where PostgreSQL is deployed:

    kubectl get pods -n <namespace>

    For example, if the PostgreSQL pod is named postgresql-0 in the database namespace:

    kubectl get pods -n database
  2. Port Forward to the PostgreSQL Pod: Forward the PostgreSQL port (default is 5432) to a local port:

    kubectl port-forward <pod-name> <local-port>:5432 -n <namespace>

    For example, to forward local port 5432 to the PostgreSQL pod:

    kubectl port-forward postgresql-0 5432:5432 -n database
  3. Connect to the PostgreSQL Database: Use a PostgreSQL client to connect to the database using the local port. The JDBC URL to connect to the PostgreSQL database from your local machine would be:

    jdbc:postgresql://localhost:<local-port>/<database-name>

    For example, if the database name is mydb, the JDBC URL would be:

    jdbc:postgresql://localhost:5432/mydb

    You can also specify the username and password in the JDBC URL:

    jdbc:postgresql://localhost:5432/mydb?user=<username>&password=<password>

    Alternatively, you can use a PostgreSQL client like psql to connect:

    psql -h localhost -p 5432 -U <username> -d <database-name>

    For example:

    psql -h localhost -p 5432 -U postgres -d mydb
  4. Underlying Mechanism

The kubectl port-forward command sets up a proxy that listens on the specified local port (e.g., 5432) and forwards incoming traffic to the corresponding port on the PostgreSQL pod in the Kubernetes cluster. This proxy effectively creates a tunnel between your local machine and the Kubernetes cluster, allowing you to use localhost to connect to the database.

Example

When you run the following command:

kubectl port-forward postgresql-0 5432:5432 -n database

kubectl sets up a proxy that listens on localhost:5432 and forwards incoming traffic to postgresql-0:5432 in the database namespace.


Best Practices​

Security Considerations​

  • Limit Access: Only forward ports when necessary and limit access to the forwarded ports.
  • Use Namespaces: Use namespaces to isolate resources and limit the scope of port forwarding.
  • Monitor Access: Monitor access to forwarded ports and review logs for suspicious activity.

Resource Management​

  • Avoid Overloading: Avoid forwarding too many ports simultaneously to prevent overloading the local machine.
  • Use Ephemeral Ports: Use ephemeral ports (e.g., 49152-65535) for local forwarding to avoid conflicts with other services.

Monitoring and Logging​

  • Enable Logging: Enable logging for port forwarding to track access and diagnose issues.
  • Monitor Performance: Monitor the performance of the local machine and the Kubernetes cluster to ensure port forwarding does not impact performance.

Lifecycle of kubectl port-forward​

Running as a Process​

When you run the kubectl port-forward command, kubectl starts a new process on your local machine. This process listens on the specified local port and forwards traffic to the corresponding port on the pod or service in the Kubernetes cluster.

Example​

When you run the following command:

kubectl port-forward postgresql-0 5432:5432 -n database

kubectl starts a new process that listens on localhost:5432 and forwards incoming traffic to postgresql-0:5432 in the database namespace.

Termination​

The kubectl port-forward process runs until you manually terminate it. You can terminate the process by pressing Ctrl+C in the terminal where the command is running.


Temporary Nature of Port Forwarding​

Temporary Connection​

The port forwarding established by kubectl port-forward is temporary. It only lasts as long as the kubectl port-forward process is running. Once you terminate the process (e.g., by pressing Ctrl+C), the port forwarding stops, and you can no longer access the pod or service via the forwarded port.

Impact of Reboot​

If you reboot your local machine, the kubectl port-forward process will be terminated, and the port forwarding will be lost. You will need to rerun the kubectl port-forward command after rebooting to reestablish the connection.


Persisting Port Forwarding​

Using Systemd Service​

To persist port forwarding across reboots, you can create a systemd service that runs the kubectl port-forward command on startup.

Step-by-Step Guide​

  1. Create a Script: Create a script that runs the kubectl port-forward command.

    #!/bin/bash
    kubectl port-forward postgresql-0 5432:5432 -n database

    Save this script as port-forward.sh and make it executable:

    chmod +x port-forward.sh
  2. Create a Systemd Service: Create a systemd service file to run the script on startup.

    [Unit]
    Description=Kubernetes Port Forwarding for PostgreSQL

    [Service]
    ExecStart=/path/to/port-forward.sh
    Restart=always

    [Install]
    WantedBy=multi-user.target

    Save this file as /etc/systemd/system/kubectl-port-forward.service.

  3. Enable and Start the Service: Enable and start the systemd service.

    sudo systemctl enable kubectl-port-forward.service
    sudo systemctl start kubectl-port-forward.service

Using Cron Job​

Another approach is to use a cron job to run the kubectl port-forward command on startup.

Step-by-Step Guide​

  1. Create a Script: Create a script that runs the kubectl port-forward command.

    #!/bin/bash
    kubectl port-forward postgresql-0 5432:5432 -n database

    Save this script as port-forward.sh and make it executable:

    chmod +x port-forward.sh
  2. Create a Cron Job: Create a cron job to run the script on startup.

    @reboot /path/to/port-forward.sh

    Add this line to your crontab by running:

    crontab -e

Security Considerations​

Limiting Access​

  • Only Forward Ports When Necessary: Only forward ports when necessary and limit access to the forwarded ports.
  • Use Namespaces: Use namespaces to isolate resources and limit the scope of port forwarding.
  • Monitor Access: Monitor access to forwarded ports and review logs for suspicious activity.

Resource Management​

  • Avoid Overloading: Avoid forwarding too many ports simultaneously to prevent overloading the local machine.
  • Use Ephemeral Ports: Use ephemeral ports (e.g., 49152-65535) for local forwarding to avoid conflicts with other services.

Monitoring and Logging​

  • Enable Logging: Enable logging for port forwarding to track access and diagnose issues.
  • Monitor Performance: Monitor the performance of the local machine and the Kubernetes cluster to ensure port forwarding does not impact performance.

Conclusion​

The kubectl port-forward command is a versatile and powerful tool for accessing services running inside a Kubernetes cluster from your local machine. By mastering the various options and scenarios covered in this article, you can effectively use kubectl port-forward for debugging applications, accessing databases, and accessing internal services. The detailed example of accessing a PostgreSQL database deployed in a Kubernetes cluster demonstrates how to use kubectl port-forward to connect to a database from your local machine.

The port forwarding established by kubectl port-forward is temporary and only lasts as long as the kubectl port-forward process is running. If you reboot your local machine, the port forwarding will be lost, and you will need to rerun the command to reestablish the connection.

To persist port forwarding across reboots, you can create a systemd service or a cron job to run the kubectl port-forward command on startup.


References​

  1. kubectl port-forward Documentation
  2. Kubernetes Best Practices
  3. Kubernetes Security Best Practices
  4. PostgreSQL JDBC Driver Documentation
  5. Systemd Documentation
  6. Cron Documentation

By following these best practices, you can leverage kubectl port-forward to its full potential and efficiently manage your Kubernetes cluster access and debugging tasks.