跳到主要内容

15 篇博文 含有标签「redis」

查看所有标签

Distributed Counter System with Redis - II

· 阅读需 28 分钟
  1. Executive Summary
  2. System Overview
  3. Architecture Design
  4. Data Model
  5. API Design
  6. Implementation Details
  7. Performance Optimization
  8. High Availability & Disaster Recovery
  9. Security Considerations
  10. Monitoring & Observability
  11. Deployment Strategy
  12. Cost Analysis
  13. Appendices

1. Executive Summary​

1.1 Purpose​

This document outlines the design for a distributed counter system using Redis as the centralized data store. The system provides atomic, consistent, and high-performance counter operations across multiple application instances.

1.2 Business Requirements​

  • Accuracy: Counters must be accurate with zero data loss
  • Performance: Support 100K+ operations per second
  • Scalability: Horizontal scaling to handle growing traffic
  • Availability: 99.99% uptime SLA
  • Real-time: Sub-millisecond latency for counter operations

1.3 Use Cases​

  1. Page view counters (website analytics)
  2. API rate limiting (throttling requests)
  3. Inventory management (product stock tracking)
  4. Social media metrics (likes, shares, followers)
  5. Event ticketing (available seats tracking)
  6. Gaming leaderboards (player scores)
  7. IoT device metrics (sensor data aggregation)

1.4 Key Decisions​

DecisionChoiceRationale
Data StoreRedisIn-memory speed, atomic operations, built-in data structures
Deployment ModelRedis ClusterHigh availability, horizontal scalability
PersistenceRDB + AOFBalance between performance and durability
Client LibraryLettuce (Java) / ioredis (Node.js)Production-ready, connection pooling, cluster support
Caching StrategyWrite-through with local cacheReduce Redis load while maintaining consistency

2. System Overview​

2.1 High-Level Architecture​

┌─────────────────────────────────────────────────────────────────────┐
│ Client Layer (Users/Apps) │
└───────────────────────┬─────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Load Balancer (HAProxy/NGINX) │
│ • Health checks │
│ • SSL termination │
│ • Request routing │
└───────────────────────┬─────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Application │ │ Application │ │ Application │
│ Server 1 │ │ Server 2 │ │ Server N │
│ │ │ │ │ │
│ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
│ │ Local │ │ │ │ Local │ │ │ │ Local │ │
│ │ Cache │ │ │ │ Cache │ │ │ │ Cache │ │
│ │(Optional)│ │ │ │(Optional)│ │ │ │(Optional)│ │
│ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
│ │ │ │ │ │
│ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
│ │ Counter │ │ │ │ Counter │ │ │ │ Counter │ │
│ │ Client │ │ │ │ Client │ │ │ │ Client │ │
│ └─────┬────┘ │ │ └─────┬────┘ │ │ └─────┬────┘ │
└───────┼──────┘ └───────┼──────┘ └───────┼──────┘
│ │ │
└────────────────┼────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Redis Sentinel Cluster │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Sentinel Nodes (Monitoring) │ │
│ │ [Sentinel 1] [Sentinel 2] [Sentinel 3] │ │
│ │ • Health monitoring • Automatic failover │ │
│ └──────────────────┬───────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────────┴───────────────────────────────────────────┐ │
│ │ Redis Master-Replica Set │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Master │──────▶│ Replica 1 │ │ Replica 2 │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ Shard 1 │ │ Shard 1 │ │ Shard 1 │ │ │
│ │ │ Slots: │ │ (Read-only)│ │ (Read-only)│ │ │
│ │ │ 0-5460 │ │ │ │ │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Master │──────▶│ Replica 1 │ │ Replica 2 │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ Shard 2 │ │ Shard 2 │ │ Shard 2 │ │ │
│ │ │ Slots: │ │ (Read-only)│ │ (Read-only)│ │ │
│ │ │ 5461-10922 │ │ │ │ │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Master │──────▶│ Replica 1 │ │ Replica 2 │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ Shard 3 │ │ Shard 3 │ │ Shard 3 │ │ │
│ │ │ Slots: │ │ (Read-only)│ │ (Read-only)│ │ │
│ │ │ 10923-16383│ │ │ │ │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ └────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Persistence Layer │
│ • RDB Snapshots (every 5 minutes) │
│ • AOF Log (append-only file for durability) │
│ • Backup to S3/Cloud Storage (hourly) │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Monitoring & Alerting │
│ • Prometheus (metrics collection) │
│ • Grafana (visualization) │
│ • PagerDuty (alerting) │
│ • ELK Stack (logging) │
└─────────────────────────────────────────────────────────────────────┘

2.2 System Components​

2.2.1 Application Layer​

  • Counter Client Library: Handles connection pooling, retries, circuit breaking
  • Local Cache (Optional): Reduces Redis load for read-heavy workloads
  • Business Logic: Application-specific counter operations

2.2.2 Redis Layer​

  • Redis Cluster: Distributed key-value store with automatic sharding
  • Sentinel: Monitoring and automatic failover
  • Persistence: RDB snapshots + AOF logging

2.2.3 Infrastructure Layer​

  • Load Balancer: Distributes traffic across application servers
  • Monitoring: Prometheus, Grafana, ELK stack
  • Backup: Automated backups to cloud storage

2.3 Data Flow Diagrams​

2.3.1 Increment Counter Flow​

┌──────────┐
│ Client │
└────┬─────┘
│
│ 1. POST /counter/increment
│ { counter: "page_views", value: 1 }
▼
┌─────────────────┐
│ App Server │
│ │
│ 2. Check local │
│ cache │
│ (optional) │
└────┬────────────┘
│
│ 3. INCR counter:page_views
▼
┌─────────────────┐
│ Redis Master │
│ │
│ 4. Atomic │
│ increment │
│ │
│ 5. Return new │
│ value │
└────┬────────────┘
│
│ 6. Replicate to replicas
▼
┌─────────────────┐
│ Redis Replica │
│ │
│ 7. Async │
│ replication │
└─────────────────┘
│
│ 8. Return response
▼
┌──────────┐
│ Client │
│ │
│ Result: │
│ { value: 1001 }
└──────────┘

2.3.2 Batch Increment Flow (Optimized)​

Application Server
┌─────────────────────────────────────┐
│ │
│ 1. Buffer increments (100ms) │
│ • page_views: +5 │
│ • api_calls: +3 │
│ • errors: +1 │
│ │
│ 2. Flush buffer (batch) │
└─────────┬───────────────────────────┘
│
│ 3. Pipeline command:
│ INCRBY counter:page_views 5
│ INCRBY counter:api_calls 3
│ INCRBY counter:errors 1
▼
┌─────────────────────────────────────┐
│ Redis Master │
│ │
│ 4. Execute pipeline atomically │
│ • All or nothing │
│ • Single network round trip │
│ │
│ 5. Return all results │
│ [1005, 503, 42] │
└─────────┬───────────────────────────┘
│
│ 6. Response
▼
┌─────────────────────────────────────┐
│ Application Server │
│ │
│ 7. Update local cache (optional) │
│ 8. Return to client │
└─────────────────────────────────────┘

3. Architecture Design​

3.1 System Architecture Patterns​

3.1.1 Write-Through Cache Pattern​

┌────────────────────────────────────────────────────────────┐
│ Application Logic │
└─────────────────┬──────────────────────────────────────────┘
│
│ Increment counter
▼
┌───────────────┐
│ Is write op? │
└───┬───────┬───┘
│ │
Yes │ │ No (read)
▼ ▼
┌────────────┐ ┌────────────┐
│ Write to │ │ Check │
│ Redis │ │ local │
│ (MASTER) │ │ cache │
└─────┬──────┘ └─────┬──────┘
│ │
│ │ Cache miss?
│ ▼
│ ┌────────────┐
│ │ Read from │
│ │ Redis │
│ │ (REPLICA) │
│ └─────┬──────┘
│ │
▼ ▼
┌──────────────────────────┐
│ Update local cache │
│ (with TTL) │
└──────────────────────────┘

3.1.2 Circuit Breaker Pattern​

┌─────────────────────────────────────────────────────────┐
│ Client Request │
└──────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────┐
│ Circuit Breaker │
│ State? │
└────┬────┬───┬───┘
│ │ │
┌───────┘ │ └────────┐
│ CLOSED │ OPEN │ HALF-OPEN
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Execute │ │ Fail │ │ Try one │
│ normally │ │ fast │ │ request │
└────┬─────┘ │ │ └────┬─────┘
│ │ │ │
│ │ │ │ Success?
▼ ▼ ▼ ▼
┌──────────────────────────────────────┐
│ Track Success/Failure │
│ │
│ Failures > Threshold? │
│ Yes → Open circuit │
│ │
│ After timeout (30s)? │
│ Yes → Half-open │
│ │
│ Half-open success? │
│ Yes → Close circuit │
└──────────────────────────────────────┘

3.2 Deployment Architecture​

3.2.1 Multi-Region Setup​

┌──────────────────────────────────────────────────────────────────┐
│ Global Load Balancer │
│ (GeoDNS / AWS Route53) │
└────────────┬─────────────────────────────────┬───────────────────┘
│ │
┌────────┴────────┐ ┌───────┴────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ Region: US-EAST-1 │ │ Region: EU-WEST-1 │
│ │ │ │
│ ┌──────────────────┐ │ │ ┌──────────────────┐ │
│ │ Redis Cluster │ │ │ │ Redis Cluster │ │
│ │ (Primary) │◄──┼──────┼─▶│ (Replica) │ │
│ │ │ │ │ │ │ │
│ │ • 3 Shards │ │ │ │ • 3 Shards │ │
│ │ • 2 Replicas ea. │ │ │ │ • Read-only │ │
│ └──────────────────┘ │ │ └──────────────────┘ │
│ │ │ │
│ ┌──────────────────┐ │ │ ┌──────────────────┐ │
│ │ App Servers │ │ │ │ App Servers │ │
│ │ (3-10 instances) │ │ │ │ (3-10 instances) │ │
│ └──────────────────┘ │ │ └──────────────────┘ │
│ │ │ │
│ Cross-region │ │ Cross-region │
│ replication ───────────┼──────┼──────────────────────▶ │
│ (async, eventual │ │ │
│ consistency) │ │ │
└─────────────────────────┘ └─────────────────────────┘

• Latency: 10-20ms • Latency: 10-20ms
• Write: US region • Write: Replicate from US
• Read: Local • Read: Local

3.2.2 Kubernetes Deployment​

# Redis StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-cluster
spec:
serviceName: redis-cluster
replicas: 6 # 3 masters + 3 replicas
selector:
matchLabels:
app: redis-cluster
template:
metadata:
labels:
app: redis-cluster
spec:
containers:
- name: redis
image: redis:7.2-alpine
ports:
- containerPort: 6379
name: client
- containerPort: 16379
name: gossip
volumeMounts:
- name: data
mountPath: /data
- name: conf
mountPath: /conf
command:
- redis-server
- /conf/redis.conf
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Gi

---
# Application Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: counter-service
spec:
replicas: 5
selector:
matchLabels:
app: counter-service
template:
metadata:
labels:
app: counter-service
spec:
containers:
- name: app
image: counter-service:v1.0
env:
- name: REDIS_CLUSTER_NODES
value: "redis-cluster-0.redis-cluster:6379,redis-cluster-1.redis-cluster:6379"
ports:
- containerPort: 8080
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"

4. Data Model​

4.1 Redis Data Structures​

4.1.1 Simple Counter (STRING)​

Key Pattern: counter:{name}
Value: Integer (64-bit signed)
TTL: Optional (for temporary counters)

Examples:
┌────────────────────────────────┬────────────┬─────────┐
│ Key │ Value │ TTL │
├────────────────────────────────┼────────────┼─────────┤
│ counter:page_views │ 1,234,567 │ -1 │
│ counter:api:rate_limit:user123 │ 45 │ 3600s │
│ counter:daily_signups:20260107 │ 892 │ 86400s │
│ counter:product:SKU123:stock │ 150 │ -1 │
└────────────────────────────────┴────────────┴─────────┘

Commands:
- INCR counter:page_views → Increment by 1
- INCRBY counter:page_views 5 → Increment by 5
- DECR counter:product:SKU123:stock → Decrement by 1
- GET counter:page_views → Read current value
- SET counter:page_views 0 EX 3600 → Reset with TTL

4.1.2 Counter with Metadata (HASH)​

Key Pattern: counter_meta:{name}
Value: Hash with multiple fields

Example: User engagement counter
┌────────────────────────────────────────────────────────┐
│ Key: counter_meta:user:12345 │
├────────────────────┬───────────────────────────────────┤
│ Field │ Value │
├────────────────────┼───────────────────────────────────┤
│ page_views │ 1,234 │
│ likes_given │ 456 │
│ comments_posted │ 89 │
│ shares │ 23 │
│ last_active │ 2026-01-07T10:30:00Z │
│ created_at │ 2025-06-15T08:00:00Z │
└────────────────────┴───────────────────────────────────┘

Commands:
- HINCRBY counter_meta:user:12345 page_views 1
- HGETALL counter_meta:user:12345
- HGET counter_meta:user:12345 page_views
- HMSET counter_meta:user:12345 last_active "2026-01-07T10:30:00Z"

4.1.3 Time-Series Counter (SORTED SET)​

Key Pattern: ts_counter:{name}
Value: Sorted Set (timestamp → count)

Example: Hourly page views
┌──────────────────────────────────────────────────────────┐
│ Key: ts_counter:page_views:hourly │
├──────────────────────┬───────────────────────────────────┤
│ Score (Timestamp) │ Member (Count) │
├──────────────────────┼───────────────────────────────────┤
│ 1704697200 │ 15,234 (2026-01-07 08:00) │
│ 1704700800 │ 18,456 (2026-01-07 09:00) │
│ 1704704400 │ 21,789 (2026-01-07 10:00) │
│ 1704708000 │ 19,234 (2026-01-07 11:00) │
└──────────────────────┴───────────────────────────────────┘

Commands:
- ZADD ts_counter:page_views:hourly 1704697200 15234
- ZRANGE ts_counter:page_views:hourly 0 -1 WITHSCORES
- ZRANGEBYSCORE ts_counter:page_views:hourly 1704697200 1704708000
- ZCARD ts_counter:page_views:hourly → Get total entries

4.1.4 Distributed Counter with Sharding​

Strategy: Split counter across multiple keys to reduce contention

Pattern: counter:{name}:shard:{N}
Total = SUM of all shards

Example: High-traffic page view counter
┌────────────────────────────────┬────────────┐
│ Key │ Value │
├────────────────────────────────┼────────────┤
│ counter:homepage:shard:0 │ 342,156 │
│ counter:homepage:shard:1 │ 338,942 │
│ counter:homepage:shard:2 │ 345,789 │
│ counter:homepage:shard:3 │ 341,234 │
└────────────────────────────────┴────────────┘

Total = 342,156 + 338,942 + 345,789 + 341,234 = 1,368,121

Algorithm:
1. Hash counter name to determine shard: shard_id = hash(user_id) % NUM_SHARDS
2. Increment specific shard: INCR counter:homepage:shard:{shard_id}
3. Read total: MGET counter:homepage:shard:* and sum

Benefits:
• Reduces lock contention on single key
• Distributes load across cluster
• Improves write throughput

4.2 Key Naming Conventions​

Standard Pattern: {namespace}:{entity}:{id}:{attribute}

Examples:
• counter:page_views - Simple global counter
• counter:user:12345:profile_views - User-specific counter
• counter:product:SKU789:stock - Product inventory
• counter:api:rate_limit:192.168.1.1 - IP-based rate limit
• counter:daily:signups:2026-01-07 - Date-based counter
• counter:campaign:SUMMER2026:clicks - Campaign tracking

Hierarchical Structure:
counter
├── global
│ ├── page_views
│ ├── api_calls
│ └── errors
├── user
│ ├── {user_id}
│ │ ├── profile_views
│ │ ├── posts_created
│ │ └── followers
├── product
│ └── {sku}
│ ├── views
│ ├── purchases
│ └── stock
└── daily
└── {date}
├── signups
├── revenue
└── orders

4.3 Data Retention Policies​

┌──────────────────────┬──────────────┬─────────────────────────┐
│ Counter Type │ Retention │ Strategy │
├──────────────────────┼──────────────┼─────────────────────────┤
│ Real-time metrics │ 24 hours │ TTL=86400 │
│ Daily aggregates │ 90 days │ TTL=7776000 │
│ Monthly aggregates │ 2 years │ TTL=63072000 │
│ Permanent counters │ Indefinite │ No TTL │
│ Rate limit counters │ 1 hour │ TTL=3600 │
└──────────────────────┴──────────────┴─────────────────────────┘

Automatic Archival:
• Hourly job exports old counters to Data Warehouse
• Compressed and stored in S3/GCS
• Queryable via BigQuery/Athena

5. API Design​

5.1 REST API Endpoints​

5.1.1 Counter Operations​

POST /api/v1/counters/{counterName}/increment
Content-Type: application/json

{
"value": 1,
"metadata": {
"user_id": "12345",
"source": "web"
}
}

Response 200 OK:
{
"counter": "page_views",
"value": 1001,
"timestamp": "2026-01-07T10:30:45Z"
}

---

GET /api/v1/counters/{counterName}

Response 200 OK:
{
"counter": "page_views",
"value": 1001,
"created_at": "2025-01-01T00:00:00Z",
"updated_at": "2026-01-07T10:30:45Z"
}

---

POST /api/v1/counters/{counterName}/decrement
Content-Type: application/json

{
"value": 1
}

Response 200 OK:
{
"counter": "product_stock",
"value": 149,
"timestamp": "2026-01-07T10:30:50Z"
}

---

POST /api/v1/counters/{counterName}/reset

Response 200 OK:
{
"counter": "daily_views",
"value": 0,
"timestamp": "2026-01-07T00:00:00Z"
}

---

POST /api/v1/counters/batch
Content-Type: application/json

{
"operations": [
{ "counter": "page_views", "action": "increment", "value": 1 },
{ "counter": "api_calls", "action": "increment", "value": 5 },
{ "counter": "errors", "action": "increment", "value": 1 }
]
}

Response 200 OK:
{
"results": [
{ "counter": "page_views", "value": 1002 },
{ "counter": "api_calls", "value": 5678 },
{ "counter": "errors", "value": 43 }
]
}

5.2 Client Library Interface​

5.2.1 Java Client (Using Lettuce)​

public interface CounterClient {
/**
* Increment counter by 1
* @param counterName Name of the counter
* @return New counter value
*/
CompletableFuture<Long> increment(String counterName);

/**
* Increment counter by specified value
* @param counterName Name of the counter
* @param value Amount to increment
* @return New counter value
*/
CompletableFuture<Long> incrementBy(String counterName, long value);

/**
* Decrement counter by specified value
* @param counterName Name of the counter
* @param value Amount to decrement
* @return New counter value
*/
CompletableFuture<Long> decrementBy(String counterName, long value);

/**
* Get current counter value
* @param counterName Name of the counter
* @return Current value
*/
CompletableFuture<Long> get(String counterName);

/**
* Reset counter to zero
* @param counterName Name of the counter
* @return Success status
*/
CompletableFuture<Boolean> reset(String counterName);

/**
* Batch operations
* @param operations List of counter operations
* @return Map of counter names to new values
*/
CompletableFuture<Map<String, Long>> batchExecute(
List<CounterOperation> operations
);
}

// Usage Example
public class CounterService {
private final CounterClient counterClient;

public void trackPageView(String userId, String page) {
counterClient.increment("page_views")
.thenCompose(count ->
counterClient.incrementBy("user:" + userId + ":views", 1))
.whenComplete((result, error) -> {
if (error != null) {
log.error("Failed to track page view", error);
} else {
log.info("Page view tracked: {}", result);
}
});
}
}

5.2.2 Node.js Client (Using ioredis)​

class CounterClient {
constructor(redisClient) {
this.redis = redisClient;
this.localCache = new Map();
this.cacheTTL = 60000; // 60 seconds
}

/**
* Increment counter by 1
* @param {string} counterName - Name of the counter
* @returns {Promise<number>} New counter value
*/
async increment(counterName) {
const key = `counter:${counterName}`;
const value = await this.redis.incr(key);
this.updateLocalCache(counterName, value);
return value;
}

/**
* Increment counter by specified value
* @param {string} counterName - Name of the counter
* @param {number} value - Amount to increment
* @returns {Promise<number>} New counter value
*/
async incrementBy(counterName, value) {
const key = `counter:${counterName}`;
const newValue = await this.redis.incrby(key, value);
this.updateLocalCache(counterName, newValue);
return newValue;
}

/**
* Get current counter value (with local cache)
* @param {string} counterName - Name of the counter
* @returns {Promise<number>} Current value
*/
async get(counterName) {
// Check local cache first
const cached = this.localCache.get(counterName);
if (cached && Date.now() - cached.timestamp < this.cacheTTL) {
return cached.value;
}

// Fetch from Redis
const key = `counter:${counterName}`;
const value = await this.redis.get(key);
const numValue = parseInt(value) || 0;

this.updateLocalCache(counterName, numValue);
return numValue;
}

/**
* Batch operations using pipeline
* @param {Array<Object>} operations - List of operations
* @returns {Promise<Array<number>>} Results
*/
async batchExecute(operations) {
const pipeline = this.redis.pipeline();

for (const op of operations) {
const key = `counter:${op.counter}`;
switch (op.action) {
case 'increment':
pipeline.incrby(key, op.value || 1);
break;
case 'decrement':
pipeline.decrby(key, op.value || 1);
break;
case 'get':
pipeline.get(key);
break;
}
}

const results = await pipeline.exec();
return results.map(([err, value]) => err ? null : value);
}

updateLocalCache(counterName, value) {
this.localCache.set(counterName, {
value,
timestamp: Date.now()
});
}
}

// Usage Example
const counterClient = new CounterClient(redisClient);

// Track page view
await counterClient.increment('page_views');

// Batch operations
const results = await counterClient.batchExecute([
{ counter: 'page_views', action: 'increment', value: 1 },
{ counter: 'api_calls', action: 'increment', value: 5 },
{ counter: 'errors', action: 'get' }
]);

5.2.3 Python Client (Using redis-py)​

from typing import Dict, List, Optional
import redis
from redis.cluster import RedisCluster
import asyncio
from datetime import datetime, timedelta

class CounterClient:
def __init__(self, redis_client: RedisCluster):
self.redis = redis_client
self.local_cache: Dict[str, tuple[int, datetime]] = {}
self.cache_ttl = timedelta(seconds=60)

async def increment(self, counter_name: str) -> int:
"""Increment counter by 1"""
key = f"counter:{counter_name}"
value = await self.redis.incr(key)
self._update_local_cache(counter_name, value)
return value

async def increment_by(self, counter_name: str, value: int) -> int:
"""Increment counter by specified value"""
key = f"counter:{counter_name}"
new_value = await self.redis.incrby(key, value)
self._update_local_cache(counter_name, new_value)
return new_value

async def get(self, counter_name: str) -> int:
"""Get current counter value with local caching"""
# Check local cache
if counter_name in self.local_cache:
value, timestamp = self.local_cache[counter_name]
if datetime.now() - timestamp < self.cache_ttl:
return value

# Fetch from Redis
key = f"counter:{counter_name}"
value = await self.redis.get(key)
num_value = int(value) if value else 0

self._update_local_cache(counter_name, num_value)
return num_value

async def batch_execute(self, operations: List[Dict]) -> List[int]:
"""Execute multiple operations in a pipeline"""
pipeline = self.redis.pipeline()

for op in operations:
key = f"counter:{op['counter']}"
action = op['action']
value = op.get('value', 1)

if action == 'increment':
pipeline.incrby(key, value)
elif action == 'decrement':
pipeline.decrby(key, value)
elif action == 'get':
pipeline.get(key)

results = await pipeline.execute()
return [int(r) if r else 0 for r in results]

def _update_local_cache(self, counter_name: str, value: int):
self.local_cache[counter_name] = (value, datetime.now())

# Usage
counter_client = CounterClient(redis_client)

# Track page view
count = await counter_client.increment('page_views')
print(f"Page views: {count}")

# Batch operations
results = await counter_client.batch_execute([
{'counter': 'page_views', 'action': 'increment'},
{'counter': 'api_calls', 'action': 'increment', 'value': 5},
{'counter': 'errors', 'action': 'get'}
])

6. Implementation Details​

6.1 Connection Management​

6.1.1 Connection Pool Configuration​

// Java - Lettuce Configuration
@Configuration
public class RedisConfig {

@Bean
public RedisClusterClient redisClusterClient() {
List<RedisURI> nodes = Arrays.asList(
RedisURI.create("redis://node1:6379"),
RedisURI.create("redis://node2:6379"),
RedisURI.create("redis://node3:6379")
);

ClusterClientOptions options = ClusterClientOptions.builder()
.autoReconnect(true)
.maxRedirects(3)
.validateClusterNodeMembership(true)
.build();

return RedisClusterClient.create(nodes);
}

@Bean
public GenericObjectPoolConfig<StatefulRedisClusterConnection> poolConfig() {
GenericObjectPoolConfig config = new GenericObjectPoolConfig();
config.setMaxTotal(50); // Max connections
config.setMaxIdle(20); // Max idle connections
config.setMinIdle(5); // Min idle connections
config.setMaxWaitMillis(2000); // Max wait time
config.setTestOnBorrow(true); // Test before borrowing
config.setTestWhileIdle(true); // Test during idle
return config;
}
}
// Node.js - ioredis Cluster Configuration
const Redis = require('ioredis');

const cluster = new Redis.Cluster([
{ host: 'node1', port: 6379 },
{ host: 'node2', port: 6379 },
{ host: 'node3', port: 6379 }
], {
redisOptions: {
password: process.env.REDIS_PASSWORD,
tls: process.env.REDIS_TLS === 'true' ? {} : undefined
},
clusterRetryStrategy: (times) => {
return Math.min(100 * times, 2000);
},
enableReadyCheck: true,
maxRedirections: 16,
retryDelayOnFailover: 100,
retryDelayOnClusterDown: 300,
slotsRefreshTimeout: 1000,
enableOfflineQueue: true,
lazyConnect: false
});

cluster.on('error', (err) => {
console.error('Redis Cluster Error:', err);
});

cluster.on('ready', () => {
console.log('Redis Cluster Ready');
});

module.exports = cluster;

6.2 Error Handling & Retry Logic​

6.2.1 Exponential Backoff Strategy​

Retry Attempt Flow:
┌──────────────────────────────────────────────────────────┐
│ Attempt 1: Immediate │
│ ↓ Failure │
│ Wait: 100ms │
│ ↓ │
│ Attempt 2 │
│ ↓ Failure │
│ Wait: 200ms │
│ ↓ │
│ Attempt 3 │
│ ↓ Failure │
│ Wait: 400ms │
│ ↓ │
│ Attempt 4 │
│ ↓ Failure │
│ Wait: 800ms │
│ ↓ │
│ Attempt 5 │
│ ↓ Failure │
│ Give up, return error │
└──────────────────────────────────────────────────────────┘

Formula: delay = min(baseDelay * 2^attempt, maxDelay)
public class RetryableCounterClient {
private static final int MAX_RETRIES = 5;
private static final int BASE_DELAY_MS = 100;
private static final int MAX_DELAY_MS = 5000;

public CompletableFuture<Long> incrementWithRetry(String counterName) {
return retryOperation(() -> increment(counterName), 0);
}

private <T> CompletableFuture<T> retryOperation(
Supplier<CompletableFuture<T>> operation,
int attempt) {

return operation.get()
.exceptionally(error -> {
if (attempt >= MAX_RETRIES) {
throw new RuntimeException("Max retries exceeded", error);
}

if (isRetryableError(error)) {
int delay = calculateDelay(attempt);
log.warn("Retry attempt {} after {}ms", attempt + 1, delay);

return CompletableFuture
.delayedExecutor(delay, TimeUnit.MILLISECONDS)
.execute(() -> retryOperation(operation, attempt + 1));
} else {
throw new RuntimeException("Non-retryable error", error);
}
});
}

private int calculateDelay(int attempt) {
return Math.min(
BASE_DELAY_MS * (int) Math.pow(2, attempt),
MAX_DELAY_MS
);
}

private boolean isRetryableError(Throwable error) {
return error instanceof TimeoutException ||
error instanceof RedisConnectionException ||
error instanceof RedisClusterException;
}
}

6.3 Atomic Operations & Race Conditions​

6.3.1 Inventory Deduction with Stock Check​

-- Lua script for atomic stock deduction
-- KEYS[1]: counter key (e.g., "counter:product:SKU123:stock")
-- ARGV[1]: amount to deduct
-- Returns: new stock level or -1 if insufficient stock

local current = redis.call('GET', KEYS[1])
if not current then
return -1
end

local stock = tonumber(current)
local deduct = tonumber(ARGV[1])

if stock < deduct then
return -1 -- Insufficient stock
end

local new_stock = redis.call('DECRBY', KEYS[1], deduct)
return new_stock
public class InventoryService {
private final RedisCommands<String, String> redis;
private final String LUA_SCRIPT = """
local current = redis.call('GET', KEYS[1])
if not current then return -1 end
local stock = tonumber(current)
if stock < tonumber(ARGV[1]) then return -1 end
return redis.call('DECRBY', KEYS[1], ARGV[1])
""";

public boolean deductInventory(String sku, int quantity) {
String key = "counter:product:" + sku + ":stock";

Long result = redis.eval(
LUA_SCRIPT,
ScriptOutputType.INTEGER,
new String[]{key},
String.valueOf(quantity)
);

if (result == -1) {
log.warn("Insufficient stock for SKU: {}", sku);
return false;
}

log.info("Stock deducted for {}: new level = {}", sku, result);
return true;
}
}

6.3.2 Rate Limiting with Sliding Window​

-- Sliding window rate limiter
-- KEYS[1]: rate limit key
-- ARGV[1]: window size (seconds)
-- ARGV[2]: max requests
-- ARGV[3]: current timestamp

local key = KEYS[1]
local window = tonumber(ARGV[1])
local max_requests = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

-- Remove old entries outside the window
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)

-- Count current requests in window
local current = redis.call('ZCARD', key)

if current < max_requests then
-- Add new request
redis.call('ZADD', key, now, now)
redis.call('EXPIRE', key, window)
return {1, max_requests - current - 1} -- [allowed, remaining]
else
return {0, 0} -- [not allowed, remaining = 0]
end

6.4 Bulk Operations Optimization​

6.4.1 Batch Processing Pattern​

Single Operation Pattern (Inefficient):
┌─────────────────────────────────────────────┐
│ For each increment: │
│ 1. Network round trip (1-5ms) │
│ 2. Redis processing (0.1ms) │
│ 3. Network return (1-5ms) │
│ │
│ Total for 100 increments: 200-1000ms │
└─────────────────────────────────────────────┘

Pipeline Pattern (Efficient):
┌─────────────────────────────────────────────┐
│ Batch 100 operations: │
│ 1. Single network round trip (1-5ms) │
│ 2. Redis processes all (10ms) │
│ 3. Single network return (1-5ms) │
│ │
│ Total for 100 increments: 12-20ms │
│ Speedup: 10-50x faster! │
└─────────────────────────────────────────────┘
class BatchProcessor {
constructor(counterClient) {
this.client = counterClient;
this.buffer = [];
this.flushInterval = 100; // ms
this.maxBatchSize = 1000;

// Start periodic flush
setInterval(() => this.flush(), this.flushInterval);
}

async increment(counterName, value = 1) {
return new Promise((resolve, reject) => {
this.buffer.push({
counter: counterName,
action: 'increment',
value,
resolve,
reject
});

// Flush immediately if buffer is full
if (this.buffer.length >= this.maxBatchSize) {
this.flush();
}
});
}

async flush() {
if (this.buffer.length === 0) return;

const batch = this.buffer.splice(0, this.maxBatchSize);

try {
const results = await this.client.batchExecute(
batch.map(op => ({
counter: op.counter,
action: op.action,
value: op.value
}))
);

// Resolve all promises
batch.forEach((op, index) => {
op.resolve(results[index]);
});
} catch (error) {
// Reject all promises
batch.forEach(op => op.reject(error));
}
}
}

// Usage
const batchProcessor = new BatchProcessor(counterClient);

// These will be batched automatically
await Promise.all([
batchProcessor.increment('page_views'),
batchProcessor.increment('api_calls', 5),
batchProcessor.increment('errors')
]);

7. Performance Optimization​

7.1 Performance Metrics​

Target Performance SLAs:
┌────────────────────────┬──────────────┬─────────────┬──────────────┐
│ Operation │ P50 │ P95 │ P99 │
├────────────────────────┼──────────────┼─────────────┼──────────────┤
│ Single increment │ 1-2 ms │ 5 ms │ 10 ms │
│ Batch increment (100) │ 5-10 ms │ 20 ms │ 30 ms │
│ Read (with cache) │ 0.1-0.5 ms │ 1 ms │ 2 ms │
│ Read (no cache) │ 1-2 ms │ 5 ms │ 10 ms │
└────────────────────────┴──────────────┴─────────────┴──────────────┘

Throughput Targets:
• Single Redis node: 50,000-100,000 ops/sec
• 3-node cluster: 150,000-300,000 ops/sec
• With client-side caching: 500,000+ reads/sec

7.2 Caching Strategy​

7.2.1 Multi-Level Caching​

┌────────────────────────────────────────────────────────────┐
│ Application Request │
└──────────────────┬─────────────────────────────────────────┘
│
▼
┌─────────────────┐
│ L1: In-Memory │
│ (Process-local)│
│ TTL: 10-60s │
│ Size: 1,000 │
└────┬────────────┘
│ Cache miss
▼
┌─────────────────┐
│ L2: Redis │
│ (Distributed) │
│ TTL: 300s │
│ Size: Unlimited│
└────┬────────────┘
│ Cache miss
▼
┌─────────────────┐
│ L3: Database │
│ (PostgreSQL) │
│ TTL: Permanent │
└─────────────────┘

Performance Comparison:
• L1 hit: 0.1 ms (99% hit rate for hot keys)
• L2 hit: 1-2 ms (95% hit rate overall)
• L3 hit: 10-50 ms (rare, only on cold start)

7.2.2 Cache Invalidation Strategies​

public class CacheInvalidationManager {

// Strategy 1: TTL-based (simplest)
public void setWithTTL(String key, long value, int ttlSeconds) {
redis.setex(key, ttlSeconds, String.valueOf(value));
}

// Strategy 2: Write-through (consistency)
public long incrementWithInvalidation(String counterName) {
String key = "counter:" + counterName;
long newValue = redis.incr(key);

// Invalidate local caches across all app servers
cacheInvalidationBus.publish("invalidate:" + counterName);

return newValue;
}

// Strategy 3: Lazy invalidation (eventual consistency)
public long getWithLazyInvalidation(String counterName) {
LocalCacheEntry entry = localCache.get(counterName);

if (entry != null && !entry.isExpired()) {
return entry.getValue();
}

// Fetch fresh value
long value = redis.get("counter:" + counterName);
localCache.put(counterName, value, Duration.ofSeconds(60));

return value;
}
}

7.3 Sharding Strategy​

7.3.1 Consistent Hashing​

Hash Ring Distribution:
0°
│
Node C │ Node A
(240°) │ (0°)
╲ │ ╱
╲ │ ╱
╲ │ ╱
╲ │ ╱
╲│╱
180° ────────────── 0°/360°
╱│╲
╱ │ ╲
╱ │ ╲
╱ │ ╲
╱ │ ╲
Node B │ Node A
(120°) │ (360°)
│
180°

Key Mapping:
• hash("counter:page_views") = 45° → Node A
• hash("counter:api_calls") = 135° → Node B
• hash("counter:errors") = 270° → Node C

Benefits:
• Minimal key redistribution on node add/remove
• Balanced load distribution
• Built into Redis Cluster

7.3.2 Manual Sharding for Hot Keys​

public class ShardedCounterClient {
private static final int NUM_SHARDS = 10;

public long incrementSharded(String counterName) {
// Distribute writes across shards
int shardId = ThreadLocalRandom.current().nextInt(NUM_SHARDS);
String shardedKey = String.format(
"counter:%s:shard:%d",
counterName,
shardId
);

return redis.incr(shardedKey);
}

public long getTotal(String counterName) {
// Read from all shards and sum
List<CompletableFuture<Long>> futures = new ArrayList<>();

for (int i = 0; i < NUM_SHARDS; i++) {
String shardedKey = String.format(
"counter:%s:shard:%d",
counterName,
i
);
futures.add(redis.getAsync(shardedKey));
}

return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
.thenApply(v -> futures.stream()
.mapToLong(f -> f.join())
.sum())
.join();
}
}

8. High Availability & Disaster Recovery​

8.1 Redis Cluster Architecture​

High Availability Setup (3 Masters + 3 Replicas):

┌────────────────────────────────────────────────────────────┐
│ Sentinel Monitoring Layer │
│ [Sentinel 1] [Sentinel 2] [Sentinel 3] │
│ • Quorum: 2 (majority of 3) │
│ • Health check interval: 1s │
│ • Failover timeout: 3s │
└──────────────────┬─────────────────────────────────────────┘
│ Monitors
▼
┌────────────────────────────────────────────────────────────┐
│ Redis Cluster Nodes │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Master 1 │────────▶│Replica 1a│ │Replica 1b│ │
│ │ Slots: │ │(standby) │ │(standby) │ │
│ │ 0-5460 │ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Master 2 │────────▶│Replica 2a│ │Replica 2b│ │
│ │ Slots: │ │(standby) │ │(standby) │ │
│ │5461-10922│ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Master 3 │────────▶│Replica 3a│ │Replica 3b│ │
│ │ Slots: │ │(standby) │ │(standby) │ │
│ │10923- │ │ │ │ │ │
│ │16383 │ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└────────────────────────────────────────────────────────────┘

Failover Scenario:
1. Master 1 fails
2. Sentinels detect failure (1-3s)
3. Sentinels vote on promotion (quorum = 2)
4. Replica 1a promoted to Master
5. Clients automatically redirected
6. Total downtime: 3-5 seconds

8.2 Backup Strategy​

Backup Architecture:
┌────────────────────────────────────────────────────────────┐
│ Redis Master Nodes │
│ • RDB snapshots every 5 minutes │
│ • AOF log (append-only, fsync every second) │
└──────────────────┬─────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ Local Backup Storage │
│ • Last 24 hours of RDB snapshots │
│ • Rolling AOF logs │
│ • Retention: 24 hours │
└──────────────────┬─────────────────────────────────────────┘
│
│ Hourly sync
▼
┌────────────────────────────────────────────────────────────┐
│ Cloud Storage (S3/GCS) │
│ • Compressed and encrypted backups │
│ • Retention policy: │
│ - Hourly: 7 days │
│ - Daily: 30 days │
│ - Weekly: 1 year │
│ - Monthly: 7 years (compliance) │
└────────────────────────────────────────────────────────────┘

Recovery Time Objectives (RTO):
• Hot standby failover: 3-5 seconds
• Restore from local backup: 5-10 minutes
• Restore from cloud backup: 30-60 minutes
• Disaster recovery (full rebuild): 2-4 hours

Recovery Point Objectives (RPO):
• With AOF: < 1 second data loss
• With RDB only: Up to 5 minutes data loss
• After disaster: Up to 1 hour data loss

8.3 Disaster Recovery Procedures​

#!/bin/bash
# Disaster Recovery Script

DR_SCENARIO=$1 # "master_failure", "cluster_down", "data_corruption"

case $DR_SCENARIO in
"master_failure")
echo "Scenario: Redis Master Node Failure"
echo "1. Sentinels detect failure automatically"
echo "2. Wait for automatic failover (3-5s)"
echo "3. Verify new master promotion"
redis-cli -h sentinel-1 -p 26379 SENTINEL get-master-addr-by-name mymaster

echo "4. Check cluster health"
redis-cli --cluster check redis-node1:6379
;;

"cluster_down")
echo "Scenario: Entire Cluster Failure"
echo "1. Stop all Redis processes"
systemctl stop redis-cluster

echo "2. Restore from latest backup"
LATEST_BACKUP=$(aws s3 ls s3://redis-backups/ --recursive | sort | tail -n 1 | awk '{print $4}')
aws s3 cp s3://redis-backups/$LATEST_BACKUP /var/lib/redis/dump.rdb

echo "3. Start Redis cluster"
systemctl start redis-cluster

echo "4. Verify data integrity"
redis-cli --cluster check redis-node1:6379
;;

"data_corruption")
echo "Scenario: Data Corruption Detected"
echo "1. Identify corruption timestamp"
echo "2. Restore from point-in-time backup"

TARGET_TIME=$2
aws s3 cp s3://redis-backups/backup-$TARGET_TIME.rdb /var/lib/redis/dump.rdb

echo "3. Replay AOF from that point"
redis-check-aof --fix /var/lib/redis/appendonly.aof

echo "4. Restart and verify"
systemctl restart redis-cluster
;;
esac

# Health Check
echo "=== Cluster Health Check ==="
redis-cli --cluster info redis-node1:6379
redis-cli --cluster check redis-node1:6379

# Test Counter Operations
echo "=== Testing Counter Operations ==="
redis-cli INCR test:counter
redis-cli GET test:counter
redis-cli DEL test:counter

echo "Recovery completed at $(date)"

9. Security Considerations​

9.1 Security Architecture​

Security Layers:
┌────────────────────────────────────────────────────────────┐
│ Layer 1: Network Security │
│ • VPC/Private subnet │
│ • Security groups (allow only app servers) │
│ • No public internet access │
└──────────────────┬─────────────────────────────────────────┘
│
┌──────────────────┴─────────────────────────────────────────┐
│ Layer 2: Transport Security │
│ • TLS 1.3 encryption for all Redis connections │
│ • Certificate validation │
│ • Mutual TLS (mTLS) for client authentication │
└──────────────────┬─────────────────────────────────────────┘
│
┌──────────────────┴─────────────────────────────────────────┐
│ Layer 3: Authentication │
│ • Redis AUTH password (minimum 32 characters) │
│ • ACL (Access Control Lists) for fine-grained permissions │
│ • Separate credentials per application/environment │
└──────────────────┬─────────────────────────────────────────┘
│
┌──────────────────┴─────────────────────────────────────────┐
│ Layer 4: Authorization │
│ • ACL rules per user: │
│ - read-only users: ~* +@read │
│ - app users: counter:* +@write +@read │
│ - admin: * +@all │
└──────────────────┬─────────────────────────────────────────┘
│
┌──────────────────┴─────────────────────────────────────────┐
│ Layer 5: Data Protection │
│ • Encryption at rest (disk encryption) │
│ • Encrypted backups │
│ • Sensitive data masking in logs │
└────────────────────────────────────────────────────────────┘

9.2 Redis ACL Configuration​

# redis.conf - Enable ACL
aclfile /etc/redis/users.acl
requirepass your-strong-password-here

# /etc/redis/users.acl
# Admin user (full access)
user admin on >StrongAdminPass123! ~* +@all

# Application user (limited to counter operations)
user counter_app on >AppSecretPass456! ~counter:* +@write +@read +@string -@dangerous

# Read-only monitoring user
user monitor on >MonitorPass789! ~* +@read -@write -@dangerous

# Disable default user
user default off nopass ~* -@all

9.3 Security Best Practices Checklist​

✓ Network Security:
☑ Redis nodes in private subnet (no public IP)
☑ Security groups allow only app servers (port 6379)
☑ Sentinel nodes in separate security group (port 26379)
☑ VPN required for admin access

✓ Authentication & Authorization:
☑ Strong Redis AUTH password (32+ characters)
☑ ACL enabled with principle of least privilege
☑ Separate users for apps, admins, monitoring
☑ Password rotation every 90 days
☑ No hardcoded credentials (use secrets manager)

✓ Encryption:
☑ TLS 1.3 for all Redis connections
☑ Encryption at rest for data volumes
☑ Encrypted backups in S3/GCS
☑ Certificate pinning in clients

✓ Monitoring & Auditing:
☑ All Redis commands logged
☑ Failed authentication attempts alerted
☑ Abnormal traffic patterns detected
☑ Regular security audits scheduled

✓ Operational Security:
☑ Disable dangerous commands (FLUSHALL, KEYS, CONFIG)
☑ Resource limits (maxmemory, maxclients)
☑ Rate limiting at application layer
☑ Regular security patches applied

10. Monitoring & Observability​

10.1 Metrics Collection​

Monitoring Stack Architecture:
┌────────────────────────────────────────────────────────────┐
│ Redis Cluster Nodes │
│ • Expose metrics on :9121 (redis_exporter) │
└──────────────────┬─────────────────────────────────────────┘
│
│ Scrape every 15s
▼
┌────────────────────────────────────────────────────────────┐
│ Prometheus Server │
│ • Store metrics (15 days retention) │
│ • Evaluate alerting rules │
│ • Federation to long-term storage │
└──────────────────┬─────────────────────────────────────────┘
│
┌─────────┴─────────┬──────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────┐ ┌──────────────┐
│ Grafana │ │ AlertManager│ │ VictoriaDB │
│ (Dashboards) │ │ (Alerts) │ │ (Long-term) │
│ • Real-time │ │ • PagerDuty│ │ • 1y data │
│ • Historical │ │ • Slack │ │ • Analysis │
└─────────────────┘ └─────────────┘ └──────────────┘

10.2 Key Metrics to Monitor​

10.2.1 Redis Metrics​

# Critical Redis Metrics
metrics:
performance:
- redis_commands_processed_total
description: "Total commands processed"
threshold: "< 100k/s = Warning"

- redis_instantaneous_ops_per_sec
description: "Current ops/sec"
threshold: "> 80k/s = Capacity planning needed"

- redis_command_duration_seconds
description: "Command latency"
threshold: "P99 > 10ms = Alert"

memory:
- redis_memory_used_bytes
description: "Memory usage"
threshold: "> 80% = Warning, > 90% = Critical"

- redis_memory_fragmentation_ratio
description: "Memory fragmentation"
threshold: "> 1.5 = Consider restart"

- redis_evicted_keys_total
description: "Evicted keys (maxmemory-policy)"
threshold: "> 0 = Warning (data loss)"

replication:
- redis_connected_slaves
description: "Number of connected replicas"
threshold: "< 1 = Critical (no HA)"

- redis_master_repl_offset
description: "Replication lag (bytes)"
threshold: "> 10MB = Warning"

connections:
- redis_connected_clients
description: "Active client connections"
threshold: "> 10k = Warning"

- redis_rejected_connections_total
description: "Connection rejections"
threshold: "> 0 = Critical (maxclients reached)"

persistence:
- redis_rdb_last_save_timestamp_seconds
description: "Last successful save"
threshold: "> 600s = Warning (backup failed)"

- redis_aof_last_rewrite_duration_sec
description: "AOF rewrite duration"
threshold: "> 60s = Performance impact"

10.2.2 Application Metrics​

// Micrometer metrics in Java
@Service
public class CounterMetricsService {
private final MeterRegistry meterRegistry;

public void recordIncrement(String counterName, long latencyMs) {
// Counter metric
meterRegistry.counter(
"counter.operations",
"operation", "increment",
"counter_name", counterName
).increment();

// Latency histogram
meterRegistry.timer(
"counter.latency",
"operation", "increment"
).record(latencyMs, TimeUnit.MILLISECONDS);
}

public void recordError(String counterName, Exception error) {
meterRegistry.counter(
"counter.errors",
"counter_name", counterName,
"error_type", error.getClass().getSimpleName()
).increment();
}

public void recordCacheHit(boolean hit) {
meterRegistry.counter(
"counter.cache",
"result", hit ? "hit" : "miss"
).increment();
}
}

// Expose custom metrics
@RestController
public class MetricsController {
@GetMapping("/metrics/counters")
public Map<String, Object> getCounterMetrics() {
return Map.of(
"total_operations", getTotalOperations(),
"cache_hit_rate", getCacheHitRate(),
"average_latency_ms", getAverageLatency(),
"error_rate", getErrorRate()
);
}
}

10.3 Grafana Dashboards​

Dashboard: Redis Counter System Overview
┌──────────────────────────────────────────────────────────┐
│ Time Range: Last 24h Auto-refresh: 30s │
├──────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Operations │ │ Latency P99 │ │ Error Rate │ │
│ │ 125k/s │ │ 3.2 ms │ │ 0.01% │ │
│ │ ↑ 15% │ │ ↓ 0.5ms │ │ ↓ 0.005% │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Operations per Second (24h) │ │
│ │ [Line graph showing ops/sec over time] │ │
│ │ Peak: 150k at 14:00 │ │
│ │ Current: 125k │ │
│ └──────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────┐ ┌──────────────────────────┐ │
│ │ Memory Usage │ │ Top Counters (by ops) │ │
│ │ [Gauge: 65%] │ │ 1. page_views: 45k/s │ │
│ │ 13GB / 20GB │ │ 2. api_calls: 30k/s │ │
│ │ │ │ 3. user_actions: 25k/s │ │
│ └────────────────────┘ │ 4. events: 15k/s │ │
│ │ 5. clicks: 10k/s │ │
│ ┌────────────────────┐ └──────────────────────────┘ │
│ │ Cache Hit Rate │ │
│ │ [Pie chart] │ ┌──────────────────────────┐ │
│ │ Hits: 95% │ │ Replication Lag │ │
│ │ Misses: 5% │ │ [Line graph] │ │
│ └────────────────────┘ │ Master → Replica: 0.2s │ │
│ └──────────────────────────┘ │
├──────────────────────────────────────────────────────────┤
│ Recent Alerts: │
│ • None - All systems operational ✓ │
└──────────────────────────────────────────────────────────┘

10.4 Alerting Rules​

# prometheus-alerts.yml
groups:
- name: redis_counter_alerts
interval: 30s
rules:
# High latency alert
- alert: HighRedisLatency
expr: histogram_quantile(0.99, rate(redis_command_duration_seconds_bucket[5m])) > 0.010
for: 5m
labels:
severity: warning
component: redis
annotations:
summary: "High Redis latency detected"
description: "P99 latency is {{ $value }}s (threshold: 10ms)"

# Memory usage alert
- alert: RedisMemoryHigh
expr: (redis_memory_used_bytes / redis_memory_max_bytes) > 0.90
for: 5m
labels:
severity: critical
component: redis
annotations:
summary: "Redis memory usage critical"
description: "Memory usage is {{ $value | humanizePercentage }} (threshold: 90%)"

# Replication lag alert
- alert: ReplicationLagHigh
expr: redis_master_repl_offset - redis_slave_repl_offset > 10485760
for: 2m
labels:
severity: warning
component: redis
annotations:
summary: "High replication lag"
description: "Lag is {{ $value | humanize1024 }}B behind master"

# Connection pool exhaustion
- alert: ConnectionPoolExhausted
expr: rate(redis_rejected_connections_total[5m]) > 0
for: 1m
labels:
severity: critical
component: redis
annotations:
summary: "Redis rejecting connections"
description: "Connection limit reached - scale up"

# Application error rate
- alert: HighCounterErrorRate
expr: rate(counter_errors_total[5m]) > 10
for: 5m
labels:
severity: warning
component: application
annotations:
summary: "High counter operation error rate"
description: "Error rate is {{ $value }} errors/sec"

11. Deployment Strategy​

11.1 Environment Setup​

Environment Progression:
┌────────────────────────────────────────────────────────┐
│ Development (Local) │
│ • Single Redis node │
│ • No replication │
│ • Volume: 1k ops/sec │
│ • Purpose: Feature development │
└────────────────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ Staging (Cloud) │
│ • 3-node cluster (1 master + 2 replicas per shard) │
│ • Sentinel enabled │
│ • Volume: 10k ops/sec │
│ • Purpose: Integration testing, QA │
└────────────────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ Production (Multi-region) │
│ • 9-node cluster (3 masters + 2 replicas each) │
│ • Sentinel + Redis Cluster mode │
│ • Volume: 100k+ ops/sec │
│ • Purpose: Live traffic │
└────────────────────────────────────────────────────────┘

11.2 Blue-Green Deployment​

Blue-Green Deployment Strategy:
┌────────────────────────────────────────────────────────┐
│ Load Balancer │
│ Route: 100% → Blue │
└──────────────────┬─────────────────────────────────────┘
│
┌──────────┴──────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Blue (Current) │ │ Green (New) │
│ v1.0 │ │ v1.1 │
│ │ │ │
│ Redis Cluster │ │ Redis Cluster │
│ • 3 shards │ │ • 3 shards │
│ • Production │ │ • Standby │
└─────────────────┘ └─────────────────┘

Deployment Steps:
1. Deploy v1.1 to Green environment
2. Run smoke tests on Green
3. Gradually shift traffic: 10% → 50% → 100%
4. Monitor for errors/latency
5. If successful: Blue becomes standby
6. If failure: Instant rollback to Blue

11.3 Canary Deployment​

Canary Deployment Flow:
┌────────────────────────────────────────────────────────┐
│ Step 1: Deploy to 1% of users │
│ • Monitor for 1 hour │
│ • Check: error rate, latency, success rate │
└────────────────┬───────────────────────────────────────┘
│ ✓ Success
▼
┌────────────────────────────────────────────────────────┐
│ Step 2: Increase to 10% │
│ • Monitor for 30 minutes │
└────────────────┬───────────────────────────────────────┘
│ ✓ Success
▼
┌────────────────────────────────────────────────────────┐
│ Step 3: Increase to 50% │
│ • Monitor for 15 minutes │
└────────────────┬───────────────────────────────────────┘
│ ✓ Success
▼
┌────────────────────────────────────────────────────────┐
│ Step 4: Full rollout to 100% │
│ • Continue monitoring │
└────────────────────────────────────────────────────────┘

Traffic Split Configuration (Istio):
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: counter-service
spec:
hosts:
- counter-service
http:
- match:
- headers:
canary:
exact: "true"
route:
- destination:
host: counter-service
subset: v2
weight: 100
- route:
- destination:
host: counter-service
subset: v1
weight: 90
- destination:
host: counter-service
subset: v2
weight: 10

11.4 Rolling Updates​

#!/bin/bash
# Rolling update script for Redis Cluster

NODES=("node1" "node2" "node3" "node4" "node5" "node6")

for NODE in "${NODES[@]}"; do
echo "Updating $NODE..."

# 1. Check if node is master
IS_MASTER=$(redis-cli -h $NODE -p 6379 ROLE | head -1)

if [ "$IS_MASTER" == "master" ]; then
echo "$NODE is master - triggering failover first"
redis-cli -h $NODE -p 6379 CLUSTER FAILOVER
sleep 5
fi

# 2. Stop Redis
ssh $NODE "systemctl stop redis"

# 3. Update Redis binary
ssh $NODE "wget https://download.redis.io/releases/redis-7.2.3.tar.gz"
ssh $NODE "tar xzf redis-7.2.3.tar.gz && cd redis-7.2.3 && make install"

# 4. Start Redis
ssh $NODE "systemctl start redis"

# 5. Wait for node to rejoin cluster
sleep 10

# 6. Verify cluster health
redis-cli --cluster check redis-node1:6379

echo "$NODE updated successfully"
echo "Waiting 60s before next node..."
sleep 60
done

echo "Rolling update completed"

12. Cost Analysis​

12.1 Infrastructure Costs (Monthly)​

AWS Cost Breakdown (Production):
┌────────────────────────────┬──────────┬────────┬──────────┐
│ Component │ Quantity │ Unit $ │ Total $ │
├────────────────────────────┼──────────┼────────┼──────────┤
│ EC2: r6g.2xlarge (Redis) │ 9 nodes │ $250 │ $2,250 │
│ EBS: gp3 100GB │ 9 vols │ $10 │ $90 │
│ EC2: m6i.large (App) │ 5 nodes │ $70 │ $350 │
│ ALB (Load Balancer) │ 1 │ $25 │ $25 │
│ Data Transfer (out) │ 1TB │ $90 │ $90 │
│ S3 Backup Storage │ 500GB │ $12 │ $12 │
│ CloudWatch (monitoring) │ - │ - │ $50 │
│ VPN/Direct Connect │ 1 │ $100 │ $100 │
├────────────────────────────┴──────────┴────────┼──────────┤
│ Total Monthly Cost │ $2,967 │
└─────────────────────────────────────────────────┴──────────┘

Cost per Million Operations:
• At 100M ops/day: $0.99 per million ops
• At 1B ops/day: $0.10 per million ops

Cost Optimization Opportunities:
• Reserved Instances: Save 30-40% ($890/month)
• Spot Instances for non-prod: Save 60-70%
• Graviton instances (r6g): 20% cheaper than r5
• S3 Glacier for old backups: 90% storage savings

12.2 Scaling Cost Projections​

Growth Scenario Analysis:
┌─────────────┬───────────┬────────────┬─────────────┬──────────┐
│ Daily Ops │ Nodes Req │ Monthly $ │ Cost/M ops │ YoY │
├─────────────┼───────────┼────────────┼─────────────┼──────────┤
│ 100M (now) │ 9 │ $2,967 │ $0.99 │ Baseline │
│ 500M (6mo) │ 15 │ $4,500 │ $0.30 │ +52% │
│ 1B (1yr) │ 27 │ $7,200 │ $0.24 │ +60% │
│ 5B (2yr) │ 45 │ $11,800 │ $0.08 │ +64% │
└─────────────┴───────────┴────────────┴─────────────┴──────────┘

Key Insights:
• Cost per operation decreases with scale
• Non-linear scaling due to better utilization
• Breaking even with managed Redis at 500M ops/day

13. Appendices​

Appendix A: Redis Commands Reference​

# Counter Operations
INCR counter:name # Increment by 1
INCRBY counter:name 5 # Increment by 5
DECR counter:name # Decrement by 1
DECRBY counter:name 3 # Decrement by 3
GET counter:name # Get current value
SET counter:name 0 # Reset to 0

# Atomic Operations
GETSET counter:name 100 # Set and return old value
APPEND counter:name "text" # Append to string

# Expiration
EXPIRE counter:name 3600 # Set TTL (seconds)
TTL counter:name # Check remaining TTL
PERSIST counter:name # Remove TTL

# Batch Operations
MGET counter:1 counter:2 # Get multiple
MSET counter:1 10 counter:2 20 # Set multiple

# Pipeline Example
redis-cli --pipe <<EOF
INCR counter:page_views
INCR counter:api_calls
INCR counter:errors
EOF

Appendix B: Performance Benchmarks​

# redis-benchmark command
redis-benchmark -h localhost -p 6379 \
-t set,get,incr,lpush,lpop \
-n 1000000 \
-c 50 \
-d 16 \
--csv

# Results (r6g.2xlarge):
"test","rps","avg_latency_ms","min_latency","p50","p95","p99","p99.9"
"INCR",98765,0.51,0.24,0.48,0.72,1.20,2.15
"GET",102340,0.49,0.20,0.45,0.68,1.15,2.10
"SET",95420,0.53,0.25,0.50,0.75,1.25,2.20
"LPUSH",89230,0.56,0.28,0.53,0.80,1.35,2.40

Appendix C: Troubleshooting Guide​

Common Issues and Solutions:
┌─────────────────────────┬──────────────────────────────────┐
│ Issue │ Solution │
├─────────────────────────┼──────────────────────────────────┤
│ High latency │ • Check network │
│ │ • Increase connection pool │
│ │ • Use pipelining │
│ │ • Check slow log │
├─────────────────────────┼──────────────────────────────────┤
│ Memory exhaustion │ • Set maxmemory-policy │
│ │ • Add TTL to counters │
│ │ • Scale horizontally │
│ │ • Check for memory leaks │
├─────────────────────────┼──────────────────────────────────┤
│ Connection refused │ • Check Redis is running │
│ │ • Verify network rules │
│ │ • Check maxclients limit │
│ │ • Review authentication │
├─────────────────────────┼──────────────────────────────────┤
│ Replication lag │ • Check network bandwidth │
│ │ • Reduce write load │
│ │ • Optimize large values │
│ │ • Consider more replicas │
├─────────────────────────┼──────────────────────────────────┤
│ Data inconsistency │ • Check AOF/RDB config │
│ │ • Review fsync settings │
│ │ • Verify cluster slots │
│ │ • Check for split brain │
└─────────────────────────┴──────────────────────────────────┘

Appendix D: Glossary​

  • AOF (Append-Only File): Durability mechanism that logs every write
  • Circuit Breaker: Pattern to prevent cascading failures
  • Consistent Hashing: Distribution algorithm for sharding
  • Gossip Protocol: Cluster communication mechanism
  • Pipelining: Sending multiple commands without waiting for replies
  • RDB (Redis Database): Point-in-time snapshot format
  • Sentinel: High availability monitoring and failover
  • Sharding: Distributing data across multiple nodes
  • Slot: Unit of data distribution (16384 slots in cluster)
  • Write-Through: Cache strategy where writes go to both cache and DB

Distributed Counter System with Redis

· 阅读需 23 分钟

1. Executive Summary​

This document outlines the design for a centralized counter service using Redis as the backend store in a distributed system. The solution addresses the need for consistent, performant, and reliable counter operations across multiple service instances, providing atomic increments, decrements, and retrieval with support for rate limiting, usage tracking, and real-time analytics.

Key Design Goals​

  • Strong consistency for counter operations across distributed nodes
  • High availability with 99.99% uptime target
  • Scalability to handle 100K+ operations per second
  • Low latency (<5ms for counter operations)
  • Fault tolerance with automatic failover and data durability

2. System Architecture​

2.1 High-Level Architecture Diagram​

graph TB
subgraph "Client Applications"
A1[Service A]
A2[Service B]
A3[Service C]
end

subgraph "Counter Service Layer"
B1[Counter API Gateway]
B2[Load Balancer]
B3[Service Registry]
end

subgraph "Redis Cluster"
C1[Master 1<br/>Shard 1]
C2[Slave 1.1]
C3[Slave 1.2]
C4[Master 2<br/>Shard 2]
C5[Slave 2.1]
C6[Slave 2.2]
C7[Master 3<br/>Shard 3]
C8[Slave 3.1]
C9[Slave 3.2]
end

subgraph "Supporting Services"
D1[Monitoring<br/>Prometheus/Grafana]
D2[Configuration Mgmt]
D3[Backup Service]
end

A1 --> B2
A2 --> B2
A3 --> B2
B2 --> B1
B1 --> C1
B1 --> C4
B1 --> C7
C1 --> C2
C1 --> C3
C4 --> C5
C4 --> C6
C7 --> C8
C7 --> C9
B1 --> D1
B1 --> D2
C1 --> D3
C4 --> D3
C7 --> D3

2.2 Component Responsibilities​

ComponentResponsibilityTechnology Options
Counter Client SDKProvides easy-to-use API for applicationsCustom SDK in multiple languages
API GatewayRequest routing, auth, rate limitingKong, AWS API Gateway, Custom
Counter ServiceBusiness logic, validation, aggregationNode.js, Go, Java Spring
Redis ClusterPrimary data store for countersRedis 7.x with Redis Stack
MonitoringMetrics collection and alertingPrometheus, Grafana, ELK Stack
Backup ServiceData persistence and disaster recoveryRedis RDB/AOF, S3 backup

3. Redis Data Design​

3.1 Key Structure Patterns​

graph LR
subgraph "Counter Key Design"
A[Counter Type] --> B[Namespace]
B --> C[Resource ID]
C --> D[Granularity]
D --> E[Full Key]
end

E --> F["counter:api:user_12345:20240120"]
E --> G["rate_limit:ip:192.168.1.1:minute"]
E --> H["inventory:product:sku_98765:warehouse"]

3.2 Data Models Implementation​

# Counter data structures in Redis
COUNTER_SCHEMA = {
# Simple counter (e.g., page views)
"page:views:homepage": {
"type": "string",
"encoding": "int",
"ttl": 2592000, # 30 days
"operations": ["INCR", "GET", "EXPIRE"]
},

# Hashed counter (e.g., user session counts by country)
"stats:sessions:20240120": {
"type": "hash",
"fields": {
"US": "integer",
"EU": "integer",
"ASIA": "integer"
},
"operations": ["HINCRBY", "HGETALL"]
},

# Sorted set for leaderboards
"leaderboard:game:season5": {
"type": "zset",
"score": "integer",
"member": "user_id",
"operations": ["ZINCRBY", "ZRANGE"]
},

# Bitmap for daily active users
"dau:20240120": {
"type": "bitmap",
"operations": ["SETBIT", "BITCOUNT"],
"max_bit": 10000000 # 10M users
}
}

3.3 Memory Optimization Strategies​

PatternUse CaseMemory SavingsExample
Integer encodingSmall counters75% vs stringINCR counter:id
Hash field packingMultiple related counters60% vs separate keysHINCRBY stats:20240120 US 1
Ziplist encodingSmall hashes/lists50-80% savingsredis.conf: hash-max-ziplist-entries 512
BitmapsBoolean flags96% vs setSETBIT active:users 12345 1
HyperLogLogApproximate counting99% vs setPFADD daily:visitors user123

4. Core Operations Design​

4.1 Atomic Operations Flow​

sequenceDiagram
participant Client as Client SDK
participant Service as Counter Service
participant Redis as Redis Master
participant Slave as Redis Replica

Client->>Service: increment("user:123:clicks", 1, ttl=3600)
Service->>Service: Validate request & rate limit
Service->>Redis: WATCH counter:user:123:clicks
Redis->>Service: OK
Service->>Redis: MULTI
Service->>Redis: INCRBY counter:user:123:clicks 1
Service->>Redis: EXPIRE counter:user:123:clicks 3600
Service->>Redis: EXEC
alt Success
Redis->>Service: [124] (new value)
Service->>Slave: Async replication
Service->>Client: {"success": true, "value": 124}
else Conflict
Redis->>Service: NULL (transaction failed)
Service->>Service: Retry logic (max 3 attempts)
Service->>Redis: Retry operations...
end

4.2 Counter Operation Patterns​

class RedisCounterService:
def __init__(self, redis_client):
self.redis = redis_client
self.lua_scripts = self._load_lua_scripts()

def _load_lua_scripts(self):
"""Load atomic Lua scripts for complex operations"""
return {
'increment_with_min_max': """
local key = KEYS[1]
local increment = tonumber(ARGV[1])
local min_val = tonumber(ARGV[2])
local max_val = tonumber(ARGV[3])
local ttl = tonumber(ARGV[4])

local current = redis.call('GET', key) or 0
current = tonumber(current)
local new_value = current + increment

-- Apply bounds
if min_val and new_value < min_val then
new_value = min_val
end
if max_val and new_value > max_val then
new_value = max_val
end

redis.call('SET', key, new_value)
if ttl > 0 then
redis.call('EXPIRE', key, ttl)
end
return new_value
""",

'batch_increment': """
-- Increment multiple counters atomically
for i = 1, #KEYS do
redis.call('INCRBY', KEYS[i], ARGV[i])
end
return #KEYS
"""
}

def increment_with_bounds(self, key, delta=1, min_val=None,
max_val=None, ttl=None):
"""Atomically increment with bounds checking"""
script = self.lua_scripts['increment_with_min_max']
args = [delta, min_val or '', max_val or '', ttl or 0]
return self.redis.eval(script, 1, key, *args)

def get_with_snapshot(self, key):
"""Get counter value with snapshot for consistency"""
# Use Redis transactions for consistent read
pipe = self.redis.pipeline()
pipe.watch(key)
value = pipe.get(key)
pipe.unwatch()
pipe.execute()
return int(value) if value else 0

5. Scalability and Performance​

5.1 Scaling Strategy​

graph TD
A[Traffic Increase] --> B{Request Type Analysis}

B -->|Read Heavy| C[Add Read Replicas]
B -->|Write Heavy| D[Shard Counters]
B -->|Mixed| E[Combined Approach]

C --> C1[2x Replicas per Shard]
C --> C2[Read-Balancing Proxy]
C --> C3[Local Cache Layer]

D --> D1[Key-based Sharding]
D --> D2[Consistent Hashing]
D --> D3[Proxy-based Sharding]

E --> E1[Redis Cluster Mode]
E --> E2[Proxy + Replication]

C1 --> F[Target: 100K RPS]
D1 --> G[Target: 50K WPS]
E1 --> H[Target: 70K OPS]

F --> I[Monitoring & Auto-scale]
G --> I
H --> I

5.2 Sharding Strategies​

StrategyHow It WorksProsConsBest For
Key-basedshard = hash(key) % NSimple, predictableResharding difficultStable key sets
Range-basedKey ranges per shardEfficient range queriesHotspot riskSequential keys
Directory-basedLookup service maps keysFlexible, easy rebalancingSingle point of failureDynamic environments
Consistent HashingHash ring distributionMinimal rehash on scaleComplex implementationCloud auto-scaling

5.3 Performance Benchmarks​

OperationSingle Node3-Node ClusterWith PipelineNotes
INCR80,000 OPS240,000 OPS500,000 OPSAtomic counter
HINCRBY70,000 OPS210,000 OPS450,000 OPSHashed counter
GET100,000 OPS300,000 OPS800,000 OPSSimple read
MULTI/EXEC40,000 OPS120,000 OPSN/ATransaction
EVAL (Lua)60,000 OPS180,000 OPSN/AScript execution

6. High Availability and Disaster Recovery​

6.1 HA Architecture​

graph TB
subgraph "Primary Region (us-east-1)"
subgraph "Availability Zone A"
M1[Master-1<br/>Primary]
S1[Slave-1.1<br/>Sync Replica]
end

subgraph "Availability Zone B"
S2[Slave-1.2<br/>Async Replica]
S3[Slave-1.3<br/>Async Replica]
end

subgraph "Availability Zone C"
P1[Sentinel-1]
P2[Sentinel-2]
P3[Sentinel-3]
end
end

subgraph "Secondary Region (us-west-2)"
DR1[DR Replica-1<br/>Async Cross-Region]
DR2[DR Replica-2]
end

subgraph "Backup Storage"
B1[S3 Daily RDB]
B2[S3 Hourly AOF]
B3[Glacier Weekly]
end

M1 --> S1
M1 --> S2
M1 --> S3
M1 --> DR1

P1 --> M1
P2 --> M1
P3 --> M1

P1 -.->|Quorum Detection| P2
P2 -.->|Quorum Detection| P3

S1 --> B1
DR1 --> B2
B2 --> B3

style M1 fill:#e1f5e1
style DR1 fill:#fff3e0

6.2 Failover Procedures​

# Automated Failover Configuration
failover_policy:
primary_failure:
detection:
method: "sentinel_quorum"
quorum_size: 2
timeout_ms: 10000
health_checks:
- "ping_response"
- "replication_lag"
- "memory_usage"

action:
- "pause_client_connections: 500ms"
- "elect_new_master: [slave-1.1, slave-1.2]"
- "update_dns_ttl: 30"
- "notify_ops: pagerduty_high"
- "update_configuration_store"
- "resume_client_connections"

validation:
- "data_integrity_check"
- "replication_verification"
- "performance_baseline_check"

regional_failover:
trigger: "primary_region_unavailable"
timeout: "5_minutes"
action:
- "activate_dr_replica_reads"
- "promote_dr_to_primary"
- "update_global_dns"
- "notify_stakeholders"

rollback:
condition: "primary_region_recovered_24h"
procedure: "incremental_sync_back"

6.3 Backup and Recovery​

Backup TypeFrequencyRetentionRTORPOStorage
RDB Snapshot6 hours30 days5 min6 hoursS3 Standard
AOF AppendContinuous7 days1 min1 secEBS GP3
Cross-region24 hours90 days15 min24 hoursS3 Cross-region
EncryptedWeekly1 year30 min1 weekS3 + Glacier

7. Monitoring and Observability​

7.1 Key Metrics Dashboard​

quadrantChart
title Redis Counter Service Health Dashboard
x-axis "Latency Impact" --> "Business Impact"
y-axis "Frequency" --> "Severity"

quadrant-1 "Critical: Immediate Action"
quadrant-2 "High: Urgent Attention"
quadrant-3 "Medium: Planned Response"
quadrant-4 "Low: Monitor & Document"

"Memory Usage >90%": [0.85, 0.9]
"Master-Slave Lag >10s": [0.75, 0.6]
"INCR OPS >100K/sec": [0.6, 0.7]
"CPU Usage >80%": [0.7, 0.5]
"Connection Count >10K": [0.5, 0.4]
"Network Throughput >1Gbps": [0.4, 0.3]
"Key Expiration Rate": [0.3, 0.2]
"Cache Hit Ratio <90%": [0.2, 0.1]

7.2 Alerting Rules​

alerting_rules:
critical:
- alert: "RedisMasterDown"
expr: "redis_up{role=\"master\"} == 0"
for: "1m"
annotations:
severity: "critical"
summary: "Redis master node is down"
action: "Check failover status and initiate recovery"

- alert: "HighMemoryUsage"
expr: "redis_memory_used_percentage > 90"
for: "5m"
annotations:
severity: "critical"
summary: "Redis memory usage exceeded 90%"
action: "Investigate memory leak or scale up"

warning:
- alert: "ReplicaLagHigh"
expr: "redis_slave_replication_offset - redis_master_replication_offset > 1000000"
for: "10m"
annotations:
severity: "warning"
summary: "Replication lag exceeds threshold"
action: "Check network and slave performance"

- alert: "CounterIncrementRateDrop"
expr: "rate(redis_command_increment_total[5m]) < 1000"
for: "15m"
annotations:
severity: "warning"
summary: "Counter increment rate dropped significantly"
action: "Check application health and connectivity"

8. Security Implementation​

8.1 Security Layers​

graph TD
A[Client Request] --> B[API Gateway]
B --> C[Authentication]
C --> D[Authorization]
D --> E[Rate Limiting]
E --> F[Counter Service]
F --> G[Redis Proxy]
G --> H[Redis Cluster]

subgraph "Network Security"
I[VPC Isolation]
J[Security Groups]
K[Network ACLs]
L[TLS Encryption]
end

subgraph "Data Security"
M[Encryption at Rest]
N[Encryption in Transit]
O[Key Rotation]
P[Audit Logging]
end

subgraph "Access Control"
Q[IAM Roles]
R[Redis ACL]
S[Token-based Auth]
T[IP Whitelisting]
end

H --> I
H --> J
H --> K
F --> L
H --> M
G --> N
H --> O
B --> P
F --> Q
G --> R
C --> S
B --> T

8.2 Security Configuration​

# Redis security configuration (redis.conf)
requirepass "{{ strong_password }}"
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command CONFIG ""

# Enable ACL
aclfile /etc/redis/users.acl

# Network security
bind 127.0.0.1 ::1
protected-mode yes
port 6379

# TLS configuration
tls-port 6380
tls-cert-file /etc/redis/redis.crt
tls-key-file /etc/redis/redis.key
tls-ca-cert-file /etc/redis/ca.crt

# Security limits
maxclients 10000
timeout 300
tcp-keepalive 300

# Memory security
maxmemory 16gb
maxmemory-policy allkeys-lru

9. Implementation Roadmap​

Phase 1: Foundation (Weeks 1-2)​

  1. Basic Redis Setup

    • Single master with two replicas
    • Basic monitoring (CPU, memory, connections)
    • Simple counter API implementation
  2. Client SDK Development

    • Java/Python/Node.js SDKs
    • Connection pooling and retry logic
    • Basic error handling

Phase 2: Scalability (Weeks 3-4)​

  1. Redis Cluster Deployment

    • 3-shard cluster setup
    • Sharding strategy implementation
    • Load testing and optimization
  2. Advanced Features

    • Lua script optimization
    • Pipeline implementation
    • Connection management improvements

Phase 3: Production Readiness (Weeks 5-6)​

  1. High Availability

    • Sentinel deployment
    • Automated failover testing
    • Cross-region replication setup
  2. Monitoring & Alerting

    • Comprehensive dashboard setup
    • Alerting rules configuration
    • Performance baseline establishment

Phase 4: Optimization (Weeks 7-8)​

  1. Performance Tuning

    • Memory optimization
    • Network configuration
    • OS-level tuning
  2. Security Hardening

    • TLS implementation
    • ACL configuration
    • Audit logging setup

10. Cost Estimation​

ComponentMonthly CostScaling FactorNotes
Redis Instances$1,200Linear with nodesm6g.xlarge (4vCPU, 16GB)
Storage$150$0.10/GB/monthEBS gp3 volumes
Backup Storage$75$0.023/GB/monthS3 Standard + Glacier
Data Transfer$200$0.01-0.02/GBCross-AZ and cross-region
Monitoring$300$50/node/monthCloudWatch + Prometheus
Total Monthly$1,925For 3-node cluster

11. Risks and Mitigations​

RiskProbabilityImpactMitigation Strategy
Data LossLowCriticalRegular backups + cross-region replication
Performance DegradationMediumHighAuto-scaling + caching layer
Security BreachLowCriticalTLS + ACL + regular audits
Single Point of FailureMediumHighMulti-AZ + automated failover
Cost OverrunMediumMediumUsage monitoring + budget alerts
Operational ComplexityHighMediumComprehensive documentation + automation

12. Conclusion​

This design provides a robust, scalable solution for centralized counter management using Redis. The architecture balances performance, reliability, and cost-effectiveness while maintaining strong consistency guarantees.

Next Steps:​

  1. Approval of this design document
  2. Resource allocation for implementation team
  3. Infrastructure provisioning in development environment
  4. Proof-of-concept implementation and validation
  5. Gradual rollout with canary deployment strategy

Success Criteria:​

  • ✅ 99.99% availability for counter operations
  • ✅ <5ms P95 latency for increment operations
  • ✅ Seamless scaling to 100K+ operations per second
  • ✅ Zero data loss in planned failover scenarios
  • ✅ Comprehensive monitoring and alerting coverage

Design Enhancement: Counter Batching Strategy

1. Overview of Counter Batching​

1.1 The Problem Statement​

In high-throughput distributed systems, individual counter operations (INCR, DECR) can create bottlenecks due to:

IssueImpactWithout BatchingWith Batching
Network Round-tripsLatency1 RTT per operation~1 RTT per batch
Redis Command OverheadCPU UtilizationParsing/execution per commandSingle command execution
Connection ContentionThroughput LimitsMultiple connections neededFewer connections
Transaction ManagementComplexityComplex multi-key transactionsSimplified batch operations

1.2 Batching Architecture Impact​

flowchart TD
A[Client Operations] --> B{Batch Strategy Decision}

B -->|Time-Based| C[Buffer for 100ms]
B -->|Size-Based| D[Buffer until 100 ops]
B -->|Hybrid| E[Whichever comes first]

C --> F[Batch Processor]
D --> F
E --> F

subgraph F[Batch Processing Engine]
F1[Batch Aggregation]
F2[Conflict Resolution]
F3[Pipeline Optimization]
F4[Retry Management]
end

F --> G[Batched Redis Operations]
G --> H[Single Pipeline Execution]
H --> I[Results Distribution]
I --> J[Individual Client Responses]

2. Batching Strategy Implementation​

2.1 Multi-Level Batching Architecture​

class MultiLevelCounterBatcher:
def __init__(self, redis_client, config=None):
self.redis = redis_client
self.config = config or {
'client_level': {
'max_size': 50, # Operations per batch
'max_time_ms': 100, # Max batching window
'enabled': True
},
'proxy_level': {
'max_size': 500,
'max_time_ms': 10, # Smaller window for proxies
'enabled': True
},
'redis_level': {
'pipeline_limit': 1000,
'transactional': True
}
}

# Client-side buffer
self.client_buffers = defaultdict(list) # client_id -> operations
self.client_timers = {} # client_id -> timer

# Proxy/Server-side aggregator
self.proxy_buffer = []
self.proxy_timer = None

def increment_batched(self, client_id, key, value=1, **options):
"""Client-side batch request submission"""
operation = {
'type': 'INCR',
'key': key,
'value': value,
'options': options,
'timestamp': time.time(),
'client_id': client_id,
'correlation_id': str(uuid.uuid4())
}

# Add to client buffer
self.client_buffers[client_id].append(operation)

# Start timer if first operation
if client_id not in self.client_timers:
self._start_client_timer(client_id)

# Trigger flush if buffer full
if len(self.client_buffers[client_id]) >= self.config['client_level']['max_size']:
self._flush_client_buffer(client_id)

return operation['correlation_id']

def _flush_client_buffer(self, client_id):
"""Send batched operations to proxy"""
if client_id not in self.client_buffers:
return

operations = self.client_buffers[client_id]
if not operations:
return

# Group operations by key pattern for optimization
grouped_ops = self._group_and_optimize_operations(operations)

# Send to proxy (simulated)
self._send_to_proxy(grouped_ops)

# Clear buffer and timer
self.client_buffers[client_id] = []
if client_id in self.client_timers:
self.client_timers.pop(client_id)

2.2 Smart Batching Algorithms​

class SmartBatchOptimizer:
"""Intelligent batching based on operation patterns"""

@staticmethod
def optimize_batch(operations):
"""
Optimize batch operations for Redis efficiency
Returns: List of optimized command groups
"""
# Group by operation type and key pattern
groups = defaultdict(list)

for op in operations:
# Create grouping key based on operation characteristics
if op['type'] in ['INCR', 'DECR']:
# Group counters by key pattern
key_pattern = op['key'].split(':')[0] # First segment
group_key = f"{op['type']}:{key_pattern}"
elif op['type'] == 'GET':
group_key = "GET:multi"
else:
group_key = f"other:{op['type']}"

groups[group_key].append(op)

# Apply optimization strategies per group
optimized_commands = []

# 1. Counter operations: Combine multiple INCRs on same key
counter_ops = groups.get('INCR:counter', []) + groups.get('DECR:counter', [])
if counter_ops:
optimized_commands.extend(
SmartBatchOptimizer._optimize_counter_operations(counter_ops)
)

# 2. GET operations: Use MGET for multiple keys
get_ops = groups.get('GET:multi', [])
if get_ops:
optimized_commands.append(
SmartBatchOptimizer._optimize_get_operations(get_ops)
)

# 3. Mixed operations in transaction
return optimized_commands

@staticmethod
def _optimize_counter_operations(operations):
"""Optimize counter operations using Lua scripting"""
# Group by exact key
key_operations = defaultdict(int)
for op in operations:
if op['type'] == 'INCR':
key_operations[op['key']] += op['value']
elif op['type'] == 'DECR':
key_operations[op['key']] -= op['value']

# Create Lua script for batched increments
if key_operations:
lua_script = """
local results = {}
for i, key in ipairs(KEYS) do
local increment = tonumber(ARGV[i])
local new_value = redis.call('INCRBY', key, increment)
results[i] = new_value
end
return results
"""

keys = list(key_operations.keys())
args = [key_operations[k] for k in keys]

return [{
'type': 'EVAL',
'script': lua_script,
'keys': keys,
'args': args
}]

return []

@staticmethod
def _optimize_get_operations(operations):
"""Optimize multiple GET operations using MGET"""
keys = [op['key'] for op in operations]
correlation_ids = [op['correlation_id'] for op in operations]

return [{
'type': 'MGET',
'keys': keys,
'metadata': {
'correlation_ids': correlation_ids,
'client_ids': list(set(op['client_id'] for op in operations))
}
}]

2.3 Redis Pipeline Optimization​

class RedisPipelineOptimizer:
"""Optimize Redis pipeline execution"""

def __init__(self, redis_client, max_pipeline_size=1000):
self.redis = redis_client
self.max_pipeline_size = max_pipeline_size
self.pipeline_batches = []

def add_to_pipeline(self, commands):
"""Add commands to pipeline with smart batching"""
for command in commands:
self._add_command_to_appropriate_batch(command)

# Execute if any batch is ready
self._execute_ready_batches()

def _add_command_to_appropriate_batch(self, command):
"""Intelligently assign command to optimal batch"""
# Find suitable batch or create new one
suitable_batch = None

for batch in self.pipeline_batches:
if self._can_add_to_batch(batch, command):
suitable_batch = batch
break

if not suitable_batch:
suitable_batch = self._create_new_batch(command['type'])
self.pipeline_batches.append(suitable_batch)

suitable_batch['commands'].append(command)
suitable_batch['size'] += self._estimate_command_size(command)

def _can_add_to_batch(self, batch, command):
"""Check if command can be added to existing batch"""
# Check size limits
if batch['size'] + self._estimate_command_size(command) > self.max_pipeline_size:
return False

# Check compatibility (e.g., don't mix reads and writes in same pipeline)
if batch['type'] == 'read' and command['type'] in ['INCR', 'DECR', 'SET']:
return False
if batch['type'] == 'write' and command['type'] == 'GET':
return False

# Check timing - don't delay reads for too long
if batch['type'] == 'read' and time.time() - batch['created'] > 0.05: # 50ms max for reads
return False

return True

def _execute_ready_batches(self):
"""Execute batches that are ready"""
for batch in list(self.pipeline_batches):
if self._should_execute_batch(batch):
self._execute_batch(batch)
self.pipeline_batches.remove(batch)

def _should_execute_batch(self, batch):
"""Determine if batch should be executed"""
# Size-based trigger
if batch['size'] >= self.max_pipeline_size * 0.8: # 80% full
return True

# Time-based trigger
max_wait = 0.1 if batch['type'] == 'write' else 0.05 # 100ms vs 50ms
if time.time() - batch['created'] > max_wait:
return True

# Priority-based trigger (high priority operations)
if any(cmd.get('priority') == 'high' for cmd in batch['commands']):
return True

return False

def _execute_batch(self, batch):
"""Execute a batch of commands in Redis pipeline"""
pipe = self.redis.pipeline(transaction=(batch['type'] == 'write'))

# Add all commands to pipeline
for cmd in batch['commands']:
if cmd['type'] == 'INCR':
pipe.incrby(cmd['key'], cmd['value'])
elif cmd['type'] == 'GET':
pipe.get(cmd['key'])
elif cmd['type'] == 'MGET':
pipe.mget(cmd['keys'])
elif cmd['type'] == 'EVAL':
pipe.eval(cmd['script'], len(cmd['keys']), *cmd['keys'], *cmd['args'])

# Execute and handle results
try:
results = pipe.execute()
self._distribute_results(batch['commands'], results)
except Exception as e:
self._handle_batch_error(batch['commands'], e)

3. Batching Configuration Strategies​

3.1 Adaptive Batching Configuration​

# batching-config.yaml
adaptive_batching:
enabled: true
monitor_interval_seconds: 30

strategies:
time_based:
enabled: true
initial_window_ms: 100
min_window_ms: 10
max_window_ms: 500
adjustment_factor: 0.2
metrics:
- latency_p95
- throughput_ops

size_based:
enabled: true
initial_batch_size: 50
min_batch_size: 10
max_batch_size: 1000
adjustment_factor: 0.3
metrics:
- memory_usage
- network_io

priority_based:
enabled: true
priority_levels:
realtime:
max_delay_ms: 5
max_batch_size: 10
high:
max_delay_ms: 20
max_batch_size: 50
normal:
max_delay_ms: 100
max_batch_size: 200
low:
max_delay_ms: 500
max_batch_size: 1000

optimization_rules:
- name: "combine_same_key_operations"
enabled: true
condition: "operations.same_key_count > 3"
action: "combine_to_single_operation"

- name: "separate_read_write"
enabled: true
condition: "batch.contains_mixed_operations"
action: "split_by_operation_type"

- name: "adaptive_window_adjustment"
enabled: true
condition: "latency_increased > 20%"
action: "reduce_batch_window_by 30%"

- name: "emergency_flush"
enabled: true
condition: "system_memory_usage > 85%"
action: "flush_all_batches_immediately"

monitoring:
metrics_to_track:
- batch_size_distribution
- batch_latency_percentiles
- operations_per_second
- batch_efficiency_ratio
- memory_footprint_per_batch

alerts:
- metric: "batch_efficiency_ratio"
threshold: "< 0.7"
severity: "warning"
action: "review_batching_strategy"

- metric: "batch_latency_p99"
threshold: "> 200ms"
severity: "critical"
action: "reduce_batch_size_window"

3.2 Batching Performance Profiles​

class BatchingProfiler:
"""Profile and optimize batching strategies"""

PROFILES = {
'low_latency': {
'batch_window_ms': 10,
'max_batch_size': 20,
'pipeline_size': 50,
'priority': 'realtime',
'optimizations': ['separate_read_write', 'minimal_buffering']
},
'high_throughput': {
'batch_window_ms': 200,
'max_batch_size': 500,
'pipeline_size': 1000,
'priority': 'normal',
'optimizations': ['combine_same_key_operations', 'max_batching']
},
'balanced': {
'batch_window_ms': 100,
'max_batch_size': 100,
'pipeline_size': 500,
'priority': 'high',
'optimizations': ['adaptive_window', 'smart_grouping']
},
'conservative_memory': {
'batch_window_ms': 50,
'max_batch_size': 50,
'pipeline_size': 100,
'priority': 'normal',
'optimizations': ['small_batches', 'frequent_flush']
}
}

@classmethod
def get_optimal_profile(cls, metrics):
"""
Determine optimal batching profile based on system metrics
"""
# Analyze current metrics
latency_sensitive = metrics.get('latency_p95', 0) < 50 # ms
high_throughput_needed = metrics.get('ops_per_second', 0) > 10000
memory_constrained = metrics.get('memory_usage_percent', 0) > 70

if latency_sensitive and not high_throughput_needed:
return cls.PROFILES['low_latency']
elif high_throughput_needed and not latency_sensitive:
return cls.PROFILES['high_throughput']
elif memory_constrained:
return cls.PROFILES['conservative_memory']
else:
return cls.PROFILES['balanced']

@classmethod
def create_custom_profile(cls, workload_pattern):
"""
Create custom batching profile based on workload analysis
"""
analysis = cls.analyze_workload(workload_pattern)

return {
'batch_window_ms': cls._calculate_optimal_window(analysis),
'max_batch_size': cls._calculate_optimal_batch_size(analysis),
'pipeline_size': cls._calculate_pipeline_size(analysis),
'priority': cls._determine_priority(analysis),
'optimizations': cls._select_optimizations(analysis)
}

@staticmethod
def analyze_workload(operations_log):
"""Analyze operation patterns for optimal batching"""
analysis = {
'operation_types': defaultdict(int),
'key_patterns': defaultdict(int),
'inter_arrival_times': [],
'temporal_patterns': defaultdict(int)
}

for op in operations_log:
analysis['operation_types'][op['type']] += 1
key_pattern = ':'.join(op['key'].split(':')[:2]) # First two segments
analysis['key_patterns'][key_pattern] += 1

return analysis

4. Error Handling and Consistency​

4.1 Atomic Batch Processing​

class AtomicBatchProcessor:
"""Ensure atomicity and consistency for batch operations"""

def process_batch_atomically(self, batch_operations):
"""
Process batch with atomic guarantees
Returns: (success, results, failed_operations)
"""
# 1. Validate all operations
validation_results = self._validate_batch(batch_operations)
if not validation_results['valid']:
return False, None, validation_results['invalid_ops']

# 2. Create backup for rollback
backup_data = self._create_backup(batch_operations)

# 3. Execute in Redis transaction
try:
pipe = self.redis.pipeline(transaction=True)

# Add WATCH for keys involved
for key in self._get_unique_keys(batch_operations):
pipe.watch(key)

# Add all operations
for op in batch_operations:
self._add_to_pipeline(pipe, op)

# Execute
results = pipe.execute()

# 4. Verify results
if self._verify_results(batch_operations, results):
# Success - return results
return True, results, []
else:
# Verification failed - rollback
self._rollback_changes(backup_data)
return False, None, batch_operations

except redis.WatchError:
# Concurrent modification detected
self._handle_concurrent_modification(batch_operations)
return False, None, batch_operations

except Exception as e:
# Other errors - rollback
self._rollback_changes(backup_data)
self._log_batch_failure(batch_operations, e)
return False, None, batch_operations

def _create_backup(self, operations):
"""Create backup of current values for rollback"""
backup = {}
unique_keys = self._get_unique_keys(operations)

# Use MGET for efficiency
current_values = self.redis.mget(unique_keys)

for key, value in zip(unique_keys, current_values):
backup[key] = value

return backup

def _rollback_changes(self, backup_data):
"""Rollback changes using backup data"""
pipe = self.redis.pipeline(transaction=False)

for key, original_value in backup_data.items():
if original_value is None:
# Key didn't exist before
pipe.delete(key)
else:
# Restore original value
pipe.set(key, original_value)

pipe.execute()

def _handle_partial_batch_failure(self, batch_operations, results):
"""
Handle case where some operations in batch succeed and some fail
"""
successful_ops = []
failed_ops = []

for i, (op, result) in enumerate(zip(batch_operations, results)):
if isinstance(result, Exception):
failed_ops.append({
'operation': op,
'error': result,
'index': i
})
else:
successful_ops.append({
'operation': op,
'result': result,
'index': i
})

if failed_ops:
# Attempt to compensate or rollback successful operations
if self.config.get('compensation_enabled', False):
self._compensate_successful_ops(successful_ops)
else:
self._rollback_successful_ops(successful_ops, batch_operations)

return successful_ops, failed_ops

4.2 Retry and Compensation Logic​

class BatchRetryManager:
"""Manage retries for failed batch operations"""

def __init__(self, max_retries=3, backoff_factor=2):
self.max_retries = max_retries
self.backoff_factor = backoff_factor
self.retry_queue = []
self.failed_operations = {}

def schedule_retry(self, operation, error, attempt=1):
"""Schedule operation for retry"""
if attempt > self.max_retries:
self._move_to_failed(operation, error)
return

# Calculate delay with exponential backoff and jitter
delay = self._calculate_retry_delay(attempt)

retry_item = {
'operation': operation,
'error': error,
'attempt': attempt,
'scheduled_at': time.time() + delay,
'retry_id': str(uuid.uuid4())
}

heapq.heappush(self.retry_queue, (retry_item['scheduled_at'], retry_item))

def _calculate_retry_delay(self, attempt):
"""Calculate retry delay with exponential backoff and jitter"""
base_delay = 100 # ms
exponential_delay = base_delay * (self.backoff_factor ** (attempt - 1))

# Add jitter (±20%)
jitter = random.uniform(-0.2, 0.2) * exponential_delay

return (exponential_delay + jitter) / 1000 # Convert to seconds

def process_retry_queue(self):
"""Process scheduled retries"""
current_time = time.time()

while self.retry_queue and self.retry_queue[0][0] <= current_time:
scheduled_at, retry_item = heapq.heappop(self.retry_queue)

try:
# Attempt retry
result = self._execute_retry(retry_item['operation'])

# Success - notify original caller if possible
self._notify_success(retry_item, result)

except Exception as e:
# Retry failed - schedule next attempt or give up
if retry_item['attempt'] >= self.max_retries:
self._move_to_failed(retry_item['operation'], e)
else:
self.schedule_retry(
retry_item['operation'],
e,
retry_item['attempt'] + 1
)

def _execute_retry(self, operation):
"""Execute retry with appropriate strategy"""
# Different strategies based on error type
error_type = operation.get('last_error_type')

if error_type == 'network':
# For network errors, try different Redis node
return self._retry_with_node_failover(operation)
elif error_type == 'timeout':
# For timeouts, reduce batch size or increase timeout
return self._retry_with_adjusted_parameters(operation)
elif error_type == 'contention':
# For contention, add random delay and retry
time.sleep(random.uniform(0.01, 0.1))
return self._retry_with_backoff(operation)
else:
# Default retry
return self._execute_operation(operation)

5. Monitoring and Metrics for Batching​

5.1 Batching-Specific Metrics​

class BatchingMetricsCollector:
"""Collect and analyze batching performance metrics"""

METRICS = {
'efficiency': {
'batch_utilization_ratio': 'filled_batches / total_batches',
'operations_per_batch': 'total_ops / total_batches',
'command_compression_ratio': 'individual_commands / batched_commands'
},
'performance': {
'batch_latency_p95': '95th percentile batch processing time',
'batch_throughput': 'batches_processed_per_second',
'redis_command_reduction': '(individual_calls - batched_calls) / individual_calls'
},
'quality': {
'batch_success_rate': 'successful_batches / total_batches',
'partial_failure_rate': 'batches_with_partial_failures / total_batches',
'retry_rate': 'operations_requiring_retry / total_operations'
},
'resource': {
'batch_memory_footprint': 'memory_used_by_buffers',
'batch_buffer_count': 'number_of_active_buffers',
'batch_age_distribution': 'histogram_of_batch_ages'
}
}

def collect_metrics(self):
"""Collect comprehensive batching metrics"""
metrics = {}

# Efficiency metrics
metrics['batch_utilization'] = self._calculate_batch_utilization()
metrics['compression_ratio'] = self._calculate_compression_ratio()

# Performance metrics
metrics['latency_improvement'] = self._calculate_latency_improvement()
metrics['throughput_improvement'] = self._calculate_throughput_improvement()

# Resource metrics
metrics['memory_savings'] = self._calculate_memory_savings()
metrics['network_savings'] = self._calculate_network_savings()

return metrics

def _calculate_batch_utilization(self):
"""Calculate how efficiently batches are being used"""
total_batches = self.stats['batches_created']
full_batches = self.stats['batches_flushed_by_size']
time_flushed_batches = self.stats['batches_flushed_by_time']

if total_batches == 0:
return 0

# Ideal: batches filled by size, not time
utilization = full_batches / total_batches
return utilization

def _calculate_compression_ratio(self):
"""Calculate command compression ratio"""
individual_commands = self.stats['individual_operations']
batched_commands = self.stats['batched_executions']

if batched_commands == 0:
return 0

return (individual_commands - batched_commands) / individual_commands

5.2 Dashboard Visualization​

graph TD
subgraph "Batching Performance Dashboard"
A[Batch Efficiency] --> A1["Utilization: 85%"]
A --> A2["Compression Ratio: 65%"]
A --> A3["Avg Batch Size: 78 ops"]

B[Performance Impact] --> B1["Latency Reduction: 40%"]
B --> B2["Throughput Increase: 3.2x"]
B --> B3["Redis Load Reduction: 55%"]

C[Resource Usage] --> C1["Memory Buffer: 45MB"]
C --> C2["Active Batches: 12"]
C --> C3["Avg Batch Age: 45ms"]

D[Quality Metrics] --> D1["Success Rate: 99.8%"]
D --> D2["Partial Failure: 0.2%"]
D --> D3["Retry Rate: 0.5%"]
end

subgraph "Batch Size Distribution"
E["<10 ops: 5%"] --> F
E1["10-50 ops: 25%"] --> F
E2["50-100 ops: 40%"] --> F[Histogram]
E3["100-200 ops: 20%"] --> F
E4[">200 ops: 10%"] --> F
end

subgraph "Real-time Monitoring"
G[Current Batch Buffer] --> G1["Client Buffers: 8"]
G --> G2["Proxy Buffer: 1"]
G --> G3["Pending Operations: 342"]

H[Next Flush Prediction] --> H1["Time-based: 32ms"]
H --> H2["Size-based: 18 ops needed"]
end

6. Implementation Roadmap for Batching​

Phase 1: Basic Batching (Week 1-2)​

  1. Client-side buffering

    • Time-based batching (100ms windows)
    • Size-based batching (50 operations)
    • Simple flush mechanisms
  2. Basic Redis pipelining

    • Pipeline execution of batched commands
    • Error handling for batch failures
    • Basic metrics collection

Phase 2: Advanced Batching (Week 3-4)​

  1. Smart batching strategies

    • Operation type-based grouping
    • Key pattern optimization
    • Adaptive batch sizing
  2. Consistency guarantees

    • Atomic batch processing
    • Partial failure handling
    • Retry mechanisms

Phase 3: Production Optimization (Week 5-6)​

  1. Performance optimization

    • Adaptive window sizing
    • Memory usage optimization
    • Network efficiency improvements
  2. Monitoring and alerting

    • Comprehensive metrics dashboard
    • Automated optimization suggestions
    • Alerting for batch inefficiencies

Phase 4: Advanced Features (Week 7-8)​

  1. Predictive batching

    • Machine learning for optimal batch sizing
    • Workload pattern recognition
    • Proactive optimization
  2. Multi-level batching hierarchy

    • Client → Proxy → Redis optimization
    • Cross-service batch coordination
    • Global batch optimization

7. Performance Expectations with Batching​

MetricWithout BatchingWith BatchingImprovement
Operations per second10,000 ops45,000 ops4.5x
Average latency15ms8ms47% reduction
Redis connections5005090% reduction
Network packets10,000/min2,000/min80% reduction
Redis CPU usage65%40%38% reduction
Memory overheadHigh (per connection)Low (buffered)60-80% savings

8. Best Practices for Batching Implementation​

  1. Start Conservative: Begin with small batch sizes and increase gradually
  2. Monitor Aggressively: Track both efficiency and side effects
  3. Implement Graceful Degradation: Fall back to non-batched mode if issues arise
  4. Test with Real Workloads: Simulate production traffic patterns
  5. Implement Circuit Breakers: Prevent cascading failures
  6. Document Batching Behavior: Ensure all teams understand the trade-offs
  7. Plan for Rollback: Have a clear procedure to disable batching if needed

9. Conclusion​

Counter batching is a critical optimization strategy for high-throughput Redis-based counter systems. When implemented correctly, it can provide:

  • 4-5x throughput improvement
  • 40-60% latency reduction
  • 80-90% connection count reduction
  • Significant Redis resource savings

The key to successful batching implementation is starting with simple strategies, monitoring comprehensively, and gradually introducing more sophisticated optimizations based on actual workload patterns and performance metrics.

The multi-level batching architecture (client → proxy → Redis) combined with smart optimization algorithms provides the flexibility to handle diverse workload patterns while maintaining strong consistency guarantees and excellent performance characteristics.

Appendices​

Appendix A: Redis Configuration Template​

[Full redis.conf template with optimizations for counter workloads]

Appendix B: Load Testing Scripts​

[Python scripts for simulating high-volume counter operations]

Appendix C: Client SDK Examples​

[Complete code examples for Java, Python, and Node.js SDKs]

Appendix D: Disaster Recovery Runbook​

[Step-by-step procedures for various failure scenarios]

Appendix E: Performance Tuning Guide​

[Detailed instructions for OS, network, and Redis tuning]


Redis: From Zero to Hero - The Brief Guide

· 阅读需 10 分钟

Executive Summary​

This document provides a comprehensive, structured understanding of Redis architecture and core concepts, distilled from in-depth technical discussions. It focuses on building accurate mental models of Redis' internal workings, persistence mechanisms, replication strategies, and clustering architecture.

Table of Contents​

  1. Redis Fundamentals
  2. Persistence: RDB vs AOF
  3. Replication Architecture
  4. High Availability with Sentinel
  5. Redis Cluster & Sharding
  6. Data Movement & Consistency

1. Redis Fundamentals​

1.1 What is Redis?​

Redis (REmote DIctionary Server) is an open-source, in-memory data structure store that can be used as a database, cache, message broker, and streaming engine. It provides sub-millisecond response times by storing all data in RAM.

graph TB
subgraph "Redis Core Philosophy"
A[In-Memory Storage] --> B[Single-Threaded Execution]
B --> C[Rich Data Structures]
C --> D[Optional Durability]
end

subgraph "Primary Use Cases"
E[Caching Layer] --> F[Session Store]
G[Real-time Analytics] --> H[Message Queue]
I[Leaderboards] --> J[Rate Limiting]
end

1.2 Single-Threaded Advantage​

Redis uses a single-threaded event loop model which provides atomic operations without locks:

sequenceDiagram
participant Client1
participant Client2
participant Redis

Client1->>Redis: INCR counter
Client2->>Redis: INCR counter
Note over Redis: Sequential processing<br/>No race conditions
Redis-->>Client1: 101
Redis-->>Client2: 102

Key Benefits:

  • No lock contention or synchronization overhead
  • Predictable latency patterns
  • Naturally atomic operations like INCR

1.3 Data Model​

Redis is not a table store but rather a huge hash map in memory with rich data types:

Data StructureConceptual ModelPrimary Use Cases
StringBasic value/counterPage views, rate limiting
HashObject/entity storageUser profiles, configurations
ListQueue/stackMessage queues, activity feeds
SetUnique collectionTags, friends lists
Sorted SetRanking systemLeaderboards, time-series
StreamAppend-only logEvent sourcing, message streams

2. Persistence: RDB vs AOF​

2.1 RDB (Redis Database File) - Snapshotting​

RDB is a point-in-time binary snapshot of Redis memory, not a runtime state or memory block.

sequenceDiagram
participant Client as Clients
participant Main as Redis Main Process
participant Fork as Forked Child Process
participant Disk as Disk Storage

Note over Client,Main: Normal Operation
Client->>Main: Write Operations
Main->>Main: Update in memory
Main-->>Client: Responses

Note over Main,Fork: RDB Snapshot Triggered
Main->>Fork: Fork() - Create child process
Note over Fork: Copy-on-Write<br/>Child gets memory snapshot

par Main Process Continues
Client->>Main: More writes
Main->>Main: Update memory
Main-->>Client: Responses
and Child Process Writes
Fork->>Fork: Iterate through dataset
Fork->>Disk: Write RDB file
Disk-->>Fork: Write complete
Fork->>Main: Signal completion
end

Main->>Disk: Atomic rename temp→dump.rdb

Key RDB Characteristics:

  • File format: Binary representation of memory at snapshot time
  • Creation: Forked child process writes to disk (copy-on-write)
  • Use cases: Fast restarts, backups, replication bootstrap
  • Risk: Potential data loss between snapshots

2.2 AOF (Append Only File) - Write-Ahead Log​

AOF logs every write operation received by the server for durability:

graph TB
A[Write Command] --> B[Execute in Memory]
B --> C[Append to AOF Buffer]
C --> D{Sync Policy}

D --> E[appendfsync always]
D --> F[appendfsync everysec]
D --> G[appendfsync no]

E --> H["Write + Fsync every command<br/>Durability: Excellent<br/>Performance: Poor"]
F --> I["Write every command<br/>Fsync every second<br/>Balance: Good (Recommended)"]
G --> J["Write only<br/>OS decides fsync<br/>Performance: Excellent<br/>Durability: Poor"]

2.3 RDB + AOF Hybrid Approach​

Memory is the source of truth - both RDB and AOF are derived artifacts:

graph TD
Memory["Live In-Memory Data"] --> RDB["Periodic RDB Snapshot"]
Memory --> AOF["Continuous AOF Append"]

RDB --> Restart1[Fast Recovery]
AOF --> Restart2[Durable Recovery]

subgraph "Restart Priority"
AOFExists{AOF exists?}
AOFExists -->|Yes| LoadAOF
AOFExists -->|No| LoadRDB
end

Production Recommendation: Enable both RDB (for speed) and AOF (for durability) with appendfsync everysec.


3. Replication Architecture​

3.1 Master-Replica Model​

Each Redis instance has one role: master OR replica, typically deployed one instance per node for isolation.

graph TD
subgraph "Production Deployment"
Node1[Node 1] --> MasterA[Master Instance A]
Node2[Node 2] --> ReplicaA1[Replica of A]
Node3[Node 3] --> ReplicaA2[Replica of A]
end

MasterA -->|Async Replication| ReplicaA1
MasterA -->|Async Replication| ReplicaA2

subgraph "Traffic Pattern"
Write[Write Requests] --> MasterA
Read1[Read Requests] --> ReplicaA1
Read2[Read Requests] --> ReplicaA2
end

3.2 Replication Process​

No separate sync daemon - replication is handled by Redis processes themselves:

sequenceDiagram
participant Client
participant Master
participant Replica

Note over Replica,Master: Initial Sync (Full Resynchronization)
Replica->>Master: SYNC command
Master->>Master: BGSAVE to create RDB
Master->>Replica: Send RDB file
Master->>Replica: Send buffered writes during BGSAVE
Replica->>Replica: Load RDB + replay writes

Note over Client,Master: Continuous Replication
Client->>Master: Write command
Master->>Master: Execute in memory
Master->>Replica: Propagate command
Master-->>Client: Response
Replica->>Replica: Execute command

Replication Buffer & Backlog:

  • Replication buffer: Commands not yet acknowledged by replicas
  • Replication backlog: Circular buffer for reconnecting replicas
  • Key metrics: master_repl_offset vs replica_offset for lag calculation

4. High Availability with Sentinel​

4.1 Sentinel Architecture​

Redis Sentinel is a separate process that monitors Redis instances and provides automatic failover:

graph TB
subgraph "Redis Data Layer"
A[Master] --> B[Replica 1]
A --> C[Replica 2]
end

subgraph "Sentinel Cluster (Odd Number Recommended)"
D[Sentinel 1] --> E[Sentinel 2]
D --> F[Sentinel 3]
E --> F
end

D --> A
E --> A
F --> A
D --> B
E --> B
F --> B
D --> C
E --> C
F --> C

subgraph "Gossip Protocol"
G[Sentinel Communication] --> H["Publish/Subscribe<br/>Failure Detection<br/>Leader Election"]
end

4.2 Why Odd Number of Sentinels?​

Quorum and fault tolerance prevent split-brain scenarios:

graph TD
subgraph "3-Sentinel Cluster"
A[Sentinel A] --> Vote[Majority Vote]
B[Sentinel B] --> Vote
C[Sentinel C] --> Vote
end

Vote --> D["Quorum = floor(3/2) + 1 = 2"]
D --> E["Can tolerate 1 Sentinel failure"]

subgraph "Failure Tolerance"
F[Total Sentinels] --> G[Can Fail]
3 --> H[1]
5 --> I[2]
7 --> J[3]
end

4.3 Failover Process​

sequenceDiagram
participant S1 as Sentinel 1
participant S2 as Sentinel 2
participant S3 as Sentinel 3
participant OldM as Old Master
participant NewM as New Master (promoted replica)

S1->>OldM: Ping (no response)
S1->>S2: Report failure
S1->>S3: Report failure

Note over S1,S3: Quorum achieved (2/3 agree)

S1->>S1: Elect self as leader (Raft consensus)

S1->>NewM: SLAVEOF NO ONE
NewM->>S1: Promotion successful

S1->>Other Replicas: SLAVEOF new-master
S1->>All Sentinels: Update configuration

Note over S1,NewM: Clients rediscover new master via Sentinel

5. Redis Cluster & Sharding​

5.1 Cluster Architecture Fundamentals​

Redis Cluster has multiple masters, each responsible for a subset of hash slots:

graph TB
subgraph "Redis Cluster - 3 Masters"
A[Master 1<br/>Slots 0-5460] --> D[Replica 1.1]
B[Master 2<br/>Slots 5461-10922] --> E[Replica 2.1]
C[Master 3<br/>Slots 10923-16383] --> F[Replica 3.1]
end

subgraph "Hash Slot Logic"
G[Key: user:123] --> H["CRC16('user:123')<br/>= 948293"]
H --> I["Slot = 948293 % 16384<br/>= 5793"]
I --> J["Slot 5793 ∈ Master 1's range"]
end

A <--> B
B <--> C
A <--> C

subgraph "Cluster Bus"
K[Gossip Protocol] --> L["Port: Redis port + 10000<br/>e.g., 6379 → 16379"]
end

5.2 Key Concepts Clarified​

Slots are logical, not physical - they exist only as routing metadata:

ConceptDefinitionPhysical Existence
SlotLogical hash bucket (0-16383)No - only metadata
ShardMaster + its assigned slotsYes - Redis instance
Key-Value PairActual dataYes - in RAM/RDB
Cluster MetadataSlot → node mappingYes - in nodes.conf

5.3 Slot Assignment & Migration​

Slots are explicitly assigned, not auto-calculated by modulo:

sequenceDiagram
participant Admin as Administrator
participant A as Master A
participant B as Master B
participant Client

Admin->>A: CLUSTER SETSLOT 5793 MIGRATING B
Admin->>B: CLUSTER SETSLOT 5793 IMPORTING A

Note over A,B: Migration in progress

Client->>A: GET user:123 (slot 5793)
A-->>Client: -ASK 5793 B (temporary redirect)

A->>B: MIGRATE "user:123" value TTL
B->>B: Store in RAM

Admin->>Cluster: CLUSTER SETSLOT 5793 NODE B

Client->>A: GET user:123
A-->>Client: -MOVED 5793 B (permanent redirect)
Client->>B: GET user:123
B-->>Client: value

5.4 Adding New Masters​

Redis does NOT auto-rebalance when adding new masters:

flowchart TD
A[Add New Master Node] --> B[Master joins with ZERO slots]
B --> C[Manual/scripted resharding required]

C --> D[Explicit slot migration]
D --> E[Gradual key movement]
E --> F[Updated cluster metadata]

subgraph "Resharding Tools"
G["redis-cli --cluster reshard"]
H["--cluster-from <old-master>"]
I["--cluster-to <new-master>"]
J["--cluster-slots <count>"]
end

Why no auto-rebalance? Avoids massive uncontrolled data movement and potential service disruption.


6. Data Movement & Consistency​

6.1 Migration and Persistence Interaction​

Slot migration moves key-value pairs in memory, not RDB files:

graph TD
subgraph "Before Migration"
A1[Master A RAM] --> B1["Key K in slot 5793"]
A1 --> C1[Master A RDB<br/>Contains K]
end

subgraph "During Migration"
A2[Master A] -->|MIGRATE K| D[Master B RAM]
D --> E[Master B stores K]
A2 --> F[Delete K from RAM]
end

subgraph "After Migration"
A3[Master A RDB] --> G["Still contains K (stale)"]
D2[Master B RAM] --> H["Now contains K"]
D2 --> I["Next RDB will include K"]
end

subgraph "Crash Recovery Scenario"
J[Master A crashes] --> K[Restart + load RDB]
K --> L[RAM contains stale K]
M[Cluster handshake] --> N["Slot 5793 → Master B"]
L --> O[Mark K as orphan/delete]
end

6.2 Critical Principles​

  1. Key → Slot mapping is immutable: CRC16(key) % 16384 always yields same slot
  2. Slot → Node mapping is mutable: Changed during migration
  3. RDB is local snapshot: Never shared or merged between nodes
  4. Cluster metadata overrides RDB: On restart, ownership checked against cluster state

6.3 Hash Tags for Co-location​

Force keys to same slot using {} syntax for multi-key operations:

# These hash to same slot (hash input = "123")
user:{123}:profile
user:{123}:orders
user:{123}:settings

# Enables:
MGET user:{123}:profile user:{123}:orders
MULTI/EXEC transactions
Lua scripts accessing multiple keys

6.4 Memory Management​

Redis must keep all active data in RAM with explicit strategies for limits:

StrategyMechanismUse Case
Eviction Policiesmaxmemory-policy (LRU, random, etc.)Caching scenarios
Horizontal ScalingAdd more masters + reshardGrowing datasets
Redis on FlashSSD-backed (Enterprise)Large cold data

Never: Automatic disk swapping or silent data eviction without configured policy.


Summary: Redis Mental Model​

Core Architecture Truths​

  1. Single-threaded with forked persistence: Main thread handles commands, child processes for RDB/AOF
  2. RAM as source of truth: Persistence is for recovery, not runtime operation
  3. Logical slot partitioning: 16384 slots provide stable indirection layer
  4. Decentralized clustering: No central controller, nodes coordinate via gossip
  5. Explicit data movement: No automatic rebalancing, all migrations are operator-controlled

Production Implications​

  • Design keys carefully: Natural distribution avoids hotspots
  • Use hash tags judiciously: Only when multi-key operations required
  • Monitor slot distribution: Ensure even load across masters
  • Plan resharding carefully: Gradual migration during low traffic
  • Always have replicas: Protect against data loss during migration windows

Final Unified Model​

Application → Key → CRC16 → Slot (0-16383) → Cluster Metadata → Master Node → RAM → RDB/AOF
↑
Replicas (sync)

This model ensures predictable performance, linear scalability, and operational control while maintaining Redis' signature sub-millisecond response times.

Redis: From Zero to Hero - The Complete Guide

· 阅读需 42 分钟

📘 Introduction to Redis​

Redis (REmote DIctionary Server) is an open-source, in-memory data structure store that can be used as a database, cache, message broker, and streaming engine. Created by Salvatore Sanfilippo in 2009, Redis supports various data structures and provides high performance with sub-millisecond response times.

graph TD
A[Redis Use Cases] --> B[Caching Layer]
A --> C[Session Store]
A --> D[Message Queue]
A --> E[Real-time Analytics]
A --> F[Leaderboards]
A --> G[Geospatial Indexing]

B --> B1[Accelerate Database Queries]
B --> B2[Reduce Backend Load]

C --> C1[User Session Data]
C --> C2[Shopping Carts]

D --> D1[Pub/Sub Messaging]
D --> D2[Stream Processing]

1. Redis Architecture Fundamentals​

1.1 Single-Threaded Event Loop Architecture​

Redis uses a single-threaded event loop model that processes commands sequentially. This design eliminates thread synchronization overhead and ensures atomic operations.

flowchart TD
A[Client 1 Request] --> B[Socket Queue]
C[Client 2 Request] --> B
D[Client N Request] --> B

B --> E[Event Demultiplexer<br/>epoll/kqueue/select]
E --> F[Single Threaded Event Loop]

F --> G[Command Parser]
G --> H[Command Processor]

subgraph I[Memory Operations]
H --> J[String Operations]
H --> K[Hash Operations]
H --> L[Set Operations]
H --> M[Sorted Set Operations]
H --> N[List Operations]
end

I --> O[Response Formatter]
O --> P[Send to Clients]

Q[Background Threads] --> R[RDB Persistence]
Q --> S[AOF Rewriting]
Q --> T[Memory Management]

Key Architectural Insights:

  • Single-threaded core: All commands execute in sequence
  • Non-blocking I/O: Uses multiplexing for network operations
  • Background threads: Handle persistence, UNLINK operations
  • Atomic operations: Guaranteed by single-threaded execution

1.2 Memory Structure and Storage​

Redis stores all data in RAM but can persist to disk. Understanding memory allocation is crucial for performance tuning.

┌─────────────────────────────────────────────────────────────┐
│ Redis Memory Layout │
├─────────────────────────────────────────────────────────────┤
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Redis Object │ │
│ │ ┌─────────────┐ ┌──────────────────────────────┐ │ │
│ │ │ type:4 │ │ encoding:4 │ │ │
│ │ │ lru:24 │ │ refcount:4 │ │ │
│ │ │ *ptr │ │ *ptr (to SDS or others) │ │ │
│ │ └─────────────┘ └──────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Simple Dynamic String │ │
│ │ ┌─────────────┬─────────────┬─────────────┬───────┐ │ │
│ │ │ len │ alloc │ flags │ buf │ │ │
│ │ │ (4 bytes) │ (4 bytes) │ (1 byte) │ │ │ │
│ │ └─────────────┴─────────────┴─────────────┴───────┘ │ │
│ └───────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Hash Table Structure │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ dict │ │ ht[0] │ │ ht[1] │ │ │
│ │ │ ┌───────┐ │ │ ┌───────┐ │ │ ┌───────┐ │ │ │
│ │ │ │size │ │ │ │table │──┼───▶│entry │ │ │ │
│ │ │ │used │ │ │ │sizemask│ │ │ │key │ │ │ │
│ │ │ └───────┘ │ │ └───────┘ │ │ │value │ │ │ │
│ │ │ │ │ │ │ │next │ │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └───────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Memory Optimization Techniques:

  • Encoding optimization: Small integers stored directly in pointer
  • String optimizations: EMBSTR encoding for small strings (<44 bytes)
  • Hash optimization: ziplist encoding for small hashes
  • Memory defragmentation: ACTIVE_DEFRAG enabled in config

2. Core Data Structures Deep Dive​

2.1 Strings - Beyond Simple Key-Value​

graph LR
subgraph "String Operations & Use Cases"
A[String Type] --> B[Simple Cache]
A --> C[Counters]
A --> D[Bit Operations]
A --> E[Serialized Objects]

B --> B1[SET/GET with EX/NX options]
C --> C1[INCR/DECR atomic operations]
D --> D1[SETBIT/GETBIT/BITCOUNT]
E --> E1[SET JSON/protobuf data]
end

subgraph "String Memory Layout"
F[RedisObject] --> G[SDS Header]
G --> H[String Buffer]

I["embstr (≤44 bytes)"] --> J["Single allocation<br/>RedisObj + SDS + data"]
K["raw (>44 bytes)"] --> L["Separate allocations"]
end

Advanced String Operations:

# Atomic counter with expiration
SET user:1000:visits 0 EX 3600 NX
INCR user:1000:visits

# Bit operations for feature flags
SETBIT features:user:1000 0 1 # Feature A enabled
SETBIT features:user:1000 3 1 # Feature D enabled
BITCOUNT features:user:1000 # Count enabled features

# String range operations
SET log "error: database connection failed"
GETRANGE log 0 4 # Returns "error"
APPEND log " at 10:30AM"

2.2 Hashes - Memory Efficient Objects​

Hashes store field-value pairs and are perfect for representing objects.

flowchart TD
A[Hash Key] --> B{Size Check}

B -->|Small Hash<br/>(≤512 entries, ≤64 bytes/value)| C[Ziplist Encoding]
B -->|Large Hash| D[Hash Table Encoding]

subgraph C[Ziplist Structure]
C1["zlbytes (4B)"] --> C2["zltail (4B)"]
C2 --> C3["zllen (2B)"]
C3 --> C4["entry1: field_len + field"]
C4 --> C5["entry1: value_len + value"]
C5 --> C6["..."]
C6 --> C7["zlend (1B: 255)"]
end

subgraph D[Hash Table Structure]
D1[Dict Structure] --> D2[Hash Table 0]
D1 --> D3[Hash Table 1<br/>(rehashing)]

D2 --> D4[Array of DictEntry pointers]
D4 --> D5["DictEntry: key, value, next"]
D5 --> D6[Collision Chain]
end

C --> E[Memory Efficient<br/>but O(n) access]
D --> F[Fast O(1) access<br/>more memory overhead]

Hash Configuration and Optimization:

# Redis config for hash optimization
hash-max-ziplist-entries 512
hash-max-ziplist-value 64

# Hash operations in practice
HSET user:1000 name "John" age 30 email "john@example.com"
HINCRBY user:1000 age 1 # Birthday!
HGETALL user:1000
HSCAN user:1000 0 # Iterate large hashes efficiently

2.3 Lists, Sets, and Sorted Sets​

Lists: Linked lists optimized with quicklist (ziplists linked together)

graph TB
A[QuickList Node] --> B[Ziplist]
A --> C[QuickList Node]
C --> D[Ziplist]
A --> E[...]

B --> F["Entry 1<br/>prevlen + encoding + content"]
B --> G["Entry 2"]
B --> H["Entry N<br/>Max 8KB per ziplist"]

I[Config] --> J["list-max-ziplist-size -2"]
I --> K["list-compress-depth 1"]

J --> L["Each ziplist ≤ 8KB"]
K --> M["Compress every other node"]

Sets and Sorted Sets Comparison:

FeatureSetSorted Set
StorageHash table or intsetSkip list + hash table
OrderingUnorderedOrdered by score
OperationsSADD, SINTER, SUNIONZADD, ZRANGE, ZSCORE
Use CasesTags, friends, unique itemsLeaderboards, time series
MemoryMore efficient for integersAdditional score storage

Sorted Set Internal Structure (Skip List + Hash Table):

┌─────────────────────────────────────────────────────────────────┐
│ Sorted Set Architecture │
├─────────────────────────────────────────────────────────────────┤
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Hash Table │ │ Skip List │ │
│ │ key→score ptr │◀──────▶│ score→key ptr │ │
│ └─────────────────┘ └─────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────────┐ │
│ │ "player1" │ │ Level 3: ──────┼─────▶ ... │
│ │ score: 100 │ │ Level 2: ──────┼─────▶ ... │
│ │ │ │ │ Level 1: ──────┼─────▶ ... │
│ └─────────────┘ │ Level 0: 100 → "player1" │
│ │ └─────────────────┘ │
│ ┌─────────────┐ │ │
│ │ "player2" │ ┌─────────────────┐ │
│ │ score: 200 │ │ Level 2: ──────┼─────▶ ... │
│ └─────────────┘ │ Level 1: ──────┼─────▶ ... │
│ │ Level 0: 200 → "player2" │
│ └─────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

3. Persistence Mechanisms​

3.1 RDB (Redis Database File)​

RDB creates point-in-time snapshots of your dataset.

sequenceDiagram
participant C as Client
participant R as Redis Main Process
participant F as Forked Child Process
participant D as Disk

Note over R: Periodically or manually triggered
R->>R: Check save conditions<br/>or receive SAVE/BGSAVE

alt SAVE (Synchronous)
R->>D: Block all clients<br/>Write RDB directly
D-->>R: RDB saved
else BGSAVE (Asynchronous)
R->>F: Fork child process<br/>(copy-on-write)
F->>F: Iterate through database
F->>D: Write RDB to temp file
F->>R: Signal completion
R->>D: Atomic rename temp to dump.rdb
end

Note over C: Clients can continue operations<br/>during BGSAVE

RDB Configuration:

# Save conditions: save <seconds> <changes>
save 900 1 # After 15 min if ≥1 key changed
save 300 10 # After 5 min if ≥10 keys changed
save 60 10000 # After 1 min if ≥10000 keys changed

# Advanced RDB settings
rdbcompression yes # Compress RDB files
rdbchecksum yes # Add checksum
dbfilename dump.rdb # Filename
dir /var/lib/redis # Directory
stop-writes-on-bgsave-error yes

3.2 AOF (Append Only File)​

AOF logs every write operation received by the server.

flowchart TD
A[Write Command] --> B[Command Processing]
B --> C[Execute in Memory]
C --> D{AOF Enabled?}

D -->|Yes| E[Append to AOF Buffer]
E --> F{Sync Policy}

F -->|appendfsync always| G[Write + Fsync<br/>Every command]
F -->|appendfsync everysec| H[Write + Fsync every second]
F -->|appendfsync no| I[Write only<br/>OS decides when to fsync]

G --> J[Highest Durability<br/>Lowest Performance]
H --> K[Good Balance<br/>Recommended]
I --> L[Highest Performance<br/>Risk of data loss]

J --> M[AOF File Grows]
K --> M
L --> M

M --> N{AOF Rewrite Triggered?}
N -->|Yes| O[Fork child process]
O --> P[Create new AOF with minimal commands]
P --> Q[Atomic replace old AOF]

AOF Rewriting Process:

┌─────────────────────────────────────────────────────────────┐
│ AOF Rewriting Process │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Main Process │ │ Child Process │ │
│ │ │ │ │ │
│ │ ┌───────────┐ │ │ ┌───────────┐ │ │
│ │ │ AOF Buffer │ │ │ │ Memory │ │ │
│ │ │ (New cmds) │──┼─────┼─▶│ Snapshot │ │ │
│ │ └───────────┘ │ │ └───────────┘ │ │
│ │ │ │ │ │ │
│ │ ┌───────────┐ │ │ ▼ │ │
│ │ │ Old AOF │──┼─────┼─▶ Write new AOF │ │
│ │ └───────────┘ │ │ (optimized) │ │
│ │ │ │ │ │ │
│ │ │ │ │ ▼ │ │
│ │ ▼ │ │ Temp AOF File │ │
│ │ Continue │ │ │ │ │
│ │ operations │ │ ▼ │ │
│ │ │ │ Signal parent │ │
│ └────────┬───────┘ └────────┬─────────┘ │
│ │ │ │
│ └───────────┬───────────┘ │
│ ▼ │
│ Atomic file replace │
│ │ │
│ ▼ │
│ ┌───────────────┐ │
│ │ New AOF Active │ │
│ └───────────────┘ │
└─────────────────────────────────────────────────────────────┘

3.3 RDB vs AOF Comparison​

quadrantChart
title RDB vs AOF: Trade-off Analysis
x-axis "Durability/Recovery Safety" --> "Performance/Simplicity"
y-axis "Low" --> "High"

quadrant-1 "AOF: High Safety, Complex Recovery"
quadrant-2 "RDB + AOF: Balanced Approach"
quadrant-3 "RDB: Simple, Faster Recovery"
quadrant-4 "No Persistence: Maximum Performance"

"AOF fsync=always": [0.9, 0.2]
"AOF fsync=everysec": [0.7, 0.5]
"RDB (hourly saves)": [0.4, 0.7]
"RDB + AOF": [0.8, 0.6]
"No persistence": [0.1, 0.9]

Hybrid Approach Recommendation:

# Enable both RDB and AOF for best results
save 900 1
save 300 10
save 60 10000

appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec

# Auto-rewrite AOF when it grows too much
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# Load AOF if available, fall back to RDB
aof-load-truncated yes

4. Redis Replication​

4.1 Master-Replica Architecture​

Redis replication is asynchronous, with a single master and multiple replicas.

sequenceDiagram
participant M as Master
participant R1 as Replica 1
participant R2 as Replica 2
participant R3 as Replica 3

Note over R1: Replica startup/initial sync
R1->>M: SLAVEOF master_ip master_port
M->>M: BGSAVE to create RDB
M->>R1: Send RDB file
R1->>R1: Load RDB into memory
M->>R1: Send replication buffer (commands during BGSAVE)

Note over M: Normal operation - asynchronous replication
M->>M: Execute client command
M->>M: Add to replication buffer
par To R1
M->>R1: Propagate command
R1->>R1: Execute command
and To R2
M->>R2: Propagate command
R2->>R2: Execute command
and To R3
M->>R3: Propagate command
R3->>R3: Execute command
end

Note over R1: Replica can handle read queries
R1->>R1: Serve read-only commands (if enabled)
R1->>R1: Reject write commands

4.2 Replication Internals and Configuration​

Replication Buffer and Backlog:

┌─────────────────────────────────────────────────────────┐
│ Master Replication State │
├─────────────────────────────────────────────────────────┤
│ ┌─────────────────────────────────────────────────┐ │
│ │ Replication Backlog │ │
│ │ ┌───┬───┬───┬───┬───┬───┬───┬───┬───┬───┐ │ │
│ │ │ C │ C │ C │ C │ C │ C │ C │ C │ C │ C │ │ │
│ │ │ m │ m │ m │ m │ m │ m │ m │ m │ m │ m │ │ │
│ │ │ d │ d │ d │ d │ d │ d │ d │ d │ d │ d │ │ │
│ │ │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ 8 │ 9 │10 │ │ │
│ │ └───┴───┴───┴───┴───┴───┴───┴───┴───┴───┘ │ │
│ │ ↑ ↑ │ │
│ │ │ │ │ │
│ │ master_repl_offset buffer end │ │
│ └─────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Connected Replicas │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
│ │ │ Replica │ │ Replica │ │ Replica │ │ │
│ │ │ 1 │ │ 2 │ │ 3 │ │ │
│ │ ├─────────┤ ├─────────┤ ├─────────┤ │ │
│ │ │offset:7 │ │offset:5 │ │offset:9 │ │ │
│ │ │lag:0s │ │lag:2s │ │lag:0s │ │ │
│ │ └─────────┘ └─────────┘ └─────────┘ │ │
│ └─────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘

Key Configuration Parameters:

# Master configuration
repl-backlog-size 1mb # Size of replication backlog
repl-backlog-ttl 3600 # Time to keep backlog after no replicas
repl-diskless-sync yes # Diskless replication for faster sync
repl-diskless-sync-delay 5 # Delay to start diskless sync

# Replica configuration
replica-read-only yes # Replicas accept only reads
repl-ping-replica-period 10 # Ping interval (seconds)
repl-timeout 60 # Replication timeout
min-replicas-to-write 1 # Require at least N replicas
min-replicas-max-lag 10 # Maximum lag allowed

4.3 Replication Failover and Sentinel​

flowchart TD
A[Master Failure Detected] --> B{Sentinel Quorum?}

B -->|Yes| C[Sentinel Leader Election]
B -->|No| D[Continue monitoring]

C --> E[Select New Master]
E --> F{Promotion Criteria}

F -->|Replica Priority| G[Select highest priority]
F -->|Replication Offset| H[Select most up-to-date]
F -->|Run ID| I[Select lexicographically]

G --> J[Promote Selected Replica]
H --> J
I --> J

J --> K[Reconfigure Other Replicas]
K --> L[Update Client Configuration]
L --> M[Failover Complete]

subgraph N[Sentinel Cluster]
O[Sentinel 1]
P[Sentinel 2]
Q[Sentinel 3]
O <--> P
P <--> Q
O <--> Q
end

N --> R[Monitor Masters/Replicas]
R --> S[Detect Failures]
S --> A

5. Redis Cluster - Distributed Redis​

5.1 Cluster Architecture and Sharding​

Redis Cluster uses automatic sharding across multiple nodes with a hash slot model (16384 slots).

graph TB
subgraph "Redis Cluster - 6 Nodes"
A[Node A<br/>Master<br/>Slots 0-5460] --> A1[Node D<br/>Replica]
B[Node B<br/>Master<br/>Slots 5461-10922] --> B1[Node E<br/>Replica]
C[Node C<br/>Master<br/>Slots 10923-16383] --> C1[Node F<br/>Replica]
end

subgraph "Cluster Bus - Gossip Protocol"
D[Gossip Messages]
E[Node Discovery]
F[Failure Detection]
end

subgraph "Client Routing"
G[Client] --> H{Key to Slot Hash}
H --> I["CRC16(key) mod 16384"]
I --> J[Route to Correct Node]
J --> K[MOVED/ASK Redirects]
end

A <--> B
B <--> C
A <--> C
A1 <--> B1
B1 <--> C1

D -.-> A
D -.-> B
D -.-> C

Hash Slot Allocation Example:

import crc16

def key_to_slot(key):
# Only hash the part between { and } if present
if '{' in key and '}' in key:
start = key.find('{') + 1
end = key.find('}')
if end != -1 and end > start:
key = key[start:end]

# Calculate CRC16 and mod 16384
return crc16.crc16xmodem(key.encode()) % 16384

# Examples
print(key_to_slot("user:1000")) # Regular key
print(key_to_slot("{orders}:pending")) # Hash tag - all go to same slot
print(key_to_slot("{orders}:completed")) # Same slot as above

5.2 Cluster Node Communication​

sequenceDiagram
participant C as Client
participant M1 as Master Node A
participant M2 as Master Node B
participant M3 as Master Node C

C->>M1: SET user:1000 "John"

Note over M1: Key hashes to slot 5500<br/>which belongs to Node B

M1->>C: -MOVED 5500 192.168.1.2:6379
C->>C: Update slot cache: slot 5500 → Node B
C->>M2: SET user:1000 "John"
M2->>C: OK

Note over C: Subsequent operations for slot 5500<br/>go directly to Node B

C->>M2: GET user:1000
M2->>C: "John"

Note over M2: Node B fails
M3->>M3: Detect failure via gossip
M3->>M3: Start failover for Node B's slots

C->>M2: GET user:1000
Note over C: Connection fails
C->>C: Clear slot 5500 from cache
C->>M1: GET user:1000
M1->>C: -MOVED 5500 192.168.1.4:6379 # New master
C->>M3: GET user:1000
M3->>C: "John"

Cluster Configuration:

# Cluster node configuration
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-replica-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

# Create cluster (from command line)
redis-cli --cluster create \
192.168.1.1:6379 192.168.1.2:6379 192.168.1.3:6379 \
192.168.1.4:6379 192.168.1.5:6379 192.168.1.6:6379 \
--cluster-replicas 1

5.3 Cluster Failure Detection and Recovery​

flowchart TD
A[Node Failure Suspected] --> B[Mark as PFAIL<br/>Possible Failure]

B --> C{Gossip Propagation}
C --> D[Other nodes receive PFAIL reports]

D --> E{Quorum of Masters<br/>agree on FAIL?}

E -->|Yes| F[Mark as FAIL<br/>Confirmed Failure]
E -->|No| G[Continue as PFAIL]

F --> H{Failed Node Type?}

H -->|Master| I[Start Failover Election]
H -->|Replica| J[No immediate action]

I --> K[Replicas of failed master<br/>start election]

K --> L{Election Criteria}
L -->|Replica priority| M[Higher priority wins]
L -->|Replication offset| N[Most up-to-date wins]
L -->|Run ID| O[Lexicographically first wins]

M --> P[Elected Replica promoted<br/>to Master]
N --> P
O --> P

P --> Q[Update cluster configuration]
Q --> R[Broadcast new configuration]
R --> S[Cluster recovery complete]

6. Advanced Features and Modules​

6.1 Redis Streams​

Streams are an append-only log data structure for messaging and event sourcing.

graph LR
subgraph "Redis Stream"
A["Stream: orders"] --> B["Entry: 1640995200000-0<br/>field1: value1"]
B --> C["Entry: 1640995201000-0<br/>field1: value2"]
C --> D["Entry: 1640995202000-0<br/>field1: value3"]
D --> E["..."]
E --> F["Entry: 1640995210000-0<br/>field1: valueN"]
end

subgraph "Consumer Group A"
G[Consumer A1] --> H[reads entries]
I[Consumer A2] --> H
end

subgraph "Consumer Group B"
J[Consumer B1] --> K[reads entries]
end

H --> A
K --> A

L["Pending Entries List<br/>for each consumer"] --> M["XCLAIM for<br/>reprocessing"]

N["Stream CG Commands"] --> O[XADD - Add entry]
N --> P[XREAD - Read entries]
N --> Q[XGROUP - Manage groups]
N --> R[XACK - Acknowledge]
N --> S[XCLAIM - Claim pending]

Stream Operations Example:

# Add entry to stream
XADD orders * user_id 1000 product_id 123 quantity 2

# Create consumer group
XGROUP CREATE orders order-processors $ MKSTREAM

# Read as consumer
XREADGROUP GROUP order-processors consumer1 COUNT 10 STREAMS orders >

# Acknowledge processing
XACK orders order-processors 1640995200000-0

# Check pending messages
XPENDING orders order-processors

# Claim stuck messages
XCLAIM orders order-processors consumer2 3600000 1640995200000-0

6.2 RedisJSON and RedisSearch​

RedisJSON adds native JSON support, and RedisSearch provides full-text search capabilities.

graph TB
subgraph "Redis with Modules"
A[Redis Core] --> B[RedisJSON Module]
A --> C[RedisSearch Module]
A --> D[RedisGraph Module]
A --> E[RedisTimeSeries Module]
A --> F[RedisBloom Module]
end

B --> G["JSON data types<br/>JSON.GET, JSON.SET<br/>JSON.INDEX with Search"]
C --> H["Secondary indexes<br/>Full-text search<br/>Faceted search<br/>Aggregations"]

G --> I["Store and query<br/>JSON documents natively"]
H --> J["Search across<br/>all data types"]

I --> K[Example: Product Catalog]
J --> L[Example: E-commerce Search]

K --> M["JSON.SET product:1 $ '{<br/> \"name\": \"Laptop\",<br/> \"price\": 999,<br/> \"tags\": [\"electronics\"]<br/>}'"]
L --> N["FT.CREATE idx:products ON JSON SCHEMA<br/> $.name AS name TEXT<br/> $.price AS price NUMERIC"]

M --> O["JSON.GET product:1 $.name"]
N --> P["FT.SEARCH idx:products \"gaming laptop\"<br/> RETURN 2 name price"]

6.3 Redis Modules Ecosystem​

ModulePurposeKey Features
RedisJSONJSON document storeJSONPath queries, atomic updates
RedisSearchFull-text searchIndexing, stemming, Chinese support
RedisGraphGraph databaseCypher queries, graph algorithms
RedisTimeSeriesTime-series dataDownsampling, aggregation, retention
RedisBloomProbabilistic dataBloom filters, Cuckoo filters, Count-Min Sketch
RedisAITensor servingModel execution, DAG execution
RedisGearsServerless engineData processing, ETL pipelines

7. Performance Optimization​

7.1 Memory Optimization Techniques​

flowchart TD
A[Memory Optimization] --> B[Data Structure Choice]
A --> C[Encoding Optimization]
A --> D[Memory Policies]
A --> E[Sharding Strategies]

B --> B1["Use Hashes for objects<br/>Use Ziplists for small lists"]
B --> B2["Use Sets for uniqueness<br/>Sorted Sets for ranges"]

C --> C1["Small strings < 44 bytes → EMBSTR"]
C --> C2["Small lists → Ziplist encoding"]
C --> C3["Small hashes → Ziplist encoding"]

D --> D1["maxmemory-policy:<br/>volatile-lru / allkeys-lru"]
D --> D2["Active defragmentation<br/>ACTIVE_DEFRAG yes"]

E --> E1["Shard large datasets<br/>across multiple instances"]
E --> E2["Use Redis Cluster<br/>for automatic sharding"]

subgraph F[Configuration Tuning]
B1 --> G["hash-max-ziplist-entries 512<br/>hash-max-ziplist-value 64"]
C2 --> H["list-max-ziplist-size -2<br/>list-compress-depth 1"]
D1 --> I["maxmemory 16gb<br/>maxmemory-samples 5"]
end

7.2 Latency Reduction Strategies​

Identifying and Reducing Latency Sources:

Latency SourceDetectionSolution
Slow commandsSLOWLOGOptimize commands, use pipelines
Network latencyredis-cli --latencyUse persistent connections, pipelining
Memory swappingredis-cli INFO memoryIncrease memory, optimize data
Fork latencyMonitor during BGSAVEUse diskless replication
AOF fsync

7.2 Command Optimization and Best Practices​

Memory-Optimized Data Structure Selection​

graph TB
A[Data Storage Requirement] --> B{Data Characteristics}

B --> C["Small data?<br/>(<100 items)"]
B --> D["Need range queries?"]
B --> E["Need uniqueness?"]
B --> F["Need scoring/ranking?"]
B --> G["Bit-level operations?"]

C --> H[Consider ziplist encoding]
D --> I[Use Sorted Sets]
E --> J[Use Sets]
F --> I
G --> K[Use Bitmaps]

H --> L["Configure:<br/>hash-max-ziplist-entries 512<br/>hash-max-ziplist-value 64"]
I --> M["ZSET: Skip list + Hash table<br/>O(log N) operations"]
J --> N["SET: Hash table or Intset<br/>O(1) operations"]
K --> O["Bitmap: String of bits<br/>Extremely memory efficient"]

subgraph "Memory Optimization Strategies"
P[Small integers] --> Q[Use intset encoding for sets]
R[Boolean flags] --> S[Use bitmaps (0.125 bytes per flag)"]
T[Small collections] --> U[Use ziplist encoding]
V[Large strings] --> W[Consider compression before storage]
end

Memory Optimization Commands:

# Check memory usage of a key
MEMORY USAGE user:session:12345

# Get detailed memory stats
INFO MEMORY

# Find big keys (production caution)
redis-cli --bigkeys

# Sample memory usage pattern
MEMORY STATS

# Memory doctor recommendations
MEMORY DOCTOR

Pipeline vs Transaction vs Lua Script​

FeaturePipeline (Pipelining)Transaction (MULTI/EXEC)Lua Script
AtomicityNoYes (all or nothing)Yes
Network Round-trips1 for all commands1 for MULTI, 1 for EXEC, 1 for each command1 for script load/execution
IsolationNoYes (serialized)Yes
PerformanceHighestMediumHigh (in-memory execution)
Error HandlingIndividual command failuresAll fail on EXEC errorAll-or-nothing
Use CaseBulk operations without atomicityBank transfers, inventory updatesComplex atomic operations

Performance Comparison:

graph LR
A[1000 INCR Commands] --> B{Normal Execution}
A --> C{Pipelined Execution}
A --> D{Lua Script Execution}

B --> E["1000 network round-trips<br/>1000 * (send + receive)"]
C --> F["1 network round-trip<br/>Batch send + batch receive"]
D --> G["2 network round-trips<br/>1 for EVAL, 1 for response"]

E --> H[Latency: ~1000ms]
F --> I[Latency: ~5ms]
G --> J[Latency: ~2ms]

H --> K[Throughput: ~1000 ops/s]
I --> L[Throughput: ~200,000 ops/s]
J --> M[Throughput: ~500,000 ops/s]

Implementation Examples:

import redis
import time

class RedisOptimizationExamples:
def __init__(self):
self.r = redis.Redis(host='localhost', port=6379, decode_responses=True)

def pipeline_example(self):
"""Pipeline example for bulk operations"""
pipe = self.r.pipeline()

start_time = time.time()

# Queue 1000 commands
for i in range(1000):
pipe.set(f"key:{i}", f"value:{i}")

# Execute all at once
pipe.execute()

elapsed = time.time() - start_time
print(f"Pipeline: {elapsed:.4f} seconds")

def transaction_example(self):
"""Transaction example for atomic operations"""
try:
# Start transaction
pipe = self.r.pipeline()

# Watch key for changes
pipe.watch("account:1000:balance", "account:2000:balance")

# Get current balances
balance1 = int(pipe.get("account:1000:balance") or 0)
balance2 = int(pipe.get("account:2000:balance") or 0)

# Check if transfer is possible
if balance1 >= 100:
# Start transaction
pipe.multi()
pipe.decrby("account:1000:balance", 100)
pipe.incrby("account:2000:balance", 100)

# Execute
results = pipe.execute()
print(f"Transaction successful: {results}")
else:
pipe.unwatch()
print("Insufficient funds")

except redis.WatchError:
print("Transaction failed due to concurrent modification")

def lua_script_example(self):
"""Lua script for complex atomic operations"""
script = """
local current = redis.call('GET', KEYS[1])
local increment = tonumber(ARGV[1])
local max_value = tonumber(ARGV[2])

if not current then
current = 0
else
current = tonumber(current)
end

local new_value = current + increment

if new_value > max_value then
return {err = "Exceeded maximum value"}
end

redis.call('SET', KEYS[1], new_value)
return new_value
"""

# Load and execute script
script_hash = self.r.script_load(script)
result = self.r.evalsha(script_hash, 1, "counter:limited", 5, 100)
print(f"Lua script result: {result}")

# Usage
optimizer = RedisOptimizationExamples()
optimizer.pipeline_example()
optimizer.transaction_example()
optimizer.lua_script_example()

7.3 Connection Pooling and Client Configuration​

Optimal Client Configuration:

import redis
from redis.connection import ConnectionPool

class OptimizedRedisClient:
def __init__(self):
# Connection pool configuration
self.pool = ConnectionPool(
host='localhost',
port=6379,
db=0,
max_connections=50, # Maximum connections in pool
socket_connect_timeout=5, # Connection timeout
socket_keepalive=True, # Keep connections alive
socket_keepalive_options={
socket.TCP_KEEPIDLE: 60, # Start checking after 60s idle
socket.TCP_KEEPINTVL: 30, # Check every 30s
socket.TCP_KEEPCNT: 3 # Fail after 3 failed checks
},
retry_on_timeout=True, # Retry on timeout
health_check_interval=30, # Health check every 30s
decode_responses=True # Decode responses to strings
)

# Create client with connection pool
self.client = redis.Redis(connection_pool=self.pool)

def connection_metrics(self):
"""Monitor connection pool metrics"""
pool_info = self.pool.connection_kwargs
print(f"Connection Pool Info:")
print(f" Max connections: {self.pool.max_connections}")
print(f" Current connections: {len(self.pool._connections)}")
print(f" Available connections: {self.pool._available_connections}")

8. Redis Security Deep Dive​

8.1 Comprehensive Security Architecture​

graph TB
subgraph "Network Security"
A[External Access] --> B[VPC/Private Network]
B --> C[Security Groups]
C --> D[Redis Instance]

E[VPN/SSH Tunnel] --> F[Jump Host]
F --> B

G[Public Access] --> H[SSL/TLS Termination]
H --> I[API Gateway]
I --> B
end

subgraph "Authentication & Authorization"
J[Client] --> K{Authentication Method}

K --> L[Password Auth]
K --> M[Client Certificates]
K --> N[IAM Authentication (AWS)]

O[Redis ACL] --> P[User Definitions]
P --> Q["user default on ~* +@all -@dangerous"]
P --> R["user appuser on ~app:* +@read +@write"]
P --> S["user readonly on ~cache:* +@read"]

T[Command Filtering] --> U["Disable dangerous commands"]
U --> V["CONFIG, FLUSHDB, FLUSHALL, KEYS"]
end

subgraph "Encryption & Data Protection"
W[Data in Transit] --> X[TLS 1.2/1.3]
X --> Y["SSL certificates<br/>Perfect Forward Secrecy"]

Z[Data at Rest] --> AA["RDB/AOF Encryption<br/>or Disk-level encryption"]

AB[Sensitive Data] --> AC[Field-level Encryption]
AC --> AD["Application-layer encryption<br/>before storing in Redis"]
end

subgraph "Monitoring & Auditing"
AE[Audit Log] --> AF["redisAuditLog yes<br/>redisAuditLogFile /var/log/redis/audit.log"]

AG[Command Logging] --> AH["MONITOR command<br/>(debugging only)"]

AI[Intrusion Detection] --> AJ["Failed login attempts<br/>Suspicious command patterns"]
end

8.2 ACL (Access Control List) Implementation​

Redis ACL Configuration:

# Redis 6.0+ ACL Configuration
# Enable ACL
aclfile /etc/redis/users.acl

# Define users in ACL file
user default off nopass ~* +@all -@dangerous
user admin on >adminpassword ~* +@all
user webapp on >apppassword ~cache:* ~session:* ~queue:* +@read +@write -@admin
user readonly on >readonlypassword ~cache:* +@read
user replica on >replicapassword +psync +replconf +ping

# Categories of commands
# +@admin - Administrative commands
# +@read - Read-only commands
# +@write - Write commands
# +@fast - Fast commands (O(1) or O(log N))
# +@slow - Slow commands
# +@dangerous - Dangerous commands (FLUSHALL, CONFIG, etc.)

# Command rules
user analytics on >analyticspass ~analytics:* +@read +geo* +bit* -@write

# Key patterns
user service1 on >service1pass ~service1:* +@all
user service2 on >service2pass ~service2:* +@all

ACL Management Commands:

# View ACL rules
ACL LIST
ACL GETUSER username

# Create/modify users
ACL SETUSER webapp on >newpassword ~app:* +@read +@write
ACL SETUSER webapp ~session:* ~cache:*
ACL SETUSER webapp -@dangerous

# Save ACL to file
ACL SAVE

# Dry run to test permissions
ACL DRYRUN webapp get user:123
ACL DRYRUN webapp flushall

# Import/Export ACL
ACL GENPASS # Generate secure password
ACL CAT # List command categories
ACL LOG # View security log

8.3 TLS/SSL Configuration​

Redis TLS Configuration:

# redis-tls.conf
port 0
tls-port 6379
tls-cert-file /etc/redis/redis.crt
tls-key-file /etc/redis/redis.key
tls-ca-cert-file /etc/redis/ca.crt

# TLS Protocol versions
tls-protocols "TLSv1.2 TLSv1.3"

# Cipher suites
tls-ciphers "EECDH+AESGCM:EDH+AESGCM"
tls-ciphersuites "TLS_CHACHA20_POLY1305_SHA256"

# Authentication modes
tls-auth-clients yes # Require client certificates
tls-auth-clients optional # Optional client certificates

# Replication with TLS
tls-replication yes
tls-cluster yes

Client TLS Connection:

import redis
import ssl

# Create SSL context
ssl_context = ssl.create_default_context(
ssl.Purpose.SERVER_AUTH,
cafile='/path/to/ca.crt'
)
ssl_context.load_cert_chain(
certfile='/path/to/client.crt',
keyfile='/path/to/client.key'
)
ssl_context.check_hostname = False # For self-signed certs
ssl_context.verify_mode = ssl.CERT_REQUIRED

# Connect with TLS
client = redis.Redis(
host='redis.example.com',
port=6379,
ssl=True,
ssl_cert_reqs='required',
ssl_ca_certs='/path/to/ca.crt',
ssl_certfile='/path/to/client.crt',
ssl_keyfile='/path/to/client.key'
)

# Alternative: Using SSL context
client = redis.Redis(
host='redis.example.com',
port=6379,
ssl=True,
ssl_context=ssl_context
)

9. Advanced Redis Features​

9.1 Redis Streams Deep Dive​

Stream Architecture and Internals:

graph TB
subgraph "Redis Stream Structure"
A[Stream Key] --> B[Stream Header]
B --> C[Radix Tree (rax)"]

C --> D[Entry 1: 1640995200000-0]
D --> E["fields: {user: 'john', action: 'login'}"]

C --> F[Entry 2: 1640995201000-0]
F --> G["fields: {user: 'jane', action: 'purchase'}"]

C --> H[Entry N: 1640995210000-0]
H --> I["fields: {user: 'bob', action: 'logout'}"]
end

subgraph "Consumer Group Architecture"
J[Consumer Group: "order-processors"] --> K[Stream: "orders"]

J --> L[Consumer 1: "worker-1"]
J --> M[Consumer 2: "worker-2"]
J --> N[Consumer 3: "worker-3"]

L --> O[Pending Entries List]
M --> O
N --> O

O --> P["Entry: 1640995200000-0 -> worker-1<br/>Entry: 1640995201000-0 -> worker-2<br/>Entry: 1640995202000-0 -> worker-1"]

Q[Last Delivered ID] --> R["> : Next entry<br/>0-0: From beginning<br/>$: Latest"]
end

subgraph "Stream Internals"
S[Radix Tree Node] --> T["Key: timestamp-sequence<br/>Value: listpack"]

U[listpack structure] --> V["Total bytes | num elements | element1 | element2 | ... | END"]

W[Memory Optimization] --> X["Small entries: listpack<br/>Large entries: separate allocations"]
end

Complete Streams Implementation:

class RedisStreamsProcessor:
"""Advanced Redis Streams implementation"""

def __init__(self, redis_client):
self.redis = redis_client

def create_stream_system(self):
"""Create complete stream-based event system"""

# Create stream with initial data
self.redis.xadd('orders:events', {
'type': 'system_start',
'timestamp': str(int(time.time() * 1000)),
'message': 'Order processing system initialized'
})

# Create consumer groups
self.redis.xgroup_create(
'orders:events', 'email-processors', id='0',
mkstream=True
)
self.redis.xgroup_create(
'orders:events', 'analytics-processors', id='0',
mkstream=True
)

# Add sample orders
for i in range(5):
order_id = f"ORD{1000 + i}"
self.redis.xadd('orders:events', {
'order_id': order_id,
'customer_id': f"CUST{2000 + i}",
'amount': str(100 + (i * 50)),
'status': 'pending',
'timestamp': str(int(time.time() * 1000))
})

def process_with_consumer_groups(self):
"""Process streams with consumer groups"""

# Consumer 1: Process for email notifications
email_entries = self.redis.xreadgroup(
'email-processors', 'consumer-1',
{'orders:events': '>'},
count=2,
block=5000
)

for stream, entries in email_entries:
for entry_id, data in entries:
print(f"Email Consumer: Processing {entry_id}")
print(f" Order: {data[b'order_id']}, Customer: {data[b'customer_id']}")

# Send email
self.send_order_email(data)

# Acknowledge processing
self.redis.xack('orders:events', 'email-processors', entry_id)

# Consumer 2: Process for analytics
analytics_entries = self.redis.xreadgroup(
'analytics-processors', 'consumer-1',
{'orders:events': '>'},
count=2,
block=5000
)

for stream, entries in analytics_entries:
for entry_id, data in entries:
print(f"Analytics Consumer: Processing {entry_id}")

# Update analytics
self.update_analytics(data)

# Acknowledge processing
self.redis.xack('orders:events', 'analytics-processors', entry_id)

def handle_pending_messages(self):
"""Handle stuck/unacknowledged messages"""

# Check for pending messages
pending = self.redis.xpending('orders:events', 'email-processors')

if pending['pending'] > 0:
print(f"Found {pending['pending']} pending messages")

# Claim messages idle for more than 5 minutes
claimed = self.redis.xclaim(
'orders:events', 'email-processors', 'consumer-2',
min_idle_time=300000, # 5 minutes in milliseconds
message_ids=['1640995200000-0'] # Specific IDs or empty for any
)

for entry_id, data in claimed:
print(f"Claimed message {entry_id} for reprocessing")
self.process_message_with_retry(data, entry_id)

def stream_aggregation_example(self):
"""Example of stream aggregation patterns"""

# Add orders with different statuses
statuses = ['pending', 'processing', 'shipped', 'delivered', 'cancelled']

for status in statuses:
for i in range(3):
self.redis.xadd('orders:stream', {
'order_id': f"ORD{1000 + i}",
'status': status,
'amount': str(50 + (i * 25)),
'timestamp': str(int(time.time() * 1000))
})

# Aggregate by status using consumer groups
self.redis.xgroup_create('orders:stream', 'status-aggregator', id='0')

# Process and aggregate
entries = self.redis.xreadgroup(
'status-aggregator', 'aggregator-1',
{'orders:stream': '>'},
count=10
)

aggregates = {}
for stream, stream_entries in entries:
for entry_id, data in stream_entries:
status = data[b'status'].decode()
amount = float(data[b'amount'].decode())

if status not in aggregates:
aggregates[status] = {
'count': 0,
'total_amount': 0,
'average_amount': 0
}

aggregates[status]['count'] += 1
aggregates[status]['total_amount'] += amount
aggregates[status]['average_amount'] = (
aggregates[status]['total_amount'] / aggregates[status]['count']
)

# Acknowledge
self.redis.xack('orders:stream', 'status-aggregator', entry_id)

# Store aggregates in Redis
for status, stats in aggregates.items():
self.redis.hset(f'order:aggregates:{status}', mapping=stats)

return aggregates

# Usage
stream_processor = RedisStreamsProcessor(redis_client)
stream_processor.create_stream_system()
stream_processor.process_with_consumer_groups()
aggregates = stream_processor.stream_aggregation_example()
print(f"Aggregates: {aggregates}")

9.2 Redis Modules: RedisJSON and RediSearch​

RedisJSON (JSON Document Store):

import redis
from redis.commands.json.path import Path

class RedisJSONExamples:
"""RedisJSON examples for JSON document storage"""

def __init__(self):
# Connect to Redis with RedisJSON module
self.r = redis.Redis(host='localhost', port=6379, decode_responses=True)

def basic_json_operations(self):
"""Basic RedisJSON operations"""

# Store JSON document
user_data = {
"name": "John Doe",
"email": "john@example.com",
"age": 30,
"address": {
"street": "123 Main St",
"city": "New York",
"zip": "10001"
},
"tags": ["customer", "premium", "active"]
}

# Set JSON document
self.r.json().set('user:1000', Path.root_path(), user_data)

# Get entire document
user = self.r.json().get('user:1000')
print(f"User: {user}")

# Get specific fields
name = self.r.json().get('user:1000', '.name')
age = self.r.json().get('user:1000', '.age')
city = self.r.json().get('user:1000', '.address.city')
print(f"Name: {name}, Age: {age}, City: {city}")

# Update specific fields
self.r.json().set('user:1000', '.age', 31)
self.r.json().set('user:1000', '.address.zip', '10002')

# Array operations
self.r.json().arrappend('user:1000', '.tags', 'verified')
tags = self.r.json().get('user:1000', '.tags')
print(f"Tags: {tags}")

# Increment numeric field
self.r.json().numincrby('user:1000', '.age', 1)

# Delete field
self.r.json().delete('user:1000', '.tags[0]')

def json_indexing_with_redisearch(self):
"""JSON indexing with RediSearch"""

# Create index on JSON documents
schema = (
TextField("$.name", as_name="name"),
TextField("$.email", as_name="email"),
NumericField("$.age", as_name="age"),
TagField("$.tags[*]", as_name="tags"),
GeoField("$.address.location", as_name="location")
)

# Create index (requires FT.CREATE)
# self.r.ft().create_index(schema, definition=IndexDefinition(
# prefix=["user:"],
# index_type=IndexType.JSON
# ))

# Search JSON documents
# results = self.r.ft().search("@tags:{premium}")
# print(f"Premium users: {results}")

RediSearch (Full-Text Search):

class RediSearchExamples:
"""RediSearch examples for full-text search"""

def __init__(self):
self.r = redis.Redis(host='localhost', port=6379, decode_responses=True)

def create_search_index(self):
"""Create RediSearch index"""

# Define schema for products
schema = (
TextField("name", weight=5.0),
TextField("description"),
TagField("category"),
NumericField("price"),
NumericField("stock"),
TextField("brand"),
GeoField("location")
)

# Create index
try:
self.r.ft().create_index(
schema,
definition=IndexDefinition(
prefix=["product:"]
)
)
print("Index created successfully")
except Exception as e:
print(f"Index may already exist: {e}")

def index_documents(self):
"""Index documents for search"""

products = [
{
"id": "product:1001",
"name": "Wireless Bluetooth Headphones",
"description": "Noise cancelling over-ear headphones with 30hr battery",
"category": "electronics,audio",
"price": 199.99,
"stock": 45,
"brand": "AudioTech"
},
{
"id": "product:1002",
"name": "Smart Watch Series 5",
"description": "Fitness tracker with heart rate monitor and GPS",
"category": "electronics,wearables",
"price": 299.99,
"stock": 23,
"brand": "TechWear"
},
{
"id": "product:1003",
"name": "Mechanical Gaming Keyboard",
"description": "RGB backlit keyboard with cherry MX switches",
"category": "electronics,gaming",
"price": 149.99,
"stock": 67,
"brand": "GameGear"
}
]

# Index each product
for product in products:
doc_id = product.pop("id")
self.r.hset(doc_id, mapping=product)

def search_examples(self):
"""Various search examples"""

# Simple full-text search
results = self.r.ft().search("wireless headphones")
print(f"Search results: {results.total} found")

# Search with filters
results = self.r.ft().search(
"gaming",
filter=NumericFilter("price", 100, 200)
)

# Search by category
results = self.r.ft().search("@category:{electronics}")

# Search with sorting
results = self.r.ft().search(
"*",
sort_by=SortBy("price", asc=False)
)

# Aggregation queries
req = AggregationRequest("*").group_by(
"@category",
reducers.count().alias("count"),
reducers.sum("@price").alias("total_value"),
reducers.avg("@price").alias("avg_price")
).sort_by(Desc("@count"))

result = self.r.ft().aggregate(req)

# Faceted search
req = AggregationRequest("*").load(
"@category", "@brand"
).group_by(
"@category",
reducers.count().alias("category_count")
).group_by(
"@brand",
reducers.count().alias("brand_count")
)

return results

9.3 RedisTimeSeries and RedisBloom​

RedisTimeSeries for Time-Series Data:

import redis
from redistimeseries.client import Client as RedisTimeSeries

class TimeSeriesExamples:
"""RedisTimeSeries examples for time-series data"""

def __init__(self):
self.rts = RedisTimeSeries(host='localhost', port=6379)

def basic_timeseries_operations(self):
"""Basic time-series operations"""

# Create time series
self.rts.create(
"temperature:nyc",
retention_msecs=365*24*60*60*1000, # 1 year
labels={"city": "nyc", "metric": "temperature"}
)

# Add data points
timestamp = int(time.time() * 1000) # Current time in milliseconds

self.rts.add("temperature:nyc", timestamp, 72.5)
self.rts.add("temperature:nyc", timestamp + 60000, 73.1) # 1 minute later
self.rts.add("temperature:nyc", timestamp + 120000, 74.2) # 2 minutes later

# Query data
# Range query
data = self.rts.range(
"temperature:nyc",
timestamp - 3600000, # 1 hour ago
timestamp + 300000 # 5 minutes from now
)

# Get latest value
latest = self.rts.get("temperature:nyc")

# Aggregated queries
hourly_avg = self.rts.range(
"temperature:nyc",
timestamp - 24*3600000, # 24 hours ago
timestamp,
bucket_size_msec=3600000, # 1 hour buckets
aggregation_type='avg'
)

def multi_series_operations(self):
"""Multiple time-series operations"""

# Create multiple series
metrics = ["cpu", "memory", "disk", "network"]

for metric in metrics:
self.rts.create(
f"server:web01:{metric}",
labels={"server": "web01", "metric": metric}
)

# Add data to all metrics
current_time = int(time.time() * 1000)

metrics_data = {
"cpu": 45.2,
"memory": 78.5,
"disk": 32.1,
"network": 120.4
}

for metric, value in metrics_data.items():
self.rts.add(f"server:web01:{metric}", current_time, value)

# Query by labels
all_server_metrics = self.rts.mrange(
filters=["server=web01"],
from_time=current_time - 3600000,
to_time=current_time
)

# Multi-get
specific_metrics = self.rts.mget(
filters=["metric=(cpu,memory)"],
with_labels=True
)

def advanced_features(self):
"""Advanced time-series features"""

# Create rules for downsampling
self.rts.create(
"temperature:nyc:1min",
retention_msecs=7*24*3600000 # 7 days
)

# Create aggregation rule
self.rts.createrule(
"temperature:nyc",
"temperature:nyc:1min",
aggregation_type="avg",
bucket_size_msec=60000 # 1 minute
)

# Incremental compaction
self.rts.create(
"temperature:nyc:1hour",
retention_msecs=365*24*3600000 # 1 year
)

self.rts.createrule(
"temperature:nyc:1min",
"temperature:nyc:1hour",
aggregation_type="avg",
bucket_size_msec=3600000 # 1 hour
)

RedisBloom for Probabilistic Data Structures:

class RedisBloomExamples:
"""RedisBloom examples for probabilistic data structures"""

def __init__(self):
self.r = redis.Redis(host='localhost', port=6379, decode_responses=True)

def bloom_filter_example(self):
"""Bloom Filter for membership testing"""

# Create Bloom filter
self.r.execute_command(
'BF.RESERVE', 'users:bloom', '0.01', '1000000'
)

# Add items
users = ['user:1001', 'user:1002', 'user:1003', 'user:1004']
for user in users:
self.r.execute_command('BF.ADD', 'users:bloom', user)

# Check membership
exists = self.r.execute_command('BF.EXISTS', 'users:bloom', 'user:1001')
print(f"User 1001 exists: {exists}")

# Check non-existing (false positives possible)
exists = self.r.execute_command('BF.EXISTS', 'users:bloom', 'user:9999')
print(f"User 9999 exists: {exists}")

# Add multiple items
new_users = ['user:1005', 'user:1006']
self.r.execute_command('BF.MADD', 'users:bloom', *new_users)

# Check multiple items
results = self.r.execute_command(
'BF.MEXISTS', 'users:bloom', 'user:1001', 'user:9999', 'user:1005'
)
print(f"Multiple checks: {results}")

def cuckoo_filter_example(self):
"""Cuckoo Filter (more efficient than Bloom)"""

# Create Cuckoo filter
self.r.execute_command(
'CF.RESERVE', 'urls:filter', '1000000'
)

# Add URLs
urls = [
'https://example.com/page1',
'https://example.com/page2',
'https://example.com/page3'
]

for url in urls:
self.r.execute_command('CF.ADD', 'urls:filter', url)

# Check and add if not exists (like set)
added = self.r.execute_command(
'CF.ADDNX', 'urls:filter', 'https://example.com/page4'
)
print(f"Added new URL: {added}")

def count_min_sketch_example(self):
"""Count-Min Sketch for frequency counting"""

# Create sketch
self.r.execute_command(
'CMS.INITBYPROB', 'page:views', '0.001', '0.01'
)

# Increment counts
pages = ['home', 'about', 'contact', 'home', 'home', 'about']
for page in pages:
self.r.execute_command('CMS.INCRBY', 'page:views', page, '1')

# Query counts
home_views = self.r.execute_command('CMS.QUERY', 'page:views', 'home')
about_views = self.r.execute_command('CMS.QUERY', 'page:views', 'about')

print(f"Home views: {home_views}")
print(f"About views: {about_views}")

# Merge sketches (for distributed counting)
self.r.execute_command(
'CMS.INITBYPROB', 'page:views:shard2', '0.001', '0.01'
)
self.r.execute_command(
'CMS.MERGE', 'page:views:total', '2', 'page:views', 'page:views:shard2'
)

def top_k_example(self):
"""Top-K for finding most frequent items"""

# Create Top-K
self.r.execute_command('TOPK.RESERVE', 'trending:hashtags', '10', '50', '5', '0.9')

# Add items
hashtags = [
'#redis', '#python', '#docker', '#kubernetes',
'#redis', '#python', '#redis', '#docker',
'#javascript', '#python', '#redis'
]

for hashtag in hashtags:
self.r.execute_command('TOPK.ADD', 'trending:hashtags', hashtag)

# Get top items
top_items = self.r.execute_command('TOPK.LIST', 'trending:hashtags')
print(f"Top hashtags: {top_items}")

# Check if item is in top-K
in_top = self.r.execute_command('TOPK.QUERY', 'trending:hashtags', '#redis')
print(f"#redis in top: {in_top}")

10. Production Deployment and Operations​

10.1 Configuration Best Practices​

Production Redis Configuration Template:

# redis-production.conf

# NETWORK
bind 127.0.0.1 ::1 # Only localhost and IPv6 loopback
protected-mode yes # Enhanced security
port 6379
tcp-backlog 511
timeout 0 # Never timeout connections
tcp-keepalive 300 # Send TCP keepalives

# GENERAL
daemonize yes # Run as daemon
supervised systemd # Use systemd supervision
pidfile /var/run/redis/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-server.log
databases 16
always-show-logo no

# SNAPSHOTTING
save 900 1 # 15min if 1+ key changed
save 300 10 # 5min if 10+ keys changed
save 60 10000 # 1min if 10000+ keys changed
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis

# REPLICATION
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync yes
repl-diskless-sync-delay 5
repl-backlog-size 128mb
repl-backlog-ttl 3600
replica-priority 100
min-replicas-to-write 1
min-replicas-max-lag 10

# SECURITY
requirepass "{{ strong_password }}"
rename-command CONFIG "{{ random_string }}"
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command SHUTDOWN ""
rename-command DEBUG ""

# CLIENTS
maxclients 10000

# MEMORY MANAGEMENT
maxmemory 16gb
maxmemory-policy allkeys-lru
maxmemory-samples 10
lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes
lazyfree-lazy-server-del yes
replica-lazy-flush yes

# APPEND ONLY MODE
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes

# LUA SCRIPTING
lua-time-limit 5000

# REDIS CLUSTER
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-replica-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

# SLOW LOG
slowlog-log-slower-than 10000 # 10ms
slowlog-max-len 1024

# LATENCY MONITOR
latency-monitor-threshold 100 # Monitor ops > 100ms

# EVENT NOTIFICATION
notify-keyspace-events ""

# ADVANCED CONFIG
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes

10.2 Monitoring and Alerting​

Comprehensive Monitoring Dashboard:

graph TB
subgraph "Core Metrics"
A1[Memory Usage] --> A2["Used memory: 12.4GB/16GB (78%)<br/>Peak memory: 14.2GB<br/>Fragmentation ratio: 1.23"]

B1[Operations] --> B2["Ops/sec: 45,230<br/>Hits/sec: 42,100<br/>Misses/sec: 3,130<br/>Hit rate: 93.1%"]

C1[Latency] --> C2["P50: 0.8ms<br/>P95: 2.1ms<br/>P99: 5.4ms<br/>P999: 12.3ms"]

D1[Connections] --> D2["Connected clients: 245<br/>Blocked clients: 0<br/>Rejected connections: 0"]
end

subgraph "Replication Health"
E1[Master] --> E2["Role: master<br/>Connected slaves: 2<br/>Repl backlog: 42MB/128MB"]

F1[Replica Lag] --> F2["Replica 1: 0.4s lag<br/>Replica 2: 1.2s lag<br/>Replica 3: OFFLINE"]

G1[Replication Buffer] --> G2["Size: 84MB<br/>First byte offset: 9876543210<br/>Current offset: 9876587654"]
end

subgraph "Persistence"
H1[RDB] --> H2["Last save: 5min ago<br/>Save in progress: No<br/>Last bgsave status: OK"]

I1[AOF] --> I2["Size: 4.2GB<br/>Buffer size: 32MB<br/>Rewrite in progress: No"]

J1[Fork Performance] --> J2["Last fork duration: 420ms<br/>Fork rate: 0.1/sec"]
end

subgraph "Cluster Health"
K1[Cluster State] --> K2["OK<br/>16384 slots covered<br/>0 failing nodes"]

L1[Node Distribution] --> L2["Node 1: 5461 slots (OK)<br/>Node 2: 5461 slots (OK)<br/>Node 3: 5462 slots (OK)"]

M1[Key Distribution] --> M2["Keys per node:<br/>Node 1: 4.2M<br/>Node 2: 4.1M<br/>Node 3: 4.3M"]
end

subgraph "Alert Thresholds"
N1[Critical] --> N2["Memory > 90%<br/>Replica lag > 10s<br/>Node down > 5min"]

O1[Warning] --> O2["Memory > 80%<br/>Hit rate < 90%<br/>Latency P99 > 10ms"]

P1[Info] --> P2["RDB save > 2s<br/>AOF rewrite > 30s<br/>Connections > 80%"]
end

Alerting Rules Configuration:

# prometheus-alerts.yml
groups:
- name: redis_alerts
rules:
- alert: RedisDown
expr: redis_up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Redis instance is down"
description: "Redis instance {{ $labels.instance }} is not responding"

- alert: RedisOutOfMemory
expr: redis_memory_used_bytes / redis_memory_max_bytes > 0.9
for: 5m
labels:
severity: critical
annotations:
summary: "Redis memory usage is high"
description: "Redis instance {{ $labels.instance }} memory usage is {{ $value | humanizePercentage }}"

- alert: RedisHighLatency
expr: histogram_quantile(0.99, rate(redis_commands_duration_seconds_bucket[5m])) > 0.01
for: 10m
labels:
severity: warning
annotations:
summary: "Redis command latency is high"
description: "P99 latency on {{ $labels.instance }} is {{ $value | humanizeDuration }}"

- alert: RedisReplicaLag
expr: redis_master_repl_offset - redis_slave_repl_offset > 10000000
for: 5m
labels:
severity: warning
annotations:
summary: "Redis replica lag is high"
description: "Replica {{ $labels.instance }} is lagging by {{ $value }} bytes"

- alert: RedisKeyspaceMissRate
expr: rate(redis_keyspace_misses_total[5m]) / rate(redis_keyspace_hits_total[5m] + redis_keyspace_misses_total[5m]) > 0.2
for: 15m
labels:
severity: warning
annotations:
summary: "Redis cache hit rate is low"
description: "Cache hit rate on {{ $labels.instance }} is {{ $value | humanizePercentage }}"

10.3 Backup and Disaster Recovery​

Comprehensive Backup Strategy:

flowchart TD
A[Backup Strategy] --> B[Snapshot Backup]
A --> C[Continuous Backup]
A --> D[Point-in-Time Recovery]

B --> E[RDB Snapshots]
C --> F[AOF Append-only]
D --> G[RDB + AOF Combination]

E --> H["Schedule: Hourly, Daily, Weekly<br/>Retention: 30 days<br/>Storage: S3/Glacier"]
F --> I["Sync: Every second<br/>Retention: 7 days<br/>Storage: EBS/SSD"]
G --> J["Recovery Point: Any second<br/>RTO: < 15min<br/>RPO: < 1min"]

H --> K[Automated Backup Script]
I --> L[AOF Rewrite Management]
J --> M[Recovery Testing]

subgraph K[Backup Automation]
N["1. BGSAVE to start RDB creation"]
N --> O["2. Wait for completion<br/>(check LASTSAVE)"]
O --> P["3. Copy RDB to backup location"]
P --> Q["4. Verify checksum<br/>(redis-check-rdb)"]
Q --> R["5. Encrypt and upload to cloud"]
R --> S["6. Update backup catalog"]
end

subgraph M[Recovery Procedure]
T["Disaster Detection"] --> U["Failover to replica"]
U --> V["Restore from backup"]
V --> W["Validate data integrity"]
W --> X["Promote to master"]
X --> Y["Reconfigure clients"]
end

S --> Z[Monitoring & Alerting]
Y --> Z

Z --> AA["Backup Success Rate: 99.9%<br/>Recovery Test: Weekly<br/>DR Drill: Quarterly"]

Automated Backup Script:

#!/bin/bash
# redis-backup.sh

set -euo pipefail

# Configuration
REDIS_HOST="localhost"
REDIS_PORT=6379
REDIS_PASSWORD=$(cat /etc/redis/password)
BACKUP_DIR="/backup/redis"
S3_BUCKET="redis-backups"
RETENTION_DAYS=30
ENCRYPTION_KEY="/etc/redis/backup.key"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Generate timestamp
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="redis_backup_${TIMESTAMP}.rdb"

# Start RDB backup
echo "[$(date)] Starting Redis backup..."
redis-cli -h "$REDIS_HOST" -p "$REDIS_PORT" -a "$REDIS_PASSWORD" --rdb "$BACKUP_DIR/dump.rdb"

# Wait for backup to complete
sleep 2

# Verify RDB file
echo "[$(date)] Verifying RDB file..."
redis-check-rdb "$BACKUP_DIR/dump.rdb"

# Rename with timestamp
mv "$BACKUP_DIR/dump.rdb" "$BACKUP_DIR/$BACKUP_FILE"

# Compress backup
echo "[$(date)] Compressing backup..."
gzip "$BACKUP_DIR/$BACKUP_FILE"

# Encrypt backup
echo "[$(date)] Encrypting backup..."
openssl enc -aes-256-cbc -salt -in "$BACKUP_DIR/${BACKUP_FILE}.gz" \
-out "$BACKUP_DIR/${BACKUP_FILE}.gz.enc" -pass file:"$ENCRYPTION_KEY"

# Upload to S3
echo "[$(date)] Uploading to S3..."
aws s3 cp "$BACKUP_DIR/${BACKUP_FILE}.gz.enc" "s3://${S3_BUCKET}/${TIMESTAMP}/"

# Cleanup local files
rm -f "$BACKUP_DIR/${BACKUP_FILE}.gz" "$BACKUP_DIR/${BACKUP_FILE}.gz.enc"

# Cleanup old backups
echo "[$(date)] Cleaning up old backups..."
find "$BACKUP_DIR" -name "redis_backup_*.gz.enc" -mtime +$RETENTION_DAYS -delete

# Verify backup in S3
echo "[$(date)] Verifying S3 backup..."
aws s3 ls "s3://${S3_BUCKET}/${TIMESTAMP}/" | grep "${BACKUP_FILE}.gz.enc"

# Update backup catalog
echo "[$(date)] Updating backup catalog..."
cat <<EOF >> "$BACKUP_DIR/backup_catalog.log"
$TIMESTAMP,$BACKUP_FILE.gz.enc,OK,$(date +%s)
EOF

echo "[$(date)] Backup completed successfully!"

11. Performance Benchmarking and Tuning​

11.1 Comprehensive Benchmarking Suite​

import time
import statistics
import redis
from concurrent.futures import ThreadPoolExecutor

class RedisBenchmark:
"""Comprehensive Redis benchmarking tool"""

def __init__(self, host='localhost', port=6379):
self.client = redis.Redis(host=host, port=port, decode_responses=True)
self.results = {}

def benchmark_operations(self, operation_count=10000):
"""Benchmark different Redis operations"""

benchmarks = {
'SET': self._benchmark_set,
'GET': self._benchmark_get,
'INCR': self._benchmark_incr,
'HSET': self._benchmark_hset,
'SADD': self._benchmark_sadd,
'LPUSH': self._benchmark_lpush,
'ZADD': self._benchmark_zadd
}

for operation, benchmark_func in benchmarks.items():
print(f"\nBenchmarking {operation}...")
results = benchmark_func(operation_count)
self.results[operation] = results
self._print_results(operation, results)

def _benchmark_set(self, count):
"""Benchmark SET operations"""
latencies = []

for i in range(count):
start = time.perf_counter_ns()
self.client.set(f"benchmark:key:{i}", f"value:{i}")
end = time.perf_counter_ns()
latencies.append((end - start) / 1_000_000) # Convert to ms

return self._calculate_stats(latencies)

def _benchmark_get(self, count):
"""Benchmark GET operations"""
# First set some data
for i in range(count):
self.client.set(f"benchmark:key:{i}", f"value:{i}")

latencies = []
for i in range(count):
start = time.perf_counter_ns()
self.client.get(f"benchmark:key:{i}")
end = time.perf_counter_ns()
latencies.append((end - start) / 1_000_000)

return self._calculate_stats(latencies)

def _benchmark_pipeline(self, count, batch_size=100):
"""Benchmark pipeline operations"""
latencies = []

for batch_start in range(0, count, batch_size):
pipe = self.client.pipeline()

for i in range(batch_start, min(batch_start + batch_size, count)):
pipe.set(f"pipeline:key:{i}", f"value:{i}")

start = time.perf_counter_ns()
pipe.execute()
end = time.perf_counter_ns()

latency_per_op = ((end - start) / 1_000_000) / batch_size
latencies.append(latency_per_op)

return self._calculate_stats(latencies)

def _benchmark_concurrent(self, operation_count=10000, thread_count=10):
"""Benchmark concurrent operations"""
def worker(worker_id, operations_per_worker):
latencies = []
for i in range(operations_per_worker):
key = f"concurrent:{worker_id}:{i}"
start = time.perf_counter_ns()
self.client.set(key, f"value:{i}")
end = time.perf_counter_ns()
latencies.append((end - start) / 1_000_000)
return latencies

operations_per_worker = operation_count // thread_count
all_latencies = []

with ThreadPoolExecutor(max_workers=thread_count) as executor:
futures = []
for i in range(thread_count):
future = executor.submit(worker, i, operations_per_worker)
futures.append(future)

for future in futures:
all_latencies.extend(future.result())

return self._calculate_stats(all_latencies)

def _calculate_stats(self, latencies):
"""Calculate statistics from latency data"""
return {
'count': len(latencies),
'mean': statistics.mean(latencies),
'median': statistics.median(latencies),
'p95': self._percentile(latencies, 95),
'p99': self._percentile(latencies, 99),
'p999': self._percentile(latencies, 99.9),
'min': min(latencies),
'max': max(latencies),
'stddev': statistics.stdev(latencies) if len(latencies) > 1 else 0
}

def _percentile(self, data, percentile):
"""Calculate percentile"""
if not data:
return 0
sorted_data = sorted(data)
index = (percentile / 100) * (len(sorted_data) - 1)

if index.is_integer():
return sorted_data[int(index)]
else:
lower = sorted_data[int(index)]
upper = sorted_data[int(index) + 1]
fraction = index - int(index)
return lower + fraction * (upper - lower)

def _print_results(self, operation, results):
"""Print benchmark results"""
print(f" Operations: {results['count']:,}")
print(f" Mean latency: {results['mean']:.3f} ms")
print(f" Median latency: {results['median']:.3f} ms")
print(f" P95 latency: {results['p95']:.3f} ms")
print(f" P99 latency: {results['p99']:.3f} ms")
print(f" P999 latency: {results['p999']:.3f} ms")
print(f" Min/Max: {results['min']:.3f}/{results['max']:.3f} ms")
print(f" Std Dev: {results['stddev']:.3f} ms")

if results['mean'] > 0:
ops_per_second = 1000 / results['mean'] # Convert ms to seconds
print(f" Estimated throughput: {ops_per_second:,.0f} ops/sec")

# Run benchmarks
benchmark = RedisBenchmark()
benchmark.benchmark_operations(10000)
pipeline_results = benchmark._benchmark_pipeline(10000)
print("\nPipeline Benchmark:")
benchmark._print_results('Pipeline', pipeline_results)
concurrent_results = benchmark._benchmark_concurrent(10000, 10)
print("\nConcurrent Benchmark:")
benchmark._print_results('Concurrent', concurrent_results)

11.2 Tuning Guide for Different Workloads​

Workload-Specific Tuning Recommendations:

Workload TypeCharacteristicsKey Tuning ParametersConfiguration Example
CachingHigh read ratio, low latency requirementsmaxmemory-policy, hz, maxclientsmaxmemory-policy allkeys-lru, hz 100
Session StoreHigh write ratio, short TTLmaxmemory-policy, activerehashingmaxmemory-policy volatile-lru, activerehashing yes
Message QueueHigh throughput, stream processinglist-max-ziplist-size, stream-node-max-entrieslist-max-ziplist-size -2, stream-node-max-entries 500
LeaderboardSorted set operations, range querieszset-max-ziplist-entries, hzzset-max-ziplist-entries 128, hz 50
Real-time AnalyticsHigh write volume, time-series datasave configuration, appendfsyncsave "", appendfsync everysec
GeospatialRadius queries, location datahash-max-ziplist-entries, activerehashinghash-max-ziplist-entries 512, activerehashing yes

12. Learning Path and Resources​

12.1 Redis Learning Path​

graph LR
A[Beginner] --> B[Intermediate]
B --> C[Advanced]
C --> D[Expert]

subgraph A[Beginner Level]
A1[Basic Commands<br/>SET, GET, DEL]
A2[Data Types<br/>Strings, Lists, Hashes]
A3[Simple Use Cases<br/>Caching, Session Store]
A4[Redis CLI Basics]
end

subgraph B[Intermediate Level]
B1[Advanced Data Types<br/>Sets, Sorted Sets, Bitmaps]
B2[Pub/Sub & Transactions]
B3[Persistence<br/>RDB & AOF]
B4[Replication<br/>Master-Slave Setup]
B5[Performance Basics]
end

subgraph C[Advanced Level]
C1[Redis Cluster<br/>Sharding & Auto-failover]
C2[High Availability<br/>Sentinel]
C3[Lua Scripting]
C4[Memory Optimization]
C5[Monitoring & Metrics]
end

subgraph D[Expert Level]
D1[Redis Modules<br/>RedisJSON, RediSearch, etc.]
D2[Stream Processing]
D3[Advanced Tuning<br/>Kernel parameters]
D4[Disaster Recovery<br/>Backup strategies]
D5[Security Hardening<br/>TLS, ACL, RBAC]
end

A1 --> B1
A2 --> B2
A3 --> B3
A4 --> B4
B1 --> C1
B2 --> C2
B3 --> C3
B4 --> C4
C1 --> D1
C2 --> D2
C3 --> D3
C4 --> D4

12.2 Essential Resources​

Official Documentation:

Books:

  • "Redis in Action" by Josiah L. Carlson
  • "Mastering Redis" by Jeremy Nelson
  • "The Little Redis Book" by Karl Seguin

Tools and Utilities:

Community:

Conclusion​

Redis is a versatile, high-performance data store that powers many of the world's most demanding applications. From simple caching to complex stream processing, Redis offers a rich set of features that continue to evolve.

Key Takeaways:

  1. Choose the right data structure for your use case - it significantly impacts performance
  2. Understand persistence trade-offs between RDB and AOF based on your requirements
  3. Plan for scaling with replication, clustering, and sharding strategies
  4. Monitor everything - latency, memory, throughput, and replication health
  5. Secure your deployment with ACL, TLS, and proper network configuration
  6. Test your disaster recovery - regular backups and recovery drills are essential

The Redis Ecosystem Continues to Grow:

  • Redis Stack: Bundles Redis with RedisJSON, RediSearch, RedisGraph, and RedisTimeSeries
  • RedisAI: Machine learning model serving
  • RedisGears: Serverless programming engine for data processing
  • RedisBloom: Probabilistic data structures for big data applications

Remember that Redis is not just a cache - it's a multi-model database that can serve as primary data store for many applications when used correctly. Continuously monitor, test, and optimize your Redis deployment to ensure it meets your application's performance and reliability requirements.


This comprehensive guide covers Redis from basic concepts to advanced production deployment. Redis continues to evolve, so always refer to the latest documentation and best practices for your specific use case and version.

Deep Dive Redis Integration With Spring Boot Application

· 阅读需 12 分钟

Table of Contents​


Introduction​

In distributed systems, caching is a critical mechanism to improve performance by reducing the load on backend services and databases. This article delves into the design and implementation of cache mechanisms for distributed systems, covering high-level architecture, detailed designs, and sample code for various scenarios.


High-Level Architecture Design​

Diagram​

graph TD
A[Client] --> B[Load Balancer]
B --> C[Service Layer]
C --> D[Cache Layer]
C --> E[Data Storage Layer]
D --> E
F[Monitoring and Metrics] --> C
F --> D
F --> E

Explanation​

  1. Client: The client sends requests to the distributed system.
  2. Load Balancer: Distributes incoming requests across multiple instances of the service layer.
  3. Service Layer: Handles business logic and interacts with the cache and data storage layers.
  4. Cache Layer: Stores frequently accessed data to reduce the load on the data storage layer.
  5. Data Storage Layer: The primary data store (e.g., database) where data is persisted.
  6. Monitoring and Metrics: Collects and monitors performance metrics for the entire system.

Detail Designs​

Cache Layer​

The cache layer is responsible for storing frequently accessed data to reduce the load on the data storage layer. It can be implemented using distributed caching solutions like Redis or Hazelcast.

Redis Setup and Integration​

Redis is an in-memory data structure store that can be used as a distributed cache. It supports various data structures like strings, hashes, lists, sets, and sorted sets.

Step 1: Install Redis​

You can install Redis on your local machine or use a cloud-based Redis service like AWS ElastiCache or Redis Labs.

For local installation on Linux:

sudo apt-get update
sudo apt-get install redis-server

For local installation on macOS:

brew install redis
Step 2: Start Redis Server​

Start the Redis server:

redis-server
Step 3: Verify Redis Installation​

Verify that Redis is running:

redis-cli ping

If Redis is running correctly, you should see PONG as the response.

Redis Configuration​

Redis configuration can be managed through the redis.conf file. Common configurations include setting the maximum memory limit, configuring persistence, and setting up replication.

Example redis.conf snippet:

maxmemory 256mb
maxmemory-policy allkeys-lru

Redis Integration with Spring Boot​

To integrate Redis with a Spring Boot application, you need to add the necessary dependencies and configure the Redis connection.

Step 1: Add Dependencies​

Add the following dependencies to your pom.xml:

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-cache</artifactId>
</dependency>
Step 2: Configure Redis Connection​

Configure the Redis connection in your application.yml:

spring:
redis:
host: localhost
port: 6379
cache:
type: redis
Step 3: Enable Caching​

Enable caching in your Spring Boot application by adding the @EnableCaching annotation to your main application class:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cache.annotation.EnableCaching;

@SpringBootApplication
@EnableCaching
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
Step 4: Use Redis Cache​

Use the @Cacheable annotation to cache method results in Redis:

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;

@Service
public class CacheService {

@Cacheable(value = "userCache", key = "#userId")
public User getUserById(Long userId) {
// Fetch user from data storage layer
return fetchUserFromDatabase(userId);
}

private User fetchUserFromDatabase(Long userId) {
// Database query logic
return new User();
}
}

Best Practices for Redis in Kubernetes​

When deploying Redis in Kubernetes, it's important to follow best practices to ensure high availability, scalability, and security.

Step 1: Deploy Redis in Kubernetes​

You can deploy Redis in Kubernetes using a StatefulSet for stateful applications or a Deployment for stateless applications. Here's an example of deploying Redis using a StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
serviceName: redis
replicas: 3
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:latest
ports:
- containerPort: 6379
volumeMounts:
- name: redis-data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
Step 2: Create a Headless Service​

Create a headless service to manage the Redis StatefulSet:

apiVersion: v1
kind: Service
metadata:
name: redis
spec:
clusterIP: None
selector:
app: redis
ports:
- port: 6379
targetPort: 6379
Step 3: Configure Redis Access in Spring Boot​

When deploying your Spring Boot application in Kubernetes, you should use the service name to access Redis. Update your application.yml to use the service name:

spring:
redis:
host: redis
port: 6379
cache:
type: redis
Step 4: Enable Redis Persistence​

To enable Redis persistence, configure the redis.conf file to use either RDB (Redis Database Backup) or AOF (Append-Only File) persistence:

save 900 1
save 300 10
save 60 10000
appendonly yes
Step 5: Secure Redis​

Secure Redis by setting a password and enabling TLS:

requirepass yourpassword
tls-port 6379
tls-cert-file /path/to/redis.crt
tls-key-file /path/to/redis.key
tls-ca-cert-file /path/to/ca.crt
Step 6: Monitor Redis​

Use Prometheus and Grafana to monitor Redis performance and health. Deploy a Redis exporter to expose Redis metrics:

apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-exporter
spec:
replicas: 1
selector:
matchLabels:
app: redis-exporter
template:
metadata:
labels:
app: redis-exporter
spec:
containers:
- name: redis-exporter
image: oliver006/redis_exporter:latest
args:
- --redis.addr=redis:6379
- --redis.password=yourpassword
ports:
- containerPort: 9121

Create a service to expose the Redis exporter:

apiVersion: v1
kind: Service
metadata:
name: redis-exporter
spec:
selector:
app: redis-exporter
ports:
- port: 9121
targetPort: 9121

Data Storage Layer​

The data storage layer is the primary data store where data is persisted. It can be a relational database, NoSQL database, or any other storage solution.

Diagram​

graph TD
A[Service Layer] --> B[Data Storage Layer]
B --> C[Relational Database]
B --> D[NoSQL Database]

Sample Code​

import org.springframework.stereotype.Repository;

@Repository
public class UserRepository {

public User findById(Long userId) {
// Database query logic
return new User();
}

public void save(User user) {
// Database save logic
}
}

Service Layer​

The service layer handles business logic and interacts with the cache and data storage layers. It is responsible for managing the flow of data between the client, cache, and data storage.

Diagram​

graph TD
A[Client] --> B[Service Layer]
B --> C[Cache Layer]
B --> D[Data Storage Layer]

Sample Code​

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class UserService {

@Autowired
private CacheService cacheService;

@Autowired
private UserRepository userRepository;

public User getUserById(Long userId) {
User user = cacheService.getUserById(userId);
if (user == null) {
user = userRepository.findById(userId);
cacheService.saveUser(user);
}
return user;
}

public void saveUser(User user) {
userRepository.save(user);
cacheService.saveUser(user);
}
}

Load Balancer​

The load balancer distributes incoming requests across multiple instances of the service layer to ensure high availability and scalability.

Diagram​

graph TD
A[Client] --> B[Load Balancer]
B --> C[Service Instance 1]
B --> D[Service Instance 2]
B --> E[Service Instance 3]

Sample Code​

Load balancers are typically implemented using infrastructure components like NGINX, HAProxy, or cloud-provided load balancers (e.g., AWS ELB, Google Cloud Load Balancer).

Monitoring and Metrics​

Monitoring and metrics collection are essential for tracking the performance and health of the distributed system. Tools like Prometheus, Grafana, and ELK stack can be used for this purpose.

Diagram​

graph TD
A[Service Layer] --> B[Monitoring and Metrics]
B --> C[Prometheus]
B --> D[Grafana]
B --> E[ELK Stack]

Sample Code​

import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;

public class MetricsCollector {

private static final Counter cacheHits = Counter.build()
.name("cache_hits_total")
.help("Total number of cache hits.")
.register();

private static final Counter cacheMisses = Counter.build()
.name("cache_misses_total")
.help("Total number of cache misses.")
.register();

private static final Gauge cacheSize = Gauge.build()
.name("cache_size")
.help("Current size of the cache.")
.register();

public static void recordCacheHit() {
cacheHits.inc();
}

public static void recordCacheMiss() {
cacheMisses.inc();
}

public static void setCacheSize(long size) {
cacheSize.set(size);
}
}

Complex Scenario 1: Flow Control and Backpressure Handling​

Problem Statement​

In high-traffic scenarios, the cache might become a bottleneck if not properly managed. Flow control and backpressure handling are essential to prevent the cache from being overwhelmed.

Solution Architecture​

  1. Rate Limiting: Implement rate limiting to control the number of requests hitting the cache.
  2. Circuit Breaker: Use a circuit breaker to handle failures gracefully and prevent cascading failures.
  3. Backpressure Handling: Implement backpressure mechanisms to manage the flow of data between services.

Flow Chart​

graph TD
A[Client Request] --> B{Rate Limiter}
B -->|Allowed| C[Cache Lookup]
B -->|Denied| D[Fallback to Database]
C --> E{Cache Hit?}
E -->|Yes| F[Return Cached Data]
E -->|No| G[Fetch from Database]
G --> H[Update Cache]
H --> F
D --> G

Sample Code​

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;

@Service
public class UserService {

@Cacheable(value = "userCache", key = "#userId")
public User getUserById(Long userId) {
// Fetch user from database
return fetchUserFromDatabase(userId);
}

private User fetchUserFromDatabase(Long userId) {
// Database query logic
return new User();
}
}

Complex Scenario 2: Cache Invalidation and Data Synchronization​

Problem Statement​

Ensuring that the cache remains consistent with the underlying data store is challenging, especially in distributed systems.

Solution Architecture​

  1. Event-Driven Invalidation: Use event-driven mechanisms to invalidate the cache when data changes.
  2. Distributed Locking: Use distributed locking to ensure that only one service updates the cache at a time.
  3. TTL-Based Invalidation: Use Time-to-Live (TTL) to automatically invalidate cache entries after a certain period.

Sequence Diagram​

sequenceDiagram
participant Client
participant Service
participant Cache
participant Database

Client->>Service: Update User
Service->>Database: Update User in Database
Database-->>Service: User Updated
Service->>Cache: Invalidate User Cache
Cache-->>Service: Cache Invalidated
Service-->>Client: User Updated

Sample Code​

import org.springframework.cache.annotation.CacheEvict;
import org.springframework.stereotype.Service;

@Service
public class UserService {

@CacheEvict(value = "userCache", key = "#userId")
public void updateUser(Long userId, User user) {
// Update user in database
updateUserInDatabase(userId, user);
}

private void updateUserInDatabase(Long userId, User user) {
// Database update logic
}
}

Complex Scenario 3: Cache Stampede Prevention​

Problem Statement​

A cache stampede occurs when multiple requests for the same data hit the cache simultaneously, leading to a spike in database load.

Solution Architecture​

  1. Probabilistic Early Expiration: Use probabilistic early expiration to spread out cache expiration times.
  2. Locking Mechanism: Implement a locking mechanism to ensure that only one request populates the cache at a time.
  3. Background Refresh: Use background refresh to keep the cache up-to-date without impacting the user experience.

Flow Chart​

graph TD
A[Client Request] --> B{Cache Hit?}
B -->|Yes| C[Return Cached Data]
B -->|No| D{Lock Acquired?}
D -->|Yes| E[Fetch from Database]
D -->|No| F[Wait for Lock]
E --> G[Update Cache]
G --> C
F --> B

Sample Code​

import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;

import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

@Service
public class UserService {

private final Lock lock = new ReentrantLock();

@Cacheable(value = "userCache", key = "#userId")
public User getUserById(Long userId) {
try {
lock.lock();
// Double-check to ensure the cache is still empty
User cachedUser = fetchUserFromCache(userId);
if (cachedUser != null) {
return cachedUser;
}
// Fetch user from database
User user = fetchUserFromDatabase(userId);
// Update cache
updateCache(userId, user);
return user;
} finally {
lock.unlock();
}
}

private User fetchUserFromCache(Long userId) {
// Cache lookup logic
return null;
}

private User fetchUserFromDatabase(Long userId) {
// Database query logic
return new User();
}

private void updateCache(Long userId, User user) {
// Cache update logic
}
}

Conclusion​

Caching is a powerful mechanism for improving the performance and scalability of distributed systems. By understanding the high-level architecture and detailed designs, you can effectively implement cache mechanisms that handle various complex scenarios like flow control, cache invalidation, and cache stampede prevention. Use the provided sample code and diagrams to implement efficient caching in your distributed system.