跳到主要内容

2 篇博文 含有标签「system-design」

查看所有标签

Distributed Counter System with Redis - II

· 阅读需 28 分钟
  1. Executive Summary
  2. System Overview
  3. Architecture Design
  4. Data Model
  5. API Design
  6. Implementation Details
  7. Performance Optimization
  8. High Availability & Disaster Recovery
  9. Security Considerations
  10. Monitoring & Observability
  11. Deployment Strategy
  12. Cost Analysis
  13. Appendices

1. Executive Summary​

1.1 Purpose​

This document outlines the design for a distributed counter system using Redis as the centralized data store. The system provides atomic, consistent, and high-performance counter operations across multiple application instances.

1.2 Business Requirements​

  • Accuracy: Counters must be accurate with zero data loss
  • Performance: Support 100K+ operations per second
  • Scalability: Horizontal scaling to handle growing traffic
  • Availability: 99.99% uptime SLA
  • Real-time: Sub-millisecond latency for counter operations

1.3 Use Cases​

  1. Page view counters (website analytics)
  2. API rate limiting (throttling requests)
  3. Inventory management (product stock tracking)
  4. Social media metrics (likes, shares, followers)
  5. Event ticketing (available seats tracking)
  6. Gaming leaderboards (player scores)
  7. IoT device metrics (sensor data aggregation)

1.4 Key Decisions​

DecisionChoiceRationale
Data StoreRedisIn-memory speed, atomic operations, built-in data structures
Deployment ModelRedis ClusterHigh availability, horizontal scalability
PersistenceRDB + AOFBalance between performance and durability
Client LibraryLettuce (Java) / ioredis (Node.js)Production-ready, connection pooling, cluster support
Caching StrategyWrite-through with local cacheReduce Redis load while maintaining consistency

2. System Overview​

2.1 High-Level Architecture​

┌─────────────────────────────────────────────────────────────────────┐
│ Client Layer (Users/Apps) │
└───────────────────────┬─────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Load Balancer (HAProxy/NGINX) │
│ • Health checks │
│ • SSL termination │
│ • Request routing │
└───────────────────────┬─────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Application │ │ Application │ │ Application │
│ Server 1 │ │ Server 2 │ │ Server N │
│ │ │ │ │ │
│ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
│ │ Local │ │ │ │ Local │ │ │ │ Local │ │
│ │ Cache │ │ │ │ Cache │ │ │ │ Cache │ │
│ │(Optional)│ │ │ │(Optional)│ │ │ │(Optional)│ │
│ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
│ │ │ │ │ │
│ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
│ │ Counter │ │ │ │ Counter │ │ │ │ Counter │ │
│ │ Client │ │ │ │ Client │ │ │ │ Client │ │
│ └─────┬────┘ │ │ └─────┬────┘ │ │ └─────┬────┘ │
└───────┼──────┘ └───────┼──────┘ └───────┼──────┘
│ │ │
└────────────────┼────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Redis Sentinel Cluster │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Sentinel Nodes (Monitoring) │ │
│ │ [Sentinel 1] [Sentinel 2] [Sentinel 3] │ │
│ │ • Health monitoring • Automatic failover │ │
│ └──────────────────┬───────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────────┴───────────────────────────────────────────┐ │
│ │ Redis Master-Replica Set │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Master │──────▶│ Replica 1 │ │ Replica 2 │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ Shard 1 │ │ Shard 1 │ │ Shard 1 │ │ │
│ │ │ Slots: │ │ (Read-only)│ │ (Read-only)│ │ │
│ │ │ 0-5460 │ │ │ │ │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Master │──────▶│ Replica 1 │ │ Replica 2 │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ Shard 2 │ │ Shard 2 │ │ Shard 2 │ │ │
│ │ │ Slots: │ │ (Read-only)│ │ (Read-only)│ │ │
│ │ │ 5461-10922 │ │ │ │ │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Master │──────▶│ Replica 1 │ │ Replica 2 │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ Shard 3 │ │ Shard 3 │ │ Shard 3 │ │ │
│ │ │ Slots: │ │ (Read-only)│ │ (Read-only)│ │ │
│ │ │ 10923-16383│ │ │ │ │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ └────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Persistence Layer │
│ • RDB Snapshots (every 5 minutes) │
│ • AOF Log (append-only file for durability) │
│ • Backup to S3/Cloud Storage (hourly) │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Monitoring & Alerting │
│ • Prometheus (metrics collection) │
│ • Grafana (visualization) │
│ • PagerDuty (alerting) │
│ • ELK Stack (logging) │
└─────────────────────────────────────────────────────────────────────┘

2.2 System Components​

2.2.1 Application Layer​

  • Counter Client Library: Handles connection pooling, retries, circuit breaking
  • Local Cache (Optional): Reduces Redis load for read-heavy workloads
  • Business Logic: Application-specific counter operations

2.2.2 Redis Layer​

  • Redis Cluster: Distributed key-value store with automatic sharding
  • Sentinel: Monitoring and automatic failover
  • Persistence: RDB snapshots + AOF logging

2.2.3 Infrastructure Layer​

  • Load Balancer: Distributes traffic across application servers
  • Monitoring: Prometheus, Grafana, ELK stack
  • Backup: Automated backups to cloud storage

2.3 Data Flow Diagrams​

2.3.1 Increment Counter Flow​

┌──────────┐
│ Client │
└────┬─────┘
│
│ 1. POST /counter/increment
│ { counter: "page_views", value: 1 }
▼
┌─────────────────┐
│ App Server │
│ │
│ 2. Check local │
│ cache │
│ (optional) │
└────┬────────────┘
│
│ 3. INCR counter:page_views
▼
┌─────────────────┐
│ Redis Master │
│ │
│ 4. Atomic │
│ increment │
│ │
│ 5. Return new │
│ value │
└────┬────────────┘
│
│ 6. Replicate to replicas
▼
┌─────────────────┐
│ Redis Replica │
│ │
│ 7. Async │
│ replication │
└─────────────────┘
│
│ 8. Return response
▼
┌──────────┐
│ Client │
│ │
│ Result: │
│ { value: 1001 }
└──────────┘

2.3.2 Batch Increment Flow (Optimized)​

Application Server
┌─────────────────────────────────────┐
│ │
│ 1. Buffer increments (100ms) │
│ • page_views: +5 │
│ • api_calls: +3 │
│ • errors: +1 │
│ │
│ 2. Flush buffer (batch) │
└─────────┬───────────────────────────┘
│
│ 3. Pipeline command:
│ INCRBY counter:page_views 5
│ INCRBY counter:api_calls 3
│ INCRBY counter:errors 1
▼
┌─────────────────────────────────────┐
│ Redis Master │
│ │
│ 4. Execute pipeline atomically │
│ • All or nothing │
│ • Single network round trip │
│ │
│ 5. Return all results │
│ [1005, 503, 42] │
└─────────┬───────────────────────────┘
│
│ 6. Response
▼
┌─────────────────────────────────────┐
│ Application Server │
│ │
│ 7. Update local cache (optional) │
│ 8. Return to client │
└─────────────────────────────────────┘

3. Architecture Design​

3.1 System Architecture Patterns​

3.1.1 Write-Through Cache Pattern​

┌────────────────────────────────────────────────────────────┐
│ Application Logic │
└─────────────────┬──────────────────────────────────────────┘
│
│ Increment counter
▼
┌───────────────┐
│ Is write op? │
└───┬───────┬───┘
│ │
Yes │ │ No (read)
▼ ▼
┌────────────┐ ┌────────────┐
│ Write to │ │ Check │
│ Redis │ │ local │
│ (MASTER) │ │ cache │
└─────┬──────┘ └─────┬──────┘
│ │
│ │ Cache miss?
│ ▼
│ ┌────────────┐
│ │ Read from │
│ │ Redis │
│ │ (REPLICA) │
│ └─────┬──────┘
│ │
▼ ▼
┌──────────────────────────┐
│ Update local cache │
│ (with TTL) │
└──────────────────────────┘

3.1.2 Circuit Breaker Pattern​

┌─────────────────────────────────────────────────────────┐
│ Client Request │
└──────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────┐
│ Circuit Breaker │
│ State? │
└────┬────┬───┬───┘
│ │ │
┌───────┘ │ └────────┐
│ CLOSED │ OPEN │ HALF-OPEN
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Execute │ │ Fail │ │ Try one │
│ normally │ │ fast │ │ request │
└────┬─────┘ │ │ └────┬─────┘
│ │ │ │
│ │ │ │ Success?
▼ ▼ ▼ ▼
┌──────────────────────────────────────┐
│ Track Success/Failure │
│ │
│ Failures > Threshold? │
│ Yes → Open circuit │
│ │
│ After timeout (30s)? │
│ Yes → Half-open │
│ │
│ Half-open success? │
│ Yes → Close circuit │
└──────────────────────────────────────┘

3.2 Deployment Architecture​

3.2.1 Multi-Region Setup​

┌──────────────────────────────────────────────────────────────────┐
│ Global Load Balancer │
│ (GeoDNS / AWS Route53) │
└────────────┬─────────────────────────────────┬───────────────────┘
│ │
┌────────┴────────┐ ┌───────┴────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ Region: US-EAST-1 │ │ Region: EU-WEST-1 │
│ │ │ │
│ ┌──────────────────┐ │ │ ┌──────────────────┐ │
│ │ Redis Cluster │ │ │ │ Redis Cluster │ │
│ │ (Primary) │◄──┼──────┼─▶│ (Replica) │ │
│ │ │ │ │ │ │ │
│ │ • 3 Shards │ │ │ │ • 3 Shards │ │
│ │ • 2 Replicas ea. │ │ │ │ • Read-only │ │
│ └──────────────────┘ │ │ └──────────────────┘ │
│ │ │ │
│ ┌──────────────────┐ │ │ ┌──────────────────┐ │
│ │ App Servers │ │ │ │ App Servers │ │
│ │ (3-10 instances) │ │ │ │ (3-10 instances) │ │
│ └──────────────────┘ │ │ └──────────────────┘ │
│ │ │ │
│ Cross-region │ │ Cross-region │
│ replication ───────────┼──────┼──────────────────────▶ │
│ (async, eventual │ │ │
│ consistency) │ │ │
└─────────────────────────┘ └─────────────────────────┘

• Latency: 10-20ms • Latency: 10-20ms
• Write: US region • Write: Replicate from US
• Read: Local • Read: Local

3.2.2 Kubernetes Deployment​

# Redis StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-cluster
spec:
serviceName: redis-cluster
replicas: 6 # 3 masters + 3 replicas
selector:
matchLabels:
app: redis-cluster
template:
metadata:
labels:
app: redis-cluster
spec:
containers:
- name: redis
image: redis:7.2-alpine
ports:
- containerPort: 6379
name: client
- containerPort: 16379
name: gossip
volumeMounts:
- name: data
mountPath: /data
- name: conf
mountPath: /conf
command:
- redis-server
- /conf/redis.conf
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Gi

---
# Application Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: counter-service
spec:
replicas: 5
selector:
matchLabels:
app: counter-service
template:
metadata:
labels:
app: counter-service
spec:
containers:
- name: app
image: counter-service:v1.0
env:
- name: REDIS_CLUSTER_NODES
value: "redis-cluster-0.redis-cluster:6379,redis-cluster-1.redis-cluster:6379"
ports:
- containerPort: 8080
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"

4. Data Model​

4.1 Redis Data Structures​

4.1.1 Simple Counter (STRING)​

Key Pattern: counter:{name}
Value: Integer (64-bit signed)
TTL: Optional (for temporary counters)

Examples:
┌────────────────────────────────┬────────────┬─────────┐
│ Key │ Value │ TTL │
├────────────────────────────────┼────────────┼─────────┤
│ counter:page_views │ 1,234,567 │ -1 │
│ counter:api:rate_limit:user123 │ 45 │ 3600s │
│ counter:daily_signups:20260107 │ 892 │ 86400s │
│ counter:product:SKU123:stock │ 150 │ -1 │
└────────────────────────────────┴────────────┴─────────┘

Commands:
- INCR counter:page_views → Increment by 1
- INCRBY counter:page_views 5 → Increment by 5
- DECR counter:product:SKU123:stock → Decrement by 1
- GET counter:page_views → Read current value
- SET counter:page_views 0 EX 3600 → Reset with TTL

4.1.2 Counter with Metadata (HASH)​

Key Pattern: counter_meta:{name}
Value: Hash with multiple fields

Example: User engagement counter
┌────────────────────────────────────────────────────────┐
│ Key: counter_meta:user:12345 │
├────────────────────┬───────────────────────────────────┤
│ Field │ Value │
├────────────────────┼───────────────────────────────────┤
│ page_views │ 1,234 │
│ likes_given │ 456 │
│ comments_posted │ 89 │
│ shares │ 23 │
│ last_active │ 2026-01-07T10:30:00Z │
│ created_at │ 2025-06-15T08:00:00Z │
└────────────────────┴───────────────────────────────────┘

Commands:
- HINCRBY counter_meta:user:12345 page_views 1
- HGETALL counter_meta:user:12345
- HGET counter_meta:user:12345 page_views
- HMSET counter_meta:user:12345 last_active "2026-01-07T10:30:00Z"

4.1.3 Time-Series Counter (SORTED SET)​

Key Pattern: ts_counter:{name}
Value: Sorted Set (timestamp → count)

Example: Hourly page views
┌──────────────────────────────────────────────────────────┐
│ Key: ts_counter:page_views:hourly │
├──────────────────────┬───────────────────────────────────┤
│ Score (Timestamp) │ Member (Count) │
├──────────────────────┼───────────────────────────────────┤
│ 1704697200 │ 15,234 (2026-01-07 08:00) │
│ 1704700800 │ 18,456 (2026-01-07 09:00) │
│ 1704704400 │ 21,789 (2026-01-07 10:00) │
│ 1704708000 │ 19,234 (2026-01-07 11:00) │
└──────────────────────┴───────────────────────────────────┘

Commands:
- ZADD ts_counter:page_views:hourly 1704697200 15234
- ZRANGE ts_counter:page_views:hourly 0 -1 WITHSCORES
- ZRANGEBYSCORE ts_counter:page_views:hourly 1704697200 1704708000
- ZCARD ts_counter:page_views:hourly → Get total entries

4.1.4 Distributed Counter with Sharding​

Strategy: Split counter across multiple keys to reduce contention

Pattern: counter:{name}:shard:{N}
Total = SUM of all shards

Example: High-traffic page view counter
┌────────────────────────────────┬────────────┐
│ Key │ Value │
├────────────────────────────────┼────────────┤
│ counter:homepage:shard:0 │ 342,156 │
│ counter:homepage:shard:1 │ 338,942 │
│ counter:homepage:shard:2 │ 345,789 │
│ counter:homepage:shard:3 │ 341,234 │
└────────────────────────────────┴────────────┘

Total = 342,156 + 338,942 + 345,789 + 341,234 = 1,368,121

Algorithm:
1. Hash counter name to determine shard: shard_id = hash(user_id) % NUM_SHARDS
2. Increment specific shard: INCR counter:homepage:shard:{shard_id}
3. Read total: MGET counter:homepage:shard:* and sum

Benefits:
• Reduces lock contention on single key
• Distributes load across cluster
• Improves write throughput

4.2 Key Naming Conventions​

Standard Pattern: {namespace}:{entity}:{id}:{attribute}

Examples:
• counter:page_views - Simple global counter
• counter:user:12345:profile_views - User-specific counter
• counter:product:SKU789:stock - Product inventory
• counter:api:rate_limit:192.168.1.1 - IP-based rate limit
• counter:daily:signups:2026-01-07 - Date-based counter
• counter:campaign:SUMMER2026:clicks - Campaign tracking

Hierarchical Structure:
counter
├── global
│ ├── page_views
│ ├── api_calls
│ └── errors
├── user
│ ├── {user_id}
│ │ ├── profile_views
│ │ ├── posts_created
│ │ └── followers
├── product
│ └── {sku}
│ ├── views
│ ├── purchases
│ └── stock
└── daily
└── {date}
├── signups
├── revenue
└── orders

4.3 Data Retention Policies​

┌──────────────────────┬──────────────┬─────────────────────────┐
│ Counter Type │ Retention │ Strategy │
├──────────────────────┼──────────────┼─────────────────────────┤
│ Real-time metrics │ 24 hours │ TTL=86400 │
│ Daily aggregates │ 90 days │ TTL=7776000 │
│ Monthly aggregates │ 2 years │ TTL=63072000 │
│ Permanent counters │ Indefinite │ No TTL │
│ Rate limit counters │ 1 hour │ TTL=3600 │
└──────────────────────┴──────────────┴─────────────────────────┘

Automatic Archival:
• Hourly job exports old counters to Data Warehouse
• Compressed and stored in S3/GCS
• Queryable via BigQuery/Athena

5. API Design​

5.1 REST API Endpoints​

5.1.1 Counter Operations​

POST /api/v1/counters/{counterName}/increment
Content-Type: application/json

{
"value": 1,
"metadata": {
"user_id": "12345",
"source": "web"
}
}

Response 200 OK:
{
"counter": "page_views",
"value": 1001,
"timestamp": "2026-01-07T10:30:45Z"
}

---

GET /api/v1/counters/{counterName}

Response 200 OK:
{
"counter": "page_views",
"value": 1001,
"created_at": "2025-01-01T00:00:00Z",
"updated_at": "2026-01-07T10:30:45Z"
}

---

POST /api/v1/counters/{counterName}/decrement
Content-Type: application/json

{
"value": 1
}

Response 200 OK:
{
"counter": "product_stock",
"value": 149,
"timestamp": "2026-01-07T10:30:50Z"
}

---

POST /api/v1/counters/{counterName}/reset

Response 200 OK:
{
"counter": "daily_views",
"value": 0,
"timestamp": "2026-01-07T00:00:00Z"
}

---

POST /api/v1/counters/batch
Content-Type: application/json

{
"operations": [
{ "counter": "page_views", "action": "increment", "value": 1 },
{ "counter": "api_calls", "action": "increment", "value": 5 },
{ "counter": "errors", "action": "increment", "value": 1 }
]
}

Response 200 OK:
{
"results": [
{ "counter": "page_views", "value": 1002 },
{ "counter": "api_calls", "value": 5678 },
{ "counter": "errors", "value": 43 }
]
}

5.2 Client Library Interface​

5.2.1 Java Client (Using Lettuce)​

public interface CounterClient {
/**
* Increment counter by 1
* @param counterName Name of the counter
* @return New counter value
*/
CompletableFuture<Long> increment(String counterName);

/**
* Increment counter by specified value
* @param counterName Name of the counter
* @param value Amount to increment
* @return New counter value
*/
CompletableFuture<Long> incrementBy(String counterName, long value);

/**
* Decrement counter by specified value
* @param counterName Name of the counter
* @param value Amount to decrement
* @return New counter value
*/
CompletableFuture<Long> decrementBy(String counterName, long value);

/**
* Get current counter value
* @param counterName Name of the counter
* @return Current value
*/
CompletableFuture<Long> get(String counterName);

/**
* Reset counter to zero
* @param counterName Name of the counter
* @return Success status
*/
CompletableFuture<Boolean> reset(String counterName);

/**
* Batch operations
* @param operations List of counter operations
* @return Map of counter names to new values
*/
CompletableFuture<Map<String, Long>> batchExecute(
List<CounterOperation> operations
);
}

// Usage Example
public class CounterService {
private final CounterClient counterClient;

public void trackPageView(String userId, String page) {
counterClient.increment("page_views")
.thenCompose(count ->
counterClient.incrementBy("user:" + userId + ":views", 1))
.whenComplete((result, error) -> {
if (error != null) {
log.error("Failed to track page view", error);
} else {
log.info("Page view tracked: {}", result);
}
});
}
}

5.2.2 Node.js Client (Using ioredis)​

class CounterClient {
constructor(redisClient) {
this.redis = redisClient;
this.localCache = new Map();
this.cacheTTL = 60000; // 60 seconds
}

/**
* Increment counter by 1
* @param {string} counterName - Name of the counter
* @returns {Promise<number>} New counter value
*/
async increment(counterName) {
const key = `counter:${counterName}`;
const value = await this.redis.incr(key);
this.updateLocalCache(counterName, value);
return value;
}

/**
* Increment counter by specified value
* @param {string} counterName - Name of the counter
* @param {number} value - Amount to increment
* @returns {Promise<number>} New counter value
*/
async incrementBy(counterName, value) {
const key = `counter:${counterName}`;
const newValue = await this.redis.incrby(key, value);
this.updateLocalCache(counterName, newValue);
return newValue;
}

/**
* Get current counter value (with local cache)
* @param {string} counterName - Name of the counter
* @returns {Promise<number>} Current value
*/
async get(counterName) {
// Check local cache first
const cached = this.localCache.get(counterName);
if (cached && Date.now() - cached.timestamp < this.cacheTTL) {
return cached.value;
}

// Fetch from Redis
const key = `counter:${counterName}`;
const value = await this.redis.get(key);
const numValue = parseInt(value) || 0;

this.updateLocalCache(counterName, numValue);
return numValue;
}

/**
* Batch operations using pipeline
* @param {Array<Object>} operations - List of operations
* @returns {Promise<Array<number>>} Results
*/
async batchExecute(operations) {
const pipeline = this.redis.pipeline();

for (const op of operations) {
const key = `counter:${op.counter}`;
switch (op.action) {
case 'increment':
pipeline.incrby(key, op.value || 1);
break;
case 'decrement':
pipeline.decrby(key, op.value || 1);
break;
case 'get':
pipeline.get(key);
break;
}
}

const results = await pipeline.exec();
return results.map(([err, value]) => err ? null : value);
}

updateLocalCache(counterName, value) {
this.localCache.set(counterName, {
value,
timestamp: Date.now()
});
}
}

// Usage Example
const counterClient = new CounterClient(redisClient);

// Track page view
await counterClient.increment('page_views');

// Batch operations
const results = await counterClient.batchExecute([
{ counter: 'page_views', action: 'increment', value: 1 },
{ counter: 'api_calls', action: 'increment', value: 5 },
{ counter: 'errors', action: 'get' }
]);

5.2.3 Python Client (Using redis-py)​

from typing import Dict, List, Optional
import redis
from redis.cluster import RedisCluster
import asyncio
from datetime import datetime, timedelta

class CounterClient:
def __init__(self, redis_client: RedisCluster):
self.redis = redis_client
self.local_cache: Dict[str, tuple[int, datetime]] = {}
self.cache_ttl = timedelta(seconds=60)

async def increment(self, counter_name: str) -> int:
"""Increment counter by 1"""
key = f"counter:{counter_name}"
value = await self.redis.incr(key)
self._update_local_cache(counter_name, value)
return value

async def increment_by(self, counter_name: str, value: int) -> int:
"""Increment counter by specified value"""
key = f"counter:{counter_name}"
new_value = await self.redis.incrby(key, value)
self._update_local_cache(counter_name, new_value)
return new_value

async def get(self, counter_name: str) -> int:
"""Get current counter value with local caching"""
# Check local cache
if counter_name in self.local_cache:
value, timestamp = self.local_cache[counter_name]
if datetime.now() - timestamp < self.cache_ttl:
return value

# Fetch from Redis
key = f"counter:{counter_name}"
value = await self.redis.get(key)
num_value = int(value) if value else 0

self._update_local_cache(counter_name, num_value)
return num_value

async def batch_execute(self, operations: List[Dict]) -> List[int]:
"""Execute multiple operations in a pipeline"""
pipeline = self.redis.pipeline()

for op in operations:
key = f"counter:{op['counter']}"
action = op['action']
value = op.get('value', 1)

if action == 'increment':
pipeline.incrby(key, value)
elif action == 'decrement':
pipeline.decrby(key, value)
elif action == 'get':
pipeline.get(key)

results = await pipeline.execute()
return [int(r) if r else 0 for r in results]

def _update_local_cache(self, counter_name: str, value: int):
self.local_cache[counter_name] = (value, datetime.now())

# Usage
counter_client = CounterClient(redis_client)

# Track page view
count = await counter_client.increment('page_views')
print(f"Page views: {count}")

# Batch operations
results = await counter_client.batch_execute([
{'counter': 'page_views', 'action': 'increment'},
{'counter': 'api_calls', 'action': 'increment', 'value': 5},
{'counter': 'errors', 'action': 'get'}
])

6. Implementation Details​

6.1 Connection Management​

6.1.1 Connection Pool Configuration​

// Java - Lettuce Configuration
@Configuration
public class RedisConfig {

@Bean
public RedisClusterClient redisClusterClient() {
List<RedisURI> nodes = Arrays.asList(
RedisURI.create("redis://node1:6379"),
RedisURI.create("redis://node2:6379"),
RedisURI.create("redis://node3:6379")
);

ClusterClientOptions options = ClusterClientOptions.builder()
.autoReconnect(true)
.maxRedirects(3)
.validateClusterNodeMembership(true)
.build();

return RedisClusterClient.create(nodes);
}

@Bean
public GenericObjectPoolConfig<StatefulRedisClusterConnection> poolConfig() {
GenericObjectPoolConfig config = new GenericObjectPoolConfig();
config.setMaxTotal(50); // Max connections
config.setMaxIdle(20); // Max idle connections
config.setMinIdle(5); // Min idle connections
config.setMaxWaitMillis(2000); // Max wait time
config.setTestOnBorrow(true); // Test before borrowing
config.setTestWhileIdle(true); // Test during idle
return config;
}
}
// Node.js - ioredis Cluster Configuration
const Redis = require('ioredis');

const cluster = new Redis.Cluster([
{ host: 'node1', port: 6379 },
{ host: 'node2', port: 6379 },
{ host: 'node3', port: 6379 }
], {
redisOptions: {
password: process.env.REDIS_PASSWORD,
tls: process.env.REDIS_TLS === 'true' ? {} : undefined
},
clusterRetryStrategy: (times) => {
return Math.min(100 * times, 2000);
},
enableReadyCheck: true,
maxRedirections: 16,
retryDelayOnFailover: 100,
retryDelayOnClusterDown: 300,
slotsRefreshTimeout: 1000,
enableOfflineQueue: true,
lazyConnect: false
});

cluster.on('error', (err) => {
console.error('Redis Cluster Error:', err);
});

cluster.on('ready', () => {
console.log('Redis Cluster Ready');
});

module.exports = cluster;

6.2 Error Handling & Retry Logic​

6.2.1 Exponential Backoff Strategy​

Retry Attempt Flow:
┌──────────────────────────────────────────────────────────┐
│ Attempt 1: Immediate │
│ ↓ Failure │
│ Wait: 100ms │
│ ↓ │
│ Attempt 2 │
│ ↓ Failure │
│ Wait: 200ms │
│ ↓ │
│ Attempt 3 │
│ ↓ Failure │
│ Wait: 400ms │
│ ↓ │
│ Attempt 4 │
│ ↓ Failure │
│ Wait: 800ms │
│ ↓ │
│ Attempt 5 │
│ ↓ Failure │
│ Give up, return error │
└──────────────────────────────────────────────────────────┘

Formula: delay = min(baseDelay * 2^attempt, maxDelay)
public class RetryableCounterClient {
private static final int MAX_RETRIES = 5;
private static final int BASE_DELAY_MS = 100;
private static final int MAX_DELAY_MS = 5000;

public CompletableFuture<Long> incrementWithRetry(String counterName) {
return retryOperation(() -> increment(counterName), 0);
}

private <T> CompletableFuture<T> retryOperation(
Supplier<CompletableFuture<T>> operation,
int attempt) {

return operation.get()
.exceptionally(error -> {
if (attempt >= MAX_RETRIES) {
throw new RuntimeException("Max retries exceeded", error);
}

if (isRetryableError(error)) {
int delay = calculateDelay(attempt);
log.warn("Retry attempt {} after {}ms", attempt + 1, delay);

return CompletableFuture
.delayedExecutor(delay, TimeUnit.MILLISECONDS)
.execute(() -> retryOperation(operation, attempt + 1));
} else {
throw new RuntimeException("Non-retryable error", error);
}
});
}

private int calculateDelay(int attempt) {
return Math.min(
BASE_DELAY_MS * (int) Math.pow(2, attempt),
MAX_DELAY_MS
);
}

private boolean isRetryableError(Throwable error) {
return error instanceof TimeoutException ||
error instanceof RedisConnectionException ||
error instanceof RedisClusterException;
}
}

6.3 Atomic Operations & Race Conditions​

6.3.1 Inventory Deduction with Stock Check​

-- Lua script for atomic stock deduction
-- KEYS[1]: counter key (e.g., "counter:product:SKU123:stock")
-- ARGV[1]: amount to deduct
-- Returns: new stock level or -1 if insufficient stock

local current = redis.call('GET', KEYS[1])
if not current then
return -1
end

local stock = tonumber(current)
local deduct = tonumber(ARGV[1])

if stock < deduct then
return -1 -- Insufficient stock
end

local new_stock = redis.call('DECRBY', KEYS[1], deduct)
return new_stock
public class InventoryService {
private final RedisCommands<String, String> redis;
private final String LUA_SCRIPT = """
local current = redis.call('GET', KEYS[1])
if not current then return -1 end
local stock = tonumber(current)
if stock < tonumber(ARGV[1]) then return -1 end
return redis.call('DECRBY', KEYS[1], ARGV[1])
""";

public boolean deductInventory(String sku, int quantity) {
String key = "counter:product:" + sku + ":stock";

Long result = redis.eval(
LUA_SCRIPT,
ScriptOutputType.INTEGER,
new String[]{key},
String.valueOf(quantity)
);

if (result == -1) {
log.warn("Insufficient stock for SKU: {}", sku);
return false;
}

log.info("Stock deducted for {}: new level = {}", sku, result);
return true;
}
}

6.3.2 Rate Limiting with Sliding Window​

-- Sliding window rate limiter
-- KEYS[1]: rate limit key
-- ARGV[1]: window size (seconds)
-- ARGV[2]: max requests
-- ARGV[3]: current timestamp

local key = KEYS[1]
local window = tonumber(ARGV[1])
local max_requests = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

-- Remove old entries outside the window
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)

-- Count current requests in window
local current = redis.call('ZCARD', key)

if current < max_requests then
-- Add new request
redis.call('ZADD', key, now, now)
redis.call('EXPIRE', key, window)
return {1, max_requests - current - 1} -- [allowed, remaining]
else
return {0, 0} -- [not allowed, remaining = 0]
end

6.4 Bulk Operations Optimization​

6.4.1 Batch Processing Pattern​

Single Operation Pattern (Inefficient):
┌─────────────────────────────────────────────┐
│ For each increment: │
│ 1. Network round trip (1-5ms) │
│ 2. Redis processing (0.1ms) │
│ 3. Network return (1-5ms) │
│ │
│ Total for 100 increments: 200-1000ms │
└─────────────────────────────────────────────┘

Pipeline Pattern (Efficient):
┌─────────────────────────────────────────────┐
│ Batch 100 operations: │
│ 1. Single network round trip (1-5ms) │
│ 2. Redis processes all (10ms) │
│ 3. Single network return (1-5ms) │
│ │
│ Total for 100 increments: 12-20ms │
│ Speedup: 10-50x faster! │
└─────────────────────────────────────────────┘
class BatchProcessor {
constructor(counterClient) {
this.client = counterClient;
this.buffer = [];
this.flushInterval = 100; // ms
this.maxBatchSize = 1000;

// Start periodic flush
setInterval(() => this.flush(), this.flushInterval);
}

async increment(counterName, value = 1) {
return new Promise((resolve, reject) => {
this.buffer.push({
counter: counterName,
action: 'increment',
value,
resolve,
reject
});

// Flush immediately if buffer is full
if (this.buffer.length >= this.maxBatchSize) {
this.flush();
}
});
}

async flush() {
if (this.buffer.length === 0) return;

const batch = this.buffer.splice(0, this.maxBatchSize);

try {
const results = await this.client.batchExecute(
batch.map(op => ({
counter: op.counter,
action: op.action,
value: op.value
}))
);

// Resolve all promises
batch.forEach((op, index) => {
op.resolve(results[index]);
});
} catch (error) {
// Reject all promises
batch.forEach(op => op.reject(error));
}
}
}

// Usage
const batchProcessor = new BatchProcessor(counterClient);

// These will be batched automatically
await Promise.all([
batchProcessor.increment('page_views'),
batchProcessor.increment('api_calls', 5),
batchProcessor.increment('errors')
]);

7. Performance Optimization​

7.1 Performance Metrics​

Target Performance SLAs:
┌────────────────────────┬──────────────┬─────────────┬──────────────┐
│ Operation │ P50 │ P95 │ P99 │
├────────────────────────┼──────────────┼─────────────┼──────────────┤
│ Single increment │ 1-2 ms │ 5 ms │ 10 ms │
│ Batch increment (100) │ 5-10 ms │ 20 ms │ 30 ms │
│ Read (with cache) │ 0.1-0.5 ms │ 1 ms │ 2 ms │
│ Read (no cache) │ 1-2 ms │ 5 ms │ 10 ms │
└────────────────────────┴──────────────┴─────────────┴──────────────┘

Throughput Targets:
• Single Redis node: 50,000-100,000 ops/sec
• 3-node cluster: 150,000-300,000 ops/sec
• With client-side caching: 500,000+ reads/sec

7.2 Caching Strategy​

7.2.1 Multi-Level Caching​

┌────────────────────────────────────────────────────────────┐
│ Application Request │
└──────────────────┬─────────────────────────────────────────┘
│
▼
┌─────────────────┐
│ L1: In-Memory │
│ (Process-local)│
│ TTL: 10-60s │
│ Size: 1,000 │
└────┬────────────┘
│ Cache miss
▼
┌─────────────────┐
│ L2: Redis │
│ (Distributed) │
│ TTL: 300s │
│ Size: Unlimited│
└────┬────────────┘
│ Cache miss
▼
┌─────────────────┐
│ L3: Database │
│ (PostgreSQL) │
│ TTL: Permanent │
└─────────────────┘

Performance Comparison:
• L1 hit: 0.1 ms (99% hit rate for hot keys)
• L2 hit: 1-2 ms (95% hit rate overall)
• L3 hit: 10-50 ms (rare, only on cold start)

7.2.2 Cache Invalidation Strategies​

public class CacheInvalidationManager {

// Strategy 1: TTL-based (simplest)
public void setWithTTL(String key, long value, int ttlSeconds) {
redis.setex(key, ttlSeconds, String.valueOf(value));
}

// Strategy 2: Write-through (consistency)
public long incrementWithInvalidation(String counterName) {
String key = "counter:" + counterName;
long newValue = redis.incr(key);

// Invalidate local caches across all app servers
cacheInvalidationBus.publish("invalidate:" + counterName);

return newValue;
}

// Strategy 3: Lazy invalidation (eventual consistency)
public long getWithLazyInvalidation(String counterName) {
LocalCacheEntry entry = localCache.get(counterName);

if (entry != null && !entry.isExpired()) {
return entry.getValue();
}

// Fetch fresh value
long value = redis.get("counter:" + counterName);
localCache.put(counterName, value, Duration.ofSeconds(60));

return value;
}
}

7.3 Sharding Strategy​

7.3.1 Consistent Hashing​

Hash Ring Distribution:
0°
│
Node C │ Node A
(240°) │ (0°)
╲ │ ╱
╲ │ ╱
╲ │ ╱
╲ │ ╱
╲│╱
180° ────────────── 0°/360°
╱│╲
╱ │ ╲
╱ │ ╲
╱ │ ╲
╱ │ ╲
Node B │ Node A
(120°) │ (360°)
│
180°

Key Mapping:
• hash("counter:page_views") = 45° → Node A
• hash("counter:api_calls") = 135° → Node B
• hash("counter:errors") = 270° → Node C

Benefits:
• Minimal key redistribution on node add/remove
• Balanced load distribution
• Built into Redis Cluster

7.3.2 Manual Sharding for Hot Keys​

public class ShardedCounterClient {
private static final int NUM_SHARDS = 10;

public long incrementSharded(String counterName) {
// Distribute writes across shards
int shardId = ThreadLocalRandom.current().nextInt(NUM_SHARDS);
String shardedKey = String.format(
"counter:%s:shard:%d",
counterName,
shardId
);

return redis.incr(shardedKey);
}

public long getTotal(String counterName) {
// Read from all shards and sum
List<CompletableFuture<Long>> futures = new ArrayList<>();

for (int i = 0; i < NUM_SHARDS; i++) {
String shardedKey = String.format(
"counter:%s:shard:%d",
counterName,
i
);
futures.add(redis.getAsync(shardedKey));
}

return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
.thenApply(v -> futures.stream()
.mapToLong(f -> f.join())
.sum())
.join();
}
}

8. High Availability & Disaster Recovery​

8.1 Redis Cluster Architecture​

High Availability Setup (3 Masters + 3 Replicas):

┌────────────────────────────────────────────────────────────┐
│ Sentinel Monitoring Layer │
│ [Sentinel 1] [Sentinel 2] [Sentinel 3] │
│ • Quorum: 2 (majority of 3) │
│ • Health check interval: 1s │
│ • Failover timeout: 3s │
└──────────────────┬─────────────────────────────────────────┘
│ Monitors
▼
┌────────────────────────────────────────────────────────────┐
│ Redis Cluster Nodes │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Master 1 │────────▶│Replica 1a│ │Replica 1b│ │
│ │ Slots: │ │(standby) │ │(standby) │ │
│ │ 0-5460 │ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Master 2 │────────▶│Replica 2a│ │Replica 2b│ │
│ │ Slots: │ │(standby) │ │(standby) │ │
│ │5461-10922│ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Master 3 │────────▶│Replica 3a│ │Replica 3b│ │
│ │ Slots: │ │(standby) │ │(standby) │ │
│ │10923- │ │ │ │ │ │
│ │16383 │ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└────────────────────────────────────────────────────────────┘

Failover Scenario:
1. Master 1 fails
2. Sentinels detect failure (1-3s)
3. Sentinels vote on promotion (quorum = 2)
4. Replica 1a promoted to Master
5. Clients automatically redirected
6. Total downtime: 3-5 seconds

8.2 Backup Strategy​

Backup Architecture:
┌────────────────────────────────────────────────────────────┐
│ Redis Master Nodes │
│ • RDB snapshots every 5 minutes │
│ • AOF log (append-only, fsync every second) │
└──────────────────┬─────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ Local Backup Storage │
│ • Last 24 hours of RDB snapshots │
│ • Rolling AOF logs │
│ • Retention: 24 hours │
└──────────────────┬─────────────────────────────────────────┘
│
│ Hourly sync
▼
┌────────────────────────────────────────────────────────────┐
│ Cloud Storage (S3/GCS) │
│ • Compressed and encrypted backups │
│ • Retention policy: │
│ - Hourly: 7 days │
│ - Daily: 30 days │
│ - Weekly: 1 year │
│ - Monthly: 7 years (compliance) │
└────────────────────────────────────────────────────────────┘

Recovery Time Objectives (RTO):
• Hot standby failover: 3-5 seconds
• Restore from local backup: 5-10 minutes
• Restore from cloud backup: 30-60 minutes
• Disaster recovery (full rebuild): 2-4 hours

Recovery Point Objectives (RPO):
• With AOF: < 1 second data loss
• With RDB only: Up to 5 minutes data loss
• After disaster: Up to 1 hour data loss

8.3 Disaster Recovery Procedures​

#!/bin/bash
# Disaster Recovery Script

DR_SCENARIO=$1 # "master_failure", "cluster_down", "data_corruption"

case $DR_SCENARIO in
"master_failure")
echo "Scenario: Redis Master Node Failure"
echo "1. Sentinels detect failure automatically"
echo "2. Wait for automatic failover (3-5s)"
echo "3. Verify new master promotion"
redis-cli -h sentinel-1 -p 26379 SENTINEL get-master-addr-by-name mymaster

echo "4. Check cluster health"
redis-cli --cluster check redis-node1:6379
;;

"cluster_down")
echo "Scenario: Entire Cluster Failure"
echo "1. Stop all Redis processes"
systemctl stop redis-cluster

echo "2. Restore from latest backup"
LATEST_BACKUP=$(aws s3 ls s3://redis-backups/ --recursive | sort | tail -n 1 | awk '{print $4}')
aws s3 cp s3://redis-backups/$LATEST_BACKUP /var/lib/redis/dump.rdb

echo "3. Start Redis cluster"
systemctl start redis-cluster

echo "4. Verify data integrity"
redis-cli --cluster check redis-node1:6379
;;

"data_corruption")
echo "Scenario: Data Corruption Detected"
echo "1. Identify corruption timestamp"
echo "2. Restore from point-in-time backup"

TARGET_TIME=$2
aws s3 cp s3://redis-backups/backup-$TARGET_TIME.rdb /var/lib/redis/dump.rdb

echo "3. Replay AOF from that point"
redis-check-aof --fix /var/lib/redis/appendonly.aof

echo "4. Restart and verify"
systemctl restart redis-cluster
;;
esac

# Health Check
echo "=== Cluster Health Check ==="
redis-cli --cluster info redis-node1:6379
redis-cli --cluster check redis-node1:6379

# Test Counter Operations
echo "=== Testing Counter Operations ==="
redis-cli INCR test:counter
redis-cli GET test:counter
redis-cli DEL test:counter

echo "Recovery completed at $(date)"

9. Security Considerations​

9.1 Security Architecture​

Security Layers:
┌────────────────────────────────────────────────────────────┐
│ Layer 1: Network Security │
│ • VPC/Private subnet │
│ • Security groups (allow only app servers) │
│ • No public internet access │
└──────────────────┬─────────────────────────────────────────┘
│
┌──────────────────┴─────────────────────────────────────────┐
│ Layer 2: Transport Security │
│ • TLS 1.3 encryption for all Redis connections │
│ • Certificate validation │
│ • Mutual TLS (mTLS) for client authentication │
└──────────────────┬─────────────────────────────────────────┘
│
┌──────────────────┴─────────────────────────────────────────┐
│ Layer 3: Authentication │
│ • Redis AUTH password (minimum 32 characters) │
│ • ACL (Access Control Lists) for fine-grained permissions │
│ • Separate credentials per application/environment │
└──────────────────┬─────────────────────────────────────────┘
│
┌──────────────────┴─────────────────────────────────────────┐
│ Layer 4: Authorization │
│ • ACL rules per user: │
│ - read-only users: ~* +@read │
│ - app users: counter:* +@write +@read │
│ - admin: * +@all │
└──────────────────┬─────────────────────────────────────────┘
│
┌──────────────────┴─────────────────────────────────────────┐
│ Layer 5: Data Protection │
│ • Encryption at rest (disk encryption) │
│ • Encrypted backups │
│ • Sensitive data masking in logs │
└────────────────────────────────────────────────────────────┘

9.2 Redis ACL Configuration​

# redis.conf - Enable ACL
aclfile /etc/redis/users.acl
requirepass your-strong-password-here

# /etc/redis/users.acl
# Admin user (full access)
user admin on >StrongAdminPass123! ~* +@all

# Application user (limited to counter operations)
user counter_app on >AppSecretPass456! ~counter:* +@write +@read +@string -@dangerous

# Read-only monitoring user
user monitor on >MonitorPass789! ~* +@read -@write -@dangerous

# Disable default user
user default off nopass ~* -@all

9.3 Security Best Practices Checklist​

✓ Network Security:
☑ Redis nodes in private subnet (no public IP)
☑ Security groups allow only app servers (port 6379)
☑ Sentinel nodes in separate security group (port 26379)
☑ VPN required for admin access

✓ Authentication & Authorization:
☑ Strong Redis AUTH password (32+ characters)
☑ ACL enabled with principle of least privilege
☑ Separate users for apps, admins, monitoring
☑ Password rotation every 90 days
☑ No hardcoded credentials (use secrets manager)

✓ Encryption:
☑ TLS 1.3 for all Redis connections
☑ Encryption at rest for data volumes
☑ Encrypted backups in S3/GCS
☑ Certificate pinning in clients

✓ Monitoring & Auditing:
☑ All Redis commands logged
☑ Failed authentication attempts alerted
☑ Abnormal traffic patterns detected
☑ Regular security audits scheduled

✓ Operational Security:
☑ Disable dangerous commands (FLUSHALL, KEYS, CONFIG)
☑ Resource limits (maxmemory, maxclients)
☑ Rate limiting at application layer
☑ Regular security patches applied

10. Monitoring & Observability​

10.1 Metrics Collection​

Monitoring Stack Architecture:
┌────────────────────────────────────────────────────────────┐
│ Redis Cluster Nodes │
│ • Expose metrics on :9121 (redis_exporter) │
└──────────────────┬─────────────────────────────────────────┘
│
│ Scrape every 15s
▼
┌────────────────────────────────────────────────────────────┐
│ Prometheus Server │
│ • Store metrics (15 days retention) │
│ • Evaluate alerting rules │
│ • Federation to long-term storage │
└──────────────────┬─────────────────────────────────────────┘
│
┌─────────┴─────────┬──────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────┐ ┌──────────────┐
│ Grafana │ │ AlertManager│ │ VictoriaDB │
│ (Dashboards) │ │ (Alerts) │ │ (Long-term) │
│ • Real-time │ │ • PagerDuty│ │ • 1y data │
│ • Historical │ │ • Slack │ │ • Analysis │
└─────────────────┘ └─────────────┘ └──────────────┘

10.2 Key Metrics to Monitor​

10.2.1 Redis Metrics​

# Critical Redis Metrics
metrics:
performance:
- redis_commands_processed_total
description: "Total commands processed"
threshold: "< 100k/s = Warning"

- redis_instantaneous_ops_per_sec
description: "Current ops/sec"
threshold: "> 80k/s = Capacity planning needed"

- redis_command_duration_seconds
description: "Command latency"
threshold: "P99 > 10ms = Alert"

memory:
- redis_memory_used_bytes
description: "Memory usage"
threshold: "> 80% = Warning, > 90% = Critical"

- redis_memory_fragmentation_ratio
description: "Memory fragmentation"
threshold: "> 1.5 = Consider restart"

- redis_evicted_keys_total
description: "Evicted keys (maxmemory-policy)"
threshold: "> 0 = Warning (data loss)"

replication:
- redis_connected_slaves
description: "Number of connected replicas"
threshold: "< 1 = Critical (no HA)"

- redis_master_repl_offset
description: "Replication lag (bytes)"
threshold: "> 10MB = Warning"

connections:
- redis_connected_clients
description: "Active client connections"
threshold: "> 10k = Warning"

- redis_rejected_connections_total
description: "Connection rejections"
threshold: "> 0 = Critical (maxclients reached)"

persistence:
- redis_rdb_last_save_timestamp_seconds
description: "Last successful save"
threshold: "> 600s = Warning (backup failed)"

- redis_aof_last_rewrite_duration_sec
description: "AOF rewrite duration"
threshold: "> 60s = Performance impact"

10.2.2 Application Metrics​

// Micrometer metrics in Java
@Service
public class CounterMetricsService {
private final MeterRegistry meterRegistry;

public void recordIncrement(String counterName, long latencyMs) {
// Counter metric
meterRegistry.counter(
"counter.operations",
"operation", "increment",
"counter_name", counterName
).increment();

// Latency histogram
meterRegistry.timer(
"counter.latency",
"operation", "increment"
).record(latencyMs, TimeUnit.MILLISECONDS);
}

public void recordError(String counterName, Exception error) {
meterRegistry.counter(
"counter.errors",
"counter_name", counterName,
"error_type", error.getClass().getSimpleName()
).increment();
}

public void recordCacheHit(boolean hit) {
meterRegistry.counter(
"counter.cache",
"result", hit ? "hit" : "miss"
).increment();
}
}

// Expose custom metrics
@RestController
public class MetricsController {
@GetMapping("/metrics/counters")
public Map<String, Object> getCounterMetrics() {
return Map.of(
"total_operations", getTotalOperations(),
"cache_hit_rate", getCacheHitRate(),
"average_latency_ms", getAverageLatency(),
"error_rate", getErrorRate()
);
}
}

10.3 Grafana Dashboards​

Dashboard: Redis Counter System Overview
┌──────────────────────────────────────────────────────────┐
│ Time Range: Last 24h Auto-refresh: 30s │
├──────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Operations │ │ Latency P99 │ │ Error Rate │ │
│ │ 125k/s │ │ 3.2 ms │ │ 0.01% │ │
│ │ ↑ 15% │ │ ↓ 0.5ms │ │ ↓ 0.005% │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Operations per Second (24h) │ │
│ │ [Line graph showing ops/sec over time] │ │
│ │ Peak: 150k at 14:00 │ │
│ │ Current: 125k │ │
│ └──────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────┐ ┌──────────────────────────┐ │
│ │ Memory Usage │ │ Top Counters (by ops) │ │
│ │ [Gauge: 65%] │ │ 1. page_views: 45k/s │ │
│ │ 13GB / 20GB │ │ 2. api_calls: 30k/s │ │
│ │ │ │ 3. user_actions: 25k/s │ │
│ └────────────────────┘ │ 4. events: 15k/s │ │
│ │ 5. clicks: 10k/s │ │
│ ┌────────────────────┐ └──────────────────────────┘ │
│ │ Cache Hit Rate │ │
│ │ [Pie chart] │ ┌──────────────────────────┐ │
│ │ Hits: 95% │ │ Replication Lag │ │
│ │ Misses: 5% │ │ [Line graph] │ │
│ └────────────────────┘ │ Master → Replica: 0.2s │ │
│ └──────────────────────────┘ │
├──────────────────────────────────────────────────────────┤
│ Recent Alerts: │
│ • None - All systems operational ✓ │
└──────────────────────────────────────────────────────────┘

10.4 Alerting Rules​

# prometheus-alerts.yml
groups:
- name: redis_counter_alerts
interval: 30s
rules:
# High latency alert
- alert: HighRedisLatency
expr: histogram_quantile(0.99, rate(redis_command_duration_seconds_bucket[5m])) > 0.010
for: 5m
labels:
severity: warning
component: redis
annotations:
summary: "High Redis latency detected"
description: "P99 latency is {{ $value }}s (threshold: 10ms)"

# Memory usage alert
- alert: RedisMemoryHigh
expr: (redis_memory_used_bytes / redis_memory_max_bytes) > 0.90
for: 5m
labels:
severity: critical
component: redis
annotations:
summary: "Redis memory usage critical"
description: "Memory usage is {{ $value | humanizePercentage }} (threshold: 90%)"

# Replication lag alert
- alert: ReplicationLagHigh
expr: redis_master_repl_offset - redis_slave_repl_offset > 10485760
for: 2m
labels:
severity: warning
component: redis
annotations:
summary: "High replication lag"
description: "Lag is {{ $value | humanize1024 }}B behind master"

# Connection pool exhaustion
- alert: ConnectionPoolExhausted
expr: rate(redis_rejected_connections_total[5m]) > 0
for: 1m
labels:
severity: critical
component: redis
annotations:
summary: "Redis rejecting connections"
description: "Connection limit reached - scale up"

# Application error rate
- alert: HighCounterErrorRate
expr: rate(counter_errors_total[5m]) > 10
for: 5m
labels:
severity: warning
component: application
annotations:
summary: "High counter operation error rate"
description: "Error rate is {{ $value }} errors/sec"

11. Deployment Strategy​

11.1 Environment Setup​

Environment Progression:
┌────────────────────────────────────────────────────────┐
│ Development (Local) │
│ • Single Redis node │
│ • No replication │
│ • Volume: 1k ops/sec │
│ • Purpose: Feature development │
└────────────────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ Staging (Cloud) │
│ • 3-node cluster (1 master + 2 replicas per shard) │
│ • Sentinel enabled │
│ • Volume: 10k ops/sec │
│ • Purpose: Integration testing, QA │
└────────────────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ Production (Multi-region) │
│ • 9-node cluster (3 masters + 2 replicas each) │
│ • Sentinel + Redis Cluster mode │
│ • Volume: 100k+ ops/sec │
│ • Purpose: Live traffic │
└────────────────────────────────────────────────────────┘

11.2 Blue-Green Deployment​

Blue-Green Deployment Strategy:
┌────────────────────────────────────────────────────────┐
│ Load Balancer │
│ Route: 100% → Blue │
└──────────────────┬─────────────────────────────────────┘
│
┌──────────┴──────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Blue (Current) │ │ Green (New) │
│ v1.0 │ │ v1.1 │
│ │ │ │
│ Redis Cluster │ │ Redis Cluster │
│ • 3 shards │ │ • 3 shards │
│ • Production │ │ • Standby │
└─────────────────┘ └─────────────────┘

Deployment Steps:
1. Deploy v1.1 to Green environment
2. Run smoke tests on Green
3. Gradually shift traffic: 10% → 50% → 100%
4. Monitor for errors/latency
5. If successful: Blue becomes standby
6. If failure: Instant rollback to Blue

11.3 Canary Deployment​

Canary Deployment Flow:
┌────────────────────────────────────────────────────────┐
│ Step 1: Deploy to 1% of users │
│ • Monitor for 1 hour │
│ • Check: error rate, latency, success rate │
└────────────────┬───────────────────────────────────────┘
│ ✓ Success
▼
┌────────────────────────────────────────────────────────┐
│ Step 2: Increase to 10% │
│ • Monitor for 30 minutes │
└────────────────┬───────────────────────────────────────┘
│ ✓ Success
▼
┌────────────────────────────────────────────────────────┐
│ Step 3: Increase to 50% │
│ • Monitor for 15 minutes │
└────────────────┬───────────────────────────────────────┘
│ ✓ Success
▼
┌────────────────────────────────────────────────────────┐
│ Step 4: Full rollout to 100% │
│ • Continue monitoring │
└────────────────────────────────────────────────────────┘

Traffic Split Configuration (Istio):
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: counter-service
spec:
hosts:
- counter-service
http:
- match:
- headers:
canary:
exact: "true"
route:
- destination:
host: counter-service
subset: v2
weight: 100
- route:
- destination:
host: counter-service
subset: v1
weight: 90
- destination:
host: counter-service
subset: v2
weight: 10

11.4 Rolling Updates​

#!/bin/bash
# Rolling update script for Redis Cluster

NODES=("node1" "node2" "node3" "node4" "node5" "node6")

for NODE in "${NODES[@]}"; do
echo "Updating $NODE..."

# 1. Check if node is master
IS_MASTER=$(redis-cli -h $NODE -p 6379 ROLE | head -1)

if [ "$IS_MASTER" == "master" ]; then
echo "$NODE is master - triggering failover first"
redis-cli -h $NODE -p 6379 CLUSTER FAILOVER
sleep 5
fi

# 2. Stop Redis
ssh $NODE "systemctl stop redis"

# 3. Update Redis binary
ssh $NODE "wget https://download.redis.io/releases/redis-7.2.3.tar.gz"
ssh $NODE "tar xzf redis-7.2.3.tar.gz && cd redis-7.2.3 && make install"

# 4. Start Redis
ssh $NODE "systemctl start redis"

# 5. Wait for node to rejoin cluster
sleep 10

# 6. Verify cluster health
redis-cli --cluster check redis-node1:6379

echo "$NODE updated successfully"
echo "Waiting 60s before next node..."
sleep 60
done

echo "Rolling update completed"

12. Cost Analysis​

12.1 Infrastructure Costs (Monthly)​

AWS Cost Breakdown (Production):
┌────────────────────────────┬──────────┬────────┬──────────┐
│ Component │ Quantity │ Unit $ │ Total $ │
├────────────────────────────┼──────────┼────────┼──────────┤
│ EC2: r6g.2xlarge (Redis) │ 9 nodes │ $250 │ $2,250 │
│ EBS: gp3 100GB │ 9 vols │ $10 │ $90 │
│ EC2: m6i.large (App) │ 5 nodes │ $70 │ $350 │
│ ALB (Load Balancer) │ 1 │ $25 │ $25 │
│ Data Transfer (out) │ 1TB │ $90 │ $90 │
│ S3 Backup Storage │ 500GB │ $12 │ $12 │
│ CloudWatch (monitoring) │ - │ - │ $50 │
│ VPN/Direct Connect │ 1 │ $100 │ $100 │
├────────────────────────────┴──────────┴────────┼──────────┤
│ Total Monthly Cost │ $2,967 │
└─────────────────────────────────────────────────┴──────────┘

Cost per Million Operations:
• At 100M ops/day: $0.99 per million ops
• At 1B ops/day: $0.10 per million ops

Cost Optimization Opportunities:
• Reserved Instances: Save 30-40% ($890/month)
• Spot Instances for non-prod: Save 60-70%
• Graviton instances (r6g): 20% cheaper than r5
• S3 Glacier for old backups: 90% storage savings

12.2 Scaling Cost Projections​

Growth Scenario Analysis:
┌─────────────┬───────────┬────────────┬─────────────┬──────────┐
│ Daily Ops │ Nodes Req │ Monthly $ │ Cost/M ops │ YoY │
├─────────────┼───────────┼────────────┼─────────────┼──────────┤
│ 100M (now) │ 9 │ $2,967 │ $0.99 │ Baseline │
│ 500M (6mo) │ 15 │ $4,500 │ $0.30 │ +52% │
│ 1B (1yr) │ 27 │ $7,200 │ $0.24 │ +60% │
│ 5B (2yr) │ 45 │ $11,800 │ $0.08 │ +64% │
└─────────────┴───────────┴────────────┴─────────────┴──────────┘

Key Insights:
• Cost per operation decreases with scale
• Non-linear scaling due to better utilization
• Breaking even with managed Redis at 500M ops/day

13. Appendices​

Appendix A: Redis Commands Reference​

# Counter Operations
INCR counter:name # Increment by 1
INCRBY counter:name 5 # Increment by 5
DECR counter:name # Decrement by 1
DECRBY counter:name 3 # Decrement by 3
GET counter:name # Get current value
SET counter:name 0 # Reset to 0

# Atomic Operations
GETSET counter:name 100 # Set and return old value
APPEND counter:name "text" # Append to string

# Expiration
EXPIRE counter:name 3600 # Set TTL (seconds)
TTL counter:name # Check remaining TTL
PERSIST counter:name # Remove TTL

# Batch Operations
MGET counter:1 counter:2 # Get multiple
MSET counter:1 10 counter:2 20 # Set multiple

# Pipeline Example
redis-cli --pipe <<EOF
INCR counter:page_views
INCR counter:api_calls
INCR counter:errors
EOF

Appendix B: Performance Benchmarks​

# redis-benchmark command
redis-benchmark -h localhost -p 6379 \
-t set,get,incr,lpush,lpop \
-n 1000000 \
-c 50 \
-d 16 \
--csv

# Results (r6g.2xlarge):
"test","rps","avg_latency_ms","min_latency","p50","p95","p99","p99.9"
"INCR",98765,0.51,0.24,0.48,0.72,1.20,2.15
"GET",102340,0.49,0.20,0.45,0.68,1.15,2.10
"SET",95420,0.53,0.25,0.50,0.75,1.25,2.20
"LPUSH",89230,0.56,0.28,0.53,0.80,1.35,2.40

Appendix C: Troubleshooting Guide​

Common Issues and Solutions:
┌─────────────────────────┬──────────────────────────────────┐
│ Issue │ Solution │
├─────────────────────────┼──────────────────────────────────┤
│ High latency │ • Check network │
│ │ • Increase connection pool │
│ │ • Use pipelining │
│ │ • Check slow log │
├─────────────────────────┼──────────────────────────────────┤
│ Memory exhaustion │ • Set maxmemory-policy │
│ │ • Add TTL to counters │
│ │ • Scale horizontally │
│ │ • Check for memory leaks │
├─────────────────────────┼──────────────────────────────────┤
│ Connection refused │ • Check Redis is running │
│ │ • Verify network rules │
│ │ • Check maxclients limit │
│ │ • Review authentication │
├─────────────────────────┼──────────────────────────────────┤
│ Replication lag │ • Check network bandwidth │
│ │ • Reduce write load │
│ │ • Optimize large values │
│ │ • Consider more replicas │
├─────────────────────────┼──────────────────────────────────┤
│ Data inconsistency │ • Check AOF/RDB config │
│ │ • Review fsync settings │
│ │ • Verify cluster slots │
│ │ • Check for split brain │
└─────────────────────────┴──────────────────────────────────┘

Appendix D: Glossary​

  • AOF (Append-Only File): Durability mechanism that logs every write
  • Circuit Breaker: Pattern to prevent cascading failures
  • Consistent Hashing: Distribution algorithm for sharding
  • Gossip Protocol: Cluster communication mechanism
  • Pipelining: Sending multiple commands without waiting for replies
  • RDB (Redis Database): Point-in-time snapshot format
  • Sentinel: High availability monitoring and failover
  • Sharding: Distributing data across multiple nodes
  • Slot: Unit of data distribution (16384 slots in cluster)
  • Write-Through: Cache strategy where writes go to both cache and DB

Distributed Counter System with Redis

· 阅读需 23 分钟

1. Executive Summary​

This document outlines the design for a centralized counter service using Redis as the backend store in a distributed system. The solution addresses the need for consistent, performant, and reliable counter operations across multiple service instances, providing atomic increments, decrements, and retrieval with support for rate limiting, usage tracking, and real-time analytics.

Key Design Goals​

  • Strong consistency for counter operations across distributed nodes
  • High availability with 99.99% uptime target
  • Scalability to handle 100K+ operations per second
  • Low latency (<5ms for counter operations)
  • Fault tolerance with automatic failover and data durability

2. System Architecture​

2.1 High-Level Architecture Diagram​

graph TB
subgraph "Client Applications"
A1[Service A]
A2[Service B]
A3[Service C]
end

subgraph "Counter Service Layer"
B1[Counter API Gateway]
B2[Load Balancer]
B3[Service Registry]
end

subgraph "Redis Cluster"
C1[Master 1<br/>Shard 1]
C2[Slave 1.1]
C3[Slave 1.2]
C4[Master 2<br/>Shard 2]
C5[Slave 2.1]
C6[Slave 2.2]
C7[Master 3<br/>Shard 3]
C8[Slave 3.1]
C9[Slave 3.2]
end

subgraph "Supporting Services"
D1[Monitoring<br/>Prometheus/Grafana]
D2[Configuration Mgmt]
D3[Backup Service]
end

A1 --> B2
A2 --> B2
A3 --> B2
B2 --> B1
B1 --> C1
B1 --> C4
B1 --> C7
C1 --> C2
C1 --> C3
C4 --> C5
C4 --> C6
C7 --> C8
C7 --> C9
B1 --> D1
B1 --> D2
C1 --> D3
C4 --> D3
C7 --> D3

2.2 Component Responsibilities​

ComponentResponsibilityTechnology Options
Counter Client SDKProvides easy-to-use API for applicationsCustom SDK in multiple languages
API GatewayRequest routing, auth, rate limitingKong, AWS API Gateway, Custom
Counter ServiceBusiness logic, validation, aggregationNode.js, Go, Java Spring
Redis ClusterPrimary data store for countersRedis 7.x with Redis Stack
MonitoringMetrics collection and alertingPrometheus, Grafana, ELK Stack
Backup ServiceData persistence and disaster recoveryRedis RDB/AOF, S3 backup

3. Redis Data Design​

3.1 Key Structure Patterns​

graph LR
subgraph "Counter Key Design"
A[Counter Type] --> B[Namespace]
B --> C[Resource ID]
C --> D[Granularity]
D --> E[Full Key]
end

E --> F["counter:api:user_12345:20240120"]
E --> G["rate_limit:ip:192.168.1.1:minute"]
E --> H["inventory:product:sku_98765:warehouse"]

3.2 Data Models Implementation​

# Counter data structures in Redis
COUNTER_SCHEMA = {
# Simple counter (e.g., page views)
"page:views:homepage": {
"type": "string",
"encoding": "int",
"ttl": 2592000, # 30 days
"operations": ["INCR", "GET", "EXPIRE"]
},

# Hashed counter (e.g., user session counts by country)
"stats:sessions:20240120": {
"type": "hash",
"fields": {
"US": "integer",
"EU": "integer",
"ASIA": "integer"
},
"operations": ["HINCRBY", "HGETALL"]
},

# Sorted set for leaderboards
"leaderboard:game:season5": {
"type": "zset",
"score": "integer",
"member": "user_id",
"operations": ["ZINCRBY", "ZRANGE"]
},

# Bitmap for daily active users
"dau:20240120": {
"type": "bitmap",
"operations": ["SETBIT", "BITCOUNT"],
"max_bit": 10000000 # 10M users
}
}

3.3 Memory Optimization Strategies​

PatternUse CaseMemory SavingsExample
Integer encodingSmall counters75% vs stringINCR counter:id
Hash field packingMultiple related counters60% vs separate keysHINCRBY stats:20240120 US 1
Ziplist encodingSmall hashes/lists50-80% savingsredis.conf: hash-max-ziplist-entries 512
BitmapsBoolean flags96% vs setSETBIT active:users 12345 1
HyperLogLogApproximate counting99% vs setPFADD daily:visitors user123

4. Core Operations Design​

4.1 Atomic Operations Flow​

sequenceDiagram
participant Client as Client SDK
participant Service as Counter Service
participant Redis as Redis Master
participant Slave as Redis Replica

Client->>Service: increment("user:123:clicks", 1, ttl=3600)
Service->>Service: Validate request & rate limit
Service->>Redis: WATCH counter:user:123:clicks
Redis->>Service: OK
Service->>Redis: MULTI
Service->>Redis: INCRBY counter:user:123:clicks 1
Service->>Redis: EXPIRE counter:user:123:clicks 3600
Service->>Redis: EXEC
alt Success
Redis->>Service: [124] (new value)
Service->>Slave: Async replication
Service->>Client: {"success": true, "value": 124}
else Conflict
Redis->>Service: NULL (transaction failed)
Service->>Service: Retry logic (max 3 attempts)
Service->>Redis: Retry operations...
end

4.2 Counter Operation Patterns​

class RedisCounterService:
def __init__(self, redis_client):
self.redis = redis_client
self.lua_scripts = self._load_lua_scripts()

def _load_lua_scripts(self):
"""Load atomic Lua scripts for complex operations"""
return {
'increment_with_min_max': """
local key = KEYS[1]
local increment = tonumber(ARGV[1])
local min_val = tonumber(ARGV[2])
local max_val = tonumber(ARGV[3])
local ttl = tonumber(ARGV[4])

local current = redis.call('GET', key) or 0
current = tonumber(current)
local new_value = current + increment

-- Apply bounds
if min_val and new_value < min_val then
new_value = min_val
end
if max_val and new_value > max_val then
new_value = max_val
end

redis.call('SET', key, new_value)
if ttl > 0 then
redis.call('EXPIRE', key, ttl)
end
return new_value
""",

'batch_increment': """
-- Increment multiple counters atomically
for i = 1, #KEYS do
redis.call('INCRBY', KEYS[i], ARGV[i])
end
return #KEYS
"""
}

def increment_with_bounds(self, key, delta=1, min_val=None,
max_val=None, ttl=None):
"""Atomically increment with bounds checking"""
script = self.lua_scripts['increment_with_min_max']
args = [delta, min_val or '', max_val or '', ttl or 0]
return self.redis.eval(script, 1, key, *args)

def get_with_snapshot(self, key):
"""Get counter value with snapshot for consistency"""
# Use Redis transactions for consistent read
pipe = self.redis.pipeline()
pipe.watch(key)
value = pipe.get(key)
pipe.unwatch()
pipe.execute()
return int(value) if value else 0

5. Scalability and Performance​

5.1 Scaling Strategy​

graph TD
A[Traffic Increase] --> B{Request Type Analysis}

B -->|Read Heavy| C[Add Read Replicas]
B -->|Write Heavy| D[Shard Counters]
B -->|Mixed| E[Combined Approach]

C --> C1[2x Replicas per Shard]
C --> C2[Read-Balancing Proxy]
C --> C3[Local Cache Layer]

D --> D1[Key-based Sharding]
D --> D2[Consistent Hashing]
D --> D3[Proxy-based Sharding]

E --> E1[Redis Cluster Mode]
E --> E2[Proxy + Replication]

C1 --> F[Target: 100K RPS]
D1 --> G[Target: 50K WPS]
E1 --> H[Target: 70K OPS]

F --> I[Monitoring & Auto-scale]
G --> I
H --> I

5.2 Sharding Strategies​

StrategyHow It WorksProsConsBest For
Key-basedshard = hash(key) % NSimple, predictableResharding difficultStable key sets
Range-basedKey ranges per shardEfficient range queriesHotspot riskSequential keys
Directory-basedLookup service maps keysFlexible, easy rebalancingSingle point of failureDynamic environments
Consistent HashingHash ring distributionMinimal rehash on scaleComplex implementationCloud auto-scaling

5.3 Performance Benchmarks​

OperationSingle Node3-Node ClusterWith PipelineNotes
INCR80,000 OPS240,000 OPS500,000 OPSAtomic counter
HINCRBY70,000 OPS210,000 OPS450,000 OPSHashed counter
GET100,000 OPS300,000 OPS800,000 OPSSimple read
MULTI/EXEC40,000 OPS120,000 OPSN/ATransaction
EVAL (Lua)60,000 OPS180,000 OPSN/AScript execution

6. High Availability and Disaster Recovery​

6.1 HA Architecture​

graph TB
subgraph "Primary Region (us-east-1)"
subgraph "Availability Zone A"
M1[Master-1<br/>Primary]
S1[Slave-1.1<br/>Sync Replica]
end

subgraph "Availability Zone B"
S2[Slave-1.2<br/>Async Replica]
S3[Slave-1.3<br/>Async Replica]
end

subgraph "Availability Zone C"
P1[Sentinel-1]
P2[Sentinel-2]
P3[Sentinel-3]
end
end

subgraph "Secondary Region (us-west-2)"
DR1[DR Replica-1<br/>Async Cross-Region]
DR2[DR Replica-2]
end

subgraph "Backup Storage"
B1[S3 Daily RDB]
B2[S3 Hourly AOF]
B3[Glacier Weekly]
end

M1 --> S1
M1 --> S2
M1 --> S3
M1 --> DR1

P1 --> M1
P2 --> M1
P3 --> M1

P1 -.->|Quorum Detection| P2
P2 -.->|Quorum Detection| P3

S1 --> B1
DR1 --> B2
B2 --> B3

style M1 fill:#e1f5e1
style DR1 fill:#fff3e0

6.2 Failover Procedures​

# Automated Failover Configuration
failover_policy:
primary_failure:
detection:
method: "sentinel_quorum"
quorum_size: 2
timeout_ms: 10000
health_checks:
- "ping_response"
- "replication_lag"
- "memory_usage"

action:
- "pause_client_connections: 500ms"
- "elect_new_master: [slave-1.1, slave-1.2]"
- "update_dns_ttl: 30"
- "notify_ops: pagerduty_high"
- "update_configuration_store"
- "resume_client_connections"

validation:
- "data_integrity_check"
- "replication_verification"
- "performance_baseline_check"

regional_failover:
trigger: "primary_region_unavailable"
timeout: "5_minutes"
action:
- "activate_dr_replica_reads"
- "promote_dr_to_primary"
- "update_global_dns"
- "notify_stakeholders"

rollback:
condition: "primary_region_recovered_24h"
procedure: "incremental_sync_back"

6.3 Backup and Recovery​

Backup TypeFrequencyRetentionRTORPOStorage
RDB Snapshot6 hours30 days5 min6 hoursS3 Standard
AOF AppendContinuous7 days1 min1 secEBS GP3
Cross-region24 hours90 days15 min24 hoursS3 Cross-region
EncryptedWeekly1 year30 min1 weekS3 + Glacier

7. Monitoring and Observability​

7.1 Key Metrics Dashboard​

quadrantChart
title Redis Counter Service Health Dashboard
x-axis "Latency Impact" --> "Business Impact"
y-axis "Frequency" --> "Severity"

quadrant-1 "Critical: Immediate Action"
quadrant-2 "High: Urgent Attention"
quadrant-3 "Medium: Planned Response"
quadrant-4 "Low: Monitor & Document"

"Memory Usage >90%": [0.85, 0.9]
"Master-Slave Lag >10s": [0.75, 0.6]
"INCR OPS >100K/sec": [0.6, 0.7]
"CPU Usage >80%": [0.7, 0.5]
"Connection Count >10K": [0.5, 0.4]
"Network Throughput >1Gbps": [0.4, 0.3]
"Key Expiration Rate": [0.3, 0.2]
"Cache Hit Ratio <90%": [0.2, 0.1]

7.2 Alerting Rules​

alerting_rules:
critical:
- alert: "RedisMasterDown"
expr: "redis_up{role=\"master\"} == 0"
for: "1m"
annotations:
severity: "critical"
summary: "Redis master node is down"
action: "Check failover status and initiate recovery"

- alert: "HighMemoryUsage"
expr: "redis_memory_used_percentage > 90"
for: "5m"
annotations:
severity: "critical"
summary: "Redis memory usage exceeded 90%"
action: "Investigate memory leak or scale up"

warning:
- alert: "ReplicaLagHigh"
expr: "redis_slave_replication_offset - redis_master_replication_offset > 1000000"
for: "10m"
annotations:
severity: "warning"
summary: "Replication lag exceeds threshold"
action: "Check network and slave performance"

- alert: "CounterIncrementRateDrop"
expr: "rate(redis_command_increment_total[5m]) < 1000"
for: "15m"
annotations:
severity: "warning"
summary: "Counter increment rate dropped significantly"
action: "Check application health and connectivity"

8. Security Implementation​

8.1 Security Layers​

graph TD
A[Client Request] --> B[API Gateway]
B --> C[Authentication]
C --> D[Authorization]
D --> E[Rate Limiting]
E --> F[Counter Service]
F --> G[Redis Proxy]
G --> H[Redis Cluster]

subgraph "Network Security"
I[VPC Isolation]
J[Security Groups]
K[Network ACLs]
L[TLS Encryption]
end

subgraph "Data Security"
M[Encryption at Rest]
N[Encryption in Transit]
O[Key Rotation]
P[Audit Logging]
end

subgraph "Access Control"
Q[IAM Roles]
R[Redis ACL]
S[Token-based Auth]
T[IP Whitelisting]
end

H --> I
H --> J
H --> K
F --> L
H --> M
G --> N
H --> O
B --> P
F --> Q
G --> R
C --> S
B --> T

8.2 Security Configuration​

# Redis security configuration (redis.conf)
requirepass "{{ strong_password }}"
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command CONFIG ""

# Enable ACL
aclfile /etc/redis/users.acl

# Network security
bind 127.0.0.1 ::1
protected-mode yes
port 6379

# TLS configuration
tls-port 6380
tls-cert-file /etc/redis/redis.crt
tls-key-file /etc/redis/redis.key
tls-ca-cert-file /etc/redis/ca.crt

# Security limits
maxclients 10000
timeout 300
tcp-keepalive 300

# Memory security
maxmemory 16gb
maxmemory-policy allkeys-lru

9. Implementation Roadmap​

Phase 1: Foundation (Weeks 1-2)​

  1. Basic Redis Setup

    • Single master with two replicas
    • Basic monitoring (CPU, memory, connections)
    • Simple counter API implementation
  2. Client SDK Development

    • Java/Python/Node.js SDKs
    • Connection pooling and retry logic
    • Basic error handling

Phase 2: Scalability (Weeks 3-4)​

  1. Redis Cluster Deployment

    • 3-shard cluster setup
    • Sharding strategy implementation
    • Load testing and optimization
  2. Advanced Features

    • Lua script optimization
    • Pipeline implementation
    • Connection management improvements

Phase 3: Production Readiness (Weeks 5-6)​

  1. High Availability

    • Sentinel deployment
    • Automated failover testing
    • Cross-region replication setup
  2. Monitoring & Alerting

    • Comprehensive dashboard setup
    • Alerting rules configuration
    • Performance baseline establishment

Phase 4: Optimization (Weeks 7-8)​

  1. Performance Tuning

    • Memory optimization
    • Network configuration
    • OS-level tuning
  2. Security Hardening

    • TLS implementation
    • ACL configuration
    • Audit logging setup

10. Cost Estimation​

ComponentMonthly CostScaling FactorNotes
Redis Instances$1,200Linear with nodesm6g.xlarge (4vCPU, 16GB)
Storage$150$0.10/GB/monthEBS gp3 volumes
Backup Storage$75$0.023/GB/monthS3 Standard + Glacier
Data Transfer$200$0.01-0.02/GBCross-AZ and cross-region
Monitoring$300$50/node/monthCloudWatch + Prometheus
Total Monthly$1,925For 3-node cluster

11. Risks and Mitigations​

RiskProbabilityImpactMitigation Strategy
Data LossLowCriticalRegular backups + cross-region replication
Performance DegradationMediumHighAuto-scaling + caching layer
Security BreachLowCriticalTLS + ACL + regular audits
Single Point of FailureMediumHighMulti-AZ + automated failover
Cost OverrunMediumMediumUsage monitoring + budget alerts
Operational ComplexityHighMediumComprehensive documentation + automation

12. Conclusion​

This design provides a robust, scalable solution for centralized counter management using Redis. The architecture balances performance, reliability, and cost-effectiveness while maintaining strong consistency guarantees.

Next Steps:​

  1. Approval of this design document
  2. Resource allocation for implementation team
  3. Infrastructure provisioning in development environment
  4. Proof-of-concept implementation and validation
  5. Gradual rollout with canary deployment strategy

Success Criteria:​

  • ✅ 99.99% availability for counter operations
  • ✅ <5ms P95 latency for increment operations
  • ✅ Seamless scaling to 100K+ operations per second
  • ✅ Zero data loss in planned failover scenarios
  • ✅ Comprehensive monitoring and alerting coverage

Design Enhancement: Counter Batching Strategy

1. Overview of Counter Batching​

1.1 The Problem Statement​

In high-throughput distributed systems, individual counter operations (INCR, DECR) can create bottlenecks due to:

IssueImpactWithout BatchingWith Batching
Network Round-tripsLatency1 RTT per operation~1 RTT per batch
Redis Command OverheadCPU UtilizationParsing/execution per commandSingle command execution
Connection ContentionThroughput LimitsMultiple connections neededFewer connections
Transaction ManagementComplexityComplex multi-key transactionsSimplified batch operations

1.2 Batching Architecture Impact​

flowchart TD
A[Client Operations] --> B{Batch Strategy Decision}

B -->|Time-Based| C[Buffer for 100ms]
B -->|Size-Based| D[Buffer until 100 ops]
B -->|Hybrid| E[Whichever comes first]

C --> F[Batch Processor]
D --> F
E --> F

subgraph F[Batch Processing Engine]
F1[Batch Aggregation]
F2[Conflict Resolution]
F3[Pipeline Optimization]
F4[Retry Management]
end

F --> G[Batched Redis Operations]
G --> H[Single Pipeline Execution]
H --> I[Results Distribution]
I --> J[Individual Client Responses]

2. Batching Strategy Implementation​

2.1 Multi-Level Batching Architecture​

class MultiLevelCounterBatcher:
def __init__(self, redis_client, config=None):
self.redis = redis_client
self.config = config or {
'client_level': {
'max_size': 50, # Operations per batch
'max_time_ms': 100, # Max batching window
'enabled': True
},
'proxy_level': {
'max_size': 500,
'max_time_ms': 10, # Smaller window for proxies
'enabled': True
},
'redis_level': {
'pipeline_limit': 1000,
'transactional': True
}
}

# Client-side buffer
self.client_buffers = defaultdict(list) # client_id -> operations
self.client_timers = {} # client_id -> timer

# Proxy/Server-side aggregator
self.proxy_buffer = []
self.proxy_timer = None

def increment_batched(self, client_id, key, value=1, **options):
"""Client-side batch request submission"""
operation = {
'type': 'INCR',
'key': key,
'value': value,
'options': options,
'timestamp': time.time(),
'client_id': client_id,
'correlation_id': str(uuid.uuid4())
}

# Add to client buffer
self.client_buffers[client_id].append(operation)

# Start timer if first operation
if client_id not in self.client_timers:
self._start_client_timer(client_id)

# Trigger flush if buffer full
if len(self.client_buffers[client_id]) >= self.config['client_level']['max_size']:
self._flush_client_buffer(client_id)

return operation['correlation_id']

def _flush_client_buffer(self, client_id):
"""Send batched operations to proxy"""
if client_id not in self.client_buffers:
return

operations = self.client_buffers[client_id]
if not operations:
return

# Group operations by key pattern for optimization
grouped_ops = self._group_and_optimize_operations(operations)

# Send to proxy (simulated)
self._send_to_proxy(grouped_ops)

# Clear buffer and timer
self.client_buffers[client_id] = []
if client_id in self.client_timers:
self.client_timers.pop(client_id)

2.2 Smart Batching Algorithms​

class SmartBatchOptimizer:
"""Intelligent batching based on operation patterns"""

@staticmethod
def optimize_batch(operations):
"""
Optimize batch operations for Redis efficiency
Returns: List of optimized command groups
"""
# Group by operation type and key pattern
groups = defaultdict(list)

for op in operations:
# Create grouping key based on operation characteristics
if op['type'] in ['INCR', 'DECR']:
# Group counters by key pattern
key_pattern = op['key'].split(':')[0] # First segment
group_key = f"{op['type']}:{key_pattern}"
elif op['type'] == 'GET':
group_key = "GET:multi"
else:
group_key = f"other:{op['type']}"

groups[group_key].append(op)

# Apply optimization strategies per group
optimized_commands = []

# 1. Counter operations: Combine multiple INCRs on same key
counter_ops = groups.get('INCR:counter', []) + groups.get('DECR:counter', [])
if counter_ops:
optimized_commands.extend(
SmartBatchOptimizer._optimize_counter_operations(counter_ops)
)

# 2. GET operations: Use MGET for multiple keys
get_ops = groups.get('GET:multi', [])
if get_ops:
optimized_commands.append(
SmartBatchOptimizer._optimize_get_operations(get_ops)
)

# 3. Mixed operations in transaction
return optimized_commands

@staticmethod
def _optimize_counter_operations(operations):
"""Optimize counter operations using Lua scripting"""
# Group by exact key
key_operations = defaultdict(int)
for op in operations:
if op['type'] == 'INCR':
key_operations[op['key']] += op['value']
elif op['type'] == 'DECR':
key_operations[op['key']] -= op['value']

# Create Lua script for batched increments
if key_operations:
lua_script = """
local results = {}
for i, key in ipairs(KEYS) do
local increment = tonumber(ARGV[i])
local new_value = redis.call('INCRBY', key, increment)
results[i] = new_value
end
return results
"""

keys = list(key_operations.keys())
args = [key_operations[k] for k in keys]

return [{
'type': 'EVAL',
'script': lua_script,
'keys': keys,
'args': args
}]

return []

@staticmethod
def _optimize_get_operations(operations):
"""Optimize multiple GET operations using MGET"""
keys = [op['key'] for op in operations]
correlation_ids = [op['correlation_id'] for op in operations]

return [{
'type': 'MGET',
'keys': keys,
'metadata': {
'correlation_ids': correlation_ids,
'client_ids': list(set(op['client_id'] for op in operations))
}
}]

2.3 Redis Pipeline Optimization​

class RedisPipelineOptimizer:
"""Optimize Redis pipeline execution"""

def __init__(self, redis_client, max_pipeline_size=1000):
self.redis = redis_client
self.max_pipeline_size = max_pipeline_size
self.pipeline_batches = []

def add_to_pipeline(self, commands):
"""Add commands to pipeline with smart batching"""
for command in commands:
self._add_command_to_appropriate_batch(command)

# Execute if any batch is ready
self._execute_ready_batches()

def _add_command_to_appropriate_batch(self, command):
"""Intelligently assign command to optimal batch"""
# Find suitable batch or create new one
suitable_batch = None

for batch in self.pipeline_batches:
if self._can_add_to_batch(batch, command):
suitable_batch = batch
break

if not suitable_batch:
suitable_batch = self._create_new_batch(command['type'])
self.pipeline_batches.append(suitable_batch)

suitable_batch['commands'].append(command)
suitable_batch['size'] += self._estimate_command_size(command)

def _can_add_to_batch(self, batch, command):
"""Check if command can be added to existing batch"""
# Check size limits
if batch['size'] + self._estimate_command_size(command) > self.max_pipeline_size:
return False

# Check compatibility (e.g., don't mix reads and writes in same pipeline)
if batch['type'] == 'read' and command['type'] in ['INCR', 'DECR', 'SET']:
return False
if batch['type'] == 'write' and command['type'] == 'GET':
return False

# Check timing - don't delay reads for too long
if batch['type'] == 'read' and time.time() - batch['created'] > 0.05: # 50ms max for reads
return False

return True

def _execute_ready_batches(self):
"""Execute batches that are ready"""
for batch in list(self.pipeline_batches):
if self._should_execute_batch(batch):
self._execute_batch(batch)
self.pipeline_batches.remove(batch)

def _should_execute_batch(self, batch):
"""Determine if batch should be executed"""
# Size-based trigger
if batch['size'] >= self.max_pipeline_size * 0.8: # 80% full
return True

# Time-based trigger
max_wait = 0.1 if batch['type'] == 'write' else 0.05 # 100ms vs 50ms
if time.time() - batch['created'] > max_wait:
return True

# Priority-based trigger (high priority operations)
if any(cmd.get('priority') == 'high' for cmd in batch['commands']):
return True

return False

def _execute_batch(self, batch):
"""Execute a batch of commands in Redis pipeline"""
pipe = self.redis.pipeline(transaction=(batch['type'] == 'write'))

# Add all commands to pipeline
for cmd in batch['commands']:
if cmd['type'] == 'INCR':
pipe.incrby(cmd['key'], cmd['value'])
elif cmd['type'] == 'GET':
pipe.get(cmd['key'])
elif cmd['type'] == 'MGET':
pipe.mget(cmd['keys'])
elif cmd['type'] == 'EVAL':
pipe.eval(cmd['script'], len(cmd['keys']), *cmd['keys'], *cmd['args'])

# Execute and handle results
try:
results = pipe.execute()
self._distribute_results(batch['commands'], results)
except Exception as e:
self._handle_batch_error(batch['commands'], e)

3. Batching Configuration Strategies​

3.1 Adaptive Batching Configuration​

# batching-config.yaml
adaptive_batching:
enabled: true
monitor_interval_seconds: 30

strategies:
time_based:
enabled: true
initial_window_ms: 100
min_window_ms: 10
max_window_ms: 500
adjustment_factor: 0.2
metrics:
- latency_p95
- throughput_ops

size_based:
enabled: true
initial_batch_size: 50
min_batch_size: 10
max_batch_size: 1000
adjustment_factor: 0.3
metrics:
- memory_usage
- network_io

priority_based:
enabled: true
priority_levels:
realtime:
max_delay_ms: 5
max_batch_size: 10
high:
max_delay_ms: 20
max_batch_size: 50
normal:
max_delay_ms: 100
max_batch_size: 200
low:
max_delay_ms: 500
max_batch_size: 1000

optimization_rules:
- name: "combine_same_key_operations"
enabled: true
condition: "operations.same_key_count > 3"
action: "combine_to_single_operation"

- name: "separate_read_write"
enabled: true
condition: "batch.contains_mixed_operations"
action: "split_by_operation_type"

- name: "adaptive_window_adjustment"
enabled: true
condition: "latency_increased > 20%"
action: "reduce_batch_window_by 30%"

- name: "emergency_flush"
enabled: true
condition: "system_memory_usage > 85%"
action: "flush_all_batches_immediately"

monitoring:
metrics_to_track:
- batch_size_distribution
- batch_latency_percentiles
- operations_per_second
- batch_efficiency_ratio
- memory_footprint_per_batch

alerts:
- metric: "batch_efficiency_ratio"
threshold: "< 0.7"
severity: "warning"
action: "review_batching_strategy"

- metric: "batch_latency_p99"
threshold: "> 200ms"
severity: "critical"
action: "reduce_batch_size_window"

3.2 Batching Performance Profiles​

class BatchingProfiler:
"""Profile and optimize batching strategies"""

PROFILES = {
'low_latency': {
'batch_window_ms': 10,
'max_batch_size': 20,
'pipeline_size': 50,
'priority': 'realtime',
'optimizations': ['separate_read_write', 'minimal_buffering']
},
'high_throughput': {
'batch_window_ms': 200,
'max_batch_size': 500,
'pipeline_size': 1000,
'priority': 'normal',
'optimizations': ['combine_same_key_operations', 'max_batching']
},
'balanced': {
'batch_window_ms': 100,
'max_batch_size': 100,
'pipeline_size': 500,
'priority': 'high',
'optimizations': ['adaptive_window', 'smart_grouping']
},
'conservative_memory': {
'batch_window_ms': 50,
'max_batch_size': 50,
'pipeline_size': 100,
'priority': 'normal',
'optimizations': ['small_batches', 'frequent_flush']
}
}

@classmethod
def get_optimal_profile(cls, metrics):
"""
Determine optimal batching profile based on system metrics
"""
# Analyze current metrics
latency_sensitive = metrics.get('latency_p95', 0) < 50 # ms
high_throughput_needed = metrics.get('ops_per_second', 0) > 10000
memory_constrained = metrics.get('memory_usage_percent', 0) > 70

if latency_sensitive and not high_throughput_needed:
return cls.PROFILES['low_latency']
elif high_throughput_needed and not latency_sensitive:
return cls.PROFILES['high_throughput']
elif memory_constrained:
return cls.PROFILES['conservative_memory']
else:
return cls.PROFILES['balanced']

@classmethod
def create_custom_profile(cls, workload_pattern):
"""
Create custom batching profile based on workload analysis
"""
analysis = cls.analyze_workload(workload_pattern)

return {
'batch_window_ms': cls._calculate_optimal_window(analysis),
'max_batch_size': cls._calculate_optimal_batch_size(analysis),
'pipeline_size': cls._calculate_pipeline_size(analysis),
'priority': cls._determine_priority(analysis),
'optimizations': cls._select_optimizations(analysis)
}

@staticmethod
def analyze_workload(operations_log):
"""Analyze operation patterns for optimal batching"""
analysis = {
'operation_types': defaultdict(int),
'key_patterns': defaultdict(int),
'inter_arrival_times': [],
'temporal_patterns': defaultdict(int)
}

for op in operations_log:
analysis['operation_types'][op['type']] += 1
key_pattern = ':'.join(op['key'].split(':')[:2]) # First two segments
analysis['key_patterns'][key_pattern] += 1

return analysis

4. Error Handling and Consistency​

4.1 Atomic Batch Processing​

class AtomicBatchProcessor:
"""Ensure atomicity and consistency for batch operations"""

def process_batch_atomically(self, batch_operations):
"""
Process batch with atomic guarantees
Returns: (success, results, failed_operations)
"""
# 1. Validate all operations
validation_results = self._validate_batch(batch_operations)
if not validation_results['valid']:
return False, None, validation_results['invalid_ops']

# 2. Create backup for rollback
backup_data = self._create_backup(batch_operations)

# 3. Execute in Redis transaction
try:
pipe = self.redis.pipeline(transaction=True)

# Add WATCH for keys involved
for key in self._get_unique_keys(batch_operations):
pipe.watch(key)

# Add all operations
for op in batch_operations:
self._add_to_pipeline(pipe, op)

# Execute
results = pipe.execute()

# 4. Verify results
if self._verify_results(batch_operations, results):
# Success - return results
return True, results, []
else:
# Verification failed - rollback
self._rollback_changes(backup_data)
return False, None, batch_operations

except redis.WatchError:
# Concurrent modification detected
self._handle_concurrent_modification(batch_operations)
return False, None, batch_operations

except Exception as e:
# Other errors - rollback
self._rollback_changes(backup_data)
self._log_batch_failure(batch_operations, e)
return False, None, batch_operations

def _create_backup(self, operations):
"""Create backup of current values for rollback"""
backup = {}
unique_keys = self._get_unique_keys(operations)

# Use MGET for efficiency
current_values = self.redis.mget(unique_keys)

for key, value in zip(unique_keys, current_values):
backup[key] = value

return backup

def _rollback_changes(self, backup_data):
"""Rollback changes using backup data"""
pipe = self.redis.pipeline(transaction=False)

for key, original_value in backup_data.items():
if original_value is None:
# Key didn't exist before
pipe.delete(key)
else:
# Restore original value
pipe.set(key, original_value)

pipe.execute()

def _handle_partial_batch_failure(self, batch_operations, results):
"""
Handle case where some operations in batch succeed and some fail
"""
successful_ops = []
failed_ops = []

for i, (op, result) in enumerate(zip(batch_operations, results)):
if isinstance(result, Exception):
failed_ops.append({
'operation': op,
'error': result,
'index': i
})
else:
successful_ops.append({
'operation': op,
'result': result,
'index': i
})

if failed_ops:
# Attempt to compensate or rollback successful operations
if self.config.get('compensation_enabled', False):
self._compensate_successful_ops(successful_ops)
else:
self._rollback_successful_ops(successful_ops, batch_operations)

return successful_ops, failed_ops

4.2 Retry and Compensation Logic​

class BatchRetryManager:
"""Manage retries for failed batch operations"""

def __init__(self, max_retries=3, backoff_factor=2):
self.max_retries = max_retries
self.backoff_factor = backoff_factor
self.retry_queue = []
self.failed_operations = {}

def schedule_retry(self, operation, error, attempt=1):
"""Schedule operation for retry"""
if attempt > self.max_retries:
self._move_to_failed(operation, error)
return

# Calculate delay with exponential backoff and jitter
delay = self._calculate_retry_delay(attempt)

retry_item = {
'operation': operation,
'error': error,
'attempt': attempt,
'scheduled_at': time.time() + delay,
'retry_id': str(uuid.uuid4())
}

heapq.heappush(self.retry_queue, (retry_item['scheduled_at'], retry_item))

def _calculate_retry_delay(self, attempt):
"""Calculate retry delay with exponential backoff and jitter"""
base_delay = 100 # ms
exponential_delay = base_delay * (self.backoff_factor ** (attempt - 1))

# Add jitter (±20%)
jitter = random.uniform(-0.2, 0.2) * exponential_delay

return (exponential_delay + jitter) / 1000 # Convert to seconds

def process_retry_queue(self):
"""Process scheduled retries"""
current_time = time.time()

while self.retry_queue and self.retry_queue[0][0] <= current_time:
scheduled_at, retry_item = heapq.heappop(self.retry_queue)

try:
# Attempt retry
result = self._execute_retry(retry_item['operation'])

# Success - notify original caller if possible
self._notify_success(retry_item, result)

except Exception as e:
# Retry failed - schedule next attempt or give up
if retry_item['attempt'] >= self.max_retries:
self._move_to_failed(retry_item['operation'], e)
else:
self.schedule_retry(
retry_item['operation'],
e,
retry_item['attempt'] + 1
)

def _execute_retry(self, operation):
"""Execute retry with appropriate strategy"""
# Different strategies based on error type
error_type = operation.get('last_error_type')

if error_type == 'network':
# For network errors, try different Redis node
return self._retry_with_node_failover(operation)
elif error_type == 'timeout':
# For timeouts, reduce batch size or increase timeout
return self._retry_with_adjusted_parameters(operation)
elif error_type == 'contention':
# For contention, add random delay and retry
time.sleep(random.uniform(0.01, 0.1))
return self._retry_with_backoff(operation)
else:
# Default retry
return self._execute_operation(operation)

5. Monitoring and Metrics for Batching​

5.1 Batching-Specific Metrics​

class BatchingMetricsCollector:
"""Collect and analyze batching performance metrics"""

METRICS = {
'efficiency': {
'batch_utilization_ratio': 'filled_batches / total_batches',
'operations_per_batch': 'total_ops / total_batches',
'command_compression_ratio': 'individual_commands / batched_commands'
},
'performance': {
'batch_latency_p95': '95th percentile batch processing time',
'batch_throughput': 'batches_processed_per_second',
'redis_command_reduction': '(individual_calls - batched_calls) / individual_calls'
},
'quality': {
'batch_success_rate': 'successful_batches / total_batches',
'partial_failure_rate': 'batches_with_partial_failures / total_batches',
'retry_rate': 'operations_requiring_retry / total_operations'
},
'resource': {
'batch_memory_footprint': 'memory_used_by_buffers',
'batch_buffer_count': 'number_of_active_buffers',
'batch_age_distribution': 'histogram_of_batch_ages'
}
}

def collect_metrics(self):
"""Collect comprehensive batching metrics"""
metrics = {}

# Efficiency metrics
metrics['batch_utilization'] = self._calculate_batch_utilization()
metrics['compression_ratio'] = self._calculate_compression_ratio()

# Performance metrics
metrics['latency_improvement'] = self._calculate_latency_improvement()
metrics['throughput_improvement'] = self._calculate_throughput_improvement()

# Resource metrics
metrics['memory_savings'] = self._calculate_memory_savings()
metrics['network_savings'] = self._calculate_network_savings()

return metrics

def _calculate_batch_utilization(self):
"""Calculate how efficiently batches are being used"""
total_batches = self.stats['batches_created']
full_batches = self.stats['batches_flushed_by_size']
time_flushed_batches = self.stats['batches_flushed_by_time']

if total_batches == 0:
return 0

# Ideal: batches filled by size, not time
utilization = full_batches / total_batches
return utilization

def _calculate_compression_ratio(self):
"""Calculate command compression ratio"""
individual_commands = self.stats['individual_operations']
batched_commands = self.stats['batched_executions']

if batched_commands == 0:
return 0

return (individual_commands - batched_commands) / individual_commands

5.2 Dashboard Visualization​

graph TD
subgraph "Batching Performance Dashboard"
A[Batch Efficiency] --> A1["Utilization: 85%"]
A --> A2["Compression Ratio: 65%"]
A --> A3["Avg Batch Size: 78 ops"]

B[Performance Impact] --> B1["Latency Reduction: 40%"]
B --> B2["Throughput Increase: 3.2x"]
B --> B3["Redis Load Reduction: 55%"]

C[Resource Usage] --> C1["Memory Buffer: 45MB"]
C --> C2["Active Batches: 12"]
C --> C3["Avg Batch Age: 45ms"]

D[Quality Metrics] --> D1["Success Rate: 99.8%"]
D --> D2["Partial Failure: 0.2%"]
D --> D3["Retry Rate: 0.5%"]
end

subgraph "Batch Size Distribution"
E["<10 ops: 5%"] --> F
E1["10-50 ops: 25%"] --> F
E2["50-100 ops: 40%"] --> F[Histogram]
E3["100-200 ops: 20%"] --> F
E4[">200 ops: 10%"] --> F
end

subgraph "Real-time Monitoring"
G[Current Batch Buffer] --> G1["Client Buffers: 8"]
G --> G2["Proxy Buffer: 1"]
G --> G3["Pending Operations: 342"]

H[Next Flush Prediction] --> H1["Time-based: 32ms"]
H --> H2["Size-based: 18 ops needed"]
end

6. Implementation Roadmap for Batching​

Phase 1: Basic Batching (Week 1-2)​

  1. Client-side buffering

    • Time-based batching (100ms windows)
    • Size-based batching (50 operations)
    • Simple flush mechanisms
  2. Basic Redis pipelining

    • Pipeline execution of batched commands
    • Error handling for batch failures
    • Basic metrics collection

Phase 2: Advanced Batching (Week 3-4)​

  1. Smart batching strategies

    • Operation type-based grouping
    • Key pattern optimization
    • Adaptive batch sizing
  2. Consistency guarantees

    • Atomic batch processing
    • Partial failure handling
    • Retry mechanisms

Phase 3: Production Optimization (Week 5-6)​

  1. Performance optimization

    • Adaptive window sizing
    • Memory usage optimization
    • Network efficiency improvements
  2. Monitoring and alerting

    • Comprehensive metrics dashboard
    • Automated optimization suggestions
    • Alerting for batch inefficiencies

Phase 4: Advanced Features (Week 7-8)​

  1. Predictive batching

    • Machine learning for optimal batch sizing
    • Workload pattern recognition
    • Proactive optimization
  2. Multi-level batching hierarchy

    • Client → Proxy → Redis optimization
    • Cross-service batch coordination
    • Global batch optimization

7. Performance Expectations with Batching​

MetricWithout BatchingWith BatchingImprovement
Operations per second10,000 ops45,000 ops4.5x
Average latency15ms8ms47% reduction
Redis connections5005090% reduction
Network packets10,000/min2,000/min80% reduction
Redis CPU usage65%40%38% reduction
Memory overheadHigh (per connection)Low (buffered)60-80% savings

8. Best Practices for Batching Implementation​

  1. Start Conservative: Begin with small batch sizes and increase gradually
  2. Monitor Aggressively: Track both efficiency and side effects
  3. Implement Graceful Degradation: Fall back to non-batched mode if issues arise
  4. Test with Real Workloads: Simulate production traffic patterns
  5. Implement Circuit Breakers: Prevent cascading failures
  6. Document Batching Behavior: Ensure all teams understand the trade-offs
  7. Plan for Rollback: Have a clear procedure to disable batching if needed

9. Conclusion​

Counter batching is a critical optimization strategy for high-throughput Redis-based counter systems. When implemented correctly, it can provide:

  • 4-5x throughput improvement
  • 40-60% latency reduction
  • 80-90% connection count reduction
  • Significant Redis resource savings

The key to successful batching implementation is starting with simple strategies, monitoring comprehensively, and gradually introducing more sophisticated optimizations based on actual workload patterns and performance metrics.

The multi-level batching architecture (client → proxy → Redis) combined with smart optimization algorithms provides the flexibility to handle diverse workload patterns while maintaining strong consistency guarantees and excellent performance characteristics.

Appendices​

Appendix A: Redis Configuration Template​

[Full redis.conf template with optimizations for counter workloads]

Appendix B: Load Testing Scripts​

[Python scripts for simulating high-volume counter operations]

Appendix C: Client SDK Examples​

[Complete code examples for Java, Python, and Node.js SDKs]

Appendix D: Disaster Recovery Runbook​

[Step-by-step procedures for various failure scenarios]

Appendix E: Performance Tuning Guide​

[Detailed instructions for OS, network, and Redis tuning]