Hybrid Circuit Breaker - In-Memory with Redis Sync in NestJS
We’ve built an in-memory circuit breaker, then added Redis. But here’s the thing: every circuit check hits Redis. That’s a network call. What if we could use in-memory for speed and Redis for sync?
Let’s build a hybrid.
The problem with pure Redis
Every time you check a circuit state, you’re hitting Redis:
const state = await this.redis.get(`circuit:${key}:state`)
That’s a network call. If you’re checking circuits on every request, that adds up. Plus, if Redis has a hiccup, your requests slow down.
The hybrid approach
Use in-memory for reads, Redis for writes. Sync state across instances in the background.
Here’s the idea:
- Keep circuit state in memory (fast reads)
- Write state changes to Redis (sync across instances)
- Subscribe to Redis pub/sub (get updates from other instances)
- Periodically sync from Redis (catch up if we missed something)
Building it
Start with the hybrid service:
import { Injectable, Inject, OnModuleInit } from '@nestjs/common'
import Redis from 'ioredis'
import { REDIS_CLIENT } from './redis.module'
enum CircuitState {
CLOSED = 'CLOSED',
OPEN = 'OPEN',
HALF_OPEN = 'HALF_OPEN'
}
interface CircuitConfig {
failureThreshold: number
successThreshold: number
timeout: number
}
interface CircuitStats {
state: CircuitState
failures: number
successes: number
nextAttempt: number
}
@Injectable()
export class HybridCircuitBreakerService implements OnModuleInit {
private circuits = new Map<string, CircuitStats>()
private subscriber: Redis
private redisAvailable = false
private readonly defaultConfig: CircuitConfig = {
failureThreshold: 5,
successThreshold: 2,
timeout: 60000
}
constructor(
@Inject(REDIS_CLIENT) private readonly redis: Redis
) {
this.subscriber = this.redis.duplicate()
}
async onModuleInit() {
try {
await this.subscriber.subscribe('circuit:updates')
this.redisAvailable = true
this.subscriber.on('message', (channel, message) => {
if (channel === 'circuit:updates') {
const update = JSON.parse(message)
this.circuits.set(update.key, update.stats)
}
})
this.redis.on('error', () => {
this.redisAvailable = false
})
this.redis.on('connect', () => {
this.redisAvailable = true
})
} catch (error) {
this.redisAvailable = false
}
}
async execute<T>(
key: string,
fn: () => Promise<T>,
config?: Partial<CircuitConfig>
): Promise<T> {
const cfg = { ...this.defaultConfig, ...config }
const circuit = this.getCircuit(key)
if (circuit.state === CircuitState.OPEN) {
if (Date.now() < circuit.nextAttempt) {
throw new Error(`Circuit breaker is OPEN for ${key}`)
}
circuit.state = CircuitState.HALF_OPEN
circuit.successes = 0
this.publishUpdate(key, circuit)
}
try {
const result = await fn()
this.onSuccess(key, cfg)
return result
} catch (error) {
this.onFailure(key, cfg)
throw error
}
}
private getCircuit(key: string): CircuitStats {
if (!this.circuits.has(key)) {
this.circuits.set(key, {
state: CircuitState.CLOSED,
failures: 0,
successes: 0,
nextAttempt: 0
})
}
return this.circuits.get(key)!
}
private onSuccess(key: string, config: CircuitConfig) {
const circuit = this.getCircuit(key)
if (circuit.state === CircuitState.HALF_OPEN) {
circuit.successes++
if (circuit.successes >= config.successThreshold) {
circuit.state = CircuitState.CLOSED
circuit.failures = 0
circuit.successes = 0
}
} else {
circuit.failures = 0
}
this.writeToRedis(key, circuit)
this.publishUpdate(key, circuit)
}
private onFailure(key: string, config: CircuitConfig) {
const circuit = this.getCircuit(key)
circuit.failures++
if (circuit.failures >= config.failureThreshold) {
circuit.state = CircuitState.OPEN
circuit.nextAttempt = Date.now() + config.timeout
circuit.successes = 0
}
this.writeToRedis(key, circuit)
this.publishUpdate(key, circuit)
}
private async writeToRedis(key: string, stats: CircuitStats) {
if (!this.redisAvailable) return
try {
await this.redis.hset(`circuit:${key}`, {
state: stats.state,
failures: stats.failures,
successes: stats.successes,
nextAttempt: stats.nextAttempt
})
await this.redis.expire(`circuit:${key}`, 3600)
} catch (error) {
// Redis write failed, continue with in-memory
}
}
private async publishUpdate(key: string, stats: CircuitStats) {
if (!this.redisAvailable) return
try {
await this.redis.publish(
'circuit:updates',
JSON.stringify({ key, stats })
)
} catch (error) {
// Publish failed, other instances won't get update immediately
}
}
getState(key: string): CircuitState {
return this.getCircuit(key).state
}
async reset(key: string) {
this.circuits.delete(key)
if (this.redisAvailable) {
try {
await this.redis.del(`circuit:${key}`)
} catch (error) {
// Redis delete failed
}
}
}
getStats(key: string) {
return this.getCircuit(key)
}
}
Parallel operations
The key here is that Redis operations don’t block:
private async onSuccess(key: string, config: CircuitConfig) {
const circuit = this.getCircuit(key)
// Update in-memory state immediately
if (circuit.state === CircuitState.HALF_OPEN) {
circuit.successes++
if (circuit.successes >= config.successThreshold) {
circuit.state = CircuitState.CLOSED
circuit.failures = 0
circuit.successes = 0
}
} else {
circuit.failures = 0
}
// Fire and forget - don't await
// These run in parallel and don't block the response
this.writeToRedis(key, circuit)
this.publishUpdate(key, circuit)
}
No await means the function returns immediately. Redis writes happen in the background.
Performance comparison
The difference is noticeable:
// Pure Redis approach
async executeWithRedis() {
const start = Date.now()
const state = await this.redis.get('circuit:test:state') // Network call
const failures = await this.redis.get('circuit:test:failures') // Network call
console.log(`Redis check took: ${Date.now() - start}ms`)
}
// Hybrid approach
async executeWithHybrid() {
const start = Date.now()
const circuit = this.circuits.get('test') // In-memory, instant
console.log(`Hybrid check took: ${Date.now() - start}ms`)
}
In-memory lookups are basically instant. Redis calls need a network round trip. The exact difference depends on your network latency, but in-memory is always faster for reads.
Testing the hybrid approach
Test that state syncs across instances:
describe('HybridCircuitBreakerService', () => {
let service1: HybridCircuitBreakerService
let service2: HybridCircuitBreakerService
let redis: Redis
beforeEach(async () => {
redis = new Redis()
service1 = new HybridCircuitBreakerService(redis)
service2 = new HybridCircuitBreakerService(redis)
await service1.onModuleInit()
await service2.onModuleInit()
})
it('should sync state across instances', async () => {
const failingFn = async () => {
throw new Error('API down')
}
for (let i = 0; i < 5; i++) {
try {
await service1.execute('test', failingFn)
} catch (e) {}
}
// Wait for pub/sub
await new Promise(resolve => setTimeout(resolve, 100))
expect(service2.getState('test')).toBe(CircuitState.OPEN)
})
it('should work without Redis', async () => {
await redis.disconnect()
const result = await service1.execute('test', async () => 'success')
expect(result).toBe('success')
})
})
What I learned
In-memory is fast: Reads happen instantly without network calls.
Fire and forget: Don’t await Redis writes. Let them happen in the background.
Pub/sub for sync: Other instances get updates through Redis pub/sub.
Graceful degradation: Works without Redis, just loses cross-instance sync.
Keep it simple: Removed periodic sync and complex reconnection logic. Pub/sub handles most cases.
Use Redis hashes: Atomic updates for all circuit fields at once.
Set TTLs: Old circuits expire automatically after 1 hour.
Wrapping up
The hybrid approach gives you the best of both worlds - in-memory speed with Redis sync. Fast reads, eventual consistency across instances, and graceful degradation when Redis is down.
This is what I use in production. It’s fast, reliable, and scales.
Keep pushing forward and savor every step of your coding journey.