What it is: Separate the read model (Query) from the write model (Command) into distinct services, databases, or code paths.
When to use:
Implementation:
Write Path: Client → Command API → Write DB (normalized, PostgreSQL)
Read Path: Client → Query API → Read DB (denormalized, Redis / Elasticsearch)
Sync: Write DB → CDC (Debezium) → Message Queue → Read DB updater
Trade-offs:
Real-world users: Amazon (order service), LinkedIn (feed)
What it is: Store state as a sequence of immutable events rather than current state. Rebuild current state by replaying events.
When to use:
Implementation:
Event Store: append-only log (Kafka, EventStoreDB)
Snapshots: periodic snapshots to speed up state rebuild
Projections: consumers build read models from events
Trade-offs:
What it is: Manage distributed transactions across microservices via a sequence of local transactions, each publishing an event. If a step fails, compensating transactions undo previous steps.
Two variants:
When to use:
Choreography Example:
OrderService creates order →
[event: OrderCreated] →
PaymentService charges card →
[event: PaymentProcessed] →
InventoryService reserves stock →
[event: StockReserved] →
ShippingService books courier
Compensating Transactions (on failure):
ShippingService fails →
[event: ShippingFailed] →
InventoryService releases stock →
PaymentService refunds card →
OrderService marks order failed
Trade-offs:
What it is: A proxy that monitors calls to a service. If failure rate exceeds threshold, the circuit "opens" and calls fail fast instead of waiting for timeout.
States:
CLOSED → calls pass through; monitor failure rate
OPEN → calls fail immediately; no calls to downstream
HALF-OPEN → let a probe call through; if success, close; if fail, stay open
When to use:
Implementation tools: Hystrix (deprecated), Resilience4j, Polly (.NET), Envoy proxy
Thresholds (starting point):
Trade-offs:
What it is: Isolate components so a failure in one doesn't consume resources of others. Named after the watertight compartments in ship hulls.
Types:
When to use:
Example:
Without bulkhead:
[Recommendation Service hangs] → fills shared thread pool → [Payment Service starves]
With bulkhead:
[Recommendation Service hangs] → fills its own thread pool (10 threads) → [Payment Service unaffected, has its own 50 threads]
What it is: Incrementally replace a legacy monolith by routing new functionality to new microservices, while keeping the monolith alive for unchanged features.
Migration steps:
Phase 1: Deploy proxy in front of monolith (no user impact)
Phase 2: Route one feature to new microservice
Phase 3: Verify; deprecate that feature in monolith
Phase 4: Repeat for each feature
Phase 5: Monolith is empty; decommission
When to use:
Trade-offs:
What it is: Solve the dual-write problem (write to DB AND publish to queue atomically) by writing the event to an "outbox" table in the same DB transaction, then having a separate process relay it to the queue.
Problem it solves:
❌ WRONG (dual-write race):
BEGIN;
UPDATE orders SET status='paid';
COMMIT;
// Crash here → event never published, DB and queue are inconsistent
publish(PaymentProcessed);
✅ CORRECT (outbox):
BEGIN;
UPDATE orders SET status='paid';
INSERT INTO outbox (event_type, payload) VALUES ('PaymentProcessed', {...});
COMMIT;
// Relay process reads outbox and publishes to Kafka
// At-least-once delivery guaranteed; make consumers idempotent
Relay options: Debezium (CDC), polling relay, transaction log tailing
What it is: A hashing scheme where adding or removing nodes requires only K/N keys to be remapped (K = keys, N = nodes), instead of remapping all keys.
When to use:
Virtual nodes: Assign multiple positions per physical node on the hash ring to ensure even distribution even with few nodes.
What it is: A mechanism for consumers to signal producers to slow down when they can't keep up, preventing memory exhaustion and cascade failures.
Strategies:
When to use:
What it is: In a distributed system, elect a single node to perform a privileged task (e.g., writing to DB, sending scheduled jobs, coordinating work).
Algorithms:
When to use:
Tools: etcd, ZooKeeper, Consul, Redis (Redlock — use with caution)
What it is: A distributed algorithm that ensures all participants in a transaction either all commit or all abort.
Phases:
Phase 1 (Prepare): Coordinator asks all participants "can you commit?"
All say YES → proceed to Phase 2
Any says NO → abort
Phase 2 (Commit): Coordinator tells all participants to commit
When to use (sparingly):
Why to avoid:
Read-Through:
Client → Cache (miss) → Cache fetches from DB → Returns to client
Cache is always populated on miss. Simple for clients. Risk: cold start.
Write-Through:
Client → Cache → Cache writes to DB synchronously → Confirms
Strong consistency. Higher write latency. Good for read-heavy with consistency need.
Write-Behind (Write-Back):
Client → Cache → Confirms immediately → Async flush to DB
Very low write latency. Risk of data loss if cache fails before flush. Good for high-throughput counters, analytics.
Cache-Aside (Lazy Loading):
Client → Cache (miss) → Client fetches from DB → Client writes to Cache
Most common. Application owns cache logic. Risk: thundering herd on cold start.