This chapter synthesizes core architectural patterns for evolving data systems, emphasizing immutable trade-offs. It begins with immutable data structures (LSM-Trees) and eventual consistency as foundational principles. The LSM-Tree SSTable merge code illustrates the sequential I/O and write amplification trade-off of compaction. Distributed consistency models are explored through Two-Phase Commit (2PC), highlighting its blocking risk and atomicity guarantee, and the Saga pattern, which uses compensating transactions for eventual consistency. A comprehensive comparison table contrasts eight architectural patterns (B-Tree, LSM-Tree, Columnar, Single/Multi/Leaderless Replication, Lambda, Kappa) across dimensions like primary data structure, update pattern, amplification, consistency model, scalability, and fault tolerance. The chapter concludes with an evolution diagram describing a system centered on an immutable log as the source of truth, feeding various derived data systems (search index, cache, analytics DB, graph DB) via ingestion paths like transactional writes and CDC. Schema evolution handlers and failure recovery mechanisms (checkpointing, replay, idempotent writers) ensure the system can evolve and recover deterministically. The narrative consistently frames choices around the immutable trade-offs of read vs. write amplification and consistency vs. availability.
Read Chapter
- 5 min read
-
Lambda and Kappa Architectures 5 min read -
Verifiability and Future Directions 8 min read