Database Internals: Deep Insights into How Distributed Data Systems Work: Summary & Key Insights

Every database begins with the simple promise of persistence: data once written should never be lost. To fulfill this promise, storage engines handle the mechanics of reading, writing, and organizing data on physical devices. They’re the beating heart of any database system, determining the trade-offs between durability, performance, and efficiency.

In this part of the book, I explore how storage engines are built and how they differ across systems like MySQL’s InnoDB, PostgreSQL’s heap manager, and Cassandra’s SSTable-based structure. Each engine has a different philosophy about when to write to disk, how to handle concurrency, and what to prioritize—speed, safety, or simplicity. I guide readers through the anatomy of these systems: how buffers, caches, and background threads interact to turn raw data into durable state.

The essence lies in knowing that not all storage engines fit all workloads. A system handling frequent writes must minimize disk seeks; one serving analytical queries must prioritize efficient sequential reads. Understanding the structure and capabilities of storage engines gives you the insight to pick what best serves your application’s behavior—not just its scale.

Underneath every storage engine lies a data structure that defines how efficiently it can store and retrieve records. This section dissects those structures in depth—starting with the venerable B-tree, cornerstone of relational systems, and the Log-Structured Merge (LSM) tree, backbone of modern NoSQL engines.

B-trees excel in balanced workloads, offering predictable latency for reads and writes. Their pages remain sorted and indexed, enabling efficient range scans and updates. Yet under write-heavy conditions, this balance becomes costly: random I/O and frequent page splits can stall throughput.

LSM trees, inspired by log-structured storage systems, invert the process. Instead of writing directly to sorted pages, they batch incoming data into immutable segments, merging them asynchronously. This architecture privileges sequential writes—a huge advantage for modern SSDs—but introduces complexities for reads and compaction management.

By comparing these structures across real implementations—RocksDB, LevelDB, and InnoDB—I emphasize that each data structure embodies a philosophy. Performance doesn’t emerge from magic; it’s a reflection of how the system trades write amplification against read complexity. And when you, as an engineer, grasp this, you begin to think in terms of trade-offs rather than assumptions.