Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing: Summary & Key Insights

For decades, batch processing reigned supreme. Early systems like MapReduce were designed to chew through vast, bounded datasets. The implicit assumption was that the data had an end—that it could be collected, stored, and processed as a whole. While this model proved remarkably powerful, the world began to outgrow it.

Modern realities—user interactions on the web, IoT sensors, financial transactions, mobile apps—produce data continuously, unboundedly. Batch systems, in that sense, felt like snapshots of a living reality. They couldn’t keep up with a world in motion. We began to need insight not tomorrow but now.

This realization led to an ideological shift. Instead of viewing data as finite collections, we began to treat it as unending streams of events. Systems like MillWheel emerged to handle data in real time while maintaining correctness despite latency, network faults, and reordering. Yet this transition wasn’t simply about speed. It demanded rethinking computation itself—how to define completeness in a world that never stops producing data.

Streaming processing introduced a new mindset: to compute continuously and incrementally rather than periodically and exhaustively. It was a shift from taking static pictures to filming living processes.

Streaming data is unbounded. It has no natural conclusion, no ‘final record.’ Every second new events arrive, perhaps late, perhaps out of order, perhaps duplicated. That incessant flow introduces beautiful complexity. Traditional algorithms that assume a finite input suddenly break down when faced with infinity.

In writing this book, I wanted to help readers internalize this difference—not as an abstract concept, but as a practical foundation. In a batch system, you compute once you know all your data. In a streaming system, you never have all your data. The art lies in deciding when enough data is sufficient to emit meaningful results.

This results in a fundamental change in how we model processes. Unboundedness forces us to think in terms of continuous computation: we define what it means to process data as it arrives, and how to integrate updates over time. The elegance of streaming comes from embracing this open-endedness rather than fighting it.