Data building blocks
Batch and stream processing
How to move and transform large data sets reliably. Batch jobs run over bounded data, streams run over events as they arrive, and the same ideas of time, state and idempotent writes apply to both.
Your OLTP database answers "what is the state of order 981 right now?" Other questions need every row or every event that ever happened. Examples are revenue per category per day, the click rate of each ad in the last minute, or the training set for a model. Data processing systems answer these questions. They read large inputs, transform them, and write derived outputs.
There are two families. Batch runs over a bounded data set, such as yesterday's logs, and then finishes. Streaming runs over an unbounded flow of events and updates results continuously. You need both: batch for correctness and backfills, streaming for freshness.
Concepts in this section
Read the pages in this order.
| Step | Concept | What it answers |
|---|---|---|
| 1 | MapReduce | How does one job split across many machines? |
| 2 | Spark and shuffles | Why do some steps cost far more than others? |
| 3 | Event time and watermarks | How do you aggregate events that arrive late? |
| 4 | Exactly-once in streams | How do you avoid double counting after a crash? |
| 5 | Lambda vs Kappa | Do you need two pipelines or one? |
| 6 | Ad-click aggregation | How do the pieces fit in one design? |
When batch is enough
Batch is simpler to build, test, debug and backfill. If results are needed hourly or daily, such as finance reports, billing, training sets or most dashboards, a scheduled batch job is usually the right tool. Orchestrators like Airflow or Dagster run these jobs.
Use streaming when the business truly needs seconds or minutes of freshness. Examples are fraud checks, ad pacing, alerting and live leaderboards.
Warehouses, lakes and lakehouses
Processed data lands in a store that analysts and jobs can query.
- Warehouse. A managed columnar SQL engine with its own storage, such as Snowflake, BigQuery or Redshift.
- Data lake. Files in object storage, usually Parquet. Parquet is columnar and compressed, so queries read only the columns they need. Plain files lack transactions and safe concurrent writes.
- Lakehouse tables. Apache Iceberg, Delta Lake and Apache Hudi add a metadata layer over Parquet. They provide atomic commits, time travel and row-level updates, and several engines can read the same tables.
- ELT with dbt. Load raw data first, then transform it with versioned SQL models.
A common modern stack is: OLTP databases send changes into Kafka, Flink serves real-time products and writes to Iceberg tables in S3, and dbt models feed dashboards and ML features.
Common mistakes
- Choosing streaming because it sounds better. A daily batch job that meets the requirement costs far less to run.
- Pointing heavy analytics at the OLTP primary. Long scans can evict hot data and slow checkout.
- Using processing time for business metrics. Replays and lag change the numbers.
- Streaming into a lake every few seconds. That creates many tiny files. Schedule compaction.
Key takeaways
- Batch processes bounded data and is simpler to backfill. Streaming processes unbounded data for freshness.
- MapReduce and Spark move data through shuffles, which usually decide the cost.
- Aggregate by event time, and use watermarks to decide when a window is final.
- Exactly-once results need idempotent or transactional sinks.
- Columnar files and lakehouse tables, with dbt, are the standard analytics stack.
Further learning
Tyler Akidau explains streams and tables, windows and watermarks in one of the clearest talks on the topic.
Designing Data-Intensive Applications (Kleppmann)Designing Data-Intensive Applications (DDIA) — an O’Reilly book by Martin Kleppmann (The Wild Boar Book)dataintensive.netRelated
Event streaming and Kafka
How a durable log lets many consumers read the same events at their own pace, and how Kafka builds that log from partitions, replicas and consumer groups.
Event-driven architecture, CQRS and event sourcing
Building systems around events, from change data capture to event sourcing and read models, and knowing when it simplifies a system versus creating a distributed mess.
Product analytics
Designing event tracking plans, reading funnels and cohorts, writing the SQL that answers product questions, and avoiding mix-shift illusions. Start here for the map.