Data building blocks
Lambda vs Kappa
Two ways to keep analytics fast and correct. Lambda runs a batch layer and a speed layer with duplicated logic. Kappa uses one streaming pipeline and replays the log when it needs to reprocess.
Early stream processors were fast, but teams did not trust them for correct numbers. The Lambda architecture answered that with two pipelines. The Kappa architecture asked whether one correct streaming pipeline would be enough. Most teams today sit between the two.
Concept
Before you start. A batch layer recomputes results from the full history. A speed layer streams approximate, recent results. A replay reads a stored log again from an earlier point. The log is the ordered, retained record of events, such as a Kafka topic.
The big idea
Think of a newspaper. The website shows breaking news quickly. The next day's print edition is checked and corrected. Lambda keeps both, and merges them. Kappa only has the website, and when a story needs correcting, it republishes from the original source.
The Kappa version works only when the source keeps enough history, and when the stream processor is correct over time.
Walking through it
| Lambda | Kappa | |
|---|---|---|
| Code paths | Two, batch and streaming | One, streaming |
| Reprocessing | Rerun the batch layer | Replay the log into a new job |
| Needs | Batch storage of all history | Long log retention or tiered storage |
| Main weakness | Duplicate logic and reconciliation | Replaying years of events can be slow |
Here is how a Kappa fix works. You change the rule for a click metric.
- Start version 2 of the job, reading the log from the beginning.
- Let it catch up to the live stream. Its results are not yet read by anyone.
- Switch the dashboards to version 2.
- Stop version 1.
Lambda would fix the same bug in two codebases, then reconcile any differences between them.
In practice, many teams run a hybrid. One streaming job gives fresh results. Batch backfills read from the data lake, not from Kafka, when old history must be recomputed. Shared SQL or Beam code keeps the logic in one place.
Common mistakes
- Keeping two copies of the same logic. They drift apart, and the numbers stop matching.
- Going Kappa with short log retention. You cannot replay what the log no longer holds.
- Replaying years of events from Kafka for a backfill. A data lake is usually faster and cheaper for that.
Check yourself
- A metric rule has a bug. Under Kappa, what is the fix? Start a new job that replays the log from the beginning, let it catch up, switch readers, then stop the old job.
- Why does Lambda need reconciliation? The batch and speed layers compute the same metric in two systems, so their results can disagree.
Key takeaways
- Lambda pairs a batch layer with a speed layer, at the cost of duplicate logic.
- Kappa uses one streaming pipeline and reprocesses by replaying the log.
- Kappa needs a correct stream processor and enough retained history.
- The common modern pattern is streaming for freshness, plus lake backfills for old history.
Further learning
Jay Kreps' essay is the original case for moving away from Lambda.
Questioning the Lambda Architecture (Jay Kreps)The Lambda Architecture has its merits, but alternatives are worth exploring.O’Reilly Media
Next, see both ideas applied to one design in ad-click aggregation.
Related
Batch and stream processing
How to move and transform large data sets reliably. Batch jobs run over bounded data, streams run over events as they arrive, and the same ideas of time, state and idempotent writes apply to both.
Exactly-once in streams
A stream processor restores its state and rewinds its input after a crash. The state counts each event once, but the output can repeat. End-to-end exactly-once needs an idempotent or transactional sink.
Event streaming and Kafka
How a durable log lets many consumers read the same events at their own pace, and how Kafka builds that log from partitions, replicas and consumer groups.