Event time, windows and watermarks

A stream processor counts events forever, so it must decide when a count is final. Phones go offline, networks retry and consumers lag. If you count by when events arrive, the numbers change with every delay. Counting by when events happened gives stable answers. This page explains how to do that, and how to deal with the events that arrive late.

The big idea

Think of taking attendance at a school. You wait a little for late students before you submit the count. The waiting period is the watermark. The count for a class is final when the wait is over.

The analogy breaks because you cannot wait forever. Each choice trades freshness against completeness. A short wait gives fresh results and drops more stragglers. A long wait catches more events, but every result is older.

Walking through it

Count clicks in a 1-minute window, [12:00, 12:01). The watermark is the largest event time seen, minus 30 seconds. Each arrival is shown as event time, then processing time.

12:00:59 event, arrives 12:01:00   max seen 12:00:59   watermark 12:00:29
12:01:10 event, arrives 12:01:12   max seen 12:01:10   watermark 12:00:40
12:01:30 event, arrives 12:01:31   max seen 12:01:30   watermark 12:01:00
                                   watermark passed 12:01, window fires
12:00:40 event, arrives 12:03:00   late, inside allowed lateness
                                   window updated and re-emitted

The window fires when the watermark reaches 12:01:00. That happens once an event with event time 12:01:30 arrives, because 12:01:30 minus 30 seconds is 12:01:00.

The late event at 12:00:40 arrives after the window fired. Allowed lateness keeps the window's state for a while, so the late event can update the result. After the allowed lateness ends, the state is dropped. Events later than that go to a side output for separate handling.

There are three common window shapes:

• Tumbling: fixed, non-overlapping. Clicks per minute.
• Sliding: fixed length, overlapping. The last 5 minutes, updated every minute, so each event counts in 5 windows.
• Session: closes after a gap of inactivity, such as 30 minutes with no events. Its length depends on the data.

Common mistakes

• Aggregating by processing time. Replaying yesterday's data gives different numbers.
• Picking a watermark delay without checking lateness. Measure how late events really arrive first.
• Forgetting that updated results need handling downstream. A re-emitted window must overwrite the old value, not add to it.

Check yourself

1. With a 30-second watermark delay, what is the watermark after an event with event time 12:01:30? 12:01:00. The window [12:00, 12:01) fires.
2. Why does an event that arrives after the watermark not always get dropped? Allowed lateness keeps the window's state for a while, so the late event can update the result.

Key takeaways

• Aggregate by event time, not processing time, so results can be reproduced.
• The watermark is the stream's claim that no earlier events will arrive.
• A short watermark delay means fresher, less complete results. A long one means the reverse.
• Allowed lateness and side outputs decide what happens to late events.
• Tumbling, sliding and session windows answer different questions.

Further learning

Next, see how stateful stream processors avoid double counting after a crash in exactly-once in streams.

Tyler Akidau, a designer of Google's Dataflow model, explains streams, tables, windows and watermarks clearly.

Streaming 101 (Tyler Akidau)A high-level tour of modern data-processing concepts.O’Reilly Media