Consumer groups, offsets and lag

Consumers that share a group ID split a topic's partitions among themselves. Each partition goes to exactly one consumer in the group. That keeps order within each partition while the group processes in parallel. Different groups are fully independent, and each reads every record at its own pace.

Billing is nearly caught up. Analytics is far behind, perhaps because it was just deployed and is reading history. Neither affects the other or the producer.

The group's maximum parallelism equals the partition count. With 3 partitions, a fourth consumer in the billing group sits idle. So partition count is a capacity decision you make early.

Offsets and commits

A consumer records its progress by committing offsets to an internal topic, __consumer_offsets. If a partition moves to another consumer, the new owner resumes from the last committed offset. When you commit decides your delivery guarantee:

• Commit before processing: at-most-once. A crash after the commit loses the message.
• Commit after processing: at-least-once. A crash before the commit repeats the message.

Kafka's default is enable.auto.commit=true, which commits about every 5 seconds in the background, whether or not processing finished. For data that matters, turn it off and commit after processing:

from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "billing",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
    "isolation.level": "read_committed",
})
consumer.subscribe(["orders"])

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    handle_order_event(msg.key(), msg.value())  # your handler; must be idempotent
    consumer.commit(message=msg, asynchronous=False)

The handler must be idempotent, because a crash after processing and before the commit repeats the message. See deduplicating in message consumers.

Lag: the health metric

Consumer lag is the gap between a partition's latest offset and the group's committed offset. Steady low lag means consumers keep up. Growing lag means they don't. Alert on lag measured in time, such as "billing is 5 minutes behind", not only in record counts. Ten thousand records behind is fine for one topic and a crisis for another.

Rebalancing

When a consumer joins, leaves or crashes, the group rebalances and partitions move. The original protocol was stop-the-world. Every consumer gave up all its partitions, so one pod restart paused the whole group for seconds. Cooperative incremental rebalancing moves only the partitions that must move. The newer consumer group protocol (KIP-848, generally available in Kafka 4.0) moves assignment logic to the broker and makes rebalances much less disruptive. Static membership, set with group.instance.id, lets a consumer restart within a timeout without triggering a rebalance at all. That helps rolling deploys a lot.

Common mistakes

• Leaving auto-commit on for important work. Offsets advance whether or not processing finished.
• Running more consumers than partitions. The extra consumers sit idle.
• Alerting on record counts only. Lag in time tells you how stale the data is.
• Non-idempotent handlers. Replays are normal after a restart or a rebalance.

Check yourself

1. A billing group has 3 partitions and you add a fourth consumer. What happens? The new consumer sits idle. Each partition has one owner, and only three can own one.

2. You commit offsets before processing each record. Which guarantee do you get? At-most-once. A crash after the commit but before processing loses that record.

Key takeaways

• A consumer group shares partitions, and each partition has one owner.
• Parallelism is capped at the partition count.
• Commit after processing for at-least-once delivery, and make the handler idempotent.
• Track lag in time, and use static membership to avoid rebalances during deploys.

Further learning

Next, see what Kafka's exactly-once guarantee does and doesn't cover in exactly-once inside Kafka.