Kafka Consumers Explained: Pull, Offsets, and Parallelism

Kafka is built for high throughput, scalability, and fault tolerance. At the core of this is its consumer model. Unlike traditional messaging systems, Kafka gives consumers full control over how they read data. This post explains how Kafka consumers work by focusing on three things: how they pull data, how offsets work, and how parallelism is achieved with consumer groups.

Pulling Data from Kafka

Kafka producers push data to brokers. Consumers pull data from brokers. This setup is intentional. It gives consumers control over how fast they process data.

In push-based systems, if the producer is faster than the consumer, the consumer can get overwhelmed or crash. Kafka avoids this problem by letting consumers decide when to fetch data. This helps with backpressure and makes the system more reliable.

Pulling also helps with batching. A consumer can fetch many messages in a single request. This reduces the number of network calls. In contrast, push systems must send each message one by one or hold back messages without knowing if the consumer is ready.

One downside of pull-based systems is wasteful polling. A consumer might keep asking for data even if nothing is available. Kafka avoids this by letting the consumer wait until enough data is ready before responding. This keeps CPU usage low and throughput high.

Kafka also avoids a model where brokers pull data from producers. That design would need every producer to store its own data. It would require more coordination and increase the risk of disk failure. Instead, Kafka stores data on the broker, where it can be managed and replicated more easily.

How Kafka Offsets Work

Kafka splits topics into partitions. Each message in a partition has a number called an offset. The offset marks the position of the message in the log.

Offsets give consumers control. A consumer chooses where to start reading and can track what has already been processed. If a consumer crashes, it can pick up where it left off by using its last committed offset.

Kafka does not track this progress for the consumer. The consumer is responsible for managing its own offsets. This is part of what makes Kafka scalable and efficient.

By default, a consumer starts at offset zero. This means it will read all messages that are still available. It can also be configured to start at the latest offset to only read new data.

Kafka only keeps data for a limited time. If a consumer tries to read from an offset that is too old, Kafka will return an error. In that case, the consumer must reset to the earliest or latest available offset.

Key Terms

• Offset: The position of a message in a partition.
• Log End Offset: The offset where the next message will be written.
• Committed Offset: The offset of the last message the consumer has finished processing.

Delivery Options

• At-most-once: The consumer commits the offset before processing. If it crashes during processing, the message is lost.
• At-least-once: The consumer commits the offset after processing. If it crashes before committing, the message may be processed again.
• Exactly-once: This uses Kafka transactions. The message and its offset are written together. If anything fails, nothing is committed. This guarantees no duplication and no loss.

Parallelism with Consumer Groups

Kafka uses consumer groups to scale out processing. A consumer group is a set of consumers working together to read from a topic.

Kafka assigns each partition to only one consumer in the group. This avoids duplication and ensures order within each partition.

When the group changes (for example, when consumers are added or removed), Kafka reassigns partitions to the available consumers.

Examples

• 100 partitions and 100 consumers: each consumer handles one partition.
• 100 partitions and 50 consumers: each consumer handles two partitions.
• 50 partitions and 100 consumers: only 50 consumers do work, the rest are idle.

Kafka does not let multiple consumers read from the same partition in the same group. This would require the broker to manage shared state, which adds complexity. Instead, Kafka puts the responsibility on the consumer to track offsets. This makes the broker faster and simpler.

The number of partitions controls how much you can scale out. More partitions allow for more parallelism. Choosing the right number of partitions is important for performance and resource usage.

Conclusion

Kafka gives consumers control over how they pull data, where they start, and how they scale. Pull-based reads avoid overload. Offsets make it easy to recover from failure. Consumer groups allow you to scale out processing.

This design makes Kafka fast, reliable, and efficient at any scale.

Appendix: Quick Reference

• Partition: A subset of a topic. Used for parallel processing and message ordering.
• Offset: A number showing a message’s position in a partition.
• Consumer Group: A set of consumers that share the work of reading from a topic.
• Rebalancing: The process where Kafka reassigns partitions when consumers join or leave a group.
• Delivery Types:
  • At-most-once: Fast, but may lose messages.
  • At-least-once: Reliable, but may duplicate messages.
  • Exactly-once: Most accurate, but needs Kafka transactions.