Apache Kafka Complete Guide: Architecture, Topics, Partitions, Offsets, Consumer Groups & Kafka vs RabbitMQ
Learn Apache Kafka from basics to advanced concepts with practical examples, Windows commands, architecture diagrams and interview-focused explanations.
In this guide, we will understand Apache Kafka, its architecture, topics, partitions, offsets, brokers, leaders and followers, consumer groups, backpressure, message retention, scalability, Kafka vs RabbitMQ and common Kafka interview questions.
- What is Apache Kafka?
- Why Do We Need Kafka?
- Kafka Architecture
- Kafka Topics
- Kafka Partitions
- Kafka Offsets
- Kafka Brokers
- Leaders and Followers
- Consumer Groups
- How Kafka Handles Millions of Messages
- Kafka Backpressure
- Kafka Pub/Sub Architecture
- Kafka vs RabbitMQ
- Run Kafka on Windows
- Kafka Metadata
- Kafka Monitoring
- Kafka Interview Questions
- Quick Revision
1. What is Apache Kafka?
Apache Kafka is a distributed event streaming platform used to publish, store, process and consume large volumes of events in a reliable and scalable way.
Kafka is commonly used in microservices, real-time analytics, log processing, event-driven architectures and data pipelines.
2. Why Do We Need Kafka?
Traditional point-to-point communication can become difficult when many applications need to exchange large numbers of events.
Kafka provides a common event-streaming layer between producers and consumers.
- High throughput
- Horizontal scalability
- Durable event storage
- Multiple independent consumers
- Fault tolerance through replication
- Ability to replay retained events
3. Kafka Architecture
A basic Kafka system contains producers, a Kafka cluster and consumers.
Producer
A producer publishes records to Kafka topics. Kafka determines the partition where the record should be written.
Kafka Broker
A broker is a Kafka server responsible for storing and serving data. A Kafka cluster normally contains multiple brokers.
Consumer
A consumer reads records from Kafka topics. Consumers can belong to consumer groups.
4. What is a Kafka Topic?
A topic is a logical name used to categorize a stream of events.
For example:
- orders
- payments
- user-events
- notifications
5. What is a Kafka Partition?
A topic can be divided into multiple partitions. Each partition is an ordered, append-only log.
Partitions provide Kafka with its main mechanism for parallelism and horizontal scalability.
6. What is a Kafka Offset?
An offset is the sequential position of a record inside a Kafka partition.
Partition 0 Offset 0 → Message A Offset 1 → Message B Offset 2 → Message C Offset 3 → Message D
Consumers use offsets to keep track of their position while reading records.
Important Offset Strategies
| Strategy | Meaning |
|---|---|
| At-most-once | Message may be processed zero or one time. Duplicates are avoided at the cost of possible loss. |
| At-least-once | Message is processed one or more times. Duplicate processing is possible. |
| Exactly-once | Processing is designed so the resulting effect occurs exactly once under the supported Kafka processing model. |
Kafka also has an internal topic called __consumer_offsets used to store consumer-group offset information.
7. What is a Kafka Broker?
A broker is a Kafka server that stores partitions and serves producer and consumer requests.
Multiple brokers together form a Kafka cluster. Partitions can be distributed across these brokers to increase capacity and availability.
8. Kafka Leaders and Followers
Kafka replicates partitions across brokers for fault tolerance.
- The partition leader handles writes for that partition.
- Followers replicate the leader's data.
- If a leader fails, Kafka can elect another suitable replica.
9. Kafka Consumer Groups
A consumer group is a group of consumers sharing the work of consuming partitions from a topic.
↓
Consumer 1
↓
Consumer 2
↓
Consumer 3
If a topic has three partitions and a consumer group has three active consumers, the work can be distributed across those three consumers.
If the group has more consumers than partitions, some consumers will remain idle because there are not enough partitions to assign.
Multiple Consumer Groups
Different consumer groups can independently consume the same topic. This makes Kafka useful for event-driven architectures.
Order Service
Analytics Service
Notification Service
10. How Does Kafka Handle Millions of Messages?
Kafka is designed for high-throughput workloads. Several design choices contribute to its performance.
1. Partitioning
Data is distributed across partitions, allowing multiple brokers and consumers to process data in parallel.
2. Batching
Producers can send multiple records together rather than making a separate network request for every message.
3. Compression
Compression can reduce network and storage overhead.
4. Sequential I/O
Kafka's append-only log design works efficiently with sequential writes.
5. Distributed Architecture
Multiple brokers can share the workload.
11. Kafka Backpressure
Kafka consumers generally use a pull-based model. Consumers fetch records from Kafka rather than having Kafka continuously push every record into them.
This allows a slower consumer to control how quickly it fetches data.
12. Kafka Pub/Sub Architecture
Kafka supports publish-subscribe style architectures through topics and independent consumer groups.
Unlike a traditional queue where a consumed message may disappear immediately, Kafka can retain events for a configured period and allow consumers to replay them.
13. Kafka vs RabbitMQ
Kafka and RabbitMQ are both widely used messaging technologies, but their architectural models are different.
| Feature | Apache Kafka | RabbitMQ |
|---|---|---|
| Basic model | Distributed event log / event streaming platform | Message broker / queue-based messaging |
| Storage | Messages are retained according to topic retention policies | Messages are commonly removed after successful consumption/acknowledgement |
| Consumption | Pull-based consumer model | Broker commonly delivers messages to consumers |
| Scaling | Partitions provide horizontal scaling | Scaling uses queues, consumers and broker features |
| Replay | Strong support through retained offsets | Not the primary design model |
| Routing | Topics and partitions | Exchanges, bindings and queues |
| Best suited for | High-throughput event streaming, analytics and event-driven systems | Task queues, routing and traditional application messaging |
14. Run Apache Kafka on Windows
Step 1: Start ZooKeeper
bin\windows\zookeeper-server-start.bat config\zookeeper.properties
Step 2: Start Kafka Broker
bin\windows\kafka-server-start.bat config\server.properties
Step 3: Create a Topic
bin\windows\kafka-topics.bat --create --topic orders --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
Step 4: Start a Console Producer
bin\windows\kafka-console-producer.bat --topic orders --bootstrap-server localhost:9092
Step 5: Start a Console Consumer
bin\windows\kafka-console-consumer.bat --topic orders --bootstrap-server localhost:9092 --from-beginning
--bootstrap-server specifies the Kafka broker address.
--partitions specifies the number of partitions.
--replication-factor specifies the number of replicas.
15. What is Kafka Metadata?
Kafka clients need metadata about the cluster so they know where partitions are located and which broker is responsible for them.
Kafka metadata can include information such as:
- Topics
- Partitions
- Partition leaders
- Replica assignments
- Broker information
- Topic configuration
- Other cluster configuration information
16. Kafka Monitoring
Kafka should be monitored carefully in production because high traffic, consumer lag, broker failures and disk usage can affect system reliability.
Popular monitoring approaches include:
- Prometheus + Grafana
- Datadog
- Burrow
- Kafka UI tools
- Confluent monitoring tools
Important Metrics
- Consumer lag
- Messages/records per second
- Request latency
- Broker health
- Disk utilization
- Under-replicated partitions
17. Apache Kafka Interview Questions
Kafka is a distributed event streaming platform used to publish, store and consume streams of events at scale.
A topic is a logical category or stream to which producers publish records and from which consumers read records.
A partition is an ordered append-only log inside a topic. Multiple partitions allow parallel processing and scaling.
Kafka guarantees ordering within a partition. It does not provide a single global ordering across multiple partitions.
An offset identifies the position of a record within a partition.
A consumer group is a group of consumers that cooperatively consume partitions of a topic.
Yes. Each consumer group maintains its own consumption position and can independently consume the same topic.
Some consumers will remain idle because each partition can be assigned to at most one active consumer within the same group.
A broker is a Kafka server responsible for storing partitions and handling client requests.
Kafka is primarily designed around durable distributed event logs and high-throughput event streaming, while RabbitMQ is a message broker focused on queues, routing and message delivery.
Kafka's pull-based consumption model allows consumers to control how quickly they fetch and process records.
KRaft is Kafka's built-in metadata quorum architecture that allows Kafka to operate without ZooKeeper.
Post a Comment
0 Comments