Apache Kafka and KRaft: A Quick Guide Without ZooKeeper
A visual overview of what KRaft is, how it differs from ZooKeeper, its architecture, basic configuration steps, and useful commands.
- Complexity
- Intermediate
- Reading
- 5 min read
- Published
While the image is fitted, the left and right arrows move to the previous and next infographic. Once it is zoomed in, the arrows move the image instead. The plus and minus keys zoom, and the zero key fits it again.
Loading the image…
About this infographic
What is KRaft: Apache Kafka's native mode (Kafka Raft Metadata mode) that replaces ZooKeeper to manage metadata using a Raft-based controller quorum. Key benefits highlighted: no external dependencies, internal metadata, controller quorum, reduced operational complexity. Changes compared to ZooKeeper: architecture shifts from two systems (Kafka + ZooKeeper) to a single system; components change from Brokers + ZooKeeper to Brokers + controller quorum; external metadata management vs. internal metadata management in Kafka (Raft); reduced operational complexity; native Raft-based consistency; external single point of failure (ZooKeeper) eliminated. How it works: producers and consumers flow through brokers; partitions with replicas for fault tolerance; the controller quorum maintains metadata state; leader election per partition; brokers register with the quorum and receive partition assignments. KRaft architecture: C1–C3 controller quorum (Raft) for metadata; broker cluster for data traffic; storage of partitions and replicas; separation of data and metadata flows. Key concepts covered: KRaft, controller quorum, broker, topic, partition, replica, metadata log, bootstrap (startup addresses for clients and tools). Basic configuration (server.properties shown): - process.roles=broker,controller - node.id=1 - controller.quorum.voters=1@node1:9093,2@node2:9093,3@node3:9093 - listeners=PLAINTEXT://:9092,CONTROLLER://:9093 - inter.broker.listener.name=PLAINTEXT - controller.listener.names=CONTROLLER - log.dirs=/var/lib/kafka/data - metadata.log.dir=/var/lib/kafka/meta Key commands illustrated: generate cluster UUID, format storage, start server, create/describe topics, check metadata quorum status. Configuration steps: format storage with cluster ID; define node IDs; configure the controller quorum; define listeners/ports and directories; start brokers/controllers; create topics and verify metadata; validate the cluster and quorum health. Advantages: simpler and lighter deployments; fewer components; eliminates dependency on ZooKeeper; improved startup and metadata performance; greater operational reliability. When to use KRaft: for new deployments; for migrations from ZooKeeper; with modern versions of Kafka 3.3+ (recommended in the slide). Considerations: check compatibility; use an odd number of controllers (e.g., 3 or 5); plan for redundancy and log storage; prepare a migration strategy from ZooKeeper; consider disk performance. Key takeaways: no ZooKeeper; better performance; scalable architecture; greater fault tolerance.
Key takeaways
- KRaft replaces ZooKeeper by managing metadata within Kafka using a Raft-based quorum.
- The architecture is simplified: from two systems to a single one, reducing complexity and external points of failure.
- Metadata (partition leaders, broker registry, assignments) resides in the controller quorum.
- Odd numbers of controllers (for example, 3 or 5) are recommended for Raft consensus.
- Basic configuration includes defining process roles, node.id, controller.quorum.voters, listeners, and data/meta routes.
- The data stream remains in brokers; the metadata stream is managed by controllers.
- Key benefits: simpler deployments, improved metadata performance, scalability, and reliability.
- Use Cases: New Deployments and Migrations in Kafka 3.3+ (as shown on the slide).
