Kafka and Zookeeper

Why Kafka
Apache Kafka is a fast and scalable messaging queue, capable of handling heavy loads in context of read and write, i.e. IO-related, stuff. You can find more about Kafka on http://kafka.apache.org/.

Why Zookeeper

Apache Kafka requires a running ZooKeeper instance, which is used for reliable distributed coordination

Download Kafka from 

http://kafka.apache.org/downloads.html

Download Binaries file


Download Zookeeper from

http://archive.apache.org/dist/zookeeper/zookeeper-3.4.8/



Unzip Kafka and Zookeeper



Setup Enviornment Variable for Zookeeper

Now we will Set Path Variable for both Zookeeper and Kafka

%ZOOKEEPER_HOME%\bin;C:\Users\NEERAJ\Downloads\kafka_2.11-0.11.0.2\bin\windows

Zookeeper Setup


Go to

C:\Users\NEERAJ\Downloads\zookeeper-3.4.8\conf

1.)Rename file  “zoo_sample.cfg” to “zoo.cfg”
2.) Open zoo.cfg  in Notepad and Change location of "dataDir" in config file to dataDir=\zookeeper-3.4.8\data
3.) Run ZooKeeper by opening a new cmd and type "zkserver" to run

ZooKeeper is up and running on port 2181


Kafka Setup

1.) Edit server.properties under "C:\Users\NEERAJ\Downloads\kafka_2.11-0.11.0.2\config"
2.) Set log.dirs=C:\Users\NEERAJ\Downloads\kafka_2.11-0.11.0.2\kafka-logs

Running Kafka Server

Pls ensure your Zookeeper is running

In command prompt got to config location of Kafka and type "kafka-server-start server.properties"

Something like below

C:\Users\NEERAJ\Downloads\kafka_2.11-0.11.0.2\config>kafka-server-start server.properties

Kafka is running on Port 9092


Create Topic, Producer and Consumer

To Create Topic , Use below command

Go To windows folder under Kafka

Topic

Execute below command in CMD

C:\Users\NEERAJ\Downloads\kafka_2.11-0.11.0.2\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Created topic "test".


As you can see Topic has been created

Producer

Now Open a New CMD

Execute command "kafka-console-producer.bat --broker-list localhost:9092 --topic test"

C:\Users\NEERAJ\Downloads\kafka_2.11-0.11.0.2\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic test

Now your producer is ready to produce messages

Consumer


Now Open a New CMD

Execute command "kafka-console-consumer.bat --zookeeper localhost:2181 --topic test"

C:\Users\NEERAJ\Downloads\kafka_2.11-0.11.0.2\bin\windows>kafka-console-consumer.bat --zookeeper localhost:2181 --topic test


Note: You can see that all these "kafka-topics.bat", "kafka-console-producer.bat" and "kafka-console-consumer.bat" are present under windows folder

Now you can type message in producer and see if consumer receive those messages :)


Refer below Screenshots

Producer

Consumer

As you can see Consumer is receiving messages

Some Other Useful Commands

List Topics: kafka-topics.bat --list --zookeeper localhost:2181 

Command for New Kafka version: bin/kafka-topics.sh --list --bootstrap-server <BROKER-LIST>

Describe Topic: kafka-topics.bat --describe --zookeeper localhost:2181 --topic [Topic Name]

Read messages from the beginning: kafka-console-consumer.bat --zookeeper localhost:2181 --topic [Topic Name] --from-beginning

Delete Topic: kafka-run-class.bat kafka.admin.TopicCommand --delete --topic [topic_to_delete] --zookeeper localhost:2181