Post

Part 4 - Run a multiple instances of Apache Kafka

Posted
By Ashim Khadka
1 min read
Part 4 - Run a multiple instances of Apache Kafka

To follow the below sections, you need to set up the Apache Kafka. And for that, you can follow the steps define in startup section.

After the installation, go to the Kafka directory, something like this kafka_2.13-2.8.0 and follow through the steps to create a topic, start producing and consuming the messages.

Zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

Check status

  • Add 4lw.commands.whitelist=* in zookeeper.properties to whitelist command like stat, conf
  • Check the status of zookeeper using telnet localhost 2181

Create Configuration for Kafka Instance

  • Make a copy of existing server.properties files to
    • For instance 0 - server-0.properties
    • For instance 1 - server-1.properties
    • For instance 2 - server-2.properties
  • Update broker.id
    • For instance 0 - broker.id=0
    • For instance 1 - broker.id=1
    • For instance 2 - broker.id=2
  • Add server listeners
    • For instance 0 - listeners=PLAINTEXT://localhost:9090
    • For instance 1 - listeners=PLAINTEXT://localhost:9091
    • For instance 2 - listeners=PLAINTEXT://localhost:9092
  • Update log.dirs
    • For instance 0 - log.dirs=/tmp/kafka-logs-0
    • For instance 1 - log.dirs=/tmp/kafka-logs-1
    • For instance 2 - log.dirs=/tmp/kafka-logs-2

You can also find these configurations in samples directory.

Start Multiple Instance

Start Instance 0

bin/kafka-server-start.sh config/server-0.properties

Start Instance 1

bin/kafka-server-start.sh config/server-1.properties

Start Instance 2

bin/kafka-server-start.sh config/server-2.properties

Check Broker Registration using Zookeeper

  • telnet localhost 2181
  • type stat

Create Topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic replica_topic

Describe Topic

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic replica_topic Output:

1
2

Topic: replica_topic    TopicId: 6pL_IeaHRzW_OA4X_in1Og PartitionCount: 1       ReplicationFactor: 3    Configs:
    Topic: replica_topic    Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2

Kafka Console Producer

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic replica_topic

Kafka Console Consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:9090 --topic replica_topic --from-beginning

Kafka Series

kafka
kafka-series
This post is licensed under CC BY 4.0 by the author.