Apache Kafka Part III – Multiple Brokers

Introduction

In my previous post, we installed Kafka in a single node, single broker configuration. In this post, we will explorer what it takes to run in a multi broker configuration. We will keep working in our Ubuntu environment, and we will leave our existing broker in place. We will add two additional broker instances on our local VM instance, which will result in the following configuration:

Kafka - Multiple Brokers

If you are continuing to follow along with my previous post I do recommend that you reboot your Ubuntu VM instance before going along with the example, this way we are sure that all ports are freed up.

Starting Zookeeper

The ZooKeeper starting procedure and configuration files remain the same, so you can start the ZooKeeper server as follows:

Starting the additional Kafka brokers

To get to three brokers, we need two additional instances on top of the one broker we already have defined with the config/server.properties file. Different server properties are required for each broker. Each property file needs to define unique, different values for the following properties:

  • broker.id – the unique ID of the broker instance
  • port – the port that the broker will be listening on.
  • log.dir – the directory for the broker’s log

The easiest way to implement this is the copy the existing config file into two new config files (one for each new broker):

Next, we need to edit both files and set the following properties:

config/server-1.properties:

config/server-2.properties:

We can follow a similar procedure for each new broker. Of course, if production environments, brokers instances will run on different machines, but this gives us a feel for what needs to be done.

We can now start each broker in a new terminal window.

Broker 1 (default broker):

Second broker:

And finally the third broker:

Creating a replicated topic

Since we now have three different brokers running, let’s create a new topic with a replication factor of three:

To check which broker is doing what we can use the describe topics command as follows:

The output of the describe topics command is shown below:

DescribeTopics

The first output line gives a summary off all partitions, showing topic name, partition count and the replication factor:

Each additional line gives information about one partition. Since we only have one partition for this topic there is only one line:

  • The Leader is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions. In our case we see that our last broker (with broker.id 2) is the leader.
  • The Replicas is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive. Here we see that all brokers replicate the topic.
  • Isr is the set of “in-sync” replicas. This is the subset of replicas that are currently alive and caught up to the leader. In this case we see that replicas are in sync.

Starting a producers to send  messages

Let’s publish a few messages to our new telemetry-replicated topic:

SendToReplicatedTopic

Note that we sent the messages to the first broker  with port number 9092

Consuming the messages

Now, let’s consume the published messages:

ConsumingMessagesSo, we see that everything still works as we expected it. Let’s make things a bit more interesting in the next section and test out our fault tolerance.

Testing Fault Tolerance

We know that the broker with ID = 2 was the leader. This is the last broker. Let’s go ahead and kill that instance. First we need to get the Linux process id of the instance:

since we used config/server-2.properties to start the broker we can grep the ps output on that. This produces the following output:

We need to use the second process id, since this is the real server process (the first is simple the shell script that started the process). Now that we have the process id we can kill it:

The full terminal output is shown below:

KillProcessFullOutput

Now, let’s do a describe topics again. This time we get the following output:

We see a few things now:

  • Leadership has switched to the first broker (with id = 0)
  • The in-sync replica set no longer contains the broker that was killed (with id = 2)

But because of the replicas of Kafka topics, the messages are still available even though the leader that took the writes originally was down. We can test this by starting the consumer again:

As we can see, all messages are still there and are ready to be consumed.

Conclusion

In this post, we created a single-node, multiple broker configuration to get a feel for how to work with a Kafka cluster. We learned that each broker has its own config file with a unique broker id, port and log directory.

We then created a replicated topic and published and consumed messages. We tested fault-tolerance by “shooting” our  leading broker, and saw that because of the replication, the published messages remain highly available, even if they were published to the killed leader.

In the next couple posts, we will start writing our own consumers. We’ll start out in java, and then move on to Python and .NET languages.

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *