Kafka Docker Environment

This is a tutorial to get a Kafka environment using Docker. You can dive and check details in the git repository here.

Cloning git repository

$ git clone https://github.com/rancavil/kafka.git
$ cd kafka

Building the image

We are using Kafka version 3.0.0.

$ docker build -t kafka:3.0.0 .

Creating the container

We are going to create a container with Kafka.

$ docker run -d --name <container-name> -p 9092:9092 --hostname <your-hostname/ip address> kafka:3.0.0

Kafka container will listen through port 9092 (the standard port).

If we want to know what’s happening inside the container, execute the following command to see the log.

$ docker logs -f <container-name>

Testing

We can use the following example to verify if Kafka is working. We are going to use Python and their Kafka module. We will write a consumer and a producer.

The producer will send messages to the topic (called stream-1) and the consumer will read from there.

First, you have to install the Python Kafka module.

$ pip install kafka-python

Writing producer.py

#!/usr/bin/env python

import socket
import kafka as k


if __name__ == "__main__":
    hostname = socket.gethostname()
    producer = k.KafkaProducer(
            bootstrap_servers=[f"{hostname}:9092"],
            api_version=(0, 10, 1)
    )
    print("Sending data")
    for i in range(1,101):
        msg = f"Data {i}"
        print(f"Sending {msg}")
        future = producer.send("stream-1",msg.encode("utf-8"))
        result = future.get(timeout=10)

    producer.flush()

The producer will write a simple text message through the topic (stream-1).

Writing consumer.py

#!/usr/bin/env python

import socket
import kafka as k

if __name__ == '__main__':
    hostname = socket.gethostname()
    consumer = k.KafkaConsumer("stream-1",
            bootstrap_servers=[f"{hostname}:9092"],
            api_version=(0, 10, 1)
    )
    print("Receiving Data")
    for msg in consumer:
        print (msg.value)

The consumer will read from the topic (stream-1) a simple plain text message.

Important: you must set up <hostname/ip address>. If you are in a local environment you can use localhost

You have to execute:

The producer.

$ python producer.py
Sending Data::
Sendind Data 1
Sendind Data 2
Sendind Data 3
....
...
..
.

In another terminal, you must execute the consumer. The consumer will wait until messages start to come.

$ python consumer.py
Receiving Data::
Data 1
Data 2
Data 3
....
...
..
.

Another way to test how the Kafka container is working.

Creating a topic

We will create a topic called TOPIC-TEST.

$ docker exec -i kafka-1 bin/kafka-topics.sh --create --topic TOPIC-TEST --bootstrap-server localhost:9092 --partitions 2 --replication-factor 1

Writing some messages (events)

We must execute and write messages (events) using the TOPIC-TEST.

$ docker exec -i <container-name> bin/kafka-console-producer.sh — topic TOPIC-TEST — bootstrap-server localhost:9092
This is an event with data
Another event

Reading messages (events)

This command executes a Kafka consumer and waits for events.

$ docker exec -i <container-name> bin/kafka-console-consumer.sh — topic TOPIC-TEST — from-beginning — bootstrap-server localhost:9092
This is an event with data
Another event

Summary

The tutorial aims to provide an easy way to get a local docker environment for starting and working with Kafka as a development tool.