Installing and Verifying Apache Kafka
Creating a Local Kafka Server Environment with Docker
This article starts Apache Kafka by using a Dockerfile.
Installing Docker
First, Docker and Docker Compose must be installed. You can check whether they are installed with the following version commands.
% docker -v
Docker version 20.10.11, build dea9396
% docker-compose -v
Docker Compose version v2.2.1
If they are not installed, install them first. This article does not explain Docker installation.
Installing an Apache Kafka Server in Docker
Next, install Apache Kafka by using Docker.
Clone the Apache Kafka Docker Repository from GitHub
git clone https://github.com/wurstmeister/kafka-docker.git
% git clone https://github.com/wurstmeister/kafka-docker.git
Cloning into 'kafka-docker'...
remote: Enumerating objects: 1100, done.
remote: Counting objects: 100% (82/82), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 1100 (delta 48), reused 50 (delta 38), pack-reused 1018
Receiving objects: 100% (1100/1100), 285.65 KiB | 7.52 MiB/s, done.
Resolving deltas: 100% (608/608), done.
Edit the docker-compose.yml File
Here we only run Kafka with Docker, so we will not consider a distributed Kafka environment and will start with a single server.
For that purpose, use docker-compose-single-broker.yml from the repository instead of docker-compose.yml.
Because Docker will run locally, edit docker-compose-single-broker.yml as follows.
... omitted ...
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
... omitted ...
Run docker-compose
docker-compose -f docker-compose-single-broker.yml up -d
The -f option is used when running a Compose file whose name is not docker-compose.yml.
Because we will use docker-compose-single-broker.yml, specify that file name after the -f option.
The -d option runs the containers in the background.
% docker-compose -f docker-compose-single-broker.yml up -d
[+] Running 17/17
⠿ zookeeper Pulled 20.8s
⠿ a3ed95caeb02 Pull complete 1.1s
... omitted ...
=> => writing image sha256:5609038ba9d28b5380b4403f4905867e9fa39cc48b03e6f32466be47b5e9b35d 0.0s
=> => naming to docker.io/library/kafka-docker_kafka 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Running 3/3
⠿ Network kafka-docker_default Created 0.1s
⠿ Container kafka-docker_kafka_1 Started 0.7s
⠿ Container kafka-docker_zookeeper_1 Started 0.8s
Check Whether Docker Started Correctly
After running the command above, you can confirm that Kafka and ZooKeeper are running.
docker ps -a
% docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
30cf1bccb37f kafka-docker_kafka "start-kafka.sh" 2 minutes ago Up 2 minutes 0.0.0.0:9092->9092/tcp kafka-docker_kafka_1
c578f5c36248 wurstmeister/zookeeper "/bin/sh -c '/usr/sb..." 2 minutes ago Up 2 minutes 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp kafka-docker_zookeeper_1
Installing a Local Kafka Client
Now the Kafka server that will receive messages through Docker is running. Next, install a Kafka client locally and send and receive messages to verify the setup.
For compatibility, first check the version of the Kafka server started above and install the same client version.
Enter the cloned repository and check the contents of the Dockerfile.
... omitted ...
ARG kafka_version=2.8.1
ARG scala_version=2.13
... omitted ...
You can find the Kafka and Scala versions near the top of the file.
Download the same Kafka version from the DOWNLOAD page on the official Kafka website.
The following example downloads it with curl.
curl https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-2.8.1.tgz -o kafka_2.13.2.8.1.tgz
% curl https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-2.8.1.tgz -o kafka_2.13-2.8.1.tgz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 68.1M 100 68.1M 0 0 130k 0 0:08:55 0:08:55 --:--:-- 115k
After the download finishes, extract the archive as follows.
tar xzvf kafka_2.13-2.8.1.tgz
Create a Topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic devkuma-topic
% cd kafka_2.13-2.8.1
% bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic devkuma-topic
Created topic devkuma-topic.
Run the Producer
bin/kafka-console-producer.sh --topic devkuma-topic --broker-list localhost:9092
When you run the CLI command below, it waits for messages to be sent.
bin/kafka-console-producer.sh --topic devkuma-topic --broker-list localhost:9092
>
If > appears and the terminal waits, it is ready to send messages.
Run the Consumer Before sending messages from the Producer console, open a Consumer console that will receive them.
Open another terminal window and run the Consumer console as follows.
bin/kafka-console-consumer.sh --topic devkuma-topic --bootstrap-server localhost:9092 --from-beginning
When you run the CLI command below, it waits to receive messages.
$ bin/kafka-console-consumer.sh --topic devkuma-topic --bootstrap-server localhost:9092 --from-beginning
If it runs and waits without displaying anything, it is ready to receive messages.
Send Messages from the Producer and Confirm Receipt in the Consumer Return to the terminal running the Producer console and type anything.
% bin/kafka-console-producer.sh --topic devkuma-topic --broker-list localhost:9092
>hello
>devkuma
>welcome to devkuma
You should then see the messages received in the terminal running the Consumer console.
% bin/kafka-console-consumer.sh --topic devkuma-topic --bootstrap-server localhost:9092 --from-beginning
hello
devkuma
welcome to devkuma