Install Kafka+Zookeeper using Docker container in Windows Subsystem for Linux(WSL)
The article is about how to install Single Node multiple broker setup of Kafka and Zookeeper in Windows subsystem for Linux(WSL). Previous article has installation guide for windows. This helps to simulate installing Kafka and Zookeeper in Linux from scratch. Installation 1. Install Windows subsystem for Linux in Windows machine The document contains step-by-step process for installation of ubuntu https://learn.microsoft.com/en-us/windows/wsl/install Click on search bar and type ubuntu and open the app. Setup username and password in the ubuntu app. C:\Users\suman>wsl --install Ubuntu is already installed. Launching Ubuntu... Installing, this may take a few minutes... Please create a default UNIX user account. The username does not need to match your Windows username. For more information visit: https://aka.ms/wslusers Enter new UNIX username: sumanthshastry New password: Retype new password: passwd: password updated successfully The operation completed successfully. Installation successful! To run a command as administrator (user "root"), use "sudo ". See "man sudo_root" for details. Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.133.1-microsoft-standard-WSL2 x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage This message is shown once a day. To disable it please create the /home/sumanthshastry/.hushlogin file. sumanthshastry@shastry:~$ 2. Install required software's Update Advanced Package tool apt-get update root@shastry:~# apt-get update Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1205 kB] Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB] Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [219 kB] Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1476 kB] Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [244 kB] Get:10 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [846 kB] Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [161 kB] Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] Get:13 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] Get:14 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7476 B] Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] Get:16 http://archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB] Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB] Get:18 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB] Get:19 http://archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB] Get:20 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B] Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1421 kB] Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [279 kB] Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1504 kB] Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [247 kB] Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1052 kB] Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [237 kB] Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B] Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B] Fetched 29.8 MB in 7s (4478 kB/s) Reading package lists... Done root@shastry:~# Install docker apt-get install do
The article is about how to install Single Node multiple broker setup of Kafka and Zookeeper in Windows subsystem for Linux(WSL). Previous article has installation guide for windows. This helps to simulate installing Kafka and Zookeeper in Linux from scratch.
Installation
1. Install Windows subsystem for Linux in Windows machine
- The document contains step-by-step process for installation of ubuntu https://learn.microsoft.com/en-us/windows/wsl/install
- Click on search bar and type ubuntu and open the app.
- Setup username and password in the ubuntu app.
C:\Users\suman>wsl --install
Ubuntu is already installed.
Launching Ubuntu...
Installing, this may take a few minutes...
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: sumanthshastry
New password:
Retype new password:
passwd: password updated successfully
The operation completed successfully.
Installation successful!
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.133.1-microsoft-standard-WSL2 x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
This message is shown once a day. To disable it please create the
/home/sumanthshastry/.hushlogin file.
sumanthshastry@shastry:~$
2. Install required software's
- Update Advanced Package tool apt-get update
root@shastry:~# apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1205 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [219 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1476 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [244 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [846 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [161 kB]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB]
Get:13 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB]
Get:14 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7476 B]
Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B]
Get:16 http://archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1421 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [279 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1504 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [247 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1052 kB]
Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [237 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB]
Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB]
Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B]
Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB]
Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB]
Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB]
Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB]
Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B]
Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 29.8 MB in 7s (4478 kB/s)
Reading package lists... Done
root@shastry:~#
Install docker apt-get install docker
root@shastry:~# apt-get install docker
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
wmdocker
The following NEW packages will be installed:
docker wmdocker
0 upgraded, 2 newly installed, 0 to remove and 79 not upgraded.
Need to get 14.3 kB of archives.
After this operation, 58.4 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 wmdocker amd64 1.5-2 [13.0 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 docker all 1.5-2 [1316 B]
Fetched 14.3 kB in 1s (28.4 kB/s)
Selecting previously unselected package wmdocker.
(Reading database ... 24208 files and directories currently installed.)
Preparing to unpack .../wmdocker_1.5-2_amd64.deb ...
Unpacking wmdocker (1.5-2) ...
Selecting previously unselected package docker.
Preparing to unpack .../archives/docker_1.5-2_all.deb ...
Unpacking docker (1.5-2) ...
Setting up wmdocker (1.5-2) ...
Setting up docker (1.5-2) ...
Processing triggers for man-db (2.10.2-1) ...
root@shastry:~#
- Install docker compose apt-get install docker-compose
root@shastry:~# apt-get install docker-compose
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
bridge-utils containerd dns-root-data dnsmasq-base docker.io pigz python3-attr python3-certifi python3-chardet python3-distutils python3-docker python3-dockerpty python3-docopt python3-dotenv python3-idna
python3-jsonschema python3-lib2to3 python3-pyrsistent python3-requests python3-setuptools python3-texttable python3-urllib3 python3-websocket runc ubuntu-fan
Suggested packages:
ifupdown aufs-tools btrfs-progs cgroupfs-mount | cgroup-lite debootstrap docker-doc rinse zfs-fuse | zfsutils python-attr-doc python-jsonschema-doc python3-openssl python3-socks python-requests-doc
python-setuptools-doc
The following NEW packages will be installed:
bridge-utils containerd dns-root-data dnsmasq-base docker-compose docker.io pigz python3-attr python3-certifi python3-chardet python3-distutils python3-docker python3-dockerpty python3-docopt
python3-dotenv python3-idna python3-jsonschema python3-lib2to3 python3-pyrsistent python3-requests python3-setuptools python3-texttable python3-urllib3 python3-websocket runc ubuntu-fan
0 upgraded, 26 newly installed, 0 to remove and 79 not upgraded.
Need to get 71.2 MB of archives.
After this operation, 274 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 pigz amd64 2.6-1 [63.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 bridge-utils amd64 1.7-1ubuntu3 [34.4 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 runc amd64 1.1.7-0ubuntu1~22.04.2 [4267 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 containerd amd64 1.7.2-0ubuntu1~22.04.1 [36.0 MB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dns-root-data all 2023112702~ubuntu0.22.04.1 [5136 B]
Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dnsmasq-base amd64 2.90-0ubuntu0.22.04.1 [374 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-lib2to3 all 3.10.8-1~22.04 [77.6 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-distutils all 3.10.8-1~22.04 [139 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-chardet all 4.0.0-1 [98.0 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-docker all 5.0.3-1 [89.3 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-dockerpty all 0.4.1-2 [11.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-docopt all 0.6.2-4 [26.9 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-dotenv all 0.19.2-1 [20.5 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-attr all 21.2.0-1 [44.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-setuptools all 59.6.0-1.2ubuntu0.22.04.1 [339 kB]
Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyrsistent amd64 0.18.1-1build1 [55.5 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jsonschema all 3.2.0-0ubuntu2 [43.1 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-texttable all 1.6.4-1 [11.4 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 docker-compose all 1.29.2-1 [95.8 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 docker.io amd64 24.0.5-0ubuntu1~22.04.1 [28.9 MB]
Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 ubuntu-fan all 0.12.16 [35.2 kB]
Fetched 71.2 MB in 9s (7698 kB/s)
Preconfiguring packages ...
Selecting previously unselected package pigz.
(Reading database ... 40597 files and directories currently installed.)
Preparing to unpack .../00-pigz_2.6-1_amd64.deb ...
Unpacking pigz (2.6-1) ...
Selecting previously unselected package bridge-utils.
Preparing to unpack .../01-bridge-utils_1.7-1ubuntu3_amd64.deb ...
Unpacking bridge-utils (1.7-1ubuntu3) ...
Selecting previously unselected package runc.
Preparing to unpack .../02-runc_1.1.7-0ubuntu1~22.04.2_amd64.deb ...
Unpacking runc (1.1.7-0ubuntu1~22.04.2) ...
Selecting previously unselected package containerd.
Preparing to unpack .../03-containerd_1.7.2-0ubuntu1~22.04.1_amd64.deb ...
Unpacking containerd (1.7.2-0ubuntu1~22.04.1) ...
Selecting previously unselected package dns-root-data.
Preparing to unpack .../04-dns-root-data_2023112702~ubuntu0.22.04.1_all.deb ...
Unpacking dns-root-data (2023112702~ubuntu0.22.04.1) ...
Selecting previously unselected package dnsmasq-base.
Preparing to unpack .../05-dnsmasq-base_2.90-0ubuntu0.22.04.1_amd64.deb ...
Unpacking dnsmasq-base (2.90-0ubuntu0.22.04.1) ...
Selecting previously unselected package python3-lib2to3.
Preparing to unpack .../06-python3-lib2to3_3.10.8-1~22.04_all.deb ...
Unpacking python3-lib2to3 (3.10.8-1~22.04) ...
Selecting previously unselected package python3-distutils.
Preparing to unpack .../07-python3-distutils_3.10.8-1~22.04_all.deb ...
Unpacking python3-distutils (3.10.8-1~22.04) ...
Selecting previously unselected package python3-certifi.
Preparing to unpack .../08-python3-certifi_2020.6.20-1_all.deb ...
Unpacking python3-certifi (2020.6.20-1) ...
Selecting previously unselected package python3-chardet.
Preparing to unpack .../09-python3-chardet_4.0.0-1_all.deb ...
Unpacking python3-chardet (4.0.0-1) ...
Selecting previously unselected package python3-idna.
Preparing to unpack .../10-python3-idna_3.3-1_all.deb ...
Unpacking python3-idna (3.3-1) ...
Selecting previously unselected package python3-urllib3.
Preparing to unpack .../11-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ...
Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ...
Selecting previously unselected package python3-requests.
Preparing to unpack .../12-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ...
Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
Selecting previously unselected package python3-websocket.
Preparing to unpack .../13-python3-websocket_1.2.3-1_all.deb ...
Unpacking python3-websocket (1.2.3-1) ...
Selecting previously unselected package python3-docker.
Preparing to unpack .../14-python3-docker_5.0.3-1_all.deb ...
Unpacking python3-docker (5.0.3-1) ...
Selecting previously unselected package python3-dockerpty.
Preparing to unpack .../15-python3-dockerpty_0.4.1-2_all.deb ...
Unpacking python3-dockerpty (0.4.1-2) ...
Selecting previously unselected package python3-docopt.
Preparing to unpack .../16-python3-docopt_0.6.2-4_all.deb ...
Unpacking python3-docopt (0.6.2-4) ...
Selecting previously unselected package python3-dotenv.
Preparing to unpack .../17-python3-dotenv_0.19.2-1_all.deb ...
Unpacking python3-dotenv (0.19.2-1) ...
Selecting previously unselected package python3-attr.
Preparing to unpack .../18-python3-attr_21.2.0-1_all.deb ...
Unpacking python3-attr (21.2.0-1) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../19-python3-setuptools_59.6.0-1.2ubuntu0.22.04.1_all.deb ...
Unpacking python3-setuptools (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package python3-pyrsistent:amd64.
Preparing to unpack .../20-python3-pyrsistent_0.18.1-1build1_amd64.deb ...
Unpacking python3-pyrsistent:amd64 (0.18.1-1build1) ...
Selecting previously unselected package python3-jsonschema.
Preparing to unpack .../21-python3-jsonschema_3.2.0-0ubuntu2_all.deb ...
Unpacking python3-jsonschema (3.2.0-0ubuntu2) ...
Selecting previously unselected package python3-texttable.
Preparing to unpack .../22-python3-texttable_1.6.4-1_all.deb ...
Unpacking python3-texttable (1.6.4-1) ...
Selecting previously unselected package docker-compose.
Preparing to unpack .../23-docker-compose_1.29.2-1_all.deb ...
Unpacking docker-compose (1.29.2-1) ...
Selecting previously unselected package docker.io.
Preparing to unpack .../24-docker.io_24.0.5-0ubuntu1~22.04.1_amd64.deb ...
Unpacking docker.io (24.0.5-0ubuntu1~22.04.1) ...
Selecting previously unselected package ubuntu-fan.
Preparing to unpack .../25-ubuntu-fan_0.12.16_all.deb ...
Unpacking ubuntu-fan (0.12.16) ...
Setting up python3-dotenv (0.19.2-1) ...
Setting up python3-attr (21.2.0-1) ...
Setting up python3-texttable (1.6.4-1) ...
Setting up python3-docopt (0.6.2-4) ...
Setting up dnsmasq-base (2.90-0ubuntu0.22.04.1) ...
Setting up runc (1.1.7-0ubuntu1~22.04.2) ...
Setting up dns-root-data (2023112702~ubuntu0.22.04.1) ...
Setting up python3-chardet (4.0.0-1) ...
Setting up python3-certifi (2020.6.20-1) ...
Setting up python3-idna (3.3-1) ...
Setting up bridge-utils (1.7-1ubuntu3) ...
Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ...
Setting up python3-pyrsistent:amd64 (0.18.1-1build1) ...
Setting up pigz (2.6-1) ...
Setting up containerd (1.7.2-0ubuntu1~22.04.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up python3-lib2to3 (3.10.8-1~22.04) ...
Setting up python3-websocket (1.2.3-1) ...
Setting up python3-dockerpty (0.4.1-2) ...
Setting up python3-distutils (3.10.8-1~22.04) ...
Setting up ubuntu-fan (0.12.16) ...
Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service → /lib/systemd/system/ubuntu-fan.service.
Setting up python3-setuptools (59.6.0-1.2ubuntu0.22.04.1) ...
Setting up docker.io (24.0.5-0ubuntu1~22.04.1) ...
Adding group `docker' (GID 117) ...
Done.
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up python3-jsonschema (3.2.0-0ubuntu2) ...
Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
Setting up python3-docker (5.0.3-1) ...
Setting up docker-compose (1.29.2-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
Processing triggers for man-db (2.10.2-1) ...
root@shastry:~#
- Install OpenJDK 17
apt-get install openjdk-17-jdk
root@shastry:~# apt-get install openjdk-17-jdk
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
adwaita-icon-theme alsa-topology-conf alsa-ucm-conf at-spi2-core ca-certificates-java dconf-gsettings-backend dconf-service fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra
gsettings-desktop-schemas gtk-update-icon-cache hicolor-icon-theme humanity-icon-theme java-common libasound2 libasound2-data libatk-bridge2.0-0 libatk-wrapper-java libatk-wrapper-java-jni libatk1.0-0
libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data libavahi-common3 libcairo-gobject2 libcairo2 libcups2 libdatrie1 libdconf1 libdeflate0 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2
libdrm-radeon1 libfontconfig1 libfontenc1 libfreetype6 libgail-common libgail18 libgdk-pixbuf-2.0-0 libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgif7 libgl1 libgl1-amber-dri libgl1-mesa-dri
libglapi-mesa libglvnd0 libglx-mesa0 libglx0 libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice-dev libice6 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2 libllvm15 libnspr4
libnss3 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpciaccess0 libpcsclite1 libpixman-1-0 libpthread-stubs0-dev librsvg2-2 librsvg2-common libsensors-config libsensors5 libsm-dev libsm6
libthai-data libthai0 libtiff5 libwebp7 libx11-dev libx11-xcb1 libxau-dev libxaw7 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0 libxcb-render0 libxcb-shape0 libxcb-shm0 libxcb-sync1
libxcb-xfixes0 libxcb1-dev libxcomposite1 libxcursor1 libxdamage1 libxdmcp-dev libxfixes3 libxft2 libxi6 libxinerama1 libxkbfile1 libxmu6 libxpm4 libxrandr2 libxrender1 libxshmfence1 libxt-dev libxt6
libxtst6 libxv1 libxxf86dga1 libxxf86vm1 openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless session-migration ubuntu-mono x11-common x11-utils x11proto-dev xorg-sgml-doctools xtrans-dev
...
root@shastry:~#
3. Create kafka folder and docker compose file
mkdir /opt/kafka
cd /opt/kafka
4. Create Docker compose file
- Copy contents of docker-compose.yml from git and save it
root@shastry:/opt/kafka# pwd
/opt/kafka
root@shastry:/opt/kafka# cat docker-compose.yml
version: '2.1'
services:
zookeep1:
image: zookeeper:3.4.14
hostname: zookeep1
ports:
- "2181:2181"
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zookeep1:2888:3888 server.2=zookeep2:2888:3888 server.3=zookeep3:2888:3888
volumes:
- ./kafka-windows-docker-data/zookeep1/data:/data
- ./kafka-windows-docker-data/zookeep1/datalog:/datalog
zookeep2:
image: zookeeper:3.4.14
hostname: zookeep2
ports:
- "2182:2182"
environment:
ZOO_MY_ID: 2
ZOO_PORT: 2182
ZOO_SERVERS: server.1=zookeep1:2888:3888 server.2=zookeep2:2888:3888 server.3=zookeep3:2888:3888
volumes:
- ./kafka-windows-docker-data/zookeep2/data:/data
- ./kafka-windows-docker-data/zookeep2/datalog:/datalog
zookeep3:
image: zookeeper:3.4.14
hostname: zookeep3
ports:
- "2183:2183"
environment:
ZOO_MY_ID: 3
ZOO_PORT: 2183
ZOO_SERVERS: server.1=zookeep1:2888:3888 server.2=zookeep2:2888:3888 server.3=zookeep3:2888:3888
volumes:
- ./kafka-windows-docker-data/zookeep3/data:/data
- ./kafka-windows-docker-data/zookeep3/datalog:/datalog
kafka1:
image: confluentinc/cp-kafka:7.5.0
user: root
hostname: kafka1
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeep1:2181,zookeep2:2182,zookeep3:2183"
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
volumes:
- ./kafka-windows-docker-data/kafka1/data:/var/lib/kafka/data
depends_on:
- zookeep1
- zookeep2
- zookeep3
kafka2:
image: confluentinc/cp-kafka:7.5.0
user: root
hostname: kafka2
ports:
- "9093:9093"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka2:19093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeep1:2181,zookeep2:2182,zookeep3:2183"
KAFKA_BROKER_ID: 2
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
volumes:
- ./kafka-windows-docker-data/kafka2/data:/var/lib/kafka/data
depends_on:
- zookeep1
- zookeep2
- zookeep3
kafka3:
image: confluentinc/cp-kafka:7.5.0
user: root
hostname: kafka3
ports:
- "9094:9094"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka3:19094,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeep1:2181,zookeep2:2182,zookeep3:2183"
KAFKA_BROKER_ID: 3
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
volumes:
- ./kafka-windows-docker-data/kafka3/data:/var/lib/kafka/data
depends_on:
- zookeep1
- zookeep2
- zookeep3
root@shastry:/opt/kafka#
5. Download kafka client
- Download kafka client from http://mirror.cogentco.com/pub/apache/kafka/3.6.1/kafka_2.12-3.6.1.tgz
root@shastry:/opt/kafka# wget http://mirror.cogentco.com/pub/apache/kafka/3.6.1/kafka_2.12-3.6.1.tgz
--2024-03-03 21:03:58-- http://mirror.cogentco.com/pub/apache/kafka/3.6.1/kafka_2.12-3.6.1.tgz
Resolving mirror.cogentco.com (mirror.cogentco.com)... 204.157.3.70
Connecting to mirror.cogentco.com (mirror.cogentco.com)|204.157.3.70|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 113609072 (108M) [application/x-gzip]
Saving to: ‘kafka_2.12-3.6.1.tgz’
kafka_2.12-3.6.1.tgz 100%[===================================================================================================================>] 108.35M 2.14MB/s in 89s
2024-03-03 21:05:28 (1.21 MB/s) - ‘kafka_2.12-3.6.1.tgz’ saved [113609072/113609072]
root@shastry:/opt/kafka#
- untar downloaded kafka client
root@shastry:/opt/kafka# ls -ltr
total 110952
-rw-r--r-- 1 root root 113609072 Dec 5 22:23 kafka_2.12-3.6.1.tgz
-rw-r--r-- 1 root root 3611 Mar 3 21:01 docker-compose.yml
root@shastry:/opt/kafka# tar -xf kafka_2.12-3.6.1.tgz
root@shastry:/opt/kafka# ls -ltr
total 110956
drwxr-xr-x 7 root root 4096 Nov 24 15:13 kafka_2.12-3.6.1
-rw-r--r-- 1 root root 113609072 Dec 5 22:23 kafka_2.12-3.6.1.tgz
-rw-r--r-- 1 root root 3611 Mar 3 21:01 docker-compose.yml
root@shastry:/opt/kafka# mv kafka_2.12-3.6.1 kafka_cli
root@shastry:/opt/kafka# ls -ltr
total 110956
drwxr-xr-x 7 root root 4096 Nov 24 15:13 kafka_cli
-rw-r--r-- 1 root root 113609072 Dec 5 22:23 kafka_2.12-3.6.1.tgz
-rw-r--r-- 1 root root 3611 Mar 3 21:01 docker-compose.yml
root@shastry:/opt/kafka#
6. Deploy kafka and zookeeper
docker-compose up -d --build
root@shastry:/opt/kafka# docker-compose up -d --build
Creating network "kafka_default" with the default driver
Pulling zookeep1 (zookeeper:3.4.14)...
3.4.14: Pulling from library/zookeeper
69692152171a: Pull complete
ce2b89b60818: Pull complete
a3c211c6bdc9: Pull complete
4c25606a3064: Pull complete
d7d77cede5c8: Pull complete
b2a5fc6a5ffe: Pull complete
49bd81bc3707: Pull complete
b6b7d3138b59: Pull complete
Digest: sha256:3882d9493d387ba77b7b69e2a031b9396477ec29483d51ceaed645c1389182e5
Status: Downloaded newer image for zookeeper:3.4.14
Pulling kafka1 (confluentinc/cp-kafka:7.5.0)...
7.5.0: Pulling from confluentinc/cp-kafka
57168402cb72: Pull complete
6edc125f768c: Pull complete
6086b74cc48f: Pull complete
04b3375cab83: Pull complete
f80eb2db9d48: Pull complete
95d7897d98ce: Pull complete
bd3de482b201: Pull complete
0fbb79a570de: Pull complete
8592ceb925f7: Pull complete
be07a4b85081: Pull complete
210860798c03: Pull complete
Digest: sha256:fbbb6fa11b258a88b83f54d4f0bddfcffbf2279f99d66a843486e3da7bdfbf41
Status: Downloaded newer image for confluentinc/cp-kafka:7.5.0
Creating kafka_zookeep3_1 ... done
Creating kafka_zookeep2_1 ... done
Creating kafka_zookeep1_1 ... done
Creating kafka_kafka1_1 ... done
Creating kafka_kafka3_1 ... done
Creating kafka_kafka2_1 ... done
root@shastry:/opt/kafka# docker-compose ps
Name Command State Ports
----------------------------------------------------------------------------------------------------------------------------------
kafka_kafka1_1 /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp,:::9092->9092/tcp
kafka_kafka2_1 /etc/confluent/docker/run Up 9092/tcp, 0.0.0.0:9093->9093/tcp,:::9093->9093/tcp
kafka_kafka3_1 /etc/confluent/docker/run Up 9092/tcp, 0.0.0.0:9094->9094/tcp,:::9094->9094/tcp
kafka_zookeep1_1 /docker-entrypoint.sh zkSe ... Up 0.0.0.0:2181->2181/tcp,:::2181->2181/tcp, 2888/tcp, 3888/tcp
kafka_zookeep2_1 /docker-entrypoint.sh zkSe ... Up 2181/tcp, 0.0.0.0:2182->2182/tcp,:::2182->2182/tcp, 2888/tcp, 3888/tcp
kafka_zookeep3_1 /docker-entrypoint.sh zkSe ... Up 2181/tcp, 0.0.0.0:2183->2183/tcp,:::2183->2183/tcp, 2888/tcp, 3888/tcp
root@shastry:/opt/kafka#
7. Create kafka topics
- Create new topic with topicName “test” with 3 partitions and 3 replication-factor
root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-topics.sh --bootstrap-server localhost:9092 --topic test --create --partitions 3 --replication-factor 3
Created topic test.
root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-topics.sh --bootstrap-server localhost:9092 --topic test1 --create --partitions 3 --replication-factor 3
Created topic test1.
root@shastry:/opt/kafka/kafka_cli/bin#
- List down the topics
root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-topics.sh --bootstrap-server localhost:9092 --list
test
test1
root@shastry:/opt/kafka/kafka_cli/bin#
- Describe a specific topic
root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-topics.sh --bootstrap-server localhost:9092 --topic test1 --describe
Topic: test1 TopicId: Lp-rUh0bTi22G_5pbSzTAA PartitionCount: 3 ReplicationFactor: 3 Configs:
Topic: test1 Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: test1 Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: test1 Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
root@shastry:/opt/kafka/kafka_cli/bin#
8. Produce and consume messages
- Produce few messages
root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test1
>Hello
>from
>Bharath
>
- Consume messages from beginning.
root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1 --from-beginning
Hello
from
Bharath
Reference Links
https://learn.microsoft.com/en-us/windows/wsl/install
http://kafka.apache.org/documentation.html
https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html
https://www.conduktor.io/kafka/how-to-start-kafka-using-docker/
https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk
