tl;dr – A simple port forward from the container to the host will not work, and no hosts files should be modified. What exact IP/hostname + port do you want to connect to? Make sure that value is set as
advertised.listenerson the broker. Make sure that address and the servers listed as part of
bootstrap.serversare actually resolvable (
pingan IP/hostname, use
netcatto check ports…)
Below gives answers for commonly used Kafka images, but it’s all the same Apache Kafka running in a container.
You’re just dependent on how it is configured. And which variables make it so.
The below answer uses
confluentincdocker images to address the question that was asked, not
wurstmeister/kafka. More specifically, the latter images are not well-maintained despite being the one of the most popular Kafka docker image.
The following sections try to aggregate all the details needed to use another image.
Refer their README section on listener configuration, Also read their Connectivity wiki.
If you want a small container, try these. The images are much smaller than the Confluent ones and are much more well maintained than
wurstmeister. Refer their README for listener configuration.
Docs on it are mentioned here.
Note: advertised host and port settings are deprecated. Advertised listeners covers both. Similar to the Confluent containers, Debezium can use
KAFKA_prefixed broker settings to update its properties.
spotify/kafkais deprecated and outdated.
lensesio/boxare great for an all in one solution, but are bloated if you only want Kafka
Dockerfile– Why? Is something incomplete with these others? Start with a pull request, not starting from scratch.
For supplemental reading, a fully-functional
docker-compose, and network diagrams, see this blog by @rmoff
The Confluent quickstart (Docker) document assumes all produce and consume requests will be within the Docker network.
You could fix the problem of connecting to
kafka:9092 by running your Kafka client code within its own container as that uses the Docker network bridge, but otherwise you’ll need to add some more environment variables for exposing the container externally, while still having it work within the Docker network.
First add a protocol mapping of
PLAINTEXT_HOST:PLAINTEXT that will map the listener protocol to a Kafka protocol
Then setup two advertised listeners on different ports. (
kafka here refers to the docker container name; it might also be named
broker, so double check your service + hostnames). Notice the protocols match the right side values of the mappings above
When running the container, add
-p 29092:29092 for the host port mapping
(with the above settings)
And if something still doesn’t work,
KAFKA_LISTENERS can be set to include
<PROTOCOL>://0.0.0.0:<PORT> where both options match the advertised setting and Docker-forwarded port
Client on same machine, not in a container
Advertising localhost and the associated port will let you connect outside of the container, as you’d expect.
In other words, when running any Kafka Client outside the Docker network (including CLI tools you might have installed locally), use
localhost:29092 for bootstrap servers and
localhost:2181 for Zookeeper (requires Docker port forwarding)
Client on another machine
If trying to connect from an external server, you’ll need to advertise the external hostname/ip of the host as well as/in place of localhost.
Simply advertising localhost with a port forward will not work because Kafka protocol will still continue to advertise the listeners you’ve configured.
This setup requires Docker port forwarding and router port forwarding (and firewall / security group changes) if not in the same local network, for example, your container is running in the cloud and you want to interact with it from your local machine.
Client in a container, on the same host
This is the least error-prone configuration; you can use DNS service names directly.
When running an app in the Docker network, use
kafka:9092 (see advertised
PLAINTEXT listener config above) for bootstrap servers and
zookeeper:2181 for Zookeeper, just like any other Docker service communication (doesn’t require any port forwarding)
If you use separate
docker run commands, or Compose files, you need to define a shared
See the example Compose file for the full Confluent stack or more minimal one for a single broker.
Connect to Kafka on host from Docker (ksqlDB)
For anyone interested in Kubernetes deployments: https://operatorhub.io/?keyword=Kafka