External clients may face connection issues when setting up a Kafka broker using Docker Compose. This guide explores common problems and solutions to enable successful communication between external clients and Kafka brokers within a Docker-Compose setup.

Quick FIX

 You need to set KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so clients can connect to it correctly. Otherwise, they’ll try to connect to the internal host address—and if that’s not reachable, then problems ensue.

Let's list down the KAFKA_LISTENERS in a typical docker-compose environment 

KAFKA_LISTENERS:  (what interfaces Kafka binds to)

  LISTENER_INTERNAL = kafka listener inside docker-compose environment

  LISTENER_LOCAL = kafka listener for local or from the host machine where the docker-compose is being executed

  LISTENER_EXTERNAL = kafka listerner for clients connecting from outside host computer or outside cluster network

Let's list down the corresponding KAFKA_ADVERTISED_LISTENERS of the above KAFKA_LISTENERS

 KAFKA_ADVERTISED_LISTENERS: (how clients can connect)

 LISTENER_INTERNAL = you dont want your client to connect in this listener as this will be used internally by kafka

  LISTENER_LOCAL = this are your client that will only connect from your host machine where docker-compose was executed

  LISTENER_EXTERNAL = this are the clients that will connect outside host computer or outside network

 

Sample Configuration

  kafka0:
    image: "confluentinc/cp-enterprise-kafka:5.2.1"
    ports:
      - '9092:9092'
      - '29094:29094'
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 0
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: LISTENER_INTERNAL://kafka0:29092,LISTENER_LOCAL://kafka0:9092,LISTENER_EXTERNAL://kafka0:29094
      KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka0:29092,LISTENER_LOCAL://localhost:9092,LISTENER_EXTERNAL://never-gonna-give-you-up:29094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_LOCAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100

Note:

1. kafka0 - docker-compose service name 
2. 9092 = exposed ports for local clients
3. 29094 = exposed ports for external clients
4. 29092 = not exposed, and been defined from the KAFKA_LISTENERS: LISTENER_INTERNAL

Behind the scene: 

1. When a client connects to Kafka broker,
   1. How Kafka is resolving the configuration between KAFKA_ADVERTISED_LISTENERS and KAFKA_LISTENERS 

     Its in the PORT where your client specified, 

When kafka received the request it will use the port and bind it to corresponding LISTENERS

If the client used UN ADVERTISED_LISTENER thogh it can connect to the docker instance, Kafka is bouncing it off

Plist the LISTENER_EXTERNAL host / IP must be rechable from external host or network by the client