I'm trying to get one of my services up and running locally within the docker network, but keep getting "Consumer failed to start: Connection refused - connect(2) for 127.0.0.1:9092" exception.

zookeeper: image: wurstmeister/zookeeper:3.4.6 restart: always expose: - "2181"
kafka: depends_on: - zookeeper environment: KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE hostname: kafka image: wurstmeister/kafka:2.11-2.0.0 restart: always ports: - "9092:9092" expose: - "9093" links: - zookeeper:zookeeper

That's what I've set in my service's docker-compose for the consumers:

 consumers: environment: - BROKER_URL=host.docker.internal:9092

That's the error message:

..`block in execute'","/usr/local/bundle/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'"],
"message":"Consumer failed to start: Connection refused - connect(2) for 127.0.0.1:9092"},
"message":"Consumer failed to start: Connection refused - connect(2) for 127.0.0.1:9092",
"@timestamp":"2020-05-24T18:08:09.434+00:00","@version":"1",
"severity":"ERROR","host":"87c5265c6a39"}

UPDATE: When checking Kafkacat:

kafkacat -L -b 127.0.0.1:9092
Metadata for all topics (from broker 0: 127.0.0.1:9092/0): 1 brokers: broker 0 at 127.0.0.1:9092 (controller) 7 topics: ...

When checking Zookeeper CLI (within Docker):

[zk: localhost:2181(CONNECTED) 2] ls /brokers/ids
[1001]

I've tried already pretty much everything I could find, but still not able to understand what am I missing.

Any help would be appreciated, thanks!

3

1 Answer

You’re connecting to the wrong listener. When you connect to 9092 the broker returns localhost for your client to subsequently connect to. Connect to 9093 and it will use kafka which your consumer in the Docker network should be able to resolve.

Ref:

1