Jaeger and Kafka with Docker Compose

[theprobugmaker]

I can't make Jaeger connect to the Kafka cluster. That is my docker-compose, what am I doing wrong?

version: "3"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    networks:
      - kafka-net
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    hostname: kafka
    container_name: kafka
    networks:
      - kafka-net
    depends_on:
      - zookeeper
    environment:
      # KAFKA_BROKER_ID: 1
      # KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      # KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      # KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      # KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      # KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      # KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      ALLOW_PLAINTEXT_LISTENER: "yes"
      KAFKA_LISTENERS-INTERNAL: //kafka:29092,EXTERNAL://localhost:9092
      KAFKA_ADVERTISED: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    ports:
      - 9092:9092
      - 29092:29092

  jaeger:
    image: jaegertracing/all-in-one:1.26
    hostname: jaeger
    container_name: jaeger
    networks:
      - kafka-net
    environment:
      SPAN_STORAGE_TYPE: kafka
      KAFKA_BROKERS: kafka:9092
      KAFKA_PRODUCER_BROKERS: kafka:29092
      KAFKA_TOPIC: jaegerspans
    ports:
      - 5775:5775/udp
      - 6831:6831/udp
      - 6832:6832/udp
      - 5778:5778
      - 16686:16686
      - 14268:14268
      - 9411:9411

networks:
  kafka-net:
    driver: bridge

[joe-elliott]

Any errors in the logs?

I don't think SPAN_STORAGE_TYPE=kafka for the all in one works. The only supported backend storage options are cassandra and elastic. kafka is usable as a queue between collector -> ingester and is only a valid SPAN_STORAGE_TYPE for the collector in that set up.

[theprobugmaker]

Thank you for the answer @joe-elliott, I tried to change like this (I was having the same error when using all-in-one):

  kafka:
    image: confluentinc/cp-kafka:latest
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    ports:
      - 9092:9092
      
  agent:
    image: jaegertracing/jaeger-agent:1.26
    hostname: agent
    container_name: agent
    depends_on:
      - collector
    ports:
      - 1111:6831/udp
    command: ["--reporter.grpc.host-port=collector1:14250", "--log-level=debug"]

  collector:
    image: jaegertracing/jaeger-collector:1.26
    hostname: collector
    container_name: collector
    depends_on:
      - kafka
    environment:
      SPAN_STORAGE_TYPE: "kafka"
    ports:
      - 14250
      - 14268:14268
    command:
      [
        "--kafka.producer.brokers=kafka:9092",
        "--kafka.producer.topic=jaegerspans",
        "--log-level=debug",
      ]

But I'm still getting this error:

{"level":"info","ts":1632917859.2473636,"caller":"kafka/factory.go:69","msg":"Kafka factory","producer builder":{"Brokers":["kafka:9092"],"RequiredAcks":1,"Compression":0,"CompressionLevel":0,"ProtocolVersion":"","BatchLinger":0,"BatchSize":0,"BatchMinMessages":0,"BatchMaxMessages":0,"Authentication":"none","Kerberos":{"ServiceName":"kafka","Realm":"","UseKeyTab":false,"Username":"","ConfigPath":"/etc/krb5.conf","KeyTabPath":"/etc/security/kafka.keytab"},"TLS":{"Enabled":false,"CAPath":"","CertPath":"","KeyPath":"","ServerName":"","ClientCAPath":"","SkipHostVerify":false},"PlainText":{"Username":"","Mechanism":"PLAIN"}},"topic":"jaegerspans"}
{"level":"fatal","ts":1632917860.193468,"caller":"./main.go:76","msg":"Failed to init storage factory","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)","stacktrace":"main.main.func1\n\t./main.go:76\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/[email protected]/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/[email protected]/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/[email protected]/command.go:902\nmain.main\n\t./main.go:138\nruntime.main\n\truntime/proc.go:255"}

[joe-elliott]

Do some basic network debugging between your collector container and kafka. Confirm the port is reachable as expected on the provided hostname.

This looks suspicious to me:

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092

listenting on http://localhost:9092 often means it will only accept connections from localhost on that port.

[theprobugmaker]

@joe-elliott Yes it does looks suspicious. The weird part is that I have a Kafka Connect service as well and it's using that same port (to connect) and is working. I will try to look for another example where this host isn't using localhost and see if it works.

[theprobugmaker]

I just went back looking and the Kafka Connect is actually working cause it's NOT using that port, meaning that it's probably the problem. 😨

[theprobugmaker]

It worked 🎉The problem is what you pointed out, I'm not supposed to be using 9092 as it's a localhost port but instead 29092 that is a port for the kafka DNS.
All of the examples I found online was using the localhost port and that is why I couldn't figure out the problem. :(