Lag exporter cannot get list of consumer groups

[AlexeyRaga]

Describe the bug

Please help me to understand and workaround this issue! :)

I am trying to configure kafka-lag-exporter to monitor AWS MSK using IAM authentication.
The service runs, and IAM auth works (I can tell because when I don't give role permissions it complains), but then the lag exporter discovers no consumer groups or topics.

I see that there is a strange timeout that happens almost immediately after the request starts (according to the log timestamp) despite very same logs saying that the timeout is supposed to be 10000ms. But I may be mis-interpreting logs, and I see that it then retries without an error...

To Reproduce
Kafka cluster: AWS MSK + IAM Authentication

Config options:

env_vars = {
  "KAFKA_LAG_EXPORTER_CLUSTERS.0.name"              = aws_msk_cluster.kafka_cluster.cluster_name
  "KAFKA_LAG_EXPORTER_CLUSTERS.0.bootstrap-brokers" = aws_msk_cluster.kafka_cluster.bootstrap_brokers_sasl_iam

  "KAFKA_LAG_EXPORTER_CLUSTERS.0.admin-client-properties.security.protocol"                  = "SASL_SSL"
  "KAFKA_LAG_EXPORTER_CLUSTERS.0.admin-client-properties.sasl.mechanism"                     = "AWS_MSK_IAM"
  "KAFKA_LAG_EXPORTER_CLUSTERS.0.admin-client-properties.sasl.jaas.config"                   = "software.amazon.msk.auth.iam.IAMLoginModule required;"
  "KAFKA_LAG_EXPORTER_CLUSTERS.0.admin-client-properties.sasl.client.callback.handler.class" = "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

  "KAFKA_LAG_EXPORTER_CLUSTERS.0.consumer-properties.security.protocol"                  = "SASL_SSL"
  "KAFKA_LAG_EXPORTER_CLUSTERS.0.consumer-properties.sasl.mechanism"                     = "AWS_MSK_IAM"
  "KAFKA_LAG_EXPORTER_CLUSTERS.0.consumer-properties.sasl.jaas.config"                   = "software.amazon.msk.auth.iam.IAMLoginModule required;"
  "KAFKA_LAG_EXPORTER_CLUSTERS.0.consumer-properties.sasl.client.callback.handler.class" = "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

  "ROOT_LOG_LEVEL" = "DEBUG"
  "KAFKA_LAG_EXPORTER_LOG_LEVEL" = "DEBUG"
  "KAFKA_LAG_EXPORTER_KAFKA_LOG_LEVEL" = "DEBUG"
}

Here is the policy that I use to give access to MSK:

data "aws_iam_policy_document" "kafka_lag_exporter" {
  statement {
    actions = [
      "kafka-cluster:*"
    ]

    resources = [
      aws_msk_cluster.kafka_cluster.arn,
      "${aws_msk_cluster.kafka_cluster.arn}/*"
    ]
  }
}

Please find DEBUG logs attached: kafka-log-exporter-log.txt

Environment

• Version: seglo/kafka-lag-exporter:0.8.2
• Version of Apache Kafka cluster: MSK, Kafka 3.3.1
• Run with Helm or Standalone: Standalone, as a docker container on ECS