Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: fix kafka transport default partitioning #246

Merged
merged 2 commits into from
Dec 6, 2023
Merged

bug: fix kafka transport default partitioning #246

merged 2 commits into from
Dec 6, 2023

Conversation

lspgn
Copy link
Member

@lspgn lspgn commented Dec 4, 2023

  • set default partitioner to roundrobin

* set default partitioner to roundrobin
@lspgn lspgn added the transport Related to transport libraries (Kafka) label Dec 5, 2023
@simPod
Copy link
Contributor

simPod commented Dec 5, 2023

I have tried to deploy the binary built from this PR but it seems to have no effect 🤔 Messages are still produced to only a single partition. I don't use -mapping so I suppose the key is nil.

image

@simPod
Copy link
Contributor

simPod commented Dec 5, 2023

I can see it is being produced with a key
image

@simPod
Copy link
Contributor

simPod commented Dec 5, 2023

I have tried a build where I removed this line

Key: sarama.ByteEncoder(key),
and now it's distributed randomly

image

Unfortunately, I have no way to debug where is the key being created at the moment. I've requested tcpdump installation on the server where the sflow is being sent to so I can capture some packets, create sflow fixtures for goflow and debug it.

@lspgn
Copy link
Member Author

lspgn commented Dec 6, 2023

Could you check again @simPod?

I think I found the problem: empty key had a fixed hash which didn't work with the RoundRobinPartitioner.

I'm observing it working with the expected behavior even when key is fixed. Make sure -transport.kafka.hashing=false

For reference in Bitnami Kafka docker:

Create a topic with 9 partitions

$ /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic flow-messages --partitions 9

Get the offsets

$ /opt/bitnami/kafka/bin/kafka-get-offsets.sh --topic flow-messages --bootstrap-server localhost:9092
flow-messages:0:1506
flow-messages:1:1506
flow-messages:2:4009
flow-messages:3:1506
flow-messages:4:1506
flow-messages:5:1506
flow-messages:6:8212
flow-messages:7:1506
flow-messages:8:1504

@simPod
Copy link
Contributor

simPod commented Dec 6, 2023

I can confirm it works without enabling -transport.kafka.hashing. This can be now closed I guess #248

@lspgn lspgn merged commit 5ba0210 into main Dec 6, 2023
1 check passed
@lspgn lspgn deleted the bug/kafka branch December 6, 2023 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
transport Related to transport libraries (Kafka)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants