Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with protobuf deserializer #291

Open
Welkin-Y opened this issue Sep 28, 2023 · 1 comment
Open

Problem with protobuf deserializer #291

Welkin-Y opened this issue Sep 28, 2023 · 1 comment

Comments

@Welkin-Y
Copy link

Problem description
Hello, I am currently working on the flink-connector project trying to add protobuf support. I've noticed that after serializing a message to a ByteBuffer, if I convert it to a byte[] using .array(), and then convert it back to a ByteBuffer using ByteBuffer.wrap(), there will be an exception when passing it to the deserialize function. I noticed that in the test cases, there are no scenarios covering conversions between ByteBuffers and byte arrays. However, such a convertion is needed for flink-connector interfaces.

Problem location

To reproduce the exception, you can modify this file by changing line 84 as follows:, you can regenerate the exception by changing line 84 as follows:

c80e7a2f340e1b7d45c1a848cdf5c19

The error message is like this:

768133634d1465c845f564f4843f7e1

Suggestions for an improvement
I'm not sure if it's related to the deserialize method. From my experiments, I found that the issue arises because, after converting a ByteBuffer to a byte[] and back, the limit information of the ByteBuffer is lost, leading to a parsing error in protobuf.

@crazyzhou
Copy link
Contributor

@Welkin-Y When parsing bytebuffer into byte[], do you have the information of the limit? Directly using array() may lose such infomation, there are some usages like: https://github.com/pravega/flink-connectors/blob/053269788245cccd53257cd70178097d4b61bf6c/src/main/java/io/pravega/connectors/flink/util/FlinkPravegaUtils.java#L204 Maybe you can have a try.

@shshashwat Please also help to see if there are anything can be improvement from this side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants