Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove kafka printer #995

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

gregschohn
Copy link
Collaborator

@gregschohn gregschohn commented Sep 22, 2024

Description

Switch out the file format for InputStreamOfTraffic streams and remove KafkaPrinter.
It was protobuf delimited and now it's Base64 w/ no line wraps with each record being on its own line. Other patches will be forthcoming to generate those files from the migration console.
Change the kafkaExport.sh script to leverage kafka-console-consumer directly.

  • Category Refactoring
  • Why these changes are required? To speed up builds and to remove code that was hard to test
  • What is the old behavior before changes and new behavior after changes? kafkaExport.sh will no longer delete groups created automatically during an export, but there are more options for exports. The file format has also changed, but it should be easier to do quick copy-pastes since the new format is base64.

Issues Resolved

https://opensearch.atlassian.net/browse/MIGRATIONS-2018

Testing

gradle and manual testing.
Manual testing - created exports, unzipped them and reran them through the replayer.

Check List

  • New functionality includes testing
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

It was protobuf delimited and now it's Base64 w/ no line wraps with each record being on its own line.  Other patches will be forthcoming to generate those files from the migration console.

Signed-off-by: Greg Schohn <[email protected]>
Copy link

codecov bot commented Sep 22, 2024

Codecov Report

Attention: Patch coverage is 38.88889% with 11 lines in your changes missing coverage. Please review.

Project coverage is 79.72%. Comparing base (d99beb6) to head (90aa6e7).

Files with missing lines Patch % Lines
...ns/replay/traffic/source/InputStreamOfTraffic.java 50.00% 7 Missing ⚠️
...search/migrations/utils/kafka/Base64Formatter.java 0.00% 4 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main     #995      +/-   ##
============================================
+ Coverage     78.98%   79.72%   +0.73%     
+ Complexity     2723     2715       -8     
============================================
  Files           375      375              
  Lines         13869    13692     -177     
  Branches        970      942      -28     
============================================
- Hits          10955    10916      -39     
+ Misses         2340     2205     -135     
+ Partials        574      571       -3     
Flag Coverage Δ
gradle-test 77.65% <38.88%> (+0.85%) ⬆️
python-test 89.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant