Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CDAP-21027 : Upgrading hadoop version to 3.3.6 #1871

Merged
merged 1 commit into from
Jul 26, 2024
Merged

CDAP-21027 : Upgrading hadoop version to 3.3.6 #1871

merged 1 commit into from
Jul 26, 2024

Conversation

sahusanket
Copy link
Contributor

@sahusanket sahusanket commented Jun 5, 2024 •
edited
Loading

  • Upgrading to Hadoop 3
  • This makes current HBASE and CASSANDRA incompatible.
  • Upgrading them has Major Compilation Changes, which requires larger effort to revaluate and retest the plugins
  • Created https://cdap.atlassian.net/browse/CDAP-21051 and https://cdap.atlassian.net/browse/CDAP-21052 to track the re-evaluation or depreciation process.
  • Removed Hbase and Cassandra from the BUILD process.
  • Still removed / upgraded hbase and cassandra libs related to vulnerabilities for safety. ( it is in broken state)

Testing :

  • testing of some core plugins like Deduplicate, record duplicator, joiner , avro , parquet are done on Sandbox and PROD Image.

@sahusanket sahusanket added the build Trigger unit test build label Jun 5, 2024
@sahusanket sahusanket self-assigned this Jun 5, 2024
@sahusanket sahusanket force-pushed the CDAP-21027_hadoop_upgrade branch 2 times, most recently from 7098661 to b4d8381 Compare June 6, 2024 11:44
This was referenced Jul 22, 2024
Merged
Merged
@sahusanket sahusanket requested a review from tivv July 26, 2024 08:20
@sahusanket sahusanket changed the title Hadoop Upgrade core plugins - 1 CDAP-21027 : Upgrading hadoop version to 3.3.6 Jul 26, 2024
tivv
tivv approved these changes Jul 26, 2024
View reviewed changes
Copy link
Contributor

@tivv tivv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with minor comments

pom.xml Outdated
@@ -29,7 +29,7 @@
<modules>
<module>hydrator-common</module>
<module>core-plugins</module>
<module>cassandra-plugins</module>
<!-- <module>cassandra-plugins</module> TO EVALUATE https://cdap.atlassian.net/browse/CDAP-21051 -->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please mark it as TODO. It's easier to track as TODO is picked up by many tools, including IDE

@sahusanket sahusanket merged commit efcf4d0 into develop Jul 26, 2024
5 checks passed
@sahusanket sahusanket deleted the CDAP-21027_hadoop_upgrade branch July 26, 2024 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tivv tivv tivv approved these changes

Assignees

@sahusanket sahusanket

Labels
build Trigger unit test build
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sahusanket @tivv