Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sedona-spark-shaded package should exclude some repetitive dependencies. #1584

Open
freamdx opened this issue Sep 12, 2024 · 2 comments
Open

Comments

@freamdx
Copy link

freamdx commented Sep 12, 2024

pom.xml should exclude any dependencies that exist in spark jars, eg:
edu.ucar:cdm-core exclude guava/httpclient/protobuf-java
... ... <dependency> <groupId>edu.ucar</groupId> <artifactId>cdm-core</artifactId> <exclusions> <exclusion> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> </exclusion> <exclusion> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> </exclusion> <exclusion> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> </exclusion> </exclusions> </dependency> ... ... <artifactSet> <excludes> <exclude>org.scala-lang:scala-library</exclude> <exclude>org.apache.commons:commons-*</exclude> <exclude>commons-pool:commons-pool</exclude> <exclude>commons-lang:commons-lang</exclude> <exclude>commons-io:commons-io</exclude> <exclude>commons-logging:commons-logging</exclude> </excludes> </artifactSet> ... ...

@jiayuasu
Copy link
Member

@freamdx would you mind create a PR to fix this yourself?

@Kontinuation
Copy link
Member

Kontinuation commented Sep 13, 2024

A few notes:

  • cdm-core has to be bundled in the shaded package, we include it in the shaded pom.xml to make it compile rather than provided.
  • guava is better shaded than excluded, since we don't know whether the guava jar shipped with spark is compatible with sedona or not, and shading guava is a common practice.
  • other package exclusion seems to be OK. Apache commons has good backward compatibility and the versions shipped with spark is later than what we bundled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants