Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-285] Added docker images for spark-sedona-zeppelin and spark-sedona-jupyterlab #843

Closed
wants to merge 8 commits into from

Conversation

kartikeyhadiya
Copy link
Contributor

@kartikeyhadiya kartikeyhadiya commented May 26, 2023

Added docker images for spark-sedona:1.3.1-zeppelin and spark-sedona1.4.0-jupyterlab

Issue: #842

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

Added docker images for spark-sedona:1.3.1-zeppelin and spark-sedona1.4.0-jupyterlab

How was this patch tested?

Local environment with docker.

Did this PR include necessary documentation updates?

  • Yes, I have added readmes in respective folders.

@jiayuasu
Copy link
Member

@kartikeyhadiya Thanks for your contribution. A few comments:

  1. Since this is a major contribution, please create a JIRA ticket first.
  2. Please do not duplicate any data files in the source repo. The source repo already has a couple of data files in binder. Please try to reuse those via Dockerfile.
  3. Please do not directly copy jars to the source repo. Please download jars on the fly via Dockerfile.
  4. All source code such as .sh script must have Apache License as the file header.
  5. Once everything is done, you will need to add the docker compilation to GitHub Action with a Yaml file.

@kartikeyhadiya
Copy link
Contributor Author

kartikeyhadiya commented May 27, 2023

Thanks for your input.
For 2 & 3 point i just thought that if anyone wants to replicate they can just download the docker folder and start working but, I will make the necessary changes.
Can you please elaborate on 5 point? I have never used github actions or any CICD, are you referring something like this https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action or https://josephrodriguezg.medium.com/build-and-publish-docker-images-with-github-actions-78be3b3fbb9b ?

@kartikeyhadiya
Copy link
Contributor Author

I have created jira ticket and included it in main comment

@jiayuasu jiayuasu changed the title Added docker images for spark-sedona:1.3.1-zeppelin and spark-sedona1.4.0-jupyterlab [SEDONA-285] Added docker images for spark-sedona-zeppelin and spark-sedona-jupyterlab May 27, 2023
@jiayuasu
Copy link
Member

kartikeyhadiya and others added 2 commits May 29, 2023 11:13
Hadiya Kartikey and others added 5 commits May 29, 2023 15:46
… License as file header to .dockerfiles and .sh script
…pelin and jupyterlab. By default spark 3.3.2 version used but can be changed with --build-args. Added workflow for pushing image to dockerhub
Updated dokcerhub-deploy.yml workflow
@kartikeyhadiya
Copy link
Contributor Author

kartikeyhadiya commented Jun 22, 2023

Hey, @jiayuasu sorry for so much delay had a busy month at work.

  • I have made all the changes you proposed, please can you have a look. I didn't make github action for zeppelin docker file as I think if zeppelin is not updated sooner or later it will be of no use, still I have kept a dockerfile for that.
  • For publishing to dockerhub, please create 2 secrets named "DOCKERHUB_USERNAME" and "DOCKERHUB_PASSWORD". For now I have kept my repository as a placeholder, please let me know the repository to which I need to publish so I can make that changes.

Please verify and let me know if any other suggestions/changes required.

@kartikeyhadiya
Copy link
Contributor Author

Hey, @jiayuasu just following up on this.

@jiayuasu
Copy link
Member

@kartikeyhadiya Thanks for the great work here. I will take a look your fork and try to run it myself.

@kartikeyhadiya
Copy link
Contributor Author

Thank you, please let me know if anything else is required.

@jiayuasu
Copy link
Member

jiayuasu commented Aug 2, 2023

@kartikeyhadiya Due to the complexity of this task, I have taken over this and created another PR based on this PR: #939 Thanks for your help!

@jiayuasu jiayuasu closed this Aug 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants