Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

packaging: Integrate github workflows to automate the air-gapping process of wave university #2181

Merged
merged 28 commits into from
Feb 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
f94114d
packaging: Initial workflow files
sulhicader Oct 29, 2023
33c3f71
packaging: changed the workflow tigger on #2163
sulhicader Oct 29, 2023
f0c6a4f
packaging: changed the workflow file names on #2163
sulhicader Oct 29, 2023
3a00254
packaging: Added to set version in app.toml on #2163
sulhicader Nov 16, 2023
5b64c3f
packaging: Added workflows from workflow-library on #2163
sulhicader Nov 16, 2023
c817498
packaging: Changed workflow trigger mode to push on #2163
sulhicader Nov 16, 2023
cd63621
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
2c6f361
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
bc9991d
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
5ee8dbf
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
e6b32d2
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
fd3610a
packaging: Added workflow directory to each step on #2163
sulhicader Nov 17, 2023
acad043
packaging: Added workflow directory to each step on #2163
sulhicader Nov 17, 2023
51a91b6
packaging: Added workflow directory to each step on #2163
sulhicader Nov 17, 2023
cdc11bc
Attached new flows to publish-university
sulhicader Nov 20, 2023
66dbde3
packaging: Attached new flows to publish-university on #2163
sulhicader Nov 20, 2023
ca86f56
packaging: Reverted some changes on #2163
sulhicader Nov 20, 2023
f1407f3
Merge remote-tracking branch 'origin/main' into feat/issue-2163
sulhicader Nov 20, 2023
dc9a1c2
packaging: Reverted some changes on #2163
sulhicader Nov 20, 2023
4ad597c
packaging: Removed generated files on #2163
sulhicader Nov 20, 2023
a2a07a6
Merge remote-tracking branch 'origin/main' into feat/issue-2163
sulhicader Jan 31, 2024
b43ef51
Handled review comments #2163
sulhicader Jan 31, 2024
72a07bd
Handled review comments #2163
sulhicader Jan 31, 2024
41bf0f7
Handled review comments #2163
sulhicader Jan 31, 2024
1ad2d02
Finalized workflow for university #2163
sulhicader Jan 31, 2024
3b2c8be
Changed the base image version
sulhicader Jan 31, 2024
f579812
Merge branch 'main' into feat/issue-2163
sulhicader Feb 1, 2024
1179b62
Merge branch 'main' into feat/issue-2163
mturoci Feb 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 65 additions & 1 deletion .github/workflows/publish-university.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,73 @@ jobs:

- name: Build university
run: make publish-university

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@master
with:
packages_dir: university/dist
password: ${{ secrets.PYPI_UNIVERSITY_TOKEN }}

create-bundle:
name: Create Wave Bundle
runs-on: ubuntu-latest

outputs:
build-version: ${{ env.VERSION }}

steps:
- uses: actions/checkout@v3

- name: Download H2O CLI
working-directory: ./university
run: mkdir -p .bin && curl -o .bin/h2o https://h2oai-cloud-release.s3.amazonaws.com/releases/ai/h2o/h2o-cloud/latest/cli/linux-amd64/h2o

- name: Change permissions
working-directory: ./university
run: chmod +x .bin/h2o

- name: Set version
working-directory: ./university
run: sed -i -r -e "s/\{\{VERSION\}\}/${{ env.VERSION }}/g" app.toml

- name: Get App Version
id: get-build-version
run: |
echo "VERSION=${{ env.VERSION }}" >> "$GITHUB_OUTPUT"

- name: Make air-gapped bundle
working-directory: ./university
run: |
.bin/h2o bundle \
--docker-base-image 524466471676.dkr.ecr.us-east-1.amazonaws.com/q8s/launcher:v0.31.0-310 \
--docker-use-buildkit \
--generate-helm-charts \
--helm-chart-version ${{ env.VERSION }} \
--helm-chart-name university \
--helm-app-bundle-image-repo 524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/university-bundle \
--helm-app-runtime-image-repo 524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/university \
--generate-dockerfile

- uses: actions/upload-artifact@v3
with:
name: wave-bundle
path: |
./university/*.Dockerfile
./university/*.wave

- uses: actions/upload-artifact@v3
with:
name: wave-bundle-helm
path: university/helm/

build-and-publish:
needs: create-bundle
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout

uses: ./.github/workflows/wave-bundle-docker-build-publish.yaml
with:
build-version: ${{ needs.create-bundle.outputs.build-version }}
bundle-artifact: wave-bundle
wave-app-name: university
114 changes: 114 additions & 0 deletions .github/workflows/wave-bundle-docker-build-publish.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
name: Build and Publish Docker image from generated Dockerfile

on:
workflow_call:
inputs:
build-version:
type: string
description: The version of the application/image to be pushed
required: true
bundle-artifact:
type: string
description: The name of the artifact containing the generated Dockerfiles and the wave bundle
required: true
working-directory:
type: string
description: Path to the working directory, where docker build will be executed
default: .
wave-app-name:
type: string
description: The name of the wave app
required: true

jobs:
docker:
name: Build and Publish App Docker Image
runs-on: ubuntu-latest
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout

steps:
- uses: actions/checkout@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Download Wave bundle and Dockerfiles
uses: actions/download-artifact@v3
with:
name: ${{ inputs.bundle-artifact }}
path: ./

- name: Rename Dockerfiles
run: |
mv ./*bundle.Dockerfile ./generated.bundle.Dockerfile
mv ./*runtime.Dockerfile ./generated.runtime.Dockerfile

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v2
with:
role-to-assume: arn:aws:iam::524466471676:role/workflows-library-wave-bundling-github-actions
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: us-east-1

- name: "Login to Amazon ECR"
uses: aws-actions/amazon-ecr-login@v1

- name: "Ensure ECR Repository for the bundle"
# Tries to describe the repository and if it fails, creates it
run: |
aws ecr describe-repositories \
--repository-names h2oai/${{ inputs.wave-app-name }}-bundle 2>/dev/null ||
aws ecr create-repository \
--repository-name h2oai/${{ inputs.wave-app-name }}-bundle \
--image-tag-mutability IMMUTABLE \
--tag \
Key=GithubRepo,Value=github.com/h2oai/${{ inputs.wave-app-name }} \
Key=ManagedBy,Value=GitHubActions \
Key=CreatedByWorkflow,Value=${{ github.workflow_ref }}

- name: "Ensure ECR Repository for the runtime"
# Tries to describe the repository and if it fails, creates it
run: |
aws ecr describe-repositories \
--repository-names h2oai/${{ inputs.wave-app-name }} 2>/dev/null ||
aws ecr create-repository \
--repository-name h2oai/${{ inputs.wave-app-name }} \
--image-tag-mutability IMMUTABLE \
--tag \
Key=GithubRepo,Value=github.com/h2oai/${{ inputs.wave-app-name }} \
Key=ManagedBy,Value=GitHubActions \
Key=CreatedByWorkflow,Value=${{ github.workflow_ref }}

- name: Build and Export Bundle Image
uses: docker/build-push-action@v4
id: bundle-build
with:
push: true
context: ${{ inputs.working-directory }}
file: ./generated.bundle.Dockerfile
platforms: linux/amd64
provenance: false
tags: |
524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/${{ inputs.wave-app-name }}-bundle:${{ inputs.build-version }}

- name: Build and Export Runtime Image
uses: docker/build-push-action@v4
id: runtime-build
with:
push: true
context: ${{ inputs.working-directory }}
file: ./generated.runtime.Dockerfile
platforms: linux/amd64
provenance: false
tags: |
524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/${{ inputs.wave-app-name }}:${{ inputs.build-version }}

- name: Published Images Summary
run: |
echo "#### Image Tags" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
echo "docker image push 524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/${{ inputs.wave-app-name }}-bundle:${{ inputs.build-version }}" >> $GITHUB_STEP_SUMMARY
echo "docker image push 524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/${{ inputs.wave-app-name }}:${{ inputs.build-version }}" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
Loading