Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only upload builder artifacts from workflow runs triggered after merge #18

Merged
merged 43 commits into from
Feb 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
6023e39
only upload artifacts on pr merge
James-Mart Jan 12, 2024
a4b26f9
remove release branch triggers
James-Mart Jan 12, 2024
2aada96
base psibase-contributor image on temporary image during PRs
James-Mart Feb 4, 2024
9b52d88
list available images
James-Mart Feb 4, 2024
bee9325
try pushing to local registry
James-Mart Feb 4, 2024
56370e4
try changing docker host
James-Mart Feb 4, 2024
4316c2b
trying to see image in local registry
James-Mart Feb 4, 2024
e480935
pull image before save to archive
James-Mart Feb 5, 2024
b059dca
attempt
James-Mart Feb 5, 2024
7f91bb5
look at images after pull
James-Mart Feb 5, 2024
345a533
image name includes the registry, apparently
James-Mart Feb 5, 2024
72181f7
also upload psibase-contributor image as artifact
James-Mart Feb 5, 2024
df6cd74
use v4 of upload/download actions
James-Mart Feb 6, 2024
3037937
Add base image
James-Mart Feb 6, 2024
6228195
add base image to local registry before reading it
James-Mart Feb 6, 2024
f72cc5c
remove the base image to save some disk space
James-Mart Feb 6, 2024
c6bf48d
try freeing up space
James-Mart Feb 6, 2024
4cadbc5
try freeing up space
James-Mart Feb 6, 2024
34ff939
remove large packages
James-Mart Feb 6, 2024
2d61374
upload pr artifact for psibase-contributor
James-Mart Feb 6, 2024
c24d073
fix typo
James-Mart Feb 6, 2024
bf99879
centralize all workflow dispatching for full control
James-Mart Feb 11, 2024
88d8ccf
Merge remote-tracking branch 'origin/main' into fix-workflow-runs
James-Mart Feb 11, 2024
5ea0b08
extract common builder workflow, parameterized over Ubuntu version
James-Mart Feb 11, 2024
bd317cf
fix invalid use of paths-ignore
James-Mart Feb 11, 2024
b75df8b
add checkout repo step
James-Mart Feb 11, 2024
6c1d618
fix outdated filenames
James-Mart Feb 11, 2024
38baefc
fix pattern checking
James-Mart Feb 12, 2024
6118e7e
fix globbing
James-Mart Feb 12, 2024
4dffc65
fix string parsing
James-Mart Feb 12, 2024
9714a0f
fix arg passing
James-Mart Feb 12, 2024
26a703e
try evaluating boolean differently
James-Mart Feb 12, 2024
be52157
fix all boolean evaluation
James-Mart Feb 12, 2024
7b93cac
add missing local registry service
James-Mart Feb 12, 2024
aa5a9ed
correctly pull base and tools images from either local or published a…
James-Mart Feb 12, 2024
afca006
Add some documentation on all of the CI/CD workflows
James-Mart Feb 12, 2024
c02f8c2
better temp artifact cleanup
James-Mart Feb 12, 2024
6b5948f
try more idiomatic way of building the archive for PR workflow runs
James-Mart Feb 15, 2024
a4c8d9d
fix dispatching logic
James-Mart Feb 15, 2024
5d15cf0
collapse dispatcher logic
James-Mart Feb 15, 2024
f75341d
fix check-patterns
James-Mart Feb 15, 2024
dff2891
check is_pr within each workflow instead of passing in a redundant va…
James-Mart Feb 15, 2024
63aa5b6
dont rely on 'latest' tag for tool-config image
James-Mart Feb 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/scripts/check-patterns.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/bin/bash

# Enable globstar for recursive globbing
shopt -s globstar

ALL_CHANGED_FILES="$1"
IFS=' ' read -r -a changed <<< $ALL_CHANGED_FILES

TOOL_CONFIG_PATTERNS=("docker/tool-config.Dockerfile .github/workflows/tool-config.yml docker/conf/**")
BUILDER_2004_PATTERNS=("docker/ubuntu-2004-builder.Dockerfile .github/workflows/builder-ubuntu.yml")
BUILDER_2204_PATTERNS=("docker/ubuntu-2204-builder.Dockerfile .github/workflows/builder-ubuntu.yml")
CONTRIB_PATTERNS=("docker/psibase-contributor.Dockerfile .github/workflows/contributor.yml")

matches_pattern() {
local pattern="$1"
for file in ${changed[@]}; do
if [[ $file == $pattern ]]; then
return 0 # Success
fi
done
return 1
}


run_tc="0"
for pattern in ${TOOL_CONFIG_PATTERNS[@]}; do
if matches_pattern $pattern; then
run_tc="1"
break
fi
done

run_2004="0"
for pattern in ${BUILDER_2004_PATTERNS[@]}; do
if matches_pattern $pattern; then
run_2004="1"
break
fi
done

run_2204="0"
for pattern in ${BUILDER_2204_PATTERNS[@]}; do
if matches_pattern $pattern; then
run_2204="1"
break
fi
done

run_contrib="0"
if [[ "$run_tc" == "1" ]] || [[ "$run_2204" == "1" ]]; then
run_contrib="1"
else
for pattern in ${CONTRIB_PATTERNS[@]}; do
if matches_pattern $pattern; then
run_contrib="1"
break
fi
done
fi

echo "${run_tc} ${run_2004} ${run_2204} ${run_contrib}"
18 changes: 18 additions & 0 deletions .github/scripts/latest-tag.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/bin/bash

# references:
# https://github.community/t/how-to-check-if-a-container-image-exists-on-ghcr/154836/3
# https://gist.github.com/eggplants/a046346571de66656f4d4d34de69fdd0

USER_IMAGE="$1"

# get token ('{"token":"***"}' -> '***')
TOKEN="$(
curl "https://ghcr.io/token?scope=repository:${USER_IMAGE}:pull" |
awk -F'"' '$0=$4'
)"
_curl(){ curl -s -H "Authorization: Bearer ${TOKEN}" "$1" 2>&1; }

# get most recent tag
LAST_TAG=$(_curl "https://ghcr.io/v2/${USER_IMAGE}/tags/list" | jq -r '.tags[-1]')
echo "${LAST_TAG}"
98 changes: 98 additions & 0 deletions .github/workflows/_dispatcher.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# This file is the dispatcher for the rest of the workflows that have complex dependencies.
# The `cli.yml` doesn't need to be dispatched via this dispatcher because it is only run manually.

name: Dispatcher

env:
GITHUB_OUTPUT: ""

on:
push:
branches:
- main
paths:
- "docker/**"
- ".github/**"
- "!.github/workflows/cli.yml"
- "!docker/psibase.Dockerfile"
- "!docker/psinode.Dockerfile"
pull_request:
types: [opened, synchronize, reopened]
paths:
- "docker/**"
- ".github/**"
- "!.github/workflows/cli.yml"
- "!docker/psibase.Dockerfile"
- "!docker/psinode.Dockerfile"

jobs:
determine-actions:
name: Set up the dispatching variables
runs-on: ubuntu-latest
outputs:
run_tc: ${{ steps.schedule-builders.outputs.run_tc }}
run_2004: ${{ steps.schedule-builders.outputs.run_2004 }}
run_2204: ${{ steps.schedule-builders.outputs.run_2204 }}
run_contrib: ${{ steps.schedule-builders.outputs.run_contrib }}

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
submodules: false
fetch-depth: 0

- name: Get changed files
swatanabe marked this conversation as resolved.
Show resolved Hide resolved
id: changed-files
uses: tj-actions/changed-files@v42

- name: (Debug) Print changed files
env:
ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
run: |
for file in $ALL_CHANGED_FILES; do
echo $file
done

- name: Determine what to dispatch
id: schedule-builders
env:
ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
run: |
necessary_jobs=$(./.github/scripts/check-patterns.sh "$ALL_CHANGED_FILES")
read -r run_tc run_2004 run_2204 run_contrib <<< "$necessary_jobs"
echo "run_tc=${run_tc}" >> $GITHUB_OUTPUT
echo "run_2004=${run_2004}" >> $GITHUB_OUTPUT
echo "run_2204=${run_2204}" >> $GITHUB_OUTPUT
echo "run_contrib=${run_contrib}" >> $GITHUB_OUTPUT

build-tool-config:
name: "Build tool config"
needs: determine-actions
if: ${{ needs.determine-actions.outputs.run_tc == '1' }}
uses: ./.github/workflows/tool-config.yml

build-2004:
name: "Build 20.04 builder"
needs: determine-actions
if: ${{ needs.determine-actions.outputs.run_2004 == '1' }}
uses: ./.github/workflows/builder-ubuntu.yml
with:
ubuntu_version: "2004"

build-2204:
name: "Build 22.04 builder"
needs: determine-actions
if: ${{ needs.determine-actions.outputs.run_2204 == '1' }}
uses: ./.github/workflows/builder-ubuntu.yml
with:
ubuntu_version: "2204"

build-contributor:
name: "Build contributor"
needs: [determine-actions, build-tool-config, build-2204]
if: ${{ needs.determine-actions.outputs.run_contrib == '1' }}
uses: ./.github/workflows/contributor.yml
with:
new_tools: ${{ needs.build-tool-config.result == 'success'}}
new_base: ${{ needs.build-2204.result == 'success'}}
81 changes: 81 additions & 0 deletions .github/workflows/builder-ubuntu.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: Generate ubuntu builder images
# Generates an image that encapsulate an environment on which it is possible to build psibase.

env:
GITHUB_OUTPUT: ""

on:
workflow_call:
inputs:
ubuntu_version:
description: "On what version of ubuntu should the build run?"
type: string
required: true
default: "2204"

jobs:
ubuntu-builder:
name: psibase-builder-ubuntu-${{ inputs.ubuntu_version }}
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
submodules: false
fetch-depth: 0

- name: Preparation
id: prep
run: |
OWNER="${{ github.repository_owner }}"
IMAGE="psibase-builder-ubuntu-${{ inputs.ubuntu_version }}"
REGISTRY="ghcr.io"
TAG="${{ github.sha }}"

TAGS="${REGISTRY}/${OWNER}/${IMAGE}:${TAG}"
echo "tags=${TAGS,,}" >> $GITHUB_OUTPUT

- name: Building ${{ steps.prep.outputs.tags }}
run: true

- name: Config docker buildx network
uses: docker/setup-buildx-action@v3
with:
buildkitd-flags: --debug

- name: Login in to registry
if: ${{ github.event_name != 'pull_request' }}
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build & Publish Image
if: ${{ github.event_name != 'pull_request' }}
uses: docker/build-push-action@v4
with:
context: .
push: true
file: docker/ubuntu-${{ inputs.ubuntu_version }}-builder.Dockerfile
tags: ${{ steps.prep.outputs.tags }}
platforms: linux/amd64
outputs: type=image,annotation-index.org.opencontainers.image.description=Psibase build environment based on Ubuntu ${{ inputs.ubuntu_version }}

- name: (PR Only) - Build image archive
if: ${{ github.event_name == 'pull_request' }}
uses: docker/build-push-action@v4
with:
context: .
file: docker/ubuntu-${{ inputs.ubuntu_version }}-builder.Dockerfile
tags: ${{ steps.prep.outputs.tags }}
platforms: linux/amd64
outputs: type=docker,dest=builder-${{ inputs.ubuntu_version }}-image.tar

- name: (PR only) - Upload image archive as artifact
if: ${{ github.event_name == 'pull_request' }}
uses: actions/upload-artifact@v4
with:
name: builder-${{ inputs.ubuntu_version }}-image
path: builder-${{ inputs.ubuntu_version }}-image.tar
retention-days: 1
24 changes: 15 additions & 9 deletions .github/workflows/cli.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,29 +25,35 @@ jobs:
run: |
REGISTRY="ghcr.io"
IMAGE="${REGISTRY}/${{ github.repository_owner }}/psinode"
TAGS="${IMAGE}:${{ github.sha }},${IMAGE}:${{ github.event.inputs.version }}"
TAGS="${IMAGE}:${{ github.event.inputs.version }}"
echo "tags=${TAGS,,}" >> $GITHUB_OUTPUT
- name: Showtag
id: showtag
run: echo ${{ steps.prep.outputs.tags }}
- name: Building ${{ steps.prep.outputs.tags }}
run: true

- name: Docker Buildx setup
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
with:
buildkitd-flags: --debug

- name: Login in to registry
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Set TOOL_CONFIG version
id: tool_cfg_img
run: |
latest_tag=$(./.github/scripts/latest-tag.sh "gofractally/http-tool-config")
echo "TOOL_CONFIG_IMAGE=ghcr.io/gofractally/http-tool-config:${latest_tag}" >> $GITHUB_OUTPUT

- name: Build & Publish Image
uses: docker/build-push-action@v4
with:
build-args: |
RELEASE_TAG=${{ github.event.inputs.version }}
TOOL_CONFIG_IMAGE=${{ steps.tool_cfg_img.outputs.TOOL_CONFIG_IMAGE }}
context: .
push: true
no-cache: true
Expand All @@ -69,19 +75,19 @@ jobs:
run: |
REGISTRY="ghcr.io"
IMAGE="${REGISTRY}/${{ github.repository_owner }}/psibase"
TAGS="${IMAGE}:${{ github.sha }},${IMAGE}:${{ github.event.inputs.version }}"
TAGS="${IMAGE}:${{ github.event.inputs.version }}"
echo "tags=${TAGS,,}" >> $GITHUB_OUTPUT
- name: Showtag
id: showtag
run: echo ${{ steps.prep.outputs.tags }}

- name: Docker Buildx setup
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
with:
buildkitd-flags: --debug

- name: Login in to registry
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
Expand Down
Loading
Loading