Skip to content

Commit

Permalink
Make workflow dispatch checkbox from source into action (infra) (#1517)
Browse files Browse the repository at this point in the history
* Refactor workflow into action

* Fix typo and set working directory

* Provide secrets via additioanl launcher

* Use launcher from the correct location

* Print override to screen

* Propagate the checkbox_revision from the action input

* Partially support noprovision

* Allow degraded wait_for_ssh

* Call the clean_machine script in the workflow

* Also set and refresh zapper snap

* Documented new parameter and reset default value

* Disable failfast

* Shallow fetch the Checkbox repo

* Try to change the path in an action aware way

* Fix typo

Co-authored-by: Paolo Gentili <[email protected]>

* Document parameters

* Use the new git_get scriplet

* Check the copy path before copying

* Try to move the scriplet to /bin

* Use the new name of the script

* Now git_get_shallow is in main

* Change the action target branch to the dispatch action

---------

Co-authored-by: Paolo Gentili <[email protected]>
  • Loading branch information
Hook25 and p-gentili authored Oct 2, 2024
1 parent 85ee304 commit 92a28be
Show file tree
Hide file tree
Showing 5 changed files with 124 additions and 43 deletions.
83 changes: 83 additions & 0 deletions .github/actions/checkbox_source_deb/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: Submit a Checkbox Test plan (or subset of it) to the lab
inputs:
data_source:
description: "Target image and provisioning data (ex. `url:` or `distro:`)"
required: false
default: null
queue:
description: "Queue that will run the testing (ex. 202012-28526)"
required: true
test_plan:
description: "Test plan to run (ex. com.canonical.certification::sru)"
required: true
match:
description: "Subset of jobs to run (ex. .*wireless.*)"
required: false
default: ".*"
launcher_override:
description: "Launcher with additional values that will take priority over the defaults"
default: ""
required: false
checkbox_revision:
description: "Revision of checkbox that has to be provisioned (ex. commit_hash, branch name, can be `beta`)"
required: true
zapper_channel:
description: "Zapper channel to be used, will be ignored if no Zapper (ex. edge, beta, stable)"
required: false
default: "beta"
runs:
using: composite
steps:
- uses: actions/checkout@v4
- name: Install dependencies
shell: bash
run: |
sudo apt install -y -qq gettext
- name: Build test resource
shell: bash
env:
INPUT_DATA_SOURCE: ${{ inputs.data_source }}
INPUT_QUEUE: ${{ inputs.queue }}
INPUT_MATCH: ${{ inputs.match || '.*' }}
INPUT_TEST_PLAN: ${{ inputs.test_plan }}
INPUT_LAUNCHER_OVERRIDE: ${{ inputs.launcher_override }}
INPUT_CHECKBOX_REVISION: ${{ inputs.checkbox_revision }}
INPUT_ZAPPER_CHANNEL: ${{ inputs.zapper_channel || 'beta' }}
working-directory: ${{ github.action_path }}/../../../tools/lab_dispatch
run: |
echo "::group::Building the testflinger job"
if [ -n "$INPUT_DATA_SOURCE" ]; then
INPUT_DATA_SOURCE="provision_data: $INPUT_DATA_SOURCE"
fi
envsubst '$INPUT_CHECKBOX_REVISION $INPUT_DATA_SOURCE $INPUT_QUEUE $INPUT_ZAPPER_CHANNEL' < generic_source.yaml | tee job.yaml
echo "::endgroup::"
echo "::group::Building the Checkbox launcher"
# this goes from .template. (missing secret, testplan, match etc. to .partial.)
# this is partial as some values are filled in on the agent (like wireless access points names)
envsubst '$INPUT_TEST_PLAN $INPUT_MATCH' < resources/checkbox.no-manifest.template.conf | tee resources/checkbox.no-manifest.partial.conf
echo "::endgroup::"
echo "::group::Dumping launcher overrides"
echo "$INPUT_LAUNCHER_OVERRIDE" | tee launcher_override.conf
echo "::endgroup::"
- name: Workaroud cwd
shell: bash
run: |
# this allows us to dispatch the action and the attachments with relative
# paths even when called form outside the Checkbox repo
action_path=$(realpath ${{ github.action_path }}/../../../tools/)
workdir_path=$(realpath tools/)
if [ ! -e "$workdir_path" ]; then
cp -rT "$action_path" "$workdir_path"
fi
if [ "$action_path" = "$workdir_path" ]; then
echo "Skipping copy as the action is already running in workdir"
else
cp -rT "$action_path" "$workdir_path"
fi
- name: Submit and monitor job
uses: canonical/testflinger/.github/actions/submit@main
with:
poll: true
job-path: tools/lab_dispatch/job.yaml
50 changes: 23 additions & 27 deletions .github/workflows/dispatch_lab_job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ on:
# - queue: machine that will run the job (ex. 202012-28526)
# - test_plan: Checkbox test plan to run (ex. com.canonical.certification::sru)
# - match: subset of jobs to run (ex. .*wireless.*)
# - zapper_channel: refreshes the zapper snap to the channel if provided, default is beta (ex. "beta")
#
# One possible matrix_to_create would therefore look like this:
# matrix_to_create=[{ data_source: "distro: desktop-22-04-2-uefi", queue: "202012-28526", match: ".*wireless.*", test_plan: "com.canonical.certification::sru" }]'
Expand All @@ -24,39 +25,34 @@ jobs:
run-matrix:
runs-on: [self-hosted, testflinger]
strategy:
fail-fast: false
matrix:
spec: ${{ fromJson(inputs.matrix_to_create) }}
defaults:
run:
working-directory: tools/lab_dispatch
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install dependencies
run: |
sudo apt install gettext
- name: Build test resource
env:
INPUT_DATA_SOURCE: ${{ matrix.spec.data_source }}
INPUT_QUEUE: ${{ matrix.spec.queue }}
INPUT_MATCH: ${{ matrix.spec.match }}
INPUT_TEST_PLAN: ${{ matrix.spec.test_plan }}
INPUT_PASSWORD_SECRET: ${{ secrets.INPUT_PASSWORD_SECRET }}
run: |
echo "::group::Building the testflinger job"
export INPUT_CHECKBOX_REVISION="$(git rev-parse HEAD)"
envsubst '$INPUT_CHECKBOX_REVISION $INPUT_DATA_SOURCE $INPUT_QUEUE' < generic_source.yaml | tee job.yaml
echo "::endgroup::"

echo "::group::Building the Checkbox launcher"
- name: Checkout repository
uses: actions/checkout@v4

- name: Get current commit SHA
id: get_sha
run: echo "sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT

# this goes from .template. (missing secret, testplan, match etc. to .partial.)
# this is partial as some values are filled in on the agent (like wireless access points names)
envsubst '$INPUT_TEST_PLAN $INPUT_MATCH $INPUT_PASSWORD_SECRET' < resources/checkbox.no-manifest.template.conf | tee resources/checkbox.no-manifest.partial.conf
echo "::endgroup::"
- name: Submit and monitor job
uses: canonical/testflinger/.github/actions/submit@main
- name: Run the spec
uses: canonical/checkbox/.github/actions/checkbox_source_deb@main
with:
poll: true
job-path: tools/lab_dispatch/job.yaml
data_source: ${{ matrix.spec.data_source }}
queue: ${{ matrix.spec.queue }}
test_plan: ${{ matrix.spec.test_plan }}
match: ${{ matrix.spec.match }}
zapper_channel: ${{ matrix.spec.zapper_channel }}
launcher_override: |
[environment]
WPA_BG_PSK = ${{ secrets.INPUT_PASSWORD_SECRET }}
WPA_N_PSK = ${{ secrets.INPUT_PASSWORD_SECRET }}
WPA_AC_PSK = ${{ secrets.INPUT_PASSWORD_SECRET }}
WPA_AX_PSK = ${{ secrets.INPUT_PASSWORD_SECRET }}
WPA3_AX_PSK = ${{ secrets.INPUT_PASSWORD_SECRET }}
checkbox_revision: ${{ steps.get_sha.outputs.sha }}
1 change: 1 addition & 0 deletions tools/lab_dispatch/build_install_deb.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ def install_local_package(repo_root, deb_name_glob):
"DEBIAN_FRONTEND=noninteractive",
"apt-get",
"--fix-broken",
"--allow-downgrades",
"-y",
"install",
]
Expand Down
28 changes: 17 additions & 11 deletions tools/lab_dispatch/generic_source.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
job_queue: $INPUT_QUEUE
global_timeout: 3600
output_timeout: 1800
provision_data:
$INPUT_DATA_SOURCE
$INPUT_DATA_SOURCE
test_data:
attachments:
- local: "tools/lab_dispatch/resources/manifest.conf"
Expand All @@ -11,6 +10,8 @@ test_data:
agent: "checkbox.no-manifest.partial.conf"
- local: "tools/lab_dispatch/build_install_deb.py"
agent: "build_install_deb.py"
- local: "tools/lab_dispatch/launcher_override.conf"
agent: "launcher_override.conf"
test_cmds: |
#!/usr/bin/env bash
Expand All @@ -28,23 +29,28 @@ test_data:
source install_tools.sh $TOOLS_PATH
# ensure device is available before continuing
wait_for_ssh
wait_for_ssh --allow-degraded
_run sudo add-apt-repository ppa:checkbox-dev/edge
_run install_packages git python3 python3-pip dpkg-dev
refresh_zapper_if_needed --channel "$INPUT_ZAPPER_CHANNEL"
wait_for_ssh
wait_for_ssh --allow-degraded
_run clean_machine --im-sure
_run sudo add-apt-repository ppa:checkbox-dev/edge
_put $RESOURCES_PATH/build_install_deb.py :
_run git clone https://github.com/canonical/checkbox.git
_run git -C checkbox checkout $CHECKBOX_REVISION
# clone the Checkbox revision without history (easier for slow systems, preserves RAM/storage/networking)
_run git_get_shallow https://github.com/canonical/checkbox.git commit $CHECKBOX_REVISION
_run wait_for_packages_complete
_run python3 build_install_deb.py --clean checkbox/checkbox-ng \
checkbox/checkbox-support checkbox/providers/resource \
checkbox/providers/base checkbox/providers/sru
_run sudo systemctl restart checkbox-ng
_run wait_for_packages_complete
git clone https://github.com/canonical/checkbox.git
git -C checkbox checkout $CHECKBOX_REVISION
git_get_shallow https://github.com/canonical/checkbox.git commit $CHECKBOX_REVISION
pipx install --spec checkbox/checkbox-ng checkbox-ng
# retrieve manifest
Expand All @@ -62,9 +68,9 @@ test_data:
which envsubst || install_packages gettext
envsubst < $RESOURCES_PATH/checkbox.no-manifest.partial.conf > checkbox.no-manifest.conf
# then insert the manifest entries via the stacker
stacker --output checkbox.conf checkbox.no-manifest.conf $MANIFEST_FILE
stacker --output checkbox.conf checkbox.no-manifest.conf $MANIFEST_FILE $RESOURCES_PATH/launcher_override.conf
wait_for_ssh
wait_for_ssh --allow-degraded
check_for_checkbox_service
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,10 @@ type = silent
[environment]
ROUTERS = multiple
WPA_BG_SSID = $WPA_BG_SSID
WPA_BG_PSK = $INPUT_PASSWORD_SECRET
WPA_N_SSID = $WPA_N_SSID
WPA_N_PSK = $INPUT_PASSWORD_SECRET
WPA_AC_SSID = $WPA_AC_SSID
WPA_AC_PSK = $INPUT_PASSWORD_SECRET
WPA_AX_SSID = $WPA_AX_SSID
WPA_AX_PSK = $INPUT_PASSWORD_SECRET
WPA3_AX_SSID = $WPA3_AX_SSID
WPA3_AX_PSK = $INPUT_PASSWORD_SECRET
OPEN_BG_SSID = $OPEN_BG_SSID
OPEN_N_SSID = $OPEN_N_SSID
OPEN_AC_SSID = $OPEN_AC_SSID
Expand Down

0 comments on commit 92a28be

Please sign in to comment.