Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests: new approach to run tests in ubuntu and debian using snapd deb from the repo #14496

Open
wants to merge 38 commits into
base: master
Choose a base branch
from
Open

tests: new approach to run tests in ubuntu and debian using snapd deb from the repo #14496

Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
bb15877
tests: new approach to run tests in ubuntu and debian using snapd deb…
sergiocazzolato Sep 12, 2024
a2637fd
Squashed 'tests/lib/external/snapd-testing-tools/' changes from 1c8ef…
sergiocazzolato Sep 12, 2024
63b2c2f
Merge commit 'a2637fda70a9d9f9fdc3d299712da9090735223f' into tests-up…
sergiocazzolato Sep 12, 2024
d1f0f31
download snapd snap-confine and ubuntu-core-launcher
sergiocazzolato Sep 12, 2024
6d5d458
skip tests that trust on deb pkg testkeys
sergiocazzolato Sep 16, 2024
ab0b03f
fix shellcheck
sergiocazzolato Sep 17, 2024
3c76431
revert kill timeout
sergiocazzolato Sep 17, 2024
7d78737
fix tests using snapd.tool and store-state test
sergiocazzolato Sep 18, 2024
87cbe02
new tests.info tool
sergiocazzolato Sep 19, 2024
1338735
skip tests which are testing with re-exec:0 and snap is comming from …
sergiocazzolato Sep 19, 2024
b7d5efb
update the systems where snapd comes from the repo
sergiocazzolato Sep 19, 2024
d9aa6db
skip another test when re-exec:0
sergiocazzolato Sep 20, 2024
3f2209b
Merge branch 'master' into tests-update-snapd-package
sergiocazzolato Sep 20, 2024
5e50960
Merge commit 'b0f4d8daff730868b7a60aebf75aac53c24b92c7' into tests-up…
sergiocazzolato Sep 20, 2024
b0f4d8d
Squashed 'tests/lib/external/snapd-testing-tools/' changes from 33c1d…
sergiocazzolato Sep 20, 2024
6cb8cb6
Adjust failing test to new re-exec scenario
sergiocazzolato Sep 23, 2024
9ddfe30
fix other 2 tests using correct paths
sergiocazzolato Sep 23, 2024
06f3a15
Squashed 'tests/lib/external/snapd-testing-tools/' changes from 91213…
sergiocazzolato Sep 23, 2024
c589280
Merge commit '06f3a157d82cfa92350c7d9a0634690d05874764' into tests-up…
sergiocazzolato Sep 23, 2024
73548a7
Merge branch 'master' into tests-update-snapd-package
sergiocazzolato Sep 23, 2024
744c89a
some tests fixes
sergiocazzolato Sep 23, 2024
2e32c65
New set of test fixes
sergiocazzolato Sep 24, 2024
514dbb8
update tests info command to check re-execution
sergiocazzolato Sep 24, 2024
ec96de4
fix uc16 which preparing
sergiocazzolato Sep 24, 2024
e048240
check SPREAD_SYSTEM is not core in tests.info
sergiocazzolato Sep 24, 2024
cb9b585
use .skip in tests to determine if a phase has to be skipped
sergiocazzolato Sep 25, 2024
463424e
skip updating core snap when the snap pkg is not built from local
sergiocazzolato Sep 26, 2024
5758688
save installed core snap when snap deb is used from the repo
sergiocazzolato Sep 26, 2024
cd751ac
copy also .orig core snap
sergiocazzolato Sep 27, 2024
8e5bfcb
change how we copy the core snap
sergiocazzolato Sep 27, 2024
d7f37b9
change the snapname used for core snap
sergiocazzolato Sep 27, 2024
1624ebf
skip saving the core snap
sergiocazzolato Sep 27, 2024
5e1e520
Fix error using snapcraft for testing in arm
sergiocazzolato Sep 27, 2024
37487c1
make sure fips tests dont run with reexec=0 and snapd.deb from the repo
sergiocazzolato Sep 27, 2024
c0ccc68
simplify how fips tests are skipped
sergiocazzolato Sep 27, 2024
b14c80d
fix indentation on test
sergiocazzolato Sep 27, 2024
cde70a7
Make sure we save core snap in TESTSTMP/core_snap
sergiocazzolato Sep 27, 2024
b50e533
use tests.exec tool to skip tests
sergiocazzolato Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions spread.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ environment:
SNAPD_NO_MEMORY_LIMIT: '$(HOST: echo "${SPREAD_SNAPD_NO_MEMORY_LIMIT:-}")'

SNAPD_PUBLISHED_VERSION: '$(HOST: echo "$SPREAD_SNAPD_PUBLISHED_VERSION")'
sergiocazzolato marked this conversation as resolved.
Show resolved Hide resolved
# Use the snapd package from the repository when possible
SNAPD_DEB_FROM_REPO: '$(HOST: echo "${SPREAD_SNAPD_DEB_FROM_REPO:-true}")'
# Build and use snapd from current branch
BUILD_SNAPD_FROM_CURRENT: '$(HOST: echo "${SPREAD_BUILD_SNAPD_FROM_CURRENT:-true}")'

Expand Down
1 change: 1 addition & 0 deletions tests/bin/tests.info
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../lib/tools/tests.info
4 changes: 3 additions & 1 deletion tests/lib/prepare-restore.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -587,7 +587,9 @@ prepare_project() {
# go mod runs as root and will leave strange permissions
chown test:test -R "$SPREAD_PATH"

if [ "$BUILD_SNAPD_FROM_CURRENT" = true ]; then
if tests.info is-snapd-pkg-repo; then
sergiocazzolato marked this conversation as resolved.
Show resolved Hide resolved
( cd "${GOHOME}" && tests.pkgs download snapd snap-confine)
elif [ "$BUILD_SNAPD_FROM_CURRENT" = true ]; then
case "$SPREAD_SYSTEM" in
ubuntu-*|debian-*)
build_deb
Expand Down
2 changes: 1 addition & 1 deletion tests/lib/prepare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1192,7 +1192,7 @@ setup_reflash_magic() {
# FIXME: install would be better but we don't have dpkg on
# the image
# unpack our freshly build snapd into the new snapd snap
dpkg-deb -x "$SPREAD_PATH"/../snapd_*.deb "$UNPACK_DIR"
dpkg-deb -x "$GOHOME"/snapd_*.deb "$UNPACK_DIR"
# Debian packages don't carry permissions correctly and we use
# post-inst hooks to fix that on classic systems. Here, as a special
# case, fix the void directory we just unpacked.
Expand Down
10 changes: 8 additions & 2 deletions tests/lib/tools/snapd.tool
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,14 @@ main() {
esac
done

LIBEXEC_DIR="$(os.paths libexec-dir)"
exec "$LIBEXEC_DIR/snapd/$tool" "$@"
# Use the tool from the snapd snap when re-exec is enabled
# Otherwise use the tool from the distro installation
if tests.info is-reexec-done; then
exec /snap/snapd/current/usr/lib/snapd/"$tool" "$@"
sergiocazzolato marked this conversation as resolved.
Show resolved Hide resolved
else
LIBEXEC_DIR="$(os.paths libexec-dir)"
exec "$LIBEXEC_DIR/snapd/$tool" "$@"
fi
}

main "$@"
60 changes: 60 additions & 0 deletions tests/lib/tools/tests.info
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/bin/bash

set -eu

zyga marked this conversation as resolved.
Show resolved Hide resolved
show_help() {
echo "usage: tests.info <CMD>"
echo
echo "Supported commands:"
echo " is-snapd-pkg-repo: indicates when the snap packages is from the repository"
sergiocazzolato marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be called is-snapd-deb-from-repo since it only works for debs? or is there a missing TODO to make this work on other systems as well?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, if at some point the re-execution is implemented for rpms, then we could make other systems download the rpm as we do with the deb.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the name is fine and should not include the word "deb" -- we will soon enable this for RPMs in tests. My only personal preference would be to call it "is-snapd-from-archive" as it reads better than abbreviated repository IMO. No need to change this though :)

echo " is-reexec-req: indicates re-execution has been explicitly enabled through the env"
sergiocazzolato marked this conversation as resolved.
Show resolved Hide resolved
echo " is-reexec-done: indicates re-execution to the snapd snap is being effectively performed"
sergiocazzolato marked this conversation as resolved.
Show resolved Hide resolved
}

is_snapd_pkg_repo() {
[ "$SNAPD_DEB_FROM_REPO" = true ] && \
[ -n "$(command -v apt)" ] && \
os.query is-classic && \
not os.query is-trusty
}

is_reexec_req() {
snap debug execution snap | grep -q "is-reexec-enabled: true"
}

is_reexec_done() {
snap debug execution snap | grep -q "is-reexecd: true"
sergiocazzolato marked this conversation as resolved.
Show resolved Hide resolved
}

main() {
if [ $# -eq 0 ]; then
show_help
exit 0
fi

local subcommand="$1"
local action=
while [ $# -gt 0 ]; do
case "$subcommand" in
-h|--help)
show_help
exit 0
;;
*)
action=$(echo "$subcommand" | tr '-' '_')
shift
break
;;
esac
done

if [ -z "$(declare -f "$action")" ]; then
echo "tests.env: no such command: $subcommand"
show_help
exit 1
fi

"$action" "$@"
}

main "$@"
9 changes: 6 additions & 3 deletions tests/main/classic-custom-device-reg/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ environment:
SEED_DIR: /var/lib/snapd/seed

prepare: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking if this is really the case. Snapd classic will re-exec into snapd snap that may be built with test keys. Does it really fail?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are some scenarios where we don't have snapd snap installed, so it uses the keys from the deb.
I'll add a comment there

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this sounds good. If We identify all the cases that don't have snapd snap installed then we can work on fixing them. I assume that the point of the test was not to test that specific behavior and that we can fix them over time.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, my idea is to create a following pr to cleanup some scenarios and fix other tests

echo "This test needs test keys to be trusted"
exit
fi
Expand Down Expand Up @@ -52,7 +53,8 @@ prepare: |
systemd-run --unit fakedevicesvc fakedevicesvc localhost:11029

restore: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand All @@ -62,7 +64,8 @@ restore: |
systemctl start snapd.socket snapd.service

execute: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down
9 changes: 6 additions & 3 deletions tests/main/classic-firstboot/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ environment:
SEED_DIR: /var/lib/snapd/seed

prepare: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down Expand Up @@ -51,7 +52,8 @@ prepare: |
cp ./test-snapd-service_1.0_all.snap "$SEED_DIR/snaps/test-snapd-service.snap"

restore: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand All @@ -60,7 +62,8 @@ restore: |
systemctl start snapd.socket snapd.service

execute: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down
9 changes: 6 additions & 3 deletions tests/main/classic-prepare-image-no-core/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ environment:
SEED_DIR: /var/lib/snapd/seed

prepare: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down Expand Up @@ -49,7 +50,8 @@ prepare: |
systemd-run --unit fakedevicesvc fakedevicesvc localhost:11029

restore: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand All @@ -61,7 +63,8 @@ restore: |
rm -rf "$ROOT"

execute: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd deb are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down
9 changes: 6 additions & 3 deletions tests/main/classic-prepare-image/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ environment:
SEED_DIR: /var/lib/snapd/seed

prepare: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down Expand Up @@ -48,7 +49,8 @@ prepare: |
systemd-run --unit fakedevicesvc fakedevicesvc localhost:11029

restore: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand All @@ -60,7 +62,8 @@ restore: |
rm -rf "$ROOT"

execute: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down
9 changes: 6 additions & 3 deletions tests/main/classic-snapd-firstboot/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ environment:
SEED_DIR: /var/lib/snapd/seed

prepare: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down Expand Up @@ -51,7 +52,8 @@ prepare: |
cp ./basic18_1.0_all.snap "$SEED_DIR/snaps/basic18.snap"

restore: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand All @@ -60,7 +62,8 @@ restore: |
systemctl start snapd.socket snapd.service

execute: |
if [ "$TRUST_TEST_KEYS" = "false" ]; then
# In this scenario, the keys from the snapd pkg are used
if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-pkg-repo; then
echo "This test needs test keys to be trusted"
exit
fi
Expand Down
10 changes: 10 additions & 0 deletions tests/main/command-chain/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ environment:
ENVDUMP: /var/snap/command-chain/current/env

prepare: |
# No needed to test the snap pkg when it is coming from the repository
if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-pkg-repo; then
exit
fi

echo "Build command chain snap"
snap pack command-chain
snap install --dangerous command-chain_1.0_all.snap
Expand All @@ -28,6 +33,11 @@ prepare: |
fi

execute: |
# No needed to test the snap pkg when it is coming from the repository
if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-pkg-repo; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get why there's no need to test this. Is the test failing?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a bug in the deb which makes snapd fail to pack snaps.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also I don't see any value running those tests with reexec=0 when we use snapd deb from the repo.

exit
fi

echo "Test that command-chain runs for hooks"
[ "$(cat "$BREADCRUMB")" = "chain1 chain2 configure" ]
MATCH '^CHAIN_1_RAN=1$' < "$ENVDUMP"
Expand Down
5 changes: 5 additions & 0 deletions tests/main/debug-execution/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ debug: |
grep -n '' snap-*.out || true

execute: |
# TODO: remove this check once snapd 2.65 is released
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this todo cannot be removed because we will always have a distribution with old snapd.

I think the test needs more nuance to execute the SNAP_REEXEC=1 without other conditions.

if tests.info is-snapd-pkg-repo; then
exit 0
fi

snap debug execution snap > snap-default.out
SNAP_REEXEC=0 snap debug execution snap > snap-no-reexec.out
SNAP_REEXEC=1 snap debug execution snap > snap-yes-reexec.out
Expand Down
10 changes: 10 additions & 0 deletions tests/main/install-errors/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,20 @@ environment:
SNAP_REEXEC/withreexec: 1

prepare: |
# No needed to test the snap pkg when it is coming from the repository
if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-pkg-repo; then
exit 0
fi

echo "Given a snap with a failing command is installed"
"$TESTSTOOLS"/snaps-state install-local "$SNAP_NAME"

execute: |
# No needed to test the snap pkg when it is coming from the repository
if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-pkg-repo; then
exit 0
fi

echo "Install unexisting snap prints error"
if snap install unexisting.canonical; then
echo "Installing unexisting snap should fail"
Expand Down
9 changes: 9 additions & 0 deletions tests/main/install-sideload-epochs/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,19 @@ environment:
SNAP_REEXEC/reexec1: 1

prepare: |
# No needed to test the snap pkg when it is coming from the repository
if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-pkg-repo; then
exit
fi
snap pack test-snapd-epoch-1
snap pack test-snapd-epoch-2

execute: |
# No needed to test the snap pkg when it is coming from the repository
if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-pkg-repo; then
exit
fi

rx="cannot refresh \"[^ \"]*\" to local snap with epoch [^ ]*, because it can't read the current epoch"
snap try test-snapd-epoch-1
not snap try test-snapd-epoch-2 2> try.err
Expand Down
10 changes: 10 additions & 0 deletions tests/main/install-sideload/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ environment:
SNAP_REEXEC/reexec1: 1

prepare: |
# No needed to test the snap pkg when it is coming from the repository
if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-pkg-repo; then
exit
fi

for snap in basic test-snapd-tools basic-desktop test-snapd-devmode snap-hooks-bad-install; do
snap pack "$TESTSLIB"/snaps/$snap
done
Expand All @@ -26,6 +31,11 @@ prepare: |
fi

execute: |
# No needed to test the snap pkg when it is coming from the repository
if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-pkg-repo; then
exit
fi

echo "Sideloaded snap shows status"
expected='^basic 1.0 installed$'
snap install --dangerous ./basic_1.0_all.snap | MATCH "$expected"
Expand Down
3 changes: 3 additions & 0 deletions tests/main/interfaces-personal-files/task.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,9 @@ execute: |

echo "Find snap-confine"
SNAP_CONFINE=$(os.paths libexec-dir)/snapd/snap-confine
if tests.info is-snapd-pkg-repo; then
SNAP_CONFINE="/snap/snapd/current$SNAP_CONFINE"
fi
# on Ubuntu Core we need to use the correct path to ensure it is
# appropriately confined by apparmor as it may be from the snapd
# snap
Expand Down
Loading
Loading