tests: run tests based on the change included in the pr (#14338)

* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 1c8efb77e1..4ab8b00afb

4ab8b00afb New spread-filter tool (#53)
cdf5cfd47b Remove centos-7 support
f3996cc3fa change the spread label
1e309f41c6 change how legacy parameter is determine in remote.pull
c43c35f7e3 run remote refresh and wait-for for xenial (skip bionic)
5262d30da7 make sure the test jobs are executed in runners with the spread label
cb74259b7a add openstack systems
0b41fd40d3 fix tests.pkgs on arch-linux
558e109793 run fedora-40 spread tests in openstack
6f6187416d fix list implementation
b4a5439c9b added more type annotatios to log_helper
58da1e36c3 mypy cleaned
1ff651e680 update wording of remote.pull
18615b1667 just usc scp -O when ssh version is -ge 9
cc68c9868b Added type annotations for log-filter
66f90d10cd Adding -O to scp command to make it compatible in uc24 tests
496cb7b5b3 removing support for centos-8
f2eef30db4 Updated the log helper and log parser
5a375ebf73 Formatting for python utils
d3eed3faa5 fix codespell in CODE_OF_CONDUCT.md
18bcca6b14 new log helper
d60381fcd9 add run number to filtered filename
5dde2d67b8 consider the tests execution in main
6b9a3aabcc change filtered log name
b2756aa579 default file is .filtered.log
500b9dace4 Fix tests workflow
45db26a3d2 fix shellcheck error in log-filter
fe45c27b7d create a var to store filter params
5a9b66d7dc filter spread results
51f9b055af New tool used to filter the spread.log output
b8d20c1d5b fix snaps.name test with correct siffix spelling
f640ac72e3 Add missing test details
f0754df304 Filter the error y debug output in log-parser
fc10196efd Add suggestions to details
94ac5ffe58 Add details on tests
501578c719 add more checks in os.query to check is-core_xx
e8929207ff fix os-query for ubuntu comparing with core
226114641f os.query won't check SPREAD_SYSTEM anymore to compare core systems
b89ec98b23 use local variables in os.query tool
dacfd81de9 fix is_core functions
1db5214d5f Improve the remote docs (#36)
2e4a3153a2 1 more comment
3a0fc57e1e add explanation about why we check for ( Do | Doing )
4cf8e635bf fix os.query test after merge
b89b4f8647 fix artifacts name
d30cee6da0 Merge remote-tracking branch 'upstream/main'
5ef5dcbe8f Tests use artifacts in spread tests (#51)
555c43d2ab Support auto-refresh with Do instead of Doing
96c2b0c19c remove tests support for ubuntu 23.04 (EoL)
74082c0c34 Tests improve remote wait (#49)
5121bfb659 remove support for opensuse leap 15.4 (#48)
30df700d08 Add new systems support (#47)
1f08938925 Support check amazon linux version (#46)
43533bdd97 Change the exit value checking for test formats (#45)
3c88244c04 Update check-test-format to support a dir and a list of files (#44)
510d95f429 add extra check for error in auto-refresh detection function
3289d4031b Try open the log with latin-1 encoding when utf-8 is not working
9db785499f improved how the tools are waiting for system reboot
2a5c4414a3 fix shellcheck errors
5e7b63883d Fixes for osquery and tests pkgs (#43)
4c9145e2ac support reboot waiting for auto-refresh
45768f5188 show changes in unknown status after refresh
8013c30c2a Remove support for ubuntu 22.10
b32b80bf54 Fix remote.rait-for test in bionic
5675c625e9 Enable fedora 38
55f4471957 Support for new oss
f2e88b357c New tool used to query spread json reports
cacd35ede0 utils/spread-shellcheck: explain disabled warnings (#42)
c82afb2dee Support --no-install-recommends parameter when installing dependencies with tests.pkgs
b84eea92e2 spread-shellcheck: fix quotes in environment variables (#41)
ab1e51c29f New comparison in os-query for core systems (#40)
e5ae22a5d4 systemd units can be overwritten
63540b845a Fix error messages in remote pull and push
75e8a426a5 make sure the unit is removed in tests.systemd test
9089ff5c02 Update tests to use the new tests.systemd stop-unit
44ecd5e56a Move tests.systemd stop-units to stop-unit
01a2a83b4b Update tests.systemd to have stop units as systemd.sh
162e93bd35 update tests.systemd CLI options to be the same than retry command
14aa43a405 new feature to re-run failed spread tests (#39)
604cb782db Fix shellcheck in systemd tool
bfc71082c8 Update the tests.systemd to allow parameters waiting for service status
8a2d0a99df Adding quiet tool and removing set +-x from tests.pkgs
d90935d2a4 A comment explaining about the default values for wait-for
3232c5dba7 Add support for ubuntu 23.04
a7164fba07 remove fedora 35 support, add fedora 37 support
89b9eb5301 Update systems supported
92bb6a0664 Include snap-sufix in the snaps.name tool

git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 4ab8b00afbfe73017a88c58018c72a2d5beea3be

* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 4ab8b00afb..e43b9e314d

e43b9e314d rename test libraries to avoid static check errors in snapd

git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: e43b9e314d0a078521d0ee3ec115f11c03005c12

* Squashed 'tests/lib/external/snapd-testing-tools/' changes from e43b9e314d..33aa8d1e53

33aa8d1e53 fix details in remote.retry test

git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 33aa8d1e53161e740d7ac720d13284fb9ddacb82

* tests: run tests based on the change included in the pr

This change runs the tests based on the changes in cluded in a pr.

There are defined some set of rules which can be used to determine which
tests to run. In this change we define 3 set of rules:
 1. main is used for most of the systems.
 2. nested is used for nested tests systems
 3. trusty is used for ubuntu 14.04 only

* fix actions script (bash)

* show the suggested tests instead of running them

* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 33aa8d1e53..1d4350a818

1d4350a818 Make sure the tools are at the begining of the path
45c2f45004 address comments
8743ceaf8c tools: fix support for multi-arch packages on apt-based systems (#54)

git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 1d4350a818c69da0240f4848ba08dc9411293e5a

* udpate how suggested tests are displayed

* udpate rule names

* updated tests to run on nested 24.04

* Adding README in the rules directory

.github/workflows/test.yaml

@@ -616,107 +616,133 @@ jobs:
             backend: google-distro-1
             systems: 'amazon-linux-2-64 amazon-linux-2023-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: arch-linux
             backend: google-distro-2
             systems: 'arch-linux-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: centos
             backend: google-distro-2
             systems: 'centos-9-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: debian-req
             backend: google-distro-1
             systems: 'debian-11-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: debian-not-req
             backend: google-distro-1
             systems: 'debian-12-64 debian-sid-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: fedora
             backend: google-distro-1
             systems: 'fedora-38-64 fedora-39-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: fedora-os
             backend: openstack
             systems: 'fedora-40-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: opensuse
             backend: google-distro-2
             systems: 'opensuse-15.5-64 opensuse-15.6-64 opensuse-tumbleweed-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-trusty
             backend: google
             systems: 'ubuntu-14.04-64'
             tests: 'tests/smoke/ tests/main/canonical-livepatch tests/main/canonical-livepatch-14.04'
+            rules: 'trusty'
           - group: ubuntu-xenial-bionic
             backend: google
             systems: 'ubuntu-16.04-64 ubuntu-18.04-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-focal-jammy
             backend: google
             systems: 'ubuntu-20.04-64 ubuntu-22.04-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-noble
             backend: google
             systems: 'ubuntu-24.04-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-no-lts
             backend: google
             systems: ''
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-daily
             backend: google
             systems: 'ubuntu-24.10-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-core-16
             backend: google-core
             systems: 'ubuntu-core-16-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-core-18
             backend: google-core
             systems: 'ubuntu-core-18-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-core-20
             backend: google-core
             systems: 'ubuntu-core-20-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-core-22
             backend: google-core
             systems: 'ubuntu-core-22-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-core-24
             backend: google-core
             systems: 'ubuntu-core-24-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-arm
             backend: google-arm
             systems: 'ubuntu-20.04-arm-64 ubuntu-core-22-arm-64'
             tests: 'tests/...'
+            rules: 'main'
           - group: ubuntu-secboot
             backend: google
             systems: 'ubuntu-secboot-20.04-64'
             tests: 'tests/...'
+            rules: 'main'
 
           - group: nested-ubuntu-16.04
             backend: google-nested
             systems: 'ubuntu-16.04-64'
             tests: 'tests/nested/...'
+            rules: 'nested'
           - group: nested-ubuntu-18.04
             backend: google-nested
             systems: 'ubuntu-18.04-64'
             tests: 'tests/nested/...'
+            rules: 'nested'
           - group: nested-ubuntu-20.04
             backend: google-nested
             systems: 'ubuntu-20.04-64'
             tests: 'tests/nested/...'
+            rules: 'nested'
           - group: nested-ubuntu-22.04
             backend: google-nested
             systems: 'ubuntu-22.04-64'
             tests: 'tests/nested/...'
+            rules: 'nested'
           - group: nested-ubuntu-24.04
             backend: google-nested
             systems: 'ubuntu-24.04-64'
             tests: 'tests/nested/...'
+            rules: 'nested'
     steps:
     - name: Cleanup job workspace
       id: cleanup-job-workspace
@@ -742,6 +768,7 @@ jobs:
         path: "${{ github.workspace }}/.test-results"
         key: "${{ github.job }}-results-${{ github.run_id }}-${{ matrix.group }}-${{ steps.get-previous-attempt.outputs.previous_attempt }}"
 
+
     - name: Prepare test results env and vars
       id: prepare-test-results-env
       run: |
@@ -756,6 +783,16 @@ jobs:
           # This step has to be after the cache is restored
           mkdir -p "$TEST_RESULTS_DIR"        
 
+    - name: Get changed files
+      id: changed-files
+      uses: tj-actions/[email protected]
+
+    - name: Save changes files
+      run: |
+          CHANGED_FILES="${{ steps.changed-files.outputs.all_changed_files }}"
+          echo "CHANGED_FILES=$CHANGED_FILES" >> $GITHUB_ENV
+          echo "The changed files found are: $CHANGED_FILES"
+
     - name: Check failed tests to run
       if: "!contains(github.event.pull_request.labels.*.name, 'Run all')"
       run: |
@@ -774,17 +811,25 @@ jobs:
       if: "!contains(github.event.pull_request.labels.*.name, 'Skip spread')"
       run: |
           RUN_TESTS=""
+          SUGGESTED_TESTS=""
           # Save previous failed test results in FAILED_TESTS env var
           if [ -n "$FAILED_TESTS" ]; then
               RUN_TESTS="$FAILED_TESTS"
           else
               for SYSTEM in ${{ matrix.systems }}; do
                   for TESTS in ${{ matrix.tests }}; do
-                      RUN_TESTS="$RUN_TESTS ${{ matrix.backend }}:$SYSTEM:$TESTS"
+                    RUN_TESTS="$RUN_TESTS ${{ matrix.backend }}:$SYSTEM:$TESTS"
+                  done
+                  CHANGES_PARAM=""
+                  for CHANGE in $CHANGED_FILES; do
+                    CHANGES_PARAM="$CHANGES_PARAM -c $CHANGE"
                   done
+                  SUGGESTED_TESTS="$SUGGESTED_TESTS $(./tests/lib/external/snapd-testing-tools/utils/spread-filter -r ./tests/lib/spread/rules/${{ matrix.rules }}.yaml -p "${{ matrix.backend }}:$SYSTEM" $CHANGES_PARAM)"
               done
           fi
           echo RUN_TESTS="$RUN_TESTS"  >> $GITHUB_ENV
+          echo "Suggested tests by spread-filter tool"
+          echo "$SUGGESTED_TESTS"
 
     - name: Setup grafana parameters
       if: "!contains(github.event.pull_request.labels.*.name, 'Skip spread')"

tests/lib/spread/rules/README

@@ -0,0 +1,34 @@
+This directory include the rules files used to determine which
+tests have to be executed based on a list of changed files.
+
+Each rules file is used depending on the system being evaluated.
+This is an example of rules:
+
+rules:
+  RULE_NAME_1:
+    from: [FROM_LIST]
+    to: [TO_LIST]
+
+  RULE_NAME_2:
+    ...
+
+where:
+RULE_NAME_1: is a descriptive name of the specific rule
+FROM_LIST: is a list of python regex used to determine if a file matches this rule
+TO_LIST: is a list of spread tests to run in case a change matches a regex from the FROM_LIST
+
+The spread-filter tool, based on the list of changes, determine which tests need to
+be executed. For each changed file, the spread-filter will use the TO_LIST for the first rule
+matching it. 
+
+Finally, the spread-filter calculates the union of all the tests and prints all the tests to
+be executed.
+
+In case the FROM_LIST is [.*], this means that any change match this rule. This
+can be used for example as the last rule to make sure that the changes which didn't match any
+rule are matched by the last one. 
+
+In case the TO_LIST includes the value $SELF, means that in case of a test, just
+this test is executed.
+
+

tests/lib/spread/rules/main.yaml

@@ -0,0 +1,23 @@
+rules:
+  tests:
+    from:
+      - tests/main/.*
+      - tests/core/.*
+      - tests/completion/.*
+      - tests/cross/.*
+      - tests/regression/.*
+      - tests/smoke/.
+      - tests/unit/.*
+      - tests/upgrade/.*
+    to: [$SELF]
+
+  unit:
+    from: [.*_test.go]
+    to:
+     - tests/unit/go
+     - tests/unit/c-unit-tests-clang
+     - tests/unit/c-unit-tests-gcc
+
+  rest:
+    from: [.*]
+    to: [tests/]

tests/lib/spread/rules/nested.yaml

@@ -0,0 +1,9 @@
+rules:
+  nested:
+    from:
+      - tests/nested/.*
+    to: [$SELF]
+
+  rest:
+    from: [.*]
+    to: [tests/nested/]

tests/lib/spread/rules/trusty.yaml

@@ -0,0 +1,7 @@
+rules:
+  all:
+    from: [.*]
+    to:
+      - tests/smoke/
+      - tests/main/canonical-livepatch
+      - tests/main/canonical-livepatch-14.04