Find a new location for tmt scripts

[thrix]

For compatibility and various functional reasons, tmt provides several scripts in the test environment like tmt-reboot, tmt-abort, etc. See here for more details https://tmt.readthedocs.io/en/stable/spec/plans.html#tmt.

As we want to support distributions using rpm-ostree and bootc distributions (Fedora CoreOS, RHEL Image Mode, etc.), the original path where these scripts were stored, i.e. /usr/local/bin cannot be used, as on these distros these paths are read-only.

We are opening this discussion and poll so you can vote for the new location of these scripts. The new path will be included in the default executable paths via /etc/profile.d scripts. See this merge request which will introduce this change.

To reserve a Fedora 40 in Image mode machine for experimenting you can use this testing-farm CLI command using the container wrapper which has all the workarounds needed to reserve this system on Public Ranch:

IMAGE=quay.io/testing-farm/cli:1478751011 testing-farm reserve --compose Fedora-Cloud-Base-40-x86_64-bootc --worker-image quay.io/testing-farm/worker-public:1462003722

If you have more suggestions, let me know.

[LecrisUT]

Need a bit more background on this. These are generated from the template inside tmt to be executed inside the test environment? If so shouldn't be under the tmt run directory?

[thrix]

@happz the reservation implementation is a workaround and we want to do that, so that need for this use case should disappear in the future.

I am voting for not caring about isolation :) and documenting is a multiple tmt invocations are not supported in genral, but they work in case users do not change the TMT_SCRIPTS_DEST_DIR (and we can extend the docs for different cases if we know about them).

@LecrisUT WDYT?

[thrix]

Any opinions @happz @martinhoyer @psss @lukaszachy @seberm ?

[kkaarreell]

Maybe I misunderstood but multiple invocations are a real thing, with provision -h connect.

[LecrisUT]

Ok, needed to read the relevant stories on tmt-abort and tmt-reboot. Iiuc these scripts are meant to be used by the test runners, so they would be copied in the TMT_SCRIPTS_DEST_DIR of the provisioned machine. Since they would be copied every time, I think it makes sense to have them run-specific at some point. The other point I was considering is if it could be used within the importlib_resources, but that only works for the tmt controller, not for the provisioned guests.

As an intermediate I've voted for /var/tmp/tmt/scripts, primarily because it indicates that the folders can be deleted without breaking some functionality, and there is already precedence of using that folder for the controller.

But at the very least all of these paths should be made configurable, e.g. for homebrew deployment/guests this would not work without sudo access, so the defaults would need to be changed. If it would be exposed in provision step, that would also be beneficial.

Tangent: what was the reason for /var/tmp/tmt instead of ~/.local/tmt?

[happz]

But at the very least all of these paths should be made configurable, e.g. for homebrew deployment/guests this would not work without sudo access, so the defaults would need to be changed. If it would be exposed in provision step, that would also be beneficial.

This particular path should be modifiable by setting the TMT_SCRIPTS_DEST_DIR envvar when running tmt. If you mean adding tmt keys, then execute would probably be better, it owns these scripts, no other step comes into contact with them, but that's just an immediate feeling. And sudo would be added automagically when using non-superuser user, so that could work for homebrew? Disclaimer, I have no experience with that tool.

[kkaarreell]

Btw, it Is quite Common to set noexec for /var mount. I suppose this won't be the case for any of the images used.

[thrix]

Btw, it Is quite Common to set noexec for /var mount. I suppose this won't be the case for any of the images used.

Interesting, need to understand that more, we did not hit it so no. The question is if there are tests which could hit this. Do you know about some test environment that would have it like this?

[thrix]

@kkaarreell yeah, so for hardened systems this could pose a real problem really, meh. So for these systems users will be need to manually override this via setting TMT_SCRIPTS_DEST_DIR to another path, which is executable. Btw, we already have this problem anyway in Testing Farm, where the tmt workdir is set to /var/ARTIFACTS and thus if a script is runinng just ./test.sh it won't work anyway. Maybe that makes the best practice rather to use bash test.sh for the test entrypoint :)

[comps]

I think an alternate approach would be worth considering.

When we were making tests/tools for ImageBuilder (a.k.a. OSBuild), there was temptation to hack in some executable temporary directory, or some custom systemd replacement that would run our "post-install" scripts on first boot, or something similar. All this because OSBuild is so strict and tight that you can't do anything custom.

Then we looked at it from a customer PoV and found a documented way to do all we wanted - just use a custom RPM that is part of the image-building DNF transaction, runs whatever as RPM scriptlets, and places whatever files as any regular RPM.

The point is - there probably is some standard way to modify ostree/bootc images that a customer would use, and maybe that's the approach testing-farm should use. Pre-install tmt on the images and have helper scripts in standard /usr/libexec (or in /usr/bin, why not).

The alternative is to keep moving against the wind, finding "exploits", like /var (or /usr/share) allowing execution, that will eventually get patched. Same with fapolicyd prohibiting execution of anything that isn't RPM-installed and hashed. The solution is to just follow standard OS software processes that any customer would use.

[happz]

Just a small clarification:

...

The point is - there probably is some standard way to modify ostree/bootc images that a customer would use, and maybe that's the approach testing-farm should use. Pre-install tmt on the images and have helper scripts in standard /usr/libexec (or in /usr/bin, why not).

• not testing-farm, but tmt itself. tmt "installs" these scripts on guests on its own before running tests on them.
• tmt itself does not need to be preinstalled in images, as tmt runs on different machine, connecting to one spawned from the image.

[kkaarreell]

I am not sure whether we should not make a step back and first outline how real tests (with their runtine dependencies) are going be executed. Is this already described somewhere?

[comps]

Just a small clarification:

...
The point is - there probably is some standard way to modify ostree/bootc images that a customer would use, and maybe that's the approach testing-farm should use. Pre-install tmt on the images and have helper scripts in standard /usr/libexec (or in /usr/bin, why not).

* not testing-farm, but tmt itself. tmt "installs" these scripts on guests on its own before running tests on them.

* tmt itself does not need to be preinstalled in images, as tmt runs on different machine, connecting to one spawned from the image.

Sure, and I think tmt itself should keep doing that (for any regular non-ostree systems), and maybe this GH issue has a good point that tmt should find a new more compatible default, if possible.

I'm just pointing out that (IMO) it's an uphill battle and maybe a better solution would be to modify the base image to be testing-compatible, whatever that means, rather than trying to "hack" it from a running ssh session.

(Similarly to how we used to test SELinux MLS by adding a testing user + permissive domain that could change role to anybody. Or how we allow root login because using sudo for everything is PITA, even if tmt supports it.)

[thrix]

@comps well, trying now full tests, but if you look at the code:

https://github.com/teemtee/tmt/pull/3229/files

It is not that hackish and keeps the test environment setup the same for all current distributions we support.

With what you advise, would mean we would have a requirement outside the power tmt, seriously hitting usability (are you testing on RHEL image mode? did you forget to add tmt scripts?). We try to avoid exceptions if possible, and that is what we are doing here.

[thrix]

I am not sure whether we should not make a step back and first outline how real tests (with their runtine dependencies) are going be executed. Is this already described somewhere?

Not sure, I completely follow your thoughts @kkaarreell, the tests will be executed the same way as in RHEL package mode. Note that we need this change to make at least reservations work, so we can get our users an easily accessible environment in Testing Farm for experimenting. This is a low-hanging fruit.

[thrix]

Thank you for all the discussion here, I am thinking about taking this approach rather, to make this change less intrusive:

1. for non rpm-ostree test environment, we would default to /usr/local/bin and omit adding the profile.d script

2. for rpm-ostree-based test environment, we would default to what has most votes here

If the distribution is using rpm-ostree we would detect by checking if /run/ostree-booted exists.

[kkaarreell]

FTR, I had a discussion with Miro today and we have agreed on a reasonable plan going forward. I am OK with this change.