How to handle and present guest logs

[happz]

TL;DR: I wanted to test the handling of guest logs in Artemis, and while artemis plugin lets me ask for logs: how should tmt make guest logs accessible to me as a tmt user??

There may be logs related to guests and their creation by the infrastructure hosting them. There is a so-called "console log", be it a virtual or actual console server, then there is Beaker with several logs saved during provisioning, dmesg, storage.log, etc.

Then there are users who are very interested in getting their hands on these logs, users like folks testing upgrades or kernel, folks that often render their guests unresponsive and need to dig deeper. In Artemis, we are working on exposing these logs, because the need for them is real, and we do know how to present them in Testing Farm, but, the question is, how to handle this with tmt alone?

We do have a couple of plugins that can offer interesting logs, beaker, I bet virtual & Testcloud could be convinced to share VM's console as well, artemis does have plugin support even though the backend is lacking behind a bit - how should we capture/download/fetch/acquire/whatever the logs, how do we present them?

• shall there be an optional --fetch-log key/option for the user to express their desire?
• fetching logs while tests are running, fetching them only after execute, before releasing guests?
• saving logs somewhere under the provision directory in the workdir would be trivial, a directory per guest name?
• is there a place where tmt could report the locations of those logs? It does not seem to fit results.yaml job description, but who knows

My two cents, after observing AWS and Beaker for some time:

• it's often useful to get a post-mortem log. Beaker keeps the console.log updated continuously, AWS offers this only with Nitro-based instances, Xen-based dump console log only after interesting life-time events (power on, power off, reboot, instance terminated, etc.). Fetching logs in runtime might be useful to have at least something should the infra alone break down, but having a console log recording broken Grub config after reboot would be appreciated more.
• that said, a reachable plaintext file is "good enough" - as long as we can point user to it, those interested can less it when needed. It would be nice to have a place similar to results.yaml for these logs though - we do have $workdir/run.yaml, would it make sense to add logs list into it, with the same semantics as the one in results.yaml? I or report plugins could peek into run and display/create pointers to all logs.

[happz]

We discussed this feature with @skycastlelily and she's willing to work on exposing guest logs to users. To not get crushed by the vast possibilities, I proposed to start small and extend the coverage:

• an internal API will be created. We need to extend tmt.steps.provision.Guest with at least one method, let's call it acquire_log() for now, which would be responsible for getting the content of a given log. The initial signature might be acquire_log(self, log_name: str) -> Optional[str]: log names are quite a free-form thing, besides a couple of well-known names - console, maybe dmesg - the majority of names would be dictated by the plugin. For example, Beaker can deliver more logs than a podman for sure.
• for now, it's either the content or nothing. In the future, we will need a way to signal "log not supported", to avoid constant calls to guests and getting None all the time. But since we would start with console, it seems like a good simplification.
• the second method to add to the Guest class would be one to fetch all available logs - let's say at the end run, in finish or after a crash, this one would be useful to get all available data. Let's say acquire_all_logs(self) -> dict[str, str] - log name => log content mapping. There is no need for Optional, either the log is supported or it's simply not added to the mapping.
• the third part would be a simple "fetch me guest logs" helper, a simple function that would be given a guest and a directory, and it would acquire logs and save them in files in said directory. In the future, this method might get incorporated into the Guest class, or into the Step class, but for now, I'd make it a stand-alone function. We would then call it from interesting places during prototyping, e.g. at the end of finish before releasing guests. This would make the change minimal but let us hook into tmt's workflows easily.

At first, there would be no recognized logs, all plugins would simply return None for all names, and we would improve that. beaker plugin offers plenty of logs, but it has long provisioning times, artemis is quite complicated, virtual does not seem to offer easy access to VM's console log (will investigate), container is quite a special as it has no log besides stdout/stderr we already capture. connect will probably need custom scripting like the one we have for reboot handling, and local offers a big collection, but will need sudo.

So, from these, it seems to me beaker and local might be the lowest-hanging plugins for extending the basic no-op implementation. beaker guests can be simply not returned and re-used. We will implement the basic fetching of Beaker logs, make sure they save correctly, and extend the support further.