-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement the check-result
key and store the original result
#3185
Comments
General comment on the way check works: It calls Are there concerns switching over to reading the journal instead? And fallback to |
Good point, thanks! Check is a optional step and not necessarily 'dmesg'. For example, there can be "journal" check plugin implemented. That said, it was discussed before. I'd like to look into incorporating |
@psss I'm thinking whether it wouldn't make sense to have
wdyt? |
@martinhoyer Would it make sense to use the |
@happz hmm not sure. Afaik we don't want to change the result of the check itself. Let's see if @psss wants check-result or is open to this per-check setting idea in the first place :) |
True, wrong place,
That would be ugly, not just the implementation but also the specification.
Per-check control can be added later, I believe, if needed. |
I was thinking more about doing one or the other. i.e. either specifying it as part of check object(s), or the original idea. I'm having a hard time visualizing it how it would work with subresults and when users would like to just do a simple |
That's why I would start with a single
I for one would start with the simpler case, let's hear from others.
|
Thanks @happz! I'll work on it today. |
I'd agree with @happz that introducing new key like Which leads to the interesting idea of not introducing a new key name test: test.sh
check:
- name: avc
result: xfail This could be useful for covering expected I also see @happz's point, starting with a single "global" |
I would find it useful to have the setting per test. I am not an active |
That should be supported in any case, the debate above is mostly about whether the control should be per test, or even more granular, per test check. The most granular way, "per test check", seems to be getting most support. |
Yes :-) |
Similarly to the
result
key it would be good to have acheck-result
key which would decide how the check results should affect the final result of a test, e.g.respect
orignore
. The original result of the test execution itself should be stored in theresults.yaml
as well so that it can be inspected if needed.There is an expectation that check results should by default affect the final result so that possible problems are revealed for example in Testing Farm viewer, so the default values should probably be
respect
.The text was updated successfully, but these errors were encountered: