Expecting tests to fail

[leingang]

Is there a way to mark a test as expected to fail?

Specifically, I have some fontspec code which selects a font that I don't have on my machine. It's not going to succeed until I get that font, and I don't have a timeline for getting it. I don't want my checks to flag this test as failing every time.

I suppose I could just save the test and its loudly complaining log file, but that would make it look like the test succeeded. I don't really want that either.

I know you can do such a thing in pytest, which is why I'm asking.

[josephwright]

You just save a .tlg for the failure: the test then passes as you get the expected outcome. That's the same as testing error messages or similar: exit status for TeX is non-zero, but the test passes.

[davidcarlisle]

an alternative is to put the font tests in a separate directory and config-whatever.lua but don't specify that config in your main build.lua

[leingang]

I think this is similar to how pytest does it. The expectations (usually pass, but not always) are layered over the tests with decorator functions. In both setups, the test-driving suite needs to know what you want to happen.

[zauguin]

What's actually the expected result here? Do you expect the failure or do you want to ignore it?

In other words, if the test would succeed against your expectation, should that lead to a test failure?

[leingang]

That question clarified everything in my head, thanks. If my test is looking for a font, and I don't have the font, and it starts to succeed against my expectation, it would mean either:

• the missing font has been installed. In this case, I would save the result of the successful run and expect success from then on.
• or the test is flawed. In this case, I'd revise the test until it failed according to expectations, and save the result.

In both cases, it requires my attention, which is what tests are for after all.

For now, I can put something in the .lvt file which reminds me that the expected result is an error, and save the result.