Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: nested test case #612

Open
aherlihy opened this issue Dec 8, 2022 · 1 comment
Open

Feature request: nested test case #612

aherlihy opened this issue Dec 8, 2022 · 1 comment

Comments

@aherlihy
Copy link

aherlihy commented Dec 8, 2022

Hi, I would like to make a feature request to allow for nested test cases, or at least some way to separate tests by namespace (like "context" or "describe" in JS testing frameworks like mocha). My problem is that I'm composing test cases programmatically so as of right now, only 1 level of nesting means I have to bunch a series of asserts into one test case when they can fail independently. These tests are naturally very hierarchical and I have relatively computationally expensive set-up for those series of test cases so I would prefer not to have to rerun the entire set up for each test.

As with Mocha, I would not expect to be able to mutate state defined in an outer test case from an inner test case - I would expect that to fail or to have undefined behavior.

If you believe I can accomplish what I want to do without nesting, then I'd be very appreciative of any suggestions! I will do my best to explain my use case in more detail below.

I'm testing a compiler and I have a set of directories representing a program to test + input state + an expected output. In the MUnit test class, I'm using a Fixture to iterate over the directories and construct a context for each directory. Then in the test itself, I loop over the queries that need to be run, but because nested isn't supported I can't have each query fail independently. However since the context is created in a fixture, I can't move test down a layer because you can't access fixures outside of "test" statements. Further, the compiler itself has various pluggable layers, so I would love a way to nest the tests per layer of the compiler. What I'm doing right now is forcing everything to be at the top level but naming the classes so they are "nested" by in the name, but that ends up with ridiculously long hierarchical test names which doesn't let me take advantage of the nice nesting UI of IntelliJ.

So instead of these lines I would love to be able to write (and nest further, but simplified for this example):

   directories.foreach(testdir => {
      test(testdir) {
        val g = graph()
        g.queries.map((hint, query) => {
          test(hint) { // these can fail independently
            assertTrue(query.solve())
          }
      }

It would be great to be able to have fixtures for each level, but I would be happy even with simply a nested test namespace alone.
Thank you!

@SethTisue
Copy link
Contributor

SethTisue commented Sep 4, 2024

for further context, utest's nesting support is described at https://github.com/com-lihaoyi/utest?tab=readme-ov-file#nesting-tests

though there is this substantial caveat:

Note that tests within the suite can nested within each other, but only directly. E.g. you cannot define tests within if-statements or for-loops

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants