Add Explore Tests Button

[rprouse]

We should be able to explore and run individual tests. See #14.

[CharliePoole]

It seems as if this is moving towards a mini-gui. If that's the case, an explore tests button may be redundant. That is, a gui normally just displays the tests in the first place.

Of course, I may not understand, not being able to run this. :-(

[tofutim]

Charlie, the nunit runner looks great. I would also like to be able to run individual tests, or at least individual test suites. Our particular tests have about 300 tests, and sometimes I don't want to run all the tests.

[tofutim]

In the meantime it might be sufficient to be able to re-run failed tests for debugging purposes. I'm not even sure how to do that.

[rprouse]

@tofutim, I think the ability to explore and run individual tests will be one of the first features I will add after we get this released.

[tofutim]

And also watch progress as the tests are run. Mine runs for about 3 min and the UI is blank and I wonder what is going on! :)

[tofutim]

Did you ever play with the TouchRunner in iOS from Xamarin - there are bugs but it does have a kind of an explorer.

[rprouse]

If by the TouchRunner you mean the NUnitLite based runner that shipped with Xamarin, then yes and that is roughly what I was thinking.

[tofutim]

Yes, that one works, but it is a bit buggy in places.

On Tue, Nov 17, 2015 at 9:39 AM, Rob Prouse [email protected]
wrote:

If by the TouchRunner you mean the NUnitLite based runner that shipped
with Xamarin, then yes and that is roughly what I was thinking.

—
Reply to this email directly or view it on GitHub
#29 (comment).

[borrrden]

More than a bit buggy. I've had to manually change and rebuild it several times just to get functionality which I consider fairly basic (I.e. properly running parameterized test cases or methods). I was hoping to move toward this runner but the one big button that says "run all tests" is not granular enough. What is the status of this?

[rprouse]

@Borrden, as @CharliePoole says, it may not end up being an explore button, that may just happen automatically at load. I am working on changes in the main NUnit project that will likely be used to enable this issue.

You mentioned you had problems with parameterized tests and methods. Can you enter issues, elaborate or even better contribute your fixes?

[borrrden]

Sorry I was referring to the NUnit 2.x runners for iOS and Android when I said that. I wanted to get off of them because of their buginess but this runner doesn't have the feature set I need yet. That's what I wanted to say. I don't think it will do much good to file issues about that :)

[rprouse]

@borrrden you're right, filing issues against the Xamarin 2.x runners won't help since Xamarin wrote them, not us 😄 I've emailed them a few times to see if they would help by donating their code, but no luck so far.

Is running individual tests the main feature you are looking for, or are there other misses that we should be looking at?

[borrrden]

Running individual tests (or test fixtures) is the key functionality I want. It takes quite a while to run all of the tests in our suite and I want to repeat failing tests, etc. By "donating their code" do you mean license wise or the source? The source is out there, and I have forked it for my organization (including my edits, but there are some edits on NUnitLite that were small enough to simply write a patch file for):

https://github.com/couchbasedeps/Touch.Unit
https://github.com/couchbaselabs/Andr.Unit

[rprouse]

Thanks for the links to the code, I didn't realize it was out there, but then again, I didn't look very hard 😄

It is under the Apache license with is open, but incompatible with the MIT license. That and the fact that it is the old NUnitLite make it a bit difficult to use outright, but I can draw ideas from it.

How would you envision exploring and running individual tests working? My thought is that you would launch on the device and it would automatically explore all tests, populating the test list. From there you would drill down to the test and run tests individually.

The issue with the current runner is that it flattens the test hierarchy to make it 'easier'. I think this scenario might end up making it harder since there will be so many tests.

Care to take a stab at a verbal design based on your usage and needs?

[borrrden]

I have not had problems with the way that the 2.x based runners work (other than the fact that they don't work all the time). The idea, to me, is sound (drilling down into tests). Let's take a small example:

I have a TestFixture class called MyTest. It has two configurations (TestFixture("TypeA") and TestFixture("TypeB")). Inside there is a test method TestMethod that also has two configurations (TestCase(true) and TestCase(false)). I would expect to see the following

Start with a list of assemblies that the runner has loaded. On click:

Show a list of test fixtures. Either combined (i.e. showing just MyTest) or not (i.e. showing MyTest("TypeA") and MyTest("TypeB").

Upon clicking one of the entries, if the former was combined then show a list of configurations (MyTest("TypeA") and MyTest("TypeB")) otherwise,

Show a list of test methods from that fixture. Again, either flattened (just showing TestMethod) or not (showing TestMethod(true) and TestMethod(false)). If flattened, then show the configurations upon click. Otherwise run the test on click. If the test passes, update the cell with a success message (preferably with green color), otherwise push into a screen that shows the details of the failure (and also update the cell in red about the failure). If the test is a status other than success or failed then update the cell with that status in a gray color (or some other color other than red or green).

In addition, each step should have a button on the bottom that will run all of the tests that are current displayed on the screen (along with any child tests, so if Run All is clicked on the page with all of the fixtures, all fixtures in all configurations get run with all test cases with all parameters). Failures should not present a new screen in this case, but simply update the cell.

Bonus: If you can provide a way to distinguish between rereunning a test case, and viewing the test case's failure details once more that would be cool!

[1Cor125]

Any update on the capability to rerun failed tests and/or explore and run specific tests? Is this on the roadmap?

[velocitysystems]

+1

[rprouse]

Sorry, this project isn't getting the love or updates it deserves. We do accept pull requests though 😄