All test passed, but it displays "Overall Failed"

[thenderson21]

When I run my unit test it displays "Overall Failed" regardless of the tests status.

[rprouse]

Very odd. It is working for me. It shows two tests. Are they simple enough to copy here so that I can try them? If you drill into the individual tests, is there any info in them that may shed some light?

[CharliePoole]

@rprouse I don't know how much of the runner code resembles our nunitlite code, but in the past such an error has generally indicated that there was a failure of one time startup rather than a test case. If you are saving the XML result, it should indicate what happened.

[thenderson21]

I not sure how to get the xml output. Unfortunately the tests are for a library that I isn't public. The test themselves were pretty simple.

    [TestFixture, Description ("Testing SocketCluster Client")]
    public class SocketClusterClientTests
    {
        SCSocket SocketClient;

        bool Connected = false;


        [OneTimeSetUp]
        public void SetUpFixture ()
        {
            var options = new SCClientOptions { Hostname = "ws://socket.talkfusion.com", Logging = SCLogingLevels.Debug };
            SocketClient = SocketCluster.Connection (options);

            SocketClient.Connected += (obj) => Connected = true;
            SocketClient.Disconnected += (arg1, arg2) => Connected = false;

        }

        [SetUp]
        public async Task SetUpTest ()
        {
            await SocketClient.ConnectAsync ();
        }

        [TearDown]
        public async Task TearDownTests ()
        {
            if (SocketClient.State != SCConnectionState.Closed) {
                await SocketClient.DisconnectAsync ();
                if (RetryUntilSuccessOrTimeout (() => !Connected, TimeSpan.FromMilliseconds (1000))) {
                } else {
                    throw new Exception ("Cound Not Tear Down test");
                }
            }
        }

        [Test, Description ("Testing connection to SocketCluster Server.")]
        public async Task Connecting ()
        {
            if (RetryUntilSuccessOrTimeout (() => Connected, TimeSpan.FromMilliseconds (500))) {
                Assert.That (SocketClient.State, Is.EqualTo (SCConnectionState.Open), "SocketClient.State is incorrect.");
                Assert.That (SocketClient.Id, Is.Not.Null.And.Not.Empty, "Missing or Invalid SocketClient.Id");
            } else {
                Assert.Fail ("Failed to connect to the server in under 500 miliseconds");
            }
        }

        [Test, Description ("Testing disconnection from SocketCluster Server.")]
        public async Task Disconnecting ()
        {
            if (RetryUntilSuccessOrTimeout (() => Connected, TimeSpan.FromMilliseconds (500))) {
                await SocketClient.DisconnectAsync ();

                if (RetryUntilSuccessOrTimeout (() => !Connected, TimeSpan.FromMilliseconds (500))) {
                    Assert.That (SocketClient.State, Is.EqualTo (SCConnectionState.Closed), "SocketClient.State is incorrect.");
                } else {
                    Assert.Fail ("Failed to disconnect from the server in under 500 miliseconds");
                }
            } else {
                Assert.Fail ("Failed to connect to the server in under 500 miliseconds");
            }
        }

        bool RetryUntilSuccessOrTimeout (Func<bool> task, TimeSpan timeSpan, int interval = 25)
        {
            bool success = false;
            int elapsed = 0;
            while ((!success) && (elapsed < timeSpan.TotalMilliseconds)) {
                Thread.Sleep (interval);
                elapsed += interval;
                success = task ();
            }
            return success;
        }
    }

Today is seams random. Maybe 1 in 10 runs, with give me that show up as failed with no failed tests. I have also noted that the Overall Success or Fail mess seams to be missing letters depending on platform. See droid image above.

[rprouse]

It looks like your OneTimeSetup is failing occasionally. I will need to add functionality to surface those errors.

As for the UI issues on some platforms, I am aware of those. I haven't tested and tweaked the UI on enough platforms yet.

Thanks for the report...

[rprouse]

Entered #44 to track the UI truncation issues.

[rprouse]

Actually, I think it is the exception that you are throwing in your teardown that is causing the problem.

[thenderson21]

I thought of that too. It show up as a test failure on the individual tests, with the stack trace and exception showing up in the test failed screen.

[thenderson21]

It looks like if there is an exception in [OneTimeSetUp] or [SetUp] it Shows overall fail even if the tests succeed. The exceptions are hidden. I Don't see them in the application output, or in the application anywhere.

[OneTimeSetUp]
public void SetUpFixture (){
    throw new Exception ("foo");    //Is hidden but causes the overall status to fail
}

[ChrisMaddock]

I can only repro this with a [(OneTime)TearDown] exception.

Repro below:

[TestFixture]
    public class TestCaseSample
    {
        [OneTimeTearDown]
        public void OneTimeSetUp()
        {
            throw new Exception("Fails");
        }

        [Test]
        public void TestAddWithResult()
        {
            Assert.IsTrue(true);
        }
    }
}

[CharliePoole]

That's as expected if you are only reporting test cases. All the tests pass and are reported. Then OneTimeTearDown runs and throws an exception. The suite is reported as a failure but (I'm guessing) your runner does nothing with that report.

The NUnit console runner historically did the same thing. We had to add code to specifically report a suite failure if the site was TearDown. Look at the console runner's event listener as an example.

[ChrisMaddock]

I see, thanks Charlie. We'll need to decide what we want that to look like for the xamarin runner.