Java Judge: Time_limit not working

[JSteegmans]

As the title says: it seems that the timeout is not working: it is provided in the testscript and should be a possible config option, however when looking at the run script it seems that the "time_limit" variable is never read, nor is it passed to the java command.

I have two questions:

1. Could we possibly fix this so we can set a timeout and don't have students code testing indefinitely
2. Would the timeout be per test function or for the entire testsuite? I assume for the entire suite but I'd like to hear what you all consider to be logical

[bmesuere]

Hi,

I didn't write the Java judge myself, but I quickly checked the code and it seems you are right: the value isn't used by the judge itself.

The variable is listed because this is a value that is passed from Dodona to the judge. It is the timeout after which the docker container running the tests will be killed by Dodona and a time limit exceeded status will be shown to the students. You thus don't have to worry that bad code will be running forever. If you create an exercise, you can choose after how many seconds it will be killed by setting the appropriate config value.

Some judges use this value themselves as well to proactively terminate testing to provide a more user friendly message to the student, but this is not the case with the java judge.

[ninewise]

The Java judge indeed relies on the timeout mechanism built into Dodona. This will mark the last test started within the time limit (which did not also end within the time limit) as failed with a message clarifying the overall time limit was exceeded.

It is however also possible to write tests with individual time limits. I believe @rien has experience.

Do note that any timing of tests will not be accurate. The load on the runner servers has a large impact. Consider it more of a 'I still need time to run other tests' or a 'this should roughly take a second and surely no more than 10' rather than a 'the submissions fails if it takes more than 3 seconds'.

[rien]

With JUnit you can indeed specify timeouts per testcase like this:

import org.junit.Assert;
import org.junit.Test;

public class TimeTest {
    @Test(timeout = 1000) // 1000 ms = 1 second
    public void fooTest() {
          foo();
    }
    @Test(timeout = 2000) // 2 seconds
    public void barTest() {
          bar();
    }
}

More information about this is on their wiki page.

But as @ninewise mentions, the runtimes on Dodona have a very high variability, so it is only good for filtering the really slow solutions (we normally multiply the expected runtime by 10).

If we want to test the time complexity of algorithms, we like to create snitch-classes which count how many times an operation (e.g. list.get(i) has occurred and compare if this is below a certain expected value. This is much more reliable time-wise.

For example, if we want students to implement binary search on a list, we use a Sequence class and its snitching descendant MonitoringSequence:

import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;

public class Sequence {
    private int[] numbers;

    public Sequence(int... numbers) {
        this.numbers = numbers;
    }

    public int size() {
        return numbers.length;
    }

    public int get(int index) {
        return numbers[index];
    }
}

public class MonitoredSequence extends Sequence {
    private Map<String, Integer> callCounts;

    public MonitoredSequence(int... numbers) {
        super(numbers);
        callCounts = new HashMap<String, Integer>();
    }

    private void countCall(String method) {
        callCounts.compute(method, (k, v) -> v == null ? 1 : v + 1);
    }

    public int callCount(String method) {
        return callCounts.getOrDefault(method, 0);
    }

    public int size() {
        countCall("size");
        return super.size();
    }

    public int get(int index) {
        countCall("get");
        return super.get(index);
    }
}

Which allows us to create tests like this:

import org.junit.Assert;
import org.junit.Test;

public class BinarySearchTest {
    @Test
    public void binarySearchTest() {
          Sequence seq = new MonitoredSequence(randomNumbers(1000));
          int index = randomIndex(1000);
          int found = binarySearch(seq, seq.get(index));
          Assert.assertEquals("Your algorithm is not correct.", index, found);
          Assert.assertTrue("Your algorithm is not Θ(log n))", seq.callCount("get") < 50);
    }
}

This is a bit more work, but much more reliable.

Let us know if you want some more information about this.