testrunner - a I/O-based testing tool for C and C++ programs
- -n, --no-ws-hints
-
Disables whitespace-hints in HTML report
- -p, --protected-mode
-
Runs in protected-mode, with details of protected testcases redacted
- -h, --help
-
Prints help information
- -V, --version
-
Prints version information
- -c, --config <CONFIG_FILE>
-
Sets testcase config file [default: test.toml]
- -J, --jobs <JOBS>
-
Sets number of tests to run in parallel
- -o, --html-output <HTML_OUTPUT>
-
Generates HTML report [default: testreport.html]
- -j, --json-output <JSON_OUTPUT>
-
Generates JSON report [default: testreport.json]
- --sudo <USER>
-
Runs tested program as user <USER>
The testrunner requires a config-file specifying the testcases to be run. The name and/or path of this file may be set using --config, which defaults to test.toml.
By default, the testrunner will generate a HTML testreport testreport.html in the current directory. The option --html-output may be used to change the name and/or path of this file.
For processing of the testresults using another program, a JSON testreport can be generated using --json-output.
The option --protected-mode allows generating a restricted testreport. Details of testcases marked as protected will be omitted in the HTML testreport. The JSON testreport will always include all information.
By default, the testrunner will run multiple testcases in parallel, depending on the number of logical CPU cores. The option --jobs may be used to manually set the number of testcases to run in parallel.
The whitespace-hints in the HTML testreport may add a significant space-overhead; the option --no-ws-hints allows to disable these whitespace-hints to reduce the file size.
The option --sudo may be used to run the tested program under another user. In this case, the testrunner itself must be run as root to work properly.
testrunner is a input/output-based testing tool; it aims to provide a comprehensive tool for checking programs for correctness in the context of teaching C and C++ (correct program behaviour on I/O, compiler warnings, memory leaks, …), while also generating an easy-to-understand testreport.
Testcases are configured in a TOML configuration file, along with additional files containing input and reference output
of the testcases. The configured input will be sent to the tested program, capturing its output and comparing it with the
expected reference output. An additional file may be checked as well.
The programs exit-code is compared with an expected exit-code.
Valgrind may be used to detect memory usage errors and memory leaks in the tested program.
The testrunner can compile a project using a Makefile, in which case it is also able to detect compiler warnings during compilation. Currently, parsing the output of GCC and Clang is supported; other compilers may work as well, as long as their output format is similar enough.
Testresults are compiled into a HTML testreport. This testreport includes all information gathered by the testrunner,
prepared in a user-friendly format aimed at less experienced users.
A summary provides an overview of all testresults; detailed results are provided for testcases not marked as protected.
The programs actual output and the expected output are shown in a two-way line-diff,
including inline-diff-hints and whitespace-hints for easier detection of small differences in the output.
A hexdump-like format is used for the two-way diff of binary files.
The testrunner is meant to be distributed to users along with public testcases for local testing. Using protected-mode, public and secret/private testcases may be tested on a central testsystem, providing the users only with basic information about the results of these secret/private testcases.