Use Caraya to generate JUnit test reports for Azure Pipelines.
Explore the docs »
View Demo
Report Bug
Request Feature
Caraya JUnit for Azure Pipelines is an extension for Caraya that generates JUnit test reports compatible with Azure Pipelines. Test reports are generated according to the JUnit result format and can be published to Azure Pipelines using the Publish Test Results v2 task.
Caraya already includes JUnit support. Why do we need another JUnit extension for Azure Pipelines?
Here are a few reasons:
-
Azure Pipelines does not display test suites
This is a known issue and there is a suggestion to fix it. Please vote for it 💖 -
Assertion labels are indistinguishable
By default Caraya uses the Assertion VI name when the assertion label is empty (which it is by default). For example, "Assert Equal Value_Variant". This label is not very helpful, especially the in the default view.
File view is a little more readable.
-
Test duration for all test cases is zero
By default, the test duration of a test case in Caraya is zero unless the user explicitly monitors the test duration and provides it to the assertion function. -
Error message and stack trace are meaningless
The default JUnit report produces meaningless error messages and stack traces. A typical failure has the error message "test failure" and stack trace "FAIL".
-
Not executed assertions are reported as skipped
When an error is passed to an assertion function, it is reported as "skipped" in the default JUnit report. That was probably a design decision. Skipped tests, however, do not count towards failed tests in Azure Pipelines. They simply appear under "others" (just like disabled tests).
Why not improve the JUnit test report of Caraya?
I've thought about it and decided against it because the differences introduced by this project would likely be considered breaking changes by other users.
How does this project improve test reports in Azure Pipelines?
This project is build from scratch and aims at providing useful test reports in Azure Pipelines. Here are a few things it improves:
-
Test results are reported with their fully qualified name
This makes the default view much more verbose.
-
Test duration is calculated for every test case
Test duration is calculated from the timestamp of each assertion. User specified test durations still take precedence, of course. -
Meaningful error messages and stack traces are reported
This extension tries its best to figure out the best error message and stack trace for a failure, based on assertion results and errors.
-
Any failure is considered a failed test
This includes errors passed to an assertion function as well as any error reported by an assertion. Tests are only considered skipped when no error occurred and the assertion was not executed.
To get a local copy up and running follow these simple steps.
- LabVIEW™ 2013 or later
- VI Package Manager
- Clone the repo
git clone https://github.com/logmanoriginal/caraya-junit-for-azure-pipelines.git
- Install packages
Or apply
start .vipc
.vipcmanually.
Here is an example that uses the JUnit report generator to create a test report for a project.
See the open issues for a list of proposed features (and known issues).
Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Keep in mind that LabVIEW™ VIs are binary files, which are difficult to merge.
- Only change a single VI or library.
- Avoid conflicts with other pull requests (don't work on the same libraries or VIs).
- Send VI Snippets (via issues) instead of pull requests when possible.
Distributed under the BSD-3-Clause License. See LICENSE for more information.
Project Link: https://github.com/logmanoriginal/caraya-junit-for-azure-pipelines