Coverage is reported as decreased in PRs when it really isn't

[notmessenger]

(Sorry for the reference to Ciena-internal PR numbers but they might be needed for additional context in the research of this, and I have included screenshots of the relevant information for discussion of this in an open manner since this is an open-source repository)

ISSUE

In MCP-UI PRs the coverage numbers are sometimes being presented as having decreased when in reality they have not ("SCENARIO 1" below). This is occurring when a PR is rebuilt because of merges into the master branch. Sometimes after another rebuild then it reports that there has been no change.

In other scenarios ("SCENARIO 2" below) a PR is submitted that makes changes to a config file, nothing that would affect coverage percentages, and it will report that coverage has dropped but then it is rebuilt and it does not report the same.

SUSPECTED CAUSE

Based on the patterns in the reporting numbers, some of which are presented below, and the scenarios described above, I suspect that this is due to a rounding error either in our code when we make the comparisons or in the underlying code generating the coverage reporting statistics.

SCENARIO 1: MCP PR 583

The first message written of

Code Coverage: 47.54% (increased 0.09% from 47.45%)

reflects the coverage reporting for this PR and everything is as expected.

The second message written of

Code Coverage: 47.52% (no change)

is the result of this PR being rebuilt due to an update to the master branch. Sometimes these numbers might change, sometimes they might not - just depends on what the change was. So this is an expected result as well.

The third message written of

Code Coverage: 47.52% (increased 0.02% from 47.50%)

gets a little interesting because I would not have expected to have seen the "from 47.50%" text but instead "from 47.52%". If the coverage number from the master branch is what is being used (because that is what is being merged in during the build) then it should have been the 47.52% from the second message written.

The fourth message written of

Code Coverage: 47.51% (dropped 0.01% from 47.52%)

is the one that causes me to suspect a rounding bug because this is the number reported, which represent the current code in the master branch combined with the code in this PR. Yet when the master branch is then merged into this PR and the coverage number are ran again there is no longer a drop of 0.01%. This same discrepancy plays itself out in "Scenario 2" below as well.

SCENARIO 2: MCP PR 584

This PR changed a value in the package.json file and the first message written of

Code Coverage: 47.51% (dropped 0.01% from 47.52%)

indicates the coverage has decreased, which should not be the case, as well as it is the same 0.01% difference seen in the fourth message written of "Scenario 1".

The second message written of

Code Coverage: 47.50% (dropped 0.02% from 47.52%)

was a result of the PR being rebuilt due to a manual request to do so (in order to clear the previous build failure due to coverage reporting) and not due to any changes in the master branch. This means that the code being evaluated is EXACTLY IDENTICAL yet the code coverage reporting yields different results.

At the time of this issue being created this PR was undergoing its 3rd build, without any changes to the master build, so the outcome of it can be checked at a later time, but there is enough information currently present and captured to represent the scenario.