Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report on test coverage #675

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions src/cli/test/reporters/coverage.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
import type { AggregatedResult } from '@jest/test-result';
import istanbulReport from 'istanbul-lib-report';
import istanbulReports from 'istanbul-reports';

export const createWriteOverride = () => {
const chunks: Uint8Array[] = [];

const output = () => Buffer.concat(chunks).toString('utf8');

const write = (
buffer: Uint8Array | string,
encodingOrCb?: BufferEncoding | ((err?: Error) => void),
cb?: (err?: Error) => void,
) => {
chunks.push(
typeof buffer === 'string'
? Buffer.from(
buffer,
typeof encodingOrCb === 'string' ? encodingOrCb : undefined,
)
: buffer,
);

if (typeof encodingOrCb === 'function') {
encodingOrCb();
} else {
cb?.();
}

return true;
};

return {
output,
write,
};
};

/**
* Renders out test coverage using the Istanbul `text` reporter.
*
* This is unfortunately hacky in a couple of ways;
*
* 1. Jest does not support custom coverage reporters (facebook/jest#9112), so
* we rely on the default `CoverageReporter` running before us and use the
* `coverageMap` that it places on the aggregated result state.
*
* 2. `istanbul-reports` does not support writing to a custom stream, so we need
* to temporarily override `process.stdout.write` 😱.
*
* {@link https://github.com/facebook/jest/blob/v27.3.1/packages/jest-reporters/src/CoverageReporter.ts#L103}
//
*/
export const renderCoverageText = (
coverageMap: AggregatedResult['coverageMap'],
) => {
if (!coverageMap) {
// Coverage was not stored on the aggregated result by `CoverageReporter`.
// Maybe `collectCoverage` / `--coverage` was not specified.
return;
}

const reportContext = istanbulReport.createContext({
coverageMap,
});

const overrideWrite = createWriteOverride();

const originalWrite = process.stdout.write.bind(this);
process.stdout.write = overrideWrite.write;

try {
istanbulReports.create('text').execute(reportContext);
} finally {
process.stdout.write = originalWrite;
}

return overrideWrite.output();
};
25 changes: 24 additions & 1 deletion src/cli/test/reporters/github/index.ts
Original file line number Diff line number Diff line change
@@ -1,25 +1,30 @@
import type { Context, Reporter } from '@jest/reporters';
import type { AggregatedResult } from '@jest/test-result';
import stripAnsi from 'strip-ansi';

import * as Buildkite from '../../../../api/buildkite';
import * as GitHub from '../../../../api/github';
import {
buildNameFromEnvironment,
enabledFromEnvironment,
} from '../../../../api/github/environment';
import { log } from '../../../../utils/logging';
import { renderCoverageText } from '../coverage';

import { generateAnnotationEntries } from './annotations';

export default class GitHubReporter implements Pick<Reporter, 'onRunComplete'> {
async onRunComplete(
_contexts: Set<Context>,
{ testResults }: AggregatedResult,
{ coverageMap, testResults }: AggregatedResult,
): Promise<void> {
if (!enabledFromEnvironment()) {
return;
}

try {
const coverage = renderCoverageText(coverageMap);

const entries = generateAnnotationEntries(testResults);

const build = buildNameFromEnvironment();
Expand All @@ -35,11 +40,29 @@ export default class GitHubReporter implements Pick<Reporter, 'onRunComplete'> {
? '`skuba test` passed.'
: '`skuba test` found issues that require triage.';

if (coverage) {
const buildkiteOutput: string = [
'`skuba test` coverage:',
Buildkite.md.terminal(coverage),
].join('\n\n');

await Buildkite.annotate(buildkiteOutput, {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@samchungy how do you feel about having this reporter support both Buildkite and GitHub 😬

If we want coverage integrations across both, it may be wasteful to define two reporters and to run renderCoverageText twice.

Copy link
Contributor

@samchungy samchungy Nov 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love it but renderCoverageText is kinda costly I guess. Even the loop processing this I guess.

Do reporters run in series? I wonder if we could store it in memory? 🤔

If we do this -> might want to rename the reporter? maybe reporters/annotate/index.ts?

context: `skuba-test-${
displayName ? `-${Buffer.from(displayName).toString('hex')}` : ''
}`,
scopeContextToStep: true,
style: isOk ? 'success' : 'error',
});
}

await GitHub.createCheckRun({
name,
annotations,
conclusion: isOk ? 'success' : 'failure',
summary,
text: coverage
? Buildkite.md.terminal(stripAnsi(coverage))
: undefined,
title: `${build} ${isOk ? 'passed' : 'failed'}`,
});
}
Expand Down