Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Same configuration in different react apps is causing failure in mutation tests at all or finds just few tests #4773

Open
Vaclovas opened this issue Mar 5, 2024 · 6 comments
Labels
⁉ Question Further information is requested

Comments

@Vaclovas
Copy link

Vaclovas commented Mar 5, 2024

Question

Why same configuration for a very similar app is causing test failure? Mainly, tests are not even running or only a few tests can be found.

Stryker environment

stryker config:
{
"reporters": [
"html", "clear-text", "progress"
],
"packageManager": "npm",
"timeoutMS": 30000,
"ignorePatterns": ["/src/lib/icons/*.tsx"],
"tsconfigFile": "tsconfig.json",
"mutator": { "excludedMutations": ["BlockStatement",
"StringLiteral",
"ArrayDeclaration",
"ObjectLiteral"] },
"disableTypeChecks": "
/.{js,ts,jsx,tsx,html,vue}",
"testRunner": "jest",
"coverageAnalysis": "perTest",
"tempDirName": "stryker-tmp",
"mutate": [
"/src//**/
.ts",
"/src///*.tsx",
"!
/src//*.spec.ts",
"!
/src/**/*.spec.tsx"
]
}

nx + jest+ react and
"@stryker-mutator/core": "8.2.6",
"@stryker-mutator/jest-runner": "8.2.6",
"jest": "29.4.1",
"jest-environment-jsdom": "29.4.1",
"jest-environment-node": "29.4.1",
"jest-slow-test-reporter": "1.0.0",

Additional context

Here is the first app:
image

here is another app:
image
What is intresting, that even components are covered by unit tests, they are marked as "NoCoverage", jest finds these tests and succesfully adds those to coverage report, but stryker - is failing. Assumption can be, that configuration is bad, but some of the files are found and counted into the report:
image
So, actually, only 6 spec.ts/tsx files are identified from 256 files. I've tried to search for differences, remove specific imports, change naming - nothing happens, tried some recipies from github and troubleshooting guide.

Maybe you could assist with this?

@Vaclovas Vaclovas added the ⁉ Question Further information is requested label Mar 5, 2024
@Vaclovas Vaclovas changed the title Same configuration in different react apps causing in failing mutation tests at all or finds just few tests Same configuration in different react apps is causing failure in mutation tests at all or finds just few tests Mar 5, 2024
@Vaclovas
Copy link
Author

Vaclovas commented Mar 6, 2024

ok, seems like I've found the issue which is skipoing those tests - this is aliased imports... Can you provide info, why those are not working?

@nicojs
Copy link
Member

nicojs commented Mar 9, 2024

Hmm interesting. I think this is an issue with Jest. Could you try running Stryker with this change in your config?

{
  "reporters": ["html", "clear-text", "progress"],
  "packageManager": "npm",
  "timeoutMS": 30000,
  "ignorePatterns": ["/src/lib/icons/*.tsx"],
  "tsconfigFile": "tsconfig.json",
  "mutator": { "excludedMutations": ["BlockStatement", "StringLiteral", "ArrayDeclaration", "ObjectLiteral"] },
  "disableTypeChecks": "/.{js,ts,jsx,tsx,html,vue}",
  "testRunner": "jest",
  "coverageAnalysis": "perTest",
  "tempDirName": "stryker-tmp",
-  "mutate": ["/src//**/.ts", "/src///*.tsx", "!/src//*.spec.ts", "!/src/**/*.spec.tsx"]
+  "mutate": ["/src//**/.ts", "/src///*.tsx", "!/src//*.spec.ts", "!/src/**/*.spec.tsx"],
+  "jest": {
+    "enableFindRelatedTests": false
+  }
}

@Vaclovas
Copy link
Author

Vaclovas commented Mar 13, 2024

Hmm interesting. I think this is an issue with Jest. Could you try running Stryker with this change in your config?

{
  "reporters": ["html", "clear-text", "progress"],
  "packageManager": "npm",
  "timeoutMS": 30000,
  "ignorePatterns": ["/src/lib/icons/*.tsx"],
  "tsconfigFile": "tsconfig.json",
  "mutator": { "excludedMutations": ["BlockStatement", "StringLiteral", "ArrayDeclaration", "ObjectLiteral"] },
  "disableTypeChecks": "/.{js,ts,jsx,tsx,html,vue}",
  "testRunner": "jest",
  "coverageAnalysis": "perTest",
  "tempDirName": "stryker-tmp",
-  "mutate": ["/src//**/.ts", "/src///*.tsx", "!/src//*.spec.ts", "!/src/**/*.spec.tsx"]
+  "mutate": ["/src//**/.ts", "/src///*.tsx", "!/src//*.spec.ts", "!/src/**/*.spec.tsx"],
+  "jest": {
+    "enableFindRelatedTests": false
+  }
}
  1. Jest is running successfully and tests are being identified.
  2. After the configuration change, all tests can be seen in the tests tab, but all cases survived, none were identified as killed.
  3. If I'm changing aliased import to relate one - Stryker is able to find a test and killed cases ar identified.
    1. Running time increased by a huge amount.

@nicojs
Copy link
Member

nicojs commented Mar 21, 2024

This sounds like this known issue: https://stryker-mutator.io/docs/stryker-js/troubleshooting/#all-mutants-survive---jest-runner

Could you try enabling --inPlace? That should do the trick. Note: using --inPlace doesn't allow you to change your code when StrykerJS is running.

@Vaclovas
Copy link
Author

Vaclovas commented May 3, 2024

Hello,
--inPlace is not changing any results.
I've created a project boilerplate for this:
project.zip
open project, run "npm i" inside the root, after that continue the same on shell and "ui-shared" folders. Run "npx stryker run" with changing FieldLabel.spec.tsx file. Comment/uncomment import line for the same file import, and see how results are changing:
image

report difference:
image

@nicojs
Copy link
Member

nicojs commented Sep 9, 2024

Hi @Vaclovas thanks for your detailed report.

The problem is that StrykerJS will try to use Jest's --findRelatedTests feature by default. This will make sure only the tests that actually cover files with mutants are run (a performance optimization). However, this doesn't work with the '@scp import.

You can verify this by running npx jest --findRelatedTests yourself:

nicojs@NicoJ01 project/ui-shared (master)$ npx jest --findRelatedTests src/lib/field-label/FieldLabel.tsx 
No tests found, exiting with code 1
Run with `--passWithNoTests` to exit with code 0
No files found in ~/tmp/project/ui-shared.
Make sure Jest's configuration does not exclude this directory.
To set up Jest, make sure a package.json file exists.
Jest Documentation: https://jestjs.io/docs/configuration
Pattern: src/lib/field-label/FieldLabel.tsx - 0 matches

There are 2 ways to solve this:

  1. Disable findRelatedTests for StrykerJS. This comes with a performance penalty:
    {
      "$schema": "../node_modules/@stryker-mutator/core/schema/stryker-schema.json",
      "reporters": [
        "html", "clear-text", "progress"
      ],
      "packageManager": "npm",
      "timeoutMS": 30000,
      "ignorePatterns": ["**/src/lib/icons/*.tsx"],
      "tsconfigFile": "tsconfig.json",
      "mutator": { "excludedMutations": ["StringLiteral", "BooleanLiteral", "ObjectLiteral"] },
      "disableTypeChecks": "**/*.{js,ts,jsx,tsx,html,vue}",
      "testRunner": "jest",
      "coverageAnalysis": "perTest",
    - "tempDirName": "stryker-tmp"
    + "tempDirName": "stryker-tmp",
    + "jest": {
    +   "enableFindRelatedTests": false
    + }
  2. Configure moduleNameMapping in ui-shared/jest.config.ts.
    /* eslint-disable */
    export default {
      displayName: 'ui-shared',
      preset: './jest.preset.js',
      transform: {
        '^(?!.*\\.(js|jsx|ts|tsx|css|json)$)': '@nx/react/plugins/jest',
        '^.+\\.[tj]sx?$': ['babel-jest', { presets: ['@nx/react/babel'] }],
      },
      moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx'],
      coverageDirectory: '../coverage/ui-shared',
      testEnvironment: 'jsdom',
    - setupFilesAfterEnv: ['<rootDir>/testSetup.ts']
    + setupFilesAfterEnv: ['<rootDir>/testSetup.ts'],
    + moduleNameMapper: {
    +  '^@scp/(.*)$': '<rootDir>/src/$1',
    + },
    };

After choosing one of these 2 solutions, you can run stryker without --inPlace

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
⁉ Question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants