Upgrading from 4.0.6 to 4.2.0 analysis no longer works for project

[martincostello]

Describe the bug

Attempting to upgrade from 4.0.6 to 4.2.0, analysis is failing for one of our test projects, causing an error.

Specifying --dev-mode doesn't appear to add any additional details.

4.0.6

Running mutation tests for '../src/Polly/Polly.csproj'. Test Project: 'Polly.Specs/Polly.Specs.csproj'

   _____ _              _               _   _ ______ _______  
  / ____| |            | |             | \ | |  ____|__   __| 
 | (___ | |_ _ __ _   _| | _____ _ __  |  \| | |__     | |    
  \___ \| __| '__| | | | |/ / _ \ '__| | . ` |  __|    | |    
  ____) | |_| |  | |_| |   <  __/ |    | |\  | |____   | |    
 |_____/ \__|_|   \__, |_|\_\___|_| (_)|_| \_|______|  |_|    
                   __/ |                                      
                  |___/                                       


Version: 4.0.6

A new version of Stryker.NET (4.2.0) is available. Please consider upgrading 
using `dotnet tool update -g dotnet-stryker`

[09:02:00 INF] Analysis starting.
[09:02:06 WRN] Could not find a project analysis for the chosen target framework net8.0. 
The available target frameworks are: net462,net472,net6.0,netstandard2.0. 
The first available framework will be selected, which is net462.
[09:02:06 INF] Found project D:\a\Polly\Polly\src\Polly\Polly.csproj to mutate.
[09:02:06 INF] Analysis complete.
[09:02:06 INF] Building test project D:\a\Polly\Polly\test\Polly.Specs\Polly.Specs.csproj (1/1)
[09:03:18 INF] Number of tests found: 1986 for project D:\a\Polly\Polly\src\Polly\Polly.csproj. Initial test run started.

4.2.0

Running mutation tests for '../src/Polly/Polly.csproj'. Test Project: 'Polly.Specs/Polly.Specs.csproj'

   _____ _              _               _   _ ______ _______  
  / ____| |            | |             | \ | |  ____|__   __| 
 | (___ | |_ _ __ _   _| | _____ _ __  |  \| | |__     | |    
  \___ \| __| '__| | | | |/ / _ \ '__| | . ` |  __|    | |    
  ____) | |_| |  | |_| |   <  __/ |    | |\  | |____   | |    
 |_____/ \__|_|   \__, |_|\_\___|_| (_)|_| \_|______|  |_|    
                   __/ |                                      
                  |___/                                       


Version: 4.2.0

[12:05:57 INF] Analysis starting.
[12:05:57 INF] Analyzing 1 test project(s).
[12:06:11 INF] Analysis complete.
[12:06:11 INF] Time Elapsed 00:00:13.4378512
Stryker.NET failed to mutate your project. For more information see the logs 
below:


No project references found. Please add a project reference to your test project
and retry.

Logs

See the mutation-report artifact here.

Expected behavior

Mutation tests work.

Desktop (please complete the following information):

• OS: Windows
• Type of project: framework and core
• Framework Version: net462,net472,net6.0,netstandard2.0
• Stryker Version: 4.2.0

Additional context

Failing upgrade: App-vNext/Polly#2293

[rouke-broersma]

Looks like we used to be unable to properly detect some information, falling back to a default that happens to work. I suspect that we are now able to find a result we can work with, which happens to actually be incorrect. Strange. I base this on these logs that used to appear:

[09:02:06 WRN] Could not find a project analysis for the chosen target framework net8.0. 
The available target frameworks are: net462,net472,net6.0,netstandard2.0. 
The first available framework will be selected, which is net462.

I see you also changed the target frameworks ordering, did you do this after you noticed stryker failed or before?

[dupdob]

humm
target framework selection has been redesigned to provide more consistent results, i.e ensuring Stryker picks compatible framework versions across projects in a solution.
fallback logic may have changed, or dropped altogether.
Update: there is no visible issue and their is a dedicated unit test for this situation.

will try to reproduce it via integration testing

[martincostello]

I see you also changed the target frameworks ordering, did you do this after you noticed stryker failed or before?

It failed before, and I swapped it to see if it would fix it, but alas it didn't.

[dupdob]

I do have a problem when specifying a non supported target framewok, but it fails differently, and I have no idea why.

I am surprised that --def-mode does not provide extra information. Both hints toward an non anticipated failure mode.
For the short time, I will fix the problem I reproduce.

[dupdob]

I opened a PR where Stryker no longer asks for the configured target framework when analyzing a project.
I assume this will fix this issue, but bear in mind that when I tried to reproduce this issue, I did get extra error messages (such as a warning because project analysis failed).

[martincostello]

Thanks - if there are genuine issues with our setup I'm fine with us having to fix through them to take the upgrade (e.g. tweaking settings).

[dupdob]

well, if could either adapt the target framework option so that it matches one that is an actual target for the project.
Or get rid of it. Because, the fix will revert the previous behavior which is to pick one framework at random 😄