VSBuild@1 fails to build appx WITHOUT the task updating

[Skintkingle]

Description

COPIED FROM Azure Pipelines Issue

Last week, we were using VSBuild@1 (version 1.245.0), last used Friday 13th September at 8pm UTC to successfully build and package an APPX.
On sunday (and today) we are using the same version (1.245.0), and the build is now failing with the following error:

fatal error CMF1106: failed to open input PDB file for reading (PDB error code = 13)

I have raised this here INSTEAD of azure-pipelines-tasks because the task hasn't changed between the successful execution and the failure, so I don't beleive it's the task that has changed to make this fail, rather the hosting environment/agent

The task is configured as such (and has been like this for many months):

- task: VSBuild@1
  displayName: Build Our Product
  inputs:
    platform: 'x64' 
    solution: 'Our Solution.sln'
    msbuildArgs: '/p:AppxBundlePlatforms="x64" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=SideLoadOnly /p:AppxPackageSigningEnabled=false'

Critically - we can run this on the same commit that previously passed, and it now fails.

This is preventing us from releasing any changes and as such, for us, is critical to get a fix or workaround

Wondering if the fix deployed for this event was the cause

Which would be related to #10601

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

20240912.1.0

Is it regression?

yes, 20240811.1.0

Expected behavior

For vsbuild@1 to build appx packages

Actual behavior

failed with the above error

Repro steps

_GenerateAppxSymbolPackage:
  C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\\x64\mspdbcmf.exe /check "D:\a\1\s\Our App\bin\x64\Debug\net8.0-windows10.0.19041.0\win-x64\msixpublish\Our App.pdb" 
  Microsoft (R) FastLink To Full PDB Converter Version 14.41.33923.0
  Copyright (C) Microsoft Corporation.  All rights reserved.
  
  D:\a\1\s\Our App\bin\x64\Debug\net8.0-windows10.0.19041.0\win-x64\msixpublish\Our App.pdb : fatal error CMF1106: failed to open input PDB file for reading (PDB error code = 13)
##[error]C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets(3715,5): Error MSB6006: "mspdbcmf.exe" exited with code 1106.
##[debug]Processed: ##vso[task.logissue type=Error;sourcepath=C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets;linenumber=3715;columnnumber=5;code=MSB6006;]"mspdbcmf.exe" exited with code 1106.
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets(3715,5): error MSB6006: "mspdbcmf.exe" exited with code 1106. [D:\a\1\s\OurApp.Package\OurApp.Package.wapproj]

[Wuppie007]

Also run into this is issue.

The last succesfully completed pipeline for me was with runner image version 20240811.1.0. At this image the "Fastlink to Full PDB Converter" version was 14.33.31424.0.

[kishorekumar-anchala]

Hi @Skintkingle ,

Thank you for bringing this issue to us. We are looking into this issue and will update you on this issue after investigating.

[Skintkingle]

Thanks for looking into this, This is currently stopping us from releasing and as such is quite a critical blocker for us. Do you have any ideas of a workaround for the time being?

[kishorekumar-anchala]

HI @Skintkingle ,

Could you please try to. build with windows-2019 image .

[Skintkingle]

Hi @kishorekumar-anchala
Unfortunately we can't use windows-2019 as our app is a .Net 8 app and the visual studio version installed for windows-2019 can't build .Net 8 projects. Its MSBuild is too old. :(

We have also tried downloading .Net 8 with the UseDotNet@2 Task, but fundamentally, given this installs the dotnet tool for that .Net version, and not MSBuild, we're left in the same situation.

[josephsleiman16]

Also run into this issue and also unable to use windows-2019 image which only supports .NET 5.0 and lower. This is preventing us from releasing and is also a critical blocker for us, so any better workaround or permanent fix would be great.

[yosefmah]

This is impacting us too. We are unable to build or deploy updates to our internal applications so it is very critical for us to get a workaround at least for the time being.

[kishorekumar-anchala]

HI @Skintkingle ,

Could you please share your yaml file to investigate further.

[Skintkingle]

Sure, it is very simple though.

I've also removed everything after the failing step (as it's never got to) :)

pool:
  vmImage: windows-latest

steps:
- task: NuGetAuthenticate@1
  displayName: Authenticate with NuGet feed

- task: DotNetCoreCLI@2
  displayName: Restore Our Solution
  inputs:
    command: 'restore'
    projects: 'Our Solution.sln'
    feedsToUse: 'select'
    vstsFeed: '<ourInternalFeed>'

- task: DotNetCoreCLI@2
  displayName: Build Our Web App
  inputs:
    command: 'build'
    projects: 'OurApp.Web/OurApp.Web.csproj'
    
 - task: VSBuild@1
  displayName: Build Our Product
  inputs:
    platform: 'x64' 
    solution: 'Our Solution.sln'
    msbuildArgs: '/p:AppxBundlePlatforms="x64" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=SideLoadOnly /p:AppxPackageSigningEnabled=false'

[kishorekumar-anchala]

Hi @Skintkingle ,

In VsBuild@1 task , kindly modify platform value to Any CPU from x64 and try . it works for me. thank you !

[josephsleiman16]

I have practically the same yaml file and the same error occurring on the exact same task. Changing the platform from 'x64' to 'Any CPU' is not compatible with our solution as it is not configured to build with Any CPU, so this is not a valid fix or workaround, even. This is becoming a more urgent issue as it is almost a week now we cannot deploy releases.

[Skintkingle]

This also doesn't work for us. We get the same error when platform is set to Any CPU

[kishorekumar-anchala]

HI @Skintkingle @josephsleiman16 ,

Might be issue with the Vstask@1 , have you tried to change the version of vsbuild to 2 ? could you please try this , as sometimes it will fix the issue.

and also confirm what is the image version you're using ? 20240912.1.0 ?

[Skintkingle]

there is only VSBuild@1. The task is unchanged between it working and it not working.

I can confirm these tests this morning were run against a newer version of the runner: 20240915.1.0

[kishorekumar-anchala]

Hi @Skintkingle , Could you please raise issue with azure tasks team. https://github.com/microsoft/azure-pipelines-tasks

[Skintkingle]

If you sincerely beleive that it's a tasks issue and not an image issue (given the runner image has changed between it working and not working, but the task hasn't changed)

[Skintkingle]

I am skeptical that it's the tasks fault though. Please don't stop investigating.

[Skintkingle]

Further to this, I have tried using MSBuild@1 rather than VSBuild@1 on the slim chance it's the tasks fault, and I get exactly the same error at exactly the same point in the MSBuild task. This has almost certainly go to do with the build environment (or its' configuration or available tooling or something like that)

The MSBuild task looked like this:

- task: MSBuild@1
  displayName: Build Our Product with MSBuild
  inputs:
    solution: 'Our Solution.sln'
    msbuildVersion: '17.0'
    msbuildArchitecture: 'x64'
    platform: 'x64'
    msbuildArguments: '/p:AppxBundlePlatforms="x64" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=SideLoadOnly /p:AppxPackageSigningEnabled=false'

[josephsleiman16]

Agree with @Skintkingle, this is almost certainly a build issue with the runner image changing. Our builds were working on Friday on runner image 20240811.1.0 and then on monday, using runner image 20240912.1.0 they stopped working, despite being the exact same commits using the exact same yaml pipelines and tasks.

[yosefmah]

Agreed here as well. Does not seem to be an issue with the task at all from our perspective.

[Skintkingle]

@kishorekumar-anchala are you still looking into this or not anymore?

[Wuppie007]

To add more information, I've added the command /p:AppxSymbolPackageEnabled=False, to not run into the "PDB file error".

- task: MSBuild@1
  inputs:
    solution: '**/msix project.wapproj'
    platform: AnyCPU
    configuration: Release
    msbuildArguments: '
     /p:OutputPath=NonPackagedApp
     /p:UapAppxPackageBuildMode=SideLoadOnly  
     /p:AppxBundle=Never
     /p:AppxSymbolPackageEnabled=False
     /p:AppxPackageOutput=$(Build.ArtifactStagingDirectory)\outputApp.msix 
     /p:AppxPackageSigningEnabled=false'
  displayName: 'Package the App'

This results in the following error: (I have removed some paths and names because of company names etc)

_GenerateAppxPackageFile:
  C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64\MakeAppx.exe pack /l /h sha256 /f obj\Release\package.map.txt /o /p D:\a\1\a\AppAttach\<omitted>.msix  
  <omitted>-> D:\a\1\a\AppAttach\<omitted>.msix
##[error]C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets(6573,5): Error APPX0002: Task 'GenerateAppInstallerFile' failed. Could not find a part of the path 'D:\a\1\s\<omitted>\trunk\<omitted>\AppPackages\<omitted>_neutral.appinstaller'.

I've checked the last succesfull pipeline, and the used buildnummer fro MakeAppx.exe is unchanged (10.0.22621.0)

[Skintkingle]

adding `/p:AppxSymbolPackageEnabled=false' actually worked around the issue, for us. So thanks for that idea @Wuppie007 ! It's definitely a workaround though. So I'm still keen to see a proper resolution

[Wuppie007]

It seems to be that de parameter /p:AppxPackageOutput=$(Build.ArtifactStagingDirectory)\outputApp.msix is causing trouble. When removing this, the pipeline succeeds. But then there are several files created instead of one *.msix file