The first NuGet restore is significantly slower when restoring from pre-installed nuget cache

[tsvietOK]

We use Azure VMs as build agents for our CI/CD. We have our own Nuget (ProGet) server. We have our VM Windows Server 2022 image where we would want to put the Nuget cache, improve build speed, and avoid downloading all packages from our Nuget server each time a new agent is deployed. Just for reference, it takes 2 minutes to restore (download) packages from our Nuget server:

As an experiment, we ran our build on agent, gathered nuget cache from %userprofile%.nuget\packages folder, and put it in the C:\NugetPackages folder in our VM image. Also, we have set env var NUGET_PACKAGES to C:\NugetPackages, nuget.config also contains line <add key="globalPackagesFolder" value="C:\NugetPackages" />. When we run a build on a fresh agent created from the new image (with preinstalled nuget cache), nuget restore takes 10-40 minutes:

The second build on the same agent (VM) takes the same time as usual:

We would like to know where is the potential bottleneck. We tried to use a faster disk/CPU, but the results are the same, downloading packages from our server is faster than having a preinstalled nuget cache.

OS: Windows Server 2022
VM size: Standard_D16ads_v5
Disk: Premium SSD LRS (255GB, P15 - 1100 IOPS, 125 MB/s) (Ephemeral disk also tested, Standard HDD LRS, 255GB, 75000 IOPS, 954 MB/s, same results)
Tool: Cake 2.2.0
Nuget 6.2.1
Cache size (unarchived): 6.3GB.