Assassins Creed Valhalla - Freeze on Nvidia GPUs

[shelterx]

I'm sure you know AC Valhalla doesn't run well on Nvidia cards, I thought I had it working but now I just freeze in the exact same spot. Not sure if it's an vkd3d problem, wine problem or driver problem.

I tried basically EVERYTHING, different settings, different drivers, different wine/proton versions. I also tried both Xorg and Wayland. I tried with G-Sync on/off.
I think it runs with ingame FPS capped at 30 though, but I really don't see the point in doing so.

This always seem to happen:
0680:err:sync:RtlpWaitForCriticalSection section 0000000005BF0B50 "?" wait timed out in thread 0680, blocked by 0694, retrying (60 sec)
Update:
After reading some reports it might be related to when the GPU hits 100%. Keeping the game below that seems to reduce the freezing, I can basically verify this because on High settings with an 90 FPS lock I didn't crash at that same place, GPU never hit 100% either. Closest was 98%,

Nvidia RTX 3060 Ti.

[adolfintel]

I'm having weird issues with this game too but on AMD, most notably it runs at like half res and I have no idea why 😆

[Saancreed]

different drivers

@shelterx Including 510 series?

[shelterx]

different drivers

@shelterx Including 510 series?

Not that far back no. Someone wrote it was a driver issue. I haven't seen anyone verifying it tho'.

[Saancreed]

Well, give it a try, it was the last series that didn't exhibit that problem for me.

On my current machine I can't even do that again myself because 510 doesn't support Ada GPUs.

[shelterx]

Yeah, I might try, it's not always easy. But if it is the newer driver versions that's causing problems, how do we push for a fix?

Because now I actually played Valhalla yesterday on driver version 530.41.03 with the settings below:
FPS Locked at 75, mainly to keep the GPU usage down.
All settings on Max, except for Shadows (high), Antialiasing (low), Motion blur (off).
Fullscreen / 144Hz
VSync Off

The above kept my GPU usage below critical values, of course it sometimes hit +90% but it seemed to work way better than before. I also tried locking the FPS at 80 but then I hit a spike at %96 GPU usage and the game froze.

[shelterx]

I couldn't get the 510 driver working. But the freeze is indeed due to GPU usage, it froze again now when hitting 95%, very weird bug/behavior.

[shelterx]

Freeze again, have a look at the GPU usage... I'm not saying it's 100% the issue but the game sure works better if the usage is below 90%

Update: I just tried to raise the FPS to unlimited after a fairly stable session (I did 2 quests and a Raid), the GPU usage instantly hit 97% and the game froze.

[shelterx]

@Saancreed Wow, you know what, I got the 510 driver working with a bit of messing around, had to revert to an older kernel and install the driver in a bit of a hacky way. But AC Valhalla seems to work now, even the benchmark works, it NEVER worked with the 520+ drivers!

EDIT:
Game works just fine with the 510 Nvidia drivers, I played for a long time without issues. Even having Ubisoft overlay enabled works.
However if you enable Force composition pipeline in Nvidia display settings, the game will freeze almost instantly after starting it. So don't do that.

[shelterx]

A long shot here but If the issue exists with the 515 driver and above, maybe someone can try and set NVreg_DynamicPowerManagementVideoMemoryThreshold to 200Mb on the newer drivers and see if the game works or not. 200 was default in the 510 drivers, they changed it to 1000Mb in the 515 drivers.

Update:
Did not help.

[shelterx]

Some more info, all drivers from 515 to 530 breaks the game. Which makes me wonder if they break something else too?

Vulkan dev 470 works but it breaks other stuff and the performance isn't good. Vulkan dev 515 (and up?) breaks the game.

[shelterx]

@HansKristian-Work
Here's what happens before the lockup with drivers > 510.
Not sure if it tells anything... might be an Nvidia mess up but everything else I've tried seems to work with these drivers so that's a bit odd...

vkd3d.log

[kodatarule]

This sounds like an nvidia driver bug, please refer to https://forums.developer.nvidia.com/t/nvidia-driver-bug-causes-vkd3d-resident-evil-ac-valhalla-to-corrupt-shaders-crash/248365

[shelterx]

Most certainly is. Since it apparently affects other games too. Oh well,all we can do is wait.

[shelterx]

Seems like Vulkan Dev driver 525.47.24 fixes the issue. I can run the benchmark now too without the game locking up instantly.
Need to do some more testing tho'.

Update:
Yea, on first start up it worked just fine, it took a very unusal long time for the game to load, I played some and tried the benchmark. Then I quit the game and when I started it again when the shaders were cached, it froze like usual. Very odd. Never had that happening before.

Update 2:
I can't get it to run again, whatever caused the game to start much slower the first time sorf of made it work. But now I removed all shader caches and tried again, it freezes.
Which makes me wonder if there's a race condition somewhere.
Starting the game with 30 FPS limit works, like before, if you then change it to unlimited, it freezes, like before. I have no idea why the game initially worked, even the benchmark loop., the benchmark ran at about 75-90FPS with GPU usage at 99%

[shelterx]

Ok, so alt-tabbing in and out of the game a few times, with some pauses before alt-tabbing back, when eivor is on the screen while loading the world, and right when the final load indicator shows. STARTS the game...........
You need to have fullscreen enabled in options, benchmark runs fine too...

The above trick worked EVERY time. But don't alt tab back too quickly.

Smells like a race condition to me, it's sooo weird. I also don't get why this fixes the game, but startinng it with 30FPS limit then switching to unlimited, does not.
At least I know how to get it running now.

There's also no 0680:err:sync:RtlpWaitForCriticalSection section 0000000005BF0B50 "?" wait timed out in thread 0680, blocked by 0694, retrying (60 sec) error message anymore when doing this.

Btw, loading and saving etc also works.

[kakra]

I found reports where people report the same issue in Windows, and there it helped some people to remove the shader/graphics cache which the game places in "My Documents". For me, that didn't help. Sometimes the game works while limited to 45 fps, sometimes while limited to 30 fps, but it is never reliable, results are inconsistent, after the next reboot eventually it will freeze anyways even with the same settings. If uncapped, it freezes within seconds, reproducible. (NVIDIA 525)

OTOH, it works rock solid on the Steam Deck with 30-40 fps (fps limited to 40).

[shelterx]

Mmm, the alt-tab trick I figured out works on PC, 100%. I think it really boils down to not having the game active/visible when the world finishes loading, if you alt-tab out right when the screen turns black after showing Eivor on the screen, the game starts fine and you can run with uncapped FPS. Like I said before, it's a very odd "workaround" that circumvents whatever weird bug that causes the freeze.

EDIT:
I played for several hours now, so alt-tabbing does indeed work.

[jpeaglesandkatz]

Doesn't work on linux

Pretty amazing that although Valhalla is not supported on linux that Ubi wouldn't make an effort to fix this. It is in their best interest if more people are playing their games.... Unbelievable.. They just don't care about customers, only about making more money from silly micro transactions.

[jpeaglesandkatz]

Correction: after a few tries this does seem to work.. For how long I don't know.. but running uncapped at it seems GPU utilization always statys under 96/97% but running higher framerates.... Well on next gamelaunches it didn't work.. Not sure how to time that alt-tab... hit and miss... only worked once out of many attempts so I would call that a lucky one time success and not even sure alt-tabbing made the diference.

Running this game is a mess...

[kakra]

Pretty amazing that although Valhalla is not supported on linux that Ubi wouldn't make an effort to fix this. It is in their best interest if more people are playing their games.... Unbelievable..

This is purely a business decision: Probably less than 1% of players use Linux but it probably causes much more than 1% of costs. Change that by promoting Linux as a platform, and at some (far) point in the future it will receive more love by game devs. But there's nothing you could force. Also, Valve mostly committed to making games work under Proton rather than game devs doing it themselves (by making wine/Proton as compatible to Windows behavior as possible). That said, any help from devs supporting Valve's efforts by providing debugging and logging would surely be appreciated but apparently, Valhalla is already EOL and will no longer receive patches. Additionally, the freezing also happens for some Windows users, it hasn't been fixed for them either. So, GPU driver devs have to fix stuff, or Protons devs for that matter.

We have reports for different games that NVIDIA 510 worked in almost all cases where games now show problems, 515 introduced some rare problems, 520 some more, 525 even more, and most of them seem to be freezes of game engines. Then, there's 530 which adds graphical glitches on top for very few games. Unfortunately, it becomes harder and harder to use the older drivers with newer kernels. I'd rather see NVIDIA fix their stuff than Ubi.

[shelterx]

Seems to work now with the Nvidia 535.43.02 Beta driver? Game starts fine and benchmark runs fine, without doing any alt-tabbing. I have tried several times now. Unless something funky is happening like the last time I thought it was working and found out the alt-tab trick.

[kakra]

@shelterx I cannot verify that.

With 535, the Steam client even refuses to open a window, I'm seeing very short triangle flashes when opening web browser windows, and Elite Dangerous still shows graphical glitches - so it's no option for me to run this driver. Additionally, it looks like it got worse even if it may fix this particular game. I know this is a beta driver but please NVIDIA, fix your stuff.

BTW: Do you use single or dual monitors?

[shelterx]

@kakra ok, yeah i realized that verifying this might be tricky. The latest Ubisoft launcher might not work, some say it works with proton experimental tho'. I still run the older Ubisoft Connect launcher.

I use two monitors but I disable the second one when gaming.

[kakra]

Maybe multi-monitor is part of the problem here. The NVIDIA driver has a long history of multi-monitor issues where G-SYNC would not work properly, vsync is out of sync (very visible tearing with vastly fluctuating frame times[1]), full composition pipeline is needed etc. Actually, I have three monitors attached but the third one becomes disabled after each reboot even with appropriate configs in xorg.conf, and I need to manually enable it again (which I often do not because it is a TV for couch gaming). It would be interesting do hear if the problem persists if you enabled your monitor while playing the game.

[1]: This is mainly an xorg issue which doesn't have proper protocols for multi-monitor vsync.

[shelterx]

@kakra Steam Runtime ran fine for me with both monitors enabled and the 535 driver but we are getting slighty off topic now.

[jpeaglesandkatz]

I haven't been able to find a 510 nvidia driver or somewhat older.. And one that actually compiles with a new 6.2.4.1 kernel..... Let alone on how to isntall it. Opensuse tumbleweed btw.

[shelterx]

@jpeaglesandkatz Here are the patches for 6.2 and 6.3.
Frogging-Family/nvidia-all#160

Btw, try alt-tab right before the world shows up, then wait like 10 seconds, then alt-tab back. It should work but I'm not sure if it does for everyone. I played several hours uncapped. I use KDE/X11.
But yea, the GPU usage is usally max ~97% if you manage to get in the game, I don't think I ever see it go up to 99-100%

[giomasce]

I'm seeing a similar issue. After having a look with the Vulkan validation layers, it seems that the application is destroying resources while they're still in use by the GPU. There's still a lot of guesswork, though, and I am not sure what a good solution/workaround would be if that turned out to be true.

[kakra]

I'm seeing - only sometimes - a scheduling timeout from the nvidia driver in dmesg. But because vkd3d is handling translation to Vulkan, it should probably prevent that resources are being destroyed while still in use? Maybe defer destruction until released?

[shelterx]

I'm pretty sure the freeze bug is fixed in the recent Nvidia release and dev drivers. I won't close the issue yet tho', since I only tested this on a single system.

[Saancreed]

It also appears to be fixed for me on 535.54.03 (but I haven't checked dev or beta drivers); I can now uncap my FPS, let the GPU hit ~97% usage and stay near that value for some time and I haven't seen any more hangs so far.

[K0bin]

But because vkd3d is handling translation to Vulkan, it should probably prevent that resources are being destroyed while still in use? Maybe defer destruction until released?

D3D12 has roughly the same rules as Vulkan here. So for most resources, it's the responsibility of the application to ensure that doesnt happen.

[kakra]

D3D12 has roughly the same rules as Vulkan here. So for most resources, it's the responsibility of the application to ensure that doesnt happen.

Yes but the "responsibility" here may be vkd3d as it is translating from D3D12. So while the game may work fine in D3D12, does (or did) it behave wrong after translation in vkd3d? OTOH, I've seen reports of native Windows users with the same or similar issues. So after all, the game may indeed behave wrong but drivers may work around it. In this case, the D3D12 driver is vkd3d as seen from the game's perspective.

I'm happy, tho, that 535.54.03 may have fixed that, and it seems to perform better overall, too - but unfortunately, I cannot use that driver yet (I tried yesterday) as it shows render glitches in some DXVK titles, primarily Elite Dangerous (that glitch exists since the 530 driver series). It has been reported in the Proton tracker for that game.

Also, I've experienced myself that this game may actually work with 60 fps for some time after a driver update - and only later start to freeze again. People using Windows reported that it may help to clear the shader and graphic caches of the game when it happens. So it may just be a bug in the game, and we do know nothing yet until it proved working for some longer time.

[Saancreed]

I just checked latest Vulkan dev driver and the issue still persists there, seems like 535 is the only branch where it's working fine again.

I'm happy, tho, that 535.54.03 may have fixed that, and it seems to perform better overall, too - but unfortunately, I cannot use that driver yet (I tried yesterday) as it shows render glitches in some DXVK titles, primarily Elite Dangerous (that glitch exists since the 530 driver series).

Does setting __GL_b5f2b3=0xFFFFFFFF as environment variable not fix it for you?

[shelterx]

I just checked latest Vulkan dev driver and the issue still persists there, seems like 535 is the only branch where it's working fine again.

Weird, it worked for me. Now I have to triple check.

[kakra]

Does setting __GL_b5f2b3=0xFFFFFFFF as environment variable not fix it for you?

Oh, wow, what a creative env name... I'll check my ED installation with that after updating the driver, thanks.

[Saancreed]

Weird, it worked for me. Now I have to triple check.

For the record, here's my log: ACValhalla-2023-06-17T14:01:52+02:00.log

Oh, wow, what a creative env name... I'll check my ED installation with that after updating the driver, thanks.

Ye, there is about a thousand of such __GL_xxxxxx variables that Nvidia's drivers check which seem to be obfuscated/undocumented on purpose. You can dump a list of them with some creative tracing like ltrace -e snprintf vkcube |& grep -oP '(?<=("))__GL_[A-Za-z0-9]+(?=("))' | sort -u but I have no idea what most of them do, their behavior is often hard to observe.

If you have Nvidia account, you could also report the result in https://forums.developer.nvidia.com/t/graphical-regression-in-elite-dangerous/247010

[kakra]

Confirming that 535 fixes the freezes for me, I played some hours in a row now with unlimited fps.

[shelterx]

@Saancreed
Okay, so this is so confusing, I ran the game 3 times, and also the benchmark with the 525.47.27 vulkan dev driver. I did nothing special when testing. It was even a few days apart from the first test.
I thought, let me launch it a 4th time and play for a longer time, now the game freezes after loading, just like before, both the benchmark and the last save.

So yeah, vulkan dev doesn't work. But I have no idea why it worked for me the first times I tested it.

[shakakibara12]

@shelterx Can you try 525.47.31 vulkan dev driver released on July 11th, 2023.
from release notes: Fix 64-bit shader disk cache issues

[shelterx]

@shakakibara12 There's a new release driver from 2023-07-22, it doesn't mention the disk cache tho'. I rather not test the Vulkan driver since the release driver from June seems to work, at least regarding Valhalla.

[Blisto91]

It includes the cache fix if you mean 535.86.05

[shelterx]

This is fixed for most people now by using newer Nvidia drivers. It can still have minor issues but restarting the game usually helps.

[shelterx]

Closed.