Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid file magic number. The appinfo.vdf version might not be supported by the current version of Protontricks #304

Closed
Richard-Payne opened this issue Jun 27, 2024 · 57 comments

Comments

@Richard-Payne
Copy link

Richard-Payne commented Jun 27, 2024

DEVELOPER EDIT:

See this comment for the current workaround.


Describe the bug
When I run protontricks I get this error:

Invalid file magic number. The appinfo.vdf version might not be supported by the current version of Protontricks

To Reproduce
Steps to reproduce the behavior:

  1. Run command protontricks 1250410 /home/richard/Downloads/MSFS_Plugin_x64_Software_8.0.313.0.exe
  2. Command fails and error is displayed:
    Traceback (most recent call last): File "/usr/bin/protontricks", line 8, in <module> sys.exit(cli()) ^^^^^ File "/usr/lib/python3.12/site-packages/protontricks/cli/main.py", line 32, in cli main(args) File "/usr/lib/python3.12/site-packages/protontricks/cli/util.py", line 159, in wrapper return cli_func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/protontricks/cli/main.py", line 339, in main proton_app = find_proton_app( ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/protontricks/steam.py", line 848, in find_proton_app tool_app = find_steam_compat_tool_app( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/protontricks/steam.py", line 609, in find_steam_compat_tool_app appinfo_sections = [ ^ File "/usr/lib/python3.12/site-packages/protontricks/steam.py", line 531, in iter_appinfo_sections raise SyntaxError( SyntaxError: Invalid file magic number. The appinfo.vdf version might not be supported by the current version of Protontricks - please check for updates.

Expected behavior
It runs the exe I'm passing in the game prefix

System (please complete the following information):

  • Distro: Manjaro
  • Protontricks: flatpak, Manjaro Extras repo and AUR git package
  • Protontricks version: AUR is protontricks (1.11.2.dev7+gf2d605d). Flatpak and Manjaro Extras are both 1.11.1
  • Steam version:
    Steam Beta Update
    Steam Version: 1719453458
    Steam Client Build Date: Thu, Jun 27 2:35 AM UTC -08:00
    Steam Web Build Date: Thu, Jun 27 12:36 AM UTC -08:00
    Steam API Version: SteamClient021

Additional context

https://gist.github.com/Richard-Payne/070d5b08ede33cfa10440e932f48149c

@MrBareBones
Copy link

Yeah I'm running into this same issue on my Steam Deck after the Beta Update with game recording. I wonder if that has something to do with it?

@Matoking
Copy link
Owner

The current workaround is to downgrade from Steam beta.


There have been several changes to the ~/.steam/steam/appcache/appinfo.vdf file in the Steam beta:

  • The file header is 8 bytes longer.
  • The binary VDF segment for each app entry in appinfo.vdf is different and the vdf Python library that Protontricks uses doesn't seem to be able to parse it. Each segment is preceded by a SHA1 checksum; calculating the checksum for the VDF segment ourselves results in the same checksum. So it's not that we're trying to parse the wrong data, the data itself seems to now use a different format that the library can't parse (yet).
  • The contents of the binary segments also seem to have changed radically, with things like redundant appinfo.appid fields having been removed. This is a bit hard to determine since I can't parse the file to a more human readable format yet. But for comparison's sake, the appinfo.vdf from 2023 is 15.6 MiB in size, while the current appinfo.vdf in Steam beta is 12.9 MiB.

Protontricks needs to read this file since it contains app compatibility mappings (or at least used to, I'll have to double check), such as what Proton versions to use by default. Cyberpunk 2077 was previously set to use Proton Experimental by default due to DualSense support. These app compatibility mappings aren't very commonplace, though, so if push comes to shove we can skip parsing the newer appinfo.vdf file. I'll try working on a proper solution first, though.

@dsalt
Copy link

dsalt commented Jun 28, 2024

Workaround: set PROTON_VERSION to whatever's appropriate.

@PickMeNow
Copy link

Having the same problem. how can I use PROTON_VERSION, do I just set PROTON_VERSION=EXPERIMENTAL in the environment variables?

@syrvolga
Copy link

syrvolga commented Jun 30, 2024

@PickMeNow Use this in a terminal for the flatpak version:

export PROTON_VERSION="Proton Experimental"
flatpak run com.github.Matoking.protontricks --gui

@Matoking
Copy link
Owner

I've submitted a PR to fix the appinfo.vdf support in the VDF parser library at ValvePython/vdf#61.

@Dark-Thoughts
Copy link

Dark-Thoughts commented Jul 2, 2024

Pretty much the same one whenever I try to install MO2 via https://github.com/rockerbacon/modorganizer2-linux-installer.
Annoying since I need it to load the game with all its mods and Steam keeps breaking it, saying it is corrupt, does a file check and then undoes the changes made to launch it that way.

INFO: steam not found in '/home/*****/.var/app/com.valvesoftware.Steam/.local/share/Steam'
Traceback (most recent call last):
  File "/app/bin/protontricks", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/app/lib/python3.11/site-packages/protontricks/cli/main.py", line 32, in cli
    main(args)
  File "/app/lib/python3.11/site-packages/protontricks/cli/util.py", line 159, in wrapper
    return cli_func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/protontricks/cli/main.py", line 339, in main
    proton_app = find_proton_app(
                 ^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/protontricks/steam.py", line 848, in find_proton_app
    tool_app = find_steam_compat_tool_app(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/protontricks/steam.py", line 609, in find_steam_compat_tool_app
    appinfo_sections = [
                       ^
  File "/app/lib/python3.11/site-packages/protontricks/steam.py", line 609, in <listcomp>
    appinfo_sections = [
                       ^
  File "/app/lib/python3.11/site-packages/protontricks/steam.py", line 531, in iter_appinfo_sections
    raise SyntaxError(
SyntaxError: Invalid file magic number. The appinfo.vdf version might not be supported by the current version of Protontricks - please check for updates.
ERROR: Protontricks did not find a valid prefix directory.  Stdout was:\n

I've got this before the recording beta and it persists when I opt out of it too.

Edit: This was with both the Fedora repo version and the Flatpak one. Using the pipx install one works, in case someone else faces the same issue. https://github.com/Matoking/protontricks#pipx

Matoking added a commit that referenced this issue Jul 5, 2024
Steam beta introduced a new version of appinfo.vdf with a space-saving
optimization. Field keys are stored in a separate table at the end of
the file, with the actual VDF segments having to be parsed using the
table to map the indices to actual field names.

`vdf` library does not support serializing appinfo.vdf using this
format, at least yet, so just use appinfo.vdf V28 in tests for the time
being. This might need to be fixed in the future once appinfo.vdf V28 is
phased out.

Fixes #304
@Matoking
Copy link
Owner

Matoking commented Jul 5, 2024

If you're using pipx, you can use the following commands to install development versions of vdf and protontricks with the required fixes:

# Uninstall Protontricks first
$ pipx uninstall protontricks

# Install Protontricks and vdf with the fixes
$ pipx install --preinstall git+https://github.com/Matoking/vdf.git@support_new_bvdf git+https://github.com/Matoking/protontricks.git@appinfo_v29

Releasing a new version of Protontricks with the fix in place requires that a new version of vdf is also published with support for the new format. I do not maintain that library and the final fix might be different, so I cannot release a new version of Protontricks until then, unfortunately.

@Matoking Matoking pinned this issue Jul 5, 2024
@testc2n14
Copy link

@PickMeNow Use this in a terminal for the flatpak version:

export PROTON_VERSION="Proton Experimental"
flatpak run com.github.Matoking.protontricks --gui

i tried that on my steam deck and i am still getting the error, i feel like i have been stupid. i i'm going to try and compile the python file next

@testc2n14
Copy link

wait no i apperntly did something wrong i got it working

@JohnTheCoolingFan
Copy link

Set PROTON_VERSION via flatseal to "Proton Experimental" and selected Proton experimental for the game as well (I use GE proton by default). Flatpak install, both steam and protontricks. Getting a different error now.

Protontricks was closed due to the following error:

Proton installation could not be found!

=============

Please include this entire error message when making a bug report.
Log messages:

Running inside Flatpak sandbox, version 1.14.6.
Found Steam directory at /home/jtcf/.var/app/com.valvesoftware.Steam/.local/share/Steam
Using default Steam Runtime at /home/jtcf/.var/app/com.valvesoftware.Steam/.local/share/Steam/ubuntu12_32/steam-runtime
WINETRICKS environment variable is not available. Searching from $PATH.
Found 2 Steam library folders
Multiple compatdata directories found for app 387990
Currently logged-in Steam user: coolercircutin
Found 0 Steam shortcuts running using Steam compatibility tools
Using 'yad' as GUI provider
$PROTON_VERSION was set but matching Proton installation could not be found.

@jurassicjordan
Copy link

How does one set the environment variable in flatseal? I added PROTON_VERSION=“proton experimental” to the environment variables list and it made no change

@JohnTheCoolingFan
Copy link

@jurassicjordan Are you sure there's no change? Is the error message the same? In my case the error message changed.

@jurassicjordan
Copy link

@jurassicjordan Are you sure there's no change? Is the error message the same? In my case the error message changed.

Same error, only resolved when running it in the command line. Not in flatseal, I ended up using a shell script for now

@ERROR2401
Copy link

I've been having this error as well, and I'm in the steam beta on a steam deck.
the log seems to be mainly reading every game that either doesn't have a proton prefix or isn't a windows program or whatever, twice (twice meaning it goes through each non-steam game once before going through them again)

@Matoking
Copy link
Owner

Matoking commented Jul 10, 2024

OUTDATED, V1.12.0 HAS BEEN RELEASED WITH THE FIXES

Instructions are below for posterity:


I've pushed a build of Protontricks to Flathub beta which contains a fix for this Steam beta issue. This fix is based on the appinfo_v29 branch which you can also install manually.

To summarize how to install this version depending on your current installation method:

Flatpak/Flathub

Discover

Enable "Flathub beta" repository under Settings. You can then look up Protontricks and select "Flathub beta" as the source:

flathub-beta

Command-line

Add the flathub-beta repository if it isn't already added and install Protontricks:

flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo
flatpak install flathub-beta com.github.Matoking.protontricks

pipx

Uninstall Protontricks first and then install these specific branches using these two commands:

# Uninstall Protontricks first
pipx uninstall protontricks

# Install Protontricks and vdf with the fixes
pipx install git+https://github.com/Matoking/protontricks.git@appinfo_v29
pipx runpip protontricks install --force-reinstall git+https://github.com/Matoking/vdf.git@support_new_bvdf

Arch AUR

Install python-vdf-solstice from AUR if you haven't already using the AUR helper of your choice (eg. with Paru the command is paru -S python-vdf-solstice). If pacman prompts you to delete existing python-vdf package, allow it to do so.

Download the PKGBUILD file from protontricks-git either manually or using the AUR helper of your choice (eg. with Paru the command is paru -G protontricks-git).

Change the following line in PKGBUILD from:

source=('git+https://github.com/Matoking/protontricks.git')

to

source=('git+https://github.com/Matoking/protontricks.git#branch=appinfo_v29')

Build and install the package by running the following commands in the directory with the PKGBUILD file:

makepkg
sudo pacman -U protontricks-git-*.tar.zst

Note that it's unlikely that the beta branch on Flathub will be maintained once a new version of Protontricks can be released. Please uninstall the beta version of Protontricks once a new stable release is out!

Also, the development build contains other unreleased changes as documented here.

@stiinkyratz
Copy link

entered in konsole:

(deck@steamdeck ~)$ $ flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo
$ flatpak install flathub-beta com.github.Matoking.protontricks
bash: $: command not found
bash: $: command not found

and if i enter them in separate lines nothing happens at all

@lucifertdark
Copy link

You copied the $ from the start of each line above, take those out & try the commands again, they'll work for sure.

@Matoking
Copy link
Owner

Matoking commented Sep 3, 2024

i'm having the same issue as mentioned further up after installing with pipx,

TypeError: binary_loads() got an unexpected keyword argument 'key_table'

i've checked my verison of pipx and it does support --preinstall. i tried using the pipx comomand you provided and cut off the protontricks install, it executes with no error but no install confirmation either. i must be missing something silly here but not sure what :D

I'm guessing you're running Ubuntu or something similar? The distro ships an older version of pipx which has an unfortunate bug making --preinstall not work correctly. This was discussed and solved in a different thread here, but I forgot to update the installation instructions...

Anyway, updated the installation instructions for pipx. The ones detailed should work for all pipx versions.

@ProtoDot
Copy link

@Matoking Well now python-vdf-solstice isn't on the AUR anymore, so I can't use your workaround. Any other ideas?

@d3-X-t3r
Copy link

d3-X-t3r commented Sep 11, 2024

@Matoking Well now python-vdf-solstice isn't on the AUR anymore, so I can't use your workaround. Any other ideas?

It's still there in Cachy's repos, so you can add their repos to your /etc/pacman.conf (and optionally, lower the priority of their repos if you don't want to upgrade/touch existing packages, so that only the stuff that's not present in your main repos will be picked up).

@d3-X-t3r
Copy link

@Matoking Well now python-vdf-solstice isn't on the AUR anymore, so I can't use your workaround. Any other ideas?

It looks like python-vdf-solstice on the AUR has been merged into python-vdf, if you check the AUR page you can see that the upstream URL has been updated to the Solstice version. So you should be able to follow the original instructions with just python-vdf from AUR (which is already listed as a dependency in the PKGBUILD, so no need to install it manually).

@ezrawalker745
Copy link

apologies if i'm missing something with a lot of the complex talk in this issue.
I installed the beta on steamdeck through commandline (i couldn't install flathub beta for some reason), when opening files through protontricks launcher, i am still met with the invalid file magic number error. checking protontricks itself, the files i open can be run through explorer. i am not sure if this is some mistake on my part.

@karlosvoneous1968
Copy link

karlosvoneous1968 commented Sep 13, 2024

I entered this into the konsole:

flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo
flatpak install flathub-beta com.github.Matoking.protontricks

now I get this error when using mod organiser 2 install script on stem deck:

error: Multiple branches available for com.github.Matoking.protontricks, you must specify one of: com.github.Matoking.protontricks/x86_64/stable, com.github.Matoking.protontricks/x86_64/beta
ERROR: missing dependencies protontricks

I am using stable version of steam and protontricks is installed so I don't understand this, any help appreciated, please.

@lzq960605
Copy link

I entered this into the konsole:

flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo flatpak install flathub-beta com.github.Matoking.protontricks

now I get this error when using mod organiser 2 install script on stem deck:

error: Multiple branches available for com.github.Matoking.protontricks, you must specify one of: com.github.Matoking.protontricks/x86_64/stable, com.github.Matoking.protontricks/x86_64/beta ERROR: missing dependencies protontricks

I am using stable version of steam and protontricks is installed so I don't understand this, any help appreciated, please.

flatpak remote-add --user --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo

flatpak install flathub-beta com.github.Matoking.protontricks

@karlosvoneous1968
Copy link

I entered this into the konsole:
flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo flatpak install flathub-beta com.github.Matoking.protontricks
now I get this error when using mod organiser 2 install script on stem deck:
error: Multiple branches available for com.github.Matoking.protontricks, you must specify one of: com.github.Matoking.protontricks/x86_64/stable, com.github.Matoking.protontricks/x86_64/beta ERROR: missing dependencies protontricks
I am using stable version of steam and protontricks is installed so I don't understand this, any help appreciated, please.

flatpak remote-add --user --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo

flatpak install flathub-beta com.github.Matoking.protontricks

@Izq960605 you are a 😇. This worked. A big thank you 👏. I can now mod The Elder Scrolls games.

@M3L6H
Copy link

M3L6H commented Sep 14, 2024

@Staudey Ah fantastic thanks, I've just managed to get it working.

For anyone else looking for a NixOS workaround;

Flake inputs;

    protontricks = {
      url = "github:Matoking/protontricks/appinfo_v29";  
      flake = false;
    };
    vdf-patch = {
      url = "github:Matoking/vdf/support_new_bvdf";
      flake = false;
    };

Overlay;

    nixpkgs.overlays = [
      (final: prev: { 
        vdf-patch = prev.python312Packages.vdf.overrideAttrs (oldAttrs: rec {
          src = inputs.vdf-patch;
        });
        protontricks-beta = prev.protontricks.overrideAttrs (oldAttrs: rec {
          src = inputs.protontricks;
          propagatedBuildInputs = [
            pkgs.python312Packages.setuptools # implicit dependency, used to find data/icon_placeholder.png
            final.vdf-patch
            pkgs.python312Packages.pillow 
          ];
        });
      })
    ];

This is incredibly rough and could probably be done better, but I'm new to nix and it works.

For nixos noobs like me who are confused why this wasn’t working (I was using protontricks installed through protontweaks), the above overlay creates a package protontricks-beta. It took me way too long to notice that and change it to protontricks. Alternatively, if installing protontricks directly, make sure to specify protontricks-beta in your environment.systemPackages.

@JManch
Copy link

JManch commented Sep 14, 2024

Here's a slightly cleaner overlay for Nix users. It avoids having to add flake inputs and replaces the existing protontricks package:

nixpkgs.overlays = [
  (final: prev: {
    protontricks =
      (prev.protontricks.overrideAttrs {
        src = final.fetchFromGitHub {
          owner = "Matoking";
          repo = "protontricks";
          rev = "f7b1fa33b0438dbd72f7222703f8442e40edc510";
          hash = "sha256-t794WEMJx/JNX3gTMHfgquFWB7yXkleW07+QURm1NPM=";
        };
      }).override {
        vdf = final.python312Packages.vdf.overrideAttrs {
          src = final.fetchFromGitHub {
            owner = "Matoking";
            repo = "vdf";
            rev = "981cad270c2558aeb8eccaf42cfcf9fabbbed199";
            hash = "sha256-OPonFrYrEFYtx0T2hvSYXl/idsm0iDPwqlnm1KbTPIo=";
          };
        };
      };
  })
];

@Kepsz
Copy link

Kepsz commented Sep 14, 2024

@Matoking Well now python-vdf-solstice isn't on the AUR anymore, so I can't use your workaround. Any other ideas?

It looks like python-vdf-solstice on the AUR has been merged into python-vdf, if you check the AUR page you can see that the upstream URL has been updated to the Solstice version. So you should be able to follow the original instructions with just python-vdf from AUR (which is already listed as a dependency in the PKGBUILD, so no need to install it manually).

Yeah, python-vdf is updated in AUR. But I'm a bit confused, what else is needed for protontricks to work with the current version of Steam?

@LarnuUK
Copy link

LarnuUK commented Sep 14, 2024

Any ideas when the release to Beta will be sent to release? The update to Steam is out of beta now, and is part of the release version, so it would be great to see this fix go out soon.

@chewi
Copy link

chewi commented Sep 15, 2024

I agree, but we need to work out what's happening with ValvePython/vdf#61. If they don't merge and release those changes, they will be need to be published on PyPI under a new name with a new maintainer.

@Jademalo
Copy link

@JManch Oh neat, I didn't think to override vdf only within the protontricks build like that, that's definitely a slightly cleaner way of doing it and means you don't have to redefine the propagated build inputs to prevent the base vdf being overwritten.

Here's that cleaner version with flakes, and with the environment.systemPackages bit added. For reference @M3L6H , I don't personally like overwriting base packages with beta releases personally, which is why I used protontricks-beta instead of overwriting it. Apologies if you were tripped up by that!

protontricks = {
  url = "github:Matoking/protontricks/appinfo_v29";  
  flake = false;
};
vdf = {
  url = "github:Matoking/vdf/support_new_bvdf";
  flake = false;
};
nixpkgs.overlays = [
  (final: prev: {
    protontricks-beta = (prev.protontricks.overrideAttrs {
      src = inputs.protontricks;
    }).override {
      vdf = final.python312Packages.vdf.overrideAttrs {
        src = inputs.vdf-patch;
      };
    };
  })
];
environment.systemPackages = [pkgs.protontricks-beta];

Matoking added a commit that referenced this issue Sep 15, 2024
Steam beta introduced a new version of appinfo.vdf with a space-saving
optimization. Field keys are stored in a separate table at the end of
the file, with the actual VDF segments having to be parsed using the
table to map the indices to actual field names.

`vdf` library does not support serializing appinfo.vdf using this
format, at least yet, so just use appinfo.vdf V28 in tests for the time
being. This might need to be fixed in the future once appinfo.vdf V28 is
phased out.

Fixes #304
@Matoking
Copy link
Owner

Any ideas when the release to Beta will be sent to release? The update to Steam is out of beta now, and is part of the release version, so it would be great to see this fix go out soon.

I'll look into publishing a version of Protontricks with a vendored copy of vdf with the necessary fixes applied. Distro package maintainers probably won't like that, but the current situation is far from ideal.

@chewi
Copy link

chewi commented Sep 15, 2024

I'll look into publishing a version of Protontricks with a vendored copy of vdf with the necessary fixes applied. Distro package maintainers probably won't like that, but the current situation is far from ideal.

I am a distro maintainer. I've dealt with it in Gentoo for now by pointing our vdf package at your fork.

Matoking added a commit that referenced this issue Sep 16, 2024
Steam beta introduced a new version of appinfo.vdf with a space-saving
optimization. Field keys are stored in a separate table at the end of
the file, with the actual VDF segments having to be parsed using the
table to map the indices to actual field names.

`vdf` library does not support serializing appinfo.vdf using this
format, at least yet, so just use appinfo.vdf V28 in tests for the time
being. This might need to be fixed in the future once appinfo.vdf V28 is
phased out.

Fixes #304
@Matoking
Copy link
Owner

Matoking commented Sep 16, 2024

Protontricks 1.12.0 is now out which contains a fix for this issue. Thanks for your patience!

If you followed the installation instructions in this thread, uninstall the beta version from Flathub and use the default stable branch instead. The beta branch won't be maintained anymore.

Also, as always, it might take a few hours for the new version to be published on Flathub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests