symbolicli quietly fails to symbolize if cabextract is not found in PATH

[tsondergaard]

I'm trying to use symbolicli to analyze minidumps in offline mode and it only symbolizes very few symbols in the stacktraces, the rest come back unresolved.

Environment

Commit 271386c built with cargo build --release on Fedora 37 x86_64.

Steps to Reproduce

My ~/.symboliclirc:

cache_dir = "/home/ts/.symbolicli/cache"

[[sources]]
id = "company"
type = "http"
url = "https://dev.my-company.com/symbols"
layout = { type = "symstore_index2", casing = "default" }

[[sources]]
id = "microsoft"
type = "http"
url = "https://msdl.microsoft.com/download/symbols"

Command executed:

target/release/symbolicli --log-level=info --offline --format=pretty ~/Downloads/general2d-test2-session-ezhiZjhl-pid-820-user-jenkins-1.dmp

Expected Result

I hoped to get a stacktrace back with function names resolved

Actual Result

Functions are mostly not resolved, with a few exceptions:

[ts@spark symbolicator]$ target/release/symbolicli --log-level=info --offline --format=pretty ~/Downloads/general2d-test2-session-ezhiZjhl-pid-820-user-jenkins-1.dmp 
2023-05-01T10:38:12.889493Z  INFO symbolicli: successfully parsed local event
2023-05-01T10:38:12.889527Z  INFO symbolicli: symbolicating minidump
 Frame #0                                                
         Trust:  context 
   Instruction:  0x7ffdbadd0cf1 
        Module:  Qt5Widgets.dll +0x40cf1 
 Frame #1                                                
         Trust:  scan 
   Instruction:  0x7ffdbadd67b4 
        Module:  Qt5Widgets.dll +0x467b4 
 Frame #2                                                
         Trust:  scan 
   Instruction:  0x7ffe180efde5 
        Module:  ucrtbase.dll +0xfde5 
      Function:  malloc_base + 0x35 
 Frame #3                                                
         Trust:  cfi 
   Instruction:  0xffffffff 
 Frame #4                                                
         Trust:  scan 
   Instruction:  0x7ffdbadca5ec 
        Module:  Qt5Widgets.dll +0x3a5ec 
 Frame #5                                                
         Trust:  scan 
   Instruction:  0x7ffdbadc16e6 
        Module:  Qt5Widgets.dll +0x316e6 
 Frame #6                                                
         Trust:  scan 
   Instruction:  0x7ffdbadc1465 
        Module:  Qt5Widgets.dll +0x31465 
 Frame #7                                                
         Trust:  scan 
   Instruction:  0x7ffdda641244 
        Module:  Qt5QuickWidgets.dll +0x1244 
 Frame #8                                                
         Trust:  scan 
   Instruction:  0x7ffdba9e4cf7 
        Module:  Qt5WebEngineWidgets.dll +0x14cf7 
 Frame #9                                                
         Trust:  scan 
   Instruction:  0x7ffdba9e747f 
        Module:  Qt5WebEngineWidgets.dll +0x1747f 
 Frame #10                                               
         Trust:  scan 
   Instruction:  0x7ffda613c560 
        Module:  Qt5WebEngineCore.dll +0x556c560 
 Frame #11                                               
         Trust:  scan 
   Instruction:  0x7ffdba9d59b9 
        Module:  Qt5WebEngineWidgets.dll +0x59b9 
...
...
 Frame #38                                               
         Trust:  cfi 
   Instruction:  0x1dcf2390000 
 Frame #39                                               
         Trust:  scan 
   Instruction:  0x7ffe1a95b7ff 
        Module:  ntdll.dll +0x2b7ff 
      Function:  RtlpLowFragHeapAllocFromContext + 0x1af 
 Frame #40                                               
         Trust:  cfi 
   Instruction:  0x1dcd7a2d190 
...
...
 Frame #53                                               
         Trust:  scan 
   Instruction:  0x7ffda64ca740 
        Module:  Qt5WebEngineCore.dll +0x58fa740 
 Frame #54                                               
         Trust:  scan 
   Instruction:  0x7ffe180efde5 
        Module:  ucrtbase.dll +0xfde5 
      Function:  malloc_base + 0x35 
 Frame #55                                               
         Trust:  cfi 
   Instruction:  0x7ffda4b7e7af 
        Module:  Qt5WebEngineCore.dll +0x3fae7af 
 Frame #56                                               
         Trust:  scan 
   Instruction:  0x7ffe1a955ba0 
        Module:  ntdll.dll +0x25ba0 
      Function:  RtlpFreeHeapInternal + 0x490 

Using the same symbol servers Visual Studio 2022 symbolizes the same minidump correctly.

I had a look in the files in the cache directory and a bunch of them report a weird looking error. I think this may be the cause of the problem?

[ts@spark cache]$ grep -r malformed
objects/v1/b0/6748cb/e13484f30cb8f48e61061899b0d8cf1e1598f803168a48fa845d18c4:malformedNo such file or directory (os error 2)
objects/v1/fe/786073/a671e3740cb068b47c7331feb57b10ba55c4a495e7cc1338828edf8f:malformedNo such file or directory (os error 2)
objects/v1/bb/f7115d/903c7c55ca0e56f9cfbc827ef0f25cb7fcedfeb42ee070d570cf549f:malformedNo such file or directory (os error 2)
objects/v1/61/41b795/384cef38654142b39a4b604beea40e2db0783265f523d2ddc380e90d:malformedNo such file or directory (os error 2)
objects/v1/7b/e672a2/af096997ee5cdba7d620e21052734ca856b45a74a915497c0c88ebbb:malformedNo such file or directory (os error 2)
objects/v1/57/dfb68e/6fbd0d6a466ca90e26be17c5c626204be672a64ddd05f9afaec2dada:malformedNo such file or directory (os error 2)
objects/v1/57/a1196c/ac3c25e3185a4e0637f71d0c36e078dff629c82ec835317f35a8a3d5:malformedNo such file or directory (os error 2)
objects/v1/65/ff4a37/08fa05c12f50bffa8353d22871ce218fbc70f07376614c96431b46cc:malformedNo such file or directory (os error 2)
objects/v1/c5/2642b9/e6e6297508d52025a4e6ffc0d08c4cc4e2a15f781d5806ccf74d7db9:malformedNo such file or directory (os error 2)
object_meta/v1/b0/6748cb/e13484f30cb8f48e61061899b0d8cf1e1598f803168a48fa845d18c4:malformedNo such file or directory (os error 2)
object_meta/v1/fe/786073/a671e3740cb068b47c7331feb57b10ba55c4a495e7cc1338828edf8f:malformedNo such file or directory (os error 2)
object_meta/v1/bb/f7115d/903c7c55ca0e56f9cfbc827ef0f25cb7fcedfeb42ee070d570cf549f:malformedNo such file or directory (os error 2)
object_meta/v1/61/41b795/384cef38654142b39a4b604beea40e2db0783265f523d2ddc380e90d:malformedNo such file or directory (os error 2)
object_meta/v1/7b/e672a2/af096997ee5cdba7d620e21052734ca856b45a74a915497c0c88ebbb:malformedNo such file or directory (os error 2)
object_meta/v1/57/dfb68e/6fbd0d6a466ca90e26be17c5c626204be672a64ddd05f9afaec2dada:malformedNo such file or directory (os error 2)
object_meta/v1/57/a1196c/ac3c25e3185a4e0637f71d0c36e078dff629c82ec835317f35a8a3d5:malformedNo such file or directory (os error 2)
object_meta/v1/65/ff4a37/08fa05c12f50bffa8353d22871ce218fbc70f07376614c96431b46cc:malformedNo such file or directory (os error 2)
object_meta/v1/c5/2642b9/e6e6297508d52025a4e6ffc0d08c4cc4e2a15f781d5806ccf74d7db9:malformedNo such file or directory (os error 2)

The accompanying .txt file in the cache to one of those reporting malformed above points to a debug source file that exists:

[ts@spark cache]$ cat object_meta/v1/c5/2642b9/e6e6297508d52025a4e6ffc0d08c4cc4e2a15f781d5806ccf74d7db9
malformedNo such file or directory (os error 2)[ts@spark cache]$ cat object_meta/v1/c5/2642b9/e6e6297508d52025a4e6ffc0d08c4cc4e2a15f781d5806ccf74d7db9.txt 
scope: global

source: company
location: https://dev.my-company.com/symbols/Qt/Qt5WebEngineCore.dll/63EA5B206c6a000/Qt5WebEngineCore.dl_

Could this possibly be related to the use of layout type "symstore_index2"?

[tsondergaard]

I've found in crates/symbolicator-service/src/caching/fs.rs:

        // States a cache item can be in:
        // * negative/empty: An empty file. Represents a failed download. mtime is used to indicate
        //   when the failed download happened (when the file was created)
        // * malformed: A file with the content `b"malformed"`. Represents a failed symcache
        //   conversion. mtime indicates when we attempted to convert.
        // * ok (don't really have a name): File has any other content, mtime is used to keep track
        //   of last use.

So I guess I have a failed symcache conversion. I have run symbolicli with --log-level=trace without seeing any log messages about failed symcache conversion, so I am still not sure how to troubleshoot this issue.

[tsondergaard]

I tried rebuilding symbolicator today on a different machine and all of a sudden the problem is gone. It now is able to symbolize the minidump correctly. I'll close the issue.

[tsondergaard]

Back on the machine where I first had this problem it still occurs with the same build and I did some troubleshooting and managed to identify the problem. symbolicli tries to run cabextract and if that program is not found in the PATH symbolization quietly fails and the only evidence is "malformedNo such file or directory (os error 2)" in the cache files. I'll reopen the issue - symbolicli should emit a message at log-level warning or error that cabextract was not found.