Crash on shutdown in Tracy-enabled builds

[marta-lokhova]

stellar-core v21.1.0 is consistently crashing on shutdown when Tracy is enabled (MacOS, clang with libc++ build)

2024-07-03T10:41:32.726 GD3YI [Ledger INFO] Got consensus: [seq=52406447, prev=1daa13, txs=210, ops=669, sv: [ SIGNED@FT SCV 2 txH: 6d4463, ct: 1720028491, upgrades: [ ] ]]
2024-07-03T10:41:32.728 GD3YI [Tx INFO] applying ledger 52406447 (classic phase: 1 component(s): [{discounted txs:209, ops:668, base_fee:100}], soroban phase: 1 component(s): [{discounted txs:1, ops:1, base_fee:100}])
2024-07-03T10:41:32.806 GD3YI [Ledger INFO] Closed ledger: [seq=52406447, hash=f58197]
^C2024-07-03T10:41:37.398 GD3YI [default INFO] got signal 2, shutting down
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 37.60.244.254:11625, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 194.233.85.232:11625, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 45.63.76.47:11625, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 74.118.143.110:11625, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 152.53.46.120:11625, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 3.93.232.149:11625, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 210.57.210.29:11725, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 54.145.255.114:11625, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 46.243.86.148:11625, reason shutdown
2024-07-03T10:41:37.407 GD3YI [Overlay INFO] Dropping peer 74.118.143.236:11625, reason shutdown
2024-07-03T10:41:38.409 GD3YI [default INFO] Application destructing
2024-07-03T10:41:38.410 GD3YI [default INFO] Joining 11 worker threads
2024-07-03T10:41:38.411 GD3YI [default INFO] Joined all 12 threads
2024-07-03T10:41:38.411 GD3YI [default INFO] Application destroyed
[1]    4563 segmentation fault  stellar-core run --conf  --console

Trace:

0   libsystem_c.dylib             	       0x19667bbd0 strerror_r + 4
1   libc++.1.dylib                	       0x19670a06c std::__1::__do_message::message(int) const + 80
2   libc++.1.dylib                	       0x19670a39c std::__1::system_error::__init(std::__1::error_code const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) + 132
3   libc++.1.dylib                	       0x19670a548 std::__1::system_error::system_error(std::__1::error_code, char const*) + 56
4   libc++.1.dylib                	       0x19670aa08 std::__1::__throw_system_error(int, char const*) + 68
5   libc++.1.dylib                	       0x1966fe72c std::__1::mutex::lock() + 40
6   stellar-core                  	       0x104ea3d20 lock_guard + 12 (lock_guard.h:35) [inlined]
7   stellar-core                  	       0x104ea3d20 lock_guard + 12 (lock_guard.h:34) [inlined]
8   stellar-core                  	       0x104ea3d20 operator new(unsigned long) + 44 (main.cpp:307)
9   libc++.1.dylib                	       0x196702e98 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__grow_by_and_replace(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, char const*) + 152
10  libc++.1.dylib                	       0x196703a64 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::append(char const*, unsigned long) + 112
11  libc++.1.dylib                	       0x19670a3bc std::__1::system_error::__init(std::__1::error_code const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) + 164
12  libc++.1.dylib                	       0x19670a548 std::__1::system_error::system_error(std::__1::error_code, char const*) + 56
13  libc++.1.dylib                	       0x19670aa08 std::__1::__throw_system_error(int, char const*) + 68
14  libc++.1.dylib                	       0x1966fe72c std::__1::mutex::lock() + 40
15  stellar-core                  	       0x104ea3d20 lock_guard + 12 (lock_guard.h:35) [inlined]
16  stellar-core                  	       0x104ea3d20 lock_guard + 12 (lock_guard.h:34) [inlined]
17  stellar-core                  	       0x104ea3d20 operator new(unsigned long) + 44 (main.cpp:307)

It appears to be introduced in #4272

[marta-lokhova]

seems like it's not just in shutdown, I'm hitting the same error in new-db path as well when building with tracy