Queries fail with storaged error - E_LEADER_LEASE_FAILED(-3531)

[b0n0r]

Describe the bug (required)

We have a 3 node baremetal nebula graph setup. Running latest 3.8.0 (git 74584886). Each node runs all 3 parts (storaged,metad,graphd).

There are 3 spaces, the environment has been running for a while without issues. After power failure and powercycle of all nodes we ended up with 1 of the 3 spaces unable to return query results with the following error: Storage Error: part: 2, error: E_LEADER_LEASE_FAILED(-3531).

Each space has partition number set to 15 and replica factor set to 3.
The spaces are: Space, SpaceAF, SpaceNew.

Queries in spaces (SpaceAF and SpaceNew) work without issues, the space with name "Space" seems to be problematic.

Logs:

E20240527 09:20:20.773510 287491 StorageAccessExecutor.h:47] Traverse failed, error E_LEADER_LEASE_FAILED, part 2
E20240527 09:20:20.773654 287491 StorageAccessExecutor.h:151] Storage Error: part: 2, error: E_LEADER_LEASE_FAILED(-3531).
E20240527 09:20:20.773830 287493 QueryInstance.cpp:151] Storage Error: part: 2, error: E_LEADER_LEASE_FAILED(-3531)., query: MATCH p=(a)-[r]-(b) WHERE id(a)=='xxx-xxx-xxx' RETURN a,r,b LIMIT 10;

Host	Port	Status	Leader count	Leader distribution	Partition distribution	Version
10.y.z.1	9779	ONLINE	9	Space:5, SpaceAF:1, SpaceNew:3	Space:15, SpaceAF:15, SpaceNew:15	3.8.0
10.y.z.2	9779	ONLINE	24	Space:9, SpaceAF:8, SpaceNew:7	Space:15, SpaceAF:15, SpaceNew:15	3.8.0
10.y.z.3	9779	ONLINE	12	Space:1, SpaceAF:6, SpaceNew:5	Space:15, SpaceAF:15, SpaceNew:15	3.8.0

We have also tried to run BALANCE LEADER without success.

Any pointers on how to debug this further?

Your Environments (required)

• OS: Linux Ubuntu 20.04.6 LTS, Kernel 5.4.x x86_64
• Compiler: g++ --version or clang++ --version
• CPU: Intel(R) Xeon(R) Gold
• Commit id 74584886