Sortition anchor block affirmation map is no longer compatible with heaviest affirmation map

[rafaelcr]

Describe the bug
I'm experiencing very strange behavior with my local mainnet Stacks node (2.4.0.0.0)... it's been stuck at block 107054 for days now, and when looking at the logs I detected an infinite loop of it trying to reach the chain tip until this happens (excerpt of a very long log):

stacks-blockchain      | INFO [1686859665.683728] [src/chainstate/coordinator/mod.rs:2051] [chains-coordinator-0.0.0.0:20443] Revalidate already-processed snapshot 822b854d8bbedce9ed2fbef8e2339ab33c79a0bd5dc218c35dd90aa6a2245bf8 height 794500 to have canonical tip b508790be76c1c434bbaec849245f188760ecfa9/9e8584155596f782383cb3584f7a20f279cbe427fe38333dbec6ddcf76e8ff58 height 107054
stacks-blockchain      | INFO [1686859665.763535] [src/net/http.rs:1637] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Handle HTTPRequest, verb: GET, peer_addr: 172.31.0.2:59356, path: /v2/info, query: 
stacks-blockchain      | INFO [1686859665.844653] [src/chainstate/coordinator/mod.rs:2051] [chains-coordinator-0.0.0.0:20443] Revalidate already-processed snapshot db93ae6afc6ca42dcec52b38598f278d91973c04cf52f335756df2439b99b555 height 794501 to have canonical tip b508790be76c1c434bbaec849245f188760ecfa9/9e8584155596f782383cb3584f7a20f279cbe427fe38333dbec6ddcf76e8ff58 height 107054
stacks-blockchain      | INFO [1686859666.182216] [src/chainstate/coordinator/mod.rs:1190] [chains-coordinator-0.0.0.0:20443] Sortition anchor block affirmation map `pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppaa` and/or Stacks affirmation map `ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp` is no longer compatible with heaviest affirmation map pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp in reward cycles 59-61
stacks-blockchain      | INFO [1686859666.186172] [src/chainstate/coordinator/mod.rs:1361] [chains-coordinator-0.0.0.0:20443] Re-playing sortitions starting within reward cycle 60 burn height 792051
stacks-blockchain      | INFO [1686859667.274667] [src/chainstate/coordinator/mod.rs:605] [chains-coordinator-0.0.0.0:20443] Anchor block selected for cycle 60: a4e0f1912185408c0218deab3de15b28c8ba3955/ef05d37d3a64b9b1de3abdefcba10dd53f17cfd1d98c738a31b413ecc03c4dfb (txid 912c1409eb3eb37226937a0f980ce4c0efa6eacf2cf764e23d02366982e45f18)
stacks-blockchain      | INFO [1686859667.274767] [src/chainstate/coordinator/mod.rs:1981] [chains-coordinator-0.0.0.0:20443] Anchor block ef05d37d3a64b9b1de3abdefcba10dd53f17cfd1d98c738a31b413ecc03c4dfb (txid 912c1409eb3eb37226937a0f980ce4c0efa6eacf2cf764e23d02366982e45f18) for reward cycle 59 is affirmed by the network (ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppa), but must be downloaded
stacks-blockchain      | INFO [1686859667.274785] [src/chainstate/coordinator/mod.rs:2116] [chains-coordinator-0.0.0.0:20443] Burnchain block processing will continue in spite of missing affirmed anchor block ef05d37d3a64b9b1de3abdefcba10dd53f17cfd1d98c738a31b413ecc03c4dfb
stacks-blockchain      | INFO [1686859667.276181] [src/chainstate/burn/db/sortdb.rs:3630] [chains-coordinator-0.0.0.0:20443] Begin reward-cycle sortition with absent anchor block=Some((ef05d37d3a64b9b1de3abdefcba10dd53f17cfd1d98c738a31b413ecc03c4dfb, 912c1409eb3eb37226937a0f980ce4c0efa6eacf2cf764e23d02366982e45f18))
stacks-blockchain      | INFO [1686859667.276325] [src/chainstate/coordinator/mod.rs:2051] [chains-coordinator-0.0.0.0:20443] Revalidate already-processed snapshot 224ff234257cbea2a22b81e1aba4e6dcbe1b39c1deb4781bbc77e213e82896ce height 792051 to have canonical tip b508790be76c1c434bbaec849245f188760ecfa9/9e8584155596f782383cb3584f7a20f279cbe427fe38333dbec6ddcf76e8ff58 height 107054

It appears I'm missing a block. I learned there was an announcement for this on testnet here but not sure if this is related for mainnet.

Steps To Reproduce
This node was freshly created from a Hiro archive backup shortly before Stacks 2.4 was activated

Expected behavior
Normal chain tip sync behavior

Environment (please complete the following information):

• OS: Ubuntu
• Node version: 2.4.0.0.0

[wileyj]

i don't think this is a blockchain issue, but i can check to verify - can you give me more details about how you performed the restore of the data (i.e. which specific versions were used?)

[wileyj]

additioanlly, if you have the archive file names that would also be helpful.

[rafaelcr]

@wileyj I'm not sure how to find out the exact date of archives I used... do you know if they leave a trace? I'm using the stacks-blockchain-docker project.

The only file I could find was this one:

-rw-r--r--  1 root     root     5597344722 May  4 14:42 stacks-blockchain-api-pg-15-7.1.9-latest.dump
-rw-r--r--  1 root     root            132 May  4 14:43 stacks-blockchain-api-pg-15-7.1.9-latest.dump.sha256

The SHA file contains:

8f52d773d74647d299aac8222d9c57bfb50d408cc43299b9e15e034bba739c77  /hirosystems/data/stacks-blockchain-api-pg-15-7.1.9-20230504.dump

I believe I used this API version with a 2.4.0.0.0 node from the same date, and later on upgraded my API version to 7.2.1

[wileyj]

As noted on the call, I don't believe this is a blockchain issue. i'll move it to the docker repo

if you could confirm the order of operations for me though, i'd appreciate it.
i.e.
git pull stacks-blockchain-docker
updated .env to use latest versions
ran scripts/seed-chainstate.sh
upgrade api to 7.2.1 a few days later
issue discovered

[wileyj]

@rafaelcr i haven't been able to reproduce this - but incidentally, someone else just had the same issue that i was able to get logs from.
i'm still of the opinion that it's 1 of two things:

1. the archive used wasn't aligned on teh same block for chainstate and the db
2. there was an error importing the postgres data, and the script isn't handling it correctly so it appears everything should work fine.

item 1 may be hard to narrow down because we'd ahve to know exactly which archive files were used, i'm thinking it's 2 but i'm not sure what the error may be.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.