interop: local head tracking

[axelKingsley]

It appears that the Chain Monitors, while tracking new log events, do not track the head pointers in a way that is usable by the rest of the system. Here I write as much context as I can:

Cross Safe Maintenance

The ChainsDB currently has a routine which runs Cross-Safety Maintenance. It attempts to advance the x-safety pointer as far as possible, and updates the value when its done. The x-safety pointer can't go beyond the local-safety pointer.

It was here we noticed that cross safety was never advancing, and it appears to be due to the local heads never advancing either.

Local Head Pointers

We expect/desire that there are pointers into the database which represent the local safety of a block of messages. We would want this data to write into the Heads Storage, which holds all these pointers as index values into the database (which is why I'm calling them pointers)

The current flow is:

Chain Monitor ->
Head Monitor ->
Update Processor "callback"

** Issue 1**: The Update Processor only has one underlying processor at the moment, the Unsafe Processor. We will need to add processors to the Safe and Finalized stages in order to process those updates at all.

Inside this underlying unsafeBlockProcessor (of type HeadUpdateProcessor) we have a notion of which block number we are on for the sake of making the right data requests. Once we get all the blocks, we unroll them into their receipts and logs, and append them all to the database.

Issue 2: Appending to the database happens in a space where the Heads Storage is totally untracked. Chain Monitors have a store (underlying is a ChainsDB), so maybe we can use that directly...

Another potential solution would be for ChainsDB to maintain local heads asynchronously, in the same way as it maintains x-heads. If every block left a checkpoint in the database, the local update routine could just use the Monitor's "latest" block number, resolve that to an index, and use that as the local head.

[axelKingsley]

Going to journal in this ticket because I think this might take a minute, and I'd like to make sure this work is visible for potential contribution.

Currently

I've made some changes to my local (based on this PR):

• Rather than interact with a LogStorage, Chain Monitors now interact with Storage which includes a minimal interface to apply new Chain Heads to the Heads Storage
• I've added a bunch of debug logging, some of which I might promote to actual logs at the end of the investigation.
• I've added an update to the chain heads at the end of ProcessLogs, which acts on one block at a time. For now this assumes an unsafe heads update, which is fine because that's where the data is coming from when logs are updated.

As Safe and Finalized get added, we can make those just do a Heads update based on the database state. these processors can be responsible for responding to reorg when the data is available.

Findings so far:
Looks like there's more bootstrapping issue. When the database is empty, it's very hard to add a new items, because there's no checkpoints. This causes things like 0-value iterators which EOF immediately, and exacerbate issues. I plan on making fixes to how these db functions work, or to how the traversal works.

Log snippet:

XLXL: Apply before  &{map[900200:{0 0 0 0 0 0} 900201:{0 0 0 0 0 0}]}
XLXL doing log &{0xa5657c5380C0c5B67dD935920C53816c7E9D7691 [0xe00bbfe6f6f8f1bbed2da38e3f5a139c6f9da594ab248a3cf8b44fc73627772c 0xb3647df63b3983e35710f3a5e8d915c0e79d23d66e1ebb44161b2d216640d2b4] [] 2 0x431ae55003703d72a9f27753614735ba1fabf00d1d8948153034256c292637bb 3 0xd1ecf85e746cb3b421478c63618e55a9d647d1f3dc9f26de65c7a6ad835e1c40 0 false}
XLXL AddLog
XLXL AddLog Done {2 0}
XLXL lastBlock  2
XLXL: LastLogInBlock 900201 2 &{0x14002a246c0 2 {2 0} false 0}
XLXL: NextLog 2 0 <nil>
XLXL: NextLog 0 0 EOF
XLXL lastIndex err block 2 not found in chain 900201
XLXL: Apply after  &{map[900200:{0 0 0 0 0 0} 900201:{0 0 0 0 0 0}]}
[... a later write]
XLXL adding log %v in block %v, but currently at log %v 0 2 0

Explanation:

• All of AddLog happened and the log context now points at block 2
• an attempted check for the last index of the block (which would be used to update the local head) fails because without a checkpoint, the table can't find the block
• The Heads Storage never updates
• And then in subsequent attempted writes, we hit an "out of order" error in adding to the database!

[axelKingsley]

Another thing that's turned out to be required is exposing LastEntryIdx -- when EOFs are hit when scanning for Executing Messages, we need to use the last entry in the DB. I've exposed that to all relevant interfaces.

[axelKingsley]

There's a core issue when attempting to get a search checkpoint near the front of the database. if the index is 0, it's assumed there was no checkpoint before-or-at the requested block number.

[axelKingsley]

Ok, I've got things in place such that local heads are tracking, and x-heads are following as expected. Here's a snippet:

XLXL Latest Chain Heads:  &{map[900200:{2 0 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{3 2 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{6 3 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{7 6 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{10 7 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{10 10 0 0 0 0} 900201:{2 0 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{10 10 0 0 0 0} 900201:{3 2 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{10 10 0 0 0 0} 900201:{6 3 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{10 10 0 0 0 0} 900201:{7 6 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{10 10 0 0 0 0} 900201:{10 7 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{11 10 0 0 0 0} 900201:{10 10 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{14 11 0 0 0 0} 900201:{10 10 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{15 14 0 0 0 0} 900201:{10 10 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{15 15 0 0 2 0} 900201:{10 10 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{15 15 2 0 2 0} 900201:{10 10 0 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{15 15 2 0 2 0} 900201:{10 10 2 0 0 0}]}
XLXL Latest Chain Heads:  &{map[900200:{15 15 2 0 2 0} 900201:{10 10 2 0 2 0}]}
XLXL Latest Chain Heads:  &{map[900200:{18 15 2 2 2 2} 900201:{10 10 2 2 2 2}]}
XLXL Latest Chain Heads:  &{map[900200:{19 18 2 2 2 2} 900201:{10 10 2 2 2 2}]}
XLXL Latest Chain Heads:  &{map[900200:{19 19 2 2 2 2} 900201:{11 10 2 2 2 2}]}
XLXL Latest Chain Heads:  &{map[900200:{19 19 2 2 2 2} 900201:{14 11 2 2 2 2}]}

Explanation: each line is a printout of the ChainsDB's Head structure, just after any update it handles.
ChainA and ChainB are taking turns sending 5 transactions each with logs. The local unsafe head stays promoted to the tip of the database, and the x-unsafe head is constantly updating to follow it.

Eventually a Safe and Finalized head are found on the network for both chains, and their x-heads track there too.

The tech required to make this happen is a new HeadProcessor which is now attached to the Chain Monitor. It uses the last log index for the block to update the local head.

More details in: #11991