Why did this server migration fail?

[johnheenan]

Quting from a previous answer at #9073 (comment)

And an IP change normally doesn't cause an interruption in the p2p connection

OK. Keys and signatures have been exchanged and verified. A home modem reboots. New IP address. New type 257 node_announcment message with a NAT IP address for propagation. Peer accepts because message signature is OK and a channel is open, even when sending IP address is different, because what the IP address is is not relevant, only the verifable signature is. Gossip is propagated.

What about the following. Home modem disconnects. Home server reboots. Home modem comes back online with a new IP address. Since channel keys can be verified by channel peer, all the channels come straight back up but non channel nodes take time to get updated information. Is this correct?

The reason I am asking this is because a node owner got results below when they migrated to another server with another IP address (I assume on a VPS, so not behind a modem using NAT). Results at #9070 (reply in thread)

    "num_pending_channels": 0,
    "num_active_channels": 0,
    "num_inactive_channels": 0,
    "num_peers": 12,

No channels reopened and none pending.

But this when they went back to the previous host (which I assume was offline when the above results were obtained).

    "num_pending_channels":  0,
    "num_active_channels":  11,
    "num_inactive_channels":  0,
    "num_peers":  12,

These are some repeated log messages from #9070 (comment) for the new server that failed to reopen channels

Pruned peer *redacted* from persistent connections, peer has no open channels
Pruned unconnected node
resend failed: unable to fetch channel sync messages for peer 

What is the best conclusion as to what happened?

1. A longer timeout should have been allowed between shutting down old server and starting new server?
2. The data migration was faulty?
3. The old server was not offline?
4. Something else?

[guggero]

This has likely nothing to do with peers and/or connections. If no channels are shown (neither pending, active or inactive), it means the channel database hasn't been migrated properly (the channel.db file).

Channels being opened/closed is an on-chain activity, so it involves an on-chain transaction. Whether a channel is active or not depends on it being connected with the peer.

We have a node migration guide, was that followed?

[johnheenan]

Thanks for your response. I will pass the infomration on.

[johnheenan]

The migration was done by a script. The approach is to dump a database to backup, shut down docker containers, copy across database backup and docker volumes to other server, start database container, restore databse, start other containers.

While it is likely the channel.db was copied across correctly at the time, perhaps the database missed an essential update because it was backed up before LND was shut down. Will have a careful look at your migration guide.

[guggero]

it was backed up before LND was shut down

That definitely does NOT work with the default database format (bbolt) that lnd uses.

[johnheenan]

Sorry, the database backed up is postgres and does not include lnd data. Will probably run my own migration tests to pinpoint how the migration data is becoming faulty.

Thanks for your help. My understanding has improved immensely and I know not what to waste time on.

[johnheenan]

One line fix at btcpayserver/btcpayserver-docker#928

No channels were opened because channels.db was excluded in the backup for migration

Thanks again