0dt upgrades: environmentd: fatal: incompatible persist version 0.115.0-dev.0, current: 0.114.0, make sure to upgrade the catalog one version forward at a time

[def-]

What version of Materialize are you using?

75a54f5

What is the issue?

I believe what is happening is that #29025 and #29035 don't work together
#29025 tries to exit with 0 when a deployment has been fenced out, but #29035 again causes us to panic (exit code 134), so environmentd is rebooted again and again. This causes the upgrade test failures in https://buildkite.com/materialize/nightly/builds/9225

platform-checks-mz_2-1              | environmentd: fatal: incompatible persist version 0.115.0-dev.0, current: 0.114.0, make sure to upgrade the catalog one version forward at a time
platform-checks-mz_2-1              | thread 'main' panicked at src/environmentd/src/bin/environmentd/main.rs:604:9: environmentd: fatal: incompatible persist version 0.115.0-dev.0, current: 0.114.0, make sure to upgrade the catalog one version forward at a time

I'm not sure what the cleanest way to fix this in code is, maybe the catalog version check should use exit!(0, ... instead of panicking. Or the exit!(0, "this deployment has been fenced out"); should happen earlier? The tests still seem to run fine, the old environmentd just keeps restarting and panicking causing logs to fill up with garbage, so for now from testing side I'll silence the panics until we have a better solution

This started occurring now because the release version was bumped and thus the older version we pick also contains #29035 . This is a general problem where we only notice problems with the catalog versioning when we actually bump the release version. Maybe we should have a test that creates a new build with the next version and tries upgrading to it, so that we can catch those issues earlier. I'm not that worried about test issues, but product issues could also hide like that. I'll give it a try.

[jkosh44]

IMO failing the version check should be treated as a panic and use the same exit code as all other panics. I'll look into why it's exiting with 134 and see if I can update it to use exit code 0.

[jkosh44]

Another tricky thing is testing the fix. It looks like it's version 0.114.0 that's stuck in a crash loop, so no changes we make will take affect until it's included in a release.

[def-]

That's why I made #29201, which allows us to test it immediately by creating a new version with the next minor. It seems to work: https://buildkite.com/materialize/nightly/builds/9232

[jkosh44]

The issue seems to be that if the error bubbles up to main, then it will unconditionally panic instead of potentially halting on certain errors.

materialize/src/environmentd/src/bin/environmentd/main.rs

Lines 603 to 605 in 62d8ae7

	if let Err(err) = run(args) {
	panic!("environmentd: fatal: {}", err.display_with_causes());
	}

[jkosh44]

I had a discussion with @benesch and this is my understanding of the issue. Before #29035, 0dt worked like this:

1. Envd version n is running.
2. We start envd version n + 1, which eventually increments the epoch and binary version in the catalog.
3. Envd version n sees a higher epoch in the catalog and halts, causing the process to exit with a non-zero exit code.
4. Envd version n is restarted because it exited with a non-zero exit code.
5. Envd version n initializes the catalog.
6. Envd version n enters the preflight checks, looks at the most recent deploy generation in the catalog, sees that it's larger than the deploy version it was started with, and exits with a zero exit code.
7. Envd version n is NOT restarted because it exited with a zero exit code.

After #29035, we see the same first four steps, and continue like this:

5. Envd version n tries to initialize the catalog, but fails because the catalog has been opened with a higher binary version, and exits with a non-zero exit code.
6. Go back to step (4).

So we end up in an infinite crash loop.

My proposal to fix this is the following:

• The durable catalog will store in memory the deploy generation it was started with.
• In the durable catalog, instead of immediately returning a fence error, always drain all updates from the persist shard. If there is a deploy generation greater than the deploy generation stored in memory return a DeployGenerationFence error, otherwise return the original fence error.
• Whenever the adapter/Coordinator/envd sees a DeployGenerationFence error, exit with exit code 0.

[benesch]

My proposal to fix this is the following:

• The durable catalog will store in memory the deploy generation it was started with.
• In the durable catalog, instead of immediately returning a fence error, always drain all updates from the persist shard. If there is a deploy generation greater than the deploy generation stored in memory return a DeployGenerationFence error, otherwise return the original fence error.
• Whenever the adapter/Coordinator/envd sees a DeployGenerationFence error, exit with exit code 0.

Proposal LGTM. I put up #29214 to tackle the last bullet—hoping you can fill in the first two bullets next week, Joe!

[benesch]

Reopening since #29269 was reverted.