ETCD becomes unavailable on update

[alexander-demicev]

See https://gitlab.com/sylva-projects/sylva-core/-/issues/1687 for all details describing the issue

[tmmorin]

thanks for opening this issue

one precision: the symptom is less clear that "etcd isn't available"... what we clearly see is some controllers failing to get leases from the apiServer, but some controller apparently still can, and we aren't sure that reading from the api is an issue

there are "bad certificates" issues in etcd logs, but they possibly are harmless (they relate to a node which isn't anymore an etcd cluster member)

so at this point we aren't really sure what the best issue title is ... ;-)

[tmmorin]

here are two dumps of CI jobs that exhibited the issue:

• https://gitlab.com/sylva-projects/sylva-core/-/jobs/7951588127/artifacts/download (on capo)
• https://gitlab.com/-/project/42451983/uploads/b173fc74cea9cf2d9596e8bb55bd3aff/artifact-7915786264.zip (on capm3)

The relevant info is in management-cluster-dump directory, it has the usual per-ns kubectl cluster-info dumps with pod logs, but also resource dumps for many CRDs, it also has a "clusterctl-describe.txt" which is convenient to see what was CAPI state.