HelmRelease uninstalled unexpectedly

[larntz]

Today I updated flux (v2.1.1 -> v2.2.2) in a cluster and one of our HelmRelease's was unexpectedly uninstalled/installed.

We tracked this down to a change in targetNamespace, and now know exactly how this behaves in v2.2.2.

There are a couple of interesting factors here which is why I am opening this issue.

1. HelmRelease manifest did not have targetNamespace set, but storageNamespace was set.
2. The output of helm ls -a -n storage-namespace showed the namespace to not be flux-system before the upgrade (I know this because we have two identical clusters).

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
generic-chart   not-storage-ns  1               2024-01-04 18:40:00.041038688 +0000 UTC deployed       0.50.19    1.50.5

I'm not sure how this chart was deployed with helm ls showing a namespace other than the release namespace, but it has been this way for ~2 years. I'm not sure if this was done with flux, the helm cli, or something else (this was before my time).

We've never had this issue when upgrading flux in the past; until version 2.2.2 (and helm controller 0.37.2).

I see the text below as part of the helm controller 0.37.0 release notes. We did not expect this uninstall behavior because we did not specify the targetNamespace in the HelmRelease manifest.

Changes to a HelmRelease .spec which require a Helm uninstall for the changes to be successfully applied are now detected. For example, a change in .spec.targetNamespace or .spec.releaseName.

I have a few questions about this and how we might prevent or detect it before deploying.

1. I am wondering if there is some way we could prevent the uninstall behavior. I didn't see anything in the HelmRelease spec that seemed like it would work, but I wanted to ask in case I'm missing something or there is another way we could prevent this in the future.
2. I'm also wondering if could point me to a code change that would have caused this. Previous versions of flux either did not detect this as a targetNamespace change or ignored it. I didn't see any warnings/errors logged about a targetNamespace mismatch or that in the future this would behave as an uninstall/install. I'd like to understand how it worked previously vs how it works now with regard to uninstall when the targetNamespace changes.
3. Lastly, is there a way to do a dry-run with a flux HelmRelease to know if it will perform uninstall or just upgrade?

Appreciate any information you can provide would be greatly appreciated. Thank you!

[stefanprodan]

A reinstall is performed if any of these fields change:

helm-controller/internal/action/verify.go

Lines 51 to 74 in 0114b20

	// ReleaseTargetChanged returns a reason and true if the given release and/or
	// chart name have been mutated in such a way that it no longer has the same
	// release target as recorded in the Status.History of the object, by comparing
	// the (storage) namespace, and release and chart names.
	// This can be used to e.g. trigger a garbage collection of the old release
	// before installing the new one.
	// If no change is detected, an empty string is returned along with false.
	func ReleaseTargetChanged(obj *v2.HelmRelease, chartName string) (string, bool) {
	cur := obj.Status.History.Latest()
	switch {
	case obj.Status.StorageNamespace == "", cur == nil:
	return "", false
	case obj.GetStorageNamespace() != obj.Status.StorageNamespace:
	return targetStorageNamespace, true
	case obj.GetReleaseNamespace() != cur.Namespace:
	return targetReleaseNamespace, true
	case release.ShortenName(obj.GetReleaseName()) != cur.Name:
	return targetReleaseName, true
	case chartName != cur.ChartName:
	return targetChartName, true
	default:
	return "", false
	}
	}