Changes to Otel CRD handling in operator Chart 0.57.0

[swiatekm]

As of operator Chart 0.57.0, operator CRDs are now installed as templates. This was changed to support installing a conversion webhook, as the operator chart added a new OpenTelemetryCollector CRD version in the subsequent 0.58.0 version. See the upgrading guide for more details.

This means that we can't use the CRDs shipped by the operator Chart anymore. The reason is that Helm checks if the definitions exist in the cluster before applying resources. If CRDs are treated as normal resources, this means they won't exist at the time of this check, and installation will fail. See this issue for more details.

The solution to this is to ensure we ship the CRDs as CRDs, and not as templates. To achieve this, we need to disable CRD creation in the operator Chart and ensure they're installed in some other way. There are several ways of accomplishing this:

1. Tell the user to manually install the CRDs at install time. Afterwards, upgrades should work correctly.
2. Ship the CRDs on our own.
3. Depend on just the CRD part of the opentelemetry-kube-stack chart.

In my view, 3 is the best long-term decision. If open-telemetry/opentelemetry-helm-charts#677 is implemented, it would make life even easier for us.