It takes 4+ hours for operator to successfully reconcile on some clusters

[ibuziuk]

Describe the bug

2024-09-26T14:01:43Z INFO Binary info {"Go version": "go1.19.13"}
2024-09-26T14:01:43Z INFO Binary info {"OS": "linux", "Arch": "amd64"}
2024-09-26T14:01:43Z INFO Address {"Metrics": ":60000"}
2024-09-26T14:01:43Z INFO Address {"Probe": ":6789"}
2024-09-26T14:01:43Z INFO Operator is running on {"Infrastructure": "OpenShift v4.x"}

time="2024-09-26T18:18:43Z" level=info msg="Deployment che-gateway is in the rolling update state."
time="2024-09-26T18:18:44Z" level=info msg="Deployment che-gateway is in the rolling update state."
2024-09-26T18:19:03Z INFO sync Object updated {"namespace": "crw", "kind": "v1alpha1.DevWorkspaceOperatorConfig", "name": "devworkspace-config"}
time="2024-09-26T18:19:04Z" level=info msg="Custom resource status devspaces updated with version: 3.16.1"
time="2024-09-26T18:19:04Z" level=info msg="Successfully reconciled."
time="2024-09-26T18:19:06Z" level=info msg="Successfully reconciled."

Che version

7.92@latest

Steps to reproduce

Rollout a new version of operator

Expected behavior

Successful reconciliation is fast

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub

Environment

Dev Sandbox (workspaces.openshift.com)

Eclipse Che Logs

No response

Additional context

No response

[tolusha]

I am thinking about adding more debug logs to find the root of the problem.

[1] https://github.com/eclipse-che/che-operator/blob/main/pkg/deploy/reconcile_manager.go#L57