Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore CRs that are not in control of the OSSMC installation. #712

Merged
merged 1 commit into from
Nov 13, 2023

Conversation

jmazzitelli
Copy link
Contributor

The first CR created (i.e. the oldest one) is the one that controls OSSMC.

fixes: kiali/kiali#6792

@jmazzitelli jmazzitelli self-assigned this Oct 27, 2023
@jmazzitelli
Copy link
Contributor Author

jmazzitelli commented Oct 27, 2023

I'm not sure if this will do what I want. This is just my first attempt at this. I'll test it and see what breaks.
UPDATE: I ran a long set of commands to exercise this stuff and it all seems to work. See the followup comment below with the full set of instructions on how to test this PR.

@jmazzitelli jmazzitelli force-pushed the 6792-ignore-newer-crs branch 3 times, most recently from 3400bf8 to c0f853e Compare October 27, 2023 21:49
@jmazzitelli
Copy link
Contributor Author

jmazzitelli commented Oct 27, 2023

How to test to see the behavior of this PR.

  1. Create an OpenShift cluster and install Istio
    hack/crc-openshift.sh start && hack/istio/install-istio-via-istioctl.sh -c oc
  2. Build and push the Kiali and operator images
    make build build-ui cluster-push
    NOTE: you must log into the CRC openshift image registry for the push to work - run make cluster-status and the end of the output shows you the login command if you forget what it is
  3. Build and push the OSSMC plugin image - don't forget to change your CWD back to the kiali repo when you are done
    cd <your local ossmc plugin repo> && make cluster-push
  4. Create a couple namespaces to be used in the test
    oc create ns ossmc1 && oc create ns ossmc-another
  5. Run the operator locally (you need ansible installed on your box along with the ansible collections)
    make run-operator
  6. The operator is now running in the foreground. You have to start another command line window to do the rest.
  7. Install Kiali (because OSSMC will not install without Kiali already installed)
    make kiali-create
  8. Wait for Kiali to start:
    oc wait pod --for condition=Ready -n istio-system -l app=kiali
  9. Create the first OSSMConsole CR which will install OSSMC:
    make -e OSSMCONSOLE_NAMESPACE=ossmc1 ossmconsole-create
  10. Wait for the CR's status to say it is all done.
    oc get ossmconsole ossmconsole -n ossmc1 -ojsonpath={.status} | jq
  11. Make sure the OSSMC pod is running in the ossmc1 namespace now:
    oc get pods -n ossmc1
  12. The list of plugins deployed in OpenShift Console should include OSSMC now:
$ oc get consoles.operator.openshift.io cluster -o jsonpath='{.spec.plugins}{"\n"}'
["ossmconsole"]
  1. Now create a second OSSMConsole CR - this should be ignored. Only one CR should be created; any more and the operator should ignore them. Only the oldest CR will be processed. This second one will be ignored
    make -e OSSMCONSOLE_NAMESPACE=ossmc-another ossmconsole-create
  2. Look at the second CR's status and it should have a message about it being ignored
$ oc get ossmconsole ossmconsole -n ossmc-another -ojsonpath='{.status.progress.message}{"\n"}'
1. Ignoring this CR. The CR in control of OSSMC is [ossmc1/ossmconsole].
  1. Make sure nothing is in the ossmc-another namespace - the operator should not have deployed OSSMC here
    oc get pods -n ossmc-another
  2. The list of plugins deployed in OpenShift Console should still include OSSMC:
$ oc get consoles.operator.openshift.io cluster -o jsonpath='{.spec.plugins}{"\n"}'
["ossmconsole"]
  1. Very important is that the operator will not continually attempt to reconcile this second CR. So look at the output of the make run-operator in your first command line window - it should not be continually spewing log messages. It should be paused and waiting on more changes to CRs.
  2. Now delete that second CR and make sure it can be deleted but the operator will not uninstall OSSMC:
    make -e OSSMCONSOLE_NAMESPACE=ossmc-another ossmconsole-delete
  3. Confirm the CR is now gone:
    oc get ossmconsole ossmconsole -n ossmc-another
  4. The list of plugins deployed in OpenShift Console should still include OSSMC - it should not have been uninstalled:
$ oc get consoles.operator.openshift.io cluster -o jsonpath='{.spec.plugins}{"\n"}'
["ossmconsole"]
  1. To make sure the operator processes the original CR correctly even if a second CR exists, create that second CR again (it will be ignored again)
    make -e OSSMCONSOLE_NAMESPACE=ossmc-another ossmconsole-create
  2. Now delete the first CR - this is the controlling CR so removing it should cause OSSMC to be uninstalled
    make -e OSSMCONSOLE_NAMESPACE=ossmc1 ossmconsole-delete
  3. Make sure the OSSMC pod is now terminated and gone from the ossmc1 namespace:
    oc get pods -n ossmc1
  4. The list of plugins deployed in OpenShift Console should not include OSSMC:
$ oc get consoles.operator.openshift.io cluster -o jsonpath='{.spec.plugins}{"\n"}'
[]
  1. Here's the interesting thing now. That second CR is the only one left and thus the oldest. It is considered the controlling CR. But OSSMC has been uninstalled. So this CR can be deleted and it will try to uninstall OSSMC again. Fortunately, this is ultimately a no-op since the operator will just try to uninstall something that isn't there. So delete the CR, and see that nothing breaks:
    make -e OSSMCONSOLE_NAMESPACE=ossmc-another ossmconsole-delete
  2. There should be no CRs left:
    oc get ossmconsole --all-namespaces
  3. The list of plugins deployed in OpenShift Console should not include OSSMC:
$ oc get consoles.operator.openshift.io cluster -o jsonpath='{.spec.plugins}{"\n"}'
[]

The test is done. You can delete the Kiali CR (make kiali-delete) and when that is done you can Control-C the make run-operator process to kill the operator.

@jmazzitelli jmazzitelli marked this pull request as ready for review October 27, 2023 23:13
@jmazzitelli
Copy link
Contributor Author

No need to get this into the 1.76 release. Let's wait for the following sprint.

The first CR created (i.e. the oldest one) is the one that controls OSSMC.

fixes: kiali/kiali#6792
@ScriptingShrimp
Copy link

just FYI, on OCP I do get following

The Kubernetes version v1.24.16+ec2a592 is not supported by Istio 1.20-alpha.9887d5dd7e6cac976e40e1f027f936cccc8c0ab5. The minimum supported Kubernetes version is 1.25.
Proceeding with the installation, but you might experience problems. See https://istio.io/latest/docs/setup/platform-setup/ for a list of supported versions.

WARNING: Istio is being upgraded from 1.19.3 to 1.20.0.
         Running this command will overwrite it; use revisions to upgrade alongside the existing version.
         Before upgrading, you may wish to use 'istioctl x precheck' to check for upgrade warnings.

@ScriptingShrimp

This comment was marked as outdated.

@ScriptingShrimp

This comment was marked as outdated.

@jmazzitelli

This comment was marked as outdated.

@ScriptingShrimp
Copy link

I go oc login <url of OCP cluster> , export DORP=podman since docker does not work for me and make clean-all
Than I follow your steps, from Kiali directory:

  1. hack/istio/install-istio-via-istioctl.sh -c oc installs Istio 1.19 OK
  2. make cluster-push fails with
...
Successfully tagged quay.io/kiali/kiali-operator:dev
5cac215765fbe70bce5a05528c1d9c095aadc66c484e6599534ab4fc4d0d7954
make[1]: Leaving directory '/home/scsh/work/scriptingShrimp/kiali/operator'
Re-tag the already built Kiali operator container image for a remote cluster using podman
podman tag quay.io/kiali/kiali-operator:dev default-route-openshift-image-registry.apps.pmarektst.maistra.upshift.redhat.com/kiali/kiali-operator:dev
Pushing Kiali operator image to remote cluster using podman: default-route-openshift-image-registry.apps.pmarektst.maistra.upshift.redhat.com/kiali/kiali-operator:dev
podman push --tls-verify=false default-route-openshift-image-registry.apps.pmarektst.maistra.upshift.redhat.com/kiali/kiali-operator:dev
Getting image source signatures
Error: trying to reuse blob sha256:bccb88911f57c99be76f3ea4814f624921377f67626e556cb789c0a180e7de32 at destination: checking whether a blob sha256:bccb88911f57c99be76f3ea4814f624921377f67626e556cb789c0a180e7de32 exists in default-route-openshift-image-registry.apps.pmarektst.maistra.upshift.redhat.com/kiali/kiali-operator: authentication required
make: *** [make/Makefile.cluster.mk:224: cluster-push-operator] Error 125

this error goes away after podman login --tls-verify=false -u kubeadmin -p $(/usr/bin/oc whoami -t) default-route-openshift-image-registry.apps.pmarektst.maistra.upshift.redhat.com
2. rerunning make cluster-push fails with Apparently, the kiali UI is not built. Build the front-end by running 'yarn && yarn build' inside the kiali UI directory
cd frontend && yarn && yarn build - build OK, running again make cluster-push fails on Preparing container image files cp: cannot stat '/home/scsh/go/bin/kiali*': No such file or directory make: *** [make/Makefile.container.mk:11: .prepare-kiali-image-files] Error 1 so running make build to build the binary and rerunnig again make cluster-push - now it passes
3. from openshift-servicemesh-plugin folder running make cluster-push OK
4. OK

@ScriptingShrimp
Copy link

before executing step no. 5 I installed Ansible, but ansible-galaxy collection install -r requirements.yml --force-with-deps command hangs (I don't have working token I guess, so its doing nothing)
5. make looks good, but I'm stuck at the same error during make run-operator:

Waiting for the OSSMConsole CRD to be established
/usr/bin/oc wait --for condition=established --timeout=60s crd ossmconsoles.kiali.io
customresourcedefinition.apiextensions.k8s.io/ossmconsoles.kiali.io condition met
ansible-galaxy collection install -r /home/scsh/work/scriptingShrimp/kiali/operator/requirements.yml --force-with-deps
Starting galaxy collection install process
Process install dependency map

with verbose setting I did get two extra line

Process install dependency map
Opened /home/scsh/.ansible/galaxy_token
Galaxy token file /home/scsh/.ansible/galaxy_token malformed, unable to read it

without token it just ends before "malformed" error

@jmazzitelli
Copy link
Contributor Author

Right, anytime you use our dev make target "cluster-push" you have to ensure you are logged into the OpenShift image registry. When you run "make cluster-status", the output gives you the image registry login command in case you forget what that command is.

And yes, before you can push the image, you have to build it with "make build build-ui".

I wrote those test instructions with the assumption a Kiali dev would be doing this review, which is why I guess I skipped the more obvious steps - they are only obvious to the devs :)

@jmazzitelli
Copy link
Contributor Author

Sometimes the Ansible Galaxy server is down. It is possible that's what your token issue is.

You do not have to run the operator locally for this test. You can deploy it into the cluster. Just change the instructions a bit - and when it tells you to look at the output of the operator, you just look at the operator pod logs instead.

To deploy the Kiali operator in the cluster, use make operator-create (that would replace step 5).

@ScriptingShrimp
Copy link

step 13. did failed for me
step 14. returns 5. Finished
step 15. returns

NAME                           READY   STATUS    RESTARTS   AGE
ossmconsole-6b7868bbcf-t7vdm   1/1     Running   0          8m55s

this might be caused by interchanging step 5 make run-operator for make operator-create - since I do not deploy operator built from this branch.

I give up since building operator does not work for me.

Copy link
Contributor

@jshaughn jshaughn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOK. Is this actually required in master if we stay with 1.73 for remaining 2.x releases?

@jmazzitelli
Copy link
Contributor Author

jmazzitelli commented Nov 10, 2023

Is this actually required in master if we stay with 1.73 for remaining 2.x releases?

We can put this in master to get this in 1.77... we don't necessarily need it in 1.73 (but there is a cherry pick PR to get it into 1.73 - it is a nice-to-have)

@jmazzitelli jmazzitelli merged commit 8fad9d2 into kiali:master Nov 13, 2023
1 check passed
@jmazzitelli jmazzitelli deleted the 6792-ignore-newer-crs branch November 13, 2023 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

[operator] only process one OSSMConsole CR
3 participants