Cluster self healing have not works with Cilium CNI

[VasilevMaksim]

Create simple cluster with KOPS (https://kops.sigs.k8s.io/getting_started/gce/#creating-a-cluster)
k8s: v1.29.3

Recreate cluster node with gcloud CLI:
$ gcloud compute instances delete --zone=us-central1-a control-plane-us-central1-a-vjvg nodes-us-central1-a-pfb4

Expected Result: Cluster goes back to normal operation

Actual Result: After nodes goes back, cluster network unhealthily.

In case with in-tree cloud provider with feature-gates DisableCloudProviders: false, and cloud-provider=gce flag in the kubelet, api and kube-controller-manager, cluster self healing works properly.

[k8s-ci-robot]

This issue is currently awaiting triage.

If the repository mantainers determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[aojea]

it will be good to have more details on the logs from the cloud controller manager and the kubelet, please also add the name of the node with the problem

[VasilevMaksim]

I have two instances
$ gcloud compute instances list --zones=us-central1-a
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
control-plane-us-central1-a-0095 us-central1-a e2-medium 10.0.16.4 34.27.84.33 RUNNING
nodes-us-central1-a-kmrd us-central1-a e2-medium 10.0.16.3 34.172.132.171 RUNNING

And delete both in one command for self hilling test:
$ gcloud compute instances delete --zone=us-central1-a control-plane-us-central1-a-0095 nodes-us-central1-a-kmrd

After nodes goes up, i check network connectivity and my cluster network have not work

CloudController logs:
kube-system-cloud-controller-manager-nsgjw-1718184872911347000.log

Kubelet logs:
kubelet_control-plane.txt
kubelet_nodes-us-central1-a-kmrd.txt

[aojea]

Jun 12 09:32:16 nodes-us-central1-a-kmrd kubelet[3925]: E0612 09:32:16.217738 3925 pod_workers.go:1298] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized" pod="kube-system/coredns-78ccb5b8c5-wwmx7" podUID="964c96b3-52ad-4a93-991a-5de72c93f916"

the cilium logs seems to be the key to understand the problem