metal, vsphere-cluster-resource: retrieve EKS-A binary at runtime #867

etungsten · 2023-09-12T16:42:19Z

Issue number:
N/A

Description of changes:


    metal,vsphere-cluster-resource: retrieve EKS-A binary at runtime
    
    Adds a new field for specifying the EKS-A release manifest URL to both
    the metal and vSphere cluster resource agent configurations. At runtime,
    the agents will download the EKS-A binary archive tagged as the latest
    release in the EKS-A release manifest. If a manifest is not specified,
    then the agents defaults to using the upstream official EKS-A release
    manifest.

    don't build EKS-A binary into the testsys agent images
    
    We now fetch the EKS-A binaries at agent runtime. No longer need to bake
    in the EKS-A binary.

Testing done:
Able to create vSphere cluster using default EKS-A release manifest:

[2023-09-12T23:28:23Z INFO  bottlerocket_agents::clusters] Using EKS-A release manifest 'https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml'                    
[2023-09-12T23:28:23Z INFO  bottlerocket_agents::clusters] Fetching EKS-A binary archive from 'https://anywhere-assets.eks.amazonaws.com/releases/eks-a/47/artifacts/eks-a/v0.17.2/l
inux/amd64/eksctl-anywhere-v0.17.2-linux-amd64.tar.gz'

Full logs:

$ kubectl --kubeconfig testsys.kubeconfig logs -f x86-64-vmware-ked61420d-716c-43f8-bfac-572ba59680ba-creatijwhh4 -n testsys                                                        
[2023-09-12T23:28:23Z INFO  resource_agent::agent] Initializing Agent                                                                                                               
[2023-09-12T23:28:23Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Getting vSphere secret                                                                
[2023-09-12T23:28:23Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Creating working directory                                                            
[2023-09-12T23:28:23Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Checking existing cluster                                                             
[2023-09-12T23:28:23Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Creation policy is 'IfNotExists' and cluster 'x86-64-vmware-k8s-128' does not exist: creating cluster                                                                                                                                                                     
[2023-09-12T23:28:23Z INFO  bottlerocket_agents::clusters] Using EKS-A release manifest 'https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml'                    
[2023-09-12T23:28:23Z INFO  bottlerocket_agents::clusters] Fetching EKS-A binary archive from 'https://anywhere-assets.eks.amazonaws.com/releases/eks-a/47/artifacts/eks-a/v0.17.2/l
inux/amd64/eksctl-anywhere-v0.17.2-linux-amd64.tar.gz'                                                                                                                              
[2023-09-12T23:28:23Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Creating cluster                                                                      
[2023-09-12T23:28:23Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Downloading OVA 'bottlerocket-vmware-k8s-1.27-x86_64-v1.15.0.ova'                     
[2023-09-12T23:28:33Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Importing OVA and creating a VM template out of it                                    
[2023-09-12T23:28:46Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Tagging VM template                                                                   
2023-09-12T23:29:42.104Z        V4      Reading bundles manifest        {"url": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/47/manifest.yaml"}                      
2023-09-12T23:29:42.456Z        V4      Relative network path specified, using path /SDDC-Datacenter/network/sddc-cgw-network-2               
....

Using a custom EKS-A release manifest, cluster creation works:

[2023-09-12T23:46:27Z INFO  bottlerocket_agents::clusters] Using EKS-A release manifest 'https://.cloudfront.net/baremetal/eksctl-anywhere-manifest.yaml'
[2023-09-12T23:46:27Z INFO  bottlerocket_agents::clusters] Fetching EKS-A binary archive from 'https://.cloudfront.net/baremetal/eksctl-anywhere-linux-amd64.tar.gz'

Full logs:

$ kubectl --kubeconfig testsys.kubeconfig logs -f x86-64-vmware-k6aec44d1-503c-4f92-abf0-1ca35d41ae2d-creatif5r6l -n testsys
[2023-09-12T23:46:27Z INFO  resource_agent::agent] Initializing Agent
[2023-09-12T23:46:27Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Getting vSphere secret
[2023-09-12T23:46:27Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Creating working directory
[2023-09-12T23:46:27Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Checking existing cluster
[2023-09-12T23:46:27Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Creation policy is 'IfNotExists' and cluster 'x86-64-vmware-k8s-128' does not exist: creating cluster
[2023-09-12T23:46:27Z INFO  bottlerocket_agents::clusters] Using EKS-A release manifest 'https://.cloudfront.net/baremetal/eksctl-anywhere-manifest.yaml'
[2023-09-12T23:46:27Z INFO  bottlerocket_agents::clusters] Fetching EKS-A binary archive from 'https://.cloudfront.net/baremetal/eksctl-anywhere-linux-amd64.tar.gz'
[2023-09-12T23:46:29Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Creating cluster
[2023-09-12T23:46:29Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Downloading OVA 'bottlerocket-vmware-k8s-1.28-x86_64-v1.15.0.ova'
[2023-09-12T23:46:33Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Importing OVA and creating a VM template out of it
[2023-09-12T23:46:45Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Tagging VM template
2023-09-12T23:47:41.247Z        V4      Reading bundles manifest        {"url": "https://dev-release-assets.eks-anywhere.model-rocket.aws.dev/bundle-release.yaml"}
2023-09-12T23:47:41.345Z        V4      Relative network path specified, using path /SDDC-Datacenter/network/sddc-cgw-network-2
2023-09-12T23:47:41.345Z        V1      SSHUsername is not set or is empty for VSphereMachineConfig, using default      {"c": "x86-64-vmware-k8s-128-node", "user": "ec2-user"}
2023-09-12T23:47:41.432Z        V2      Pulling docker image    {"image": "public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.17.1-eks-a-v0.0.0-dev-build.7550"}
2023-09-12T23:47:48.032Z        V3      Initializing long running container     {"name": "eksa_1694562461432403955", "image": "public.ecr.aws/l0g8r8j6/eks-anywhere-cli-tools:v0.17.1-eks-a-v0.0.0-dev-build.7550"}
2023-09-12T23:47:51.714Z        V4      Task start      {"task_name": "setup-validate"}
2023-09-12T23:47:51.714Z        V0      Performing setup and validations
2023-09-12T23:47:51.732Z        V0      ✅ Connected to server
2023-09-12T23:47:52.091Z        V0      ✅ Authenticated to vSphere
2023-09-12T23:47:52.833Z        V0      ✅ Datacenter validated
2023-09-12T23:47:53.165Z        V0      ✅ Network validated
2023-09-12T23:47:54.162Z        V3      CloneMode not set, defaulting to fullClone      {"VSphereMachineConfig": "x86-64-vmware-k8s-128-node"}
2023-09-12T23:47:54.162Z        V4      Relative datastore path specified, using path /SDDC-Datacenter/datastore/WorkloadDatastore
2023-09-12T23:47:54.541Z        V0      ✅ Datastore validated
2023-09-12T23:47:54.826Z        V0      ✅ Folder validated
2023-09-12T23:47:55.116Z        V0      ✅ Resource pool validated
2023-09-12T23:47:57.555Z        V0      ✅ Machine config tags validated
2023-09-12T23:47:57.555Z        V0      ✅ Control plane and Workload templates validated
2023-09-12T23:47:58.958Z        V0      Provided sshAuthorizedKey is not set or is empty, auto-generating new key pair...       {"vSphereMachineConfig": "x86-64-vmware-k8s-128-node"}
2023-09-12T23:48:00.293Z        V0      Private key saved to x86-64-vmware-k8s-128/eks-a-id_rsa. Use 'ssh -i x86-64-vmware-k8s-128/eks-a-id_rsa <username>@<Node-IP-Address>' to login to your cluster node
2023-09-12T23:48:02.425Z        V0      ✅ [email protected] user vSphere privileges validated
2023-09-12T23:48:02.425Z        V0      ✅ Vsphere Provider setup is valid
2023-09-12T23:48:02.425Z        V0      ✅ Validate OS is compatible with registry mirror configuration
2023-09-12T23:48:02.425Z        V0      ✅ Validate certificate for registry mirror
2023-09-12T23:48:02.425Z        V0      ✅ Validate authentication for git provider
2023-09-12T23:48:02.425Z        V0      ✅ Validate cluster's eksaVersion matches EKS-A version
2023-09-12T23:48:03.335Z        V0      ✅ Validate cluster name
2023-09-12T23:48:03.335Z        V0      ✅ Validate gitops
2023-09-12T23:48:03.335Z        V0      ✅ Validate identity providers' name
2023-09-12T23:48:04.054Z        V0      ✅ Validate management cluster has eksa crds
2023-09-12T23:48:05.054Z        V0      ✅ Validate management cluster name is valid
2023-09-12T23:48:05.929Z        V0      ✅ Validate management cluster eksaVersion compatibility
2023-09-12T23:48:05.929Z        V4      Task finished   {"task_name": "setup-validate", "duration": "14.214849493s"}
2023-09-12T23:48:05.929Z        V4      ----------------------------------
2023-09-12T23:48:05.929Z        V4      Task start      {"task_name": "bootstrap-cluster-init"}
2023-09-12T23:48:05.929Z        V4      Task finished   {"task_name": "bootstrap-cluster-init", "duration": "2.044µs"}
2023-09-12T23:48:05.930Z        V4      ----------------------------------
2023-09-12T23:48:05.930Z        V4      Task start      {"task_name": "workload-cluster-init"}
2023-09-12T23:48:05.930Z        V0      Creating new workload cluster
2023-09-12T23:48:06.846Z        V3      Waiting for external etcd to be ready   {"cluster": "x86-64-vmware-k8s-128"}
2023-09-12T23:50:15.185Z        V3      External etcd is ready
2023-09-12T23:50:15.185Z        V3      Waiting for control plane to be available
...
2023-09-12T23:53:12.975Z        V4      Nodes ready     {"total": 3}
2023-09-12T23:53:12.975Z        V4      Task finished   {"task_name": "workload-cluster-init", "duration": "5m7.045290773s"}
2023-09-12T23:53:12.975Z        V4      ----------------------------------
2023-09-12T23:53:12.975Z        V4      Task start      {"task_name": "install-resources-on-management-cluster"}
2023-09-12T23:53:12.975Z        V4      Task finished   {"task_name": "install-resources-on-management-cluster", "duration": "1.514µs"}
2023-09-12T23:53:12.975Z        V4      ----------------------------------
2023-09-12T23:53:12.975Z        V4      Task start      {"task_name": "capi-management-move"}
2023-09-12T23:53:12.975Z        V4      Task finished   {"task_name": "capi-management-move", "duration": "616ns"}
2023-09-12T23:53:12.975Z        V4      ----------------------------------
2023-09-12T23:53:12.975Z        V4      Task start      {"task_name": "eksa-components-install"}
2023-09-12T23:53:12.975Z        V0      Creating EKS-A CRDs instances on workload cluster
2023-09-12T23:53:12.978Z        V4      Applying eksa yaml resources to cluster
2023-09-12T23:53:13.616Z        V1      Applying Bundles to cluster
2023-09-12T23:53:14.317Z        V1      Applying EKSARelease to cluster
2023-09-12T23:53:14.942Z        V4      Applying eksd manifest to cluster
....
2023-09-12T23:53:20.060Z        V4      Task finished   {"task_name": "eksa-components-install", "duration": "7.084643615s"}
2023-09-12T23:53:20.060Z        V4      ----------------------------------
2023-09-12T23:53:20.060Z        V4      Task start      {"task_name": "gitops-manager-install"}
2023-09-12T23:53:20.060Z        V0      Installing GitOps Toolkit on workload cluster
2023-09-12T23:53:20.060Z        V0      GitOps field not specified, bootstrap flux skipped
2023-09-12T23:53:20.060Z        V4      Task finished   {"task_name": "gitops-manager-install", "duration": "88.987µs"}
2023-09-12T23:53:20.060Z        V4      ----------------------------------
2023-09-12T23:53:20.060Z        V4      Task start      {"task_name": "write-cluster-config"}
2023-09-12T23:53:20.060Z        V0      Writing cluster config file
2023-09-12T23:53:20.062Z        V4      Task finished   {"task_name": "write-cluster-config", "duration": "2.409873ms"}
2023-09-12T23:53:20.062Z        V4      ----------------------------------
2023-09-12T23:53:20.063Z        V4      Task start      {"task_name": "delete-kind-cluster"}
2023-09-12T23:53:20.063Z        V0      🎉 Cluster created!
2023-09-12T23:53:20.063Z        V4      Task finished   {"task_name": "delete-kind-cluster", "duration": "15.9µs"}
2023-09-12T23:53:20.063Z        V4      ----------------------------------
2023-09-12T23:53:20.063Z        V4      Task start      {"task_name": "install-curated-packages"}
--------------------------------------------------------------------------------------
The Amazon EKS Anywhere Curated Packages are only available to customers with the 
Amazon EKS Anywhere Enterprise Subscription
--------------------------------------------------------------------------------------
...
2023-09-12T23:53:20.855Z        V4      Tasks completed {"duration": "5m29.14094338s"}
2023-09-12T23:53:20.856Z        V3      Logging out from current govc session
2023-09-12T23:53:21.484Z        V3      Cleaning up long running container      {"name": "eksa_1694562461432403955"}
[2023-09-12T23:53:22Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Scaling default NodeGroup machinedeployments replicas to 0
machinedeployment.cluster.x-k8s.io/x86-64-vmware-k8s-128-md-0 scaled
[2023-09-12T23:53:22Z INFO  vsphere_k8s_cluster_resource_agent::vsphere_k8s_cluster_provider] Cluster created

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

bottlerocket/agents/src/clusters.rs

bottlerocket/agents/src/bin/vsphere-k8s-cluster-resource-agent/vsphere_k8s_cluster_provider.rs

Adds a new field for specifying the EKS-A release manifest URL to both the metal and vSphere cluster resource agent configurations. At runtime, the agents will download the EKS-A binary archive tagged as the latest release in the EKS-A release manifest. If a manifest is not specified, then the agents defaults to using the upstream official EKS-A release manifest.

We now fetch the EKS-A binaries at agent runtime. No longer need to bake in the EKS-A binary.

bcressey

🎉

etungsten requested review from ecpullen, bcressey and stmcginnis September 12, 2023 16:42

stmcginnis approved these changes Sep 12, 2023

View reviewed changes

bcressey reviewed Sep 12, 2023

View reviewed changes

bottlerocket/agents/src/clusters.rs Outdated Show resolved Hide resolved

bottlerocket/agents/src/bin/vsphere-k8s-cluster-resource-agent/vsphere_k8s_cluster_provider.rs Outdated Show resolved Hide resolved

etungsten force-pushed the fetch-eksa-dynamically branch from c33efb5 to 429846a Compare September 12, 2023 21:08

ecpullen approved these changes Sep 12, 2023

View reviewed changes

etungsten requested review from bcressey and stmcginnis September 12, 2023 21:17

etungsten force-pushed the fetch-eksa-dynamically branch 3 times, most recently from 57e942f to 59ba440 Compare September 12, 2023 21:37

etungsten added 2 commits September 12, 2023 15:01

don't bake EKS-A binary into the testsys agent images

57e677a

We now fetch the EKS-A binaries at agent runtime. No longer need to bake in the EKS-A binary.

etungsten force-pushed the fetch-eksa-dynamically branch from 59ba440 to 57e677a Compare September 12, 2023 22:02

bcressey approved these changes Sep 12, 2023

View reviewed changes

zmrow approved these changes Sep 12, 2023

View reviewed changes

etungsten marked this pull request as ready for review September 12, 2023 23:54

etungsten merged commit 034a9a0 into bottlerocket-os:develop Sep 12, 2023
4 checks passed

etungsten deleted the fetch-eksa-dynamically branch September 12, 2023 23:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metal, vsphere-cluster-resource: retrieve EKS-A binary at runtime #867

metal, vsphere-cluster-resource: retrieve EKS-A binary at runtime #867

etungsten commented Sep 12, 2023 •

edited

Loading

bcressey left a comment

metal, vsphere-cluster-resource: retrieve EKS-A binary at runtime #867

metal, vsphere-cluster-resource: retrieve EKS-A binary at runtime #867

Conversation

etungsten commented Sep 12, 2023 • edited Loading

bcressey left a comment

Choose a reason for hiding this comment

etungsten commented Sep 12, 2023 •

edited

Loading