Infrastructure for Orka (2024 and beyond)

[UlisesGascon]

I plan to work on it during the weekend, so I can provide a good overview on the next build meeting on Tuesday.

Current tasks on MacOS infra

• Cleaned up all the macos machines that are available via SSH
• Open discussion to apply upgrade in the Firewall: Schedule Maintenance for Orka CISCO Firewall #3642
• Review Orka state: Infrastructure for Orka (2024 and beyond) #3686 (comment)
• Plan the final allocation for resources in Orka (see: Infrastructure for Orka (2024 and beyond) #3686 (comment))
  • MacOS 13
  • Nearform MacOS Machines (see: Replacement for NearForm Mac OS machines #3638)
• Provide access for @ryanaslett to the infra-mac related resources: LFIT access to secrets #3658
• Prepare and deliver and onboarding session for @ryanaslett to the Orka infrastructure
• Add new VMs for MacOS 13 Intel (@ryanaslett)

Blocked until ARM nodes are provided

• Confirm org decision regarding new ARM nodes (discussion ongoing in the mailing list)
• Add new VMs for MacOS 13 ARM
• Add new VMs for MacOS 11 ARM

[UlisesGascon]

Current Orka state

updated on April 19, 2024

SSH port	Node: macpro-4	Node: macpro-5	Node: macpro-6
8822	release-macos11-x64-1	empty	test-macos11-x64-1
8823	empty	empty	test-macos11-x64-2
8824	empty	test-macos1015-x64-2	test-macos1015-x64-1
8825	empty	empty	empty

[UlisesGascon]

Next Orka state

updated on April 22, 2024

Intel Nodes

SSH port	Node: macpro-4	Node: macpro-5	Node: macpro-6
8822	release-macos11-x64-1	test-macos13-x64-2	test-macos11-x64-1
8823	test-macos13-x64-1	release-macos13-x64-1	test-macos11-x64-2
8824	empty	test-macos1015-x64-2	test-macos1015-x64-1
8825	empty	empty	empty

ARM Nodes

We assume that ARM Nodes can handle only 2 VMs and not +4 as Intel in the past due license limitations. This needs to be confirmed with support AFAIK?

SSH port	Node: arm-1	Node: arm-2	Node: arm-3
8822	test-macos11-arm64-1	release-macos13-arm64-1	empty
8823	release-macos11-arm64-1	test-macos13-arm64-1	test-macos13-arm64-2

How Nearform machines are "relocated"?

• release-nearform-macos11.0-arm64-1 -> release-orka-macos11-arm64-1
• test-nearform-macos11.0-arm64-1 -> test-orka-macos11-arm64-1

[targos]

release-macos13-x64-2
release-macos13-arm64-2

I don't think it's necessary to have two identical release machines.

[targos]

test-nearform-macos11.0-arm64-1

Are these typos?

[UlisesGascon]

Great feedback @targos! I updated the tables

I don't think it's necessary to have two identical release machines.

We have space for redundancy, but let's remove them for now.

Are these typos?

I made a better reference for the "relocated" machines

[targos]

release-macos13-x64-2
release-macos13-arm64-2

I don't think it's necessary to have two identical release machines.

Actually, I think we should have one x64 and two arm64 machines, because there are two jobs that run on macos-arm64 during a release (osx11-release-pkg and osx11-arm64-release-tar).

[ryanaslett]

Some questions/thoughts/suggestions:

1. Requirements Question: Do we still need to support 10.15 and/or 11? from (https://github.com/nodejs/node/blob/main/BUILDING.md#supported-platforms) I see:

Node.js does not support a platform version if a vendor has expired support for it. In other words, Node.js does not support running on End-of-Life (EoL) platforms. This is true regardless of entries in the table below.

And the table lists MacOS 11>.

And that table may be outdated as it seems as though MacOS 11 was EOL as of November 2023 ?

2. ARM support in Orka:

We assume that ARM Nodes can handle only 2 VMs and not +4 as Intel in the past due license limitations. This needs to be confirmed with support AFAIK?

https://orkadocs.macstadium.com/docs/apple-arm-based-support confirms this:

IMPORTANT

You can deploy up to 2 VMs per Apple silicon-based node.

3. From what I can gather macOS infra seems to be brittle, with nodes often running into disk issues/maintenance issues.

#3592
#3685
(https://github.com/nodejs/build/issues?q=is%3Aissue+macos+is%3Aclosed+disk) etc.

My suggestion to avoid Jenkins worker decay is to lean into an ephemeral node strategy so that each build has a fresh Orka instance to run on.

We can do that with the following Jenkins plugin for Orka:
https://plugins.jenkins.io/macstadium-orka/#plugin-content-ephemeral-agents

We would first need to set up a packer build process to create our VM images so that Orka would have a baseline image to create:
https://orkadocs.macstadium.com/docs/packer

The packer process can leverage our existing ansible playbooks:
https://developer.hashicorp.com/packer/integrations/hashicorp/ansible/latest/components/provisioner/ansible.

This strategy would require that we have an Orka3.0 cluster. Rather than trying to do an upgrade of the existing cluster, I propose that we ask macstadium to allow us to provision a new cluster with the resources we need in it (enough arm/intel backing nodes for our macos11/13 testing and release), get it built/provisioned and working, and then decommission/return all the existing macstadium/orka machines.

I believe this would end up with us using roughly the same amount of resources, so should be palatable for macstadium to support this transition.

[mhdawson]

This strategy would require that we have an Orka3.0 cluster. Rather than trying to do an upgrade of the existing cluster, I propose that we ask macstadium to allow us to provision a new cluster with the resources we need in it (enough arm/intel backing nodes for our macos11/13 testing and release), get it built/provisioned and working, and then decommission/return all the existing macstadium/orka machines.

+1 from me if Macstadium will support that

[UlisesGascon]

Quick update from our last call with MacStadium:

Next week we will have a new Orka cluster (v3) that includes 2 nodes (Intel and ARM):

• Mac Studio - G1MC M1M/10/32/16/64GB/2TB/10G
• Mac mini G4E - i7/3.2Ghz/6C/64G/1T/SSD/10G

---

Pending:

Dependencies

• Add Jenkins Plugin for Orka  #3860
• Get a new Orka Cluster running Orka 3 (on going)

✅ Setup Jenkins <-> Orka

• Add documentation in the secrets repo: https://github.com/nodejs-private/secrets/pull/342
• Create a namespace orka-test for test ci
• Create a namespace orka-release for release ci
• Create a service account for test ci sa-jenkins-test
• Create a service account for release ci sa-jenkins-release
• Generate tokens for Jenkins to connect to the release namespace
• Generate tokens for Jenkins to connect to the test namespace
• Setup the Orka Plugin in Jenkins for test CI: https://ci.nodejs.org/manage/cloud/Orka%20Cluster/
• Setup the Orka Plugin in Jenkins for release CI: https://ci-release.nodejs.org/manage/cloud/Orka%20Cluster/
• Add a VPN connection between Jenkins test CI and Orka: IMPORTANT: evaluate VPN connection between Jenkins test CI and the Orka cluster  #3883
• Add a VPN connection between Jenkins release CI and Orka: (Waiting until tuesday, see: IMPORTANT: evaluate VPN connection between Jenkins test CI and the Orka cluster  #3883 (comment))
• Ensure the cloud connection in Jenkins test environment
• Ensure the cloud connection in Jenkins release environment

Current status: Completed.

✅ Create Image templates

• Define images for each MacOS Version/arch and use (Test or release) using packer
  • Team agreement on Proposal: Drop support for MacOS prior to 13 #3876
  • MacOS 13 (Ventura) test Arm (macos-13-arm-test.pkr.hcl): Add Packer support for MacOS 13 test images in Orka #3882
  • MacOS 13 (Ventura) test Intel (macos-13-intel-test.pkr.hcl): Add Packer support for MacOS 13 test images in Orka #3882
  • MacOS 13 (Ventura) release Arm (macos-13-arm-release.pkr.hcl): Add Packer support for MacOS 13 release images in Orka #3893
  • MacOS 13 (Ventura) release Intel (macos-13-intel-release.pkr.hcl): Add Packer support for MacOS 13 release images in Orka #3893
  • Update the secrets repo: https://github.com/nodejs-private/secrets/pull/348
• Add Packer to the Build project: Add initial support for Packer in Orka #3872
• Add GitHub Action to do the validation (NO auto-deployments): Add initial support for Packer in Orka #3872
• Add Packer secrets in to the secrets repo: https://github.com/nodejs-private/secrets/pull/343

Current status: Completed.

✅ Trigger Ephemeral VMs from Jenkins

• Setup Ephemeral nodes from Jenkins on demand for the test CI and probe that they can build and test node.
• Setup Ephemeral nodes from Jenkins on demand for the release CI and probe that they can build, test and sign node.

Current status: Completed

Jobs and Agents Migration

• Add the MacOS13 to the ci test existing jobs (commit-test, night builds, v8, CITGM...) to check that all the current setup can run all the pipelines
• Add the MacOS13 to the ci release existing jobs to check that all the current setup can run all the pipelines
• Configure Jenkins Plugin to be compatible with MacOS11 and MacOS10.15 -mmacosx-version-min (see: Proposal: Drop support for MacOS prior to 13 #3876) in the test ci
• Configure Jenkins Plugin to be compatible with MacOS11 and MacOS10.15 -mmacosx-version-min (see: Proposal: Drop support for MacOS prior to 13 #3876) in the release ci
• Remove labels from legacy jenkins agents and check that the jobs are working as expected in the test ci
• Remove labels from legacy jenkins agents and check that the jobs are working as expected in the release ci

Current status: @UlisesGascon working on the setup.

Clean up

• Decommission the Bare metal machines
  • Open a support ticket
  • Remove MacStadium test from inventory #3871
  • Remove MacStadium release from inventory #3868
• Decommission Orka Cluster (Intel Nodes)
  • Open a support ticket
  • Remove Orka test from inventory #3869
  • Remove Orka release from inventory #3870
• Remove MacOS references from the Ansible playbooks
  • docs: remove macos references from manual steps #3892

Other

• Add a good documentation in the Build repo about the current Orka/Jenkins setup
• Support Marketing activities related (@UlisesGascon will provide more details)
• Elaborate a separate plan for MacOS 14 (Sonoma) and MacOS 15 (Sequoia)
• Improve SSH keys management (see: Add GitHub host key to known hosts #3896 (comment))

Deadline
The idea is to try to achieve this transition in 30 days.

Important

We don't expect any downtime will doing the migration as we will have a new cluster working on isolation will the current system is in place until we are ready to transfer the operations to the new cluster and then decommission the HW.

Challenges

• Lack of support for MacOS 11 ARM in Orka: details
• Cannot deploy more than 2 VMs on an ARM host.
• Error: admission webhook "vimage.kb.io" denied the request: cannot delete image "macos13-intel-test-latest.img". The image is being used by one or more VMs: vm-ttdzh. Remove the VMs and try again
• When Jenkins create a cloud agent if this agent failed then is not removed from the cluster
• Evaluate how to use namespace
• New VMs are much slower than expected (@UlisesGascon investigating)
• Unify the HCL templates into a single one if possible (see: Orka template updates #3906 (comment))

[UlisesGascon]

Current status

I will be on PTO from the 19th to the 25th. I made some changes to the templates to add the missing dependencies (#3906).

So, @ryanaslett, in case you want to help with this during my time off:

• Check why the iojs+release-Ulises-test-orca is not passing. The current error (10:09:52 Makefile:1030: *** No xz command, cannot continue. Stop.) (details) is related to the PATH (I think), as xz is included on all the machines already since my last PR.
• Try to modify the base image for release and test machines to include and use Xcode 15.2, based on the discussion with @targos. Documentation
• Keep running and patching the other pipelines that we cloned with @mhdawson.

Probably the next errors in the CI will be related to the users; currently, we only have the admin user. Maybe we need to create a separate one like iojs in order to make the CI pipelines work.