Remove Kubernetes 1.23 jobs

[xmudrii]

Experimenting with removing Kubernetes 1.23 jobs (ref #29230, #29379, #29327)

This PR removes Kubernetes 1.23 jobs.

Disclaimer: this is done in a hacky way and I'll try my best to explain what's going on and what issues we have.

Version markers

For context, version markers are explained here: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/kubernetes-versions.md

We have the following version markers in play:

• k8s-master
  • Current and expected usage: points to the master branch in k/k
• k8s-beta
  • Current usage: points to the latest stable release in time frame from cutting the release branch to cutting the next release branch
  • Expected usage: points to the latest stable release in time frame from cutting the release branch to cutting the .0 release
• k8s-stable1 to k8s-stable4
  • Current usage and expected usage: points to the appropriate stable release

The biggest issue that we have with k8s-beta is that we use it for way too long. It should be used only in the "transition period" between cutting the release branch and publishing the .0 release. However, we don't push out k8s-beta after .0 release, instead we continue using it until the next branch cut.

This can be quite confusing for those eventually depending on those markers. We discussed it previously on Slack and some idea was that we should fix this: https://kubernetes.slack.com/archives/CJH2GBF7Y/p1680813414229639

Fixing this is going to be important especially for the next part.

Removing jobs for the EOL release branch

Removing jobs for the EOL release branch is often done by generating jobs for the new release branch + some manual clean ups. In other words, we basically did this: make -C releng prepare-release-branch

There are two problems with that:

• this is going to also create jobs for 1.28 which we don't want at this point in the 1.28 release cycle
• the logic behind this Make target is implemented on premises that k8s-beta is used all the time and not only in the transition period

When we run that command, the result will be:

• 1.28 jobs are generated (these can be removed manually very easily)
• 1.27 - 1.24 jobs are correctly pushed/rotated to stable1 - stable4
• jobs in generated.yaml are updated
• testgrid dashboards for 1.28 are generated

Bold steps are problematic:

• generated.yaml is auto-generated and shouldn't be touched
• even if we touch generated.yaml and remove 1.28 jobs, tests are going to fail because we have 1.28 testgrid dashboards that are not used at all

Mitigations

What we need actually is just config-rotator and regenerating generated.yaml. In other words, we don't need config-forker and all the logic that generates 1.28 jobs.

It's possible to execute both manually, however, before we can do that, we have to figure out how to approach k8s-beta jobs. The best idea I was able to come up is to just comment k8s-beta in releng/test_config.yaml. That way, regenerating generated.yaml is actually going to remove all k8sbeta tests which we in fact don't need at this stage because that marker shouldn't be in use until we don't cut 1.28 release branch.

With that done, the following commands can be used to do the magic:

go run ./releng/config-rotator -config-file $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.24.yaml -new stable4 -old stable3
go run ./releng/config-rotator -config-file $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.25.yaml -new stable3 -old stable2
go run ./releng/config-rotator -config-file $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.26.yaml -new stable2 -old stable1
go run ./releng/config-rotator -config-file $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.27.yaml -new stable1 -old beta

releng/generate_tests.py --yaml-config-path=/Users/marko/Projects/src/k8s.io/test-infra/releng/test_config.yaml

At this stage, we're almost ready. Concretely:

• there are some testgrid dashboards that are not in use after removing 1.23 jobs and those dashboards must be removed
• because we don't have k8s-beta jobs anymore we have to comment some dashboards
• we need to clean up old jobs, i.e. remove 1.23.yaml and some Windows jobs

After that, we should be in a good state. 🎉

Generating 1.28 jobs

Generating 1.28 jobs is not going to be trivial. We can't use the Make target (make -C releng prepare-release-branch) because we already used config-rotator and using it again in the same release cycle would end up in skipping one version marker. For example, 1.28 would be added as k8s-beta, but 1.27 would end up jumping from stable1 to stable2. Instead, 1.27 should stay at stable1 until we don't push out 1.28 to stable1 next time we're removing jobs.

Instead of using the Make target, we have to:

• uncommeting k8s-beta marker and relevant jobs and dashboards
• run config-forker manually to create 1.28 jobs
• regenerate generated.yaml so k8sbeta jobs are added
• add testgrid dashboards for 1.28

In fact, most of those tasks should be pretty easy. The real challenge is making sure we adhere to this, i.e. that we don't forget about this issue. This is especially important because the handbook is instructing to use the Make target.

Follow ups

We have an issue with some fork annotations. Some of them are lost in 1.23.yaml and we need to look into that. That's mostly like irrelevant to this PR, it's mostly like that we forgot to update those annotations when introducing stable4 marker a while ago.

[xmudrii]

/assign @cici37 @ramrodo @palnabarun
cc @kubernetes/release-engineering

[xmudrii]

/hold
for discussion

[xmudrii]

Added some notes to guide the review process. I need to add some TODOs and I'll do that later today.

[xmudrii]

I removed this file because it seems related to presubmits for k/k 1.23. We should double check with Windows folks to make sure this is okay.

[xmudrii]

This is auto-generated and will be back in place after uncommenting k8s-beta.

[xmudrii]

This job is removed in 1.23 so we don't need this tab at all.

[xmudrii]

This job is removed in 1.23 so we don't need this tab at all.

[ramrodo]

@xmudrii Thanks for the great explanation and links for reference! I understand better the version markers now.

I think that adding the TODOs comments to be aware when we release 1.28 will be a great guide.

Maybe we could add this issue/guide to the Handbook to have this issue present and be followed in the upcoming release.

/lgtm

[dims]

/approve

Thanks for the extensive notes. I do shudder at when we have to create 1.28 jobs for sure!

[xmudrii]

I added TODO(1.28) comments where needed. I also created:

• Revisit values for fork-per-release-periodic-interval and fork-per-release-cron annotations #29388 to track fixing annotations
• Update release branch jobs creation documentation sig-release#2227 to track updating the handbook
• Refactor and rewrite Python scripts in releng/ in Go #29390 to propose refactoring Python scripts in Go

I also added this on meeting agenda for the SIG Release meeting planned for May 2nd.

@cici37 @ramrodo @dims PTAL again, I pushed a commit to add TODOs. When ready, please unhold the PR.

[dims]

/lgtm

will leave it to release folks to unhold.

[ramrodo]

Great! Everything looks good to me
/unhold
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, ramrodo, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes-sigs/sig-windows/OWNERS [dims]
• config/jobs/kubernetes/generated/OWNERS [dims,xmudrii]
• config/jobs/kubernetes/sig-release/OWNERS [dims,xmudrii]
• config/testgrids/OWNERS [dims]
• releng/OWNERS [dims,xmudrii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@xmudrii: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key release-1.23-windows-presubmits.yaml using file ``
• key release-1.23-windows.yaml using file ``
• key generated.yaml using file config/jobs/kubernetes/generated/generated.yaml
• key 1.23.yaml using file ``
• key 1.24.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.24.yaml
• key 1.25.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.25.yaml
• key 1.26.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.26.yaml
• key 1.27.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.27.yaml

In response to this:

Experimenting with removing Kubernetes 1.23 jobs (ref #29230, #29379, #29327)

This PR removes Kubernetes 1.23 jobs.

Disclaimer: this is done in a hacky way and I'll try my best to explain what's going on and what issues we have.

Version markers

For context, version markers are explained here: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/kubernetes-versions.md

We have the following version markers in play:

• k8s-master
• Current and expected usage: points to the master branch in k/k
• k8s-beta
• Current usage: points to the latest stable release in time frame from cutting the release branch to cutting the next release branch
• Expected usage: points to the latest stable release in time frame from cutting the release branch to cutting the .0 release
• k8s-stable1 to k8s-stable4
• Current usage and expected usage: points to the appropriate stable release

The biggest issue that we have with k8s-beta is that we use it for way too long. It should be used only in the "transition period" between cutting the release branch and publishing the .0 release. However, we don't push out k8s-beta after .0 release, instead we continue using it until the next branch cut.

This can be quite confusing for those eventually depending on those markers. We discussed it previously on Slack and some idea was that we should fix this: https://kubernetes.slack.com/archives/CJH2GBF7Y/p1680813414229639

Fixing this is going to be important especially for the next part.

Removing jobs for the EOL release branch

Removing jobs for the EOL release branch is often done by generating jobs for the new release branch + some manual clean ups. In other words, we basically did this: make -C releng prepare-release-branch

There are two problems with that:

• this is going to also create jobs for 1.28 which we don't want at this point in the 1.28 release cycle
• the logic behind this Make target is implemented on premises that k8s-beta is used all the time and not only in the transition period

When we run that command, the result will be:

• 1.28 jobs are generated (these can be removed manually very easily)
• 1.27 - 1.24 jobs are correctly pushed/rotated to stable1 - stable4
• jobs in generated.yaml are updated
• testgrid dashboards for 1.28 are generated

Bold steps are problematic:

• generated.yaml is auto-generated and shouldn't be touched
• even if we touch generated.yaml and remove 1.28 jobs, tests are going to fail because we have 1.28 testgrid dashboards that are not used at all

Mitigations

What we need actually is just config-rotator and regenerating generated.yaml. In other words, we don't need config-forker and all the logic that generates 1.28 jobs.

It's possible to execute both manually, however, before we can do that, we have to figure out how to approach k8s-beta jobs. The best idea I was able to come up is to just comment k8s-beta in releng/test_config.yaml. That way, regenerating generated.yaml is actually going to remove all k8sbeta tests which we in fact don't need at this stage because that marker shouldn't be in use until we don't cut 1.28 release branch.

With that done, the following commands can be used to do the magic:

go run ./releng/config-rotator -config-file $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.24.yaml -new stable4 -old stable3
go run ./releng/config-rotator -config-file $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.25.yaml -new stable3 -old stable2
go run ./releng/config-rotator -config-file $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.26.yaml -new stable2 -old stable1
go run ./releng/config-rotator -config-file $(pwd)/config/jobs/kubernetes/sig-release/release-branch-jobs/1.27.yaml -new stable1 -old beta

releng/generate_tests.py --yaml-config-path=/Users/marko/Projects/src/k8s.io/test-infra/releng/test_config.yaml

At this stage, we're almost ready. Concretely:

• there are some testgrid dashboards that are not in use after removing 1.23 jobs and those dashboards must be removed
• because we don't have k8s-beta jobs anymore we have to comment some dashboards
• we need to clean up old jobs, i.e. remove 1.23.yaml and some Windows jobs

After that, we should be in a good state. 🎉

Generating 1.28 jobs

Generating 1.28 jobs is not going to be trivial. We can't use the Make target (make -C releng prepare-release-branch) because we already used config-rotator and using it again in the same release cycle would end up in skipping one version marker. For example, 1.28 would be added as k8s-beta, but 1.27 would end up jumping from stable1 to stable2. Instead, 1.27 should stay at stable1 until we don't push out 1.28 to stable1 next time we're removing jobs.

Instead of using the Make target, we have to:

• uncommeting k8s-beta marker and relevant jobs and dashboards
• run config-forker manually to create 1.28 jobs
• regenerate generated.yaml so k8sbeta jobs are added
• add testgrid dashboards for 1.28

In fact, most of those tasks should be pretty easy. The real challenge is making sure we adhere to this, i.e. that we don't forget about this issue. This is especially important because the handbook is instructing to use the Make target.

Follow ups

We have an issue with some fork annotations. Some of them are lost in 1.23.yaml and we need to look into that. That's mostly like irrelevant to this PR, it's mostly like that we forgot to update those annotations when introducing stable4 marker a while ago.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.