ci-automation: add hetzner testing

[tormath1]

In this PR, we bring the "glue" between Flatcar CI and Hetzner Mantle implementation.

NOTE: For this specific cloud provider there is no need to implement garbage collection as the provider is providing us a temporary project to run our tests, the project is deleted right after

Testing done

CI 🟢 : http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/4600/cldsv/

---

• No changelog entry as it's not user facing.
• To be merged with: platform: Hetzner API implementation mantle#536

Closes flatcar/Flatcar#1412

[apricote]

cx plans have limited availability. To reduce flaky pipelines I would recommend cpx11 instead.

https://status.hetzner.com/incident/aa5ce33b-faa5-4fd0-9782-fde43cd270cf

• Title: Limited availability of CX plans
• Status: In progress
• Start: 2024-06-17 10:00 UTC+0
• Description: Due to increased demand, instances on CX plans (CX22 to CX52) are currently subject to limited availability.
  We will gradually release new capacities as soon as they are available.

[tormath1]

@apricote AMD tests look perfect but ARM tests are not happy:

Failed tests:
 cl.internet
 cl.basic
 coreos.ignition.security.tls
 cl.flannel.udp
 coreos.ignition.resource.local

With:

 bash: line 1: ./kolet: cannot execute binary file: Exec format error
     --- FAIL: cl.basic/CloudConfig (1.84s)
             cluster.go:85: kolet: Process exited with status 126
         cluster.go:82: kolet:
 bash: line 1: ./kolet: cannot execute binary file: Exec format error
     --- FAIL: cl.basic/PortSSH (1.84s)
             cluster.go:85: kolet: Process exited with status 126
 --- FAIL: cl.flannel.udp (1096.63s)
         harness.go:628: Cluster failed starting machines: machine "52347143" failed basic checks: ssh unreachable or system not ready: failure checking if machine is running: systemctl is-system-running returned stdout: "starting", stderr: "", err: Process exited with status 1, systemctl list-jobs returned stdout: "JOB UNIT                        TYPE  STATE\n736 etcd-member.service         start running\n583 multi-user.target           start waiting\n718 flannel-docker-opts.service start waiting\n717 flanneld.service            start waiting\n\n4 jobs listed.", stderr: "", err: <nil>
 --- FAIL: coreos.ignition.resource.local (1187.20s)
         resource.go:338: starting client: machine "52347217" failed to start: ssh journalctl failed: time limit exceeded: dial tcp 49.13.137.171:22: connect: connection refused
 --- FAIL: coreos.ignition.security.tls (1193.94s)
         security.go:137: starting client: machine "52347233" failed to start: ssh journalctl failed: time limit exceeded: dial tcp 188.245.121.230:22: connect: connection refused
 FAIL, output in _kola_temp/hetzner-2024-08-27-2021-34

kolet is a binary (part of Kola) uploaded on the tested machine and it runs some "native" test directly on the machine, the "exec format error" usually means that is has been compiled with the wrong arch.

For the other tests, I will try to repro.

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/10768868921

[apricote]

@tormath1 Can you try again with the patches from flatcar/mantle#553? It works now for me on cl.basic. Did not try the other tests.

[tormath1]

@tormath1 Can you try again with the patches from flatcar/mantle#553? It works now for me on cl.basic. Did not try the other tests.

@apricote All good now - a bit flaky on the Kubernetes test but I think it was transient:

         cluster.go:125: failed to pull image "registry.k8s.io/kube-apiserver:v1.28.7": output: E0902 12:46:59.711737    1962 remote_image.go:167] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.28.7\": failed to resolve reference \"registry.k8s.io/kube-apiserver:v1.28.7\": unexpected status from HEAD request to https://registry.k8s.io/v2/kube-apiserver/manifests/v1.28.7: 403 Forbidden" image="registry.k8s.io/kube-apiserver:v1.28.7"
         cluster.go:125: time="2024-09-02T12:46:59Z" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.28.7\": failed to resolve reference \"registry.k8s.io/kube-apiserver:v1.28.7\": unexpected status from HEAD request to https://registry.k8s.io/v2/kube-apiserver/manifests/v1.28.7: 403 Forbidden"

[krnowak]

Could you add a comment about why we override the user agent?

[tormath1]

@apricote any idea why we need this? I see it's similar to this: https://github.com/hetznercloud/tps-action/blob/dee5dd2546322c28ed8f74b910189066e8b6f31a/get-token.sh#L19 but not sure why we need it.

[krnowak]

This looks like a debugging aid?

hetznercloud/tps-action#5

[apricote]

It works without the user agent. Having the user agent in requests is helpful for us operating the service as we can easily figure out who is affected by any errors, who is sending too many requests...

Of course, the user agent can easily be faked by anyone, so we don't fully rely on it

[tormath1]

Thanks @apricote for sharing the details. Let's keep the user agent like this in this case as it can help you to monitor the TPS system.

[krnowak]

The user-agent issue is a minor thing. Regardless of whether it gets removed or not, the PR looks good.

[tormath1]

cherry-picked to:

• flatcar-4081
• flatcar-4054
• flatcar-3975