Move GPU CI pipelines from old daint to new daint

[msimberg]

• don't allow failures on gh200 anymore

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.01% (target: -1.00%)	✅ ∅ (target: 90.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (17f3c6f)	18346	13774	75.08%
Head commit (3ba13b4)	18346 (+0)	13776 (+2)	75.09% (+0.01%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#1239)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

[aurianer]

Thanks a lot!

[msimberg]

To do.

[msimberg]

Exporting NVIDIA_VISIBLE_DEVICES=all and NVIDIA_DRIVER_CAPABILITIES="compute,utility" seems to be what was required to get the container images to load the correct drivers etc. and avoid

cudaErrorInsufficientDriver (CUDA driver version is insufficient for CUDA runtime version)

These are from https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html#constraints.

These work when testing manually, but don't seem to be work in CI yet.

[msimberg]

All right, we're making some progress:

• The GCC 12/CUDA 12 pipeline is working now.
• The clang/cuda one isn't working because it uses valgrind and valgrind doesn't seem to like some of the instructions used. I'll see if using a less specific arm instruction set may help valgrind here.
• The CUDA 11 pipeline isn't working. I would've expected it to be compatible despite the old CUDA version. I'll see if changing required driver versions etc. helps at all.

I may end up disabling the test steps for the latter two in this PR to reenable them in separate PRs.

[msimberg]

To do.

[msimberg]

Remove pipeline?

[msimberg]

The clang/cuda configuration with valgrind no longer complains about illegal instructions: good. It now reports many issues, which I don't know yet if they're real or not.

I'll aim to get the GCC 12/CUDA 12 pipeline running properly (still some tweaks needed on the CSCS CI side apparently) and then I'll attempt to revive the two other CUDA configurations separately, possibly introducing another valgrind configuration on x86.