Flaky CI - build-and-test - lidar_centerpoint

[xmfcx]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I'm convinced that this is not my fault but a bug.

Description

In build-and-test jobs, lidar_centerpoint package test sometimes fails on non-cuda job.

See most recently failed jobs here: https://github.com/autowarefoundation/autoware.universe/commits/main/

common-runner-x64-01 is hosted on aws, a 4c4t, 16GB RAM machine.
leo-copper is a 4c8t machine with 32GB RAM hosted in the office.

Examples:

Workflow	Runner	Success?
refactor(map_projection_loader): apply static analysis (#7497)	common-runner-x64-01	❌
refactor(yabloc_image_processing): apply static analysis (#7489)	common-runner-x64-01	✅
chore: sync files (#7368)	leo-copper	✅
fix(vehicle_cmd_gate): fix unnecessary modification (#7488)	leo-copper	❌
feat(map_loader): warn if some pcds from the metadata file are missin…	common-runner-x64-01	✅
ci(pre-commit): autoupdate (#7499)	common-runner-x64-01	❌

Expected behavior

Actual behavior

7: /__w/autoware.universe/autoware.universe/build/lidar_centerpoint/test_nms: error while loading shared libraries: libnvinfer.so.8: cannot open shared object file: No such file or directory

https://github.com/autowarefoundation/autoware.universe/actions/runs/9519392668/job/26242490387#step:9:21781

Steps to reproduce

Versions

No response

Possible causes

No response

Additional context

No response

[xmfcx]

Closing until we see this again. We had a lot of CI changes.

• build-and-test workflow is failing due to CUDA runtime version #7668
• Tracking tests disabled due to lack of gpu capable machines #7724
• ci(build-and-test-x): fix cache and merge clang-tidy #7723

[xmfcx]

Looking at it again, it was due to cache issue:

• https://github.com/autowarefoundation/autoware.universe/actions/runs/9540376376/job/26292029505#step:9:245

This should not have gotten a cache. cc: @mitsudome-r

Solved by:

• ci(build-and-test-x): fix cache and merge clang-tidy #7723
• fix: caching strategies of colcon-build/test and clang-tidy autoware-github-actions#304