Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking tests disabled due to lack of gpu capable machines #7724

Open
4 tasks done
xmfcx opened this issue Jun 27, 2024 · 4 comments
Open
4 tasks done

Tracking tests disabled due to lack of gpu capable machines #7724

xmfcx opened this issue Jun 27, 2024 · 4 comments
Assignees
Labels
type:ci Continuous Integration (CI) processes and testing. (auto-assigned)

Comments

@xmfcx
Copy link
Contributor

xmfcx commented Jun 27, 2024

Checklist

  • I've read the contribution guidelines.
  • I've searched other issues and no duplicate issues were found.
  • I've agreed with the maintainers that I can plan this task.

Description

Story behind can be tracked from:

Basically the tests disabled here:

  • ✅ work on machines with Nvidia GPU
  • ❌ fail on machines without Nvidia GPU

When we upgrade our testing infrastructure to have CUDA capable GPUs, we can re-enable these tests.

cc. @knzo25 @mitsudome-r

Sister issue:

Purpose

To make CI pass on no-gpu machines. (Especially github hosted runners)

Possible approaches

  • Commenting out the failing tests in the respective CMakeLists.txt file.
  • Adding a reference to here

Definition of done

  • build-and-test passes on the main branch.

Table of tracked tests

Link to commented section Package Name Related PR
# Temporary disabled, tracked by:
# https://github.com/autowarefoundation/autoware.universe/issues/7724
# ament_auto_add_gtest(test_voxel_generator
# test/test_voxel_generator.cpp
# )
lidar_centerpoint #7722
# add_executable(test_preprocess_kernel
# test/test_preprocess_kernel.cpp
# lib/utils.cpp
# )
#
# target_include_directories(test_preprocess_kernel PUBLIC
# ${test_preprocess_kernel_SOURCE_DIR}
# )
#
# target_link_libraries(test_preprocess_kernel
# centerpoint_cuda_lib
# gtest
# gtest_main
# )
#
# ament_add_test(test_preprocess_kernel
# GENERATE_RESULT_FOR_RETURN_CODE_ZERO
# COMMAND "$<TARGET_FILE:test_preprocess_kernel>"
# )
#
# add_executable(test_postprocess_kernel
# test/test_postprocess_kernel.cpp
# lib/utils.cpp
# )
#
# target_include_directories(test_postprocess_kernel PUBLIC
# ${test_postprocess_kernel_SOURCE_DIR}
# )
#
# target_link_libraries(test_postprocess_kernel
# centerpoint_cuda_lib
# gtest
# gtest_main
# )
#
# ament_add_test(test_postprocess_kernel
# GENERATE_RESULT_FOR_RETURN_CODE_ZERO
# COMMAND "$<TARGET_FILE:test_postprocess_kernel>"
# )
lidar_centerpoint #7731
@xmfcx xmfcx added the type:ci Continuous Integration (CI) processes and testing. (auto-assigned) label Jun 27, 2024
@xmfcx xmfcx self-assigned this Jun 27, 2024
@xmfcx xmfcx changed the title Tracking disabled tests due to lack of gpu capable machines Tracking tests disabled due to lack of gpu capable machines Jun 27, 2024
@xmfcx
Copy link
Contributor Author

xmfcx commented Jun 27, 2024

@mitsudome-r if somehow we gain access to gpu capable machines either by:

We can use this PR as reference:

I will also create a documentation on how I set up the self hosted runners with custom init scripts.

Copy link

stale bot commented Aug 27, 2024

This pull request has been automatically marked as stale because it has not had recent activity.

@stale stale bot added the status:stale Inactive or outdated issues. (auto-assigned) label Aug 27, 2024
@knzo25
Copy link
Contributor

knzo25 commented Aug 30, 2024

@xmfcx
Is there any updates on this issue?
We would like to have the cuda tests enabled (https://github.com/autowarefoundation/autoware.universe/blob/main/perception/autoware_lidar_centerpoint/CMakeLists.txt)

@stale stale bot removed the status:stale Inactive or outdated issues. (auto-assigned) label Aug 30, 2024
@xmfcx
Copy link
Contributor Author

xmfcx commented Sep 3, 2024

@knzo25 please have a peek at:

Through this new system, we have access to gpu1.small machines with 15 GB, 4vCPU and an unknown GPU, I didn't make any tests.

Full list: https://aws.amazon.com/codebuild/pricing/ (select us east n. virginia)

But this machine is pricier.

We only need GPU for tests, not for building with CUDA.

So one option would be to separate build and test in the workflow and test on this new machine.

But it requires someone to work on this task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:ci Continuous Integration (CI) processes and testing. (auto-assigned)
Projects
Status: No status
Development

No branches or pull requests

3 participants