feat(autoware_tensorrt_yolox): add GPU - CUDA device option

[ismetatabay]

Description

This PR adds a parameter to specify which GPU will be used for tensorrt_yolox inference on multiple GPU devices.

Related links

Parent Issue:

• Link: tensorrt_yolox node does not support GPU selection #8178

How was this PR tested?

This PR has been tested with 8 cameras on a 3-GPU computer. Since GPU-0 is used for the lidar centerpoint and other things, the 8 cameras are divided between GPU-1 and GPU-2. The NVIDIA-SMI output can be found in the following image.

Notes for reviewers

None.

Interface changes

Change type	Parameter Name	Type	Default Value	Description
Added	gpu_id	int	0	GPU ID to select CUDA Device

Effects on system behavior

None.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[kminoda]

@ismetatabay Please make sure to submit the PR after all the CIs are passing (json schema check is failing)

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 22 lines in your changes missing coverage. Please review.

Project coverage is 23.99%. Comparing base (a64566e) to head (d3889b5).
Report is 3 commits behind head on main.

Files	Patch %	Lines
...ion/autoware_tensorrt_yolox/src/tensorrt_yolox.cpp	0.00%	11 Missing ⚠️
...utoware_tensorrt_yolox/src/tensorrt_yolox_node.cpp	0.00%	5 Missing ⚠️
..._detector/src/traffic_light_fine_detector_node.cpp	0.00%	5 Missing ⚠️
...include/autoware/tensorrt_yolox/tensorrt_yolox.hpp	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8245      +/-   ##
==========================================
- Coverage   24.11%   23.99%   -0.13%     
==========================================
  Files        1399     1397       -2     
  Lines      102423   102224     -199     
  Branches    38926    38778     -148     
==========================================
- Hits        24702    24529     -173     
+ Misses      75223    75187      -36     
- Partials     2498     2508      +10     

Flag	Coverage Δ		*Carryforward flag
differential	0.00% <0.00%> (?)
total	24.00% <ø> (-0.12%)	⬇️	Carriedforward from 8f97a9d

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[manato]

@ismetatabay
We appreciate your contribution. I guess engine generation by TensorRT would be better to be performed on the target device (GPU) on which inference will run. From this perspective, I wonder we need to set target device before engine generation; so I would appreciate it if you consider that order of operation.

[ismetatabay]

@manato -san, thank you for your valuable comments. Since I am using the same 3xGPU model for our tests, the created engine file for GPU 0 (default) is valid for the remaining two GPUs. If there is another different model GPU, it will be a problem. Therefore, we can update the tensorrt_common package to handle different GPUs as well. What do you think about that?

[manato]

@ismetatabay
Thank you for your consideration!
Yes, my main concern was such kinds of environments that different model GPUs exist. Regarding building engine on the specific device, it seems to be enough to call cudaSetDevice() before an engine builder starts to work, according to FAQs: Q.How do I use TensorRT on multiple GPUs? (I believe you can confirm this behavior by monitoring which GPU is used using nvidia-smi during engine building). If this is the case, I think we don't need to modify tensorrt_common because we just need to call cudaSetDevice before constructing tensorrt_common::TrtCommon.

Sorry for repeatedly asking, but could you consider to move this cuda-related codes into tensorrt_yolox.cpp for further encapsulation so that other nodes who use yolox as a module can enjoy your improvement?

[ismetatabay]

Thank you @manato -san, you are right. I am okay with moving everything to tensorrt_yolox.cpp. With this update, traffic_light_fine_detector can also select GPU as you mentioned. However, if I move everything to TrtYoloX class, I still get an illegal memory access error from CUDA unless I call cudaSetDevice before creating the trt_yolox_ object in the TrtYoloXNode constructor. Do you have any suggestions on that?

autoware.universe/perception/autoware_tensorrt_yolox/src/tensorrt_yolox_node.cpp

Lines 92 to 97 in 3e79b8f

	trt_yolox_ = std::make_unique<tensorrt_yolox::TrtYoloX>(
	model_path, precision, label_map_.size(), score_threshold, nms_threshold, build_config,
	preprocess_on_gpu, calibration_image_list_path, norm_factor, cache_dir, batch_config,
	max_workspace_size, color_map_path);

[manato]

Thank you @ismetatabay -san for your thoughtful consideration!

I guess the cause of the issue you observed is this line; the line attempts to create cudaStream on the default (0) device during construction of an instance, and the stream will be used for memory copy before/after inference. Could you please try to move to call makeCudaStream() after setting the device in the class constructor and see if it makes a difference?

[ismetatabay]

Hello @manato -san, sorry for the late update. I was on leave and only had a chance to update it today. I have moved the updates to the tensorrt_yolox. Could you please check it? Thank you 🙌

[manato]

Hi @ismetatabay -san, thank you for your continuous updates. The modifications look good to me!

[Shin-kyoto]

/review

[Shin-kyoto]

/improve

[github-actions]

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Key issues to review Possible Bug The setCudaDeviceId method returns a boolean, but the constructor throws an exception if it fails. This inconsistency in error handling should be addressed. Consider making the method throw an exception instead of returning a boolean. Logging The use of std::cout for logging GPU selection is not ideal for production code. Consider using a proper logging framework.

[Shin-kyoto]

Thank you for the changes.
Finally, could you please perform a functionality check with all the changes included and document the verification method and results?

[ismetatabay]

Hello @Shin-kyoto -san, @manato -san, @Owen-Liuyuxuan -san,

Thank you for your review and valuable comments. Today, I tested the latest version of the PR with our test vehicle, and everything worked as expected. The vehicle has 8 cameras and 3 GPUs; I divided the 8 cameras between GPU 1 and GPU 2 (4 cameras each) for the tests. The NVIDIA SMI output is shown in the following image.

I used the following launch files to launch the tensorrt_yolox package at our vehicle:

• camera_node_container.launch.py
• camera.launch.xml

If it is okay for you, I think we can merge this PR.

[Owen-Liuyuxuan]

I have no further issue.

[Shin-kyoto]

LGTM

• I confirmed that the content of this PR is enough to merge.
• I did NOT check this PR by running autoware.

[manato]

@ismetatabay
Thank you for your continuous updating to solve my concerns. LGTM!
I confirmed this modification causes no problem at least x86 laptop with 1 GPU.