Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956

Open
wants to merge 87 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 85 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
e6d39e6
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Jul 11, 2024
1e1fd44
style(pre-commit): autofix
pre-commit-ci[bot] Jul 11, 2024
f17c6b0
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Jul 12, 2024
9806852
Merge branch 'feat/add_tensorrt_bevdet' of github.com:cyn-liu/autowar…
cyn-liu Jul 12, 2024
eca8ed0
style(pre-commit): autofix
pre-commit-ci[bot] Jul 12, 2024
acb33f0
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Jul 19, 2024
741e917
Merge branch 'feat/add_tensorrt_bevdet' of github.com:cyn-liu/autowar…
cyn-liu Jul 19, 2024
3baff54
style(pre-commit): autofix
pre-commit-ci[bot] Jul 19, 2024
0dac361
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Jul 23, 2024
67f1d92
Merge branch 'feat/add_tensorrt_bevdet' of github.com:cyn-liu/autowar…
cyn-liu Jul 23, 2024
3149fd3
style(pre-commit): autofix
pre-commit-ci[bot] Jul 23, 2024
956fd28
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Jul 31, 2024
35561b9
style(pre-commit): autofix
pre-commit-ci[bot] Jul 31, 2024
71712be
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Jul 31, 2024
1f0ecd4
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Jul 31, 2024
cc1e3d8
style(pre-commit): autofix
pre-commit-ci[bot] Jul 31, 2024
b2d7ef0
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Jul 31, 2024
eda1452
style(pre-commit): autofix
pre-commit-ci[bot] Jul 31, 2024
b1d1162
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 1, 2024
9ba2488
style(pre-commit): autofix
pre-commit-ci[bot] Aug 1, 2024
70066bc
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 1, 2024
3a840a5
style(pre-commit): autofix
pre-commit-ci[bot] Aug 1, 2024
d9e37a8
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 2, 2024
646e36a
Merge branch 'main' into feat/add_tensorrt_bevdet
cyn-liu Aug 2, 2024
afceea0
feat(tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 9, 2024
6d61812
style(pre-commit): autofix
pre-commit-ci[bot] Aug 9, 2024
bd95d71
Merge branch 'main' into feat/add_tensorrt_bevdet
cyn-liu Aug 9, 2024
6e370e3
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 9, 2024
43d3118
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 9, 2024
63ffd19
style(pre-commit): autofix
pre-commit-ci[bot] Aug 9, 2024
156e10f
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 12, 2024
58d4356
style(pre-commit): autofix
pre-commit-ci[bot] Aug 12, 2024
f13adc7
Merge branch 'main' into feat/add_tensorrt_bevdet
cyn-liu Aug 12, 2024
415be2e
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 13, 2024
8c28774
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 13, 2024
a5da3ef
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 13, 2024
a33aa0b
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 13, 2024
03f24cb
use an optimized image_transport
Owen-Liuyuxuan Aug 14, 2024
67ed09a
clean up un-needed include
Owen-Liuyuxuan Aug 14, 2024
c944950
Merge pull request #1 from Owen-Liuyuxuan/bev/feat/optimize_transport
cyn-liu Aug 15, 2024
5d17ef9
style(pre-commit): autofix
pre-commit-ci[bot] Aug 15, 2024
1fbf791
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 15, 2024
e1a5796
style(pre-commit): autofix
pre-commit-ci[bot] Aug 15, 2024
9b475dd
feat(autoware_tensorrt_bevdet): add new 3d object detection method
cyn-liu Aug 15, 2024
8253b07
style(pre-commit): autofix
pre-commit-ci[bot] Aug 15, 2024
2b8efc3
Replace printf with std::cout or std::cerr
cyn-liu Aug 15, 2024
474e7a6
style(pre-commit): autofix
pre-commit-ci[bot] Aug 15, 2024
d425c0a
fix: use english comment
cyn-liu Aug 19, 2024
6676d93
style(pre-commit): autofix
pre-commit-ci[bot] Aug 19, 2024
6477a76
Merge branch 'main' into feat/add_tensorrt_bevdet
cyn-liu Aug 20, 2024
30d208c
fix: replace cpp std log with ros2 log
cyn-liu Aug 20, 2024
38b1d81
style(pre-commit): autofix
pre-commit-ci[bot] Aug 20, 2024
87b040f
fix cppcheck ci
cyn-liu Aug 20, 2024
de1da9f
fix: use english comment
cyn-liu Aug 21, 2024
399b7f3
style(pre-commit): autofix
pre-commit-ci[bot] Aug 21, 2024
d684399
using RCLCPP_DEBUG instead of print
Owen-Liuyuxuan Aug 25, 2024
cf8dfff
using cameara info and TF
Owen-Liuyuxuan Aug 25, 2024
d24a423
debug and make it runnable with ROS2 nuscenes dataset
Owen-Liuyuxuan Aug 26, 2024
f4e3a7f
fix documents and clean up
Owen-Liuyuxuan Aug 26, 2024
e03a881
try clean up spelling
Owen-Liuyuxuan Aug 26, 2024
f8cf04e
try fixing cspell
Owen-Liuyuxuan Aug 26, 2024
751bb07
Merge pull request #2 from Owen-Liuyuxuan/bev_updates
cyn-liu Aug 27, 2024
56adbea
style(pre-commit): autofix
pre-commit-ci[bot] Aug 27, 2024
32a5cfa
fix copyright
cyn-liu Aug 28, 2024
4c43d2b
Merge branch 'main' into feat/add_tensorrt_bevdet
cyn-liu Aug 28, 2024
aee264f
delete specific onnx and engine file path in tier4_perception_launch
cyn-liu Aug 29, 2024
5c071fe
fix cppcheck-differential
cyn-liu Aug 29, 2024
30ea6b5
fix(bevdet.cpp): modify function naming
cyn-liu Sep 2, 2024
b063490
fix: add the namespace autoware::tensorrt_bevdet
cyn-liu Sep 2, 2024
4e30fef
style(pre-commit): autofix
pre-commit-ci[bot] Sep 2, 2024
4946e55
trying fix cppcheck-differential and add maintainer
cyn-liu Sep 2, 2024
7127f09
style(pre-commit): autofix
pre-commit-ci[bot] Sep 2, 2024
f9bbd09
fix: git H and W from CameraInfo and replace uint32_t with int
cyn-liu Sep 3, 2024
534b1c2
style(pre-commit): autofix
pre-commit-ci[bot] Sep 3, 2024
c6161c4
fix: change the class name to UpperCamelCase style
cyn-liu Sep 4, 2024
47b7349
fix: add _ suffix to member variables in Class AdjFrame
cyn-liu Sep 4, 2024
3a5e1c1
style(pre-commit): autofix
pre-commit-ci[bot] Sep 4, 2024
f4a71fb
fix: add _ suffix to member variables in Class BEVDet
cyn-liu Sep 10, 2024
881b457
style(pre-commit): autofix
pre-commit-ci[bot] Sep 10, 2024
cea3f9e
fix: add how to use in readme.md
cyn-liu Sep 10, 2024
8a07984
fix: replace ptr with std::vector
cyn-liu Sep 18, 2024
b6b1f91
style(pre-commit): autofix
pre-commit-ci[bot] Sep 18, 2024
7dc00d7
trying fix pre-commit.ci error
cyn-liu Sep 18, 2024
dbc0337
trying fix pre-commit.ci error
cyn-liu Sep 18, 2024
dc6fd9b
style(pre-commit): autofix
pre-commit-ci[bot] Sep 18, 2024
13b4c5b
replace shared_ptr with std::vector
cyn-liu Sep 23, 2024
93556de
style(pre-commit): autofix
pre-commit-ci[bot] Sep 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,28 @@
<arg name="radar_object_clustering_param_path" value="$(var radar_object_clustering_param_path)"/>
</include>
</group>
<group if="$(eval '&quot;$(var mode)&quot;==&quot;camera&quot;')">
<!-- Camera-based bev detectors-->
<include file="$(find-pkg-share tier4_perception_launch)/launch/object_recognition/detection/detector/camera_bev_detector.launch.xml">
<arg name="input/camera0/image" value="$(var input/camera0/image)"/>
<arg name="input/camera0/info" value="$(var input/camera0/info)"/>
<arg name="input/camera1/image" value="$(var input/camera1/image)"/>
<arg name="input/camera1/info" value="$(var input/camera1/info)"/>
<arg name="input/camera2/image" value="$(var input/camera2/image)"/>
<arg name="input/camera2/info" value="$(var input/camera2/info)"/>
<arg name="input/camera3/image" value="$(var input/camera3/image)"/>
<arg name="input/camera3/info" value="$(var input/camera3/info)"/>
<arg name="input/camera4/image" value="$(var input/camera4/image)"/>
<arg name="input/camera4/info" value="$(var input/camera4/info)"/>
<arg name="input/camera5/image" value="$(var input/camera5/image)"/>
<arg name="input/camera5/info" value="$(var input/camera5/info)"/>
<arg name="input/camera6/image" value="$(var input/camera6/image)"/>
<arg name="input/camera6/info" value="$(var input/camera6/info)"/>
<arg name="input/camera7/image" value="$(var input/camera7/image)"/>
<arg name="input/camera7/info" value="$(var input/camera7/info)"/>
<arg name="number_of_cameras" value="$(var number_of_cameras)"/>
</include>
</group>
<!-- DetectionByTracker -->
<group if="$(var switch/detector/tracker_based)">
<include file="$(find-pkg-share tier4_perception_launch)/launch/object_recognition/detection/detector/tracker_based_detector.launch.xml">
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
<?xml version="1.0"?>
<launch>
<!-- Camera parameters -->
<arg name="input/camera0/image"/>
<arg name="input/camera0/info"/>
<arg name="input/camera1/image"/>
<arg name="input/camera1/info"/>
<arg name="input/camera2/image"/>
<arg name="input/camera2/info"/>
<arg name="input/camera3/image"/>
<arg name="input/camera3/info"/>
<arg name="input/camera4/image"/>
<arg name="input/camera4/info"/>
<arg name="input/camera5/image"/>
<arg name="input/camera5/info"/>
<arg name="input/camera6/image"/>
<arg name="input/camera6/info"/>
<arg name="input/camera7/image"/>
<arg name="input/camera7/info"/>
<arg name="number_of_cameras"/>

<!-- Model parameters -->
<arg name="data_path" default="$(env HOME)/autoware_data" description="packages data and artifacts directory path"/>
<arg name="bevdet_model_name" default="bevdet_one_lt_d"/>

Check warning on line 24 in launch/tier4_perception_launch/launch/object_recognition/detection/detector/camera_bev_detector.launch.xml

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (bevdet)

Check warning on line 24 in launch/tier4_perception_launch/launch/object_recognition/detection/detector/camera_bev_detector.launch.xml

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (bevdet)
<arg name="bevdet_model_path" default="$(var data_path)/tensorrt_bevdet"/>

Check warning on line 25 in launch/tier4_perception_launch/launch/object_recognition/detection/detector/camera_bev_detector.launch.xml

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (bevdet)

Check warning on line 25 in launch/tier4_perception_launch/launch/object_recognition/detection/detector/camera_bev_detector.launch.xml

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (bevdet)
<!-- BEVDet -->
<group>
<push-ros-namespace namespace="bevdet"/>

Check warning on line 28 in launch/tier4_perception_launch/launch/object_recognition/detection/detector/camera_bev_detector.launch.xml

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (bevdet)
<include file="$(find-pkg-share autoware_tensorrt_bevdet)/launch/tensorrt_bevdet.launch.xml">
<arg name="input/img_fl" value="$(var input/camera0/image)"/>
<arg name="input/img_fl/camera_info" value="$(var input/camera0/info)"/>
<arg name="input/img_f" value="$(var input/camera1/image)"/>
<arg name="input/img_f/camera_info" value="$(var input/camera1/info)"/>
<arg name="input/img_fr" value="$(var input/camera2/image)"/>
<arg name="input/img_fr/camera_info" value="$(var input/camera2/info)"/>
<arg name="input/img_bl" value="$(var input/camera3/image)"/>
<arg name="input/img_bl/camera_info" value="$(var input/camera3/info)"/>
<arg name="input/img_b" value="$(var input/camera4/image)"/>
<arg name="input/img_b/camera_info" value="$(var input/camera4/info)"/>
<arg name="input/img_br" value="$(var input/camera5/image)"/>
<arg name="input/img_br/camera_info" value="$(var input/camera5/info)"/>
<arg name="output/boxes" value="boxes"/>
<arg name="data_path" value="$(var data_path)"/>
<arg name="model_name" value="$(var bevdet_model_name)"/>
<arg name="model_path" value="$(var bevdet_model_path)"/>
</include>
</group>
</launch>
102 changes: 102 additions & 0 deletions perception/autoware_tensorrt_bevdet/CMakeLists.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

@cyn-liu cyn-liu Sep 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I modify

find_package(ament_cmake_auto REQUIRED)
ament_auto_package()

to

find_package(autoware_cmake REQUIRED)
autoware_package()

in CMakeLists.txt, I get the following error:

--- stderr: autoware_trt_bevdet                                 
In file included from /usr/local/cuda/include/thrust/system/cuda/detail/util.h:36,
                 from /usr/local/cuda/include/thrust/system/cuda/detail/internal/copy_cross_system.h:41,
                 from /usr/local/cuda/include/thrust/system/cuda/detail/copy.h:100,
                 from /usr/local/cuda/include/thrust/system/detail/adl/copy.h:42,
                 from /usr/local/cuda/include/thrust/detail/copy.inl:22,
                 from /usr/local/cuda/include/thrust/detail/copy.h:90,
                 from /usr/local/cuda/include/thrust/detail/allocator/copy_construct_range.inl:21,
                 from /usr/local/cuda/include/thrust/detail/allocator/copy_construct_range.h:45,
                 from /usr/local/cuda/include/thrust/detail/contiguous_storage.inl:23,
                 from /usr/local/cuda/include/thrust/detail/contiguous_storage.h:234,
                 from /usr/local/cuda/include/thrust/detail/temporary_array.h:40,
                 from /usr/local/cuda/include/thrust/system/detail/generic/scan_by_key.inl:26,
                 from /usr/local/cuda/include/thrust/system/detail/generic/scan_by_key.h:142,
                 from /usr/local/cuda/include/thrust/detail/scan.inl:27,
                 from /usr/local/cuda/include/thrust/scan.h:1560,
                 from /usr/local/cuda/include/thrust/system/detail/generic/reduce_by_key.inl:37,
                 from /usr/local/cuda/include/thrust/system/detail/generic/reduce_by_key.h:87,
                 from /usr/local/cuda/include/thrust/detail/reduce.inl:28,
                 from /usr/local/cuda/include/thrust/reduce.h:781,
                 from /usr/local/cuda/include/thrust/system/detail/generic/transform_reduce.inl:19,
                 from /usr/local/cuda/include/thrust/system/detail/generic/transform_reduce.h:51,
                 from /usr/local/cuda/include/thrust/detail/transform_reduce.inl:25,
                 from /usr/local/cuda/include/thrust/transform_reduce.h:194,
                 from /usr/local/cuda/include/thrust/system/detail/generic/count.inl:19,
                 from /usr/local/cuda/include/thrust/system/detail/generic/count.h:49,
                 from /usr/local/cuda/include/thrust/detail/count.inl:26,
                 from /usr/local/cuda/include/thrust/count.h:231,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/iou3d_nms.hpp:12,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/postprocess.hpp:4,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/bevdet.hpp:6,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/bevdet_node.hpp:3,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/src/bevdet_node.cpp:1:
/usr/local/cuda/include/cub/detail/device_synchronize.cuh:33: error: ignoring ‘#pragma nv_exec_check_disable ’ [-Werror=unknown-pragmas]
   33 | #pragma nv_exec_check_disable
      | 
In file included from /usr/local/cuda/include/thrust/system/cuda/detail/util.h:36,
                 from /usr/local/cuda/include/thrust/system/cuda/detail/internal/copy_cross_system.h:41,
                 from /usr/local/cuda/include/thrust/system/cuda/detail/copy.h:100,
                 from /usr/local/cuda/include/thrust/system/detail/adl/copy.h:42,
                 from /usr/local/cuda/include/thrust/detail/copy.inl:22,
                 from /usr/local/cuda/include/thrust/detail/copy.h:90,
                 from /usr/local/cuda/include/thrust/detail/allocator/copy_construct_range.inl:21,
                 from /usr/local/cuda/include/thrust/detail/allocator/copy_construct_range.h:45,
                 from /usr/local/cuda/include/thrust/detail/contiguous_storage.inl:23,
                 from /usr/local/cuda/include/thrust/detail/contiguous_storage.h:234,
                 from /usr/local/cuda/include/thrust/detail/temporary_array.h:40,
                 from /usr/local/cuda/include/thrust/system/detail/generic/scan_by_key.inl:26,
                 from /usr/local/cuda/include/thrust/system/detail/generic/scan_by_key.h:142,
                 from /usr/local/cuda/include/thrust/detail/scan.inl:27,
                 from /usr/local/cuda/include/thrust/scan.h:1560,
                 from /usr/local/cuda/include/thrust/system/detail/generic/reduce_by_key.inl:37,
                 from /usr/local/cuda/include/thrust/system/detail/generic/reduce_by_key.h:87,
                 from /usr/local/cuda/include/thrust/detail/reduce.inl:28,
                 from /usr/local/cuda/include/thrust/reduce.h:781,
                 from /usr/local/cuda/include/thrust/system/detail/generic/transform_reduce.inl:19,
                 from /usr/local/cuda/include/thrust/system/detail/generic/transform_reduce.h:51,
                 from /usr/local/cuda/include/thrust/detail/transform_reduce.inl:25,
                 from /usr/local/cuda/include/thrust/transform_reduce.h:194,
                 from /usr/local/cuda/include/thrust/system/detail/generic/count.inl:19,
                 from /usr/local/cuda/include/thrust/system/detail/generic/count.h:49,
                 from /usr/local/cuda/include/thrust/detail/count.inl:26,
                 from /usr/local/cuda/include/thrust/count.h:231,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/iou3d_nms.hpp:12,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/postprocess.hpp:4,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/bevdet.hpp:6,
                 from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/src/bevdet.cpp:1:
/usr/local/cuda/include/cub/detail/device_synchronize.cuh:33: error: ignoring ‘#pragma nv_exec_check_disable ’ [-Werror=unknown-pragmas]
   33 | #pragma nv_exec_check_disable
      | 
In file included from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/src/bevdet.cpp:6:
/home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/gatherbev_plugin.hpp:31:20: error: ‘{anonymous}::GATHERBEV_PLUGIN_VERSION’ defined but not used [-Werror=unused-variable]
   31 | static const char *GATHERBEV_PLUGIN_VERSION {"1"};
      |                    ^~~~~~~~~~~~~~~~~~~~~~~~
/home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/gatherbev_plugin.hpp:30:20: error: ‘{anonymous}::GATHERBEV_PLUGIN_NAME’ defined but not used [-Werror=unused-variable]
   30 | static const char *GATHERBEV_PLUGIN_NAME {"GatherBEV"};
      |                    ^~~~~~~~~~~~~~~~~~~~~
In file included from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/src/bevdet.cpp:5:
/home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/preprocess_plugin.hpp:31:20: error: ‘{anonymous}::PRE_PLUGIN_VERSION’ defined but not used [-Werror=unused-variable]
   31 | static const char *PRE_PLUGIN_VERSION {"1"};
      |                    ^~~~~~~~~~~~~~~~~~
/home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/preprocess_plugin.hpp:30:20: error: ‘{anonymous}::PRE_PLUGIN_NAME’ defined but not used [-Werror=unused-variable]
   30 | static const char *PRE_PLUGIN_NAME {"Preprocess"};
      |                    ^~~~~~~~~~~~~~~
In file included from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/src/bevdet.cpp:4:
/home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/bevpool_plugin.hpp:31:20: error: ‘{anonymous}::BEVPOOL_PLUGIN_VERSION’ defined but not used [-Werror=unused-variable]
   31 | static const char *BEVPOOL_PLUGIN_VERSION {"1"};
      |                    ^~~~~~~~~~~~~~~~~~~~~~
/home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/bevpool_plugin.hpp:30:20: error: ‘{anonymous}::BEVPOOL_PLUGIN_NAME’ defined but not used [-Werror=unused-variable]
   30 | static const char *BEVPOOL_PLUGIN_NAME {"BEVPool"};
      |                    ^~~~~~~~~~~~~~~~~~~
In file included from /home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/src/bevdet.cpp:3:
/home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/alignbev_plugin.hpp:30:20: error: ‘{anonymous}::ALIGN_PLUGIN_VERSION’ defined but not used [-Werror=unused-variable]
   30 | static const char *ALIGN_PLUGIN_VERSION {"1"};
      |                    ^~~~~~~~~~~~~~~~~~~~
/home/ros/new-workspace/ws_lc/autoware_ws/autoware_20240611/src/universe/autoware.universe/perception/autoware_trt_bevdet/include/autoware/trt_bevdet/alignbev_plugin.hpp:29:20: error: ‘{anonymous}::ALIGN_PLUGIN_NAME’ defined but not used [-Werror=unused-variable]
   29 | static const char *ALIGN_PLUGIN_NAME {"AlignBEV"};
      |                    ^~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
gmake[2]: *** [CMakeFiles/autoware_trt_bevdet_node.dir/build.make:139: CMakeFiles/autoware_trt_bevdet_node.dir/src/bevdet.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
cc1plus: all warnings being treated as errors
gmake[2]: *** [CMakeFiles/autoware_trt_bevdet_node.dir/build.make:125: CMakeFiles/autoware_trt_bevdet_node.dir/src/bevdet_node.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:137: CMakeFiles/autoware_trt_bevdet_node.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
---
Failed   <<< autoware_trt_bevdet [28.7s, exited with code 2]

Summary: 23 packages finished [48.1s]
  1 package failed: autoware_trt_bevdet
  1 package had stderr output: autoware_trt_bevdet

Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
cmake_minimum_required(VERSION 3.17)
project(autoware_tensorrt_bevdet)

Check warning on line 2 in perception/autoware_tensorrt_bevdet/CMakeLists.txt

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (bevdet)

add_compile_options(-W)
add_compile_options(-std=c++17)
set(CMAKE_CXX_FLAGS_RELEASE "-Wno-deprecated-declarations -O2")
set(CUDA_NVCC_FLAGS "${CUDA_NVCC_FLAGS} -std=c++17 -O0 -Xcompiler -fPIC -g -w -gencode=arch=compute_86,code=sm_86")

Check warning on line 7 in perception/autoware_tensorrt_bevdet/CMakeLists.txt

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (gencode)
find_package(tensorrt_common)
if(NOT ${tensorrt_common_FOUND})
message(WARNING "The tensorrt_common package is not found. Please check its dependencies.")
return()
endif()

find_package(ament_cmake REQUIRED)
find_package(cudnn_cmake_module REQUIRED)
find_package(tensorrt_cmake_module REQUIRED)
find_package(rclcpp REQUIRED)
find_package(yaml-cpp REQUIRED)
find_package(Eigen3 REQUIRED)
find_package(OpenCV REQUIRED)
find_package(PCL REQUIRED)
find_package(sensor_msgs REQUIRED)
find_package(pcl_conversions REQUIRED)
find_package(cv_bridge REQUIRED)
find_package(tf2_geometry_msgs REQUIRED)
find_package(autoware_perception_msgs REQUIRED)

find_package(CUDA REQUIRED)
find_package(CUDAToolkit)
find_package(CUDNN)
find_package(TENSORRT)
if(NOT (CUDAToolkit_FOUND AND CUDNN_FOUND AND TENSORRT_FOUND))
message(WARNING "cuda, cudnn, tensorrt libraries are not found")
return()
endif()

include_directories(
include
SYSTEM
${YAML_CPP_INCLUDE_DIRS}
${EIGEN3_INCLUDE_DIRS}
${OpenCV_INCLUDE_DIRS}
${PCL_INCLUDE_DIRS}
${cv_bridge_INCLUDE_DIRS}
${tf2_geometry_msgs_INCLUDE_DIRS}
${autoware_perception_msgs_INCLUDE_DIRS}
)

cuda_add_executable(${PROJECT_NAME}_node
src/bevdet_node.cpp

Check warning on line 50 in perception/autoware_tensorrt_bevdet/CMakeLists.txt

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (bevdet)
src/bevdet.cpp

Check warning on line 51 in perception/autoware_tensorrt_bevdet/CMakeLists.txt

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (bevdet)
src/preprocess.cu
src/iou3d_nms.cu
src/postprocess.cu
src/data.cpp
src/cpu_jpegdecoder.cpp

Check warning on line 56 in perception/autoware_tensorrt_bevdet/CMakeLists.txt

View workflow job for this annotation

GitHub Actions / spell-check-differential

Unknown word (jpegdecoder)
src/nvjpegdecoder.cpp

src/preprocess_plugin.cu
src/bevpool_plugin.cu
src/alignbev_plugin.cu
src/gatherbev_plugin.cu
)

# Link libraries to executable
ament_target_dependencies(${PROJECT_NAME}_node
"rclcpp"
"sensor_msgs"
"pcl_conversions"
"cv_bridge"
"autoware_perception_msgs"
"tf2_geometry_msgs"
)

target_link_libraries(${PROJECT_NAME}_node
yaml-cpp
libnvinfer.so
libnvonnxparser.so
libz.so
libjpeg.so
rclcpp::rclcpp
stdc++fs
${NVINFER}
${TENSORRT_LIBRARIES}
${CUDA_LIBRARIES}
${CUBLAS_LIBRARIES}
${CUDNN_LIBRARY}
${OpenCV_LIBS}
${PCL_LIBRARIES}
)

install(TARGETS ${PROJECT_NAME}_node
DESTINATION lib/${PROJECT_NAME}
)

install(DIRECTORY
launch
config
DESTINATION share/${PROJECT_NAME}/
)

ament_package()
69 changes: 69 additions & 0 deletions perception/autoware_tensorrt_bevdet/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# tensorrt_bevdet <!-- cspell: ignore bevdet -->

## Purpose

tensorrt_bevdet is a dynamic 3D bev object detection package based on 6 surround view cameras.

## Inner-workings / Algorithms

BEVDet is a BEV perception algorithm based on panoramic cameras. It unifies multi-view images into the perspective of BEV for 3D object detection task. In this implementation, BEVDet network to inference with TensorRT.

## Inputs / Outputs

### Inputs

| Name | Type | Description |
| ---------------------------------- | ------------------------------ | ----------------------------------- |
| `~/input/topic_img_fl` | `sensor_msgs::msg::Image` | input front_left camera image |
| `~/input/topic_img_f` | `sensor_msgs::msg::Image` | input front camera image |
| `~/input/topic_img_fr` | `sensor_msgs::msg::Image` | input front_right camera image |
| `~/input/topic_img_bl` | `sensor_msgs::msg::Image` | input back_left camera image |
| `~/input/topic_img_b` | `sensor_msgs::msg::Image` | input back camera image |
| `~/input/topic_img_br` | `sensor_msgs::msg::Image` | input back_right camera image |
| `~/input/topic_img_fl/camera_info` | `sensor_msgs::msg::CameraInfo` | input front_left camera parameters |
| `~/input/topic_img_f/camera_info` | `sensor_msgs::msg::CameraInfo` | input front camera parameters |
| `~/input/topic_img_fr/camera_info` | `sensor_msgs::msg::CameraInfo` | input front_right camera parameters |
| `~/input/topic_img_bl/camera_info` | `sensor_msgs::msg::CameraInfo` | input back_left camera parameters |
| `~/input/topic_img_b/camera_info` | `sensor_msgs::msg::CameraInfo` | input back camera parameters |
| `~/input/topic_img_br/camera_info` | `sensor_msgs::msg::CameraInfo` | input back_right camera parameters |

### Outputs

| Name | Type | Description |
| ---------------- | ------------------------------------------------ | ---------------- |
| `~/output/boxes` | `autoware_perception_msgs::msg::DetectedObjects` | detected objects |

## How to Use Tensorrt BEVDet Node

1. launch `tensorrt_bevdet_node`

```bash

ros2 launch autoware_tensorrt_bevdet tensorrt_bevdet_node.launch.xml
```

2. play ros2 bag of nuScenes data

Please refer to open source repository [ros2_dataset_bridge](https://github.com/Owen-Liuyuxuan/ros2_dataset_bridge) to publish the ROS 2 topics.

## Limitation

The model is trained on open-source dataset `NuScenes` and has poor generalization on its own dataset, If you want to use this model to infer your data, you need to retrain it.

## Trained Models

You can download the onnx format of trained models by clicking on the links below.

- BEVDet: [bevdet_one_lt_d.onnx](https://drive.google.com/file/d/1eMGJfdCVlDPBphBTjMcnIh3wdW7Q7WZB/view?usp=sharing)

The model was trained in NuScenes database for 20 epochs.

If you want to train model using the [TIER IV's internal database(~2600 key frames)](https://drive.google.com/file/d/1UaarK88HZu09sf7Ix-bEVl9zGNGFwTVL/view?usp=sharing), please refer to the following repositories:[BEVDet adapted to TIER IV dataset](https://github.com/cyn-liu/BEVDet/tree/train_export)

## References/External links

[1] <https://github.com/HuangJunJie2017/BEVDet/tree/dev2.1>

[2] <https://github.com/LCH1238/BEVDet/tree/export>

[3] <https://github.com/LCH1238/bevdet-tensorrt-cpp/tree/one>
14 changes: 14 additions & 0 deletions perception/autoware_tensorrt_bevdet/config/bevdet.param.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
/**:
ros__parameters:
# weight files
onnx_path: "$(var model_path)/$(var model_name).onnx"
engine_path: "$(var model_path)/$(var model_name).engine"
data_params:
N: 6 # camera num
H: 900 # image height
cyn-liu marked this conversation as resolved.
Show resolved Hide resolved
W: 1600 # image width
cams: ["CAM_FRONT_LEFT", "CAM_FRONT", "CAM_FRONT_RIGHT", "CAM_BACK_LEFT", "CAM_BACK", "CAM_BACK_RIGHT"]
post_process_params:
# post-process params
score_threshold: 0.2
class_names: ["car", "truck", "construction_vehicle", "bus", "trailer", "barrier", "motorcycle", "bicycle", "pedestrian", "traffic_cone"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
bev_range: [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]
mean: [123.675, 116.28, 103.53]
std: [58.395, 57.12, 57.375]
use_depth: true
use_adj: true
adj_num: 8
transform_size: 6
cam_params_size: 27
sampling: nearest # nearest # or bicubic
data_config:
Ncams: 6
cams: [CAM_FRONT_LEFT, CAM_FRONT, CAM_FRONT_RIGHT, CAM_BACK_LEFT, CAM_BACK, CAM_BACK_RIGHT]
resize_radio: 0.44
crop: [140, 0]
flip: true
input_size: [256, 704]
resize: [-0.06, 0.11]
resize_test: 0.0
rot: [-5.4, 5.4]
src_size: [900, 1600]
grid_config:
depth: [1.0, 60.0, 0.5]
x: [-51.2, 51.2, 0.8]
y: [-51.2, 51.2, 0.8]
z: [-5, 3, 8]
model:
bevpool_channels: 80
coder:
code_size: 9
max_num: 500
post_center_range: [-61.2, -61.2, -10.0, 61.2, 61.2, 10.0]
score_threshold: 0.1
common_head:
channels: [2, 1, 3, 2, 2]
names: [reg, height, dim, rot, vel]
down_sample: 16
tasks:
- class_names:
[
car,
truck,
construction_vehicle,
bus,
trailer,
barrier,
motorcycle,
bicycle,
pedestrian,
traffic_cone,
]
num_class: 10
test_cfg:
max_per_img: 500
max_pool_nms: false
min_radius: [4, 12, 10, 1, 0.85, 0.175]
nms_rescale_factor:
- [1.0, 0.7, 0.7, 0.4, 0.55, 1.1, 1.0, 1.0, 1.5, 3.5]
nms_thr: [0.2]
nms_type: [rotate]
post_center_limit_range: [-61.2, -61.2, -10.0, 61.2, 61.2, 10.0]
post_max_size: 500
pre_max_size: 1000
score_threshold: 0.1
Loading
Loading