VINS-application

Mainly focused on Build process and explanation

VINS-Mono and Fusion application of different sets of cameras and imu on different board including desktop and jetson xavier

This repository contains many branches! as following :

Branch : ZED-mini, Pointgrey_myAHRS, intel D435i, FlightGoggles
- Including config.yaml files and Calibration data
- git clone -b <branch_name> --single-branch https://github.com/engcang/VINS-application
Tested on : Jetson Xavier NX, Jetson Xavier AGX, Jetson TX2, Intel i7-6700k, i7-8700k, i9-10900k, i5-9600k

Result clips : here

VINS-Fusion for PX4 with Masking: here

Index

1. Algorithm & Gpu, Cpu version

2. Parameters

3. Prerequisites

● Ceres solver and Eigen : Mandatory for VINS (build Eigen first)

● OpenCV with CUDA : Necessary for GPU version

● CV_Bridge and image_proc with Built OpenCV : Necessary for GPU version

● USB performance : Have to improve performance of sensors with USB

● IMU-Camera Calibration : Synchronization, time offset, extrinsic parameter

● IMU-Camera rotational extrinsic : Rotational extrinsic between IMU and Cam

● Installation

● Trouble shooting

4. Comparison & Application

1. Algorithm & GPU, CPU version

Mainly use Ceres-solver with Eigen, performance of VINS is strongly proportional to CPU performance and some parameters
CPU version
GPU version

2. Parameters

Camera frame rate
- lower - low time delay, poor performance
- higher - high time delay, better performance
- has to be set from camera launch file : 10~30hz

from src/VINS/config/<config_file_name>.yaml

Max tracking Feature number max_cnt
- 100~150, same correlation as camera frame rates
time offset estimated_td : 1, td : value from kalibr
GPU acceleration use_gpu : 1, use_gpu_acc_flow : 1 (for GPU version)
Thread numbers multiple_thread : enabling multi-threads

3. Prerequisites

● Ceres solver and Eigen : Mandatory for VINS

Eigen latest Stable version below new home or old home

$ wget -O eigen.zip https://gitlab.com/libeigen/eigen/-/archive/3.3.7/eigen-3.3.7.zip #check version
$ unzip eigen.zip
$ cd eigen-3.3.7
& mkdir build && cd build
$ cmake .. && sudo make install

Eigen 3.3.90 version or later for using slicing and Indexing as here

$ git clone https://gitlab.com/libeigen/eigen.git
$ cd eigen 
$ mkdir build && cd build
$ cmake .. && sudo make install

Ceres solver home

$ sudo apt-get install -y cmake libgoogle-glog-dev libatlas-base-dev libsuitesparse-dev
$ wget http://ceres-solver.org/ceres-solver-1.14.0.tar.gz
$ tar zxf ceres-solver-1.14.0.tar.gz
$ mkdir ceres-bin
$ mkdir solver && cd ceres-bin
$ cmake ../ceres-solver-1.14.0 -DEXPORT_BUILD_DIR=ON -DCMAKE_INSTALL_PREFIX="../solver"  #good for build without being root privileged and at wanted directory
$ make -j8 # 8 : number of cores
$ make test
$ make install

● OpenCV with CUDA : Necessary for GPU version

Install CUDA and Graphic Driver :
- for upper than 18.04,

    $ sudo apt install gcc make
    $ sudo ubuntu-drivers devices
    $ sudo ubuntu-drivers autoinstall
    $ sudo reboot
    
    # get cuda install script at https://developer.nvidia.com/cuda-downloads
    $ sudo sh cuda_<version>_linux.run
        # if want to install only graphic driver, get graphic driver install script at 
        # sudo ./NVIDIA_<graphic_driver_installer>.run --dkms
        # --dkms option is recommended when you also install NVIDIA driver, to register it along with kernel
        # otherwise, NVIDIA graphic driver will be gone after kernel upgrade via $ sudo apt upgrade
    $ sudo reboot
    
    $ gedit ~/.bashrc
    # type
    export PATH=<CUDA_PATH>/bin:$PATH #ex: /usr/local/cuda-10.1
    export LD_LIBRARY_PATH=<CUDA_PATH>/lib64:$LD_LIBRARY_PATH #ex : /usr/local/cuda-10.1

check CUDA version using nvcc --version

# check installed cuda version
$ nvcc --version
# if nvcc --version does not print out CUDA,
$ gedit ~/.profile
# type below and save
export PATH=<CUDA_PATH>/bin:$PATH #ex: /usr/local/cuda-10.1
export LD_LIBRARY_PATH=<CUDA_PATH>/lib64:$LD_LIBRARY_PATH #ex : /usr/local/cuda-10.1
$ source ~/.profile

Build OpenCV with CUDA - references : link 1, link 2
- for Xavier do as below or sh file from jetsonhacks here
- If want to use C API (e.g. Darknet YOLO) consider :
  - Recommend version is 3.4.0 because darknet has to use C API with OpenCV refer
  - or Patch as here to use other version
    - should comment the /usr/local/include/opencv2/highgui/highgui_c.h line 139 as here after install

$ sudo apt-get purge libopencv* python-opencv
$ sudo apt-get update
$ sudo apt-get install -y build-essential pkg-config
$ sudo apt-get install -y cmake libavcodec-dev libavformat-dev libavutil-dev \
    libglew-dev libgtk2.0-dev libgtk-3-dev libjpeg-dev libpng-dev libpostproc-dev \
    libswscale-dev libtbb-dev libtiff5-dev libv4l-dev libxvidcore-dev \
    libx264-dev qt5-default zlib1g-dev libgl1 libglvnd-dev pkg-config \
    libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev mesa-utils #libeigen3-dev # recommend to build from source : http://eigen.tuxfamily.org/index.php?title=Main_Page
$ sudo apt-get install python2.7-dev python3-dev python-numpy python3-numpy
$ mkdir <opencv_source_directory> && cd <opencv_source_directory>
$ wget -O opencv.zip https://github.com/opencv/opencv/archive/3.4.1.zip # check version
$ unzip opencv.zip
$ cd <opencv_source_directory>/opencv && mkdir build && cd build
# check your BIN version : http://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/
# 8.6 for RTX3080 7.2 for Xavier, 5.2 for GTX TITAN X
# -D BUILD_opencv_cudacodec=OFF #for cuda10-opencv3.4
$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
      -D CMAKE_C_COMPILER=gcc-6 \
      -D CMAKE_CXX_COMPILER=g++-6 \
      -D CMAKE_INSTALL_PREFIX=/usr/local \
      -D WITH_CUDA=ON \
      -D CUDA_ARCH_BIN=7.2 \
      -D CUDA_ARCH_PTX="" \
      -D ENABLE_FAST_MATH=ON \
      -D CUDA_FAST_MATH=ON \
      -D WITH_CUBLAS=ON \
      -D WITH_LIBV4L=ON \
      -D WITH_GSTREAMER=ON \
      -D WITH_GSTREAMER_0_10=OFF \
      -D WITH_QT=ON \
      -D WITH_OPENGL=ON \
      -D BUILD_opencv_cudacodec=OFF \
      -D CUDA_NVCC_FLAGS="--expt-relaxed-constexpr" \
      -D WITH_TBB=ON \
      ../
$ time make -j8 # 8 : numbers of core
$ sudo make install
$ sudo rm -r <opencv_source_directory> #optional

● when build error :

Please include the appropriate gl headers before including cuda_gl_interop.h => reference 1, 2, 3
modules/cudacodec/src/precomp.hpp:60:37: fatal error: dynlink_nvcuvid.h: No such file or directory compilation terminated. --> for CUDA version 10
- => reference here
- cmake ... -D BUILD_opencv_cudacodec=OFF ...
CUDA_nppicom_LIBRARY not found => reference here
- $ sudo apt-get install nvidia-cuda-toolkit
- or Edit FindCUDA.cmake and OpenCVDetectCUDA.cmake

(Optional) if also contrib for OpenCV should be built,
- add -D OPENCV_EXTRA_MODULES_PATH option as below:

$ cd <opencv_source_directory>
$ wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/3.4.1.zip #check version
$ unzip opencv_contrib.zip
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
      -D CMAKE_C_COMPILER=gcc-6 \
      -D CMAKE_CXX_COMPILER=g++-6 \
      -D CMAKE_INSTALL_PREFIX=/usr/local \
      -D WITH_CUDA=ON \
      -D CUDA_ARCH_BIN=7.2 \
      -D CUDA_ARCH_PTX="" \
      -D ENABLE_FAST_MATH=ON \
      -D CUDA_FAST_MATH=ON \
      -D WITH_CUBLAS=ON \
      -D WITH_LIBV4L=ON \
      -D WITH_GSTREAMER=ON \
      -D WITH_GSTREAMER_0_10=OFF \
      -D WITH_QT=ON \
      -D WITH_OPENGL=ON \
      -D BUILD_opencv_cudacodec=OFF \
      -D CUDA_NVCC_FLAGS="--expt-relaxed-constexpr" \
      -D WITH_TBB=ON \
      -D OPENCV_EXTRA_MODULES_PATH=../opencv_contrib-3.4.1/modules \
      ../
$ time make -j1 # important, use only one core to prevent compile error
$ sudo make install

● CV_Bridge and image_proc with built OpenCV : Necessary for whom built OpenCV manually from above

● CV_bridge

For GPU version, if OpenCV with CUDA was built manually, build cv_bridge manually also

$ cd ~/catkin_ws/src && git clone https://github.com/ros-perception/vision_opencv
# since ROS Noetic is added, we have to checkout to melodic tree
$ cd vision_opencv && git checkout origin/melodic
$ gedit vision_opencv/cv_bridge/CMakeLists.txt

Edit OpenCV PATHS in CMakeLists and include cmake file

#when error, try both lines
find_package(OpenCV 3 REQUIRED PATHS /usr/local/share/OpenCV NO_DEFAULT_PATH
#find_package(OpenCV 3 HINTS /usr/local/share/OpenCV NO_DEFAULT_PATH
  COMPONENTS
    opencv_core
    opencv_imgproc
    opencv_imgcodecs
  CONFIG
)
include(/usr/local/share/OpenCV/OpenCVConfig.cmake) #under catkin_python_setup()

$ cd .. && catkin build cv_bridge

● image_proc (Not recommended)

$ cd ~/catkin_ws/src && git clone https://github.com/ros-perception/image_pipeline

Edit OpenCV PATHS in CMakeLists in

1. depth_image_proc/CMakeLists.txt, 2. image_proc/CMakeLists.txt, 
3. image_view/CMakeLists.txt, 4. stereo_image_proc/CMakeLists.txt
find_package(OpenCV 3 REQUIRED PATHS /usr/local/share/OpenCV NO_DEFAULT_PATH)
5. image_publisher/CMakeLists.txt
find_package(OpenCV 3 REQUIRED PATHS /usr/local/share/OpenCV NO_DEFAULT_PATH COMPONENTS core)
6. image_rotate/CMakeLists.txt
find_package(OpenCV 3 REQUIRED PATHS /usr/local/share/OpenCV NO_DEFAULT_PATH COMPONENTS core imgproc)

$ cd ~/catkin_ws && catkin build

● USB performance : Have to improve performance of sensors with USB

Link : here for x86_64 desktops
TX1/TX2 : here
For Xavier : here

$ sudo ./flash.sh -k kernel -C "usbcore.usbfs_memory_mb=1000" -k kernel-dtb jetson-xavier mmcblk0p1

● Calibration : Kalibr -> synchronization, time offset, extrinsic parameter

Kalibr -> synchronization, time offset
For ZED cameras : here
When Calibrating Fisheye camera like T265
- Try with MEI camera model, as here, which is omni-radtan in Kalibr
- and try this Pull to deal with NaNs here
ImportError: No module named Image reference

$ gedit kalibr/aslam_offline_calibration/kalibr/python/kalibr_camera_calibration/MulticamGraph.py
#import Image
from PIL import Image

focal length initialization error

 $ gedit kalibr/aslam_cv/aslam_cameras/include/aslam/cameras/implementation/PinholeProjection.hpp
 # edit if sentence in line 781
 # comment from line 782 to 795
 f_guesses.push_back(2000.0) #initial guess of focal length!!!!

cameras are not connected

 $ gedit kalibr/aslam_offline_calibration/kalibr/python/kalibr_calibrate_cameras
 # comment from line 201 to 205

● IMU-Camera rotational extrinsic example

Between ROS standard body(IMU) and camera
Left view : Between ROS standard body(IMU) and down-pitched (look downward) camera

● Installation

git clone and build from source

$ cd ~/catkin_ws/src
$ git clone https://github.com/HKUST-Aerial-Robotics/VINS-Fusion #CPU
or 
$ git clone https://github.com/pjrambo/VINS-Fusion-gpu #GPU
$ cd .. && catkin build camera_models # camera models first
$ catkin build

Before build VINS-Fusion, process below could be required.

For GPU version, Edit CMakeLists.txt for loop_fusion and vins_estimator

$ cd ~/catkin_ws/src/VINS-Fusion-gpu/loop_fusion && gedit CMakeLists.txt
or
$ cd ~/catkin_ws/src/VINS-Fusion-gpu/vins_estimator && gedit CMakeLists.txt

##For loop_fusion : line 19
#find_package(OpenCV)
include(/usr/local/share/OpenCV/OpenCVConfig.cmake)

##For vins_estimator : line 20
#find_package(OpenCV REQUIRED)
include(/usr/local/share/OpenCV/OpenCVConfig.cmake)

● Trouble shooting

Aborted error when running vins_node :

 $ echo "export MALLOC_CHECK_=0" >> ~/.bashrc
 $ source ~/.bashrc

If want to try to deal with NaNs, refer here

4. Comparison & Application

/tf vs VINS-Mono on FlightGoggles: youtube, with CPU youtube
Loop Fusion vs vins node on FlightGoggles: youtube
Real World VINS-Mono with pointgrey cam, myAHRS+ imu on Jetson Xavier: youtube, moved faster : youtube
Real World VINS(GPU+version) with pointgrey, myAHRS at Intel i7-8700k, TITAN RTX: youtube
Real World VINS(GPU+version, Stereo) with Intel D435i, on Xavier, max CPU clocked: youtube and youtube2 : screen
VINS mono VS ROVIO: youtube
VINS-Mono vs ROVIO vs ORB-SLAM2: youtube
VINS-Fusion (Stereo) vs S-MSCKF on FlightGoggles: youtube
VINS-Fusion (Stereo) based autonomous flight and 3D mapping using RGB-D camera: youtube
VINS-Fusion (Stereo) with Intel D435i and Pixhawk4 mini fused with T265 camera: here
VINS-Fusion (stereo) with Intel D435i and Pixhawk4 mini on 1km long underground tunnel: here

Name		Name	Last commit message	Last commit date
Latest commit History 122 Commits
vins-fusion-px4		vins-fusion-px4
LICENSE		LICENSE
OpenGLHeader.patch		OpenGLHeader.patch
README.md		README.md
extrinsic.png		extrinsic.png
gpuGraph.py		gpuGraph.py
pitching.png		pitching.png

License

alexjunholee/vins-application

Folders and files

Latest commit

History

Repository files navigation

VINS-application

Mainly focused on Build process and explanation

This repository contains many branches! as following :

Result clips : here

VINS-Fusion for PX4 with Masking: here

Index

1. Algorithm & Gpu, Cpu version

2. Parameters

3. Prerequisites

● Ceres solver and Eigen : Mandatory for VINS (build Eigen first)

● OpenCV with CUDA : Necessary for GPU version

● CV_Bridge and image_proc with Built OpenCV : Necessary for GPU version

● USB performance : Have to improve performance of sensors with USB

● IMU-Camera Calibration : Synchronization, time offset, extrinsic parameter

● IMU-Camera rotational extrinsic : Rotational extrinsic between IMU and Cam

● Installation

● Trouble shooting

4. Comparison & Application

1. Algorithm & GPU, CPU version

2. Parameters

from src/VINS/config/<config_file_name>.yaml

3. Prerequisites

● Ceres solver and Eigen : Mandatory for VINS

● OpenCV with CUDA : Necessary for GPU version

● when build error :

● CV_Bridge and image_proc with built OpenCV : Necessary for whom built OpenCV manually from above

● CV_bridge

● image_proc (Not recommended)

● USB performance : Have to improve performance of sensors with USB

● Calibration : Kalibr -> synchronization, time offset, extrinsic parameter

● IMU-Camera rotational extrinsic example

● Installation

● Trouble shooting

4. Comparison & Application

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages