当前位置：首页 > news >正文

无人机避障——感知篇（Orin nx采用zed2双目相机进行Vins-Fusion-GPU定位，再通过位姿和深度图建图完成实时感知）

news 2025/6/14 7:04:04

本文是在Orin nx上已经安装并用zed2相机跑通Vins-Fusion感知建图的基础上进行Vins-Fusion-gpu进行修改的内容，一些基本的库文件，依赖安装，只讲解opecv4.6.0带GPU加速配置和cv_bridge配置以及zed2相机的配置，其他就不再这里赘述，详情请看我的其他文章：

无人机避障——感知篇（Orin nx采用zed2双目相机进行Vins-Fusion定位，再通过位姿和深度图建图完成实时感知）-CSDN博客

无人机避障——感知篇（在Ubuntu20.04的Orin nx上基于ZED2实现Vins Fusion）-CSDN博客

无人机避障——感知部分（Ubuntu 20.04 复现Vins Fusion跑数据集）胎教级教程_vinsfusion ubuntu20.04-CSDN博客

安装opencv4.6.0带GPU加速配置和cv_bridge安装：

参考文章：

Jetson Orin NX 开发指南（5）: 安装 OpenCV 4.6.0 并配置 CUDA 以支持 GPU 加速_jetson xavier nx 配置 opencv cuda加速-CSDN博客

按上面文章是可以进行opencv4.6.0的安装的，但是cv_bridge虽然安装了但是配置上还会有一些问题，首先需要删除4.2.0版本的ROS自带的opencv：

sudo apt purge libopencv*4.2* python3-opencv libopencv-dev

然后NX中就只剩下刚刚安装的cv_bridge牵连的是opencv4.6.0。

编译Vins-Fusion-GPU：

mkdir -p ~/vins_gpu_ws/src/vins-fusion-gpu/src/
cd ~/vins_gpu_ws/src/vins-fusion-gpu/src/git clone https://github.com/pjrambo/VINS-Fusion-gpu.git

下载完成进入VINS-Fusion-gpu，修改 vins_estimator/CMakeLists.txt 文件和修改 loop_fusion/CMakeLists.txt 文件。

在vins_estimator/CMakeLists.txt 的20行和loop_fusion/CMakeLists.txt 的19行，将opencv的位置进行替换：

# include(/home/dji/opencv/build/OpenCVConfig.cmake)
include("~/Documents/opencv-4.6.0/build/OpenCVConfig.cmake")

在vins_estimator/CMakeLists.txt 的8行和loop_fusion/CMakeLists.txt 的8行，加入cv_bridge的路径：

set(cv_bridge_DIR "/home/nvidia/cv_bridge_pkg/devel/share/cv_bridge/cmake")

然后输入以下内容：

sed -i 's/CV_FONT_HERSHEY_SIMPLEX/cv::FONT_HERSHEY_SIMPLEX/g' `grep CV_FONT_HERSHEY_SIMPLEX -rl ./`
sed -i 's/CV_LOAD_IMAGE_GRAYSCALE/cv::IMREAD_GRAYSCALE/g' `grep CV_LOAD_IMAGE_GRAYSCALE -rl ./`
sed -i 's/CV_BGR2GRAY/cv::COLOR_BGR2GRAY/g' `grep CV_BGR2GRAY -rl ./`
sed -i 's/CV_RGB2GRAY/cv::COLOR_RGB2GRAY/g' `grep CV_RGB2GRAY -rl ./`
sed -i 's/CV_GRAY2RGB/cv::COLOR_GRAY2RGB/g' `grep CV_GRAY2RGB -rl ./`
sed -i 's/CV_GRAY2BGR/cv::COLOR_GRAY2BGR/g' `grep CV_GRAY2BGR -rl ./`
sed -i 's/CV_CALIB_CB_ADAPTIVE_THRESH/cv::CALIB_CB_ADAPTIVE_THRESH/g' `grep CV_CALIB_CB_ADAPTIVE_THRESH -rl ./`
sed -i 's/CV_CALIB_CB_NORMALIZE_IMAGE/cv::CALIB_CB_NORMALIZE_IMAGE/g' `grep CV_CALIB_CB_NORMALIZE_IMAGE -rl ./`
sed -i 's/CV_CALIB_CB_FILTER_QUADS/cv::CALIB_CB_FILTER_QUADS/g' `grep CV_CALIB_CB_FILTER_QUADS -rl ./`
sed -i 's/CV_CALIB_CB_FAST_CHECK/cv::CALIB_CB_FAST_CHECK/g' `grep CV_CALIB_CB_FAST_CHECK -rl ./`
sed -i 's/CV_ADAPTIVE_THRESH_MEAN_C/cv::ADAPTIVE_THRESH_MEAN_C/g' `grep CV_ADAPTIVE_THRESH_MEAN_C -rl ./`
sed -i 's/CV_THRESH_BINARY/cv::THRESH_BINARY/g' `grep CV_THRESH_BINARY -rl ./`
sed -i 's/CV_SHAPE_CROSS/cv::MORPH_CROSS/g' `grep CV_SHAPE_CROSS -rl ./`
sed -i 's/CV_SHAPE_RECT/cv::MORPH_RECT/g' `grep CV_SHAPE_RECT -rl ./`
sed -i 's/CV_TERMCRIT_EPS/cv::TermCriteria::EPS/g' `grep CV_TERMCRIT_EPS -rl ./`
sed -i 's/CV_TERMCRIT_ITER/cv::TermCriteria::MAX_ITER/g' `grep CV_TERMCRIT_ITER -rl ./`
sed -i 's/CV_RETR_CCOMP/cv::RETR_CCOMP/g' `grep CV_RETR_CCOMP -rl ./`
sed -i 's/CV_CHAIN_APPROX_SIMPLE/cv::CHAIN_APPROX_SIMPLE/g' `grep CV_CHAIN_APPROX_SIMPLE -rl ./`
sed -i 's/CV_AA/cv::LINE_AA/g' `grep CV_AA -rl ./`
sed -i 's/CV_LOAD_IMAGE_UNCHANGED/cv::IMREAD_UNCHANGED/g' `grep CV_LOAD_IMAGE_UNCHANGED -rl ./`
sed -i 's/CV_MINMAX/cv::NORM_MINMAX/g' `grep CV_MINMAX -rl ./`

原因参考博客：

Jetson Orin NX 开发指南（6）: VINS-Fusion-gpu 的编译和运行-CSDN博客

编译通过之后运行数据集：

# 新开终端
cd ~/vins_gpu_ws/src/vins-fusion-gpu && source devel/setup.bash && roslaunch vins vins_rviz.launch# 新开终端
cd ~/vins_gpu_ws/src/vins-fusion-gpu && source devel/setup.bash && rosrun vins vins_node src/VINS-Fusion-gpu/config/euroc/euroc_stereo_imu_config.yaml# 新开终端
cd ~/vins_gpu_ws/src/vins-fusion-gpu && source devel/setup.bash && rosrun loop_fusion loop_fusion_node src/VINS-Fusion-gpu/euroc/euroc_stereo_imu_config.yaml# 新开终端跑数据集
cd ~/data_set && rosbag play MH_01_easy.bag

编译之后运行实际ZED2：

参考内容我的以下文章自己新建zed的相机yaml文件：

无人机避障——感知篇（在Ubuntu20.04的Orin nx上基于ZED2实现Vins Fusion）-CSDN博客

但是需要修改 zed2_stereo_config.yaml文件：

修改如下：

%YAML:1.0#common parameters
#support: 1 imu 1 cam; 1 imu 2 cam: 2 cam; 
imu: 1         
num_of_cam: 2  #实时相机
imu_topic: "/zed2/zed_node/imu/data_raw"
image0_topic: "/zed2/zed_node/left/image_rect_gray"
image1_topic: "/zed2/zed_node/right/image_rect_gray"# 录制bag包
# imu_topic: "/zed2/zed_node/imu/data_raw2"
# image0_topic: "/zed2/zed_node/left/image_rect_color2"
# image1_topic: "/zed2/zed_node/right/image_rect_color2"output_path: "/home/vins_gpu_ws/output/"cam0_calib: "cam0.yaml"
cam1_calib: "cam1.yaml"
image_width: 640
image_height: 360# Extrinsic parameter between IMU and Camera.
estimate_extrinsic: 0   # 0  Have an accurate extrinsic parameters. We will trust the following imu^R_cam, imu^T_cam, don't change it.# 1  Have an initial guess about extrinsic parameters. We will optimize around your initial guess.body_T_cam0: !!opencv-matrixrows: 4cols: 4dt: ddata: [0.00621782, 0.00255719, 0.9999774, 0.02442757,-0.99997099, -0.00438481, 0.00622899, 0.02442823,0.00440064, -0.99998712, 0.00252985, 0.00964505,0, 0, 0, 1]body_T_cam1: !!opencv-matrixrows: 4cols: 4dt: ddata: [0.00376341, 0.00237248, 0.9999901, 0.02559884,-0.99998414, -0.00418019, 0.00377331, -0.09545715,0.0041891, -0.99998845, 0.00235671, 0.01015661,0, 0, 0, 1]#Multiple thread support
multiple_thread: 1
use_gpu: 1
use_gpu_acc_flow: 0#feature traker paprameters
max_cnt: 350            # max feature number in feature tracking
min_dist: 30            # min distance between two features 
freq: 10                # frequence (Hz) of publish tracking result. At least 10Hz for good estimation. If set 0, the frequence will be same as raw image 
F_threshold: 1.0        # ransac threshold (pixel)
show_track: 1          # publish tracking image as topic
flow_back: 1            # perform forward and backward optical flow to improve feature tracking accuracy#optimization parameters
max_solver_time: 0.04  # max solver itration time (ms), to guarantee real time
max_num_iterations: 8   # max solver itrations, to guarantee real time
keyframe_parallax: 10.0 # keyframe selection threshold (pixel)#imu parameters       The more accurate parameters you provide, the better performance
# acc_n: 1.4402862002020933e-02          # accelerometer measurement noise standard deviation. 
# gyr_n: 1.3752563738546138e-03         # gyroscope measurement noise standard deviation.     
# acc_w: 5.3890784193863061e-04        # accelerometer bias random work noise standard deviation.  
# gyr_w: 4.5861836272840561e-05       # gyroscope bias random work noise standard deviation.     
# g_norm: 9.81007     # gravity magnitude
acc_n: 0.1          # accelerometer measurement noise standard deviation. 
gyr_n: 0.01         # gyroscope measurement noise standard deviation.     
acc_w: 0.001        # accelerometer bias random work noise standard deviation.  
gyr_w: 0.0001       # gyroscope bias random work noise standard deviation.     
g_norm: 9.81007     # gravity magnitude#unsynchronization parameters
estimate_td: 0                      # online estimate time offset between camera and imu
td: 0.0                             # initial value of time offset. unit: s. readed image clock + td = real image clock (IMU clock)#loop closure parameters
load_previous_pose_graph: 0        # load and reuse previous pose graph; load from 'pose_graph_save_path'
pose_graph_save_path: "/home/nvidia/vins_gpu_ws/output/pose_graph/" # save and load path
save_image: 1                      # save image in pose graph for visualization prupose; you can close this function by setting 0

[注意]：其中这两个量是控制是否运行gpu的，我一开始全部置为1，发现zed2实际测试太卡了，后面就这么设置。

use_gpu: 1
use_gpu_acc_flow: 0

参考我的前面博客写一个bash文件进行一键启动操作：

无人机避障——感知篇（Orin nx采用zed2双目相机进行Vins-Fusion定位，再通过位姿和深度图建图完成实时感知）-CSDN博客

bash文件：

# run.sh文件#!/bin/bash# Start roscore
gnome-terminal -- bash -c "roscore"
# Start RViz
#gnome-terminal -- bash -c "cd ~/vins_gpu_ws/src/vins-fusion-gpu && source devel/setup.bash && roslaunch vins vins_rviz.launch"# Start VINS-Fusion node
sleep 5
gnome-terminal -- bash -c "cd ~/vins_gpu_ws/src/vins-fusion-gpu && source devel/setup.bash && rosrun vins vins_node src/VINS-Fusion-gpu/config/zed/zed2_stereo_config.yaml"#回环检测
sleep 5
gnome-terminal -- bash -c "cd ~/vins_gpu_ws/src/vins-fusion-gpu && source devel/setup.bash && rosrun loop_fusion loop_fusion_node src/VINS-Fusion-gpu/config/zed/zed2_stereo_config.yaml"## 实时相机
sleep 5
gnome-terminal -- bash -c "cd ~/vins_gpu_ws/src/vins-fusion-gpu && source /home/nvidia/zed_ws/devel/setup.bash && roslaunch zed_wrapper zed2.launch"## 实时建栅格地图
sleep 5
gnome-terminal -- bash -c "cd vins_ws && source devel/setup.bash && source /home/nvidia/ego_planner_grid/devel/setup.bash && roslaunch plan_env grid_map.launch"# Play rosbag
# sleep 5
# gnome-terminal -- bash -c "source devel/setup.bash && rosbag play /home/nvidia/data_set/MH_01_easy.bag"# Keep the terminal open until you manually close it
echo "Press Enter to close the terminals"
read