动态环境下基于语义信息与几何约束的视觉SLAM系统

doi:10.11959/j.issn.2096-6652.202342

智能科学与技术学报 ›› 2023, Vol. 5 ›› Issue (4): 477-485.doi: 10.11959/j.issn.2096-6652.202342

动态环境下基于语义信息与几何约束的视觉SLAM系统

李嘉铭, 解明扬(), 张民, 王从庆

南京航空航天大学自动化学院，江苏南京 211106

收稿日期:2023-08-14 修回日期:2023-10-11 出版日期:2023-12-15 发布日期:2023-12-15
通讯作者: 解明扬 E-mail:myxie@nuaa.edu.cn
作者简介:李嘉铭（1999- ），男，南京航空航天大学自动化学院硕士研究生，主要研究方向为移动机器人自主导航。
解明扬（1988- ），男，博士，南京航空航天大学自动化学院副研究员，主要研究方向为智能机器人技术。
张民（1973- ），男，博士，南京航空航天大学自动化学院副研究员，主要研究方向为多无人机/编队协同控制。
王从庆（1960- ），男，博士，南京航空航天大学自动化学院教授，主要研究方向为智能机器人与智能控制。
基金资助:
启元实验室创新基金项目(S20210201102);南京航空航天大学科研与实践创新计划(xcxjh20220342)

Visual SLAM based on semantic information and geometric constraints in dynamic environment

Jiaming LI, Mingyang XIE(), Min ZHANG, Congqing WANG

College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China

Received:2023-08-14 Revised:2023-10-11 Online:2023-12-15 Published:2023-12-15
Contact: Mingyang XIE E-mail:myxie@nuaa.edu.cn
Supported by:
The Qiyuan Lab Innovation Fund Project(S20210201102);Research and Practice Innovation Program of NUAA(xcxjh20220342)

摘要/Abstract

摘要：

目前同时定位与地图构建技术多假设外部环境是静态的，忽略了动态目标对SLAM系统的影响，这一假设很大程度上影响了无人系统自主导航的定位精度与鲁棒性。针对这一问题，提出了一种结合目标检测语义信息与多视图极线几何约束的动态SLAM系统，根据运动概率来判别与剔除系统中的动态特征点，并在TUM数据集及实机平台上进行了算法性能测试。实验结果表明在高动态环境下，该系统相较于ORB-SLAM2在绝对轨迹误差上至少可减小94%以上，相对位移轨迹和旋转角度误差分别至少降低了41%和40%，表明该算法能够有效剔除动态特征点，提高了动态环境下视觉SLAM系统的定位精度和鲁棒性。

关键词: 动态SLAM, 深度学习, 目标检测, 极线几何

Abstract:

Most existing visual SLAM systems assume that the external environment is static, ignoring the influence of dynamic objects on the SLAM system. This assumption largely affects the accuracy and robustness of autonomous navigation. To address this issue, a dynamic SLAM system was proposed, which combined semantic information based on object detection and geometric information from multi-view geometry constraints by defining and discriminating the dynamic feature points in the system based on the moving probability. Experiment results on the public TUM dataset and our robot in real environment showed that, when comparing with ORB-SLAM2, the absolute trajectory error could be reduced larger than 94%, and the average relative position and attitude errors were reduced at least 41% and 40%, respectively, in high dynamic environments. It means that the proposed SLAM system effectively removes dynamic feature points, thus improving the localization accuracy and robustness of the visual SLAM system within high dynamic environments.

Key words: dynamic SLAM, deep learning, object detection, epipolar geometry

中图分类号:

TP242.6

李嘉铭, 解明扬, 张民, 等. 动态环境下基于语义信息与几何约束的视觉SLAM系统[J]. 智能科学与技术学报, 2023, 5(4): 477-485.

Jiaming LI, Mingyang XIE, Min ZHANG, et al. Visual SLAM based on semantic information and geometric constraints in dynamic environment[J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(4): 477-485.

图/表 11

图1

图2

图3

图4

图5

图6

表1

表2

表3

表4

表5

参考文献 25

1	SMITH R C, CHEESEMAN P. On the representation and estimation of spatial uncertainty[J]. The International Journal of Robotics Research, 1986, 5(4): 56-68.
2	王霞, 左一凡. 视觉SLAM研究进展[J]. 智能系统学报, 2020, 15(5): 825-834.
	WANG X, ZUO Y F. Advances in visual SLAM research[J]. CAAI Transactions on Intelligent Systems, 2020, 15(5): 825-834.
3	MUR-ARTAL R, MONTIEL J M M, TARDóS J D. ORB-SLAM: a versatile and accurate monocular SLAM system[J]. IEEE Transactions on Robotics, 2015, 31(5): 1147-1163.
4	MUR-ARTAL R, TARDóS J D. ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras[J]. IEEE Transactions on Robotics, 2017, 33(5): 1255-1262.
5	CAMPOS C, ELVIRA R, RODRíGUEZ J J G, et al. ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM[J]. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.
6	LABBé M, MICHAUD F. Online global loop closure detection for large-scale multi-session graph-based SLAM[C]//Proceedings of 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE Press, 2014: 2661-2666.
7	CARUSO D, ENGEL J, CREMERS D. Large-scale direct SLAM for omnidirectional cameras[C]//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2015: 141-148.
8	郑南宁. 人工智能新时代[J]. 智能科学与技术学报, 2019, 1(1): 1-3.
	ZHENG N N. The new era of artificial intelligence[J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(1): 1-3.
9	YU C, LIU Z X, LIU X J, et al. DS-SLAM: a semantic visual SLAM towards dynamic environments[C]//Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2019: 1168-1174.
10	BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
11	BESCOS B, FáCIL J M, CIVERA J, et al. DynaSLAM: tracking, mapping, and inpainting in dynamic scenes[J]. IEEE Robotics and Automation Letters, 2018, 3(4): 4076-4083.
12	HE K M, GKIOXARI G, DOLLáR P, et al. Mask R-CNN[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway: IEEE Press, 2017: 2980-2988.
13	BESCOS B, CAMPOS C, TARDóS J D, et al. DynaSLAM II: tightly-coupled multi-object tracking and SLAM[J]. IEEE Robotics and Automation Letters, 2021, 6(3): 5191-5198.
14	JI T T, WANG C, XIE L H. Towards real-time semantic RGB-D SLAM in dynamic environments[C]//Proceedings of 2021 IEEE International Conference on Robotics and Automation (ICRA). Piscataway: IEEE Press, 2021: 11175-11181.
15	YUAN X, CHEN S. SaD-SLAM: a visual SLAM based on semantic and depth information[C]//Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2020: 4930-4935.
16	LIU Y B, MIURA J. RDS-SLAM: real-time dynamic SLAM using semantic segmentation methods[J]. IEEE Access, 2021, 9: 23772-23785.
17	ZHONG F W, WANG S, ZHANG Z Q, et al. Detect-SLAM: making object detection and SLAM mutually beneficial[C]//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE Press, 2018: 1001-1010.
18	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot MultiBox detector[M]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 21-37.
19	SOARES J C V, GATTASS M, MEGGIOLARO M A. Crowd-SLAM: visual SLAM towards crowded environments using object detection[J]. Journal of Intelligent & Robotic Systems, 2021, 102(2): 1-16.
20	REDMON J, FARHADI A. YOLOv3: an incremental improvement[EB].2018, arXiv preprint, arXiv:1804.02767.
21	ZHANG C Y, HUANG T, ZHANG R C, et al. PLD-SLAM: a new RGB-D SLAM method with point and line features for indoor dynamic scene[J]. ISPRS International Journal of Geo-Information, 2021, 10(3): 163.
22	WANG Y N, XU K, TIAN Y B, et al. DRG-SLAM: a semantic RGB-D SLAM using geometric features for indoor dynamic scene[C]//Proceedings of 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2022: 1352-1359.
23	LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
24	JOCHER G, CHAURASIA A, STOKEN A, et al. Ultralytics/YOLOv5: v7.0-YOLOv5 sota realtime instance segmentation[Z]. Zenodo, 2022.
25	STURM J, ENGELHARD N, ENDRES F, et al. A benchmark for the evaluation of RGB-D SLAM systems[C]//Proceedings of 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE Press, 2012: 573-580.

序列	ORB-SLAM2/m		YOLO-Only/m		Our System/m		Improvement(O)/%		Improvement(Y)/%
序列	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD
walk_half	0.5763	0.2952	0.1304	0.0267	0.0305	0.0169	94.71	94.28	76.61	36.70
walk_xyz	0.7405	0.3623	0.2584	0.0874	0.0163	0.0079	97.80	97.82	93.69	90.96
walk_static	0.3779	0.1537	0.0082	0.0036	0.0069	0.0028	97.99	97.59	7.32	22.22
walk_rpy	0.7021	0.3425	0.2013	0.1378	0.0213	0.0166	96.97	95.15	89.42	87.95

序列	ORB-SLAM2/m		YOLO-Only/m		Our System/m		Improvement(O)/%		Improvement(Y)/%
序列	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD
walk_half	0.0280	0.0190	0.0183	0.0117	0.0163	0.0101	41.78	46.84	10.93	13.68
walk_xyz	0.0438	0.0344	0.0150	0.0093	0.0120	0.0076	72.60	77.91	20.00	18.28
walk_static	0.0184	0.0146	0.0063	0.0035	0.0057	0.0031	69.02	78.77	9.52	11.43
walk_rpy	0.0416	0.0339	0.0303	0.0208	0.0216	0.0144	48.08	57.52	28.71	30.77

序列	ORB-SLAM2/度		YOLO-Only/度		Our System/度		Improvement(O)/%		Improvement(Y)/%
序列	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD
walk_half	0.6849	0.4271	0.4740	0.2689	0.4072	0.2064	40.55	51.67	14.09	23.24
walk_xyz	0.9306	0.7095	0.4245	0.2942	0.3927	0.2733	57.80	61.48	7.49	7.10
walk_static	0.3863	0.2827	0.1793	0.0943	0.1735	0.0873	55.09	69.12	3.23	7.42
walk_rpy	0.9246	0.7079	0.6277	0.3828	0.4839	0.2900	47.66	59.03	22.91	24.24

序列	DS-SLAM		RDS-SLAM		Detect-SLAM		Our System
序列	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD
walk_half	0.0303	0.0159	0.0807	0.0454	0.0514	—	0.0305	0.0169
walk_xyz	0.0247	0.0161	0.0571	0.0229	0.0241	—	0.0163	0.0079
walk_static	0.0081	0.0036	0.0206	0.0120	—	—	0.0069	0.0028
walk_rpy	0.4442	0.2350	0.1604	0.0873	0.2959	—	0.0213	0.0166

系统	平均每帧处理时间/ms	选用神经网络	硬件平台
ORB-SLAM2	23	—	AMD Ryzen7 5 800H CPU
DS-SLAM	76.5	SegNet	Nvidia Quadro P4 000
RDS-SLAM	50-65	SegNet	Nvidia RTX 2 080Ti
Detect-SLAM	>340	SSD	Nvidia GTX 960M
Our System	62	YOLOv5m6	Nvidia GTX 3 060Laptop

动态环境下基于语义信息与几何约束的视觉SLAM系统

Visual SLAM based on semantic information and geometric constraints in dynamic environment

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 25

相关文章 15

Metrics

推荐阅读 0

[1]	余唯一, 陈涛, 张军平, 单洪明. 基于深度学习的MRI脑卒中病灶分割方法综述[J]. 智能科学与技术学报, 2023, 5(3): 293-312.
[2]	靳聪, 王洁, 郭子淳, 王晶. 融合音画同步的唇形合成研究[J]. 智能科学与技术学报, 2023, 5(3): 397-405.
[3]	黄哲, 王永才, 李德英. 3D目标检测方法研究综述[J]. 智能科学与技术学报, 2023, 5(1): 7-31.
[4]	顾宙瑜, 於跃成, 者甜甜. Rapider-YOLOX：高效的轻量级目标检测网络[J]. 智能科学与技术学报, 2023, 5(1): 92-103.
[5]	卢经纬, 程相, 王飞跃. 求解微分方程的人工智能与深度学习方法：现状及展望[J]. 智能科学与技术学报, 2022, 4(4): 461-476.
[6]	张俊, 许沛东, 陈思远, 高天露, 戴宇欣, 张科, 赵杭, 高杰迈, 白昱阳, 李金星, 张浩然, 李湘, 陈玖香. 物理-数据-知识混合驱动的人机混合增强智能系统管控方法[J]. 智能科学与技术学报, 2022, 4(4): 571-583.
[7]	陈妍, 罗雪琴, 梁伟, 谢永芳. 基于情感信息融合注意力机制的抑郁症识别[J]. 智能科学与技术学报, 2022, 4(4): 600-609.
[8]	栗仁武, 张凌霄, 高林, 李淳芃, 蒋浩. 基于点云的类级别物体姿态估计[J]. 智能科学与技术学报, 2022, 4(2): 246-254.
[9]	李琳辉, 周彬, 任威威, 连静. 行人轨迹预测方法综述[J]. 智能科学与技术学报, 2021, 3(4): 399-411.
[10]	田庆, 胡蓉, 李佐勇, 蔡远征, 余兆钗. 基于SE-YOLOv5s的绝缘子检测[J]. 智能科学与技术学报, 2021, 3(3): 312-321.
[11]	李颖, 陈龙, 黄钊宏, 孙杨, 蔡国榕. 基于多尺度卷积神经网络特征融合的植株叶片检测技术[J]. 智能科学与技术学报, 2021, 3(3): 304-311.
[12]	张阳, 胡月, 辛东嵘. 一种考虑时空关联的深度学习短时交通流预测方法[J]. 智能科学与技术学报, 2021, 3(2): 172-178.
[13]	刘文, 胡琨林, 李岩, 刘钊. 移动目标轨迹预测方法研究综述[J]. 智能科学与技术学报, 2021, 3(2): 149-160.
[14]	刘栋军, 王宇涵, 凌文芬, 彭勇, 孔万增. 基于脑机协同智能的情绪识别[J]. 智能科学与技术学报, 2021, 3(1): 65-75.
[15]	赵亮, 谢志峰, 张坤鹏, 郑玉卿, 付园坤. 无线网络信号传输建模：一种区间二型模糊集成深度学习方法[J]. 智能科学与技术学报, 2020, 2(4): 401-411.