动态环境下基于语义信息与几何约束的视觉SLAM系统

doi:10.11959/j.issn.2096-6652.202342

Abstract

Abstract:

Most existing visual SLAM systems assume that the external environment is static, ignoring the influence of dynamic objects on the SLAM system. This assumption largely affects the accuracy and robustness of autonomous navigation. To address this issue, a dynamic SLAM system was proposed, which combined semantic information based on object detection and geometric information from multi-view geometry constraints by defining and discriminating the dynamic feature points in the system based on the moving probability. Experiment results on the public TUM dataset and our robot in real environment showed that, when comparing with ORB-SLAM2, the absolute trajectory error could be reduced larger than 94%, and the average relative position and attitude errors were reduced at least 41% and 40%, respectively, in high dynamic environments. It means that the proposed SLAM system effectively removes dynamic feature points, thus improving the localization accuracy and robustness of the visual SLAM system within high dynamic environments.

Key words: dynamic SLAM, deep learning, object detection, epipolar geometry

CLC Number:

TP242.6

Jiaming LI, Mingyang XIE, Min ZHANG, et al. Visual SLAM based on semantic information and geometric constraints in dynamic environment[J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(4): 477-485.

Figures/Tables 11

References 25

1	SMITH R C, CHEESEMAN P. On the representation and estimation of spatial uncertainty[J]. The International Journal of Robotics Research, 1986, 5(4): 56-68.
2	王霞, 左一凡. 视觉SLAM研究进展[J]. 智能系统学报, 2020, 15(5): 825-834.
	WANG X, ZUO Y F. Advances in visual SLAM research[J]. CAAI Transactions on Intelligent Systems, 2020, 15(5): 825-834.
3	MUR-ARTAL R, MONTIEL J M M, TARDóS J D. ORB-SLAM: a versatile and accurate monocular SLAM system[J]. IEEE Transactions on Robotics, 2015, 31(5): 1147-1163.
4	MUR-ARTAL R, TARDóS J D. ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras[J]. IEEE Transactions on Robotics, 2017, 33(5): 1255-1262.
5	CAMPOS C, ELVIRA R, RODRíGUEZ J J G, et al. ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM[J]. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.
6	LABBé M, MICHAUD F. Online global loop closure detection for large-scale multi-session graph-based SLAM[C]//Proceedings of 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE Press, 2014: 2661-2666.
7	CARUSO D, ENGEL J, CREMERS D. Large-scale direct SLAM for omnidirectional cameras[C]//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2015: 141-148.
8	郑南宁. 人工智能新时代[J]. 智能科学与技术学报, 2019, 1(1): 1-3.
	ZHENG N N. The new era of artificial intelligence[J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(1): 1-3.
9	YU C, LIU Z X, LIU X J, et al. DS-SLAM: a semantic visual SLAM towards dynamic environments[C]//Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2019: 1168-1174.
10	BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
11	BESCOS B, FáCIL J M, CIVERA J, et al. DynaSLAM: tracking, mapping, and inpainting in dynamic scenes[J]. IEEE Robotics and Automation Letters, 2018, 3(4): 4076-4083.
12	HE K M, GKIOXARI G, DOLLáR P, et al. Mask R-CNN[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway: IEEE Press, 2017: 2980-2988.
13	BESCOS B, CAMPOS C, TARDóS J D, et al. DynaSLAM II: tightly-coupled multi-object tracking and SLAM[J]. IEEE Robotics and Automation Letters, 2021, 6(3): 5191-5198.
14	JI T T, WANG C, XIE L H. Towards real-time semantic RGB-D SLAM in dynamic environments[C]//Proceedings of 2021 IEEE International Conference on Robotics and Automation (ICRA). Piscataway: IEEE Press, 2021: 11175-11181.
15	YUAN X, CHEN S. SaD-SLAM: a visual SLAM based on semantic and depth information[C]//Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2020: 4930-4935.
16	LIU Y B, MIURA J. RDS-SLAM: real-time dynamic SLAM using semantic segmentation methods[J]. IEEE Access, 2021, 9: 23772-23785.
17	ZHONG F W, WANG S, ZHANG Z Q, et al. Detect-SLAM: making object detection and SLAM mutually beneficial[C]//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE Press, 2018: 1001-1010.
18	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot MultiBox detector[M]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 21-37.
19	SOARES J C V, GATTASS M, MEGGIOLARO M A. Crowd-SLAM: visual SLAM towards crowded environments using object detection[J]. Journal of Intelligent & Robotic Systems, 2021, 102(2): 1-16.
20	REDMON J, FARHADI A. YOLOv3: an incremental improvement[EB].2018, arXiv preprint, arXiv:1804.02767.
21	ZHANG C Y, HUANG T, ZHANG R C, et al. PLD-SLAM: a new RGB-D SLAM method with point and line features for indoor dynamic scene[J]. ISPRS International Journal of Geo-Information, 2021, 10(3): 163.
22	WANG Y N, XU K, TIAN Y B, et al. DRG-SLAM: a semantic RGB-D SLAM using geometric features for indoor dynamic scene[C]//Proceedings of 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2022: 1352-1359.
23	LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
24	JOCHER G, CHAURASIA A, STOKEN A, et al. Ultralytics/YOLOv5: v7.0-YOLOv5 sota realtime instance segmentation[Z]. Zenodo, 2022.
25	STURM J, ENGELHARD N, ENDRES F, et al. A benchmark for the evaluation of RGB-D SLAM systems[C]//Proceedings of 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE Press, 2012: 573-580.

Metrics

Recommended 0

No Suggested Reading articles found!

序列	ORB-SLAM2/m		YOLO-Only/m		Our System/m		Improvement(O)/%		Improvement(Y)/%
序列	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD
walk_half	0.5763	0.2952	0.1304	0.0267	0.0305	0.0169	94.71	94.28	76.61	36.70
walk_xyz	0.7405	0.3623	0.2584	0.0874	0.0163	0.0079	97.80	97.82	93.69	90.96
walk_static	0.3779	0.1537	0.0082	0.0036	0.0069	0.0028	97.99	97.59	7.32	22.22
walk_rpy	0.7021	0.3425	0.2013	0.1378	0.0213	0.0166	96.97	95.15	89.42	87.95

序列	ORB-SLAM2/m		YOLO-Only/m		Our System/m		Improvement(O)/%		Improvement(Y)/%
序列	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD
walk_half	0.0280	0.0190	0.0183	0.0117	0.0163	0.0101	41.78	46.84	10.93	13.68
walk_xyz	0.0438	0.0344	0.0150	0.0093	0.0120	0.0076	72.60	77.91	20.00	18.28
walk_static	0.0184	0.0146	0.0063	0.0035	0.0057	0.0031	69.02	78.77	9.52	11.43
walk_rpy	0.0416	0.0339	0.0303	0.0208	0.0216	0.0144	48.08	57.52	28.71	30.77

序列	ORB-SLAM2/度		YOLO-Only/度		Our System/度		Improvement(O)/%		Improvement(Y)/%
序列	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD
walk_half	0.6849	0.4271	0.4740	0.2689	0.4072	0.2064	40.55	51.67	14.09	23.24
walk_xyz	0.9306	0.7095	0.4245	0.2942	0.3927	0.2733	57.80	61.48	7.49	7.10
walk_static	0.3863	0.2827	0.1793	0.0943	0.1735	0.0873	55.09	69.12	3.23	7.42
walk_rpy	0.9246	0.7079	0.6277	0.3828	0.4839	0.2900	47.66	59.03	22.91	24.24

序列	DS-SLAM		RDS-SLAM		Detect-SLAM		Our System
序列	RMSE	STD	RMSE	STD	RMSE	STD	RMSE	STD
walk_half	0.0303	0.0159	0.0807	0.0454	0.0514	—	0.0305	0.0169
walk_xyz	0.0247	0.0161	0.0571	0.0229	0.0241	—	0.0163	0.0079
walk_static	0.0081	0.0036	0.0206	0.0120	—	—	0.0069	0.0028
walk_rpy	0.4442	0.2350	0.1604	0.0873	0.2959	—	0.0213	0.0166

系统	平均每帧处理时间/ms	选用神经网络	硬件平台
ORB-SLAM2	23	—	AMD Ryzen7 5 800H CPU
DS-SLAM	76.5	SegNet	Nvidia Quadro P4 000
RDS-SLAM	50-65	SegNet	Nvidia RTX 2 080Ti
Detect-SLAM	>340	SSD	Nvidia GTX 960M
Our System	62	YOLOv5m6	Nvidia GTX 3 060Laptop

Visual SLAM based on semantic information and geometric constraints in dynamic environment

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 25

Related Articles 15

Metrics

Recommended 0

[1]	Weiyi YU, Tao CHEN, Junping ZHANG, Hongming SHAN. A survey of deep learning-based MRI stroke lesion segmentation methods [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(3): 293-312.
[2]	Cong JIN, Jie WANG, Zichun GUO, Jing WANG. Lipsynthesis incorporating audio-visual synchronisation [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(3): 397-405.
[3]	Zhe HUANG, Yongcai WANG, Deying LI. A survey of 3D object detection algorithms [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(1): 7-31.
[4]	Zhouyu GU, Yuecheng YU, Tiantian Zhe. Rapider-YOLOX: lightweight object detection network with high precision [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(1): 92-103.
[5]	Jingwei LU, Xiang CHENG, Fei-Yue WANG. Artificial intelligence and deep learning methods for solving differential equations: the state of the art and prospects [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 461-476.
[6]	Jun ZHANG, Peidong XU, Siyuan CHEN, Tianlu GAO, Yuxin DAI, Ke ZHANG, Hang ZHAO, Jiemai GAO, Yuyang BAI, Jinxing LI, Haoran ZHANG, Xiang LI, Jiuxiang CHEN. A hybrid physics-data-knowledge driven approach for human-machine hybrid-augmented intelligence-based system management and control [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 571-583.
[7]	Yan CHEN, Xueqin LUO, Wei LIANG, Yongfang XIE. Depression recognition based on emotional information fused with attentional mechanism [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 600-609.
[8]	Renwu LI, Lingxiao ZHANG, Lin GAO, Chunpeng LI, Hao JIANG. Category-level object pose estimation from depth point cloud [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 246-254.
[9]	Linhui LI, Bin ZHOU, Weiwei REN, Jing LIAN. Review of pedestrian trajectory prediction methods [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(4): 399-411.
[10]	Qing TIAN, Rong HU, Zuoyong LI, Yuanzheng CAI, Zhaochai YU. Insulator detection based on SE-YOLOv5s [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(3): 312-321.
[11]	Ying LI, Long CHEN, Zhaohong HUANG, Yang SUN, Guorong CAI. Plant leaf detection technology based on multi-scale CNN feature fusion [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(3): 304-311.
[12]	Yang ZHANG, Yue HU, Dongrong XIN. A deep learning short-term traffic flow prediction method considering spatial-temporal association [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(2): 172-178.
[13]	Wen LIU, Kunlin HU, Yan LI, Zhao LIU. A review of prediction methods for moving target trajectories [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(2): 149-160.
[14]	Dongjun LIU, Yuhan WANG, Wenfen LING, Yong PENG, Wanzeng KONG. Emotion recognition based on brain and machine collaborative intelligence [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 65-75.
[15]	Liang ZHAO, Zhifeng XIE, Kunpeng ZHANG, Yuqing ZHENG, Yuankun FU. Modeling signal propagation in wireless network:an interval type-2 fuzzy ensemble deep learning approach [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 401-411.