基于深度学习的自动驾驶多模态轨迹预测方法：现状及展望

doi:10.11959/j.issn.2096-6652.202317

智能科学与技术学报 ›› 2023, Vol. 5 ›› Issue (2): 180-199.doi: 10.11959/j.issn.2096-6652.202317

基于深度学习的自动驾驶多模态轨迹预测方法：现状及展望

黄峻¹^,²^,³, 田永林¹^,⁴, 戴星原¹^,⁵, 王晓³^,⁶^,⁷^,⁸, 平之行⁴^,⁵^,⁹

¹ 中国科学院自动化研究所复杂系统管理与控制国家重点实验室，北京 100190
² 澳门科技大学，澳门 999078
³ 青岛智能产业技术研究院，山东青岛 266000
⁴ 北京怀柔平行传感智能研究院，北京 101400
⁵ 中国科学院自动化研究所北京市智能化技术与系统工程技术研究中心，北京 100190
⁶ 安徽大学人工智能学院，安徽合肥 230601
⁷ 自主无人系统技术教育部工程研究中心，安徽合肥 230601
⁸ 安徽省无人系统与智能技术工程研究中心，安徽合肥 230601
⁹ 北方自动控制技术研究所，山西太原 030006

修回日期:2023-05-26 出版日期:2023-06-15 发布日期:2023-06-10
作者简介:黄峻（1998- ），男，中国科学院自动化研究所复杂系统管理与控制国家重点实验室轨迹规划工程师，主要研究方向为智能交通系统、深度学习、强化学习
田永林（1994- ），男，博士，中国科学院自动化研究所博士后，主要研究方向为计算机视觉、智能交通
戴星原（1993- ），男，博士，中国科学院自动化研究所助理研究员，主要研究方向为人工智能、智能交通系统、认知自动驾驶
王晓（1988- ），女，博士，安徽大学人工智能学院教授，青岛智能产业技术研究院院长，主要研究方向为社会交通、动态网群组织、平行智能和社交网络分析
平之行（1961- ），男，北方自动控制技术研究所资深研究员，主要研究方向为平行系统、平行智能、社会计算、知识自动化
基金资助:
国家自然科学基金项目(62173329)

Deep learning-based multimodal trajectory prediction methods for autonomous driving: state of the art and perspectives

Jun HUANG¹^,²^,³, Yonglin TIAN¹^,⁴, Xingyuan DAI¹^,⁵, Xiao WANG³^,⁶^,⁷^,⁸, Zhixing PING⁴^,⁵^,⁹

¹ The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
² Macau University of Science and Technology, Macau 999780, China
³ Qingdao Academy of Intelligent Industries, Qingdao 266000, China
⁴ Beijing Huairou Academy of Parallel Sensing, Beijing 101400, China
⁵ Beijing Engineering Research Center of Intelligent Systems and Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
⁶ School of Artificial Intelligence, Anhui University, Hefei 230601, China
⁷ Engineering Research Center of Autonomous Unmanned System Technology, Ministry of Education, Hefei 230601, China
⁸ Anhui Provincial Engineering Research Center for Unmanned System and Intelligent Technology, Hefei 230601, China
⁹ North Automatic Control Technology Institute, Taiyuan 030006, China

Revised:2023-05-26 Online:2023-06-15 Published:2023-06-10
Supported by:
The National Natural Science Foundation of China(62173329)

摘要/Abstract

摘要：

对周围车辆轨迹的精确预测可以辅助自动驾驶车辆做出合理的即时决策。虽然相比传统轨迹预测算法，深度学习方法已取得较好效果，但是自动驾驶车辆在异构高动态复杂变化环境下实现多模态高精度预测仍存在信息丢失、交互和不确定性难以建模、预测缺乏可解释性等问题。Transformer具备的长距离建模能力和并行计算能力使其不仅在自然语言处理领域取得巨大成功，而且在扩展至自动驾驶多模态轨迹预测任务时也解决了以上问题。基于此，对过去基于深度神经网络的方法，特别是对基于Transformer的方法进行全面总结与回顾；同时分析了Transformer相较于传统序列网络、图神经网络、生成模型的优势，并结合现有难题进行针对性分析与分类。Transformer模型可以更好地应用于多模态轨迹预测任务，此类模型具有更好的泛化性和可解释性。最后，对多模态轨迹预测未来发展方向进行了展望。

关键词: Transformer, 序列网络, 图神经网络, 生成模型, 轨迹预测, 多模态

Abstract:

Although deep learning methods have achieved better results than traditional trajectory prediction algorithms, there are still problems such as information loss, interaction and uncertainty difficulties in modelling, and lack of interpretability of predictions when implementing multimodal high-precision prediction for autonomous vehicles in heterogeneous, highly dynamic and complex changing environments.The newly developed Transformer's long-range modelling capability and parallel computing ability make it a great success not only in the field of natural language processing, but also in solving the above problems when extended to the task of multimodal trajectory prediction for autonomous driving.Based on this, the aim of this paper is to provide a comprehensive summary and review of past deep neural network-based approaches, in particular the Transformer-based approach.The advantages of Transformer over traditional sequential network, graphical neural network and generative model were also analyzed and classified in relation to existing challenges, simultaneously.Transformer models can be better applied to multimodal trajectory prediction tasks, and that such models have better generalisation and interpretability.Finally, the future directions of multimodal trajectory prediction were presented.

Key words: Transformer, sequential network, graph neural network, generative model, trajectory prediction, multimodal

中图分类号:

TP29

黄峻, 田永林, 戴星原, 等. 基于深度学习的自动驾驶多模态轨迹预测方法：现状及展望[J]. 智能科学与技术学报, 2023, 5(2): 180-199.

Jun HUANG, Yonglin TIAN, Xingyuan DAI, et al. Deep learning-based multimodal trajectory prediction methods for autonomous driving: state of the art and perspectives[J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(2): 180-199.

图/表 4

图

表1

图

表2

多模态轨迹预测问题及基于Transformer模型的解决办法"

研究问题	解决办法
时间、空间维度编码问题：	mmTransformer：通过使用3个堆叠Transformer对环境信息、历史轨迹及车辆特征信息分别编码；
时间维度、空间维度信息如何编码？	LAformer：时间密集型多模态轨迹预测模型，通过两阶段方法生成随时间变化、可调整的多模态轨迹并建模
是否会有信息丢失？	交互；
	AgentFormer：时间编码器代替位置编码器，通过时间戳保留时间信息
交互问题（自注意力计算）：	mmTransformer：对交互信息编码，对车辆间依赖关系建模；
智能体间交互如何建模？	AgentFormer：Agent-Aware机制分别计算智能体间和智能体自身内在注意力；
有什么样的依赖关系？	TrajFormer：通过自注意力解决提取局部特征受限问题，更好地提取环境交互信息；
是否有交互信息丢失问题？如何解决？	MPTP：通过补丁合并和移位窗口方法降低自注意力计算复杂性；
自注意力计算效率如何提升？	Gatformer：对智能体-智能体、智能体-环境交互建模
	Spatio-Channel TF：通过基于挤压激励网络的Channel模块捕获相邻Channel之间的相互作用，进而捕获多智
多智能体交互问题：	能体间的交互信息；
多智能体交互信息如何捕获？	GatFormer：采用灵活的图结构，降低计算复杂度，通过交互权重分配提高性能和模型的可解释性，同时保证
	鲁棒性
可解释性问题：
可解释性差的原因？	MPTP：Swim Transformer编码环境信息，GAR编码历史轨迹，生成可解释性高的预测轨迹
如何提高可解释性？
规模化问题：
规模化难的原因？	Wayformer：分析前融合、后融合、分层融合的利弊，为TF如何扩展到大型多维序列提供解方案
编码模块如何优化？

表2

参考文献 86

[87]	ZHANG Y , ZOU Y , TANG J ,et al. A lane-changing prediction method based on temporal convolution network[J]. arXiv preprint, 2020,arXiv:2011.01224.
[88]	GILLES T , SABATINI S , TSISHKOU D ,et al. HOME:heatmap output for future motion estimation[C]// Proceedings of 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). Piscataway:IEEE Press, 2021: 500-507.
[89]	GILLES T , SABATINI S , TSISHKOU D ,et al. GOHOME:graphoriented heatmap output for future motion estimation[C]// Proceedings of 2022 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2022: 9107-9114.
[90]	GILLES T , SABATINI S , TSISHKOU D ,et al. Thomas:trajectory heatmap output with learned multi-agent sampling[J]. arXiv preprint. 2021,arXiv:2110.06607.
[91]	YE M S , CAO T Y , CHEN Q F . TPCN:temporal point cloud networks for motion forecasting[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2021: 11313-11322.
[92]	GIULIARI F , HASAN I , CRISTANI M ,et al. Transformer networks for trajectory forecasting[C]// Proceedings of 2020 25th International Conference on Pattern Recognition (ICPR). Piscataway:IEEE Press, 2020: 10335-10342.
[93]	ALTCHé F , DE LA FORTELLE A . An LSTM network for highway trajectory prediction[C]// Proceedings of 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2017: 353-359.
[94]	ZYNER A , WORRALL S , NEBOT E . Naturalistic driver intention and path prediction using recurrent neural networks[J]. IEEE Transactions on Intelligent Transportation Systems, 2019,21(4): 1584-1594.
[95]	PARK S H , KIM B , KANG C M ,et al. Sequence-to-sequence prediction of vehicle trajectory via LSTM encoder-decoder architecture[C]// Proceedings of 2018 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2018: 1672-1678.
[96]	SONG H , DING W , CHEN Y ,et al. Pip:planning-informed trajectory prediction for autonomous driving[C]// Proceedings of Computer VisionECCV 2020:16th European Conference.[S.l.:s.n.], 2020: 598-614.
[97]	XIN L , WANG P , CHAN C Y ,et al. Intention-aware long horizon trajectory prediction of surrounding vehicles using dual LSTM networks[C]// Proceedings of 2018 21st International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018: 1441-1446.
[98]	SHIRAZI M S , MORRIS B T . Trajectory prediction of vehicles turning at intersections using deep neural networks[J]. Machine Vision and Applications, 2019,30(6): 1097-1109.
[99]	XING Y , LV C , CAO D P . Personalized vehicle trajectory prediction based on joint time-series modeling for connected vehicles[J]. IEEE Transactions on Vehicular Technology, 2020,69(2): 1341-1352.
[100]	ZHANG T , SONG W J , FU M Y ,et al. Vehicle motion prediction at intersections based on the turning intention and prior trajectories model[J]. IEEE/CAA Journal of Automatica Sinica, 2021,8(10): 1657-1666.
[101]	BAE S , SAXENA D , NAKHAEI A ,et al. Cooperation-aware lane change maneuver in dense traffic based on model predictive control with recurrent neural network[C]// Proceedings of 2020 American Control Conference (ACC). Piscataway:IEEE Press, 2020: 1209-1216.
[102]	MO X Y , XING Y , LV C . Interaction-aware trajectory prediction of connected vehicles using CNN-LSTM networks[C]// Proceedings of IECON 2020 the 46th Annual Conference of the IEEE Industrial Electronics Society. Piscataway:IEEE Press, 2020: 5057-5062.
[103]	HUANG Z , WANG J , PI L ,et al. LSTM based trajectory prediction model for cyclist utilizing multiple interactions with environment[J]. Pattern Recognition, 2021,112:107800.
[104]	WANG S B , ZHAO P , YU B ,et al. Vehicle trajectory prediction by knowledge-driven LSTM network in urban environments[J]. Journal of Advanced Transportation, 2020,2020: 1-20.
[105]	JU C , WANG Z , LONG C ,et al. Interaction-aware Kalman neural networks for trajectory prediction[C]// Proceedings of 2020 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2020: 1793-1800.
[106]	LIN L , LI W Z , BI H K ,et al. Vehicle trajectory prediction using LSTMs with spatial-temporal attention mechanisms[J]. IEEE Intelligent Transportation Systems Magazine, 2022,14(2): 197-208.
[107]	SHI X J , GAO Z H , LAUSEN L ,et al. Deep learning for precipitation nowcasting:a benchmark and a new model[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. New York:ACM, 2017: 5622-5632.
[108]	DING W C , CHEN J , SHEN S J . Predicting vehicle behaviors over an extended horizon using behavior interaction network[C]// Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2019: 8634-8640.
[1]	LEVINSON J , ASKELAND J , BECKER J ,et al. Towards fully autonomous driving:systems and algorithms[C]// Proceedings of 2011 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2011: 163-168.
[2]	MAURER M , GERDES J C , LENZ B ,et al. Autonomous driving:technical,legal and social aspects[M]. Springer Berlin: Nature, 2016.
[3]	YURTSEVER E , LAMBERT J , CARBALLO A ,et al. A survey of autonomous driving:common practices and emerging technologies[J]. IEEE Access, 2020,8: 58443-58469.
[4]	张慧, 王坤峰, 王飞跃 . 深度学习在目标视觉检测中的应用进展与展望[J]. 自动化学报, 2017,43(8): 1289-1305.
	ZHANG H , WANG K F , WANG F Y . Advances and perspectives on applications of deep learning in visual object detection[J]. Acta Automatica Sinica, 2017,43(8): 1289-1305.
[5]	田永林, 王雨桐, 王建功 ,等. 视觉 Transformer 研究的关键问题:现状及展望[J]. 自动化学报, 2022,48(4): 957-979.
	TIAN Y L , WANG Y T , WANG J G ,et al. Key problems and progress of vision transformers:the state of the art and prospects[J]. Acta Automatica Sinica, 2022,48(4): 957-979.
[6]	卢经纬, 郭超, 戴星原 ,等. 问答ChatGPT之后:超大预训练模型的机遇和挑战[J]. 自动化学报, 2023,49(4): 705-717.
	LU J W , GUO C , DAI X Y ,et al. The ChatGPT after:opportunities and challenges of very large scale pre-trained models[J]. Acta Automatica Sinica, 2023,49(4): 705-717.
[7]	王飞跃, 缪青海, 张军平 ,等. 探讨AI for Science的影响与意义:现状与展望[J]. 智能科学与技术学报, 2023,5(1): 1-6.
	WANG F Y , MIAO Q H , ZHANG J P ,et al. The DAOs to AI for Science by DeSci:the state of the art and perspective[J]. Chinese Journal of Intelligent Science and Technology, 2023,5(1): 1-6.
[109]	MERCAT J , GILLES T , EL ZOGHBY N ,et al. Multi-head attention for multi-modal joint vehicle motion forecasting[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2020: 9638-9644.
[110]	ZHANG K P , ZHAO L , DONG C X ,et al. AI-TP:attention-based interaction-aware trajectory prediction for autonomous driving[J]. IEEE Transactions on Intelligent Vehicles, 2023,8(1): 73-83.
[111]	TANG C , ZHAN W , TOMIZUKA M . Exploring social posterior collapse in variational autoencoder for interaction modeling[J]. Advances in Neural Information Processing Systems, 2021,34: 8481-8494.
[112]	DEVLIN J , CHANG M W , LEE K ,et al. Bert:pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint, 2018,arXiv:1810.04805.
[113]	LEE N , CHOI W , VERNAZA P ,et al. DESIRE:distant future prediction in dynamic scenes with interacting agents[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017: 2165-2174.
[114]	ZHAO H , GAO J , LAN T ,et al. Tnt:target-driven trajectory prediction[C]// Proceedings of Conference on Robot Learning. New York:PMLR, 2021: 895-904.
[115]	LIU M , CHENG H , CHEN L ,et al. LAformer:trajectory prediction for autonomous driving with lane-aware scene constraints[J]. arXiv preprint, 2023,arXiv:2302.13933.
[116]	YUAN Y , WENG X S , OU Y L ,et al. AgentFormer:agent-aware transformers for socio-temporal multi-agent forecasting[C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2021: 9793-9803.
[117]	PINI S , PERONE C S , AHUJA A ,et al. Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Experts[J]. arXiv preprint, 2022,arXiv:2211.02131.
[118]	CHEN W , WANG F , SUN H . S2tnet:spatio-temporal transformer networks for trajectory prediction in autonomous driving[C]// Proceedings of Asian Conference on Machine Learning. New York:PMLR, 2021: 454-469.
[119]	BHAT M , FRANCIS J , OH J . Trajformer:trajectory prediction with local self-attentive contexts for autonomous driving[J]. arXiv preprint, 2020,arXiv.2011.14910.
[8]	李娟娟, 秦蕊, 丁文文 ,等. 基于Web3的去中心化自治组织与运营新框架[J]. 自动化学报, 2023,49(5): 985-998.
	LI J J , QIN R , DING W W ,et al. A new framework for Web3-powered decentralized autonomous organizations and operations[J]. Acta Automatica Sinica, 2023,49(5): 985-998.
[9]	LECUN Y , BOSER B , DENKER J S ,et al. Backpropagation applied to handwritten zip code recognition[J]. Neural Computation, 1989,1(4): 541-551.
[10]	YE P J , WANG X , ZHENG W B ,et al. Parallel cognition:hybrid intelligence for human-machine interaction and management[J]. Frontiers of Information Technology ＆ Electronic Engineering, 2022,23(12): 1765-1779.
[11]	WANG X , YANG J , HAN J P ,et al. Metaverses and DeMetaverses:from digital twins in CPS to parallel intelligence in CPSS[J]. IEEE Intelligent Systems, 2022,37(4): 97-102.
[12]	XIE G , SHANGGUAN A Q , FEI R ,et al. Motion trajectory prediction based on a CNN-LSTM sequential model[J]. Science China Information Sciences, 2020,63(11): 1-21.
[13]	GIRSHICK R . Fast R-CNN[C]// Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2015: 1440-1448.
[14]	HE K M , GKIOXARI G , DOLLáR P , et al . Mask R-CNN[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2980-2988.
[15]	GRAVES A . Generating sequences with recurrent neural networks[J]. arXiv preprint. 2013,2013,arXiv:1308.0850.
[16]	BAHDANAU D , CHO K , BENGIO Y . Neural machine translation by jointly learning to align and translate[J]. arXiv preprint, 2014,arXiv:1409.0473.
[17]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[120]	ZHAO J , LI X , XUE Q ,et al. Spatial-channel transformer network for trajectory prediction on the traffic scenes[J]. arXiv preprint, 2021,arXiv:2101.11472.
[121]	HU J , SHEN L , SUN G . Squeeze-and-excitation networks[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 7132-7141.
[122]	HUANG Z Y , MO X Y , LV C . Multi-modal motion prediction with transformer-based neural network for autonomous driving[C]// Proceedings of 2022 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2022: 2605-2611.
[123]	ZHANG K P , FENG X L , WU L ,et al. Trajectory prediction for autonomous driving using spatial-temporal graph attention transformer[J]. IEEE Transactions on Intelligent Transportation Systems, 2022,23(11): 22343-22353.
[124]	LIU Z , LIN Y T , CAO Y ,et al. Swin transformer:hierarchical vision transformer using shifted windows[C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2021: 9992-10002.
[125]	NAYAKANTI N , AL-RFOU R , ZHOU A ,et al. Wayformer:motion forecasting via simple ＆ efficient attention networks[J]. arXiv preprint, 2022,arXiv:2207.05844.
[126]	CHAMI I , ABU-EL-HAIJA S , PEROZZI B ,et al. Machine learning on graphs:A model and comprehensive taxonomy[J]. Journal of Machine Learning Research, 2022,23(89): 1-64.
[127]	LIANG M , YANG B , HU R ,et al. Learning Lane Graph Representations for Motion Forecasting[C]// Proceedings of European Conference on Computer Vision. Cham:Springer, 2020: 541-556.
[128]	BEHBAHANI F , SHIARLIS K , CHEN X ,et al. Learning from demonstration in the wild[C]// Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2019: 775-781.
[129]	CASAS S , GULINO C , LIAO R J ,et al. SpAGNN:spatially-aware graph neural networks for relational behavior forecasting from sensor data[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2020: 9491-9497.
[130]	RHINEHART N , MCALLISTER R , KITANI K ,et al. PRECOG:PREdiction conditioned on goals in visual multi-agent settings[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2020: 2821-2830.
[18]	FU R , ZHANG Z , LI L . Using LSTM and GRU neural network methods for traffic flow prediction[C]// Proceedings of 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC). Piscataway:IEEE Press, 2016: 324-328.
[19]	DEY R , SALEM F M . Gate-variants of Gated Recurrent Unit (GRU) neural networks[C]// Proceedings of 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS). Piscataway:IEEE Press, 2017: 1597-1600.
[20]	CHUNG J , GULCEHRE C , CHO K H ,et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv preprint, 2014,arXiv:1412.3555.
[21]	MESSAOUD K , YAHIAOUI I , VERROUST-BLONDET A ,et al. Nonlocal social pooling for vehicle trajectory prediction[C]// Proceedings of 2019 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2019: 975-980.
[22]	MESSAOUD K , YAHIAOUI I , VERROUST-BLONDET A , ,et al. Attention based vehicle trajectory prediction[J]. IEEE Transactions on Intelligent Vehicles, 2021,6(1): 175-185.
[23]	MESSAOUD K , DEO N , TRIVEDI M M ,et al. Trajectory prediction for autonomous driving based on multi-head attention with joint agentmap representation[C]// Proceedings of 2021 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2021: 165-170.
[24]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. New York:ACM, 2017: 6000-6010.
[25]	LIU Y C , ZHANG J H , FANG L J ,et al. Multimodal motion prediction with stacked transformers[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2021: 7573-7582.
[26]	ZHANG K , LI L . Explainable multimodal trajectory prediction using attention models[J]. Transportation Research Part C:Emerging Technologies, 2022,143:103829.
[27]	WU Z H , PAN S R , CHEN F W ,et al. A comprehensive survey on graph neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021,32(1): 4-24.
[28]	SCARSELLI F , GORI M , TSOI A C ,et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2009,20(1): 61-80.
[131]	CASAS S , GULINO C , SUO S ,et al. Implicit Latent Variable Model for Scene-Consistent Motion Forecasting[C]// Proceeding of European Conference on Computer Vision. Cham:Springer, 2020: 624-641.
[132]	LI X , YING X W , CHUAH M C . GRIP:graph-based interactionaware trajectory prediction[C]// Proceedings of 2019 IEEE Intelligent Transportation Systems Conference (ITSC). Piscataway:IEEE Press, 2019: 3960-3966.
[133]	LI X , YING X , CHUAH M C . Grip++:enhanced graph-based interaction-aware trajectory prediction for autonomous driving[J]. arXiv preprint, 2019,arXiv:1907.07792.
[134]	CHANDRA R , GUAN T R , PANUGANTI S ,et al. Forecasting trajectory and behavior of road-agents using spectral clustering in graphLSTMs[J]. IEEE Robotics and Automation Letters, 2020,5(3): 4882-4890.
[135]	GAO J Y , SUN C , ZHAO H ,et al. VectorNet:encoding HD maps and agent dynamics from vectorized representation[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 11522-11530.
[136]	GU J R , SUN C , ZHAO H . DenseTNT:end-to-end trajectory prediction from dense goal sets[C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2021: 15283-15292.
[137]	CHAI Y , SAPP B , BANSAL M ,et al. Multipath:multiple probabilistic anchor trajectory hypotheses for behavior prediction[J]. arXiv preprint, 2019,arXiv:1910.05449.
[138]	HUANG Y F , BI H K , LI Z X ,et al. STGAT:modeling spatialtemporal interactions for human trajectory prediction[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2019: 6271-6280.
[139]	GUPTA A , JOHNSON J , LI F F ,et al. Social GAN:socially acceptable trajectories with generative adversarial networks[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 2255-2264.
[140]	LI J C , MA H B , TOMIZUKA M . Conditional generative neural system for probabilistic trajectory prediction[C]// Proceedings of 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway:IEEE Press, 2019: 6150-6156.
[141]	YANG B , HE C , WANG P ,et al. TPPO:a novel trajectory predictor with pseudo oracle[J]. arXiv preprint, 2020,arXiv:2002.01852.
[29]	DIEHL F , BRUNNER T , LE M T ,et al. Graph neural networks for modelling traffic participant interaction[C]// Proceedings of 2019 IEEE Intelligent Vehicles Symposium. Piscataway:IEEE Press, 2019: 695-701.
[30]	KIPF T N , WELLING M . Semi-supervised classification with graph convolutional networks[J]. arXiv preprint, 2016,arXiv:1609.02907.
[31]	ZHANG S , TONG H H , XU J J ,et al. Graph convolutional networks:a comprehensive review[J]. Computational Social Networks, 2019,6(1): 1-23.
[32]	VELI?KOVI? P , CUCURULL G , CASANOVA A ,et al. Graph attention networks[J]. arXiv preprint, 2017,arXiv:1710.10903.
[33]	JI H Y , WANG X , SHI C ,et al. Heterogeneous graph propagation network[J]. IEEE Transactions on Knowledge and Data Engineering, 2023,35(1): 521-532.
[34]	WANG X , HE X N , CAO Y X ,et al. KGAT:knowledge graph attention network for recommendation[C]// Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery ＆ Data Mining. New York:ACM, 2019: 950-958.
[35]	KIPF T N , WELLING M . Variational graph auto-encoders[J]. arXiv preprint, 2016,arXiv:1611.07308.
[36]	PAN S , HU R , LONG G ,et al. Adversarially regularized graph autoencoder for graph embedding[J]. arXiv preprint, 2018,arXiv:1802.04407.
[37]	ZHOU D W , ZHENG L C , HAN J W ,et al. A data-driven graph generative model for temporal interaction networks[C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery ＆ Data Mining. New York:ACM, 2020: 401-411.
[38]	BONGINI P , BIANCHINI M , SCARSELLI F . Molecular generative graph neural networks for drug discovery[J]. Neurocomputing, 2021,450: 242-252.
[39]	YAN S , XIONG Y , LIN D . Spatial temporal graph convolutional networks for skeleton-based action recognition[J]. arXiv preprint, 2018,arXiv:1801.07455.
[142]	SADEGHIAN A , KOSARAJU V , SADEGHIAN A ,et al. SoPhie:an attentive GAN for predicting paths compliant to social and physical constraints[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019: 1349-1358.
[143]	LIU S H , LIU H B , BI H K ,et al. CoL-GAN:plausible and collisionless trajectory prediction by attention-based GAN[J]. IEEE Access, 2020,8: 101662-101671.
[144]	CHOI S , KWEON N , YANG C ,et al. DSA-GAN:driving style attention generative adversarial network for vehicle trajectory prediction[C]// Proceedings of 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). Piscataway:IEEE Press, 2021: 1515-1520.
[145]	KINGMA D P , WELLING M . Auto-encoding variational bayes[J]. arXiv preprint, 2013,arXiv:1312.6114.
[146]	BALDI P . Autoencoders,unsupervised learning and deep architectures[C]// Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning workshop. New York:ACM, 2011: 37-50.
[147]	GóMEZ-HUéLAMO C , CONDE M V , ORTIZ M ,et al. Exploring attention GAN for vehicle motion prediction[C]// Proceedings of 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2022: 4011-4016.
[148]	WANG W , LIU C , ZHAO D . How much data are enough? A statistical approach with case study on longitudinal driving behavior[J]. IEEE Transactions on Intelligent Vehicles, 2017,2(2): 85-98.
[149]	王晓, 杨林瑶, 胡斌 ,等. 平行推理:一种基于ACP方法的虚实互动的知识协同框架[J]. 智能科学与技术学报, 2023,5(1): 69-82.
	WANG X , YANG L Y , HU B ,et al. Parallel reasoning:a virtual-real interactive knowledge collaboration framework based on ACP approach[J]. Chinese Journal of Intelligent Science and Technology, 2023,5(1): 69-82.
[150]	田永林, 陈苑文, 杨静 ,等. 元宇宙与平行系统:发展现状、对比及展望[J]. 智能科学与技术学报, 2023,5(1): 121-132.
	TIAN Y L , CHEN Y W , YANG J ,et al. Metaverses and parallel systems:the state of the art,comparisons and prospects[J]. Chinese Journal of Intelligent Science and Technology, 2023,5(1): 121-132.
[40]	GUO S N , LIN Y F , FENG N ,et al. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019,33(1): 922-929.
[41]	GOODFELLOW I , POUGET-ABADIE J , MIRZA M ,et al. Generative adversarial networks[J]. Communications of the ACM, 2020,63(11): 139-144.
[42]	CRESWELL A , WHITE T , DUMOULIN V ,et al. Generative adversarial networks:an overview[J]. IEEE Signal Processing Magazine, 2018,35(1): 53-65.
[43]	HO J , ERMON S . Generative adversarial imitation learning[C]// Proceedings of the 30th International Conference on Neural Information Processing Systems. New York:ACM, 2016: 4572-4580.
[44]	SONG J M , REN H Y , SADIGH D ,et al. Multi-agent generative adversarial imitation learning[C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York:ACM, 2018: 7472-7483.
[45]	KOSARAJU V , SADEGHIAN A , MARTíN-MARTíN R , ,et al. SocialBiGAT:multimodal trajectory forecasting using bicycle-GAN and graph attention networks[J]. arXiv preprint, 2019,arXiv:1907.03395.
[46]	ZHAO T , ZHAO R , ESKENAZI M . Learning discourse-level diversity for neural dialog models using conditional variational autoencoders[J]. arXiv preprint, 2017,arXiv:1703.10960.
[47]	BAO J M , CHEN D , WEN F ,et al. CVAE-GAN:fine-grained image generation through asymmetric training[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2764-2773.
[48]	IVANOVIC B , LEUNG K , SCHMERLING E ,et al. Multimodal deep generative models for trajectory prediction:a conditional variational autoencoder approach[J]. IEEE Robotics and Automation Letters, 2021,6(2): 295-302.
[49]	HONG J , SAPP B , PHILBIN J . Rules of the road:predicting driving behavior with a convolutional model of semantic interactions[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019: 8446-8454.
[50]	LEFèVRE S , VASQUEZ D , LAUGIER C . A survey on motion prediction and risk assessment for intelligent vehicles[J]. ROBOMECH Journal, 2014,1(1): 1-14.
[51]	GONZáLEZ D , PéREZ J , MILANéS V ,et al. A review of motion planning techniques for automated vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2016,17(4): 1135-1145.
[52]	LIU J B , MAO X Y , FANG Y Q ,et al. A survey on deep-learning approaches for vehicle trajectory prediction in autonomous driving[C]// Proceedings of 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO). Piscataway:IEEE Press, 2021: 978-985.
[53]	MOZAFFARI S , AL-JARRAH O Y , DIANATI M ,et al. Deep learningbased vehicle behavior prediction for autonomous driving applications:a review[J]. IEEE Transactions on Intelligent Transportation Systems, 2022,23(1): 33-47.
[54]	LEON F , GAVRILESCU M . A review of tracking and trajectory prediction methods for autonomous driving[J]. Mathematics, 2021,9(6): 660.
[55]	GOMES I , WOLF D . A review on intention-aware and interactionaware trajectory prediction for autonomous vehicles[J]. 2022
[56]	SHIRAZI M S , MORRIS B T . Looking at intersections:a survey of intersection monitoring,behavior and safety analysis of recent studies[J]. IEEE Transactions on Intelligent Transportation Systems, 2017,18(1): 4-24.
[57]	HUANG Y J , DU J T , YANG Z R ,et al. A survey on trajectory-prediction methods for autonomous driving[J]. IEEE Transactions on Intelligent Vehicles, 2022,7(3): 652-674.
[58]	KAELBLING L P , LITTMAN M L , MOORE A W . Reinforcement learning:a survey[J]. Journal of Artificial Intelligence Research, 1996,4: 237-285.
[59]	NAYAK A , ESKANDARIAN A , DOERZAPH Z . Uncertainty estimation of pedestrian future trajectory using Bayesian approximation[J]. IEEE Open Journal of Intelligent Transportation Systems, 2022,3: 617-630.
[60]	WANG X , TANG K , DAI X Y ,et al. Safety-balanced driving-style aware trajectory planning in intersection scenarios with uncertain environment[J]. IEEE Transactions on Intelligent Vehicles, 2023,8(4): 2888-2898.
[61]	DEO N , TRIVEDI M M . Multi-modal trajectory prediction of surrounding vehicles with maneuver based LSTMs[C]// Proceedings of 2018 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2018: 1179-1184.
[62]	SHOUNO O . Deep unsupervised learning of a topological map of vehicle maneuvers for characterizing driving styles[C]// Proceedings of 2018 21st International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018: 2917-2922.
[63]	LIPTON Z C . The mythos of model interpretability:in machine learning,the concept of interpretability is both important and slippery[J]. Queue, 2018,16(3): 31-57.
[64]	CHAKRABORTY S , TOMSETT R , RAGHAVENDRA R ,et al. Interpretability of deep learning models:A survey of results[C]// 2017 IEEE SmartWorld,Ubiquitous Intelligence ＆ Computing,Advanced ＆ Trusted Computed,Scalable Computing ＆ Communications,Cloud ＆ Big Data Computing,Internet of People and Smart City Innovation. Piscataway:IEEE Press, 2017: 1-6.
[65]	GUNNING D , STEFIK M , CHOI J ,et al. XAI-explainable artificial intelligence[J]. Science Robotics, 2019,4(37): eaay7120.
[66]	DU M N , LIU N H , HU X . Techniques for interpretable machine learning[J]. Communications of the ACM, 2019,63(1): 68-77.
[67]	HUME D . An inquiry concerning human understanding[M]. New York: Oxford University Press, 2003.
[68]	YUAN Y , KITANI K . Diverse trajectory forecasting with determinantal point processes[J]. arXiv preprint, 2019,arXiv:1907.04967.
[69]	KIM J , CANNY J . Interpretable learning for self-driving cars by visualizing causal attention[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2961-2969.
[70]	CAESAR H , BANKITI V , LANG A H ,et al. nuScenes:a multimodal dataset for autonomous driving[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 11618-11628.
[71]	HOUSTON J , ZUIDHOF G , BERGAMINI L ,et al. One thousand and one hours:Self-driving motion prediction dataset[C]// Proceedings of Conference on Robot Learning. New York:PMLR, 2021: 409-418.
[72]	SUN P , KRETZSCHMAR H , DOTIWALLA X ,et al. Scalability in perception for autonomous driving:waymo open dataset[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 2443-2451.
[73]	CHANG M F , LAMBERT J , SANGKLOY P ,et al. Argoverse:3D tracking and forecasting with rich maps[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019: 8740-8749.
[74]	WILSON B , QI W , AGARWAL T ,et al. Argoverse 2:next generation datasets for self-driving perception and forecasting[J]. arXiv preprint, 2023,arXiv:2301.00493.
[75]	ZHAN W , SUN L , WANG D ,et al. Interaction dataset:an international,adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps[J]. arXiv preprint, 2019,arXiv:1910.03088.
[76]	KRAJEWSKI R , BOCK J , KLOEKER L ,et al. The highD dataset:a drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems[C]// Proceedings of 2018 21st International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018: 2118-2125.
[77]	HUANG X Y , CHENG X J , GENG Q C ,et al. The ApolloScape dataset for autonomous driving[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway:IEEE Press, 2018: 954-960.
[78]	GEIGER A , LENZ P , STILLER C ,et al. Vision meets robotics:the KITTI dataset[J]. International Journal of Robotics Research, 2013,32(11): 1231-1237.
[79]	FANG L J , JIANG Q H , SHI J P ,et al. TPNet:trajectory proposal net work for motion prediction[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 6796-6805.
[80]	NIKHIL N , MORRIS B T . Convolutional Neural Network for Trajectory Prediction[C]// European Conference on Computer Vision. Cham:Springer, 2019: 186-196.
[81]	CHOU F C , LIN T H , CUI H G ,et al. Predicting motion of vulnerable road users using high-definition maps and efficient ConvNets[C]// Proceedings of 2020 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2020: 1655-1662.
[82]	CUI H G , RADOSAVLJEVIC V , CHOU F C ,et al. Multimodal trajectory predictions for autonomous driving using deep convolutional networks[C]// Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2019: 2090-2096.
[83]	PHAN-MINHT , GRIGOREE C , BOULTONF A ,et al. CoverNet:multimodal behavior prediction using trajectory sets[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 14062-14071.
[151]	MIAO Q H , LV Y S , HUANG M ,et al. Parallel learning:overview and perspective for computational learning across Syn2Real and Sim2Real[J]. IEEE/CAA Journal of Automatica Sinica, 2023,10(3): 603-631.
[84]	CUI H G , NGUYEN T , CHOU F C ,et al. Deep kinematic models for kinematically feasible vehicle trajectory predictions[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2020: 10563-10569.
[85]	DEO N , TRIVEDI M M . Convolutional social pooling for vehicle trajectory prediction[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway:IEEE Press, 2018: 1468-1476.
[86]	MARZINOTTO A , COLLEDANCHISE M , SMITH C ,et al. Towards a unified behavior trees framework for robot control[C]// Proceedings of 2014 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2014: 5420-5427.

数据集	年份	交通参与者	传感器	场景	数据内容	基于此数据集的算法
NuScenes^[70]	2020年	车辆、行人、自行车	雷达、相机	城市	驾驶轨迹、高清地图	MHA-JAM
Lyft^[71]	2020年	车辆、行人、自行车	雷达、相机	城市	驾驶轨迹、高清地图	Gatformer
Waymo^[72]	2020年	车辆、行人、自行车	雷达、相机	城市	驾驶轨迹、高清地图	Wayformer、DenseTNT
Argoverse^[73-74]	2019年	车辆	雷达、相机	无信号灯路口	驾驶轨迹、高清地图	mmTransformer、HOME、TrajFormer、VertorNet
INTERACTION^[75]	2019年	车辆、行人	无人机、相机	高速公路、无信号灯路口、有信号灯路口、环岛	驾驶轨迹、高清地图	GOHOME、THOMAS、TNT
HighD^[76]	2018年	车辆	无人机	城市路口	驾驶轨迹、车道	MHA-LSTM
Apolloscape^[77]	2018年	车辆、行人、自行车	雷达、相机	城市高速公路	驾驶轨迹	AI-TP、S2TNet、GRIP++
KITTI^[78]	2013年	行人、自行车	雷达、相机	城市高速公路	图像、点云	DESIRE
NGSIM	2006年	车辆	相机	高速公路路口	驾驶轨迹、车道	Maneuver based LSTM、GRIP++

基于深度学习的自动驾驶多模态轨迹预测方法：现状及展望

Deep learning-based multimodal trajectory prediction methods for autonomous driving: state of the art and perspectives

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 4

参考文献 86

相关文章 10

Metrics

推荐阅读 0

[1]	杜泉成, 王晓, 李灵犀, 宁焕生. 行人轨迹预测方法关键问题研究：现状及展望[J]. 智能科学与技术学报, 2023, 5(2): 143-162.
[2]	黄哲, 王永才, 李德英. 3D目标检测方法研究综述[J]. 智能科学与技术学报, 2023, 5(1): 7-31.
[3]	丛晓峰, 桂杰, 章军. 基于视觉Transformer的多损失融合水下图像增强网络[J]. 智能科学与技术学报, 2022, 4(4): 522-532.
[4]	覃缘琪, 季青原, 葛俊, 戴星原, 陈圆圆, 王晓. 城市交通路网动态短时推理与精准预测研究[J]. 智能科学与技术学报, 2022, 4(3): 380-395.
[5]	郭陈凤, 伍冬睿. 基于典型相关分析的多视图学习方法综述[J]. 智能科学与技术学报, 2022, 4(1): 14-26.
[6]	李琳辉, 周彬, 任威威, 连静. 行人轨迹预测方法综述[J]. 智能科学与技术学报, 2021, 3(4): 399-411.
[7]	刘文, 胡琨林, 李岩, 刘钊. 移动目标轨迹预测方法研究综述[J]. 智能科学与技术学报, 2021, 3(2): 149-160.
[8]	凌文芬, 陈思含, 彭勇, 孔万增. 基于3D分层卷积融合的多模态生理信号情绪识别[J]. 智能科学与技术学报, 2021, 3(1): 76-84.
[9]	吕宝粮, 张亚倩, 郑伟龙. 情感脑机接口研究综述[J]. 智能科学与技术学报, 2021, 3(1): 36-48.
[10]	李姗,李永超,邹颖,杨琳,王茵,姚志军,胡斌. 基于多模态影像下的抑郁症大脑异常[J]. 智能科学与技术学报, 2020, 2(2): 116-125.