基于深度学习的自动驾驶多模态轨迹预测方法：现状及展望

doi:10.11959/j.issn.2096-6652.202317

Abstract

Abstract:

Although deep learning methods have achieved better results than traditional trajectory prediction algorithms, there are still problems such as information loss, interaction and uncertainty difficulties in modelling, and lack of interpretability of predictions when implementing multimodal high-precision prediction for autonomous vehicles in heterogeneous, highly dynamic and complex changing environments.The newly developed Transformer's long-range modelling capability and parallel computing ability make it a great success not only in the field of natural language processing, but also in solving the above problems when extended to the task of multimodal trajectory prediction for autonomous driving.Based on this, the aim of this paper is to provide a comprehensive summary and review of past deep neural network-based approaches, in particular the Transformer-based approach.The advantages of Transformer over traditional sequential network, graphical neural network and generative model were also analyzed and classified in relation to existing challenges, simultaneously.Transformer models can be better applied to multimodal trajectory prediction tasks, and that such models have better generalisation and interpretability.Finally, the future directions of multimodal trajectory prediction were presented.

Key words: Transformer, sequential network, graph neural network, generative model, trajectory prediction, multimodal

CLC Number:

TP29

Jun HUANG, Yonglin TIAN, Xingyuan DAI, et al. Deep learning-based multimodal trajectory prediction methods for autonomous driving: state of the art and perspectives[J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(2): 180-199.

Figures/Tables 4

研究问题	解决办法
时间、空间维度编码问题：	mmTransformer：通过使用3个堆叠Transformer对环境信息、历史轨迹及车辆特征信息分别编码；
时间维度、空间维度信息如何编码？	LAformer：时间密集型多模态轨迹预测模型，通过两阶段方法生成随时间变化、可调整的多模态轨迹并建模
是否会有信息丢失？	交互；
	AgentFormer：时间编码器代替位置编码器，通过时间戳保留时间信息
交互问题（自注意力计算）：	mmTransformer：对交互信息编码，对车辆间依赖关系建模；
智能体间交互如何建模？	AgentFormer：Agent-Aware机制分别计算智能体间和智能体自身内在注意力；
有什么样的依赖关系？	TrajFormer：通过自注意力解决提取局部特征受限问题，更好地提取环境交互信息；
是否有交互信息丢失问题？如何解决？	MPTP：通过补丁合并和移位窗口方法降低自注意力计算复杂性；
自注意力计算效率如何提升？	Gatformer：对智能体-智能体、智能体-环境交互建模
	Spatio-Channel TF：通过基于挤压激励网络的Channel模块捕获相邻Channel之间的相互作用，进而捕获多智
多智能体交互问题：	能体间的交互信息；
多智能体交互信息如何捕获？	GatFormer：采用灵活的图结构，降低计算复杂度，通过交互权重分配提高性能和模型的可解释性，同时保证
	鲁棒性
可解释性问题：
可解释性差的原因？	MPTP：Swim Transformer编码环境信息，GAR编码历史轨迹，生成可解释性高的预测轨迹
如何提高可解释性？
规模化问题：
规模化难的原因？	Wayformer：分析前融合、后融合、分层融合的利弊，为TF如何扩展到大型多维序列提供解方案
编码模块如何优化？

References 86

[87]	ZHANG Y , ZOU Y , TANG J ,et al. A lane-changing prediction method based on temporal convolution network[J]. arXiv preprint, 2020,arXiv:2011.01224.
[88]	GILLES T , SABATINI S , TSISHKOU D ,et al. HOME:heatmap output for future motion estimation[C]// Proceedings of 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). Piscataway:IEEE Press, 2021: 500-507.
[89]	GILLES T , SABATINI S , TSISHKOU D ,et al. GOHOME:graphoriented heatmap output for future motion estimation[C]// Proceedings of 2022 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2022: 9107-9114.
[90]	GILLES T , SABATINI S , TSISHKOU D ,et al. Thomas:trajectory heatmap output with learned multi-agent sampling[J]. arXiv preprint. 2021,arXiv:2110.06607.
[91]	YE M S , CAO T Y , CHEN Q F . TPCN:temporal point cloud networks for motion forecasting[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2021: 11313-11322.
[92]	GIULIARI F , HASAN I , CRISTANI M ,et al. Transformer networks for trajectory forecasting[C]// Proceedings of 2020 25th International Conference on Pattern Recognition (ICPR). Piscataway:IEEE Press, 2020: 10335-10342.
[93]	ALTCHé F , DE LA FORTELLE A . An LSTM network for highway trajectory prediction[C]// Proceedings of 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2017: 353-359.
[94]	ZYNER A , WORRALL S , NEBOT E . Naturalistic driver intention and path prediction using recurrent neural networks[J]. IEEE Transactions on Intelligent Transportation Systems, 2019,21(4): 1584-1594.
[95]	PARK S H , KIM B , KANG C M ,et al. Sequence-to-sequence prediction of vehicle trajectory via LSTM encoder-decoder architecture[C]// Proceedings of 2018 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2018: 1672-1678.
[96]	SONG H , DING W , CHEN Y ,et al. Pip:planning-informed trajectory prediction for autonomous driving[C]// Proceedings of Computer VisionECCV 2020:16th European Conference.[S.l.:s.n.], 2020: 598-614.
[97]	XIN L , WANG P , CHAN C Y ,et al. Intention-aware long horizon trajectory prediction of surrounding vehicles using dual LSTM networks[C]// Proceedings of 2018 21st International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018: 1441-1446.
[98]	SHIRAZI M S , MORRIS B T . Trajectory prediction of vehicles turning at intersections using deep neural networks[J]. Machine Vision and Applications, 2019,30(6): 1097-1109.
[99]	XING Y , LV C , CAO D P . Personalized vehicle trajectory prediction based on joint time-series modeling for connected vehicles[J]. IEEE Transactions on Vehicular Technology, 2020,69(2): 1341-1352.
[100]	ZHANG T , SONG W J , FU M Y ,et al. Vehicle motion prediction at intersections based on the turning intention and prior trajectories model[J]. IEEE/CAA Journal of Automatica Sinica, 2021,8(10): 1657-1666.
[101]	BAE S , SAXENA D , NAKHAEI A ,et al. Cooperation-aware lane change maneuver in dense traffic based on model predictive control with recurrent neural network[C]// Proceedings of 2020 American Control Conference (ACC). Piscataway:IEEE Press, 2020: 1209-1216.
[102]	MO X Y , XING Y , LV C . Interaction-aware trajectory prediction of connected vehicles using CNN-LSTM networks[C]// Proceedings of IECON 2020 the 46th Annual Conference of the IEEE Industrial Electronics Society. Piscataway:IEEE Press, 2020: 5057-5062.
[103]	HUANG Z , WANG J , PI L ,et al. LSTM based trajectory prediction model for cyclist utilizing multiple interactions with environment[J]. Pattern Recognition, 2021,112:107800.
[104]	WANG S B , ZHAO P , YU B ,et al. Vehicle trajectory prediction by knowledge-driven LSTM network in urban environments[J]. Journal of Advanced Transportation, 2020,2020: 1-20.
[105]	JU C , WANG Z , LONG C ,et al. Interaction-aware Kalman neural networks for trajectory prediction[C]// Proceedings of 2020 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2020: 1793-1800.
[106]	LIN L , LI W Z , BI H K ,et al. Vehicle trajectory prediction using LSTMs with spatial-temporal attention mechanisms[J]. IEEE Intelligent Transportation Systems Magazine, 2022,14(2): 197-208.
[107]	SHI X J , GAO Z H , LAUSEN L ,et al. Deep learning for precipitation nowcasting:a benchmark and a new model[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. New York:ACM, 2017: 5622-5632.
[108]	DING W C , CHEN J , SHEN S J . Predicting vehicle behaviors over an extended horizon using behavior interaction network[C]// Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2019: 8634-8640.
[1]	LEVINSON J , ASKELAND J , BECKER J ,et al. Towards fully autonomous driving:systems and algorithms[C]// Proceedings of 2011 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2011: 163-168.
[2]	MAURER M , GERDES J C , LENZ B ,et al. Autonomous driving:technical,legal and social aspects[M]. Springer Berlin: Nature, 2016.
[3]	YURTSEVER E , LAMBERT J , CARBALLO A ,et al. A survey of autonomous driving:common practices and emerging technologies[J]. IEEE Access, 2020,8: 58443-58469.
[4]	张慧, 王坤峰, 王飞跃 . 深度学习在目标视觉检测中的应用进展与展望[J]. 自动化学报, 2017,43(8): 1289-1305.
	ZHANG H , WANG K F , WANG F Y . Advances and perspectives on applications of deep learning in visual object detection[J]. Acta Automatica Sinica, 2017,43(8): 1289-1305.
[5]	田永林, 王雨桐, 王建功 ,等. 视觉 Transformer 研究的关键问题:现状及展望[J]. 自动化学报, 2022,48(4): 957-979.
	TIAN Y L , WANG Y T , WANG J G ,et al. Key problems and progress of vision transformers:the state of the art and prospects[J]. Acta Automatica Sinica, 2022,48(4): 957-979.
[6]	卢经纬, 郭超, 戴星原 ,等. 问答ChatGPT之后:超大预训练模型的机遇和挑战[J]. 自动化学报, 2023,49(4): 705-717.
	LU J W , GUO C , DAI X Y ,et al. The ChatGPT after:opportunities and challenges of very large scale pre-trained models[J]. Acta Automatica Sinica, 2023,49(4): 705-717.
[7]	王飞跃, 缪青海, 张军平 ,等. 探讨AI for Science的影响与意义:现状与展望[J]. 智能科学与技术学报, 2023,5(1): 1-6.
	WANG F Y , MIAO Q H , ZHANG J P ,et al. The DAOs to AI for Science by DeSci:the state of the art and perspective[J]. Chinese Journal of Intelligent Science and Technology, 2023,5(1): 1-6.
[109]	MERCAT J , GILLES T , EL ZOGHBY N ,et al. Multi-head attention for multi-modal joint vehicle motion forecasting[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2020: 9638-9644.
[110]	ZHANG K P , ZHAO L , DONG C X ,et al. AI-TP:attention-based interaction-aware trajectory prediction for autonomous driving[J]. IEEE Transactions on Intelligent Vehicles, 2023,8(1): 73-83.
[111]	TANG C , ZHAN W , TOMIZUKA M . Exploring social posterior collapse in variational autoencoder for interaction modeling[J]. Advances in Neural Information Processing Systems, 2021,34: 8481-8494.
[112]	DEVLIN J , CHANG M W , LEE K ,et al. Bert:pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint, 2018,arXiv:1810.04805.
[113]	LEE N , CHOI W , VERNAZA P ,et al. DESIRE:distant future prediction in dynamic scenes with interacting agents[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017: 2165-2174.
[114]	ZHAO H , GAO J , LAN T ,et al. Tnt:target-driven trajectory prediction[C]// Proceedings of Conference on Robot Learning. New York:PMLR, 2021: 895-904.
[115]	LIU M , CHENG H , CHEN L ,et al. LAformer:trajectory prediction for autonomous driving with lane-aware scene constraints[J]. arXiv preprint, 2023,arXiv:2302.13933.
[116]	YUAN Y , WENG X S , OU Y L ,et al. AgentFormer:agent-aware transformers for socio-temporal multi-agent forecasting[C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2021: 9793-9803.
[117]	PINI S , PERONE C S , AHUJA A ,et al. Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Experts[J]. arXiv preprint, 2022,arXiv:2211.02131.
[118]	CHEN W , WANG F , SUN H . S2tnet:spatio-temporal transformer networks for trajectory prediction in autonomous driving[C]// Proceedings of Asian Conference on Machine Learning. New York:PMLR, 2021: 454-469.
[119]	BHAT M , FRANCIS J , OH J . Trajformer:trajectory prediction with local self-attentive contexts for autonomous driving[J]. arXiv preprint, 2020,arXiv.2011.14910.
[8]	李娟娟, 秦蕊, 丁文文 ,等. 基于Web3的去中心化自治组织与运营新框架[J]. 自动化学报, 2023,49(5): 985-998.
	LI J J , QIN R , DING W W ,et al. A new framework for Web3-powered decentralized autonomous organizations and operations[J]. Acta Automatica Sinica, 2023,49(5): 985-998.
[9]	LECUN Y , BOSER B , DENKER J S ,et al. Backpropagation applied to handwritten zip code recognition[J]. Neural Computation, 1989,1(4): 541-551.
[10]	YE P J , WANG X , ZHENG W B ,et al. Parallel cognition:hybrid intelligence for human-machine interaction and management[J]. Frontiers of Information Technology ＆ Electronic Engineering, 2022,23(12): 1765-1779.
[11]	WANG X , YANG J , HAN J P ,et al. Metaverses and DeMetaverses:from digital twins in CPS to parallel intelligence in CPSS[J]. IEEE Intelligent Systems, 2022,37(4): 97-102.
[12]	XIE G , SHANGGUAN A Q , FEI R ,et al. Motion trajectory prediction based on a CNN-LSTM sequential model[J]. Science China Information Sciences, 2020,63(11): 1-21.
[13]	GIRSHICK R . Fast R-CNN[C]// Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2015: 1440-1448.
[14]	HE K M , GKIOXARI G , DOLLáR P , et al . Mask R-CNN[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2980-2988.
[15]	GRAVES A . Generating sequences with recurrent neural networks[J]. arXiv preprint. 2013,2013,arXiv:1308.0850.
[16]	BAHDANAU D , CHO K , BENGIO Y . Neural machine translation by jointly learning to align and translate[J]. arXiv preprint, 2014,arXiv:1409.0473.
[17]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[120]	ZHAO J , LI X , XUE Q ,et al. Spatial-channel transformer network for trajectory prediction on the traffic scenes[J]. arXiv preprint, 2021,arXiv:2101.11472.
[121]	HU J , SHEN L , SUN G . Squeeze-and-excitation networks[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 7132-7141.
[122]	HUANG Z Y , MO X Y , LV C . Multi-modal motion prediction with transformer-based neural network for autonomous driving[C]// Proceedings of 2022 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2022: 2605-2611.
[123]	ZHANG K P , FENG X L , WU L ,et al. Trajectory prediction for autonomous driving using spatial-temporal graph attention transformer[J]. IEEE Transactions on Intelligent Transportation Systems, 2022,23(11): 22343-22353.
[124]	LIU Z , LIN Y T , CAO Y ,et al. Swin transformer:hierarchical vision transformer using shifted windows[C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2021: 9992-10002.
[125]	NAYAKANTI N , AL-RFOU R , ZHOU A ,et al. Wayformer:motion forecasting via simple ＆ efficient attention networks[J]. arXiv preprint, 2022,arXiv:2207.05844.
[126]	CHAMI I , ABU-EL-HAIJA S , PEROZZI B ,et al. Machine learning on graphs:A model and comprehensive taxonomy[J]. Journal of Machine Learning Research, 2022,23(89): 1-64.
[127]	LIANG M , YANG B , HU R ,et al. Learning Lane Graph Representations for Motion Forecasting[C]// Proceedings of European Conference on Computer Vision. Cham:Springer, 2020: 541-556.
[128]	BEHBAHANI F , SHIARLIS K , CHEN X ,et al. Learning from demonstration in the wild[C]// Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2019: 775-781.
[129]	CASAS S , GULINO C , LIAO R J ,et al. SpAGNN:spatially-aware graph neural networks for relational behavior forecasting from sensor data[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2020: 9491-9497.
[130]	RHINEHART N , MCALLISTER R , KITANI K ,et al. PRECOG:PREdiction conditioned on goals in visual multi-agent settings[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2020: 2821-2830.
[18]	FU R , ZHANG Z , LI L . Using LSTM and GRU neural network methods for traffic flow prediction[C]// Proceedings of 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC). Piscataway:IEEE Press, 2016: 324-328.
[19]	DEY R , SALEM F M . Gate-variants of Gated Recurrent Unit (GRU) neural networks[C]// Proceedings of 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS). Piscataway:IEEE Press, 2017: 1597-1600.
[20]	CHUNG J , GULCEHRE C , CHO K H ,et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv preprint, 2014,arXiv:1412.3555.
[21]	MESSAOUD K , YAHIAOUI I , VERROUST-BLONDET A ,et al. Nonlocal social pooling for vehicle trajectory prediction[C]// Proceedings of 2019 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2019: 975-980.
[22]	MESSAOUD K , YAHIAOUI I , VERROUST-BLONDET A , ,et al. Attention based vehicle trajectory prediction[J]. IEEE Transactions on Intelligent Vehicles, 2021,6(1): 175-185.
[23]	MESSAOUD K , DEO N , TRIVEDI M M ,et al. Trajectory prediction for autonomous driving based on multi-head attention with joint agentmap representation[C]// Proceedings of 2021 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2021: 165-170.
[24]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. New York:ACM, 2017: 6000-6010.
[25]	LIU Y C , ZHANG J H , FANG L J ,et al. Multimodal motion prediction with stacked transformers[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2021: 7573-7582.
[26]	ZHANG K , LI L . Explainable multimodal trajectory prediction using attention models[J]. Transportation Research Part C:Emerging Technologies, 2022,143:103829.
[27]	WU Z H , PAN S R , CHEN F W ,et al. A comprehensive survey on graph neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021,32(1): 4-24.
[28]	SCARSELLI F , GORI M , TSOI A C ,et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2009,20(1): 61-80.
[131]	CASAS S , GULINO C , SUO S ,et al. Implicit Latent Variable Model for Scene-Consistent Motion Forecasting[C]// Proceeding of European Conference on Computer Vision. Cham:Springer, 2020: 624-641.
[132]	LI X , YING X W , CHUAH M C . GRIP:graph-based interactionaware trajectory prediction[C]// Proceedings of 2019 IEEE Intelligent Transportation Systems Conference (ITSC). Piscataway:IEEE Press, 2019: 3960-3966.
[133]	LI X , YING X , CHUAH M C . Grip++:enhanced graph-based interaction-aware trajectory prediction for autonomous driving[J]. arXiv preprint, 2019,arXiv:1907.07792.
[134]	CHANDRA R , GUAN T R , PANUGANTI S ,et al. Forecasting trajectory and behavior of road-agents using spectral clustering in graphLSTMs[J]. IEEE Robotics and Automation Letters, 2020,5(3): 4882-4890.
[135]	GAO J Y , SUN C , ZHAO H ,et al. VectorNet:encoding HD maps and agent dynamics from vectorized representation[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 11522-11530.
[136]	GU J R , SUN C , ZHAO H . DenseTNT:end-to-end trajectory prediction from dense goal sets[C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2021: 15283-15292.
[137]	CHAI Y , SAPP B , BANSAL M ,et al. Multipath:multiple probabilistic anchor trajectory hypotheses for behavior prediction[J]. arXiv preprint, 2019,arXiv:1910.05449.
[138]	HUANG Y F , BI H K , LI Z X ,et al. STGAT:modeling spatialtemporal interactions for human trajectory prediction[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2019: 6271-6280.
[139]	GUPTA A , JOHNSON J , LI F F ,et al. Social GAN:socially acceptable trajectories with generative adversarial networks[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 2255-2264.
[140]	LI J C , MA H B , TOMIZUKA M . Conditional generative neural system for probabilistic trajectory prediction[C]// Proceedings of 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway:IEEE Press, 2019: 6150-6156.
[141]	YANG B , HE C , WANG P ,et al. TPPO:a novel trajectory predictor with pseudo oracle[J]. arXiv preprint, 2020,arXiv:2002.01852.
[29]	DIEHL F , BRUNNER T , LE M T ,et al. Graph neural networks for modelling traffic participant interaction[C]// Proceedings of 2019 IEEE Intelligent Vehicles Symposium. Piscataway:IEEE Press, 2019: 695-701.
[30]	KIPF T N , WELLING M . Semi-supervised classification with graph convolutional networks[J]. arXiv preprint, 2016,arXiv:1609.02907.
[31]	ZHANG S , TONG H H , XU J J ,et al. Graph convolutional networks:a comprehensive review[J]. Computational Social Networks, 2019,6(1): 1-23.
[32]	VELI?KOVI? P , CUCURULL G , CASANOVA A ,et al. Graph attention networks[J]. arXiv preprint, 2017,arXiv:1710.10903.
[33]	JI H Y , WANG X , SHI C ,et al. Heterogeneous graph propagation network[J]. IEEE Transactions on Knowledge and Data Engineering, 2023,35(1): 521-532.
[34]	WANG X , HE X N , CAO Y X ,et al. KGAT:knowledge graph attention network for recommendation[C]// Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery ＆ Data Mining. New York:ACM, 2019: 950-958.
[35]	KIPF T N , WELLING M . Variational graph auto-encoders[J]. arXiv preprint, 2016,arXiv:1611.07308.
[36]	PAN S , HU R , LONG G ,et al. Adversarially regularized graph autoencoder for graph embedding[J]. arXiv preprint, 2018,arXiv:1802.04407.
[37]	ZHOU D W , ZHENG L C , HAN J W ,et al. A data-driven graph generative model for temporal interaction networks[C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery ＆ Data Mining. New York:ACM, 2020: 401-411.
[38]	BONGINI P , BIANCHINI M , SCARSELLI F . Molecular generative graph neural networks for drug discovery[J]. Neurocomputing, 2021,450: 242-252.
[39]	YAN S , XIONG Y , LIN D . Spatial temporal graph convolutional networks for skeleton-based action recognition[J]. arXiv preprint, 2018,arXiv:1801.07455.
[142]	SADEGHIAN A , KOSARAJU V , SADEGHIAN A ,et al. SoPhie:an attentive GAN for predicting paths compliant to social and physical constraints[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019: 1349-1358.
[143]	LIU S H , LIU H B , BI H K ,et al. CoL-GAN:plausible and collisionless trajectory prediction by attention-based GAN[J]. IEEE Access, 2020,8: 101662-101671.
[144]	CHOI S , KWEON N , YANG C ,et al. DSA-GAN:driving style attention generative adversarial network for vehicle trajectory prediction[C]// Proceedings of 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). Piscataway:IEEE Press, 2021: 1515-1520.
[145]	KINGMA D P , WELLING M . Auto-encoding variational bayes[J]. arXiv preprint, 2013,arXiv:1312.6114.
[146]	BALDI P . Autoencoders,unsupervised learning and deep architectures[C]// Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning workshop. New York:ACM, 2011: 37-50.
[147]	GóMEZ-HUéLAMO C , CONDE M V , ORTIZ M ,et al. Exploring attention GAN for vehicle motion prediction[C]// Proceedings of 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2022: 4011-4016.
[148]	WANG W , LIU C , ZHAO D . How much data are enough? A statistical approach with case study on longitudinal driving behavior[J]. IEEE Transactions on Intelligent Vehicles, 2017,2(2): 85-98.
[149]	王晓, 杨林瑶, 胡斌 ,等. 平行推理:一种基于ACP方法的虚实互动的知识协同框架[J]. 智能科学与技术学报, 2023,5(1): 69-82.
	WANG X , YANG L Y , HU B ,et al. Parallel reasoning:a virtual-real interactive knowledge collaboration framework based on ACP approach[J]. Chinese Journal of Intelligent Science and Technology, 2023,5(1): 69-82.
[150]	田永林, 陈苑文, 杨静 ,等. 元宇宙与平行系统:发展现状、对比及展望[J]. 智能科学与技术学报, 2023,5(1): 121-132.
	TIAN Y L , CHEN Y W , YANG J ,et al. Metaverses and parallel systems:the state of the art,comparisons and prospects[J]. Chinese Journal of Intelligent Science and Technology, 2023,5(1): 121-132.
[40]	GUO S N , LIN Y F , FENG N ,et al. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019,33(1): 922-929.
[41]	GOODFELLOW I , POUGET-ABADIE J , MIRZA M ,et al. Generative adversarial networks[J]. Communications of the ACM, 2020,63(11): 139-144.
[42]	CRESWELL A , WHITE T , DUMOULIN V ,et al. Generative adversarial networks:an overview[J]. IEEE Signal Processing Magazine, 2018,35(1): 53-65.
[43]	HO J , ERMON S . Generative adversarial imitation learning[C]// Proceedings of the 30th International Conference on Neural Information Processing Systems. New York:ACM, 2016: 4572-4580.
[44]	SONG J M , REN H Y , SADIGH D ,et al. Multi-agent generative adversarial imitation learning[C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York:ACM, 2018: 7472-7483.
[45]	KOSARAJU V , SADEGHIAN A , MARTíN-MARTíN R , ,et al. SocialBiGAT:multimodal trajectory forecasting using bicycle-GAN and graph attention networks[J]. arXiv preprint, 2019,arXiv:1907.03395.
[46]	ZHAO T , ZHAO R , ESKENAZI M . Learning discourse-level diversity for neural dialog models using conditional variational autoencoders[J]. arXiv preprint, 2017,arXiv:1703.10960.
[47]	BAO J M , CHEN D , WEN F ,et al. CVAE-GAN:fine-grained image generation through asymmetric training[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2764-2773.
[48]	IVANOVIC B , LEUNG K , SCHMERLING E ,et al. Multimodal deep generative models for trajectory prediction:a conditional variational autoencoder approach[J]. IEEE Robotics and Automation Letters, 2021,6(2): 295-302.
[49]	HONG J , SAPP B , PHILBIN J . Rules of the road:predicting driving behavior with a convolutional model of semantic interactions[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019: 8446-8454.
[50]	LEFèVRE S , VASQUEZ D , LAUGIER C . A survey on motion prediction and risk assessment for intelligent vehicles[J]. ROBOMECH Journal, 2014,1(1): 1-14.
[51]	GONZáLEZ D , PéREZ J , MILANéS V ,et al. A review of motion planning techniques for automated vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2016,17(4): 1135-1145.
[52]	LIU J B , MAO X Y , FANG Y Q ,et al. A survey on deep-learning approaches for vehicle trajectory prediction in autonomous driving[C]// Proceedings of 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO). Piscataway:IEEE Press, 2021: 978-985.
[53]	MOZAFFARI S , AL-JARRAH O Y , DIANATI M ,et al. Deep learningbased vehicle behavior prediction for autonomous driving applications:a review[J]. IEEE Transactions on Intelligent Transportation Systems, 2022,23(1): 33-47.
[54]	LEON F , GAVRILESCU M . A review of tracking and trajectory prediction methods for autonomous driving[J]. Mathematics, 2021,9(6): 660.
[55]	GOMES I , WOLF D . A review on intention-aware and interactionaware trajectory prediction for autonomous vehicles[J]. 2022
[56]	SHIRAZI M S , MORRIS B T . Looking at intersections:a survey of intersection monitoring,behavior and safety analysis of recent studies[J]. IEEE Transactions on Intelligent Transportation Systems, 2017,18(1): 4-24.
[57]	HUANG Y J , DU J T , YANG Z R ,et al. A survey on trajectory-prediction methods for autonomous driving[J]. IEEE Transactions on Intelligent Vehicles, 2022,7(3): 652-674.
[58]	KAELBLING L P , LITTMAN M L , MOORE A W . Reinforcement learning:a survey[J]. Journal of Artificial Intelligence Research, 1996,4: 237-285.
[59]	NAYAK A , ESKANDARIAN A , DOERZAPH Z . Uncertainty estimation of pedestrian future trajectory using Bayesian approximation[J]. IEEE Open Journal of Intelligent Transportation Systems, 2022,3: 617-630.
[60]	WANG X , TANG K , DAI X Y ,et al. Safety-balanced driving-style aware trajectory planning in intersection scenarios with uncertain environment[J]. IEEE Transactions on Intelligent Vehicles, 2023,8(4): 2888-2898.
[61]	DEO N , TRIVEDI M M . Multi-modal trajectory prediction of surrounding vehicles with maneuver based LSTMs[C]// Proceedings of 2018 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2018: 1179-1184.
[62]	SHOUNO O . Deep unsupervised learning of a topological map of vehicle maneuvers for characterizing driving styles[C]// Proceedings of 2018 21st International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018: 2917-2922.
[63]	LIPTON Z C . The mythos of model interpretability:in machine learning,the concept of interpretability is both important and slippery[J]. Queue, 2018,16(3): 31-57.
[64]	CHAKRABORTY S , TOMSETT R , RAGHAVENDRA R ,et al. Interpretability of deep learning models:A survey of results[C]// 2017 IEEE SmartWorld,Ubiquitous Intelligence ＆ Computing,Advanced ＆ Trusted Computed,Scalable Computing ＆ Communications,Cloud ＆ Big Data Computing,Internet of People and Smart City Innovation. Piscataway:IEEE Press, 2017: 1-6.
[65]	GUNNING D , STEFIK M , CHOI J ,et al. XAI-explainable artificial intelligence[J]. Science Robotics, 2019,4(37): eaay7120.
[66]	DU M N , LIU N H , HU X . Techniques for interpretable machine learning[J]. Communications of the ACM, 2019,63(1): 68-77.
[67]	HUME D . An inquiry concerning human understanding[M]. New York: Oxford University Press, 2003.
[68]	YUAN Y , KITANI K . Diverse trajectory forecasting with determinantal point processes[J]. arXiv preprint, 2019,arXiv:1907.04967.
[69]	KIM J , CANNY J . Interpretable learning for self-driving cars by visualizing causal attention[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2961-2969.
[70]	CAESAR H , BANKITI V , LANG A H ,et al. nuScenes:a multimodal dataset for autonomous driving[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 11618-11628.
[71]	HOUSTON J , ZUIDHOF G , BERGAMINI L ,et al. One thousand and one hours:Self-driving motion prediction dataset[C]// Proceedings of Conference on Robot Learning. New York:PMLR, 2021: 409-418.
[72]	SUN P , KRETZSCHMAR H , DOTIWALLA X ,et al. Scalability in perception for autonomous driving:waymo open dataset[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 2443-2451.
[73]	CHANG M F , LAMBERT J , SANGKLOY P ,et al. Argoverse:3D tracking and forecasting with rich maps[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019: 8740-8749.
[74]	WILSON B , QI W , AGARWAL T ,et al. Argoverse 2:next generation datasets for self-driving perception and forecasting[J]. arXiv preprint, 2023,arXiv:2301.00493.
[75]	ZHAN W , SUN L , WANG D ,et al. Interaction dataset:an international,adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps[J]. arXiv preprint, 2019,arXiv:1910.03088.
[76]	KRAJEWSKI R , BOCK J , KLOEKER L ,et al. The highD dataset:a drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems[C]// Proceedings of 2018 21st International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018: 2118-2125.
[77]	HUANG X Y , CHENG X J , GENG Q C ,et al. The ApolloScape dataset for autonomous driving[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway:IEEE Press, 2018: 954-960.
[78]	GEIGER A , LENZ P , STILLER C ,et al. Vision meets robotics:the KITTI dataset[J]. International Journal of Robotics Research, 2013,32(11): 1231-1237.
[79]	FANG L J , JIANG Q H , SHI J P ,et al. TPNet:trajectory proposal net work for motion prediction[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 6796-6805.
[80]	NIKHIL N , MORRIS B T . Convolutional Neural Network for Trajectory Prediction[C]// European Conference on Computer Vision. Cham:Springer, 2019: 186-196.
[81]	CHOU F C , LIN T H , CUI H G ,et al. Predicting motion of vulnerable road users using high-definition maps and efficient ConvNets[C]// Proceedings of 2020 IEEE Intelligent Vehicles Symposium (IV). Piscataway:IEEE Press, 2020: 1655-1662.
[82]	CUI H G , RADOSAVLJEVIC V , CHOU F C ,et al. Multimodal trajectory predictions for autonomous driving using deep convolutional networks[C]// Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2019: 2090-2096.
[83]	PHAN-MINHT , GRIGOREE C , BOULTONF A ,et al. CoverNet:multimodal behavior prediction using trajectory sets[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 14062-14071.
[151]	MIAO Q H , LV Y S , HUANG M ,et al. Parallel learning:overview and perspective for computational learning across Syn2Real and Sim2Real[J]. IEEE/CAA Journal of Automatica Sinica, 2023,10(3): 603-631.
[84]	CUI H G , NGUYEN T , CHOU F C ,et al. Deep kinematic models for kinematically feasible vehicle trajectory predictions[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2020: 10563-10569.
[85]	DEO N , TRIVEDI M M . Convolutional social pooling for vehicle trajectory prediction[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway:IEEE Press, 2018: 1468-1476.
[86]	MARZINOTTO A , COLLEDANCHISE M , SMITH C ,et al. Towards a unified behavior trees framework for robot control[C]// Proceedings of 2014 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2014: 5420-5427.

Metrics

Recommended 0

No Suggested Reading articles found!

数据集	年份	交通参与者	传感器	场景	数据内容	基于此数据集的算法
NuScenes^[70]	2020年	车辆、行人、自行车	雷达、相机	城市	驾驶轨迹、高清地图	MHA-JAM
Lyft^[71]	2020年	车辆、行人、自行车	雷达、相机	城市	驾驶轨迹、高清地图	Gatformer
Waymo^[72]	2020年	车辆、行人、自行车	雷达、相机	城市	驾驶轨迹、高清地图	Wayformer、DenseTNT
Argoverse^[73-74]	2019年	车辆	雷达、相机	无信号灯路口	驾驶轨迹、高清地图	mmTransformer、HOME、TrajFormer、VertorNet
INTERACTION^[75]	2019年	车辆、行人	无人机、相机	高速公路、无信号灯路口、有信号灯路口、环岛	驾驶轨迹、高清地图	GOHOME、THOMAS、TNT
HighD^[76]	2018年	车辆	无人机	城市路口	驾驶轨迹、车道	MHA-LSTM
Apolloscape^[77]	2018年	车辆、行人、自行车	雷达、相机	城市高速公路	驾驶轨迹	AI-TP、S2TNet、GRIP++
KITTI^[78]	2013年	行人、自行车	雷达、相机	城市高速公路	图像、点云	DESIRE
NGSIM	2006年	车辆	相机	高速公路路口	驾驶轨迹、车道	Maneuver based LSTM、GRIP++

Deep learning-based multimodal trajectory prediction methods for autonomous driving: state of the art and perspectives

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 4

References 86

Related Articles 8

Metrics

Recommended 0

[1]	Quancheng DU, Xiao WANG, Lingxi LI, Huansheng NING. Key problems and progress of pedestrian trajectory prediction methods: the state of the art and prospects [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(2): 143-162.
[2]	Zhe HUANG, Yongcai WANG, Deying LI. A survey of 3D object detection algorithms [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(1): 7-31.
[3]	Xiaofeng CONG, Jie GUI, Jun ZHANG. Underwater image enhancement network based on visual Transformer with multiple loss functions fusion [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 522-532.
[4]	Yuanqi QIN, Qingyuan JI, Jun GE, Xingyuan DAI, Yuanyuan CHEN, Xiao WANG. Short-term traffic state reasoning and precise prediction in urban networks [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(3): 380-395.
[5]	Linhui LI, Bin ZHOU, Weiwei REN, Jing LIAN. Review of pedestrian trajectory prediction methods [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(4): 399-411.
[6]	Wen LIU, Kunlin HU, Yan LI, Zhao LIU. A review of prediction methods for moving target trajectories [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(2): 149-160.
[7]	Bao-Liang LU, Yaqian ZHANG, Wei-Long ZHENG. A survey of affective brain-computer interface [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 36-48.
[8]	Shan LI,Yongchao LI,Ying ZOU,Lin YANG,Yin WANG,Zhijun YAO,Ubin H. Brain abnormalities in depression based on multimodal imaging [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(2): 116-125.