具身智能驾驶：概念、方法、现状与展望

doi:10.11959/j.issn.2096-6652.202404

Abstract

Abstract:

Embodied intelligence transcends the boundaries of traditional artificial intelligence by emphasizing the importance of interaction between machines and the physical world, facilitating the development of intelligent entities that combine hardware and software to learn from and adapt to their environments, thereby solving real-world problems. Inspired by this philosophy, the concept and framework of embodied intelligent driving are introduced, aiming at integrating the idea of embodied intelligence into the development and application of autonomous vehicles. Through the continuous interaction between physical agents, virtual agents, and real traffic scenes, intelligent driving systems can achieve precise perception, efficient execution, and autonomous evolution in complex scenes, enhancing the long-term adaptability of autonomous vehicles in open traffic environments. Based on the embodied intelligent driving framework, the relevant technologies are summarize and the development status and existing problems of such technologies are analyzed. Furthermore, thoughts and prospects in this field are demonstrated by exploring the important roles and application potential of virtual-real interactive data intelligence, foundation models and foundation intelligence, continuous learning and parallel intelligence. This paper is expected to promote innovative research and the application on embodied intelligent driving in a wider range of scenarios, and provide new ideas and solutions for the development of mobile robot systems such as intelligent vehicles.

Key words: embodied intelligence, autonomous driving, embodied perception, embodied execution, embodied evolution

CLC Number:

TP39

Tianyu SHEN, Zhiwei LI, Lili FAN, et al. Embodied intelligent driving: concept, methods, the state of the art and beyond[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(1): 17-32.

Figures/Tables 9

References 87

1	WANG Y T, WANG X, WANG X X, et al. The ChatGPT after: building knowledge factories for knowledge workers with knowledge automation[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(11): 2041-2044.
2	TIAN Y L, LI X, ZHANG H, et al. VistaGPT: generative parallel transformers for vehicles with intelligent systems for transport automation[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(9): 4198-4207.
3	WANG F Y, MIAO Q H, LI X, et al. What does ChatGPT say: the DAO from algorithmic intelligence to linguistic intelligence[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(3): 575-579.
4	MEMARIAN B, DOLECK T. Embodied AI in education: a review on the body, environment, and mind[J]. Education and Information Technologies, 2024, 29(1): 895-916.
5	刘华平, 郭迪, 孙富春, 等. 基于形态的具身智能研究: 历史回顾与前沿进展[J]. 自动化学报, 2023, 49(6): 1131-1154.
	LIU H P, GUO D, SUN F C, et al. Morphology-based embodied intelligence: historical retrospect and research progress[J]. Acta Automatica Sinica, 2023, 49(6): 1131-1154.
6	GUPTA A, SAVARESE S, GANGULI S, et al. Embodied intelligence via learning and evolution[J]. Nature Communications, 2021, 12(1): 5721.
7	HOWARD D, EIBEN A E, KENNEDY D F, et al. Evolving embodied intelligence from materials to machines[J]. Nature Machine Intelligence, 2019, 1(1): 12-19.
8	JIN D D, ZHANG L. Embodied intelligence weaves a better future[J]. Nature Machine Intelligence, 2020, 2: 663-664.
9	AI Y F, LIU Y H, GAO Y, et al. PMWorld: a parallel testing platform for autonomous driving in mines[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(1): 1402-1411.
10	YANG J J, HUANG Q K, GE S R, et al. On intelligent mining with parallel intelligence[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(10): 4296-4300.
11	CHEN L, XIE J K, ZHANG X T, et al. Mining 5.0: concept and framework for intelligent mining systems in CPSS[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(6): 3533-3536.
12	GU Y X, WANG Q L, QIN X L. Real-time streaming perception system for autonomous driving[C]//Proceedings of the 2021 China Automation Congress (CAC). Piscataway: IEEE Press, 2021: 5239-5244.
13	DENG Z Y, SHI Y J, SHEN W M. V2X-lead: LiDAR-based end-to-end autonomous driving with vehicle-to-everything communication integration[C]//Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway: IEEE Press, 2023: 7471-7478.
14	THRUN S. Toward robotic cars[J]. Communications of the ACM, 2010, 53(4): 99-106.
15	SCHWARTZ R, Safe integration of autonomous vehicles in urban tra?ic[J]. Nature Machine Intelligence, 2020.
16	LIU Y F, LIU Y Z, SHEN C. Combining minds and machines: investigating the fusion of cognitive architectures and generative models for general embodied intelligence[J]. Proceedings of the AAAI Symposium Series, 2024, 2(1): 307-314.
17	TURING A M. Computing machinery and intelligence[M]//Parsing the Turing Test. Dordrecht: Springer Netherlands, 2007: 23-65.
18	On-Road Automated Driving (ORAD) Committee. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles[M]. [S.l.]: SAE International, 2021.
19	沈宇, 王晓, 韩双双, 等. 代理技术Agent在智能车辆与驾驶中的应用现状[J]. 指挥与控制学报, 2019, 5(2): 87-98.
	SHEN Y, WANG X, HAN S S, et al. Agent-based technology in intelligent vehicles and driving: state-of-the-art and prospect[J]. Journal of Command and Control, 2019, 5(2): 87-98.
20	WANG J G, WANG X, SHEN T Y, et al. Parallel vision for long-tail regularization: initial results from IVFC autonomous driving testing[J]. IEEE Transactions on Intelligent Vehicles, 2022, 7(2): 286-299.
21	BAI X Y, HU Z Y, ZHU X G, et al. TransFusion: robust LiDAR-camera fusion for 3D object detection with transformers[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2022: 1080-1089.
22	RONG Y, WEI X Y, LIN T W, et al. DynStatF: an efficient feature fusion strategy for LiDAR 3D object detection[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE Press, 2023: 3238-3247.
23	TIAN Y L, ZHANG X J, WANG X, et al. ACF-net: asymmetric cascade fusion for 3D detection with LiDAR point clouds and images[J]. IEEE Transactions on Intelligent Vehicles, 2023: 1-12.
24	SINDAGI V A, ZHOU Y, TUZEL O. MVX-net: multimodal VoxelNet for 3D object detection[C]//Proceedings of the 2019 International Conference on Robotics and Automation (ICRA). Piscataway: IEEE Press, 2019: 7276-7282.
25	孙水发, 汤永恒, 王奔, 等. 动态场景的三维重建研究综述[J]. 计算机科学与探索, 2024, 18(4): 831-860.
	SUN S F, TANG Y H, WANG B, et al. Review of research on 3D reconstruction of dynamic scenes[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(4): 831-860.
26	GRETTON A, SMOLA A, HUANG J Y, et al. Covariate shift and local learning by distribution matching[M]. Cambridge: MIT Press, 2009: 131-160.
27	GANIN Y, LEMPITSKY V. Unsupervised domain adaptation by backpropagation[EB]. arXiv preprint, 2014, arXiv: 1409.7495.
28	BERTHELOT D, ROELOFS R, SOHN K, et al. AdaMatch: a unified approach to semi-supervised learning and domain adaptation[EB]. arXiv preprint, 2021, arXiv: 2106.04732.
29	MENG D J, XIAO W, ZHANG L J, et al. Vehicle trajectory prediction based predictive collision risk assessment for autonomous driving in highway scenarios[J]. arXiv preprint, 2023, arXiv: 2304.05610.
30	WANG X W, ALONSO-MORA J, WANG M. Probabilistic risk metric for highway driving leveraging multi-modal trajectory predictions[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(10): 19399-19412.
31	FANG L J, JIANG Q H, SHI J P, et al. TPNet: trajectory proposal network for motion prediction[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2020: 6796-6805.
32	景荣荣, 吴兰, 张坤鹏. 基于Transformer的自动驾驶交互感知轨迹预测[J]. 科学技术与工程, 2023, 23(26): 11414-11423.
	JING R R, WU L, ZHANG K P. Transformer-based interaction-aware trajectory prediction for autonomous driving[J]. Science Technology and Engineering, 2023, 23(26): 11414-11423.
33	李升波, 占国建, 蒋宇轩, 等. 类脑学习型自动驾驶决控系统的关键技术[J]. 汽车工程, 2023, 45(9): 1499-1515.
	LI S B, ZHAN G J, JIANG Y X, et al. Key technologies of brain-inspired decision and control intelligence for autonomous driving systems[J]. Automotive Engineering, 2023, 45(9): 1499-1515.
34	PENG X Y, CHANG C, WANG F Y, et al. Robust multitask learning with sample gradient similarity[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2024, 54(1): 497-506.
35	曹佳钰, 冷甦鹏, 张科. 面向自动驾驶应用的车联多智能体信息融合协同决策机制研究[J]. 物联网学报, 2020, 4(3): 69-77.
	CAO J Y, LENG S P, ZHANG K. Multi-agent driven collaborative decision mechanism of\r information fusion for autonomous driving vehicles[J]. Chinese Journal on Internet of Things, 2020, 4(3): 69-77.
36	WANG L Y, ZHANG X X, SU H, et al. A comprehensive survey of continual learning: theory, method and application[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, accepted.
37	RANNEN A, ALJUNDI R, BLASCHKO M B, et al. Encoder based lifelong learning[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway: IEEE Press, 2017: 1329-1337.
38	LIU X L, WU C S, MENTA M, et al. Generative feature replay for class-incremental learning[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE Press, 2020: 915-924.
39	JOSEPH K J, BALASUBRAMANIAN V N. Meta-consolidation for continual learning[EB]. arXiv preprint, 2020, arXiv: 2010.00352.
40	WANG R Q, BAO Y X, ZHANG B C, et al. Anti-retroactive interference for lifelong learning[M]//Lecture Notes in Computer Science. Cham: Springer Nature, 2022: 163-178.
41	SHI Y J, ZHOU K Q, LIANG J, et al. Mimicking the oracle: an initial phase decorrelation approach for class incremental learning[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2022: 16701-16710.
42	ZHANG G W, WANG L Y, KANG G L, et al. SLCA: slow learner with classifier alignment for continual learning on a pre-trained model[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE Press, 2023: 19091-19101.
43	KIM G, XIAO C N, KONISHI T, et al. A theoretical study on solving continual learning[EB]. arXiv preprint, 2022, arXiv: 2211.02633.
44	MCCLOSKEY M, COHEN N J. Catastrophic interference in connectionist networks: the sequential learning problem[M]//Psychology of Learning and Motivation. Amsterdam: Elsevier, 1989: 109-165.
45	MASANA M, LIU X L, TWARDOWSKI B, et al. Class-incremental learning: survey and performance evaluation on image classification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(5): 5513-5533.
46	GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2012: 3354-3361.
47	CAESAR H, BANKITI V, LANG A H, et al. NuScenes: a multimodal dataset for autonomous driving[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2020: 11618-11628.
48	SUN P, KRETZSCHMAR H, DOTIWALLA X, et al. Scalability in perception for autonomous driving: waymo open dataset[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2020: 2443-2451.
49	CHANG M F, LAMBERT J, SANGKLOY P, et al. Argoverse: 3D tracking and forecasting with rich maps[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2019: 8740-8749.
50	HOUSTON J, ZUIDHOF G, BERGAMINI L, et al. One thousand and one hours: self-driving motion prediction dataset[EB]. arXiv preprint, 2020, arXiv: 2006.14480.
51	FENG D P, QI Y H, ZHONG S P, et al. S3E: a large-scale multimodal dataset for collaborative SLAM[EB]. arXiv preprint, 2022, arXiv: 2210.13723.
52	ALIBEIGI M, LJUNGBERGH W, TONDERSKI A, et al. Zenseact open dataset: a large-scale and diverse multimodal dataset for autonomous driving[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE Press, 2023: 20121-20131.
53	DOSOVITSKIY A, ROS G, CODEVILLA F, et al. CARLA: an open urban driving simulator[EB]. arXiv preprint, 2017, arXiv: 1711.03938.
54	HUANG X Y, CHENG X J, GENG Q C, et al. The ApolloScape dataset for autonomous driving[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE Press, 2018: 1067-10676.
55	GEYER J, KASSAHUN Y, MAHMUDI M, et al. A2D2: audi autonomous driving dataset[EB]. arXiv preprint, 2020: arXiv: 2004.06320.
56	CHEN M, LIU P F, ZHAO H C. LiDAR-camera fusion: dual transformer enhancement for 3D object detection[J]. Engineering Applications of Artificial Intelligence, 2023, 120: 105815.
57	AN P, LIANG J X, HONG X, et al. Leveraging self-paced semi-supervised learning with prior knowledge for 3D object detection on a LiDAR-camera system[J]. Remote Sensing, 2023, 15(3): 627.
58	ZHANG L, LI X, TANG K C, et al. FS-net: LiDAR-camera fusion with matched scale for 3D object detection in autonomous driving[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(11): 12154-12165.
59	LI Y W, YU A W, MENG T J, et al. DeepFusion: lidar-camera deep fusion for multi-modal 3D object detection[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2022: 17161-17170.
60	KOH J, KIM J, YOO J H, et al. Joint 3D object detection and tracking using spatio-temporal representation of camera image and LiDAR point clouds[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(1): 1210-1218.
61	HUANG K M, HAO Q. Joint multi-object detection and tracking with camera-LiDAR fusion for autonomous driving[C]//Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). New York: ACM, 2021: 6983-6989.
62	ZHANG K P, LIU Y H, MEI F, et al. Boost correlation features with 3D-MiIoU-based camera-LiDAR fusion for MODT in autonomous driving[J]. Remote Sensing, 2023, 15(4): 874.
63	WANG X Y, FU C Y, LI Z K, et al. DeepFusionMOT: a 3D multi-object tracking framework based on camera-LiDAR fusion with deep association[J]. IEEE Robotics and Automation Letters, 2022, 7(3): 8260-8267.
64	ZHU Z M, NIE J H, WU H, et al. MSA-MOT: multi-stage association for 3D multimodality multi-object tracking[J]. Sensors, 2022, 22(22): 8650.
65	WANG X Y, FU C Y, HE J W, et al. StrongFusionMOT: a multi-object tracking method based on LiDAR-camera fusion[J]. IEEE Sensors Journal, 2023, 23(11): 11241-11252.
66	BüCHNER M, VALADA A. 3D multi-object tracking using graph neural networks with cross-edge modality attention[J]. IEEE Robotics and Automation Letters, 2022, 7(4): 9707-9714.
67	XIE B Q, YANG Z M, YANG L, et al. AMMF: attention-based multi-phase multi-task fusion for small contour object 3D detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(2): 1692-1701.
68	CHIU H K, LI J, AMBRU? R, et al. Probabilistic 3D multi-modal, multi-object tracking for autonomous driving[C]//Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA). Piscataway: IEEE Press, 2021: 14227-14233.
69	SHAHVERDY M, FATHY M, BERANGI R, et al. Driver behavior detection and classification using deep convolutional neural networks[J]. Expert Systems with Applications, 2020, 149: 113240.
70	JEONG D, BAEK M, LEE S S. Long-term prediction of vehicle trajectory based on a deep neural network[C]//Proceedings of the 2017 International Conference on Information and Communication Technology Convergence (ICTC). Piscataway: IEEE Press, 2017: 725-727.
71	YANG L, LEI W M, ZHANG W, et al. Dual-flow network with attention for autonomous driving[J]. Frontiers in Neurorobotics, 2023, 16: 978225.
72	KIM C J, LEE M J, HWANG K H, et al. End-to-end deep learning-based autonomous driving control for high-speed environment[J]. The Journal of Supercomputing, 2021: 1961-1982.
73	KATUWANDENIYA K, MIRO J V, DANTANARAYANA L. End-to-end joint intention estimation for shared control personal mobility navigation[C]//Proceedings of the 2020 16th International Conference on Control, Automation, Robotics and Vision (ICARCV). Piscataway: IEEE Press, 2020: 1-6.
74	LI X, WANG Y T, WANG K F, et al. The ParallelEye-CS dataset: constructing artificial scenes for evaluating the visual intelligence of intelligent vehicles[C]//Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV). Piscataway: IEEE Press, 2018: 37-42.
75	LI X, WANG K F, GU X F, et al. ParallelEye pipeline: an effective method to synthesize images for improving the visual intelligence of intelligent vehicles[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(9): 5545-5556.
76	MIAO Q H, LV Y S, HUANG M, et al. Parallel learning: overview and perspective for computational learning across Syn2Real and Sim2Real[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(3): 603-631.
77	LI L, LIN Y L, ZHENG N N, et al. Parallel learning: a perspective and a framework[J]. IEEE/CAA Journal of Automatica Sinica, 2017, 4(3): 389-395.
78	SHEN Y, LIU Y H, TIAN Y L, et al. Parallel sensing in metaverses: virtual-real interactive smart systems for "6S" sensing[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(12): 2047-2054.
79	王坤峰, 苟超, 王飞跃. 平行视觉: 基于ACP的智能视觉计算方法[J]. 自动化学报, 2016, 42(10): 1490-1500.
	WANG K F, GOU C, WANG F Y. Parallel vision: an ACP-based approach to intelligent vision computing[J]. Acta Automatica Sinica, 2016, 42(10): 1490-1500.
80	李轩, 王飞跃. 面向智能驾驶的平行视觉感知: 基本概念、框架与应用[J]. 中国图象图形学报, 2021, 26(1): 67-81.
	LI X, WANG F Y. Parallel visual perception for intelligent driving: basic concept, framework and application[J]. Journal of Image and Graphics, 2021, 26(1): 67-81.
81	LIU Y H, SHEN L, FAN L L, et al, Parallel radars: from digital twins to digital intelligence for smart radar systems[J]. Sensors, 2022, 22(24): 9930.
82	LI X, TIAN Y L, YE P J, et al. A novel scenarios engineering methodology for foundation models in metaverse[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(4): 2148-2159.
83	WANG F Y. New control paradigm for industry 5.0: from big models to foundation control and management[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(8): 1643-1646.
84	LI J J, LIANG X L, QIN R, et al. From DAO to TAO: finding the essence of decentralization[C]//Proceedings of the 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Piscataway: IEEE Press, 2023: 4283-4288.
85	LI J J, WANG F Y. The TAO of blockchain intelligence for intelligent web 3.0[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(12): 2183-2186.
86	WANG Z R, LV C, WANG F Y. A new era of intelligent vehicles and intelligent transportation systems: digital twins and parallel intelligence[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(4): 2619-2627.
87	王飞跃, 王雨桐. 数字科学家与平行科学:AI4S和S4AI的本源与目标[J]. 中国科学院院刊, 2024, 39(1): 27-33.
	WANG F Y, WANG Y T. Digital scientists and parallel sciences: the origin and goal of AI for Science and Science for AI[J]. Bulletin of Chinese Academy of Sciences, 2024, 39(1): 27-33.

Metrics

Recommended 0

No Suggested Reading articles found!

数据集	年份	传感器类型
数据集	年份	前视相机	环视相机	激光雷达	雷达	车身信号	全球定位系统	惯性传感器	立体相机	高精度地图	高分辨率相机
KITTI^[46]	2012	✓		✓			✓	✓
NuScenes^[47]	2019	✓	✓	✓	✓		✓			✓
Argoverse^[49]	2019		✓	✓						✓
Waymo^[48]	2019		✓	✓
ONCE^[50]	2021		✓	✓
S3E^[51]	2022			✓				✓	✓
ZOD^[52]	2023			✓				✓			✓

数据集	年份	传感器类型
数据集	年份	前视相机	环视相机	激光雷达	雷达	车身信号	全球定位系统	惯性传感器	立体相机	高精度地图	高分辨率相机
CARLA^[53]	2017	✓		✓			✓	✓		✓
ApolloScape^[54]	2018	✓		✓			✓	✓		✓	✓
A2D2^[55]	2020	✓		✓		✓	✓
NuScenes^[47]	2019	✓	✓	✓	✓		✓	✓		✓

[1]	Quancheng DU, Xiao WANG, Lingxi LI, Huansheng NING. Key problems and progress of pedestrian trajectory prediction methods: the state of the art and prospects [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(2): 143-162.
[2]	Zhe HUANG, Yongcai WANG, Deying LI. A survey of 3D object detection algorithms [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(1): 7-31.

Embodied intelligent driving: concept, methods, the state of the art and beyond

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 87

Related Articles 2

Metrics

Recommended 0