边缘智能下基于强化学习的车联网路由协议

doi:10.11959/j.issn.1000-436x.2023187

通信学报 ›› 2023, Vol. 44 ›› Issue (11): 110-119.doi: 10.11959/j.issn.1000-436x.2023187

• 专题：复杂环境下分布式边缘智能 • 上一篇

边缘智能下基于强化学习的车联网路由协议

刘冰艺¹^,², 刘煜昊¹, 韩玮祯¹^,³, 夏振厂¹, 吴黎兵⁴, 熊盛武¹

¹ 武汉理工大学计算机科学与人工智能学院，湖北武汉 430070
² 武汉理工大学三亚科教创新园，海南三亚 572000
³ 武汉理工大学重庆研究院，重庆 401135
⁴ 武汉大学国家网络安全学院，湖北武汉 430070

修回日期:2023-07-27 出版日期:2023-11-01 发布日期:2023-11-01
作者简介:刘冰艺（1990− ），男，湖北武汉人，博士，武汉理工大学副教授、博士生导师，主要研究方向为无线网络、车载自组织网络、物联网等
刘煜昊（1999− ），男，湖北潜江人，武汉理工大学硕士生，主要研究方向为车载自组织网络、强化学习
韩玮祯（1996− ），男，江苏常州人，武汉理工大学博士生，主要研究方向为车载自组织网络、强化学习
夏振厂（1987− ），男，河南周口人，博士，武汉理工大学讲师、硕士生导师，主要研究方向为车联网、网络拥塞控制、强化学习等
吴黎兵（1972− ），男，湖北武汉人，博士，武汉大学教授、博士生导师，主要研究方向为分布式计算、网络安全、无线感知网络等
熊盛武（1966− ），男，湖北武汉人，博士，武汉理工大学教授、博士生导师，主要研究方向为智能网联汽车、机器学习、数据挖掘等
基金资助:
国家自然科学基金资助项目(62272357);国家自然科学基金资助项目(62176194);国家自然科学基金资助项目(62202348);国家自然科学基金资助项目(U20A20177);国家自然科学基金资助项目(62272348);湖北省重点研发计划基金资助项目(2022BAA052);海南省重点研发计划基金资助项目(ZDYF2021GXJS014);重庆市科学基金资助项目(cstc2021jcyj-msxm4264);武汉理工大学重庆研究院研究基金资助项目(ZD2021-04);武汉理工大学重庆研究院研究基金资助项目(ZL2021-05)

Edge intelligence-assisted routing protocol for Internet of vehicles via reinforcement learning

Bingyi LIU¹^,², Yuhao LIU¹, Weizhen HAN¹^,³, Zhenchang XIA¹, Libing WU⁴, Shengwu XIONG¹

¹ School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan 430070, China
² Sanya Science and Education Innovation Park, Wuhan University of Technology, Sanya 572000, China
³ Chongqing Research Institute, Wuhan University of Technology, Chongqing 401135, China
⁴ School of Cyber Science and Engineering, Wuhan University, Wuhan 430070, China

Revised:2023-07-27 Online:2023-11-01 Published:2023-11-01
Supported by:
The National Natural Science Foundation of China(62272357);The National Natural Science Foundation of China(62176194);The National Natural Science Foundation of China(62202348);The National Natural Science Foundation of China(U20A20177);The National Natural Science Foundation of China(62272348);The Key Research and Development Program of Hubei Province(2022BAA052);The Key Research and Development Program of Hainan Province(ZDYF2021GXJS014);The Science Foundation of Chongqing(cstc2021jcyj-msxm4264);The Research Project of Chongqing Research Institute of Wuhan University of Technology(ZD2021-04);The Research Project of Chongqing Research Institute of Wuhan University of Technology(ZL2021-05)

摘要/Abstract

摘要：

为实现复杂城市车联网环境下高可靠、自适应的数据包路由协议，提出一个端-边-云边缘智能架构，该架构包括终端用户层、边缘协作层和云计算层。在所提边缘智能架构的基础上，设计了一个基于多智能体强化学习的数据包路由协议。实验结果表明，相比于现有的紧急消息传输机制、基于交叉路口雾节点的分布式路由协议和基于双深度Q网络的路由协议，所提协议在消息传输时延和接收率方面分别取得29.65%～44.06%和17.08%～25.38%的优化。

关键词: 边缘智能, 车联网, 多智能体强化学习, 数据包路由

Abstract:

To achieve a highly reliable and adaptive packet routing protocol in a complex urban Internet of vehicles, an end-edge-cloud edge intelligence architecture was proposed which consisted of an end user layer, an edge collaboration layer, and a cloud computing layer.Based on the proposed edge intelligence architecture, an packet routing protocol based on multi-intelligent reinforcement learning technologies was designed.The experimental results show that the proposed protocol could significantly improve the transmission delay and the packet reception rate in the interval of 29.65%～44.06% and 17.08%～25.38% compared to the state-of-the-art transmission mechanism for emergency data (TMED), intersection fog-based distributed routing protocol (IDR), and double deep Q-net based routing protocol (DRP).

Key words: edge intelligence, Internet of vehicles, multi-agent reinforcement learning, packet routing

中图分类号:

U495

刘冰艺, 刘煜昊, 韩玮祯, 夏振厂, 吴黎兵, 熊盛武. 边缘智能下基于强化学习的车联网路由协议[J]. 通信学报, 2023, 44(11): 110-119.

Bingyi LIU, Yuhao LIU, Weizhen HAN, Zhenchang XIA, Libing WU, Shengwu XIONG. Edge intelligence-assisted routing protocol for Internet of vehicles via reinforcement learning[J]. Journal on Communications, 2023, 44(11): 110-119.

图/表 10

图1

图2

图3

图4

表1

图5

图6

表2

图7

图8

参考文献 24

[1]	YU Y F . Mobile edge computing towards 5G:vision,recent progress,and open challenges[J]. China Communications, 2016,13: 89-99.
[2]	LIU B Y , HAN W Z , JIANG W ,et al. A novel V2V-based temporary warning network for safety message dissemination in urban environments[J]. IEEE Internet of Things Journal, 2022,9(24): 25136-25149.
[3]	LIU B Y , HAN W Z , WANG E S ,et al. Multi-agent attention double actor-critic framework for intelligent traffic light control in urban scenarios with hybrid traffic[J]. IEEE Transactions on Mobile Computing, 2023,PP(99): 1-13.
[4]	YAO L , WANG J , WANG X ,et al. V2X routing in a VANET based on the hidden Markov model[J]. IEEE Transactions on Intelligent Transportation Systems, 2018,19(3): 889-899.
[5]	TAKAHASHI S , YOSHIDA M , RAMONET A G ,et al. Shadowing-fading-based intersection geographic opportunistic routing protocol for urban VANETs[C]// Proceedings of 24th International Conference on Advanced Communication Technology (ICACT). Piscataway:IEEE Press, 2022: 179-184.
[6]	GOUDARZI F , ASGARI H , AL-RAWESHIDY H S . Traffic-aware VANET routing for city environments—a protocol based on ant colony optimization[J]. IEEE Systems Journal, 2018,13(1): 571-581.
[7]	QIU T , WANG X , CHEN C ,et al. TMED:a spider-web-like transmission mechanism for emergency data in vehicular ad hoc networks[J]. IEEE Transactions on Vehicular Technology, 2018,67(9): 8682-8694.
[8]	LIU B Y , FANG Z P , WANG W ,et al. A region-based collaborative management scheme for dynamic clustering in green VANET[J]. IEEE Transactions on Green Communications and Networking, 2022,6(3): 1276-1287.
[9]	SUN G , ZHANG Y J , YU H F ,et al. Intersection fog-based distributed routing for V2V communication in urban vehicular ad hoc networks[J]. IEEE Transactions on Intelligent Transportation Systems, 2020,21(6): 2409-2426.
[10]	LIN D , KANG J , SQUICCIARINI A ,et al. MoZo:a moving zone based routing protocol using pure V2V communication in VANETs[J]. IEEE Transactions on Mobile Computing, 2017,16(5): 1357-1370.
[11]	BHOI S K , SAHU P K , SINGH M ,et al. Local traffic aware unicast routing scheme for connected car system[J]. IEEE Transactions on Intelligent Transportation Systems, 2020,21(6): 2360-2375.
[12]	SUTTON R S , BARTO A G . Introduction to reinforcement learning[M]. Cambridge: MIT Press, 1998.
[13]	LIU B Y , DENG D X , RAO W B ,et al. CPA-MAC:a collision prediction and avoidance MAC for safety message dissemination in MEC-assisted VANETs[J]. IEEE Transactions on Network Science and Engineering, 2022,9(2): 783-794.
[14]	LUO L , SHENG L , YU H F ,et al. Intersection-based V2X routing via reinforcement learning in vehicular ad hoc networks[J]. IEEE Transactions on Intelligent Transportation Systems, 2022,23(6): 5446-5459.
[15]	WU J Q , FANG M , LI H K ,et al. RSU-assisted traffic-aware routing based on reinforcement learning for urban VANETs[J]. IEEE Access, 2020,8: 5733-5748.
[16]	LI F , SONG X Y , CHEN H J ,et al. Hierarchical routing for vehicular ad hoc networks via reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2019,68(2): 1852-1865.
[17]	ZHAO N , WU H , YU F R ,et al. Deep-reinforcement-learning-based latency minimization in edge intelligence over vehicular networks[J]. IEEE Internet of Things Journal, 2022,9(2): 1300-1312.
[18]	ZHOU Z , CHEN X , LI E ,et al. Edge intelligence:paving the last mile of artificial intelligence with edge computing[J]. Proceedings of the IEEE, 2019,107(8): 1738-1762.
[19]	SHAO X , HASEGAWA G , DONG M X ,et al. An online orchestration mechanism for general-purpose edge computing[J]. IEEE Transactions on Services Computing, 2022,16(2): 927-940.
[20]	QIAO G H , LENG S P , ZHANG K ,et al. Collaborative task offloading in vehicular edge multi-access networks[J]. IEEE Communications Magazine, 2018,56(8): 48-54.
[21]	HSU K C , REN A Z , NGUYEN D P ,et al. Sim-to-Lab-to-Real:safe reinforcement learning with shielding and generalization guarantees[J]. Artificial Intelligence, 2023,314:103811.
[22]	YU C , VELU A , VINITSKY E ,et al. The surprising effectiveness of PPO in cooperative multi-agent games[J]. Advances in Neural Information Processing Systems, 2022,35: 24611-24624.
[23]	SCHULMAN J , MORITZ P , LEVINE S ,et al. High-dimensional continuous control using generalized advantage estimation[J]. arXiv Preprint,arXiv:1506.02438, 2015.
[24]	SOMMER C , GERMAN R , DRESSLER F . Bidirectionally coupled network and road traffic simulation for improved IVC analysis[J]. IEEE Transactions on Mobile Computing, 2011,10(1): 3-15.

参数	值
虚拟地图大小	5 000 m×5 000 m
MAC协议	IEEE 802.11p
车道数量	双向四车道
道路长度/m	1 000
最大车速/(m·s^-1)	15
交叉路口数量	25
RSU传输功率/ mW	15
车辆传输功率/ mW	6
智能体数量/个	10
每轮训练步长/步	200
最小信噪比阈值/ dBm	-110

边缘智能下基于强化学习的车联网路由协议

Edge intelligence-assisted routing protocol for Internet of vehicles via reinforcement learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 24

相关文章 15

Metrics

推荐阅读 0

传输距离/m	MARP	TMED	IDR	DRP
3 000	16	18	18	17
4 000	20	22	22	21
5 000	26	29	28	28
6 000	30	33	32	31

[1]	陈发堂, 张若凡. 可重构智能反射面辅助的车联网资源分配算法研究[J]. 通信学报, 2023, 44(9): 70-78.
[2]	燕锋, 林晓薇, 李正浩, 徐霞, 夏玮玮, 沈连丰. 智能电网中基于多智能体强化学习的频谱分配算法[J]. 通信学报, 2023, 44(9): 12-24.
[3]	张海波, 王大斌, 王汝言, 王冬宇. 车联网中基于模糊评价密度聚类的共谋节点检测方法[J]. 通信学报, 2023, 44(7): 114-123.
[4]	李云, 高倩, 姚枝秀, 夏士超, 梁吉申. 移动边缘计算中智能服务编排和算网资源分配联合优化方法[J]. 通信学报, 2023, 44(7): 51-63.
[5]	张海波, 兰凯, 陈舟, 王汝言, 邹灿, 王明月. 车联网中基于环的匿名高效批量认证与组密钥协商协议[J]. 通信学报, 2023, 44(6): 103-116.
[6]	刘雪娇, 钟强, 夏莹杰. 基于双层分片区块链的车联网跨信任域高效认证方案[J]. 通信学报, 2023, 44(5): 213-223.
[7]	张海波, 曹钰坤, 刘开健, 王汝言. 车联网中基于区块链的分布式信任管理方案[J]. 通信学报, 2023, 44(5): 148-157.
[8]	刘雪娇, 曹天聪, 夏莹杰. 区块链架构下高效的车联网跨域数据安全共享研究[J]. 通信学报, 2023, 44(3): 186-197.
[9]	张雷, 王玉, 田建杰, 张琳, 章天骄. 基于IRS辅助的MIMO车联网系统联合波束成形设计[J]. 通信学报, 2023, 44(2): 59-69.
[10]	顾晓丹, 吴文甲, 凌振. 用户密集环境下基于边缘智能的直播视频传输优化机制[J]. 通信学报, 2023, 44(11): 55-66.
[11]	陈发堂, 刘小玲, 王丹, 张若凡. RIS辅助太赫兹频段车载网络容量优化[J]. 通信学报, 2023, 44(10): 103-111.
[12]	王平, 杨志伟, 李贺举. 智能反射面赋能的联邦边缘学习及其在车联网中的应用[J]. 通信学报, 2023, 44(10): 46-57.
[13]	程翔, 张浩天, 杨宗辉, 黄子蔚, 李思江, 余安澜. 车联网通信感知一体化研究：现状与发展趋势[J]. 通信学报, 2022, 43(8): 188-202.
[14]	曾嵘, 杭潇. 车联网环境下可重构智能反射面辅助无线信道估计算法[J]. 通信学报, 2022, 43(8): 142-150.
[15]	许文俊, 吴思雷, 王凤玉, 林兰, 李国军, 张治. 基于多智能体强化学习的大规模灾后用户分布式覆盖优化[J]. 通信学报, 2022, 43(8): 1-16.