基于多智能体元强化学习的车联网协同服务缓存和计算卸载

doi:10.11959/j.issn.1000-436x.2021104

通信学报 ›› 2021, Vol. 42 ›› Issue (6): 118-130.doi: 10.11959/j.issn.1000-436x.2021104

基于多智能体元强化学习的车联网协同服务缓存和计算卸载

宁兆龙¹^,², 张凯源², 王小洁¹, 郭磊¹

¹ 重庆邮电大学通信与信息工程学院，重庆 400065
² 大连理工大学软件学院，辽宁大连 116620

修回日期:2021-03-24 出版日期:2021-06-25 发布日期:2021-06-01
作者简介:宁兆龙（1986− ），男，辽宁沈阳人，博士，重庆邮电大学教授，主要研究方向为边缘计算、车联网、网络优化等
张凯源（1994− ），男，黑龙江哈尔滨人，大连理工大学硕士生，主要研究方向为人工智能、边缘计算
王小洁（1988− ），女，河北张家口人，博士，重庆邮电大学特聘教授，主要研究方向为物联网、人工智能、边缘计算
郭磊（1980− ），男，四川眉山人，博士，重庆邮电大学教授，主要研究方向为网络优化、网络通信、光网络等
基金资助:
国家重点研发计划基金资助项目(2018YFE0206800);国家自然科学基金资助项目(62025105);国家自然科学基金资助项目(61971084);国家自然科学基金资助项目(62001073);重庆英才计划基金资助项目(CQYC2020058659)

Cooperative service caching and peer offloading in Internet of vehicles based on multi-agent meta-reinforcement learning

Zhaolong NING¹^,², Kaiyuan ZHANG², Xiaojie WANG¹, Lei GUO¹

¹ School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
² School of Software, Dalian University of Technology, Dalian 116620, China

Revised:2021-03-24 Online:2021-06-25 Published:2021-06-01
Supported by:
The National Key Research and Development Program of China(2018YFE0206800);The National Natural Science Foundation of China(62025105);The National Natural Science Foundation of China(61971084);The National Natural Science Foundation of China(62001073);Chongqing Talent Program(CQYC2020058659)

摘要/Abstract

摘要：

为了降低求解优化问题的难度，提出一种双层的多路侧单元（RSU）协同缓存框架将问题进行解耦。外层采用多智能体元强化学习方法，在每个本地智能体进行决策学习的同时，采用长短期记忆网络作为元智能体来平衡本地决策并加速学习过程，从而得到最优的 RSU 缓存策略；内层采用拉格朗日乘子法求解最佳协同卸载策略，实现 RSU 间的任务分配。基于杭州真实交通数据的实验表明，所提方法具有理想的能效性能，并且能够在大规模任务流下保持网络稳健性。

关键词: 车联网, 边缘服务缓存, 协同卸载, 元强化学习

Abstract:

In order to reduce computation complexity, a two-layer mutli-RSU (road side unit) service caching and peer offloading algorithm (MPO) was proposed to decouple the optimization problem.In the designed MPO, the outer layer utilized multi-agent meta-reinforcement learning, which established long short-term memory (LSTM) network as the meta-agent to balance decisions of local agents and accelerate learning progress.The inner layer utilized lagrange multiplier method to achieve optimal decision for RSU peer offloading.Experimental results based on real traffic data in Hangzhou demonstrate that the proposed method outperforms other methods and remains robust under large-scale workloads.

Key words: Internet of vehicles, edge service caching, cooperative offloading, meta-reinforcement learning

中图分类号:

TN92

宁兆龙, 张凯源, 王小洁, 郭磊. 基于多智能体元强化学习的车联网协同服务缓存和计算卸载[J]. 通信学报, 2021, 42(6): 118-130.

Zhaolong NING, Kaiyuan ZHANG, Xiaojie WANG, Lei GUO. Cooperative service caching and peer offloading in Internet of vehicles based on multi-agent meta-reinforcement learning[J]. Journal on Communications, 2021, 42(6): 118-130.

图/表 12

表1

主要变量及其含义"

变量	含义
$α_{n s}^{t}$	t时刻由RSUn缓存计算任务s
$β_{i j}^{t}$	t时刻由RSUi卸载到RSUj的计算任务数量
$π_{n}^{t}$	t时刻RSUn的任务接受速率
$T_{i}^{f, t}$	车联网系统i在t时刻的计算时延
T^{g ,t}	车联网系统在t时刻的拥塞时延
r_t	t时刻车联网系统的奖励函数

表1

图1

图2

图3

图4

表2

图5

图6

图7

图8

图9

图10

参考文献 22

[1]	NING Z L , DONG P R , WANG X J ,et al. Mobile edge computing enabled 5G health monitoring for Internet of medical things:a decentralized game theoretic approach[J]. IEEE Journal on Selected Areas in Communications, 2021,39(2): 463-478.
[2]	谢人超, 廉晓飞, 贾庆民 ,等. 移动边缘计算卸载技术综述[J]. 通信学报, 2018,39(11): 138-155.
	XIE R C , LIAN X F , JIA Q M ,et al. Survey on computation offloading in mobile edge computing[J]. Journal on Communications, 2018,39(11): 138-155.
[3]	ZHUANG W H , YE Q , LYU F ,et al. SDN/NFV-empowered future IoV with enhanced communication,computing,and caching[J]. Proceedings of the IEEE, 2020,108(2): 274-291.
[4]	WANG X J , NING Z L , GUO S . Multi-agent imitation learning for pervasive edge computing:a decentralized computation offloading algorithm[J]. IEEE Transactions on Parallel and Distributed Systems, 2021,32(2): 411-425.
[5]	刘雷, 陈晨, 冯杰 ,等. 车载边缘计算中任务卸载和服务缓存的联合智能优化[J]. 通信学报, 2021,42(1): 18-26.
	LIU L , CHEN C , FENG J ,et al. Joint intelligent optimization of task offloading and service caching for vehicular edge computing[J]. Journal on Communications, 2021,42(1): 18-26.
[6]	CHEN Z Q , DUAN L Y , WANG S Q ,et al. Toward knowledge as a service over networks:a deep learning model communication paradigm[J]. IEEE Journal on Selected Areas in Communications, 2019,37(6): 1349-1363.
[7]	张彦, 张科, 曹佳钰 . 边缘智能驱动的车联网[J]. 物联网学报, 2018,2(4): 40-48.
	ZHANG Y , ZHANG K , CAO J Y . Internet of vehicles empowered by edge intelligence[J]. Chinese Journal on Internet of Things, 2018,2(4): 40-48.
[8]	WANG X J , NING Z L , GUO S ,et al. Imitation learning enabled task scheduling for online vehicular edge computing[J]. IEEE Transactions on Mobile Computing, 2020,PP(99): 1.
[9]	NING Z L , DONG P R , WANG X J ,et al. When deep reinforcement learning meets 5G-enabled vehicular networks:a distributed offloading framework for traffic big data[J]. IEEE Transactions on Industrial Informatics, 2020,16(2): 1352-1361.
[10]	NING Z L , DONG P R , WANG X J ,et al. Deep reinforcement learning for vehicular edge computing:an intelligent offloading system[J]. ACM Transactions on Intelligent Systems and Technology, 2019,10(6): 60.
[11]	TRAN T X , HAJISAMI A , PANDEY P ,et al. Collaborative mobile edge computing in 5G networks:new paradigms,scenarios,and challenges[J]. IEEE Communications Magazine, 2017,55(4): 54-61.
[12]	亓晋, 孙海蓉, 巩锟 ,等. 移动边缘计算中基于信誉值的智能计算卸载模型研究[J]. 通信学报, 2020,41(7): 141-151.
	QI J , SUN H R , GONG K ,et al. Research on intelligent computing offloading model based on reputation value in mobile edge computing[J]. Journal on Communications, 2020,41(7): 141-151.
[13]	XU J , CHEN L X , ZHOU P . Joint service caching and task offloading for mobile edge computing in dense networks[C]// IEEE INFOCOM 2018 - IEEE Conference on Computer Communications. Piscataway:IEEE Press, 2018: 207-215.
[14]	WANG X J , NING Z L , WANG L . Offloading in Internet of vehicles:a fog-enabled real-time traffic management system[J]. IEEE Transactions on Industrial Informatics, 2018,14(10): 4568-4578.
[15]	CHEN M , HAO Y X . Task offloading for mobile edge computing in software defined ultra-dense network[J]. IEEE Journal on Selected Areas in Communications, 2018,36(3): 587-597.
[16]	TAN L T , HU R Q , HANZO L . Twin-timescale artificial intelligence aided mobility-aware edge caching and computing in vehicular networks[J]. IEEE Transactions on Vehicular Technology, 2019,68(4): 3086-3099.
[17]	DAI Y Y , XU D , MAHARJAN S ,et al. Joint computation offloading and user association in multi-task mobile edge computing[J]. IEEE Transactions on Vehicular Technology, 2018,67(12): 12313-12325.
[18]	TAO O Y , ZHI Z , XU C . Follow me at the edge:mobility-aware dynamic service placement for mobile edge computing[C]// 2018 IEEE/ACM 26th International Symposium on Quality of Service. Piscataway:IEEE Press, 2018: 1-10.
[19]	NING Z L , DONG P R , KONG X J ,et al. A cooperative partial computation offloading scheme for mobile edge computing enabled Internet of Things[J]. IEEE Internet of Things Journal, 2019,6(3): 4804-4814.
[20]	QIAO G H , LENG S P , ZHANG K ,et al. Collaborative task offloading in vehicular edge multi-access networks[J]. IEEE Communications Magazine, 2018,56(8): 48-54.
[21]	NEWELL G F . Simple queueing systems[M]. Dordrecht: Springer Netherlands, 1982.
[22]	HERINGS J J , PEETERS R . Stationary equilibria in stochastic games:structure,selection,and computation[J]. Research Memorandum, 2000,118(1): 32-60.

参数	值
任务所需CPU周期数h/个	2×10⁷
RSU i的计算能力（CPU周期数）F_i/GHz	3
缓存服务s大小 p_s/MB	0.2
RSU i的服务缓存能力C_i/MB	0.6
网络传输一个计算任务的时延τ/ms	200
计算任务上传到云端的时延T_cloud/ms	800

基于多智能体元强化学习的车联网协同服务缓存和计算卸载

Cooperative service caching and peer offloading in Internet of vehicles based on multi-agent meta-reinforcement learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 22

相关文章 15

Metrics

推荐阅读 0

[1]	张海波, 兰凯, 陈舟, 王汝言, 邹灿, 王明月. 车联网中基于环的匿名高效批量认证与组密钥协商协议[J]. 通信学报, 2023, 44(6): 103-116.
[2]	张海波, 曹钰坤, 刘开健, 王汝言. 车联网中基于区块链的分布式信任管理方案[J]. 通信学报, 2023, 44(5): 148-157.
[3]	刘雪娇, 钟强, 夏莹杰. 基于双层分片区块链的车联网跨信任域高效认证方案[J]. 通信学报, 2023, 44(5): 213-223.
[4]	刘雪娇, 曹天聪, 夏莹杰. 区块链架构下高效的车联网跨域数据安全共享研究[J]. 通信学报, 2023, 44(3): 186-197.
[5]	张雷, 王玉, 田建杰, 张琳, 章天骄. 基于IRS辅助的MIMO车联网系统联合波束成形设计[J]. 通信学报, 2023, 44(2): 59-69.
[6]	曾嵘, 杭潇. 车联网环境下可重构智能反射面辅助无线信道估计算法[J]. 通信学报, 2022, 43(8): 142-150.
[7]	程翔, 张浩天, 杨宗辉, 黄子蔚, 李思江, 余安澜. 车联网通信感知一体化研究：现状与发展趋势[J]. 通信学报, 2022, 43(8): 188-202.
[8]	秦鹏, 和昊婷, 赵雄文, 伏阳, 张钰, 王淼, 王硕, 武雪. 基于停放车辆路边单元环境感知的车联网资源高效分配[J]. 通信学报, 2022, 43(7): 113-125.
[9]	孙雁飞, 尹嘉峥, 亓晋, 胡筱旋, 陈梦婷, 董振江. 基于动态图嵌入的车联网拓扑控制[J]. 通信学报, 2022, 43(6): 133-142.
[10]	朱思峰, 蔡江昊, 柴争义, 孙恩林. 车联网云边协同计算场景下的多目标优化卸载决策[J]. 通信学报, 2022, 43(6): 223-234.
[11]	亓伟敬, 宋清洋, 郭磊. 面向软件定义多模态车联网的双时间尺度RAN切片资源分配[J]. 通信学报, 2022, 43(4): 60-70.
[12]	莫梓嘉, 高志鹏, 杨杨, 林怡静, 孙山, 赵晨. 面向车联网数据隐私保护的高效分布式模型共享策略[J]. 通信学报, 2022, 43(4): 83-94.
[13]	丛玉良, 孙闻晞, 薛科, 钱志鸿, 陈绵书. 基于改进的混合遗传算法的车联网任务卸载策略研究[J]. 通信学报, 2022, 43(10): 77-85.
[14]	陈九九, 冯春燕, 郭彩丽, 杨洋, 孙启政, 朱美逸. 车联网中视频语义驱动的资源分配算法[J]. 通信学报, 2021, 42(7): 1-11.
[15]	崔杰, 陈学峰, 张静, 魏璐, 仲红. 基于公交车缓存的车联网位置隐私保护方案[J]. 通信学报, 2021, 42(7): 150-161.