基于多智能体元强化学习的车联网协同服务缓存和计算卸载

doi:10.11959/j.issn.1000-436x.2021104

Abstract

Abstract:

In order to reduce computation complexity, a two-layer mutli-RSU (road side unit) service caching and peer offloading algorithm (MPO) was proposed to decouple the optimization problem.In the designed MPO, the outer layer utilized multi-agent meta-reinforcement learning, which established long short-term memory (LSTM) network as the meta-agent to balance decisions of local agents and accelerate learning progress.The inner layer utilized lagrange multiplier method to achieve optimal decision for RSU peer offloading.Experimental results based on real traffic data in Hangzhou demonstrate that the proposed method outperforms other methods and remains robust under large-scale workloads.

Key words: Internet of vehicles, edge service caching, cooperative offloading, meta-reinforcement learning

CLC Number:

TN92

Zhaolong NING, Kaiyuan ZHANG, Xiaojie WANG, Lei GUO. Cooperative service caching and peer offloading in Internet of vehicles based on multi-agent meta-reinforcement learning[J]. Journal on Communications, 2021, 42(6): 118-130.

Figures/Tables 12

变量	含义
$α_{n s}^{t}$	t时刻由RSUn缓存计算任务s
$β_{i j}^{t}$	t时刻由RSUi卸载到RSUj的计算任务数量
$π_{n}^{t}$	t时刻RSUn的任务接受速率
$T_{i}^{f, t}$	车联网系统i在t时刻的计算时延
T^{g ,t}	车联网系统在t时刻的拥塞时延
r_t	t时刻车联网系统的奖励函数

References 22

[1]	NING Z L , DONG P R , WANG X J ,et al. Mobile edge computing enabled 5G health monitoring for Internet of medical things:a decentralized game theoretic approach[J]. IEEE Journal on Selected Areas in Communications, 2021,39(2): 463-478.
[2]	谢人超, 廉晓飞, 贾庆民 ,等. 移动边缘计算卸载技术综述[J]. 通信学报, 2018,39(11): 138-155.
	XIE R C , LIAN X F , JIA Q M ,et al. Survey on computation offloading in mobile edge computing[J]. Journal on Communications, 2018,39(11): 138-155.
[3]	ZHUANG W H , YE Q , LYU F ,et al. SDN/NFV-empowered future IoV with enhanced communication,computing,and caching[J]. Proceedings of the IEEE, 2020,108(2): 274-291.
[4]	WANG X J , NING Z L , GUO S . Multi-agent imitation learning for pervasive edge computing:a decentralized computation offloading algorithm[J]. IEEE Transactions on Parallel and Distributed Systems, 2021,32(2): 411-425.
[5]	刘雷, 陈晨, 冯杰 ,等. 车载边缘计算中任务卸载和服务缓存的联合智能优化[J]. 通信学报, 2021,42(1): 18-26.
	LIU L , CHEN C , FENG J ,et al. Joint intelligent optimization of task offloading and service caching for vehicular edge computing[J]. Journal on Communications, 2021,42(1): 18-26.
[6]	CHEN Z Q , DUAN L Y , WANG S Q ,et al. Toward knowledge as a service over networks:a deep learning model communication paradigm[J]. IEEE Journal on Selected Areas in Communications, 2019,37(6): 1349-1363.
[7]	张彦, 张科, 曹佳钰 . 边缘智能驱动的车联网[J]. 物联网学报, 2018,2(4): 40-48.
	ZHANG Y , ZHANG K , CAO J Y . Internet of vehicles empowered by edge intelligence[J]. Chinese Journal on Internet of Things, 2018,2(4): 40-48.
[8]	WANG X J , NING Z L , GUO S ,et al. Imitation learning enabled task scheduling for online vehicular edge computing[J]. IEEE Transactions on Mobile Computing, 2020,PP(99): 1.
[9]	NING Z L , DONG P R , WANG X J ,et al. When deep reinforcement learning meets 5G-enabled vehicular networks:a distributed offloading framework for traffic big data[J]. IEEE Transactions on Industrial Informatics, 2020,16(2): 1352-1361.
[10]	NING Z L , DONG P R , WANG X J ,et al. Deep reinforcement learning for vehicular edge computing:an intelligent offloading system[J]. ACM Transactions on Intelligent Systems and Technology, 2019,10(6): 60.
[11]	TRAN T X , HAJISAMI A , PANDEY P ,et al. Collaborative mobile edge computing in 5G networks:new paradigms,scenarios,and challenges[J]. IEEE Communications Magazine, 2017,55(4): 54-61.
[12]	亓晋, 孙海蓉, 巩锟 ,等. 移动边缘计算中基于信誉值的智能计算卸载模型研究[J]. 通信学报, 2020,41(7): 141-151.
	QI J , SUN H R , GONG K ,et al. Research on intelligent computing offloading model based on reputation value in mobile edge computing[J]. Journal on Communications, 2020,41(7): 141-151.
[13]	XU J , CHEN L X , ZHOU P . Joint service caching and task offloading for mobile edge computing in dense networks[C]// IEEE INFOCOM 2018 - IEEE Conference on Computer Communications. Piscataway:IEEE Press, 2018: 207-215.
[14]	WANG X J , NING Z L , WANG L . Offloading in Internet of vehicles:a fog-enabled real-time traffic management system[J]. IEEE Transactions on Industrial Informatics, 2018,14(10): 4568-4578.
[15]	CHEN M , HAO Y X . Task offloading for mobile edge computing in software defined ultra-dense network[J]. IEEE Journal on Selected Areas in Communications, 2018,36(3): 587-597.
[16]	TAN L T , HU R Q , HANZO L . Twin-timescale artificial intelligence aided mobility-aware edge caching and computing in vehicular networks[J]. IEEE Transactions on Vehicular Technology, 2019,68(4): 3086-3099.
[17]	DAI Y Y , XU D , MAHARJAN S ,et al. Joint computation offloading and user association in multi-task mobile edge computing[J]. IEEE Transactions on Vehicular Technology, 2018,67(12): 12313-12325.
[18]	TAO O Y , ZHI Z , XU C . Follow me at the edge:mobility-aware dynamic service placement for mobile edge computing[C]// 2018 IEEE/ACM 26th International Symposium on Quality of Service. Piscataway:IEEE Press, 2018: 1-10.
[19]	NING Z L , DONG P R , KONG X J ,et al. A cooperative partial computation offloading scheme for mobile edge computing enabled Internet of Things[J]. IEEE Internet of Things Journal, 2019,6(3): 4804-4814.
[20]	QIAO G H , LENG S P , ZHANG K ,et al. Collaborative task offloading in vehicular edge multi-access networks[J]. IEEE Communications Magazine, 2018,56(8): 48-54.
[21]	NEWELL G F . Simple queueing systems[M]. Dordrecht: Springer Netherlands, 1982.
[22]	HERINGS J J , PEETERS R . Stationary equilibria in stochastic games:structure,selection,and computation[J]. Research Memorandum, 2000,118(1): 32-60.

Metrics

Recommended 0

No Suggested Reading articles found!

参数	值
任务所需CPU周期数h/个	2×10⁷
RSU i的计算能力（CPU周期数）F_i/GHz	3
缓存服务s大小 p_s/MB	0.2
RSU i的服务缓存能力C_i/MB	0.6
网络传输一个计算任务的时延τ/ms	200
计算任务上传到云端的时延T_cloud/ms	800

Cooperative service caching and peer offloading in Internet of vehicles based on multi-agent meta-reinforcement learning

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 22

Related Articles 15

Metrics

Recommended 0

[1]	Haibo ZHANG, Yukun CAO, Kaijian LIU, Ruyan WANG. Distributed trust management scheme based on blockchain in Internet of vehicles [J]. Journal on Communications, 2023, 44(5): 148-157.
[2]	Xuejiao LIU, Qiang ZHONG, Yingjie XIA. Efficient authentication scheme for cross-trust domain of IoV based on double-layer shard blockchain [J]. Journal on Communications, 2023, 44(5): 213-223.
[3]	Lei ZHANG, Yu WANG, Jianjie TIAN, Lin ZHANG, Tianjiao ZHANG. Joint beam forming design for IRS-aided MIMO Internet of vehicles system [J]. Journal on Communications, 2023, 44(2): 59-69.
[4]	Rong ZENG, Xiao HANG. Reconfigurable intelligent surface assist wireless channel estimation algorithm in Internet of vehicles environment [J]. Journal on Communications, 2022, 43(8): 142-150.
[5]	Xiang CHENG, Haotian ZHANG, Zonghui YANG, Ziwei HUANG, Sijiang LI, Anlan YU. Integrated sensing and communications for Internet of vehicles：current status and development trend [J]. Journal on Communications, 2022, 43(8): 188-202.
[6]	Peng QIN, Haoting HE, Xiongwen ZHAO, Yang FU, Yu ZHANG, Miao WANG, Shuo WANG, Xue WU. Efficient resource allocation with context-awareness for parked car road side unit-based Internet of vehicles [J]. Journal on Communications, 2022, 43(7): 113-125.
[7]	Sifeng ZHU, Jianghao CAI, Zhengyi CHAI, Enlin SUN. Multi-objective optimal offloading decision for cloud-edge collaborative computing scenario in Internet of vehicles [J]. Journal on Communications, 2022, 43(6): 223-234.
[8]	Yanfei SUN, Jiazheng YIN, Jin QI, Xiaoxuan HU, Mengting CHEN, Zhenjiang DONG. Topology control based on dynamic graph embedding in Internet of vehicles [J]. Journal on Communications, 2022, 43(6): 133-142.
[9]	Yuliang CONG, Wenxi SUN, Ke XUE, Zhihong QIAN, Mianshu CHEN. Research on task offloading strategy of Internet of vehicles based on improved hybrid genetic algorithm [J]. Journal on Communications, 2022, 43(10): 77-85.
[10]	Jiujiu CHEN, Chunyan FENG, Caili GUO, Yang YANG, Qizheng SUN, Meiyi ZHU. Video semantics-driven resource allocation algorithm in Internet of vehicles [J]. Journal on Communications, 2021, 42(7): 1-11.
[11]	Jie CUI, Xuefeng CHEN, Jing ZHANG, Lu WEI, Hong ZHONG. Bus cache-based location privacy protection scheme in the Internet of vehicles [J]. Journal on Communications, 2021, 42(7): 150-161.
[12]	Jiujun CHENG, Guiyuan YUAN, Jie CUI, Aiguo ZHOU, Bo LYU, Guangyao LI. Spatio-temporal data analysis and accessibility method for IoV in an urban scene [J]. Journal on Communications, 2021, 42(6): 52-61.
[13]	Tao HAN, Wei HE, Jun DAI, Yong ZUO, Yang YANG, Xiaohu GE. Connectivity analysis of IoV based on scale-free network [J]. Journal on Communications, 2021, 42(4): 100-108.
[14]	Shichang XUAN, Hao TANG, Wu YANG. Method for detecting collusion attack node in road condition information sharing based on reputation point [J]. Journal on Communications, 2021, 42(4): 158-168.
[15]	Xiaodong YANG, Wanting XI, Jiaqi WANG, Aijia CHEN, Caifen WANG. Electronic evidence sharing scheme of Internet of vehicles based on signcryption and blockchain [J]. Journal on Communications, 2021, 42(12): 236-246.