移动边缘网络中基于双深度Q学习的高能效资源分配方法

doi:10.11959/j.issn.1000-436X.2020218

通信学报 ›› 2020, Vol. 41 ›› Issue (12): 148-161.doi: 10.11959/j.issn.1000-436X.2020218

移动边缘网络中基于双深度Q学习的高能效资源分配方法

喻鹏¹, 张俊也¹, 李文璟¹, 周凡钦¹, 丰雷¹, 付澍², 邱雪松¹

¹ 北京邮电大学网络与交换技术国家重点实验室，北京 100876
² 重庆大学微电子与通信工程学院，重庆 400044

修回日期:2020-09-19 出版日期:2020-12-25 发布日期:2020-12-01
作者简介:喻鹏（1986- ），男，湖北随州人，博士，北京邮电大学副教授、博士生导师，主要研究方向为5G/6G网络智能管控等。
张俊也（1998- ），女，山西太原人，北京邮电大学硕士生，主要研究方向为5G/6G网络智能管控、移动边缘计算等。
李文璟（1973- ），女，山西太谷人，博士，北京邮电大学教授、博士生导师，主要研究方向为无线网络管理和自组织网络等。
周凡钦（1988- ），男，河南渑池人，博士，北京邮电大学在站博士后，主要研究方向为无线异构网络的资源管理和负载均衡等。
丰雷（1987- ），男，北京人，北京邮电大学副教授、硕士生导师，主要研究方向为无线网络和智能电网的资源管理等。
付澍（1985- ），男，贵州贵阳人，博士，重庆大学副教授、硕士生导师，主要研究方向为星地通信、NOMA、物联网、网络一体化等。
邱雪松（1973- ），男，江西上饶人，博士，北京邮电大学教授、博士生导师，主要研究方向为网络管理与通信软件等。
基金资助:
国家重点研发计划基金资助项目(2018YFE0205502)

Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning

Peng YU¹, Junye ZHANG¹, Wenjing LI¹, Fanqin ZHOU¹, Lei FENG¹, Shu FU², Xuesong QIU¹

¹ State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
² School of Microelectronics Communication Engineering, Chongqing University, Chongqing 400044, China

Revised:2020-09-19 Online:2020-12-25 Published:2020-12-01
Supported by:
The National Key Research and Development Program of China(2018YFE0205502)

摘要/Abstract

摘要：

为了提升移动边缘网络中系统的能量使用效率，面向多任务、多终端设备、多边缘网关、多边缘服务器共存网络架构的下行通信过程，提出了一种基于双深度Q学习（DDQL）的通信、计算、存储融合资源分配方法。以任务平均能耗最小化为优化目标，设置任务时延和通信、计算、存储资源限制等约束条件，构建了对应的资源分配模型。依据模型特征，基于DDQL框架，提出了适用于通信和计算资源智能决策、存储资源按需分配的资源分配模型和算法。仿真结果表明，所提出的基于DDQL资源分配方法可以有效地解决多任务资源分配问题，具有较好的收敛性和较低的时间复杂度，在保障业务服务质量的同时，相对于基于随机算法、贪心算法、粒子群优化算法、深度Q学习等方法，降低了至少5%的任务平均能耗。

关键词: 移动边缘网络, 融合资源分配, 高能效, 双深度Q学习

Abstract:

To improve the system energy efficiency in mobile edge networks, a resource allocation method based on double deep Q-learning(DDQL) for integration of communication, computing, storage resources was proposed for the downlink communication process under the network architecture of multiple tasks, end devices, edge gateways and edge servers.A resource allocation model was constructed, which took the minimization of average energy consumption of tasks as the optimization goal and set the constraints of task delay limits and communication, computing, and storage resource limits.According to the model characteristics, a suitable resource allocation model and method based on DDQL framework was proposed to make intelligent allocation decisions for communication and computing resources and allocate storage resources on demand.Simulation results show that the proposed DDQL-based solution can effectively solve the multi-task resource allocation problem with good converge and low time complexity, and it reduces the average energy consumption of tasks by at least 5% compared with the solving methods based on random algorithm, greedy algorithm, particle swarm optimization algorithm and deep Q-learning while ensuring the quality of service.

Key words: mobile edge network, integrated resource allocation, energy-efficient, DDQL

中图分类号:

TN929.5

喻鹏, 张俊也, 李文璟, 周凡钦, 丰雷, 付澍, 邱雪松. 移动边缘网络中基于双深度Q学习的高能效资源分配方法[J]. 通信学报, 2020, 41(12): 148-161.

Peng YU, Junye ZHANG, Wenjing LI, Fanqin ZHOU, Lei FENG, Shu FU, Xuesong QIU. Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning[J]. Journal on Communications, 2020, 41(12): 148-161.

图/表 14

图1

图2

表1

算法时间复杂度"

算法	时间复杂度
暴力搜索	$O (K \prod_{k \in K} (N_{k}^{u} N_{k}^{p} N_{k}^{f}_{k}))$
RA	O(T^RAK)
GA	O(T^GAK)
PSO	O(T ^PSO MK)
DDQL/DQL	O(T^StepK)

表1

图3

表2

表3

表4

图4

图5

图6

图7

图8

图9

图10

参考文献 32

[1]	陈亮, 余少华 . 6G 移动通信发展趋势初探（特邀）[J]. 光通信研究, 2019,45(4): 1-8.
	CHEN L , YU S H . Preliminary Study on the Trend of 6G Mobile Communication (special invitation)[J]. Study on Optical Communications, 2019,45(4): 1-8.
[2]	田辉, 范绍帅, 吕昕晨 ,等. 面向 5G 需求的移动边缘计算[J]. 北京邮电大学学报, 2017,40(2): 1-10.
	TIAN H , FAN S S , LYU X C ,et al. Mobile edge computing for 5G requirements[J]. Journal of Beijing University of Posts and Telecommunications, 2017,40(2): 1-10.
[3]	STRINATI E C , BARBAROSSA S , GONZALEZ-JIMENEZ J L ,et al. 6G:the next frontier:from holographic messaging to artificial intelligence using subterahertz and visible light communication[J]. IEEE Vehicular Technology Magazine, 2019,14(3): 42-50.
[4]	毕成 . 5G无线网络节能技术研究[J]. 电信工程技术与标准化, 2020,33(8): 1-5.
	BI C . Research on energy saving technology of 5G wireless network[J]. Telecom Engineering Technics and Standardization, 2020,33(8): 1-5.
[5]	赵亚军, 郁光辉, 徐汉青 . 6G移动通信网络:愿景、挑战与关键技术[J]. 中国科学:信息科学, 2019,8(49): 963-987.
	ZHAO Y J , YU G H , XU H Q . 6G mobile communication networks:vision,challenges,and key technologies[J]. Scientia Sinica (Informationis), 2019,8(49): 963-987.
[6]	CAI S , CHE Y , DUAN J ,et al. Green 5G heterogeneous networks through dynamic small-cell operation[J]. IEEE Journal on Selected Areas in Communications, 2016,34(5): 1103-1115.
[7]	ZHOU Y , TIAN L , LIU L ,et al. Fog computing enabled future mobile communication networks:a convergence of communication and computing[J]. IEEE Communications Magazine, 2019,57(5): 20-27.
[8]	LIU L , ZHOU Y , YUAN J ,et al. Economically optimal MS association for multimedia content delivery in cache-enabled heterogeneous cloud radio access networks[J]. IEEE Journal on Selected Areas in Communications, 2019,37(7): 1584-1593.
[9]	BI S , HUANG L , ZHANG Y J A . Joint optimization of service caching placement and computation offloading in mobile edge computing systems[J]. IEEE Transactions on Wireless Communications, 2020,19(7): 4947-4963.
[10]	MAO Y , ZHANG J , LETAIEF K B . Dynamic computation offloading for mobile-edge computing with energy harvesting devices[J]. IEEE Journal on Selected Areas in Communications, 2016,34(12): 3590-3605.
[11]	LIU L , ZHOU Y , GARCIA V ,et al. Load aware joint CoMP clustering and inter-cell resource scheduling in heterogeneous ultra dense cellular networks[J]. IEEE Transactions on Vehicular Technology, 2017,67(3): 2741-2755.
[12]	BADRI H , BAHREINI T , GROSU D ,et al. Risk-based optimization of resource provisioning in mobile edge computing[C]// 2018 IEEE/ACM Symposium on Edge Computing (SEC). Piscataway:IEEE Press, 2018: 328-330.
[13]	WANG L , JIAO L , LI J ,et al. Moera:mobility-agnostic online resource allocation for edge computing[J]. IEEE Transactions on Mobile Computing, 2019,18(8): 1843-1856.
[14]	CHEN X , JIAO L , LI W ,et al. Efficient multi-user computation offloading for mobile-edge cloud computing[J]. IEEE/ACM Transactions on Networking, 2015,24(5): 2795-2808.
[15]	ZENG D , GU L , PAN S ,et al. Resource management at the network edge:a deep reinforcement learning approach[J]. IEEE Network, 2019,33(3): 26-33.
[16]	梁应敞, 谭俊杰, NIYATO D . 智能无线通信技术研究概况[J]. 通信学报, 2020,41(7): 1-17.
	LIANG Y C , TAN J J , NIYATO D . Overview on intelligent wireless communication technology[J]. Journal on Communications, 2020,41(7): 1-17.
[17]	XU J , CHEN L , REN S . Online learning for offloading and autoscaling in energy harvesting mobile edge computing[J]. IEEE Transactions on Cognitive Communications and Networking, 2017,3(3): 361-373.
[18]	ZHANG C , PATRAS P , HADDADI H . Deep learning in mobile and wireless networking:a survey[J]. IEEE Communications Surveys ＆Tutorials, 2019,21(3): 2224-2287.
[19]	ZHAO N , LIANG Y C , NIYATO D ,et al. Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks[J]. IEEE Transactions on Wireless Communications, 2019,18(11): 5141-5152.
[20]	LI J , GAO H , LYU T ,et al. Deep reinforcement learning based computation offloading and resource allocation for MEC[C]// 2018 IEEE Wireless Communications and Networking Conference. Piscataway:IEEE Press, 2018: 1-6.
[21]	WANG J , ZHAO L , LIU J ,et al. Smart resource allocation for mobile edge computing:a deep reinforcement learning approach[J]. IEEE Transactions on Emerging Topics in Computing, 2019,,doi:10.1109/TETC.2019.2902661.
[22]	LI Y , QI F , WANG Z ,et al. Distributed edge computing offloading algorithm based on deep reinforcement learning[J]. IEEE Access, 2020,8: 85204-85215.
[23]	VAN H H , GUEZ A , SILVER D . Deep reinforcement learning with double Q-learning[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto:AAAI Press, 2016: 2094-2100.
[24]	CHEN X , ZHANG H , WU C ,et al. Optimized computation of-floading performance in virtual edge computing systems via deep reinforcement learning[J]. IEEE Internet of Things Journal, 2018,6(3): 4005-4018.
[25]	MAO Y , YOU C , ZHANG J ,et al. A survey on mobile edge computing:the communication perspective[J]. IEEE Communications Surveys ＆ Tutorials, 2017,19(4): 2322-2358.
[26]	刘全, 翟建伟, 章宗长 ,等. 深度强化学习综述[J]. 计算机学报, 2018,41(1): 1-27.
	LIU Q , ZHAI J W , ZHANG Z Z ,et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018,41(1): 1-27.
[27]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[28]	CAO X , WANG F , XU J ,et al. Joint computation and communication cooperation for energy-efficient mobile edge computing[J]. IEEE Internet of Things Journal, 2018,6(3): 4188-4200.
[29]	TRAN T X , POMPILI D . Joint task offloading and resource allocation for multi-server mobile-edge computing networks[J]. IEEE Transactions on Vehicular Technology, 2018,68(1): 856-868.
[30]	YANG X , YU P , FENG L ,et al. A deep reinforcement learning based mechanism for cell outage compensation in 5G UDN[C]// 2019 IFIP/IEEE Symposium on Integrated Network and Service Management. Piscataway:IEEE Press, 2019: 476-481.
[31]	WANG S , PAN C , YIN C . Joint heterogeneous tasks offloading and resource allocation in mobile edge computing systems[C]// 2018 10th International Conference on Wireless Communications and Signal Processing. Piscataway:IEEE Press, 2018: 1-6.
[32]	ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M ,et al. Deep reinforcement learning:a brief survey[J]. IEEE Signal Processing Magazine, 2017,34(6): 26-38.

参数	数值
带宽B_g,d/ MHz	10
路径损耗h_g,d/ dB	15.3+37.6lg(D_g,d )
噪声功率谱密度N₀ /( dBm.Hz^-1)	-100
EG g 最大发射功率P_g /W	10
ES s最大时钟频率F_s /GHz	20
ES s存储空间B_s/TB	1
计算能耗参数κ	10^-27
每个任务最大发射功率P_max/W	1
每个任务最小发射功率P_min/W	0.1
每个任务最大时钟频率F_max/ GHz	2
每个任务最小时钟频率F_min/ GHz	0.5
任务k返回数据量l_k /MB	1,10]
任务k所需存储空间b_k /MB	10,1 024]
任务k所需计算周期数c_k /Mcycle	10,500]
任务k最大时延T_k/s	0.1,1]
任务最大能耗E_max/J	1

参数	数值
折扣因子γ	0.9
探索利用率ε最大值	0.9
探索利用率ε最小值	0.1
探索利用率ε减小速率/步	0.01
回放记忆单元容量/个	500
抽样大小/个	32
DNN各层神经元个数/个	3K,64,32,128,5K+1

参数	数值
粒子群规模M	50
个体经验学习因子c₁	1
社会经验学习因子c₂	1
惯性因子ω	0.6
最大速度v_max	2
最小速度v_min	-2

移动边缘网络中基于双深度Q学习的高能效资源分配方法

Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 32

相关文章 2

Metrics

推荐阅读 0

[1]	蔡艳, 吴凡, 朱洪波. D2D协作边缘缓存系统中基于传输时延的缓存策略[J]. 通信学报, 2021, 42(3): 183-189.
[2]	卢旭,程良伦,罗世亮. 无线传感器网络自适应目标跟踪节点调度算法[J]. 通信学报, 2015, 36(4): 70-80.