移动边缘网络中基于双深度Q学习的高能效资源分配方法

doi:10.11959/j.issn.1000-436X.2020218

Abstract

Abstract:

To improve the system energy efficiency in mobile edge networks, a resource allocation method based on double deep Q-learning(DDQL) for integration of communication, computing, storage resources was proposed for the downlink communication process under the network architecture of multiple tasks, end devices, edge gateways and edge servers.A resource allocation model was constructed, which took the minimization of average energy consumption of tasks as the optimization goal and set the constraints of task delay limits and communication, computing, and storage resource limits.According to the model characteristics, a suitable resource allocation model and method based on DDQL framework was proposed to make intelligent allocation decisions for communication and computing resources and allocate storage resources on demand.Simulation results show that the proposed DDQL-based solution can effectively solve the multi-task resource allocation problem with good converge and low time complexity, and it reduces the average energy consumption of tasks by at least 5% compared with the solving methods based on random algorithm, greedy algorithm, particle swarm optimization algorithm and deep Q-learning while ensuring the quality of service.

Key words: mobile edge network, integrated resource allocation, energy-efficient, DDQL

CLC Number:

TN929.5

Peng YU, Junye ZHANG, Wenjing LI, Fanqin ZHOU, Lei FENG, Shu FU, Xuesong QIU. Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning[J]. Journal on Communications, 2020, 41(12): 148-161.

Figures/Tables 14

算法	时间复杂度
暴力搜索	$O (K \prod_{k \in K} (N_{k}^{u} N_{k}^{p} N_{k}^{f}_{k}))$
RA	O(T^RAK)
GA	O(T^GAK)
PSO	O(T ^PSO MK)
DDQL/DQL	O(T^StepK)

References 32

[1]	陈亮, 余少华 . 6G 移动通信发展趋势初探（特邀）[J]. 光通信研究, 2019,45(4): 1-8.
	CHEN L , YU S H . Preliminary Study on the Trend of 6G Mobile Communication (special invitation)[J]. Study on Optical Communications, 2019,45(4): 1-8.
[2]	田辉, 范绍帅, 吕昕晨 ,等. 面向 5G 需求的移动边缘计算[J]. 北京邮电大学学报, 2017,40(2): 1-10.
	TIAN H , FAN S S , LYU X C ,et al. Mobile edge computing for 5G requirements[J]. Journal of Beijing University of Posts and Telecommunications, 2017,40(2): 1-10.
[3]	STRINATI E C , BARBAROSSA S , GONZALEZ-JIMENEZ J L ,et al. 6G:the next frontier:from holographic messaging to artificial intelligence using subterahertz and visible light communication[J]. IEEE Vehicular Technology Magazine, 2019,14(3): 42-50.
[4]	毕成 . 5G无线网络节能技术研究[J]. 电信工程技术与标准化, 2020,33(8): 1-5.
	BI C . Research on energy saving technology of 5G wireless network[J]. Telecom Engineering Technics and Standardization, 2020,33(8): 1-5.
[5]	赵亚军, 郁光辉, 徐汉青 . 6G移动通信网络:愿景、挑战与关键技术[J]. 中国科学:信息科学, 2019,8(49): 963-987.
	ZHAO Y J , YU G H , XU H Q . 6G mobile communication networks:vision,challenges,and key technologies[J]. Scientia Sinica (Informationis), 2019,8(49): 963-987.
[6]	CAI S , CHE Y , DUAN J ,et al. Green 5G heterogeneous networks through dynamic small-cell operation[J]. IEEE Journal on Selected Areas in Communications, 2016,34(5): 1103-1115.
[7]	ZHOU Y , TIAN L , LIU L ,et al. Fog computing enabled future mobile communication networks:a convergence of communication and computing[J]. IEEE Communications Magazine, 2019,57(5): 20-27.
[8]	LIU L , ZHOU Y , YUAN J ,et al. Economically optimal MS association for multimedia content delivery in cache-enabled heterogeneous cloud radio access networks[J]. IEEE Journal on Selected Areas in Communications, 2019,37(7): 1584-1593.
[9]	BI S , HUANG L , ZHANG Y J A . Joint optimization of service caching placement and computation offloading in mobile edge computing systems[J]. IEEE Transactions on Wireless Communications, 2020,19(7): 4947-4963.
[10]	MAO Y , ZHANG J , LETAIEF K B . Dynamic computation offloading for mobile-edge computing with energy harvesting devices[J]. IEEE Journal on Selected Areas in Communications, 2016,34(12): 3590-3605.
[11]	LIU L , ZHOU Y , GARCIA V ,et al. Load aware joint CoMP clustering and inter-cell resource scheduling in heterogeneous ultra dense cellular networks[J]. IEEE Transactions on Vehicular Technology, 2017,67(3): 2741-2755.
[12]	BADRI H , BAHREINI T , GROSU D ,et al. Risk-based optimization of resource provisioning in mobile edge computing[C]// 2018 IEEE/ACM Symposium on Edge Computing (SEC). Piscataway:IEEE Press, 2018: 328-330.
[13]	WANG L , JIAO L , LI J ,et al. Moera:mobility-agnostic online resource allocation for edge computing[J]. IEEE Transactions on Mobile Computing, 2019,18(8): 1843-1856.
[14]	CHEN X , JIAO L , LI W ,et al. Efficient multi-user computation offloading for mobile-edge cloud computing[J]. IEEE/ACM Transactions on Networking, 2015,24(5): 2795-2808.
[15]	ZENG D , GU L , PAN S ,et al. Resource management at the network edge:a deep reinforcement learning approach[J]. IEEE Network, 2019,33(3): 26-33.
[16]	梁应敞, 谭俊杰, NIYATO D . 智能无线通信技术研究概况[J]. 通信学报, 2020,41(7): 1-17.
	LIANG Y C , TAN J J , NIYATO D . Overview on intelligent wireless communication technology[J]. Journal on Communications, 2020,41(7): 1-17.
[17]	XU J , CHEN L , REN S . Online learning for offloading and autoscaling in energy harvesting mobile edge computing[J]. IEEE Transactions on Cognitive Communications and Networking, 2017,3(3): 361-373.
[18]	ZHANG C , PATRAS P , HADDADI H . Deep learning in mobile and wireless networking:a survey[J]. IEEE Communications Surveys ＆Tutorials, 2019,21(3): 2224-2287.
[19]	ZHAO N , LIANG Y C , NIYATO D ,et al. Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks[J]. IEEE Transactions on Wireless Communications, 2019,18(11): 5141-5152.
[20]	LI J , GAO H , LYU T ,et al. Deep reinforcement learning based computation offloading and resource allocation for MEC[C]// 2018 IEEE Wireless Communications and Networking Conference. Piscataway:IEEE Press, 2018: 1-6.
[21]	WANG J , ZHAO L , LIU J ,et al. Smart resource allocation for mobile edge computing:a deep reinforcement learning approach[J]. IEEE Transactions on Emerging Topics in Computing, 2019,,doi:10.1109/TETC.2019.2902661.
[22]	LI Y , QI F , WANG Z ,et al. Distributed edge computing offloading algorithm based on deep reinforcement learning[J]. IEEE Access, 2020,8: 85204-85215.
[23]	VAN H H , GUEZ A , SILVER D . Deep reinforcement learning with double Q-learning[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto:AAAI Press, 2016: 2094-2100.
[24]	CHEN X , ZHANG H , WU C ,et al. Optimized computation of-floading performance in virtual edge computing systems via deep reinforcement learning[J]. IEEE Internet of Things Journal, 2018,6(3): 4005-4018.
[25]	MAO Y , YOU C , ZHANG J ,et al. A survey on mobile edge computing:the communication perspective[J]. IEEE Communications Surveys ＆ Tutorials, 2017,19(4): 2322-2358.
[26]	刘全, 翟建伟, 章宗长 ,等. 深度强化学习综述[J]. 计算机学报, 2018,41(1): 1-27.
	LIU Q , ZHAI J W , ZHANG Z Z ,et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018,41(1): 1-27.
[27]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[28]	CAO X , WANG F , XU J ,et al. Joint computation and communication cooperation for energy-efficient mobile edge computing[J]. IEEE Internet of Things Journal, 2018,6(3): 4188-4200.
[29]	TRAN T X , POMPILI D . Joint task offloading and resource allocation for multi-server mobile-edge computing networks[J]. IEEE Transactions on Vehicular Technology, 2018,68(1): 856-868.
[30]	YANG X , YU P , FENG L ,et al. A deep reinforcement learning based mechanism for cell outage compensation in 5G UDN[C]// 2019 IFIP/IEEE Symposium on Integrated Network and Service Management. Piscataway:IEEE Press, 2019: 476-481.
[31]	WANG S , PAN C , YIN C . Joint heterogeneous tasks offloading and resource allocation in mobile edge computing systems[C]// 2018 10th International Conference on Wireless Communications and Signal Processing. Piscataway:IEEE Press, 2018: 1-6.
[32]	ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M ,et al. Deep reinforcement learning:a brief survey[J]. IEEE Signal Processing Magazine, 2017,34(6): 26-38.

Metrics

Recommended 0

No Suggested Reading articles found!

参数	数值
带宽B_g,d/ MHz	10
路径损耗h_g,d/ dB	15.3+37.6lg(D_g,d )
噪声功率谱密度N₀ /( dBm.Hz^-1)	-100
EG g 最大发射功率P_g /W	10
ES s最大时钟频率F_s /GHz	20
ES s存储空间B_s/TB	1
计算能耗参数κ	10^-27
每个任务最大发射功率P_max/W	1
每个任务最小发射功率P_min/W	0.1
每个任务最大时钟频率F_max/ GHz	2
每个任务最小时钟频率F_min/ GHz	0.5
任务k返回数据量l_k /MB	1,10]
任务k所需存储空间b_k /MB	10,1 024]
任务k所需计算周期数c_k /Mcycle	10,500]
任务k最大时延T_k/s	0.1,1]
任务最大能耗E_max/J	1

参数	数值
折扣因子γ	0.9
探索利用率ε最大值	0.9
探索利用率ε最小值	0.1
探索利用率ε减小速率/步	0.01
回放记忆单元容量/个	500
抽样大小/个	32
DNN各层神经元个数/个	3K,64,32,128,5K+1

参数	数值
粒子群规模M	50
个体经验学习因子c₁	1
社会经验学习因子c₂	1
惯性因子ω	0.6
最大速度v_max	2
最小速度v_min	-2

Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 32

Related Articles 7

Metrics

Recommended 0

[1]	Yan CAI, Fan WU, Hongbo ZHU. Caching strategy based on transmission delay for D2D cooperative edge caching system [J]. Journal on Communications, 2021, 42(3): 183-189.
[2]	Ying HU,Bao-feng JI,Yong-ming HUANG,Fei YU,Lv-xi YANG. Energy-efficient resource allocation algorithm for massive MIMO OFDMA downlink system [J]. Journal on Communications, 2015, 36(7): 1-49.
[3]	Xu LU,Liang-lun CHENG,Shi-liang LUO. Adaptive node scheduling algorithm for target tracking in wireless sensor networks [J]. Journal on Communications, 2015, 36(4): 70-80.
[4]	UYing H,IBao-feng J,UANGYong-ming H,UFei Y,ANGLV-xi Y. Energy-efficient optimal algorithm based on uplink multi-user very large MIMO system [J]. Journal on Communications, 2015, 36(1): 68-74.
[5]	Xu LIU,Jin-bao LI,Long-jiang GUO,Bin FAN,Jing-hua ZHU. Energy-efficient algorithm for mobile target prediction and tracking in wireless sensor networks [J]. Journal on Communications, 2010, 31(9A): 135-143.
[6]	Shu-gang LIU,Hong-li LIU,Jie ZHAN,Yao-nan WANG. Research of energy-efficient hybrid communication algorithm in wireless sensor networks [J]. Journal on Communications, 2009, 30(1): 12-17.
[7]	Yu-hong LUO,Jian-xin WANG,Song-qiao CHEN. Distributed multicast algorithm for building an energy-efficient shared tree in ad hoc networks [J]. Journal on Communications, 2006, 27(6): 1-9.