基于深度强化学习的能源互联网智能巡检任务分配机制

doi:10.11959/j.issn.1000-436x.2021071

通信学报 ›› 2021, Vol. 42 ›› Issue (5): 191-204.doi: 10.11959/j.issn.1000-436x.2021071

基于深度强化学习的能源互联网智能巡检任务分配机制

徐思雅¹, 邢逸斐¹, 郭少勇¹, 杨超², 邱雪松¹, 孟洛明¹

¹ 北京邮电大学网络技术与交换重点实验室，北京 100876
² 国网辽宁省电力有限公司信息通信分公司，辽宁沈阳 110004

修回日期:2021-03-10 出版日期:2021-05-25 发布日期:2021-05-01
作者简介:徐思雅（1988- ），女，北京人，博士，北京邮电大学讲师，主要研究方向为信息通信网络管理、SDN/NFV、移动边缘计算和人工智能等
邢逸斐（1997- ），男，河北保定人，北京邮电大学硕士生，主要研究方向为智能电网网络管理、移动边缘计算等
郭少勇（1985- ），男，河北邢台人，博士，北京邮电大学副教授，主要研究方向为区块链、物联网等
杨超（1988- ），男，山东平度人，博士，国网辽宁省电力有限公司工程师，主要研究方向为电力物联网、人工智能工程化应用等
邱雪松（1973- ），男，江西上饶人，博士，北京邮电大学教授、博士生导师，主要研究方向为网络与业务管理、物联网与区块链
孟洛明（1955- ），男，河南洛阳人，博士，北京邮电大学教授、博士生导师，主要研究方向为通信网、网络管理、通信软件等
基金资助:
国家重点研发计划基金资助项目(2019YFB2102302);国家自然科学基金资助项目(61702048);工业互联网创新发展工程基金资助项目（基于泛在电力物联网的工业互联网测试床）

Deep reinforcement learning based task allocation mechanism for intelligent inspection in energy Internet

Siya XU¹, Yifei XING¹, Shaoyong GUO¹, Chao YANG², Xuesong QIU¹, Luoming MENG¹

¹ State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
² Information and Communication Branch, State Grid Liaoning Electric Power Co., Ltd., Shenyang 110004, China

Revised:2021-03-10 Online:2021-05-25 Published:2021-05-01
Supported by:
The National Key Research and Development Program of China(2019YFB2102302);The National Natural Science Foundation of China(61702048);Innovative Development Project of Industrial Internet (Construction of Industrial In-ternet Test Bed Based on Ubiquitous Power Internet of Things)

摘要/Abstract

摘要：

在能源互联网中引入无人机进行电力线路巡查，并借助移动边缘计算技术实现巡检任务的接入和处理，可降低服务成本，提高工作效率。但是，由于无人机数据传输需求和地理位置的动态变化，易造成边缘服务器负载不均衡，致使巡检业务处理时延和网络能耗较高。为解决以上问题，提出基于深度强化学习的能源互联网智能巡检任务分配机制。首先，综合考虑无人机和边缘节点的运动轨迹、业务差异化的服务需求、边缘节点有限的服务能力等，建立面向时延、能耗等多目标联合优化的双层边缘网络任务卸载模型。进而，基于 Lyapunov 优化理论和双时间尺度机制，采用近端策略优化的深度强化学习算法，对固定边缘汇聚层和移动边缘接入层边缘节点间的连接关系和卸载策略进行求解。仿真结果表明，所提机制能够在保证系统稳定的情况下降低服务时延和系统能耗。

关键词: 巡检无人机, 任务卸载, 近端策略优化, 李雅普诺夫优化, 人工智能

Abstract:

In order to reduce the cost and improve efficiency of power line inspection, UAV (unmanned aerial vehicle), which use mobile edge computing technology to access and process service data, are used to inspect power lines in the energy internet.However, due to the dynamic changes of UAV data transmission demand and geographical location, the edge server load will be unbalanced, which causes higher service processing delay and network energy consumption.Thus, an intelligent inspection task allocation mechanism for energy internet based on deep reinforcement learning was proposed.First, a two-layer edge network task offloading model was established to archive joint optimization of multi-objectives, such as delay and energy consumption.It was designed by comprehensively considering the route of UAV and edge nodes, different demands of services and limited service capabilities of edge nodes.Furthermore, based on Lyapunov optimization theory and dual-time-scaled mechanism, proximal policy optimization algorithm based deep reinforcement learning was used to solve the connection relationship and offloading strategy of edge servers between fixed edge sink layer and mobile edge access layer.The simulation results show that, the proposed mechanism can reduce the service request delay and system energy consumption while ensuring the stability of system.

Key words: patrol UAV, ask offloading, proximal policy optimization, Lyapunov optimization, artificial intelligence

中图分类号:

TN92

徐思雅, 邢逸斐, 郭少勇, 杨超, 邱雪松, 孟洛明. 基于深度强化学习的能源互联网智能巡检任务分配机制[J]. 通信学报, 2021, 42(5): 191-204.

Siya XU, Yifei XING, Shaoyong GUO, Chao YANG, Xuesong QIU, Luoming MENG. Deep reinforcement learning based task allocation mechanism for intelligent inspection in energy Internet[J]. Journal on Communications, 2021, 42(5): 191-204.

图/表 10

图1

图2

表1

系统参数符号"

参数	含义
$t = {t_{1}, t_{2}, \dots, t_{n}}$	短时隙集合
$T = {T_{1}, T_{2}, \dots, T_{m}}$	长时隙集合
$U = {u_{1}, \dots, u_{i}, \dots, u_{x}}$	UAV的集合
$S = {s_{1}, \dots, s_{j}, \dots, s_{y}}$	Sub-MECS的集合
$M = {m_{1}, \dots, m_{k}, \dots, m_{z}}$	M-MECS的集合
$R_{i}^{u} (t)$	u _i在时隙t产生的任务请求
$x^{h} (t)$	时隙t的计算卸载策略
$λ^{sub} (t) = {λ_{1}^{sub} (t), \dots, λ_{y}^{sub} (t)}$	Sub-MECS 在时隙t的任务到达率
$λ^{main} (t) = {λ_{1}^{main} (t), \dots, λ_{z}^{main} (t)}$	M-MECS 在时隙t 的任务到达率
$P_{i}^{u, c}$	每时隙中UAV的运行功耗
$E_{i}^{u s} (t) 和 E_{i}^{u m} (t)$	UAV在时隙t向Sub-MECS和M-MECS的传输能耗
$E_{j}^{s, c} (t)$	Sub-MECS在时隙t处理能耗
$P_{k}^{m, c} (t)$	M-MECS在时隙t的运行功耗
$D_{i}^{u s} (t) 和 D_{i}^{u m} (t)$	UAV在时隙t向Sub-MECS和M-MECS的传输时延
$D_{j}^{s, c} (t)$	Sub-MECS在时隙t处理时延
$D_{k}^{m, c} (t)$	M-MECS在时隙t处理时延
$D_{k}^{cloud} (t)$	M-MECS在时隙云传输时延

表1

图3

图4

图5

图6

图7

图8

图9

参考文献 32

[1]	李君海, 张苗苗, 熊道洋 . 基于实时信息传输技术的无人机巡检管控平台[J]. 测绘与空间地理信息, 2020,43(6): 165-167,171.
	LI J H , ZHANG M M , XIONG D Y . An UAV power inspection management platform based on real-time message transmission technology[J]. Geomatics ＆ Spatial Information Technology, 2020,43(6): 165-167,171.
[2]	陈兰波 . 电力线路无人机巡检方案研究[J]. 科技与创新, 2020(11): 36-38,41.
	CHEN L B . Research on UAV patrol scheme of power line[J]. Science and Technology ＆ Innovation, 2020(11): 36-38,41.
[3]	严波, 林世忠, 张振威 ,等. 无人机电力巡检技术应用分析[J]. 自动化应用, 2019(12): 155-156.
	YAN B , LIN S Z , ZHANG Z W ,et al. Application analysis of UAV power inspection technology[J]. Automation Application, 2019(12): 155-156.
[4]	HAN D S , LI S J , CHEN Z X . Hybrid energy ratio allocation algorithm in a multi-base-station collaboration system[J]. IEEE Access, 2019,7: 147001-147009.
[5]	JEONG S , SIMEONE O , KANG J . Mobile edge computing via a UAV-mounted cloudlet:optimization of bit allocation and path planning[J]. IEEE Transactions on Vehicular Technology, 2018,67(3): 2049-2063.
[6]	WU Q Q , ZENG Y , ZHANG R . Joint trajectory and communication design for multi-UAV enabled wireless networks[J]. IEEE Transactions on Wireless Communications, 2018,17(3): 2109-2121.
[7]	WEI X L , TANG C G , FAN J H ,et al. Joint optimization of energy consumption and delay in cloud-to-thing continuum[J]. IEEE Internet of Things Journal, 2019,6(2): 2325-2337.
[8]	HU X Y , WONG K K , YANG K ,et al. UAV-assisted relaying and edge computing:scheduling and trajectory optimization[J]. IEEE Transactions on Wireless Communications, 2019,18(10): 4738-4752.
[9]	HUA M , WANG Y , ZHANG Z M ,et al. Power-efficient communication in UAV-aided wireless sensor networks[J]. IEEE Communications Letters, 2018,22(6): 1264-1267.
[10]	ZHANG T K , XU Y , LOO J ,et al. Joint computation and communication design for UAV-assisted mobile edge computing in IoT[J]. IEEE Transactions on Industrial Informatics, 2020,16(8): 5505-5516.
[11]	ATEYA A A , MUTHANNA A , KIRICHEK R ,et al. Energy-and latency-aware hybrid offloading algorithm for UAVs[J]. IEEE Access, 2019,7: 37587-37600.
[12]	王立科 . 无人机技术在电力巡检信息化管理中的应用研究[J]. 机电信息, 2019(27): 88-89.
	WANG L K . Research on application of UAV technology in information management of power patrol inspection[J]. Mechanical and Electrical Information, 2019(27): 88-89.
[13]	马青岷 . 无人机电力巡检及三维模型重建技术研究[D]. 济南:山东大学, 2017.
	MA Q M . Research of powerline inspection based on UAV and 3D model technology[D]. Jinan:Shandong University, 2017.
[14]	SHUO W , XING Z , ZHI Y ,et al. Cooperative edge computing with sleep control under nonuniform traffic in mobile edge networks[J]. IEEE Internet of Things Journal, 2019,6(3): 4295-4306.
[15]	SHAW J A . Radiometry and the Friis transmission equation[J]. American Journal of Physics, 2013,81(1): 33-37.
[16]	DOWNEY C . Understanding wireless range calculations[J]. Electronic Design, 2013.
[17]	MAO Y Y , ZHANG J , SONG S H ,et al. Power-delay tradeoff in multi-user mobile-edge computing systems[C]// 2016 IEEE Global Communications Conference. Piscataway:IEEE Press, 2016: 1-6.
[18]	BURD T D , BRODERSEN R W . Processor design for portable systems[J]. Journal of VLSI Signal Processing Systems for Signal,Image and Video Technology, 1996,13(2/3): 203-221.
[19]	FAN X B , WEBER W D , BARROSO L A . Power provisioning for a warehouse-sized computer[J]. ACM SIGARCH Computer Architecture News, 2007,35(2): 13-23.
[20]	CHEN L X , ZHOU S , XU J . Computation peer offloading for energy-constrained mobile edge computing in small-cell networks[J]. IEEE/ACM Transactions on Networking, 2018,26(4): 1619-1632.
[21]	YANG L , CAO J N , TANG S J ,et al. A framework for partitioning and execution of data stream applications in mobile cloud computing[C]// 2012 IEEE Fifth International Conference on Cloud Computing. Piscataway:IEEE Press, 2012: 794-802.
[22]	CHEN L X , XU J , ZHOU S . Computation peer offloading in mobile edge computing with energy budgets[C]// GLOBECOM 2017 - 2017 IEEE Global Communications Conference. Piscataway:IEEE Press, 2017: 1-6.
[23]	NEELY J M . Stochastic network optimization with application to communication and queueing systems[J]. Synthesis Lectures on Communication Networks, 2010,3(1): 211.
[24]	XU S Y , LIU Q C , GONG B ,et al. RJCC:reinforcement-learningbased joint communicational-and-computational resource allocation mechanism for smart city IoT[J]. IEEE Internet of Things Journal, 2020,7(9): 8059-8076.
[25]	HEESS N , DHRUVA T B , SRIRAM S ,et al. Emergence of locomotion behaviours in rich environments[J]. arXiv Preprint,arXiv:1707.02286, 2017.
[26]	SCHULMAN J , WOLSKI F , DHARIWAL P ,et al. Proximal policy optimization algorithms[J]. arXiv Preprint,arXiv:1707.06347, 2017.
[27]	王志夫 . 基于深度强化学习的双足机器人步行运动控制[D]. 济南:山东大学, 2020.
	WANG Z F . Deep reinforcement learning based walking control of biped robot[D]. Jinan:Shandong University, 2020.
[28]	王鸿涛 . 基于强化学习的机械臂自学习控制[D]. 哈尔滨:哈尔滨工业大学, 2019.
	WANG H T . Self learning control of mechanical arm based on reinforcement learning[D]. Harbin:Harbin Institute of Technology, 2019.
[29]	胡彦娟 . 移动边缘计算中任务卸载及资源分配算法研究[D]. 重庆:重庆邮电大学, 2020.
	HU Y J . Research on task offloading and resource allocation algorithm in mobile edge computing[D]. Chongqing:Chongqing University of Posts and Telecommunications, 2020.
[30]	GAO Z B , WEN B , HUANG L F ,et al. Q-learning-based power control for LTE enterprise femtocell networks[J]. IEEE Systems Journal, 2017,11(4): 2699-2707.
[31]	LI R , ZHANG C Y , PATRAS P ,et al. Learning driven mobility control of airborne base stations in emergency networks[J]. ACM SIGMETRICS Performance Evaluation Review, 2019,46(3): 163-166.
[32]	KWAK J , KIM Y , LEE J ,et al. DREAM:dynamic resource and task allocation for energy minimization in mobile cloud systems[J]. IEEE Journal on Selected Areas in Communications, 2015,33(12): 2510-2523.

基于深度强化学习的能源互联网智能巡检任务分配机制

Deep reinforcement learning based task allocation mechanism for intelligent inspection in energy Internet

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 32

相关文章 15

Metrics

推荐阅读 0

[1]	蒋丽, 谢胜利, 田辉. 面向数字孪生边缘网络的区块链分片及资源自适应优化机制[J]. 通信学报, 2023, 44(3): 12-23.
[2]	江沸菠, 彭于波, 董莉. 面向6G的深度图像语义通信模型[J]. 通信学报, 2023, 44(3): 198-208.
[3]	夏景明, 刘玉风, 谈玲. 基于蜂窝网络的多无人机能量消耗最优化算法研究[J]. 通信学报, 2023, 44(2): 185-197.
[4]	王敬宇, 庄子睿. 知识定义多模态网络按需服务体系研究[J]. 通信学报, 2022, 43(4): 71-82.
[5]	牛志升, 周盛, 孙宇璇. 面向“双碳”战略的绿色通信与网络：挑战与对策[J]. 通信学报, 2022, 43(2): 1-14.
[6]	苏新, 孟蕾蕾, 周一青, CELIMUGE Wu. 基于深度强化学习的海洋移动边缘计算卸载方法[J]. 通信学报, 2022, 43(10): 133-145.
[7]	吕品, 许嘉, 李陶深, 徐文彪. 面向自动驾驶的边缘计算技术研究综述[J]. 通信学报, 2021, 42(3): 190-208.
[8]	刘传宏, 郭彩丽, 杨洋, 冯春燕, 孙启政, 陈九九. 人工智能物联网中面向智能任务的语义通信方法[J]. 通信学报, 2021, 42(11): 97-108.
[9]	黄韬, 刘江, 汪硕, 张晨, 刘韵洁. 未来网络技术与发展趋势综述[J]. 通信学报, 2021, 42(1): 130-150.
[10]	伏玉笋,杨根科. 人工智能在移动通信中的应用：挑战与实践[J]. 通信学报, 2020, 41(9): 190-201.
[11]	梁应敞,谭俊杰,Dusit Niyato. 智能无线通信技术研究概况[J]. 通信学报, 2020, 41(7): 1-17.
[12]	赵临东,庄文芹,陈建新,周亮. 异构蜂窝网络中分层任务卸载：建模与优化[J]. 通信学报, 2020, 41(4): 34-44.
[13]	林艳, 闫帅, 张一晋, 李春国, 束锋. 基于交通流量预测的车联网双边拍卖边缘计算迁移方案[J]. 通信学报, 2020, 41(12): 205-214.
[14]	沈纲祥. 基于人工智能技术的光通信网络应用研究[J]. 通信学报, 2020, 41(1): 162-168.
[15]	孙鸿宇,何远,王基策,董颖,朱立鹏,王鹤,张玉清. 人工智能技术在安全漏洞领域的应用[J]. 通信学报, 2018, 39(8): 1-17.