基于深度强化学习的无人机自主部署及能效优化策略

doi:10.11959/j.issn.2096-3750.2019.00106

物联网学报 ›› 2019, Vol. 3 ›› Issue (2): 47-55.doi: 10.11959/j.issn.2096-3750.2019.00106

基于深度强化学习的无人机自主部署及能效优化策略

周毅^1,²,马晓勇^1,²,郜富晓^1,²,李伟^1,²,承楠³,路宁⁴

¹ 河南大学计算机与信息工程学院，河南开封 475004
² 河南省车联网协同技术国际联合实验室，河南开封 475004
³ 滑铁卢大学电气与计算机工程学院，安大略滑铁卢 N2L 3G1
⁴ 汤普森河大学计算机科学系，不列颠哥伦比亚坎路普斯 V2C 0C8

修回日期:2019-05-10 出版日期:2019-06-30 发布日期:2019-07-17
作者简介:周毅（1981- ），男，河南信阳人，河南大学副教授、博士生导师，主要研究方向为车联网、空地协同组网、平行增强学习、协作机器人等。|马晓勇（1993- ），男，河南洛阳人，河南大学硕士生，主要研究方向为无人机组网、边缘计算等。|郜富晓（1992- ），女，河南洛阳人，河南大学硕士生，主要研究方向为空地协同组网、深度学习等。|李伟（1979- ），女，河南济源人，河南大学副教授，主要研究方向为车联网优化控制、协作通信等。|承楠（1987- ），男，辽宁锦州人，滑铁卢大学在站博士后，主要研究方向为车联网、人工智能、空地协同等。|路宁（1984- ），男，山西长治人，汤普森河大学助理教授，主要研究方向为车联网、移动边缘计算等。
基金资助:
国家自然科学基金资助项目(61701170);河南省国际科技合作项目(182102410050);河南省青年骨干教师计划(2016GGJS-018)

Autonomous deployment and energy efficiency optimization strategy of UAV based on deep reinforcement learning

Yi ZHOU^1,²,Xiaoyong MA^1,²,Fuxiao GAO^1,²,Wei LI^1,²,Nan CHENG³,Ning LU⁴

¹ School of Computer and Information Engineering,Henan University,Kaifeng 475004,China
² International Joint Research Laboratory for Cooperative Vehicular Networks of Henan,Kaifeng 475004,China
³ Department of Electrical and Computer Engineering,University of Waterloo,Waterloo N2L 3G1,Canada
⁴ Department of Computing Science,Thompson Rivers University,Kamloops V2C 0C8,Canada

Revised:2019-05-10 Online:2019-06-30 Published:2019-07-17
Supported by:
The National Natural Science Foundation of China(61701170);Henan International Science ＆ Technology Cooperation Program(182102410050);Henan Young Scholar Promotion Program(2016GGJS-018)

摘要/Abstract

摘要：

利用无人机组建空中移动基站，可为地面终端用户提供更灵活、高效的接入服务。受无人机覆盖范围和有限能量的约束，研究如何建立快速、高效、节能的空地协同网络至关重要，无人机需要根据复杂动态场景进行最优覆盖部署，同时要减少部署过程中的路径损耗和能量消耗。基于深度强化学习提出了无人机自主部署和能效优化策略，建立无人机覆盖状态集合，以能效作为奖励函数，利用深度神经网络和Q-learning引导无人机自主决策，部署最佳位置。仿真结果表明，该方法的部署时间能够有效减少60%，能耗可降低10%~20%。

关键词: 空地协作组网, 无人机, 自主部署, 能效优化, 深度强化学习

Abstract:

Utilizing a UAV to build aerial mobile small cell can provide more flexible and efficient access services for ground terminal users.Constrained by the coverage and limited energy of the UAV,it is necessary to study how to build a fast,efficient and energy-saving air-ground collaborative network.To deal with complex dynamic scenarios,the UAV needs to deploy an optimal coverage position,and meanwhile reduce both path loss and energy consumption in the deployment process.Based on the deep reinforcement learning,a strategy of autonomous UAV deployment and efficiency optimization was proposed.The coverage state set of UAV was established,and the energy efficiency was used as a reward function.Depth neural network and Q-learning were used to guide UAV to make autonomous decision and deploy the optimal position.The simulation results show that the deployment time of the proposed method can be effectively reduced by 60%,while the energy consumption can be reduced by 10%~20%.

Key words: aerial-ground cooperative networking, unmanned aerial vehicle, autonomous deployment, efficiency optimization, deep reinforcement learning

中图分类号:

TP393

周毅,马晓勇,郜富晓,李伟,承楠,路宁. 基于深度强化学习的无人机自主部署及能效优化策略[J]. 物联网学报, 2019, 3(2): 47-55.

Yi ZHOU,Xiaoyong MA,Fuxiao GAO,Wei LI,Nan CHENG,Ning LU. Autonomous deployment and energy efficiency optimization strategy of UAV based on deep reinforcement learning[J]. Chinese Journal on Internet of Things, 2019, 3(2): 47-55.

图/表 7

图1

图2

图3

图4

图5

图6

图7

参考文献 20

[1]	LU N , ZHOU Y , SHI C ,et al. Planning while flying:a measurement-aided dynamic planning of drone small cells[J]. IEEE Internet of Things Journal, 2019,6(2): 2693-2705.
[2]	BOR-YALINIZ R , EL-KEYI A , YANIKOMEROGLU H ,et al. Efficient 3D placement of an aerial base station in next generation cellular networks[C]// 2016 IEEE International Conference on Communications(ICC). IEEE, 2016: 1-5.
[3]	ZHOU Y , CHENG N , LU N ,et al. Multi-UAV aided networks:aerial-ground cooperative vehicular networking architecture[J]. IEEE Vehicular Technology Magazine, 2015,10(4): 36-44.
[4]	CHENG N , XU W , SHI W ,et al. Air-ground integrated mobile edge networks:architecture,challenges and opportunities[J]. IEEE Communications Magazine, 2018,56(8): 26-32.
[5]	MOZAFFARI M , SAAD W , BENNIS M ,et al. Unmanned aerial vehicle with underlaid device-to-device communications:performance and tradeoffs[J]. IEEE Transactions on Wireless Communications, 2016,15(6): 3949-3963.
[6]	ZENG Y , ZHANG R . Energy-efficient UAV communication with trajectory optimization[J]. IEEE Transactions on Wireless Communications, 2017,16(6): 3747-3760.
[7]	AL-HOURANI A , KANDEEPAN S , JAMALIPOUR A . Modeling air-to-ground path loss for low altitude platforms in urban environments[C]// 2014 IEEE Global Communications Conference. IEEE, 2014: 2898-2904.
[8]	LYU J , ZENG Y , ZHANG R ,et al. Placement optimization of UAV-mounted mobile base stations[C]// IEEE Communications Letters. 2017,21(3): 604-607.
[9]	AL-HOURANI A , KANDEEPAN S , LARDNER S . Optimal LAP altitude for maximum coverage[J]. IEEE Wireless Communications, 2014,3: 569-572.
[10]	KALANTARI E , YANIKOMEROGLU H , YONGACOGLU A . On the number and 3D placement of drone base stations in wireless cellular networks[C]// 2016 IEEE 84th Vehicular Technology Conference (VTC-Fall). IEEE, 2016: 1-6.
[11]	ALZENAD M , EL-KEYI A , LAGUM F ,et al. 3D placement of an unmanned aerial vehicle base station (UAV-BS) for energy-efficient maximal coverage[C]// IEEE Wireless Communications Letters, 2017,6(4): 434-437.
[12]	RUAN L , WANG J , CHEN J ,et al. Energy-efficient multi-UAV coverage deployment in UAV networks:a game-theoretic framework[J]. China Communications, 2018,15(10): 194-209.
[13]	CHEN Y , LI N , WANG C ,et al. A 3D placement of unmanned aerial vehicle base station based on multi-population genetic algorithm for maximizing users with different QoS requirements[C]// 2018 IEEE 18th International Conference on Communication Technology (ICCT). IEEE, 2018: 967-972.
[14]	LAI C , CHEN C , WANG L . On-demand density-aware UAV base station 3D placement for arbitrarily distributed users with guaranteed data rates[C]// IEEE Wireless Communications Letters. IEEE, 2019:1.
[15]	WANG L , HU B , CHEN S . Energy efficient placement of a drone base station for minimum required transmit power[C]// IEEE Wireless Communications Letters. IEEE, 2018.
[16]	SHAKHATREH H , KHREISHAH A , ALSARHAN A ,et al. Efficient 3D placement of a UAV using particle swarm optimization[C]// 2017 8th International Conference on Information and Communication Systems (ICICS). IEEE, 2017: 258-263.
[17]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[18]	XU Y , LIU Z , WANG X . Monocular vision based autonomous landing of quadrotor through deep reinforcement learning[C]// 2018 37th Chinese Control Conference (CCC). IEEE, 2018: 10014-10019.
[19]	ZHANG Y , SUN P , YIN Y ,et al. Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning[C]// 2018 IEEE Intelligent Vehicles Symposium(IV). IEEE, 2018: 1251-1256.
[20]	GOODFELLOW I , BENGIO Y , COURVILLE A . Deep learning[M]. Cambridge: MIT PressPress, 2016.

基于深度强化学习的无人机自主部署及能效优化策略

Autonomous deployment and energy efficiency optimization strategy of UAV based on deep reinforcement learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 20

相关文章 15

Metrics

推荐阅读 0

[1]	王志宏, 冷甦鹏, 熊凯. 面向无人机集群协同感知的多智能体资源分配策略[J]. 物联网学报, 2023, 7(1): 18-26.
[2]	廖岑卉珊, 陈俊彦, 梁观平, 谢小兰, 卢小烨. 基于深度强化学习的SDN服务质量智能优化算法[J]. 物联网学报, 2023, 7(1): 73-82.
[3]	俞汉清, 林艳, 贾林琼, 李强, 张一晋. 面向多目标救援的通信受限无人机集群分布式策略[J]. 物联网学报, 2022, 6(3): 103-112.
[4]	李茜雯, 陈健锋, 崔苗, 张广驰. 可充电无人机辅助数据采集系统的飞行路线与通信调度优化[J]. 物联网学报, 2022, 6(3): 113-123.
[5]	罗梓珲, 江呈羚, 刘亮, 郑霄龙, 马华东. 基于深度强化学习的智能车间调度方法研究[J]. 物联网学报, 2022, 6(1): 53-64.
[6]	王巍, 谷壬倩, 彭力, 赵继军, 魏忠诚, 常存喜. 基于无人机的物联网空基中继鲁棒优化[J]. 物联网学报, 2022, 6(1): 101-112.
[7]	王巍, 梁雅静, 彭力, 魏忠诚, 赵继军. 设备接入受限的UAV空基应急物联网节点分簇部署研究[J]. 物联网学报, 2021, 5(3): 97-105.
[8]	梅海波, 杨鲲, 范新宇. 基于深度增强学习的无人机赋能雾无线电接入网络的能效优化[J]. 物联网学报, 2021, 5(2): 48-59.
[9]	林椿珉, 曾烈康, 陈旭. 边缘智能驱动的高能效无人机自主导航算法研究[J]. 物联网学报, 2021, 5(2): 87-96.
[10]	嵇介曲, 朱琨, 易畅言, 王然. 多无人机辅助移动边缘计算中的任务卸载和轨迹优化[J]. 物联网学报, 2021, 5(1): 27-35.
[11]	牟治宇,张煜,范典,刘君,高飞飞. 基于深度强化学习的无人机数据采集和路径规划研究[J]. 物联网学报, 2020, 4(3): 42-51.
[12]	刘亚林,戴弘宁,王曲北剑. 无人机辅助的物联网通信技术及其应用[J]. 物联网学报, 2019, 3(4): 48-55.
[13]	丁瑞金,高飞飞,邢玲. 基于深度强化学习的物联网智能路由策略[J]. 物联网学报, 2019, 3(2): 56-63.
[14]	宋庆恒,郑福春. 基于无人机的物联网无线通信的潜力与方法[J]. 物联网学报, 2019, 3(1): 82-89.
[15]	郭芳,张玉艳,赵龙,郑侃,王文博. 基于无人机辅助的V2I无线传输系统性能优化[J]. 物联网学报, 2017, 1(2): 46-53.