基于多智能体强化学习的大规模灾后用户分布式覆盖优化

doi:10.11959/j.issn.1000-436x.2022131

Abstract

Abstract:

In order to quickly restore emergency communication services for large-scale post-disaster users, a distributed intellicise coverage optimization architecture based on multi-agent reinforcement learning (RL) was proposed, which could address the significant differences and dynamics of communication services caused by a large number of access users, and the difficulty of expansion caused by centralized algorithms.Specifically, a distributed k-sums clustering algorithm considering service differences of users was designed in the network characterization layer, which could make each unmanned aerial vehicle base station (UAV-BS) adjust the local networking natively and simply, and obtain states of cluster center for multi-agent RL.In the trajectory control layer, multi-agent soft actor critic (MASAC) with distributed-training-distributed-execution structure was designed for UAV-BS to control trajectory as intelligent nodes.Furthermore, ensemble learning and curriculum learning were integrated to improve the stability and convergence speed of training process.The simulation results show that the proposed distributed k-sums algorithm is superior to the k-means in terms of average load efficiency and clustering balance, and MASAC based trajectory control algorithm can effectively reduce communication interruptions and improve the spectrum efficiency, which outperforms the existing RL algorithms.

Key words: emergency communication, coverage optimization, multi-agent reinforcement learning, distributed training

CLC Number:

TN929.5

Wenjun XU, Silei WU, Fengyu WANG, Lan LIN, Guojun LI, Zhi ZHANG. Large-scale post-disaster user distributed coverage optimization based on multi-agent reinforcement learning[J]. Journal on Communications, 2022, 43(8): 1-16.

Figures/Tables 17

参数	含义
N ,M	受灾用户、无人机基站的数量
$N_{j}, M_{j}$	无人机基站 j可观测的用户、无人机基站数量
$κ_{1}, κ_{2}$	用户激活状态Beta分布的参数
$μ_{i}, σ_{i}$	用户i的传输业务类型分布的均值与标准差
$u_{i}, u_{j}$	用户i、无人机基站 j对应的簇中心用户
$f_{c}$	地面通信或空地通信的中心频率
$p_{i}, p_{u}, p_{j}$	用户i、簇中心用户u、无人机基站 j的位置
$P_{1}, P_{2}$	簇内用户、簇中心用户的传输功率
L ,G	路径损耗、信道增益
$N_{0}$	接收端的噪声功率
$d_{i}, D_{i}$	用户i的新传输任务、总传输任务的大小
$R_{i}, R_{j}$	用户i、无人机基站j的频谱效率
B	地面通信资源块的带宽
$n_{i, u}$	用户i占用的资源块数量
$N_{c}$	簇内用户负载阈值
η	平均负载效率
t₀	历史观测数据集存储的帧数
λ	用户优先参数
$\hat{μ}, \hat{σ}$	用户优先参数分布的均值、标准差
$y_{i, j}$	用户i是否处于簇 j的分簇标识
$g_{i_{1}, i_{2}}$	用户i₁对于用户i₂的不相似性度量
$s_{t}, a_{t}, r_{t}$	强化学习状态、动作、奖励函数
γ	折扣因子
θ	深度神经网络参数
α	最大熵强化学习温度因子
η	学习步长
W	集成学习神经网络的组数

参数	取值	参数	取值
用户数目N	500	观测用户数 $N_{j}$	$\frac{2 N}{M}$
邻近基站数 $M_{j}$	3	带宽B/MHz	1
空地频率 $f_{c}^{air}$ /GHz	4	地面频率 $f_{c}^{ground}$ /GHz	2
噪声 $N_{0}$ /(dBm·Hz^-1)	-174	无人机最大速度/(m·s^-1)	10
簇中心功率P₁/dBm	100	用户功率P₂/dBm	20
负载阈值N_c	5	总任务时长T₀/s	500
激活参数 $κ_{1}$	2	激活参数 $κ_{2}$	5
惩罚值 $ξ_{collision}$	500	惩罚值 $ξ_{outage}$	10
步长 $η_{1}$	0.001	步长 $η_{2, 3}$	0.000 1
步长 $η_{4, 5, 6}$	0.001	集成学习维度W	10
折扣因子γ	0.99	经验回放样本数	128

References 25

[1]	DEEPAK G C , LADAS A , SAMBO Y A ,et al. An overview of post-disaster emergency communication systems in the future networks[J]. IEEE Wireless Communications, 2019,26(6): 132-139.
[2]	GUO H Z , LI J Y , LIU J J ,et al. A survey on space-air-ground-sea integrated network security in 6G[J]. IEEE Communications Surveys＆ Tutorials, 2022,24(1): 53-87.
[3]	ZHOU Y Q , LIU L , WANG L ,et al. Service-aware 6G:an intelligent and open network based on the convergence of communication,computing and caching[J]. Digital Communications and Networks, 2020,6(3): 253-260.
[4]	ZHANG P , XU W J , GAO H ,et al. Toward wisdom-evolutionary and primitive-concise 6G:a new paradigm of semantic communication networks[J]. Engineering, 2022,8: 60-73.
[5]	张平, 许晓东, 韩书君 ,等. 智简无线网络赋能行业应用[J]. 北京邮电大学学报, 2020,43(6): 1-9.
	ZHANG P , XU X D , HAN S J ,et al. Entropy reduced mobile networks empowering industrial applications[J]. Journal of Beijing University of Posts and Telecommunications, 2020,43(6): 1-9.
[6]	ZHOU Y Q , TIAN L , LIU L ,et al. Fog computing enabled future mobile communication networks:a convergence of communication and computing[J]. IEEE Communications Magazine, 2019,57(5): 20-27.
[7]	KANG Z Y , YOU C S , ZHANG R . 3D placement for multi-UAV relaying:an iterative Gibbs-sampling and block coordinate descent optimization approach[J]. IEEE Transactions on Communications, 2021,69(3): 2047-2062.
[8]	YIN S X , LI L H , YU F R . Resource allocation and basestation placement in downlink cellular networks assisted by multiple wireless powered UAVs[J]. IEEE Transactions on Vehicular Technology, 2020,69(2): 2171-2184.
[9]	ZHANG Y X , CHENG W C . Trajectory and power optimization for multi-UAV enabled emergency wireless communications networks[C]// Proceedings of International Conference on Communications Workshops. Piscataway:IEEE Press, 2019: 1-6.
[10]	LI X , WANG Q , LIU J ,et al. Trajectory design and generalization for UAV enabled networks:a deep reinforcement learning approach[C]// Proceedings of Wireless Communications and Networking Conference. Piscataway:IEEE Press, 2020: 1-6.
[11]	LIU X , LIU Y W , CHEN Y . Reinforcement learning in multiple-UAV networks:deployment and movement design[J]. IEEE Transactions on Vehicular Technology, 2019,68(8): 8036-8049.
[12]	CHALLITA U , SAAD W , BETTSTETTER C . Interference management for cellular-connected UAVs:a deep reinforcement learning approach[J]. IEEE Transactions on Wireless Communications, 2019,18(4): 2125-2140.
[13]	ZHAO N , LIU Z H , CHENG Y Q . Multi-agent deep reinforcement learning for trajectory design and power allocation in multi-UAV networks[J]. IEEE Access, 8: 139670-139679.
[14]	QIN Z Q , LIU Z H , HAN G J ,et al. Distributed UAV-BSs trajectory optimization for user-level fair communication service with multi-agent deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2021,70(12): 12290-12301.
[15]	LOWE R , WU Y , TAMAR A ,et al. Multi-agent actor-critic for mixed cooperative-competitive environments[J]. arXiv Preprint,arXiv:1706.02275, 2017.
[16]	HAARNOJA T , TANG H R , ABBEEL P ,et al. Reinforcement learning with deep energy-based policies[J]. arXiv Preprint,arXiv:1702.08165, 2017.
[17]	NAVARRO-ORTIZ J , ROMERO-DIAZ P , SENDRA S ,et al. A survey on 5G usage scenarios and traffic models[J]. IEEE Communications Surveys ＆ Tutorials, 2020,22(2): 905-929.
[18]	3GPP. Technical specification group (TSG) RAN WG4; RF system scenarios:TR 25.942 v2.1.3[S]. 2000.
[19]	WANG L T , SUN L T , TOMIZUKA M ,et al. Socially-compatible behavior design of autonomous vehicles with verification on real human data[J]. IEEE Robotics and Automation Letters, 2021,6(2): 3421-3428.
[20]	PEI S , NIE F , WANG R ,et al. Efficient clustering based on a unified view of k-means and ratio-cut[J]. Advances in Neural Information Processing Systems, 2020,33: 14855-14866.
[21]	HAARNOJA T , ZHOU A , ABBEEL P ,et al. Soft actor-critic:off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]// International Conference on Machine Learning. New York:PMLR, 2018: 1861-1870.
[22]	HASSELT H V , GUEZ A , SILVER D . Deep reinforcement learning with double Q-learning[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto:AAAI Press, 2016: 2094-2100.
[23]	DONG X B , YU Z W , CAO W M ,et al. A survey on ensemble learning[J]. Frontiers of Computer Science, 2020,14(2): 241-258.
[24]	NARVEKAR S , PENG B , LEONETTI M ,et al. Curriculum learning for reinforcement learning domains:a framework and survey[J]. arXiv Preprint,arXiv:2003.04960, 2020.
[25]	SUTTON R S , BARTO A G . Reinforcement learning:an introduction[M]. Massachusetts: MIT Press, 1998.

Metrics

Recommended 0

No Suggested Reading articles found!

Large-scale post-disaster user distributed coverage optimization based on multi-agent reinforcement learning

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 25

Related Articles 6

Metrics

Recommended 0

[1]	Li WANG, Qing WEI, Lianming XU, Yuan SHEN, Ping ZHANG, Aiguo FEI. Research on low-energy-consumption deployment of emergency UAV network for integrated communication-navigating-sensing [J]. Journal on Communications, 2022, 43(7): 1-20.
[2]	Qingsong HU,Wei YANG,Enjie DING,Shiyin LI,Binghao LI. State-of-the-art and trend of emergency rescue communication technologies for coal mine [J]. Journal on Communications, 2019, 40(5): 163-179.
[3]	Fang-min LI,Le ZENG,Yi SHEN,Tao ZHANG. Design and implementation of a layer-II fast handoff protocol in emergency communication system [J]. Journal on Communications, 2017, 38(Z2): 8-16.
[4]	Lian-feng SHEN,Ya-ping ZHU,Zhao-ming DING,Feng YAN,Shu-guang DENG. Study on network reconfiguration algorithms in software-defined sensor networks [J]. Journal on Communications, 2016, 37(7): 38-49.
[5]	Zheng GUAN,Wen-hua QIAN,Mei-le YU. Cognitive network spectrum access for emergency communications [J]. Journal on Communications, 2012, 33(Z2): 182-188.
[6]	Wei WANG,Wen-hong ZHAO. Topology modeling and analysis of emergency communications network [J]. Journal on Communications, 2012, 33(Z1): 201-209.