面向巨型星座的智能负载均衡算法

doi:10.11959/j.issn.2096-8930.2023042

摘要/Abstract

摘要：

针对巨型星座中卫星数量众多容易引发局部拥塞的问题，提出基于协作多智能体深度强化学习的巨型星座负载均衡算法。首先对巨型星座中的卫星进行分簇设计，实现巨型星座的分布式管理，降低网络管理开销。然后，利用Q-混合多智能体神经网络深度强化学习设计各卫星自主决策的路由规划方案，实现多传输任务的簇内协同。此外，提出基于自动编码器的簇状态压缩机制，提高多智能体深度强化学习的效率。仿真结果表明，所提算法相比于传统的单任务路由算法，传输成功率可提升40%以上，证明所提算法能够避免局部拥塞的发生，提高巨型星座的传输效率。

关键词: 巨型星座, 多智能体深度强化学习, 负载均衡, 路由算法

Abstract:

To overcome the local traffic congestion caused by the huge number of satellites in mega-constellation, a load balancing algorithm based on multi-agent deep reinforcement learning was proposed.Firstly, the satellites in the mega constellation were divided into clusters to perform the distributed management of the mega constellation, which could reduce the overhead of whole network.Then, based on the coordinated multi-agent deep reinforcement learning model, routing planning, which could be individually operated by satellites in the mega constellation, was designed to achieve the intra-cluster coordination.Additionally, a cluster state compression mechanism with autoencoder was proposed to compress the state space and improve the efficiency of multi-agent deep reinforcement learning.Finally, simulation results showed that compared with the traditional single-task routing algorithm, the proposed algorithm could increase the transmission success rate by more than 40% and the proposed algorithm could efficiently avoid local traffic congestion.

Key words: mega constellation, multi-agent deep reinforcement learning, load balancing, routing algorithm

中图分类号:

TP393

罗树欣, 张超, 肖勇, 刘建平. 面向巨型星座的智能负载均衡算法[J]. 天地一体化信息网络, 2023, 4(4): 49-60.

Shuxin LUO, Chao ZHANG, Yong XIAO, Jianping LIU. Intelligent Load Balancing Algorithm of Mega Constellation[J]. Space-Integrated-Ground Information Networks, 2023, 4(4): 49-60.

图/表 12

图1

图2

图3

图4

图5

图6

图7

表1

图8

图9

表2

图10

参考文献 34

[1]	SHENG M , ZHOU D , BAI W G ,et al. 6G service coverage with mega satellite constellations[J]. China Communications, 2022,19(1): 64-76.
[2]	阮永井, 胡敏, 云朝明 . 低轨巨型星座构型设计与控制研究进展与展望[J]. 中国空间科学技术, 2022,42(1): 1-15.
	RUAN Y J , HU M , YUN C M . Advances and prospects of the configuration design and control research of the LEO mega- constellations[J]. Chinese Space Science and Technology, 2022,42(1): 1-15.
[3]	CHAUDHRY A U , YANIKOMEROGLU H . Laser intersatellite links in a starlink constellation:a classification and analysis[J]. IEEE Vehicular Technology Magazine, 2021,16(2): 48-56.
[4]	RADHAKRISHNAN R , EDMONSON W W , AFGHAH F ,et al. Survey of inter-satellite communication for small satellite systems:physical layer to network layer view[J]. IEEE Communications Surveys ＆ Tutorials, 2016,18(4): 2442-2473.
[5]	YANG X . Low earth orbit (LEO) mega constellations:satellite and terrestrial integrated communication networks[D]. Guildford:University of Surrey, 2019.
[6]	陈全, 杨磊, 郭剑鸣 ,等. 低轨巨型星座网络:组网技术与研究现状[J]. 通信学报, 2022,43(5): 177-189.
	CHEN Q , YANG L , GUO J M ,et al. LEO mega-constellation network:networking technologies and state of the art[J]. Journal on Communications, 2022,43(5): 177-189.
[7]	周笛, 盛敏, 郝琪 ,等. 巨型星座系统的网络运维与资源管控技术[J]. 天地一体化信息网络, 2020,1(1): 26-35.
	ZHOU D , SHENG M , HAO Q ,et al. Network operation,maintenance and resource management in mega constellation system[J]. Space-Integrated-Ground Information Networks, 2020,1(1): 26-35.
[8]	CHEN Q , GIAMBENE G , YANG L ,et al. Analysis of inter-satellite link paths for LEO mega-constellation networks[J]. IEEE Transactions on Vehicular Technology, 2021,70(3): 2743-2755.
[9]	QU Z C , ZHANG G X , CAO H T ,et al. LEO satellite constellation for Internet of Things[J]. IEEE Access, 2017,5: 18391-18401.
[10]	LEE Y , CHOI J P . Connectivity analysis of mega-constellation satellite networks with optical intersatellite links[J]. IEEE Transactions on Aerospace and Electronic Systems, 2021,57(6): 4213-4226.
[11]	XIE H R , ZHAN Y F , ZENG G M ,et al. LEO mega-constellations for 6G global coverage:challenges and opportunities[J]. IEEE Access, 2021,9: 164223-164244.
[12]	TANG F L , ZHANG H T , YANG L T . Multipath cooperative routing with efficient acknowledgement for LEO satellite networks[J]. IEEE Transactions on Mobile Computing, 2019,18(1): 179-192.
[13]	巢孟愿 . 卫星网络多路径路由算法与切换策略研究[D]. 杭州:浙江大学, 2014.
	CHAO M Y . Research on multipath routing algorithm and handover strategy in satellite networks[D]. Hangzhou:Zhejiang University, 2014.
[14]	ZENG G M , ZHAN Y F , PAN X H . Failure-tolerant and low-latency telecomm and in mega-constellations:the redundant multi-path routing[J]. IEEE Access, 2021,9: 34975-34985.
[15]	LIU Y L , ZHU L D . A suboptimal routing algorithm for massive LEO satellite networks[C]// Proceedings of 2018 International Symposium on Networks,Computers and Communications (ISNCC). Piscataway:IEEE Press, 2018: 1-5.
[16]	LI J , LU H C , XUE K P ,et al. Temporal netgrid model-based dynamic routing in large-scale small satellite networks[J]. IEEE Transactions on Vehicular Technology, 2019,68(6): 6009-6021.
[17]	QI X X , ZHANG B , QIU Z L . A distributed survivable routing algorithm for mega-constellations with inclined orbits[J]. IEEE Access, 2020,8: 219199-219213.
[18]	倪少杰, 岳洋, 左勇 ,等. 卫星网络路由技术现状及展望[J]. 电子与信息学报, 2023,45(2): 383-395.
	NI S J , YUE Y , ZUO Y ,et al. The status quo and prospect of satellite network routing technology[J]. Journal of Electronics ＆ Information Technology, 2023,45(2): 383-395.
[19]	郑爽, 张兴, 王文博 . 低轨卫星通信网络路由技术综述[J]. 天地一体化信息网络, 2022,3(3): 97-105.
	ZHENG S , ZHANG X , WANG W B . Survey of low earth orbit satellite communication network routing technology[J]. Space- Integrated- Ground Information Networks, 2022,3(3): 97-105.
[20]	石晓东, 李勇军, 赵尚弘 ,等. 基于SDN的卫星网络多QoS目标优化路由算法[J]. 系统工程与电子技术, 2020,42(6): 1395-1401.
	SHI X D , LI Y J , ZHAO S H ,et al. Multi-QoS objective optimization routing algorithm of satellite network based on SDN[J]. Systems Engineering and Electronics, 2020,42(6): 1395-1401.
[21]	LIU J , ZHANG X Y , ZHANG R ,et al. Reliable and low-overhead clustering in LEO small satellite networks[J]. IEEE Internet of Things Journal, 2022,9(16): 14844-14856.
[22]	TANG F X , HOFNER H , KATO N ,et al. A deep reinforcement learning-based dynamic traffic offloading in space-air-ground integrated networks (SAGIN)[J]. IEEE Journal on Selected Areas in Communications, 2022,40(1): 276-289.
[23]	QI X G , MA J L , WU D ,et al. A survey of routing techniques for satellite networks[J]. Journal of Communications and Information Networks, 2016,1(4): 66-85.
[24]	沙鹏翔 . 基于分簇的巨型低轨星座高效管理方法研究[D]. 南京:南京大学, 2021.
	SHA P X . Research on efficient management method of giant LEO constellation based on clustering[D]. Nanjing:Nanjing University, 2021.
[25]	刘全, 翟建伟, 章宗长 ,等. 深度强化学习综述[J]. 计算机学报, 2018,41(1): 1-27.
	LIU Q , ZHAI J W , ZHANG Z C ,et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018,41(1): 1-27.
[26]	裴培, 何绍溟, 王江 ,等. 一种深度强化学习制导控制一体化算法[J]. 宇航学报, 2021,42(10): 1293-1304.
	PEI P , HE S M , WANG J ,et al. Integrated guidance and control for missile using deep reinforcement learning[J]. Journal of Astronautics, 2021,42(10): 1293-1304.
[27]	NAZARI M , OROOJLOOY A , TAKá? M ,et al. Reinforcement learning for solving the vehicle routing problem[C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York:ACM, 2018: 9861-9871.
[28]	孙彧, 曹雷, 陈希亮 ,等. 多智能体深度强化学习研究综述[J]. 计算机工程与应用, 2020,56(5): 13-24.
	SUN Y , CAO L , CHEN X L ,et al. Overview of multi-agent deep reinforcement learning[J]. Computer Engineering and Applications, 2020,56(5): 13-24.
[29]	ZHANG K , YANG Z , BA?AR T . Multi-agent reinforcement learning:A selective overview of theories and algorithms[J]. Handbook of reinforcement learning and control, 2021: 321-384.
[30]	RASHID T , SAMVELYAN M , DE WITT C S ,et al. QMIX:monotonic value function factorisation for deep multi-agent reinforcement learning[EB]. 2018.
[31]	陈亮, 梁宸, 张景异 ,等. Actor-Critic 框架下一种基于改进DDPG 的多智能体强化学习算法[J]. 控制与决策, 2021,36(1): 75-82.
	CHEN L , LIANG C , ZHANG J Y ,et al. A multi-agent reinforcement learning algorithm based on improved DDPG in Actor-Critic framework[J]. Control and Decision, 2021,36(1): 75-82.
[32]	刘建伟, 高峰, 罗雄麟 . 基于值函数和策略梯度的深度强化学习综述[J]. 计算机学报, 2019,42(6): 1406-1438.
	LIU J W , GAO F , LUO X L . Survey of deep reinforcement learning based on value function and policy gradient[J]. Chinese Journal of Computers, 2019,42(6): 1406-1438.
[33]	韩冲, 王俊丽, 吴雨茜 ,等. 基于神经进化的深度学习模型研究综述[J]. 电子学报, 2021,49(2): 372-379.
	HAN C , WANG J L , WU Y X ,et al. A review of deep learning models based on neuroevolution[J]. Acta Electronica Sinica, 2021,49(2): 372-379.
[34]	LIU J H , ZHAO B K , XIN Q ,et al. DRL-ER:an intelligent energy-aware routing protocol with guaranteed delay bounds in satellite mega-constellations[J]. IEEE Transactions on Network Science and Engineering, 2021,8(4): 2872-2884.

仿真参数	值
经验池大小	200×N×M
TTL	2×N×M
折扣因子	0.99
更新周期	40
DRQN隐藏层维度	4
超网络隐藏层维度	分簇大小
起始探索概率	1
中止探索概率	0.1
学习率	0.01

算法	星座规模
算法	6× 6	12× 12	24× 24	32× 32
所提算法	85.32%	72.84%	56.49%	39.19%
DQNRA	82.95%	70.21%	52.91%	29.33%
DSRA	77.18%	65.54%	44.68%	25.74%