面向巨型星座的智能负载均衡算法

doi:10.11959/j.issn.2096-8930.2023042

Abstract

Abstract:

To overcome the local traffic congestion caused by the huge number of satellites in mega-constellation, a load balancing algorithm based on multi-agent deep reinforcement learning was proposed.Firstly, the satellites in the mega constellation were divided into clusters to perform the distributed management of the mega constellation, which could reduce the overhead of whole network.Then, based on the coordinated multi-agent deep reinforcement learning model, routing planning, which could be individually operated by satellites in the mega constellation, was designed to achieve the intra-cluster coordination.Additionally, a cluster state compression mechanism with autoencoder was proposed to compress the state space and improve the efficiency of multi-agent deep reinforcement learning.Finally, simulation results showed that compared with the traditional single-task routing algorithm, the proposed algorithm could increase the transmission success rate by more than 40% and the proposed algorithm could efficiently avoid local traffic congestion.

Key words: mega constellation, multi-agent deep reinforcement learning, load balancing, routing algorithm

CLC Number:

TP393

Shuxin LUO, Chao ZHANG, Yong XIAO, Jianping LIU. Intelligent Load Balancing Algorithm of Mega Constellation[J]. Space-Integrated-Ground Information Networks, 2023, 4(4): 49-60.

Figures/Tables 12

References 34

[1]	SHENG M , ZHOU D , BAI W G ,et al. 6G service coverage with mega satellite constellations[J]. China Communications, 2022,19(1): 64-76.
[2]	阮永井, 胡敏, 云朝明 . 低轨巨型星座构型设计与控制研究进展与展望[J]. 中国空间科学技术, 2022,42(1): 1-15.
	RUAN Y J , HU M , YUN C M . Advances and prospects of the configuration design and control research of the LEO mega- constellations[J]. Chinese Space Science and Technology, 2022,42(1): 1-15.
[3]	CHAUDHRY A U , YANIKOMEROGLU H . Laser intersatellite links in a starlink constellation:a classification and analysis[J]. IEEE Vehicular Technology Magazine, 2021,16(2): 48-56.
[4]	RADHAKRISHNAN R , EDMONSON W W , AFGHAH F ,et al. Survey of inter-satellite communication for small satellite systems:physical layer to network layer view[J]. IEEE Communications Surveys ＆ Tutorials, 2016,18(4): 2442-2473.
[5]	YANG X . Low earth orbit (LEO) mega constellations:satellite and terrestrial integrated communication networks[D]. Guildford:University of Surrey, 2019.
[6]	陈全, 杨磊, 郭剑鸣 ,等. 低轨巨型星座网络:组网技术与研究现状[J]. 通信学报, 2022,43(5): 177-189.
	CHEN Q , YANG L , GUO J M ,et al. LEO mega-constellation network:networking technologies and state of the art[J]. Journal on Communications, 2022,43(5): 177-189.
[7]	周笛, 盛敏, 郝琪 ,等. 巨型星座系统的网络运维与资源管控技术[J]. 天地一体化信息网络, 2020,1(1): 26-35.
	ZHOU D , SHENG M , HAO Q ,et al. Network operation,maintenance and resource management in mega constellation system[J]. Space-Integrated-Ground Information Networks, 2020,1(1): 26-35.
[8]	CHEN Q , GIAMBENE G , YANG L ,et al. Analysis of inter-satellite link paths for LEO mega-constellation networks[J]. IEEE Transactions on Vehicular Technology, 2021,70(3): 2743-2755.
[9]	QU Z C , ZHANG G X , CAO H T ,et al. LEO satellite constellation for Internet of Things[J]. IEEE Access, 2017,5: 18391-18401.
[10]	LEE Y , CHOI J P . Connectivity analysis of mega-constellation satellite networks with optical intersatellite links[J]. IEEE Transactions on Aerospace and Electronic Systems, 2021,57(6): 4213-4226.
[11]	XIE H R , ZHAN Y F , ZENG G M ,et al. LEO mega-constellations for 6G global coverage:challenges and opportunities[J]. IEEE Access, 2021,9: 164223-164244.
[12]	TANG F L , ZHANG H T , YANG L T . Multipath cooperative routing with efficient acknowledgement for LEO satellite networks[J]. IEEE Transactions on Mobile Computing, 2019,18(1): 179-192.
[13]	巢孟愿 . 卫星网络多路径路由算法与切换策略研究[D]. 杭州:浙江大学, 2014.
	CHAO M Y . Research on multipath routing algorithm and handover strategy in satellite networks[D]. Hangzhou:Zhejiang University, 2014.
[14]	ZENG G M , ZHAN Y F , PAN X H . Failure-tolerant and low-latency telecomm and in mega-constellations:the redundant multi-path routing[J]. IEEE Access, 2021,9: 34975-34985.
[15]	LIU Y L , ZHU L D . A suboptimal routing algorithm for massive LEO satellite networks[C]// Proceedings of 2018 International Symposium on Networks,Computers and Communications (ISNCC). Piscataway:IEEE Press, 2018: 1-5.
[16]	LI J , LU H C , XUE K P ,et al. Temporal netgrid model-based dynamic routing in large-scale small satellite networks[J]. IEEE Transactions on Vehicular Technology, 2019,68(6): 6009-6021.
[17]	QI X X , ZHANG B , QIU Z L . A distributed survivable routing algorithm for mega-constellations with inclined orbits[J]. IEEE Access, 2020,8: 219199-219213.
[18]	倪少杰, 岳洋, 左勇 ,等. 卫星网络路由技术现状及展望[J]. 电子与信息学报, 2023,45(2): 383-395.
	NI S J , YUE Y , ZUO Y ,et al. The status quo and prospect of satellite network routing technology[J]. Journal of Electronics ＆ Information Technology, 2023,45(2): 383-395.
[19]	郑爽, 张兴, 王文博 . 低轨卫星通信网络路由技术综述[J]. 天地一体化信息网络, 2022,3(3): 97-105.
	ZHENG S , ZHANG X , WANG W B . Survey of low earth orbit satellite communication network routing technology[J]. Space- Integrated- Ground Information Networks, 2022,3(3): 97-105.
[20]	石晓东, 李勇军, 赵尚弘 ,等. 基于SDN的卫星网络多QoS目标优化路由算法[J]. 系统工程与电子技术, 2020,42(6): 1395-1401.
	SHI X D , LI Y J , ZHAO S H ,et al. Multi-QoS objective optimization routing algorithm of satellite network based on SDN[J]. Systems Engineering and Electronics, 2020,42(6): 1395-1401.
[21]	LIU J , ZHANG X Y , ZHANG R ,et al. Reliable and low-overhead clustering in LEO small satellite networks[J]. IEEE Internet of Things Journal, 2022,9(16): 14844-14856.
[22]	TANG F X , HOFNER H , KATO N ,et al. A deep reinforcement learning-based dynamic traffic offloading in space-air-ground integrated networks (SAGIN)[J]. IEEE Journal on Selected Areas in Communications, 2022,40(1): 276-289.
[23]	QI X G , MA J L , WU D ,et al. A survey of routing techniques for satellite networks[J]. Journal of Communications and Information Networks, 2016,1(4): 66-85.
[24]	沙鹏翔 . 基于分簇的巨型低轨星座高效管理方法研究[D]. 南京:南京大学, 2021.
	SHA P X . Research on efficient management method of giant LEO constellation based on clustering[D]. Nanjing:Nanjing University, 2021.
[25]	刘全, 翟建伟, 章宗长 ,等. 深度强化学习综述[J]. 计算机学报, 2018,41(1): 1-27.
	LIU Q , ZHAI J W , ZHANG Z C ,et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018,41(1): 1-27.
[26]	裴培, 何绍溟, 王江 ,等. 一种深度强化学习制导控制一体化算法[J]. 宇航学报, 2021,42(10): 1293-1304.
	PEI P , HE S M , WANG J ,et al. Integrated guidance and control for missile using deep reinforcement learning[J]. Journal of Astronautics, 2021,42(10): 1293-1304.
[27]	NAZARI M , OROOJLOOY A , TAKá? M ,et al. Reinforcement learning for solving the vehicle routing problem[C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York:ACM, 2018: 9861-9871.
[28]	孙彧, 曹雷, 陈希亮 ,等. 多智能体深度强化学习研究综述[J]. 计算机工程与应用, 2020,56(5): 13-24.
	SUN Y , CAO L , CHEN X L ,et al. Overview of multi-agent deep reinforcement learning[J]. Computer Engineering and Applications, 2020,56(5): 13-24.
[29]	ZHANG K , YANG Z , BA?AR T . Multi-agent reinforcement learning:A selective overview of theories and algorithms[J]. Handbook of reinforcement learning and control, 2021: 321-384.
[30]	RASHID T , SAMVELYAN M , DE WITT C S ,et al. QMIX:monotonic value function factorisation for deep multi-agent reinforcement learning[EB]. 2018.
[31]	陈亮, 梁宸, 张景异 ,等. Actor-Critic 框架下一种基于改进DDPG 的多智能体强化学习算法[J]. 控制与决策, 2021,36(1): 75-82.
	CHEN L , LIANG C , ZHANG J Y ,et al. A multi-agent reinforcement learning algorithm based on improved DDPG in Actor-Critic framework[J]. Control and Decision, 2021,36(1): 75-82.
[32]	刘建伟, 高峰, 罗雄麟 . 基于值函数和策略梯度的深度强化学习综述[J]. 计算机学报, 2019,42(6): 1406-1438.
	LIU J W , GAO F , LUO X L . Survey of deep reinforcement learning based on value function and policy gradient[J]. Chinese Journal of Computers, 2019,42(6): 1406-1438.
[33]	韩冲, 王俊丽, 吴雨茜 ,等. 基于神经进化的深度学习模型研究综述[J]. 电子学报, 2021,49(2): 372-379.
	HAN C , WANG J L , WU Y X ,et al. A review of deep learning models based on neuroevolution[J]. Acta Electronica Sinica, 2021,49(2): 372-379.
[34]	LIU J H , ZHAO B K , XIN Q ,et al. DRL-ER:an intelligent energy-aware routing protocol with guaranteed delay bounds in satellite mega-constellations[J]. IEEE Transactions on Network Science and Engineering, 2021,8(4): 2872-2884.

Metrics

Recommended 0

No Suggested Reading articles found!

仿真参数	值
经验池大小	200×N×M
TTL	2×N×M
折扣因子	0.99
更新周期	40
DRQN隐藏层维度	4
超网络隐藏层维度	分簇大小
起始探索概率	1
中止探索概率	0.1
学习率	0.01

算法	星座规模
算法	6× 6	12× 12	24× 24	32× 32
所提算法	85.32%	72.84%	56.49%	39.19%
DQNRA	82.95%	70.21%	52.91%	29.33%
DSRA	77.18%	65.54%	44.68%	25.74%

Intelligent Load Balancing Algorithm of Mega Constellation

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 34

Related Articles 6

Metrics

Recommended 0

[1]	Siyue CHENG, Haoran LI, Weigang BAI, Di ZHOU, Yan ZHU. Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning [J]. Space-Integrated-Ground Information Networks, 2023, 4(1): 12-22.
[2]	Yang LIU, Di ZHOU, Min SHENG, Jiandong LI, Shiguang HAO, Xiaotian ZHENG. Multi-Ground Station Collaborative Measurement and Control Technology for Giant Constellation System [J]. Space-Integrated-Ground Information Networks, 2023, 4(1): 2-11.
[3]	Mo SHA, Kaixuan SUN, Huasen HE, Wenke YUAN, Zhicheng YE, Jian YANG. Software Defi ned Satellite Network Assisted Multi-Domain Routing Strategy for Aeronautical Networks [J]. Space-Integrated-Ground Information Networks, 2022, 3(3): 56-64.
[4]	Yijie WANG, Wei ZHAO, Fei CHENG, Wen CHEN, Anjie CAO. Large Scale LEO Satellite Internet Routing Algorithm Based on Load Balancing [J]. Space-Integrated-Ground Information Networks, 2022, 3(1): 27-34.
[5]	Hai LI, Shanghong ZHAO, Yongjun LI, Cong PENG. Research on Wavelength Algorithm for Cross-Layer Routing in Software Defi ned Satellite Optical Network [J]. Space-Integrated-Ground Information Networks, 2021, 2(3): 76-80.
[6]	Di ZHOU,Min SHENG,Qi HAO,Jiandong LI,Jianping LIU. Network Operation,Maintenance and Resource Management in Mega Constellation System [J]. Space-Integrated-Ground Information Networks, 2020, 1(1): 26-35.