基于深度强化学习的卫星互联网路由优化研究

doi:10.11959/j.issn.2096-8930.2022033

Abstract

Abstract:

With the rapid development of satellite communication, the satellite internet is one of the core technologies of 6G network to realize global coverage, full-time access and full scene service.The high dynamics and limited capacity of satellite network lead to a series of management and control challenges such as heterogeneous network management, dynamic resource allocation and so on.Since the machine learning-based technologies have strength in network design, the intelligent architecture of software-defi ned satellite internet was put forward.In view of the intelligent routing in satellite internet, and leverages the deep reinforcement algorithm based on double delayed deep deterministic policy gradient (TD3) to solve the network routing optimization problem.The experimental results showed that compared with DDPG algorithm, the TD3 algorithm reduced the delay by 19.19%.

Key words: satellite internet, software-defi ned networking, deep reinforcement learning, routing optimization

CLC Number:

TN927.2

Linhui WEI, Guowen LIU, Yu LIU, Yumei WANG. Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning[J]. Space-Integrated-Ground Information Networks, 2022, 3(3): 65-71.

Figures/Tables 8

References 23

[1]	吴巍 . 卫星互联网发展综述[J]. 天地一体化信息网络, 2020,1(1): 1-16.
	WU W . Survey on the development of space-integrated-ground information network[J]. Space-Integrated-Ground Information Networks, 2020,1(1): 1-16.
[2]	KREUTZ D , RAMOS F M V , VERíSSIMO P E , ,et al. Softwaredefined networking,a comprehensive survey[J]. Proceedings of the IEEE, 2015,103(1): 14-76.
[3]	TANG Z , ZHAO B K , YU W R ,et al. Software defined satellite networks,benefits and challenges[C]// Proceedings of 2014 IEEE Computers,Communications and IT Applications Conference. Piscataway,IEEE Press, 2014: 127-132.
[4]	FORTZ B , THORUP M . Internet traffic engineering by optimizing OSPF weights[C]// Proceedings of IEEE INFOCOM 2000.Conference on Computer Communications.Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat.No.00CH37064). Piscataway,IEEE Press, 2000: 519-528.
[5]	XIE J F , YU F R , HUANG T ,et al. A survey of machine learning techniques applied to software defined networking (SDN),research issues and challenges[J]. IEEE Communications Surveys ＆ Tutorials, 2019,21(1): 393-430.
[6]	WANG M W , CUI Y , WANG X ,et al. Machine learning for networking,workflow,advances and opportunities[J]. IEEE Network, 2018,32(2): 92-99.
[7]	ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M ,et al. Deep reinforcement learning,a brief survey[J]. IEEE Signal Processing Magazine, 2017,34(6): 26-38.
[8]	YAO H P , WANG L Y , WANG X D ,et al. The space-terrestrial integrated network,an overview[J]. IEEE Communications Magazine, 2018,56(9): 178-185.
[9]	安建平, 李建国, 于季弘 ,等. 空天通信网络关键技术综述[J]. 电子学报, 2022,50(2): 470-479.
	AN J P , LI J G , YU J H ,et al. Key technologies of space-air-ground communication networks,a survey[J]. Acta Electronica Sinica, 2022,50(2): 470-479.
[10]	SHI Y P , LIU J J , FADLULLAH Z M ,et al. Cross-layer data delivery in satellite-aerial-terrestrial communication[J]. IEEE Wireless Communications, 2018,25(3): 138-143.
[11]	YAO S , GUAN J F , YAN Z W ,et al. SI-STIN,a smart identifier framework for space and terrestrial integrated network[J]. IEEE Network, 2019,33(1): 8-14.
[12]	徐晖, 孙韶辉 . 面向6G的天地一体化信息网络架构研究[J]. 天地一体化信息网络, 2021,2(4): 2-9.
	XU H , SUN S H . Research on network architecture for the spaceintegrated-ground information network in 6G[J]. Space-IntegratedGround Information Networks, 2021,2(4): 2-9.
[13]	BI Y G , HAN G J , XU S ,et al. Software defined space-terrestrial integrated networks,architecture,challenges,and solutions[J]. IEEE Network, 2019,33(1): 22-28.
[14]	ZHANG N , ZHANG S , YANG P ,et al. Software defined space-airground integrated vehicular networks,challenges and solutions[J]. IEEE Communications Magazine, 2017,55(7): 101-109.
[15]	杨丹, 刘江, 张然 ,等. 基于SDN的卫星通信网络,现状、机遇与挑战[J]. 天地一体化信息网络, 2020(2): 34-41.
	YANG D , LIU J , ZHANG R ,et al. SDN-based satellite networks:progress,opportunities and challenges[J]. Space-Integrated-Ground Information Networks, 2020(2): 34-41.
[16]	MESTRES A , RODRIGUEZ-NATAL A ,, CARNER J ,et al. Knowledge-defined networking[J]. ACM SIGCOMM Computer Communication Review, 2017,47(3): 2-10.
[17]	STAMPA G , ARIAS M , SANCHEZ-CHARLES D ,et al. A deepreinforcement learning approach for software-defined networking routing optimization[EB]. 2017.
[18]	CHEN J , XIAO Z W , XING H L ,et al. STDPG,a spatiotemporal deterministic policy gradient agent for dynamic routing in SDN[C]// Proceedings of ICC 2020-2020 IEEE International Conference on Communications. Piscataway,IEEE Press, 2020: 1-6.
[19]	HUANG X H , YUAN T T , QIAO G H ,et al. Deep reinforcement learning for multimedia traffic control in software defined networking[J]. IEEE Network, 2018,32(6): 35-41.
[20]	TU Z , ZHOU H C , LI K ,et al. A routing optimization method for software-defined SGIN based on deep reinforcement learning[C]// Proceedings of 2019 IEEE Globecom Workshops. Piscataway,IEEE Press, 2019: 1-6.
[21]	SHI X J , REN P Y , DU Q H . Reinforcement learning routing in space-air-ground integrated networks[C]// Proceedings of 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP). Piscataway,IEEE Press, 2021: 1-6.
[22]	ZUO P L , WANG C , YAO Z ,et al. An intelligent routing algorithm for LEO satellites based on deep reinforcement learning[C]// Proceedings of 2021 IEEE 94th Vehicular Technology Conference. Piscataway:IEEE Press, 2021: 1-5.
[23]	李新桐, 张亚生 . 一种适用于低轨卫星的SDN网络人工智能路由方法[J]. 电子测量技术, 2020,43(22): 109-114.
	LI X T , ZHANG Y S . Artificial intelligence routing method for SDN network suitable for LEO satellites[J]. Electronic Measurement Technology, 2020,43(22): 109-114.

Metrics

Recommended 0

No Suggested Reading articles found!

参数名称	数值
轨道数/个	3
每个轨道的卫星数/颗	8
轨道倾角/°	60
高度/km	1 400

Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 23

Related Articles 15

Metrics

Recommended 0

[1]	Qiuyu BU, Jin CAO, Lifu CHENG, Ruhui MA, Hui LI. Research on Access Authentication and Re-Authentication Mechanism of User Devices in Terrestrial Default Scenario in Satellite Internet [J]. Space-Integrated-Ground Information Networks, 2023, 4(2): 31-46.
[2]	Hao LI, Linjie ZHANG, Yifei ZHANG. Research on Satellite Internet Security Simulation Test Technology [J]. Space-Integrated-Ground Information Networks, 2023, 4(2): 47-54.
[3]	Dan ZHANG, Jingjing LI, Tian LIU, Sunjie TAO, Ziping LYU. Survey on Giant Constellation Cloud Network Integration [J]. Space-Integrated-Ground Information Networks, 2023, 4(2): 71-81.
[4]	Liang ZHU, Shaobo QI, Bo YANG, Bingyu XU, Zifan LI, Shijie ZHANG. Application of the Power Grid Business Carrying on Low Earth Orbit Broadband Satellite Internet [J]. Space-Integrated-Ground Information Networks, 2023, 4(2): 103-113.
[5]	Siyue CHENG, Haoran LI, Weigang BAI, Di ZHOU, Yan ZHU. Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning [J]. Space-Integrated-Ground Information Networks, 2023, 4(1): 12-22.
[6]	Jiacheng MENG, Ningbo XIE, Zhaofeng BAI, Jiaxuan ZHU, Junxia WU, Duorui GAO, Wei WANG, Xiaoping XIE. Spaceborne Optical Switching Technology for Satellite Internet [J]. Space-Integrated-Ground Information Networks, 2022, 3(2): 47-55.
[7]	Tao CUI, Zhiyuan REN, Jun LI, Qinggui TAN, Jingling LI, Wei LIANG. Intelligent Identifi cation and Classifi cation Algorithm and Simulation of Satellite Internet Business [J]. Space-Integrated-Ground Information Networks, 2022, 3(2): 72-80.
[8]	Wenyu SUN, Weijia ZHANG, Limin WANG. Fault Detection Method of Low-Orbit Satellite Internet Based on Deep Uncertainty Estimation Network [J]. Space-Integrated-Ground Information Networks, 2022, 3(2): 89-97.
[9]	Yijie WANG, Wei ZHAO, Fei CHENG, Wen CHEN, Anjie CAO. Large Scale LEO Satellite Internet Routing Algorithm Based on Load Balancing [J]. Space-Integrated-Ground Information Networks, 2022, 3(1): 27-34.
[10]	Meilin XU, Min JIA, Qing GUO. Research on Resource Allocation of Satellite Internet Service Function Based on SDN/NFV [J]. Space-Integrated-Ground Information Networks, 2022, 3(1): 44-49.
[11]	Chen HAN, Aijun LIU, Kang AN. Research Prospect of Anti-Jamming Strategy for the Satellite Internet [J]. Space-Integrated-Ground Information Networks, 2022, 3(1): 50-55.
[12]	Zhe JI, Sheng WU, Wenbo WANG. Hierarchical Intelligence Deployment Architecture for Satellite Internet [J]. Space-Integrated-Ground Information Networks, 2022, 3(1): 56-61.
[13]	Yaoqi LIU, Hongguang LI, Yiqing ZHOU, Jinglin SHI, Yongtao SU, Jinxi QIAN. Digital Twin Satellite Internet: Architecture and Key Technologies [J]. Space-Integrated-Ground Information Networks, 2022, 3(1): 62-71.
[14]	Ruijie ZHU, Yudong ZHANG, Yating WEI, Chao XI. Security Management Technology of Multilayer Satellite Internet Based on Blockchain [J]. Space-Integrated-Ground Information Networks, 2022, 3(1): 79-86.
[15]	Xiaojin DING, Tao HONG, Rui LIU, Wentao PENG, Gengxin ZHANG. Research on Architecture of LEO Satellite Internet of Things and Key Technologies [J]. Space-Integrated-Ground Information Networks, 2021, 2(4): 10-18.