基于深度强化学习的卫星互联网路由优化研究

doi:10.11959/j.issn.2096-8930.2022033

天地一体化信息网络 ›› 2022, Vol. 3 ›› Issue (3): 65-71.doi: 10.11959/j.issn.2096-8930.2022033

所属专题：专题：天地融合软件定义网络

• 专题：天地融合软件定义网络 • 上一篇下一篇

基于深度强化学习的卫星互联网路由优化研究

魏琳慧¹, 刘国文¹, 刘雨¹^,², 望育梅¹

¹ 北京邮电大学人工智能学院，北京 100876
² 鹏城实验室，广东深圳 518000

修回日期:2022-07-24 出版日期:2022-09-20 发布日期:2022-09-01
作者简介:魏琳慧（1997-），女，北京邮电大学人工智能学院博士生，主要研究方向为卫星互联网、软件定义网络、多媒体传输技术等
刘国文（1998-），男，北京邮电大学人工智能学院硕士生，主要研究方向为卫星互联网、机器学习在低轨卫星网络中的应用等
刘雨（1978-），女，北京邮电大学人工智能学院副教授，博士生导师，鹏城实验室网络与通信研究中心副教授，主要研究方向为卫星互联网、图像处理、分布式源编码等
望育梅（1974-），女，北京邮电大学人工智能学院副教授，硕士生导师，主要研究方向为卫星互联网、多媒体信号处理、无线多媒体传输和分布式视频编码等
基金资助:
国家重点研发计划资助项目(2019YFB1803103);北京邮电大学博士生创新基金项目(CX2021113)

Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning

Linhui WEI¹, Guowen LIU¹, Yu LIU¹^,², Yumei WANG¹

¹ School of Artifi cial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China
² Peng Cheng Laboratory, Shenzhen 518000, China

Revised:2022-07-24 Online:2022-09-20 Published:2022-09-01
Supported by:
National Key Research and Development Program of China(2019YFB1803103);BUPT Excellent Ph.D.Students Foundation(CX2021113)

摘要/Abstract

摘要：

随着卫星通信技术的飞速发展，卫星互联网成为6G网络实现全球覆盖、全时接入、全场景服务的核心关键技术。卫星网络的高动态性及有限的卫星容量，导致面临以异构网络管理、动态资源分配为代表的一系列管控挑战。由于机器学习技术在网络设计等方面具有显著优势，因此提出软件定义的卫星互联网智能化架构。针对卫星互联网的智能路由问题，利用基于双延迟深度确定性策略梯度的深度强化学习算法，解决网络的实时路由优化问题。实验结果表明，TD3算法相较于DDPG算法，平均网络时延降低了19.19%。

关键词: 卫星互联网, 深度强化学习, 软件定义网络, 路由优化

Abstract:

With the rapid development of satellite communication, the satellite internet is one of the core technologies of 6G network to realize global coverage, full-time access and full scene service.The high dynamics and limited capacity of satellite network lead to a series of management and control challenges such as heterogeneous network management, dynamic resource allocation and so on.Since the machine learning-based technologies have strength in network design, the intelligent architecture of software-defi ned satellite internet was put forward.In view of the intelligent routing in satellite internet, and leverages the deep reinforcement algorithm based on double delayed deep deterministic policy gradient (TD3) to solve the network routing optimization problem.The experimental results showed that compared with DDPG algorithm, the TD3 algorithm reduced the delay by 19.19%.

Key words: satellite internet, software-defi ned networking, deep reinforcement learning, routing optimization

中图分类号:

TN927.2

魏琳慧, 刘国文, 刘雨, 望育梅. 基于深度强化学习的卫星互联网路由优化研究[J]. 天地一体化信息网络, 2022, 3(3): 65-71.

Linhui WEI, Guowen LIU, Yu LIU, Yumei WANG. Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning[J]. Space-Integrated-Ground Information Networks, 2022, 3(3): 65-71.

图/表 8

图1

图2

图3

图4

表1

图5

图6

图7

参考文献 23

[1]	吴巍 . 卫星互联网发展综述[J]. 天地一体化信息网络, 2020,1(1): 1-16.
	WU W . Survey on the development of space-integrated-ground information network[J]. Space-Integrated-Ground Information Networks, 2020,1(1): 1-16.
[2]	KREUTZ D , RAMOS F M V , VERíSSIMO P E , ,et al. Softwaredefined networking,a comprehensive survey[J]. Proceedings of the IEEE, 2015,103(1): 14-76.
[3]	TANG Z , ZHAO B K , YU W R ,et al. Software defined satellite networks,benefits and challenges[C]// Proceedings of 2014 IEEE Computers,Communications and IT Applications Conference. Piscataway,IEEE Press, 2014: 127-132.
[4]	FORTZ B , THORUP M . Internet traffic engineering by optimizing OSPF weights[C]// Proceedings of IEEE INFOCOM 2000.Conference on Computer Communications.Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat.No.00CH37064). Piscataway,IEEE Press, 2000: 519-528.
[5]	XIE J F , YU F R , HUANG T ,et al. A survey of machine learning techniques applied to software defined networking (SDN),research issues and challenges[J]. IEEE Communications Surveys ＆ Tutorials, 2019,21(1): 393-430.
[6]	WANG M W , CUI Y , WANG X ,et al. Machine learning for networking,workflow,advances and opportunities[J]. IEEE Network, 2018,32(2): 92-99.
[7]	ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M ,et al. Deep reinforcement learning,a brief survey[J]. IEEE Signal Processing Magazine, 2017,34(6): 26-38.
[8]	YAO H P , WANG L Y , WANG X D ,et al. The space-terrestrial integrated network,an overview[J]. IEEE Communications Magazine, 2018,56(9): 178-185.
[9]	安建平, 李建国, 于季弘 ,等. 空天通信网络关键技术综述[J]. 电子学报, 2022,50(2): 470-479.
	AN J P , LI J G , YU J H ,et al. Key technologies of space-air-ground communication networks,a survey[J]. Acta Electronica Sinica, 2022,50(2): 470-479.
[10]	SHI Y P , LIU J J , FADLULLAH Z M ,et al. Cross-layer data delivery in satellite-aerial-terrestrial communication[J]. IEEE Wireless Communications, 2018,25(3): 138-143.
[11]	YAO S , GUAN J F , YAN Z W ,et al. SI-STIN,a smart identifier framework for space and terrestrial integrated network[J]. IEEE Network, 2019,33(1): 8-14.
[12]	徐晖, 孙韶辉 . 面向6G的天地一体化信息网络架构研究[J]. 天地一体化信息网络, 2021,2(4): 2-9.
	XU H , SUN S H . Research on network architecture for the spaceintegrated-ground information network in 6G[J]. Space-IntegratedGround Information Networks, 2021,2(4): 2-9.
[13]	BI Y G , HAN G J , XU S ,et al. Software defined space-terrestrial integrated networks,architecture,challenges,and solutions[J]. IEEE Network, 2019,33(1): 22-28.
[14]	ZHANG N , ZHANG S , YANG P ,et al. Software defined space-airground integrated vehicular networks,challenges and solutions[J]. IEEE Communications Magazine, 2017,55(7): 101-109.
[15]	杨丹, 刘江, 张然 ,等. 基于SDN的卫星通信网络,现状、机遇与挑战[J]. 天地一体化信息网络, 2020(2): 34-41.
	YANG D , LIU J , ZHANG R ,et al. SDN-based satellite networks:progress,opportunities and challenges[J]. Space-Integrated-Ground Information Networks, 2020(2): 34-41.
[16]	MESTRES A , RODRIGUEZ-NATAL A ,, CARNER J ,et al. Knowledge-defined networking[J]. ACM SIGCOMM Computer Communication Review, 2017,47(3): 2-10.
[17]	STAMPA G , ARIAS M , SANCHEZ-CHARLES D ,et al. A deepreinforcement learning approach for software-defined networking routing optimization[EB]. 2017.
[18]	CHEN J , XIAO Z W , XING H L ,et al. STDPG,a spatiotemporal deterministic policy gradient agent for dynamic routing in SDN[C]// Proceedings of ICC 2020-2020 IEEE International Conference on Communications. Piscataway,IEEE Press, 2020: 1-6.
[19]	HUANG X H , YUAN T T , QIAO G H ,et al. Deep reinforcement learning for multimedia traffic control in software defined networking[J]. IEEE Network, 2018,32(6): 35-41.
[20]	TU Z , ZHOU H C , LI K ,et al. A routing optimization method for software-defined SGIN based on deep reinforcement learning[C]// Proceedings of 2019 IEEE Globecom Workshops. Piscataway,IEEE Press, 2019: 1-6.
[21]	SHI X J , REN P Y , DU Q H . Reinforcement learning routing in space-air-ground integrated networks[C]// Proceedings of 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP). Piscataway,IEEE Press, 2021: 1-6.
[22]	ZUO P L , WANG C , YAO Z ,et al. An intelligent routing algorithm for LEO satellites based on deep reinforcement learning[C]// Proceedings of 2021 IEEE 94th Vehicular Technology Conference. Piscataway:IEEE Press, 2021: 1-5.
[23]	李新桐, 张亚生 . 一种适用于低轨卫星的SDN网络人工智能路由方法[J]. 电子测量技术, 2020,43(22): 109-114.
	LI X T , ZHANG Y S . Artificial intelligence routing method for SDN network suitable for LEO satellites[J]. Electronic Measurement Technology, 2020,43(22): 109-114.

参数名称	数值
轨道数/个	3
每个轨道的卫星数/颗	8
轨道倾角/°	60
高度/km	1 400

基于深度强化学习的卫星互联网路由优化研究

Research on Routing Optimization in Satellite Internet Based on Deep Reinforcement Learning

在线阅读

pdf下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 23

相关文章 15

Metrics

推荐阅读 0

[1]	卜秋雨, 曹进, 程利甫, 马如慧, 李晖. 卫星互联网地面缺省场景下用户设备的接入认证及重认证机制研究[J]. 天地一体化信息网络, 2023, 4(2): 31-46.
[2]	李皓, 张林杰, 张翼飞. 卫星互联网安全仿真测试技术研究[J]. 天地一体化信息网络, 2023, 4(2): 47-54.
[3]	黄思奇, 曾德泽, 李跃鹏, 张梁钰, 高丰. 天空地融合网络架构与传输优化技术[J]. 天地一体化信息网络, 2023, 4(2): 62-70.
[4]	张丹, 李晶晶, 刘田, 陶孙杰, 吕子平. 巨型星座云网融合发展探析[J]. 天地一体化信息网络, 2023, 4(2): 71-81.
[5]	朱亮, 戚少博, 杨波, 徐冰玉, 李子凡, 张世杰. 低轨宽带卫星互联网承载电网业务应用[J]. 天地一体化信息网络, 2023, 4(2): 103-113.
[6]	成思玥, 李浩然, 白卫岗, 周笛, 朱彦. 基于多智能体深度强化学习的测运控一体化资源调度方法[J]. 天地一体化信息网络, 2023, 4(1): 12-22.
[7]	张婷婷, 武楠, 姚海鹏. 天地融合网络智能组网体系架构研究[J]. 天地一体化信息网络, 2022, 3(3): 47-55.
[8]	夏师懿, 李国通. 基于光实时延迟线的波束成形技术研究回顾[J]. 天地一体化信息网络, 2022, 3(2): 20-27.
[9]	孟佳成, 谢宁波, 白兆峰, 朱嘉轩, 武军霞, 高铎瑞, 汪伟, 谢小平. 面向卫星互联网的星载光交换技术[J]. 天地一体化信息网络, 2022, 3(2): 47-55.
[10]	崔涛, 任智源, 黎军, 谭庆贵, 李静玲, 梁薇. 卫星互联网业务智能识别分类算法与仿真[J]. 天地一体化信息网络, 2022, 3(2): 72-80.
[11]	孙文宇, 张伟嘉, 王立民. 基于深度不确定性估计网络的低轨卫星互联网故障预测方法[J]. 天地一体化信息网络, 2022, 3(2): 89-97.
[12]	汪伊婕, 赵伟, 成飞, 陈文, 曹岸杰. 基于负载均衡的大规模低轨卫星互联网路由算法[J]. 天地一体化信息网络, 2022, 3(1): 27-34.
[13]	徐媚琳, 贾敏, 郭庆. 基于SDN/NFV的卫星互联网服务功能资源分配研究[J]. 天地一体化信息网络, 2022, 3(1): 44-49.
[14]	韩晨, 刘爱军, 安康. 卫星互联网抗干扰策略研究展望[J]. 天地一体化信息网络, 2022, 3(1): 50-55.
[15]	纪哲, 吴胜, 王文博. 面向卫星互联网的层级化智能部署架构[J]. 天地一体化信息网络, 2022, 3(1): 56-61.