基于多智能体深度强化学习的测运控一体化资源调度方法

doi:10.11959/j.issn.2096-8930.2023002

天地一体化信息网络 ›› 2023, Vol. 4 ›› Issue (1): 12-22.doi: 10.11959/j.issn.2096-8930.2023002

所属专题：大规模星座组网及测控关键技术

• 专题：大规模星座组网及测控关键技术 • 上一篇下一篇

基于多智能体深度强化学习的测运控一体化资源调度方法

成思玥, 李浩然, 白卫岗, 周笛, 朱彦

西安电子科技大学通信工程学院，陕西西安 710071

修回日期:2023-03-01 出版日期:2023-03-20 发布日期:2023-03-01
作者简介:成思玥（1998-），女，西安电子科技大学综合业务网理论与关键技术国家重点实验室硕士生，主要研究方向为巨型星座系统宽带民航业务资源切片
李浩然（1991-），男，西安电子科技大学综合业务网理论与关键技术国家重点实验室讲师，主要研究方向为大规模卫星星座资源分配与任务调度、网络的群智涌现与群体行为
白卫岗（1987-），男，西安电子科技大学综合业务网理论与关键技术国家重点实验室副教授，主要研究方向为水声通信网络、卫星通信网络、空天地海一体化网络架构、组网协议及仿真系统
周笛（1991-），女，西安电子科技大学综合业务网理论与关键技术国家重点实验室副教授，主要研究方向为空间信息网络任务规划及资源管理、卫星互联网资源管控技术等
朱彦（1993-），男，西安电子科技大学综合业务网理论与关键技术国家重点实验室讲师，主要研究方向为端到端可靠传输、服务质量保障等
基金资助:
国家重点研发计划资助项目(2020YFB1806100);国家自然科学基金青年项目(62101410);秦创原引用高层次创新创业人才项目(QCYRCXM-2022-228)

Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning

Siyue CHENG, Haoran LI, Weigang BAI, Di ZHOU, Yan ZHU

School of Telecommunications Engineering, Xidian University, Xi’an 710071,China

Revised:2023-03-01 Online:2023-03-20 Published:2023-03-01
Supported by:
National Key Research＆Development Program of China(2020YFB1806100);National Natural Science Foundation of China(62101410);The Foundation of Shaanxi Province(QCYRCXM-2022-228)

摘要/Abstract

摘要：

随着卫星通信技术的发展，星座规模的不断扩大，测运控一体化成为主流趋势。星座规模大、调度对象多、复杂操作联合控制给卫星网络测运控一体化资源调度带来巨大的挑战。受制于调度算法求解效率低、约束复杂等问题，传统的测运控资源调度技术采用提前上注测控指令，按照固定部署执行任务，难以满足突发事件与紧急任务的调度需求。因此，提出一种基于多智能体演员-评判家确定性策略梯度算法的测运控一体化资源调度方法，采用集中式训练和分布式执行的方法，建立测运控一体化任务的多智能体模型，通过分析邻居智能体局部信息计算调度策略，提高任务的响应速度。依据测运控一体化资源调度问题中的模型和约束，选择影响意义大、可解释的约束，建立多智能体资源调度强化学习模型，并进行仿真测试。测试结果显示，该方法的任务收益较传统方法提高22%。

关键词: 测运控一体化, 大规模星座系统, 资源调度, 多智能体深度强化学习, 任务收益

Abstract:

With the development of satellite communication technology and the continuous expansion of the constellation scale, the integration of TT＆amp;C and observation technology has become the mainstream trend.The large constellation scale, many scheduling objects and complex operation joint control bring great challenges to the integrated resource scheduling of satellite network TT＆amp;C and observation.Subject to the low solution effi ciency and complex constraints of scheduling algorithms, the traditional TT＆amp;C resource scheduling technology adopts the advance injection TT＆amp;C instructions to perform tasks according to the fi xed deployment, which is diffi cult to meet the scheduling needs of emergencies and emergency tasks.Therefore, a kind of resource scheduling method based on multi-agent actor-Agent Actor-Critic Deterministic Policy Gradient Algorithms (MADDPG) was presented.With centralized training and distributed execution, the multi-agent model of integrated task of TT＆amp;C and observation was established.By analyzed the scheduling strategy of neighbor agent, the response speed of local information was improved.According to the model and constraints in the integrated resource scheduling problem of TT＆amp;C and observation, selected signifi cant and interpretable constraints, then established the multi-agent resource scheduling reinforcement learning model, and carried on the simulation test.The simulation results showed that the task benefi t of this method was 22% higher than the traditional method.

Key words: integration of TT＆C and observation, large-scale constellation system, resources scheduling, multi-agent deep reinforcement learning, tasks reward

中图分类号:

成思玥, 李浩然, 白卫岗, 周笛, 朱彦. 基于多智能体深度强化学习的测运控一体化资源调度方法[J]. 天地一体化信息网络, 2023, 4(1): 12-22.

Siyue CHENG, Haoran LI, Weigang BAI, Di ZHOU, Yan ZHU. Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning[J]. Space-Integrated-Ground Information Networks, 2023, 4(1): 12-22.

图/表 9

图1

图2

图3

图4

图5

图6

表1

图7

图8

参考文献 26

[1]	张威, 吴涛, 马宏 ,等. 智能一体化航天测运控网络发展探析[J]. 天地一体化信息网络, 2021,2(2): 82-89.
	ZHANG W , WU T , MA H ,et al. Discussion on the development of integrated and intelligent space TTC ＆ OC network[J]. SpaceIntegrated-Ground Information Networks, 2021,2(2): 82-89.
[2]	GABREL V . Strengthened 0-1 linear formulation for the daily satellite mission planning[J]. Journal of Combinatorial Optimization, 2006,11(3): 341-346.
[3]	王海波, 徐敏强, 王日新 ,等. 对地观测小卫星星座长期任务规划求解技术[J]. 系统工程与电子技术, 2011,33(6): 1293-1298.
	WANG H B , XU M Q , WANG R X ,et al. Long-term acquisition plan method for small satellites constellation[J]. Systems Engineering and Electronics, 2011,33(6): 1293-1298.
[4]	王沛 . 基于分支定价的多星多站集成调度方法研究[D]. 长沙:国防科学技术大学, 2011.
	WANG P . Research on branch-and-price based multi-satellite multi-station integrated scheduling method[D]. Changsha:National University of Defense Technology, 2011.
[5]	徐小辉, 胡绍林, 郭小红 ,等. 航天器测控资源调度模型及算法[J]. 中国空间科学技术, 2014,34(3): 32-37.
	XU X H , HU S L , GUO X H ,et al. Spacecraft TT ＆ C scheduling models and algorithms[J]. Chinese Space Science and Technology, 2014,34(3): 32-37.
[6]	王远振, 赵坚, 聂成 . 多卫星—地面站系统的Petri网模型研究[J]. 空军工程大学学报(自然科学版), 2003,4(2): 7-11.
	WANG Y Z , ZHAO J , NIE C . Study on Petri net model for multisatellites-ground station system[J]. Journal of Air Force Engineering University (Natural Science Edition), 2003,4(2): 7-11.
[7]	ZHANG N , FENG Z R , FENG Y J ,et al. An optimization model for multisatellite resources scheduling[C]// Proceedings of 2006 6th World Congress on Intelligent Control and Automation. Piscataway:IEEE Press, 2006: 7400-7404.
[8]	ZHANG Z J , HU F N , ZHANG N . Ant colony algorithm for satellite control resource scheduling problem[J]. Applied Intelligence, 2018,48(10): 3295-3305.
[9]	张天骄, 李济生, 李晶 ,等. 基于混合蚁群优化的天地一体化调度方法[J]. 系统工程与电子技术, 2016,38(7): 1555-1562.
	ZHANG T J , LI J S , LI J ,et al. Space-ground integrated scheduling based on the hybrid ant colony optimization[J]. Systems Engineering and Electronics, 2016,38(7): 1555-1562.
[10]	陶孙杰, 宋竹 . 一种测控数据传输一体化站网资源调度算法[J]. 电讯技术, 2018,58(7): 760-767.
	TAO S J , SONG Z . A ground-station resource scheduling algorithm based on integration of TT ＆ C and data transmission[J]. Telecommunication Engineering, 2018,58(7): 760-767.
[11]	金光, 武小悦, 高卫斌 . 卫星地面站资源调度优化模型及启发式算法[J]. 系统工程与电子技术, 2004,26(12): 1839-1841,1875.
	JIN G , WU X Y , GAO W B . Ground Station resource scheduling optimization model and its heuristic algorithm[J]. Systems Engineering and Electronics, 2004,26(12): 1839-1841,1875.
[12]	金光, 武小悦, 高卫斌 . 基于冲突的卫星地面站系统资源调度与能力分析[J]. 小型微型计算机系统, 2007,28(2): 310-312.
	JIN G , WU X Y , GAO W B . Conflict based resource scheduling and capability analysis of satellite-ground station system[J]. Journal of Chinese Computer Systems, 2007,28(2): 310-312.
[13]	刘嵩, 白国庆, 陈英武 . 地球观测网络成像任务可调度性预测方法[J]. 宇航学报, 2015,36(5): 583-588.
	LIU S , BAI G Q , CHEN Y W . Prediction method for imaging task schedulability of earth observation network[J]. Journal of Astronautics, 2015,36(5): 583-588.
[14]	杜红梅, 柯宏发 . 基于多智能体技术的航天测控资源调度模型设计[C]// 系统仿真技术及其应用. 2015(16): 78-81.
	DU H M , KE H F . Design of space TT＆C resource scheduling model based on multi-agent technology[C]// System simulation technology and its application. 2015(16): 78-81.
[15]	BADALONI S , FALDA M , GIACOMIN M . Solving temporal over-constrained problems using fuzzy techniques[J]. Journal of Intelligent ＆ Fuzzy Systems:Applications in Engineering and Technology, 2007,18(3): 255-265.
[16]	WANG H J , YANG Z , ZHOU W G ,et al. Online scheduling of image satellites based on neural networks and deep reinforcement learning[J]. Chinese Journal of Aeronautics, 2019,32(4): 1011-1019.
[17]	ZHANG T J , KE L J , LI J S ,et al. Fireworks algorithm for the satellite link scheduling problem in the navigation constellation[C]// Proceedings of 2016 IEEE Congress on Evolutionary Computation (CEC). Piscataway:IEEE Press, 2016: 4029-4037.
[18]	窦骄, 韩孟飞, 宁金枝 ,等. 小卫星测控通信技术发展与趋势[J]. 航天器工程, 2021,30(6): 113-119.
	DOU J , HAN M F , NING J Z ,et al. Development and trends of TT ＆ C and communication technology for small satellite[J]. Spacecraft Engineering, 2021,30(6): 113-119.
[19]	陈峰, 武小悦 . 天地测控资源一体化调度模型[J]. 宇航学报, 2010,31(5): 1405-1412.
	CHEN F , WU X Y . Space and ground TT ＆ C resource integrated scheduling model[J]. Journal of Astronautics, 2010,31(5): 1405-1412.
[20]	孟欢 . 测控资源调度预测与服务功能链优化映射研究[D]. 天津:天津大学, 2019.
	MENG H . Research on measurement and control resource scheduling prediction and service function chain optimization mapping[D]. Tianjin:Tianjin University, 2019.
[21]	安元元 . 低轨卫星测控系统调度策略的设计与实现[D]. 西安:中国科学院大学(中国科学院国家授时中心), 2021.
	AN Y Y . Design and realization of schedule strategy of the LEO satellite telemetry and telecontrol system[D]. Xi'an:National Time Service Center,Chinese Academy of Sciences, 2021.
[22]	雷瀚, 杨帆, 刘建平 ,等. 航天地基测运控资源调度多任务需求分析及统一的任务模型研究[C]// 第三届体系工程学术会议论文集——复杂系统与体系工程管理. 2021: 100-110.
	LEI H , YANG F , LIU J P ,et al. Multi-task demand analysis and unified task model research of space-based survey,transportation and control resource scheduling[C]// Proceedings of the 3rd Symposium on Systems Engineering-Complex Systems and Systems Engineering Management. 2021: 100-110.
[23]	唐成圆 . 面向测运控的卫星网络常规任务规划算法研究[D]. 西安:西安电子科技大学, 2020.
	TANG C Y . Routine task planning algorithm for telemetry tracking and operation control center in satellite network[D]. Xi'an:Xidian University, 2020.
[24]	武艺 . 基于深度强化学习的多星测控资源调度方法研究[D]. 重庆:重庆大学, 2020.
	WU Y . Research on A scheduling method of TT＆C resources for multi-satellite based on deep reinforcement learning[D]. Chongqing:Chongqing University, 2020.
[25]	李长德, 徐伟, 徐梁 ,等. 基于深度神经网络的多星测控调度方法[J]. 中国空间科学技术, 2022,42(1): 65-72.
	LI C D , XU W , XU L ,et al. Multi-satellite TT＆C scheduling method based on deep neural network[J]. Chinese Space Science and Technology, 2022,42(1): 65-72.
[26]	LOWE R , WU Y , TAMAR A ,et al. Multi-agent actor-critic for mixed cooperative-competitive environments[EB]. 2017.

参数	值	参数	值
运行总次数/次	1 000	经验池大小	500 000条
运行步数/步	8 400	训练批次大小	256
开始训练的次数	50次	软更新参数τ	0.01
动作网络学习率	0.000 1	动作选择参数ε	0.9
策略网络学习率	0.001	计算损失参数γ	0.95

基于多智能体深度强化学习的测运控一体化资源调度方法

Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning

在线阅读

pdf下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 26

相关文章 2

Metrics

推荐阅读 0

[1]	纪翔, 刘润滋, 要趁红, 陈剑豪, 慕桃. 卫星网络测控资源鲁棒调度方法[J]. 天地一体化信息网络, 2023, 4(1): 23-30.
[2]	贾敏,郭家,徐媚琳,郭庆,顾学迈. SDN/NFV架构下基于MPTCP协同SDN的空间网络资源调度技术[J]. 天地一体化信息网络, 2020, 1(1): 71-77.