基于多智能体深度强化学习的测运控一体化资源调度方法

doi:10.11959/j.issn.2096-8930.2023002

Abstract

Abstract:

With the development of satellite communication technology and the continuous expansion of the constellation scale, the integration of TT＆amp;C and observation technology has become the mainstream trend.The large constellation scale, many scheduling objects and complex operation joint control bring great challenges to the integrated resource scheduling of satellite network TT＆amp;C and observation.Subject to the low solution effi ciency and complex constraints of scheduling algorithms, the traditional TT＆amp;C resource scheduling technology adopts the advance injection TT＆amp;C instructions to perform tasks according to the fi xed deployment, which is diffi cult to meet the scheduling needs of emergencies and emergency tasks.Therefore, a kind of resource scheduling method based on multi-agent actor-Agent Actor-Critic Deterministic Policy Gradient Algorithms (MADDPG) was presented.With centralized training and distributed execution, the multi-agent model of integrated task of TT＆amp;C and observation was established.By analyzed the scheduling strategy of neighbor agent, the response speed of local information was improved.According to the model and constraints in the integrated resource scheduling problem of TT＆amp;C and observation, selected signifi cant and interpretable constraints, then established the multi-agent resource scheduling reinforcement learning model, and carried on the simulation test.The simulation results showed that the task benefi t of this method was 22% higher than the traditional method.

Key words: integration of TT＆C and observation, large-scale constellation system, resources scheduling, multi-agent deep reinforcement learning, tasks reward

CLC Number:

Siyue CHENG, Haoran LI, Weigang BAI, Di ZHOU, Yan ZHU. Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning[J]. Space-Integrated-Ground Information Networks, 2023, 4(1): 12-22.

Figures/Tables 9

References 26

[1]	张威, 吴涛, 马宏 ,等. 智能一体化航天测运控网络发展探析[J]. 天地一体化信息网络, 2021,2(2): 82-89.
	ZHANG W , WU T , MA H ,et al. Discussion on the development of integrated and intelligent space TTC ＆ OC network[J]. SpaceIntegrated-Ground Information Networks, 2021,2(2): 82-89.
[2]	GABREL V . Strengthened 0-1 linear formulation for the daily satellite mission planning[J]. Journal of Combinatorial Optimization, 2006,11(3): 341-346.
[3]	王海波, 徐敏强, 王日新 ,等. 对地观测小卫星星座长期任务规划求解技术[J]. 系统工程与电子技术, 2011,33(6): 1293-1298.
	WANG H B , XU M Q , WANG R X ,et al. Long-term acquisition plan method for small satellites constellation[J]. Systems Engineering and Electronics, 2011,33(6): 1293-1298.
[4]	王沛 . 基于分支定价的多星多站集成调度方法研究[D]. 长沙:国防科学技术大学, 2011.
	WANG P . Research on branch-and-price based multi-satellite multi-station integrated scheduling method[D]. Changsha:National University of Defense Technology, 2011.
[5]	徐小辉, 胡绍林, 郭小红 ,等. 航天器测控资源调度模型及算法[J]. 中国空间科学技术, 2014,34(3): 32-37.
	XU X H , HU S L , GUO X H ,et al. Spacecraft TT ＆ C scheduling models and algorithms[J]. Chinese Space Science and Technology, 2014,34(3): 32-37.
[6]	王远振, 赵坚, 聂成 . 多卫星—地面站系统的Petri网模型研究[J]. 空军工程大学学报(自然科学版), 2003,4(2): 7-11.
	WANG Y Z , ZHAO J , NIE C . Study on Petri net model for multisatellites-ground station system[J]. Journal of Air Force Engineering University (Natural Science Edition), 2003,4(2): 7-11.
[7]	ZHANG N , FENG Z R , FENG Y J ,et al. An optimization model for multisatellite resources scheduling[C]// Proceedings of 2006 6th World Congress on Intelligent Control and Automation. Piscataway:IEEE Press, 2006: 7400-7404.
[8]	ZHANG Z J , HU F N , ZHANG N . Ant colony algorithm for satellite control resource scheduling problem[J]. Applied Intelligence, 2018,48(10): 3295-3305.
[9]	张天骄, 李济生, 李晶 ,等. 基于混合蚁群优化的天地一体化调度方法[J]. 系统工程与电子技术, 2016,38(7): 1555-1562.
	ZHANG T J , LI J S , LI J ,et al. Space-ground integrated scheduling based on the hybrid ant colony optimization[J]. Systems Engineering and Electronics, 2016,38(7): 1555-1562.
[10]	陶孙杰, 宋竹 . 一种测控数据传输一体化站网资源调度算法[J]. 电讯技术, 2018,58(7): 760-767.
	TAO S J , SONG Z . A ground-station resource scheduling algorithm based on integration of TT ＆ C and data transmission[J]. Telecommunication Engineering, 2018,58(7): 760-767.
[11]	金光, 武小悦, 高卫斌 . 卫星地面站资源调度优化模型及启发式算法[J]. 系统工程与电子技术, 2004,26(12): 1839-1841,1875.
	JIN G , WU X Y , GAO W B . Ground Station resource scheduling optimization model and its heuristic algorithm[J]. Systems Engineering and Electronics, 2004,26(12): 1839-1841,1875.
[12]	金光, 武小悦, 高卫斌 . 基于冲突的卫星地面站系统资源调度与能力分析[J]. 小型微型计算机系统, 2007,28(2): 310-312.
	JIN G , WU X Y , GAO W B . Conflict based resource scheduling and capability analysis of satellite-ground station system[J]. Journal of Chinese Computer Systems, 2007,28(2): 310-312.
[13]	刘嵩, 白国庆, 陈英武 . 地球观测网络成像任务可调度性预测方法[J]. 宇航学报, 2015,36(5): 583-588.
	LIU S , BAI G Q , CHEN Y W . Prediction method for imaging task schedulability of earth observation network[J]. Journal of Astronautics, 2015,36(5): 583-588.
[14]	杜红梅, 柯宏发 . 基于多智能体技术的航天测控资源调度模型设计[C]// 系统仿真技术及其应用. 2015(16): 78-81.
	DU H M , KE H F . Design of space TT＆C resource scheduling model based on multi-agent technology[C]// System simulation technology and its application. 2015(16): 78-81.
[15]	BADALONI S , FALDA M , GIACOMIN M . Solving temporal over-constrained problems using fuzzy techniques[J]. Journal of Intelligent ＆ Fuzzy Systems:Applications in Engineering and Technology, 2007,18(3): 255-265.
[16]	WANG H J , YANG Z , ZHOU W G ,et al. Online scheduling of image satellites based on neural networks and deep reinforcement learning[J]. Chinese Journal of Aeronautics, 2019,32(4): 1011-1019.
[17]	ZHANG T J , KE L J , LI J S ,et al. Fireworks algorithm for the satellite link scheduling problem in the navigation constellation[C]// Proceedings of 2016 IEEE Congress on Evolutionary Computation (CEC). Piscataway:IEEE Press, 2016: 4029-4037.
[18]	窦骄, 韩孟飞, 宁金枝 ,等. 小卫星测控通信技术发展与趋势[J]. 航天器工程, 2021,30(6): 113-119.
	DOU J , HAN M F , NING J Z ,et al. Development and trends of TT ＆ C and communication technology for small satellite[J]. Spacecraft Engineering, 2021,30(6): 113-119.
[19]	陈峰, 武小悦 . 天地测控资源一体化调度模型[J]. 宇航学报, 2010,31(5): 1405-1412.
	CHEN F , WU X Y . Space and ground TT ＆ C resource integrated scheduling model[J]. Journal of Astronautics, 2010,31(5): 1405-1412.
[20]	孟欢 . 测控资源调度预测与服务功能链优化映射研究[D]. 天津:天津大学, 2019.
	MENG H . Research on measurement and control resource scheduling prediction and service function chain optimization mapping[D]. Tianjin:Tianjin University, 2019.
[21]	安元元 . 低轨卫星测控系统调度策略的设计与实现[D]. 西安:中国科学院大学(中国科学院国家授时中心), 2021.
	AN Y Y . Design and realization of schedule strategy of the LEO satellite telemetry and telecontrol system[D]. Xi'an:National Time Service Center,Chinese Academy of Sciences, 2021.
[22]	雷瀚, 杨帆, 刘建平 ,等. 航天地基测运控资源调度多任务需求分析及统一的任务模型研究[C]// 第三届体系工程学术会议论文集——复杂系统与体系工程管理. 2021: 100-110.
	LEI H , YANG F , LIU J P ,et al. Multi-task demand analysis and unified task model research of space-based survey,transportation and control resource scheduling[C]// Proceedings of the 3rd Symposium on Systems Engineering-Complex Systems and Systems Engineering Management. 2021: 100-110.
[23]	唐成圆 . 面向测运控的卫星网络常规任务规划算法研究[D]. 西安:西安电子科技大学, 2020.
	TANG C Y . Routine task planning algorithm for telemetry tracking and operation control center in satellite network[D]. Xi'an:Xidian University, 2020.
[24]	武艺 . 基于深度强化学习的多星测控资源调度方法研究[D]. 重庆:重庆大学, 2020.
	WU Y . Research on A scheduling method of TT＆C resources for multi-satellite based on deep reinforcement learning[D]. Chongqing:Chongqing University, 2020.
[25]	李长德, 徐伟, 徐梁 ,等. 基于深度神经网络的多星测控调度方法[J]. 中国空间科学技术, 2022,42(1): 65-72.
	LI C D , XU W , XU L ,et al. Multi-satellite TT＆C scheduling method based on deep neural network[J]. Chinese Space Science and Technology, 2022,42(1): 65-72.
[26]	LOWE R , WU Y , TAMAR A ,et al. Multi-agent actor-critic for mixed cooperative-competitive environments[EB]. 2017.

Metrics

Recommended 0

No Suggested Reading articles found!

参数	值	参数	值
运行总次数/次	1 000	经验池大小	500 000条
运行步数/步	8 400	训练批次大小	256
开始训练的次数	50次	软更新参数τ	0.01
动作网络学习率	0.000 1	动作选择参数ε	0.9
策略网络学习率	0.001	计算损失参数γ	0.95

Resource Scheduling Method for Integration of TT＆C and Observation Based on Multi-Agent Deep Reinforcement Learning

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 26

Related Articles 5

Metrics

Recommended 0

[1]	Peisen WANG, Yueneng WANG, Boming CHENG, Yue YANG, Neng YE. Multi-User Detection Algorithm for Asynchronous Non-Orthogonal Multiple Access Based on Gaussian Approximation [J]. Space-Integrated-Ground Information Networks, 2023, 4(1): 73-81.
[2]	Yingte CHAI, Gang WANG, Chao ZHANG, Guangxi ZHU, Wei FU, Zhengqiang GUO. Satellite-Earth Integration Intelligent Resource Management and Control Architecture of Giant Sensing Constellation [J]. Space-Integrated-Ground Information Networks, 2022, 3(3): 13-22.
[3]	Guangxi ZHU, Gang WANG, Chao ZHANG, Yingte CHAI, Wei FU, Zhengqiang GUO. Intelligent Planning Framework for Star-Walk Mission Based on Multimodal Observation Requirements Information [J]. Space-Integrated-Ground Information Networks, 2022, 3(3): 23-29.
[4]	Qiulin LYU, Xiaojin DING, Gengxin ZHANG. Intelligent Excavation Technologies for Satellite Spectrum Signals [J]. Space-Integrated-Ground Information Networks, 2022, 3(3): 30-36.
[5]	Tingting ZHANG, Nan WU, Haipeng YAO. Research on Intelligent Networking Architecture for the Integrated Space-Terrestrial Networks [J]. Space-Integrated-Ground Information Networks, 2022, 3(3): 47-55.