基于分层强化学习的中继卫星网络任务动态调度方法

doi:10.11959/j.issn.1000-436x.2023130

Abstract

Abstract:

In recent years, with the increasing number of various emergency tasks, how to control the impact on common tasks while ensuring system revenue has become a huge challenge for the dynamic scheduling of relay satellite networks.Aiming at this problem, with the goal of maximizing the total revenue of emergency tasks and minimizing the damage to common tasks, a dynamic task scheduling method for relay satellite networks based on hierarchical reinforcement learning was proposed.Specifically, in order to take into account the long-term and short-term performance of the system at the same time, a two-layer scheduling framework implemented by upper-level and lower-level DQN was designed.The upper-level DQN was responsible for determining the temporary optimization goal based on long-term performance, and the lower-level DQN determined the scheduling strategy for current task according to the optimization goal.Simulation results show that compared with traditional deep learning methods and the heuristic methods dealing with dynamic scheduling problems, the proposed method can improve the total revenue of urgent tasks while reducing the damage to common tasks.

Key words: relay satellite networks, task scheduling, deep reinforcement learning, multi-objective optimization, dynamic scheduling

CLC Number:

TN92

Runzi LIU, Tianci MA, Weihua WU, Chenhong YAO, Qinghai YANG. Dynamic task scheduling method for relay satellite networks based on hierarchical reinforcement learning[J]. Journal on Communications, 2023, 44(7): 207-217.

Figures/Tables 9

References 15

[1]	王家胜 . 中国数据中继卫星系统及其应用拓展[J]. 航天器工程, 2013,22(1): 1-6.
	WANG J . China’s data relay satellite system and its application prospect[J]. Spacecraft Engineering, 2013,22(1): 1-6.
[2]	贺川, 朱晓敏, 邱涤珊 . 面向应急成像观测任务的多星协同调度方法[J]. 系统工程与电子技术, 2012,34(4): 726-731.
	HE C , ZHU X M , QIU D S . Cooperative scheduling method of multi-satellites for imaging reconnaissance in emergency condition[J]. Systems Engineering and Electronics, 2012,34(4): 726-731.
[3]	WU G H , MA M H , ZHU J H ,et al. Multi-satellite observation integrated scheduling method oriented to emergency tasks and common tasks[J]. Journal of Systems Engineering and Electronics, 2012,23(5): 723-733.
[4]	李飞龙, 李广侠, 李志强 ,等. 基于多层分簇的北斗卫星导航系统拓扑结构与路由策略[J]. 通信学报, 2014,35(10): 31-41.
	LI F L , LI G X , LI Z Q ,et al. Topology structure and routing policy based on multilayered clusters in Beidou satellite navigation system[J]. Journal on Communications, 2014,35(10): 31-41.
[5]	HAN S M , BEAK S W , CHO K R ,et al. Satellite mission scheduling using genetic algorithm[C]// Proceedings of 2008 SICE Annual Conference. Piscataway:IEEE Press, 2008: 1226-1230.
[6]	HUI-CHCNG H , JIANG W , LI Y J . Solving on agile satellites mission planning based on tabu search-parallel genetic algorithms[C]// Proceed ings of 2013 International Conference on Management Science and Proceedings of Engineering 20th Annual Conference Proceedings. Piscataway:IEEE Press, 2013: 120-125.
[7]	HE L J , LI J D , SHENG M ,et al. Dynamic scheduling of hybrid tasks with time windows in data relay satellite networks[J]. IEEE Transactions on Vehicular Technology, 2019,68(5): 4989-5004.
[8]	DENG B Y , JIANG C X , KUANG L L ,et al. Two-phase task scheduling in data relay satellite systems[J]. IEEE Transactions on Vehicular Technology, 2018,67(2): 1782-1793.
[9]	DAI C Q , LI C , FU S ,et al. Dynamic scheduling for emergency tasks in space data relay network[J]. IEEE Transactions on Vehicular Technology, 2021,70(1): 795-807.
[10]	ZHANG Z J , ZHANG N , FENG Z R . Multi-satellite control resource scheduling based on ant colony optimization[J]. Expert Systems with Applications, 2014,41(6): 2816-2823.
[11]	刘润滋, 盛敏, 唐成圆 ,等. 基于任务拆分聚合的中继卫星系统任务规划方法[J]. 通信学报, 2017,38(S1): 110-117.
	LIU R Z , SHENG M , TANG C Y ,et al. Tasking planning based on task splitting and merging in relay satellite network[J]. Journal on Communications, 2017,38(S1): 110-117.
[12]	ZHU X M , SIM K M , JIANG J Q ,et al. Agent-based dynamic scheduling for earth-observing tasks on multiple airships in emergency[J]. IEEE Systems Journal, 2016,10(2): 661-672.
[13]	ROJANASOONTHON S , BARD J . A GRASP for parallel machine scheduling with time windows[J]. INFORMS Journal on Computing, 2005,17(1): 32-51.
[14]	ZHOU D , SHENG M , LI J D ,et al. Aerospace integrated networks innovation for empowering 6G:a survey and future challenges[J]. IEEE Communications Surveys ＆ Tutorials, 2023,25(2): 975-1019.
[15]	LUO S , ZHANG L X , FAN Y S . Dynamic multi-objective scheduling for flexible job shop by deep reinforcement learning[J]. Computers ＆Industrial Engineering, 2021,159:107489.

Metrics

Recommended 0

No Suggested Reading articles found!

Dynamic task scheduling method for relay satellite networks based on hierarchical reinforcement learning

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 15

Related Articles 15

Metrics

Recommended 0

[1]	Ningjiang CHEN, Linming LIAN, Pingjie OU, Xuemei YUAN. D2D cooperative caching strategy based on graph collaborative filtering model [J]. Journal on Communications, 2023, 44(7): 136-148.
[2]	Feng ZENG, Zheng ZHANG, Zhigang CHEN. Computation offloading and resource allocation strategy based on deep reinforcement learning [J]. Journal on Communications, 2023, 44(7): 124-135.
[3]	Biao JIN, Yikang LI, Zhiqiang YAO, Yulin CHEN, Jinbo XIONG. GenFedRL: a general federated reinforcement learning framework for deep reinforcement learning agents [J]. Journal on Communications, 2023, 44(6): 183-197.
[4]	Yuancheng LI, Yongtai QIN. Deep reinforcement learning based algorithm for real-time QoS optimization of software-defined security middle platform [J]. Journal on Communications, 2023, 44(5): 181-192.
[5]	Weifeng LU, Ning LI, Jia XU, Lijie XU, Jian XU. Joint scheduling algorithm for correlative tasks in multi-access edge computing [J]. Journal on Communications, 2023, 44(4): 87-98.
[6]	Guoliang XU, Feng TAN, Yongyi RAN, Feng CHEN. Joint beam hopping and coverage control optimization algorithm for multibeam satellite system [J]. Journal on Communications, 2023, 44(4): 78-86.
[7]	Yu ZHANG, Min CHENG. Joint optimization of edge computing and caching in NDN [J]. Journal on Communications, 2022, 43(8): 164-175.
[8]	Zongxuan SHA, Ru HUO, Chuang SUN, Shuo WANG, Tao HUANG. Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(8): 30-40.
[9]	Sifeng ZHU, Jianghao CAI, Zhengyi CHAI, Enlin SUN. Multi-objective optimal offloading decision for cloud-edge collaborative computing scenario in Internet of vehicles [J]. Journal on Communications, 2022, 43(6): 223-234.
[10]	Xianchao ZHANG, Yao ZHAO, Haijun YE, Rui FAN. Intelligent transmit power control algorithm for the multi-user interference of wireless network [J]. Journal on Communications, 2022, 43(2): 15-21.
[11]	Xin SU, Leilei MENG, Yiqing ZHOU, Wu CELIMUGE. Maritime mobile edge computing offloading method based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(10): 133-145.
[12]	Li’na DU, Li ZHUO, Shuo YANG, Jiafeng LI, Jing ZHANG. Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services [J]. Journal on Communications, 2021, 42(9): 205-217.
[13]	Wenchen HE, Shaoyong GUO, Xuesong QIU, Liandong CHEN, Suxiang ZHANG. Node selection method in federated learning based on deep reinforcement learning [J]. Journal on Communications, 2021, 42(6): 62-71.
[14]	Tianyi ZHU,Fenghua LI,Wei JIN,Yunchuan GUO,Liang FANG,Lin CHENG. Cross-domain access control policy mapping mechanism for balancing interoperability and autonomy [J]. Journal on Communications, 2020, 41(9): 29-48.
[15]	Guolin SUN,Ruijie OU,Guisong LIU. Deep reinforcement learning-based resource reservation algorithm for emergency Internet-of-things slice [J]. Journal on Communications, 2020, 41(9): 8-20.