通信学报 ›› 2014, Vol. 35 ›› Issue (7): 122-128.doi: 10.3969/j.issn.1000-436x.2014.07.015

• 论文Ⅱ • 上一篇    下一篇

基于节点识别的慢任务调度算法

崔云飞1,2,李新明2,李艺2,刘东2   

  1. 1 北京航天飞行控制中心，北京 100094
    2 装备学院 复杂电子系统仿真重点实验室，北京 101416
  • 出版日期:2014-07-25 发布日期:2017-06-24
  • 基金资助:
    国家自然科学基金资助项目;国家重大科技专项基金项目

Slow task scheduling algorithm based on node identification

Yun-fei CUI1,2,Xin-ming LI2,Yi LI2,Dong LIU2   

  1. 1 Beijing Aerospace Control Center, Beijing 100094, China
    2 National Key Laboratory of Complex Electronic System Simulation, Academy of Equipment, Beijing 101416, China
  • Online:2014-07-25 Published:2017-06-24
  • Supported by:
    The National Natural Science Foundation of China;The National Science and Technology Major Project

在线阅读

1

PDF下载

438

摘要：

为了降低大数据处理集群在执行任务过程中的慢任务对作业执行效率的影响，提出了一种识别慢任务、备份慢任务、减少慢任务相结合的调度算法——TQST 算法。首先，通过判断节点能力和任务执行时间，建立慢节点、非常慢节点和慢任务队列；其次，根据预判备份执行价值确定如何启动慢任务的备份任务，提高了备份执行的作用；然后，在节点识别的基础上，规避为非常慢节点分配任务，从根本上减少慢任务的产生，提高作业执行效率。实验结果表明，TQST算法在作业响应时间等方面优于已有的慢任务调度算法。

关键词: 大数据, 慢任务, 备份任务, Map-Reduce

Abstract:

In order to reduce the influence of the slow task, produced in big data processing, a scheduling algorithm (TQST) combining recognition, speculation and seduction of slow task was proposed. First of all, through the judgment of node ability and task execution time, slow node queue, very slow node queue and slow task queue were established. Secondly, according to the anticipation speculative execution value to decide how to start speculative task. Then, in the basis of node identification, avoid distributing tasks to very slow node, radically reduce slow task production, improve job execution efficiency. The experimental results show that TQST algorithm previous existing slow task scheduling al-gorithm in term of the job response time.

Key words: big data, slow task, speculative task, Map-Reduce

No Suggested Reading articles found!