Abstract:

In order to reduce the influence of the slow task, produced in big data processing, a scheduling algorithm (TQST) combining recognition, speculation and seduction of slow task was proposed. First of all, through the judgment of node ability and task execution time, slow node queue, very slow node queue and slow task queue were established. Secondly, according to the anticipation speculative execution value to decide how to start speculative task. Then, in the basis of node identification, avoid distributing tasks to very slow node, radically reduce slow task production, improve job execution efficiency. The experimental results show that TQST algorithm previous existing slow task scheduling al-gorithm in term of the job response time.

Key words: big data, slow task, speculative task, Map-Reduce