Abstract:

Hadoop has become a major platform for big data storage and large data mining nowadays.Although Hadoop platform achieves high performance parallel computing through a distributed cluster of machines,the bottlenecks will inevitably appear on a machine when cluster load increases,because the cluster is composed of inexpensive host.Aiming at this problem,a bottleneck detection algorithms based on information gain was proposed.The algorithm detected cluster's bottlenecks resource by computing the information gain of each resource.The experiments show that the bottleneck detection algorithm is feasible.

Key words: big data, Hadoop, information gain, bottleneck detection