基于多变量动态算法的Hadoop负载均衡优化与实现

doi:10.3969/j.issn.1000-0801.2012.12.019

Abstract

Abstract:

Cloud computing system and Hadoop load balance mechanism were introduced particularly.When Hadoop system needs load balance,the user must call the system's balancer with the threshold being input in advance which restrict the performance of the balancer greatly.In this case,a multivariable dynamic evaluate threshold algorithm was used to optimize and realize the balancer.Multivariable dynamic evaluate threshold algorithm is one of the algorithm based on the current state of the cluster.It used the disk space utilization and cluster busy degree to dynamic evaluates the threshold of balancer.This algorithm not only can dynamic evaluates the balancer's threshold,but also can balance the desired balanced result and the running time by means of selection suitable weight parameters.Results of the experiment shows that comparing with the system default balancer,which needs a fixed threshold.The balancer using multivariable dynamic evaluate threshold algorithm would be more satisfactory.

Key words: cloud computing, HDFS, load balance, threshold

Juan Wu,Hai Huang,Feng Qian,Yongjun Li,Zhibin Shou. Optimization and Realization of Hadoop Load Balance Mechanism Based on a Multivariable Dynamic Evaluate Threshold Algorithm[J]. Telecommunications Science, 2012, 28(12): 124-128.

	容量（GB）	已占用空间（GB）	已占用空间百分比	区块
DataNode1	71.93	38.45	53.46%	649
DataNode2	72.42	42.37	58.5%	719
DataNode3	71.5	18.19	25.44%	328
DataNode4	71.5	17.25	24.12%	324
DataNode5	72.42	15.63	21.59%	295
DataNode6	71.5	43.63	61.02%	757
DataNode7	71.5	43.94	61.45%	764
DataNode8	71.5	38.74	54.18%	666

	容量（GB）	已占用空间（GB）	已占用空间百分比	区块
DataNode1	71.93	38.45	53.46%	657
DataNode2	72.42	42.37	52.33%	637
DataNode3	71.5	18.19	37.97%	439
DataNode4	71.5	17.25	37.42%	427
DataNode5	72.42	15.63	36.07%	403
DataNode6	71.5	43.63	54.34%	665
DataNode7	71.5	43.94	53.27%	652
DataNode8	71.5	38.74	52.68%	614

	容量（GB）	已占用空间（GB）	已占用空间百分比	区块
DataNode1	71.93	33.52	46.6%	570
DataNode2	72.42	32.55	44.94%	554
DataNode3	71.5	31.09	43.48%	546
DataNode4	71.5	30.19	42.4%	541
DataNode5	72.42	28.99	40.04%	529
DataNode6	71.5	34.75	48.61%	615
DataNode7	71.5	34.51	48.26%	639
DataNode8	71.5	32.62	45.63%	603

2	（美）怀特 . Hadoop权威指南. 北京：清华大学出版社， 2011
	Borthakur D . The Hadoop distributed file system:architecture and design. , 2011
	林伟伟 . 一种改进的Hadoop数据放置策略. 华南理工大学学报（自然科学版）， 2012,40（1）:152～158
	Myint J , Naing T T . A data placement algorithm with binary weighted tree on PC cluster-based cloud storage system[C]. Cloud and Service Computing （CSC）,2011 International Conference on, Hong kong,China, 2011:315～320

[1]	Xue HAN. Damage toughness assessment method of power backbone communication network based on power big data [J]. Telecommunications Science, 2023, 39(5): 136-143.
[2]	Wenjuan XING, Bo LEI, Qianying ZHAO. Development status and trend prospect of computing power infrastructure [J]. Telecommunications Science, 2022, 38(6): 51-61.
[3]	Pingke DENG, Tongxu ZHANG, Nanxiang SHI, Tong ZHANG, Tianzhu SHAO, Shaowen ZHENG. Computing satellite networks—the novel development of computing-empowered space-air-ground integrated networks [J]. Telecommunications Science, 2022, 38(6): 71-81.
[4]	Jiangxing WU. Revolution of the development paradigm of network technology system—network of networks [J]. Telecommunications Science, 2022, 38(6): 3-12.
[5]	Kun MA, Lingyu XU, Xiaoping SHEN, Zhicheng GONG, Jianping LAN, Shuangxi CHEN, Jun QIAN. Virtual machine scheduling model based on Shapley value modified genetic algorithm in cloud computing [J]. Telecommunications Science, 2022, 38(12): 1-10.
[6]	Zhiyong LIU, Zhongjiang HE, Yilong RUAN, Junfeng SHAN, Chao ZHANG. Big data security features and operation practices [J]. Telecommunications Science, 2021, 37(5): 160-169.
[7]	Jian XU,Li JIN. A quantifiable cloud computing platform security evaluation model [J]. Telecommunications Science, 2020, 36(7): 163-167.
[8]	Sining LUO,Hualong WANG,Hongyu LI,Wei PENG. Improved ant colony algorithm based cloud computing user task scheduling algorithm [J]. Telecommunications Science, 2020, 36(2): 95-100.
[9]	Shunlan LIU, Yide XIAO, Jianrong BAO. Dual threshold cooperative spectrum sensing algorithm based on stochastic resonance [J]. Telecommunications Science, 2020, 36(12): 33-40.
[10]	Qichao SUN,Jianrong BAO,Chao LIU,Bin JIANG,Fang ZHU,Jianhai HE. Improved Polar selective decoding and forwarding cooperation [J]. Telecommunications Science, 2020, 36(1): 77-87.
[11]	Bo LEI, Zengyi LIU, Xuliang WANG, Mingchuan ·ANG, ·unqing CHEN. Computing network:a new multi-access edge computing [J]. Telecommunications Science, 2019, 35(9): 44-51.
[12]	Yongmei GAO,Guanjie CHENG. Data-intensive service deployment based on edge computing [J]. Telecommunications Science, 2019, 35(7): 87-99.
[13]	Zelin LU,Qiangzhi LI. Evolutionary logic and operational mechanism of E-commerce platform [J]. Telecommunications Science, 2019, 35(7): 152-158.
[14]	Zhilan HUANG,Yongbing FAN,Ning FAN,Nan CHEN,Linze WU,Baohong LIN. Solution and key technology of integrating vFW in cloud platform [J]. Telecommunications Science, 2019, 35(5): 140-148.
[15]	Youkang ZHU,Guangxue YUE,Xiaohui YANG,Jiansheng LIU. A survey on edge computing offloading [J]. Telecommunications Science, 2019, 35(4): 74-94.

Optimization and Realization of Hadoop Load Balance Mechanism Based on a Multivariable Dynamic Evaluate Threshold Algorithm

RichHTML

PDF下载

Knowledge

Cited

Abstract

Cite this article

share this article

Figures/Tables 6

References 2

Related Articles 15

Metrics

Recommended 0