基于Storm平台的数据迁移合并节能策略

doi:10.11959/j.issn.1000-436x.2019226

Abstract

Abstract:

Storm is suffering the problems of high energy consumption but low efficiency.Aiming at this problem,the resource constraint model,the optimal principle of data reorganization in executors and node voltage reduction principle were proposed based on the analysis of the architecture and topology of Storm,and further the energy-efficient strategy for data migration and merging was put forward in Storm(DMM-Storm),which was composed of resource constraint algorithm,data migration and merging algorithm as well as node voltage reduction algorithm.The resource constraint algorithm estimates whether work nodes are appropriate for data migration according to the resource constraint model.The data migration and merging algorithm designs an optimal method to migrate data according to the the optimal principle of data reorganization in executors.The node voltage reduction algorithm reduces voltage of work nodes according to node voltage reduction principle.The experimental results show that the DMM-Storm can reduce energy consumption efficiently without affecting the performance of cluster compared with the existing researches.

Key words: big data, Storm, resource constraint, data migration, energy consumption

CLC Number:

TP311

Yonglin PU,Jiong YU,Liang LU,Ziyang LI,Chen BIAN,Bin LIAO. Energy-efficient strategy for data migration and merging in Storm[J]. Journal on Communications, 2019, 40(12): 68-85.

Figures/Tables 17

References 42

[1]	孟小峰, 慈祥 . 大数据管理:概念、技术与挑战[J]. 计算机研究与发展, 2013,50(1): 146-169.
	MENG X F , CI X . Big data management:concepts,techniques and challenges[J]. Journal of Computer Research and Development, 2013,50(1): 146-169.
[2]	RANJAN R . Streaming big data processing in datacenter clouds[J]. IEEE Cloud Computing, 2014,1(1): 78-83.
[3]	CHEN C L P , ZHANG C Y . Data-intensive applications,challenges,techniques and technologies:a survey on big data[J]. Information Sciences, 2014,275(11): 314-347.
[4]	孙大为 . 大数据流式计算:应用特征和技术挑战[J]. 大数据, 2015,1(3): 99-105.
	SUN D W . Big data stream computing:features and challenges[J]. Big Data Research, 2015,1(3): 99-105.
[5]	KAMBATLA K , KOLLIAS G , KUMAR V ,et al. Trends in big data analytics[J]. Journal of Parallel and Distributed Computing, 2014,74(7): 2561-2573.
[6]	杨挺, 王萌, 张亚健 ,等. 云计算数据中心 HDFS 差异性存储节能优化算法[J]. 计算机学报, 2019(4): 721-735.
	YANG T , WANG M , ZHANG Y J ,et al. HDFS differential storage energy-saving optimal algorithm in cloud data center[J]. Chinese Journal of Computers, 2019(4): 721-735.
[7]	余晓晖, . 数据中心能效测评指南[R]. “云计算发展与政策论坛”技术报告,(2012-03-16)[2019-07-04].
	YU X H . Data center energy efficiency assessment guide[R]. Cloud Computing Development and Policy Forum Technical Report,(2012-03-16)[2019-07-04].
[8]	陈小燕, 干丽萍, 郭文平 . 大数据可视化工具比较及应用[J]. 计算机教育, 2018,282(6): 100-105.
	CHEN X Y , GAN L P , GUO W P . Comparison and application of big data visualization tools[J]. Computer Education, 2018,282(6): 100-105.
[9]	SUN D , ZHANG G , YANG S ,et al. Re-Stream:real-time and energy-efficient resource scheduling in big data stream computing environments[J]. Information Sciences, 2015,319: 92-112.
[10]	鲁亮, 于炯, 卞琛 ,等. 大数据流式计算框架 Storm 的任务迁移策略[J]. 计算机研究与发展, 2018,55(1): 71-92.
	LU L , YU J , BIAN C ,et al. A task migration strategy in big data stream computing with Storm[J]. Journal of Computer Research and Development, 2018,55(1): 71-92.
[11]	BORTHAKUR D , GRAY J , SARMA J S ,et al. Apache Hadoop goes realtime at Facebook[C]// The 2011 ACM SIGMOD International Conference on Management of Data. ACM, 2011: 1071-1080.
[12]	NEUMEYER L , ROBBINS B , NAIR A , KESARI A . S4:distributed stream computing platform[C]// The 10th IEEE International Conference on Data Mining Workshops (ICDMW 2010). IEEE, 2010: 170-177.
[13]	李梓杨, 于炯, 卞琛 ,等. 基于流网络的Flink平台弹性资源调度策略[J]. 通信学报, 2019,40(8): 85-101.
	LI Z Y , YU J , BIAN C ,et al. Flow-network based auto rescale strategy for Flink[J]. Journal on Communications, 2019,40(8): 85-101.
[14]	卞琛, 于炯, 修位蓉 ,等. 基于分配适应度的 Spark 渐进填充分区映射算法[J]. 通信学报, 2017,38(9): 133-147.
	BIAN C , YU J , XIU W R ,et al. Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree[J]. Journal on Communications, 2017,38(9): 133-147.
[15]	KULKARNI S , BHAGAT N , FU M ,et al. Twitter heron:Stream processing at scale[C]// The 2015 ACM SIGMOD International Conference on Management of Data. ACM, 2015: 239-250.
[16]	ANDERSON Q . Storm real-time processing cookbook[M]. Birmingham: Packt PublishingPress, 2013: 4-8.
[17]	TA V D , LIU C M , NKABINDE G W . Big data stream computing in healthcare real-time analytics[C]// The 2016 IEEE International Conference on Cloud Computing and Big Data Analysis (ICCCBDA). IEEE, 2016: 37-42.
[18]	MISHNE G , DALTON J , LI Z ,et al. Fast data in the era of big data:Twitter’s real-time related query suggestion architecture[C]// The 2013 ACM SIGMOD International Conference on Management of Data. ACM, 2013: 1147-1158.
[19]	DING W L , HAN Y B , ZHAO Z F ,et al. Stream-oriented availability services for endpoint-to-endpoint data transmission[C]// The 2012 In ternational Conference on Cloud and Service Computing. IEEE, 2012: 212-218.
[20]	SHIN D J , PARK S K , KIM S M ,et al. Adaptive page grouping for energy efficiency in hybrid PRAM-DRAM main memory[C]// ACM Research in Applied Computation Symposium. ACM, 2012: 395-402.
[21]	BONAMY R , BILAVARN S , MULLER F . An energy-aware scheduler for dynamically reconfigurable multi-core systems[C]// International Symposium on Reconfigurable Communication-Centric Systems-On-Chip. IEEE, 2015: 1-6.
[22]	KIM H S , SHIN D I , YU Y J ,et al. Towards energy proportional cloud for data processing frameworks[M]. San Jose: USENIX AssociationPress, 2010: 1-8.
[23]	FAISAL S M , TZIANTZIOULIS G , GOK A M ,et al. Edge importance identification for energy efficient graph processing[C]// IEEE International Conference on Big Data. IEEE, 2015: 347-354.
[24]	SONG J , MA Z , THOMAS R ,et al. Energy efficiency optimization in big data processing platform by improving resources utilization[J]. Sustainable Computing:Informatics and Systems, 2019,21: 80-89.
[25]	MU J , PEI Y , LI W ,et al. Research on energy saving optimization strategy of substation operation based on big data technology[C]// 2018 Chinese Control And Decision Conference (CCDC). IEEE, 2018: 3567-3571.
[26]	DE MATTEIS T , MENCAGLI G . Keep calm and react with foresight:strategies for low-latency and energy-efficient elastic data stream processing[J]. Journal of Systems and Software, 2016,51(8): 1-12.
[27]	LEVERICH J , KOZYRAKIS C . On the energy (in) efficiency of Hadoop clusters[J]. ACM SIGOPS Operating Systems Review, 2010,44(1): 61-65.
[28]	LANG W , PATEL J M . Energy management for MapReduce clusters[J]. Proceedings of the VLDB Endowment, 2010,3(1-2): 129-139.
[29]	宋杰, 李甜甜, 朱志良 ,等. 云数据管理系统能耗基准测试与分析[J]. 计算机学报, 2017,36(7): 1485-1499.
	SONG J , LI T T , ZHU Z L ,et al. Benchmarking and analyzing the energy consumption of cloud data management system[J]. Chinese Journal of Computers, 2013,36(7): 1485-1499.
[30]	廖彬, 张陶, 于炯 ,等. MapReduce 能耗建模及优化分析[J]. 计算机研究与发展, 2016,53(9): 2107-2131.
	LIAO B , ZHANG T , YU J ,et al. Energy consumption modeling and optimization analysis for MapReduce[J]. Journal of Computer Research and Development, 2016,53(9): 2107-2131.
[31]	LIAO B , YU J , ZHANG T ,et al. Energy-efficient algorithms for distributed storage system based on block storage structure reconfiguration[J]. Journal of Computer Research ＆ Development, 2015,48(2): 71-86.
[32]	SHIN D J , PARK S K , KIM S M ,et al. Adaptive page grouping for energy efficiency in hybrid PRAM-DRAM main memory[C]// ACM Research in Applied Computation Symposium. ACM, 2012: 395-402.
[33]	ZHOU S , CHELMIS C , PRASANNA V K . High-Throughput and Energy-Efficient Graph Processing on FPGA[C]// International Symposium on Field-Programmable Custom Computing Machines. IEEE, 2016: 103-110.
[34]	廖彬, 张陶, 于炯 ,等. 温度感知的MapReduce节能任务调度策略[J]. 通信学报, 2016,37(1): 61-75.
	LIAO B , ZHANG T , YU J . Temperature aware energy-efficient task scheduling strategies for MapReduce[J]. Journal on Communications, 2016,37(1): 61-75.
[35]	VASUDEVAN V , FRANKLIN J , ANDERSEN D . FAWN damentally power-efficient clusters[C]// The 12th Workshop on Hot Topics in Operating Systems (HotOS 09?). Usenix Association, 2009: 1-5.
[36]	廖彬, 于炯, 孙华 ,等. 基于存储结构重配置的分布式存储系统节能算法[J]. 计算机研究与发展, 2013,50(1): 3-18.
	LIAO B , YU J , SUN H ,et al. Energy-efficient algorithms for distributed storage system based on data storage structure reconfiguration[J]. Journal of Computer Research and Development, 2013,50(1): 3-18.
[37]	GUO B , YU J , LIAO B ,et al. A green framework for DBMS based on energy-aware query optimization and energy-efficient query processing[J]. Journal of Network and Computer Applications, 2017,84: 118-130.
[38]	WANG Z , WANG H , ZHAO W ,et al. Energy optimization of parallel programs in a heterogeneous system by combining processor core-shutdown and dynamic voltage scaling[J]. Future Generation Computer Systems, 2019,92: 198-209.
[39]	CORDESCHI N , SHOJAFAR M , AMENDOLA D ,et al. Energy-efficient adaptive networked datacenters for the QoS support of real-time applications[J]. The Journal of Supercomputing, 2014,71(2): 448-478.
[40]	PANDA A , CHATHA K S . An embedded architecture for energy-efficient stream computing[J]. IEEE Embedded Systems Letters, 2014,6(3): 57-60.
[41]	ZONG Z , MANZANARES A , RUAN X ,et al. EAD and PEBD:two energy-aware duplication scheduling algorithms for parallel tasks on homogeneous clusters[J]. IEEE Transactions on Computers, 2010,60(3): 360-374.
[42]	蒲勇霖, 于炯, 鲁亮 ,等. Storm平台下工作节点的内存电压调控节能策略[J]. 通信学报, 2018,39(10): 101-121.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy for work node by DRAM voltage regulation in Storm[J]. Journal on Communications, 2018,39(10): 101-121.

Metrics

Recommended 0

No Suggested Reading articles found!

节点	CPU	内存	网络带宽
NimbusZooKeeper₁ (Leader)	Intel core i7 4790 3.6 GHz Quad Core	8 GB DDR3 1 066 MHz	100 Mbit/s LAN
Supervisor₁~Supervisor₁₆	Intel core i7 4790 3.6 GHz Quad Core	8 GB DDR3 1 066 MHz	100 Mbit/s LAN
Zookeeper₂、Zookeeper₃ (Follower)	Intel core i7 4790 3.6 GHz Quad Core	8 GB DDR3 1 066 MHz	100 Mbit/s LAN

参数	数值
OS	CentOS 6.4
Storm	1.2.2
JDK	1.7 Open JDK
Zookeeper	3.4.11
Python	2.6
MySQL	5.1.73
VMware	12.1.1

基准测试	参数	数值
	component.spout_num	60
	component.split_bolt_num	120
WordCount	component.count_bolt_num	120
	topology.works	16
	topology.acker.executors	16
	topology.max.spout.pending	200
	component.spout_num	60
	component.sort_bolt_num	120
RollingSort	emit.frequency	10
	chunk.size	2 000 000
	message.size	100 000
	topology.level	3
Sol	message.size	2 000
	component.spout_num	60
	component.bolt_num	120
	component.spout_num	60
	component.split_bolt_num	120
RollingCount	component.rolling_count_bolt_num	120
	window.length	150
	emit.frequency	30

参数	数值
component.spout_num	60
component.split_bolt_num	120
component.rolling_count_bolt_num	120
topology.max.spout.pending	200
topology.workers	16
topology.acker.executors	16
T	30 s
β	65%
γ	50 000 tuple/s
执行DVRCP内存电压的取值范围	1.25~1.50 V
执行DVRNP内存电压的取值范围	1.36~1.50 V

基准测试	算法	CPU占用率	网络带宽占用率	内存占用率
	原系统	53.8%	49.4%	53.5%
RollingCount	DMMNE	72.5%	70.5%	73.5%
	DMMCE	73.1%	69.2%	74.3%
	原系统	72.6%	71.3%	61.4%
WordCount	DMMNE	86.3%	88.3%	80.7%
	DMMCE	85.2%	89.2%	79.8%
	原系统	24.5%	69.6%	51.3%
Sol	DMMNE	60.6%	88.7%	72.8%
	DMMCE	63.4%	87.3%	71.9%
	原系统	58.7%	51.2%	49.8%
RollingSort	DMMNE	72.4%	75.4%	74.8%
	DMMCE	73.7%	76.4%	75.6%

Energy-efficient strategy for data migration and merging in Storm

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 42

Related Articles 15

Metrics

Recommended 0

实验组	最小功耗/W	最大功耗/W	平均功耗/W
test11	1 000.787 46	1 183.516 94	1 083.995 012
test12	840.833 95	985.700 53	905.268 979 5
test13	820.316 05	985.104 32	911.453 409 2
test21	1 015.426 56	1 195.926 06	1 125.715 989
test22	867.373 92	936.333 72	915.091 409 7
test23	852.070 68	947.363 31	911.314 227 2
test31	962.508 52	1192.577 08	1 076.023 367
test32	856.864 18	965.197 93	926.058 879 5
test33	832.394 69	959.708 03	905.334 361 6

[1]	Weifeng LU, Ning LI, Jia XU, Lijie XU, Jian XU. Joint scheduling algorithm for correlative tasks in multi-access edge computing [J]. Journal on Communications, 2023, 44(4): 87-98.
[2]	Xueyong YU, Lixiang QIU, Jianing SONG, Hongbo ZHU. Security communication and energy efficiency optimization strategy in UAV-aided edge computing [J]. Journal on Communications, 2023, 44(3): 45-54.
[3]	Wei JIN, Fenghua LI, Mingjie YU, Yunchuan GUO, Ziyan ZHOU, Liang FANG. HDFS-oriented cryptographic key resource control mechanism [J]. Journal on Communications, 2022, 43(9): 27-41.
[4]	Cuiran LI, Xuejie WANG, Jianli XIE, Anqi LYU. Routing algorithm for railway monitoring linear WSN based on improved PSO [J]. Journal on Communications, 2022, 43(5): 155-165.
[5]	Yimin MAO, Qianhu DENG, Zhigang CHEN. Parallel association rules incremental mining algorithm based on information entropy and genetic algorithm [J]. Journal on Communications, 2021, 42(5): 122-136.
[6]	Liang YUAN, Xiao YU, Enjie DING, Xiaohu ZHAO, Shimin FENG, Da ZHANG, Tongyu LIU, Weidong WANG, Yanqiu HUANG. Research on key technologies of human-machine-environment states perception in mine Internet of things [J]. Journal on Communications, 2020, 41(2): 1-12.
[7]	. Dual-architecture Internet supporting intelligent governance of cyber content [J]. Journal on Communications, 2019, 40(9): 1-14.
[8]	Leyi SHI,Hongbin GUO,Xiao WEN,Jianlan LI,Yuwen CUI,Mengfei MA,Hui SUN. Research on end hopping and spreading for active cyber defense [J]. Journal on Communications, 2019, 40(5): 125-135.
[9]	Yu FU, Yihan YU, Xiaoping WU. Differential privacy protection technology and its application in big data environment [J]. Journal on Communications, 2019, 40(10): 157-168.
[10]	Wei LIU,Shu XIONG,Wei DU,Wei WANG. Research on Cloudlet selection strategy for data streaming applications in mobile cloud environment [J]. Journal on Communications, 2019, 40(1): 87-101.
[11]	Yonglin PU,Jiong YU,Liang LU,Chen BIAN,Bin LIAO,Ziyang LI. Energy-efficient strategy for work node by DRAM voltage regulation in storm [J]. Journal on Communications, 2018, 39(10): 97-117.
[12]	Kun-fang ZHANG,Ming-ming LU,Lin ZHENG. Big data based metro crowd delivery system [J]. Journal on Communications, 2017, 38(Z2): 99-112.
[13]	Le-tian SHA,Fu XIAO,Wei CHEN,Jing SUN,Ru-chuan WANG. Sensitive information leakage awareness method for big data platform based on multi-attributes decision-making and taint tracking [J]. Journal on Communications, 2017, 38(7): 56-69.
[14]	Min NIE,Li SHI,Guang YANG,Chang-xing PEI. Influence of thunderstorm cloud on the performance of satellite-to-ground quantum link and parameters simulation [J]. Journal on Communications, 2017, 38(5): 31-38.
[15]	Jun-xin SHEN,Ying-qian CHEN. Research on network analysis method for development ability of big data industry in underdeveloped area [J]. Journal on Communications, 2017, 38(12): 153-159.