流式大数据平台下的弹性数据迁移能效优化策略

doi:10.11959/j.issn.1000-436x.2024006

Abstract

Abstract:

Focused on the problem that the stream computing platform was suffering from the high energy consumption and low efficiency due to the lack of consideration for energy efficiency in designing process, an energy-efficient optimization strategy based on elastic data migration in big data streaming platform (EEDM-BDSP) was proposed.Firstly, models of the load prediction and the resource judgment were set up, and the load prediction algorithm was designed, which predicted the load tendency and determine node resource occupancy, so as to find nodes of resource overload and redundancy.Secondly, models of the resource constraint and the optimal data migration were set up, and the optimal data migration algorithm was proposed, which data migration for the purpose of improving node resource utilization.Finally, model of the energy consumption was set up to calculate the energy consumption saved by the cluster after data migration.The experimental results show that the EEDM-BDSP changes node resources in the cluster can responded on time, the resource utilization and the energy-efficient are improved.

Key words: stream computing, load prediction, resource constraint, data migration, energy-efficient

CLC Number:

TN311

Yonglin PU, Xiaolong XU, Jiong YU, Ziyang LI, Binglei GUO. Energy-efficient optimization strategy based on elastic data migration in big data streaming platform[J]. Journal on Communications, 2024, 45(2): 188-200.

Figures/Tables 16

References 24

[1]	LV X , YE R Q , LIU M . Green cloud 2022[R]. 2023.
[2]	WU H Q . Green ICT impact on energy conservation and emission reduction[J]. China Communications, 2008,5(3): 79-84.
[3]	KIM H , CHOI H , KANG H ,et al. A systematic review of the smart energy conservation system:from smart homes to sustainable smart cities[J]. Renewable and Sustainable Energy Reviews, 2021:doi.org/10.1016/J.RSER.2021.110755.
[4]	SEAGATE. Data age 2025[R]. 2023.
[5]	蒲勇霖, 于炯, 鲁亮 ,等. storm平台下工作节点的内存电压调控节能策略[J]. 通信学报, 2018,39(10): 97-117.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy for work node by DRAM voltage regulation in storm[J]. Journal on Communications, 2018,39(10): 97-117.
[6]	蒲勇霖, 于炯, 鲁亮 ,等. 基于 Storm 平台的数据恢复节能策略[J]. 计算机研究与发展, 2021,58(3): 479-496.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy based on data recovery in storm[J]. Journal of Computer Research and Development, 2021,58(3): 479-496.
[7]	SHIN D , JANG H , OH K ,et al. An energy-efficient DRAM cache architecture for mobile platforms with PCM-based main memory[J]. ACM Transactions on Embedded Computing Systems, 2022,21(1): 1-22.
[8]	DAI X X , YANG P , ZHANG X Y ,et al. RESPIRE:reducing spatial-temporal redundancy for efficient edge-based industrial video analytics[J]. IEEE Transactions on Industrial Informatics, 2022,18(12): 9324-9334.
[9]	ZENG Q S , DU Y Q , HUANG K B ,et al. Energy-efficient resource management for federated edge learning with CPU-GPU heterogeneous computing[J]. IEEE Transactions on Wireless Communications, 2021,20(12): 7947-7962.
[10]	SAMRIYA J K , TIWARI R , Cheng X C ,et al. Network intrusion detection using ACO-DNN model with DVFS based energy optimization in cloud framework[J]. Sustainable Computing:Informatics and Systems, 2022,35:100746.
[11]	PIETRI I , ZHUANG S C , CASAS M ,et al. Evaluating scientific workflow execution on an asymmetric multicore processor[C]// Proceedings of European Conference on Parallel Processing. Berlin:Springer, 2018: 439-451.
[12]	SHIN D J , PARK S K , KIM S M ,et al. Adaptive page grouping for energy efficiency in hybrid PRAM-DRAM main memory[C]// Proceedings of the 2012 ACM Research in Applied Computation Symposium. New York:ACM Press, 2012: 395-402.
[13]	FEI X W , LI K L , YANG W D ,et al. Analysis of energy efficiency of a parallel AES algorithm for CPU-GPU heterogeneous platforms[C]// Proceedings of 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). Piscataway:IEEE Press, 2019: 499-508.
[14]	JAVADPOUR A , SANGAIAH A K , PINTO P ,et al. An energy-optimized embedded load balancing using DVFS computing in cloud data centers[J]. Computer Communications, 2023,197: 255-266.
[15]	蒲勇霖, 于炯, 鲁亮 ,等. 基于 Storm 平台的数据迁移合并节能策略[J]. 通信学报, 2019,40(12): 68-85.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy for data migration and merging in Storm[J]. Journal on Communications, 2019,40(12): 68-85.
[16]	CHENG D Z , CHEN Y , ZHOU X B ,et al. Adaptive scheduling of parallel jobs in spark streaming[C]// Proceedings of IEEE Conference on Computer Communications. Piscataway:IEEE Press, 2017: 1-9.
[17]	XU X L , XUE Y , QI L Y ,et al. Load-aware edge server placement for mobile edge computing in 5G networks[C]// Proceedings of ServiceOriented Computing. Cham:Springer International Publishing, 2019: 494-507.
[18]	DAYARATHNA M , LI Y L , WEN Y G ,et al. Energy consumption analysis of data stream processing:a benchmarking approach[J]. Software:Practice and Experience, 2017,47(10): 1443-1462.
[19]	SUN D , ZHANG G , YANG S ,et al. Re-Stream:real-time and energy-efficient resource scheduling in big data stream computing environments[J]. Information Sciences, 2015,319: 92-112.
[20]	YANG S , LI F , SHEN M ,et al. Cloudlet placement and task allocation in mobile edge computing[J]. IEEE Internet of Things Journal, 2019,6(3): 5853-5863.
[21]	HUANG X F , ZHOU S . Dynamic compression ratio selection for edge inference systems with hard deadlines[J]. IEEE Internet of Things Journal, 2020,7(9): 8800-8810.
[22]	MATTEIS D T , MENCAGLI G . Keep calm and react with foresight[J]. ACM SIGPLAN Notices, 2016,51(8): 1-12.
[23]	蒲勇霖, 于炯, 鲁亮 ,等. Storm平台下的线程重分配与数据迁移节能策略[J]. 软件学报, 2021,32(8): 2557-2579.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy based on executor reallocation and data migration in storm[J]. Journal of Software, 2021,32(8): 2557-2579.
[24]	YANG H , LI X , QIANG W ,et al. A network traffic forecasting method based on SA optimized ARIMA-BP neural network[J]. Computer Networks, 2021,193:108102.

Metrics

Recommended 0

No Suggested Reading articles found!

硬件配置项	参数
CPU	Intel core i7 4790 3.6 GHz Quad Core
内存	8 GB DDR3 1 066 MHz
硬盘	1 TB, 7 200 r/min
网络	100 Mbit/s LAN
测电器	GB/T17215-2003

软件配置项	参数
OS	CentOS 6.8
Flink	1.6.0
Hadoop	2.7.4
JDK	jdk1.8.0-181-Linux-X64
Zookeeper	3.4.10
Kafka	2.10-0.8.2.0
MySQL	5.7.27
VMware	12.1.1
MATLAB	R2014a (8.3.0.532) 64 bit

配置项	参数值	说明
JobManager.heap.size/MB	2 048	主节点内存
TaskManager.heap.size/MB	2 048	工作节点内存
TaskManager.numberOfTaskSlots	2	节点线程数目
high-availability	Zookeeper	开启HA模式
state.backend	rocksdb	状态数据存储
state.backend.incremental	true	增量式快照
TaskManager.memory.segment-size	32 758	内存分块大小

Energy-efficient optimization strategy based on elastic data migration in big data streaming platform

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 24

Related Articles 13

Metrics

Recommended 0

[1]	Peng YU, Junye ZHANG, Wenjing LI, Fanqin ZHOU, Lei FENG, Shu FU, Xuesong QIU. Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning [J]. Journal on Communications, 2020, 41(12): 148-161.
[2]	Ziyang LI,Jiong YU,Yuefei WANG,Chen BIAN,Yonglin PU,Yitian ZHANG,Yu LIU. Load prediction based elastic resource scheduling strategy in Flink [J]. Journal on Communications, 2020, 41(10): 92-108.
[3]	Ziyang LI,Jiong YU,Chen BIAN,Yitian ZHANG,Yonglin PU,Yuefei WANG,Liang LU. Flow-network based auto rescale strategy for Flink [J]. Journal on Communications, 2019, 40(8): 85-101.
[4]	Leyi SHI,Hongbin GUO,Xiao WEN,Jianlan LI,Yuwen CUI,Mengfei MA,Hui SUN. Research on end hopping and spreading for active cyber defense [J]. Journal on Communications, 2019, 40(5): 125-135.
[5]	Yonglin PU,Jiong YU,Liang LU,Ziyang LI,Chen BIAN,Bin LIAO. Energy-efficient strategy for data migration and merging in Storm [J]. Journal on Communications, 2019, 40(12): 68-85.
[6]	Yonglin PU,Jiong YU,Liang LU,Chen BIAN,Bin LIAO,Ziyang LI. Energy-efficient strategy for work node by DRAM voltage regulation in storm [J]. Journal on Communications, 2018, 39(10): 97-117.
[7]	Ying HU,Bao-feng JI,Yong-ming HUANG,Fei YU,Lv-xi YANG. Energy-efficient resource allocation algorithm for massive MIMO OFDMA downlink system [J]. Journal on Communications, 2015, 36(7): 1-49.
[8]	Xu LU,Liang-lun CHENG,Shi-liang LUO. Adaptive node scheduling algorithm for target tracking in wireless sensor networks [J]. Journal on Communications, 2015, 36(4): 70-80.
[9]	UYing H,IBao-feng J,UANGYong-ming H,UFei Y,ANGLV-xi Y. Energy-efficient optimal algorithm based on uplink multi-user very large MIMO system [J]. Journal on Communications, 2015, 36(1): 68-74.
[10]	Xiao-ping FAN,Zhe-yuan XIONG,Zhi-jie CHEN,Shao-qiang LIU,Zhi-hua QU. Video coding in wireless multimedia sensor networks:a survey [J]. Journal on Communications, 2011, 32(9): 137-146.
[11]	Xu LIU,Jin-bao LI,Long-jiang GUO,Bin FAN,Jing-hua ZHU. Energy-efficient algorithm for mobile target prediction and tracking in wireless sensor networks [J]. Journal on Communications, 2010, 31(9A): 135-143.
[12]	Shu-gang LIU,Hong-li LIU,Jie ZHAN,Yao-nan WANG. Research of energy-efficient hybrid communication algorithm in wireless sensor networks [J]. Journal on Communications, 2009, 30(1): 12-17.
[13]	Yu-hong LUO,Jian-xin WANG,Song-qiao CHEN. Distributed multicast algorithm for building an energy-efficient shared tree in ad hoc networks [J]. Journal on Communications, 2006, 27(6): 1-9.