流式大数据平台下的弹性数据迁移能效优化策略

doi:10.11959/j.issn.1000-436x.2024006

通信学报 ›› 2024, Vol. 45 ›› Issue (2): 188-200.doi: 10.11959/j.issn.1000-436x.2024006

• 学术论文 • 上一篇

流式大数据平台下的弹性数据迁移能效优化策略

蒲勇霖¹, 许小龙¹, 于炯², 李梓杨², 国冰磊²

¹ 南京信息工程大学软件学院，江苏南京 210044
² 新疆大学软件学院，新疆乌鲁木齐 830002
³ 湖北文理学院计算机工程学院，湖北襄阳 441053

修回日期:2023-11-08 出版日期:2024-02-01 发布日期:2024-02-01
作者简介:蒲勇霖（1991− ），男，山东淄博人，博士，南京信息工程大学讲师、硕士生导师，主要研究方向为边缘计算、协同计算、绿色计算等
许小龙（1988− ），男，江苏南通人，博士，南京信息工程大学教授、博士生导师，主要研究方向为边缘计算、服务计算等
于炯（1964− ），男，北京人，博士，新疆大学教授、博士生导师，主要研究方向为网格计算、并行计算、分布式系统
李梓杨（1993− ），男，新疆乌鲁木齐人，博士，新疆大学副教授、硕士生导师，主要研究方向为大数据分析、机器学习
国冰磊（1991− ），女，湖北襄阳人，博士，湖北文理学院讲师，主要研究方向为分布式数据库系统、绿色计算
基金资助:
国家自然科学基金资助项目(62262064);新疆维吾尔自治区重点研发计划基金资助项目(2022295358);江苏省高等学校自然科学基金资助项目(23KJB520019);新疆维吾尔自治区自然科学基金资助项目(2022D01C56);湖北省自然科学基金资助项目(2022CFB805)

Energy-efficient optimization strategy based on elastic data migration in big data streaming platform

Yonglin PU¹, Xiaolong XU¹, Jiong YU², Ziyang LI², Binglei GUO²

¹ School of Software, Nanjing University of Information Science and Technology, Nanjing 210044, China
² School of Software, Xinjiang University, Urumqi 830002, China
³ School of Computer Engineering, Hubei University of Arts and Science, Xiangyang 441053, China

Revised:2023-11-08 Online:2024-02-01 Published:2024-02-01
Supported by:
The National Natural Science Foundation of China(62262064);The Key Research and Development Project of Xinjiang Uygur Autonomous Region(2022295358);The Natural Science Foundation of Jiangsu Higher Education Institutions(23KJB520019);The Natural Science Foundation of Xinjiang Uygur Autonomous Region(2022D01C56);The Natural Sci-ence Foundation of Hubei(2022CFB805)

摘要/Abstract

摘要：

针对流式计算框架在最初设计时缺乏能效方面的考虑，导致其存在高能耗与低效率的问题，提出一种流式大数据平台下的弹性数据迁移节能优化策略。首先，建立负载预测模型与资源判定模型，并进一步设计负载预测算法，通过预测负载变化趋势确定节点资源占用，找到资源过载与过剩节点；其次，建立资源约束模型与最优数据迁移模型，由此提出最优数据迁移算法，以提高节点资源利用率为目的进行数据迁移；最后，建立能耗模型，计算集群进行数据迁移后节约的能耗。实验结果表明，数据迁移节能优化策略能够对集群内节点资源变化做出及时响应，并在提高节点资源利用率的基础上，有效提高集群数据处理的能效。

关键词: 流式计算, 负载预测, 资源约束, 数据迁移, 能效

Abstract:

Focused on the problem that the stream computing platform was suffering from the high energy consumption and low efficiency due to the lack of consideration for energy efficiency in designing process, an energy-efficient optimization strategy based on elastic data migration in big data streaming platform (EEDM-BDSP) was proposed.Firstly, models of the load prediction and the resource judgment were set up, and the load prediction algorithm was designed, which predicted the load tendency and determine node resource occupancy, so as to find nodes of resource overload and redundancy.Secondly, models of the resource constraint and the optimal data migration were set up, and the optimal data migration algorithm was proposed, which data migration for the purpose of improving node resource utilization.Finally, model of the energy consumption was set up to calculate the energy consumption saved by the cluster after data migration.The experimental results show that the EEDM-BDSP changes node resources in the cluster can responded on time, the resource utilization and the energy-efficient are improved.

Key words: stream computing, load prediction, resource constraint, data migration, energy-efficient

中图分类号:

TN311

蒲勇霖, 许小龙, 于炯, 李梓杨, 国冰磊. 流式大数据平台下的弹性数据迁移能效优化策略[J]. 通信学报, 2024, 45(2): 188-200.

Yonglin PU, Xiaolong XU, Jiong YU, Ziyang LI, Binglei GUO. Energy-efficient optimization strategy based on elastic data migration in big data streaming platform[J]. Journal on Communications, 2024, 45(2): 188-200.

图/表 16

图1

图2

图3

图4

表1

表2

图5

表3

图6

图7

图8

图9

图10

图11

图12

图13

参考文献 24

[1]	LV X , YE R Q , LIU M . Green cloud 2022[R]. 2023.
[2]	WU H Q . Green ICT impact on energy conservation and emission reduction[J]. China Communications, 2008,5(3): 79-84.
[3]	KIM H , CHOI H , KANG H ,et al. A systematic review of the smart energy conservation system:from smart homes to sustainable smart cities[J]. Renewable and Sustainable Energy Reviews, 2021:doi.org/10.1016/J.RSER.2021.110755.
[4]	SEAGATE. Data age 2025[R]. 2023.
[5]	蒲勇霖, 于炯, 鲁亮 ,等. storm平台下工作节点的内存电压调控节能策略[J]. 通信学报, 2018,39(10): 97-117.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy for work node by DRAM voltage regulation in storm[J]. Journal on Communications, 2018,39(10): 97-117.
[6]	蒲勇霖, 于炯, 鲁亮 ,等. 基于 Storm 平台的数据恢复节能策略[J]. 计算机研究与发展, 2021,58(3): 479-496.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy based on data recovery in storm[J]. Journal of Computer Research and Development, 2021,58(3): 479-496.
[7]	SHIN D , JANG H , OH K ,et al. An energy-efficient DRAM cache architecture for mobile platforms with PCM-based main memory[J]. ACM Transactions on Embedded Computing Systems, 2022,21(1): 1-22.
[8]	DAI X X , YANG P , ZHANG X Y ,et al. RESPIRE:reducing spatial-temporal redundancy for efficient edge-based industrial video analytics[J]. IEEE Transactions on Industrial Informatics, 2022,18(12): 9324-9334.
[9]	ZENG Q S , DU Y Q , HUANG K B ,et al. Energy-efficient resource management for federated edge learning with CPU-GPU heterogeneous computing[J]. IEEE Transactions on Wireless Communications, 2021,20(12): 7947-7962.
[10]	SAMRIYA J K , TIWARI R , Cheng X C ,et al. Network intrusion detection using ACO-DNN model with DVFS based energy optimization in cloud framework[J]. Sustainable Computing:Informatics and Systems, 2022,35:100746.
[11]	PIETRI I , ZHUANG S C , CASAS M ,et al. Evaluating scientific workflow execution on an asymmetric multicore processor[C]// Proceedings of European Conference on Parallel Processing. Berlin:Springer, 2018: 439-451.
[12]	SHIN D J , PARK S K , KIM S M ,et al. Adaptive page grouping for energy efficiency in hybrid PRAM-DRAM main memory[C]// Proceedings of the 2012 ACM Research in Applied Computation Symposium. New York:ACM Press, 2012: 395-402.
[13]	FEI X W , LI K L , YANG W D ,et al. Analysis of energy efficiency of a parallel AES algorithm for CPU-GPU heterogeneous platforms[C]// Proceedings of 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). Piscataway:IEEE Press, 2019: 499-508.
[14]	JAVADPOUR A , SANGAIAH A K , PINTO P ,et al. An energy-optimized embedded load balancing using DVFS computing in cloud data centers[J]. Computer Communications, 2023,197: 255-266.
[15]	蒲勇霖, 于炯, 鲁亮 ,等. 基于 Storm 平台的数据迁移合并节能策略[J]. 通信学报, 2019,40(12): 68-85.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy for data migration and merging in Storm[J]. Journal on Communications, 2019,40(12): 68-85.
[16]	CHENG D Z , CHEN Y , ZHOU X B ,et al. Adaptive scheduling of parallel jobs in spark streaming[C]// Proceedings of IEEE Conference on Computer Communications. Piscataway:IEEE Press, 2017: 1-9.
[17]	XU X L , XUE Y , QI L Y ,et al. Load-aware edge server placement for mobile edge computing in 5G networks[C]// Proceedings of ServiceOriented Computing. Cham:Springer International Publishing, 2019: 494-507.
[18]	DAYARATHNA M , LI Y L , WEN Y G ,et al. Energy consumption analysis of data stream processing:a benchmarking approach[J]. Software:Practice and Experience, 2017,47(10): 1443-1462.
[19]	SUN D , ZHANG G , YANG S ,et al. Re-Stream:real-time and energy-efficient resource scheduling in big data stream computing environments[J]. Information Sciences, 2015,319: 92-112.
[20]	YANG S , LI F , SHEN M ,et al. Cloudlet placement and task allocation in mobile edge computing[J]. IEEE Internet of Things Journal, 2019,6(3): 5853-5863.
[21]	HUANG X F , ZHOU S . Dynamic compression ratio selection for edge inference systems with hard deadlines[J]. IEEE Internet of Things Journal, 2020,7(9): 8800-8810.
[22]	MATTEIS D T , MENCAGLI G . Keep calm and react with foresight[J]. ACM SIGPLAN Notices, 2016,51(8): 1-12.
[23]	蒲勇霖, 于炯, 鲁亮 ,等. Storm平台下的线程重分配与数据迁移节能策略[J]. 软件学报, 2021,32(8): 2557-2579.
	PU Y L , YU J , LU L ,et al. Energy-efficient strategy based on executor reallocation and data migration in storm[J]. Journal of Software, 2021,32(8): 2557-2579.
[24]	YANG H , LI X , QIANG W ,et al. A network traffic forecasting method based on SA optimized ARIMA-BP neural network[J]. Computer Networks, 2021,193:108102.

硬件配置项	参数
CPU	Intel core i7 4790 3.6 GHz Quad Core
内存	8 GB DDR3 1 066 MHz
硬盘	1 TB, 7 200 r/min
网络	100 Mbit/s LAN
测电器	GB/T17215-2003

软件配置项	参数
OS	CentOS 6.8
Flink	1.6.0
Hadoop	2.7.4
JDK	jdk1.8.0-181-Linux-X64
Zookeeper	3.4.10
Kafka	2.10-0.8.2.0
MySQL	5.7.27
VMware	12.1.1
MATLAB	R2014a (8.3.0.532) 64 bit

配置项	参数值	说明
JobManager.heap.size/MB	2 048	主节点内存
TaskManager.heap.size/MB	2 048	工作节点内存
TaskManager.numberOfTaskSlots	2	节点线程数目
high-availability	Zookeeper	开启HA模式
state.backend	rocksdb	状态数据存储
state.backend.incremental	true	增量式快照
TaskManager.memory.segment-size	32 758	内存分块大小

流式大数据平台下的弹性数据迁移能效优化策略

Energy-efficient optimization strategy based on elastic data migration in big data streaming platform

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 24

相关文章 15

Metrics

推荐阅读 0

[1]	孙钢灿, 王硕, 宁冰, 郝万明. 子连接有源可重构智能表面辅助的宽带无蜂窝网络能效优化[J]. 通信学报, 2024, 45(2): 127-136.
[2]	曾晓婉, 王海军, 黄蕾, 马东堂. 无人机辅助D2D通信网络安全通信资源分配算法[J]. 通信学报, 2024, 45(2): 115-126.
[3]	胡九川, 范东睿, 程建聪, 严龙, 叶笑春, 李灵枝, 万良易, 钟海斌. 内存与片上渗透缓存之间数据迁移的理论分析[J]. 通信学报, 2021, 42(8): 217-225.
[4]	王哲, 李陶深, 葛丽娜, 张桂芬, 吴敏. 基于深度学习的传感云sink节点最优能效SWIPT波束成形设计[J]. 通信学报, 2021, 42(7): 176-188.
[5]	赵飞,郝万明,孙钢灿,周一青,王飞,王毅. 基于SWIPT的毫米波大规模MIMO-NOMA系统下安全能效资源优化[J]. 通信学报, 2020, 41(8): 79-86.
[6]	徐勇军,谢豪,陈前斌,林金朝,刘期烈. 基于不完美CSI的异构NOMA网络能效优化算法[J]. 通信学报, 2020, 41(7): 131-140.
[7]	叶迎晖,施丽琴,卢光跃. 反向散射辅助的无线供能通信网络中用户能效公平性研究[J]. 通信学报, 2020, 41(7): 84-94.
[8]	赖英旭,蒲叶玮,刘静. 基于最小代价路径的交换机迁移方法研究[J]. 通信学报, 2020, 41(2): 131-142.
[9]	喻鹏, 张俊也, 李文璟, 周凡钦, 丰雷, 付澍, 邱雪松. 移动边缘网络中基于双深度Q学习的高能效资源分配方法[J]. 通信学报, 2020, 41(12): 148-161.
[10]	徐勇军,谷博文,陈前斌,林金朝. 基于能效最大的无线供电反向散射网络资源分配算法[J]. 通信学报, 2020, 41(10): 202-210.
[11]	李梓杨,于炯,王跃飞,卞琛,蒲勇霖,张译天,刘宇. Flink环境下基于负载预测的弹性资源调度策略[J]. 通信学报, 2020, 41(10): 92-108.
[12]	施丽琴,叶迎晖,卢光跃. 无线供能边缘计算网络中系统计算能效最大化资源分配方案[J]. 通信学报, 2020, 41(10): 59-69.
[13]	李梓杨,于炯,卞琛,张译天,蒲勇霖,王跃飞,鲁亮. 基于流网络的Flink平台弹性资源调度策略[J]. 通信学报, 2019, 40(8): 85-101.
[14]	石乐义,郭宏彬,温晓,李剑蓝,崔玉文,马猛飞,孙慧. 端信息跳扩混合的主动网络防御技术研究[J]. 通信学报, 2019, 40(5): 125-135.
[15]	蒲勇霖,于炯,鲁亮,李梓杨,卞琛,廖彬. 基于Storm平台的数据迁移合并节能策略[J]. 通信学报, 2019, 40(12): 68-85.