一种基于改进的链式MapReduce的并行ETL应用

doi:10.3969/j.issn.1000-0801.2013.12.001

电信科学 ›› 2013, Vol. 29 ›› Issue (12): 1-8.doi: 10.3969/j.issn.1000-0801.2013.12.001

• 研究与开发 • 下一篇

一种基于改进的链式MapReduce的并行ETL应用

吴斌,刘心光

北京邮电大学计算机学院通信软件工程中心北京100876

出版日期:2013-12-20 发布日期:2017-07-04
基金资助:
国家自然科学基金资助项目

A Parallel ETL Tool Based on an Improved Chain-MapReduce Framework

Bin Wu,Xinguang Liu

Telecommunication and Software Engineering Center, School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China

Online:2013-12-20 Published:2017-07-04

摘要/Abstract

摘要：

介绍了并行ETL 的相关工作和常见的处理多MapReduce 作业流程的方法；提出一种改进的链式MapReduce 框架，并将此框架应用于一个并行ETL 工具，同时提出一些针对ETL 处理的流程级优化规则，使ETL流程产生更少的MapReduce作业，从而减少I/O以及网络传输的消耗；利用某省份手机上网数据与Hive进行了大数据对比实验，结果表明，本ETL工具的性能平均比Hive快10%～20%。

关键词: ETL, 优化规则, 改进的链式MapReduce

Abstract:

The related work in parallel ETL and common methods to deal with multiple MapReduce jobs were introduced. Then an improved chain-MapReduce framework was presented, based on this framework,a parallel ETL tool was designed. Several optimization rules on ETL which will make the ETL process generate less MapReduce jobs to avoid unnecessary I/O and network cost were presented. The ETL tool on real queries and real big datasets were evaluated. Compared with Hive, the tool reduces time on average by 10% to 20%.

Key words: improved chain-MapReduce, ETL, optimization rule

吴斌,刘心光. 一种基于改进的链式MapReduce的并行ETL应用[J]. 电信科学, 2013, 29(12): 1-8.

Bin Wu,Xinguang Liu. A Parallel ETL Tool Based on an Improved Chain-MapReduce Framework[J]. Telecommunications Science, 2013, 29(12): 1-8.

图/表 11

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

表1

参考文献 12

1	Dean J , Ghemawat S . MapReduce:simplified data processing on large clusters . Proceedings of the 6th Conference on Symposium on Opearting Systems Resign＆Implementation , San Francisco, USA, 2004
2	Apache. Hive. , 2013
3	Chen S T . Cheetah:a high performance, custom data warehouse on top of MapReduce. Proceedings of the VLDB Endowment, 2010,3（1）:1459～1468
4	Melnik S , Gubarev A , Long J J ， et al. Dremel: interactive analysis of web-scale datasets. Communications of the ACM, 2011,54（6）:114～123
5	Cloudera. Impala. ,
6	Olston C , Reed B , Srivastava U ， et al. Pig Latin: a not-so-foreign language for data processing. Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data， 2008: 1099～1110
7	IBM. JAQL. ,
8	Chambers C , Raniwala A , Perry F ， et al. FlumeJava: easy, efficient data-parallel pipelines. ACM SIGPLAN Notices, 2010,45（6）: 363～375
9	Cascading. , 2012
10	Horton Works. Tez. , 2013
11	Bu Y Y , Howe B , Balazinska M ， et al. HaLoop: efficient iterative data processing on large clusters. Proceedings of the VLDB Endowment, 2010,3（1/2）: 285～296
12	Bu Y Y , Ekanayake J , Li H , Zhang B J ， et al. Twister: a runtime for iterative MapReduce. Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, Chicago, Iuinois, 2010: 810～818

流程名	结果条数	Hive 0.11.0	新链式ETL	性能提升（减少时间百分比）
case 1	120 079	4 min 49 s	4 min 16 s	11%
case 2	47 533 740	6 min 39 s	6 min 6 s	8%
case 3	170 440 935	21 min 7 s（14 min 13 s+6 min 54 s）	14 min 52 s（11 min 25 s+3 min 27 s）	30%
case 4	886 981 116	23 min 50 s（15 min 11 s+8 min 39 s）	19 min 54 s（13 min 13 s+6 min 41 s）	17%

[1]	汪保友,钱晶,袁时金. 基于Hadoop的电信大数据采集方案研究与实现[J]. 电信科学, 2017, 33(1): 135-142.
[2]	谷红勋,杨珂. 基于大数据的移动用户行为分析系统与应用案例[J]. 电信科学, 2016, 32(3): 139-146.
[3]	雷蕾,李景文,宫大鹏,王睿,苏雷,陈宁江. 基于Hadoop的OSS域数据建模与采集方法研究[J]. 电信科学, 2015, 31(1): 2015021-.
[4]	雷蕾,李景文,宫大鹏,王睿,苏雷,陈宁江. 基于Hadoop的OSS域数据建模与采集方法研究[J]. 电信科学, 2015, 31(1): 128-138.

一种基于改进的链式MapReduce的并行ETL应用

A Parallel ETL Tool Based on an Improved Chain-MapReduce Framework

在线阅读

PDF下载

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 12

相关文章 4

Metrics

推荐阅读 0