面向大数据处理的数据流编程模型和工具综述

doi:10.11959/j.issn.2096-0271.2020024

Abstract

Abstract:

The application of big data and artificial intelligence is promoted by data mining and intelligent analysis of a large number of static data using big data computing platform.In the face of the growing demand for real-time dynamic data processing generated by the Internet of things,dataflow computing has been gradually introduced into some big data processing platforms.Aiming at the programming model of data flow,the traditional software engineering design method for dataflow analysis and the structure definition and model reference provided by the current dataflow programming model for big data processing platform was compared,the differences and shortcomings were analyzed,and the main features and key elements of the dataflow programming model were summarized.The main methods of dataflow programming and the combination with the mainstream programming tools were analyzed,and the basic framework and programming mode of visual dataflow programming tools were presented according to the dataflow computing business requirements of big data processing.

Key words: data flow, programming model, big data processing, programming tool

CLC Number:

TP391

Xiaofeng ZOU, Wangdong YANG, Xuecheng RONG, Kenli LI, Keqin LI. A survey of dataflow programming models and tools for big data processing[J]. Big Data Research, 2020, 6(3): 59-72.

Figures/Tables 11

References 24

[24]	YANG Y . The design and implementation of COStream graph editor[D]. Wuhan:Huazhong University of Science and Technology, 2016.
[25]	ALVES T A O , GOLDSTEIN B F , FRANCA F M G ,et al. A minimalistic dataflow programming library for Python[J]. Operations Research Letters, 2014,5(1): 51-54.
[26]	LIN Y A , . Map-reduce for machine learning on multicore[C]// The 20th Annual Conference on Neural Information Processing Systems. Massachusetts:MIT Press, 2006: 281-288.
[27]	TEAM A H B . Apache HBase reference guide[M]. California:[s.n.]. 2016.
[28]	HUAI Y , CHAUHAN A , GATES A ,et al. Major technical advancements in Apache Hive[C]// The 2014 ACM SIGMOD International Conference on Management of Data. New York:ACM Press, 2014: 1235-1246.
[29]	GARG N . Apache Kafka[M]. England: Packt Publishing LtdPress, 2013.
[1]	VAVILAPALLI V K , MURTHY A C , DOUGLAS C ,et al. Apache Hadoop YARN:yet another resource negotiator[C]// The 4th Annual Symposium on Cloud Computing. New York:ACM Press, 2013: 1-16.
[2]	IQBAL M H , SOOMRO T R . Big data analysis:Apache Storm perspective[J]. International Journal of Computer Trends and Technology, 2015,19(1): 9-14.
[3]	CARBONE P , KATSIFODIMOS A , EWEN S ,et al. Apache Flink:stream and batch processing in a single engine[J]. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2015,36(4): 28-38.
[4]	ZAHARIA M , XIN R S , WENDELL P ,et al. Apache Spark:a unified engine for big data processing[J]. Communications of the ACM, 2016,59(11): 56-65.
[5]	MCDERMID J . Software engineering:apractitioner’s approach[J]. Software Engineering Journal, 1995,10(6):266.
[6]	JILANIC A A A , NADEEM A , KIM T H ,et al. Formal representations of the data flow diagram:a survey[C]// The 2008 Advanced Software Engineering and Its Applications. Piscataway:IEEE Press, 2008: 153-158.
[7]	REPA V . Object-oriented analysis with data flow diagram[M]. Heidelbery: SpringerPress, 2013.
[8]	DENNIS J B , FOSSEEN J B , LINDERMAN J P . Data flow schemas[C]// International Symposium on Theoretical Programming. Heidelbery:Springer, 1972: 187-216.
[9]	KRAMER R , GUPTA R , SOFFA M L . The combining DAG:a technique for parallel data flow analysis[J]. IEEE Transactions on Parallel and Distributed Systems, 1994,5(8): 805-813.
[10]	DENNIS J B . Data flow supercomputers[J]. Computer, 1980(11): 48-56.
[11]	MILUTINOVIC V , SALOM J , TRIFUNOVIC N ,et al. Guide to dataflow supercomputing[M]. Heidelbery: SpringerPress, 2015.
[12]	LI A , BRAAK G J , CORPORAAL H ,et al. Fine-grained synchronizations and dataflow programming on GPUs[C]// The 29th ACM on International Conference on Supercomputing. New York:ACM Press, 2015: 109-118.
[13]	HALBWACHS N , CASPI P , RAYMOND P ,et al. The synchronous data flow programming language LUSTRE[J]. Proceedings of the IEEE, 1991,79(9): 1305-1320.
[14]	苏志超 . 神威?太湖之光上数据流编程模型的设计与实现[D]. 合肥:中国科学技术大学, 2018.
	SU Z C . Design and implement of a dataflow programming model on sunway taihulight[D]. Hefei:University of Science and Technology of China, 2018.
[15]	杨瑞瑞 . 面向多核 CPU/众核 GPU 异构集群的数据流编程模型研究[D]. 武汉:华中科技大学, 2017.
	YANG R R . A research of dataflow programming model oriented multi-core CPU/many-core GPU heterogeneous cluster[D]. Wuhan:Huazhong University of Science and Technology, 2017.
[16]	张维维, 魏海涛, 于俊清 ,等. COStream:一种面向数据流的编程语言和编译器实现[J]. 计算机学报, 2013,36(10): 1993-2006.
	ZHANG W W , WEI H T , YU J Q ,et al. COStream:a language for dataflow application and compiler[J]. Chinese Journal of Computers, 2013,36(10): 1993-2006.
[17]	AKIDAU T , SCHMIDT E , WHITTLE S ,et al. The dataflow model:a practical approach to balancing correctness,latency,and cost in massive-scale,unbounded,out-of-order data processing[J]. Proceedings of the VLDB Endowment, 2015,8(12): 1792-1803.
[18]	LAUDERDALE C , KHAN R . Towards a codelet-based runtime for exascale computing:position paper[C]// The 2nd International Workshop on Adaptive Self-Tuning Computing Systems for the Exaflop Era. New York:ACM Press, 2012: 21-26.
[19]	THIES W , KARCZMAREK M , AMARASINGHE S . StreamIt:a language for streaming applications[C]// The 11th International Conference on Compiler Construction. Heidelberg:Springer, 2002: 179-196.
[20]	ABADI M , BARHAM P , CHEN J ,et al. TensorFlow:a system for large-scale machine learning[C]// The 12th USENIX Symposium on Operating Systems Design and Implementation. Berkeley:USENIX Association, 2016: 265-283.
[21]	SUETTERLEIN J , ZUCKERMAN , STEPHANE ,et al. An implementation of the Codelet model[M]. Heidelberg: SpringerPress, 2013.
[22]	杨秋吉, 于俊清, 莫斌生 ,等. 面向 Storm 的数据流编程模型与编译优化方法研究[J]. 计算机工程与科学, 2016,38(12): 2409-2418.
	YANG Q J , YU J Q , MO B S ,et al. A data flow programming model and compiler optimization for Storm[J]. Computer Engineering and Science, 2016,38(12): 2409-2418.
[23]	BLUME P A . The LabVIEW style book[M]. Upper Saddle River: Prentice HallPress, 2007.
[24]	杨燕 . COStream 数据流程序图形编辑器的设计与实现[D]. 武汉:华中科技大学, 2016.

Metrics

Recommended 0

No Suggested Reading articles found!

数据流编程模型	平台适应性	扩展性	适应场景
Apache Beam	能够适应不同的分布式计算平台	具有较好的可扩展性	能够适应复杂的数据流的处理
SWARM	自身提供计算引擎	具有硬件可扩展性	提供较底层的实现，目前提供的数据流处理功能较弱
StreamIt	适应特定的处理器平台	可扩展性较差	复杂的并行场景适应性较差
COStream	不同平台需要重新编译	可扩展性一般	高层的业务场景描述能力不够
TensorFlow	目前支持的平台越来越多	具有较好的可扩展性	主要针对深度学习

数据流编程工具	适应领域	可扩展性	编程模式	易用性	描述语言
LabVIEW	虚拟仪器仿真专用领域	较弱	图形化设计数据流图	较好	专用的G语言
COStream	通用的数据流计算编程	一般	提供编程库和图形化编程	一般	C语言
Oceanus-ML	大数据分析	一般	提供编程库和图形化编程	较好	扩展SQL
TensorFlow	深度学习	较好	提供编程函数库，利用现有的Python编程工具	较差	Python
Sucuri	并行数据流计算编程	较弱	提供编程库和图形化编程	较差	Python

A survey of dataflow programming models and tools for big data processing

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 24

Related Articles 10

Metrics

Recommended 0

[1]	Hong MEI, Xiaoyong DU, Hai JIN, Xueqi CHENG, Yunpeng CHAI, Xuanhua SHI, Xiaolong JIN, Yasha WANG, Chi LIU. Big data technologies forward-looking [J]. Big Data Research, 2023, 9(1): 1-20.
[2]	Ming GONG, Xiangyu JIANG, Ying CHEN, Zhaofeng LIU. Software infrastructures for Chinese supercomputers from the perspective of lattice QCD applications [J]. Big Data Research, 2021, 7(5): 31-39.
[3]	Chenhao ZHANG, Limin XIAO, Guangjun QIN, Yao SONG, Shixuan JIANG, Jiye WANG. A wide-area collaborative scheduling system oriented to big data processing applications [J]. Big Data Research, 2021, 7(5): 82-97.
[4]	Yangyong ZHU, Yun XIONG. Primary exploration of transborder data folw supervision [J]. Big Data Research, 2021, 7(1): 135-144.
[5]	Nifei BI, Guangyao DING, Qihang CHEN, Chen XU, Aoying ZHOU. Dataflow model and its applications in big data processing [J]. Big Data Research, 2020, 6(3): 73-86.
[6]	Huayou SU, Songzhu MEI, Rongchun LI, Yong DOU. The usage of dataflow model in GPU and big data processing [J]. Big Data Research, 2020, 6(3): 117-128.
[7]	Weigang WU, Liang CHANG, Jiangtao REN, Tianlong GU. High performance big data computing systems for government governance [J]. Big Data Research, 2020, 6(2): 41-56.
[8]	Bo HE. Legislation and enforcement of cross-border data flows rules in Russia [J]. Big Data Research, 2016, 2(6): 129-134.
[9]	Shaomin MU, Fujiang WEN, Changqing SONG. Training mode of graduate students majored in agricultural big data [J]. Big Data Research, 2016, 2(1): 53-58.
[10]	Yihua Huang. Research Progress on Big Data Machine Learning System [J]. Big Data Research, 2015, 1(1): 35-54.