大数据 ›› 2020, Vol. 6 ›› Issue (3): 59-72.doi: 10.11959/j.issn.2096-0271.2020024

• 专题:面向大数据处理的数据流计算技术 • 上一篇    下一篇

面向大数据处理的数据流编程模型和工具综述

邹骁锋,阳王东(),容学成,李肯立,李克勤   

  1. 湖南大学信息科学与工程学院,湖南 长沙 410008
  • 出版日期:2020-05-15 发布日期:2020-05-15
  • 作者简介:邹骁锋(1996- ),男,湖南大学信息科学与工程学院博士生,主要研究方向为并行计算、数据挖掘和机器学习|阳王东(1974- ),男,湖南大学信息科学与工程学院教授,主要研究方向为分布式并行计算、机器学习|容学成(1996- ),男,湖南大学信息科学与工程学院硕士生,主要研究方向为大数据和机器学习|李肯立(1970- ),男,博士,湖南大学信息科学与工程学院教授,主要研究方向为高性能计算、人工智能和大数据|李克勤(1963- ),男,博士,湖南大学信息科学与工程学院教授,主要研究方向为并行计算、边缘计算和大数据
  • 基金资助:
    国家重点研发计划基金资助项目(2018YFB1003401)

A survey of dataflow programming models and tools for big data processing

Xiaofeng ZOU,Wangdong YANG(),Xuecheng RONG,Kenli LI,Keqin LI   

  1. College of Computer Science and Electronic Engineering,Hunan University,Changsha 410008,China
  • Online:2020-05-15 Published:2020-05-15
  • Supported by:
    The National Key Research and Development Program of China(2018YFB1003401)

摘要:

利用大数据计算平台对大量的静态数据进行数据挖掘和智能分析助推了大数据和人工智能应用的落地。在面临互联网、物联网产生的日益庞大的实时动态数据的处理需求时,数据流计算被逐步引入目前的一些大数据处理平台中。针对数据流的编程模型,比较了传统软件工程的面向数据流的分析和设计方法与目前针对大数据处理平台的数据流编程模型提供的结构定义和模型参考,分析了两者的差异和不足,总结了数据流编程模型的主要特征和关键要素。分析了目前数据流编程的主要方式以及与主流编程工具的结合,针对大数据处理的数据流计算业务需求,给出了可视化数据流编程工具的基本框架和编程模式。

关键词: 数据流, 编程模型, 大数据处理, 编程工具

Abstract:

The application of big data and artificial intelligence is promoted by data mining and intelligent analysis of a large number of static data using big data computing platform.In the face of the growing demand for real-time dynamic data processing generated by the Internet of things,dataflow computing has been gradually introduced into some big data processing platforms.Aiming at the programming model of data flow,the traditional software engineering design method for dataflow analysis and the structure definition and model reference provided by the current dataflow programming model for big data processing platform was compared,the differences and shortcomings were analyzed,and the main features and key elements of the dataflow programming model were summarized.The main methods of dataflow programming and the combination with the mainstream programming tools were analyzed,and the basic framework and programming mode of visual dataflow programming tools were presented according to the dataflow computing business requirements of big data processing.

Key words: data flow, programming model, big data processing, programming tool

中图分类号: 

No Suggested Reading articles found!