分布式数据流计算系统的数据缓存技术综述

doi:10.11959/j.issn.2096-0271.2020027

Abstract

Abstract:

Dataflow model is adopted by several dataflow systems for its advantages of high parallel computing,pipeline processing and functional programming.In distributed dataflow systems and heterogeneous dataflow systems,due to the speed mismatch between the data production of data source operators and the data consumption of data sink operators,data could be delayed and operators could be idle.In order to support an efficient dataflow system,a dataflow cache system was desired to ensure efficient caching and movement of dataflow.Several distributed dataflow systems and distributed message queuing systems were analyzed,and the support degree of current message queuing system to data flow caching system was summarized.Finally,the cache technique was introduced,and the demands and research directions of future dataflow caching systems were analyzed.

Key words: dataflow, cache, distributed system, message queue

CLC Number:

TP391

Xuchu YUAN, Guo FU, Jize BI, Yanfeng ZHANG, Tiezheng NIE, Yu GU, Yubin BAO, Ge YU. Survey on data caching technology of distributed dataflow system[J]. Big Data Research, 2020, 6(3): 101-116.

Figures/Tables 6

References 41

[1]	YAZDANPANAH F , ALVAREZ-MARTINEZ C , JIMENEZ-GONZALEZ D ,et al. Hybrid dataflow/von-neumann architectures[J]. IEEE Transactions on Parallel and Distributed Systems, 2014,25(6): 1489-1509.
[2]	NOWATZKI A , GANGADHAR V , SANKARALINGAM K . Computer with hybrid von-neumann/dataflow execution architecture:U.S.Patent 10,216,693[J].2019-2-26.
[3]	DENNIS J B , . First version of a data flow procedure language[C]// Programming Symposium. Heidelberg:Springer, 1974: 362-376.
[4]	DENNIS J B , FOSSEEN J B , LINDMAN J P . Data flow schemas[C]// International Sympoisum on Theoretical Programming. Heidelberg:Springer, 1972: 187-216.
[5]	DENNIS J B . Fresh breeze:a multiprocessor chip architecture guided by modular programming principles[J]. ACM SIGARCH Computer Architecture News, 2003,31(1): 7-15.
[6]	NAJJAR W A , LEE E A , GAO G R . Advances in the dataflow computational model[J]. Parallel Computing, 1999,25(13-14): 1907-1929.
[7]	ZAHARIA M , CHOWDHURY M , FRANKLIN M J ,et al. Spark:cluster computing with working sets[C]// The 2nd USENIX Conference on Hot Topics in Cloud Computing. Berkeley:USENIX Association, 2010:10.
[8]	MENG X , BRADLEY J , YAVUZ B ,et al. Mllib:machine learning in Apache Spark[J]. The Journal of Machine Learning Research, 2016,17(1): 1235-1241.
[9]	KARAU H , KONWINSKI A , WENDELL P ,et al. Learning Spark:lightning-fast big data analysis[M]. Sebastopol: O’Reilly Media,Inc.Press, 2015.
[10]	CARBONE P , KATSIFODIMOS A , EWEN S ,et al. Apache Flink:stream and batch processing in a single engine[J]. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2015,36(4): 28-38.
[11]	FRIEDMAN E , TZOUMAS K . Introduction to Apache Flink:stream processing for real time and beyond[M]. Sebastopol: O’Reilly Media,Inc.Press, 2016.
[12]	ABADI M , BARHAM P , CHEN J ,et al. TensorFlow:a system for large-scale machine learning[C]// The 12th USENIX Symposium on Operating Systems Design and Implementation. Berkeley:USENIX Association, 2016: 265-283.
[13]	ABADI M , AGARWAL A , BARHAM P ,et al. TensorFlow:large-scale machine learning on heterogeneous distributed systems[J]. Computer Science, 2016,arXiv:1603.04467.
[14]	WONGSUPHASAWAT K , SMILKOV D , WEXLEX J ,et al. Visualizing dataflow graphs of deep learning models in TensorFlow[J]. IEEE Transactions on Visualization and Computer Graphics, 2017,24(1): 1-12.
[15]	AKIDAU T , BRADSHAW R , CHAMBERS C ,et al. The dataflow model:a practical approach to balancing correctness,latency,and cost in massive-scale,unbounded,out-of-order data processing[J]. Proceedings of VLDB Endowment, 2015,8(12): 1792-1803.
[16]	倪炜 . 分布式消息中间件实践[M]. 北京: 电子工业出版社, 2018.
	NI W . Practice of distributed message middleware[M]. Beijing: Publishing House of Electronics IndustryPress, 2018.
[17]	NARKHEDE N , SHAPIRA G , PALINO T . Kafka:the definitive guide:real-time data and stream processing at scale[M]. Sebastopol: O’Reilly Media,Inc.Press, 2017.
[18]	郑奇煌 . Kafka技术内幕[M]. 北京: 人民邮电出版社, 2017.
	ZHENG Q H . Inside of Kafka technology[M]. Beijing: The People’s Posts and Telecommunications PressPress, 2017.
[19]	DOBBELAERE P , ESMAILI K S . Kafka versus RabbitMQ:a comparative study of two industry reference publish/subscribe implementations:industry paper[C]// The 11th ACM International Conference. New York:ACM Press, 2017: 227-238.
[20]	AYANOGLU E , AYTAS Y , NAHUM D . Mastering RabbitMQ[M]. Birmingham: Packt Publishing LtdPress, 2016.
[21]	ROSTANSKI M , GROCHLA K , SEMAN A . Evaluation of highly available and fault-tolerant middleware clustered architectures using RabbitMQ[C]// The 2014 Federated Conference on Computer Science and Information Systems. Piscataway:IEEE Press, 2014: 879-884.
[22]	IONESCU V M , . The analysis of the performance of RabbitMQ and ActiveMQ[C]// Roedunet International Conference-Networking in Education ＆Research. Piscataway:IEEE Press, 2015: 132-137.
[23]	HENJES R , SCHLOSSER D , MENTH M ,et al. Throughput performance of the ActiveMQ JMS server[M]. Heidelberg: SpringerPress, 2007.
[24]	丁威, 周继锋 . RocketMQ技术内幕[M]. 北京: 机械工业出版社, 2018.
	DING W , ZHOU J F . Inside of RocketMQ technology[M]. Beijing: China Machine PressPress, 2018.
[25]	郭嘉凯 . RocketMQ:从阿里巴巴走向世界[J]. 软件和集成电路, 2018(11):13.
	GUO J K . RocketMQ:from Alibaba to the world[J]. Software and Integrated Circuit, 2018(11):13.
[26]	李国杰 . 一种新的体系结构——数据流计算机[J]. 电子计算机动态, 1981(11): 3-10.
	LI G J . A new architecture:dataflow computer[J]. Computer Review, 1981(11): 3-10.
[27]	ZUCKERMAN S , SUETTLERLEIN J , KANUERHASE R ,et al. Using a Codelet program execution model for exascale machines:position paper[C]// The 1st International Workshop on Adaptive SelfTuning Computing Systems for the Exaflop Era. New York:ACM Press, 2011: 64-69.
[28]	高光荣 . 大数据的流动之美——数据流与大数据：挑战与机遇[J]. 中国计算机学会通讯, 2013,9(12): 16-19.
	GAO G R . The flowing beauty of big data,dataflow and big data:challenge and opportunity[J]. Communication of the CCF, 2013,9(12): 16-19.
[29]	NEUMEYER L , ROBBINS B , NAIR A ,et al. S4:distributed stream computing platform[C]// The 2010 IEEE International Conference on Data Mining Workshops. Piscataway:IEEE Press, 2010: 170-177.
[30]	SUETTLERLEIN J , ZUCKERMAN S , GAO G R . An implementation of the Codelet model[M]. Heidelberg: SpringerPress, 2013: 633-644.
[31]	SUETTERLEIN J . Darts:a runtime based on the Codelet execution model[D]. Newark:University of Delaware, 2014.
[32]	LAUDERDALE C , KHAN R . Towards a Codelet-based runtime for exascale computing:position paper[C]// The International Workshop on Adaptive Selftuning Computing Systems for the Exaflop Era. New York:ACM Press, 2012: 21-26.
[33]	DEAN J , CHEMAWAT S . MapReduce:simplified data processing on large clusters[J]. Communications of the ACM, 2008,51(1): 107-113.
[34]	TOSHNIWAL A , TANEJA S , SHUKLA A ,et al. Storm@twitter[C]// The 2014 ACM SIGMOD International Conference on Management of Data. New York:ACM Press, 2014: 147-156.
[35]	李明, 王晓鹏 . Storm源码分析[M]. 北京: 人民邮电出版社, 2014.
	LI M , WANG X P . Source code analysis of Storm[M]. Beijing: The People’s Posts and Telecommunications PressPress, 2014.
[36]	WU Y , ZHENG L , HEILIG B ,et al. HAMR:a dataflow-based real-time in-memory cluster computing engine[J]. The International Journal of High Performance Computing Applications, 2017,31(5): 361-374.
[37]	MURRAY D G , MCSHERRY F , ISAACS R ,et al. Naiad:a timely dataflow system[C]// The 24th ACM Symposium on Operating Systems Principles. New York:ACM Press, 2013: 439-455.
[38]	SHVACHKO K , KUANG H , RADIA S ,et al. The Hadoop distributed file system[C]// The 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies. Piscataway:IEEE Press, 2010: 1-10.
[39]	HUNT P , KONAR M , JUNQUEIRA F P ,et al. ZooKeeper:wait-free coordination for internetscale systems[C]// The 2010 USENIX Annual Technical Conference. Berkeley:USENIX Association, 2010:11.
[40]	JUNQUEIRA F , REED B . Zookeeper:distributed process coordination[M]. Sebastopol: O’Reilly Media,Inc.Press, 2013.
[41]	吴璨, 王小宁, 肖海力 ,等. 分布式消息系统研究综述[J]. 计算机科学, 2019(B06): 1-5.
	WU C , WANG X N , XIAO H L ,et al. Survey on distributed message system[J]. Computer Science, 2019(B06): 1-5.

Metrics

Recommended 0

No Suggested Reading articles found!

对比项	Kafka	RabbitMQ	ActiveMQ	RocketMQ	Pulsar
实现语言	Scala/Java	Erlang	Java	Java	Java
客户端软件开发工具包	Java、.NET、C++等	Java、Scala等	Java、C、C++等	Java、C++、Go等	Java、C++、Go等
通信协议	TCP	AMQP	OpenWire、STOMP、REST、XMPP、AMQP	自定义	TCP
消息顺序性保证	顺序（一个分区内）	顺序（一个用户）	顺序（独有消费者和排他性队列模式）	全局顺序	全局顺序
消息延时与定时投递	不支持	支持	支持	支持	不支持
批量传输	支持	不支持	支持	支持	支持
广播	不支持	支持	支持	支持	支持
持久化	磁盘文件	内存、文件	内存、文件、数据库	磁盘文件	磁盘文件
消息优先级	不支持	支持	支持	不支持	支持
事务消息	支持	支持	支持	支持	不支持
集群管理服务	ZooKeeper		ZooKeeper	nameserver（自带）	BookKeeper、ZooKeeper

Survey on data caching technology of distributed dataflow system

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 41

Related Articles 9

Metrics

Recommended 0

[1]	Jiantong HUO, Limin XIAO, Zhisheng HUO, Yaowen XU. Research and implementation of edge cache system in global virtual data space across WAN [J]. Big Data Research, 2021, 7(5): 57-81.
[2]	Kai ZHANG, Yang CHE. Method of accelerating deep learning with optimized distributed cache in containers [J]. Big Data Research, 2021, 7(5): 150-163.
[3]	Zhenlong DENG, Zhiguang CHEN. An optimization of MPI-IO interface for non-volatile memory [J]. Big Data Research, 2021, 7(2): 172-181.
[4]	Nifei BI, Guangyao DING, Qihang CHEN, Chen XU, Aoying ZHOU. Dataflow model and its applications in big data processing [J]. Big Data Research, 2020, 6(3): 73-86.
[5]	Xiaochun TANG, Ying FU, Zhao DING, Anqi MAO, Zhanhuai LI. State-of-art research of cluster resource management in dataflow computing model [J]. Big Data Research, 2020, 6(3): 87-100.
[6]	Huayou SU, Songzhu MEI, Rongchun LI, Yong DOU. The usage of dataflow model in GPU and big data processing [J]. Big Data Research, 2020, 6(3): 117-128.
[7]	Xuehan KE,Rong CHEN. Graph processing engine based on graph query system [J]. Big Data Research, 2019, 5(4): 16-26.
[8]	Hao WU,Kang CHEN,Yongwei WU,Weimin ZHENG. Research on the consensus of big data systems based on RDMA and NVM [J]. Big Data Research, 2019, 5(4): 89-99.
[9]	Xingda WEI, Rong CHEN, Haibo CHEN. Optimizing distributed systems with remote direct memory access [J]. Big Data Research, 2018, 4(4): 3-14.