面向数据流处理的元组跟踪方法

doi:10.3969/j.issn.1000-0801.2013.10.010

电信科学 ›› 2013, Vol. 29 ›› Issue (10): 49-57.doi: 10.3969/j.issn.1000-0801.2013.10.010

面向数据流处理的元组跟踪方法

杜华明^1,³,张鹏^2,³,徐克付^2,³,谭建龙^2,³,李焱⁴

¹ 中国科学技术大学软件学院合肥 230051
² 中国科学院信息工程研究所北京 100093
³ 信息内容安全技术国家工程实验室北京 100093
⁴ 国家计算机网络应急技术处理协调中心北京 100029

出版日期:2013-10-15 发布日期:2017-06-19
基金资助:
国家高技术研究发展计划（“863”计划）基金资助项目;国家自然科学基金资助项目;中国科学院战略性先导专项基金资助项目

TTDSP:A Cost-Effective Approach to Tracking Tuple in Data Stream Processing

Huaming Du^1,³,Peng Zhang^2,³,Kefu Xu^2,³,Jianlong Tan^2,³,Yan Li⁴

¹ School of Software Engineering,University of Science and Technology of China,Hefei 230051,China
² Institute of Information Engineering,Chinese Academy of Sciences,Beijing 100093,China
³ National Engineering Laboratory for Information Security Technologies,Beijing 100093,China
⁴ National Computer Network Emergency Response and Coordination Center,Beijing 100029,China

Online:2013-10-15 Published:2017-06-19

摘要/Abstract

摘要：

为了保证数据流中的每个元组得到可靠处理，传统的方法需要在内存中保存每个元组，直到它们被数据流处理系统正常处理，因此会带来很大的内存开销。为此提出了一种既能够保证元组得到可靠处理，又能够节省内存开销的元组跟踪方法。该方法包括内存分配策略、元组跟踪单元选择策略和校验值更新策略，这3个策略使得元组跟踪单元只保留元组标识符的异或校验值而不是元组减少内存开销，同时通过改进一致性散列变换实现元组跟踪单元的负载均衡。内存开销和负载均衡的相关实验表明，该方法能够有效实现对元组的跟踪和可靠处理。

关键词: 数据流处理, 可靠性, 数据流, 负载均衡

Abstract:

The traditional data stream processing systems will keep all the tuples in the memory until they have been processed in order to provide reliable tuple processing.Unfortunately,the strategy will take up much memory.To address this issue,a cost-effective approach to tracking tuples-TTDSP was proposed.The approach includes three strategies,namely memory allocation strategy,tuple acker selection strategy and checksum updating strategy,which make tuple acker to keep only the XOR checksum not the tuple in memory.Moreover,the tuple acker are load balancing through the improved consistent Hash.The experiments on memory overhead and load balancing show that this approach is able to track and process tuples effectively and reliably.

Key words: data stream processing, reliability,data stream, load balancing

杜华明,张鹏,徐克付,谭建龙,李焱. 面向数据流处理的元组跟踪方法[J]. 电信科学, 2013, 29(10): 49-57.

Huaming Du,Peng Zhang,Kefu Xu,Jianlong Tan,Yan Li. TTDSP:A Cost-Effective Approach to Tracking Tuple in Data Stream Processing[J]. Telecommunications Science, 2013, 29(10): 49-57.

图/表 4

参考文献 12

1	孟小峰，慈祥 . 大数据管理：概念，技术与挑战. 计算机研究与发展， 2010,50（1）: 146～169
2	Kumar R . Two computational paradigm for big data. , 2012
3	Hwang J H , Balazinska M , Rasin A ,et al. High-availability algorithms for distributed stream processing. Proceedings of 21st International Conference on Data Engineering （ICDE 2005）, Tokyo,Japan, 2005
4	Two computational paradigm for big dataS4 distributed stream computing platformarticle-title>. , 2013
5	Guaranteeing message processing. , 2013
6	Gu X H , Papadimitriou S , Yu P S ,et al. Toward predictive failure management for distributed stream processing systems. Proceedings of 2008 the 28th International Conference on Distributed Computing Systems（ICDCS'08）, Washington,DC,USA, 2008
7	Brito A , Fetzer C , Felber P . Minimizing latency in fault-tolerant distributed stream processing systems. Proceedings of 2009 the 29th IEEE International Conference on Distributed Computing Systems（ICDCS'09）, Washington,DC,USA, 2009
8	Brito A , Fetzer C , Felber P . Multithreading-enabled active replication for event stream processing operators. Proceedings of the 28th IEEE International Symposium on Reliable Distributed Systems（SRDS'09）, Niagara Falls,New York,USA, 2009
9	Brito A , Fetzer C , Felber P . Minimizing latency in fault-tolerant distributed stream processing systems. Proceedings of 2009 the 29th IEEE International Conference on Distributed Computing Systems（ICDCS'09）, Washington,DC,USA, 2009
10	Sebepou Z , Magoutis K . CEC:continuous eventual check pointing for data stream processing operators. Proceedings of 2011 IEEE/IFIP 41st International Conference on Dependable Systems Networks（DSN）, Hong Kong,China, June2011
11	Sebepou Z , Magoutis K . Scalable storage support for data stream processing. Proceedings of the 26th Symposium on Mass Storage Systems and Technologies（MSST）, Incline Village,Nevada, May2010
12	Gulisano V , Jimenez-Peris R , Patino-Martnez M ,et al. StreamCloud:an elastic and scalable data streaming system. IEEE Transactions on Parallel and Distributed Systems, 2010,23（12）: 2351～2365

[1]	丁雨, 李晨凯, 韩会梅, 卢为党, 任元红, 高原, 曹江. 基于5G无人机通信的多智能体异构网络选择方法[J]. 电信科学, 2022, 38(8): 28-36.
[2]	王兆辉, 沈剑良, 张霞, 陈艇. FALB：一种FC协议链路聚合算法[J]. 电信科学, 2021, 37(9): 105-111.
[3]	粟栗, 陆黎, 张星, 刘畅. 基于图数据库的5G数据流转安全防护机制[J]. 电信科学, 2021, 37(4): 28-36.
[4]	许方敏,伍丽娇,杨帆,赵成林. 时间敏感网络（TSN）及无线TSN技术[J]. 电信科学, 2020, 36(8): 81-91.
[5]	罗斯宁,王化龙,李弘宇,彭蔚. 基于改进蚁群算法的云计算用户任务调度算法[J]. 电信科学, 2020, 36(2): 95-100.
[6]	雷宇骁,李更丰,黄玉雄,别朝红. 基于支持向量机的发/输电系统快速蒙特卡洛可靠性评估方法[J]. 电信科学, 2019, 35(6): 15-24.
[7]	邵帅,尚志军,王忠锋,辛国良,李悦悦,孔祥余,彭绪静. 基于QoS的电力光纤到户网络规划方法[J]. 电信科学, 2019, 35(6): 78-88.
[8]	乐光学,戴亚盛,杨晓慧,朱友康,游真旭,刘建生. 边缘计算多约束可信协同任务迁移策略[J]. 电信科学, 2019, 35(11): 36-50.
[9]	高会生,沈青. 通道不对称时延对继电保护可靠性影响的量化分析[J]. 电信科学, 2019, 35(1): 90-96.
[10]	李阳春,樊勇兵,陈楠,陈天. 面向分层解耦的虚拟节点可靠性设计[J]. 电信科学, 2018, 34(9): 160-168.
[11]	王文博,叶庆卫,周宇,陆志华. 基于排队论综合指标评估的动态负载均衡算法[J]. 电信科学, 2018, 34(7): 86-91.
[12]	刘海鹏. 基于负载均衡的MEH静态切换机制[J]. 电信科学, 2018, 34(7): 118-127.
[13]	吴宇,王小燕,杨昀,王华,窦燕,方晶晶. 多网协同的集团客户语音专线可靠性优化新方案[J]. 电信科学, 2018, 34(12): 155-162.
[14]	王兴鹏,许晓荣,胡安迪,冯维. Nakagami-m衰落信道下存在窃听攻击的认知中继网络安全性与可靠性折中方案[J]. 电信科学, 2018, 34(12): 24-32.
[15]	刘川,黄在朝,陶静,贾惠彬. 考虑排队时延的系统保护通信网络路由选择算法[J]. 电信科学, 2018, 34(10): 47-52.

面向数据流处理的元组跟踪方法

TTDSP:A Cost-Effective Approach to Tracking Tuple in Data Stream Processing

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 4

参考文献 12

相关文章 15

Metrics

推荐阅读 0