基于RDMA和NVM的大数据系统一致性协议研究

doi:10.11959/j.issn.2096-0271.2019034

大数据 ›› 2019, Vol. 5 ›› Issue (4): 89-99.doi: 10.11959/j.issn.2096-0271.2019034

基于RDMA和NVM的大数据系统一致性协议研究

吴昊^1,²,陈康^1,²,武永卫^1,²,郑纬民^1,²

¹ 清华大学计算机科学与技术系，北京 100084
² 北京信息科学与技术国家研究中心，北京 100084

出版日期:2019-07-15 发布日期:2019-08-09
作者简介:吴昊（1992- ），男，清华大学计算机科学与技术系硕士生，主要研究方向为分布式系统。|陈康（1976- ），男，清华大学计算机科学与技术系副教授，中国计算机学会（CCF）会员，主要研究方向为分布式系统、存储系统等。|武永卫（1974- ），男，清华大学计算机科学与技术系教授，CCF高级会员，主要研究方向为并行和分布式处理、云计算和存储等。|郑纬民（1946- ），男，清华大学计算机科学与技术系教授、博士生导师，CCF会士，主要研究方向为计算机架构、操作系统、存储和分布式计算等。
基金资助:
国家重点研发计划基金资助项目(2016YFB1000504);国家自然科学基金资助项目(61433008);国家自然科学基金资助项目(61373145);国家自然科学基金资助项目(61572280);国家自然科学基金资助项目(1435216);中国博士后科学基金资助项目(2018M630162)

Research on the consensus of big data systems based on RDMA and NVM

Hao WU^1,²,Kang CHEN^1,²,Yongwei WU^1,²,Weimin ZHENG^1,²

¹ Department of Computer Science and Technology,Tsinghua University,Beijing 100084,China
² Beijing National Research Center for Information Science and Technology,Beijing 100084,China

Online:2019-07-15 Published:2019-08-09
Supported by:
National Key Research and Development Program of China(2016YFB1000504);The National Natural Science Foundation of China(61433008);The National Natural Science Foundation of China(61373145);The National Natural Science Foundation of China(61572280);The National Natural Science Foundation of China(1435216);China Postdoctoral Science Foundation(2018M630162)

摘要/Abstract

摘要：

分布式的存储系统以及计算系统是构造大数据处理系统的基础。系统的高可用性是任何一个分布式系统的基石，高可用技术一般依赖于一致性协议。讨论了经典的非拜占庭的分布式一致性协议以及新技术发展下的RDMA通信协议与NVM存储介质，通过RDMA和NVM的结合获得了更高性能的高可用系统。改进了一致性协议，使其能够更好地利用RDMA与NVM的特性。实现的系统在保证系统数据一致和可用的同时，有效地提高了协议实现的性能。实验表明，相比于现有的系统，实现的系统能够得到40%的性能提高。

关键词: 大数据, 分布式系统, 一致性协议, 远程内存直接访问, 非易失存储器

Abstract:

Distributed storage systems and computing systems are the foundation for constructing big data processing systems.High availability of the system is the cornerstone of any distributed system.High-availability technologies generally rely on consensus protocols.The classic non-Byzantine distributed consensus protocol was discussed,as well as the RDMA communication protocol and NVM storage media under the development of new technologies to achieve higher performance high availability systems by combining them.The consensus protocol to make the better use of the features of RDMA and NVM was modified.The implemented system effectively improves the performance of the protocol while ensuring the consistency and availability of the system data.Experiments show that the system implemented in this paper can achieve 40% performance improvement compared to existing systems.

Key words: big data, distributed system, consensus protocol, RDMA, NVM

中图分类号:

TP302.8

吴昊,陈康,武永卫,郑纬民. 基于RDMA和NVM的大数据系统一致性协议研究[J]. 大数据, 2019, 5(4): 89-99.

Hao WU,Kang CHEN,Yongwei WU,Weimin ZHENG. Research on the consensus of big data systems based on RDMA and NVM[J]. Big Data Research, 2019, 5(4): 89-99.

图/表 7

图1

表1

表2

图2

图3

表3

图4

参考文献 32

[1]	GRAY J , LAMPORT L . Consensus on transaction commit[J]. ACM Transactions on Database Systems (TODS), 2006,31(1): 133-160.
[2]	SKEEN D , . Nonblocking commit protocols[C]// The 1981 ACM SIGMOD International Conference on Management of Data,April 29-May 1,1981,Ann Arbor,USA. New York:ACM Press, 1981: 133-142.
[3]	GIFFORD D K , . Weighted voting for replicated data[C]// The 7th ACM Symposium on Operating Systems Principles,December 10-12,1979,Pacific Grove,USA. New York:ACM Press, 1979: 150-162.
[4]	LAMPORT L . Paxos made simple[J]. ACM Sigact News, 2001,32(4): 18-25.
[5]	CHANDRA T D , GRIESEMER R , REDSTONE J . Paxos made live:an engineering perspective[C]// The 26th Annual ACM Symposium on Principles of Distributed Computing,August 12-15,2007,Portland,USA. New York:ACM Press, 2007: 398-407.
[6]	HUNT P , KONAR M , JUNQUEIRA F P ,et al. ZooKeeper:wait-free coordination for Internet-scale systems[C]// USENIX Annual Technical Conference,June 23-25,2010,Boston,USA.[S.l.:s.n]. 2010.
[7]	ONGARO D , OUSTERHOUT J . In search of an understandable consensus algorithm[C]// 2014 USENIX Annual Technical Conference,Junuary 17-20,2014,Philadelphia,USA.[S.l.:s.n. ], 2014: 305-319.
[8]	LAMPORT L . Fast Paxos[J]. Distributed Computing, 2006,19(2): 79-103.
[9]	ZHAO W . Fast Paxos made easy:theory and implementation[J]. International Journal of Distributed Systems and Technologies (IJDST), 2015,6(1): 15-33.
[10]	LAMPORT L B . Generalized Paxos:U.S.patent 7698465[P].2010-04-13.
[11]	SUTRA P , SHAPIRO M . Fast genuine generalized consensus[C]// 2011 IEEE 30th International Symposium on Reliable Distributed Systems,October 4-7,2011,Madrid,Spain. Piscataway:IEEE Press, 2011: 255-264.
[12]	MORARU I , ANDERSEN D G , KAMINSKY M . There is more consensus in egalitarian parliaments[C]// The 24th ACM Symposium on Operating Systems Principles,November 3-6,2013,Farminton,USA. New York:ACM Press, 2013: 358-372.
[13]	BURROWS M , . The chubby lock service for loosely-coupled distributed systems[C]// The 7th Symposium on Operating Systems Design and Implementation,November 6-8,2006,Seattle,USA. Berkeley:USENIX Association, 2006: 335-350.
[14]	BAKER J , BOND C , CORBETT J C ,et al. Megastore:providing scalable,highly available storage forinter active services[C]// The 5th Biennial CIDR Conference,January 9-11,2011,Asilomar,USA.[S.l.:s.n]. 2011: 223-234.
[15]	HUNT P , KONAR M , JUNQUEIRA F P ,et al. ZooKeeper:wait-free coordination for internet-scalesystems[C]// USENIX Annual Technical Conference,June 23-25,2010,Boston,USA. Berkeley:USENIX Association, 2010.
[16]	GUO C , WU H , DENG Z ,et al. RDMA over commodity ethernet at scale[C]// The 2016 ACM SIGCOMM Conference,August 22-26,2016,Florianopolis,Brazil. New York:ACM Press, 2016: 202-215.
[17]	TSAI S Y , ZHANG Y . Lite kernel RDMA support for datacenter applications[C]// The 26th Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China. New York:ACM Press, 2017: 306-324.
[18]	MACARTHUR P , RUSSELL R D . A performance study to guide RDMA programming decisions[C]// 2012 IEEE 14th International Conference on High Performance Computing and Communication,June 25-27,2012,Liverpool,UK. Piscataway:IEEE Press, 2012: 778-785.
[19]	CHEN A . A review of emerging nonvolatile memory (NVM) technologies and applications[J]. Solid-State Electronics, 2016(125): 25-38.
[20]	MAOW , LIU JN , TONG W ,et al. A review of storage technology research based on phase change memory[J]. Chinese Journal of Computers, 2015,38(5):944.
[21]	WANG C , JIANG J , CHEN X ,et al. Apus:fast and scalable Paxos on RDMA[C]// The 2017 Symposiumon Cloud Computing,September 25-27,2017,Santa Clara,USA. New York:ACM Press, 2017: 94-107.
[22]	KIM D , MEMARIPOUR A , BADAM A ,et al. Hyperloop:group-based NIC-offloading to accelerate replicated transactions in multi-tenant storage systems[C]// The 2018 Conference of the ACM Special Interest Group on Data Communication,August 20-25,2018,Budapest,Hungary. New York:ACM Press, 2018: 297-312.
[23]	DANG H T , SCIASCIA D , CANINI M ,et al. Netpaxos:consensus at network speed[C]// The 1st ACM SIGCOMM Symposium on Software Defined Networking Research,June 17-18,2015,Santa Clara,USA. New York:ACM Press, 2015.
[24]	MCKEOWN N , ANDERSON T , BALAKRISHNAN H ,et al. OpenFlow:enabling innovation in campus networks[J]. ACM SIGCOMM Computer Communication Review, 2008,38(2): 69-74.
[25]	LI J , MICHAEL E , SHARMA N K ,et al. Just say noto Paxos overhead:replacing consensus with network ordering[C]// The 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI16),November 2-4,2016,Savannah,USA. Berkeley:USENIX Association, 2016: 467-483.
[26]	BOSSHART P , DALY D , GIBB G ,et al. P4:programming protocol-independent packet processors[J]. ACM SIGCOMM Computer Communication Review, 2014,44(3): 87-95.
[27]	DANG H T , CANINI M , PEDONE F ,et al. Paxos made switch-y[J]. ACM SIGCOMM Computer Communication Review, 2016,46(2): 18-24.
[28]	SULAIMAN N , OBAID Z A , MARHABAN M ,et al. Design and implementation of FPGAbased systems-areview[J]. Australian Journal of Basic and Applied Sciences, 2009,3(4): 3575-3596.
[29]	ISTVáN Z , SIDLER D , ALONSO G ,et al. Consensus in a box:inexpensive coordination in hardware[C]// The 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI16),March 16-18,2016,Santa Clara,USA. Berkeley:USENIX Association, 2016: 425-438.
[30]	POKE M , HOEFLER T . DARE:highperformance state machine replication on RDMA networks[C]// The 24th International Symposium on High-Performance Parallel and Distributed Computing,June 15-19,2015,Portland,USA. New York:ACM Press, 2015: 107-118.
[31]	TALEB Y , STUTSMAN R , ANTONIU G ,et al. Tailwind:fast and atomic RDMA-based replication[C]// The 2018 USENIX Annual Technical Conference,July 11-13,2018,Boston,USA.[S.l.:s.n. ], 2018: 851-863.
[32]	PANDA D K , KOOP M , BALAJI P . Tutorial:infiniband and 10-Gigabit Ethernet for dummies[C]// IEEE Symposium on High Performance Interconnects,August 25-27,2009,New York,USA. Piscataway:IEEE Press, 2009.

原语	UD	UC	RC
Send	√	√	√
Receive	√	√	√
RDMA Write		√	√
RDMA Read			√
Atomic			√

特性	磁盘	PCM	DRAM
读能耗/(J/GB)	65	1	0.8
写能耗/(J/GB)	65	6	1.2
读时延	~5 ms	~60 ns	20~50 ns
写时延	~5 ms	50~120 ns	20~50 ns
最大擦写次数/次	∞	10⁶~10⁸	∞

配置	具体信息
操作系统	CentOS 7.2
CPU	Intel Xeon CPU E5-2643 v4 12核，主频3.40 GHz
内存	128 GB
网卡	Mellnaox ConnectX-3 Infiniband网卡，56 Gbit/s

基于RDMA和NVM的大数据系统一致性协议研究

Research on the consensus of big data systems based on RDMA and NVM

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 32

相关文章 15

Metrics

推荐阅读 0

[1]	钱海红, 王茂异, 熊贇. 高等教育数字化转型的现状与发展研究[J]. 大数据, 2023, 9(3): 56-70.
[2]	梅宏, 杜小勇, 金海, 程学旗, 柴云鹏, 石宣化, 靳小龙, 王亚沙, 刘驰. 大数据技术前瞻[J]. 大数据, 2023, 9(1): 1-20.
[3]	沈阳, 余梦珑. 元宇宙与大数据：时空智能中的数据洞察与价值连接[J]. 大数据, 2023, 9(1): 103-110.
[4]	陈静. 人文大数据及其在数字人文领域中的应用[J]. 大数据, 2022, 8(6): 3-14.
[5]	罗煜楚, 吴昊, 郭宇涵, 谭绍聪, 刘灿, 蒋瑞珂, 袁晓如. 数字人文中的可视化[J]. 大数据, 2022, 8(6): 74-93.
[6]	郑童哲恒, 李斌, 冯敏萱, 常博林, 王东波. 历史典籍的结构化探索——《史记·列传》数字人文知识库的构建与可视化研究[J]. 大数据, 2022, 8(6): 40-55.
[7]	李汶龙, 袁媛, 安筱鹏. 刍议大数据治理的三大基础思维[J]. 大数据, 2022, 8(4): 34-45.
[8]	汤奇峰, 邵志清, 叶雅珍. 数据交易中的权利确认和授予体系[J]. 大数据, 2022, 8(3): 40-53.
[9]	王陈慧子, 蔡玮. 元宇宙数字经济：现状、特征与发展建议[J]. 大数据, 2022, 8(3): 140-150.
[10]	杨玫, 李玮, 乔思渊, 刘巍. 中国大数据产业产值测算方法研究[J]. 大数据, 2022, 8(3): 151-160.
[11]	李德仁, 张过, 蒋永华, 沈欣, 刘伟玲. 论大数据视角下的地球空间信息学的机遇与挑战[J]. 大数据, 2022, 8(2): 3-14.
[12]	仇晓兰, 胡玉新, 上官松涛, 付琨. 遥感卫星大数据高精度一体化处理技术[J]. 大数据, 2022, 8(2): 15-27.
[13]	刘伟权, 王程, 臧彧, 胡倩, 于尚书, 赖柏锜. 基于遥感大数据的信息提取技术综述[J]. 大数据, 2022, 8(2): 28-57.
[14]	刘建强, 叶小敏, 兰友国. 我国海洋卫星遥感大数据及其应用服务[J]. 大数据, 2022, 8(2): 75-88.
[15]	杨何群, 王晓峰, 高彦青, 陆一闻, 麻炳欣, 王昕瑶. 数值天气预报对卫星大数据的需求分析[J]. 大数据, 2022, 8(2): 89-102.