基于RDMA高速网络的高性能分布式系统

doi:10.11959/j.issn.2096-0271.2018036

大数据 ›› 2018, Vol. 4 ›› Issue (4): 3-14.doi: 10.11959/j.issn.2096-0271.2018036

• 专题：大数据驱动的智能计算体系架构 • 下一篇

基于RDMA高速网络的高性能分布式系统

魏星达,陈榕,陈海波

上海交通大学并行与分布式系统研究所，上海 200240

出版日期:2018-07-15 发布日期:2018-08-08
作者简介:魏星达（1992-），男，上海交通大学并行与分布式系统研究所硕士生，主要研究方向为分布式系统、数据库事务处理、利用新型硬件的分布式系统。|陈榕（1981-），男，博士，上海交通大学并行与分布式系统研究所副教授，ACM/IEEE/CCF会员，主要研究方向为系统软件、并行与分布式系统。在SOSP、OSDI、EuroSys、USENIX ATC等国际著名会议发表多篇学术论文，并获得EuroSys 2015、ACM APSys 2017与IEEE ICPP 2017的最佳论文奖。|陈海波（1982-），男，上海交通大学并行与分布式系统研究所教授、博士生导师，主要研究方向为操作系统与并行分布式系统。在操作系统、基于新型硬件的事务处理系统与大数据查询系统等领域做出了引领性工作。入选2014年国家“万人计划”青年拔尖人才计划，获得2011年全国优秀博士学位论文奖、2015年CCF青年科学家奖、2017年CCF NASAC-东软青年软件创新奖与2018年CCF青竹奖。目前担任ACM SIGOPS ChinaSys副主席、ACM APSys指导委员会主席、《ACM Transactions on Storage》编委，ACM CCS 2018系统安全领域主席。在国际著名学术会议与学术期刊上共发表学术论文100余篇，获得ACM EuroSys 2015、ACM APSys 2013/2017与IEEE ICPP 2007的最佳论文奖与IEEE HPCA 2014的最佳论文提名奖。研究工作也获得Google Faculty Research Award、IBM X10 Innovation Award、NetApp Faculty Fellowship与华为创新价值成果奖等企业奖励。
基金资助:
国家自然科学基金资助项目(61772335)

Optimizing distributed systems with remote direct memory access

Xingda WEI,Rong CHEN,Haibo CHEN

Institute of Parallel and Distributed Systems,Shanghai Jiao Tong University,Shanghai 200240,China

Online:2018-07-15 Published:2018-08-08
Supported by:
The National Natural Science Foundation of China(61772335)

摘要/Abstract

摘要：

高速的RDMA网络设备已经被广泛部署在现代数据中心。RDMA可以从两方面加速分布式系统：首先可以提供一种快速的消息处理机制，其次RDMA提供了新的硬件原语。这极大地提升了处理器的利用率以及对RDMA的使用率，但是需要重新设计系统。介绍了RDMA的研究进展，概述了近年来利用RDMA加速分布式系统的工作，包括基于RDMA重新设计的系统以及如何更好地利用RDMA的设计，并给出了未来的研究方向。

关键词: 分布式系统, 键值存储系统, 图处理系统, 联机事务处理系统, 远程过程调用

Abstract:

Fast network devices with RDMA support have been price-compatible with traditional network primitives such as Ethernet,and it’s now widely deployed in modern data centers.RDMA can be used in two ways.Firstly,it can optimize the messaging primitive in distributed applications.The second way is to redesign the applications with RDMA’s onesided features.One-sided features provide high CPU utilizations and high network performance,but the system should be redesigned.The research progress of RDMA was introduced.An overview on the research efforts on using RDMA for distributed systems was presented.The works on how to use RDMA to redesign systems and the works on how to better leverage RDMA were included.The future research directions were also put forward.

Key words: distributed system, key-value stores system, graph processing system, OLTP system, remote procedure call

中图分类号:

TP316

魏星达, 陈榕, 陈海波. 基于RDMA高速网络的高性能分布式系统[J]. 大数据, 2018, 4(4): 3-14.

Xingda WEI, Rong CHEN, Haibo CHEN. Optimizing distributed systems with remote direct memory access[J]. Big Data Research, 2018, 4(4): 3-14.

图/表 3

参考文献 33

[1]	FAN B , ANDERSEN D G , KAMINSKY M . MemC3:compact and concurrent MemCache with dumber caching and smarter hashing[C]// The 10th USENIX Conference on Networked Systems Design and Implementation,April2-5,2013,San Francisco,USA. Berkeley:USENIX Association, 2013: 371-384.
[2]	GUO C , WU H , DENG Z ,et al. RDMA over commodity ethernet at scale[C]// The 2016 ACM SIG-COMM Conference,August 22-26,2016,Florianopolis,Brazil. New York:ACM Press, 2016: 202-215.
[3]	TSAI S Y , ZHANG Y . Lite kernel rdma support for datacenter applications[C]// The 26th ACM Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China.[S.l.:s.n]. 2017: 306-324.
[4]	NARAYANAN D , HODSON O , CASTRO M ,et al. FaRM:fast remote memory[C]// The 11th USENIX Conference on Networked Systems Design and Implementation,April 2-4,2014,Seattle,USA. Berkeley:USENIX Association, 2014: 401-414.
[5]	DRAGOJEVI? A , NARAYANAN D , NIGHTINGALE E B ,et al. No compromises:distributed transactions with consistency,availability,and performance[C]// The 25th Symposium on Operating Systems Principles,October 4-7,2015,Monterey,USA. New York:ACM Press, 2015: 54-70.
[6]	NEWMARCH J . Remote procedure call[D]. Pittsburgh:Carnegie Mellon University, 1981.
[7]	KALIA A , KAMINSKY M , ANDERSEN D G . Design guidelines for high performance RDMA systems[C]// 2016 USENIX Annual Technical Conference,June 22-24,Denver,USA. Berkeley:USENIX Association, 2016: 437-450.
[8]	KALIA A , KAMINSKY M , ANDERSEN D G . FaSST:fast,scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs[C]// The 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16),November 2-4,2016,Savannah,USA. Berkeley:USENIX Association, 2016: 185-201.
[9]	LU Y , SHU J , CHEN Y ,et al. Octopus:an RDMA-enabled distributed persistent memory file system[C]// 2017 USENIX Annual Technical Conference (USENIX ATC 17),July 12-14,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017: 773-785.
[10]	SU M , ZHANG M , CHEN K ,et al. RFP:when RPC is faster than server-bypass with RDMA[C]// The 12th European Conference on Computer Systems,April 23-26,2017,Belgrade,Serbia. New York:ACM Press, 2017: 1-15.
[11]	KALIA A , KAMINSKY M , ANDERSEN D G . Using RDMA efficiently for key-value services[C]// The 2014 ACM Conference on SIGCOMM,August 17-22,2014,Chicago,USA. New York:ACM Press, 2014,44(4): 295-306.
[12]	TSAI S Y , ZHANG Y . Lite kernel rdma support for datacenter applications[C]// The 26th ACM Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China. New York:ACM Press, 2017: 306-324.
[13]	BRONSON N , AMSDEN Z , CABRERA G ,et al. TAO:Facebook's distributed data store for the social graph[C]// The 2013 USENIX Conference on Annual Technical Conference,June 26-28,2013,San Jose,USA. Berkeley:USENIX Association, 2013: 49-60.
[14]	DECANDIA G , HASTORUN D , JAMPANI M ,et al. Dynamo:amazon’s highly available key-value store[J]. ACM SIGOPS Operating Systems Review, 2007,41(6): 205-220.
[15]	ATIKOGLU B , XU Y H , FRACHTENBERG E ,et al. Workload analysis of a large-scale key-value store[J]. ACM SIGMETRICS Performance Evaluation Review, 2012,40(1): 53-64.
[16]	NISHTALA R , FUGAL H , GRIMM S ,et al. Scaling memcache at Facebook[C]// The 10th USENIX Conference on Networked Systems Design and Implementation (NSDI’13),April 2-5,2013,Lombard,USA. Berkeley:USENIX Association, 2013: 385-398.
[17]	MITCHELL C , GENG Y , LI J . Using onesided RDMA reads to build a fast,CPUefficient key-value store[C]// The 2013 USENIX Annual Technical Conference,June 26-28,2013,San Jose,USA. Berkeley:USENIX Association, 2013: 103-114.
[18]	PAGH R , RODLER F F . Cuckoo hashing[J]. Journal of Algorithms, 2004,51(2): 122-144.
[19]	HERLIHY M , SHAVIT N , TZAFRIR M . Hopscotch Hashing[C]// The 22nd International Symposium on Distributed Computing,September 22-24,2008,Arcachon,France. Berkeley:USENIX Association, 2008: 350-364.
[20]	MONTGOMERY C M K , NELSON L , SEN S ,et al. Balancing CPU and network in the cell distributed B-Tree store[C]// 2016 USENIX Annual Technical Conference,June 22-24,2016,Denver,USA. Berkeley:USENIX Association, 2016.
[21]	ZAMANIAN E , BINNIG C , HARRIS T ,et al. The end of a myth:distributed transactions can scale[J]. Proceedings of the VLDB Endowment, 2017,10(6): 685-696.
[22]	BERENSON H , BERNSTEIN P , GRAYJ ,et al. A critique of ANSI SQL isolation levels[C]// The 1995 ACM SIGMOD International Conference on Management of Data,May 22-25,1995,San Jose,USA. New York:ACM Press, 1995: 1-10.
[23]	WU M , YANG F , XUE J ,et al. GraM:Scaling graph computation to the trillions[C]// The 6th ACM Symposium on Cloud Computing (SoCC’15),August 27-29,2015,Kohala Coast,USA. New York:ACM Press, 2015: 408-421.
[24]	WEI X , SHEN S , CHEN R ,et al. Replication-driven live reconfiguration for fast distributed transaction processing[C]// 2017 USENIX Annual Technical Conference,July 12-14,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017: 335-347.
[25]	POKE M , HOEFLER T . Dare:highperformance state machine replication on RDMA networks[C]// The 24th International Symposium on High-Performance Parallel and Distributed Computing,June 15-19,2015,Portland,USA. New York:ACM Press, 2015: 107-118.
[26]	WANG C , JIANG J , CHEN X ,et al. APUS:fast and scalable paxos on RDMA[C]// The 2017 Symposium on Cloud Computing,September 25-27,2017,Santa Clara,USA. New York:ACM Press, 2017: 94-107.
[27]	WEI X , SHI J , CHEN Y ,et al. Fast inmemory transaction processing using RDMA and HTM[C]// The 25th Symposium on Operating Systems Principles,October 4-7,2015,Monterey,USA. New York:ACM Press, 2015: 87-104.
[28]	DEAN J , BARROSO L A . The tail at scale[J]. Communications of the ACM, 2013,56(2): 74-80.
[29]	SHI J , YAO Y , CHEN R ,et al. Fast and concurrent RDF queries with RDMAbased distributed graph exploration[C]// The 12th USENIX Conference on Operating Systems Design and Implementation,November 2-4,2016,Savannah,USA. Berkeley:USENIX Association, 2016: 317-332.
[30]	AJOUX P , BRONSON N , KUMAR S ,et al. Challenges to adopting stronger consistency at scale[C]// The 15th USENIX Conference on Hot Topics in Operating Systems,May 18-20,2015,Kartause Ittingen,Switzerland. Berkeley:USENIX Association, 2015.
[31]	DAGLIS A , USTIUGOV D , NOVAKOVIC?S ,et al. Sabres:atomic object reads for in-memory rack-scale computing[C]// The 49th Annual IEEE/ACM International Symposium on Microarchitecture,October 15-19,2016,Taipei,China. Piscataway:IEEE Press, 2016: 1-13.
[32]	RAIKIN S , LISS L , SHACHAR A ,et al. Remote transactional memory:20150269116[P].2015-09-24.
[33]	LI B , RUAN Z , XIAO W ,et al. KV-direct:high-performance in-memory keyvalue store with programmable NIC[C]// The 26th Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China. New York:ACM Press, 2017: 137-152.

连接模式/请求	send/recv	one-sided write	one-sided read/atomics
RC	√	√	√
UC	√	√	×
UD	√	×	×

基于RDMA高速网络的高性能分布式系统

Optimizing distributed systems with remote direct memory access

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 3

参考文献 33

相关文章 3

Metrics

推荐阅读 0

[1]	袁旭初, 付国, 毕继泽, 张岩峰, 聂铁铮, 谷峪, 鲍玉斌, 于戈. 分布式数据流计算系统的数据缓存技术综述[J]. 大数据, 2020, 6(3): 101-116.
[2]	柯学翰,陈榕. 基于图查询系统的图计算引擎[J]. 大数据, 2019, 5(4): 16-26.
[3]	吴昊,陈康,武永卫,郑纬民. 基于RDMA和NVM的大数据系统一致性协议研究[J]. 大数据, 2019, 5(4): 89-99.