基于RDMA高速网络的高性能分布式系统

doi:10.11959/j.issn.2096-0271.2018036

Abstract

Abstract:

Fast network devices with RDMA support have been price-compatible with traditional network primitives such as Ethernet,and it’s now widely deployed in modern data centers.RDMA can be used in two ways.Firstly,it can optimize the messaging primitive in distributed applications.The second way is to redesign the applications with RDMA’s onesided features.One-sided features provide high CPU utilizations and high network performance,but the system should be redesigned.The research progress of RDMA was introduced.An overview on the research efforts on using RDMA for distributed systems was presented.The works on how to use RDMA to redesign systems and the works on how to better leverage RDMA were included.The future research directions were also put forward.

Key words: distributed system, key-value stores system, graph processing system, OLTP system, remote procedure call

CLC Number:

TP316

Xingda WEI, Rong CHEN, Haibo CHEN. Optimizing distributed systems with remote direct memory access[J]. Big Data Research, 2018, 4(4): 3-14.

Figures/Tables 3

References 33

[1]	FAN B , ANDERSEN D G , KAMINSKY M . MemC3:compact and concurrent MemCache with dumber caching and smarter hashing[C]// The 10th USENIX Conference on Networked Systems Design and Implementation,April2-5,2013,San Francisco,USA. Berkeley:USENIX Association, 2013: 371-384.
[2]	GUO C , WU H , DENG Z ,et al. RDMA over commodity ethernet at scale[C]// The 2016 ACM SIG-COMM Conference,August 22-26,2016,Florianopolis,Brazil. New York:ACM Press, 2016: 202-215.
[3]	TSAI S Y , ZHANG Y . Lite kernel rdma support for datacenter applications[C]// The 26th ACM Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China.[S.l.:s.n]. 2017: 306-324.
[4]	NARAYANAN D , HODSON O , CASTRO M ,et al. FaRM:fast remote memory[C]// The 11th USENIX Conference on Networked Systems Design and Implementation,April 2-4,2014,Seattle,USA. Berkeley:USENIX Association, 2014: 401-414.
[5]	DRAGOJEVI? A , NARAYANAN D , NIGHTINGALE E B ,et al. No compromises:distributed transactions with consistency,availability,and performance[C]// The 25th Symposium on Operating Systems Principles,October 4-7,2015,Monterey,USA. New York:ACM Press, 2015: 54-70.
[6]	NEWMARCH J . Remote procedure call[D]. Pittsburgh:Carnegie Mellon University, 1981.
[7]	KALIA A , KAMINSKY M , ANDERSEN D G . Design guidelines for high performance RDMA systems[C]// 2016 USENIX Annual Technical Conference,June 22-24,Denver,USA. Berkeley:USENIX Association, 2016: 437-450.
[8]	KALIA A , KAMINSKY M , ANDERSEN D G . FaSST:fast,scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs[C]// The 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16),November 2-4,2016,Savannah,USA. Berkeley:USENIX Association, 2016: 185-201.
[9]	LU Y , SHU J , CHEN Y ,et al. Octopus:an RDMA-enabled distributed persistent memory file system[C]// 2017 USENIX Annual Technical Conference (USENIX ATC 17),July 12-14,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017: 773-785.
[10]	SU M , ZHANG M , CHEN K ,et al. RFP:when RPC is faster than server-bypass with RDMA[C]// The 12th European Conference on Computer Systems,April 23-26,2017,Belgrade,Serbia. New York:ACM Press, 2017: 1-15.
[11]	KALIA A , KAMINSKY M , ANDERSEN D G . Using RDMA efficiently for key-value services[C]// The 2014 ACM Conference on SIGCOMM,August 17-22,2014,Chicago,USA. New York:ACM Press, 2014,44(4): 295-306.
[12]	TSAI S Y , ZHANG Y . Lite kernel rdma support for datacenter applications[C]// The 26th ACM Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China. New York:ACM Press, 2017: 306-324.
[13]	BRONSON N , AMSDEN Z , CABRERA G ,et al. TAO:Facebook's distributed data store for the social graph[C]// The 2013 USENIX Conference on Annual Technical Conference,June 26-28,2013,San Jose,USA. Berkeley:USENIX Association, 2013: 49-60.
[14]	DECANDIA G , HASTORUN D , JAMPANI M ,et al. Dynamo:amazon’s highly available key-value store[J]. ACM SIGOPS Operating Systems Review, 2007,41(6): 205-220.
[15]	ATIKOGLU B , XU Y H , FRACHTENBERG E ,et al. Workload analysis of a large-scale key-value store[J]. ACM SIGMETRICS Performance Evaluation Review, 2012,40(1): 53-64.
[16]	NISHTALA R , FUGAL H , GRIMM S ,et al. Scaling memcache at Facebook[C]// The 10th USENIX Conference on Networked Systems Design and Implementation (NSDI’13),April 2-5,2013,Lombard,USA. Berkeley:USENIX Association, 2013: 385-398.
[17]	MITCHELL C , GENG Y , LI J . Using onesided RDMA reads to build a fast,CPUefficient key-value store[C]// The 2013 USENIX Annual Technical Conference,June 26-28,2013,San Jose,USA. Berkeley:USENIX Association, 2013: 103-114.
[18]	PAGH R , RODLER F F . Cuckoo hashing[J]. Journal of Algorithms, 2004,51(2): 122-144.
[19]	HERLIHY M , SHAVIT N , TZAFRIR M . Hopscotch Hashing[C]// The 22nd International Symposium on Distributed Computing,September 22-24,2008,Arcachon,France. Berkeley:USENIX Association, 2008: 350-364.
[20]	MONTGOMERY C M K , NELSON L , SEN S ,et al. Balancing CPU and network in the cell distributed B-Tree store[C]// 2016 USENIX Annual Technical Conference,June 22-24,2016,Denver,USA. Berkeley:USENIX Association, 2016.
[21]	ZAMANIAN E , BINNIG C , HARRIS T ,et al. The end of a myth:distributed transactions can scale[J]. Proceedings of the VLDB Endowment, 2017,10(6): 685-696.
[22]	BERENSON H , BERNSTEIN P , GRAYJ ,et al. A critique of ANSI SQL isolation levels[C]// The 1995 ACM SIGMOD International Conference on Management of Data,May 22-25,1995,San Jose,USA. New York:ACM Press, 1995: 1-10.
[23]	WU M , YANG F , XUE J ,et al. GraM:Scaling graph computation to the trillions[C]// The 6th ACM Symposium on Cloud Computing (SoCC’15),August 27-29,2015,Kohala Coast,USA. New York:ACM Press, 2015: 408-421.
[24]	WEI X , SHEN S , CHEN R ,et al. Replication-driven live reconfiguration for fast distributed transaction processing[C]// 2017 USENIX Annual Technical Conference,July 12-14,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017: 335-347.
[25]	POKE M , HOEFLER T . Dare:highperformance state machine replication on RDMA networks[C]// The 24th International Symposium on High-Performance Parallel and Distributed Computing,June 15-19,2015,Portland,USA. New York:ACM Press, 2015: 107-118.
[26]	WANG C , JIANG J , CHEN X ,et al. APUS:fast and scalable paxos on RDMA[C]// The 2017 Symposium on Cloud Computing,September 25-27,2017,Santa Clara,USA. New York:ACM Press, 2017: 94-107.
[27]	WEI X , SHI J , CHEN Y ,et al. Fast inmemory transaction processing using RDMA and HTM[C]// The 25th Symposium on Operating Systems Principles,October 4-7,2015,Monterey,USA. New York:ACM Press, 2015: 87-104.
[28]	DEAN J , BARROSO L A . The tail at scale[J]. Communications of the ACM, 2013,56(2): 74-80.
[29]	SHI J , YAO Y , CHEN R ,et al. Fast and concurrent RDF queries with RDMAbased distributed graph exploration[C]// The 12th USENIX Conference on Operating Systems Design and Implementation,November 2-4,2016,Savannah,USA. Berkeley:USENIX Association, 2016: 317-332.
[30]	AJOUX P , BRONSON N , KUMAR S ,et al. Challenges to adopting stronger consistency at scale[C]// The 15th USENIX Conference on Hot Topics in Operating Systems,May 18-20,2015,Kartause Ittingen,Switzerland. Berkeley:USENIX Association, 2015.
[31]	DAGLIS A , USTIUGOV D , NOVAKOVIC?S ,et al. Sabres:atomic object reads for in-memory rack-scale computing[C]// The 49th Annual IEEE/ACM International Symposium on Microarchitecture,October 15-19,2016,Taipei,China. Piscataway:IEEE Press, 2016: 1-13.
[32]	RAIKIN S , LISS L , SHACHAR A ,et al. Remote transactional memory:20150269116[P].2015-09-24.
[33]	LI B , RUAN Z , XIAO W ,et al. KV-direct:high-performance in-memory keyvalue store with programmable NIC[C]// The 26th Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China. New York:ACM Press, 2017: 137-152.

Metrics

Recommended 0

No Suggested Reading articles found!

连接模式/请求	send/recv	one-sided write	one-sided read/atomics
RC	√	√	√
UC	√	√	×
UD	√	×	×

Optimizing distributed systems with remote direct memory access

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 3

References 33

Related Articles 3

Metrics

Recommended 0

[1]	Xuchu YUAN, Guo FU, Jize BI, Yanfeng ZHANG, Tiezheng NIE, Yu GU, Yubin BAO, Ge YU. Survey on data caching technology of distributed dataflow system [J]. Big Data Research, 2020, 6(3): 101-116.
[2]	Xuehan KE,Rong CHEN. Graph processing engine based on graph query system [J]. Big Data Research, 2019, 5(4): 16-26.
[3]	Hao WU,Kang CHEN,Yongwei WU,Weimin ZHENG. Research on the consensus of big data systems based on RDMA and NVM [J]. Big Data Research, 2019, 5(4): 89-99.