大数据环境下的存储系统构建：挑战、方法和趋势

doi:10.11959/j.issn.2096-0271.2019030

大数据 ›› 2019, Vol. 5 ›› Issue (4): 27-40.doi: 10.11959/j.issn.2096-0271.2019030

• 专题：大数据的系统结构 • 上一篇下一篇

大数据环境下的存储系统构建：挑战、方法和趋势

陈游旻,李飞,舒继武

清华大学计算机科学与技术系，北京 100084

出版日期:2019-07-15 发布日期:2019-08-09
作者简介:陈游旻（1993- ），男，清华大学计算机科学与技术系博士生，主要研究方向为文件系统、分布式系统。|李飞（1993- ），男，清华大学计算机科学与技术系硕士生，主要研究方向为闪存存储系统。|舒继武（1968- ），男，博士，清华大学计算机科学与技术系教授，教育部长江学者特聘教授，IEEE Fellow，博士生导师。近年来主要从事基于非易失存储器件的新型存储系统与技术、基于Flash器件的固态存储系统与技术、网络（云/大数据）存储系统与关键技术、数据存储可靠性等方面的研究工作。
基金资助:
：国家重点研发计划基金资助项目(2018YFB1003301)

Building storage systems in big data era:challenges,methods and trends

Youmin CHEN,Fei LI,Jiwu SHU

Department of Computer Science and Technology,Tsinghua University,Beijing 100084,China

Online:2019-07-15 Published:2019-08-09
Supported by:
National Key Research and Development Program of China(2018YFB1003301)

摘要/Abstract

摘要：

互联网规模的迅速扩展促使全球数据总量呈现爆炸式的增长。物联网、电子商务等新的应用对数据存储及处理的实时性提出了更高的要求，迫切需要结合新型存储介质，以构建大规模、高性能存储系统。分别从闪存存储、持久性内存存储两种存储系统构建方案出发，详细阐述了其各自面临的挑战，并总结了现有的解决方案。最后，展望了未来数据中心及存储系统构建的若干发展趋势。

关键词: 存储系统, 闪存, 非易失内存

Abstract:

The rapid expansion of the Internet has led to an explosive growth in global data.New applications,such as the Internet of things and e-commerce,demand higher requirements for the latency and performance of data storage and processing.Thus,big data is not only more “big” but also more “fast”,and there is an urgent need to combine new storage media to build large-scale,high-performance storage systems.Two storage system design schemes were started,namely,flash storage and persistent memory storage.Their respective challenges were illustrated in detail,and the existing solutions were summarized.Finally,the future trends in the construction of data centers and storage systems were looked into.

Key words: storage system, flash memory, non-volatile memory

中图分类号:

TP316.4

陈游旻, 李飞, 舒继武. 大数据环境下的存储系统构建：挑战、方法和趋势[J]. 大数据, 2019, 5(4): 27-40.

Youmin CHEN, Fei LI, Jiwu SHU. Building storage systems in big data era:challenges,methods and trends[J]. Big Data Research, 2019, 5(4): 27-40.

图/表 4

参考文献 23

[1]	SHARMA S . Trends in server efficiency and power usage in data centers[R]. Auckland:SPEC 2016 ASIA SUMMIT, 2016.
[2]	AJOUX P , BRONSON N , KUMAR S ,et al. Challenges to adopting stronger consistency at scale[C]// The 15th Workshop on Hot Topics in Operating Systems (HotOS XV),May 18-20,2015,Switzerland. Berkeley:USENIX Association, 2015.
[3]	AGRAWAL N , PRABHAKARAN V , WOBBER T ,et al. Design tradeoffs for SSD performance[C]// USENIX Annual Technical Conference,June 22-27,2008,Boston,USA. Berkeley:USENIX Association, 2008:57.
[4]	LU Y Y , SHU J W , ZHENG W . Extending the lifetime of flash-based storage through reducing write amplification from file systems[C]// The 11th USENIX Conference on File and Storage Technologies,February 12-15,San Jose,USA. Berkeley:USENIX Association, 2013: 257-270.
[5]	LEE S , LIU M , JUN S ,et al. Applicationmanaged flash[C]// The 14th USENIX Conference on File and Storage Technologies,February 22-25,2016,San Jose,USA. Berkeley:USENIX Association, 2016: 339-353.
[6]	OUYANG J , LIN S , JIANG S ,et al. SDF:software-defined flash for webscale internet storage systems[J]. ACM SIGPLAN Notices, 2014,49(4): 471-484.
[7]	ZHANG J , LU Y , SHU J ,et al. FlashKV:accelerating KV performance with openchannel SSDs[J]. ACM Transactions on Embedded Computing Systems, 2017,16(5s):139.
[8]	BJ?RLING M , GONZáLEZ J , BONNET P . LightNVM:the Linux open-channel{SSD} subsystem[C]// The 15th USENIX Conference on File and Storage Technologies,February 27-March 2,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017: 359-374.
[9]	HUANG F B , . Achieving both performance isolation and uniform lifetime for virtualized SSDs[C]// The 15th USENIX Conference on File and Storage Technologies,February 27-March 2,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017.
[10]	ZHANG J , SHU J , LU Y . ParaFS:a log-structured file system to exploit the internal parallelism of flash devices[C]// 2016 USENIX Annual Technical Conference,June 22-24,2016,Denver,USA. Berkeley:USENIX Association, 2016: 87-100.
[11]	CARLSON M . Storage for hyperscalers[R]. Santa Clara:SNIA 2016, 2016.
[12]	YANG J , PLASSON N , GILLIS G ,et al. Don’t stack your log on my log[C]// The 2nd Workshop on Interactions of NVM/Flash with Operating Systems and Workloads,October 5,2014,Broomfield,USA.[S.l.:s.n. ], 2014.
[13]	IZRAELEVITZ J , YANG J , ZHANG L ,et al. Basic performance measurements of the Intel Optane DC persistent memory module[J]. Computer Science, 2019,arXiv:1903.05714.
[14]	SWANSON S . Redrawing the boundary between software and storage for fast non-volatile memories[R]. San Diego:University of California, 2015.
[15]	CONDIT J , NIGHTINGALE E B , FROST C ,et al. Better I/O through byteaddressable,persistent memory[C]// The ACM SIGOPS 22nd Symposium on Operating Systems Principles,October 11-14,2009,Big Sky,USA. New York:ACM Press, 2009: 133-146.
[16]	NARAYANAN D , HODSON O . Wholesystem persistence[J]. ACM SIGARCH Computer Architecture News, 2012,40(1): 401-410.
[17]	LU Y Y , SHU J W , SUN L . Blurred persistence in transactional persistent memory[C]// The 31st Symposium on Mass Storage Systems and Technologies (MSST),May 30-June 5,2015,Santa Clara,USA. Piscataway:IEEE Press, 2015: 1-13.
[18]	DULLOOR S R , KUMAR S , KESHAVAMURTHY A ,et al. System software for persistent memory[C]// The 9th European Conference on Computer Systems,April 14-16,2014,Amsterdam,the Netherlands. New York:ACM Press, 2014:15.
[19]	XU J , SWANSON S . NOVA:a logstructured file system for hybrid volatile/non-volatile main memories[C]// The 14th USENIX Conference on File and Storage Technologies,February 22-25,2016,Santa Clara,USA. Berkeley:USENIX Association, 2016: 323-338.
[20]	VOLOS H , NALLI S , PANNEERSELVAM S ,et al. Aerie:flexible file-system interfaces to storage-class memory[C]// The 9th European Conference on Computer Systems,April 14-16,2014,Amsterdam,the Netherlands. New York:ACM Press, 2014:14.
[21]	KWON Y , FINGLER H , HUNT T ,et al. Strata:a cross media file system[C]// The 26th Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China. New York:ACM Press, 2017: 460-477.
[22]	陈游旻, 陆游游, 罗圣美 ,等. 基于RDMA的分布式存储系统研究综述[J]. 计算机研究与发展, 2019,56(2): 227-239.
	CHEN Y M , LU Y Y , LUO S M ,et al. Survey on RDMA-based distributed storage systems[J]. Journal of Computer Research and Development, 2019,56(2): 227-239.
[23]	LU Y Y , SHU J W , CHEN Y M ,et al. Octopus:an RDMA-enabled distributed persistent memory file system[C]// 2017 USENIX Annual Technical Conference (USENIX ATC 17),July 12-14,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017: 773-785.

设备型号	接口类型	带宽/(MB/s)		时延/ms		IOPS
设备型号	接口类型	读	写	读	写	读	写
希捷Savvio 15K.3	SAS 6 Gbit/s	202	202	2	2	543	428
Intel S3710	SATA 3 Gbit/s	550	520	0.055	0.066	85×10³	45×10³
Intel P3700	PCIe 3.0 x4	2 800	2 000	0.02	0.02	460×10³	175×10³

大数据环境下的存储系统构建：挑战、方法和趋势

Building storage systems in big data era:challenges,methods and trends

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 4

参考文献 23

相关文章 4

Metrics

推荐阅读 0

[1]	王永锋, 陈志广. 面向非易失性内存的持久索引数据结构研究综述[J]. 大数据, 2021, 7(6): 78-88.
[2]	邓镇龙, 陈志广. 面向非易失内存的MPI-IO接口优化[J]. 大数据, 2021, 7(2): 172-181.
[3]	魏星达, 陈榕, 陈海波. 基于RDMA高速网络的高性能分布式系统[J]. 大数据, 2018, 4(4): 3-14.
[4]	肖利民, 霍志胜. 大数据存储系统I/O性能优化技术研究进展[J]. 大数据, 2017, 3(6): 65-84.