大数据环境下的存储系统构建：挑战、方法和趋势

doi:10.11959/j.issn.2096-0271.2019030

Abstract

Abstract:

The rapid expansion of the Internet has led to an explosive growth in global data.New applications,such as the Internet of things and e-commerce,demand higher requirements for the latency and performance of data storage and processing.Thus,big data is not only more “big” but also more “fast”,and there is an urgent need to combine new storage media to build large-scale,high-performance storage systems.Two storage system design schemes were started,namely,flash storage and persistent memory storage.Their respective challenges were illustrated in detail,and the existing solutions were summarized.Finally,the future trends in the construction of data centers and storage systems were looked into.

Key words: storage system, flash memory, non-volatile memory

CLC Number:

TP316.4

Youmin CHEN, Fei LI, Jiwu SHU. Building storage systems in big data era:challenges,methods and trends[J]. Big Data Research, 2019, 5(4): 27-40.

Figures/Tables 4

References 23

[1]	SHARMA S . Trends in server efficiency and power usage in data centers[R]. Auckland:SPEC 2016 ASIA SUMMIT, 2016.
[2]	AJOUX P , BRONSON N , KUMAR S ,et al. Challenges to adopting stronger consistency at scale[C]// The 15th Workshop on Hot Topics in Operating Systems (HotOS XV),May 18-20,2015,Switzerland. Berkeley:USENIX Association, 2015.
[3]	AGRAWAL N , PRABHAKARAN V , WOBBER T ,et al. Design tradeoffs for SSD performance[C]// USENIX Annual Technical Conference,June 22-27,2008,Boston,USA. Berkeley:USENIX Association, 2008:57.
[4]	LU Y Y , SHU J W , ZHENG W . Extending the lifetime of flash-based storage through reducing write amplification from file systems[C]// The 11th USENIX Conference on File and Storage Technologies,February 12-15,San Jose,USA. Berkeley:USENIX Association, 2013: 257-270.
[5]	LEE S , LIU M , JUN S ,et al. Applicationmanaged flash[C]// The 14th USENIX Conference on File and Storage Technologies,February 22-25,2016,San Jose,USA. Berkeley:USENIX Association, 2016: 339-353.
[6]	OUYANG J , LIN S , JIANG S ,et al. SDF:software-defined flash for webscale internet storage systems[J]. ACM SIGPLAN Notices, 2014,49(4): 471-484.
[7]	ZHANG J , LU Y , SHU J ,et al. FlashKV:accelerating KV performance with openchannel SSDs[J]. ACM Transactions on Embedded Computing Systems, 2017,16(5s):139.
[8]	BJ?RLING M , GONZáLEZ J , BONNET P . LightNVM:the Linux open-channel{SSD} subsystem[C]// The 15th USENIX Conference on File and Storage Technologies,February 27-March 2,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017: 359-374.
[9]	HUANG F B , . Achieving both performance isolation and uniform lifetime for virtualized SSDs[C]// The 15th USENIX Conference on File and Storage Technologies,February 27-March 2,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017.
[10]	ZHANG J , SHU J , LU Y . ParaFS:a log-structured file system to exploit the internal parallelism of flash devices[C]// 2016 USENIX Annual Technical Conference,June 22-24,2016,Denver,USA. Berkeley:USENIX Association, 2016: 87-100.
[11]	CARLSON M . Storage for hyperscalers[R]. Santa Clara:SNIA 2016, 2016.
[12]	YANG J , PLASSON N , GILLIS G ,et al. Don’t stack your log on my log[C]// The 2nd Workshop on Interactions of NVM/Flash with Operating Systems and Workloads,October 5,2014,Broomfield,USA.[S.l.:s.n. ], 2014.
[13]	IZRAELEVITZ J , YANG J , ZHANG L ,et al. Basic performance measurements of the Intel Optane DC persistent memory module[J]. Computer Science, 2019,arXiv:1903.05714.
[14]	SWANSON S . Redrawing the boundary between software and storage for fast non-volatile memories[R]. San Diego:University of California, 2015.
[15]	CONDIT J , NIGHTINGALE E B , FROST C ,et al. Better I/O through byteaddressable,persistent memory[C]// The ACM SIGOPS 22nd Symposium on Operating Systems Principles,October 11-14,2009,Big Sky,USA. New York:ACM Press, 2009: 133-146.
[16]	NARAYANAN D , HODSON O . Wholesystem persistence[J]. ACM SIGARCH Computer Architecture News, 2012,40(1): 401-410.
[17]	LU Y Y , SHU J W , SUN L . Blurred persistence in transactional persistent memory[C]// The 31st Symposium on Mass Storage Systems and Technologies (MSST),May 30-June 5,2015,Santa Clara,USA. Piscataway:IEEE Press, 2015: 1-13.
[18]	DULLOOR S R , KUMAR S , KESHAVAMURTHY A ,et al. System software for persistent memory[C]// The 9th European Conference on Computer Systems,April 14-16,2014,Amsterdam,the Netherlands. New York:ACM Press, 2014:15.
[19]	XU J , SWANSON S . NOVA:a logstructured file system for hybrid volatile/non-volatile main memories[C]// The 14th USENIX Conference on File and Storage Technologies,February 22-25,2016,Santa Clara,USA. Berkeley:USENIX Association, 2016: 323-338.
[20]	VOLOS H , NALLI S , PANNEERSELVAM S ,et al. Aerie:flexible file-system interfaces to storage-class memory[C]// The 9th European Conference on Computer Systems,April 14-16,2014,Amsterdam,the Netherlands. New York:ACM Press, 2014:14.
[21]	KWON Y , FINGLER H , HUNT T ,et al. Strata:a cross media file system[C]// The 26th Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China. New York:ACM Press, 2017: 460-477.
[22]	陈游旻, 陆游游, 罗圣美 ,等. 基于RDMA的分布式存储系统研究综述[J]. 计算机研究与发展, 2019,56(2): 227-239.
	CHEN Y M , LU Y Y , LUO S M ,et al. Survey on RDMA-based distributed storage systems[J]. Journal of Computer Research and Development, 2019,56(2): 227-239.
[23]	LU Y Y , SHU J W , CHEN Y M ,et al. Octopus:an RDMA-enabled distributed persistent memory file system[C]// 2017 USENIX Annual Technical Conference (USENIX ATC 17),July 12-14,2017,Santa Clara,USA. Berkeley:USENIX Association, 2017: 773-785.

Metrics

Recommended 0

No Suggested Reading articles found!

设备型号	接口类型	带宽/(MB/s)		时延/ms		IOPS
设备型号	接口类型	读	写	读	写	读	写
希捷Savvio 15K.3	SAS 6 Gbit/s	202	202	2	2	543	428
Intel S3710	SATA 3 Gbit/s	550	520	0.055	0.066	85×10³	45×10³
Intel P3700	PCIe 3.0 x4	2 800	2 000	0.02	0.02	460×10³	175×10³

Building storage systems in big data era:challenges,methods and trends

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 4

References 23

Related Articles 5

Metrics

Recommended 0

[1]	Yongfeng WANG, Zhiguang CHEN. A survey of persistent index data structures on non-volatile memory [J]. Big Data Research, 2021, 7(6): 78-88.
[2]	Zhenlong DENG, Zhiguang CHEN. An optimization of MPI-IO interface for non-volatile memory [J]. Big Data Research, 2021, 7(2): 172-181.
[3]	Xiaoyuan WANG, Xiaofei LIAO, Haikun LIU, Hai JIN. Big data oriented hybrid memory systems [J]. Big Data Research, 2018, 4(4): 15-34.
[4]	Xin LI, Xuan CHEN, Zhiqiu HUANG. Analysis on hybrid memory architecture for big data application [J]. Big Data Research, 2018, 4(3): 61-80.
[5]	Limin XIAO, Zhisheng HUO. Review of I/O performance optimization technology for big data storage system [J]. Big Data Research, 2017, 3(6): 65-84.