面向大数据异构系统的神威并行存储系统

doi:10.11959/j.issn.2096-0271.2020031

Abstract

Abstract:

With the integration of big data applications and traditional high-performance computing applications and the introduction of heterogeneous computing,the traditional parallel storage system for high-performance computing faces the problems of poor I/O support,performance interference,and low efficiency.By introducing multi-level storage architecture into the system architecture,the cache mapping mechanism was designed to reduce the I/O load.The I/O forwarding strategy was adjusted in the forwarding service layer to balance the I/O load.In the back-end storage layer,the high availability function of the system was adjusted to solve the conflict between the big data I/O access mode and the original high availability functions.After optimized design and improvement,the parallel storage system can better adapt to the heterogeneous multi-core architecture,making some applications get more than 10 times of I/O performance improvement.

Key words: big data, high performance computing, Sunway TaihuLight, heterogeneous, parallel storage

CLC Number:

TP316

Xiaobin HE, Jinhu JIANG. Sunway parallel storage system for big data heterogeneous system[J]. Big Data Research, 2020, 6(4): 30-39.

Figures/Tables 6

References 27

[1]	YAN Z F , LIN Y Z , PENG L ,et al. Harmonia:a high throughput B+tree for GPUs[C]// The 24th Symposium on Principles and Practice of Parallel Programming.[S.l.:s.n]. 2019.
[2]	ZHANG W , YAN Z , LIN Y ,et al. A high throughput B+tree for SIMD architectures[J]. IEEE Transactions on Parallel and Distributed Systems, 2020,31(3): 707-720.
[3]	WANG X , ZHANG W H , WANG Z G ,et al. Eunomia:scaling concurrent search trees under contention using HTM[J]. ACM SIGPLAN Notices, 2017,52(8): 385-399.
[4]	ANDERSON M , SMITH S , SUNDARAM N ,et al. Bridging the gap between HPC and big data frameworks[J]. Proceedings of the VLDB Endowment, 2017,10(8): 901-912.
[5]	XUAN P F , DENTON J , LUO F ,et al. Big data analytics on traditional HPC infrastructure using two-level storage[C]// The 2015 International Workshop on Data-Intensive Scalable Computing Systems.[S.l.:s.n. ], 2015.
[6]	PAUL A K , GOYAL A , WANG F Y ,et al. I/O load balancing for big data HPC applications[C]// 2017 IEEE International Conference on Big Data. Piscataway:IEEE Press, 2017.
[7]	ISLAM N S , SHANKAR D , LU X Y ,et al. Accelerating I/O performance of big data analytics on HPC clusters through RDMA-based key-value store[C]// The 44th International Conference on Parallel Processing. Piscataway:IEEE Press, 2015.
[8]	QIU J , JHA S , LUCKOW A ,et al. Towards HPC-ABDS:an initial high-performance big data stack[J]. ACM, 2014,1(1): 1-22.
[9]	XUAN P F , LIGON W B , SRIMANI P K ,et al. Accelerating big data analytics on HPC clusters using two-level storage[J]. Parallel Computing, 2017,61(1): 18-34.
[10]	朱传家, 刘鑫, 方佳瑞 . 基于“神威·太湖之光”的Caffe分布式扩展研究[J]. 计算机应用与软件, 2020,37(1): 15-20.
	ZHU C J , LIU X , FANG J R . Distributed optimization study for Caffe on Sunway TaihuLight supercomputer[J]. Computer Applications and Software, 2020,37(1): 15-20.
[11]	FU H H , LIAO J F , YANG J Z ,et al. The Sunway TaihuLight supercomputer:system and applications[J]. Science China (Information Sciences), 2016,59(7):072001.
[12]	CHEN Q , CHEN K , CHEN Z N ,et al. Lessons learned from optimizing the Sunway storage system for higher application I/O performance[J]. Journal of Computer Science and Technology, 2020,35(1): 47-60.
[13]	WU Y , RODR?GUEZ J , BOURILKOV D ,et al. Utilizing Lustre file system with DCache for CMS analysis[J]. Journal of Physics:Conference Series, 2010,219:062068.
[14]	HEBENSTREIT M . Performance evaluation of Intel^?SSD-Based Lustre*Cluster file systems at the Intel^?CRTDC[J]. 2010.
[15]	KOO D , KIM J S , HWANG S ,et al. Utilizing progressive file layout leveraging SSDs in HPC cloud environments[C]// 2016 IEEE 1st International Workshops on Foundations and Applications of Self*Systems. Piscataway:IEEE Press, 2016.
[16]	XIN L , LU Y , LU Y T ,et al. masFS:file system based on memory and SSD in compute nodes for high performance computers[C]// IEEE International Conference on Parallel ＆ Distributed Systems. Piscataway:IEEE Press, 2016.
[17]	CHEN S , LIU L , ZHANG W H ,et al. Architectural support for NVRAM persistence in GPUs[J]. IEEE Transactions on Parallel and Distributed Systems, 2019(219):1.
[18]	MCKUSICK M K , QUINLAN S . GFS:evolution on fast-forward[J]. Queue, 2009,7(7): 10-20.
[19]	CHEN M , MAO S W , ZHANG Y ,et al. Big data storage[M]. Heidelberg: SpringerPress, 2014: 33-49.
[20]	WEETS J F , KAKHANI M K , KUMAR A . Limitations and challenges of HDFS and MapReduce[C]// 2015 International Conference on Green Computing and Internet of Things. Piscataway:IEEE Press, 2015.
[21]	ANDREU-PEREZ J , CAO F , HAGRAS H ,et al. A self-adaptive online brainmachine interface of a humanoid robot through a general type-2 fuzzy inference system[J]. IEEE Transactions on Fuzzy Systems, 2018,26(1): 101-116.
[22]	XIE J , MENG F J , WANG H L ,et al. Research on scheduling scheme for Hadoop clusters[C]// The 2nd International Conference on Computer and Applications.[S.l.:s.n]. 2013: 49-52.
[23]	YU W K , WANG Y D , QUE X Y ,et al. Virtual shuffling for efficient data movement in MapReduce[J]. IEEE Transactions on Computers, 2015,64(2): 556-568.
[24]	何晓斌, 蒋金虎, 魏巍 ,等. 神威蓝光计算机轻量级文件系统LWFS的优化和测试[J]. 高性能计算技术, 2012(5): 41-45.
	HE X B , JIANG J H , WEI W ,et al. Accelerating and evaluation of LWFS in Sunway BlueLight computer system[J]. High Performance Computing Technology, 2012(5): 41-45.
[25]	VISHWANATH V , HERELD M , ISKRA K ,et al. Accelerating I/O forwarding in IBM blue gene/p systems[C]// The 2010 ACM/IEEE International Conference for High Performance Computing,Networking,Storage and Analysis. Piscataway:IEEE Press, 2010.
[26]	JI X , YANG B , ZHANG T Y ,et al. Automatic,application-aware I/O forwarding resource allocation[C]// The 17th USENIX Conference on File and Storage Technologies.[S.l.:s.n]. 2019.
[27]	BRAAM P . The Lustre storage architecture[J]. Computer Science, 2019,arXiv:1903.01955.

Metrics

Recommended 0

No Suggested Reading articles found!

Sunway parallel storage system for big data heterogeneous system

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 27

Related Articles 15

Metrics

Recommended 0

[1]	Haihong QIAN, Maoyi WANG, Yun XIONG. Digital transformation in higher education:a systematic review [J]. Big Data Research, 2023, 9(3): 56-70.
[2]	Chuanyao ZHANG, Shijing SI, Jianzong WANG, Jing XIAO. Federated meta learning: a review [J]. Big Data Research, 2023, 9(2): 122-146.
[3]	Hong MEI, Xiaoyong DU, Hai JIN, Xueqi CHENG, Yunpeng CHAI, Xuanhua SHI, Xiaolong JIN, Yasha WANG, Chi LIU. Big data technologies forward-looking [J]. Big Data Research, 2023, 9(1): 1-20.
[4]	Yang SHEN, Menglong YU. Metaverse and big data: data insight and value connection in spatio-temporal intelligence [J]. Big Data Research, 2023, 9(1): 103-110.
[5]	Tongzheheng ZHENG, Bin LI, Minxuan FENG, Bolin CHANG, Dongbo WANG. Explore the structuration of historical books:the construction and quantitative analysis of digital humanities database of the Biographies of the Shiji [J]. Big Data Research, 2022, 8(6): 40-55.
[6]	Jing CHEN. Humanities big data and its application in the field of digital humanities [J]. Big Data Research, 2022, 8(6): 3-14.
[7]	Yuchu LUO, Hao WU, Yuhan GUO, Shaocong TAN, Can LIU, Ruike JIANG, Xiaoru YUAN. Visualization in digital humanities [J]. Big Data Research, 2022, 8(6): 74-93.
[8]	Wenlong LI, Yuan YUAN, Xiaopeng AN. Modus operandi of big data governance: some preliminary observations [J]. Big Data Research, 2022, 8(4): 34-45.
[9]	Qifeng TANG, Zhiqing SHAO, Yazhen YE. Authenticating and licensing architecture of data rights in data trade [J]. Big Data Research, 2022, 8(3): 40-53.
[10]	Chenhuizi WANG, Wei CAI. Digital economics in metaverse: state-of-the-art, characteristics, and vision [J]. Big Data Research, 2022, 8(3): 140-150.
[11]	Mei YANG, Wei LI, Siyuan QIAO, Wei LIU. Research on calculation method of China’s big data industry output value [J]. Big Data Research, 2022, 8(3): 151-160.
[12]	Deren LI, Guo ZHANG, Yonghua JIANG, Xin SHEN, Weiling LIU. Opportunities and challenges of geo-spatial information science from the perspective of big data [J]. Big Data Research, 2022, 8(2): 3-14.
[13]	Xiaolan QIU, Yuxin HU, Songtao SHANGGUAN, Kun FU. Remote sensing satellite big data high-recision integration processing technology [J]. Big Data Research, 2022, 8(2): 15-27.
[14]	Weiquan LIU, Cheng WANG, Yu ZANG, Qian HU, Shangshu YU, Baiqi LAI. A survey on information extraction technology based on remote sensing big data [J]. Big Data Research, 2022, 8(2): 28-57.
[15]	Jianqiang LIU, Xiaomin YE, Youguo LAN. Remote sensing big data from Chinese ocean satellites and its application service [J]. Big Data Research, 2022, 8(2): 75-88.