GVDS：面向广域高性能计算环境的虚拟数据空间

doi:10.11959/j.issn.2096-0271.2021017

Abstract

Abstract:

The wide-area high-performance computing environment is the core information infrastructure to support technology innovation, economic development, and national defense.However, heterogeneous storage resources are geographically distributed in wide-area high-performance computing environments, resulting in the barriers between applications and data.The requirements of unified data management and efficient data access cannot be met.A method of establishing virtual data space and a data access optimization method was presented, and a global virtual data space (GVDS) for wide-area high-performance computing environments to satisfy the requirements was implemented.GVDS aggregates geographically distributed and heterogeneous storage resources, creating a unified virtual data space to provide unified and efficient data access.Sharing and collaborative processing of geographically distributed data were achieved in widearea environments.The experimental results indicate that compared with the state-of-the-art wide-area storage system in the field of high-performance computing, such as OneData and GFFS, GVDS has similar functions and improves the read bandwidth significantly.

Key words: global virtual data space, wide-area high-performance computing environment, efficient data access, heterogeneous storage resource

CLC Number:

TP316

Limin XIAO, Yao SONG, Guangjun QIN, Hanjie ZHOU, Chaobo WANG, Bing WEI, Wei WEI, Zhisheng HUO. GVDS: a global virtual data space for wide-area high-performance computing environments[J]. Big Data Research, 2021, 7(2): 123-146.

Figures/Tables 17

References 36

[1]	SLOTA R , DUTKA L , KRYZA B ,et al. Storage systems for organizationally distributed environments-PLGrid plus case study[C]// International Conference on Parallel Processing and Applied Mathematics. Heidelberg:Springer, 2014: 724-733.
[2]	XIE X , XIAO N , XU Z W ,et al. CNGrid software 2:service oriented approach to grid computing[C]// The UK e-Science All Hands Meeting.[S.l.:s.n.], 2005: 701-708.
[3]	DEPEI Q , . CNGrid:a test-bed for grid technologies in China[C]// The 10th IEEE International Workshop on Future Trends of Distributed Computing Systems. Piscataway:IEEE Press, 2004: 135-139.
[4]	Cluster File Systems,Inc. Lustre:a scalable,high-performance file system[Z]. 2002.
[5]	WEIL S A , BRANDT S A , MILLER E L ,et al. Ceph:a scalable,high-performance distributed file system[C]// The 7th Symposium on Operating Systems Design and Implementation. New York:ACM Press, 2006: 307-320.
[6]	SCHMUCK F B , HASKIN R L . GPFS:a shared-disk file system for large computing clusters[C]// The Conference on File and Storage Technologies. New York:ACM Press, 2002: 231-244.
[7]	DABEK F , KAASHOEK M F , KARGER D ,et al. Wide-area cooperative storage with CFS[C]// The 18th ACM Symposium on Operating Systems Principles. New York:ACM Press, 2001: 202-215.
[8]	DABEK F , BRUNSKILL E , KAASHOEK M F ,et al. Building peer-to-peer systems with chord,a distributed lookup service[C]// The 8th Workshop on Hot Topics in Operating Systems. Piscataway:IEEE Press, 2001: 81-86.
[9]	GRAFFI K , GROSS C , STINGL D ,et al. Lifesocial.KOM:a secure and p2p-based solution for online social networks[C]// 2011 IEEE Consumer Communications and Networking Conference. Piscataway:IEEE Press, 2011: 554-558.
[10]	CHAN Y W , HO T H , SHIH P C ,et al. Malugo:a peer-to-peer storage system[J]. International Journal of Ad Hoc and Ubiquitous Computing, 2010,5(4): 209.
[11]	TOKA L , DELL’AMICO M , MICHIARDI P . Data transfer scheduling for P2P storage[C]// 2011 IEEE International Conference on Peer-to-Peer Computing. Piscataway:IEEE Press, 2011: 132-141.
[12]	SHEN H Y , LI Z , LI J . A DHT-aided chunkdriven overlay for scalable and efficient peer-to-peer live streaming[J]. IEEE Transactions on Parallel and Distributed Systems, 2012,24(11): 2125-2137.
[13]	KARGER D , LEHMAN E , LEIGHTON T ,et al. Consistent hashing and random trees:distributed caching protocols for relieving hot spots on the World Wide Web[C]// The 29th Annual ACM Symposium on Theory of Computing. New York:ACM Press, 1997: 65-663.
[14]	CALDER B , WANG J , OGUS A ,et al. Windows Azure storage:a highly available cloud storage service with strong consistency[C]// The 23rd ACM Symposium on Operating Systems Principles. New York:ACM Press, 2011: 143-157.
[15]	SIMMS C K , PIKE G G , BALOG D . Wide area filesystem performance using Lustre on the TeraGrid[R]. 2007.
[16]	TATEBE O , HIRAGA K , SODA N . Gfarm grid file system[J]. New Generation Computing, 2010,28: 257-275.
[17]	THOMSON A , ABADI D J . CalvinFS:consistent WAN replication and scalable metadata management for distributed file systems[C]// The 13th USENIX Conference on File and Storage Technologies. New York:ACM Press, 2015: 1-14.
[18]	WRZESZCZ M , TRZEPLA K , SOTA R ,et al. Metadata organization and management for globalization of data access with OneData[C]// International Conference on Parallel Processing and Applied Mathematics. Cham:Springer, 2015: 312-321.
[19]	DUTKA U , SOTA R , WRZESZCZ M ,et al. Uniform and efficient access to data in organizationally distributed environments[M]// eScience on Distributed Computing Infrastructure. Cham: Springer, 2014: 178-194.
[20]	GRIMSHAW A , MORGAN M , KALYANARAMAN A . GFFS-the XSEDE global federated file system[J]. Parallel Processing Letters, 2013,23(2): 1340005.
[21]	胡正丁, 薛巍 . 面向异构众核超级计算机的大规模稀疏计算性能优化研究[J]. 大数据, 2020,6(4): 40-55.
	HU Z D , XUE W . Research on performance optimization for largescale sparse computation over many-core heterogenous supercomputer[J]. Big Data Research, 2020,6(4): 40-55.
[22]	KUNSZT P , BADINO P , FROHNER A ,et al. Data storage,access and catalogs in gLite[C]// 2005 IEEE International Symposium on Mass Storage Systems and Technology. Piscataway:IEEE Press, 2005: 166-170.
[23]	FOSTER I , CZAJKOWSKI K , FERGUSON D E ,et al. Modeling and managing state in distributed systems:The role of OGSI and WSRF[J]. Proceedings of the IEEE, 2005,93(3): 604-612.
[24]	CHERVENAK A , FOSTER I , KESSELMAN C ,et al. The data grid:towards an architecture for the distributed management and analysis of large scientific datasets[J]. Journal of Network and Computer Applications, 2000,23(3): 187-200.
[25]	ALLCOCK W , BRESNAHAN J , BESTER J . GridFTP protocol specification[Z]. 2002.
[26]	FITZGERALD S , FOSTER I , KESSELMAN C ,et al. A directory service for configuring high-performance distributed computations[C]// The 6th IEEE International Symposium on High Performance Distributed Computing. Piscataway:IEEE Press, 1997: 365-375.
[27]	CHEN M , BANGERA G B , HILDEBRAND D ,et al. vNFS:maximizing NFS performance with compounds and vectorized I/O[J]. ACM Transactions on Storage, 2017,13(3): 1-24.
[28]	RYU J , LEE D , SHIN K G ,et al. ClusterFetch:a lightweight prefetcher for intensive disk reads[J]. IEEE Transactions on Computers, 2017,67(2): 284-290.
[29]	WU Z , BUTKIEWICZ M , PERKINS D ,et al. SPANStore:cost-effective georeplicated storage spanning multiple cloud services[C]// The 24th ACM Symposium on Operating Systems Principles. New York:ACM Press, 2013: 292-308.
[30]	SHAO Y , LI C , TANG H . A data replica placement strategy for IoT workflows in collaborative edge and cloud environm ents[J]. Computer Networks, 2019,148: 46-59.
[31]	CHANG W C , WANG P C . Write-aware replica placement for cloud computing[J]. IEEE Journal on Selected Areas in Commun ications, 2019,37(3): 656-667.
[32]	The reference implementation of the linux FUSE (filesystem in userspace) interface[Z]. 2019.
[33]	CHOPADE R , DHAVASE N S . MongoDB,Couchbase:performance comparison for image dataset[C]// 2017 2nd International Conference for Convergence in Technology. Piscataway:IEEE Press, 2017: 255-258.
[34]	A network filesystem client to connect to SSH servers[Z]. 2019.
[35]	SHEPLER S , NOVECK D , EISLER M . Network file system (NFS) version 4 minor version 1 protocol[Z]. 2010.
[36]	Fio-flexible I/O tester rev.3.16[Z]. 2019.

Metrics

Recommended 0

No Suggested Reading articles found!

GVDS: a global virtual data space for wide-area high-performance computing environments

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 36

Related Articles 2

Metrics

Recommended 0

[1]	Jiantong HUO, Limin XIAO, Zhisheng HUO, Yaowen XU. Research and implementation of edge cache system in global virtual data space across WAN [J]. Big Data Research, 2021, 7(5): 57-81.
[2]	Chenhao ZHANG, Limin XIAO, Guangjun QIN, Yao SONG, Shixuan JIANG, Jiye WANG. A wide-area collaborative scheduling system oriented to big data processing applications [J]. Big Data Research, 2021, 7(5): 82-97.