面向算力网络的跨域数据管理方法

doi:10.11959/j.issn.2096-0271.2023068

Abstract

Abstract:

Cross-domain computing power networks wish to integrate computational and data resources from multiple computing centers, but existing methods do not pay enough attention to cross-domain file and data management.In this paper, a lightweight data access scheme for cross-domain computing power networks was proposed: (1) accessing parallel file system storage resources of remote computing centers through file system protocol conversion; (2) local caching as a supplement to cope with high IOPS applications; and (3) mounting remote or local storage to specified directories through container binding technology.The prototype system based on this scheme had been deployed on highperformance computing centers in multiple universities.The measured data and user experience showed that the scheme in this paper could meet the requirements of common high-performance computing applications.

Key words: computing power network, parallel file system, data management, heterogeneous storage resource

CLC Number:

TP316

Weizheng LU, Qizhi DAI, Ce ZHANG. Cross-domain data management for computing power networks[J]. Big Data Research, 2024, 10(2): 94-108.

Figures/Tables 10

References 22

[1]	FOSTER I T , KESSELMAN C . The grid:blueprint for a new computing infrastructure[M]. San Francisco: Morgan Kaufman Publishers, 1998.
[2]	XU Z W , CHI X B , XIAO N . Highperformance computing environment:a review of twenty years of experiments in China[J]. National Science Review, 2016,3(1): 36-48.
[3]	钱德沛, 栾钟治, 刘轶 . 从网格到“东数西算”：构建国家算力基础设施[J]. 北京航空航天大学学报, 2022,48(9): 1561-1574.
	QIAN D P , LUAN Z Z , LIU Y . From grid to “East-west Computing Transfer”:constructing national computing infrastructure[J]. Journal of Beijing University of Aeronautics and Astronautics, 2022,48(9): 1561-1574.
[4]	秦广军, 肖利民, 张广艳 ,等. 面向国家高性能计算环境的虚拟数据空间系统[J]. 大数据, 2021,7(2): 101-122.
	QIN G J , XIAO L M , ZHANG G Y ,et al. Virtual data space system for national high-performance computing environment[J]. Big Data Research, 2021,7(2): 101-122.
[5]	何小雨, 邓笋根, 栾海晶 ,等. 国家高性能计算环境的虚拟数据空间运行支撑技术研究[J]. 大数据, 2021,7(2): 158-171.
	HE X Y , DENG S G , LUAN H J ,et al. Study of technique support on the operation of virtual data space in national highperformance computing environment[J]. Big Data Research, 2021,7(2): 158-171.
[6]	肖利民, 宋尧, 秦广军 ,等. GVDS：面向广域高性能计算环境的虚拟数据空间[J]. 大数据, 2021,7(2): 123-146.
	XIAO L M , SONG Y , QING J ,et al. GVDS:a global virtual data space for wide-area high-performance computing environments[J]. Big Data Research, 2021,7(2): 123-146.
[7]	YOO A B , JETTE M A , GRONDONA M . SLURM:simple linux utility for resource management[C]// Proceedings of Job Scheduling Strategies for Parallel Processing. Seattle:Springer, 2003: 44-60.
[8]	YIN F , SHI F . A comparative survey of big data computing and hpc:from a parallel programming model to a cluster architecture[J]. International Journal of Parallel Programming, 2022,50(1): 27-64.
[9]	BURROWS M . Lustre:building a file system for 1000-node clusters[C]// Proceedings of the 2003 Linux Symposium.[S.l.:s.n.], 2003: 380-386.
[10]	SCHMUCK F B , HASKIN R L . GPFS:a shared-disk file system for largecomputing clusters[C]// The FAST 2002 Conferenceon File and Storage Technologies. Monterey:USENIX, 2002: 231-244.
[11]	CATLETT C , ALLCOCK W E , ANDREWS P ,et al. TeraGrid:analysis of organization,system architecture,and middleware enabling new types of applications[C]// Proceedings of the 2006 International Advanced Research Workshop on High Performance Computing and Grids. Amsterdam:IOS Press, 2006: 225-249.
[12]	TOWN J , BOISSEAU J , ROSKIES J ,et al. XSEDE:extreme science and engineering discovery environment (OAC 15-48562)[R]. 2020.
[13]	NEWHOUSE S . Seeking new horizons:EGI’s role in 2020 (EGI-1098-D230-V3)[R]. 2021.
[14]	HENSCHEL R , SIMMS S , HANCOCK D ,et al. Demonstrating lustre over a 100Gbps wide area network of 3500km[C]// Proceedings ofthe International Conference on High Performance Computing,Networking,Storage and Analysis. Piscataway:IEEE Press, 2012: 1-8.
[15]	THOMSON A , ABADI D J . CalvinFS:consistent WAN replication and scalable metadata management for distributed file systems[C]// Proceedings of the 13th USENIX Conference on File and Storage Technologies. Berkeley:USENIX, 2015: 1-14.
[16]	CORBETT J C , DEAN J , EPSTEIN M ,et al. Spanner:Google’s globally-distributed database[J]. ACM Transactions on Computer Systems, 2013,31(3): 8.
[17]	CAO W , LIU Z J , WANG P ,et al. PolarFS:an ultra-low latency and failure resilient distributed file system for shared storage cloud database[J]. Proceedings of the VLDB Endowment, 2018,11(12): 1849-1862.
[18]	A network filesystem client to connect to SSH servers[Z]. 2023.
[19]	DUTKA U , SOTA R , WRZESZCZ M ,et al. Uniform and efficient access to data in organizationally distributed environments[C]// Proceeding of eScience on Distributed Computing Infrastructure. Cham:Springer, 2014: 178-194.
[20]	DONENFELD J A . Wireguard:next generation kernel network tunnel[C]// Proceeding of 24th Annual Network and Distributed System Security Symposium. San Diego:The Internet Society, 2017: 1-12.
[21]	DONGARRA J , LUSZCZEK P , PETITET A . The LINPACK benchmark:past,present,and future[J]. Concurrency and Computation:Practice and Experience, 2003,15(9): 803-820.
[22]	Fio - flexible I/O tester rev.3.25[Z]. 2020.

Metrics

Recommended 0

No Suggested Reading articles found!

方案	EaaS	CNGrid	分散管理
跨域文件操作	cp、mv、vim等文件操作命令	sceput2、sceget2等专用命令	scp、rsync等传输命令
数据管理	文件只需创建一次，并在多算力中心共享	文件需上传到多算力中心	文件需上传到多算力中心
软件管理	常用软件数据安装一次，多算力中心共享	每个算力中心安装自己的软件	每个算力中心安装自己的软件
算力调度	统一的调度软件	统一的调度软件	每个算力中心使用自己的调度软件

对比项	HPL/（GFLOPS）	随机读IOPS	顺序读带宽/（Gbit/s）	顺序写带/（Gbit/s）
裸金属物理机	4 980	184.5	14.53	9.43
容器	4 910	183	14.16	9.53
百分比	98.6%	99.2%	97.5%	101.1%

Cross-domain data management for computing power networks

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 22

Related Articles 15

Metrics

Recommended 0

[1]	Weiming LI, Tong LI, Dafang ZHANG, Longchao DAI, Yunpeng CHAI. Distributed consensus algorithms for crossdomain data management: state-of-the-art, challenges and perspectives [J]. Big Data Research, 2023, 9(4): 3-15.
[2]	Jilin ZHANG, Xiaowei GU, Yizhao ZHANG, Xiaolin ZHENG, Chaochao CHEN. Research and application of cross-domain data authorization and operation [J]. Big Data Research, 2023, 9(4): 83-97.
[3]	Hong MEI, Xiaoyong DU, Hai JIN, Xueqi CHENG, Yunpeng CHAI, Xuanhua SHI, Xiaolong JIN, Yasha WANG, Chi LIU. Big data technologies forward-looking [J]. Big Data Research, 2023, 9(1): 1-20.
[4]	Rui ZHU, Hongzhi WANG, Shuangshuang CUI, Kaixin ZHANG, Yu YAN. Cloud-edge-end collaborative big data management for metaverse [J]. Big Data Research, 2023, 9(1): 63-77.
[5]	Feng ZHI, Feng TIAN, Ruofan ZHAO. Classification of big data in metrology [J]. Big Data Research, 2022, 8(1): 60-72.
[6]	Qi XU, Ziming ZOU, Yaqin YUAN, Xiaoyan HU, Jizhou TONG, Wenzhen MA. Data management process model for the science and technology programs [J]. Big Data Research, 2022, 8(1): 15-23.
[7]	Hao HU, Fazhi QI, Xiaokang SUN, Qi LUO. HEPS scientific data management policy research and applications [J]. Big Data Research, 2022, 8(1): 5-14.
[8]	Limin XIAO, Yao SONG, Guangjun QIN, Hanjie ZHOU, Chaobo WANG, Bing WEI, Wei WEI, Zhisheng HUO. GVDS: a global virtual data space for wide-area high-performance computing environments [J]. Big Data Research, 2021, 7(2): 123-146.
[9]	Hong DAI, Qun ZHANG, Haolin LU, Junzhi BIN. Comparative analysis between bank industry data governance guidelines and DCMM [J]. Big Data Research, 2020, 6(5): 118-128.
[10]	Minghe YU, Tiezheng NIE, Guoliang LI. Data curation technologies and applications [J]. Big Data Research, 2019, 5(6): 30-46.
[11]	Hong DAI, Qun ZHANG, Zhuo YIN. Study on big data governance standard system [J]. Big Data Research, 2019, 5(3): 47-54.
[12]	Xiaoyong DU, Yueguo CHEN, Ju FAN, Wei LU. Data wrangling:a key technique of data governance [J]. Big Data Research, 2019, 5(3): 13-22.
[13]	Pingwen ZHANG, Weinan E, Xiaoru YUAN, Yiming FU. Big data analysis and application technology innovation platform [J]. Big Data Research, 2018, 4(4): 86-93.
[14]	Youhuan LI, Lei ZOU. Graph stream:model,algorithm and system [J]. Big Data Research, 2018, 4(4): 44-55.
[15]	Zhihui WANG, Xuchen ZHOU, Yun ZHU. Privacy preservation in self-governing openness of data [J]. Big Data Research, 2018, 4(2): 42-49.