国家高性能计算环境的虚拟数据空间运行支撑技术研究

doi:10.11959/j.issn.2096-0271.2021019

Abstract

Abstract:

The national high performance computing environment (NHPCE) is characterized by wide area dispersion, heterogeneous system and storage isolation autonomy, which posed great challenges to storage management and data sharing.Firstly, the concept of virtual data space system was described.Secondly, how the virtual data space system effectively reduce the cost of cross domain access through viewing-access, data sharing, computing environment docking , was analyzed.Thirdly, the virtual data space system were deeply integrated with the NHPCE by means of modularization and its functions were added to the NHPCE.Finally, the unified virtual and real space user management framework was used to achieve cross domain unified, transparent and secure storage service and the support of large-scale computing application, which is of great significance to the development of NHPCE.

Key words: virtual data space system, national high performance computing environment, data distribution, user view, environment integration

CLC Number:

TP31

Xiaoyu HE, Sungen DENG, Haijing LUAN, Beifang NIU. Study of technique support on the operation of virtual data space in national high performance computing environment[J]. Big Data Research, 2021, 7(2): 158-171.

Figures/Tables 17

References 13

[1]	吴璨, 王小宁, 肖海力 ,等. 基于消息总线的高性能计算环境系统软件优化设计与实现[J]. 高技术通讯, 2020,30(3): 248-258.
	WU C , WANG X N , XIAO H L ,et al. Design and implement of middleware for high performance computing environment based on message bus[J]. Chinese High Technology Letters, 2020,30(3): 248-258.
[2]	王晓东, 赵一宁, 肖海力 ,等. 多节点系统异常日志流量模式检测方法[J]. 软件学报, 2020,31(10): 3295-3308.
	WANG X D , ZHAO Y N , XIAO H L ,et al. Multi-node system abnormal log flow mode detection method[J]. Journal of Software, 2020,31(10): 3295-3308.
[3]	和荣, 王小宁, 肖海力 . 高性能计算资源聚合服务在中国科技云门户的快速集成研究与实现[J]. 科研信息化技术与应用, 2019,10(3): 71-78.
	HE R , WANG X N , XIAO H L . Research and implementation of high performance computing resource aggregation service in CSTCloud portal[J]. e-Science Technology＆ Application, 2019,10(3): 71-78.
[4]	吴燦, 王小宁, 肖海力 ,等. 分布式消息系统研究综述[J]. 计算机科学, 2019,46(z1): 1-5,34.
	WU C , WANG X N , XIAO H L ,et al. Survey on distributed message system[J]. Computer Science, 2019,46(z1): 1-5,34.
[5]	吴燦, 王小宁, 肖海力 ,等. 高性能计算环境中间件的优化设计与实现[J]. 计算机应用研究, 2019,36(1): 163-167.
	WU C , WANG X N , XIA O H L ,et al. Design and implementation of high performance computing environment middleware[J]. Application Research of Computers, 2019,36(1): 163-167.
[6]	迟学斌 . 高性能计算环境与应用[J]. 国防科技工业, 2018(5): 21-22.
	CHI X B . High performance computing enviromment and applications[J]. Defence Science ＆ Technologg Industry, 2018(5): 21-22.
[7]	迟学斌 . 国家高性能计算环境发展报告(2002—2017年)[M]. 北京: 科学出版社, 2018.
	CHI X B . Development report on national high performance computing environment(2002—2017)[M]. Beijing: China Science Publishing ＆Media Ltd., 2018.
[8]	和荣, 王小宁, 卢莎莎 ,等. 高性能计算环境通用计算平台[J]. 计算机系统应用, 2019,28(12): 55-62.
	HE R , WANG X N , LU S S ,et al. Platform for high performance computing environment[J]. Computer Systems ＆Applications, 2019,28(12): 55-62.
[9]	胡正丁, 薛巍 . 面向异构众核超级计算机的大规模稀疏计算性能优化研究[J]. 大数据, 2020,6(4): 40-55.
	HU Z D , XUE W . Research on performance optimization for large-scale sparse computation over many-core heterogenous supercomputer[J]. Big Data Research, 2020,6(4): 40-55.
[10]	BIRD I G . LHC com puting (WLCG):past,present,and future[J]. Grid and Cloud Computing:Concepts and Practical Applications, 2016,192: 1.
[11]	FOATER I , KESSELM AN C . The grid:blueprint for a new computing infrastructure[M]. San Francisco: Morgan Kaufmann, 2004.
[12]	CHOLIA S , SKINNER D , BOVERHOF J . NEWT:a RESTful service for building high performance computing web applications[C]// 2010 Gateway Computing Environments Workshop(GCE). Piscataway:IEEE Press, 2010.
[13]	汤小春, 符莹, 丁朝 ,等. 数据流计算环境下的集群资源管理技术[J]. 大数据, 2020,6(3): 87-100.
	TANG X C , FU Y , DING Z ,et al. State-of-art research of cluster resource management in dataflow computing model[J]. Big Data Research, 2020,6(3): 87-100.

Metrics

Recommended 0

No Suggested Reading articles found!

接口名称	接口描述	接口类型	请求URL	请求类别	请求说明	参数
vhpcs	查询高性能计算环境提供的HPC列表信息	REST	/resources/hpcs	GET	HPC信息查询	无
vapps	查询高性能计算环境提供的应用信息	REST	/resources/applications	GET	计算环境提供的应用信息查询	host：可选；字符串；表示计算环境中HPC的名称
vqueues	查询高性能计算环境中的应用可以使用的计算队列信息	REST	/resources/applications/{appName}	GET	appName表示应用的名字	host：可选；字符串；表示计算环境中HPC的名称
vjobs	查询高性能计算环境中的作业信息，包括所有已经提交的作业	REST	/jobs	GET	作业信息查询	host：字符串；表示计算环境中HPC的名称，被查询的作业在该HPC上计算任务

接口名称	接口描述	接口类型	请求URL	请求类别	请求说明	参数
vlu	查询虚拟数据空间用户信息	REST	/users/search/:name	GET	账户信息查询	账户名
vlg	虚拟数据空间用户组以区域为粒度，即一个区域内的主人和成员为一个用户组，区域主人具有增加/删除成员、修改/查询组的权限的功能	REST	/zone/share	POST	区域共享	区域名、账户名
		REST	/zone/sharecancel	POST	区域共享取消	区域名、账户名
		REST	/auth/modify	POST	区域权限修改	账户ID
		REST	/auth/search	POST	区域权限查询	账户ID
vmkuser	创建虚拟数据空间用户	REST	/users/registration	POST	账户注册	账户注册信息
vmoduser	修改虚拟数据空间用户信息	REST	/users/modify	POST	账户信息修改	账户名
vrsync	同步本地的数据与虚拟空间的数据，由于虚拟数据空间支持POSIX接口操作，故直接调用linux sync命令即可	-	-	-	同linux sync命令	同linux sync命令

[1]	Hongrun REN, Yangyong ZHU. Data pipeline model: exploration of the overthe-counter form of streaming data [J]. Big Data Research, 2023, 9(3): 15-28.
[2]	Weijun GAO, Kai WANG. Research on public welfare donation traceability system based on consortium blockchains [J]. Big Data Research, 2023, 9(3): 150-167.
[3]	Chaoran LUO, Yun MA, Xiang JING, Gang HUANG. Internet of data: a solution for dataspace infrastructure and its technical challenges [J]. Big Data Research, 2023, 9(2): 110-121.
[4]	Xiangquan GUI, Zhili GUO, Yi YANG, Bingfeng QIN. Tourism points exchange system design based on blockchain technology [J]. Big Data Research, 2023, 9(2): 147-162.
[5]	Yazhen YE, Yangyong ZHU. Data-Commerce-Ecosystem: data goods, data businessman and data commerce [J]. Big Data Research, 2023, 9(1): 111-125.
[6]	Lingli ZHANG, Qikai CHU, Guijuan WANG, Weihan ZHANG, Hui PU, Zhenjin SONG, Yadong WU. Text sentiment visual analysis technology and its application in humanities [J]. Big Data Research, 2022, 8(6): 56-73.
[7]	Aili LI, Zishuai ZHANG, Yin LIN, Qiuju WANG, Jianan YANG, Weicheng MENG, Yanfeng ZHANG. Research on emotion monitoring of public based on social network big data [J]. Big Data Research, 2022, 8(6): 105-126.
[8]	Zhengxun XIA, Jianfei TANG, Shengmei LUO, Yan ZHANG. Exploration and practice of trusted AI governance framework [J]. Big Data Research, 2022, 8(4): 145-164.
[9]	Yazhen YE, Yangyong ZHU. BoxedData: a data product form based on databox [J]. Big Data Research, 2022, 8(3): 15-25.
[10]	Qifeng TANG, Zhiqing SHAO, Yazhen YE. Authenticating and licensing architecture of data rights in data trade [J]. Big Data Research, 2022, 8(3): 40-53.
[11]	Xiaolong ZHANG, Long ZHI, Jian GAO, Zhongchen MIAO, Yuefeng LIN, Yali XIANG, Yun XIONG. A semi-supervised learning financial news classification algorithm [J]. Big Data Research, 2022, 8(2): 134-144.
[12]	Feng ZHI, Feng TIAN, Ruofan ZHAO. Classification of big data in metrology [J]. Big Data Research, 2022, 8(1): 60-72.
[13]	Yongfeng WANG, Zhiguang CHEN. A survey of persistent index data structures on non-volatile memory [J]. Big Data Research, 2021, 7(6): 78-88.
[14]	Jinfeng MA, Kaifeng RAO, Ruonan LI, Jing ZHANG, Hua ZHENG. Research on the integration of water environment model and big data technology [J]. Big Data Research, 2021, 7(6): 103-119.
[15]	Yaodong CHENG, Yaosong CHENG, Yujiang BI, Yu GAO, Haibo LI, Lu WANG, Qiuling YAO. Data processing system for HEP based on domestic processor architecture [J]. Big Data Research, 2021, 7(5): 17-30.

Study of technique support on the operation of virtual data space in national high performance computing environment

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 13

Related Articles 15

Metrics

Recommended 0