面向大数据处理应用的广域存算协同调度系统

doi:10.11959/j.issn.2096-0271.2021050

Abstract

Abstract:

Based on the high-performance computing global virtual data space system, a wide-area collaborative scheduling system for big data processing applications was designed and implemented.This system can address the issue of how big data processing applications unified use wide-area storage and computing resources.And it can collaborative schedule of application data and computing tasks based on the computing characteristics of the application and data layout through collaborative scheduling, load balancing scheduling, data locality scheduling strategies.By unified scheduling of application data and computing tasks in the wide-area environment, it can coordinate the utilization of wide-area computing and storage resources, and effectively improve the running performance of big data processing applications.The actual test results in the national high-performance computing environment show that the scheduling method proposed can support big data processing applications effectively, and the running efficiency of typical applications such as wide-area target collaborative recognition and molecular docking can be increased by 3~4 times.

Key words: wide-area collaborative scheduling, big data processing application, global virtual data space, high-performance computing environment

CLC Number:

TP316

Chenhao ZHANG, Limin XIAO, Guangjun QIN, Yao SONG, Shixuan JIANG, Jiye WANG. A wide-area collaborative scheduling system oriented to big data processing applications[J]. Big Data Research, 2021, 7(5): 82-97.

Figures/Tables 12

References 22

[1]	佩瑟鲁·拉吉阿诺帕马·拉曼德维亚·纳加拉杰 ,等. 高性能计算系统与大数据分析[M]. 齐宁,庞建民,张铮,等,译.北京: 机械工业出版社, 2019: 17-20.
	RAJ P , RAMAN A , NAGARAJ D ,et al. High-performance big-data analytics computing systems and approaches[M]. Translated by QI N,PANG J M,ZHANG Z,et al. Beijing: China Machine Press, 2019: 17-20.
[2]	彭宇, 庞景月, 刘大同 ,等. 大数据:内涵、技术体系与展望[J]. 电子测量与仪器学报, 2015,29(4): 469-482.
	PENG Y , PANG J Y , LIU D T ,et al. Big data:connotation,technical framework and its development[J]. Journal of Electronic Measurement and Instrumentation, 2015,29(4): 469-482.
[3]	陈国良, 毛睿, 蔡晔 . 高性能计算及其相关新兴技术[J]. 深圳大学学报(理工版), 2015,32(1): 25-31.
	CHEN G L , MAO R , CAI Y . High performance computing and related new technologies[J]. Journal of Shenzhen University Science and Engineering, 2015,32(1): 25-31.
[4]	TOWNS J , GAITHER K , BLOOD P ,et al. XSEDE:extreme science and engineering discovery environment(OAC 15-48562)[R]. 2020.
[5]	NEWHOUSE S . Seeking new horizons:EGI’s role in 2020(EGI-1098-D230-V3)[R]. 2021.
[6]	VILJOEN M , DUTKA ? ,, KRYZA B ,et al. Towards European open science commons:the EGI open data platform and the EGI DataHub[J]. Procedia Computer Science, 2016,97: 148-152.
[7]	WRZESZCZ M , TRZEPLA K , S?OTA R , ,et al. Metadata organization and management for globalization of data access with onedata[C]// Parallel Processing and Applied Mathematics. Cham:Springer, 2016: 312-321.
[8]	历军 . 高性能计算应用概览[M]. 北京: 清华大学出版社, 2018: 304-307.
	LI J . Overview of high-performance computing applications[M]. Beijing: Tsinghua University Press, 2018: 304-307.
[9]	XU Z W , CHI X B , XIAO N . Highperformance computing environment:a review of twenty years of experiments in China[J]. National Science Review, 2016,3(1): 36-48.
[10]	秦广军, 肖利民, 张广艳 ,等. 面向国家高性能计算环境的虚拟数据空间系统[J]. 大数据, 2021,7(2): 101-122.
	QIN G J , XIAO L M , ZHANG G Y ,et al. Virtual data space system for national high-performance computing environment[J]. Big Data Research, 2021,7(2): 101-122.
[11]	肖利民, 宋尧, 秦广军 ,等. GVDS:面向广域高性能计算环境的虚拟数据空间[J]. 大数据, 2021,7(2): 123-146.
	XIAO L M , SONG Y , QIN G J ,et al. GVDS:a global virtual data space for wide-area high-performance computing environments[J]. Big Data Research, 2021,7(2): 123-146.
[12]	HEY T , TREFETHEN A . The data deluge:an e-science perspective[M]// Grid computing:making the global infrastructure a reality. Chichester: John Wiley ＆ Sons,Ltd, 2003: 809-824.
[13]	FREY J , TANNENBAUM T , LIVNY M ,et al. Condor-G:a computation management agent for multi-institutional grids[C]// Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing. Piscataway:IEEE, 2001: 55-63.
[14]	KOSAR T , BALMAN M . A new paradigm:data-aware scheduling in grid computing[J]. Future Generation Computer Systems, 2009,25(4): 406-413.
[15]	ZHAO L P , YANG Y N , MUNIR A ,et al. Optimizing geo-distributed data analytics with coordinated task scheduling and routing[J]. IEEE Transactions on Parallel and Distributed Systems, 2020,31(2): 279-293.
[16]	WANG K , QIAO K , SADOOGHI I ,et al. Load-balanced and locality-aware scheduling for data-intensive workloads at extreme scales[J]. Concurrency and Computation:Practice and Experience, 2016,28(1): 70-94.
[17]	LI C L , BAI J P , TANG J H . Joint optimization of data placement and scheduling for improving user experience in edge computing[J]. Journal of Parallel and Distributed Computing, 2019,125: 93-105.
[18]	HE L , QIAN Z C . Intent-based resource matching strategy in cloud[J]. Information Sciences, 2020,538: 1-18.
[19]	BRYK P , MALAWSKI M , JUVE G ,et al. Storage-aware algorithms for scheduling of workflow ensembles in clouds[J]. Journal of Grid Computing, 2016,14(2): 359-378.
[20]	HU M L , LUO J , WANG Y ,et al. Adaptive scheduling of task graphs with dynamic resilience[J]. IEEE Transactions on Computers, 2017,66(1): 17-23.
[21]	尹伶艳 . 广域云环境下数据与计算的协同调度[D]. 天津:天津大学, 2014.
	YIN L Y . Joint scheduling of data and computation in geo-distributed cloud systems[D]. Tianjin:Tianjin University, 2014.
[22]	SAIKIA S , BORDOLOI M . Molecular docking:challenges,advances and its use in drug discovery perspective[J]. Current Drug Targets, 2019,20(5): 501-521.

Metrics

Recommended 0

No Suggested Reading articles found!

节点	CPU核数/个	存储容量/TB
中科院网络信息中心	112	715.75
上海超级计算中心	136	1.45
国家超算广州中心	24	1 208.121
国家超算济南中心	56	266.44
国家超算长沙中心	32	87.4

A wide-area collaborative scheduling system oriented to big data processing applications

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 22

Related Articles 3

Metrics

Recommended 0

[1]	Jiantong HUO, Limin XIAO, Zhisheng HUO, Yaowen XU. Research and implementation of edge cache system in global virtual data space across WAN [J]. Big Data Research, 2021, 7(5): 57-81.
[2]	Guangjun QIN, Limin XIAO, Guangyan ZHANG, Beifang NIU, Zhiguang CHEN. Virtual data space system for national highperformance computing environment [J]. Big Data Research, 2021, 7(2): 101-122.
[3]	Limin XIAO, Yao SONG, Guangjun QIN, Hanjie ZHOU, Chaobo WANG, Bing WEI, Wei WEI, Zhisheng HUO. GVDS: a global virtual data space for wide-area high-performance computing environments [J]. Big Data Research, 2021, 7(2): 123-146.