Hadoop下水环境模拟集群运算模式

doi:10.11959/j.issn.2096-0271.2019051

Abstract

Abstract:

Water environment numerical models are effective tools for the simulation,analysis and prediction of the processes of pollutant transport and transformation in water.The development of high-performance batch computation of water environment models has long been a hot topic.The distributed cluster computing mode based on big data technology is a promising approach for massive data management and batch computation,which provides a viable solution to large-scale water environment simulations.The adaptability of water environmental models under the framework of big data technology was explored,and a distributed cluster computing mode for water environment simulations was proposed.Moreover,the feasibility of adapting Delft3D model for cluster computing under Hadoop MapReduce environment was verified with real examples.

Key words: water environment simulation, cluster calculation, big data, Hadoop

CLC Number:

TP311

Jinfeng MA, Li TANG, Kaifeng RAO, Gang HONG, Mei MA. Cluster computing mode for water environment simulation based on Hadoop[J]. Big Data Research, 2019, 5(6): 73-84.

Figures/Tables 9

References 17

[1]	陈永灿, 刘昭伟, 朱德军 . 水动力及水环境模拟方法与应用[M]. 北京: 科学出版社, 2012.
	CHEN Y C , LIU Z W , ZHU D J . Hydrodynamic and water environment simulation methods and applications[M]. Beijing: Science PressPress, 2012.
[2]	彭泽洲, 杨天行, 梁秀娟 . 水环境数学模型及其应用[M]. 北京: 化学工业出版社, 2007: 98-103.
	PENG Z Z , YANG T X , LIANG X J . Mathematical model of water environment and its application[M]. Beijing: Chemical Industry PressPress, 2007.
[3]	NGUYEN T T , KEUPERS I , WILLEMS P . Conceptual river water quality model with flexible model structure[J]. Environmental Modelling ＆ Software, 2018,104: 102-117.
[4]	MELCHING C S . Application of a water quality model for determining instream aeration station location and operational rules:a case study[J]. Water Science and Engineering, 2018,11(1): 8-16.
[5]	KEUPERS I , WILLEMS P . Development and testing of a fast conceptual river water quality model[J]. Water Research, 2017,113: 62-71.
[6]	REDER K , ALCAMO J , FLORKE M . A sensitivity and uncertainty analysis of a continental-scale water quality model of pathogen pollution in African rivers[J]. Ecological Modelling, 2017,351: 129-139.
[7]	贾仰文, 王浩, 严登华 . 黑河流域水循环系统的分布式模拟(Ⅰ)-模型开发与验证[J]. 水利学报, 2006(5): 534-542.
	JIA Y W , WANG H , YAN D H . Distributed model of hydrological cycle system in Heihe River basin Ⅰ-model development and verification[J]. Journal of Hydraulic Engineering, 2006(5): 534-542.
[8]	王永桂 . 流域大尺度水环境模型的高效能集群计算方法研究及其在三峡库区的应用[D]. 武汉:武汉大学, 2015.
	WANG Y G . Research on high productivity cluster computing technology of large scale water environment model and the application in the Three Gorges Area[D]. Wuhan:Wuhan University, 2015.
[9]	WANG H , FU X D , WANG G Q ,et al. A common parallel computing framework for modeling hydrological processes of river basins[J]. Parallel Computing, 2011,37(6): 302-315.
[10]	左一鸣, 崔广柏 . 二维水动力模型的并行计算研究[J]. 水科学进展, 2008,19(6): 846-850.
	ZUO Y M , CUI G B . Parallel computation for 2D flow model[J]. Advances in Water Science, 2008,19(6): 846-850.
[11]	许栋 PAyet D 及春宁 .等. 浅水方程大规模并行计算模拟城市洪水演进[J]. 天津大学学报(自然科学与工程技术版), 2016,49(4): 341-348.
	XU D , PAYET D , JI C N ,et al. Simulation of flood propagation in cities using large scale parallel computation of shallow water equations[J]. Journal of Tianjin University, 2016,49(4): 341-348.
[12]	靖争 . 湖泊水环境数值模拟及非静压水流并行计算方法研究[D]. 武汉:华中科技大学, 2017.
	JING Z . Numerical simulations of the lake water environment and parallel computations of the non-hydrostatic flows[D]. Wuhan:Huazhong University of Science and Technology, 2017.
[13]	赵旭东, 梁书秀, 孙昭晨 ,等. 基于GPU并行算法的水动力数学模型建立及其效率分析[J]. 大连理工大学学报, 2014,54(2): 204-209.
	ZHAO X D , LIANG S X , SUN Z C ,et al. Foundation and analysis of computational efficiency for hydrodynamic model based on GPU parallel algorithm[J]. Journal of Dalian University of Technology, 2014,54(2): 204-209.
[14]	宋利祥, 范光伟, 张文明 ,等. 洪水演进高速模拟系统设计与实现[J]. 水电能源科学, 2017,35(12): 33-36.
	SONG L X , FAN G W , ZHANG W M ,et al. Design and implementation of high speed simulation for flood routing[J]. Water Resources and Power, 2017,35(12): 33-36.
[15]	ZHANG D , CHEN X , YAO H ,et al. Moving SWAT model calibration and uncertainty analysis to an enterprise Hadoop-based cloud[J]. Environmental Modelling ＆Software, 2016,84(C): 140-148
[16]	WHITE T . Hadoop:the definitive guide[M]. Sebastopol: O’Reilly Media,Inc.Press, 2010.
[17]	MURTHY A , VAVILAPALLI V , EADLINE D ,et al. Apache Hadoop YARN[M]. Boston: Addison-WesleyPress, 2014.

Metrics

Recommended 0

No Suggested Reading articles found!

机器名	硬件配置	集群中角色
Master	HP ProLiant DL580 g7，32核， 2.67 GHz，内存64 GB，硬盘容量900 GB	NameNode、JobTracker
Master2	HP ProLiant DL580 g7，32核， 2.67 GHz，内存64 GB，硬盘容量900 GB	Secondary NameNode
s200～s204	ThinkCentre M8411t，8核，3.4 GHz，内存32 GB，硬盘容量2 TB	DataNode、TaskTracker

Cluster computing mode for water environment simulation based on Hadoop

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 17

Related Articles 15

Metrics

Recommended 0

[1]	Haihong QIAN, Maoyi WANG, Yun XIONG. Digital transformation in higher education:a systematic review [J]. Big Data Research, 2023, 9(3): 56-70.
[2]	Hong MEI, Xiaoyong DU, Hai JIN, Xueqi CHENG, Yunpeng CHAI, Xuanhua SHI, Xiaolong JIN, Yasha WANG, Chi LIU. Big data technologies forward-looking [J]. Big Data Research, 2023, 9(1): 1-20.
[3]	Yang SHEN, Menglong YU. Metaverse and big data: data insight and value connection in spatio-temporal intelligence [J]. Big Data Research, 2023, 9(1): 103-110.
[4]	Tongzheheng ZHENG, Bin LI, Minxuan FENG, Bolin CHANG, Dongbo WANG. Explore the structuration of historical books:the construction and quantitative analysis of digital humanities database of the Biographies of the Shiji [J]. Big Data Research, 2022, 8(6): 40-55.
[5]	Jing CHEN. Humanities big data and its application in the field of digital humanities [J]. Big Data Research, 2022, 8(6): 3-14.
[6]	Yuchu LUO, Hao WU, Yuhan GUO, Shaocong TAN, Can LIU, Ruike JIANG, Xiaoru YUAN. Visualization in digital humanities [J]. Big Data Research, 2022, 8(6): 74-93.
[7]	Wenlong LI, Yuan YUAN, Xiaopeng AN. Modus operandi of big data governance: some preliminary observations [J]. Big Data Research, 2022, 8(4): 34-45.
[8]	Qifeng TANG, Zhiqing SHAO, Yazhen YE. Authenticating and licensing architecture of data rights in data trade [J]. Big Data Research, 2022, 8(3): 40-53.
[9]	Chenhuizi WANG, Wei CAI. Digital economics in metaverse: state-of-the-art, characteristics, and vision [J]. Big Data Research, 2022, 8(3): 140-150.
[10]	Mei YANG, Wei LI, Siyuan QIAO, Wei LIU. Research on calculation method of China’s big data industry output value [J]. Big Data Research, 2022, 8(3): 151-160.
[11]	Deren LI, Guo ZHANG, Yonghua JIANG, Xin SHEN, Weiling LIU. Opportunities and challenges of geo-spatial information science from the perspective of big data [J]. Big Data Research, 2022, 8(2): 3-14.
[12]	Xiaolan QIU, Yuxin HU, Songtao SHANGGUAN, Kun FU. Remote sensing satellite big data high-recision integration processing technology [J]. Big Data Research, 2022, 8(2): 15-27.
[13]	Weiquan LIU, Cheng WANG, Yu ZANG, Qian HU, Shangshu YU, Baiqi LAI. A survey on information extraction technology based on remote sensing big data [J]. Big Data Research, 2022, 8(2): 28-57.
[14]	Jianqiang LIU, Xiaomin YE, Youguo LAN. Remote sensing big data from Chinese ocean satellites and its application service [J]. Big Data Research, 2022, 8(2): 75-88.
[15]	Hequn YANG, Xiaofeng WANG, Yanqing GAO, Yiwen LU, Bingxin MA, Xinyao WANG. Analysis of satellite big data requirements in numerical weather prediction [J]. Big Data Research, 2022, 8(2): 89-102.