水环境模型与大数据技术融合研究

doi:10.11959/j.issn.2096-0271.2021064

Abstract

Abstract:

Applications of water environment models are greatly limited by complex internal structure of the model and timeconsuming calculations, significant computation burdens arise during the process of parameter calibration, multi-scenario analysis, and decision-making optimization.How to integrate water environment model and big data technology, deeply explore the potential of model application and give full play to its application value is a research hotspot.The bottlenecks faced by the water environment model in the process of practical application were summarized, and the potential of big data technology in solving these problems was analyzed.Based on the existing big data technology, a framework for the integration of water environment model and big data technology was proposed to solve the problem of large-scale calculation, large-scale storage and application analysis of water environment model.The problems faced in the integration of model and big data technologies were described, and specific technical ways of implementation were proposed.A case study for calibration of SWAT model was used to demonstrate feasibility of the proposed framework.Finally, the future research direction of water environment modeling in the context of big data was discussed, and the conclusion was pointed out that the research on surrogate modeling of complex water environment model and on water environment simulation and optimization framework is the future development trend.

Key words: water environment simulation, big data, Hadoop, MapReduce, integration

CLC Number:

TP311

Jinfeng MA, Kaifeng RAO, Ruonan LI, Jing ZHANG, Hua ZHENG. Research on the integration of water environment model and big data technology[J]. Big Data Research, 2021, 7(6): 103-119.

Figures/Tables 9

References 34

[1]	National Academies of Sciences,Engineering, Engineering, Engineering,and Medicine, Division on Earth and Life Studies, Water Science and Technology Board,,et al. Future water priorities for the nation:directions for the U.S.geological survey water mission area[M]. Washington D.C.: National Academies Press, 2018.
[2]	REICHSTEIN M , CAMPS-VALLS G ,, STEVENS B ,et al. Deep learning and process understanding for data-driven earth system science[J]. Nature, 2019,566(7743): 195-204.
[3]	陈永灿, 刘昭伟, 朱德军 . 水动力及水环境模拟方法与应用[M]. 北京: 科学出版社, 2012.
	CHEN Y C , LIU Z W , ZHU D J . Hydrodynamic and water environment simulation methods and applications[M]. Beijing: Science Press, 2012.
[4]	季振刚 . 水动力学和水质——河流、湖泊及河口数值模拟[M].李建平,冯立成,赵万星,译. 北京: 海洋出版社, 2017.
	JI Z G . Hydrodynamics and water quality:modeling rivers,lakes,and estuaries[M]. Translated by LI J P,FENG L C,ZHAO W X. Beijing: China Ocean Press, 2017.
[5]	RAZAVI S , TOLSON B A , MATOTT L S ,et al. Reducing the computational cost of automatic calibration through model preemption[J]. Water Resources Research, 2010,46(11).
[6]	ZHANG X S , SRINIVASAN R , BOSCH D . Calibration and uncertainty analysis of the SWAT model using genetic algorithms and bayesian model averaging[J]. Journal of Hydrology, 2009,374(3-4): 307-317.
[7]	LIN F R , WU N J , TU C H ,et al. Automatic calibration of an unsteady river flow model by using dynamically dimensioned search algorithm[J]. Mathematical Problems in Engineering, 2017(7): 1-19.
[8]	马金锋, 唐力, 饶凯锋 ,等. Hadoop下水环境模拟集群运算模式[J]. 大数据, 2019,5(6): 73-84.
	MA J F , TANG L , RAO K F ,et al. Cluster computing mode for water environment simulation based on Hadoop[J]. Big Data Research, 2019,5(6): 73-84.
[9]	ZHANG D J , CHEN X W , YAO H X ,et al. Moving SWAT model calibration and uncertainty analysis to an enterprise Hadoop-based cloud[J]. Environmental Modelling ＆ Software, 2016,84: 140-148.
[10]	GEORGE L . HBase:the definitive guide(fisrt edition)[M]. Sebastopol: O’Reilly Media, 2011.
[11]	KARAU H , KONWINSKI A , WENDELL P ,et al. Spark快速大数据分析[M].王道远,译. 北京: 人民邮电出版社, 2015.
	KARAU H , KONWINSKI A , WENDELL P ,et al. Learning Spark:lightning-fast data analysis[M]. Translated by WANG D Y. Beijing: Posts ＆ Telecom Press, 2015.
[12]	WHITE T . Hadoop:the definitive guide(second edition)[M]. Sebastopol: O’Reilly Media, 2010.
[13]	吴信东, 嵇圣硙 . MapReduce与Spark用于大数据分析之比较[J]. 软件学报, 2018,29(6): 1770-1791.
	WU X D , JI S W . Comparative study on MapReduce and Spark for big data analytics[J]. Journal of Software, 2018,29(6): 1770-1791.
[14]	SHIVHARE N , DIKSHIT P K S , DWIVEDI S B . A comparison of SWAT model calibration techniques for hydrological modeling in the Ganga river watershed[J]. Engineering, 2018,4(5): 643-652.
[15]	MURTHY A C , VAVILAPALLI V K . Apache Hadoop YARN:moving beyond MapReduce and batch processing with Apache Hadoop 2[M]. Boston: AddisonWesley Professional, 2014.
[16]	YI X , ZOU R , GUO H C . Global sensitivity analysis of a three-dimensional nutrients-algae dynamic model for a large shallow lake[J]. Ecological Modelling, 2016,327: 74-84.
[17]	SONG X M , ZHANG J Y , ZHAN C S ,et al. Global sensitivity analysis in hydrological modeling:review of concepts,methods,theoretical framework,and applications[J]. Journal of Hydrology, 2015,523: 739-757.
[18]	JIANG L , LI Y P , ZHAO X ,et al. Parameter uncertainty and sensitivity analysis of water quality model in Lake Taihu,China[J]. Ecological Modelling, 2018,375: 1-12.
[19]	CAMPOLONGO F , SALTELLI A . Sensitivity analysis of an environmental model:an application of different analysis methods[J]. Reliability Engineering ＆System Safety, 1997,57(1): 49-69.
[20]	刘松, 佘敦先, 张利平 ,等. 基于Morris和Sobol的水文模型参数敏感性分析[J]. 长江流域资源与环境, 2019,28(6): 1296-1303.
	LIU S , SHE D X , ZHANG L P ,et al. Global sensitivity analysis of hydrological model parameters based on Morris and Sobol methods[J]. Resources and Environment in the Yangtze Basin, 2019,28(6): 1296-1303.
[21]	BENNETT N D , CROKE B F W , GUARISO G ,et al. Characterising performance of environmental models[J]. Environmental Modelling ＆ Software, 2013,40: 1-20.
[22]	BAE S , SEO D . Analysis and modeling of algal blooms in the Nakdong River,Korea[J]. Ecological Modelling, 2018,372: 53-63.
[23]	NASH J E , SUTCLIFFE J V . River flow forecasting through conceptual models part I—a discussion of principles[J]. Journal of Hydrology, 1970,10(3): 282-290.
[24]	GHAITH M , LI Z . Propagation of parameter uncertainty in SWAT:a probabilistic forecasting method based on polynomial chaos expansion and machine learning[J]. Journal of Hydrology, 2020,10: 124854
[25]	任婷玉, 梁中耀, 刘永 ,等. 基于贝叶斯优化的三维水动力-水质模型参数估值方法[J]. 环境科学学报, 2019,39(6): 2024-2032.
	REN T Y , LIANG Z Y , LIU Y ,et al. The parameters estimation method based on Bayesian optimization for complex water quality models[J]. Acta Scientiae Circumstantiae, 2019,39(6): 2024-2032.
[26]	RAZAVI S , TOLSON B A , BURN D H . Review of surrogate modeling in water resources[J]. Water Resources Research, 2012,48(7).
[27]	SIMPSON T W , MAUERY T M , KORTE J J ,et al. Kriging models for global approximation in simulation-based multidisciplinary design optimization[J]. AIAA Journal, 2001,39(12): 2233-2241.
[28]	DOWLA F U , ROGERS L L . Solving problems in environmental engineering and geosciences with artificial neural networks[M]. Cambridge: MIT Press, 1995.
[29]	REGIS R G , SHOEMAKER C A . A stochastic radial basis function method for the global optimization of expensive functions[J]. INFORMS Journal on Computing, 2007,19(4): 497-509.
[30]	FEN C S , CHAN C C , CHENG H C . Assessing a response surface-based optimization approach for soil vapor extraction system design[J]. Journal of Water Resources Planning and Management, 2009,135(3): 198-207.
[31]	ZHANG X S , SRINIVASAN R , LIEW M V . Approximating SWAT model using artificial neural network and support vector machine[J]. JAWRA Journal of the American Water Resources Association, 2009,45(2): 460-474.
[32]	ZENG L Z , SHI L S , ZHANG D X ,et al. A sparse grid based Bayesian method for contaminant source identification[J]. Advances in Water Resources, 2012,37: 1-9.
[33]	BOYLE J S , KLEIN S A , LUCAS D D ,et al. The parametric sensitivity of CAM5’s MJO[J]. Journal of Geophysical Research:Atmospheres, 2015,120(4): 1424-1444.
[34]	LIANG Z Y , ZOU R , CHEN X ,et al. Simulate the forecast capacity of a complicated water quality model using the long short-term memory approach[J]. Journal of Hydrology, 2020,581: 124432

Metrics

Recommended 0

No Suggested Reading articles found!

参数名称	最小值	最大值	参数意义
CN2	35	98	径流曲线数
ALPHA_BNK	0	1	河岸基流α因子
SOL_K	0	2 000	饱和导水率
CH_K2	-0.01	500	主通道冲积层中的有效导水率
CANMX	0	100	最大冠层储藏量
SOL_AWC	0	1	土壤有效含水率
SOL_BD	0.9	2.5	土壤密度
CH_N2	0	0.3	主河道曼宁系数

Research on the integration of water environment model and big data technology

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 34

Related Articles 15

Metrics

Recommended 0

[1]	Haihong QIAN, Maoyi WANG, Yun XIONG. Digital transformation in higher education:a systematic review [J]. Big Data Research, 2023, 9(3): 56-70.
[2]	Hong MEI, Xiaoyong DU, Hai JIN, Xueqi CHENG, Yunpeng CHAI, Xuanhua SHI, Xiaolong JIN, Yasha WANG, Chi LIU. Big data technologies forward-looking [J]. Big Data Research, 2023, 9(1): 1-20.
[3]	Yang SHEN, Menglong YU. Metaverse and big data: data insight and value connection in spatio-temporal intelligence [J]. Big Data Research, 2023, 9(1): 103-110.
[4]	Jing CHEN. Humanities big data and its application in the field of digital humanities [J]. Big Data Research, 2022, 8(6): 3-14.
[5]	Yuchu LUO, Hao WU, Yuhan GUO, Shaocong TAN, Can LIU, Ruike JIANG, Xiaoru YUAN. Visualization in digital humanities [J]. Big Data Research, 2022, 8(6): 74-93.
[6]	Tongzheheng ZHENG, Bin LI, Minxuan FENG, Bolin CHANG, Dongbo WANG. Explore the structuration of historical books:the construction and quantitative analysis of digital humanities database of the Biographies of the Shiji [J]. Big Data Research, 2022, 8(6): 40-55.
[7]	Wenlong LI, Yuan YUAN, Xiaopeng AN. Modus operandi of big data governance: some preliminary observations [J]. Big Data Research, 2022, 8(4): 34-45.
[8]	Qifeng TANG, Zhiqing SHAO, Yazhen YE. Authenticating and licensing architecture of data rights in data trade [J]. Big Data Research, 2022, 8(3): 40-53.
[9]	Chenhuizi WANG, Wei CAI. Digital economics in metaverse: state-of-the-art, characteristics, and vision [J]. Big Data Research, 2022, 8(3): 140-150.
[10]	Mei YANG, Wei LI, Siyuan QIAO, Wei LIU. Research on calculation method of China’s big data industry output value [J]. Big Data Research, 2022, 8(3): 151-160.
[11]	Deren LI, Guo ZHANG, Yonghua JIANG, Xin SHEN, Weiling LIU. Opportunities and challenges of geo-spatial information science from the perspective of big data [J]. Big Data Research, 2022, 8(2): 3-14.
[12]	Xiaolan QIU, Yuxin HU, Songtao SHANGGUAN, Kun FU. Remote sensing satellite big data high-recision integration processing technology [J]. Big Data Research, 2022, 8(2): 15-27.
[13]	Weiquan LIU, Cheng WANG, Yu ZANG, Qian HU, Shangshu YU, Baiqi LAI. A survey on information extraction technology based on remote sensing big data [J]. Big Data Research, 2022, 8(2): 28-57.
[14]	Jianqiang LIU, Xiaomin YE, Youguo LAN. Remote sensing big data from Chinese ocean satellites and its application service [J]. Big Data Research, 2022, 8(2): 75-88.
[15]	Hequn YANG, Xiaofeng WANG, Yanqing GAO, Yiwen LU, Bingxin MA, Xinyao WANG. Analysis of satellite big data requirements in numerical weather prediction [J]. Big Data Research, 2022, 8(2): 89-102.