水环境模型与大数据技术融合研究

doi:10.11959/j.issn.2096-0271.2021064

摘要/Abstract

摘要：

水环境模型内部结构复杂且计算耗时，造成参数率定、多情景分析及决策优化过程中面临高负荷计算难题，这极大地限制了其应用价值的发挥。如何融合水环境模型和大数据技术，深入挖掘模型应用潜力和充分发挥其应用价值是一个研究热点。总结了水环境模型在实际应用过程中面临的瓶颈，分析了大数据技术在解决这些问题上具有的潜力。基于现有成熟的大数据技术，提出了水环境模型与大数据技术融合框架，解决了水环境模型规模计算、规模存储和应用分析问题。阐述了模型与大数据技术融合过程中面临的问题，提出了具体的实现技术思路。通过SWAT模型率定应用案例，证明融合框架的可行性。最后探讨了大数据背景下水环境模型的未来研究方向，指出开展复杂水环境模型的代理模型研究和水环境模拟优化框架研究是未来的发展趋势。

关键词: 水环境模拟, 大数据, Hadoop, MapReduce, 融合

Abstract:

Applications of water environment models are greatly limited by complex internal structure of the model and timeconsuming calculations, significant computation burdens arise during the process of parameter calibration, multi-scenario analysis, and decision-making optimization.How to integrate water environment model and big data technology, deeply explore the potential of model application and give full play to its application value is a research hotspot.The bottlenecks faced by the water environment model in the process of practical application were summarized, and the potential of big data technology in solving these problems was analyzed.Based on the existing big data technology, a framework for the integration of water environment model and big data technology was proposed to solve the problem of large-scale calculation, large-scale storage and application analysis of water environment model.The problems faced in the integration of model and big data technologies were described, and specific technical ways of implementation were proposed.A case study for calibration of SWAT model was used to demonstrate feasibility of the proposed framework.Finally, the future research direction of water environment modeling in the context of big data was discussed, and the conclusion was pointed out that the research on surrogate modeling of complex water environment model and on water environment simulation and optimization framework is the future development trend.

Key words: water environment simulation, big data, Hadoop, MapReduce, integration

中图分类号:

TP311

马金锋, 饶凯锋, 李若男, 张京, 郑华. 水环境模型与大数据技术融合研究[J]. 大数据, 2021, 7(6): 103-119.

Jinfeng MA, Kaifeng RAO, Ruonan LI, Jing ZHANG, Hua ZHENG. Research on the integration of water environment model and big data technology[J]. Big Data Research, 2021, 7(6): 103-119.

图/表 9

图1

图2

图3

图4

图5

图6

表1

图7

图8

参考文献 34

[1]	National Academies of Sciences,Engineering, Engineering, Engineering,and Medicine, Division on Earth and Life Studies, Water Science and Technology Board,,et al. Future water priorities for the nation:directions for the U.S.geological survey water mission area[M]. Washington D.C.: National Academies Press, 2018.
[2]	REICHSTEIN M , CAMPS-VALLS G ,, STEVENS B ,et al. Deep learning and process understanding for data-driven earth system science[J]. Nature, 2019,566(7743): 195-204.
[3]	陈永灿, 刘昭伟, 朱德军 . 水动力及水环境模拟方法与应用[M]. 北京: 科学出版社, 2012.
	CHEN Y C , LIU Z W , ZHU D J . Hydrodynamic and water environment simulation methods and applications[M]. Beijing: Science Press, 2012.
[4]	季振刚 . 水动力学和水质——河流、湖泊及河口数值模拟[M].李建平,冯立成,赵万星,译. 北京: 海洋出版社, 2017.
	JI Z G . Hydrodynamics and water quality:modeling rivers,lakes,and estuaries[M]. Translated by LI J P,FENG L C,ZHAO W X. Beijing: China Ocean Press, 2017.
[5]	RAZAVI S , TOLSON B A , MATOTT L S ,et al. Reducing the computational cost of automatic calibration through model preemption[J]. Water Resources Research, 2010,46(11).
[6]	ZHANG X S , SRINIVASAN R , BOSCH D . Calibration and uncertainty analysis of the SWAT model using genetic algorithms and bayesian model averaging[J]. Journal of Hydrology, 2009,374(3-4): 307-317.
[7]	LIN F R , WU N J , TU C H ,et al. Automatic calibration of an unsteady river flow model by using dynamically dimensioned search algorithm[J]. Mathematical Problems in Engineering, 2017(7): 1-19.
[8]	马金锋, 唐力, 饶凯锋 ,等. Hadoop下水环境模拟集群运算模式[J]. 大数据, 2019,5(6): 73-84.
	MA J F , TANG L , RAO K F ,et al. Cluster computing mode for water environment simulation based on Hadoop[J]. Big Data Research, 2019,5(6): 73-84.
[9]	ZHANG D J , CHEN X W , YAO H X ,et al. Moving SWAT model calibration and uncertainty analysis to an enterprise Hadoop-based cloud[J]. Environmental Modelling ＆ Software, 2016,84: 140-148.
[10]	GEORGE L . HBase:the definitive guide(fisrt edition)[M]. Sebastopol: O’Reilly Media, 2011.
[11]	KARAU H , KONWINSKI A , WENDELL P ,et al. Spark快速大数据分析[M].王道远,译. 北京: 人民邮电出版社, 2015.
	KARAU H , KONWINSKI A , WENDELL P ,et al. Learning Spark:lightning-fast data analysis[M]. Translated by WANG D Y. Beijing: Posts ＆ Telecom Press, 2015.
[12]	WHITE T . Hadoop:the definitive guide(second edition)[M]. Sebastopol: O’Reilly Media, 2010.
[13]	吴信东, 嵇圣硙 . MapReduce与Spark用于大数据分析之比较[J]. 软件学报, 2018,29(6): 1770-1791.
	WU X D , JI S W . Comparative study on MapReduce and Spark for big data analytics[J]. Journal of Software, 2018,29(6): 1770-1791.
[14]	SHIVHARE N , DIKSHIT P K S , DWIVEDI S B . A comparison of SWAT model calibration techniques for hydrological modeling in the Ganga river watershed[J]. Engineering, 2018,4(5): 643-652.
[15]	MURTHY A C , VAVILAPALLI V K . Apache Hadoop YARN:moving beyond MapReduce and batch processing with Apache Hadoop 2[M]. Boston: AddisonWesley Professional, 2014.
[16]	YI X , ZOU R , GUO H C . Global sensitivity analysis of a three-dimensional nutrients-algae dynamic model for a large shallow lake[J]. Ecological Modelling, 2016,327: 74-84.
[17]	SONG X M , ZHANG J Y , ZHAN C S ,et al. Global sensitivity analysis in hydrological modeling:review of concepts,methods,theoretical framework,and applications[J]. Journal of Hydrology, 2015,523: 739-757.
[18]	JIANG L , LI Y P , ZHAO X ,et al. Parameter uncertainty and sensitivity analysis of water quality model in Lake Taihu,China[J]. Ecological Modelling, 2018,375: 1-12.
[19]	CAMPOLONGO F , SALTELLI A . Sensitivity analysis of an environmental model:an application of different analysis methods[J]. Reliability Engineering ＆System Safety, 1997,57(1): 49-69.
[20]	刘松, 佘敦先, 张利平 ,等. 基于Morris和Sobol的水文模型参数敏感性分析[J]. 长江流域资源与环境, 2019,28(6): 1296-1303.
	LIU S , SHE D X , ZHANG L P ,et al. Global sensitivity analysis of hydrological model parameters based on Morris and Sobol methods[J]. Resources and Environment in the Yangtze Basin, 2019,28(6): 1296-1303.
[21]	BENNETT N D , CROKE B F W , GUARISO G ,et al. Characterising performance of environmental models[J]. Environmental Modelling ＆ Software, 2013,40: 1-20.
[22]	BAE S , SEO D . Analysis and modeling of algal blooms in the Nakdong River,Korea[J]. Ecological Modelling, 2018,372: 53-63.
[23]	NASH J E , SUTCLIFFE J V . River flow forecasting through conceptual models part I—a discussion of principles[J]. Journal of Hydrology, 1970,10(3): 282-290.
[24]	GHAITH M , LI Z . Propagation of parameter uncertainty in SWAT:a probabilistic forecasting method based on polynomial chaos expansion and machine learning[J]. Journal of Hydrology, 2020,10: 124854
[25]	任婷玉, 梁中耀, 刘永 ,等. 基于贝叶斯优化的三维水动力-水质模型参数估值方法[J]. 环境科学学报, 2019,39(6): 2024-2032.
	REN T Y , LIANG Z Y , LIU Y ,et al. The parameters estimation method based on Bayesian optimization for complex water quality models[J]. Acta Scientiae Circumstantiae, 2019,39(6): 2024-2032.
[26]	RAZAVI S , TOLSON B A , BURN D H . Review of surrogate modeling in water resources[J]. Water Resources Research, 2012,48(7).
[27]	SIMPSON T W , MAUERY T M , KORTE J J ,et al. Kriging models for global approximation in simulation-based multidisciplinary design optimization[J]. AIAA Journal, 2001,39(12): 2233-2241.
[28]	DOWLA F U , ROGERS L L . Solving problems in environmental engineering and geosciences with artificial neural networks[M]. Cambridge: MIT Press, 1995.
[29]	REGIS R G , SHOEMAKER C A . A stochastic radial basis function method for the global optimization of expensive functions[J]. INFORMS Journal on Computing, 2007,19(4): 497-509.
[30]	FEN C S , CHAN C C , CHENG H C . Assessing a response surface-based optimization approach for soil vapor extraction system design[J]. Journal of Water Resources Planning and Management, 2009,135(3): 198-207.
[31]	ZHANG X S , SRINIVASAN R , LIEW M V . Approximating SWAT model using artificial neural network and support vector machine[J]. JAWRA Journal of the American Water Resources Association, 2009,45(2): 460-474.
[32]	ZENG L Z , SHI L S , ZHANG D X ,et al. A sparse grid based Bayesian method for contaminant source identification[J]. Advances in Water Resources, 2012,37: 1-9.
[33]	BOYLE J S , KLEIN S A , LUCAS D D ,et al. The parametric sensitivity of CAM5’s MJO[J]. Journal of Geophysical Research:Atmospheres, 2015,120(4): 1424-1444.
[34]	LIANG Z Y , ZOU R , CHEN X ,et al. Simulate the forecast capacity of a complicated water quality model using the long short-term memory approach[J]. Journal of Hydrology, 2020,581: 124432

参数名称	最小值	最大值	参数意义
CN2	35	98	径流曲线数
ALPHA_BNK	0	1	河岸基流α因子
SOL_K	0	2 000	饱和导水率
CH_K2	-0.01	500	主通道冲积层中的有效导水率
CANMX	0	100	最大冠层储藏量
SOL_AWC	0	1	土壤有效含水率
SOL_BD	0.9	2.5	土壤密度
CH_N2	0	0.3	主河道曼宁系数