进化集成学习算法综述

doi:10.11959/j.issn.2096-6652.202103

智能科学与技术学报 ›› 2021, Vol. 3 ›› Issue (1): 18-35.doi: 10.11959/j.issn.2096-6652.202103

进化集成学习算法综述

胡毅¹, 瞿博阳², 梁静¹, 王杰¹, 王艳丽¹

¹ 郑州大学电气工程学院，河南郑州 450001
² 中原工学院电子信息学院，河南郑州 450007

修回日期:2020-08-25 出版日期:2021-03-15 发布日期:2021-03-01
作者简介:胡毅（1988- ），男，郑州大学电气工程学院博士生，主要研究方向为进化计算、机器学习、智能控制及其应用。
瞿博阳（1984- ），男，博士，中原工学院电子信息学院教授、院长，主要研究方向为机器学习、遗传与进化算法、群集智能和多目标优化。
梁静（1981- ），女，博士，郑州大学电气工程学院教授、院长，主要研究方向为计算智能、智能优化及机器学习。
王杰（1959- ），男，博士，郑州大学电气工程学院教授，主要研究方向为模式识别与智能控制。
王艳丽（1982- ），女，郑州大学电气工程学院博士生，主要研究方向为智能计算和机器学习。
基金资助:
国家自然科学基金资助项目(61922072);河南省高等学校重点科研资助项目(18A470015);中国博士后科学基金资助项目(2017M622373)

A survey on evolutionary ensemble learning algorithm

Yi HU¹, Boyang QU², Jing LIANG¹, Jie WANG¹, Yanli WANG¹

¹ School of Electrical Engineering, Zhengzhou University, Zhengzhou 450001, China
² School of Electronic and Information, Zhongyuan University of Technology, Zhengzhou 450007, China

Revised:2020-08-25 Online:2021-03-15 Published:2021-03-01
Supported by:
The National Natural Science Foundation of China(61922072);The Key Scientific Research Projects of Henan Province under Grant(18A470015)

摘要/Abstract

摘要：

进化集成学习结合了集成学习和进化算法两方面的优势，并在机器学习、数据挖掘和模式识别等领域被广泛应用。首先对进化集成学习算法的理论基础、组成结构及分类情况进行了概述。然后根据进化算法在集成学习中的优化任务，从样本选择、特征选择、集成模型参数组合优化、集成模型结构优化以及集成模型融合策略优化几个方面对当前进化集成学习领域的一些代表性研究成果进行了详细的综述，并分析和总结了各种进化集成学习算法的特点。最后对现有的进化集成研究的优缺点进行探讨，并给出了未来的研究方向。

关键词: 集成学习, 机器学习, 进化算法, 分类, 回归, 聚类

Abstract:

Evolutionary ensemble learning integrates advantages of ensemble learning and evolutionary algorithm and is widely used in machine learning, data mining, and pattern recognition.Firstly, the theoretical basis, formation, and taxonomy are introduced.Secondly, according to the optimization task of evolutionary algorithm in ensemble learning, some representative researches on evolutionary ensemble learning field were analysed from the aspects of instance selection, feature selection, parameter optimization, structure optimization, and fusion strategy optimization, and the characteristics of different evolutionary ensemble learning methods were summarized.Finally, the pros and cons of the current researches on evolutionary ensemble learning were analysed, and research directions in the future work were given.

Key words: ensemble learning, machine learning, evolutionary algorithm, classification, regression, clustering

中图分类号:

TP181

胡毅, 瞿博阳, 梁静, 等. 进化集成学习算法综述[J]. 智能科学与技术学报, 2021, 3(1): 18-35.

Yi HU, Boyang QU, Jing LIANG, et al. A survey on evolutionary ensemble learning algorithm[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 18-35.

图/表 2

参考文献 86

[1]	MAHMOOD A , LI T , YANG Y ,et al. Semi-supervised evolutionary ensembles for web video categorization[J]. Knowledge-Based Systems, 2015,76: 53-66.
[2]	NAG K , PAL N R . A multiobjective genetic programming-based ensemble for simultaneous feature selection and classification[J]. IEEE Transactions on Cybernetics, 2015,46(2): 499-510.
[3]	DITTERRICH T . Machine learning research:four current direction[J]. Artificial Intelligence Magazine, 1997,4: 97-136.
[4]	FERNANDEZ A , CARMONA C J , JOSE DEL JESUS M ,et al. A pareto-based ensemble with feature and instance selection for learning from multi-class imbalanced datasets[J]. International Journal of Neural Systems, 2017,27(6): 1750028.
[5]	ROSALES-PEREZ A , GARCIA S , GONZALEZ J A ,et al. An evolutionary multiobjective model and instance selection for support vector machines with pareto-based ensembles[J]. IEEE Transactions on Evolutionary Computation, 2017,21(6): 863-877.
[6]	姚旭, 王晓丹, 张玉玺 ,等. 基于随机子空间和 AdaBoost 的自适应集成方法[J]. 电子学报, 2013,41(4): 810-814.
	YAO X , WANG X D , ZHANG Y X ,et al. A self-adaption ensemble algorithm based on random subspace and AdaBoost[J]. Acta Electronica Sinica, 2013,41(4): 810-814.
[7]	LIU Y , DIAO X C , CAO J J ,et al. A method for entity resolution in high dimensional data using ensemble classifiers[J]. Mathematical Problems in Engineering, 2017(12): 1-11.
[8]	李毓, 张春霞, 王冠伟 . 线性回归模型的 Boosting 变量选择方法[J]. 工程数学学报, 2015,32(5): 677-689.
	LI Y , ZHANG C X , WANG G W . Boosting variable selection algorithm for linear regression models[J]. Chinese Journal of Engineering Mathematics, 2015,32(5): 677-689.
[9]	JIMéNEZ F , SANCHEZ G , GARCIA J M ,et al. Multi-objective evolutionary feature selection for online sales forecasting[J]. Neurocomputing, 2017,234: 75-92.
[10]	HONG Y , KWONG S , CHANG Y ,et al. Unsupervised feature selection using clustering ensembles and population based incremental learning algorithm[J]. Pattern Recognition, 2008,41(9): 2742-2756.
[11]	CHANDRA A , YAO X . Ensemble learning using multi-objective evolutionary algorithms[J]. Journal of Mathematical Modelling and Algorithms, 2006,5(4): 417-445.
[12]	CHEN H , YAO X . Multiobjective neural network ensembles based on regularized negative correlation learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010,22(12): 1738-1751.
[13]	盛伟国, 单鹏霄 . 一种基于小生境的负相关神经网络集成算法[J]. 浙江工业大学学报, 2016(5): 482-486.
	SHENG W G , SHAN P X . Negative correlation neural network ensemble based on a niching algorithm[J]. Journal of Zhejiang University of Technology, 2016(5): 482-486.
[14]	SHENG W G , SHAN P , CHEN S ,et al. A niching evolutionary algorithm with adaptive negative correlation learning for neural network ensemble[J]. Neurocomputing, 2017,247: 173-182.
[15]	PADILHA C A , BARONE D A C , NETO A D D . A multi-level approach using genetic algorithms in an ensemble of least squares support vector machines[J]. Knowledge-Based Systems, 2016,106: 85-95.
[16]	YAO X , FISCHER M , BROWN G . Neural network ensembles and their application to traffic flow prediction in telecommunications networks[C]// Proceedings of 2001 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2001: 693-698.
[17]	JIN Y , OKABE T , SENDHOFF B . Neural network regularization and ensembling using multi-objective evolutionary algorithms[C]// Proceedings of The 2004 Congress on Evolutionary Computation. Piscataway:IEEE Press, 2004: 1-8.
[18]	胡梦月, 胡志坚, 仉梦林 ,等. 基于改进 AdaBoost.RT 和 KELM的风功率预测方法研究[J]. 电网技术, 2017,41(2): 536-542.
	HU M Y , HU Z J , ZHANG M L ,et al. Research on wind power forecasting method based on improved AdaBoost.RT and KELM algorithm[J]. Power System Technology, 2017,41(2): 536-542.
[19]	BUI L T , DINH T T H ,et al. A novel evolutionary multi-objective ensemble learning approach for forecasting currency exchange rates[J]. Data ＆ Knowledge Engineering, 2018,114: 40-66.
[20]	HANDL J , KNOWLES J . An evolutionary approach to multiobjective clustering[J]. IEEE Transactions on Evolutionary Computation, 2007,11(1): 56-76.
[21]	MOHAMMADI M , NIKANJAM A , RAHMANI A . An evolutionary approach to clustering ensemble[C]// Proceedings of the 4th International Conference on Natural Computation. Piscataway:IEEE Press, 2008: 77-82.
[22]	OBO T , KUBOTA N , LOO C K . Evolutionary ensemble learning of fuzzy randomized neural network for posture recognition[C]// Proceedings of 2016 World Automation Congress. Piscataway:IEEE Press, 2016: 1-6.
[23]	OJHA V K , ABRAHAM A , SNASEL V . Ensemble of heterogeneous flexible neural trees using multiobjective genetic programming[J]. Applied Soft Computing, 2017,52: 909-924.
[24]	MINKU L L , YAO X . Software effort estimation as a multiobjective learning problem[J]. ACM Transactions on Software Engineering and Methodology, 2013,22(4): 1-32.
[25]	SMITH C , JIN Y . Evolutionary multi-objective generation of recurrent neural network ensembles for time series prediction[J]. Neurocomputing, 2014,143: 302-311.
[26]	MIN S H . A genetic algorithm-based heterogeneous random subspace ensemble model for bankruptcy prediction[J]. International Journal of Applied Engineering Research, 2016,11: 2927-2931.
[27]	EKBAL A , SAHA S . A multiobjective simulated annealing approach for classifier ensemble:named entity recognition in indian languages as case studies[J]. Expert Systems with Applications, 2011,38(12): 14760-14772.
[28]	任克强, 高晓林, 谢斌 . 基于 AFSA 和 PSO 融合优化的 AdaBoost人脸检测算法[J]. 小型微型计算机系统, 2016,37(4): 861-865.
	REN K Q , GAO X L , XIE B . AdaBoost face detection algorithm based on fusion optimization of AFSA and PSO[J]. Journal of Chinese Computer Systems, 2016,37(4): 861-865.
[29]	ONAN A , KORUKOGLU S , BULUT H . A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification[J]. Expert Systems with Applications, 2016,62: 1-16.
[30]	ABRAHAM A , GROSAN C , HAN S Y ,et al. Evolutionary multiobjective optimization approach for evolving ensemble of intelligent paradigms for stock market modeling[C]// Proceedings of 2005 Mexican International Conference on Artificial Intelligence. Heidelberg:Springer-Verlag, 2005: 673-681.
[31]	CHEN Y , YANG B , ABRAHAM A . Flexible neural trees ensemble for stock index modeling[J]. Neurocomputing, 2007,70(4-6): 697-703.
[32]	ALAMANIOTIS M , IKONOMOPOULOS A , TSOUKALAS L H . Evolutionary multiobjective optimization of kernel-based very-shortterm load forecasting[J]. IEEE Transactions on Power Systems, 2012,27(3): 1477-1484.
[33]	YOON H S , AHN S Y , LEE S H ,et al. Heterogeneous clustering ensemble method for combining different cluster results[C]// Proceedings of the International Workshop on Data Mining for Biomedical Applications. Heidelberg:Springer-Verlag, 2006: 82-92.
[34]	LUO H , JING F , XIE X . Combining multiple clusterings using information theory based genetic algorithm[C]// Proceedings of 2006 International Conference on Computational Intelligence and Security. Piscataway:IEEE Press, 2006: 84-89.
[35]	QIAN X , ZHANG X , JIAO L ,et al. Unsupervised texture image segmentation using multiobjective evolutionary clustering ensemble algorithm[C]// Proceedings of 2008 IEEE Congress on Evolutionary Computation. Piscataway:IEEE Press, 2008: 3561-3567.
[36]	WAHID A , GAO X , ANDREAE P . Multi-objective multi-view clustering ensemble based on evolutionary approach[C]// Proceedings of 2015 IEEE Congress on Evolutionary Computation. Piscataway:IEEE Press, 2015: 1696-1703.
[37]	WANG Y , LIU X , XIANG L . Ga-based membrane evolutionary algorithm for ensemble clustering[J]. Computational Intelligence and Neuroscience, 2017.
[38]	LI X , ZHANG S , WONG K C . Single-cell RNA-seq interpretations using evolutionary multiobjective ensemble pruning[J]. Bioinformatics, 2019,35(16): 2809-2817.
[39]	REN Y , ZHANG L , SUGANTHAN P N . Ensemble classification and regression-recent developments,applications and future directions[J]. IEEE Computational Intelligence Magazine, 2016,11(1): 41-53.
[40]	MENDES-MOREIRA J , SOARES C , JORGE A M ,et al. Ensemble approaches for regression:a survey[J]. ACM Computing Surveys, 2012,45(1): 1-40.
[41]	VEGA-PONS S ， RUIZ-SHULCLOPER J . A survey of clustering ensemble algorithms[J]. International Journal of Pattern Recognition and Artificial Intelligence, 2011,25(3): 337-372.
[42]	MENDES-MOREIRA J , SOARES C , JORGE A M ,et al. Ensemble approaches for regression:a survey[J]. ACM Computing Surveys, 2012,45(1): 1-40.
[43]	FREUND Y , SCHAPIRE R E ,et al. Experiments with a new boosting algorithm[C]// Proceedings of the 13th International Conference on Machine Learning.[S.l.:s.n.], 1996: 148-156.
[44]	BREIMAN L . Bagging predictors[J]. Machine Learning, 1996,24(2): 123-140.
[45]	TING K M , WITTEN I H . Stacking bagged and dagged models[C]// Proceedings of the 14th International Conference on Machine Learning.San Francisco:Morgan Kaufmann Publishers Inc. , 1997: 367-375.
[46]	HO T K . The random subspace method for constructing decision forests[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998,20(8): 832-844.
[47]	WOLPERT D H . Stacked generalization[J]. Neural Networks, 1992,5(2): 241-259.
[48]	MORENO-SECO F , INESTA J M , DE LEON P J P ,et al. Comparison of classifier fusion methods for classification in pattern recognition tasks[C]// Proceedings of the Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Heidelberg:Springer-Verlag, 2006: 705-713.
[49]	DIETTERICH T G , . Ensemble methods in machine learning[C]// Proceedings of the 1st International Workshop on Multiple Classifier Systems. Heidelberg:Springer-Verlag, 2000: 1-15.
[50]	JAMES G M . Variance and bias for general loss functions[J]. Machine Learning, 2003,51(2): 115-135.
[51]	KLEINBERG E . Stochastic discrimination[J]. Annals of Mathematics and Artificial Intelligence, 1990,1(1-4): 207-239.
[52]	SCHAPIRE R E , FREUND Y , BARTLETT P ,et al. Boosting the margin:a new explanation for the effectiveness of voting methods[J]. The Annals of Statistics, 1998,26(5): 1651-1686.
[53]	FRANK E , PFAHRINGER B . Improving on bagging with input smearing[C]// Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining. Berlin:Springer, 2006: 97-106.
[54]	BREIMAN L . Randomizing outputs to increase prediction accuracy[J]. Machine Learning, 2000,40(3): 229-242.
[55]	LIN H T , LI L . Infinite ensemble learning with support vector machines[C]// European Conference on Machine Learning. Heidelberg:Springer-Verlag, 2005: 242-254.
[56]	BI Y . The impact of diversity on the accuracy of evidential classifier ensembles[J]. International Journal of Approximate Reasoning, 2012,53(4): 584-607.
[57]	张春霞, 张讲社 . 选择性集成学习算法综述[J]. 计算机学报, 2011,34(8): 1399-1410.
	ZHANG C X , ZHANG J S . A survey of selective ensemble learning algorithms[J]. Chinese Journal of Computers, 2011,34(8): 1399-1410.
[58]	SAGI O , ROKACH L . Ensemble learning:a survey[J]. Wiley Interdisciplinary Reviews:Data Mining and Knowledge Discovery, 2018,8(4): e1249.
[59]	PACL′IK P , LANDGREBE T C , TAX D M ,et al. On deriving the second-stage training set for trainable combiners[C]// Proceedings of the 6th International Workshop on Multiple Classifier Systems. Heidelberg:Springer-Verlag, 2005: 136-146.
[60]	张春霞 . 集成学习中有关算法的研究[D]. 西安:西安交通大学, 2010.
	ZHANG C X . Research on some algorithms in ensemble learning[D]. Xi’an:Xi’an Jiaotong University, 2010.
[61]	ROONEY N , PATTERSON D , ANAND S ,et al. Dynamic integration of regression models[C]// Proceedings of the 5th International Workshop on Multiple Classifier Systems. Heidelberg:Springer-Verlag, 2004: 164-173.
[62]	VALENTINI G , MASULLI F . Ensembles of learning machines[C]// Proceedings of the 13th Italian Workshop on Neural Nets. Heidelberg:Springer-Verlag, 2002: 3-20.
[63]	ZHOU Z H , WU J , TANG W . Ensembling neural networks:many could be better than all[J]. Artificial Intelligence, 2002,137(1-2): 239-263.
[64]	WANG Y , LI B . Multi-strategy ensemble evolutionary algorithm for dynamic multi-objective optimization[J]. Memetic Computing, 2010,2(1): 3-24.
[65]	YEN G G , HE Z . Performance metric ensemble for multiobjective evolutionary algorithms[J]. IEEE Transactions on Evolutionary Computation, 2013,18(1): 131-144.
[66]	QIU X , SUGANTHAN P N , AMARATUNGA G A . Ensemble incremental learning random vector functional link network for short-term electric load forecasting[J]. Knowledge-Based Systems, 2018,145: 182-196.
[67]	QIU X , REN Y , SUGANTHAN P N ,et al. Empirical mode decomposition based ensemble deep learning for load demand time series forecasting[J]. Applied Soft Computing, 2017,54: 246-255.
[68]	YUE C , LIANG J , QU B ,et al. Multimodal multiobjective optimization in feature selection[C]// Proceedings of 2019 IEEE Congress on Evolutionary Computation. Piscataway:IEEE Press, 2019: 302-309.
[69]	YUE C , QU B , LIANG J . A multiobjective particle swarm optimizer using ring topology for solving multimodal multiobjective problems[J]. IEEE Transactions on Evolutionary Computation, 2017,22(5): 805-817.
[70]	HU Y , WANG J , LIANG J ,et al. A self-organizing multimodal multi-objective pigeon-inspired optimization algorithm[J]. Science China-information Sciences, 2019,62(7): 1-17.
[71]	AIN Q U , SAHAF H A , XUE B ,et al. A genetic programming approach to feature construction for ensemble learning in skin cancer detection[C]// Proceedings of the 2020 Genetic and Evolutionary Computation Conference. New York:ACM Press, 2020: 1186-1194.
[72]	ZENG J , XU H Y , GONG G L ,et al. An heterogeneous stacking ensemble model for laser-induced breakdown spectroscopy elemental quantitative analysis[J]. IEEE Access, 2020,8: 136141-136150.
[73]	HUANG T , DUAN D T , GONG Y J ,et al. Concurrent optimization of multiple base learners in neural network ensembles:an adaptive niching differential evolution approach[J]. Neurocomputing, 2020,396: 24-38.
[74]	YAN S , WANG Y , KONG D ,et al. Hybrid chain-hypergraph p systems for multiobjective ensemble clustering[J]. IEEE Access, 2019,9: 143511-143523.
[75]	MINKU L L , YAO X . An analysis of multi-objective evolutionary algorithms for training ensemble models based on different performance measures in software effort estimation[C]// Proceedings of the 9th International Conference on Predictive Models in Software Engineering.[S.l.:s.n.], 2013: 1-10.
[76]	LIU W , WU W , WANG Y ,et al. Selective ensemble learning method for belief-rule-base classification system based on PAES[J]. Big Data Mining and Analytics, 2019,2(4): 306-318.
[77]	FACELI K , DE CARVALHO A C , DE SOUTO M C . Multi-objective clustering ensemble[J]. International Journal of Hybrid Intelligent Systems, 2007,4(3): 145-156.
[78]	LIU R , LIU Y , LI Y . An improved method for multi-objective clustering ensemble algorithm[C]// Proceedings of the 2012 IEEE Congress on Evolutionary Computation. Piscataway:IEEE Press, 2012: 1-8.
[79]	GOUDOS S , ATHANASIADOU G E . Application of an ensemble method to uav power modeling for cellular communications[J]. IEEE Antennas and Wireless Propagation Letters, 2019(99): 1.
[80]	DAI H , SHENG W . A multi-objective clustering ensemble algorithm with automatic k-determination[C]// Proceedings of the 2019 IEEE 4th International Conference on Cloud Computing and Big Data Analysis. Piscataway:IEEE Press, 2019: 333-337.
[81]	MINKU F L , LUDERMIR T B . Clustering and co-evolution to construct neural network ensembles:an experimental study[J]. Neural Networks, 2008,21(9): 1363-1379.
[82]	RAHMAN A , VERMA B . Cluster oriented ensemble classifiers using multi-objective evolutionary algorithm[C]// Proceedings of the 2013 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2013: 1-6.
[83]	COLETTA L F , HRUSCHKA E R , ACHARYA A ,et al. A differential evolution algorithm to optimise the combination of classifier and cluster ensembles[J]. International Journal of Bio-Inspired Computation, 2015,7(2): 111-124.
[84]	JAN Z M , VERMA B K . Evolutionary classifier and cluster selection approach for ensemble classification[J]. ACM Transactions on Knowledge Discovery from Data, 2019.
[85]	JIANG F , HE J , TIAN T . A clustering-based ensemble approach with improved pigeon-inspired optimization and extreme learning machine for air quality prediction[J]. Applied Soft Computing, 2019,85: 105827.
[86]	ONAN A , KORUKOGLU S , BULUT H . A hybrid ensemble pruning approach based on consensus clustering and multi-objective evolutionary algorithm for sentiment classification[J]. Information Processing and Management, 2017,53(4): 814-833.

进化集成学习算法综述

A survey on evolutionary ensemble learning algorithm

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 2

参考文献 86

相关文章 9

Metrics

推荐阅读 0

[1]	许辉. 基于深度Q学习的蜂窝车联网边路资源分配算法[J]. 智能科学与技术学报, 2023, 5(1): 83-91.
[2]	王云, 王美蕴, 周健, 邹媛媛, 李少远. 基于改进层次聚类和GL-APSO算法的配电网动态重构[J]. 智能科学与技术学报, 2022, 4(3): 410-417.
[3]	胡东伟, 冯晓璐. 大脑建模的理论框架及热点问题[J]. 智能科学与技术学报, 2021, 3(4): 412-434.
[4]	邵虹, 张鸣坤, 崔文成. 基于分层卷积神经网络的皮肤镜图像分类方法[J]. 智能科学与技术学报, 2021, 3(4): 474-481.
[5]	刘晨熙, 刘炳宏, 张贤, 龙新南, 彭木根. 面向智能服务的雾无线接入网络：原理、技术与挑战[J]. 智能科学与技术学报, 2021, 3(1): 10-17.
[6]	赵亮, 谢志峰, 张坤鹏, 郑玉卿, 付园坤. 无线网络信号传输建模：一种区间二型模糊集成深度学习方法[J]. 智能科学与技术学报, 2020, 2(4): 401-411.
[7]	徐志栋,陈炳阳,王晓,张卫山. 基于胶囊网络的方面级情感分类研究[J]. 智能科学与技术学报, 2020, 2(3): 284-292.
[8]	郝晓辰,冀亚坤,郑立召,史鑫,赵彦涛. 基于差分进化的水泥烧成系统动态优化算法[J]. 智能科学与技术学报, 2020, 2(2): 186-193.
[9]	曹汉平,张晓晶,祝睿杰,黄潇拉. 数字金融时代机器学习模型在实时反欺诈中的应用与实践[J]. 智能科学与技术学报, 2019, 1(4): 342-351.