进化集成学习算法综述

doi:10.11959/j.issn.2096-6652.202103

Abstract

Abstract:

Evolutionary ensemble learning integrates advantages of ensemble learning and evolutionary algorithm and is widely used in machine learning, data mining, and pattern recognition.Firstly, the theoretical basis, formation, and taxonomy are introduced.Secondly, according to the optimization task of evolutionary algorithm in ensemble learning, some representative researches on evolutionary ensemble learning field were analysed from the aspects of instance selection, feature selection, parameter optimization, structure optimization, and fusion strategy optimization, and the characteristics of different evolutionary ensemble learning methods were summarized.Finally, the pros and cons of the current researches on evolutionary ensemble learning were analysed, and research directions in the future work were given.

Key words: ensemble learning, machine learning, evolutionary algorithm, classification, regression, clustering

CLC Number:

TP181

Yi HU, Boyang QU, Jing LIANG, et al. A survey on evolutionary ensemble learning algorithm[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 18-35.

Figures/Tables 2

References 86

[1]	MAHMOOD A , LI T , YANG Y ,et al. Semi-supervised evolutionary ensembles for web video categorization[J]. Knowledge-Based Systems, 2015,76: 53-66.
[2]	NAG K , PAL N R . A multiobjective genetic programming-based ensemble for simultaneous feature selection and classification[J]. IEEE Transactions on Cybernetics, 2015,46(2): 499-510.
[3]	DITTERRICH T . Machine learning research:four current direction[J]. Artificial Intelligence Magazine, 1997,4: 97-136.
[4]	FERNANDEZ A , CARMONA C J , JOSE DEL JESUS M ,et al. A pareto-based ensemble with feature and instance selection for learning from multi-class imbalanced datasets[J]. International Journal of Neural Systems, 2017,27(6): 1750028.
[5]	ROSALES-PEREZ A , GARCIA S , GONZALEZ J A ,et al. An evolutionary multiobjective model and instance selection for support vector machines with pareto-based ensembles[J]. IEEE Transactions on Evolutionary Computation, 2017,21(6): 863-877.
[6]	姚旭, 王晓丹, 张玉玺 ,等. 基于随机子空间和 AdaBoost 的自适应集成方法[J]. 电子学报, 2013,41(4): 810-814.
	YAO X , WANG X D , ZHANG Y X ,et al. A self-adaption ensemble algorithm based on random subspace and AdaBoost[J]. Acta Electronica Sinica, 2013,41(4): 810-814.
[7]	LIU Y , DIAO X C , CAO J J ,et al. A method for entity resolution in high dimensional data using ensemble classifiers[J]. Mathematical Problems in Engineering, 2017(12): 1-11.
[8]	李毓, 张春霞, 王冠伟 . 线性回归模型的 Boosting 变量选择方法[J]. 工程数学学报, 2015,32(5): 677-689.
	LI Y , ZHANG C X , WANG G W . Boosting variable selection algorithm for linear regression models[J]. Chinese Journal of Engineering Mathematics, 2015,32(5): 677-689.
[9]	JIMéNEZ F , SANCHEZ G , GARCIA J M ,et al. Multi-objective evolutionary feature selection for online sales forecasting[J]. Neurocomputing, 2017,234: 75-92.
[10]	HONG Y , KWONG S , CHANG Y ,et al. Unsupervised feature selection using clustering ensembles and population based incremental learning algorithm[J]. Pattern Recognition, 2008,41(9): 2742-2756.
[11]	CHANDRA A , YAO X . Ensemble learning using multi-objective evolutionary algorithms[J]. Journal of Mathematical Modelling and Algorithms, 2006,5(4): 417-445.
[12]	CHEN H , YAO X . Multiobjective neural network ensembles based on regularized negative correlation learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010,22(12): 1738-1751.
[13]	盛伟国, 单鹏霄 . 一种基于小生境的负相关神经网络集成算法[J]. 浙江工业大学学报, 2016(5): 482-486.
	SHENG W G , SHAN P X . Negative correlation neural network ensemble based on a niching algorithm[J]. Journal of Zhejiang University of Technology, 2016(5): 482-486.
[14]	SHENG W G , SHAN P , CHEN S ,et al. A niching evolutionary algorithm with adaptive negative correlation learning for neural network ensemble[J]. Neurocomputing, 2017,247: 173-182.
[15]	PADILHA C A , BARONE D A C , NETO A D D . A multi-level approach using genetic algorithms in an ensemble of least squares support vector machines[J]. Knowledge-Based Systems, 2016,106: 85-95.
[16]	YAO X , FISCHER M , BROWN G . Neural network ensembles and their application to traffic flow prediction in telecommunications networks[C]// Proceedings of 2001 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2001: 693-698.
[17]	JIN Y , OKABE T , SENDHOFF B . Neural network regularization and ensembling using multi-objective evolutionary algorithms[C]// Proceedings of The 2004 Congress on Evolutionary Computation. Piscataway:IEEE Press, 2004: 1-8.
[18]	胡梦月, 胡志坚, 仉梦林 ,等. 基于改进 AdaBoost.RT 和 KELM的风功率预测方法研究[J]. 电网技术, 2017,41(2): 536-542.
	HU M Y , HU Z J , ZHANG M L ,et al. Research on wind power forecasting method based on improved AdaBoost.RT and KELM algorithm[J]. Power System Technology, 2017,41(2): 536-542.
[19]	BUI L T , DINH T T H ,et al. A novel evolutionary multi-objective ensemble learning approach for forecasting currency exchange rates[J]. Data ＆ Knowledge Engineering, 2018,114: 40-66.
[20]	HANDL J , KNOWLES J . An evolutionary approach to multiobjective clustering[J]. IEEE Transactions on Evolutionary Computation, 2007,11(1): 56-76.
[21]	MOHAMMADI M , NIKANJAM A , RAHMANI A . An evolutionary approach to clustering ensemble[C]// Proceedings of the 4th International Conference on Natural Computation. Piscataway:IEEE Press, 2008: 77-82.
[22]	OBO T , KUBOTA N , LOO C K . Evolutionary ensemble learning of fuzzy randomized neural network for posture recognition[C]// Proceedings of 2016 World Automation Congress. Piscataway:IEEE Press, 2016: 1-6.
[23]	OJHA V K , ABRAHAM A , SNASEL V . Ensemble of heterogeneous flexible neural trees using multiobjective genetic programming[J]. Applied Soft Computing, 2017,52: 909-924.
[24]	MINKU L L , YAO X . Software effort estimation as a multiobjective learning problem[J]. ACM Transactions on Software Engineering and Methodology, 2013,22(4): 1-32.
[25]	SMITH C , JIN Y . Evolutionary multi-objective generation of recurrent neural network ensembles for time series prediction[J]. Neurocomputing, 2014,143: 302-311.
[26]	MIN S H . A genetic algorithm-based heterogeneous random subspace ensemble model for bankruptcy prediction[J]. International Journal of Applied Engineering Research, 2016,11: 2927-2931.
[27]	EKBAL A , SAHA S . A multiobjective simulated annealing approach for classifier ensemble:named entity recognition in indian languages as case studies[J]. Expert Systems with Applications, 2011,38(12): 14760-14772.
[28]	任克强, 高晓林, 谢斌 . 基于 AFSA 和 PSO 融合优化的 AdaBoost人脸检测算法[J]. 小型微型计算机系统, 2016,37(4): 861-865.
	REN K Q , GAO X L , XIE B . AdaBoost face detection algorithm based on fusion optimization of AFSA and PSO[J]. Journal of Chinese Computer Systems, 2016,37(4): 861-865.
[29]	ONAN A , KORUKOGLU S , BULUT H . A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification[J]. Expert Systems with Applications, 2016,62: 1-16.
[30]	ABRAHAM A , GROSAN C , HAN S Y ,et al. Evolutionary multiobjective optimization approach for evolving ensemble of intelligent paradigms for stock market modeling[C]// Proceedings of 2005 Mexican International Conference on Artificial Intelligence. Heidelberg:Springer-Verlag, 2005: 673-681.
[31]	CHEN Y , YANG B , ABRAHAM A . Flexible neural trees ensemble for stock index modeling[J]. Neurocomputing, 2007,70(4-6): 697-703.
[32]	ALAMANIOTIS M , IKONOMOPOULOS A , TSOUKALAS L H . Evolutionary multiobjective optimization of kernel-based very-shortterm load forecasting[J]. IEEE Transactions on Power Systems, 2012,27(3): 1477-1484.
[33]	YOON H S , AHN S Y , LEE S H ,et al. Heterogeneous clustering ensemble method for combining different cluster results[C]// Proceedings of the International Workshop on Data Mining for Biomedical Applications. Heidelberg:Springer-Verlag, 2006: 82-92.
[34]	LUO H , JING F , XIE X . Combining multiple clusterings using information theory based genetic algorithm[C]// Proceedings of 2006 International Conference on Computational Intelligence and Security. Piscataway:IEEE Press, 2006: 84-89.
[35]	QIAN X , ZHANG X , JIAO L ,et al. Unsupervised texture image segmentation using multiobjective evolutionary clustering ensemble algorithm[C]// Proceedings of 2008 IEEE Congress on Evolutionary Computation. Piscataway:IEEE Press, 2008: 3561-3567.
[36]	WAHID A , GAO X , ANDREAE P . Multi-objective multi-view clustering ensemble based on evolutionary approach[C]// Proceedings of 2015 IEEE Congress on Evolutionary Computation. Piscataway:IEEE Press, 2015: 1696-1703.
[37]	WANG Y , LIU X , XIANG L . Ga-based membrane evolutionary algorithm for ensemble clustering[J]. Computational Intelligence and Neuroscience, 2017.
[38]	LI X , ZHANG S , WONG K C . Single-cell RNA-seq interpretations using evolutionary multiobjective ensemble pruning[J]. Bioinformatics, 2019,35(16): 2809-2817.
[39]	REN Y , ZHANG L , SUGANTHAN P N . Ensemble classification and regression-recent developments,applications and future directions[J]. IEEE Computational Intelligence Magazine, 2016,11(1): 41-53.
[40]	MENDES-MOREIRA J , SOARES C , JORGE A M ,et al. Ensemble approaches for regression:a survey[J]. ACM Computing Surveys, 2012,45(1): 1-40.
[41]	VEGA-PONS S ， RUIZ-SHULCLOPER J . A survey of clustering ensemble algorithms[J]. International Journal of Pattern Recognition and Artificial Intelligence, 2011,25(3): 337-372.
[42]	MENDES-MOREIRA J , SOARES C , JORGE A M ,et al. Ensemble approaches for regression:a survey[J]. ACM Computing Surveys, 2012,45(1): 1-40.
[43]	FREUND Y , SCHAPIRE R E ,et al. Experiments with a new boosting algorithm[C]// Proceedings of the 13th International Conference on Machine Learning.[S.l.:s.n.], 1996: 148-156.
[44]	BREIMAN L . Bagging predictors[J]. Machine Learning, 1996,24(2): 123-140.
[45]	TING K M , WITTEN I H . Stacking bagged and dagged models[C]// Proceedings of the 14th International Conference on Machine Learning.San Francisco:Morgan Kaufmann Publishers Inc. , 1997: 367-375.
[46]	HO T K . The random subspace method for constructing decision forests[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998,20(8): 832-844.
[47]	WOLPERT D H . Stacked generalization[J]. Neural Networks, 1992,5(2): 241-259.
[48]	MORENO-SECO F , INESTA J M , DE LEON P J P ,et al. Comparison of classifier fusion methods for classification in pattern recognition tasks[C]// Proceedings of the Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Heidelberg:Springer-Verlag, 2006: 705-713.
[49]	DIETTERICH T G , . Ensemble methods in machine learning[C]// Proceedings of the 1st International Workshop on Multiple Classifier Systems. Heidelberg:Springer-Verlag, 2000: 1-15.
[50]	JAMES G M . Variance and bias for general loss functions[J]. Machine Learning, 2003,51(2): 115-135.
[51]	KLEINBERG E . Stochastic discrimination[J]. Annals of Mathematics and Artificial Intelligence, 1990,1(1-4): 207-239.
[52]	SCHAPIRE R E , FREUND Y , BARTLETT P ,et al. Boosting the margin:a new explanation for the effectiveness of voting methods[J]. The Annals of Statistics, 1998,26(5): 1651-1686.
[53]	FRANK E , PFAHRINGER B . Improving on bagging with input smearing[C]// Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining. Berlin:Springer, 2006: 97-106.
[54]	BREIMAN L . Randomizing outputs to increase prediction accuracy[J]. Machine Learning, 2000,40(3): 229-242.
[55]	LIN H T , LI L . Infinite ensemble learning with support vector machines[C]// European Conference on Machine Learning. Heidelberg:Springer-Verlag, 2005: 242-254.
[56]	BI Y . The impact of diversity on the accuracy of evidential classifier ensembles[J]. International Journal of Approximate Reasoning, 2012,53(4): 584-607.
[57]	张春霞, 张讲社 . 选择性集成学习算法综述[J]. 计算机学报, 2011,34(8): 1399-1410.
	ZHANG C X , ZHANG J S . A survey of selective ensemble learning algorithms[J]. Chinese Journal of Computers, 2011,34(8): 1399-1410.
[58]	SAGI O , ROKACH L . Ensemble learning:a survey[J]. Wiley Interdisciplinary Reviews:Data Mining and Knowledge Discovery, 2018,8(4): e1249.
[59]	PACL′IK P , LANDGREBE T C , TAX D M ,et al. On deriving the second-stage training set for trainable combiners[C]// Proceedings of the 6th International Workshop on Multiple Classifier Systems. Heidelberg:Springer-Verlag, 2005: 136-146.
[60]	张春霞 . 集成学习中有关算法的研究[D]. 西安:西安交通大学, 2010.
	ZHANG C X . Research on some algorithms in ensemble learning[D]. Xi’an:Xi’an Jiaotong University, 2010.
[61]	ROONEY N , PATTERSON D , ANAND S ,et al. Dynamic integration of regression models[C]// Proceedings of the 5th International Workshop on Multiple Classifier Systems. Heidelberg:Springer-Verlag, 2004: 164-173.
[62]	VALENTINI G , MASULLI F . Ensembles of learning machines[C]// Proceedings of the 13th Italian Workshop on Neural Nets. Heidelberg:Springer-Verlag, 2002: 3-20.
[63]	ZHOU Z H , WU J , TANG W . Ensembling neural networks:many could be better than all[J]. Artificial Intelligence, 2002,137(1-2): 239-263.
[64]	WANG Y , LI B . Multi-strategy ensemble evolutionary algorithm for dynamic multi-objective optimization[J]. Memetic Computing, 2010,2(1): 3-24.
[65]	YEN G G , HE Z . Performance metric ensemble for multiobjective evolutionary algorithms[J]. IEEE Transactions on Evolutionary Computation, 2013,18(1): 131-144.
[66]	QIU X , SUGANTHAN P N , AMARATUNGA G A . Ensemble incremental learning random vector functional link network for short-term electric load forecasting[J]. Knowledge-Based Systems, 2018,145: 182-196.
[67]	QIU X , REN Y , SUGANTHAN P N ,et al. Empirical mode decomposition based ensemble deep learning for load demand time series forecasting[J]. Applied Soft Computing, 2017,54: 246-255.
[68]	YUE C , LIANG J , QU B ,et al. Multimodal multiobjective optimization in feature selection[C]// Proceedings of 2019 IEEE Congress on Evolutionary Computation. Piscataway:IEEE Press, 2019: 302-309.
[69]	YUE C , QU B , LIANG J . A multiobjective particle swarm optimizer using ring topology for solving multimodal multiobjective problems[J]. IEEE Transactions on Evolutionary Computation, 2017,22(5): 805-817.
[70]	HU Y , WANG J , LIANG J ,et al. A self-organizing multimodal multi-objective pigeon-inspired optimization algorithm[J]. Science China-information Sciences, 2019,62(7): 1-17.
[71]	AIN Q U , SAHAF H A , XUE B ,et al. A genetic programming approach to feature construction for ensemble learning in skin cancer detection[C]// Proceedings of the 2020 Genetic and Evolutionary Computation Conference. New York:ACM Press, 2020: 1186-1194.
[72]	ZENG J , XU H Y , GONG G L ,et al. An heterogeneous stacking ensemble model for laser-induced breakdown spectroscopy elemental quantitative analysis[J]. IEEE Access, 2020,8: 136141-136150.
[73]	HUANG T , DUAN D T , GONG Y J ,et al. Concurrent optimization of multiple base learners in neural network ensembles:an adaptive niching differential evolution approach[J]. Neurocomputing, 2020,396: 24-38.
[74]	YAN S , WANG Y , KONG D ,et al. Hybrid chain-hypergraph p systems for multiobjective ensemble clustering[J]. IEEE Access, 2019,9: 143511-143523.
[75]	MINKU L L , YAO X . An analysis of multi-objective evolutionary algorithms for training ensemble models based on different performance measures in software effort estimation[C]// Proceedings of the 9th International Conference on Predictive Models in Software Engineering.[S.l.:s.n.], 2013: 1-10.
[76]	LIU W , WU W , WANG Y ,et al. Selective ensemble learning method for belief-rule-base classification system based on PAES[J]. Big Data Mining and Analytics, 2019,2(4): 306-318.
[77]	FACELI K , DE CARVALHO A C , DE SOUTO M C . Multi-objective clustering ensemble[J]. International Journal of Hybrid Intelligent Systems, 2007,4(3): 145-156.
[78]	LIU R , LIU Y , LI Y . An improved method for multi-objective clustering ensemble algorithm[C]// Proceedings of the 2012 IEEE Congress on Evolutionary Computation. Piscataway:IEEE Press, 2012: 1-8.
[79]	GOUDOS S , ATHANASIADOU G E . Application of an ensemble method to uav power modeling for cellular communications[J]. IEEE Antennas and Wireless Propagation Letters, 2019(99): 1.
[80]	DAI H , SHENG W . A multi-objective clustering ensemble algorithm with automatic k-determination[C]// Proceedings of the 2019 IEEE 4th International Conference on Cloud Computing and Big Data Analysis. Piscataway:IEEE Press, 2019: 333-337.
[81]	MINKU F L , LUDERMIR T B . Clustering and co-evolution to construct neural network ensembles:an experimental study[J]. Neural Networks, 2008,21(9): 1363-1379.
[82]	RAHMAN A , VERMA B . Cluster oriented ensemble classifiers using multi-objective evolutionary algorithm[C]// Proceedings of the 2013 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2013: 1-6.
[83]	COLETTA L F , HRUSCHKA E R , ACHARYA A ,et al. A differential evolution algorithm to optimise the combination of classifier and cluster ensembles[J]. International Journal of Bio-Inspired Computation, 2015,7(2): 111-124.
[84]	JAN Z M , VERMA B K . Evolutionary classifier and cluster selection approach for ensemble classification[J]. ACM Transactions on Knowledge Discovery from Data, 2019.
[85]	JIANG F , HE J , TIAN T . A clustering-based ensemble approach with improved pigeon-inspired optimization and extreme learning machine for air quality prediction[J]. Applied Soft Computing, 2019,85: 105827.
[86]	ONAN A , KORUKOGLU S , BULUT H . A hybrid ensemble pruning approach based on consensus clustering and multi-objective evolutionary algorithm for sentiment classification[J]. Information Processing and Management, 2017,53(4): 814-833.

Metrics

Recommended 0

No Suggested Reading articles found!

A survey on evolutionary ensemble learning algorithm

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 2

References 86

Related Articles 8

Metrics

Recommended 0

[1]	Hui XU. Sidelink resource allocation algorithm of C-V2X based on deep Q learning [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(1): 83-91.
[2]	Yun WANG, Meiyun WANG, Jian ZHOU, Yuanyuan ZOU, Shaoyuan LI. Dynamic configuration of distribution network based on improved hierarchical clustering and GL-APSO algorithm [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(3): 410-417.
[3]	Dongwei HU, Xiaolu FENG. Theoretical framework of brain modelling and highlighted problems [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(4): 412-434.
[4]	Hong SHAO, Mingkun ZHANG, Wencheng CUI. Classification method of dermoscopic image based on hierarchical convolution neural network [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(4): 474-481.
[5]	Chenxi LIU, Binghong LIU, Xian ZHANG, Xinnan LONG, Mugen PENG. Intelligent service oriented fog radio access network:principles, technologies and challenges [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 10-17.
[6]	Liang ZHAO, Zhifeng XIE, Kunpeng ZHANG, Yuqing ZHENG, Yuankun FU. Modeling signal propagation in wireless network:an interval type-2 fuzzy ensemble deep learning approach [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 401-411.
[7]	Zhidong XU,Bingyang CHEN,Xiao WANG,Weishan ZHANG. Research on capsule network-based for aspect-level sentiment classification [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(3): 284-292.
[8]	Hanping CAO,Xiaojing ZHANG,Ruijie ZHU,Xiaola HUANG. Application and practice of machine learning model in real-time anti-fraud in the era of digital finance [J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(4): 342-351.