智能启发算法在机器学习中的应用研究综述

doi:10.11959/j.issn.1000-436x.2019242

摘要/Abstract

摘要：

针对机器学习算法在应用中存在的问题，构建基于智能启发算法的机器学习模型优化体系。首先，介绍已有智能启发算法类型及其建模过程。然后，从智能启发算法在机器学习算法中的应用，包括神经网络等参数结构优化、特征优化、集成约简、原型优化、加权投票集成和核函数学习等方面说明智能启发算法的优势。最后，结合实际需求展望智能启发算法及在机器学习领域的发展方向。

关键词: 参数结构优化, 特征优化, 集成约简, 原型优化, 加权投票集成, 核函数学习

Abstract:

Aiming at the problems existing in the application of machine learning algorithm,an optimization system of the machine learning model based on the heuristic algorithm was constructed.Firstly,the existing types of heuristic algorithms and the modeling process of heuristic algorithms were introduced.Then,the advantages of the heuristic algorithm were illustrated from its applications in machine learning,including the parameter and structure optimization of neural network and other machine learning algorithms,feature optimization,ensemble pruning,prototype optimization,weighted voting ensemble and kernel function learning.Finally,the heuristic algorithms and their development directions in the field of machine learning were given according to the actual needs.

Key words: parameter and structure optimization, feature optimization, ensemble pruning, prototype optimization, weighted voting ensemble, kernel function learning

中图分类号:

TP39

沈焱萍,郑康锋,伍淳华,杨义先. 智能启发算法在机器学习中的应用研究综述[J]. 通信学报, 2019, 40(12): 124-137.

Yanping SHEN,Kangfeng ZHENG,Chunhua WU,Yixian YANG. Survey of research on application of heuristic algorithm in machine learning[J]. Journal on Communications, 2019, 40(12): 124-137.

图/表 8

图1

图2

图3

图4

表1

表2

智能启发算法常用评估度量及适应度函数定义形式"

使用场景及常用评估度量	所结合机器学习算法	适应度函数常见定义形式
参数优化、加权集成：准确率	ELM、SVM、DT、FLN（fast learning network）、KNN、NB、C4.5	F₁=acc,acc表示准确率，有些文章采用错误率、F值等 $F_{2} = w_{1} acc + w_{2} (1 - \frac{\sum_{j = 1}^{n_{F}} f_{j}}{n_{F}})$ ，n_F表示特征总数，f_j=1表示
特征优化：准确率、特征约简率	SVM、ELM、KNN
		特征j被选中，否则f_j=0
集成约简：准确率、子模型差异性	ELM	F₃=w₁acc+w₂diversity，diversity表示所用子模型差异性，目前存在多种差异性度量标准，包括成对度量和非成对度量
原型优化：准确率、样本约简率	KNN	F₄=w₁acc+w₂RedRate,RedRate表示样本约简率
核函数学习：准确率、核相似性度量	SVM、ELM	$F_{5} = KA (k_{p}, k_{q})$ ，KA表示实际核矩阵k_p和理想核矩阵k_q的相似性度量

表2

图5

表3

参考文献 95

[1]	MITCHELL T.M . Machine learning[M]. New York: McGraw-Hill EducationPress, 1997.
[2]	KIM P . Deep learning for beginners:with Matlab examples[M]. Charleston: Create Space Independent Publishing PlatformPress, 2016.
[3]	郑泽宇, 顾思宇 . Tensorflow:实战Google深度学习框架[M]. 北京: 电子工业出版社, 2017.
	ZHENG Z Y , GU S Y . Tensorflow:a practical Google deep learning framework[M]. Beijing: Publishing House of Electronics IndustryPress, 2017.
[4]	KARAMI A , GUERRERO-ZAPATA M . A fuzzy anomaly detection system based on hybrid PSO-Kmeans algorithm in content-centric networks[J]. Neurocomputing, 2015,149: 1253-1269.
[5]	YANG X S , GANDOMI A H . Bat algorithm:a novel approach for global engineering optimization[J]. Engineering Computations, 2012,29(5): 464-483.
[6]	YANG X S , HE X . Firefly algorithm:recent advances and applications[J]. International Journal of Swarm Intelligence, 2013,1(1):36.
[7]	MIRJALILI S . Dragonfly algorithm:a new meta-heuristic optimization technique for solving single-objective,discrete,and multi-objective problems[J]. Neural Computing ＆ Applications, 2016,27(4): 1053-1073.
[8]	MIRJALILI S , LEWIS A . The whale optimization algorithm[J]. Advances in Engineering Software, 2016,95: 51-67.
[9]	MIRJALILI S , MIRJALILI S M , LEWIS A . Grey wolf optimization[J]. Advances in engineering software, 2014,69(7): 46-61.
[10]	MIRJALILI S . The ant lion optimizer[J]. Advances in Engineering Software, 2015,83: 80-98.
[11]	MIRJALILI S , GANDOMI A H , MIRJALILI S Z ,et al. Salp swarmalgorithm:a bio-inspired optimizer for engineering design problems[J]. Advances in Engineering Software, 2017: 163-191.
[12]	TALBI E G . Hybrid Metaheuristics[M]. Berlin: SpringerPress, 2013: 3-76.
[13]	Davis L , . Bit-climbing,representational bias,and test suite design[C]// ICGA. CA, 1991: 18-23.
[14]	KIRKPATRICK S , JR D G , VECCHI M P . Optimization by simulated annealing[J]. Science, 1983,220: 671-680.
[15]	GLOVER F . Tabu search - Part I[J]. ORSA Journal on Computing, 1989,1(3): 190-206.
[16]	GLOVER F . Tabu search - Part II[J]. ORSA Journal on Computing, 1990,2(1): 4-32.
[17]	KENNEDY J , EBERHART R . Particle swarm optimization[C]// The 1995 IEEE international conference on neural networks. IEEE, 1995: 1942-1948.
[18]	HOLLAND J H . Genetic algorithms[J]. Scientific American, 1992,267(1): 66-72.
[19]	RECHENBERG I , EIGEN M . Evolutions strategie:optimierung technischer systeme nach prinzipien der biologischen evolution[M]. Stuttgart: Frommann-HolzboogPress, 1973.
[20]	KOZA J R . Genetic programming as a means for programming computers by natural selection[J]. Statistics ＆ Computing, 1994,4(2): 87-112.
[21]	STORN R , PRICE K V . Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces[J]. Journal of Global Optimization, 1997,11(10): 341-359.
[22]	RASHEDI E , NEZAMABADI-POUR H , SARYAZDI S . GSA:a gravitational search algorithm[J]. Information Science, 2009,179(13): 2232-2248.
[23]	KAVEH A , TALATAHARI S . A novel heuristic optimization method:charged system search[J]. Acta Mechanica, 2010,213(3-4): 267-289.
[24]	FORMATO R A . Central force optimization:a new metaheuristic with applications in applied electromagnetics[J]. Progress in Electromagnetics Research, 2007,77: 425-491.
[25]	EROL O K , EKSIN I . A new optimization method:big bang-big crunch[J]. Advances in Engineering Software, 2006,37(2): 106-111.
[26]	DORIGO M , BIRATTARI M , STUTZLE T . Ant colony optimization[J]. IEEE Computational Intelligence Magazine, 2007,1(4): 28-39.
[27]	BASTURK B , KARABOGA D . An artificial bee colony(ABC) algorithm for numeric function optimization[C]// The IEEE swarm intelligence symposium. IEEE, 2006: 12-14.
[28]	YANG X S , DEB S . Cuckoo search via Levy flights[C]// 2009 World Congress on Nature ＆ Biologically Inspired Computing. IEEE, 2009: 210-214.
[29]	RAO R V , SAVSANI V J , VAKHARIA D P . Teaching-learning-based optimization:an optimization method for continuous non-linear large scale problems[J]. Information Sciences, 2012,183(1): 1-15.
[30]	GEEM Z W , KIM J H , LOGANATHAN G V . A new heuristic optimization algorithm:harmony search[J]. Simulation, 2001,76(2): 60-68.
[31]	HE S , WU Q H , SAUNDERS J R . Group search optimizer:an optimization algorithm inspired by animal searching behavior[J]. IEEE Transactions on Evolutionary Computation, 2009,13(5): 973-990.
[32]	WOLPERT D H , MACREADY W G . No free lunch theorems for optimization[J]. IEEE Transactions on Evolutionary Computation, 1997,1(1): 67-82.
[33]	KOZA J R . Genetic programming:a paradigm for genetically breeding populations of computer programs to solve problems[D]. Stanford:Stanford University, 1990.
[34]	MAHDAVI S , SHIRI M E , RAHNAMAYAN S . Metaheuristics in large-scale global continues optimization:a survey[J]. Information Sciences, 2015,295: 407-428.
[35]	ALI M , MHOAMMED B A D , ISMAIL A ,et al. A new intrusion detection system based on fast learning network and particle swarm optimization[J]. IEEE Access, 2018,PP(99):1
[36]	XIA H , HOI S C H . MKBoost:a framework of multiple kernel boosting[J]. IEEE Transactions on Knowledge and Data Engineering, 2013,25(7): 1574-1586.
[37]	YOUNG S R , ROSE D C , KARNOWSKI T P ,et al. Optimizing deep learning hyper-parameters through an evolutionary algorithm[C]// The Workshop on Machine Learning in High-Performance Computing Environments. ACM, 2015: 1-5.
[38]	AHILA R , SADASIVAM V , MANIMALA K . An integrated PSO for parameter determination and feature selection of ELM and its application in classification of power system disturbances[J]. Applied Soft Computing, 2015,32: 23-37.
[39]	ZHU Q , QIN A , SUGANTHAN P ,et al. Evolutionary extreme learning machine[J]. Pattern Recognition, 2005,38(10): 1759-1763.
[40]	ZHANG Y , WU J , CAI Z ,et al. Memetic extreme learning machine[J]. Pattern Recognition, 2016,58: 135-148.
[41]	CAMILLERI M , NERI F , PAPOUTSIDAKIS M . An algorithmic approach to parameter selection in machine learning using meta-optimization techniques[J]. Wseas Transactions on Systems, 2014,13: 203-212.
[42]	KARDAN A A , KAVIAN A , ESMAEILI A . Simultaneous feature selection and feature weighting with K selection for KNN classification using BBO algorithm[C]// Information ＆ Knowledge Technology. IEEE, 2013: 349-354.
[43]	COSTA K A , PEREIRA L A , NAKAMURA R Y ,et al. A nature-inspired approach to speed up optimum-path forest clustering and its application to intrusion detection in computer networks[J]. Information Sciences, 2015,294: 95-108.
[44]	GHEISARI S , MEYBODI M R . BNC-PSO:structure learning of Bayesian networks by particle swarm optimization[J]. Information Sciences, 2016,348: 272-289.
[45]	HUANG C L , DUN J F . A distributed PSO-SVM hybrid system with feature selection and parameter optimization[J]. Applied Soft Computing, 2008,8(4): 1381-1391.
[46]	BHADRA T , BANDYOPADHYAY S , MAULIK U . Differential evolution based optimization of SVM parameters for meta classifier design[J]. Procedia Technology, 2012(4): 50-57.
[47]	DING S , LI H , SU C ,et al. Evolutionary artificial neural networks:a review[J]. Artificial Intelligence Review, 2013,39(3): 251-260.
[48]	PANDAY D , RENATO C D A , LANE P . Feature weighting as a tool for unsupervised feature selection[J]. Information Processing Letters, 2018,129: 44-52.
[49]	姚旭, 王晓丹, 张玉玺 ,等. 特征选择方法综述[J]. 控制与决策, 2012,27(2): 161-166.
	YAO X , WANG X D , ZHANG Y X ,et al. Summary of feature selection algorithms[J]. Control and Decision, 2012,27(2): 161-166.
[50]	KASHEF S , NEZAMABADI-POUR H . An advanced ACO algorithm for feature subset selection[J]. Neurocomputing, 2015,147: 271-279.
[51]	ZELENKOV Y , FEDOROVA E , CHEKRIZOV D . Two-step classification method based on genetic algorithm for bankruptcy forecasting[J]. Expert Systems with Applications, 2017,88: 393-401.
[52]	DIAO R , SHEN Q . Nature inspired feature selection meta-heuristics[J]. Artificial Intelligence Review, 2015,44(3): 311-340.
[53]	MATEOS G , GUTIRREZ J , SANTOS J C . On the evolutionary weighting of neighbors and features in the k-nearest neighbor rule[J]. Neurocomputing, 2019,326-327: 54-60.
[54]	BHARTI K K , SINGH P K . Opposition chaotic fitness mutation based adaptive inertia weight BPSO for feature selection in text clustering[J]. Applied Soft Computing, 2016,43: 20-34.
[55]	LI A , CHEN H L , HUANG H ,et al. An enhanced grey wolf optimization based feature selection wrapped kernel extreme learning machine for medical diagnosis[J]. Computational and Mathematical Methods in Medicine, 2017,2017: 1-15.
[56]	ZHANG L , MISTRY K , LIM C P ,et al. Feature selection using firefly optimization for classification and regression models[J]. Decision Support Systems, 2018,106: 64-85.
[57]	CRUZ R M O , SABOURIN R , CAVALCANTI G D C ,et al. META-DES:a dynamic ensemble selection framework using meta-learning[J]. Pattern Recognition, 2015,48(5): 1925-1935.
[58]	ZHOU Z H , WU J X , TANG W . Ensembling neural networks:many could be better than all[J]. Artificial Intelligence, 2002,137(1): 239-263.
[59]	DIAO R , CHAO F , PENG T ,et al. Feature selection inspired classier ensemble reduction[J]. IEEE Transactions on Cybernetics, 2013,44: 1259-1268.
[60]	SHEN Y , ZHENG K , WU C ,et al. An ensemble method based on selection using bat algorithm for intrusion detection[J]. The Computer Journal, 2017: 1-13.
[61]	KRAWCZYK B . One-class classifier ensemble pruning and weighting with firefly algorithm[J]. Neurocomputing, 2015,150: 490-500.
[62]	HU W M , GAO J , WANG Y ,et al. Online AdaBoost-based parameterized methods for dynamic distributed network intrusion detection[J]. IEEE Transactions on Cybernetics, 2013,44(1): 66-82.
[63]	ZHANG L , SRISUKKHAM W , NEOH S C ,et al. Classifier ensemble reduction using a modified firefly algorithm:an empirical evaluation[J]. Expert Systems with Applications, 2017:93.
[64]	GARCIA S , DERRAC J , CANO J ,et al. Prototype selection for nearest neighbor classification:taxonomy and empirical study[J]. IEEE Transactions on Pattern Analysis ＆ Machine Intelligence, 2012,34(3): 417-435.
[65]	TRIGUERO I , GARCIA S , HERRERA F . A taxonomy and experimental study on prototype generation for nearest neighbor classification[J]. IEEE Transactions on Systems Man ＆ Cybernetics Part C Applications ＆ Reviews, 2012,42(1): 86-100.
[66]	VERBIESTA N , CORNELISA C , HERRERAB F . FRPS:a fuzzy rough prototype selection method[J]. Pattern Recognition, 2013,46(10): 2770-2782.
[67]	RICO J R , INESTA J M . New rank methods for reducing the size of the training set using the nearest neighbor rule[J]. Pattern Recognition Letters, 2012,33(5): 654-660.
[68]	KOHONEN T . The self-organizative map[J]. Proceedings of the IEEE, 1990,78(9): 1464-1480.
[69]	HAMAMOTO Y , UCHIMURA S , TOMITA S . A bootstrap technique for nearest neighbor classifier design[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997,19(1): 73-79.
[70]	LOZANO M , SOTOCA J M , SáNCHEZ J S ,et al. Experimental study on prototype optimization algorithms for prototype-based classification in vector spaces[J]. Pattern Recognition, 2006,39(10): 1827-1838.
[71]	NANNI L , LUMINI A . Particle swarm optimization for prototype reduction[J]. Neurocomputing, 2009,72(4): 1092-1097.
[72]	HU W W , TAN Y . Prototype generation using multiobjective particle swarm optimization for nearest neighbor classification[J]. IEEE Transactions on Cybernetics, 2015,46(12): 2719-2731.
[73]	TRIGUERO I , GARCíA S , HERRERA F . Differential evolution for optimizing the positioning of prototypes in nearest neighbor classification[J]. Pattern Recognition, 2011,44(4): 901-916.
[74]	REZAEI M , NEZAMABADI-POUR H . Using gravitational search algorithm in prototype generation for nearest neighbor classification[J]. Neurocomputing, 2015,157: 256-263.
[75]	PEREZ-RODRIGUEZ J, A G,GARCIA-PEDRAJAS N ， ARROYO-PE?A A G GARCIA-PEDRAJAS N . Simultaneous instance and feature selection and weighting using evolutionary computation:Proposal and study[J]. Applied Soft Computing, 2015,37(C): 416-443.
[76]	DERRAC J , TRIGUERO I , GARCIA S ,et al. Integrating instance selection,instance weighting,and feature weighting for nearest neighbor classifiers by coevolutionary algorithms[J]. IEEE Transactions on Systems,Man,and Cybernetics,Part B (Cybernetics), 2012,42(5): 1383-1397.
[77]	ESCALANTE H J , GRAFF M , MORALES-REYES A . PGGP:prototype generation via genetic programming[J]. Applied Soft Computing, 2016,40: 569-580.
[78]	VERBIEST N , VLUYMANS S , CORNELIS C ,et al. Improving nearest neighbor classification using ensembles of evolutionary generated prototype subsets[J]. Applied Soft Computing, 2016,44(C): 75-88.
[79]	CAO J , KWONG S , WANG R ,et al. Class-specific soft voting based multiple extreme learning machines ensemble[J]. Neurocomputing, 2015,149: 275-284.
[80]	ABUROMMAN A A , REAZ M B I . A novel SVM-kNN-PSO ensemble method for intrusion detection system[J]. Applied Soft Computing, 2016,38: 360-372.
[81]	ZHANG Y , ZHANG H R , CAI J ,et al. A weighted voting classifier based on differential evolution[J]. Abstract and Applied Analysis, 2014,2014: 1-6.
[82]	ONAN A , KORUKOGLU S , BULUT H . A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification[J]. Expert Systems with Applications, 2016,62: 1-16.
[83]	EKBAL A , SAHA S . Weighted vote-based classifier ensemble for named entity recognition:a genetic algorithm-based approach[J]. ACM Transactions on Asian Language Information Processing, 2011,10(2): 1-37.
[84]	MA C , OUYANG J , CHEN H L ,et al. A novel kernel extreme learning machine algorithm based on self-adaptive artificial bee colony optimization strategy[J]. International Journal of Systems Science, 2015: 1-16.
[85]	GAUTHAMA RAMAN M R , SOMU N , KIRTHIVASAN K ,et al. An efficient intrusion detection system based on hypergraph-genetic algorithm for parameter optimization and feature selection in support vector machine[J]. Knowledge-Based Systems, 2017,134: 1-12.
[86]	ZHANG X L , CHEN X F , HE Z J . An ACO-based algorithm for parameter optimization of support vector machines[J]. Expert Systems with Applications, 2010,37(9): 6618-6628.
[87]	KUANG F , XU W , ZHANG S . A novel hybrid KPCA and SVM with GA model for intrusion detection[J]. Applied Soft Computing, 2014,18(C): 178-184.
[88]	KUANG F , ZHANG S , JIN Z ,et al. A novel SVM by combining kernel principal component analysis and improved chaotic particle swarm optimization for intrusion detection[J]. Soft Computing, 2015,19(5): 1187-1199.
[89]	BAMAKAN S M H , WANG H , YINGJIE T ,et al. An effective intrusion detection framework based on MCLP/SVM optimized by time-varying chaos particle swarm optimization[J]. Neurocomputing, 2016,199(C): 90-102.
[90]	BAO Y , HU Z , XIONG T . A PSO and pattern search based memetic algorithm for SVMs parameters optimization[J]. Neurocomputing, 2013,117: 98-106.
[91]	AVCI D , DOGANTEKIN A . An expert diagnosis system for Parkinson disease based on genetic algorithm-wavelet kernel-extreme learning machine[J]. Parkinson’s Disease, 2016: 1-9.
[92]	NIAZMARDI S , SAFARI A , HOMAYOUNI S . A novel multiple kernel learning framework for multiple feature classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations ＆ Remote Sensing, 2017,10(8): 3734-3743.
[93]	NANDA S J , PANDA G . A survey on nature inspired metaheuristic algorithms for partitional clustering[J]. Swarm ＆ Evolutionary Computation, 2014,16: 1-18.
[94]	YANG X S . Nature-inspired algorithms and applied optimization[M]. Berlin: SpringerPress, 2018: 1-25.
[95]	FONG S , DEB S , YANG X S . How meta-heuristic algorithms contribute to deep learning in the hype of big data analytics[C]// Advances in Intelligent Systems and Computing. Springer, 2018: 3-25.

编码方式	分类	优缺点	应用场景
	二进制编码	方便定义；当编码长度有限时，其表达精度往往不够，如需满足一定的精度要求，应增加搜索空间的维度，这势必影响进化过程的速度	用于离散问题，如特征选择、集成约简等
线性编码	实数编码	方便定义，克服了二进制编码的缺点；但需重新设计操作符	用于连续函数优化问题，例如权重调整或参数优化等
	符号编码	符号编码避免数值编码对优化过程造成不必要的干扰，和实际求解问题相关，方便定义；需调整算子更新策略	用于路径优化问题、基因表达问题
非线性编码	树形编码等	避免线性编码表达的单一性；定义更新过程稍显复杂	用于各种组合优化方案

参数	所用机器学习方法	所用优化方法
神经元个数和权重	CNN、ELM、FLN	ACO、PSO、DE 、GA
正则化参数	SVM	PSO、DE
结构学习	BN、ID3、KNN、OPFC	PSO、GA、BBO、BA、firefly、GSA、HS