基于XGBoost的特征选择算法

doi:10.11959/j.issn.1000-436x.2019154

摘要/Abstract

摘要：

分类问题中的特征选择一直是一个重要而又困难的问题。这类问题中要求特征选择算法不仅能够帮助分类器提高分类准确率，同时还要尽可能地减少冗余特征。因此，为了在分类问题中更好地进行特征选择，提出了一种新型的包裹式特征选择算法XGBSFS。该算法借鉴极端梯度提升（XGBoost）算法中构建树的思想过程，通过从3个重要性度量的角度来衡量特征的重要性，避免单一重要性度量的局限性；然后通过改进的序列浮动前向搜索策略（ISFFS）搜索特征子集，使最终得到的特征子集有较高的质量。在8个UCI数据集的对比实验中表明，所提算法具有很好的性能。

关键词: 特征选择, 极端梯度提升, 序列浮动搜索策略

Abstract:

Feature selection in classification has always been an important but difficult problem.This kind of problem requires that feature selection algorithms can not only help classifiers to improve the classification accuracy,but also reduce the redundant features as much as possible.Therefore,in order to solve feature selection in the classification problems better,a new wrapped feature selection algorithm XGBSFS was proposed.The thought process of building trees in XGBoost was used for reference,and the importance of features from three importance metrics was measured to avoid the limitation of single importance metric.Then the improved sequential floating forward selection (ISFFS) was applied to search the feature subset so that it had high quality.Compared with the experimental results of eight datasets in UCI,the proposed algorithm has good performance.

Key words: feature selection, XGBoost, sequential floating forward selection

中图分类号:

TP18

李占山, 刘兆赓. 基于XGBoost的特征选择算法[J]. 通信学报, 2019, 40(10): 101-108.

Zhanshan LI, Zhaogeng LIU. Feature selection algorithm based on XGBoost[J]. Journal on Communications, 2019, 40(10): 101-108.

图/表 13

表1

表2

表3

表4

表5

表6

表7

表8

表9

表10

表11

表12

表13

参考文献 26

[1]	ZHOU T , LU H L , WANG W W ,et al. GA-SVM based feature selection and parameter optimization in hospitalization expense modeling[J]. Applied Soft Computing, 2019(75): 323-332.
[2]	LI J D , CHENG K W , WANG S H ,et al. Feature selection:a data perspective[J]. ACM Computing Surveys, 2017,50(6): 1-45.
[3]	周志华 . 机器学习[M]. 北京: 清华大学出版社, 2016.
	ZHOU Z H . Machine learning[M]. Beijing: Tsinghua University PressPress, 2016.
[4]	LIU H , YU L . Toward integrating feature selection algorithms for classification and clustering[J]. IEEE Transactions on Knowledge and Data Engineering, 2005,17(4): 491-502.
[5]	ALMUALLIM H , DIETTERICH T G . Learning boolean concepts in the presence of many irrelevant features[J]. Artificial Intelligence, 1994,69(1-2): 279-305.
[6]	KAMATH U , DE J K , SHEHU A . Effective automated feature construction and selection for classification of biological sequences[J]. Plos One, 2014,9(7):e99982
[7]	GUYON I , ELISSEEFF A . An introduction to variable and feature selection[J]. Journal of Machine Learning Research, 2003,3(6): 1157-1182.
[8]	ZAKERI A , HOKMABADI A . Efficient feature selection method using real-valued grasshopper optimization algorithm[J]. Expert Systems with Applications, 2019(119): 61-72.
[9]	XUE B , ZHANG M , BROWNE W N ,et al. A survey on evolutionary computation approaches to feature selection[J]. IEEE Transactions on Evolutionary Computation, 2016,20(4): 606-626.
[10]	GHAEMI M , FEIZI-DERAKHSHI M R . Feature selection using forest optimization algorithm[J]. Pattern Recognition, 2016(60): 121-129.
[11]	ZHANG Y , SONG X , GONG D . A return-cost-based binary firefly algorithm for feature selection[J]. Information Sciences, 2017,418-419: 561-574.
[12]	CHEN T , GUESTRIN C . XGBoost:a scalable tree boosting system[C]// ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016: 785-794.
[13]	FRIEDMAN J , HASTIE T , TIBSHIRANI R . Special invited paper.additive logistic regression:a statistical view of boosting[J]. The Annals of Statistics, 2000,28(2): 337-374.
[14]	LU Y , LIU L , LUAN S ,et al. The diagnostic value of texture analysis in predicting WHO grades of meningiomas based on ADC maps:an attempt using decision tree and decision forest[J]. European Radiology, 2019,29(3): 1318-1328.
[15]	PANG L , WANG J , ZHAO L ,et al. A novel protein subcellular localization method with CNN-XGBoost model for alzheimer’s disease[J]. Frontiers in Genetics, 2019(9): 1-7.
[16]	PAN B . Application of XGBoost algorithm in hourly PM2.5 concentration prediction[J]. IOP Conference Series:Earth and Environmental Science, 2018(113): 012-127.
[17]	MACEDO F , ROSáRIO OLIVEIRA M , PACHECO A ,et al. Theoretical foundations of forward feature selection methods based on mutual information[J]. Neurocomputing, 2019(325): 67-89.
[18]	VERGARA J R ， ESTéVEZ P A . A review of feature selection methods based on mutual information[J]. Neural Computing and Applications, 2014,24(1): 175-186.
[19]	SHI F , YAO Y , BIN Y ,et al. Computational identification of deleterious synonymous variants in human genomes using a feature-based approach[J]. BMC Medical Genomics, 2019,12(1): 81-88.
[20]	XUE B , ZHANG M , BROWNE W N . Novel initialisation and updating mechanisms in PSO for feature selection in classification[J]. Applications of Evolutionary Computation, 2013(7835): 428-438.
[21]	GHOSH A , DATTA A , GHOSH S . Self-adaptive differential evolution for feature selection in hyperspectral image data[J]. Applied Soft Computing, 2013,13(4): 1969-1977.
[22]	DUA D , EFI K T . UCI machine learning repository[J]. The UCI Machine Learning Repository, 2019.
[23]	BROWN G , POCOCK A C , ZHAO M J ,et al. Conditional likelihood maximisation:a unifying framework for information theoretic feature selection[J]. Journal of Machine Learning Research, 2012(13): 27-66.
[24]	CADENAS J M , GARRIDO M C ， MARTíNEZ R . Feature subset selection filter–wrapper based on low quality data[J]. Expert Systems with Applications, 2013,40(16): 6241-6252.
[25]	MAFARJA M M , MIRJALILI S . Hybrid whale optimization algorithm with simulated annealing for feature selection[J]. Neurocomputing, 2017(260): 302-312.
[26]	EMARY E , ZAWBAA H M , HASSANIEN A E . Binary grey wolf optimization approaches for feature selection[J]. Neurocomputing, 2016(172): 371-381.

数据集	特征数/个	样本数/个	分类数/类
Wine	13	178	3
Vehicle	18	846	4
Segmentation	19	2 310	7
Ionosphere	34	351	2
Sonar	60	208	2
LSVT	310	126	2
CNAE-9	856	1 080	9
Arcene	10 000	200	2

SFFS策略	特征重要性判别描述
SFFS1	仅使用FScore进行序列浮动前向搜索
SFFS2	仅使用AverageGain进行序列浮动前向搜索
SFFS3	仅使用AverageCover进行序列浮动前向搜索
ISFFS	应用本文提出的改进的序列浮动前向搜索

算法名称	描述/发表年份
Rc-BBFA¹	基于收益成本的萤火虫算法的特征选择^[11]/2017
FSFOA²	基于森林优化算法的特征选择^[10]/2016
JMI³	基于联合互信息的特征选择^[23]/2012
MIM⁴	基于互信息最大化的特征选择^[23]/2012

数据集	SFFS1	SFFS2	SFFS3	ISFFS
Wine	91.67	88.34	84.26	97.97
Vehicle	71.48	69.17	70.12	75.95
Segmentation	95.09	96.64	95.17	96.67
Ionosphere	94.15	94.15	93.59	96.42
Sonar	84.76	89.21	90.32	95.71
LSVT	86.05	87.89	88.42	91.32
CNAE-9	89.54	91.57	87.78	91.88
Arcene	95.67	95.33	94.00	97.83

数据集	SFFS1	SFFS2	SFFS3	ISFFS
Wine	63.85	71.54	59.23	60.77
Vehicle	59.45	61.11	61.11	63.89
Segmentation	52.63	66.31	62.63	67.37
Ionosphere	85.00	83.82	81.47	81.77
Sonar	87.67	87.00	85.83	82.17
LSVT	98.58	98.23	98.10	98.61
CNAE-9	94.19	94.42	92.22	92.73
Arcene	99.87	99.89	99.89	99.86