基于XGBoost的特征选择算法

doi:10.11959/j.issn.1000-436x.2019154

Abstract

Abstract:

Feature selection in classification has always been an important but difficult problem.This kind of problem requires that feature selection algorithms can not only help classifiers to improve the classification accuracy,but also reduce the redundant features as much as possible.Therefore,in order to solve feature selection in the classification problems better,a new wrapped feature selection algorithm XGBSFS was proposed.The thought process of building trees in XGBoost was used for reference,and the importance of features from three importance metrics was measured to avoid the limitation of single importance metric.Then the improved sequential floating forward selection (ISFFS) was applied to search the feature subset so that it had high quality.Compared with the experimental results of eight datasets in UCI,the proposed algorithm has good performance.

Key words: feature selection, XGBoost, sequential floating forward selection

CLC Number:

TP18

Zhanshan LI, Zhaogeng LIU. Feature selection algorithm based on XGBoost[J]. Journal on Communications, 2019, 40(10): 101-108.

Figures/Tables 13

References 26

[1]	ZHOU T , LU H L , WANG W W ,et al. GA-SVM based feature selection and parameter optimization in hospitalization expense modeling[J]. Applied Soft Computing, 2019(75): 323-332.
[2]	LI J D , CHENG K W , WANG S H ,et al. Feature selection:a data perspective[J]. ACM Computing Surveys, 2017,50(6): 1-45.
[3]	周志华 . 机器学习[M]. 北京: 清华大学出版社, 2016.
	ZHOU Z H . Machine learning[M]. Beijing: Tsinghua University PressPress, 2016.
[4]	LIU H , YU L . Toward integrating feature selection algorithms for classification and clustering[J]. IEEE Transactions on Knowledge and Data Engineering, 2005,17(4): 491-502.
[5]	ALMUALLIM H , DIETTERICH T G . Learning boolean concepts in the presence of many irrelevant features[J]. Artificial Intelligence, 1994,69(1-2): 279-305.
[6]	KAMATH U , DE J K , SHEHU A . Effective automated feature construction and selection for classification of biological sequences[J]. Plos One, 2014,9(7):e99982
[7]	GUYON I , ELISSEEFF A . An introduction to variable and feature selection[J]. Journal of Machine Learning Research, 2003,3(6): 1157-1182.
[8]	ZAKERI A , HOKMABADI A . Efficient feature selection method using real-valued grasshopper optimization algorithm[J]. Expert Systems with Applications, 2019(119): 61-72.
[9]	XUE B , ZHANG M , BROWNE W N ,et al. A survey on evolutionary computation approaches to feature selection[J]. IEEE Transactions on Evolutionary Computation, 2016,20(4): 606-626.
[10]	GHAEMI M , FEIZI-DERAKHSHI M R . Feature selection using forest optimization algorithm[J]. Pattern Recognition, 2016(60): 121-129.
[11]	ZHANG Y , SONG X , GONG D . A return-cost-based binary firefly algorithm for feature selection[J]. Information Sciences, 2017,418-419: 561-574.
[12]	CHEN T , GUESTRIN C . XGBoost:a scalable tree boosting system[C]// ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016: 785-794.
[13]	FRIEDMAN J , HASTIE T , TIBSHIRANI R . Special invited paper.additive logistic regression:a statistical view of boosting[J]. The Annals of Statistics, 2000,28(2): 337-374.
[14]	LU Y , LIU L , LUAN S ,et al. The diagnostic value of texture analysis in predicting WHO grades of meningiomas based on ADC maps:an attempt using decision tree and decision forest[J]. European Radiology, 2019,29(3): 1318-1328.
[15]	PANG L , WANG J , ZHAO L ,et al. A novel protein subcellular localization method with CNN-XGBoost model for alzheimer’s disease[J]. Frontiers in Genetics, 2019(9): 1-7.
[16]	PAN B . Application of XGBoost algorithm in hourly PM2.5 concentration prediction[J]. IOP Conference Series:Earth and Environmental Science, 2018(113): 012-127.
[17]	MACEDO F , ROSáRIO OLIVEIRA M , PACHECO A ,et al. Theoretical foundations of forward feature selection methods based on mutual information[J]. Neurocomputing, 2019(325): 67-89.
[18]	VERGARA J R ， ESTéVEZ P A . A review of feature selection methods based on mutual information[J]. Neural Computing and Applications, 2014,24(1): 175-186.
[19]	SHI F , YAO Y , BIN Y ,et al. Computational identification of deleterious synonymous variants in human genomes using a feature-based approach[J]. BMC Medical Genomics, 2019,12(1): 81-88.
[20]	XUE B , ZHANG M , BROWNE W N . Novel initialisation and updating mechanisms in PSO for feature selection in classification[J]. Applications of Evolutionary Computation, 2013(7835): 428-438.
[21]	GHOSH A , DATTA A , GHOSH S . Self-adaptive differential evolution for feature selection in hyperspectral image data[J]. Applied Soft Computing, 2013,13(4): 1969-1977.
[22]	DUA D , EFI K T . UCI machine learning repository[J]. The UCI Machine Learning Repository, 2019.
[23]	BROWN G , POCOCK A C , ZHAO M J ,et al. Conditional likelihood maximisation:a unifying framework for information theoretic feature selection[J]. Journal of Machine Learning Research, 2012(13): 27-66.
[24]	CADENAS J M , GARRIDO M C ， MARTíNEZ R . Feature subset selection filter–wrapper based on low quality data[J]. Expert Systems with Applications, 2013,40(16): 6241-6252.
[25]	MAFARJA M M , MIRJALILI S . Hybrid whale optimization algorithm with simulated annealing for feature selection[J]. Neurocomputing, 2017(260): 302-312.
[26]	EMARY E , ZAWBAA H M , HASSANIEN A E . Binary grey wolf optimization approaches for feature selection[J]. Neurocomputing, 2016(172): 371-381.

Metrics

Recommended 0

No Suggested Reading articles found!

数据集	特征数/个	样本数/个	分类数/类
Wine	13	178	3
Vehicle	18	846	4
Segmentation	19	2 310	7
Ionosphere	34	351	2
Sonar	60	208	2
LSVT	310	126	2
CNAE-9	856	1 080	9
Arcene	10 000	200	2

SFFS策略	特征重要性判别描述
SFFS1	仅使用FScore进行序列浮动前向搜索
SFFS2	仅使用AverageGain进行序列浮动前向搜索
SFFS3	仅使用AverageCover进行序列浮动前向搜索
ISFFS	应用本文提出的改进的序列浮动前向搜索

算法名称	描述/发表年份
Rc-BBFA¹	基于收益成本的萤火虫算法的特征选择^[11]/2017
FSFOA²	基于森林优化算法的特征选择^[10]/2016
JMI³	基于联合互信息的特征选择^[23]/2012
MIM⁴	基于互信息最大化的特征选择^[23]/2012

数据集	SFFS1	SFFS2	SFFS3	ISFFS
Wine	91.67	88.34	84.26	97.97
Vehicle	71.48	69.17	70.12	75.95
Segmentation	95.09	96.64	95.17	96.67
Ionosphere	94.15	94.15	93.59	96.42
Sonar	84.76	89.21	90.32	95.71
LSVT	86.05	87.89	88.42	91.32
CNAE-9	89.54	91.57	87.78	91.88
Arcene	95.67	95.33	94.00	97.83

数据集	SFFS1	SFFS2	SFFS3	ISFFS
Wine	63.85	71.54	59.23	60.77
Vehicle	59.45	61.11	61.11	63.89
Segmentation	52.63	66.31	62.63	67.37
Ionosphere	85.00	83.82	81.47	81.77
Sonar	87.67	87.00	85.83	82.17
LSVT	98.58	98.23	98.10	98.61
CNAE-9	94.19	94.42	92.22	92.73
Arcene	99.87	99.89	99.89	99.86

Feature selection algorithm based on XGBoost

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 26

Related Articles 14

Metrics

Recommended 0

Wine	CA	DR	验证方法
XGBSFS	97.97%	60.77%	70%为训练集，30%为验证集
Rc-BBFA	99.66%	38.46%	70%为训练集，30%为验证集
FSFOA	98.07%	50.00%	70%为训练集，30%为验证集
JMI	92.41%	58.46%	70%为训练集，30%为验证集
MIM	94.81%	74.61%	70%为训练集，30%为验证集

[1]	Shuangyan YI, Yongsheng LIANG, Jingjing LU, Wei LIU, Tao HU, Zhenyu HE. Robust feature selection method via joint low-rank reconstruction and projection reconstruction [J]. Journal on Communications, 2023, 44(3): 209-219.
[2]	Xiaorong ZHU,Peipei ZHANG. Fault detection and diagnosis method for heterogeneous wireless network based on GAN [J]. Journal on Communications, 2020, 41(8): 110-119.
[3]	Yonghao LI, Liang HU, Ping ZHANG, Wanfu GAO. Multi-label feature selection based on dynamic graph Laplacian [J]. Journal on Communications, 2020, 41(12): 47-59.
[4]	Li ZHANG,Cong WANG. Multi-label feature selection algorithm based on joint mutual information of max-relevance and min-redundancy [J]. Journal on Communications, 2018, 39(5): 111-122.
[5]	Zhen ZHANG,Peng WEI,Yufeng LI,Julong LAN,Ping XU,Bo CHEN. Feature selection algorithm based on improved particle swarm joint taboo search [J]. Journal on Communications, 2018, 39(12): 60-68.
[6]	Yong WANG,Huiyi ZHOU,Hao FENG,Miao YE,Wenlong KE. Network traffic classification method basing on CNN [J]. Journal on Communications, 2018, 39(1): 14-23.
[7]	Yu-xiang ZHANG,Yu SUN,Jia-hai YANG,Da-lei ZHOU,Xiang-fei MENG,Chun-jing XIAO. Feature importance analysis for spammer detection in Sina Weibo [J]. Journal on Communications, 2016, 37(8): 24-33.
[8]	Xiao-nian WU,Xiao-jin PENG,Yu-yang YANG,Kun FANG. Two-level feature selection method based on SVM for intrusion detection [J]. Journal on Communications, 2015, 36(4): 19-26.
[9]	Chun-hua JU,Fu-guang BAO. Research on a multidimensional personalized recommendation model based on a situation and characteristics of the users [J]. Journal on Communications, 2012, 33(Z1): 17-27.
[10]	Liang CHEN,Jian GONG. Fast application-level traffic classification using NetFlow records [J]. Journal on Communications, 2012, 33(1): 145-152.
[11]	Tie-ming CHEN,Ji-xia MA,Yi-guang XUAN,Jia-mei CAI. Quick feature selection method and its application on network intrusion detection [J]. Journal on Communications, 2010, 31(9A): 233-238.
[12]	Ying ZHUO,Chun-ye GONG,Zheng-hu GONG. Research and implementation of network transmission situation awareness [J]. Journal on Communications, 2010, 31(9): 55-64.
[13]	Bo WANG,Yan JIA,Shu-qiang YANG,Bin ZHOU. Feature selection algorithm for uncertain text classification [J]. Journal on Communications, 2009, 30(8): 32-38.
[14]	Yang LI,Li GUO,Tian-bo LU,Zhi-hong TIAN. Research on performance optimizations for TCM-KNN network anomaly detection algorithm [J]. Journal on Communications, 2009, 30(7): 13-19.