基于Spark和三路交互信息的并行深度森林算法

doi:10.11959/j.issn.1000-436x.2023143

Abstract

Abstract:

To address issues such as excessive redundancy and irrelevant features, long class vectors, slow model convergence, and low efficiency of parallel training in parallel deep forests, a parallel deep forest algorithm based on Spark and three-way interactive information was proposed.Firstly, a feature selection based on feature interaction (FSFI) strategy was proposed to filter the original features and eliminate irrelevant and redundant features.Secondly, a multi-granularity vector elimination (MGVE) strategy was proposed, which fused similar class vectors and shortened the class vector length.Subsequently, the cascade forest feature enhancement (CFFE) strategy was proposed to improve the utilization of information and accelerate the convergence speed of the model.Finally, a multi-level load balancing (MLB) strategy was proposed, combined with the Spark framework, to improve the parallelization efficiency through adaptive sub-forest division and heterogeneous skew data partitioning.Experimental results demonstrate that the proposed algorithm significantly improves the model classification effect and reduces the parallelization training time.

Key words: Spark framework, parallel deep forest algorithm, feature selection, multilevel load balancing

CLC Number:

TN92

Yimin MAO, Zhan ZHOU, Zhigang CHEN. Parallel deep forest algorithm based on Spark and three-way interactive information[J]. Journal on Communications, 2023, 44(8): 228-240.

Figures/Tables 9

References 16

[1]	ZHOU Z , FENG J . Deep forest:towards an alternative to deep neural networks[J]. arXiv Preprint,arXiv:1702.08835, 2017.
[2]	戴瑾, 王天宇, 王少尉 . 基于深度森林的网络流量分类方法[J]. 国防科技大学学报, 2020,42(4): 30-34.
	DAI J , WANG T Y , WANG S W . Network traffic classification method based on deep forest[J]. Journal of National University of Defense Technology, 2020,42(4): 30-34.
[3]	牛振东, 石鹏飞, 朱一凡 ,等. 基于深度随机森林的商品类超短文本分类研究[J]. 北京理工大学学报, 2021,41(12): 1277-1285.
	NIU Z D , SHI P F , ZHU Y F ,et al. Research on classification of commodity ultra-short text based on deep random forest[J]. Transactions of Beijing Institute of Technology, 2021,41(12): 1277-1285.
[4]	邵怡韦, 陈嘉宇, 林翠颖 ,等. 小训练样本下齿轮箱故障诊断:一种基于改进深度森林的方法[J]. 航空学报, 2022,43(8): 112-126.
	SHAO Y W , CHEN J Y , LIN C Y ,et al. Gearbox fault diagnosis with small training samples:an improved deep forest based method[J]. Acta Aeronautica et Astronautica Sinica, 2022,43(8): 112-126.
[5]	李璐, 杜兰, 何浩男 ,等. 基于深度森林的多级特征融合SAR目标识别[J]. 电子与信息学报, 2021,43(3): 606-614.
	LI L , DU L , HE H N ,et al. Multi-level feature fusion SAR automatic target recognition based on deep forest[J]. Journal of Electronics ＆Information Technology, 2021,43(3): 606-614.
[6]	卢喜东, 段哲民, 钱叶魁 ,等. 一种基于深度森林的恶意代码分类方法[J]. 软件学报, 2020,31(5): 1454-1464.
	LU X D , DUAN Z M , QIAN Y K ,et al. Malicious code classification method based on deep forest[J]. Journal of Software, 2020,31(5): 1454-1464.
[7]	毛伊敏, 甘德瑾, 廖列法 ,等. 基于Spark框架和ASPSO的并行划分聚类算法[J]. 通信学报, 2022,43(3): 148-163.
	MAO Y M , GAN D J , LIAO L F ,et al. Parallel division clustering algorithm based on Spark framework and ASPSO[J]. Journal on Communications, 2022,43(3): 148-163.
[8]	LI X B , SUN Y , ZHANG F Z ,et al. Potential off-grid user prediction system based on Spark[J]. ZTE Communications, 2019,17(2): 26-37.
[9]	LIU Z P , SU N , QIN Y W ,et al. A deep random forest model on Spark for network intrusion detection[J]. Mobile Information Systems, 2020,2020: 1-16.
[10]	CHEN Z X , WANG T , CAI H B ,et al. BLB-gcForest:a high-performance distributed deep forest with adaptive sub-forest splitting[J]. IEEE Transactions on Parallel and Distributed Systems, 2022,33(11): 3141-3152.
[11]	庞明 . 新型深度森林模型的研究[D]. 南京:南京大学, 2020.
	PANG M . Study on new deep forest model[D]. Nanjing:Nanjing University, 2020.
[12]	李占山, 杨云凯, 张家晨 . 基于熵权法的过滤式特征选择算法[J]. 东北大学学报(自然科学版), 2022,43(7): 921-929.
	LI Z S , YANG Y K , ZHANG J C . Filtering feature selection algorithm based on entropy weight method[J]. Journal of Northeastern University (Natural Science), 2022,43(7): 921-929.
[13]	顾翔元, 郭继昌, 李重仪 ,等. 基于对称不确定性和三路交互信息的特征子集选择算法[J]. 天津大学学报(自然科学与工程技术版), 2021,54(2): 214-220.
	GU X Y , GUO J C , LI C Y ,et al. Feature subset selection algorithm based on symmetric uncertainty and three-way interaction information[J]. Journal of Tianjin University (Science and Technology), 2021,54(2): 214-220.
[14]	肖利军, 郭继昌, 顾翔元 . 一种采用冗余性动态权重的特征选择算法[J]. 西安电子科技大学学报, 2019,46(5): 155-161.
	XIAO L J , GUO J C , GU X Y . Algorithm for selection of features based on dynamic weights using redundancy[J]. Journal of Xidian University, 2019,46(5): 155-161.
[15]	张俐, 王枞, 郭文明 . 利用近似马尔可夫毯的最大相关最小冗余特征选择算法[J]. 西安交通大学学报, 2018,52(10): 141-145.
	ZHANG L , WANG C , GUO W M . A feature selection algorithm for maximum relevance minimum redundancy using approximate Markov blanket[J]. Journal of Xi’an Jiaotong University, 2018,52(10): 141-145.
[16]	杨晓晖, 张圣昌 . 基于多粒度级联孤立森林算法的异常检测模型[J]. 通信学报, 2019,40(8): 133-142.
	YANG X H , ZHANG S C . Anomaly detection model based on multi-grained cascade isolation forest algorithm[J]. Journal on Communications, 2019,40(8): 133-142.

Metrics

Recommended 0

No Suggested Reading articles found!

节点类型	主机名	IP地址
Master	M	192.168.110.1
Slaver	S₁	192.168.110.2
Slaver	S₂	192.168.110.3
Slaver	S₃	192.168.110.4
Slaver	S₄	192.168.110.5
Slaver	S₅	192.168.110.6
Slaver	S₆	192.168.110.7

数据集	样本数/条	特征数/种	数据特点
Farm Ads	4413	54 877	样本少特征多
Susy	5 000 000	18	样本多特征少
Connect-4	67 557	42	样本特征适中
FMA	106 574	518	样本特征适中

数据集	算法策略	加速比	准确率
	所有策略	4.53(+0.79)	96.74 %(+3.23%)
	仅FSFI	3.91(+0.17)	95.39 %(+1.88%)
Farm Ads	仅MGVE	3.86(+0.12)	93.83 %(+0.32%)
	仅CFFE	3.81(+0.06)	94.82 %(+1.31%)
	仅MBL	4.31(+0.57)	93.62 %(+0.11%)
	无任何策略	3.74(±0)	93.51%(±0)
	所有策略	5.24(+1.23)	98.06 %(+3.91%)
	仅FSFI	4.26(+0.25)	95.71 %(+1.56%)
Susy	仅MGVE	4.15(+0.14)	94.54 %(+0.39%)
	仅CFFE	4.09(+0.08)	96.53%(+2.38%)
	仅MBL	4.94(+0.93)	94.43 %(+0.28%)
	无任何策略	4.01(±0)	94.15 %(±0)
	所有策略	4.97(+0.99)	97.4%(+3.52%)
	仅FSFI	4.20(+0.22)	95.51 %(+1.63%)
Connect-4	仅MGVE	4.13(+0.15)	94.26 %(+0.38%)
	仅CFFE	4.06(+0.08)	95.82 %(+1.94%)
	仅MBL	4.69(+0.71)	94.07 %(+0.19%)
	无任何策略	3.98(±0)	93.88 %(±0)
	所有策略	4.96(+1.04)	98.75 %(+4.03%)
	仅FSFI	4.15(+0.23)	96.94 %(+2.22%)
FMA	仅MGVE	4.08(+0.16)	95.32%(+0.6%)
	仅CFFE	4.01(+0.09)	96.53 %(+1.81%)
	仅MBL	4.69(+0.77)	94.93 %(+0.21%)
	无任何策略	3.92(±0)	94.72%(±0)

Parallel deep forest algorithm based on Spark and three-way interactive information

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 16

Related Articles 15

Metrics

Recommended 0

[1]	Shuangyan YI, Yongsheng LIANG, Jingjing LU, Wei LIU, Tao HU, Zhenyu HE. Robust feature selection method via joint low-rank reconstruction and projection reconstruction [J]. Journal on Communications, 2023, 44(3): 209-219.
[2]	Yimin MAO, Dejin GAN, Liefa LIAO, Zhigang CHEN. Parallel division clustering algorithm based on Spark framework and ASPSO [J]. Journal on Communications, 2022, 43(3): 148-163.
[3]	Yonghao LI, Liang HU, Ping ZHANG, Wanfu GAO. Multi-label feature selection based on dynamic graph Laplacian [J]. Journal on Communications, 2020, 41(12): 47-59.
[4]	Zhanshan LI, Zhaogeng LIU. Feature selection algorithm based on XGBoost [J]. Journal on Communications, 2019, 40(10): 101-108.
[5]	Li ZHANG,Cong WANG. Multi-label feature selection algorithm based on joint mutual information of max-relevance and min-redundancy [J]. Journal on Communications, 2018, 39(5): 111-122.
[6]	Zhen ZHANG,Peng WEI,Yufeng LI,Julong LAN,Ping XU,Bo CHEN. Feature selection algorithm based on improved particle swarm joint taboo search [J]. Journal on Communications, 2018, 39(12): 60-68.
[7]	Yong WANG,Huiyi ZHOU,Hao FENG,Miao YE,Wenlong KE. Network traffic classification method basing on CNN [J]. Journal on Communications, 2018, 39(1): 14-23.
[8]	Yu-xiang ZHANG,Yu SUN,Jia-hai YANG,Da-lei ZHOU,Xiang-fei MENG,Chun-jing XIAO. Feature importance analysis for spammer detection in Sina Weibo [J]. Journal on Communications, 2016, 37(8): 24-33.
[9]	Xiao-nian WU,Xiao-jin PENG,Yu-yang YANG,Kun FANG. Two-level feature selection method based on SVM for intrusion detection [J]. Journal on Communications, 2015, 36(4): 19-26.
[10]	Chun-hua JU,Fu-guang BAO. Research on a multidimensional personalized recommendation model based on a situation and characteristics of the users [J]. Journal on Communications, 2012, 33(Z1): 17-27.
[11]	Liang CHEN,Jian GONG. Fast application-level traffic classification using NetFlow records [J]. Journal on Communications, 2012, 33(1): 145-152.
[12]	Tie-ming CHEN,Ji-xia MA,Yi-guang XUAN,Jia-mei CAI. Quick feature selection method and its application on network intrusion detection [J]. Journal on Communications, 2010, 31(9A): 233-238.
[13]	Ying ZHUO,Chun-ye GONG,Zheng-hu GONG. Research and implementation of network transmission situation awareness [J]. Journal on Communications, 2010, 31(9): 55-64.
[14]	Bo WANG,Yan JIA,Shu-qiang YANG,Bin ZHOU. Feature selection algorithm for uncertain text classification [J]. Journal on Communications, 2009, 30(8): 32-38.
[15]	Yang LI,Li GUO,Tian-bo LU,Zhi-hong TIAN. Research on performance optimizations for TCM-KNN network anomaly detection algorithm [J]. Journal on Communications, 2009, 30(7): 13-19.