融合Focal Loss的网络威胁情报实体抽取

doi:10.11959/j.issn.1000-436x.2022132

Abstract

Abstract:

Cyber threat intelligence contains a wealth of knowledge of threat behavior.Timely analysis and process of threat intelligence can promote the transformation of defense from passive to active.Nowadays, most threat intelligence that exists in the form of natural language texts contains a large amount of unstructured data, which needs to be converted into structured data for subsequent processing using entity extraction methods.However, since threat intelligence contains numerous terminology such as vulnerability names, malware and APT organizations, and the distribution of entities are extremely unbalanced, the performance of extraction methods in general field are severely limited when applied to threat intelligence.Therefore, an entity extraction model integrated with Focal Loss was proposed, which improved the cross-entropy loss function and balanced sample distribution by introducing balance factor and modulation coefficient.In addition, for the problem that threat intelligence had a complex structure and a wide range of sources, and contained a large number of professional words, token and character features were added to the model, which effectively improved OOV (out of vocabulary) problem in threat intelligence.Experiment results show that compared with existing mainstream model BiLSTM and BiLSTM-CRF, the F1 scores of the proposed model is increased by 7.07% and 4.79% respectively, which verifies the effectiveness of introducing Focal Loss and character features.

Key words: cyber security, threat intelligence, entity extraction, label imbalance

CLC Number:

TP391

Yuanbo GUO, Yongfei LI, Qingli CHEN, Chen FANG, Yangyang HU. Fusion of Focal Loss’s cyber threat intelligence entity extraction[J]. Journal on Communications, 2022, 43(7): 85-92.

Figures/Tables 11

模型	P	R	F1
BiLSTM	$76 . 32 %$	31.50%	44.59%
BiLSTM-CRF	71.77%	34.80%	46.87%
(CNN-BiLSTM)-BiLSTM-CRF_FL	72.06%	$40 . 26 %$	$51 . 66 %$

模型	损失函数	P	R	F1
(CNN-BiLSTM)-BiLSTM-CRF_FL	Focal Loss	72.06%	$40 . 26 %$	$51 . 66 %$
(CNN-BiLSTM)-BiLSTM-CRF	BCE	$73 . 30 %$	39.22%	51.10%

模型	字符特征	P	R	F1
(CNN-BiLSTM)-BiLSTM-CRF_FL	CNNChar+BiLSTMChar+Word	72.06%	$40 . 26 %$	$51 . 66 %$
(BiLSTM)-BiLSTM-CRF_FL	BiLSTMChar+Word	$72 . 28 %$	39.45%	51.04%
(CNN)-BiLSTM-CRF_FL	CNNChar +Word	68.46%	38.87%	49.59%
BiLSTM-CRF_FL	Word2Vec	71.45%	35.96%	47.84%

实体类型	BiLSTM-CRF_FL			(BiLSTM)-BiLSTM-CRF_FL			(CNN)-BiLSTM-CRF_FL			(CNN-BiLSTM)-BiLSTM-CRF_FL
实体类型	P	R	F1	P	R	F1	P	R	F1	P	R	F1
TIME	83.79%	75.96%	79.68%	88.42%	59.59%	60.00%	86.40%	77.71%	81.82%	89.11%	79.14%	$83 . 83 %$
VIRUS	51.11%	19.26%	27.98%	59.59%	29.12%	$39 . 12 %$	50.06%	22.83%	31.36%	54.47%	27.75%	36.77%
SOFTWARE	35.81%	19.26%	25.05%	34.41%	21.53%	26.49%	32.64%	20.59%	25.25%	32.56%	22.36%	$26 . 51 %$
TYPE	54.47%	38.09%	44.83%	53.67%	38.84%	$45 . 07 %$	53.65%	35.62%	42.81%	55.81%	36.20%	43.92%
LOCATION	86.35%	83.13%	84.71%	87.03%	83.77%	85.37%	85.94%	83.51%	84.71%	86.45%	84.82%	$85 . 63 %$
ORG	61.37%	42.11%	49.95%	63.78%	44.42%	52.37%	62.52%	44.66%	52.10%	65.80%	44.63%	$53 . 19 %$
ATTACKER	63.87%	17.49%	27.46%	60.00%	17.33%	26.89%	58.57%	17.50%	26.95%	60.94%	20.05%	$30 . 17 %$
VERSION	0.25%	0.10%	0.14%	0.25%	0.10%	0.14%	0.50%	0.08%	0.14%	—	—	—
OS	50.98%	45.78%	48.24%	48.20%	44.47%	46.26%	47.83%	43.80%	45.73%	51.06%	46.30%	$48 . 56 %$
EVENT	35.47%	8.04%	13.11%	36.08%	9.46%	$14 . 99 %$	34.34%	9.37%	14.72%	30.34%	9.33%	14.27%
ATTACK	60.50%	17.83%	27.54%	59.89%	18.86%	28.69%	61.80%	20.11%	$30 . 35 %$	59.29%	18.58%	28.29%

模型	实体抽取结果
实例1	We reported these apps to[Microsoft]_ORGand they subsequently removed them from their[store]_SOFTWARE.
BiLSTM	We reported these apps to[Microsoft]_ORGand they subsequently removed[them]from their[store].
BiLSTM-CRF	We reported these apps to[Microsoft]_ORGand they subsequently removed them from their[store]_SOFTWARE.
(CNN-BiLSTM)-BiLSTM-CRF_FL	We reported these apps to[Microsoft]_ORGand they subsequently removed them from their[store]_SOFTWARE.
实例2	The malware ([Android.Doublehidden]_VIRUS) is localized in the Persian language, which aims to compromise Android apps.
BiLSTM	The malware ([Android.Doublehidden]) is localized in the Persian language, which aims to compromise[Android]_OS[apps].
BiLSTM-CRF	The malware ([Android.Doublehidden]) is localized in the Persian language, which aims to compromise[Android]_OS[apps].
(CNN-BiLSTM)-BiLSTM-CRF_FL	The malware ([Android.Doublehidden]_VIRUS) is localized in the Persian language, which aims to compromise[Android]_OSapps.
实例3	This command was then seen searching for all the computer objects in the Active Directory database with filter condition like server or 2003 or 7 (returning all[Windows Server]_OS,[Windows Server]_OS[2003]_VERSION, or[Windows]_OS[7]_VERSIONinstances).
BiLSTM	This command was then seen searching for all the computer objects in the Active Directory database with filter condition like server or 2003 or 7 (returning all[Windows]_OS[Server],[Windows]_OS[Server][2003], or[Windows]_OS[7]instances).
BiLSTM-CRF	This command was then seen searching for all the computer objects in the Active Directory database with filter condition like server or 2003 or 7 (returning all[Windows]_OS[Server],[Windows Server][2003], or[Windows]_OS[7]instances).
(CNN-BiLSTM)-BiLSTM-CRF_FL	This command was then seen searching for all the computer objects in the Active Directory database with filter condition like server or 2003 or 7 (returning all[Windows]_OS[Server],[Windows]_OS[Server][2003], or[Windows]_OS[7]instances).

References 21

[1]	DALY M K , . Advanced persistent threat[C]// Proceedings of 23rd Large Installation System Administration Conference. Berkeley:USENIX Association, 2009: 1-6.
[2]	MCMILLAN R . Definition:threat intelligence[R]. Garter Research, 2013.
[3]	李涛, 郭渊博, 琚安康 . 融合对抗主动学习的网络安全知识三元组抽取[J]. 通信学报, 2020,41(10): 80-91.
	LI T , GUO Y B , JU A K . Knowledge triple extraction in cybersecurity with adversarial active learning[J]. Journal on Communications, 2020,41(10): 80-91.
[4]	HOHENECKER P , MTUMBUKA F , KOCIJAN V ,et al. Systematic comparison of neural architectures and training approaches for open information extraction[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg:Association for Computational Linguistics, 2020: 8554-8565.
[5]	LIN T Y , GOYAL P , GIRSHICK R ,et al. Focal loss for dense object detection[C]// Proceedings of 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE Press, 2017: 2999-3007.
[6]	LEI J B , TANG B Z , LU X Q ,et al. A comprehensive study of named entity recognition in Chinese clinical text[J]. Journal of the American Medical Informatics Association, 2013,21(5): 808-814.
[7]	刘显敏, 李建中 . 基于键规则的 XML 实体抽取方法[J]. 计算机研究与发展, 2014,51(1): 64-75.
	LIU X M , LI J Z . Key-based method for extracting entities from XML data[J]. Journal of Computer Research and Development, 2014,51(1): 64-75.
[8]	MULWAD V , LI W J , JOSHI A ,et al. Extracting information about security vulnerabilities from web text[C]// Proceedings of 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology. Piscataway:IEEE Press, 2011: 257-260.
[9]	BRIDGES R A , HUFFER K M T , JONES C L ,et al. Cybersecurity automated information extraction techniques:drawbacks of current methods,and enhanced extractors[C]// Proceedings of 2017 16th IEEE International Conference on Machine Learning and Applications. Piscataway:IEEE Press, 2017: 437-442.
[10]	JONES C L , BRIDGES R A , HUFFER K M T ,et al. Towards a relation extraction framework for cyber-security concepts[C]// Proceedings of the 10th Annual Cyber and Information Security Research Conference.[S.l.:s.n.], 2015: 1-4.
[11]	HUANG Z , XU W , YU K . Bidirectional LSTM-CRF models for sequence tagging[J]. arXiv Preprint,arXiv:150801991, 2015.
[12]	SARHAN I , SPRUIT M . Open-CyKG:an open cyber threat intelligence knowledge graph[J]. Knowledge-Based Systems, 2021,233:107524.
[13]	ZHAO J , YAN Q B , LI J X ,et al. TIMiner:automatically extracting and analyzing categorized cyber threat intelligence from social data[J]. Computers ＆ Security, 2020,95:101867.
[14]	GASMI H , LAVAL J , BOURAS A . Information extraction of cybersecurity concepts:an LSTM approach[J]. Applied Sciences, 2019,9(19): 3945.
[15]	王伟平, 宁翔凯, 宋虹 ,等. iAES:面向网络安全博客的 IOC 自动抽取方法[J]. 计算机学报, 2021,44(5): 882-896.
	WANG W P , NING X K , SONG H ,et al. An indicator of compromise extraction method based on deep learning[J]. Chinese Journal of Computers, 2021,44(5): 882-896.
[16]	WU Y M , LIU Q J , LIAO X J ,et al. Price TAG:towards semi-automatically discovery tactics,techniques and procedures of E-commerce cyber threat intelligence[J]. IEEE Transactions on Dependable and Secure Computing, 2021,PP(99): 1.
[17]	MANIKANDAN R , MADGULA K , SAHA S . TeamDL at SemEval 2018 task 8:cybersecurity text analysis using convolutional neural network and conditional random fields[C]// Proceedings of the 12th International Workshop on Semantic Evaluation. Stroudsburg:Association for Computational Linguistics, 2018: 868-873.
[18]	MA X Z , HOVY E . End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF[J]. arXiv Preprint,arXiv:160301354, 2016.
[19]	FU M M , ZHAO X M , YAN Y H . HCCL at SemEval-2018 task 8:an end-to-end system for sequence labeling from cybersecurity reports[C]// Proceedings of the 12th International Workshop on Semantic Evaluation. Stroudsburg:Association for Computational Linguistics, 2018: 874-877.
[20]	CHIU J P C , NICHOLS E . Named entity recognition with bidirectional LSTM-CNNs[J]. arXiv Preprint,arXiv:1511.08308, 2015.
[21]	SANTOS C N D , ZADROZNY B . Learning character-level representations for part-of-speech tagging[C]// Proceedings of International Conference on Machine Learning. New York:ACM Press, 2014: 1818-1826.

Metrics

Recommended 0

No Suggested Reading articles found!

实体类型	释义	数量/个	样例
TIME	时间	146	Apparently, the breach started on September 6, 2021.
VIRUS	恶意脚本	232	Buckeye uses a variant of DoublePulsar.
SOFTWARE	合法软件	46	The group has succeed in comprising Microsoft Exchange.
TYPE	攻击类型	27	Conducting DdoS attacks on key historical dates is not new.
LOCATION	国家地区	185	The top 10 regions targeted by BEC scammers are led by the U.S..
ORG	组织厂商	181	Symantec will continue to monitor their activities and respond in kind.
ATTACKER	攻击者	195	Lazarus was subsequently implicated in the WannaCry ransomware attacks.
VERSION	版本	5	Both Windows 7 and Windows Server 2008 ceased receiving support.
OS	操作系统	30	Researchers implemented ESP in Simple Gallery, a popular app on Android.
EVENT	攻击事件	39	Symantec believes that the attackers behind the Anthem breach are part of a highly resourceful cyberespionage group called Black Vine.
ATTACK	攻击方式	113	Leafminer attempts to infiltrate target networks through various means of intrusion:watering hole websites, vulnerability scans of network services

实体类型	正则表达式
CVE漏洞编码	CVE\-[0-9]{4}\-[0,9]{4,6}
Email地址	[a-zA-Z0-9_-]+@[a-zA-Z0-9_-]+(\.[a-zA-Z0-9_-]+)+
SHA1码	[a-f0-9]{40}\|[A-F0-9]{40}
MD5码	[a-f0-9]{32}\|[A-F0-9]{32}
IP地址	\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Fusion of Focal Loss’s cyber threat intelligence entity extraction

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 21

Related Articles 3

Metrics

Recommended 0

[1]	Ming XU, Baojun ZHANG, Yiming WU, Chenduo YING, Ning ZHENG. Cyber attacks and privacy protection distributed consensus algorithm for multi-agent systems [J]. Journal on Communications, 2023, 44(3): 117-127.
[2]	Xiuzhang YANG, Guojun PENG, Zichuan LI, Yangqi LYU, Side LIU, Chenguang LI. Research on entity recognition and alignment of APT attack based on Bert and BiLSTM-CRF [J]. Journal on Communications, 2022, 43(6): 58-70.
[3]	Hongbin ZHANG, Yan YIN, Dongmei ZHAO, Bin LIU. Network security situational awareness model based on threat intelligence [J]. Journal on Communications, 2021, 42(6): 182-194.