基于深度学习的系统日志异常检测研究

doi:10.11959/j.issn.2096-109x.2019055

Abstract

Abstract:

The system log reflects the running status of the system and records the activity information of specific events in the system.Therefore,the rapid and accurate detection of the system abnormal log is important to the security and stability of the system.A log anomaly detection algorithm based on GRU neural network is proposed.Log parsing is implemented based on log key technology.Log anomaly detection is realized by using anomaly detection model of execution path and anomaly detection model of parameter value.The system has the advantages of less parameters and faster training.It improves the running speed while achieving higher detection accuracy,and is suitable for log analysis of large information systems.

Key words: log anomaly detection, deep learning, GRU neural network

CLC Number:

TP390

Yidong WANG, Peishun LIU, Gbin WAN. Research on system log anomaly detection based on deep learning[J]. Chinese Journal of Network and Information Security, 2019, 5(5): 105-118.

Figures/Tables 15

References 12

[1]	XU W , HUANG L , FOX A ,et al. Detecting large-scale system problems by mining console logs[C]// ACM SIGOPS 22nd symposium on Operating systems principles. 2009: 117-132.
[2]	YU X , JOSHI P , XU J ,et al. CloudSeer:workflow monitoring of cloud infrastructures via interleaved logs[J]. ACM Sigarch Computer Architecture News, 2016,44(2): 489-502.
[3]	DU M , LI F , ZHENG G ,et al. Deeplog:anomaly detection and diagnosis from system logs through deep learning[C]// 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017: 1285-1298.
[4]	JURAFSKY D . Speech ＆ language processing[M]. Pearson Education India, 2000: 35-61.
[5]	DU M , LI F . spell:Streaming parsing of system event logs[C]// IEEE 16th International Conference on Data Mining (ICDM). 2016: 859-864.
[6]	ZHU J , HE S , LIU J ,et al. Tools and benchmarks for automated log parsing[J]. arXiv preprint arXiv:1811.03509, 2018.
[7]	HE P , ZHU J , HE S ,et al. An evaluation study on log parsing and its use in log mining[C]// 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). 2016: 654-661.
[8]	KINGMA D P , BA J . Adam:a method for stochastic optimization[J]. arXiv preprint arXiv:1412.6980, 2014.
[9]	MC MAHAN H B , HOLT G , SCULLEY D ,et al. Ad click prediction:a view from the trenches[C]// The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2013: 1222-1230.
[10]	HE S , ZHU J , HE P ,et al. Experience report:system log analysis for anomaly detection[C]// IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). 2016: 207-218.
[11]	XU W , HUANG L , FOX A ,et al. Online system problem detection by mining patterns of console logs[C]// Ninth IEEE International Conference on Data Mining. 2009: 588-597.
[12]	OLINER A , STEARLEY J . What supercomputers say:a study of five system logs[C]// 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07). 2007: 575-584.

Metrics

Recommended 0

No Suggested Reading articles found!

	训练集（本文方法）	测试集
正常会话数	4 855	558 224
异常会话数	0	16 838
Logkey数	29	29

算法	假阳性	假阴性	准确率
PCA	281	5 429	99.00%
IM	2 131	1 226	99.42%
DeepLog	839	615	99.75%
GRU	857	624	99.74%

方法	日志条数	总时间/s	平均时间/s	时间成本
DeepLog	11 175 629	11 399	1.02	1.00
GRU （本文方法）	11 175 629	9 499	0.85	0.833

模型情况	真阳性	假阴性	召回率
执行路径异常检测模型	5	17	22.70%
同时使用两种模型	22	0	100%

模型情况	假阳性	假阴性	真阳性
N(10%)	804 961	0	155 613
N(20%)	616 215	0	123 539
Y(10%)	20 619	0	155 613
Y(20%)	14 648	0	123 539

Research on system log anomaly detection based on deep learning

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 12

Related Articles 15

Metrics

Recommended 0

[1]	Xiaomeng LI, Daidou GUO, Xunfang ZHUO, Heng YAO, Chuan QIN. Carrier-independent screen-shooting resistant watermarking based on information overlay superimposition [J]. Chinese Journal of Network and Information Security, 2023, 9(3): 135-149.
[2]	Rongna XIE, Zhuhong MA, Zongyu LI, Ye TIAN. Encrypted traffic classification method based on convolutional neural network [J]. Chinese Journal of Network and Information Security, 2022, 8(6): 84-91.
[3]	Dengyong ZHANG, Huang WEN, Feng LI, Peng CAO, Lingyun XIANG, Gaobo YANG, Xiangling DING. Image inpainting forensics method based on dual branch network [J]. Chinese Journal of Network and Information Security, 2022, 8(6): 110-122.
[4]	Jiaying LIN, Wenbo ZHOU, Weiming ZHANG, Nenghai YU. Lip forgery detection via spatial-frequency domain combination [J]. Chinese Journal of Network and Information Security, 2022, 8(6): 146-155.
[5]	Jinyin CHEN, Changan WU, Haibin ZHENG. Novel defense based on softmax activation transformation [J]. Chinese Journal of Network and Information Security, 2022, 8(2): 48-63.
[6]	Baolin QIU, Ping YI. Adversarial examples defense method based on multi-dimensional feature maps knowledge distillation [J]. Chinese Journal of Network and Information Security, 2022, 8(2): 88-99.
[7]	Lijuan LI, Man LI, Hongjun BI, Huachun ZHOU. Multi-type low-rate DDoS attack detection method based on hybrid deep learning [J]. Chinese Journal of Network and Information Security, 2022, 8(1): 73-85.
[8]	Zhongyuan QIN, Zhaoxiang HE, Tao LI, Liquan CHEN. Adversarial example defense algorithm for MNIST based on image reconstruction [J]. Chinese Journal of Network and Information Security, 2022, 8(1): 86-94.
[9]	Deqing ZOU, Xiang LI, Minhuan HUANG, Xiang SONG, Hao LI, Weiming LI. Intelligent vulnerability detection system based on graph structured source code slice [J]. Chinese Journal of Network and Information Security, 2021, 7(5): 113-122.
[10]	Zhenglong WANG, Baowen ZHANG. Survey of generative adversarial network [J]. Chinese Journal of Network and Information Security, 2021, 7(4): 68-85.
[11]	Binglong LI, Jinlong TONG, Yu ZHANG, Yifeng SUN, Qingxian WANG, Chaowen CHANG. Auto forensic detecting algorithms of malicious code fragment based on TensorFlow [J]. Chinese Journal of Network and Information Security, 2021, 7(4): 154-163.
[12]	Qingyin TAN, Yingming ZENG, Ye HAN, Yijing LIU, Zheli LIU. Survey on backdoor attacks targeted on neural network [J]. Chinese Journal of Network and Information Security, 2021, 7(3): 46-58.
[13]	Luhui YANG,Huiwen BAI,Guangjie LIU,Yuewei DAI. Lightweight malicious domain name detection model based on separable convolution [J]. Chinese Journal of Network and Information Security, 2020, 6(6): 112-120.
[14]	Ximeng LIU,Lehui XIE,Yaopeng WANG,Xuru LI. Adversarial attacks and defenses in deep learning [J]. Chinese Journal of Network and Information Security, 2020, 6(5): 36-53.
[15]	Sijia DU,Haining YU,Hongli ZHANG. Survey of text classification methods based on deep learning [J]. Chinese Journal of Network and Information Security, 2020, 6(4): 1-13.