基于改进的Transformer编码器的中文命名实体识别

doi:10.11959/j.issn.2096-109x.2021041

Abstract

Abstract:

In order to improve the effect of chinese named entity recognition, a method based on the XLNETTransformer_P-CRF model was proposed, which used the Transformer_P encoder, improved the shortcomings of the traditional Transformer encoder that couldn’t obtain relative position information.Experiments show that the XLNET-Transformer_P-CRF model achieves 95.11%, 80.54%, 96.70%, and 71.46% F1 values on the four types of data sets: MSRA, OntoNotes4.0, Resume, and Weibo, which are all higher than other mainstream chinese NER model.

Key words: Chinese named entity recognition, Transformer encoder, relative position information

CLC Number:

TP391

Honghao ZHENG, Hongtao YU, Shaomei LI. Chinese NER based on improved Transformer encoder[J]. Chinese Journal of Network and Information Security, 2021, 7(5): 105-112.

Figures/Tables 7

模型	MSRA	OntoNotes4.0	Resume	微博
BiGRU	88.03%	67.38%	93.78%	54.42%
BiLSTM	88.35%	67.00%	93.25%	$56 . 23 %$
Transformer	87.67%	65.33%	92.64%	50.21%
Transformer_P	$90 . 35 %$	$71 . 14 %$	$94 . 21 %$	56.03%

模型	精确率	召回率	F1值
文献[29]	91.22%	81.71%	86.20%
文献[30]	92.20%	90.18%	91.18%
文献[31]	91.86%	88.75%	90.28%
文献[32]	91.28%	90.62%	90.95%
文献[27]	93.57%	92.79%	93.18%
文献[33]	94.01%	92.93%	93.47%
文献[34]	-	-	94.35%
本文模型 Max=4	$93 . 64 %$	$96 . 63 %$	$95 . 11 %$

模型	精确率	召回率	F1值
文献[35]	76.43%	72.32%	74.32%
文献[36]	$77 . 71 %$	72.51%	75.02%
文献[37]	65.59%	71.84%	68.57%
文献[27]	76.35%	71.56%	73.88%
文献[33]	75.06%	74.52%	74.79%
文献[34]	-	-	75.70%
本文模型 Max=4	76.68%	$84 . 80 %$	$80 . 54 %$

模型	精确率	召回率	F1值
文献[27]	94.81%	94.11%	94.46%
文献[38]	95.14%	94.79%	94.96%
文献[39]	95.28%	95.46%	95.37%
文献[34]	-	-	94.93%
本文模型Max=4	$96 . 23 %$	$97 . 18 %$	$96 . 70 %$

模型	精确率	召回率	F1值
文献[28]	-	-	56.05%
文献[40]	-	-	58.23%
文献[27]	-	-	58.79%
文献[33]	67.31%	48.61%	56.45%
文献[34]	-	-	63.42%
本文模型Max=5	$68 . 34 %$	$74 . 88 %$	$71 . 46 %$

References 40

[1]	GRISHMAN R , SUNDHEIM B . Message understanding conference-6:a brief history[C]// International Conference on Computational Linguistics, 1996: 466-471.
[2]	PIZZATO L A , MOLLA D , PARIS C . Pseudo relevance feedback using named entities for question answering[C]// Proceedings of the 2006 Australian Language Technology Workshop (ALTW-2006). 2006: 89-90.
[3]	BABYCH B , HARTLEY A . Improving machine translation quality with automatic named entity recognition[C]// Proceedings of the 7th International EAMT Workshop on MT and other Language Technology Tools,Improving MT Through other Language Technology Tools:Resources and Tools for Building MT,Association for Computational Linguistics. 2003: 1-8.
[4]	MANDL T , WOMSER-HACKER C , . The effect of named entities on effectiveness in cross-language information retrieval evaluation[C]// Proceedings of the 2005 ACM Symposium on Applied Computing. 2005: 1059-1064.
[5]	邱锡鹏 . 神经网络与深度学习[M]. 北京: 机械工业出版社, 2020.
	QIU X P . Neural networks and deep learning[M]. Beijing: China Machine Press, 2020.
[6]	LI J , SUN A , HAN J ,et al. A survey on deep learning for named entity recognition[J]. IEEE Transactions on Knowledge and Data Engineering, 2020:1.
[7]	HAMMERTON J , . Named entity recognition with long short-term memory[C]// North American Chapter of the Association for Computational Linguistics. 2003: 172-175.
[8]	HUANG Z , XU W , YU K ,et al. Bidirectional LSTM-CRF models for sequence tagging[J]. arXiv:Computation and Language, 2015.
[9]	MA X , HOVY E . End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[J]. arXiv:Learning, 2016.
[10]	CHIU J P , NICHOLS E . Named entity recognition with bidirectional LSTM-CNNs[J]. Transactions of the Association for Computational Linguistics, 2016,4(1): 357-370.
[11]	BENGIO Y , SIMARD P Y , FRASCONI P ,et al. Learning long-term dependencies with gradient descent is difficult[J]. IEEE Transactions on Neural Networks, 1994,5(2): 157-166.
[12]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[C]// Neural Information Processing Systems. 2017: 5998-6008.
[13]	GUO Q P , QIU X P , LIU P F ,et al. Star transformer[C]// NAACL. 2019: 1315-1325.
[14]	DAI Z , YANG Z , YANG Y ,et al. Transformer-XL:attentive language models beyond a fixed-length context[J]. arXiv:Learning, 2019.
[15]	HUANG A , VASWANI A , USZKOREIT J ,et al. Music transformer:generating music with long-term structure[C]// International Conference on Learning Representations, 2019.
[16]	MIKOLOV T , CHEN K , CORRADO G S ,et al. Efficient estimation of word representations in vector space[C]// International Conference on Learning Representations. 2013.
[17]	杨飘, 董文永 . 基于BERT嵌入的中文命名实体识别方法[J]. 计算机工程, 2020,46(04): 40-45,52.
	YANG P , DONG W Y . Chinese NER based on BERT embedding[J]. Computer Engineering, 2020,46(4): 40-45,52.
[18]	DEVLIN J , CHANG M , LEE K ,et al. BERT:pre-training of deep bidirectional transformers for language understanding[J]. arXiv:Computation and Language, 2018.
[19]	YANG Z , DAI Z , YANG Y ,et al. XLNet:generalized autoregressive pretraining for language understanding[J]. arXiv:Computation and Language, 2019.
[20]	PARIKH A P , TACKSTROM O , DAS D ,et al. A decomposable attention model for natural language inference[C]// Empirical Methods in Natural Language Processing. 2016: 2249-2255.
[21]	YAN H , DENG B , LI X ,et al. TENER:adapting transformer encoder for named entity recognition[J]. arXiv:Computation and Language, 2019.
[22]	SHAW P , USZKOREIT J , VASWANI A ,et al. SELF-attention with relative position representations[C]// North American Chapter of the Association for Computational Linguistics. 2018: 464-468.
[23]	LAFFERTY J , MCCALLUM A , PEREIRA F ,et al. Conditional random fields:probabilistic models for segmenting and Labeling Sequence Data[C]// International Conference on Machine Learning. 2001: 282-289.
[24]	SMITH L N , . Cyclical learning rates for training neural networks[C]// Workshop on Applications of Computer Vision, 2017: 464-472.
[25]	LEVOW G , . The third international chinese language processing Bakeoff:word segmentation and named entity recognition[C]// Meeting of the Association for Computational Linguistics. 2006: 108-117.
[26]	Ralph Weischedel. Ontonotes release 4.0 LDC2011T03[S]. 2011.
[27]	ZHANG Y , YANG J . Chinese NER using lattice LSTM[C]// Meeting of the Association for Computational Linguistics. 2018: 1554-1564.
[28]	PENG N , DREDZE M . Named entity recognition for chinese social media with jointly trained embeddings[C]// Empirical Methods in Natural Language Processing. 2015: 548-554.
[29]	CHEN A , PENG F , SHAN R ,et al. Chinese named entity recognition with conditional probabilistic models[C]// Meeting of the Association for Computational Linguistics. 2006: 173-176.
[30]	ZHANG S , QIN Y , WEN J ,et al. Word segmentation and named entity recognition for SIGHAN Bakeoff3[C]// Meeting of the Association for Computational Linguistics. 2006: 158-161.
[31]	ZHOU J S , QU W G , ZHANG F . Chinese named entity recognition via joint identification and categorization[J]. Chinese Journal of Electronics, 2013,22(2): 225-230.
[32]	DONG C H , ZHANG J J , ZONG C Q . Character based LSTM-CRF with radical-level features for Chinese named entity recognition[C]// International Conference on Computer Processing of Oriental Languages. 2016: 239-250.
[33]	SUI D , CHEN Y , LIU K ,et al. Leverage lexical knowledge for chinese named entity recognition via collaborative graph network[C]// International Joint Conference on Natural Language Processing. 2019: 3828-3838.
[34]	QIU X P , LI X N , YAN H . Flat chinese ner using flat-lattice transformer[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 6836-6842.
[35]	WANG M , CHE W , MANNING C D ,et al. Effective bilingual constraints for semi-supervised learning of named entity recognizers[C]// National Conference on Artificial Intelligence. 2013: 919-925.
[36]	CHE W , WANG M , MANNING C D ,et al. Named entity recognition with bilingual constraints[C]// North American Chapter of the Association for Computational Linguistics. 2013: 52-62.
[37]	YANG J , ZHANG Y , DONG F ,et al. Neural word segmentation with rich pretraining[C]// Meeting of the Association for Computational Linguistics. 2017: 839-849.
[38]	LIU W , XU T , XU Q ,et al. An encoding strategy based word-character LSTM for chinese NER[C]// North American Chapter of the Association for Computational Linguistics. 2019: 2379-2389.
[39]	GUI T , ZOU Y , ZHANG Q ,et al. A lexicon-based graph neural network for chinese NER[C]// International Joint Conference on Natural Language Processing. 2019: 1040-1050.
[40]	HE H , SUN X . A unified model for cross-domain and semi- supervised named entity recognition in chinese social media[C]// National Conference on Artificial Intelligence. 2017: 3216-3222.

Metrics

Recommended 0

No Suggested Reading articles found!

数据集	训练集		验证集		测试集
数据集	句子	字符	句子	字符	句子	字符
MSRA	464 00	2 169 900	—	—	4 400	172 600
OntoNotes4.0	15 700	491 900	4 300	200 500	4 300	208 100
Resume	3 800	124 100	460	13 900	480	15 100
微博	1 400	73 500	270	14 500	270	14 800

Chinese NER based on improved Transformer encoder

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 40

Related Articles 15

Metrics

Recommended 0

[1]	Saite CHEN, Weihai LI, Yuanzhi YAO, Nenghai YU. Location privacy protection method based on lightweight K-anonymity incremental nearest neighbor algorithm [J]. Chinese Journal of Network and Information Security, 2023, 9(3): 60-72.
[2]	Wanze CHEN, Liqing HUANG, Jiazhen CHEN, Feng YE, Tianqiang HUANG, Haifeng LUO. Gender forgery of faces by fusing wavelet shortcut connection generative adversarial network [J]. Chinese Journal of Network and Information Security, 2023, 9(3): 150-160.
[3]	Heli WANG, Qiao YAN. Selfish mining detection scheme based on the characters of transactions [J]. Chinese Journal of Network and Information Security, 2023, 9(2): 104-114.
[4]	Xiaochen SHEN, Yinhui GE, Bo CHEN, Ling YU. Research on construction technology of artificial intelligence security knowledge graph [J]. Chinese Journal of Network and Information Security, 2023, 9(2): 164-174.
[5]	Jingwen LI, Yawen LI. Application and risk response of deep synthesis technology [J]. Chinese Journal of Network and Information Security, 2023, 9(2): 184-190.
[6]	Long DAI, Jing ZHANG, Xuefeng FAN, Xiaoyi ZHOU. NLP neural network copyright protection based on black box watermark [J]. Chinese Journal of Network and Information Security, 2023, 9(1): 140-149.
[7]	Honghao ZHENG, Yinuo HAO, Hongtao YU, Shaomei LI, Yiteng WU. Rumor detection in social media based on eahanced Transformer [J]. Chinese Journal of Network and Information Security, 2022, 8(4): 168-174.
[8]	Yu ZHANG, Binglong LI, Xuejuan LI, Heyu ZHANG. Evidence classification method of chat text based on DSR and BGRU model [J]. Chinese Journal of Network and Information Security, 2022, 8(2): 150-159.
[9]	Xiangdong HU, Zhengguo TIAN. Methods of security situation prediction for industrial internet fused attention mechanism and BSRU [J]. Chinese Journal of Network and Information Security, 2022, 8(1): 41-51.
[10]	Pengcheng WANG, Haibin ZHENG, Jianfei ZOU, Ling PANG, Hu LI, Jinyin CHEN. Robustness evaluation of commercial liveness detection platform [J]. Chinese Journal of Network and Information Security, 2022, 8(1): 180-189.
[11]	Yaofei WANG, Weiming ZHANG, Kejiang CHEN, Wenbo ZHOU, Nenghai YU. Survey on image non-additive steganography [J]. Chinese Journal of Network and Information Security, 2021, 7(6): 1-10.
[12]	Jie QIU, Rui HAN, Zhifeng WEI, Zhiyang WANG. Research of public infrastructure system and security policy in cyberspace [J]. Chinese Journal of Network and Information Security, 2021, 7(6): 56-67.
[13]	Yuxiang CHENG, Weiming ZHANG, Weixiang LI, Nenghai YU. Binary image steganography method based on layered embedding [J]. Chinese Journal of Network and Information Security, 2021, 7(5): 49-56.
[14]	Xiao YANG,Xiuzhen CHEN,Jin MA,Haozhe LIANG,Shenghong LI. User interests-based microblog tracing algorithm [J]. Chinese Journal of Network and Information Security, 2020, 6(6): 164-173.
[15]	Zhenhua CAI,Jiayun LIN,Fang LIU. Blockchain storage:technologies and challenges [J]. Chinese Journal of Network and Information Security, 2020, 6(5): 11-20.