基于神经网络的文本表示模型新方法

doi:10.11959/j.issn.1000-436x.2017088

Abstract

Abstract:

Method of text representation model was proposed to extract word-embedding from text feature.Firstly,the word-embedding of the dual word-embedding list based on dictionary index and the corresponding part of speech index was created.Then,feature vectors was obtained further from these extracted word-embeddings by using Bi-LSTM recurrent neural network.Finally,the sentence vectors were processed by mean-pooling layer and text categorization was classified by softmax layer.The training effects and extraction performance of the combination model of Bi-LSTM and double word-embedding neural network were verified.The experimental results show that this model not only performs well in dealing with the high-quality text feature vector and the expression sequence,but also significantly outperforms other three kinds of neural networks,which includes LSTM,LSTM+context window and Bi-LSTM.

Key words: neural network, word-embedding, Bi-LSTM, text representation

CLC Number:

TP183

Shui-fei ZENG,Xiao-yan ZHANG,Xiao-feng DU,Tian-bo LU. New method of text representation model based on neural network[J]. Journal on Communications, 2017, 38(4): 86-98.

Figures/Tables 9

References 21

[1]	XU W D , AULI M , CLARK S . CCG supertagging with a recurrent neural network[C]// The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers). 2015: 250-255.
[2]	GEOFFREY E,HINTON , . Learning distributed representations of concepts[C]// The Eighth Annual Conference of the Cognitive Science Society. Amherst,Mass, 1986: 1-12.
[3]	YAO Y S , HUANG Z . Bi-directional LSTM recurrent neural network for Chinese word segmentation[J]. arXiv:1602.04874v1[cs.LG], 2016.
[4]	CHIU J P C , NICHOLS E . Named entity recognition with bidirectional LSTM-CNNs[J]. Transactions of the Association for Computational Linguistics, 2016,4: 357-370.
[5]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735.
[6]	TAN M , XIANG B , ZHOU B W . LSTM-based deep learning models for non-factoid answer selection[C]// ICLR 2016.
[7]	KIPERWASSER E , GOLDBERG Y . Simple and accurate dependency parsing using Bidirectional LSTM feature representations[J]. Transactions of the Association for Computational Linguistics, 2016,4: 313-327.
[8]	ZEILER M D . ADADELTA:an adaptive learning rate method[J]. arXir:1212.5701v1[cs.LG], 2012.
[9]	MIKOLOV T , SUTSKEVER I , CHEN K ,et al. Distributed representations of words and phrases and their compositionality[J]. Advances in Neural Information Processing Systems, 2013,26: 3111-3119.
[10]	MAAS A L , DALY R E , PHAM P T ,et al. Learning word vectors for sentiment analysis[C]// The 49th Annual Meeting of the Association for Computational Linguistics:Human Language Technologies. 2011: 142-150.
[11]	CHO K . Natural language understanding with distributed representation[J]. Nato Asi, 2015,147: 139-155.
[12]	SANTOS C D , TAN M , XIANG B . Attentive pooling networks[J]. arXir:1602.03609v1[cs.CL], 2016.
[13]	SEVERYN A , MOSCHITTI A . Modeling relational information in question-answer pairs with convolutional neural networks[J]. arXiv:1604.01178v1[cs.CL], 2016.
[14]	CROSS J , HUANG L . Incremental parsing with minimal features using bi-directional LSTM[C]// The 54th Annual Meeting of the Association for Computational Linguistics. 2016: 32-37.
[15]	TURIAN J , RATINOV L , BENGIO Y . Word representations:a simple and general method for semi-supervised learning[C]// The 48th Annual Meeting of the Association for Computational Linguistics. 2010: 384-394.
[16]	SUGAWARA H , TAKAMURA H , SASANO R ,et al. Context representation with word embeddings for WSD[M]. Computational Linguistics.Springer. Singapore, 2015: 108-119.
[17]	SUNDERMEYER M,SCHLüTER R , NEY H . LSTM neural networks for language modeling[J]. Interspeech, 2012,31(43): 601-608.
[18]	WANG P L , QIAN Y , SOONG F K ,et al. A unified tagging solution:bidirectional LSTM recurrent neural network with word embedding[J]. arXiv:1511.00215v1[cs.CL], 2015.
[19]	HUANG Z H , XU W , YU K . Bidirectional LSTM-CRF models for sequence tagging[J]. arXiv:1508.01991v1[cs.CL], 2015.
[20]	WANG L,LUíS T , MARUJO L , et al . Finding function in form:compositional character models for open vocabulary word representation[C]// The 2015 Conference on Empirical Methods in Natural Language Processing. 2016: 1520-1530.
[21]	RAO A , SPASOJEVIC N . Actionable and political text classification using word embeddings and LSTM[J]. arXiv:1607.02501v1[cs.CL], 2016.

Metrics

Recommended 0

No Suggested Reading articles found!

类别	训练集	测试集
描述	785	58
人物	333	23
地点	936	73
数字	1 068	98
实体	1 242	104
时间	590	43
未知	9	1
总数	4 963	400

参数名称	维度	层级	参数含义
W_i^0	96×96	正向LSTM	输入门当前序列表征矩阵
W_f^0	96×96	正向LSTM	遗忘门当前序列表征矩阵
W_o^0	96×96	正向LSTM	输出门当前序列表征矩阵
W_c^0	96×96	正向LSTM	当前序列候选表征矩阵
U_i^0	96×96	正向LSTM	输入门前时刻序列表征矩阵
U_f^0	96×96	正向LSTM	遗忘门前时刻序列表征矩阵
U_o^0	96×96	正向LSTM	输出门前时刻序列表征矩阵
U_c^0	96×96	正向LSTM	前时刻序列候选表征矩阵
b_i^0	1×96	正向LSTM	输入门偏置向量
b_f^0	1×96	正向LSTM	遗忘门偏置向量
b_o^0	1×96	正向LSTM	输出门偏置向量
b_c^0	1×96	正向LSTM	候选偏置向量
W_i^1	96×96	反向LSTM	输入门当前序列表征矩阵
W_f^1	96×96	反向LSTM	遗忘门当前序列表征矩阵
W_o^1	96×96	反向LSTM	输出门当前序列表征矩阵
W_c^1	96×96	反向LSTM	当前序列候选表征矩阵
U_i^1	96×96	反向LSTM	输入门前时刻序列表征矩阵
U_f^1	96×96	反向LSTM	遗忘门前时刻序列表征矩阵
U_o^1	96×96	反向LSTM	输出门前时刻序列表征矩阵
U_c^1	96×96	反向LSTM	前时刻序列候选表征矩阵
b_i^1	1×96	反向LSTM	输入门偏置向量
b_f^1	1×96	反向LSTM	遗忘门偏置向量
b_o^1	1×96	反向LSTM	输出门偏置向量
b_c^1	1×96	反向LSTM	候选偏置向量
W	7×192	SOFTMAX	分类层表征矩阵
b	1×7	SOFTMAX	分类层偏置向量

模型	训练集正确率	测试集正确率	词向量维度
LSTM	99%	82%	128
LSTM+context window	99%	86%	5×64
Bi-LSTM	99%	87%	64
Bi-LSTM+double word-embedding	99%	90%	64+32

样本编号	类别0	类别1	类别2	类别3	类别4	类别5	类别6	正确标签
24	0.000 5	0.001 748	0.874 618	0.100 758	0.000 031	0.009 069	0.013 275	2
25	0.439 797	0.093 689	0.024 485	0.000 94	0.317 784	0.115 709	0.007 597	4
26	0.000 013	0.054 079	0.879 863	0.012 894	0.050 611	0.000 197	0.002 342	3
27	0.611 823	0.160 361	0.135 009	0.000 045	0.068 51	0.000 001	0.024 251	0
28	0.000 007	0	0.000 001	0.999 979	0.000 001	0	0.000 012	3
29	0.112 567	0.040 439	0.035 534	0.000 341	0.769 876	0.030 551	0.010 691	4
30	0.000 973	0.170 807	0.441 168	0.002 186	0.192 433	0.190 177	0.002 256	2
31	0.000 488	0.000 057	0.002 153	0.995 496	0.001 251	0.000 191	0.000 364	3
32	0.067 126	0.019 039	0.004 159	0.001 812	0.899 413	0.004 31	0.004 14	4
33	0.000 2	0.000 205	0.000 045	0.000 046	0.999 353	0.000 041	0.000 11	4
34	0.000 102	0.001 531	0.978 825	0.015 233	0.000 323	0.003 705	0.000 28	2
35	0	1	0	0	0	0	0	1
36	0.000 024	0.000 004	0.000 009	0.999 919	0.000 031	0.000 001	0.000 013	3
37	0.024 869	0.541 476	0.070 165	0.015 507	0.066 96	0.279 107	0.001 916	4
38	0.331 426	0.000 09	0.520 277	0.004 777	0.127 866	0.000 296	0.015 269	3
39	0.000 106	0.000 001	0.000 042	0.999 098	0.000 045	0	0.000 708	3
40	0.000 415	0.000 068	0.000 829	0.001 67	0.993 545	0.002 899	0.000 573	4

New method of text representation model based on neural network

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 21

Related Articles 15

Metrics

Recommended 0

[1]	Jinyin CHEN, Haiyang XIONG, Haonan MA, Yayu ZHENG. CLB-Defense: based on contrastive learning defense for graph neural network against backdoor attack [J]. Journal on Communications, 2023, 44(4): 154-166.
[2]	Jianfeng LI, Zheyu LIU, Yang RONG, Zhan LI, Bolin LIAO, Linxi QU, Zhijie LIU, Kunhuang LIN. Zeroing neural network for time-varying convex quadratic programming with linear noise [J]. Journal on Communications, 2023, 44(4): 226-233.
[3]	Yun LIN, Huaitao XU, Sen WANG, Sicheng ZHANG, Long ZHUANG. Objective assessment of communication speech interference effect based on feature fusion [J]. Journal on Communications, 2023, 44(3): 105-116.
[4]	Hongyu YANG, Haiyun YANG, Liang ZHANG, Xiang CHENG. Feature dependence graph based source code loophole detection method [J]. Journal on Communications, 2023, 44(1): 103-117.
[5]	Shiwen HE, Jun YUAN, Zhenyu AN, Min ZHANG, Yongming HUANG, Yaoxue ZHANG. GNN-based optimization algorithm for joint user scheduling and beamforming [J]. Journal on Communications, 2022, 43(7): 73-84.
[6]	Tao LENG, Lijun CAI, Aimin YU, Ziyuan ZHU, Jian’gang MA, Chaofei LI, Ruicheng NIU, Dan MENG. Review of threat discovery and forensic analysis based on system provenance graph [J]. Journal on Communications, 2022, 43(7): 172-188.
[7]	Yurong LIAO, Haining WANG, Cunbao LIN, Yang LI, Yuqiang FANG, Shuyan NI. Research progress of deep learning-based object detection of optical remote sensing image [J]. Journal on Communications, 2022, 43(5): 190-203.
[8]	Fan ZHANG, Yun HUANG, Zizhuo FANG, Wei GUO. Lost-minimum post-training parameter quantization method for convolutional neural network [J]. Journal on Communications, 2022, 43(4): 114-122.
[9]	Zhengyu ZHU, Gengwang HOU, Chongwen HUANG, Gangcan SUN, Wanming HAO, Jing LIANG. Systems resource allocation algorithm for RIS-assisted D2D secure communication based on parallel CNN [J]. Journal on Communications, 2022, 43(3): 172-179.
[10]	Junyan HUO, Danni WANG, Yanzhuo MA, Shuai WAN, Fuzheng YANG. Efficient cross-component prediction for H.266/VVC based on lightweight fully connected networks [J]. Journal on Communications, 2022, 43(2): 143-155.
[11]	Hua LONG, Zhangheng HUANG, Yubin SHAO, Qingzhi DU, Shumeng SU. Research on language recognition algorithm based on improved CFCC feature extraction [J]. Journal on Communications, 2022, 43(12): 211-221.
[12]	Zhengyu ZHU, Pengfei CHEN, Zixuan WANG, Kexian GONG, Di WU, Zhongyong WANG. Short wave protocol signals recognition based on Swin-Transformer [J]. Journal on Communications, 2022, 43(11): 127-135.
[13]	Jinbo XIONG, Yongjie ZHOU, Renwan BI, Liang WAN, Youliang TIAN. Towards edge-collaborative, lightweight and privacy-preserving classification framework [J]. Journal on Communications, 2022, 43(1): 127-137.
[14]	Yiteng WU, Wei LIU, Hongtao YU. Label flipping adversarial attack on graph neural network [J]. Journal on Communications, 2021, 42(9): 65-74.
[15]	Changyin SUN, Liyan LIU, Fan JIANG, Jing JIANG. DNN-based Sub-6 GHz assisted millimeter wave network power allocation algorithm [J]. Journal on Communications, 2021, 42(9): 184-193.