融合递增词汇选择的深度学习中文输入法

doi:10.11959/j.issn.1000-0801.2022294

Abstract

Abstract:

The core task of an input method is to convert the keystroke sequences typed by users into Chinese character sequences.Input methods applying deep learning methods have advantages in learning long-range dependencies and solving data sparsity problems.However, the existing methods still have two shortcomings: the separation structure of pinyin slicing in conversion leads to error propagation, and the model is complicated to meet the demand for real-time performance of the input method.A deep-learning input method model incorporating incremental word selection methods was proposed to address these shortcomings.Various softmax optimization methods were compared.Experiments on People’s Daily data and Chinese Wikipedia data show that the model improves the conversion accuracy by 15% compared with the current state-of-the-art model, and the incremental vocabulary selection method makes the model 130 times faster without losing conversion accuracy.

Key words: Chinese input method, long short-term memory, vocabulary selection

CLC Number:

TP391

Huajian REN, Xiulan HAO, Wenjing XU. Deep learning Chinese input method with incremental vocabulary selection[J]. Telecommunications Science, 2022, 38(12): 56-64.

Figures/Tables 4

References 32

[1]	STOLCKE A , . SRILM - an extensible language modeling toolkit[C]// Proceedings of 7th International Conference on Spoken Language Processing (ICSLP 2002). ISCA:ISCA, 2002: 901-904.
[2]	XIAO Y L , LIU L M , HUANG G P ,et al. BiTIIMT:a bilingual text-infilling method for interactive machine translation[C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). Stroudsburg,PA,USA:Association for Computational Linguistics, 2022: 1958-1969.
[3]	PARASKEVOPOULOS G , PARTHASARATHY S , KHARE A ,et al. Multimodal and multiresolution speech recognition with transformers[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg,PA,USA:Association for Computational Linguistics, 2020: 2381-2387.
[4]	DING Z X , XIA R , YU J F . End-to-end emotion-cause pair extraction based on sliding window multi-label learning[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg,PA,USA:Association for Computational Linguistics, 2020: 3574-3583.
[5]	DING Z X , XIA R , YU J F . ECPE-2D:emotion-cause pair extraction based on joint two-dimensional representation,interaction and prediction[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg,PA,USA:Association for Computational Linguistics, 2020: 3161-3170.
[6]	PANDEY P , BASU P , CHAKRABORTY K ,et al. GreenTPU:improving timing error resilience of a near-threshold tensor processing unit[C]// Proceedings of DAC '19:Proceedings of the 56th Annual Design Automation Conference.[S.l:s.n.],2019. 2019: 1-6.
[7]	LIU X Y , CHEN X , WANG Y Q ,et al. Two efficient lattice rescoring methods using recurrent neural network language models[J]. IEEE/ACM Transactions on Audio,Speech,and Language Processing, 2016,24(8): 1438-1449.
[8]	LEE K , PARK C , KIM N ,et al. Accelerating recurrent neural network language model based online speech recognition system[C]// Proceedings of 2018 IEEE International Conference on Acoustics,Speech and Signal Processing. Piscataway:IEEE Press, 2018: 5904-5908.
[9]	CHEN X , LIU X Y , WANG Y Q ,et al. Efficient training and evaluation of recurrent neural network language models for automatic speech recognition[J]. IEEE/ACM Transactions on Audio,Speech,and Language Processing, 2016,24(11): 2146-2157.
[10]	CHO K , VAN MERRIENBOER B , GULCEHRE C ,et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg,PA,USA:Association for Computational Linguistics, 2014: 1724-1734.
[11]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017,22(7): 139-147.
[12]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[13]	CHEN Z , LEE K F . A new statistical approach to Chinese Pinyin input[C]// Proceedings of the 38th Annual Meeting on Association for Computational Linguistics - ACL '00. Morristown,NJ,USA:Association for Computational Linguistics, 2000: 241-247.
[14]	YANG S , ZHAO H , LU B . A machine translation approach for Chinese whole-sentence Pinyin-to-character conversion[C]// Proceed ings of the 26th Pacific Asia Conference on Language,Information,and Computation.[S.l.:s.n.], 2012: 333-342.
[15]	HUANG Y F , LI Z C , ZHANG Z S ,et al. Moon IME:neural-based Chinese pinyin aided input method with customizable association[C]// Proceedings of ACL 2018,System Demonstrations. Stroudsburg,PA,USA:Association for Computational Linguistics, 2018: 140-145.
[16]	ZHANG Z S , HUANG Y F , ZHAO H . Open vocabulary learning for neural Chinese pinyin IME[C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg,PA,USA:Association for Computational Linguistics, 2019: 1584-1594.
[17]	HUANG Y F , ZHAO H . Chinese pinyin aided IME,input what You have not keystroked yet[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg,PA,USA:Association for Computational Linguistics, 2018: 2923-2929.
[18]	TAN M H , DAI Y , TANG D Y ,et al. Exploring and adapting Chinese GPT to pinyin input method[C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). Stroudsburg,PA,USA:Association for Computational Linguistics, 2022: 1899-1909.
[19]	CHEN W L , GRANGIER D , AULI M . Strategies for training large vocabulary neural language models[C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). Stroudsburg,PA,USA:Association for Computational Linguistics, 2016: 1975-1985.
[20]	JOULIN A , CISSé M , GRANGIER D ,et al. Efficient softmax approximation for GPUs[C]// International Conference on Machine Learning.[S.l.:s.n.], 2017: 1302-1310.
[21]	SHIM K , LEE M , CHOI I ,et al. SVD-softmax:Fast softmax approximation on large vocabulary neural networks[J]. Advances in Neural Information Processing Systems, 2017,30: 5463-5473.
[22]	SHI Y Z , ZHANG W Q , LIU J ,et al. RNN language model with word clustering and class-based output layer[J]. EURASIP Journal on Audio,Speech,and Music Processing,2013, 2013:22.
[23]	Zhang M J , Wang W H , Liu X D ,et al. Navigating with graph representations for fast and scalable decoding of neural language models[J]. Advances in Neural Information Processing Systems, 2018,31: 6308-6319.
[24]	MI H T , WANG Z G , ITTYCHERIAH A . Vocabulary manipulation for neural machine translation[C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2:Short Papers). Stroudsburg,PA,USA:Association for Computational Linguistics, 2016: 124-129.
[25]	JEAN S , CHO K , MEMISEVIC R ,et al. On using very large target vocabulary for neural machine translation[C]// Proceed ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:Long Papers). Stroudsburg,PA,USA:Association for Computational Linguistics, 2015: 1-10.
[26]	YAO J , SHU R , LI X ,et al. Enabling real-time neural IME with incremental vocabulary selection[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.[S.l.:s.n.], 2019: 1-8.
[27]	MIKOLOV T , KARAFIáT M , BURGET L ,et al. Recurrent neural network based language model[C]// Proceedings of Interspeech 2010. ISCA:ISCA, 2010,2(3): 1045-1048.
[28]	ELMAN J L . Finding structure in time[J]. Cognitive Science, 1990,14(2): 179-211.
[29]	KNESER R , NEY H . Improved backing-off for M-gram language modeling[C]// Proceedings of 1995 International Conference on Acoustics,Speech,and Signal Processing. Piscataway:IEEE Press, 1995: 181-184.
[30]	JAMES F . Modified kneser-ney smoothing of n-gram models[R]. Research Institute for Advanced Computer Science, 2000.
[31]	PRESS O , WOLF L . Using the output embedding to improve language models[C]// Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics:Volume 2,Short Papers. Stroudsburg,PA,USA:Association for Computational Linguistics, 2017: 157-163.
[32]	DEVLIN J , ZBIB R , HUANG Z Q ,et al. Fast and robust neural network joint models for statistical machine translation[C]// Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). Stroudsburg,PA,USA:Association for Computational Linguistics, 2014: 1370-1380.

Metrics

Recommended 0

No Suggested Reading articles found!

模型	Top 1 Accuracy	Top 10 Accuracy
三元语法	54.4%	66.4%
谷歌拼音输入法	70.9%	82.3%
Online P2C	71.3%	81.3%
GPT P2C	73.2%	85.5%
LSTM基线	88.6%	97.6%

模型	单步时长/ms	softmax时长/ms	Top 1 Accuracy	Top 10 Accuracy
LSTM基线	172.1	169.8	86.4%	97.1%
单字模型	6.8	4.6	66.8%	79.5%
单字模型束宽度50	14.2	9.3	69.1%	86.3%
差异化softmax	43.0	39.3	84.7%	96.2%
自适应softmax	41.4	39.0	84.4%	96.1%
递增词汇选择	3.2	1.3	84.6%	95.8%
+顶部采样	4.0	2.0	86.1%	96.8%
+均匀采样	4.0	2.0	84.7%	95.8%
+自归一化	3.2	1.2	86.1%	96.9%

Deep learning Chinese input method with incremental vocabulary selection

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 4

References 32

Related Articles 1

Metrics

Recommended 0