融合篇章结构位置编码的神经机器翻译

doi:10.11959/j.issn.2096-6652.202016

摘要/Abstract

摘要：

现有的文档级神经机器翻译方法在翻译一个句子时大多只利用文档的上下文词汇信息，而忽视了跨句子的篇章语义单元之间的结构关系。针对此问题，提出了多种篇章结构位置编码策略，利用基于修辞结构理论的篇章树结构，对篇章树上位于不同篇章单元的单词之间的位置关系进行了表示。实验表明，通过位置编码的方式，在基于Transformer框架的神经机器翻译模型中有效地融合了源端的篇章结构信息，译文质量得到了显著提升。

关键词: 神经机器翻译, 篇章结构, 位置编码, 篇章分析, 修辞结构理论

Abstract:

Most of existing document-level neural machine translation (DocNMT) methods focus on exploring the utilization of the lexical information of context,which ignore the structural relationships among the cross-sentence discourse semantic units.Therefore,multiple discourse structural position encoding strategies were proposed to represent the positional relationships among the words in discourse units over the discourse tree based on rhetorical structure theory (RST).Experimental results show that the source-side discourse structural position information is effectively fused into the DocNMT models underlying the Transformer architecture by the position encoding,and the translation quality is improved significantly.

Key words: neural machine translation, discourse structure, position encoding, discourse analysis, rhetorical structure theory

中图分类号:

TP391

亢晓勉,宗成庆. 融合篇章结构位置编码的神经机器翻译[J]. 智能科学与技术学报, 2020, 2(2): 144-152.

Xiaomian KANG,Chengqing ZONG. Fusion of discourse structural position encoding for neural machine translation[J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(2): 144-152.

图/表 7

图1

图2

表1

图3

表2

表3

表4

参考文献 37

[1]	郑南宁 . 人工智能新时代[J]. 智能科学与技术学报, 2019,1(1): 1-3.
	ZHENG N N . The new era of artificial intelligence[J]. Chinese Journal of Intelligent Science and Technology, 2019,1(1): 1-3.
[2]	张钹 . 人工智能进入后深度学习时代[J]. 智能科学与技术学报, 2019,1(1): 4-6.
	ZHANG B . Artificial intelligence is entering the post deep-learning era[J]. Chinese Journal of f Intelligent Science And Technology, 2019,1(1): 4-6.
[3]	宗成庆 . 统计自然语言处理[M]. 北京: 清华大学出版社, 2013.
	ZONG C Q . Statistical natural language processing[M]. Beijing: Tsinghua University PressPress, 2013.
[4]	杜倩龙, 宗成庆, 苏克毅 . 利用上下文相似度增强词对齐效果的自然语言推理方法[J]. 智能科学与技术学报, 2020,2(1): 26-35.
	DU Q L , ZONG C Q , SU K Y . Enhancing alignment with context similarity for natural language inference[J]. Chinese Journal of Intelligent Science and Technology, 2020,2(1): 26-35.
[5]	BROWN G , BROWN G D , BROWN G R ,et al. Discourse analysis[M]. Cambridge: Cambridge University PressPress, 1983.
[6]	HALLIDAY M A K , HASAN R . Cohesion in English[M]. >[S.l.]. RoutledgePress, 2014.
[7]	GONG Z X , ZHANG M , ZHOU G D . Cache-based document-level statistical machine translation[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2011: 909-919.
[8]	XIONG D , DING Y , ZHANG M ,et al. Lexical chain based cohesion models for document-level statistical machine translation[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2013: 1563-1573.
[9]	TU M , ZHOU Y , ZONG C Q . Enhancing grammatical cohesion:generating transitional expressions for SMT[C]// The Annual Meeting of the Association for Computational Linguistics. Stroudsburg:ACL, 2014: 850-860.
[10]	HARDMEIER C . Discourse in statistical machine translation[D]. Uppsala:Uppsala University, 2014.
[11]	WANG L Y , TU Z P , WAY A ,et al. Exploiting cross-sentence context for neural machine translation[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2017: 2826-2831.
[12]	VOITA E , SERDYUKOV P , SENNRICH R ,et al. Context-aware neural machine translation learns anaphora resolution[C]// The Annual Meeting of the Association for Computational Linguistics. Stroudsburg:ACL, 2019: 1264-1274.
[13]	ZHANG J C , LUAN H B , SUN M S ,et al. Improving the transformer translation model with document-level context[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2018: 533-542.
[14]	MICULICICH L , RAM D , PAPPAS N ,et al. Document-level neural machine translation with hierarchical attention networks[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2018: 2947-2954.
[15]	YANG Z X , ZHANG J C , MENG F D ,et al. Enhancing context modeling with a query-guided capsule network for document-level translation[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2019: 1527-1537.
[16]	TU Z P , LIU Y , SHI S M ,et al. Learning to remember translation history with a continuous cache[J]. Transactions of the ACL, 2018,6: 407-420.
[17]	XIONG H , HE Z J , WU H ,et al. Modeling coherence for discourse neural machine translation[C]// The AAAI Conference on Artificial Intelligence. Palo Alto:AAAI Press, 2019: 7338-7345.
[18]	MARUF S , MARTINS A F T , HAFFARI G . Selective attention for context-aware neural machine translation[C]// TheConference of the North American Chapter of the ACL. Stroudsburg:ACL, 2019: 3092-3102.
[19]	BAWDEN R , SENNRICH R , BIRCH A ,et al. Evaluating discourse phenomena in neural machine translation[C]// The Conference of the North American Chapter of the ACL. Stroudsburg:ACL, 2018: 1304-1313.
[20]	VOITA E , SENNRICH R , TITOV I . When a good translation is wrong in context:Context-aware machine translation improves on deixis,ellipsis,and lexical cohesion[C]// The Annual Meeting of the Associationfor Computational Linguistics. Stroudsburg:ACL, 2019: 1198-1212.
[21]	KANG X M , ZONG C Q , XUE N W . A survey of discourse representations for Chinese discourse annotation[J]. ACM Transactions on Asian and Low-Resource Language Information Processing, 2019,18(3): 1-25.
[22]	ROTHWELL A D . Thematic progression as a functional resource in analysing texts[J]. Circulo de Linguistica Aplicada a la Communication, 2001,(5):2.
[23]	ASHER N , ALEX L . Logics of conversation[M]. Cambridge: Cambridge University PressPress, 2003.
[24]	MANN W C , THOMPSON S A . Rhetorical structure theory:toward a functional theory of text organization[J]. Text ＆ Talk, 1988,8(3): 243-281.
[25]	HERNAULT H , PRENDINGER H , DUVERLE D A . HILDA:adiscourse parser using support vector machine classification[J]. Dialogue＆ Discourse, 2010,1(3): 1-33.
[26]	FENG V W , HIRST G . Text-level discourse parsing with rich linguistic features[C]// The Annual Meeting of the Association for Computational Linguistics. Stroudsburg:ACL, 2012: 60-68.
[27]	JI Y F , EISENSTEIN J . Representation learning for text-level discourse parsing[C]// The Annual Meeting of the Association for Computational Linguistics. Stroudsburg:ACL, 2014: 13-24.
[28]	BHATIA P , JI Y F , EISENSTEIN J . Better document-level sentiment analysis from RST discourse parsing[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2015: 2212-2218.
[29]	GERANI S , MEHDAD Y , CARENINI G ,et al. Abstractive summarization of product reviews using discourse structure[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2014: 1602-1613.
[30]	GUZMAN F , JOTY S , LIUIS A ,et al. Using discourse structure improves machine translation evaluation[C]// The Annual Meeting of the Association for Computational Linguistics. Stroudsburg:ACL, 2014: 687-698.
[31]	TU M , ZHOU Y , ZONG C Q . A novel translation framework based on rhetorical structure theory[C]// The Annual Meeting of the Association for Computational Linguistics. Stroudsburg:ACL, 2013: 370-374.
[32]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[C]// The 31st Annual Conference on Advances in Neural Information Processing Systems. Boston:MIT Press, 2017: 5998-6008.
[33]	BAHDANAU D , CHO K , BENGIO Y . Neural machine translation by jointly learning to align and translate[J]. arXiv:1409.0473, 2014
[34]	GEHRING J , AULI M , GRAGIER D ,et al. Convolutional sequence to sequence learning[C]// The 34th International Conference on Machine Learning. New York:ACM Press, 2017: 1243-1252.
[35]	SHAW P , USZKOREIT J , VASWANI A . Self-attention with relative position representations[C]// The Conference of the North American Chapter of the Association for Computational Linguistics. Stroudsburg:ACL, 2018: 464-468
[36]	WANG X , TU Z P , WANG L Y ,et al. Self-attention with structural position representations[C]// The Conference on Empirical Methods in Natural Language Processing. Stroudsburg:ACL, 2019: 1403-1409.
[37]	SENNRICH R , DADDOW B , BIRCH A . Neural machine translation of rare words with subword units[C]// The Annual Meeting of the Associationfor Computational Linguistics. Stroudsburg:ACL, 2016: 1715-1725.

数据来源	句子数目/个	段落数目/段
英译中TED	0.23 M/0.9 K/3.9 K	15.3 K/59/261
英译德TED	0.21 M/0.9 K/3.4 K	13.7 K/60/230
英译德Europarl	1.67 M/3.6 K/5.1 K	156.5 K/330/477

融合方式	编号	模型	BLEU
无	1	Base	28.63%
	2	HierAtt	29.38%
加法融合	3	+Abs EDU-PE	29.52%
	4	+Rel EDU-PE	29.49%
	5	+Abs Depth-PE	29.68%
	6	+Rel Depth-PE	29.65%
	7	+Path-PE	29.77%
非线性融合	8	+Abs EDU-PE	29.48%
	9	+Rel EDU-PE	29.53%
	10	+Abs Depth-PE	29.76%
	11	+Rel Depth-PE	29.80%
	12	+Path-PE	29.89%

模型	英译中TED	英译德TED	英译德Europarl
Base	21.54%	28.44%	28.87%
HierAtt	22.29%	29.31%	29.76%
+Rel EDU-PE	22.41%	29.52%	29.95%
+Rel Depth-PE	22.82%*	29.61%	30.12%*
+Path-PE	22.78%*	29.83%*	30.17%*
+Rel EDU-PE+Rel Depth-PE	23.07%*	29.70%	30.21%*
+Rel EDU-PE+Rel Depth-PE+Path-PE	22.98%*	29.97%*	30.28%*

类型	FlatAtt	HierAtt
无篇章结构位置编码	22.06%	22.29%
融合篇章结构位置编码	23.05 %	22.98 %