利用上下文相似度增强词对齐效果的自然语言推理方法

doi:10.11959/j.issn.2096-6652.202003

Abstract

Abstract:

Previous approaches generally use context information to improve the word representation but ignore the importance of context similarity in aligning tokens.Furthermore,most of them uniformly weight various local decisions during aggregation for the global judgment.However,local decisions related to various tokens can influence the final decision differently.In order to process these problems,an enhanced alignment mechanism was proposed,which jointly considers both token content and context similarity in computing the alignment weight for each token pair.Besides,a selection gate mechanism to weight local decisions differently was also proposed.Experimental results show that our performance is comparable to state-of-the-art approaches but better mimics human behavior,making it more interpretable.

Key words: textual entailment, ural language inference, word alignment

CLC Number:

TP391.1

Qianlong DU,Chengqing ZONG,Keh-Yih SU. Enhancing alignment with context similarity for natural language inference[J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(1): 26-35.

Figures/Tables 7

References 33

[1]	KOUYLEKOV M , MAGNINI B . Recognizing textual entailment with tree edit distance algorithms[J]. Pascal Challenges on RTE, 2005: 17-20.
[2]	BOS J , MARKERT K . Recognizing textual entailment with robust logical inference[C]// The Conference Human Language Technology and Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2005: 404-426.
[3]	BHASKAR P , BANERJEE S , PAKRAY P ,et al. A hybrid question answering system for multiple choice question(MCQ)[C]// QA4MRE at Conference and Labs of the Evaluation Forum.[S.l.:s.n]. 2013.
[4]	ROMANO L , KOUYLEKOV M , SZPEKTOR I ,et al. Investigating a generic paraphrase-based approach for relation extraction[C]// The 11th Conference of the European Chapter of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2006: 409-416.
[5]	PADó S , CER D , GALLEY M ,et al. Measuring machine translation quality as semantic equivalence:a metric based on entailment features[J]. Machine Translation, 2009,23(2-3): 181-193.
[6]	HARABAGIU S , HICKL A . Methods for using textual entailment in open-domain question answering[C]// The 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2006: 905-912.
[7]	HARABAGIU S , HICKL A , LACATUSU F . Satisfying information needs with multi-document summaries[J]. Information Processing and Management, 2007,43(6): 1619-1642.
[8]	HEILMAN M , SMITH N A . Tree edit models for recognizing textual entailments,paraphrases,and answers to questions[C]// Human Language Technologies:Conference of the North American Chapter of the Association of Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2010: 1011-1019.
[9]	BOWMAN S R , ANGELI G , POTTS C ,et al. A large annotated corpus for learning natural language inference[C]// The 2015 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2015: 632-642.
[10]	MOU L L , MEN R , LI G ,et al. Natural language inference by tree-based convolution and heuristic matching[C]// The 54th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2016: 130-136.
[11]	PARIKH A P , TACKSTROM O , DAS D ,et al. A decomposable attention model for natural language inference[C]// The 2016 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2016: 2249-2255.
[12]	CHEN Q , ZHU X , LING Z H ,et al. Enhanced LSTM for natural language inference[C]// The 55th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2017: 1657-1668.
[13]	TAY Y , LUU A T , HUI S C ,et al. Compare,compress and propagate:enhancing neural architectures with alignment factorization for natural language inference[C]// The 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2018: 1565-1575.
[14]	POLIAK A , NARADOWSKY J , HALDAR A ,et al. Hypothesis only baselines in natural language inference[C]// The 7th Joint Conference on Lexical and Computational Semantics.[S.l.:s.n. ], 2018.
[15]	PETERS M E , NEUMANN M , IYYER M ,et al. Deep contextualized word representations[C]// The 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2018: 2227-2237.
[16]	WANG Z , HAMZA W , FLORIAN R ,et al. Bilateral multi-perspective matching for natural language sentences[C]// The 26th International Joint Conference on Artificial Intelligence. Los Altos:William Kaufman, 2017: 4144-4150.
[17]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[C]// The 31st Conference on Neural Information Processing System (NIPS 2017).[S.l.:s.n]. 2017.
[18]	DEVLIN J , CHANG M , LEE K ,et al. BERT:Pre-training of deep bidirectional transformers for language understanding[C]// The 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2019: 4171-4186.
[19]	DU Q L , SU K Y , ZONG C Q . Adopting the word-pair-dependencytriplets with individual comparison for natural language inference[C]// The 27th International Conference on Computational Linguistics (COLING). Stroudsburg:Association for Computational Linguistics, 2018: 414-425.
[20]	PENNINGTON J , SOCHER R , MANNING C D . GloVe:global vectors for word representation[C]// The 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2014: 1532-1543.
[21]	CHEN D , FISCH A , WESTON J ,et al. Reading Wikipedia to answer open-domain questions[C]// The 55th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2017: 1870-1879.
[22]	WILLIAMS A , NANGIA N , BOWMAN S R ,et al. A broad-coverage challenge corpus for sentence understanding through inference[C]// The 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2018: 1112-1122.
[23]	GLOCKNER M , SHWARTZ V , GOLDBERG Y . Breaking NLI systems with sentences that require simple lexical inferences[C]// The 56th Annual Meeting of the Association for Computational Linguistic. Stroudsburg:Association for Computational Linguistics, 2018: 650-655.
[24]	WANG S , JIANG J . Learning natural language inference with LSTM[C]// The 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2016: 1442-1451.
[25]	SHA L , CHANG B B , SUI Z F ,et al. Reading and thinking:reread LSTM unit for textual entailment recognition[C]// The 26th International Conference on Computational Linguistics (COLING). Stroudsburg:Association for Computational Linguistics, 2016: 2870-2879.
[26]	GONG Y , LUO H , ZHANG J ,et al. Natural language inference over interaction space[C]// The 6th International Conference On Learning Representations.[S.l.:s.n]. 2018.
[27]	GHAEINI R , HASAN S A , DATLA V ,et al. DR-BiLSTM:dependent reading bidirectional LSTM for natural lnaguage inference[C]// The 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2018: 1460-1469.
[28]	CHEN Q , ZHU X D , LING Z H ,et al. Neural natural language inference models enhanced with external knowledge[C]// The 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2018: 2406-2417.
[29]	GURURANGAN S , SWAYAMDIPTA S , LEVY O ,et al. Annotation artifacts in natural language inference data[C]// The 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg:Association for Computational Linguistics, 2018: 107-112.
[30]	SRIVASTAVA N , HINTON G , KRIZHEVSKY A ,et al. Dropout:a simple way to prevent neural network from overfitting[J]. The Journal of Machine Learning Research, 2014,15(1): 1929-1958.
[31]	BALAZS J A , MARRESE-TAYLOR E , LOYOLA P ,et al. Refing raw sentence representations for textual entailment recognition via attention[C]// The 2nd Workshop on Evaluating Vector Space Representations for NLP. Stroudsburg:Association for Computational Linguistics, 2017: 51-55.
[32]	CHEN Q , ZHU X D , LING Z H ,et al. Recurrent neural network-based sentence encoder with gated attention for natural language inference[C]// The 2nd Workshop on Evaluating Vector Space Representation for NLP. Stroudsburg:Association for Computational Linguistics, 2017: 36-40.
[33]	NIE Y , BANSAL M . Shortcut-stacked sentence encoders for multi-domain inference[C]// The 2nd workshop on Evaluating Vector Space Representations for NLP. Stroudsburg:Association for Computational Linguistics, 2017: 41-45.

Metrics

Recommended 0

No Suggested Reading articles found!

模型	训练集准确率	测试集准确率
（1）mLSTM word-by-word attention model^[24]	92.0%	86.1%
（2）decomposable attention model^[11]	90.5%	86.8%
（3）RHD with Individual Comparison^[19]	90.3%	87.4%
（4）BiMPM^[16]	90.9%	87.5%
（5）re-read LSTM^[25]	90.7%	87.5%
（6）ESIM^[12]	92.6%	88.0%
（7）DIIN^[26]	91.2%	88.0%
（8）DR-BiLSTM^[27]	94.1%	88.6%
（9）KIM^[28]	94.1%	88.6%
（10）ESIM + ELMo^[15]	—	88.7%
（11）Human Performance^[26]	—	87.7%
（12）proposed approach	95.4%	88.8%

模型	测试集准确率
模型	Matched	Mismatched
（1）BiLSTM^[22]	67.0%	67.6%
（2）Inner Att^[31]	72.1%	72.1%
（3）ESIM^[22]	72.3%	72.1%
（4）Gated-Att BiLSTM^[32]	73.2%	73.6%
（5）Shortcut-Stacked encoder^[33]	74.6%	73.6%
（6）RHD with Individual Comparison^[19]	75.1%	74.7%
（7）BiLSTM + ELMo + Attn^[18]	76.4%	76.1%
（8）KIM^[28]	77.2%	76.4%
（9）CAFé^[13]	78.7%	77.9%
（10）DIIN^[26]	78.8%	77.8%
（11）Human Performance^[26]	88.5%	89.2%
（12）proposed approach	79.0%	77.5%

模型	训练数据集	New-test-set
（1）decomposable attention model^[11]	SNLI	51.9%
	MultiNLI + SNLI	65.8%
（2）ESIM^[12]	SNLI	65.6%
	MultiNLI + SNLI	74.9%
（3）Residual-Stacked-Encoder^[33]	SNLI	62.2%
	MultiNLI + SNLI	68.2%
（4）WordNet Baseline^[23]	—	85.8%
（5）KIM^[28]	SNLI	83.5%
（6）BERT^[18]	SNLI	93.0%
（7）proposed approach	SNLI	93.4%

模型	SNLI	Matched
（1）proposed approach	88.8%	79.0%
（2）w/o context similarity	88.4%	77.9%
（3）w/o selection gate	88.6%	77.8%
（4）w/o context similarity ＆ selection gate	88.2%	76.9%
（5）w/o context similarity,selection gate ＆additional features	88.0%	76.3%

Enhancing alignment with context similarity for natural language inference

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 33

Related Articles 4

Metrics

Recommended 0

[1]	Yan CHEN, Xueqin LUO, Wei LIANG, Yongfang XIE. Depression recognition based on emotional information fused with attentional mechanism [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 600-609.
[2]	Yue YU, Xin LIU, Fangqing JIANG, Han ZHANG, Hui WANG, Wei ZENG. Exploration of the continual learning ability that supports the application ecological evolution of the large-scale pretraining Peng Cheng series open source models [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(1): 97-108.
[3]	Xin CHEN,Jiayu WU,Xue WU,Minxia ZHANG,Yujun ZHENG. An intelligent optimization scheduling method for community patrolling and investigation in epidemic situations [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(2): 126-134.
[4]	Xingkai SUN,Xiao WANG,Hao LU. Digital monitoring and modeling of activities:the IVFC case study [J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(4): 352-368.