基于预训练机制的自修正复杂语义分析方法

doi:10.11959/j.issn.1000-436x.2019195

通信学报 ›› 2019, Vol. 40 ›› Issue (12): 41-50.doi: 10.11959/j.issn.1000-436x.2019195

基于预训练机制的自修正复杂语义分析方法

李青¹,钟将¹(),李立力²,李琪³

¹ 重庆大学计算机学院，重庆 400044
² 重庆大学土木工程学院，重庆 400044
³ 绍兴文理学院计算机科学与工程系，浙江绍兴 312000

修回日期:2019-10-30 出版日期:2019-12-25 发布日期:2020-01-16
作者简介:李青（1989- ），女，陕西西安人，重庆大学博士生，主要研究方向为自然语言处理、复杂事件检测、医学信息学|钟将（1974- ），男，重庆人，博士，重庆大学教授，主要研究方向为自然语言处理、数据挖掘|李立力（1989- ），男，陕西铜川人，博士，重庆大学博士生，主要研究方向为桥梁健康监测、数据挖掘|李琪（1987- ），男，江苏盱眙人，博士，绍兴文理学院讲师，主要研究方向为图计算、数据挖掘
基金资助:
中央高校研究生科研创新基金资助项目(2018CDYJSY0055);重庆市研究生科研创新基金资助项目(2017YFB1402400);重庆市研究生科研创新基金资助项目(CYB18058);重庆市技术创新与应用示范基金资助项目(cstc2018jszx-cyzdX0086)

Self-correcting complex semantic analysis method based on pre-training mechanism

Qing LI¹,Jiang ZHONG¹(),Lili LI²,Qi LI³

¹ College of Computer Science,Chongqing University,Chongqing 400044,China
² School of Civil Engineering,Chongqing University,Chongqing 400044,China
³ Department of Computer Science and Engineering,Shaoxing University,Shaoxing 312000,China

Revised:2019-10-30 Online:2019-12-25 Published:2020-01-16
Supported by:
Fundamental Research Funds for the Central Universities(2018CDYJSY0055);The National Key Research and Development Program of China(2017YFB1402400);The National Key Research and Development Program of China(CYB18058);Chongqing Technological Innovation and Application Demonstration Project(cstc2018jszx-cyzdX0086)

摘要/Abstract

摘要：

面向知识服务过程中内容资源的智能化、知识化、精细化和重组化的碎片性管理需求。深层分析并挖掘语义隐层知识、技术、经验与信息，突破已有传统文本到结构化查询语言（SQL）的语义分析技术瓶颈，提出基于预训练机制的自修正复杂语义分析方法PT-Sem2SQL。设计结合Kullback-Leibler差异技术的MT-DNN预训练机制，以加强上下文语义理解深度；设计专有增强模块，捕获句内上下文语义信息的位置；并通过自修正方法优化生成模型的执行过程，以解决解码过程中的错误输出。实验结果表明，PT-Sem2SQL 能够有效提高复杂语义的解析性能，准确度优于相关工作。

关键词: 文本到SQL, 语义分析, 自然语言处理, 复杂事件处理

Abstract:

In the process of knowledge service,in order to meet the fragmentation management needs of intellectualization,knowledge ability,refinement and reorganization content resources.Through deep analysis and mining of semantic hidden knowledge,technology,experience,and information,it broke through the existing bottleneck of traditional semantic parsing technology from Text-to-SQL.The PT-Sem2SQL based on the pre-training mechanism was proposed.The MT-DNN pre-training model mechanism combining Kullback-Leibler technology was designed to enhance the depth of context semantic understanding.A proprietary enhancement module was designed that captured the location of contextual semantic information within the sentence.Optimize the execution process of the generated model by the self-correcting method to solve the error output during decoding.The experimental results show that PT-Sem2SQL can effectively improve the parsing performance of complex semantics,and its accuracy is better than related work.

Key words: Text-to-SQL, semantic parsing, natural language processing, complex event processing

中图分类号:

TP302.1

李青,钟将,李立力,李琪. 基于预训练机制的自修正复杂语义分析方法[J]. 通信学报, 2019, 40(12): 41-50.

Qing LI,Jiang ZHONG,Lili LI,Qi LI. Self-correcting complex semantic analysis method based on pre-training mechanism[J]. Journal on Communications, 2019, 40(12): 41-50.

图/表 10

图1

表1

图2

图3

图4

表2

图5

表3

表4

表5

参考文献 38

[1]	CHOI E , HE H , IYYER M ,et al. QuAC:question answering in context[C]// The 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 2174-2184.
[2]	SUHR A , IYER S , ARTZI Y . Learning to map context-dependent sentences to executable formal queries[C]// The 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 1 (Long Papers). 2018: 2238-2249.
[3]	李青, 钟将, 李立力 ,等. 领域专业知识富关联关系提取方法研究[J]. 控制与决策, 2019: 1-9.
	LI Q , ZHONG J , LI L L ,et al. Research on the extraction method of multiple semantic relations in domain knowledge[J]. Control and De-cision, 2019: 1-9.
[4]	孙加东, 赵铁军, 梁华参 . 基于结构对齐的统计机器翻译模型[J]. 通信学报, 2009,30(7): 124-129.
	SUN J D , ZHAO T J , LIANG H S . Statistical machine translation model based on structure alignment[J]. Journal on Communications, 2009,30(7): 124-129.
[5]	LI Q , ZHONG J , TAO Y ,et al. Research of the processing technology for time complex event based on LSTM[J]. Cluster Computing, 2019,22: 9571-9579.
[6]	GUO Y , SHANG X , LI Z . Identification of cancer subtypes by integrating multiple types of transcriptomics data with deep learning in breast cancer[J]. Neurocomputing, 2019,324: 20-30.
[7]	YU T , ZHANG R , YANG K ,et al. Spider:a large-scale human-labeled dataset for complex and cross-domain semantic parsing and Text-to-SQL task[C]// The 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 3911-3921.
[8]	YU T , ZHANG R , YASUNAGA M ,et al. SParC:cross-domain semantic parsing in context[J]. arXiv Preprint,arXiv:1906.02285, 2019.
[9]	ZHONG V , XIONG C , SOCHER R . Seq2sql:generating structured queries from natural language using reinforcement learning[J]. arXiv Preprint,arXiv:1709.00103, 2017.
[10]	United States.Defense advanced research projects agency . speech and natural language:proceedings of a workshop held at Hidden Valley,Pennsylvania,June 24-27,1990[M]. Morgan Kaufmann Pub, 1990.
[11]	DAHL D A , BATES M , BROWN M ,et al. Expanding the scope of the ATIS task:The ATIS-3 corpus[C]// The Workshop on Human Language Technology. Association for Computational Linguistics, 1994: 43-48.
[12]	FINEGAN-DOLLAK C , KUMMERFELD J K , ZHANG L ,et al. Improving Text-to-SQL Evaluation Methodology[C]// The 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). 2018: 351-360.
[13]	DEVLIN J , CHANG M W , LEE K ,et al. BERT:pre-training of deep bidirectional transformers for language understanding[C]// The 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 1 (Long and Short Papers). 2019: 4171-4186.
[14]	LIU X , HE P , CHEN W ,et al. Multi-task deep neural networks for natural language understanding[J]. arXiv Preprint,arXiv:1901.11504, 2019.
[15]	SOLOMON J , CHARETTE F . Hierarchical multi-task deep neural network architecture for end-to-end driving[J]. arXiv Preprint,arXiv:1902.03466, 2019.
[16]	KULLBACK S . Letter to the editor:the Kullback-Leibler distance[J]. American Statistician, 1987,41(4): 340-341.
[17]	WARREN D H D , PEREIRA F C N . An efficient easily adaptable system for interpreting natural language queries[J]. Computational Linguistics, 1982,8(3-4): 110-122.
[18]	WONG Y W , MOONEY R . Learning synchronous grammars for semantic parsing with lambda calculus[C]// The 45th Annual Meeting of the Association of Computational Linguistics. 2007: 960-967.
[19]	ZELLE J M , MOONEY R J . Learning to parse database queries using inductive logic programming[C]// The National Conference on Artificial Intelligence. 1996: 1050-1055.
[20]	ZETTLEMOYER L S , COLLINS M . Learning to map sentences to logical form:structured classification with probabilistic categorial grammars[C]// The Twenty-First Conference on Uncertainty in Artificial Intelligence. AUAI Press, 2005: 658-666.
[21]	ZHAO K , HUANG L . Type-driven incremental semantic parsing with polymorphism[C]// The 2015 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. 2015: 1416-1421.
[22]	HERZIG J , BERANT J . Decoupling structure and lexicon for zero-shot semantic parsing[C]// The 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 1619-1629.
[23]	SUTSKEVER I , VINYALS O , LE Q V . Sequence to sequence learning with neural networks[C]// Advances in Neural Information Processing Systems. 2014: 3104-3112.
[24]	BAHDANAU D , CHO K , BENGIO Y . Neural machine translation by jointly learning to align and translate[C]// International Conference on Learning Representations,abs/1409. 0473, 2014.
[25]	YU T , LI Z , ZHANG Z ,et al. TypeSQL:knowledge-based type-aware neural Text-to-SQL generation[C]// The 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 2 (Short Papers). 2018: 588-594.
[26]	DONG L , LAPATA M . Coarse-to-fine decoding for neural semantic parsing[C]// The 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). 2018: 731-742.
[27]	WANG C , BROCKSCHMIDT M , SINGH R . Pointing out SQL queries from text[J]. 2018:
[28]	PENNINGTON J , SOCHER R , MANNING C . Glove:global vectors for word representation[C]// The 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014: 1532-1543.
[29]	HWANG W , YIM J , PARK S ,et al. A comprehensive exploration on WikiSQL with table-aware word contextualization[J]. arXiv Preprint,arXiv:1902.01069, 2019.
[30]	RADFORD A , NARASIMHAN K , SALIMANS T ,et al. Improving language understanding by generative pre-training[J]. American Statistician, 2018.
[31]	YU T , YASUNAGA M , YANG K ,et al. SyntaxSQLNet:Syntax tree networks for complex and cross-domain Text-to-SQL task[C]// The 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 1653-1663.
[32]	TANG L R , MOONEY R J . Using multiple clause constructors in inductive logic programming for semantic parsing[C]// European Conference on Machine Learning. Springer, 2001: 466-477.
[33]	POPESCU A M , ETZIONI O , KAUTZ H . Towards a theory of natural language interfaces to databases[C]// The 8th International Conference on Intelligent User Interfaces. ACM, 2003: 149-157.
[34]	ZELLE J M , MOONEY R J . Learning to parse database queries using inductive logic programming[C]// The National Conference on Artificial Intelligence. 1996: 1050-1055.
[35]	IYER S , KONSTAS I , CHEUNG A ,et al. Learning a neural semantic parser from user feedback[J]. arXiv Preprint,arXiv:1704.08760, 2017.
[36]	LI F , JAGADISH H V . Constructing an interactive natural language interface for relational databases[J]. Proceedings of the VLDB Endowment, 2014,8(1): 73-84.
[37]	YAGHMAZADEH N , WANG Y , DILLIG I ,et al. SQLizer:query synthesis from natural language[J]. Proceedings of the ACM on Programming Languages, 2017,1(OOPSLA):63.
[38]	SMITH S L , KINDERMANS P J , YING C ,et al. Don’t decay the learning rate,increase the batch size[J]. arXiv Preprint,arXiv:1711.00489, 2017,41(4): 340-341.

数据集	CONTEXT	CROSS DOMAIN	QUESTION /个	TURN	QUESTION LENGTH	QUESTION VOCABULARY	SELECT	WHEREGROUP BYORDER BY	KEYWORDS
Spider	NO	YES	11 840	1.0	13.4	4 818	98.9%	55.2% 24.0% 21.5%	68.6%
SParC	YES	YES	12 726	3.0	8.1	3 794	97.4%	42.8% 20.1% 17.0%	41.2%

模型方法	测试集					验证集
模型方法	Easy	Medium	Hard	Extra Hard	All	All
seq2seq	11.9%	1.9%	1.3%	0.5%	3.7%	1.9%
SQLNet	26.2%	12.6%	6.6%	1.3%	12.4%	10.9%
TypeSQL	19.6%	7.6%	3.8%	0.8%	8.2%	8.0%
IncSQL	29.5%	13.1%	12.3%	2.5%	14.6%	13.2%
SyntaxSQLNet	48.0%	27.0%	24.3%	4.6%	27.2%	24.8%
SQLove	56.7%	32.9%	22.3%	4.3%	29.1%	27.9%
PT-Sem2SQL-SE	68.1%	40.2%	31.6%	7.3%	36.8%	33.7%
PT-Sem2SQL	73.2%	45.1%	36.2%	9.2%	40.9%	38.1%

模型方法	SELECT	WHERE	GROUP BY	ORDER BY	KEYWORDS
seq2seq	13.0%	1.5%	3.3%	5.3%	8.7%
SQLNet	44.5%	19.8%	29.5%	48.8%	64.0%
TypeSQL	36.4%	16.0%	17.2%	47.7%	66.2%
IncSQL	59.2%	41.9%	23.4%	52.7%	67.3%
SyntaxSQLNet	62.5%	34.8%	55.6%	60.9%	69.6%
SQLove	78.4%	47.8%	61.6%	66.2%	74.6%
PT-Sem2SQL-SE	81.3%	50.4%	68.0%	69.1%	77.4%
PT-Sem2SQL	89.2%	65.9%	77.1%	75.6%	86.1%

模型方法	测试集					验证集
模型方法	Easy	Medium	Hard	Extra Hard	All	All
SyntaxSQLNet	38.4%	8.7%	2.4%	1.3%	12.7%	11.9%
CD-Seq2Seq	36.1%	8.4%	3.7%	2.1%	12.6%	11.8%
SQLove	32.2%	6.9%	1.3%	0.5%	10.2%	8.9%
PT-Sem2SQL-SE	46.3%	23.8%	12.4%	7.3%	22.5%	19.7%
PT-Sem2SQL	53.1%	28.4%	15.5%	9.2%	26.6%	24.1%

模型方法	问题匹配度		上下文相关交互匹配度
模型方法	验证集	测试集	验证集	测试集
SyntaxSQLNet	18.5%	20.2%	4.3%	5.2%
CD-Seq2Seq	21.9%	23.2%	8.1%	7.5%
SQLove	20.3%	22.0%	6.9%	6.4%
PT-Sem2SQL-SE	29.7%	32.7%	13.1%	11.9%
PT-Sem2SQL	32.5%	34.1%	15.8%	13.4%

基于预训练机制的自修正复杂语义分析方法

Self-correcting complex semantic analysis method based on pre-training mechanism

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 38

相关文章 5

Metrics

推荐阅读 0

[1]	刘敖迪, 杜学绘, 王娜, 乔蕊. 基于深度学习的ABAC访问控制策略自动化生成技术[J]. 通信学报, 2020, 41(12): 8-20.
[2]	黄杨琛,贾焰,甘亮,徐菁,黄九鸣,赫中翮. 基于远程监督的多因子人物关系抽取模型[J]. 通信学报, 2018, 39(7): 103-112.
[3]	耿少峰,王永恒,李仁发,张佳. 主动式复杂事件处理方法的研究[J]. 通信学报, 2016, 37(9): 111-120.
[4]	陈皓,李瑜,虎嵩林,梁英. 基于S4框架的并行复杂事件处理系统[J]. 通信学报, 2012, 33(Z1): 165-169.
[5]	廖建新,刘秀磊,朱晓民,孙海峰,王敬宇. 扩展结构包含推理算法的本体匹配[J]. 通信学报, 2012, 33(8): 190-199.