大数据 ›› 2021, Vol. 7 ›› Issue (3): 15-29.doi: 10.11959/j.issn.2096-0271.2021023

• 专题:基于大数据的知识图谱及其应用 • 上一篇    下一篇

实体摘要系统的解释性评测

刘庆霞, 李俊宥, 程龚   

  1. 南京大学计算机软件新技术国家重点实验室,江苏 南京 210023
  • 出版日期:2021-05-15 发布日期:2021-05-01
  • 作者简介:刘庆霞(1990- ),女,南京大学计算机软件新技术国家重点实验室博士生,主要研究方向为数据摘要和智能问答。
    李俊宥(1996- ),男,南京大学计算机软件新技术国家重点实验室硕士生,主要研究方向为数据摘要和强化学习。
    程龚(1984- ),男,博士,南京大学计算机软件新技术国家重点实验室副教授,主要研究方向为语义搜索、数据摘要、智能问答。
  • 基金资助:
    国家重点研发计划资助项目(2018YFB1004300);国家自然科学基金资助项目(62072224)

An interpretive evaluation of entity summarization system

Qingxia LIU, Junyou LI, Gong CHENG   

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China
  • Online:2021-05-15 Published:2021-05-01
  • Supported by:
    The National Key Research and Development Program of China(2018YFB1004300);The National Natural Science Foundation of China(62072224)

摘要:

任务是从知识图谱中描述实体的大量三元组中选取最优子集作为摘要。现有实体摘要系统通常以较复杂的方式集成多种摘要技术特征。已开展的评测工作对现有系统进行了总体效果的评测和对比,但未能解释系统所用各摘要特征对最终效果的作用。为此,提出对实体摘要系统开展解释性评测。提出两种新指标:特征效用率和特征显著率,两者分别度量各摘要特征在标准摘要和系统生成摘要中的显示度,两者对比分析的结果在一定程度上可为系统取得的最终效果提供解释。基于3个评测集实现了这种评测新方法,运用6种常见的摘要特征,对9个非监督实体摘要系统和两个有监督实体摘要系统进行了解释性评测,相关代码和数据已开源。

关键词: 知识图谱, 实体摘要, 评测集

Abstract:

The task of entity summarization (ES) is to select an optimum subset from a large set of triples describing an entity in a knowledge graph.ES systems often integrate many and various ES features in a complex way.While state-of-the-art ES systems have been evaluated and compared by recent benchmarking efforts, it was unclear whether and how much each constituent ES feature had contributed to the performance of an ES system.An interpretive evaluation of ES systems was proposed.Two novel evaluation metrics were proposed, feature effectiveness ratio and feature significance ratio, to characterize how much ground-truth summaries and machine-generated summaries exhibit each ES feature.Their comparison would help to interpret the performance of an ES system.Based on three benchmarks, metrics with six popular ES features were implemented, and an interpretive evaluation of nine unsupervised ES systems and two supervised ES systems were presented.The code and data are open source.

Key words: knowledge graph, entity summarization, benchmark

中图分类号: 

No Suggested Reading articles found!