实体摘要系统的解释性评测

doi:10.11959/j.issn.2096-0271.2021023

Abstract

Abstract:

The task of entity summarization (ES) is to select an optimum subset from a large set of triples describing an entity in a knowledge graph.ES systems often integrate many and various ES features in a complex way.While state-of-the-art ES systems have been evaluated and compared by recent benchmarking efforts, it was unclear whether and how much each constituent ES feature had contributed to the performance of an ES system.An interpretive evaluation of ES systems was proposed.Two novel evaluation metrics were proposed, feature effectiveness ratio and feature significance ratio, to characterize how much ground-truth summaries and machine-generated summaries exhibit each ES feature.Their comparison would help to interpret the performance of an ES system.Based on three benchmarks, metrics with six popular ES features were implemented, and an interpretive evaluation of nine unsupervised ES systems and two supervised ES systems were presented.The code and data are open source.

Key words: knowledge graph, entity summarization, benchmark

CLC Number:

TP31

Qingxia LIU, Junyou LI, Gong CHENG. An interpretive evaluation of entity summarization system[J]. Big Data Research, 2021, 7(3): 15-29.

Figures/Tables 7

References 23

[1]	NOY N , GAO Y Q , JAIN A ,et al. Industry-scale knowledge graphs:lessons and challenges[J]. Communications of the ACM, 2019,62(8): 36-43.
[2]	GUNARATNA K . Semantics-based summarization of entities in knowledge graphs[D]. Dayton:Wright State University, 2017.
[3]	THALHAMMER A . Linked data entity summarization[D]. Karlsruhe:Karlsruher Institut für Technologie, 2017.
[4]	GUNARATNA K , THIRUNARAYAN K , SHETH A ,et al. Gleaning types for literals in RDF triples with application to entity summarization[C]// Proceedings of the ICWE 2016. Cham:Springer, 2016: 85-100.
[5]	LIU Q X , CHENG G , GUNARATNA K ,et al. Entity summarization:state of the art and future challenges[J]. arXiv preprint, 2019,arXiv:1910.08252v1.
[6]	LIU Q X , CHENG G , GUNARATNA K ,et al. ESBM:an entity summarization BenchMark[C]// Proceedings of the ICWE 2020. Cham:Springer, 2020: 548-564.
[7]	CHENG G , TRAN T , QU Y Z . RELIN:relatedness and informativeness-based centrality for entity summarization[C]// Proceedings of the ICWE 2011. Cham:Springer, 2011: 114-129.
[8]	SYDOW M , PIKU?A M , SCHENKEL R . The notion of diversity in graphical entity summarisation on semantic knowledge graphs[J]. Journal of Intelligent Information Systems, 2013,41(2): 109-149.
[9]	GUNARATNA K , THIRUNARAYAN K , SHETH A . FACES:diversity-aware entity summarization using incremental hierarchical conceptual clustering[C]// Proceedings of the 29th AAAI Conference on Artificial Intelligence. Palo Alto:AAAI Press, 2015: 116-122.
[10]	XU D Y , ZHENG L , QU Y Z . CD at ENSEC 2016:Generating characteristic and diverse entity summaries[C]// Proceedings of the ESWC 2016.[S.l.:s.n.], 2016.
[11]	THALHAMMER A , LASIERRA N , RETTINGER A . LinkSUM:using link analysis to summarize entity data[C]// Proceedings of the ICWE 2016. Cham:Springer, 2016: 244-261.
[12]	KROLL H , NAGEL D , BALKE W T . BAFREC:balancing frequency and rarity for entity characterization in linked open data[C]// Proceedings of the 1st International Workshop on Entity Retrieval.[S.l.:s.n.], 2018.
[13]	POURIYEH S , ALLAHYARI M , KOCHUT K ,et al. ES-LDA:entity summarization using knowledge-based topic modeling[C]// Proceedings of the 8th International Joint Conference on Natural Language Processing.[S.l.:s.n.], 2017: 316-325.
[14]	POURIYEH S , ALLAHYARI M , KOCHUT K J ,et al. Combining word embedding and knowledge-based topic modeling for entity summarization[C]// Proceedings of the 12th International Conference on Semantic Computing. Piscataway:IEEE Press, 2018: 252-255.
[15]	WEI D J , GAO S Y , LIU Y X ,et al. MPSUM:entity summarization with predicate-based matching[J]. arXiv preprint, 2020,arXiv:2005.11992.
[16]	KIM E , CHOI K S . Entity summarization based on formal concept analysis[C]// Proceedings of the EYRE 2018.[S.l.:s.n.], 2018.
[17]	WEI D J , LIU Y X , ZHU F Q ,et al. ESA:entity summarization with attention[J]. arXiv preprint, 2019,arXiv:1905.10625.
[18]	LIU Q X , CHENG G , QU Y Z . DeepLENS:deep learning for entity summarization[J]. arXiv preprint, 2020arXiv:2003.03736.
[19]	THALHAMMER A , KNUTH M , SACK H . Evaluating entity summarization using a game-based ground truth[C]// Proceedings of the ISWC 2012. Heidelberg:Springer, 2012: 350-361.
[20]	LANGER P , SCHULZE P , GEORGE S ,et al. Assigning global relevance scores to DBpedia facts[C]// Proceedings of the 2014 IEEE 30th International Conference on Data Engineering Workshops. Piscataway:IEEE Press, 2014: 248-253.
[21]	WAITELONIS J , SACK H , BOBIC T . FRanCo-a ground truth corpus for fact ranking evaluation[C]// Proceedings of the 1st International Workshop on Summarizing and Presenting Entities and Ontologies.[S.l.:s.n.], 2015.
[22]	RDF Working Group . W3C:resource description framework (RDF)[R]. 2014.
[23]	STOILOS G , STAMOU G , KOLLIAS S . A string metric for ontology alignment[C]// Proceedings of the ISWC 2005. Heidelberg:Springer, 2005: 624-637.

Metrics

Recommended 0

No Suggested Reading articles found!

评测集	ESBM-D	ESBM-L	FED
RELIN	0.242±0.120 -?-↓↓-↓↓↓↓↓	0.203±0.125 -?-↓?-↓?↓↓↓	0.118±0.080 -????↓↓?↓↓↓
DIVERSUM	0.249±0.136 ?--?↓-↓↓↓↓↓	0.207±0.127 ?--↓?-↓?↓↓↓	0.113±0.079 ?-???↓↓?↓↓↓
FACES	不适用	不适用	0.146±0.090 ??-??↓???↓↓
FACES-E	0.280±0.142 ↑?--?-↓↓↓↓↓	0.313±0.116 ↑↑--↑-↓↑??↓	0.146±0.090 ???-?↓???↓↓
CD	0.283±0.134 ↑↑-?--↓↓?↓↓	0.217±0.101 ??-↓--↓??↓↓	0.138±0.076 ????-↓↓??↓↓
LinkSUM	不适用	不适用	0.237±0.120 ↑↑↑↑↑-?↑↑??
BAFREC	0.335±0.140 ↑↑-↑↑--???↓	0.360±0.078 ↑↑-↑↑--↑↑?↓	0.195±0.098 ↑↑??↑?-??↓↓
KAFCA	0.314±0.128 ↑↑-↑↑-?-??↓	0.244±0.117 ??-↓?-↓-?↓↓	0.143±0.086 ?????↓?-?↓↓
MPSUM	0.314±0.123 ↑↑-↑?-??-?↓	0.272±0.112 ↑↑-??-↓?-↓↓	0.169±0.103 ↑↑???↓??-↓↓
ESA	0.331±0.159 ↑↑-↑↑-???-↓	0.350±0.143 ↑↑-?↑-?↑↑-↓	0.247±0.132 ↑↑↑↑↑?↑↑↑-?
DeepLENS	0.404±0.126 ↑↑-↑↑-↑↑↑↑	0.469±0.085 ↑↑-↑↑-↑↑↑↑	0.279±0.101 ↑↑↑↑↑?↑↑↑?

测评集	LFoP	DoP	GFoV	IoPV	GFoP	DoV
ESBM-D	0.561±0.165 ↓	2.478±0.747↑	0.759±0.125 ↓	1.275±0.175 ↑	0.913±0.052↓	1.016±0.054 ↑
ESBM-L	0.581±0.125 ↓	3.093±2.394↑	1.349±0.188 ↑	0.864±0.057 ↓	0.998±0.029	1.061±0.068 ↑
FED	0.821±0.205 ↓	1.699±0.480↑	1.148±0.153 ↑	0.958±0.044 ↓	1.012±0.066	1.016±0.046

摘要系统	LFoP	GFoP	GFoV	IoPV	DoP	DoV
RELIN	0.280±0.228	0.869±0.075	0.277±0.098	1.801±0.297	2.351±0.791 ?	0.749±0.253
DIVERSUM	0.649±0.192	0.910±0.048 ?	0.854±0.167	1.175±0.198	2.753±0.925	1.037±0.086
FACES-E	0.623±0.281 ?	0.914±0.079 ?	0.911±0.208	1.142±0.241	2.494±0.881 ?	0.972±0.118
CD	0.334±0.181	0.863±0.062	0.414±0.136	1.620±0.250	2.742±0.886	1.061±0.051
BAFREC	0.585±0.169 ?	0.954±0.056	0.908±0.177	1.117±0.198	2.586±0.755	0.980±0.107
KAFCA	0.361±0.248	0.850±0.083	0.646±0.244	1.377±0.294	2.505±0.829 ?	0.993±0.116 ?
MPSUM	0.434±0.201	0.876±0.072	0.730±0.250 ?	1.304±0.291 ?	2.742±0.886	0.891±0.187
ESA	0.266±0.212	0.848±0.084	0.529±0.179	1.535±0.282	2.303±0.827	0.930±0.160
DeepLENS	0.302±0.219	0.854±0.076	0.656±0.190	1.407±0.267	2.415±0.801 ?	0.957±0.115
FER	0.561±0.165	0.913±0.052	0.759±0.125	1.275±0.175	2.478±0.747	1.016±0.054

摘要系统	LFoP	GFoP	GFoV	IoPV	DoP	DoV
RELIN	0.688±0.432 ?	0.991±0.047 ?	0.600±0.137	1.154±0.050	2.775±2.012 ?	0.967±0.189
DIVERSUM	0.870±0.381	0.993±0.038 ?	1.006±0.247	0.991±0.077	3.869±3.357	1.091±0.075
FACES-E	0.536±0.163 ?	0.962±0.079	1.341±0.296 ?	0.872±0.092 ?	3.848±3.352	1.059±0.103 ?
CD	0.470±0.199	0.996±0.073 ?	1.009±0.212	0.959±0.079	3.869±3.357	1.102±0.071
BAFREC	0.562±0.201 ?	1.020±0.053	1.598±0.491	0.781±0.144	3.485±3.228	1.007±0.088
KAFCA	0.234±0.200	0.954±0.056	1.309±0.395 ?	0.884±0.108 ?	3.869±3.357	1.104±0.102
MPSUM	0.568±0.201 ?	0.979±0.046	1.249±0.428 ?	0.908±0.131 ?	3.869±3.357	1.083±0.104 ?
ESA	0.514±0.235 ?	1.029±0.037	1.241±0.352 ?	0.892±0.116 ?	3.125±2.613 ?	1.013±0.154 ?
DeepLENS	0.361±0.163	1.004±0.037 ?	1.412±0.409 ?	0.840±0.129 ?	3.496±2.343	1.000±0.081
FER	0.581±0.125	0.998±0.029	1.349±0.188	0.864±0.057	3.093±2.394	1.061±0.068

摘要系统	LFoP	GFoP	GFoV	IoPV	DoP	DoV
RELIN	0.911±0.481 ?	1.028±0.157 ?	0.652±0.329	1.123±0.097	1.473±0.579	0.761±0.209
DIVERSUM	1.339±0.220	0.962±0.056	1.043±0.191	0.989±0.069 ?	1.783±0.517	0.981±0.097 ?
FACES	0.860±0.314 ?	0.936±0.081	1.489±0.245	0.886±0.084	1.714±0.514 ?	1.019±0.124 ?
FACES-E	0.860±0.314 ?	0.936±0.081	1.489±0.245	0.886±0.084	1.714±0.514 ?	1.019±0.124 ?
CD	0.799±0.206 ?	1.042±0.075	0.699±0.226	1.118±0.076	1.783±0.517	1.060±0.066
LinkSUM	0.976±0.353	0.987±0.080 ?	1.656±0.250	0.797±0.089	1.460±0.474	1.062±0.074
BAFREC	0.928±0.273	0.949±0.078	1.658±0.304	0.811±0.089	1.768±0.516	0.975±0.119 ?
KAFCA	0.636±0.248	0.999±0.116 ?	0.864±0.363	1.024±0.092	1.699±0.518 ?	0.909±0.134
MPSUM	0.878±0.245 ?	0.918±0.067	1.344±0.289	0.949±0.095 ?	1.783±0.517	0.821±0.225
ESA	0.842±0.323 ?	1.090±0.113	0.813±0.232	1.039±0.075	1.378±0.408	0.875±0.136
DeepLENS	0.823±0.476 ?	1.056±0.095	1.166±0.375 ?	0.926±0.124 ?	1.481±0.486	0.863±0.131
FER	0.821±0.205	1.012±0.066	1.148±0.153	0.958±0.044	1.699±0.480	1.016±0.046

An interpretive evaluation of entity summarization system

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 23

Related Articles 15

Metrics

Recommended 0

[1]	Yuqi ZHANG, Xiaowen HUANG, Jitao SANG. Knowledge-enhanced policy-guided interactive reinforcement recommendation system [J]. Big Data Research, 2022, 8(5): 88-105.
[2]	Jingye SHI, Pan LI. Key technologies and application exploration of aerospace big data in the construction of new smart city [J]. Big Data Research, 2022, 8(2): 120-133.
[3]	Ling TAN, Haihong E, Zemin KUANG, Meina SONG, Yu LIU, Zhengyu CHEN, Xiaoxuan XIE, Jundi LI, Jiawei FAN, Qingchuan WANG, Xiaoyang KANG. Key technologies and research progress of medical knowledge graph construction [J]. Big Data Research, 2021, 7(4): 80-104.
[4]	Yuming SHEN, Jianfeng DU. Temporal knowledge graph completion:methods and progress [J]. Big Data Research, 2021, 7(3): 30-41.
[5]	Huifang DU, Haofen WANG, Yinghui SHI, Meng WANG. Progress, challenges and research trends of reasoning in multi-hop knowledge graph based question answering [J]. Big Data Research, 2021, 7(3): 60-79.
[6]	Zhilei HU, Xiaolong JIN, Jianyun CHEN, Guanli HUANG. Construction, reasoning and applications of event graphs [J]. Big Data Research, 2021, 7(3): 80-96.
[7]	Huajun CHEN, Wen ZHANG, Chi-Man WONG, Ganqiang YE, Bo WEN, Wei ZHANG. Large scale pre-trained knowledge graph model and e-commerce application [J]. Big Data Research, 2021, 7(3): 97-115.
[8]	Qiang CHEN, Shiya DAI. Recognition method of accounting fraud risk based on financial knowledge graph [J]. Big Data Research, 2021, 7(3): 116-129.
[9]	Bing XIE, Xin PENG, Gang YIN, Xuandong LI, Jun WEI, Hailong SUN. Big data based intelligent software development methodology and environment [J]. Big Data Research, 2021, 7(1): 3-21.
[10]	Yanzhen ZOU, Min WANG, Bing XIE, Zeqi LIN. Software knowledge graph construction and Q＆amp;A technology based on big data [J]. Big Data Research, 2021, 7(1): 22-36.
[11]	Wei CHEN, Hongjie YE, Jiahong ZHOU, Jun WEI. An approach to automatically building Docker images by using domain knowledge [J]. Big Data Research, 2021, 7(1): 64-75.
[12]	Jian ZHANG, Xiangxin MENG, Hailong SUN, Xu WANG, Xudong LIU. Data driven intelligent collaboration of software developers [J]. Big Data Research, 2021, 7(1): 76-93.
[13]	Cheng CHEN, Yueguo CHEN, Chen LIU, Xiaotong LYU, Xiaoyong DU. Constructing and analyzing intention knowledge graphs [J]. Big Data Research, 2020, 6(2): 57-68.
[14]	Genlin ZANG, Yaqiang WANG, Qingrong WU, Chunli ZHAN, Yi LI. Model and construction method of the ontology of knowledge graph of smart city [J]. Big Data Research, 2020, 6(2): 96-106.
[15]	Ying LIANG, Wei ZHANG, Zhidong YU, Hongzhou SHI. Applications of academic big data in the process of science and technology management [J]. Big Data Research, 2019, 5(5): 3-15.