基于BERT阅读理解框架的司法要素抽取方法

doi:10.11959/j.issn.2096-0271.2021057

Abstract

Abstract:

Extraction of legal elements is an important basis for judicial intelligent auxiliary applications, and its purpose is to identify the key elements involved in the judgment document.In the past, extracting legal elements usually used multi-label classification methods for modeling.These methods mainly relied on the text features of the judgment document, thereby ignoring the label features.Besides, due to the imbalanced data problem in judicial data sets, the classification method will lead to poor model performance because of too many negative examples.To solve the above problems, a legal element extraction method based on BERT reading comprehension framework was proposed.This method constructed auxiliary questions with label information and legal prior knowledge, and used the machine reading comprehension model based on BERT to establish the semantic associations between question and judgment document.And this method added special tokens before and after the label in the question to enhance the learning ability of the model.Experiments were conducted on the legal element extraction data sets of the CAIL2019.Experiment results show that the performance is improved significantly, and the F1 value has been increased by 2.7%, 11.3%, and 5.6% respectively on the data sets of marriage and family case, labor dispute case, and loan contract dispute case.

Key words: element extraction, machine reading comprehension, neural network, BERT

CLC Number:

TP391.1

Hui HUANG, Yongbin QIN, Yanping CHEN, Ruizhang HUANG. Legal element extraction method based on BERT reading comprehension framework[J]. Big Data Research, 2021, 7(6): 19-29.

Figures/Tables 10

模型	特征			婚姻家庭		劳动争议		借款合同
模型	标签	标识符	法律先验知识	F1-micro	F1-macro	F1-micro	F1-macro	F1-micro	F1-macro
RBT-CLS	—	—	—	84.2%	67.9%	80.4%	42.4%	80.1%	45.3%
	√	—	—	84.6%	72.1%	81.9%	59.3%	81.0%	57.3%
RBT-MRC	√	√	—	84.9%	72.3%	81.9%	62.2%	81.1%	57.7%
	√	√	√	$85 . 1 %$	$73 . 0 %$	$82 . 0 %$	$64 . 1 %$	$81 . 4 %$	$59 . 1 %$

References 27

[1]	ELISSEEFF A , WESTON J . A kernel method for multi-labelled classification[C]// Proceedings of the 14th International Conference on Neural Information Processing Systems:Natural and Synthetic. Cambridge:MIT Press, 2001: 681-687.
[2]	GHAMRAWI N , MCCALLUM A . Collective multi-label classification[C]// Proceedings of the 14th ACM International Conference on Information and Knowledge Management. New York:ACM Press, 2005: 195-200.
[3]	LI C , WANG B , PAVLU V ,et al. Conditional bernoulli mixtures for multilabel classification[C]// Proceedings of the 2016 International Conference on Machine Learning.[S.l.:s.n.], 2016: 2482-2491.
[4]	KIM Y . Convolutional neural networks for sentence classification[J]. arXiv preprint,2014,arXiv:1408.5882.
[5]	CONNEAU A , SCHWENK H , BARRAULT L ,et al. Very deep convolutional networks for text classification[J]. arXiv preprint,2016,arXiv:1606.01781.
[6]	SUN X , MA X H , NI Z W ,et al. A new LSTM network model combining TextCNN[C]// Proceedings of the 2018 International Conference on Neural Information Processing. Heidelberg:Springer, 2018: 416-424.
[7]	LIN J Y , SU Q , YANG P C ,et al. Semantic-unit-based dilated convolution for multi-label text classification[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2018.
[8]	PETERS M E , NEUMANN M , IYYER M ,et al. Deep contextualized word representations[J]. arXiv preprint,2018,arXiv:1802.05365.
[9]	RADFORD A , NARASIMHAN K , SALIMANS T ,et al. Improving language understanding by generative pretraining[Z]. 2018.
[10]	DEVLIN J , CHANG M W , LEE K ,et al. BERT:pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint,2018,arXiv:1810.04805.
[11]	YANG Z L , DAI Z H , YANG Y M ,et al. XLNet:generalized autoregressive pretraining for language understanding[J]. arXiv preprint,2019,arXiv:1906.08237.
[12]	LUO B F , FENG Y S , XU J B ,et al. Learning to predict charges for criminal cases with legal basis[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2017: 2727-2736.
[13]	ZHONG H X , GUO Z P , TU C C ,et al. Legal judgment prediction via topological learning[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2018: 3540-3549.
[14]	陈文哲, 秦永彬, 黄瑞章 ,等. 基于犯罪行为序列的法律条文预测方法[J]. 计算机工程与应用, 2019,55(22): 245-249,264.
	CHEN W Z , QIN Y B , HUANG R Z ,et al. Legal text prediction method based on criminal behavior sequence[J]. Computer Engineering and Applications, 2019,55(22): 245-249,264.
[15]	秦永彬, 冯丽, 陈艳平 ,等. “智慧法院”数据融合分析与集成应用[J]. 大数据, 2019,5(3): 35-46.
	QIN Y B , FENG L , CHEN Y P ,et al. “Intelligent Court” data fusion analysis and integrated application[J]. Big Data Research, 2019,5(3): 35-46.
[16]	Natural Language Computing Group,Microsoft Research Asia. R-NET:machine reading comprehension with self-matching networks[R]. 2017.
[17]	SEO M , KEMBHAVI A , FARHADI A ,et al. Bidirectional attention flow for machine comprehension[J]. arXiv preprint,2016,arXiv:1611.01603.
[18]	LEVY O , SEO M , CHOI E ,et al. Zeroshot relation extraction via reading comprehension[J]. arXiv preprint,2017,arXiv:1706.04115.
[19]	MCCANN B , KESKAR N S , XIONG C M ,et al. The natural language decathlon:multitask learning as question answering[J]. arXiv preprint,2018,arXiv:1806.08730.
[20]	LI X Y , YIN F , SUN Z J ,et al. Entityrelation extraction as multi-turn question answering[J]. arXiv preprint,2019,arXiv:1905.05529.
[21]	刘奕洋, 余正涛, 高盛祥 ,等. 基于机器阅读理解的中文命名实体识别方法[J]. 模式识别与人工智能, 2020,33(7): 653-659.
	LIU Y Y , YU Z T , GAO S X ,et al. Chinese named entity recognition method based on machine reading comprehension[J]. Pattern Recognition and Artificial Intelligence, 2020,33(7): 653-659.
[22]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[C]// Proceedings of the 31st Conference on Neural Information Processing Systems.[S.l:s.n.], 2017: 5998-6008.
[23]	SOCHER R , PERELYGIN A , WU J ,et al. Recursive deep models for semantic compositionality over a sentiment treebank[C]// Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2013: 1631-1642.
[24]	ZHOU P , SHI W , TIAN J ,et al. Attentionbased bidirectional long short-term memory networks for relation classification[C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2:Short Papers). Stroudsburg:Association for Computational Linguistics, 2016: 207-212.
[25]	LAN Z Z , CHEN M D , GOODMAN S ,et al. ALBERT:a lite BERT for self-supervised learning of language representations[J]. arXiv preprint,2019,arXiv:1909.11942.
[26]	SUN Y , WANG S H , LI Y K ,et al. ERNIE:enhanced representation through knowledge integration[J]. arXiv preprint,2019,arXiv:1904.09223.
[27]	CUI Y M , CHE W X , LIU T ,et al. Pretraining with whole word masking for Chinese BERT[J]. arXiv preprint,2019,arXiv:1906.08101.

Metrics

Recommended 0

No Suggested Reading articles found!

要素	先验知识	构造问句
不动产分割	不动产包括房屋、土地等	句子中是否包含不动产分割，不动产包括房屋、土地等。是或者否？
夫妻共同财产	夫妻共同财产包括夫妻关系存续期间的工资、生产投资收益等	句子中是否包含夫妻共同财产，包括夫妻关系存续期间的工资、生产投资收益等。是或者否？
特殊工种	特殊工种包括有害工种，如采煤工、锅炉工、下井挖掘等	句子中是否包含特殊工种，特殊工种包括有害工种，如采煤工、锅炉工、下井挖掘等。是或者否？

对比项	婚姻家庭		劳动争议		借款合同
对比项	训练集	测试集	训练集	测试集	训练集	测试集
文档/个	869	400	685	151	485	150
句子/个	25 235	11 685	26 048	5 682	17 567	5 124

参数名称	多标签分类方法	本文方法
学习率	2×10^-5(1×10^-4)	2×10^-5
优化器	AdamW	AdamW
权重衰减指数	0.01	0.01
最大文本长度	128	128
最大问句长度	—	32
迭代轮次/轮	9(30)	3
随机种子	42	42
批次大小	32	32

模型	婚姻家庭		劳动争议		借款合同
模型	F1	F1-search	F1	F1-search	F1	F1-search
TextRNN	58.4%	61.5%	48.6%	50.1%	54.6%	55.1%
Att-LSTM	69.8%	71.0%	59.2%	61.2%	61.2%	62.2%
TextCNN	71.9%	74.0%	59.0%	60.7%	64.6%	65.3%
ALBERT	69.2%	73.3%	60.0%	60.6%	59.5%	59.7%
BERT	76.4%	78.1%	61.5%	63.3%	61.8%	64.7%
ERNIE	76.2%	78.1%	61.8%	62.3%	60.4%	60.8%
RBT	75.9%	78.0%	61.4%	62.3%	62.7%	63.8%
MATCH（RBT）	78.0%	—	68.9%	—	69.8%	—
JRC（RBT）	79.1%	—	73.1%	—	70.2%	—

训练集	测试集	F1-micro	F1-macro
婚姻家庭	劳动争议	50.6%	30.0%
	借款合同	39.0%	26.8%
劳动争议	婚姻家庭	37.9%	32.3%
	借款合同	45.3%	28.4%
借款合同	婚姻家庭	25.2%	22.1%
	劳动争议	58.3%	34.3%

Legal element extraction method based on BERT reading comprehension framework

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 27

Related Articles 7

Metrics

Recommended 0

[1]	Xinhui LI, Qing SHEN, Xiongtao ZHANG. Classification algorithm for imbalance data of ECG based on PSOFS and TSK fuzzy system [J]. Big Data Research, 2022, 8(5): 139-152.
[2]	Shaolin AO, Yongbin QIN, Ruizhang HUANG, Yanping CHEN, Lijuan LIU, Qinghua ZHENG, Changheng CHEN, Shaofen CHENG. Research on auxiliary division method based on convolutional neural network [J]. Big Data Research, 2022, 8(2): 145-157.
[3]	Shenpeng ZHENG, Xiaojun CHEN, Yang XIANG, Ruchao SHEN. An entity relation extraction method based on subject mask [J]. Big Data Research, 2021, 7(3): 3-14.
[4]	Wenguang WANG. Knowledge graph reasoning: modern methods and applications [J]. Big Data Research, 2021, 7(3): 42-59.
[5]	Lihui WANG, Yongbin QIN. State of the art and future perspectives of the applications of deep learning in the medical image analysis [J]. Big Data Research, 2020, 6(6): 83-104.
[6]	Ting SONG, Zhanwei CHEN, Haifeng YANG. Aspect sentiment analysis based on a hierarchical attention network [J]. Big Data Research, 2020, 6(5): 82-91.
[7]	Yifeng WANG, Liru SUN, Liangle CUI, Yi ZHAO. Adaptive feature spectrum neural networks for special types of natural language classification [J]. Big Data Research, 2020, 6(4): 92-104.