基于3D卷积的图像序列特征提取与自注意力的车牌识别方法

doi:10.11959/j.issn.2096-6652.202128

Abstract

Abstract:

In recent years, neural networks based on self-attentive mechanism have been widely used in computer vision tasks.As the intelligent transportation system is widely used, the task difficulty of license plate recognition is increasing and the need for correct recognition is getting more pressing in the face of complex and changing traffic scenes.Therefore, a rectification-free license plate recognition method T-LPR based on self-attention was proposed.Firstly, the images were sliced and sequenced, and 3D convolution was used for feature extraction of the sliced sequences to obtain a sequence of image embedding vectors.Secondly, the sequence of embedding vectors was fed into an encoder based on Transformer Encoder, which learned the relationship between the individual embedding vectors and outputs the final encoding result.Finally, the final encoding result was classified by a classifier.Experimental results on several public datasets show that T-LPR proposed is very effective for recognizing license plates in all kinds of difficult scenarios.

Key words: license plate recognition, image embedding vector, self-attention, rectification-free

CLC Number:

TP183，TP39

Ganxiong ZENG,Xiao KE. 3D convolution-based image sequence feature extraction and self-attention for license plate recognition method[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(3): 268-279.

Figures/Tables 8

方法	All	Base	DB	FN	Rotate	Tilt	Weather	Challenge
Faster RCNN^[43]+HC	92.8%	97.2%	94.4%	90.9%	82.9%	87.3%	85.5%	76.3%
YOLO9000^[14]+HC	93.7%	98.1%	96%	88.2%	84.5%	88.5%	87.0%	80.5%
SSD300^[45]+HC	95.2%	98.3%	96.6%	95.9%	88.4%	91.5%	87.3%	83.8%
TE2E^[19]	94.4%	97.8%	94.8%	94.5%	87.9%	92.1%	86.8%	81.2%
RPnet^[7]	95.5%	98.5%	96.9%	94.3%	90.8%	92.5%	87.9%	85.1%
DAN^[21]	96.6%	98.9%	96.1%	96.4%	91.9%	93.7%	95.4%	83.1%
MORAN^[20]	98.3%	99.5%	98.1%	98.6%	98.1%	98.6%	97.6%	86.5%
MTLPR^[18]	98.8%	-	-	-	-	-	-	-
LPRNet^[12]	93.0%	97.8%	92.2%	91.9%	79.4%	85.8%	92.0%	69.8%
ANet (real data only)^[11]	98.5%	99.6%	98.8%	98.8%	96.4%	97.6%	98.5%	88.9%
ANet (real+synthetic data)^[11]	98.9%	$99 . 8 %$	$99 . 2 %$	99.1%	98.1%	$98 . 8 %$	98.6%	89.7%
T-LPR	$99 . 0 %$	99.6%	99.1%	$99 . 2 %$	$98 . 1 %$	98.6%	$98 . 9 %$	$93 . 1 %$

方法	PKUData：ACC	PKUData：ACC w/o	CLPD：ACC	CLPD：ACC w/o
Sighthound (2017)^[46]	-	89.3%	-	85.2%
TE2E^[7]	77.6%	78.4%	66.5%	78.9%
ANet (real data only)^[11]	84.8%	86.5%	70.8%	86.1%
ANet (real+synthetic data)^[11]	88.2%	90.5%	76.8%	87.6%
T-LPR	$94 . 7 %$	$97 . 5 %$	$81 . 4 %$	$92 . 1 %$

方法	ACC	ACC w/o
未预训练的T-LPR	94.7%	97.5%
预训练后的T-LPR	$97 . 9 %$	$98 . 9 %$

数据集	ViT^[24]	T-LPR
Base	99.4%	$99 . 6 %$
DB	99.0%	$99 . 1 %$
FN	$99 . 2 %$	$99 . 2 %$
Rotate	96.8%	$98 . 1 %$
Tilt	97.8%	$98 . 6 %$
Weather	$98 . 9 %$	$98 . 9 %$
Challenge	92.2%	$93 . 1 %$
PKUData	88.4%	$94 . 7 %$
CLPD	61.3%	$81 . 4 %$

References 46

[32]	GAO F , CAI Y C , GE Y S ,et al. EDF-LPR:a new encoder-decoder framework for license plate recognition[J]. IET Intelligent Transport Systems, 2020,14(8): 959-969.
[33]	SCHUSTER M , PALIWAL K K . Bidirectional recurrent neural networks[J]. IEEE Transactions on Signal Processing, 1997,45(11): 2673-2681.
[34]	GRAVES A , FERNáNDEZ S , GOMEZ F ,et al. Connectionist temporal classification:labelling unsegmented sequence data with recurrent neural networks[C]// Proceedings of the 23rd International Conference on Machine Learning. New York:ACM Press, 2006: 369-376.
[35]	CHOLLET F , . Xception:deep learning with depthwise separable convolutions[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017: 1800-1807.
[36]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[37]	SHI B G , BAI X , YAO C . An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(11): 2298-2304.
[38]	ZHANG Y P , NIE S , LIU W J ,et al. Sequence-to-sequence domain adaptation network for robust text image recognition[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019: 2735-2744.
[39]	JADERBERG M , SIMONYAN K , ZISSERMAN A ,et al. Spatial transformer networks[J]. arXiv preprint,2016,arXiv:1506.02025.
[40]	YANG M K , GUAN Y S , LIAO M H ,et al. Symmetry-constrained rectification network for scene text recognition[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE Press, 2019: 9146-9155.
[41]	LU N , YU W W , QI X B ,et al. MASTER:multi-aspect non-local network for scene text recognition[J]. Pattern Recognition, 2021,117: 107980.
[42]	BA J L , KIROS J R , HINTON G E . Layer normalization[J]. arXiv preprint,2016,arXiv:1607.06450.
[43]	REN S Q , HE K M , GIRSHICK R ,et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. arXiv preprint,2015,arXiv:1506.01497.
[1]	刘晓波, 蒋阳升, 唐优华 ,等. 综合交通大数据应用技术的发展展望[J]. 大数据, 2019,5(3): 55-68.
	LIU X B , JIANG Y S , TANG Y H ,et al. Development prospect of in-tegrated transportation big data application technology[J]. Big Data Research, 2019,5(3): 55-68.
[44]	?PA?HEL J , SOCHOR J , JURáNEK R ,et al. Holistic recognition of low quality license plates by CNN using track annotated data[C]// Proceedings of 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). Piscataway:IEEE Press, 2017: 1-6.
[45]	LIU W , ANGUELOV D , ERHAN D ,et al. SSD:single shot multibox detector[C]// Proceedings of the European conference on Computer Vision. Heidelberg:Springer, 2016: 21-37.
[2]	KHAN M A , SHARIF M , JAVED M Y ,et al. License number plate recognition system using entropy-based features selection approach with SVM[J]. IET Image Processing, 2018,12(2): 200-209.
[3]	薛磊, 杨晓敏, 吴炜 ,等. 一种基于KNN与改进SVM的车牌字符识别算法[J]. 四川大学学报(自然科学版), 2006,43(5): 1031-1036.
	XUE L , YANG X M , WU W ,et al. An algorithm based on KNN and improved SVM for licence plate recognition[J]. Journal of Sichuan University (Natural Science Edition), 2006,43(5): 1031-1036.
[4]	张钹 . 人工智能进入后深度学习时代[J]. 智能科学与技术学报, 2019,1(1): 4-6.
	ZHANG B . Artificial intelligence is entering the post deep-learning era[J]. Chinese Journal of Intelligent Science and Technology, 2019,1(1): 4-6.
[5]	LI H , WANG P , YOU M Y ,et al. Reading car license plates using deep neural networks[J]. Image and Vision Computing, 2018,72: 14-23.
[6]	JAIN V , SASINDRAN Z , RAJAGOPAL A ,et al. Deep automatic license plate recognition system[C]// Proceedings of the 10th Indian Conference on Computer Vision,Graphics and Image Processing. New York:ACM Press, 2016: 1-8.
[7]	XU Z B , YANG W , MENG A J ,et al. Towards end-to-end license plate detection and recognition:a large dataset and baseline[C]// Proceedings of the European Conference on Computer Vision. Heidelberg:Springer, 2018: 261-277.
[8]	LAROCA R , SEVERO E , ZANLORENSI L A ,et al. A robust real-time automatic license plate recognition based on the YOLO detector[C]// Proceedings of 2018 International Joint Conference on Neural Networks (IJCNN). Piscataway:IEEE Press, 2018: 1-10.
[9]	ZHUANG J F , HOU S H , WANG Z L ,et al. Towards human-level license plate recognition[C]// Proceedings of the European Conference on Computer Vision. Heidelberg:Springer, 2018: 314-329.
[10]	SILVA S M , JUNG C R . License plate detection and recognition in unconstrained scenarios[C]// Proceedings of the European Conference on Computer Vision. Heidelberg:Springer, 2018: 593-609.
[11]	ZHANG L J , WANG P , LI H ,et al. A robust attentional framework for license plate recognition in the wild[J]. IEEE Transactions on Intelligent Transportation Systems (Early Access), 2020: 1-10.
[12]	ZHERZDEV S , GRUZDEV A . LPRNet:license plate recognition via deep neural networks[J]. arXiv preprint,2018,arXiv:1806,10447.
[13]	CHEN L C , PAPANDREOU G , KOKKINOS I ,et al. DeepLab:semantic image segmentation with deep convolutional nets,atrous convolution,and fully connected CRFs[J]. arXiv preprint,2016,arXiv:1606.00915.
[14]	HE K M , GKIOXARI G , DOLLáR P ,et al. Mask R-CNN[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2980-2988.
[15]	REDMON J , FARHADI A . YOLOv3:an incremental improvement[J]. arXiv preprint,2018,arXiv:1804.02767.
[16]	REDMON J , FARHADI A . YOLO9000:better,faster,stronger[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2017: 6517-6525.
[17]	ZHANG Z G , WANG C . The research of vehicle plate recognition technical based on BP neural network[J]. AASRI Procedia, 2012,1: 74-81.
[18]	TAO X Z , LI L , LU L . A lightweight convolutional neural network for license plate character recognition[C]// Proceedings of 2019 International Symposium on Intelligence Computation and Applications. Heidelberg:Springer, 2019: 379-387.
[19]	LI H , WANG P , SHEN C H . Toward end-to-end car license plate detection and recognition with deep neural networks[J]. IEEE Transactions on Intelligent Transportation Systems, 2019,20(3): 1126-1136.
[20]	LUO C J , JIN L W , SUN Z H . MORAN:a multi-object rectified attention network for scene text recognition[J]. Pattern Recognition, 2019,90: 109-118.
[21]	WANG T W , ZHU Y Z , JIN L W ,et al. Decoupled attention network for text recognition[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2020,34(7): 12216-12224.
[22]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[J]. arXiv preprint,2017,arXiv:1706.03762.
[23]	DEVLIN J , CHANG M W , LEE K ,et al. BERT:pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint,2019,arXiv:1810.04805.
[24]	DOSOVITSKIY A , BEYER L , KOLESNIKOV A ,et al. An image is worth 16×16 words:transformers for image recognition at scale[J]. arXiv preprint,2020,arXiv:2010.11929.
[25]	CARION N , MASSA F , SYNNAEVE G ,et al. End-to-end object detection with transformers[C]// Proceedings of the European conference on Computer Vision. Heidelberg:Springer, 2020: 213-229.
[26]	ZHENG S X , LU J C , ZHAO H S ,et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[J]. arXiv preprint,2020,arXiv:2012.15840.
[27]	LI X Y , ZHANG Y Y , LIU C H ,et al. VidTr:video transformer without convolutions[J]. arXiv preprint,2021,arXiv:2104.11746.
[28]	YUAN Y L , ZOU W B , ZHAO Y ,et al. A robust and efficient approach to license plate detection[J]. IEEE Transactions on Image Processing, 2017,26(3): 1102-1114.
[29]	LI P Y , NGUYEN M , YAN W Q . Rotation correction for license plate recognition[C]// Proceedings of 2018 4th International Conference on Control,Automation and Robotics. Piscataway:IEEE Press, 2018: 400-404.
[30]	GOU C , WANG K F , YAO Y J ,et al. Vehicle license plate recognition based on extremal regions and restricted boltzmann machines[J]. IEEE Transactions on Intelligent Transportation Systems, 2016,17(4): 1096-1107.
[31]	BJ?RKLUND T , FIANDROTTI A , ANNARUMMA M ,et al. Robust license plate recognition using neural networks trained on synthetic images[J]. Pattern Recognition, 2019,93: 134-146.
[46]	MASOOD S Z , SHU G , DEHGHAN A ,et al. License plate detection and recognition using deeply learned convolutional neural networks[J]. arXiv preprint,2017,arXiv:1703.07330.

Metrics

Recommended 0

No Suggested Reading articles found!

数据集	基本描述	数量/张（总数/训练/测试）
CCPD	大规模的开放场景车牌数据集	280 000/100 000/180 000
Base	简单的正面车牌图像	200 000/100 000/100 000
DB	过曝或暗光环境下的车牌图像	20 000/0/20 000
FN	远距离或很近距离拍摄的车牌图像	20 000/0/20 000
Rotate	旋转或倾斜的车牌图像	10 000/0/10 000
Tilt	旋转或倾斜的车牌图像	10 000/0/10 000
Weather	雨、雪、雾等天气条件下的车牌图像	10 000/0/10 000
Challenge	困难车牌图像	10 000/0/10 000
PKUData	包括较完整省份的正常车牌图像	4 210/2 526/1 684
CLPD	包括完整省份和多种拍摄环境的车牌图像	1 195/0/1 195

3D convolution-based image sequence feature extraction and self-attention for license plate recognition method

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 46

Related Articles 7

Metrics

Recommended 0

[1]	Zhe HUANG, Yongcai WANG, Deying LI. A survey of 3D object detection algorithms [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(1): 7-31.
[2]	Yan CHEN, Xueqin LUO, Wei LIANG, Yongfang XIE. Depression recognition based on emotional information fused with attentional mechanism [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 600-609.
[3]	Qiang ZHANG, Wen WEN, Xiaodong ZHOU, Weihui LIU, Xiaoyu CHU. Research on the manipulator intelligent trajectory planning method based on the improved TD3 algorithm [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 223-232.
[4]	Yi HONG, Chengli SUN, Yan LENG. End-to-end speech enhancement based on ultra-lightweight channel attention [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(3): 351-358.
[5]	Li LI, Weiliang ZENG, Yonghui HUANG, Weijun SUN. Research on anti-spoofing method of face recognition based on semi-supervised learning [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(3): 370-380.
[6]	Guobin ZHANG,Xinying WANG. Research on data-driven modeling for photovoltaic characteristics based on hybrid neural network [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(2): 169-178.
[7]	Yating WEI,Zhiyong WANG,Shuyue ZHOU,Wei CHEN. Federated visualization:a new model for privacy-preserving visualization [J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(4): 415-420.