基于参数回归的快速全景图像拼接算法

doi:10.11959/j.issn.1000-436x.2023182

摘要/Abstract

摘要：

现实场景中照相机获得的图像视场角范围往往是有限的，而目前对全景图像的需求日益增大，因此针对拍摄得到的全景图像序列，提出了一种基于参数回归的快速全景图像拼接算法。将传统的图像配准任务转化为深度学习结合机器学习的方式，设计一种基于高斯差分金字塔的多尺度深度卷积神经网络（MDCNN）对待拼接图像进行特征提取，并使用LightGBM回归模型对拼接参数进行预测，获得图像之间的变换矩阵和照相机焦距完成图像对齐，并设计了一种双曲线图像融合算法消除图像之间的拼接缝。实验结果表明，所提算法能够实现图像的快速拼接，获得比已有代表性算法更清晰自然的全景拼接效果，同时对红外图像也具有很好的适应性。

关键词: 图像拼接, 全景图像, 特征提取, 参数回归, 图像融合

Abstract:

In reality, the field of view of images acquired by cameras was usually limited, and the demand for panoramic images was increasing.Therefore, a fast panoramic image stitching algorithm based on parameter regression was proposed for panoramic image sequences.The traditional image registration task was transformed into deep learning combined with machine learning, a multi-scale deep convolutional neural network (MDCNN) based on Gaussian difference pyramid was designed to extract features of stitching images, and LightGBM regression model was used to predict stitching parameters.The transformation matrix and the focal length of the camera were obtained to align the images, and a hyperbolic image fusion algorithm was designed to eliminate the stitching seam between the images.The experimental results show that the proposed algorithm can quickly mosaic images and obtain clearer and more natural panoramic mosaic effects than the existing representative algorithms.It also has good adaptability for infrared images.

Key words: image stitching, panoramic image, feature extraction, parameter regression, image fusion

中图分类号:

TP391.41

郭璠, 李小虎, 刘文韬, 唐琎. 基于参数回归的快速全景图像拼接算法[J]. 通信学报, 2023, 44(9): 36-47.

Fan GUO, Xiaohu LI, Wentao LIU, Jin TANG. Fast panoramic image stitching algorithm based on parameter regression[J]. Journal on Communications, 2023, 44(9): 36-47.

图/表 18

图1

图2

图3

图4

图5

图6

图7

图8

表1

图9

表2

图10

图11

图12

表3

图13

表4

图14

参考文献 17

[1]	HARRIS C , STEPHENS M . A combined corner and edge detector[C]// Proceedings of the Alvey Vision Conference 1988.[S.l.]: Alvey Vision Club, 1988: 147-151.
[2]	SHU T . Multi-sensor remote sensing image alignment based on fast algorithms[J]. Journal of Intelligent Systems, 2023,32(1): 1-13.
[3]	BROWN M , LOWE D G . Automatic panoramic image stitching using invariant features[J]. International Journal of Computer Vision, 2007,74(1): 59-73.
[4]	ZARAGOZA J , CHIN T J , BROWN M S ,et al. As-projectiveas-possible image stitching with moving DLT[C]// Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2013: 2339-2346.
[5]	CHANG C H , SATO Y , CHUANG Y Y . Shape-preserving half-projective warps for image stitching[C]// Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2014: 3254-3261.
[6]	LIN C C , PANKANTI S U , RAMAMURTHY K N ,et al. Adaptive as-natural-as-possible image stitching[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2015: 1155-1163.
[7]	CHEN Y S , CHUANG Y Y . Natural image stitching with the global similarity prior[C]// European Conference on Computer Vision. Berlin:Springer, 2016: 186-201.
[8]	曾国奇, 牛子凡, 郑丽丽 ,等. 基于球形变换的无人机视频图像实时拼接方法[J]. 航空学报, 2023,44(23): 1-11.
	ZENG G Q , NIU Z F , ZHEN L L ,et al. A real time video image stitching method for UAV based on spherical transformation[J]. Acta Aeronautica et Astronautica Sinica, 2023,44(23): 1-11.
[9]	ZHU S D , ZHANG Y Z , ZHANG J ,et al. ISGTA:an effective approach for multi-image stitching based on gradual transformation matrix[J]. Signal,Image and Video Processing, 2023,17(7): 3811-3820.
[10]	KHAMIYEV I , ISSA D , AKHTAR Z ,et al. Panoramic image generation using deep neural networks[J]. Soft Computing, 2023,27(13): 8679-8695.
[18]	KE G , MENG Q , FINLEY T ,et al. LightGBM:a highly efficient gradient boosting decision tree[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. New York:ACM Press, 2017: 3149-3157.
[19]	FISCHLER M A , BOLLES R C . Random sample consensuss:a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981,24(6): 381-395.
[20]	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017,60(6): 84-90.
[21]	IANDOLA F N , HAN S , MOSKEWICZ M W ,et al. SqueezeNet:AlexNet-level accuracy with 50x fewer parameters and ＜0.5MB model size[J]. arXiv Preprint,arXiv:1602.07360, 2016.
[22]	IANDOLA F , MOSKEWICZ M , KARAYEV S ,et al. Densenet:Implementing efficient convnet descriptor pyramids[J]. arXiv Preprint,arXiv:1404.1869, 2014.
[23]	HE K M , ZHANG X Y , REN S Q ,et al. Deep residual learning for image recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016: 770-778.
[24]	MITTAL A , SOUNDARARAJAN R , BOVIK A C . Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2013,20(3): 209-212.
[25]	BLAU Y , MECHREZ R , TIMOFTE R ,et al. The 2018 PIRM challenge on perceptual image super-resolution[C]// European Conference on Computer Vision. Berlin:Springer, 2019: 334-355.
[11]	DAI Q Y , FANG F M , LI J C ,et al. Edge-guided composition network for image stitching[J]. Pattern Recognition, 2021,118:108019.
[12]	ZHANG H , ZHANG J M , PERAZZI F ,et al. Deep image compositing[C]// Proceedings of 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway:IEEE Press, 2021: 365-374.
[13]	LU C N , CHANG Y C , CHIU W C . Bridging the visual gap:wide-range image blending[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2021: 843-851.
[14]	PARIS S , KORNPROBST P , TUMBLIN J ,et al. Bilateral filtering:theory and applications[J]. Foundations and Trends in Computer Graphics and Vision, 2008,4(1): 1-75.
[15]	SHLENS J . A tutorial on principal component analysis[J]. arXiv Preprint,arXiv:1404.1100, 2014.
[16]	FRIEDMAN J H . Greedy function approximation:a gradient boosting machine[J]. The Annals of Statistics, 2001,29(5): 1189-1232.
[17]	CHEN T Q , GUESTRIN C . XGBoost:a scalable tree boosting system[C]// Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York:ACM Press, 2016: 785-794.

分类网络	Top1准确率	Top5准确率
AlexNet	56.522%	79.066%
SqueezeNet	58.092%	80.420%
DenseNet	74.434%	91.972%
VGG16	73.360%	91.516%
VGG19	74.218%	91.842%
ResNet	73.314%	91.420%
MDCNN（不引入DoGPM）	74.998%	91.807%
MDCNN（不引入衍生图）	75.318%	92.684%
MDCNN（引入DoGPM和衍生图）	75.683%	92.728%

算法	MAE	RMSE	R²
Linear	0.098	0.127	0.784
RandomForest	0.028	0.038	0.822
SVM	0.036	0.040	0.835
KNN	0.026	0.037	0.579
XGBoost	0.029	0.043	0.804
BP	0.173	0.200	0.753
LightGBM	0.022	0.035	0.864

算法	NIQE	PI	拼接成功率	运行时间/s
AutoStitch^[3]	4.411	4.358	90%	5.146
APAP^[4]	6.163	7.195	60%	71.632
AANAP^[6]	5.322	5.976	80%	46.249
SPHP^[5]	5.675	6.732	85%	62.751
NISWGSP^[7]	4.831	5.368	95%	12.843
本文算法	4.372	4.373	100%	4.386

算法	拼接成功率	运行时间/s
AutoStitch	90%	36.375
APAP	25%	183.276
AANAP	60%	123.164
SPHP	45%	146.527
NISWGSP	80%	80.892
本文算法	100%	8.184