基于视觉Transformer的多损失融合水下图像增强网络

doi:10.11959/j.issn.2096-6652.202252

摘要/Abstract

摘要：

由于水中存在光的吸收和散射现象，水下机器人拍摄到的图像存在颜色失真和对比度降低的问题。针对水下图像存在的质量退化现象，提出了一种基于视觉 Transformer 的多损失融合的方式训练水下图像增强网络。图像增强网络采用编码与解码的结构，可以采用端到端的方式进行训练。将多损失的线性组合作为总体优化目标，有效地更新水下图像增强网络的参数，包括像素损失、结构损失、边缘损失和特征损失。在两个大型水下数据集上进行了量化实验，并与7种水下图像增强算法进行对比。以峰值信噪比和结构相似性为有参考评估指标，以水下评估指标为无参考评估指标进行实验。实验结果表明，提出的水下图像增强网络能够有效地解决图像的颜色失真与对比度降低问题。

关键词: 水下图像, 质量增强, 视觉Transformer, 神经网络

Abstract:

Due to the absorption and scattering of light in water, the images captured by underwater robots suffer from color distortion and reduced contrast.Aiming at alleviating the quality degradation phenomenon of underwater images, an underwater image enhancement network based on vision Transformer that be trained with multiple losses fusion strategy was proposed.The image enhancement network adopted an encoder-decoder architecture, and could be trained in an end-to-end manner.In order to effectively update the parameters of the network for enhancing underwater images, a linear combination of various losses was adopted as the overall optimization objective, including pixel loss, structure loss, edge loss and feature loss.Quantitative experiments were carried out on two large underwater datasets, and the proposed underwater image enhancement network was compared with 7 underwater image enhancement algorithms.The full reference evaluation metrics peak signal-to-noise ratio and structural similarity were calculated in experiment, and the non-referenced metric underwater image quality measure was also computed.The experimental results showed that the proposed underwater image enhancement network could effectively deal with color distortion and contrast reduction.

Key words: underwater image, quality enhancement, vision Transformer, neural network

中图分类号:

TP391

丛晓峰,桂杰,章军. 基于视觉Transformer的多损失融合水下图像增强网络[J]. 智能科学与技术学报, 2022, 4(4): 522-532.

Xiaofeng CONG,Jie GUI,Jun ZHANG. Underwater image enhancement network based on visual Transformer with multiple loss functions fusion[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 522-532.

图/表 10

图1

图2

图3

表1

UIEB数据集上的定量实验结果"

指标	UDCP	IBLA	ULAP	GB	UResNet	UWNet	WaterNet	UWT-Net
PSNR	12.368	17.299	17.601	15.078	18.684	18.913	$22 . 694$	21.163
SSIM	0.526	0.678	0.692	0.608	0.754	0.784	0.854	$0 . 870$
UIQM	1.842	1.783	1.878	2.272	2.772	2.746	2.883	$3 . 084$

表1

图4

表2

EUVP-US数据集上的定量实验结果"

指标	UDCP	IBLA	ULAP	GB	UResNet	UWNet	WaterNet	UWT-Net
PSNR	14.994	22.138	21.897	16.553	27.638	27.727	23.722	$28 . 343$
SSIM	0.570	0.721	0.761	0.606	0.839	0.832	0.818	$0 . 871$
UIQM	2.114	2.214	2.369	2.583	2.867	2.888	$2 . 889$	2.806

表2

图5

表3

L1损失消融实验结果"

指标	设置1	设置2	设置3	设置4
PSNR	$21 . 163$	19.244	20.577	19.973
SSIM	$0 . 870$	0.830	0.853	0.851
UIQM	$3 . 084$	2.900	3.041	3.053

表3

图6

图7

参考文献 33

[1]	严浙平, 曲思瑜, 邢文 . 水下图像增强方法研究综述[J]. 智能系统学报, 2022,17(5): 860-873.
	YAN Z P , QU S Y , XING W . An overview of underwater image enhancement methods[J]. CAAI Transactions on Intelligent Systems, 2022,17(5): 860-873.
[2]	DREWS JR P , DO NASCIMENTO E , MORAES F ,et al. Transmission estimation in underwater single images[C]// Proceedings of 2013 IEEE International Conference on Computer Vision Workshops. Piscataway:IEEE Press, 2013: 825-830.
[3]	DREWS JR P , NASCIMENTO E R , BOTELHO S S C ,et al. Underwater depth estimation and image restoration based on single images[J]. IEEE Computer Graphics and Applications, 2016,36(2): 24-35.
[4]	PENG Y T , COSMAN P C . Underwater image restoration based on image blurriness and light absorption[J]. IEEE Transactions on Image Processing, 2017,26(4): 1579-1594.
[5]	SONG W , WANG Y , HUANG D M ,et al. A rapid scene depth estimation model based on underwater light attenuation prior for underwater image restoration[C]// Proceedings of 2018 Advances in Multimedia Information Processing. Cham:Springer, 2018: 678-688.
[6]	LIU P , WANG G Y , QI H ,et al. Underwater image enhancement with a deep residual framework[J]. IEEE Access, 2019,7: 94614-94629.
[7]	NAIK A , SWARNAKAR A , MITTAL K . Shallow-UWnet:compressed model for underwater image enhancement (student abstract)[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021,35(18): 15853-15854.
[8]	LI C Y . Underwater scene prior inspired deep underwater image and video enhancement[J]. Pattern Recognition, 2020,98.
[9]	BERMAN D , LEVY D , AVIDAN S ,et al. Underwater single image color restoration using haze-lines and a new quantitative dataset[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021,43(8): 2822-2837.
[10]	LIU Z , LIN Y T , CAO Y ,et al. Swin Transformer:hierarchical vision transformer using shifted windows[C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE Press, 2021: 9992-10002.
[11]	ZAMIR S W , ARORA A , KHAN S ,et al. Restormer:efficient transformer for high-resolution image restoration[C]// Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2022: 5718-5729.
[12]	DOSOVITSKIY A , BEYER L , KOLESNIKOV A ,et al. An image is worth 16×16 words:transformers for image recognition at scale[J]. arXiv preprint, 2020,arXiv:2010.11929.
[13]	LI J , SKINNER K A , EUSTICE R M ,et al. WaterGAN:unsupervised generative network to enable real-time color correction of monocular underwater images[J]. IEEE Robotics and Automation Letters, 2018,3(1): 387-394.
[14]	常戬, 韩旭 . 结合导向滤波与自适应算子的水下图像增强[J]. 计算机工程与应用, 2021,已录用.
	CHANG J , HAN X . Underwater image enhancement combining guide filtering with adaptive operator[J]. Computer Engineering and Applications, 2021,Accepted.
[15]	李颖, 陈龙, 黄钊宏 ,等. 基于多尺度卷积神经网络特征融合的植株叶片检测技术[J]. 智能科学与技术学报, 2021,3(3): 304-311.
	LI Y , CHEN L , HUANG Z H ,et al. Plant leaf detection technology based on multi-scale CNN feature fusion[J]. Chinese Journal of Intelligent Science and Technology, 2021,3(3): 304-311.
[16]	王禾扬, 杨启鸣, 朱旗 . 基于深度卷积集成网络的视网膜多种疾病筛查和识别方法[J]. 智能科学与技术学报, 2021,3(3): 259-267.
	WANG H Y , YANG Q M , ZHU Q . Retinal multi-disease screening and recognition method based on deep convolution ensemble network[J]. Chinese Journal of Intelligent Science and Technology, 2021,3(3): 259-267.
[17]	邵虹, 张鸣坤, 崔文成 . 基于分层卷积神经网络的皮肤镜图像分类方法[J]. 智能科学与技术学报, 2021,3(4): 474-481.
	SHAO H , ZHANG M K , CUI W C . Classification method of dermoscopic image based on hierarchical convolution neural network[J]. Chinese Journal of Intelligent Science and Technology, 2021,3(4): 474-481.
[18]	陈龙, 丁丹丹 . 多残差联合学习的水下图像增强[J]. 中国图象图形学报, 2022,27(5): 1577-1588.
	CHEN L , DING D D . Joint multi-residual learning for underwater image enhancement[J]. Journal of Image and Graphics, 2022,27(5): 1577-1588.
[19]	范新南, 杨鑫, 史鹏飞 ,等. 特征融合生成对抗网络的水下图像增强[J]. 计算机辅助设计与图形学学报, 2022,34(2): 264-272.
	FAN X N , YANG X , SHI P F ,et al. Underwater image enhancement based on feature fusion generative adversaral networks[J]. Journal of Computer-Aided Design ＆ Computer Graphics, 2022,34(2): 264-272.
[20]	LI C Y , GUO C L , REN W Q ,et al. An underwater image enhancement benchmark dataset and beyond[J]. IEEE Transactions on Image Processing, 2020,29: 4376-4389.
[21]	GUO Y C , LI H Y , ZHUANG P X . Underwater image enhancement using a multiscale dense generative adversarial network[J]. IEEE Journal of Oceanic Engineering, 2020,45(3): 862-870.
[22]	FABBRI C , ISLAM M J , SATTAR J . Enhancing underwater imagery using generative adversarial networks[C]// Proceedings of 2018 IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 2018: 7159-7165.
[23]	BOUDIAF A , GUO Y H , GHIMIRE A ,et al. Underwater image enhancement using pre-trained transformer[C]// Proceedings of International Conference on Image Analysis and Processing.[S.l.:s.n.], 2022: 480-488.
[24]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[J]. arXiv preprint, 2017,arXiv:1706.03762.
[25]	SONG Y D , HE Z Q , QIAN H ,et al. Vision transformers for single image dehazing[J]. arXiv preprint, 2022,arXiv:2204.03883.
[26]	ZHAO H , GALLO O , FROSIO I ,et al. Loss functions for image restoration with neural networks[J]. IEEE Transactions on Computational Imaging, 2017,3(1): 47-57.
[27]	ZHANG H , PATEL V M . Densely connected pyramid dehazing network[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 3194-3203.
[28]	JOHNSON J , ALAHI A , LI F F . Perceptual losses for real-time style transfer and super-resolution[C]// Proceedings of 2016 European Conference on Computer Vision.[S.l.:s.n.], 2016: 694-711.
[29]	SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint,2014 2014,arXiv:1409.
[30]	ISLAM M J , XIA Y Y , SATTAR J . Fast underwater image enhancement for improved visual perception[J]. IEEE Robotics and Automation Letters, 2020,5(2): 3227-3234.
[31]	LI C Y , QUO J , PANG Y W ,et al. Single underwater image restoration by blue-green channels dehazing and red channel correction[C]// Proceedings of 2016 IEEE International Conference on Acoustics,Speech and Signal Processing. Piscataway:IEEE Press, 2016: 1731-1735.
[32]	WANG Y , SONG W , FORTINO G ,et al. An experimental-based review of image enhancement and image restoration methods for underwater imaging[J]. IEEE Access, 7: 140233-140251.
[33]	PANETTA K , GAO C , AGAIAN S . Human-visual-system-inspired underwater image quality measures[J]. IEEE Journal of Oceanic Engineering, 2016,41(3): 541-551.