基于双分支网络的图像修复取证方法

doi:10.11959/j.issn.2096-109x.2022084

Abstract

Abstract:

Image inpainting is a technique that uses information from known areas of an image to repair missing or damaged areas of the image.Image editing software based on it has made it easy to edit and modify the content of digital images without any specialized foundation.When image inpainting techniques are used to maliciously remove the content of an image, it will cause confidence crisis on the real image.Current researches in image inpainting forensics can only effectively detect a certain type of image inpainting.To address this problem, a passive forensic method for image inpainting was proposed, which is based on a two-branch network.The high-pass filtered convolutional network in the dual branch first used a set of high-pass filters to attenuate the low-frequency components in the image.Then features were extracted using four residual blocks, and two transposed convolutions were performed with 4x up-sampling to zoom in on the feature map.And thereafter a 5×5 convolution was used to attenuate the tessellation artifacts from the transposed convolutions to generate a discriminative feature map on the high-frequency components of the image.The dual-attention feature fusion branch in the dual branch first added a local binary pattern feature map to the image using a preprocessing block.Then the dual-attention convolution block was used to adaptively integrate the image’s local features and global dependencies to capture the differences in content and texture between the inpainted and pristine regions of the image.Additionally, the features extracted from the dual-attention convolution block were fused, and the feature maps were up-sampled identically to generate the discriminative image content and texture on the feature maps.The extensive experimental results show the proposed method improved the F1 score by 2.05% and the Intersection over Union(IoU) by 3.53% for the exemplar-based method and by 1.06% and 1.22% for the deep-learning-based method in detecting the inpainted region of the removed object.Visualization of the results shows that the edges of the removed objects can be accurately located on the detected inpainted area.

Key words: image forensics, image forgery detection, deep learning, attention mechanism

CLC Number:

TP393

Dengyong ZHANG, Huang WEN, Feng LI, Peng CAO, Lingyun XIANG, Gaobo YANG, Xiangling DING. Image inpainting forensics method based on dual branch network[J]. Chinese Journal of Network and Information Security, 2022, 8(6): 110-122.

Figures/Tables 12

References 37

[1]	SRIDEVI G , SRINIVAS KUMAR S . Image inpainting based on fractional-order nonlinear diffusion for image reconstruction[J]. Circuits,Systems,and Signal Processing, 2019,38(8): 3802-3817.
[2]	DOU L , QIAN Z , QIN C ,et al. Anti-forensics of diffusion-based image inpainting[J]. Journal of Electronic Imaging, 2020,29(4): 043026.
[3]	ZHANG Y , DING F , KWONG S ,et al. Feature pyramid network for diffusion-based image inpainting detection[J]. Information Sciences, 2021,572: 29-42.
[4]	CRIMINISI A , PéREZ P , TOYAMA K . Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on image processing, 2004,13(9): 1200-1212.
[5]	QUY TRAN B , VAN NGUYEN T , DUY TRAN D ,et al. Accelerating exemplar-based image inpainting with GPU and CUDA[C]// 2021 10th International Conference on Software and Computer Applications. 2021: 173-179.
[6]	XU T , HUANG T Z , DENG L J ,et al. Exemplar-based image inpainting using adaptive two-stage structure-tensor based priority function and nonlocal filtering[J]. Journal of Visual Communication and Image Representation, 2022,83:103430.
[7]	IIZUKA S , SIMO-SERRA E , ISHIKAWA H . Globally and locally consistent image completion[J]. ACM Transactions on Graphics (ToG), 2017,36(4): 1-14.
[8]	ZENG Y , FU J , CHAO H ,et al. Learning pyramid-context encoder network for high-quality image inpainting[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1486-1494.
[9]	SUVOROV R , LOGACHEVA E , MASHIKHIN A ,et al. Resolution-robust large mask inpainting with fourier convolutions[C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022: 2149-2159.
[10]	WU Q , SUN S-J , ZHU W ,et al. Detection of digital doctoring in exemplar-based inpainted images[C]// 2008 International Conference on Machine Learning and Cybernetics. 2008,3: 1222-1226.
[11]	BACCHUWAR K S , RAMAKRISHNAN K . A jump patch-block match algorithm for multiple forgery detection[C]// 2013 International Mutli-Conference on Automation,Computing,Communication,Control and Compressed Sensing (iMac4s). 2013: 723-728.
[12]	CHANG I C , YU J C , CHANG C C . A forgery detection algorithm for exemplar-based inpainting images using multi-region relation[J]. Image and Vision Computing, 2013,31(1): 57-71.
[13]	LIANG Z , YANG G , DING X ,et al. An efficient forgery detection algorithm for object removal by exemplar-based image inpainting[J]. Journal of Visual Communication and Image Representation, 2015,30: 75-85.
[14]	ZHAO Y Q , LIAO M , SHIH F Y ,et al. Tampered region detection of inpainting JPEG images[J]. Optik, 2013,124(16): 2487-2492.
[15]	LIU Q , SUNG A H , ZHOU B ,et al. Exposing inpainting forgery in jpeg images under recompression attacks[C]// 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA). 2016: 164-169.
[16]	ZHANG D , LIANG Z , YANG G ,et al. A robust forgery detection algorithm for object removal by exemplar-based image inpainting[J]. Multimedia Tools and Applications, 2018,77(10): 11823-11842.
[17]	ZHU X , QIAN Y , ZHAO X ,et al. A deep learning approach to patch-based image inpainting forensics[J]. Signal Processing:Image Communication, 2018,67: 90-99.
[18]	WANG X , WANG H , NIU S . An intelligent forensics approach for detecting patch-based image inpainting[J]. Mathematical Problems in Engineering, 2020,2020(8): 1-10.
[19]	WANG X , NIU S , WANG H . Image inpainting detection based on multi-task deep learning network[J]. IETE Technical Review, 2021,38(1): 149-157.
[20]	LU M , NIU S . A detection approach using LSTM-CNN for object removal caused by exemplar-based image inpainting[J]. Electronics, 2020,9(5): 858.
[21]	BARNI M , COSTANZO A , NOWROOZI E ,et al. CNN-based detection of generic contrast adjustment with JPEG post-processing[C]// 2018 25th IEEE International Conference on Image Processing (ICIP). 2018: 3803-3807.
[22]	SHAN W , YI Y , HUANG R ,et al. Robust contrast enhancement forensics based on convolutional neural networks[J]. Signal Processing:Image Communication, 2019,71: 138-146.
[23]	ZHOU P , HAN X , MORARIU V I ,et al. Learning rich features for image manipulation detection[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 1053-1061.
[24]	BARNI M , BONDI L , BONETTINI N ,et al. Aligned and non-aligned double JPEG detection using convolutional neural networks[J]. Journal of Visual Communication and Image Representation, 2017,49: 153-163.
[25]	BAYAR B , STAMM M C . Constrained convolutional neural networks:a new approach towards general purpose image manipulation detection[J]. IEEE Transactions on Information Forensics and Security, 2018,13(11): 2691-2706.
[26]	BUNK J , BAPPY J H , MOHAMMED T M ,et al. Detection and localization of image forgeries using resampling features and deep learning[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2017: 1881-1889.
[27]	BAPPY J H , SIMONS C , NATARAJ L ,et al. Hybrid lstm and encoder–decoder architecture for detection of image forgeries[J]. IEEE Transactions on Image Processing, 2019,28(7): 3286-3300.
[28]	WU Y , ABDALMAGEED W , NATARAJAN P . Mantra-net:Manipulation tracing network for detection and localization of image forgeries with anomalous features[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 9543-9552.
[29]	BARNI M , PHAN Q T , TONDI B . Copy move source-target disambiguation through multi-branch CNNs[J]. IEEE Transactions on Information Forensics and Security, 2020,16: 1825-1840.
[30]	LI H , HUANG J . Localization of deep inpainting using high-pass fully convolutional network[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 8301-8310.
[31]	WANG X , WANG H , NIU S . An image forensic method for AI inpainting using faster R-CNN[C]// International Conference on Artificial Intelligence and Security. 2019,41: 476-487.
[32]	LI A , KE Q , MA X ,et al. Noise doesn't lie:towards universal detection of deep inpainting[C]// Proceedings of International Joint Conference on Artificial Intelligence. 2021: 786-792.
[33]	WU H , ZHOU J . IID-Net:image inpainting detection network via neural architecture search and attention[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021,32(3): 1172-1185.
[34]	SZEGEDY C , LIU W , JIA Y ,et al. Going deeper with convolutions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 1-9.
[35]	ZHANG D , CHEN X , LI F ,et al. Seam-carved image tampering detection based on the cooccurrence of adjacent lbps[J]. Security and Communication Networks, 2020,2020: 1-12.
[36]	FU J , LIU J , TIAN H ,et al. Dual attention network for scene segmentation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 3146-3154.
[37]	EVERINGHAM M , ESLAMI S , VAN GOOL L ,et al. The pascal visual object classes challenge:a retrospective[J]. International Journal of Computer Vision, 2015,111(1): 98-136.

Metrics

Recommended 0

No Suggested Reading articles found!

残差块	瓶颈单元	瓶颈深度	输出深度	步长
#1	#1	32	128	1
	#2	32	128	2
#2	#1	64	256	1
	#2	64	256	2
#3	#1	128	512	1
	#2	128	512	2
#4	#1	256	1 024	1
	#2	256	1 024	2

块	类型	配置
	卷积#1	核个数：64，大小：3×3，步长：1
卷积块#1	卷积#2	核个数：64，大小：3×3，步长：1
	最大池化	窗口大小：2×2，步长：2
	卷积#1	核个数：128，大小：3×3，步长：1
卷积块#2	卷积#2	核个数：128，大小：3×3，步长：1
	最大池化	窗口大小：2×2，步长：2
	卷积#1	核个数：256，大小：3×3，步长：1
	卷积#2	核个数：256，大小：3×3，步长：1
双注意力卷积块#1	卷积#3	核个数：256，大小：3×3，步长：1
	最大池化	窗口大小：2×2，步长：2
	双重注意力	—
	卷积#1	核个数：512，大小：3×3，步长：1，空洞卷积率：2
双注意力卷积块#2	卷积#2	核个数：512，大小：3×3，步长：1，空洞卷积率：2
	卷积#3	核个数：512，大小：3×3，步长：1，空洞卷积率：2
	双重注意力	—
	卷积#1	核个数：512，大小：3×3，步长：1
双注意力卷积块#3	卷积#2	核个数：512，大小：3×3，步长：1
	卷积#3	核个数：512，大小：3×3，步长：1
	双重注意力	—

数据集	样本块图像修复数据集	深度学习图像修复数据集
训练集	19 388	19 388
验证集	2 154	2 154
随机缺失测试集	8 915	8 915
对象移除测试集	275	275

损失类型	随机缺失的样本块图像修复				随机缺失的深度学习图像修复
损失类型	召回率	精确率	交并比	F1分数值	召回率	精确率	交并比	F1分数值
焦点损失	90.32%	91.07%	84.17%	89.98%	94.08%	96.62%	91.30%	95.29%
加权交叉熵损失	93.76%	88.82%	84.92%	90.46%	97.18%	93.99%	91.70%	95.53%
标准交叉熵损失	93.10%	91.79%	86.92%	91.72%	96.29%	97.38%	94.02%	96.78%

网络	随机缺失的样本块图像修复的F1分数值	随机缺失深度学习图像修复的F1分数值
1-Base	89.568 8%	95.641 2%
2-Base	84.901 9%	95.684 4%
2-Base+DA	81.951 3%	95.937 8%
2-Base+DA+FF	86.352 6%	96.535 8%
本文提出的网络	91.724 6%	96.780 4%

Image inpainting forensics method based on dual branch network

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 37

Related Articles 15

Metrics

Recommended 0

	样本块图像修复				深度学习图像修复
方法	随机缺失测试集		对象移除测试集		随机缺失测试集		对象移除测试集
	F1分数值	交并比	F1分数值	交并比	F1分数值	交并比	F1分数值	交并比
文献[17]	89.69%	82.45%	87.53%	79.35%	91.60%	85.13%	90.64%	84.04%
文献[30]	89.56%	83.56%	91.29%	85.25%	95.64%	91.89%	95.57%	91.74%
文献[33]	62.37%	51.94%	31.36%	22.36%	96.58%	93.64%	95.11%	91.58%
本文	91.72%	86.91%	93.34%	88.78%	96.78%	94.02%	96.17%	92.80%

	样本块图像修复				深度学习图像修复
文献	JPEG压缩		高斯模糊		JPEG压缩		高斯模糊
	QF95	QF75	3×3高斯核	5×5高斯核	QF95	QF75	3×3高斯核	5×5高斯核
文献[17]	0.622 198	0.504 967	0.857 684	0.812 820	0.880 446	0.811 035	0.785 285	0.728 888
文献[18]	0.785 101	0.726 922	0.849 848	0.734 478	0.921 320	0.739 590	0.911 001	0.897 738
本文	0.791 670	0.773 700	0.885 564	0.868 461	0.937 303	0.880 019	0.932 124	0.924 232

[1]	Xiaomeng LI, Daidou GUO, Xunfang ZHUO, Heng YAO, Chuan QIN. Carrier-independent screen-shooting resistant watermarking based on information overlay superimposition [J]. Chinese Journal of Network and Information Security, 2023, 9(3): 135-149.
[2]	Rongna XIE, Zhuhong MA, Zongyu LI, Ye TIAN. Encrypted traffic classification method based on convolutional neural network [J]. Chinese Journal of Network and Information Security, 2022, 8(6): 84-91.
[3]	Jiaying LIN, Wenbo ZHOU, Weiming ZHANG, Nenghai YU. Lip forgery detection via spatial-frequency domain combination [J]. Chinese Journal of Network and Information Security, 2022, 8(6): 146-155.
[4]	Nuo CHEN, Shuren QI, Yushu ZHANG, Mingfu XUE, Zhongyun HUA. Image recoloring detection based on inter-channel correlation [J]. Chinese Journal of Network and Information Security, 2022, 8(5): 167-178.
[5]	Fan GAO, Jian WANG, Jiqiang LIU. Research on link detection technology based on dynamic browser fingerprint [J]. Chinese Journal of Network and Information Security, 2022, 8(4): 144-156.
[6]	Jinyin CHEN, Changan WU, Haibin ZHENG. Novel defense based on softmax activation transformation [J]. Chinese Journal of Network and Information Security, 2022, 8(2): 48-63.
[7]	Baolin QIU, Ping YI. Adversarial examples defense method based on multi-dimensional feature maps knowledge distillation [J]. Chinese Journal of Network and Information Security, 2022, 8(2): 88-99.
[8]	Xiangdong HU, Zhengguo TIAN. Methods of security situation prediction for industrial internet fused attention mechanism and BSRU [J]. Chinese Journal of Network and Information Security, 2022, 8(1): 41-51.
[9]	Lijuan LI, Man LI, Hongjun BI, Huachun ZHOU. Multi-type low-rate DDoS attack detection method based on hybrid deep learning [J]. Chinese Journal of Network and Information Security, 2022, 8(1): 73-85.
[10]	Zhongyuan QIN, Zhaoxiang HE, Tao LI, Liquan CHEN. Adversarial example defense algorithm for MNIST based on image reconstruction [J]. Chinese Journal of Network and Information Security, 2022, 8(1): 86-94.
[11]	Deqing ZOU, Xiang LI, Minhuan HUANG, Xiang SONG, Hao LI, Weiming LI. Intelligent vulnerability detection system based on graph structured source code slice [J]. Chinese Journal of Network and Information Security, 2021, 7(5): 113-122.
[12]	Zhenglong WANG, Baowen ZHANG. Survey of generative adversarial network [J]. Chinese Journal of Network and Information Security, 2021, 7(4): 68-85.
[13]	Binglong LI, Jinlong TONG, Yu ZHANG, Yifeng SUN, Qingxian WANG, Chaowen CHANG. Auto forensic detecting algorithms of malicious code fragment based on TensorFlow [J]. Chinese Journal of Network and Information Security, 2021, 7(4): 154-163.
[14]	Qingyin TAN, Yingming ZENG, Ye HAN, Yijing LIU, Zheli LIU. Survey on backdoor attacks targeted on neural network [J]. Chinese Journal of Network and Information Security, 2021, 7(3): 46-58.
[15]	Luhui YANG,Huiwen BAI,Guangjie LIU,Yuewei DAI. Lightweight malicious domain name detection model based on separable convolution [J]. Chinese Journal of Network and Information Security, 2020, 6(6): 112-120.