基于强化样本的伪孪生网络图像篡改定位模型

doi:10.11959/j.issn.2096-109x.2024010

网络与信息安全学报 ›› 2024, Vol. 10 ›› Issue (1): 33-47.doi: 10.11959/j.issn.2096-109x.2024010

• 学术论文 • 上一篇

基于强化样本的伪孪生网络图像篡改定位模型

王金伟¹^,²^,³^,⁴, 张子荷¹^,²^,³, 罗向阳⁴^,⁵, 马宾⁶

¹ 南京信息工程大学数字取证教育部工程研究中心，江苏南京210044
² 南京信息工程大学计算机学院，江苏南京210044
³ 南京信息工程大学网络空间安全学院，江苏南京210044
⁴ 数字工程与先进计算国家重点实验室，河南郑州450001
⁵ 信息工程大学，河南郑州 450001
⁶ 齐鲁工业大学网络空间安全学院，山东济南 250353

修回日期:2023-11-08 出版日期:2024-02-01 发布日期:2024-02-01
作者简介:王金伟（1978− ），男，内蒙古呼伦贝尔人，南京信息工程大学教授、博士生导师，主要研究方向为人工智能安全、多媒体取证和信息隐藏
张子荷（1999− ），女，河北廊坊人，南京信息工程大学硕士生，主要研究方向为信息安全、多媒体取证
罗向阳（1978− ），男，湖北荆门人，信息工程大学教授、博士生导师，主要研究方向为图像隐写、隐写分析技术
马宾（1973− ），男，山东济宁人，齐鲁工业大学教授、博士生导师，主要研究方向为可逆信息隐藏、多媒体取证、隐写与隐写分析
基金资助:
国家自然科学基金(62072250);国家自然科学基金(62172435);国家自然科学基金(U1804263);国家自然科学基金(U20B2065)

Pseudo-siamese network image tampering localization model based on reinforced samples

Jinwei WANG¹^,²^,³^,⁴, Zihe ZHANG¹^,²^,³, Xiangyang LUO⁴^,⁵, Bin MA⁶

¹ Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology, Nanjing 210044, China
² Department of Computer Science, Nanjing University of Information Science and Technology, Nanjing 210044, China
³ Department of Cyberspace Security, Nanjing University of Information Science and Technology, Nanjing 210044, China
⁴ State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou 450001, China
⁵ Force Information Engineering University, Zhengzhou 450001, China
⁶ School of Cyberspace Security, Qilu University of Technology, Jinan 250353, China

Revised:2023-11-08 Online:2024-02-01 Published:2024-02-01
Supported by:
The National Natural Science Foundation of China(62072250);The National Natural Science Foundation of China(62172435);The National Natural Science Foundation of China(U1804263);The National Natural Science Foundation of China(U20B2065)

摘要/Abstract

摘要：

随着互联网不断发展，网络上的篡改图像越来越多，掩盖篡改痕迹的手段越来越丰富。而现在大多数检测模型没有考虑到图像后处理操作对篡改检测算法的影响，限制了其在实际生活中的应用。为了解决上述问题，提出了一种通用的基于强化样本的伪孪生网络图像篡改定位模型。所提模型利用伪孪生网络，一方面学习真实图像中的篡改特征；另一方面通过约束卷积，抑制图像内容，从而能够更加关注篡改残留的痕迹信息。网络的两分支结构可以达到充分利用图像特征信息的目的。模型利用强化样本，可以自适应地生成当前最需要学习的篡改类型图片，实现对模型有针对性地训练，使得模型在各个方向上学习收敛，最终得到全局最优模型。利用数据增强思路，自动生成丰富的篡改图像以及其对应的掩膜，这很好地解决了篡改数据集有限的问题。在 4 个数据集上的大量实验证明了所提模型在像素级操作检测方面的可行性和有效性。尤其是在Columbia数据集上，算法的F1值提高了33.5%，Matthews correlation coefficirnt（MCC）得分提高了 23.3%，说明所提模型利用深度学习模型的优点，显著提高了篡改定位的检测效果。

关键词: 强化样本, 篡改定位, 伪孪生网络, 数据增强, 篡改图像

Abstract:

With the continuous development of the internet, an increasing number of images have been tampered with on the network, accompanied by a growing range of techniques to cover up tampering traces.However, most current detection models neglect the impact of image post-processing on tamper detection algorithms, limiting their real-life applications.To address these issues, a general image tampering location model based on enhanced samples and the pseudo-twin network was proposed.The pseudo-twin network enabled the model to learn tampering features in real images.On one hand, by applying convolution constraints, the image content was suppressed, allowing the model to focus more on residual trace information of tampering.The two-branch structure of the network facilitated the comprehensive utilization of image feature information.By utilizing enhanced samples, the model could dynamically generate the most crucial pictures for learning tamper types, enabling targeted training of the model.This approach ensured that the model converged in all directions, ultimately obtaining the global optimal model.The idea of data enhancement was employed to automatically generate abundant tampered images and corresponding masks, effectively resolving the limited tampering dataset issue.Extensive experiments were conducted on four datasets, demonstrating the feasibility and effectiveness of the proposed model in pixel-level tamper detection.Particularly on the Columbia dataset, the algorithm achieves a 33.5% increase in F1 score and a 23.3% increase in MCC score.These results indicate that the proposed model harnesses the advantages of deep learning models and significantly improves the effectiveness of tamper location detection.

Key words: enhanced sample, tampering positioning, pseudo-siamese network, data augmentation, tampering image

中图分类号:

TP391

王金伟, 张子荷, 罗向阳, 马宾. 基于强化样本的伪孪生网络图像篡改定位模型[J]. 网络与信息安全学报, 2024, 10(1): 33-47.

Jinwei WANG, Zihe ZHANG, Xiangyang LUO, Bin MA. Pseudo-siamese network image tampering localization model based on reinforced samples[J]. Chinese Journal of Network and Information Security, 2024, 10(1): 33-47.

图/表 20

图1

图2

图3

图4

图5

图6

图7

表1

表2

表3

表4

图8

图9

图10

图11

表5

表6

表7

表8

图12

参考文献 30

[1]	FERRARA P , BIANCHI T , DE ROSA A ,et al. Image forgery localization via fine-grained analysis of CFA artifacts[J]. IEEE Transactions on Information Forensics and Security, 2012,7(5): 1566-1577.
[2]	LI C , MA Q , XIAO L M ,et al. An image copy move forgery detection method using QDCT[C]// Proceedings of the International Conference on Internet Multimedia Computing and Service. 2016: 5-8.
[3]	AMBILI B , GEORGE N . A robust technique for splicing detection in tampered blurred images[C]// Proceedings of 2017 International Conference on Trends in Electronics and Informatics (ICEI). 2017: 897-901.
[4]	黄维, 黄添强, 张雪莉 ,等. 基于块效应网格偏移的重获取JPEG图像篡改检测[J]. 网络与信息安全学报, 2017,3(12): 24-30.
	HUANG W , HUANG T Q , ZHANG X L ,et al. JPEG recapture image tamper detection method based on block effect grid offset[J]. Chinese Journal of Network and Information Security, 2017,3(12): 24-30.
[5]	BAYAR B , STAMM M C . Constrained convolutional neural networks:a new approach towards general purpose image manipulation detection[J]. IEEE Transactions on Information Forensics and Security, 2018,13(11): 2691-2706.
[6]	WU Y , ABD-ALMAGEED W , NATARAJAN P . Image copy-move forgery detection via an end-to-end deep neural network[C]// Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). 2018: 1907-1915.
[7]	蒋小玉, 刘春晓 . 边缘与区域不一致性引导下的图像拼接篡改检测网络[J]. 中国图象图形学报, 2021,26(10): 2411-2420.
	JIANG X Y , LIU C X . Edge and region inconsistency-guided image splicing tamper detection network[J]. Journal of Image and Graphics, 2021,26(10): 2411-2420.
[8]	朱新同, 唐云祁, 耿鹏志 . 基于特征融合的篡改与深度伪造图像检测算法[J]. 信息网络安全, 2021(8): 70-81.
	ZHU X T , TANG Y Q , GENG P Z . Detection algorithm of tamper and deepfake image based on feature fusion[J]. Netinfo Security, 2021(8): 70-81.
[9]	李小红, 王晓霞 . 一种剪枝压缩CNN移动设备端图像篡改检测方法[J]. 计算机仿真, 2021,38(3): 83-86,91.
	LI X H , WANG X X . An image tamper detection method for CNN mobile devices based on pruning compression[J]. Computer Simulation, 2021,38(3): 83-86,91.
[10]	何家豪 . 图像复制移动篡改检测研究及实现[D]. 大连:大连理工大学, 2021.
	HE J H . Research and implementation of image copy mobile tampering detection[D]. Dalian:Dalian University of Technology, 2021.
[11]	ZHUANG P Y , LI H D , TAN S Q ,et al. Image tampering localization using a dense fully convolutional network[J]. IEEE Transactions on Information Forensics and Security, 2021,16: 2986-2999.
[12]	WEI Y , MA J F , WANG Z Z ,et al. Image splicing forgery detection by combining synthetic adversarial networks and hybrid dense U-net based on multiple spaces[J]. International Journal of Intelligent Systems, 2022,37(11): 8291-8308.
[13]	LU W , XU W B , SHENG Z Q . An interpretable image tampering detection approach based on cooperative game[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023,33(2): 952-962.
[14]	ZHAO T C , XU X , XU M Z ,et al. Learning self-consistency for deepfake detection[C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 2021: 15003-15013.
[15]	CHEN L , ZHANG Y , SONG Y B ,et al. Self-supervised learning of adversarial example:towards good generalizations for deepfake detection[C]// Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022: 18689-18698.
[16]	SOLIMAN M , LEHMAN C , ALREGIB G . S6:semi-supervised self-supervised semantic segmentation[C]// Proceedings of 2020 IEEE International Conference on Image Processing (ICIP). 2020: 1861-1865.
[17]	XU H , LI X , GONG W Y . Research on recognition of landslides with remote sensing images based on extreme learning machine[C]// Proceedings of 2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC). 2017: 740-747.
[18]	HALIASSOS A , MIRA R , PETRIDIS S ,et al. Leveraging real talking faces via self-supervision for robust forgery detection[C]// Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022: 14930-14942.
[19]	KRISHNA V A , REDDY A , NAGAJYOTHI D . Signature recognition using Siamese neural networks[C]// Proceedings of 2021 IEEE International Conference on Mobile Networks and Wireless Communications (ICMNWC). 2022: 1-4.
[20]	WANG Y Y , ZHANG W S , ZHANG L M ,et al. Depthwise over-parameterized Siamese network for visual tracking[C]// Proceedings of 2021 International Conference on Information Technology and Biomedical Engineering (ICITBE). 2022: 58-62.
[21]	SUN K , YAO T P , CHEN S ,et al. Dual contrastive learning for general face forgery detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022,36(2): 2316-2324.
[22]	COZZOLINO D , POGGI G , VERDOLIVA L . Splicebuster:a new blind image splicing detector[C]// Proceedings of 2015 IEEE International Workshop on Information Forensics and Security (WIFS). 2015: 1-6.
[23]	ZHOU P , HAN X T , MORARIU V I ,et al. Learning rich features for image manipulation detection[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018: 1053-1061.
[24]	ZHAO Q , CAO G , ZHOU A T ,et al. Image tampering detection via semantic segmentation network[C]// Proceedings of 2020 15th IEEE International Conference on Signal Processing (ICSP). 2020: 165-169.
[25]	CHEN J X , LIAO X , WANG W ,et al. SNIS:a signal noise separation-based network for post-processed image forgery detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023,33(2): 935-951.
[26]	WU Y , ABDALMAGEED W , NATARAJAN P . ManTra-Net:manipulation tracing network for detection and localization of image forgeries with anomalous features[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019: 9535-9544.
[27]	CHEN X R , DONG C B , JI J Q ,et al. Image manipulation detection by multi-view multi-scale supervision[C]// Proceedings of International Conference on Computer Vision. 2021.
[28]	WANG J , WU Z , CHEN J ,et al. ObjectFormer for image manipulation detection and localization[C]// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2022.
[29]	DONG J , WANG W , TAN T N . CASIA image tampering detection evaluation database[C]// Proceedings of 2013 IEEE China Summit and International Conference on Signal and Information Processing. 2013: 422-426.
[30]	HSU Y F , CHANG S F . Detecting image splicing using geometry invariants and camera characteristics consistency[C]// Proceedings of IEEE International Conference on Multimedia and Expo. 2006.

后处理操作类型	CASIA测试集F1值
JPEG压缩（质量因子Q=80）	0.532
高斯模糊（半径r=2）	0.534
边缘增强（Filteraegs=(3,3)，2,0）	0.537
改变色度（增强因子=0.7）	0.541
对比度增强（增强因子=1.5）	0.535

后处理操作	CASIA测试集F1值
高斯模糊（r=2）
均值滤波（r=6）
边缘增强（Filteraegs=(3,3),2,0）
锐化（Filteraegs=(3,3),16,0）
平滑（Filteraegs=(5,5),100,0）	0.542
细节提升（Filteraegs=(3,3),1,255）
改变亮度（增强因子=0.8）
改变色度（增强因子=0.7）
对比度增强（增强因子=1.5）
锐度增强（增强因子=3.0）
高斯模糊（r=4）
均值滤波（r=10）
边缘增强（Filteraegs=(3,3),2,0）
锐化（Filteraegs=(3,3),16,0）
平滑（Filteraegs=(5,5),100,0）	0.532
细节提升（Filteraegs=(3,3),1,255）
改变亮度（增强因子=0.9）
改变色度（增强因子=0.8）
对比度增强（增强因子=1.6）
锐度增强（增强因子=3.1）
高斯模糊（r=6）
均值滤波（r=12）
边缘增强（Filteraegs=(3,3),2,0）
锐化（Filteraegs=(3,3),16,0）
平滑（Filteraegs=(5,5),100,0）	0.538
细节提升（Filteraegs=(3,3),1,255）
改变亮度（增强因子=1.0）
改变色度（增强因子=0.9）
对比度增强（增强因子=1.7）
锐度增强（增强因子=3.2）

训练集中CASIA:自制数据集	训练集中CASIA图像数量	训练集中制作图像数量	测试集CASIAF1值
0.90:0.10	4 965	551	0.541
0.85:0.15	4 689	827	0.535
0.80:0.20	4 413	1 103	0.539
0.75:0.25	4 137	1 379	0.537
0.70:0.30	3 861	1 655	0.532
0.60:0.40	3 309	2 207	0.526
0.40:0.60	2 206	3 310	0.531
0.30:0.70	1 654	3 862	0.527

数据集	原图流	约束卷积流	强化样本模块	F1值
	√			0.458
CASIA	√	√		0.492
	√	√	√	0.542
	√			0.598
DSO-1	√	√		0.634
	√	√	√	0.683
	√			0.846
Columbia	√	√		0.894
	√	√	√	0.973

指导流	CASIA F1值
强化样本+原图流预测分类	0.542
强化样本+约束卷积流预测分类	0.528
强化样本+融合流预测分类	0.534

基于强化样本的伪孪生网络图像篡改定位模型

Pseudo-siamese network image tampering localization model based on reinforced samples

在线阅读

pdf下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 20

参考文献 30

相关文章 2

Metrics

推荐阅读 0

训练集中NIST16:自制数据集	训练集中NIST16图像数量	训练集中制作图像数量	测试集NIST16F1值
0.10:0.90	150	1353	0.628
0.20:0.80	300	1203	0.631
0.30:0.70	451	1052	0.625

数据集	F1值	MCC	召回率	精确率
CASIA	0.542	0.512	0.506	0.584
DSO-1	0.683	0.573	0.678	0.689
NIST16	0.635	0.572	0.639	0.631
Columbia	0.973	0.947	0.957	0.988

模型	CASIA		Columbia
模型	F1值	MCC	F1值	MCC
NOI3^[22]	0.176	0.089	0.448	0.207
CFA2^[1]	0.212	0.162	0.503	0.328
SNIS^[25]	0.368	—	0.681	—
RGB-N^23]	0.408	0.287	0.697	—
Zhao^[24]	0.423	0.427	0.733	0.714
ManTra^[26]	0.46	0.45	0.793	0.742
MVSS-Net^[27]	0.452	0.404	0.638	—
Lu^[13]	0.51	0.49	—	—
DenFCN^[11]	0.559	0.516	—	—
Wang^[28]	0.579	0.518	—	—
本文模型	0.542	0.512	0.973	0.947

[1]	冯冠云, 付才, 吕建强, 韩兰胜. 基于操作注意力和数据增强的内部威胁检测[J]. 网络与信息安全学报, 2023, 9(3): 102-112.
[2]	江玉朝,吉立新,高超,李邵梅. 面向Logo识别的合成数据生成方法研究[J]. 网络与信息安全学报, 2018, 4(5): 21-31.