基于噪声注意力的伪造人脸检测方法

doi:10.11959/j.issn.2096-109x.2023061

网络与信息安全学报 ›› 2023, Vol. 9 ›› Issue (4): 155-165.doi: 10.11959/j.issn.2096-109x.2023061

• 学术论文 • 上一篇

基于噪声注意力的伪造人脸检测方法

张博林¹^,²^,³, 朱春陶¹^,²^,³, 殷琪林¹^,²^,³, 付婧巧¹^,²^,³, 刘凌毅¹^,²^,³, 刘佳睿¹^,²^,³, 刘红梅¹^,²^,³, 卢伟¹^,²^,³

¹ 中山大学计算机学院，广东广州 510006
² 广东省信息安全技术重点实验室，广东广州 510006
³ 机器智能与先进计算教育部重点实验室，广东广州 510006

修回日期:2023-05-26 出版日期:2023-08-01 发布日期:2023-08-01
作者简介:张博林（1999- ），男，陕西宝鸡人，中山大学硕士生，主要研究方向为深度伪造检测
朱春陶（1997- ），女，河南周口人，中山大学硕士生，主要研究方向为深度伪造检测
殷琪林（1995- ），男，江苏泰州人，中山大学博士生，主要研究方向为数字多媒体取证
付婧巧（1987- ），女，四川南充人，中山大学硕士生，主要研究方向为自然语言处理
刘凌毅（1998- ），男，江西瑞金人，中山大学硕士生，主要研究方向为深度伪造检测
刘佳睿（1997- ），男，山东招远人，中山大学硕士生，主要研究方向为深度伪造检测
刘红梅（1969- ），女，内蒙古丰镇人，中山大学副教授，主要研究方向为图像/视频处理、多媒体内容安全
卢伟（1979- ），男，河南南阳人，中山大学教授、博士生导师，主要研究方向为多媒体信息安全、数字取证、信息隐藏、隐私保护、多媒体大数据智能
基金资助:
国家自然科学基金(U2001202);国家自然科学基金(62072480)

Noise-attention-based forgery face detection method

Bolin ZHANG¹^,²^,³, Chuntao ZHU¹^,²^,³, Qilin YIN¹^,²^,³, Jingqiao FU¹^,²^,³, Lingyi LIU¹^,²^,³, Jiarui LIU¹^,²^,³, Hongmei LIU¹^,²^,³, Wei LU¹^,²^,³

¹ School of Computer Science and Engineering, Sun Yat-sen University, GuangZhou 510006, China
² Guangdong Province Key Laboratory of Information Security Technology, GuangZhou 510006, China
³ Ministry of Education Key Laboratory of Machine Intelligence and Advanced Computing, GuangZhou 510006, China

Revised:2023-05-26 Online:2023-08-01 Published:2023-08-01
Supported by:
The National Natural Science Foundation of China(U2001202);The National Natural Science Foundation of China(62072480)

摘要/Abstract

摘要：

随着人工智能和深度神经网络的不断发展，图像生成与编辑变得越来越容易，恶意运用图像生成工具进行篡改伪造的现象层出不穷，这对多媒体安全以及社会稳定造成了极大威胁，因此研究伪造人脸的检测方法至关重要。人脸篡改伪造的方式和工具多种多样，在篡改的过程中可能留下不同程度的篡改痕迹，而这在图像噪声中都有一定程度上的反映。从图像噪声的角度出发，通过噪声去除的方式挖掘反映伪造人脸篡改痕迹的噪声成分，进一步生成噪声注意力，指导主干网络进行伪造人脸检测。使用 SRM 滤波监督噪声去除模块的训练，并将噪声去除模块所得到的噪声再次加入真实人脸图像中，形成一对有监督的训练样本，通过自监督的方式对噪声去除模块进行加强指导，实验结果说明噪声去除模块得到的噪声特征具有较好的区分度。在多个公开数据集上进行了实验，所提方法在 Celeb-DF 数据集上达到 98.32%的准确率，在FaceForensics++数据集上达到 94%以上的准确率，在 DFDC 数据集上达到 92.61%的准确率，证明了所提方法的有效性。

关键词: Deepfake检测, 图像噪声, 注意力机制, 篡改痕迹

Abstract:

With the advancement of artificial intelligence and deep neural networks, the ease of image generation and editing has increased significantly.Consequently, the occurrence of malicious tampering and forgery using image generation tools is on the rise, posing a significant threat to multimedia security and social stability.Therefore, it is crucial to research detection methods for forged faces.Face tampering and forgery can occur through various means and tools, leaving different levels of forgery traces during the tampering process.These traces can be partly reflected in the image noise.From the perspective of image noise, the noise components reflecting tampering traces of forged faces were extracted through a noise removal module.Furthermore, noise attention was generated to guide the backbone network in the detection of forged faces.The training of the noise removal module was supervised using SRM filters.In order to strengthen the guidance of the noise removal module, the noise obtained by the noise removal module was added back to the real face image, forming a pair of supervised training samples in a self-supervised manner.The experimental results illustrate that the noise features obtained by the noise removal module have a good degree of discrimination.Experiments were also conducted on several public datasets, and the proposed method achieves an accuracy of 98.32% on the Celeb-DF dataset, 92.61% on the DFDC dataset, and more than 94% on the FaceForensics++ dataset, thus proving the effectiveness of the proposed method.

Key words: Deepfake detection, image noise, attention mechanism, tempering artifact

中图分类号:

TP393

张博林, 朱春陶, 殷琪林, 付婧巧, 刘凌毅, 刘佳睿, 刘红梅, 卢伟. 基于噪声注意力的伪造人脸检测方法[J]. 网络与信息安全学报, 2023, 9(4): 155-165.

Bolin ZHANG, Chuntao ZHU, Qilin YIN, Jingqiao FU, Lingyi LIU, Jiarui LIU, Hongmei LIU, Wei LU. Noise-attention-based forgery face detection method[J]. Chinese Journal of Network and Information Security, 2023, 9(4): 155-165.

图/表 9

图1

表1

不同方法在FF++数据集上的检测准确率（ACC） Table 1 Accuracy of detection on FF++ dataset with different methods"

方法	ACC
方法	FF++C23	FF++C40
BayarConv	82.97 %	66.84 %
MesoNet	83.10 %	70.47 %
Xception	95.73 %	86.86 %
EfficientNet-B4	96.63 %	86.67 %
F3-Net	97.52 %	90.43 %
SPSL	—	93.48 %
本文	$98 . 47 %$	$95 . 01 %$

表1

表2

不同方法在Celeb-DF数据集上的检测结果Table 2 Result of detection on Celeb-DF dataset with different methods"

方法	ACC	AUC
MesoNet	—	53.6 %
I3D	76.08 %	83.00 %
C3D	78.67 %	84.00 %
Xception	89.55 %	89.91 %
GFFD	94.14 %	$98 . 30 %$
本文	$98 . 32 %$	97.77 %

表2

表3

不同方法在DFDC数据集上的检测结果Table 3 Result of detection on DFDC dataset with different methods"

方法	ACC
Xception	88.91 %
Multi-attention	91.02 %
RFM	90.01 %
GFFD	92.33 %
本文	$92 . 61 %$

表3

表4

表5

图2

表6

表7

参考文献 55

[1]	XU W , KESHMIRI S , WANG G . Adversarially approximated auto encoder for image generation and manipulation[J]. IEEE Transactions on Multimedia, 2019,21(9): 2387-2396.
[2]	GOODFELLOW I , POUGET-ABADIE J , MIRZA M ,et al. Generative adversarial nets[C]// Advances in neural Information Processing Systems. 2014:27.
[3]	LIU M , DING Y , XIA M ,et al. Stgan:a unified selective transfer network for arbitrary image attribute editing[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 3673-3682.
[4]	CHOI Y , CHOI M , KIM M ,et al. Stargan:unified generative adversarial networks for multi-domain image-to-image translation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 8789-8797.
[5]	KARRAS T , LAINE S , AITTALA M ,et al. Analyzing and improving the image quality of stylegan[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8110-8119.
[6]	王鹏程, 郑海斌, 邹健飞 ,等. 面向商用活体检测平台的鲁棒性评估[J]. 网络与信息安全学报, 2022,8(1): 180-189.
	WANG P C , ZHENG H B , ZOU J F ,et al. Robustness evaluation of commercial liveness detection platform[J]. Chinese Journal of Network and Information Security, 2022,8(1): 180-189.
[7]	WAHEED A , GOYAL M , GUPTA D ,et al. An optimized dense convolutional neural network model for disease recognition and classification in corn leaf[J]. Computers and Electronics in Agriculture, 2020,175:105456.
[8]	CHOLLET F . Xception:deep learning with depthwise separable convolutions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1251-1258.
[9]	MATERN F , RIESS C , WINTER M S 2019 I ,et al. Exploiting visual artifacts to expose deepfakes and face manipulations[C]// 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW). 2019: 83-92.
[10]	GüERA D , DELP E J . Deepfake video detection using recurrent neural networks[C]// 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). 2018: 1-6.
[11]	NIRKIN Y , WOLF L , KELLER Y ,et al. Deepfake detection based on the discrepancy between the face and its context[J]. arXiv Preprint arXiv:2008.12262, 2020.
[12]	MCCLOSKEY S , ALBRIGHT M , ACST H . Detecting GAN-generated imagery using color cues[J]. arXiv Preprint arXiv:1812.08247, 2018.
[13]	PENG B , WANG W , DONG J ,et al. Optimized 3D lighting environment estimation for image forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2016,12(2): 479-494.
[14]	YU N , DAVIS L S , FRITZ M . Attributing fake images to GANs:Learning and analyzing GAN fingerprints[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 7556-7566.
[15]	FRANK J , EISENHOFER T , SCH?NHERR L , et al . Leveraging frequency analysis for deep fake image recognition[C]// International Conference on Machine Learning. 2020: 3247-3258.
[16]	QIAN Y , YIN G , SHENG L ,et al. Thinking in frequency:face forgery detection by mining frequency-aware clues[C]// European Conference on Computer Vision. 2020: 86-103.
[17]	FRIDRICH J , KODOVSKY J . Rich models for steganalysis of digital images[J]. IEEE Transactions on Information Forensics and Security, 2012,7(3): 868-882.
[18]	LI Y , YANG X , SUN P ,et al. Celeb-df:a large-scale challenging dataset for deepfake forensics[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 3207-3216.
[19]	R?SSLER A , COZZOLINO D , VERDOLIVA L ,et al. Faceforensics++:learning to detect manipulated facial images[C]// Interna- tional Conference on Computer Vision(ICCV). 2019: 1-11.
[20]	AGARWAL S , FARID H , EL-GAALY T ,et al. Detecting deepfake videos from appearance and behavior[C]// 2020 IEEE International Workshop on Information Forensics and Security(WIFS). 2020: 1-6.
[21]	NIRKIN Y , WOLF L , KELLER Y ,et al. Deepfake detection based on discrepancies between faces and their context[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021,(8).
[22]	MCCLOSKEY S , ALBRIGHT M . Detecting GAN-generated imagery using saturation cues[C]// 2019 IEEE International Conference on Image Processing (ICIP). 2019: 4584-4588.
[23]	MARRA F , GRAGNANIELLO D , VERDOLIVA L ,et al. Do GANs leave artificial fingerprints?[C]// 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). 2019: 506-511.
[24]	AGARWAL S , FARID H , Fried O ,et al. Detecting deepfake videos from phoneme-viseme mismatches[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020: 660-661.
[25]	BHATTACHARYA U , MITTAL T , CHANDRA R ,et al. Step:spatial temporal graph convolutional networks for emotion perception from gaits[C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2020: 1342-1350.
[26]	LI Y , CHANG M C , LYU S . In ICTU oculi:exposing AI created fake videos by detecting eye blinking[C]// 2018 IEEE International Workshop on Information Forensics and Security (WIFS). 2018: 1-7.
[27]	LI Y , LYU S . Exposing deepfake videos by detecting face warping artifacts[J]. arXiv Preprint arXiv:1811.00656, 2018.
[28]	MATERN F , RIESS C , STAMMINGER M . Exploiting visual artifacts to expose deepfakes and face manipulations[C]// 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW). 2019: 83-92.
[29]	QI H , GUO Q , JUEFEI X F ,et al. Deeprhythm:exposing deepfakes with attentional visual heartbeat rhythms[C]// Proceedings of the 28th ACM International Conference on Multimedia. 2020: 4318-4327.
[30]	KOOPMAN M , RODRIGUEZ A M , GERADTS Z . Detection of deepfake video manipulation[C]// The 20th Irish Machine Vision and Image Processing Conference (IMVIP). 2018: 133-136.
[31]	LI H , LI B , TAN S ,et al. Identification of deep network generated images using disparities in color components[J]. Signal Processing, 2020(174): 107616.
[32]	DO N T , NA I S , KIM S H . Forensics face detection from GANs using convolutional neural network[C]// ISITC. 2018: 376-379.
[33]	SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[J]. arXiv Preprint arXiv:1409.1556, 2014.
[34]	TARIQ S , LEE S , KIM H ,et al. Detecting both machine and human created fake face images in the wild[C]// Proceedings of the 2nd International Workshop on Multimedia Privacy and Security. 2018: 81-87.
[35]	HE K , ZHANG X , REN S ,et al. Deep residual learning for image recognition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
[36]	NGUYEN H , YAMAGISHI J , ECHIZEN I . Capsule-forensics:Using capsule networks to detect forged images and videos[C]// 2019 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP). 2019: 2307-2311.
[37]	SABIR E , CHENG J , JAISWAL A ,et al. Recurrent convolutional strategies for face manipulation detection in videos[J]. Interfaces (GUI), 2019,3(1): 80-87.
[38]	HSU C C , ZHUANG Y X , LEE C Y . Deepfake image detection based on pairwise learning[J]. Applied Sciences, 2020,10(1): 370.
[39]	GANDHI A , JAIN S . Adversarial perturbations fool deepfake detectors[C]// 2020 International Joint Conference on Neural Networks (IJCNN). 2020: 1-8.
[40]	龚晓娟, 黄添强, 翁彬 ,等. 基于双层注意力的 Deepfake 换脸检测[J]. 网络与信息安全学报, 2021,7(2): 151-160.
	GONG X J , HUANG T Q , WENG B ,et al. Deepfake swapped face detection based on double attention[J]. Chinese Journal of Network and Information Security, 2021,7(2): 151-160.
[41]	WU X , XIE Z , GAO Y ,et al. Sstnet:detecting manipulated faces through spatial,steganalysis and temporal features[C]// 2020 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP). 2020: 2952-2956.
[42]	LIU Z , QI X , TORR P H S . Global texture enhancement for fake face detection in the wild[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8060-8069.
[43]	RONNEBERGER O , FISCHER P , BROX T . U-net:convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. 2015: 234-241.
[44]	WANG F , CHENG J , LIU W ,et al. Additive margin softmax for face verification[J]. IEEE Signal Processing Letters, 2018,25(7): 926-930.
[45]	KING D E . Dlibml:a machine learning toolkit[J]. The Journal of Machine Learning Research, 2009,10: 1755-1758.
[46]	BAYAR B , MEMBER S , STAMM M C . Constrained convolutional neural networks:a new approach towards general purpose image manipulation detection[J]. IEEE Transactions on Information Forensics and Security, 2018,13(11): 2691-2706.
[47]	AFCHAR D , NOZICK V , YAMAGISHI J ,et al. MesoNet:a compact facial video forgery detection network[C]// 2018 IEEE International Workshop on Information Forensics and Security(WIFS). 2018: 1-7.
[48]	TAN M , LE Q . Efficientnet:rethinking model scaling for convolutional neural networks[C]// International Conference on Machine Learning. 2019: 6105-6114.
[49]	LIU H , LI X , ZHOU W ,et al. Spatial-phase shallow learning:rethinking face forgery detection in frequency domain[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 772-781.
[50]	TRAN D , BOURDEV L , FERGUS R ,et al. Learning spatiotemporal features with 3D convolutional networks[C]// Proceedings of the IEEE International Conference on Computer Vision. 2015: 4489-4497.
[51]	CARREIRA J , ZISSERMAN A , COM Z ,et al. Quo vadis,action recognition? a new model and the kinetics dataset[C]// proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 6299-6308.
[52]	LUO Y , ZHANG Y , YAN J ,et al. Generalizing face forgery detection with high-frequency features[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 16317-16326.
[53]	DOLHANSKY B , HOWES R , PFLAUM B ,et al. The deepfake detection challenge (DFDC) preview dataset[J]. arXiv Preprint arXiv:1910.08854, 2019.
[54]	WANG C , DENG W . Representative forgery mining for fake face detection[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 14923-14932.
[55]	ZHAO H , ZHOU W , CHEN D ,et al. Multi-attentional deepfake detection[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 2185-2194.

方法	ACC
无SRM监督模块	96.96 %
无自监督模块	96.77 %
无SRM监督模块和自监督模块	96.03 %
本文	98.32 %

数据集	ACC	FPR	FNR
Deepfake	98.47 %	1.73 %	1.33 %
Face2Face	98.33 %	2.81 %	0.52 %
FaceShifter	98.25 %	2.33 %	1.17 %
NeuralTextures	92.93 %	9.55 %	5.07 %
Celeb-DF	98.32 %	0.36 %	4.10 %
DFDC	92.61 %	7.95 %	7.29 %

方法	参数量	FLOP	ACC
Xception	20.81 MB	8.44 GB	95.73%
F3-Net	41.63 MB	16.91GB	97.52%
本文	9.72 MB	5.08 GB	98.47%

基于噪声注意力的伪造人脸检测方法

Noise-attention-based forgery face detection method

在线阅读

pdf下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 55

相关文章 7

Metrics

推荐阅读 0

[1]	朱春陶, 尹承禧, 张博林, 殷琪林, 卢伟. 基于多域时序特征挖掘的伪造人脸检测方法[J]. 网络与信息安全学报, 2023, 9(3): 123-134.
[2]	章登勇, 文凰, 李峰, 曹鹏, 向凌云, 杨高波, 丁湘陵. 基于双分支网络的图像修复取证方法[J]. 网络与信息安全学报, 2022, 8(6): 110-122.
[3]	高凡, 王健, 刘吉强. 基于动态浏览器指纹的链接检测技术研究[J]. 网络与信息安全学报, 2022, 8(4): 144-156.
[4]	胡向东, 田正国. 融合注意力机制和BSRU的工业互联网安全态势预测方法[J]. 网络与信息安全学报, 2022, 8(1): 41-51.
[5]	贾鉴,刘林峰,吴家皋. 基于循环神经网络的空载电动出租车的充电桩推荐方法[J]. 网络与信息安全学报, 2020, 6(6): 152-163.
[6]	曲强,于洪涛,黄瑞阳. 基于注意力机制的社交垃圾文本检测方法[J]. 网络与信息安全学报, 2020, 6(1): 54-61.
[7]	张晓斌, 陈福才, 黄瑞阳. 基于CNN和双向LSTM融合的实体关系抽取[J]. 网络与信息安全学报, 2018, 4(9): 44-51.

方法	ACC
SRM滤波	96.43 %
噪声特征	97.40 %