基于多域时序特征挖掘的伪造人脸检测方法

doi:10.11959/j.issn.2096-109x.2023044

网络与信息安全学报 ›› 2023, Vol. 9 ›› Issue (3): 123-134.doi: 10.11959/j.issn.2096-109x.2023044

基于多域时序特征挖掘的伪造人脸检测方法

朱春陶¹^,²^,³, 尹承禧¹^,²^,³, 张博林¹^,²^,³, 殷琪林¹^,²^,³^,⁴, 卢伟¹^,²^,³

¹ 中山大学计算机学院，广东广州 510006
² 广东省信息安全技术重点实验室（中山大学），广东广州 510006
³ 机器智能与先进计算教育部重点实验室（中山大学），广东广州 510006
⁴ 山东省计算机网络重点实验室，山东济南 250014

修回日期:2023-04-23 出版日期:2023-06-25 发布日期:2023-06-01
作者简介:朱春陶（1997- ），女，河南周口人，中山大学硕士生，主要研究方向为深度伪造检测
尹承禧(1999- )，男，广东广州人，中山大学硕士生，主要研究方向为多媒体信息安全和数字图像取证
张博林（1999- ），男，陕西宝鸡人，中山大学硕士生，主要研究方向为深度伪造检测
殷琪林（1995- ），男，江苏泰州人，中山大学博士生，主要研究方向为数字多媒体取证
卢伟（1979- ），男，河南南阳人，中山大学教授、博士生导师，主要研究方向为多媒体信息安全、人工智能安全与对抗、信息隐藏
基金资助:
国家自然科学基金(U2001202);国家自然科学基金(62072480)

Forgery face detection method based on multi-domain temporal features mining

Chuntao ZHU¹^,²^,³, Chengxi YIN¹^,²^,³, Bolin ZHANG¹^,²^,³, Qilin YIN¹^,²^,³^,⁴, Wei LU¹^,²^,³

¹ School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China
² Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University, Guangzhou 510006, China
³ Ministry of Education Key Laboratory of Machine Intelligence and Advanced Computing, Sun Yat-sen University, Guangzhou 510006, China
⁴ Shandong Key Laboratory of Computer Networks, Shandong 250014, China

Revised:2023-04-23 Online:2023-06-25 Published:2023-06-01
Supported by:
The National Natural Science Foundation of China(U2001202);The National Natural Science Foundation of China(62072480)

摘要/Abstract

摘要：

随着计算机技术在金融服务行业中的不断发展，金融科技便利了人们的日常生活，与此同时，数字金融存在着危害性极大的安全问题。人脸生物信息作为人物身份信息的重要组成部分，广泛应用于金融行业中的支付系统、账号注册等方面；伪造人脸技术的出现不断冲击着数字金融安全体系，给国家资产安全和社会稳定造成了一定的威胁。为了应对伪造人脸带来的安全问题，提出了一种基于多域时序特征挖掘的伪造人脸检测方法。所提方法从视频在空域和频域中存在的时序特征出发，基于人脸统计特征数据分布的一致性以及时间上动作趋势的一致性，对篡改特征进行区分增强。在空域中，所提方法使用改进的长短记忆网络（LSTM）来挖掘帧间的时序特征；在频域中，利用 3D 卷积层来挖掘不同频段频谱的时序信息，并与主干网络提取到的篡改特征进行融合，进而有效地区分伪造人脸和真实人脸。所提方法在主流数据集中表现优越，证明了所提方法的有效性。

关键词: 人脸身份认证, Deepfake检测, 时序特征, 多域特征

Abstract:

Financial technology has greatly facilitated people’s daily life with the continuous development of computer technology in the financial services industry.However, digital finance is accompanied by security problems that can be extremely harmful.Face biometrics, as an important part of identity information, is widely used in payment systems, account registration, and many other aspects of the financial industry.The emergence of face forgery technology constantly impacts the digital financial security system, posing a threat to national asset security and social stability.To address the security problems caused by fake faces, a forgery face detection method based on multi-domain temporal features mining was proposed.The tampering features were distinguished and enhanced based on the consistency of statistical feature data distribution and temporal action trend in the temporal features of videos existing in the spatial domain and frequency domain.Temporal information was mined in the spatial domain using an improved LSTM, while in the frequency domain, temporal information existing in different frequency bands of the spectrum was mined using 3D convolution layers.The information was then fused with the tampering features extracted from the backbone network, thus effectively distinguishing forged faces from real ones.The effectiveness of the proposed method was demonstrated on mainstream datasets.

Key words: face authentication, Deepfake detection, temporal features, multi-domain features

中图分类号:

TP393

朱春陶, 尹承禧, 张博林, 殷琪林, 卢伟. 基于多域时序特征挖掘的伪造人脸检测方法[J]. 网络与信息安全学报, 2023, 9(3): 123-134.

Chuntao ZHU, Chengxi YIN, Bolin ZHANG, Qilin YIN, Wei LU. Forgery face detection method based on multi-domain temporal features mining[J]. Chinese Journal of Network and Information Security, 2023, 9(3): 123-134.

图/表 12

图1

图2

图3

表1

表2

表3

表4

表5

表6

表7

表8

表9

参考文献 43

[1]	DeepFakes GitHub non official project based on original deepfakes thread[EB].
[2]	THIES J , ZOLLH?FER M , STAMMINGER M , et al . Face2Face:real-time face capture and reenactment of RGB videos[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016: 2387-2395.
[3]	FaceSwap GitHub non official project based on original deepfakes thread[EB].
[4]	GOODFELLOW I J , POUGET-ABADIE J , MIRZA M ,et al. Generative adversarial networks[J]. arXiv Preprint arXiv:1406.2661. 2014.
[5]	XU W J , KESHMIRI S , WANG G H . Adversarially approximated autoencoder for image generation and manipulation[J]. IEEE Transactions on Multimedia, 2019,21(9): 2387-2396.
[6]	王鹏程, 郑海斌, 邹健飞 ,等. 面向商用活体检测平台的鲁棒性评估[J]. 网络与信息安全学报, 2022,8(1): 180-189.
	WANG P C , ZHENG H B , ZOU J F ,et al. Robustness evaluation of commercial liveness detection platform[J]. Chinese Journal of Network and Information Security, 2022,8(1): 180-189.
[7]	MATERN F , RIESS C , STAMMINGER M . Exploiting visual artifacts to expose deepfakes and face manipulations[C]// Proceedings of 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW). 2019: 83-92.
[8]	PENG B , WANG W , DONG J ,et al. Optimized 3D lighting environment estimation for image forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2017,12(2): 479-494.
[9]	NIRKIN Y , WOLF L , KELLER Y ,et al. DeepFake detection based on discrepancies between faces and their context[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022,44(10): 6111-6121.
[10]	LIU H G , LI X D , ZHOU W B ,et al. Spatial-phase shallow learning:rethinking face forgery detection in frequency domain[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 772-781.
[11]	林佳滢, 周文柏, 张卫明 ,等. 空域频域相结合的唇型篡改检测方法[J]. 网络与信息安全学报, 2022,8(6): 146-155.
	LIN J Y , ZHOU W B , ZHANG W M ,et al. Lip forgery detection via spatial-frequency domain combination[J]. Chinese Journal of Network and Information Security, 2022,8(6): 146-155.
[12]	LI Y Z , CHANG M C , LYU S W . Exposing AI created fake videos by detecting eye blinking[C]// Proceedings of 2018 IEEE International Workshop on Information Forensics and Security (WIFS). 2019: 1-7.
[13]	SUN Z K , HAN Y J , HUA Z Y ,et al. Improving the efficiency and robustness of deepfakes detection through precise geometric features[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 3608-3617.
[14]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[15]	WANG Y B , WU H X , ZHANG J J ,et al. PredRNN:a recurrent neural network for spatiotemporal predictive learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023,45(2): 2208-2225.
[16]	THIES J , ZOLLH?FER M , NIE?NER M . Deferred neural rendering[J]. ACM Transactions on Graphics, 2019,38(4): 1-12.
[17]	LIU M , BREUEL T , KAUTZ J . Unsupervised image-to-image translation networks[J]. arXiv Preprint arXiv:1703.00848, 2017.
[18]	KARRAS T , AILA T , LAINE S ,et al. Progressive growing of gans for improved quality,stability,and variation[J]. arXiv Preprint arXiv:1710.10196, 2017.
[19]	KARRAS T , LAINE S , AILA T M . A style-based generator architecture for generative adversarial networks[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 4396-4405.
[20]	HUANG X , BELONGIE S . Arbitrary style transfer in real-time with adaptive instance normalization[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). 2017: 1510-1519.
[21]	LI L Z , BAO J M , YANG H ,et al. Advancing high fidelity identity swapping for forgery detection[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 5073-5082.
[22]	PAN X Y , ZHANG X , LYU S W . Exposing image splicing with inconsistent local noise variances[C]// Proceedings of 2012 IEEE International Conference on Computational Photography (ICCP). 2012: 1-10.
[23]	FERRARA P , BIANCHI T , DE ROSA A ,et al. Image forgery localization via fine-grained analysis of CFA artifacts[J]. IEEE Transactions on Information Forensics and Security, 2012,7(5): 1566-1577.
[24]	FRIDRICH J , KODOVSKY J . Rich models for steganalysis of digital images[J]. IEEE Transactions on Information Forensics and Security, 2012,7(3): 868-882.
[25]	YANG X , LI Y Z , LYU S W . Exposing deep fakes using inconsistent head poses[C]// Proceedings of ICASSP 2019 - 2019 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP). 2019: 8261-8265.
[26]	AGARWAL S , FARID H , GU Y ,et al. Protecting world leaders against deep fakes[C]// Proceedings of CVPR Workshops. 2019,1:38.
[27]	CHOLLET F . Xception:deep learning with depthwise separable convolutions[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017: 1800-1807.
[28]	AFCHAR D , NOZICK V , YAMAGISHI J ,et al. MesoNet:a compact facial video forgery detection network[C]// Proceedings of 2018 IEEE International Workshop on Information Forensics and Security (WIFS). 2019: 1-7.
[29]	NGUYEN H H , YAMAGISHI J , ECHIZEN I . Capsule-forensics:using capsule networks to detect forged images and videos[C]// Proceedings of ICASSP 2019 - 2019 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP). 2019: 2307-2311.
[30]	LI L Z , BAO J M , ZHANG T ,et al. Face X-ray for more general face forgery detection[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 5000-5009.
[31]	DURALL R , KEUPER M , PFREUNDT F J ,et al. Unmasking DeepFakes with simple features[J]. arXiv Preprint arXiv:1911.00686, 2019.
[32]	QIAN Y Y , YIN G J , SHENG L ,et al. Thinking in frequency:face forgery detection by mining frequency-aware clues[C]// Proceedings of European Conference on Computer Vision. 2020: 86-103.
[33]	GüERA D , DELP E J . Deepfake video detection using recurrent neural networks[C]// Proceedings of 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). 2019: 1-6.
[34]	HU J , LIAO X , WANG W ,et al. Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022,32(3): 1089-1102.
[35]	LU W , LIU L Y , ZHANG B L ,et al. Detection of deepfake videos using long-distance attention[J]. IEEE Transactions on Neural networks and Learning Systems, 2023,(99): 1-14.
[36]	SHI X J , CHEN Z R , WANG H ,et al. Convolutional LSTM Network:a machine learning approach for precipitation nowcasting[C]// Proceedings of the 28th International Conference on Neural Information Processing System. 2015: 802-810.
[37]	JADERBERG M , SIMONYAN K , ZISSERMAN A ,et al. Spatial transformer networks[J]. arXiv Preprint arXiv:1506.02025, 2015.
[38]	WANG Q L , WU B G , ZHU P F ,et al. ECA-net:efficient channel attention for deep convolutional neural networks[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 11531-11539.
[39]	R?SSLER A , COZZOLINO D , VERDOLIVA L ,et al. FaceForensicss:learning to detect manipulated facial images[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2020: 1-11.
[40]	LI Y Z , YANG X , SUN P ,et al. Celeb-DF:a large-scale challenging dataset for DeepFake forensics[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 3204-3213.
[41]	DOLHANSKY B , HOWES R , PFLAUM B ,et al. The deepfake detection challenge (DFDC) preview dataset[J]. arXiv Preprint arXiv:1910.08854, 2019.
[42]	WANG C R , DENG W H . Representative forgery mining for fake face detection[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 14918-14927.
[43]	ZHAO H Q , WEI T Y , ZHOU W B ,et al. Multi-attentional deepfake detection[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 2185-2194.

检测方法	Deepfakes		Face2Face		FaceSwap		NeuralTextures
检测方法	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
Xception	0.967 9	0.976 4	0.917 9	0.931 7	0.960 7	0.970 5	0.817 9	0.823 6
Multi-attention			0.946 4	0.955 4			0.817 9	0.838 9
F³Net	0.971 4	0.978 5			0.971 4	0.974 8		0.846 4
Two-stream	0.946 4	0.980 0	0.864 8	0.940 0	0.852 7	0.940 0	0.800 5
RNN	0.817 0	0.860 0	0.708 5	0.770 0	0.668 3	0.760 0	0.691 0	0.780 0
本文方法
注：加粗指标为最优实验结果，下划线指标为次优实验结果，余同。

检测方法	Deepfakes		Face2Face		FaceSwap		NeuralTextures
检测方法	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
Xception	0.945 8	0.986 8	0.893 5	0.960 4	0.924 2	0.978 0	0.743 5	0.812 4
Multi-attention			0.897 6	0.959 6	0.942 5
RFM	0.952 4	0.992 1	0.894 0	0.956 1	0.928 9	0.980 0	0.748 3	0.822 9
F³Net
本文方法	0.954 1	0.989 1				0.978 6	0.769 3	0.839 8

检测方法	ACC
检测方法	帧级		视频级
Xception	0.978 0		0.992 3	0.990 1
Multi-attention		0.995 2	0.990 3	0.988 7
RFM	0.980 9		—	—
F³Net		0.993 9
本文方法	0.984 5	0.997 4

检测方法	ACC
检测方法	帧级		视频级
Xception	0.889 1	0.960 1	0.898 4	0.897 8
Multi-attention
RFM	0.900 1	0.967 3	—	—
F³Net				0.924 5
本文方法	0.895 0	0.958 1	0.890 6

检测方法（HQ）	AUC
检测方法（HQ）	FaceForensics++	Celeb-DF
Xception	0.985 1	0.673 1
Multi-attention	0.994 3	0.647 0
RFM	0.988 2	0.697 3
F³Net		0.635 2
本文方法	0.992 6

基于多域时序特征挖掘的伪造人脸检测方法

Forgery face detection method based on multi-domain temporal features mining

在线阅读

pdf下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 43

相关文章 15

Metrics

推荐阅读 0

检测方法（LQ）	AUC
检测方法（LQ）	FaceForensics++	Celeb-DF
Xception	0.904 2	0.617 1
Multi-attention	0.924 6	0.595 0
RFM	0.910 0	0.635 9
F³Net		0.620 4
本文方法	0.927 1

帧数			ACC
帧数	Deepfakes	Face2Face	FaceSwap	NeuralTextures
4	0.948 6	0.901 6		0.760 4
6			0.943 2
8	0.952 6	0.903 2	0.942 0	0.765 6

对照组	Deepfakes		Face2Face		FaceSwap		NeuralTextures
对照组	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
主干	0.943 4	0.983 9	0.893 3	0.954 9	0.932 8	0.973 7	0.750 5	0.826 0
主干+空域	0.950 4	0.990 0	0.904 3		0.941 7		0.759 0	0.836 6
主干+频域	0.945 1		0.896 9	0.962 7	0.940 2	0.978 9	0.750 0	0.829 1
主干+多域		0.989 1		0.961 5		0.978 6

[1]	陈先意, 顾军, 颜凯, 江栋, 许林峰, 付章杰. 针对车牌识别系统的双重对抗攻击[J]. 网络与信息安全学报, 2023, 9(3): 16-27.
[2]	叶天鹏, 林祥, 李建华, 张轩凯, 许力文. 面向雾计算的个性化轻量级分布式网络入侵检测系统[J]. 网络与信息安全学报, 2023, 9(3): 28-37.
[3]	祖立军, 曹雅琳, 门小骅, 吕智慧, 叶家炜, 李泓一, 张亮. 基于隐私风险评估的脱敏算法自适应方法[J]. 网络与信息安全学报, 2023, 9(3): 49-59.
[4]	夏锐琪, 李曼曼, 陈少真. 基于机器学习的分组密码结构识别[J]. 网络与信息安全学报, 2023, 9(3): 79-89.
[5]	袁静怡, 李子川, 彭国军. EN-Bypass：针对邮件代发提醒机制的安全评估方法[J]. 网络与信息安全学报, 2023, 9(3): 90-101.
[6]	余锋, 林庆新, 林晖, 汪晓丁. 基于生成对抗网络的隐私增强联邦学习方案[J]. 网络与信息安全学报, 2023, 9(3): 113-122.
[7]	李晓萌, 郭玳豆, 卓训方, 姚恒, 秦川. 载体独立的抗屏摄信息膜叠加水印算法[J]. 网络与信息安全学报, 2023, 9(3): 135-149.
[8]	蔡召, 荆涛, 任爽. 以太坊钓鱼诈骗检测技术综述[J]. 网络与信息安全学报, 2023, 9(2): 21-32.
[9]	潘雁, 林伟, 祝跃飞. 渐进式的协议状态机主动推断方法[J]. 网络与信息安全学报, 2023, 9(2): 81-93.
[10]	杨盼, 康绯, 舒辉, 黄宇垚, 吕小少. 基于函数摘要的二进制程序污点分析优化方法[J]. 网络与信息安全学报, 2023, 9(2): 115-131.
[11]	肖天, 江智昊, 唐鹏, 黄征, 郭捷, 邱卫东. 基于深度强化学习的高性能导向性模糊测试方案[J]. 网络与信息安全学报, 2023, 9(2): 132-142.
[12]	袁承昊, 李勇, 任爽. 多关键词动态可搜索加密方案[J]. 网络与信息安全学报, 2023, 9(2): 143-153.
[13]	侯泽洲, 任炯炯, 陈少真. 基于神经网络区分器的SIMON-like算法参数安全性评估[J]. 网络与信息安全学报, 2023, 9(2): 154-163.
[14]	郭学镜, 方毅翔, 赵怡, 张天助, 曾文超, 王俊祥. 基于传统引导机制的深度鲁棒水印算法[J]. 网络与信息安全学报, 2023, 9(2): 175-183.
[15]	余北缘, 任珊瑶, 刘建伟. 区块链资产窃取攻击与防御技术综述[J]. 网络与信息安全学报, 2023, 9(1): 1-17.