基于多域时序特征挖掘的伪造人脸检测方法

doi:10.11959/j.issn.2096-109x.2023044

Abstract

Abstract:

Financial technology has greatly facilitated people’s daily life with the continuous development of computer technology in the financial services industry.However, digital finance is accompanied by security problems that can be extremely harmful.Face biometrics, as an important part of identity information, is widely used in payment systems, account registration, and many other aspects of the financial industry.The emergence of face forgery technology constantly impacts the digital financial security system, posing a threat to national asset security and social stability.To address the security problems caused by fake faces, a forgery face detection method based on multi-domain temporal features mining was proposed.The tampering features were distinguished and enhanced based on the consistency of statistical feature data distribution and temporal action trend in the temporal features of videos existing in the spatial domain and frequency domain.Temporal information was mined in the spatial domain using an improved LSTM, while in the frequency domain, temporal information existing in different frequency bands of the spectrum was mined using 3D convolution layers.The information was then fused with the tampering features extracted from the backbone network, thus effectively distinguishing forged faces from real ones.The effectiveness of the proposed method was demonstrated on mainstream datasets.

Key words: face authentication, Deepfake detection, temporal features, multi-domain features

CLC Number:

TP393

Chuntao ZHU, Chengxi YIN, Bolin ZHANG, Qilin YIN, Wei LU. Forgery face detection method based on multi-domain temporal features mining[J]. Chinese Journal of Network and Information Security, 2023, 9(3): 123-134.

Figures/Tables 12

References 43

[1]	DeepFakes GitHub non official project based on original deepfakes thread[EB].
[2]	THIES J , ZOLLH?FER M , STAMMINGER M , et al . Face2Face:real-time face capture and reenactment of RGB videos[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016: 2387-2395.
[3]	FaceSwap GitHub non official project based on original deepfakes thread[EB].
[4]	GOODFELLOW I J , POUGET-ABADIE J , MIRZA M ,et al. Generative adversarial networks[J]. arXiv Preprint arXiv:1406.2661. 2014.
[5]	XU W J , KESHMIRI S , WANG G H . Adversarially approximated autoencoder for image generation and manipulation[J]. IEEE Transactions on Multimedia, 2019,21(9): 2387-2396.
[6]	王鹏程, 郑海斌, 邹健飞 ,等. 面向商用活体检测平台的鲁棒性评估[J]. 网络与信息安全学报, 2022,8(1): 180-189.
	WANG P C , ZHENG H B , ZOU J F ,et al. Robustness evaluation of commercial liveness detection platform[J]. Chinese Journal of Network and Information Security, 2022,8(1): 180-189.
[7]	MATERN F , RIESS C , STAMMINGER M . Exploiting visual artifacts to expose deepfakes and face manipulations[C]// Proceedings of 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW). 2019: 83-92.
[8]	PENG B , WANG W , DONG J ,et al. Optimized 3D lighting environment estimation for image forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2017,12(2): 479-494.
[9]	NIRKIN Y , WOLF L , KELLER Y ,et al. DeepFake detection based on discrepancies between faces and their context[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022,44(10): 6111-6121.
[10]	LIU H G , LI X D , ZHOU W B ,et al. Spatial-phase shallow learning:rethinking face forgery detection in frequency domain[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 772-781.
[11]	林佳滢, 周文柏, 张卫明 ,等. 空域频域相结合的唇型篡改检测方法[J]. 网络与信息安全学报, 2022,8(6): 146-155.
	LIN J Y , ZHOU W B , ZHANG W M ,et al. Lip forgery detection via spatial-frequency domain combination[J]. Chinese Journal of Network and Information Security, 2022,8(6): 146-155.
[12]	LI Y Z , CHANG M C , LYU S W . Exposing AI created fake videos by detecting eye blinking[C]// Proceedings of 2018 IEEE International Workshop on Information Forensics and Security (WIFS). 2019: 1-7.
[13]	SUN Z K , HAN Y J , HUA Z Y ,et al. Improving the efficiency and robustness of deepfakes detection through precise geometric features[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 3608-3617.
[14]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[15]	WANG Y B , WU H X , ZHANG J J ,et al. PredRNN:a recurrent neural network for spatiotemporal predictive learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023,45(2): 2208-2225.
[16]	THIES J , ZOLLH?FER M , NIE?NER M . Deferred neural rendering[J]. ACM Transactions on Graphics, 2019,38(4): 1-12.
[17]	LIU M , BREUEL T , KAUTZ J . Unsupervised image-to-image translation networks[J]. arXiv Preprint arXiv:1703.00848, 2017.
[18]	KARRAS T , AILA T , LAINE S ,et al. Progressive growing of gans for improved quality,stability,and variation[J]. arXiv Preprint arXiv:1710.10196, 2017.
[19]	KARRAS T , LAINE S , AILA T M . A style-based generator architecture for generative adversarial networks[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 4396-4405.
[20]	HUANG X , BELONGIE S . Arbitrary style transfer in real-time with adaptive instance normalization[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). 2017: 1510-1519.
[21]	LI L Z , BAO J M , YANG H ,et al. Advancing high fidelity identity swapping for forgery detection[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 5073-5082.
[22]	PAN X Y , ZHANG X , LYU S W . Exposing image splicing with inconsistent local noise variances[C]// Proceedings of 2012 IEEE International Conference on Computational Photography (ICCP). 2012: 1-10.
[23]	FERRARA P , BIANCHI T , DE ROSA A ,et al. Image forgery localization via fine-grained analysis of CFA artifacts[J]. IEEE Transactions on Information Forensics and Security, 2012,7(5): 1566-1577.
[24]	FRIDRICH J , KODOVSKY J . Rich models for steganalysis of digital images[J]. IEEE Transactions on Information Forensics and Security, 2012,7(3): 868-882.
[25]	YANG X , LI Y Z , LYU S W . Exposing deep fakes using inconsistent head poses[C]// Proceedings of ICASSP 2019 - 2019 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP). 2019: 8261-8265.
[26]	AGARWAL S , FARID H , GU Y ,et al. Protecting world leaders against deep fakes[C]// Proceedings of CVPR Workshops. 2019,1:38.
[27]	CHOLLET F . Xception:deep learning with depthwise separable convolutions[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017: 1800-1807.
[28]	AFCHAR D , NOZICK V , YAMAGISHI J ,et al. MesoNet:a compact facial video forgery detection network[C]// Proceedings of 2018 IEEE International Workshop on Information Forensics and Security (WIFS). 2019: 1-7.
[29]	NGUYEN H H , YAMAGISHI J , ECHIZEN I . Capsule-forensics:using capsule networks to detect forged images and videos[C]// Proceedings of ICASSP 2019 - 2019 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP). 2019: 2307-2311.
[30]	LI L Z , BAO J M , ZHANG T ,et al. Face X-ray for more general face forgery detection[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 5000-5009.
[31]	DURALL R , KEUPER M , PFREUNDT F J ,et al. Unmasking DeepFakes with simple features[J]. arXiv Preprint arXiv:1911.00686, 2019.
[32]	QIAN Y Y , YIN G J , SHENG L ,et al. Thinking in frequency:face forgery detection by mining frequency-aware clues[C]// Proceedings of European Conference on Computer Vision. 2020: 86-103.
[33]	GüERA D , DELP E J . Deepfake video detection using recurrent neural networks[C]// Proceedings of 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). 2019: 1-6.
[34]	HU J , LIAO X , WANG W ,et al. Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022,32(3): 1089-1102.
[35]	LU W , LIU L Y , ZHANG B L ,et al. Detection of deepfake videos using long-distance attention[J]. IEEE Transactions on Neural networks and Learning Systems, 2023,(99): 1-14.
[36]	SHI X J , CHEN Z R , WANG H ,et al. Convolutional LSTM Network:a machine learning approach for precipitation nowcasting[C]// Proceedings of the 28th International Conference on Neural Information Processing System. 2015: 802-810.
[37]	JADERBERG M , SIMONYAN K , ZISSERMAN A ,et al. Spatial transformer networks[J]. arXiv Preprint arXiv:1506.02025, 2015.
[38]	WANG Q L , WU B G , ZHU P F ,et al. ECA-net:efficient channel attention for deep convolutional neural networks[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 11531-11539.
[39]	R?SSLER A , COZZOLINO D , VERDOLIVA L ,et al. FaceForensicss:learning to detect manipulated facial images[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2020: 1-11.
[40]	LI Y Z , YANG X , SUN P ,et al. Celeb-DF:a large-scale challenging dataset for DeepFake forensics[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 3204-3213.
[41]	DOLHANSKY B , HOWES R , PFLAUM B ,et al. The deepfake detection challenge (DFDC) preview dataset[J]. arXiv Preprint arXiv:1910.08854, 2019.
[42]	WANG C R , DENG W H . Representative forgery mining for fake face detection[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 14918-14927.
[43]	ZHAO H Q , WEI T Y , ZHOU W B ,et al. Multi-attentional deepfake detection[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 2185-2194.

Metrics

Recommended 0

No Suggested Reading articles found!

检测方法	Deepfakes		Face2Face		FaceSwap		NeuralTextures
检测方法	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
Xception	0.967 9	0.976 4	0.917 9	0.931 7	0.960 7	0.970 5	0.817 9	0.823 6
Multi-attention			0.946 4	0.955 4			0.817 9	0.838 9
F³Net	0.971 4	0.978 5			0.971 4	0.974 8		0.846 4
Two-stream	0.946 4	0.980 0	0.864 8	0.940 0	0.852 7	0.940 0	0.800 5
RNN	0.817 0	0.860 0	0.708 5	0.770 0	0.668 3	0.760 0	0.691 0	0.780 0
本文方法
注：加粗指标为最优实验结果，下划线指标为次优实验结果，余同。

检测方法	Deepfakes		Face2Face		FaceSwap		NeuralTextures
检测方法	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
Xception	0.945 8	0.986 8	0.893 5	0.960 4	0.924 2	0.978 0	0.743 5	0.812 4
Multi-attention			0.897 6	0.959 6	0.942 5
RFM	0.952 4	0.992 1	0.894 0	0.956 1	0.928 9	0.980 0	0.748 3	0.822 9
F³Net
本文方法	0.954 1	0.989 1				0.978 6	0.769 3	0.839 8

检测方法	ACC
检测方法	帧级		视频级
Xception	0.978 0		0.992 3	0.990 1
Multi-attention		0.995 2	0.990 3	0.988 7
RFM	0.980 9		—	—
F³Net		0.993 9
本文方法	0.984 5	0.997 4

检测方法	ACC
检测方法	帧级		视频级
Xception	0.889 1	0.960 1	0.898 4	0.897 8
Multi-attention
RFM	0.900 1	0.967 3	—	—
F³Net				0.924 5
本文方法	0.895 0	0.958 1	0.890 6

检测方法（HQ）	AUC
检测方法（HQ）	FaceForensics++	Celeb-DF
Xception	0.985 1	0.673 1
Multi-attention	0.994 3	0.647 0
RFM	0.988 2	0.697 3
F³Net		0.635 2
本文方法	0.992 6

Forgery face detection method based on multi-domain temporal features mining

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 43

Related Articles 2

Metrics

Recommended 0

检测方法（LQ）	AUC
检测方法（LQ）	FaceForensics++	Celeb-DF
Xception	0.904 2	0.617 1
Multi-attention	0.924 6	0.595 0
RFM	0.910 0	0.635 9
F³Net		0.620 4
本文方法	0.927 1

帧数			ACC
帧数	Deepfakes	Face2Face	FaceSwap	NeuralTextures
4	0.948 6	0.901 6		0.760 4
6			0.943 2
8	0.952 6	0.903 2	0.942 0	0.765 6

对照组	Deepfakes		Face2Face		FaceSwap		NeuralTextures
对照组	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
主干	0.943 4	0.983 9	0.893 3	0.954 9	0.932 8	0.973 7	0.750 5	0.826 0
主干+空域	0.950 4	0.990 0	0.904 3		0.941 7		0.759 0	0.836 6
主干+频域	0.945 1		0.896 9	0.962 7	0.940 2	0.978 9	0.750 0	0.829 1
主干+多域		0.989 1		0.961 5		0.978 6

[1]	Wenxuan WU, Wenbo ZHOU, Weiming ZHANG, Nenghai YU. Deepfake detection method based on patch-wise lighting inconsistency [J]. Chinese Journal of Network and Information Security, 2023, 9(1): 167-177.
[2]	Jiaying LIN, Wenbo ZHOU, Weiming ZHANG, Nenghai YU. Lip forgery detection via spatial-frequency domain combination [J]. Chinese Journal of Network and Information Security, 2022, 8(6): 146-155.