空域频域相结合的唇型篡改检测方法

doi:10.11959/j.issn.2096-109x.2022075

网络与信息安全学报 ›› 2022, Vol. 8 ›› Issue (6): 146-155.doi: 10.11959/j.issn.2096-109x.2022075

空域频域相结合的唇型篡改检测方法

林佳滢¹^,², 周文柏¹^,², 张卫明¹^,², 俞能海¹^,²

¹ 中国科学院电磁空间信息重点实验室，安徽合肥 230027
² 中国科学技术大学网络空间安全学院，安徽合肥 230027

修回日期:2022-07-09 出版日期:2022-12-15 发布日期:2023-01-16
作者简介:林佳滢（1997- ），女，江西赣州人，中国科学技术大学硕士生，主要研究方向为人工智能安全、信息隐藏
周文柏（1992- ），男，安徽合肥人，中国科学技术大学特任副研究员，主要研究方向为信息隐藏、人工智能安全
张卫明（1976- ），男，河北定州人，中国科学技术大学教授、博士生导师，主要研究方向为信息隐藏、多媒体内容安全、人工智能安全
俞能海（1964- ），男，安徽无为人，中国科学技术大学教授、博士生导师，主要研究方向为多媒体信息检索、图像处理与视频通信、数字媒体内容安全
基金资助:
国家自然科学基金(U20B2047);国家自然科学基金(62072421);国家自然科学基金(62002334);国家自然科学基金(62102386);国家自然科学基金(62121002);中国科技大学探索基金项目(YD3480002001);中央高校基础研究基金(WK2100000011)

Lip forgery detection via spatial-frequency domain combination

Jiaying LIN¹^,², Wenbo ZHOU¹^,², Weiming ZHANG¹^,², Nenghai YU¹^,²

¹ Key Laboratory of Electromagnetic Space Information, Chinese Academy of Sciences, Hefei 230027, China
² School of Cyber Science, University of Science and Technology of China, Hefei 230027, China

Revised:2022-07-09 Online:2022-12-15 Published:2023-01-16
Supported by:
The National Natural Science Foundation of China(U20B2047);The National Natural Science Foundation of China(62072421);The National Natural Science Foundation of China(62002334);The National Natural Science Foundation of China(62102386);The National Natural Science Foundation of China(62121002);Exploration Fund Project of University of Science and Technology of China(YD3480002001);Fundamental Research Funds for the Central Universities(WK2100000011)

摘要/Abstract

摘要：

近年来，社交网络中的“换脸”视频层出不穷，对说话者进行唇型篡改是其中的视频代表之一，这给大众生活增添娱乐的同时，对于网络空间中的个人隐私、财产安全也带来了不小隐患。大多数唇型篡改检测方法在无损条件下取得了较好的表现，但广泛存在于社交媒体平台、人脸识别等场景中的压缩操作，在节约像素和时间冗余的同时，会对视频质量造成影响，破坏空域上像素与像素、帧与帧之间的连贯完整性，导致其检测性能的下降，从而引发对真实视频的错判情况。当空域信息无法提供足够有效的特征时，能够抵抗压缩干扰的频域信息就自然而然地成为重点研究对象。针对这一问题，通过分析频率信息在图像结构和梯度反馈上的优势，提出了空域频域相结合的唇型篡改检测方法，有效利用空域、频域信息的各自特点。对于空域上的唇型特征，设计了自适应提取网络和轻量级的注意力模块；对于频域上的频率特征，设计了不同分量的分离提取与融合模块。随后，通过对空域上的唇型特征和频域上的频率特征进行有侧重的融合，保留更多关键纹理信息。此外，在训练中设计细粒度约束，分开真假唇型特征类间距离的同时，拉近类内距离。实验结果表明，得益于频率信息，所提方法能有效改善压缩情况下的检测准确性，并具备一定的迁移性。另外，在对核心模块开展的消融实验中，相关结果验证了频率分量对于抗压缩的有效性，以及双重损失函数在训练中的约束作用。

关键词: 人脸伪造, 人脸伪造检测防御, 唇型篡改检测, 抗压缩, 深度学习

Abstract:

In recent years, numerous “face-swapping” videos have emerged in social networks, one of the representatives is the lip forgery with speakers.While making life more entertaining for the public, it poses a significant crisis for personal privacy and property security in cyberspace.Currently, under non-destructive conditions, most of the lip forgery detection methods achieve good performance.However, the compression operations are widely used in practice especially in social media platforms, face recognition and other scenarios.While saving pixel and time redundancy, the compression operations affect the video quality and destroy the coherent integrity of pixel-to-pixel and frame-to-frame in the spatial domain, and then the degradation of its detection performance and even misjudgment of the real video will be caused.When the information in the spatial domain cannot provide sufficiently effective features, the information in the frequency domain naturally becomes a priority research object because it can resist compression interference.Aiming at this problem, the advantages of frequency information in image structure and gradient feedback were analyzed.Then the lip forgery detection via spatial-frequency domain combination was proposed, which effectively utilized the corresponding characteristics of information in spatial and frequency domains.For lip features in the spatial domain, an adaptive extraction network and a light-weight attention module were designed.For frequency features in the frequency domain, separate extraction and fusion modules for different components were designed.Subsequently, by conducting a weighted fusion of lip features in spatial domain and frequency features in frequency domain, more texture information was preserved.In addition, fine-grained constraints were designed during the training to separate the inter-class distance of real and fake lip features while closing the intra-class distance.Experimental results show that, benefiting from the frequency information, the proposed method can enhance the detection accuracy under compression situation with certain transferability.On the other hand, in the ablation study conducted on the core modules, the results verify the effectiveness of the frequency component for anti-compression and the constraint of the dual loss function in training.

Key words: DeepFake forgery, DeepFake detection and defense, lip forgery detection, anti-compression, deep learning

中图分类号:

TP309.2

林佳滢, 周文柏, 张卫明, 俞能海. 空域频域相结合的唇型篡改检测方法[J]. 网络与信息安全学报, 2022, 8(6): 146-155.

Jiaying LIN, Wenbo ZHOU, Weiming ZHANG, Nenghai YU. Lip forgery detection via spatial-frequency domain combination[J]. Chinese Journal of Network and Information Security, 2022, 8(6): 146-155.

图/表 13

图1

图2

图3

图4

图5

图6

图7

图8

表1

表2

表3

表4

表5

参考文献 31

[1]	SUWAJANAKORN S , SEITZ S M , KEMELMACHER-SHLIZERMAN I , . Synthesizing Obama:learning lip sync from audio[J]. ACM Transactions on Graphics (TOG), 2017,36: 1-13.
[2]	SIAROHIN A , LATHUILIèRE S , TULYAKOV S ,et al. First order motion model for image animation[J]. ArXiv, 2019,abs/2003.00196.
[3]	YI R , YE Z , ZHANG J ,et al. Audio-driven talking face video generation with learning-based personalized head pose[J]. arXiv:2002.10137v2, 2020.
[4]	PRAJWAL K R , MUKHOPADHYAY R , NAMBOODIRI V P ,et al. A lip sync expert is all you need for speech to lip generation in the wild[C]// Proceedings of the 28th ACM International Conference on Multimedia. 2020: 484-492.
[5]	HALIASSOS A , VOUGIOUKAS K , PETRIDIS S ,et al. Lips don't lie:a generalisable and robust approach to face forgery detection[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 5037-5047.
[6]	FARHA Y A , GALL J . MS-TCN:multi-stage temporal convolutional network for action segmentation[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019: 3570-3579.
[7]	QIAN Y Y , YIN G J , SHENG L ,et al. Thinking in frequency:face forgery detection by mining frequency-aware clues[C]// Proceedings of Computer Vision – ECCV 2020. 2020: 86-103.
[8]	LI J M , XIE H T , LI J H ,et al. Frequency-aware discriminative feature learning supervised by single-center loss for face forgery detection[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 6454-6463.
[9]	CHEN S , YAO T P , CHEN Y ,et al. Local relation learning for face forgery detection[J]. arXiv:2105.02577, 2021.
[10]	REN S Q , HE K M , GIRSHICK R ,et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(6): 1137-1149.
[11]	HUANG G , LIU Z , VAN DER MAATEN L ,et al. Densely connected convolutional networks[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2261-2269.
[12]	SUN Y , WANG X G , TANG X O . Deep learning face representation from predicting 10,000 classes[C]// Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. 2014: 1891-1898.
[13]	WEN Y D , ZHANG K P , LI Z F ,et al. A discriminative feature learning approach for deep face recognition[C]// Proceedings of Computer Vision – ECCV 2016. 2016: 499-515.
[14]	R?SSLER A , COZZOLINO D , VERDOLIVA L ,et al. FaceForensics++:learning to detect manipulated facial images[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2019: 1-11.
[15]	LI Y Z , YANG X , SUN P ,et al. Celeb-DF:a large-scale challenging dataset for DeepFake forensics[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 3204-3213.
[16]	Faceswap. Faceswap github[EB].
[17]	THIES J , ZOLLH?FER M , STAMMINGER M ,et al. Face 2 face:real-time face capture and reenactment of RGB videos[J]. ArXiv, 2019,abs/2007.14808.
[18]	DeepFakes. Deepfakes github[EB].
[19]	THIES J , ZOLLH?FER M , NIE?NER M . Real-time expression transfer for facial reenactment[J]. ACM Transactions on Graphics, 2015,34(6): 1-14.
[20]	LI L Z , BAO J M , ZHANG T ,et al. Face X-ray for more general face forgery detection[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 5000-5009.
[21]	KINGMA D P , BA J . Adam:a method for stochastic optimization[J]. CoRR. 2015,abs/1412.6980.
[22]	FRIDRICH J J , KODOVSKY J , . Rich models for steganalysis of digital images[J]. IEEE Transactions on Information Forensics and Security, 2012,7(3): 868-882.
[23]	AFCHAR D , NOZICK V , YAMAGISHI J ,et al. MesoNet:a compact facial video forgery detection network[J]. 2018 IEEE International Workshop on Information Forensics and Security (WIFS), 2018: 1-7.
[24]	CHOLLET F , . Xception:deep learning with depthwise separable convolutions[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1800-1807.
[25]	COZZOLINO D , POGGI G , VERDOLIVA L . Recasting residual-based local descriptors as convolutional neural networks:an application to image forgery detection[C]// Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security. 2017: 159-164.
[26]	BAYAR B , STAMM M C . A deep learning approach to universal image manipulation detection using a new convolutional layer[C]// Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security. 2016: 5-10.
[27]	RAHMOUNI N , NOZICK V , YAMAGISHI J ,et al. Distinguishing computer graphics from natural images using convolution neural networks[J]. 2017 IEEE Workshop on Information Forensics and Security (WIFS), 2017: 1-6.
[28]	ZHOU P , HAN X T , MORARIU V I ,et al. Two-stream neural networks for tampered face detection[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017: 1831-1839.
[29]	NGUYEN H H , FANG F M , YAMAGISHI J ,et al. Multi-task learning for detecting and segmenting manipulated facial images and videos[C]// Proceedings of 2019 IEEE 10th International Conference on Biometrics Theory,Applications and Systems. 2019: 1-8.
[30]	MATERN F , RIESS C , STAMMINGER M . Exploiting visual artifacts to expose deepfakesand face manipulations[J]. 2019 IEEE Winter Applications of Computer Vision Workshops(WACVW), 2019: 83-92.
[31]	LI Y , LYU S . Exposing DeepFake videos by detecting face warping artifacts[J]. ArXiv, 2019,abs/1811.00656.

数据集	Steg.Features^[22]		Cozzolino et al^[23]		Rahmouni et al^[24]		Bayer ＆ Stamm^[25]		MesoNet^[26]		XceptionNet^[27]		本文方法
数据集	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
C23	70.97%	—	78.45%	—	79.08%	—	82.97%	—	83.10%	—	92.39%	94.86%	92.88%	95.07%
C40	55.98%	—	58.69%	—	61.18%	—	66.84%	—	70.47%	—	80.32%	81.76%	81.50%	82.81%

检测方法	ACC
检测方法	DeepFakes	Face2Face	FaceSwap	Neural Textures
Steg.Features^[22]	73.64%	73.72%	68.93%	63.33%
Cozzolino et al^[23]	85.45%	67.88%	73.79%	78.00%
Rahmouni et al^[24]	85.45%	64.23%	56.31%	60.07%
Bayer ＆ Stamm^[25]	84.55%	73.72%	82.52%	70.67%
MesoNet^[26]	87.27%	56.20%	61.17%	40.67%
XceptionNet^[27]	95.15%	83.48%	92.09%	77.89%
本文方法	95.36%	85.47%	92.18%	78.19%

检测方法	AUC
检测方法	FaceForensics++	Celeb-DF
Two-stream^[28]	70.10%	53.80%
Multi-task^[29]	76.30%	54.30%
VA-LogReg^[30]	78.00%	55.10%
FWA^[31]	80.10%	56.90%
MesoNet^[26]	84.70%	54.80%
XceptionNet^[27]	94.86%	65.50%
本文方法	95.07%	66.24%

数据集	XceptionNet		XceptionNet+低频		XceptionNet+低频+中频		XceptionNet+低频+中频+高频
数据集	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
C23	91.79%	94.08%	91.85%	94.10%	91.92%	94.27%	91.94%	94.29%
C40	80.77%	82.18%	80.89%	82.26%	81.06%	82.44%	81.10%	82.48%

数据集	Softmax Loss		Softmax Loss+Center Loss
数据集	ACC	AUC	ACC	AUC
C23	91.94%	94.29%	92.88%	95.07%
C40	81.10%	82.48%	81.50%	82.81%

空域频域相结合的唇型篡改检测方法

Lip forgery detection via spatial-frequency domain combination

在线阅读

pdf下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 31

相关文章 15

Metrics

推荐阅读 0

[1]	李晓萌, 郭玳豆, 卓训方, 姚恒, 秦川. 载体独立的抗屏摄信息膜叠加水印算法[J]. 网络与信息安全学报, 2023, 9(3): 135-149.
[2]	谢绒娜, 马铸鸿, 李宗俞, 田野. 基于卷积神经网络的加密流量分类方法[J]. 网络与信息安全学报, 2022, 8(6): 84-91.
[3]	章登勇, 文凰, 李峰, 曹鹏, 向凌云, 杨高波, 丁湘陵. 基于双分支网络的图像修复取证方法[J]. 网络与信息安全学报, 2022, 8(6): 110-122.
[4]	陈晋音, 吴长安, 郑海斌. 基于softmax激活变换的对抗防御方法[J]. 网络与信息安全学报, 2022, 8(2): 48-63.
[5]	邱宝琳, 易平. 基于多维特征图知识蒸馏的对抗样本防御方法[J]. 网络与信息安全学报, 2022, 8(2): 88-99.
[6]	李丽娟, 李曼, 毕红军, 周华春. 基于混合深度学习的多类型低速率DDoS攻击检测方法[J]. 网络与信息安全学报, 2022, 8(1): 73-85.
[7]	秦中元, 贺兆祥, 李涛, 陈立全. 基于图像重构的MNIST对抗样本防御算法[J]. 网络与信息安全学报, 2022, 8(1): 86-94.
[8]	邹德清, 李响, 黄敏桓, 宋翔, 李浩, 李伟明. 基于图结构源代码切片的智能化漏洞检测系统[J]. 网络与信息安全学报, 2021, 7(5): 113-122.
[9]	王正龙, 张保稳. 生成对抗网络研究综述[J]. 网络与信息安全学报, 2021, 7(4): 68-85.
[10]	李炳龙, 佟金龙, 张宇, 孙怡峰, 王清贤, 常朝稳. 基于TensorFlow的恶意代码片段自动取证检测算法[J]. 网络与信息安全学报, 2021, 7(4): 154-163.
[11]	谭清尹, 曾颖明, 韩叶, 刘一静, 刘哲理. 神经网络后门攻击研究[J]. 网络与信息安全学报, 2021, 7(3): 46-58.
[12]	杨路辉,白惠文,刘光杰,戴跃伟. 基于可分离卷积的轻量级恶意域名检测模型[J]. 网络与信息安全学报, 2020, 6(6): 112-120.
[13]	刘西蒙,谢乐辉,王耀鹏,李旭如. 深度学习中的对抗攻击与防御[J]. 网络与信息安全学报, 2020, 6(5): 36-53.
[14]	杜思佳,于海宁,张宏莉. 基于深度学习的文本分类研究进展[J]. 网络与信息安全学报, 2020, 6(4): 1-13.
[15]	翟明芳,张兴明,赵博. 基于深度学习的加密恶意流量检测研究[J]. 网络与信息安全学报, 2020, 6(3): 66-77.