基于块间光照不一致性的深度伪造检测算法

doi:10.11959/j.issn.2096-109x.2023011

网络与信息安全学报 ›› 2023, Vol. 9 ›› Issue (1): 167-177.doi: 10.11959/j.issn.2096-109x.2023011

基于块间光照不一致性的深度伪造检测算法

吴文轩, 周文柏, 张卫明, 俞能海

中国科学技术大学网络空间安全学院，安徽合肥 230026

修回日期:2022-05-19 出版日期:2023-02-25 发布日期:2023-02-01
作者简介:吴文轩（1997- ），男，浙江金华人，中国科学技术大学硕士生，主要研究方向为深度伪造检测
周文柏（1992- ），男，安徽合肥人，中国科学技术大学特任副研究员，主要研究方向为信息隐藏与人工智能安全
张卫明（1976- ），男，河北定州人，中国科学技术大学教授、博士生导师，主要研究方向为信息隐藏、多媒体内容安全、人工智能安全
俞能海（1964- ），男，安徽无为人，中国科学技术大学教授、博士生导师，主要研究方向为多媒体信息检索、图像处理与视频通信、数字媒体内容安全
基金资助:
国家自然科学基金(62002334);国家自然科学基金(U20B2047);国家自然科学基金(62121002);安徽省自然科学基金(2008085QF296);统筹推进世界一流大学和一流学科建设专项资金(YD3480002001);中央高校基本科研业务费专项资金(WK2100000011)

Deepfake detection method based on patch-wise lighting inconsistency

Wenxuan WU, Wenbo ZHOU, Weiming ZHANG, Nenghai YU

School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230026, China

Revised:2022-05-19 Online:2023-02-25 Published:2023-02-01
Supported by:
The National Natural Science Foundation of China(62002334);The National Natural Science Foundation of China(U20B2047);The National Natural Science Foundation of China(62121002);The Natural Science Foundation of Anhui Province(2008085QF296);Special Funds for Promoting the Construction of World-class Universities and First-class Disciplines(YD3480002001);Fundamental Research Funds for the Central Universities(WK2100000011)

摘要/Abstract

摘要：

深度伪造技术的迅速发展和广泛传播引起了社会的广泛关注，但深度伪造技术的恶意应用也给社会带来了潜在威胁。因此，如何检测出此类深度伪造内容成为热门研究课题。以往的多数深度伪造检测算法着重于捕捉像素级别的细微伪造痕迹。目前的深度伪造算法大多忽略了伪造前后的光照信息，导致原始人脸与伪造人脸之间存在一定的光照不一致性，这为使用光照不一致性来检测深度伪造提供了可能。从引入光照不一致性信息和为特定任务设计网络结构模块两个角度设计了对应的算法。针对光照的引入，通过设计对应的通道融合算法，将更多的光照不一致信息提供给网络特征提取层，从而衍生出新的网络结构。为了保证该网络结构的可移植性，将特征通道融合的过程置于网络提取信息之前，从而使所提算法能够完整移植至常见的深度伪造检测网络。针对网络结构的设计，从网络结构和损失函数设计两个角度出发，提出了基于块间相似性的光照不一致性深度伪造检测算法。对于网络结构，基于伪造图像篡改区域和背景区域不一致的特性，在网络特征层中对提取特征进行分块，通过对比块间余弦相似度得到特征层相似矩阵，使网络拟合重心更偏向于光照不一致性。在此基础上，基于特征层相似性对比方案，通过将输入图像与该图像的未篡改图像进行块间真伪性对比，为这一任务设计了独立的真实数据参考及损失函数。实验结果表明，与基线算法相比，所提算法对于深度伪造检测的准确性有明显提升。

关键词: 深度伪造检测, 光照不一致性, 色彩恒常性, 块间相似度对比

Abstract:

The rapid development and widespread dissemination of deepfake techniques has caused increased concern.The malicious application of deepfake techniques also poses a potential threat to the society.Therefore, how to detect deepfake content has become a popular research topic.Most of the previous deepfake detection algorithms focused on capturing subtle forgery traces at pixel level and have achieved some results.However, most of the deepfake algorithms ignore the lighting information before and after generation, resulting in some lighting inconsistency between the original face and the forged face, which provided the possibility of using lighting inconsistency to detect deepfake.A corresponding algorithm was designed from two perspectives: introducing lighting inconsistency information and designing a network structure module for a specific task.For the introduction of lighting task, a new network structure was derived by designing the corresponding channel fusion method to provide more lighting inconsistency information to the network feature extraction layer.In order to ensure the portability of the network structure, the process of feature channel fusion was placed before the network extraction information, so that the proposed method can be fully planted to common deepfake detection networks.For the design of the network structure, a deepfake detection method was proposed for lighting inconsistency based on patch-similarity from two perspectives: network structure and loss function design.For the network structure, based on the characteristic of inconsistency between the forged image tampering region and the background region, the extracted features were chunked in the network feature layer and the feature layer similarity matrix was obtained by comparing the patch-wise cosine similarity to make the network focus more on the lighting inconsistency.On this basis, based on the feature layer similarity matching scheme, an independent ground truth and loss function was designed for this task in a targeted manner by comparing the input image with the untampered image of this image for patch-wise authenticity.It is demonstrated experimentally that the accuracy of the proposed method is significantly improved for deepfake detection compared with the baseline method.

Key words: Deepfake detection, lighting inconsistency, color constancy, patch-similarity comparison

中图分类号:

TP393

吴文轩, 周文柏, 张卫明, 俞能海. 基于块间光照不一致性的深度伪造检测算法[J]. 网络与信息安全学报, 2023, 9(1): 167-177.

Wenxuan WU, Wenbo ZHOU, Weiming ZHANG, Nenghai YU. Deepfake detection method based on patch-wise lighting inconsistency[J]. Chinese Journal of Network and Information Security, 2023, 9(1): 167-177.

图/表 10

图1

图2

图3

图4

图5

表1

本文算法与XceptionNet算法的准确率对比Table 1 Comparison of accuracy between the proposed algorithm and XceptionNet algorithm"

数据集	准确率
数据集	XceptionNet算法	本文算法
DF_c23	98.69	$99 . 01$
NT_c23	93.67	$94 . 79$
F2F_c23	98.7	$98 . 71$
FS_c23	$99 . 08$	98.94
Total_c23	92.83	$96 . 03$
DF_c40	95.21	$97 . 83$
NT_c40	78.02	$81 . 3$
F2F_c40	90.66	$90 . 92$
FS_c40	91.99	$93 . 24$
Total_c40	84.75	$86 . 19$

表1

表2

本文算法与隐写分析算法的准确率对比Table 2 Comparison of accuracy between the proposed algorithm and steganlysis algorithm"

数据集	准确率
数据集	隐写特征算法	本文算法
DF_c23	83.61	$99 . 01$
NT_c23	73.92	$94 . 79$
F2F_c23	83.75	$98 . 71$
FS_c23	79.67	$98 . 94$
Total_c23	76.74	$96 . 03$
DF_c40	76.6	$97 . 83$
NT_c40	68.9	$81 . 3$
F2F_c40	76.72	$90 . 92$
FS_c40	71.26	$93 . 24$
Total_c40	70.72	$86 . 19$

表2

表3

本文算法与ResNet-50算法的准确率对比Table 3 Comparison of accuracy between the proposed algorithm and ResNet-50 algorithm"

数据集	准确率
数据集	ResNet-50	本文算法
DF_c23	97.55	$97 . 68$
NT_c23	91.33	$92 . 16$
F2F_c23	$98 . 25$	98.14
FS_c23	$97 . 94$	97.53
Total_c23	92.06	$95 . 41$
DF_c40	96.42	$97 . 26$
NT_c40	76.58	$78 . 65$
F2F_c40	88.36	$88 . 92$
FS_c40	88.69	$90 . 84$
Total_c40	84.11	$85 . 47$

表3

表4

其他数据集上本文算法与XceptionNet算法的准确率对比Table 4 Comparison of accuracy between the proposed algorithm and XceptionNet algorithm on other datasets"

数据集	准确率
数据集	XceptionNet算法	本文算法
DFDC	78.26	$80 . 47$
Celeb-DF	98.24	$98 . 96$

表4

表5

消融实验下本文算法与XceptionNet算法的准确率对比Table 5 Comparison of accuracy between the proposed algorithm and XceptionNet algorithm in ablation experiment"

数据集	准确率
数据集	XceptionNet	Light	Sim	Light+Sim
DF_c23	98.69	98.92	98.7	$99 . 01$
NT_c23	93.67	94.37	94.1	$94 . 79$
F2F_c23	98.7	98.34	98.68	$98 . 71$
FS_c23	$99 . 08$	97.75	99.02	98.94
Total_c23	92.83	95.74	94.86	$96 . 03$
DF_c40	95.21	97.41	96.07	$97 . 83$
NT_c40	78.02	80.37	78.14	$81 . 3$
F2F_c40	90.66	90.68	90.47	$90 . 92$
FS_c40	91.99	92.93	92.26	$93 . 24$
Total_c40	84.75	85.52	85.37	$86 . 19$

表5

参考文献 39

[1]	GOODFELLOW I , POUGET-ABADIE J , MIRZA M ,et al. Generative adversarial networks[J]. Communications of the ACM, 2020,63(11): 139-144.
[2]	HUH M , LIU A , OWENS A ,et al. Fighting fake news:Image splice detection via learned self-consistency[C]// Proceedings of Computer Vision – ECCV 2018. 2018: 106-124.
[3]	KWON P , YOU J , NAM G ,et al. KoDF:A large-scale Korean deepfake detection dataset[J]. arXiv preprintarXiv:2103.10094, 2021.
[4]	LEE S . Detecting handcrafted facial image manipulations and GAN-generated facial images using shallow-fakefacenet[J]. Applied Soft Computing, 2021,105:107256.
[5]	NGUYEN T T , NGUYEN C M , NGUYEN D T ,et al. Deep learning for deepfakes creation and detection:a survey[J]. arXiv:1909.11573, 2019.
[6]	SUN K , LIU H , YE Q X ,et al. Domain general face forgery detection by learning to weight[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021,35(3): 2638-2646.
[7]	YANG X , LI Y Z , LYU S W . Exposing deep fakes using inconsistent head poses[C]// Proceedings of ICASSP 2019-2019 IEEE International Conference on Acoustics,Speech and Signal Processing. 2019: 8261-8265.
[8]	ZHOU P , HAN X T , MORARIU V I ,et al. Two-stream neural networks for tampered face detection[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017: 1831-1839.
[9]	AFCHAR D , NOZICK V , YAMAGISHI J ,et al. MesoNet:A compact facial video forgery detection network[C]// Proceedings of 2018 IEEE International Workshop on Information Forensics and Security. 2018: 1-7.
[10]	NGUYEN H H , YAMAGISHI J , ECHIZEN I . Capsule-forensics:Using capsule networks to detect forged images and videos[C]// Proceedings of ICASSP 2019-2019 IEEE International Conference on Acoustics,Speech and Signal Processing. 2019: 2307-2311.
[11]	LI Y Z , LYU S W . Exposing DeepFake videos by detecting face warping artifacts[J]. arXiv preprint arXiv:1811.00656, 2018.
[12]	R?SSLER A , COZZOLINO D , VERDOLIVA L ,et al. FaceForensics++:manipulatedacial images[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2019: 1-11.
[13]	ZHAO H , ZHOU W , CHEN D ,et al. Multiattentional deepfake detection[J]. arXiv preprint arXiv:2103.02406, 2021.
[14]	LIU Z Z , QI X J , TORR P H S . Global texture enhancement for fake face detection in the wild[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 8057-8066.
[15]	SUN X W , WU B T , CHEN W . Identifying invariant texture violation for robust deepfake detection[J]. arXiv preprint arXiv:2012.10580, 2020.
[16]	CHUGH K , GUPTA P , DHALL A ,et al. Not made for each other audio visual dissonance-based deepfake detection and localization[C]// Proceedings of MM'20:Proceedings of the 28th ACM International Conference on Multimedia. 2020: 439-447.
[17]	ZHANG X , KARAMAN S , CHANG S F . Detecting and simulating artifacts in GAN fake images[C]// Proceedings of 2019 IEEE International Workshop on Information Forensics and Security. 2019: 1-6.
[18]	DOLHANSKY B , HOWES R , PFLAUM B ,et al. The DeepFake detection challenge (DFDC) preview dataset[EB].
[19]	THIES J , ZOLLH?FER M STAMMINGER M ,et al. Face2Face:real-time face capture and reenactment of RGB videos[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016: 2387-2395.
[20]	DeepFakes[EB].
[21]	FaceSwap[EB].
[22]	LI L Z , BAO J M , YANG H ,et al. FaceShifter:Towards high fidelity and occlusion aware face swapping[EB].
	LI L Z , BAO J M , YANG H ,et al. FaceShifter:Towards high fidelity and occlusion aware face swapping[J]. arXiv preprint arXiv:1912.13457, 2019.
[23]	KOUJAN M R , DOUKAS M C , ROUSSOS A ,et al. Head2Head:Video-based neural head synthesis[C]// Proceedings of 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition. 2020: 16-23.
[24]	ZHU H , FU C , WU Q ,et al. AOT:appearance optimal transport based identity swapping for forgery detection[C]// NeurIPS,IEEE, 2020.
	ZHU H , FU C Y , WU Q Y ,et al. AOT:appearance optimal transport based identity swapping for forgery detection[C]// Proceedings of NeurIPS. 2020.
[25]	MASI I , KILLEKAR A , MASCARENHAS R M ,et al. Two branch recurrent network for isolating deepfakes in videos[C]// Proceedings of Computer Vision – ECCV 2020, 2020: 667-684.
[26]	WU X , XIE Z , GAO Y T ,et al. SSTNet:detecting manipulated faces through spatial,steganalysis and temporal features[C]// Proceedings of ICASSP 2020 - 2020 IEEE International Conference on Acoustics,Speech and Signal Processing. 2020: 2952-2956.
[27]	SHRUTI AL , HANY F , YUMING G ,et al. Protecting world leaders against deep fakes[C]// Proceedings of CVPR Workshops. 2019.
[28]	LI L Z , BAO J M , ZHANG T ,et al. Face X-ray for more general face forgery detection[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 5000-5009.
[29]	LIU H G , LI X D , ZHOU W B ,et al. Spatial-phase shallow learning:Rethinking face forgery detection in frequency domain[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021: 772-781.
[30]	BRAINARD D H , WANDELL B A . Analysis of the retinex theory of color vision[J]. Journal of the Optical Society of America A,Optics and Image Science, 1986,3(10): 1651-1661.
[31]	BUCHSBAUM G . A spatial processor model for object colour perception[J]. Journal of the Franklin Institute, 1980,310(1): 1-26.
[32]	VAN DE WEIJER J , GEVERS T , GIJSENIJ A . Edge-based color constancy[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2007,16(9): 2207-2214.
[33]	FINLAYSON G D , TREZZI E . Shades of gray and colour constancy[C]// Proceedings of Color and Imaging Conference. 2004.
[34]	JOZE H R V , DREW M S , FINLAYSON G D ,et al. The role of bright pixels in illumination estimation[C]// Proceedings of Color and Imaging Conference. 2012.
[35]	YANG K F , GAO S B , LI Y J . Efficient illuminant estimation for color constancy using grey pixels[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015: 2254-2263.
[36]	QIAN Y L , K?M?R?INEN J K , NIKKANEN J ,et al. On finding gray pixels[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019: 8054-8062.
[37]	GATTA C , RIZZI A , MARINI D . ACE:an automatic color equalization algorithm[C]// Proceedings of Conference on Colour in Graphics. 2002.
[38]	LAND E H , MCCANN J J . Lightness and retinex theory[J]. Journal of the Optical Society of America, 1971,61(1): 1-11.
[39]	FRIDRICH J , KODOVSKY J . Rich models for steganalysis of digital images[J]. IEEE Transactions on Information Forensics and Security, 2012,7(3): 868-882.

基于块间光照不一致性的深度伪造检测算法

Deepfake detection method based on patch-wise lighting inconsistency

在线阅读

pdf下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 39

相关文章 15

Metrics

推荐阅读 0

[1]	陈先意, 顾军, 颜凯, 江栋, 许林峰, 付章杰. 针对车牌识别系统的双重对抗攻击[J]. 网络与信息安全学报, 2023, 9(3): 16-27.
[2]	叶天鹏, 林祥, 李建华, 张轩凯, 许力文. 面向雾计算的个性化轻量级分布式网络入侵检测系统[J]. 网络与信息安全学报, 2023, 9(3): 28-37.
[3]	祖立军, 曹雅琳, 门小骅, 吕智慧, 叶家炜, 李泓一, 张亮. 基于隐私风险评估的脱敏算法自适应方法[J]. 网络与信息安全学报, 2023, 9(3): 49-59.
[4]	夏锐琪, 李曼曼, 陈少真. 基于机器学习的分组密码结构识别[J]. 网络与信息安全学报, 2023, 9(3): 79-89.
[5]	袁静怡, 李子川, 彭国军. EN-Bypass：针对邮件代发提醒机制的安全评估方法[J]. 网络与信息安全学报, 2023, 9(3): 90-101.
[6]	余锋, 林庆新, 林晖, 汪晓丁. 基于生成对抗网络的隐私增强联邦学习方案[J]. 网络与信息安全学报, 2023, 9(3): 113-122.
[7]	朱春陶, 尹承禧, 张博林, 殷琪林, 卢伟. 基于多域时序特征挖掘的伪造人脸检测方法[J]. 网络与信息安全学报, 2023, 9(3): 123-134.
[8]	李晓萌, 郭玳豆, 卓训方, 姚恒, 秦川. 载体独立的抗屏摄信息膜叠加水印算法[J]. 网络与信息安全学报, 2023, 9(3): 135-149.
[9]	蔡召, 荆涛, 任爽. 以太坊钓鱼诈骗检测技术综述[J]. 网络与信息安全学报, 2023, 9(2): 21-32.
[10]	潘雁, 林伟, 祝跃飞. 渐进式的协议状态机主动推断方法[J]. 网络与信息安全学报, 2023, 9(2): 81-93.
[11]	杨盼, 康绯, 舒辉, 黄宇垚, 吕小少. 基于函数摘要的二进制程序污点分析优化方法[J]. 网络与信息安全学报, 2023, 9(2): 115-131.
[12]	肖天, 江智昊, 唐鹏, 黄征, 郭捷, 邱卫东. 基于深度强化学习的高性能导向性模糊测试方案[J]. 网络与信息安全学报, 2023, 9(2): 132-142.
[13]	袁承昊, 李勇, 任爽. 多关键词动态可搜索加密方案[J]. 网络与信息安全学报, 2023, 9(2): 143-153.
[14]	侯泽洲, 任炯炯, 陈少真. 基于神经网络区分器的SIMON-like算法参数安全性评估[J]. 网络与信息安全学报, 2023, 9(2): 154-163.
[15]	郭学镜, 方毅翔, 赵怡, 张天助, 曾文超, 王俊祥. 基于传统引导机制的深度鲁棒水印算法[J]. 网络与信息安全学报, 2023, 9(2): 175-183.