生成对抗网络研究综述

doi:10.11959/j.issn.2096-109x.2021080

摘要/Abstract

摘要：

首先介绍了生成对抗网络基本理论、应用场景和研究现状，并列举了其亟待改进的问题。围绕针对提升模型训练效率、提升生成样本质量和降低模式崩溃现象发生可能性3类问题的解决，从模型结构和训练过程两大改进方向和7个细分维度，对近年来生成对抗网络的主要研究工作、改进机理和特点进行了归纳和总结，并结合3方面对其未来的研究方向进行了探讨。

关键词: 生成对抗网络, 生成模型, 深度学习, 模式崩溃, 分布距离度量, 神经网络鲁棒性

Abstract:

Firstly, the basic theory, application scenarios and current state of research of GAN (generative adversarial network) were introduced, and the problems need to be improved were listed.Then, recent research, improvement mechanism and model features in 2 categories and 7 subcategories revolved around 3 points (improving model training efficiency, improving the quality of generated samples, and reducing the possibility of model collapse) were generalized and summarized.Finally, 3 future research directions were discussed.

Key words: generative adversarial network, generative model, deep learning, mode collapse, distribution similarity measurement, robustness of artificial neural network

中图分类号:

TP183

王正龙, 张保稳. 生成对抗网络研究综述[J]. 网络与信息安全学报, 2021, 7(4): 68-85.

Zhenglong WANG, Baowen ZHANG. Survey of generative adversarial network[J]. Chinese Journal of Network and Information Security, 2021, 7(4): 68-85.

图/表 24

图1

表1

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

图12

图13

图14

图15

图16

图17

图18

图19

图20

图21

图22

图23

参考文献 106

[1]	SMOLENSKY P . Information processing in dynamical systems:Foundations of harmony theory[R]. 1986.
[2]	HINTON G , OSINDERO S , TEH Y W . A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006 18(7): 1527-1554.
[3]	SALAKHUTDINOV R , HINTON G . Deep boltzmann machines[C]// Artificial Intelligence and Statistics. 2009: 448-455.
[4]	KINGMA D P , WELLING M . Auto-encoding variational bayes[J]. arXiv preprint arXiv:1312.6114, 2013.
[5]	GOODFELLOW I , POUGET-ABADIE J ,, MIRZA M , et al . Generative adversarial nets[C]// Advances in Neural Information Processing Systems. 2014: 2672-2680.
[6]	KOO S , . Automatic colorization with deep convolutional generative adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017: 212-217.
[7]	RADFORD A , METZ L , CHINTALA S . Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv preprint arXiv:1511.06434, 2015.
[8]	ZHU J Y , PARK T , ISOLA P ,et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the IEEE International Conference on Computer Vision. 2017: 2223-2232.
[9]	BAO J , CHEN D , WEN F ,et al. CVAE-GAN:fine-grained image generation through asymmetric training[C]// Proceedings of the IEEE International Conference on Computer Vision. 2017: 2745-2754.
[10]	SCHLEGL T , SEEB?CK P ,, WALDSTEIN S M , et al . Unsupervised anomaly detection with generative adversarial networks to guide marker discovery[C]// International Conference on Information Processing in Medical Imaging. 2017: 146-157.
[11]	ARJOVSKY M , CHINTALA S , BOTTOU L . WassersteinGAN[J]. arXiv preprint arXiv:1701.07875, 2017.
[12]	WOLTERINK J M , LEINER T , ISGUM I . Blood vessel geometry synthesis using generative adversarial networks[J]. arXiv preprint arXiv:1804.04381, 2018.
[13]	刘西蒙, 谢乐辉, 王耀鹏 ,等. 深度学习中的对抗攻击与防御[J]. 网络与信息安全学报, 2020(5): 36-53.
	LIU X M , XIE L H , WANGY P ,et al. Adversarial attacks and defenses in deep learning[J]. Chinese Journal of Network and Information Security, 2020,6(5): 36-53.
[14]	张煜, 吕锡香, 邹宇聪 ,等. 基于生成对抗网络的文本序列数据集脱敏[J]. 网络与信息安全学报, 2020(4): 109-119.
	ZHANG Y , LYU X X , ZOU Y C ,et al. Differentially private sequence generative adversarial networks for data privacy masking[J]. Chinese Journal of Network and Information Security, 2020,6(4): 109-119.
[15]	BEAULIEU-JONES B K , WU Z S , WILLIAMS C ,et al. Privacy-preserving generative deep neural networks support clinical data sharing[J]. Circulation:Cardiovascular Quality and Outcomes, 2019,12(7): e005122.
[16]	郭鹏, 钟尚平, 陈开志 ,等. 差分隐私 GAN 梯度裁剪阈值的自适应选取方法[J]. 网络与信息安全学报, 2018,4(5): 10-20.
	GUO P , ZHONG S P , CHEN K J ,et al. Adaptive selection method of differential privacy GAN gradient clipping thresholds[J]. Chinese Journal of Network and Information Security, 2018,4(5): 10-20.
[17]	王旭东, 卫红权, 高超 ,等. 身份保持约束下的人脸图像补全[J]. 网络与信息安全学报, 2018,4(8): 71-76.
	WANG X D , WEI H Q , GAO C ,et al. Identity preserving face completion with generative adversarial networks[J]. Chinese Journal of Network and Information Security, 2018,4(8): 71-76.
[18]	MIYATO T , KATAOKA T , KOYAMA M ,et al. Spectral normalization for generative adversarial networks[J]. arXiv preprint arXiv:1802.05957, 2018.
[19]	GOODFELLOW I . NIPS 2016 tutorial:generative adversarial networks[J]. arXiv preprint arXiv:1701.00160, 2016.
[20]	MIRZA M , OSINDERO S . Conditional generative adversarial nets[J]. arXiv preprint arXiv:1411.1784, 2014.
[21]	JAISWAL A , ABDALMAGEED W , WU Y ,et al. Bidirectional conditional generative adversarial networks[C]// Asian Conference on Computer Vision. 2018: 216-232.
[22]	PERARNAU G , VAN DE WEIJER J , RADUCANU B ,et al. Invertible conditional GANs for image editing[J]. arXiv preprint ar Xiv:1611.06355, 2016.
[23]	CHEN X , DUAN Y , HOUTHOOFT R ,et al. Infogan:Interpretable representation learning by information maximizing generative adversarial nets[C]// Advances in Neural Information Processing Systems. 2016: 2172-2180.
[24]	GURUMURTHY S , KIRAN SARVADEVABHATLA R , VENKATESH BABU R . DeLiGAN:Generative adversarial networks for diverse and limited data[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 166-174.
[25]	MISHRA D , JAYENDRAN A , SRIVASTAVA V ,et al. Mode matching in GANs through latent space learning and inversion[J]. arXiv preprint arXiv:1811.03692, 2018.
[26]	KWAK H , ZHANG B T . Ways of conditioning generative adversarial networks[J]. arXiv preprint arXiv:1611.01455, 2016.
[27]	ODENA A . Semi-supervised learning with generative adversarial networks[J]. arXiv preprint arXiv:1606.01583, 2016.
[28]	ODENA A , OLAH C , SHLENS J . Conditional image synthesis with auxiliary classifier GANs[C]// International Conference on Machine Learning. 2017: 2642-2651.
[29]	TOLSTIKHIN I O , GELLY S , BOUSQUET O ,et al. AdaGAN:boosting generative models[C]// Advances in Neural Information Processing Systems. 2017: 5424-5433.
[30]	GHOSH A , KULHARIA V , NAMBOODIRI V P ,et al. Multi-agent diverse generative adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 8513-8521.
[31]	HOANG Q , NGUYEN T D , LE T ,et al. MGAN:training generative adversarial nets with multiple generators[C]// International Conference on Learning Representations. 2018.
[32]	GHOSH A , KULHARIA V , NAMBOODIRI V . Message passing multi-agent GANs[J]. arXiv preprint arXiv:1612.01294, 2016.
[33]	LIN Z , KHETAN A , FANTI G ,et al. PacGAN:The power of two samples in generative adversarial networks[J]. IEEE Journal on Selected Areas in Information Theory, 2020,1(1): 324-335.
[34]	DURUGKAR I , GEMP I , MAHADEVAN S . Generative multi-adversarial networks[J]. arXiv preprint arXiv:1611.01673, 2016.
[35]	MORDIDO G , YANG H , MEINEL C . Dropout-GAN:learning from a dynamic ensemble of discriminators[J]. arXiv preprint arXiv:1807.11346, 2018.
[36]	NGUYEN T , LE T , VU H ,et al. Dual discriminator generative adversarial nets[C]// Advances in Neural Information Processing Systems. 2017: 2670-2680.
[37]	NEYSHABUR B , BHOJANAPALLI S , CHAKRABARTI A . Stabilizing GAN training with multiple random projections[J]. arXiv preprint arXiv:1705.07831, 2017.
[38]	ZHAO J , MATHIEU M , LECUN Y . Energy-based generative adversarial network[J]. arXiv preprint arXiv:1609.03126, 2016.
[39]	BERTHELOT D , SCHUMM T , METZ L . BeGAN:boundary equilibrium generative adversarial networks[J]. arXiv preprint arXiv:1703.10717, 2017.
[40]	IM D J , KIM C D , JIANG H ,et al. Generating images with recurrent adversarial networks[J]. arXiv preprint arXiv:1602.05110, 2016.
[41]	ZHANG H , XU T , LI H ,et al. StackGAN:text to photo-realistic image synthesis with stacked generative adversarial networks[C]// Proceedings of the IEEE International Conference on Computer Vision. 2017: 5907-5915.
[42]	KARRAS T , AILA T , LAINE S ,et al. Progressive growing of gans for improved quality,stability,and variation[J]. arXiv preprint arXiv:1710.10196, 2017.
[43]	LI C , XU T , ZHU J ,et al. Triple generative adversarial nets[J]. Advances in Neural Information Processing Systems, 2017,30: 4088-4098.
[44]	LEE M , SEOK J . Controllable generative adversarial network[J]. Ieee Access, 2019,7: 28158-28169.
[45]	CHAVDAROVA T , FLEURET F . SGAN:An alternative training of generative adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 9407-9415.
[46]	KIM Y , KIM M , KIM G . Memorization precedes generation:Learning unsupervised gans with memory networks[J]. arXiv preprint arXiv:1803.01500, 2018.
[47]	JAISWAL A , ABDALMAGEED W , WU Y ,et al. CapsuleGAN:Generative adversarial capsule network[C]// Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[48]	LARSEN A B L , S?NDERBY S K ,, LAROCHELLE H , et al . Autoencoding beyond pixels using a learned similarity metric[C]// International Conference on Machine Learning. 2016: 1558-1566.
[49]	ZHONG G , GAO W , LIU Y ,et al. Generative adversarial networks with decoder-encoder output noises[J]. Neural Networks, 2020.
[50]	MAKHZANI A , SHLENS J , JAITLY N ,et al. Adversarial autoencoders[J]. arXiv preprint arXiv:1511.05644, 2015.
[51]	DONAHUE J , KR?HENBüHL P ,, DARRELL T . Adversarial feature learning[J]. arXiv preprint arXiv:1605.09782, 2016.
[52]	DUMOULIN V , BELGHAZI I , POOLE B ,et al. Adversarially learned inference[J]. arXiv preprint arXiv:1606.00704, 2016.
[53]	MáTTYUS G , URTASUN R . Matching adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 8024-8032.
[54]	ZHANG H , GOODFELLOW I , METAXAS D ,et al. Self-attention generative adversarial networks[C]// International Conference on Machine Learning. 2019: 7354-7363.
[55]	WANG X , ZHANG R , SUN Y ,et al. KdGAN:Knowledge distillation with generative adversarial networks[C]// Advances in Neural Information Processing Systems. 2018: 775-786.
[56]	WANG J , YU L , ZHANG W ,et al. IrGAN:a minimax game for unifying generative and discriminative information retrieval models[C]// Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017: 515-524.
[57]	DENTON E L , CHINTALA S , FERGUS R . Deep generative image models using a laplacian pyramid of adversarial networks[C]// Advances in Neural Information Processing Systems. 2015: 1486-1494.
[58]	LLOYD S , WEEDBROOK C . Quantum generative adversarial learning[J]. Physical Review Letters, 2018,121(4).
[59]	SAATCI Y , WILSON A G . Bayesian GAN[C]// Advances in Neural Information Processing Systems. 2017: 3622-3631.
[60]	NOWOZIN S , CSEKE B , TOMIOKA R . f-GAN:training generative neural samplers using variational divergence minimization[C]// Advances in Neural Information Processing Systems. 2016: 271-279.
[61]	MAO X , LI Q , XIE H ,et al. Least squares generative adversarial networks[C]// Proceedings of the IEEE International Conference on Computer Vision. 2017: 2794-2802.
[62]	GULRAJANI I , AHMED F , ARJOVSKY M ,et al. Improved training of WassersteinGANs[C]// Advances in Neural Information Processing Systems. 2017: 5767-5777.
[63]	PETZKA H , FISCHER A , LUKOVNICOV D . On the regularization of WassersteinGANs[J]. arXiv preprint arXiv:1709.08894, 2017.
[64]	ADLER J , LUNZ S . Banach WassersteinGAN[C]// Advances in Neural Information Processing Systems. 2018: 6754-6763.
[65]	WEI X , GONG B , LIU Z ,et al. Improving the improved training of WassersteinGANs:a consistency term and its dual effect[J]. arXiv preprint arXiv:1803.01541, 2018.
[66]	GUO X , HONG J , LIN T ,et al. Relaxed wasserstein with applications to GANs[J]. arXiv preprint arXiv:1705.07164, 2017.
[67]	BELLEMARE M G , DANIHELKA I , DABNEY W ,et al. The cramer distance as a solution to biased wasserstein gradients[J]. arXiv preprint arXiv:1705.10743, 2017.
[68]	WU J , HUANG Z , THOMA J ,et al. Wasserstein divergence for GANs[C]// Proceedings of the European Conference on Computer Vision (ECCV). 2018: 653-668.
[69]	MROUEH Y , SERCU T , GOEL V . McGAN:Mean and covariance feature matching GAN[J]. arXiv preprint arXiv:1702.08398, 2017.
[70]	LI Y , SWERSKY K , ZEMEL R . Generative moment matching networks[C]// International Conference on Machine Learning. 2015: 1718-1727.
[71]	LI C L , CHANG W C , CHENG Y ,et al. MmdGAN:towards deeper understanding of moment matching network[C]// Advances in Neural Information Processing Systems. 2017: 2203-2213
[72]	MROUEH Y , SERCU T . Fisher GAN[C]// Advances in Neural Information Processing Systems. 2017: 2513-2523.
[73]	SALIMANS T , GOODFELLOW I , ZAREMBA W ,et al. Improved techniques for training GANS[C]// Advances in Neural Information Processing Systems. 2016: 2234-2242.
[74]	ARORA S , GE R , LIANG Y ,et al. Generalization and equilibrium in generative adversarial nets (GANs)[J]. arXiv preprint arXiv:1703.00573, 2017.
[75]	SALIMANS T , ZHANG H , RADFORD A ,et al. Improving GANs using optimal transport[J]. arXiv preprint arXiv:1803.05573, 2018.
[76]	PARK N , ANAND A , MONIZ J R A ,et al. MMGAN:manifold-Matching generative adversarial networks[C]// 2018 24th International Conference on Pattern Recognition (ICPR). 2018: 1343-1348.
[77]	WANG R , CULLY A , CHANG H J ,et al. MaGAN:margin adaptation for generative adversarial networks[J]. arXiv preprint arXiv:1704.03817, 2017.
[78]	LIN M . SoftmaxGAN[J]. arXiv preprint arXiv:1704.06191, 2017.
[79]	JOLICOEUR-MARTINEAU A . The relativistic discriminator:a key element missing from standard GAN[J]. arXiv preprint ar Xiv:1807.00734, 2018.
[80]	MüLLER A . Integral probability metrics and their generating classes of functions[J]. Advances in Applied Probability, 1997: 429-443.
[81]	EVANS L C . Partial differential equations and monge-kantorovich mass transfer[J]. Current Developments in Mathematics, 1997,1997(1): 65-126.
[82]	BORJI A . Pros and cons of GAN evaluation measures[J]. Computer Vision and Image Understanding, 2019,179: 41-65.
[83]	SZEGEDY C , VANHOUCKE V , IOFFE S ,et al. Rethinking the inception architecture for computer vision[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 2818-2826.
[84]	CHE T , LI Y , JACOB A P ,et al. Mode regularized generative adversarial networks[J]. arXiv preprint arXiv:1612.02136, 2016.
[85]	ZHOU Z , CAI H , RONG S ,et al. Activation maximization generative adversarial nets[J]. arXiv preprint arXiv:1703.02000, 2017.
[86]	BARRATT S , SHARMA R . A note on the inception score[J]. arXiv preprint arXiv:1801.01973, 2018.
[87]	HEUSEL M , RAMSAUER H , UNTERTHINER T ,et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium[C]// Advances in Neural Information Processing Systems. 2017: 6626-6637.
[88]	LOPEZ-PAZ D , OQUAB M . Revisiting classifier two-sample tests[J]. arXiv preprint arXiv:1610.06545, 2016.
[89]	SHMELKOV K , SCHMID C , ALAHARI K . How good is my GAN[C]// Proceedings of the European Conference on Computer Vision (ECCV). 2018: 213-229.
[90]	ZHANG Z , SONG Y , QI H . Decoupled learning for conditional adversarial networks[C]// 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). 2018: 700-708.
[91]	QI G J , ZHANG L , HU H ,et al. Global versus localized generative adversarial nets[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 1517-1525.
[92]	LEI N , GUO Y , AN D ,et al. Mode collapse and regularity of optimal transportation maps[J]. arXiv preprint arXiv:1902.02934, 2019.
[93]	LIU K , TANG W , ZHOU F ,et al. Spectral regularization for combating mode collapse in GANs[C]// Proceedings of the IEEE International Conference on Computer Vision. 2019: 6382-6390.
[94]	GUAN S , LOEW M , KO H . Data separability for neural network classifiers and the development of a separability index[J]. arXiv preprint arXiv:2005.13120, 2020.
[95]	MO S , CHO M , SHIN J . Freeze discriminator:a simple baseline for fine-tuning GANs[J]. arXiv preprint arXiv:2002.10964, 2020.
[96]	ROBB E , CHU W S , KUMAR A ,et al. Few-shot adaptation of generative adversarial networks[J]. arXiv preprint arXiv:2010.11943, 2020.
[97]	ZHANG R , CHE T , GHAHRAMANI Z ,et al. MetaGAN:an adversarial approach to few-shot learning[C]// Advances in Neural Information Processing Systems. 2018: 2365-2374.
[98]	NGUYEN K , TODOROVIC S . A self-supervised GAN for unsupervised few-shot object recognition[J]. arXiv preprint arXiv:2008.06982, 2020.
[99]	KARRAS T , AITTALA M , HELLSTEN J ,et al. Training generative adversarial networks with limited data[J]. Advances in Neural Information Processing Systems, 2020,33.
[100]	KANEKO T , USHIKU Y , HARADA T . Label-noise robust generative adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 2467-2476.
[101]	KANEKO T , HARADA T . Noise robust generative adversarial networks[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8404-8414.
[102]	SZEGEDY C , ZAREMBA W , SUTSKEVER I ,et al. Intriguing properties of neural networks[C]// 2nd International Conference on Learning Representations,ICLR 2014. 2014.
[103]	GOODFELLOW I J , SHLENS J , SZEGEDY C . Explaining and harnessing adversarial examples[J]. Stat,2015, 1050:20.
[104]	BAI T , LUO J , ZHAO J . Recent advances in understanding adversarial robustness of deep neural networks[J]. arXiv preprint arXiv:2011.01539, 2020.
[105]	BASTANI O , IOANNOU Y , LAMPROPOULOS L ,et al. Measuring neural net robustness with constraints[C]// Advances in Neural Information Processing Systems. 2016: 2613-2621.
[106]	WANG S , PEI K , WHITEHOUSE J ,et al. Efficient formal safety analysis of neural networks[C]// Advances in Neural Information Processing Systems. 2018: 6367-6377.

改进维度	改进方法	衍生模型
	基于隐变量的改进	cGAN^[20]、BiCoGAN^[21]、IcGAN^[22]、InfoGAN^[23]
输入	基于隐空间的改进	DeLiGAN^[24]、NEMGAN^[25]
	其他输入改进方法	FCGAN^[26]
输出	多分类输出	SGAN^[27]、AC-GAN^[28]
	特征输出	InfoGAN^[23]
	集成方法	AdaGAN^[29]
生成器	样本模式区分	MADGAN^[30]、MGAN^[31]
	其他改进方法	MPMGAN^[32]
	集成方法	PacGAN^[33]、GMAN^[34]、DropoutGAN^[35]
判别器	样本模式区分	D2GAN^[36]、StabilizingGAN^[37]
	其他改进方法	EBGAN^[38]、BEGAN^[39]
	多阶段模型	GRAN^[40]、StackGAN^[41]、ProgressGAN^[42]
多模块组合	辅助模块模型	TripleGAN^[43]、ControlGAN^[44]
	其他改进方法	SGAN^[45]、MemoryGAN^[46]
	神经网络结构替换	DCGAN^[7]、CapsuleGAN^[47]
模型交叉	复合生成模型	VAEGAN^[48]、DEGAN^[49]、AAE^[50]、BiGAN^[51]、ALi^[52]、MatAN^[53]
	其他领域思想	SAGAN^[54]、KDGAN^[55]、IRGAN^[56]、LapGAN^[57]、QuGAN^[58]、BayesianGAN^[59]
	基于f-散度的方法	f-GAN^[60]、LSGAN^[61]、EBGAN^[38]
分布距离度量	基于IPM的方法	WGAN^[11]、WGAN-GP^[62]、WGAN-LP^[63]、BWGAN^[64]、CT-GAN^[65]、RWGAN^[66]、CramerGAN^[67]、SN-GAN^[18]、WGAN-div^[68]、McGAN^[69]、GMMN^[70]、MMDGAN^[71]、FisherGAN^[72]
	其他方法	IGAN^[73]、MIX+GAN^[74]、OT-GAN^[75]、MMGAN^[76]
梯度计算过程		MAGAN^[77]、LSGAN^[61]、SoftmaxGAN^[78]、RGAN^[79]