机器学习安全及隐私保护研究进展

doi:10.11959/j.issn.2096-109x.2018067

摘要/Abstract

摘要：

机器学习作为实现人工智能的一种重要方法，在数据挖掘、计算机视觉、自然语言处理等领域得到广泛应用。随着机器学习应用的普及发展，其安全与隐私问题受到越来越多的关注。首先结合机器学习的一般过程，对敌手模型进行了描述。然后总结了机器学习常见的安全威胁，如投毒攻击、对抗攻击、询问攻击等，以及应对的防御方法，如正则化、对抗训练、防御精馏等。接着对机器学习常见的隐私威胁，如训练数据窃取、逆向攻击、成员推理攻击等进行了总结，并给出了相应的隐私保护技术，如同态加密、差分隐私。最后给出了亟待解决的问题和发展方向。

关键词: 机器学习, 安全威胁, 防御技术, 隐私保护

Abstract:

As an important method to implement artificial intelligence,machine learning technology is widely used in data mining,computer vision,natural language processing and other fields.With the development of machine learning,it brings amount of security and privacy issues which are getting more and more attention.Firstly,the adversary model was described according to machine learning.Secondly,the common security threats in machine learning was summarized,such as poisoning attacks,adversarial attacks,oracle attacks,and major defense methods such as regularization,adversarial training,and defense distillation.Then,privacy issues such were summarized as stealing training data,reverse attacks,and membership tests,as well as privacy protection technologies such as differential privacy and homomorphic encryption.Finally,the urgent problems and development direction were given in this field.

Key words: machine learning, security threats, defense technology, privacy

中图分类号:

TP309.2

宋蕾, 马春光, 段广晗. 机器学习安全及隐私保护研究进展[J]. 网络与信息安全学报, 2018, 4(8): 1-11.

Lei SONG, Chunguang MA, Guanghan DUAN. Machine learning security and privacy:a survey[J]. Chinese Journal of Network and Information Security, 2018, 4(8): 1-11.

图/表 9

图1

表1

图2

图3

表2

表3

图4

图5

表4

参考文献 62

[1]	GHORBEL A , GHORBEL M , JMAIEL M . Privacy in cloud computing environments:a survey and research challenges[J]. Journal of Supercomputing, 2017,73(6): 2763-2800.
[2]	SILVER D , HUANG A , MADDISON C J ,et al. Mastering the game of go with deep neural networks and tree search[J]. Nature, 2016,529(7587): 484-489.
[3]	BARRENO M , NELSON B , SEARS R ,et al. Can machine learning be secure?[C]// ACM Symposium on Information,Computer and Communications Security. 2006: 16-25.
[4]	KEARNS M , LI M . Learning in the presence of malicious errors[J]. SIAM Journal on Computing, 1993,22(4): 807-837.
[5]	BIGGIO B , NELSON B , LASKOV P . Support vector machines under adversarial label noise[J]. Journal of Machine Learning Research, 2011,20(3): 97-112.
[6]	BIGGIO B , NELSON B , LASKOV P . Poisoning attacks against support vector machines[C]// International Coference on International Conference on Machine Learning. 2012: 1467-1474.
[7]	MEI S , ZHU X . Using machine teaching to identify optimal training-set attacks on machine learners[C]// AAAI. 2015: 2871-2877.
[8]	BIGGIO B , DIDACI L , FUMERA G ,et al. Poisoning attacks to compromise face templates[C]// International Conference on Biometrics. 2013: 1-7.
[9]	KLOFT M , LASKOV P . Security analysis of online anomaly detection[J]. Journal of Machine Learning Research, 2010,13(1): 3681-3724.
[10]	C SZEGEDY , W ZAREMBA , I SUTSKEVER , ,et al. Intriguing properties of neural networks[C]// 2014 International Conference on Learning Representations.Computational and Biological Learning Society. 2014.
[11]	PAPERNOT N , MC D P , SINHA A ,et al. Towards the science of security and privacy in machine learning[J]. arXiv preprint arXiv:1611.03814, 2016.
[12]	GOODFELLOW I J , SHLENS J , SZEGEDY C . Explaining and harnessing adversarial examples[C]// International Conference on Learning Representations. 2015.
[13]	KURAKIN A , GOODFELLOW I , BENGIO S . Adversarial machine learning at scale[J]. arXiv preprint arXiv:1611.01236, 2017.
[14]	DONG Y P , LIAO F Z , PANG T Y ,et al. Boosting adversarial attacks with momentum[J]. arXiv preprint arXiv:1710.06081, 2017.
[15]	MIYATO T , MAEDA S , KOYAMA M ,et al. Virtual adversarial training:a regularization method for supervised and semi-supervised learning[J]. arXiv preprint 1704.03976, 2017.
[16]	MOOSAVI-DEZFOOLI S , FAWZI A , FROSSARD P . DeepFool:a simple and accurate method to fool deep neural networks[C]// IEEE Conference on Computer Vision and Pattern Recognition. 2016: 2574-2582.
[17]	PAPERNOT N , MCDANIEL P , JHA S ,et al. The limitations of deep learning in adversarial settings[C]// IEEE European Symposium on Security and Privacy. 2016: 372-387.
[18]	SU J , VARGAS D V , KOUICHI S . One pixel attack for fooling deep neural networks[J]. arXiv preprint arXiv:1710.08864, 2017.
[19]	LOWD D , MEEK C . Adversarial learning[C]// The eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining. 2005: 641-647.
[20]	MOOSAVI-DEZFOOLI S M , FAWZI A , FAWZI O ,et al. Universal adversarial perturbations[C]// IEEE Conference on Computer Vision and Pattern Recognition. 2017.
[21]	PAPERNOT N , MCDANIEL P , GOODFELLOW I ,et al. Practical black-box attacks against machine learning[C]// 2017 ACM on Asia Conf on Computer and Communications Security. 2017: 506-519.
[22]	PAPERNOT N , MCDANIEL P , GOODFELLOW I . Transferability in machine learning:from phenomena to black-box attacks using adversarial samples[J]. arXiv preprint arXiv:1605.07277, 2016.
[23]	GU S X , RIGAZIO L . Towards deep neural network architectures robust to adversarial examples[J]. arXiv preprint arXiv:1412.5068, 2014.
[24]	LYU C , HUANG K , LIANG H N . A unified gradient regularization family for adversarial examples[C]// IEEE International Conference on Data Mining. 2016: 301-309.
[25]	ZHAO Q Y , GRIFFIN L D . Suppressing the unusual:towards robust cnns using symmetric activation functions[J]. arXiv preprint arXiv:1603.05145, 2016.
[26]	ROZSA A , GUNTHER M , BOULT T E . Towards robust deep neural networks with BANG[J]. arXiv preprint arXiv:1612.00138, 2016.
[27]	GOODFELLOW I J , SHLENS J , SZEGEDY C . Explaining and harnessing adversarial examples[C]// International Conference on Learning Representations. Computational and Biological Learning Society, 2015.
[28]	HUANG R , XU B , SCHUURMANS D ,et al. Learning with a strong adversary[J]. arXiv preprint arXiv:1511.03034, 2015.
[29]	TRAMèR F , KURAKIN A , PAPERNOT N ,et al. ensemble adversarial training:attacks and defenses[J]. arXiv preprint arXiv:1705.07204, 2017.
[30]	PAPERNOT N , MCDANIEL P , WU X ,et al. Distillation as a defense to adversarial perturbations against deep neural networks[C]// IEEE Symp on Security and Privacy. 2016: 582-597.
[31]	HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network[J]. arXiv preprint arXiv:1503.02531, 2015.
[32]	PAPERNOT N , MCDANIEL P . Extending defensive distillation[J]. arXiv preprint arXiv:1705.05264, 2017.
[33]	BULòS R , BIGGIO B , PILLAI I ,et al. Randomized prediction games for adversarial machine learning[J]. IEEE transactions on neural networks and learning systems, 2017,28(11): 2466-2478.
[34]	HARDT M , MEGIDDO N , PAPADIMITRIOU C ,et al. Strategic classification[C]// 2016 ACM conference on innovations in theoretical computer science. 2016: 111-122.
[35]	BRüCKNER M , KANZOW C , SCHEFFER T . Static prediction games for adversarial learning problems[J]. Journal of Machine Learning Research, 2012,13(Sep): 2617-2654.
[36]	METZEN J H , GENEWEIN T , FISCHER V ,et al. On detecting adversarial perturbations[J]. arXiv preprint arXiv:1702.04267, 2017.
[37]	LU JIAJUN , ISSARANON T , FORSYTH D . SAFETYNET:Detecting and rejecting adversarial examples robustly[J]. arXiv preprint arXiv:1704.00103, 2017.
[38]	HITAJ B , ATENIESE G ， PEREZ-CRUZ F . Deep models under the GAN:information leakage from collaborative deep learning[C]// ACM Sigsac Conference. 2017: 603-618.
[39]	FREDRIKSON M , LANTZ E , JHA S ,et al. Privacy in pharmacogenetics:an end-to-end case study of personalized warfarin dosing[C]// The 23rd Usenix Security Symposium. 2014: 17-32.
[40]	FREDRIKSON M , JHA S , RISTENPART T . Model inversion attacks that exploit confidence information and basic countermeasures[C]// The 22nd ACM Sigsac Conference on Computer and Communications Security. 2015: 1322-1333.
[41]	ATENIESE G , MANCINI L V , SPOGNARDI A ,et al. Hacking smart machines with smarter ones:How to extract meaningful data from machine learning classifiers[J]. International Journal of Security and Networks, 2015,10(3): 137-150.
[42]	SHOKRI R , STRONATI M , SONG C ,et al. Membership inference attacks against machine learning models[J]. arXiv preprint arXiv:1610.05820, 2016.
[43]	TRAMER F , ZHANG F , JUELS A ,et al. Stealing machine learning models via prediction apis[J]. arXiv preprint arXiv:1609.02943, 2016.
[44]	GENTRY , CRAIG , Fully homomorphic encryption using ideal lattices[J]. Stoc, 2009,9(4): 169-178.
[45]	DOWLIN N , RAN G B , LAINE K ,et al. CryptoNets:applying neural networks to encrypted data with high throughput and accuracy[C]// Radio and Wireless Symposium. 2016: 76-78.
[46]	HESAMIFARD E , TAKABI H , GHASEMI M ,et al. Privacy-preserving machine learning in cloud[C]// The 2017 on Cloud Computing Security Workshop. 2017: 39-43.
[47]	BARYALAI M , JANG-JACCARD J , LIU D . Towards privacy-preserving classification in neural networks[C]// IEEE Privacy,Security and Trust. 2017: 392-399.
[48]	XIE P , BILENKO M , FINLEY T ,et al. Crypto-nets:neural networks over encrypted data[J]. Computer Science, 2014.
[49]	STONE M H . The generalized weierstrass approximation theorem[J]. Mathematics Magazine, 1948,21(4): 167-184.
[50]	ZHANG Q , YANG L , CHEN Z . Privacy preserving deep computation model on cloud for big data feature learning[J]. IEEE Transactions on Computers, 2016,65(5): 1351-1362.
[51]	DWORK C , MCSHERRY F , NISSIM K ,et al. Calibrating noise to sensitivity in private data analysis[C]// The Third conference on Theory of Cryptography. 2006: 265-284.
[52]	ABADI M , CHU A , GOODFELLOW I ,et al. Deep learning with differential privacy[C]// 2016 ACM Sigsac Conference on Computer and Communications Security. 2016: 308-318.
[53]	PAPERNOT N , ABADI M , ERLINGSSON U ,et al. Semi- supervised knowledge transfer for deep learning from private training data[J]. arXiv preprint arXiv:1610.05755, 2016.
[54]	BEAULIEUJONES B K , WU Z S , WILLIAMS C J ,et al. Privacy-preserving generative deep neural networks support clinical data sharing[J]. bioRxiv, 2017.
[55]	郭鹏, 钟尚平, 陈开志 ,等. 差分隐私 GAN 梯度裁剪阈值的自适应选取方法[J]. 网络与信息安全学报, 2018,4(5): 10-20.
	GUO P , ZHONG S P , CHEN K Z ,et al. Adaptive selection method of differential privacy[J]. Chinese Journal of Network and Information Security, 2018,4(5): 10-20.
[56]	SHOKRI R , SHMATIKOV V . Privacy-preserving deep learning[C]// The 22nd ACM SIGSAC Conference on Computer and Communications Security. 2015: 1310-1321.
[57]	LIU M , JIANG H , CHEN J ,et al. A collaborative privacy-preserving deep learning system in distributed mobile environment[C]// International Conference on Computational Science and Computational Intelligence. 2017: 192-197.
[58]	LE T P , AONO Y , HAYASHI T ,et al. Privacy-preserving deep learning via additively homomorphic encryption[J]. IEEE Transactions on Information Forensics ＆ Security, 2018,13(5): 1333-1345.
[59]	MCMAHAN B , RAMAGE D . Federated learning:collaborative machine learning without centralized training data[J]. Google Research Blog, 2017.
[60]	BONAWITZ K , IVANOV V , KREUTER B ,et al. Practical secure aggregation for privacy-preserving machine learning[C]// 2017 ACM Sigsac Conference on Computer and Communications Security. 2017: 1175-1191.
[61]	MCMAHAN H B , MOORE E , RAMAGE D ,et al. Federated learning of deep networks using model averaging[J]. arXiv preprint arXiv:1502.01710v5, 2016.
[62]	OSSIA S A , SHAMSABADI A S , TAHERI A ,et al. A hybrid deep learning architecture for privacy-preserving mobile analytics[J]. arXiv preprint arXiv:1703.02952, 2017.

阶段	敌手策略	敌手目标	敌手能力	敌手知识
训练阶段	投毒攻击	完整性/可用性	修改训练数据	有限知识
预测阶段	对抗攻击询问攻击	完整性/可用性完整性/可用性	制作对抗样本访问目标模型	黑盒/白盒黑盒

攻击类型	访问模型输出	获取模型内部结构	针对目标攻击	非针对目标攻击
			L-BFGS^[10]、	BIM^[13]、
白盒攻击	√	√	FGSM^[12]、	Deepfool^[16]
			JSMA^[17]
黑盒攻击	√			文献[20-22]

阶段	敌手策略	敌手目标	敌手能力	敌手知识
训练阶段	窃取训练数据	机密性	获得训练数据	有限知识
预测阶段	逆向攻击	机密性	提取模型/训练数据信息	黑盒/白盒
	成员推理攻击	机密性	访问目标模型	黑盒

隐私威胁	训练阶段	预测阶段	访问模型输出	访问模型内部结构	提取模型结构信息	提取训练数据信息
文献[38]	√		√	√		√
文献[41]		√	√	√		√
文献[43]		√	√		√
文献[39,40,42]		√	√			√