基于同态加密的高效安全联邦学习聚合框架

doi:10.11959/j.issn.1000-436x.2023015

摘要/Abstract

摘要：

为了解决联邦学习数据安全以及加密后通信开销大等问题，提出了一种基于同态加密的高效安全联邦聚合框架。在联邦学习过程中，用户数据的隐私安全问题亟须解决，然而在训练过程中采用加密方案带来的计算和通信开销又会影响训练效率。在既要保护数据安全又要保证训练效率的情况下，首先，采用Top-K梯度选择方法对模型梯度进行筛选，减少了需要上传的梯度数量，提出适合多边缘节点的候选量化协议和安全候选索引合并算法，进一步降低通信开销、加速同态加密计算。其次，由于神经网络每层模型参数具有高斯分布的特性，对选择的模型梯度进行裁剪量化，并采用梯度无符号量化协议以加速同态加密计算。最后，实验结果表明，在联邦学习的场景下，所提框架既保证了数据隐私安全，又具有较高的准确率和高效的性能。

关键词: 联邦学习, 同态加密, 隐私保护, 量化协议

Abstract:

In order to solve the problems of data security and communication overhead in federated learning, an efficient and secure federated aggregation framework based on homomorphic encryption was proposed.In the process of federated learning, the privacy and security issues of user data need to be solved urgently.However, the computational cost and communication overhead caused by the encryption scheme would affect the training efficiency.Firstly, in the case of protecting data security and ensuring training efficiency, the Top-K gradient selection method was used to screen model gradients, reducing the number of gradients that need to be uploaded.A candidate quantization protocol suitable for multi-edge terminals and a secure candidate index merging algorithm were proposed to further reduce communication overhead and accelerate homomorphic encryption calculations.Secondly, since model parameters of each layer of neural networks had characteristics of the Gaussian distribution, the selected model gradients were clipped and quantized, and the gradient unsigned quantization protocol was adopted to speed up the homomorphic encryption calculation.Finally, the experimental results show that in the federated learning scenario, the proposed framework can protect data privacy, and has high accuracy and efficient performance.

Key words: federated learning, homomorphic encryption, privacy-preserving, quantization protocol

中图分类号:

TN92

余晟兴, 陈钟. 基于同态加密的高效安全联邦学习聚合框架[J]. 通信学报, 2023, 44(1): 14-28.

Shengxing YU, Zhong CHEN. Efficient secure federated learning aggregation framework based on homomorphic encryption[J]. Journal on Communications, 2023, 44(1): 14-28.

图/表 7

图1

图2

图3

图4

表1

图5

图6

参考文献 38

[1]	TEAM I G P . EU General data protection regulation (GDPR) - an implementation and compliance guide,fourth edition[M]. Cambridgeshire: IT Governance Publishing, 2020.
[2]	EVANS D , KOLESNIKOV V , ROSULEK M . A pragmatic introduction to secure multi-party computation[J]. Foundations and Trends in Privacy and Security, 2018,2(2/3): 70-246.
[3]	DWORK C , ROTH A . The algorithmic foundations of differential privacy[J]. Foundations and Trends in Theoretical Computer Science, 2013,9(3/4): 211-407.
[4]	ACAR A , AKSU H , ULUAGAC A S ,et al. A survey on homomorphic encryption schemes[J]. ACM Computing Surveys, 2019,51(4): 1-35.
[5]	LIU X M . Hybrid privacy-preserving clinical decision support system in fog-cloud computing[J]. Future Generation Computer Systems, 2018,78: 825-837.
[6]	CHEN Z K , ZHENG Z W , LIU X M ,et al. Privacy-preserving computation tookit on floating-point numbers[C]// Mobile Multimedia Communications. Berlin:Springer, 2021: 462-476.
[7]	HU S S , LI M H , WANG Q ,et al. Outsourced biometric identification with privacy[J]. IEEE Transactions on Information Forensics and Security, 2018,13(10): 2448-2463.
[8]	MA Z R . Privacy-preserving and high-accurate outsourced disease predictor on random forest[J]. Information Sciences, 2019,496: 225-241.
[9]	LIU X M , CHOO K K R , DENG R H ,et al. Efficient and privacy-preserving outsourced calculation of rational numbers[J]. IEEE Transactions on Dependable and Secure Computing, 2018,15(1): 27-39.
[10]	LI T , SAHU A K , TALWALKAR A ,et al. Federated learning:challenges,methods,and future directions[J]. IEEE Signal Processing Magazine, 2020,37(3): 50-60.
[11]	BHOWMICK A , DUCHI J , FREUDIGER J ,et al. Protection against reconstruction and its applications in private federated learning[J]. arXiv Preprint,arXiv:1812.00984, 2018.
[12]	MELIS L , SONG C Z , CRISTOFARO E D ,et al. Exploiting unintended feature leakage in collaborative learning[C]// Proceedings of 2019 IEEE Symposium on Security and Privacy. Piscataway:IEEE Press, 2019: 691-706.
[13]	ZHANG C , LI S , XIA J ,et al. BatchCrypt:efficient homomorphic encryption for Cross-Silo federated learning[C]// Proceedings of 2020 USENIX Annual Technical Conference. Berkeley:USENIX Association, 2020: 493-506.
[14]	董业, 侯炜, 陈小军 ,等. 基于秘密分享和梯度选择的高效安全联邦学习[J]. 计算机研究与发展, 2020,57(10): 2241-2250.
	DONG Y , HOU W , CHEN X J ,et al. Efficient and secure federated learning based on secret sharing and gradients selection[J]. Journal of Computer Research and Development, 2020,57(10): 2241-2250.
[15]	STROM N , . Scalable distributed DNN training using commodity GPU cloud computing[C]// Proceedings of Interspeech 2015. Piscataway:IEEE Press, 2015: 1488-1492.
[16]	DRYDEN N , MOON T , JACOBS S A ,et al. Communication quantization for data-parallel training of deep neural networks[C]// Proceedings of 2016 2nd Workshop on Machine Learning in HPC Environments (MLHPC). Piscataway:IEEE Press, 2016: 1-8.
[17]	AJI A F , HEAFIELD K . Sparse communication for distributed gradient descent[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Boston:Association for Computational Linguistics, 2017: 440-445.
[18]	ALISTARH D , LI J , TOMIOKA R ,et al. QSGD:randomized quantization for communication-optimal stochastic gradient descent[J]. arXiv Preprint,arXiv:1610.02132, 2016.
[19]	BONAWITZ K , IVANOV V , KREUTER B ,et al. Practical secure aggregation for privacy-preserving machine learning[C]// Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. New York:ACM Press, 2017: 1175-1191.
[20]	SHAMIR A . How to share a secret[J]. Communications of the ACM, 1979,22(11): 612-613.
[21]	NIU C Y , WU F , TANG S J ,et al. Secure federated submodel learning[J]. arXiv Preprint,arXiv:1911.02254, 2019.
[22]	BLOOM B H . Space/time trade-offs in hash coding with allowable errors[J]. Communications of the ACM, 1970,13(7): 422-426.
[23]	DONG Y , CHEN X , SHEN L ,et al. EaSTFLy:efficient and secure ternary federated learning[J]. Computers ＆ Security, 2020,94:101824.
[24]	PAILLIER P , . Public-key cryptosystems based on composite degree residuosity classes[C]// Advances in Cryptology - EUROCRYPT ’99. Berlin:Springer, 1999: 223-238.
[25]	BANNER R , NAHSHAN Y , SOUDRY D . Post training 4-bit quantization of convolutional networks for rapid-deployment[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems.New York:Curran Associates Inc. , 2019: 950-7958.
[26]	WEN W , XU C , YAN F ,et al. TernGrad:ternary gradients to reduce communication in distributed deep learning[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems.New York:Curran Associates Inc. , 2017: 1508-1518.
[27]	SONG L , ZHAO K , PAN P ,et al. Communication efficient SGD via gradient sampling with Bayes prior[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2021: 12060-12069.
[28]	ANDERSON A G , BERG C P . The high-dimensional geometry of binary neural networks[J]. arXiv Preprint,arXiv:1705.07199, 2017.
[29]	SOUDRY D , HUBARA I , MEIR R . Expectation backpropagation:parameter-free training of multilayer neural networks with continuous or discrete weights[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems. Massachusetts:MIT Press, 2014: 963-971.
[30]	BANNER R , HUBARA I , HOFFER E ,et al. Scalable methods for 8-bit training of neural networks[C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems.New York:Curran Associates Inc. , 2019: 5151-5159.
[31]	TANG H L , LIAN X R , ZHANG T ,et al. DoubleSqueeze:parallel stochastic gradient descent with double-pass error-compensated compression[C]// International Conference on Machine Learning. New York:PMLR, 2019: 6155-6165.
[32]	STICH S U , CORDONNIER J B , JAGGI M . Sparsified SGD with memory[J]. arXiv Preprint,arXiv:1809.07599, 2018.
[33]	KOLOSKOVA A , STICH S U , JAGGI M . Decentralized stochastic optimization and gossip algorithms with compressed communication[C]// International Conference on Machine Learning. New York:PMLR, 2019: 3478-3487.
[34]	LIN Y J , HAN S , MAO H Z ,et al. Deep gradient compression:reducing the communication bandwidth for distributed training[J]. arXiv Preprint,arXiv:1712.01887, 2017.
[35]	COURBARIAUX M , BENGIO Y , DAVID J . Training deep neural networks with low precision multiplications[J]. arXiv Preprint,arXiv:1412.7024, 2014.
[36]	GUPTA S , AGRAWAL A , GOPALAKRISHNAN K ,et al. Deep learning with limited numerical precision[C]// International Conference on Machine Learning. New York:PMLR, 2015: 1737-1746.
[37]	LECUN Y , BOTTOU L , BENGIO Y ,et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998,86(11): 2278-2324.
[38]	KRIZHEVSKY A , NAIR V , HINTON G ,et al. The CIFAR-10 dataset[EB]. 2014.

数据集	边缘节点数量/个	方案准确率
数据集	边缘节点数量/个	ESFL(16 bit)	明文模型	BatchCrypt (16 bit）
	55	98.90%98.90%	98.79%98.79%	97.61%97.61%
MNISTMNIST	1010	98.01%98.01%	98.44%98.44%	94.16%94.16%
	2020	97.92%97.92%	98.45%98.45%	90.27%90.27%
	5050	97.33%97.33%	97.73%97.73%	73.68%73.68%
	55	64.43%64.43%	64.80%64.80%	47.44%47.44%
CIFAR10CIFAR10	1010	62.93%62.93%	60.27%60.27%	37.87%37.87%
	2020	63.86%63.86%	58.77%58.77%	29.11%29.11%
	5050	61.31%61.31%	59.82%59.82%	27.06%27.06%