面向非独立同分布数据的联邦学习数据增强方案

doi:10.11959/j.issn.1000-436x.2023007

Abstract

Abstract:

To solve the problem that the model accuracy remains low when the data are not independent and identically distributed (non-IID) across different clients in federated learning, a privacy-preserving data augmentation scheme was proposed.Firstly, a data augmentation framework for federated learning scenarios was designed.All clients generated synthetic samples locally and shared them with each other, which eased the problem of client drift caused by the difference of clients’ data distributions.Secondly, based on generative adversarial network and differential privacy, a private sample generation algorithm was proposed.It helped clients to generate informative samples while preserving the privacy of clients’ local data.Finally, a differentially private label selection algorithm was proposed to ensure the labels of synthetic samples will not leak information.Simulation results demonstrate that under multiple non-IID data partition strategies, the proposed scheme can consistently improve the model accuracy and make the model converge faster.Compared with the benchmark approaches, the proposed scheme can achieve at least 25% accuracy improvement when each client has only one class of samples.

Key words: federated learning, non-IID, generative adversarial network, differential privacy, data augmentation

CLC Number:

TP301

Lingtao TANG, Di WANG, Shengyun LIU. Data augmentation scheme for federated learning with non-IID data[J]. Journal on Communications, 2023, 44(1): 164-176.

Figures/Tables 11

参数	含义
S	中央服务器
C_i,D_i	第i个客户端节点及其本地数据集
γ	客户端虚拟样本共享比例
m_i	C _i 共享的虚拟样本数目
U_i	D _i中所有样本的对应标签集合
L	全局样本类别总数
$(x, y)$	真实样本特征及标签
$(\hat{x}, \hat{y})$	虚拟样本特征及标签
$(ε, δ)$	差分隐私机制的隐私预算
σ	差分隐私机制的噪声乘子
c	训练梯度的剪裁上界
$A$	隐私损失计算函数
$D, G$	判别器, 生成器
B	一批训练样本的数目

方法	MNIST			FMNIST			Cifar10		SVHN
方法	1-Label	2-Label	Dir(0.05)	1-Label	2-Label	Dir(0.05)	1-Label	2-Label	1-Label	2-Label
FedAvg^[1]	50.44%	90.99%	96.63%	39.13%	69.92%	77.58%	13.99%	$45 . 62 %$	9.69%	48.48%
FedProx^[6]	48.56%	87.56%	96.32%	46.48%	62.39%	77.46%	12.28%	31.46%	15.53%	$58 . 86 %$
SCAFFOLD^[7]	9.52%	91.53%	97.82%	10.00%	68.87%	75.38%	10.00%	42.71%	9.33%	34.53%
FedNova^[9]	42.40%	88.86%	96.96%	36.47%	70.13%	77.38%	10.73%	44.97%	11.16%	47.76%
FedMix^[14]	23.32%	65.64%	91.24%	36.18%	51.03%	64.28%	9.98%	43.61%	19.59%	44.89%
本文方案(γ=0.01)	75.13%	93.21%	97.11%	75.11%	$79 . 12 %$	$82 . 84 %$	$25 . 95 %$	44.27%	21.46%	51.48%
本文方案(γ=0.05)	$85 . 99 %$	$94 . 26 %$	$97 . 85 %$	$75 . 54 %$	78.56%	82.41%	23.25%	40.02%	$42 . 94 %$	52.41%
centralized training		99.16%			90.15%		64.87%		87.90%

References 37

[1]	MCMAHAN B , MOORE E , RAMAGE D ,et al. Communication-efficient learning of deep networks from decentralized data[C]// Artificial Intelligence and statistics. New York:PMLR, 2017: 1273-1282.
[2]	KONE?NY J , MCMAHAN H B , RAMAGE D ,et al. Federated optimization:distributed machine learning for on-device intelligence[J]. arXiv Preprint,arXiv:1610.02527, 2016.
[3]	ZHAO Y , LI M , LAI L Z ,et al. Federated learning with non-IID data[J]. arXiv Preprint,arXiv:1806.00582, 2018.
[4]	XU C C , HONG Z W , HUANG M L ,et al. Acceleration of federated learning with alleviated forgetting in local training[J]. arXiv Preprint,arXiv:2203.02645, 2022.
[5]	LI Q B , DIAO Y Q , CHEN Q ,et al. Federated learning on non-IID data silos:an experimental study[J]. arXiv Preprint,arXiv:2102.02079, 2021
[6]	LI T , SAHU A K , ZAHEER M ,et al. Federated optimization in heterogeneous networks[J]. arXiv Preprint,arXiv:1812.06127, 2018.
[7]	KARIMIREDDY S P , KALE S , MOHRI M ,et al. Scaffold:stochastic controlled averaging for federated learning[C]// International Conference on Machine Learning. New York:PMLR, 2020: 5132-5143.
[8]	LUO M , CHEN F , HU D P ,et al. No fear of heterogeneity:classifier calibration for federated learning with non-IID data[C]// Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Massachusetts:MIT Press, 2021: 1-13.
[9]	WANG J Y , LIU Q H , LIANG H ,et al. Tackling the objective in-consistency problem in heterogeneous federated optimization[C]// Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020). Massachusetts:MIT Press, 2020: 1-13.
[10]	HSU T M H , QI H , BROWN M . Measuring the effects of non-identical data distribution for federated visual classifica-tion[J]. arXiv Preprint,arXiv:1909.06335, 2019.
[11]	LIN T , KONG L J , STICH S U ,et al. Ensemble distillation for robust model fusion in federated learning[J]. arXiv Preprint,arXiv:2006.07242, 2020.
[12]	GOETZ J , TEWARI A . Federated learning via synthetic data[J]. arXiv Preprint,arXiv:2008.04489, 2020.
[13]	HAO W T , EL-KHAMY M , LEE J ,et al. Towards fair federated learning with zero-shot data augmentation[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway:IEEE Press, 2021: 3305-3314.
[14]	YOON T , SHIN S , HWANG S J ,et al. FedMix:approximation of mixup under mean augmented federated learning[J]. arXiv Preprint,arXiv:2107.00233, 2021.
[15]	FALLAH A , MOKHTARI A , OZDAGLAR A . Personalized federated learning:a meta-learning approach[J]. arXiv Preprint,arXiv:2002.07948, 2020.
[16]	SMITH V , CHIANG C K , SANJABI M ,et al. Federated multi-task learning[C]// Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017). Massachusetts:MIT Press, 2017: 1-11.
[17]	GHOSH A , CHUNG J , YIN D ,et al. An efficient framework for clustered federated learning[J]. IEEE Transactions on Information Theory, 2022,68(12): 8076-8091.
[18]	WAHEED A , GOYAL M , GUPTA D ,et al. CovidGAN:data augmentation using auxiliary classifier GAN for improved COVID-19 detection[J]. IEEE Access, 2020,8: 91916-91923.
[19]	SHOKRI R , STRONATI M , SONG C Z ,et al. Membership inference attacks against machine learning models[C]// Proceedings of 2017 IEEE Symposium on Security and Privacy. Piscataway:IEEE Press, 2017: 3-18.
[20]	CHEN D F , YU N , ZHANG Y ,et al. GAN-leaks:a taxonomy of membership inference attacks against generative models[C]// Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. New York:ACM Press, 2020: 343-362.
[21]	GOODFELLOW I , POUGET-ABADIE J , MIRZA M ,et al. Generative adversarial nets[C]// Proceedings of the 27th Conference on Neural Information Processing Systems (NeurIPS 2014). Massachusetts:MIT Press, 2014: 2672-2680.
[22]	MIRZA M , OSINDERO S . Conditional generative adversarial nets[J]. arXiv Preprint,arXiv:1411.1784, 2014.
[23]	RADFORD A , METZ L , CHINTALA S . Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv Preprint,arXiv:1511.06434, 2015.
[24]	ARJOVSKY M , CHINTALA S , BOTTOU L . Wasserstein generative adversarial networks[C]// International Conference on Machine Learning. New York:PMLR, 2017: 214-223.
[25]	DWORK C , MCSHERRY F , NISSIM K ,et al. Calibrating noise to sensitivity in private data analysis[C]// Theory of Cryptography Conference. Berlin:Springer, 2006: 265-284.
[26]	DWORK C , ROTH A . The algorithmic foundations of differential privacy[J]. Foundations and Trends in Theoretical Computer Science, 2013,9(3/4): 211-407.
[27]	ABADI M , CHU A , GOODFELLOW I ,et al. Deep learning with differential privacy[C]// Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. New York:ACM Press, 2016: 308-318.
[28]	XIE L Y , LIN K X , WANG S ,et al. Differentially private generative adversarial network[J]. arXiv Preprint,arXiv:1802.06739, 2018.
[29]	ZHANG X Y , JI S L , WANG T . Differentially private releasing via deep generative model[J]. arXiv Preprint,arXiv:1801.01594, 2018.
[30]	DAVODY A , ADELANI D I , KLEINBAUER T ,et al. Robust differentially private training of deep neural networks[J]. arXiv Preprint,arXiv:2006.10919, 2020.
[31]	YOUSEFPOUR A , SHILOV I , SABLAYROLLES A ,et al. Opacus:user-friendly differential privacy library in PyTorch[J]. arXiv Preprint,arXiv:2109.12298, 2021.
[32]	ZENG D , LIANG S Q , HU X J ,et al. FedLab:a flexible federated learning framework[J]. arXiv Preprint,arXiv:2107.11621, 2021.
[33]	LECUN Y , BOTTOU L , BENGIO Y ,et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998,86(11): 2278-2324.
[34]	XIAO H , RASUL K , VOLLGRAF R . Fashion-MNIST:a novel image dataset for benchmarking machine learning algorithms[J]. arXiv Preprint,arXiv:1708.07747, 2017.
[35]	KRIZHEVSKY A . Learning multiple layers of features from tiny images[D]. Toronto:University of Toronto, 2009.
[36]	NETZER Y , WANG T , COATES A ,et al. Reading digits in natural images with unsupervised feature learning[C]// Proceedings of the 25th Conference on Neural Information Processing Systems (NIPS 2011). Massachusetts:MIT Press, 2011: 1-9.
[37]	MIRONOV I , TALWAR K , ZHANG L . Renyi differential privacy of the sampled gaussian mechanism[J]. arXiv Preprint,arXiv:1908.10530, 2019.

Metrics

Recommended 0

No Suggested Reading articles found!

阶段	参数名称	参数值
数据增强阶段	虚拟样本共享比例γ	0.01
	GAN预定训练轮数/次	50
	Batch Size	256
	生成器输入噪声维度	10
	噪声乘子σ	0.5
	剪裁上界c	2
	隐私预算(ε ,δ)	50, 0.000 01
	优化算法	Adam
	优化器参数(学习率,β₁, β₂)	0.0002, 0.5, 0.999
联邦学习阶段	客户端数/个	10
	节点间总通信轮数/次	50
	客户端本地训练轮数/次	5
	每轮参与训练的客户端比例	1
	Batch Size	32
	优化算法	SGD
	优化器参数(学习率,动量)	0.01, 0.5

ε	模型准确率
1	58.81%
5	74.55%
20	75.87%
50	75.13%
∞	88.33%

Data augmentation scheme for federated learning with non-IID data

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 37

Related Articles 15

Metrics

Recommended 0

[1]	Xindi MA, Qinghua LI, Qi JIANG, Zhuo MA, Sheng GAO, Youliang TIAN, Jianfeng MA. Byzantine-robust federated learning over Non-IID data [J]. Journal on Communications, 2023, 44(6): 138-153.
[2]	Youliang TIAN, Shihong WU, Ta LI, Lindong WANG, Hua ZHOU. Federated learning optimization algorithm based on incentive mechanism [J]. Journal on Communications, 2023, 44(5): 169-180.
[3]	Jiale ZHANG, Chengcheng ZHU, Xiaobing SUN, Bing CHEN. Membership inference attack and defense method in federated learning based on GAN [J]. Journal on Communications, 2023, 44(5): 193-205.
[4]	Tao FENG, Liqiu CHEN, Junli FANG, Jianming SHI. Blockchain data sharing scheme based on localized difference privacy and attribute-based searchable encryption [J]. Journal on Communications, 2023, 44(5): 224-233.
[5]	Kaiju LI, Qiang XU, Hao WANG. Communication-efficient federated learning method via redundant data elimination [J]. Journal on Communications, 2023, 44(5): 79-93.
[6]	Shengxing YU, Zekai CHEN, Zhong CHEN, Ximeng LIU. DAGUARD: distributed backdoor attack defense scheme under federated learning [J]. Journal on Communications, 2023, 44(5): 110-122.
[7]	Hui JIANG, Tianliu HE, Min LIU, Sheng SUN, Yuwei WANG. High-performance federated continual learning algorithm for heterogeneous streaming data [J]. Journal on Communications, 2023, 44(5): 123-136.
[8]	Xin SUN, Guifu ZHANG, Hongyan XING, Wang Zenghui. Research on intrusion detection for maritime meteorological sensor network based on balancing generative adversarial network [J]. Journal on Communications, 2023, 44(4): 124-136.
[9]	Shufen ZHANG, Yanling DONG, Jingcheng XU, Haoshi WANG. AdaBoost algorithm based on target perturbation [J]. Journal on Communications, 2023, 44(2): 198-209.
[10]	Shengxing YU, Zhong CHEN. Efficient secure federated learning aggregation framework based on homomorphic encryption [J]. Journal on Communications, 2023, 44(1): 14-28.
[11]	Yanhua LIU, Jiaqi LI, Zhengui OU, Xiaoling GAO, Ximeng LIU, Weizhi MENG, Baoxu LIU. Adversarial training driven malicious code detection enhancement method [J]. Journal on Communications, 2022, 43(9): 169-180.
[12]	Chengsheng YUAN, Qiang GUO, Zhangjie FU. Copyright protection algorithm based on differential privacy deep fake fingerprint detection model [J]. Journal on Communications, 2022, 43(9): 181-193.
[13]	Yanwen WANG, Weimin LEI, Wei ZHANG, Huan MENG, Xinyi CHEN, Wenhui YE, Qingyang JING. Survey on video image reconstruction method based on generative model [J]. Journal on Communications, 2022, 43(9): 194-208.
[14]	Hanyi WANG, Xiaoguang LI, Wenqing BI, Yahong CHEN, Fenghua LI, Ben NIU. Multi-level local differential privacy algorithm recommendation framework [J]. Journal on Communications, 2022, 43(8): 52-64.
[15]	Shaoshuai FAN, Jianbo WU, Hui TIAN. Federated learning resource management for energy-constrained industrial IoT devices [J]. Journal on Communications, 2022, 43(8): 65-77.