冗余数据去除的联邦学习高效通信方法

doi:10.11959/j.issn.1000-436x.2023072

Abstract

Abstract:

To address the influence of limited network bandwidth of edge devices on the communication efficiency of federated learning, and efficiently transmit local model update to complete model aggregation, a communication-efficient federated learning method via redundant data elimination was proposed.The essential reasons for generation of redundant update parameters and according to non-IID properties and model distributed training features of FL were analyzed, a novel sensitivity and loss function tolerance definitions for coreset was given, and a novel federated coreset construction algorithm was proposed.Furthermore, to fit the extracted coreset, a novel distributed adaptive sparse network model evolution mechanism was designed to dynamically adjust the structure and the training model size before each global training iteration, which reduced the number of communication bits between edge devices and the server while also guarantees the training model accuracy.Experimental results show that the proposed method achieves 17% reduction in communication bits transmission while only 0.5% degradation in model accuracy compared with state-of-the-art method.

Key words: federated learning, communication efficiency, coreset, model evolution, accuracy

CLC Number:

TP391

Kaiju LI, Qiang XU, Hao WANG. Communication-efficient federated learning method via redundant data elimination[J]. Journal on Communications, 2023, 44(5): 79-93.

Figures/Tables 7

References 31

[1]	RODRIGUES T K , SUTO K , KATO N . Edge cloud server deployment with transmission power control through machine learning for 6G Internet of things[J]. IEEE Transactions on Emerging Topics in Computing, 2021,9(4): 2099-2108.
[2]	MCMAHAN B , MOORE E , RAMAGE D ,et al. Communication-efficient learning of deep networks from decentralized data[C]// Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS). Piscataway:IEEE Press, 2017: 1273-1282.
[3]	LI K J , XIAO C H . CBFL:a communication-efficient federated learning framework from data redundancy perspective[J]. IEEE Systems Journal, 2022,16(4): 5572-5583.
[4]	SATTLER F , WIEDEMANN S , MüLLER K R ,et al. Robust and communication-efficient federated learning from non-i.i.d.data[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019,31(9): 3400-3413.
[5]	LI K J , XIAO C H . PBFL:communication-efficient federated learning via parameter predicting[J]. The Computer Journal, 2023,66(3): 626-642.
[6]	KONECNY J , MCMAHAN H B , YU F X ,et al. Federated learning:strategies for improving communication efficiency[J]. arXiv Preprint,arXiv:1610.05492, 2016.
[7]	LI T , SAHU A K , TALWALKAR A ,et al. Federated learning:challenges,methods,and future directions[J]. IEEE Signal Processing Magazine, 2020,37(3): 50-60.
[8]	LI K J , XIAO C H . Federated learning communication-efficiency framework via corset construction[J]. Computer Journal, 2022,doi:10.1093/comjnl/bxac062.
[9]	TELLEZ D , LITJENS G , LAAK J ,et al. Neural image compression for gigapixel histopathology image analysis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021,43(2): 567-578.
[10]	LI X , HUANG K X , YANG W H ,et al. On the convergence of FedAvg on Non-IID data[C]// Proceedings of the International Conference on Learning Representations (ICLR). Piscataway:IEEE Press, 2020: 1-26.
[11]	TAO Z , LI Q . eSGD:communication efficient distributed deep learning on the edge[C]// 2018 Hot Topics in Edge Computing (HotEdge 18). Piscataway:IEEE Press, 2018: 1-6.
[12]	YU H , YANG S , ZHU S H . Parallel restarted SGD with faster convergence and less communication:demystifying why model averaging works for deep learning[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto:AAAI Press, 2019: 5693-5700.
[13]	XU J J , DU W L , JIN Y C ,et al. Ternary compression for communication-efficient federated learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022,33(3): 1162-1176.
[14]	LI S Q , QI Q , WANG J Y ,et al. GGS:general gradient sparsification for federated learning in edge computing[C]// Proceedings of 2020 IEEE International Conference on Communications (ICC). Piscataway:IEEE Press, 2020: 1-7.
[15]	HAN P C , WANG S Q , LEUNG K K . Adaptive gradient sparsification for efficient federated learning:an online learning approach[C]// Proceedings of 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS). Piscataway:IEEE Press, 2021: 300-310.
[16]	OZFATURA E , OZFATURA K , GüNDüZ D . Time-correlated sparsification for communication-efficient federated learning[C]// Proceedings of 2021 IEEE International Symposium on Information Theory (ISIT). Piscataway:IEEE Press, 2021: 461-466.
[17]	ASAD M , MOUSTAFA A , ITO T . FedOpt:towards communication efficiency and privacy preservation in federated learning[J]. Applied Sciences, 2020,10(8): 2864.
[18]	BERNSTEIN J , WANG Y , AZIZZADENESHELI K ,et al. signSGD:compressed optimisation for non-convex problems[C]// Proceedings of the 35th International Conference on Machine Learning. Piscataway:IEEE Press, 2018: 560-569.
[19]	REISIZADEH A , MOKHTARI A , HASSANI H ,et al. FedPAQ:a communication-efficient federated learning method with periodic averaging and quantization[C]// Proceedings of the 23rdInternational Conference on Artificial Intelligence and Statistics (AISTATS). Piscataway:IEEE Press, 2020: 2021-2031.
[20]	AMIRI M M , GUNDUZ D , KULKARNI S R ,et al. Federated learning with quantized global model updates[J]. arXiv Preprint,arXiv:2020.10672, 2020.
[21]	NORI M K , YUN S , KIM I M . Fast federated learning by balancing communication trade-offs[J]. IEEE Transactions on Communications, 2021,69(8): 5168-5182.
[22]	JHUNJHUNWALA D , GADHIKAR A , JOSHI G ,et al. Adaptive quantization of model updates for communication-efficient federated learning[C]// Proceedings of 2021 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP). Piscataway:IEEE Press, 2021: 3110-3114.
[23]	DU Y , YANG S , HUANG H . High-dimensional stochastic gradient quantization for communication-efficient edge learning[C]// Proceed ings of 2019 IEEE Global Conference on Signal and Information Processing (Global SIP). Piscataway:IEEE Press, 2019: 1-5.
[24]	LIAN Z , CAO Z , ZUO Y ,et al. AGQFL:communication-efficient federated learning via automatic gradient quantization in edge heterogeneous systems[C]// Proceedings of 2021 IEEE 39th International Conference on Computer Design (ICCD). Piscataway:IEEE Press, 2021: 551-558.
[25]	BāDOIU M , HAR-PELED S , INDYK P . Approximate clustering via core-sets[C]// Proceedings of the 34th Annual ACM Symposium on Theory of Computing. New York:ACM Press, 2002: 250-257.
[26]	VLADIMIR B , DAN F , HARRY L ,et al. Efficient coreset constructions via sensitivity sampling[C]// Proceedings of the 13th Asian Con ference on Machine Learning. Piscataway:IEEE Press, 2021: 948-963.
[27]	LU H L , LI M J , HE T ,et al. Robust coreset construction for distributed machine learning[J]. IEEE Journal on Selected Areas in Communications, 2020,38(10): 2400-2417.
[28]	CAMPBELL T , BRODERICK T . Bayesian coreset construction via greedy iterative geodesic ascent[C]// Proceedings of the 35th International Conference on Machine Learning. Piscataway:IEEE Press, 2018: 698-706.
[29]	FAN Y W , LI H S . Communication efficient coreset sampling for distributed learning[C]// Proceedings of 2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC). Piscataway:IEEE Press, 2018: 1-5.
[30]	MOCANU D C , MOCANU E , STONE P ,et al. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science[J]. Nature Communications, 2018,9:2383.
[31]	LI A , SUN J , WANG B ,et al. LotteryFL:personalized and communication-efficient federated learning with lottery ticket hypothesis on non-IID datasets[J]. arXiv Preprint,arXiv:2008.03371, 2020.

Metrics

Recommended 0

No Suggested Reading articles found!

符号	含义
	全局加权平均损失函数
	设备N_i第t轮迭代的本地损失函数
	设备N_i第t轮迭代的梯度值
	设备N_i的学习率
	设备N_i第t轮迭代的本地模型更新
	第t轮迭代的全局模型参数
	第T轮迭代的全局模型参数
D_i	设备的本地数据集
D	全体样本空间
N	终端设备数目
ε	损失函数容忍度
Γ	原始数据集X的核心数据集
A_T	全局模型的收敛准确率
B_T	累计通信比特数

参数	取值
N	100
r	1
B	10
E	5
η	0.01
ε	0.05
σ	0.08
β	{100,50,30}
ζ	{0,0.1,0.5,1.0}
?	{0,0.1,0.5,1.0}
(λ,K)	{(5,16), (4,8), (3,6), (2,4), (1,2)}
τ	0.01

Communication-efficient federated learning method via redundant data elimination

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 31

Related Articles 15

Metrics

Recommended 0

[1]	Xindi MA, Qinghua LI, Qi JIANG, Zhuo MA, Sheng GAO, Youliang TIAN, Jianfeng MA. Byzantine-robust federated learning over Non-IID data [J]. Journal on Communications, 2023, 44(6): 138-153.
[2]	Shengxing YU, Zekai CHEN, Zhong CHEN, Ximeng LIU. DAGUARD: distributed backdoor attack defense scheme under federated learning [J]. Journal on Communications, 2023, 44(5): 110-122.
[3]	Hui JIANG, Tianliu HE, Min LIU, Sheng SUN, Yuwei WANG. High-performance federated continual learning algorithm for heterogeneous streaming data [J]. Journal on Communications, 2023, 44(5): 123-136.
[4]	Youliang TIAN, Shihong WU, Ta LI, Lindong WANG, Hua ZHOU. Federated learning optimization algorithm based on incentive mechanism [J]. Journal on Communications, 2023, 44(5): 169-180.
[5]	Jiale ZHANG, Chengcheng ZHU, Xiaobing SUN, Bing CHEN. Membership inference attack and defense method in federated learning based on GAN [J]. Journal on Communications, 2023, 44(5): 193-205.
[6]	Shengxing YU, Zhong CHEN. Efficient secure federated learning aggregation framework based on homomorphic encryption [J]. Journal on Communications, 2023, 44(1): 14-28.
[7]	Lingtao TANG, Di WANG, Shengyun LIU. Data augmentation scheme for federated learning with non-IID data [J]. Journal on Communications, 2023, 44(1): 164-176.
[8]	Shaoshuai FAN, Jianbo WU, Hui TIAN. Federated learning resource management for energy-constrained industrial IoT devices [J]. Journal on Communications, 2022, 43(8): 65-77.
[9]	Zijia MO, Zhipeng GAO, Yang YANG, Yijing LIN, Shan SUN, Chen ZHAO. Efficient distributed model sharing strategy for data privacy protection in Internet of vehicles [J]. Journal on Communications, 2022, 43(4): 83-94.
[10]	Haiyan KANG, Yuanrui JI. Research on federated learning approach based on local differential privacy [J]. Journal on Communications, 2022, 43(10): 94-105.
[11]	Meixia TAO, Dong WANG, Rui SUN, Naifu ZHANG. TDMA-based user scheduling policies for federated learning [J]. Journal on Communications, 2021, 42(6): 1-29.
[12]	Wenchen HE, Shaoyong GUO, Xuesong QIU, Liandong CHEN, Suxiang ZHANG. Node selection method in federated learning based on deep reinforcement learning [J]. Journal on Communications, 2021, 42(6): 62-71.
[13]	Youhuizi LI, Yuyu YIN, Honghao GAO, Yi JIN, Xinheng WANG. Survey on privacy protection in non-aggregated data sharing [J]. Journal on Communications, 2021, 42(6): 195-212.
[14]	Yongming HUANG, Chong ZHENG, Zhengming ZHANG, Xiaohu YOU. Research on mobile edge computing and caching in massive wireless communication network [J]. Journal on Communications, 2021, 42(4): 44-61.
[15]	Ruiqi LI, Chunfu JIA, Yafei WANG. Multi-key homomorphic proxy re-encryption scheme based on NTRU and its application [J]. Journal on Communications, 2021, 42(3): 11-22.