群体智能中的联邦学习算法综述

doi:10.11959/j.issn.2096-6652.202218

Abstract

Abstract:

Crowd intelligence is emerging as a new artificial intelligence paradigm owing to the rapid development of the Internet.However, the data isolation and data privacy preservation problems make it difficult to share data among the crowd and to build crowd intelligent applications.Federated learning is a novel solution that aims to collaboratively build models by breaking the data barriers in crowd.Firstly, the basic ideas of federated learning and a comparison with crowd intelligence were introduced.Secondly, federated learning algorithms were divided into three categories according to the crowd organization, and further optimization techniques on privacy, accuracy and efficiency were discussed.Thirdly, federated learning operators based on linear models, tree models and neural network models were presented respectively.Finally, mainstream federated learningopensource platforms and typical applications were introduced, followed by the conclusion.

Key words: crowd intelligence, federated learning, privacy preservation

CLC Number:

TP39

Qiang YANG, Yongxin TONG, Yansheng WANG, et al. A survey on federated learning in crowd intelligence[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(1): 29-44.

Figures/Tables 9

References 98

[1]	KONE?NY J , MCMAHAN H B , YU F X ,et al. Federated learning:strategies for improving communication efficiency[J]. arXiv preprint, 2016,arXiv:1610.05492.
[2]	MCMAHAN H B , MOORE E , RAMAGE D ,et al. Communication-efficient learning of deep networks from decentralized data[J]. arXiv preprint, 2016,arXiv:1602.05629.
[3]	YANG Q , LIU Y , CHEN T J ,et al. Federated machine learning[J]. ACM Transactions on Intelligent Systems and Technology, 2019,10(2): 1-19.
[4]	YANG Q , LIU Y , CHENG Y ,et al. Federated learning-synthesis lectures on artificial intelligence and machine learning[M].[S.l:s.n.], 2019.
[5]	YANG Q , FAN L X , YU H . Federated learning[M]. Cham: Springer International Publishing, 2020.
[6]	杨强, 刘洋, 程勇 ,等. 联邦学习[M]. 北京: 电子工业出版社, 2020.
	YANG Q , LIU Y , CHENG Y ,et al. Federated learning[M]. Beijing: Publishing House of Electronics Industry, 2020.
[7]	杨强, 黄安埠, 刘洋 ,等. 联邦学习实战[M]. 北京: 电子工业出版社, 2021.
	YANG Q , HUANG A B , LIU Y ,et al. Practicing federated learning[M]. Beijing: Publishing House of Electronics Industry, 2021.
[8]	杨强, 刘洋, 陈天健 ,等. 联邦学习[J]. 中国计算机学会通讯, 2018,14(11): 49-55.
	YANG Q , LIU Y , CHEN T J ,et al. Federated learning[J]. Communications of the CCF, 2018,14(11): 49-55.
[9]	REYNOLDS C W , . Flocks,herds and schools:a distributed behavioral model[C]// Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques. New York:ACM Press, 1987: 25-34.
[10]	PANDEY S R , TRAN N H , BENNIS M ,et al. A crowdsourcing framework for on-device federated learning[J]. IEEE Transactions on Wireless Communications, 2020,19(5): 3241-3256.
[11]	SINDINGER T S . Crowdsourcing:why the power of the crowd is driving the future of business[J]. Human Resource Management International Digest, 2010,18(3): 11-16.
[12]	TONG Y X , WANG Y S , SHI D Y . Federated learning in the lens of crowdsourcing[J]. Data Engineering, 2020:26.
[13]	FINKEL J R , MANNING C D . Joint parsing and named entity recognition[C]// Proceedings of 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Morristown:Association for Computational Linguistics, 2009.
[14]	RICH C . Multitask learning[J]. Machine Learning, 1997,28(1): 41-75.
[15]	ZHANG Y , YANG Q . An overview of multi-task learning[J]. National Science Review, 2017,5(1): 30-43.
[16]	LI M , . Scaling distributed machine learning with the parameter server[C]// Proceedings of 2014 International Conference on Big Data Science and Computing. New York:ACM Press, 2014: 583-598.
[17]	SATTLER F , MüLLER K R , SAMEK W . Clustered federated learning:model-agnostic distributed multitask optimization under privacy constraints[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020,32(8): 3710-3722.
[18]	ABAD M S H , OZFATURA E , GUNDUZ D ,et al. Hierarchical federated learning ACROSS heterogeneous cellular networks[C]// Proceedings of 2020 IEEE International Conference on Acoustics,Speech and Signal Processing. Piscataway:IEEE Press, 2020: 8866-8870.
[19]	GHOSH A , CHUNG J , YIN D ,et al. An efficient framework for clustered federated learning[J]. Advances in Neural Information Processing Systems, 2020,33: 19586-19597.
[20]	LIM W Y B , NG J S , XIONG Z H ,et al. Decentralized edge intelligence:a dynamic resource allocation framework for hierarchical federated learning[J]. IEEE Transactions on Parallel and Distributed Systems, 2022,33(3): 536-550.
[21]	WARNAT-HERRESTHAL S , SCHULTZE H , SHASTRY K L ,et al. Swarm learning for decentralized and confidential clinical machine learning[J]. Nature, 2021,594(7862): 265-270.
[22]	KIM H , PARK J , BENNIS M ,et al. Blockchained on-device federated learning[J]. IEEE Communications Letters, 2020,24(6): 1279-1283.
[23]	RIVEST R L , ADLEMAN L M , DERTOUZOS M L . On data banks and privacy homomorphisms[J]. Foundations of Secure Computation, 1978,4(11): 169-180.
[24]	PHONG L T , AONO Y , HAYASHI T ,et al. Privacy-preserving deep learning via additively homomorphic encryption[J]. IEEE Transactions on Information Forensics and Security, 2018,13(5): 1333-1345.
[25]	HAO M , LI H W , LUO X Z ,et al. Efficient and privacy-enhanced federated learning for industrial artificial intelligence[J]. IEEE Transactions on Industrial Informatics, 2020,16(10): 6532-6542.
[26]	CHEN Y Q , QIN X , WANG J D ,et al. FedHealth:a federated transfer learning framework for wearable healthcare[J]. IEEE Intelligent Systems, 2020,35(4): 83-93.
[27]	ZHANG C L , LI S Y , XIA J Z ,et al. Batchcrypt:efficient homomorphic encryption for cross-silo federated learning[C]// Proceedings of 2020 USENIX Conference on Usenix Annual Technical Conference.[S.l.:s.n.], 2020: 493-506.
[28]	YAO A C , . Protocols for secure computations[C]// Proceedings of the 23rd Annual Symposium on Foundations of Computer Science. Piscataway:IEEE Press, 1982: 160-164.
[29]	COCK M D , DOWSLEY R , NASCIMENTO A C A ,et al. Privacypreserving classification of personal text messages with secure multi-party computation:an application to hate-speech detection[J]. Advances in Neural Information Processing Systems 32, 2019:3752.
[30]	SHARMA S , XING C P , LIU Y ,et al. Secure and efficient federated transfer learning[C]// Proceedings of 2019 IEEE International Conference on Big Data. Piscataway:IEEE Press, 2019: 2569-2576.
[31]	ZHU H F , MONG GOH R S , NG W K . Privacy-preserving weighted federated learning within the secret sharing framework[J]. IEEE Access, 2020,8: 198275-198284.
[32]	SHAMIR A . How to share a secret[J]. Communications of the ACM, 1979,22(11): 612-613.
[33]	BONAWITZ K , IVANOV V , KREUTER B ,et al. Practical secure aggregation for privacy-preserving machine learning[C]// Proceedings of 2017 ACM SIGSAC Conference on Computer and Communications Security. New York:ACM Press, 2017: 1175-1191.
[34]	XU G W , LI H W , LIU S ,et al. VerifyNet:secure and verifiable federated learning[J]. IEEE Transactions on Information Forensics and Security, 2020,15: 911-926.
[35]	GAO D S , LIU Y , HUANG A B ,et al. Privacy-preserving heterogeneous federated transfer learning[C]// Proceedings of 2019 IEEE International Conference on Big Data. Piscataway:IEEE Press, 2019: 2552-2559.
[36]	LIU Y , MA Z , YAN Z ,et al. Privacy-preserving federated K-means for proactive caching in next generation cellular networks[J]. Information Sciences, 2020,521: 14-31.
[37]	DWORK C , . Differential privacy[C]// Proceedings of the International Colloquium on Automata,Languages,and Programming. Heidelberg:Springer, 2006: 1-12.
[38]	GEYER R C , KLEIN T , NABI M . Differentially private federated learning:a client level perspective[J]. arXiv preprint, 2017,arXiv:1712.07557.
[39]	MCMAHAN H B , RAMAGE D , TALWAR K ,et al. Learning differentially private recurrent language models[J]. arXiv preprint, 2017,arXiv:1710.06963.
[40]	WANG Y S , TONG Y X , SHI D Y ,et al. An efficient approach for cross-silo federated learning to rank[C]// Proceedings of 2021 IEEE 37th International Conference on Data Engineering. Piscataway:IEEE Press, 2021: 1128-1139.
[41]	SHI Y X , TONG Y X , SU Z Y ,et al. Federated topic discovery:a semantic consistent approach[J]. IEEE Intelligent Systems, 2020.
[42]	JIANG D , TONG Y X , SONG Y F ,et al. Industrial federated topic modeling[J]. ACM Transactions on Intelligent Systems and Technology, 2021,12(1): 1-22.
[43]	JIANG D , SONG Y F , TONG Y X ,et al. Federated topic modeling[C]// Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York:ACM Press, 2019: 1071-1080.
[44]	HAO M , LI H W , XU G W ,et al. Towards efficient and privacy-preserving federated deep learning[C]// Proceedings of 2019 IEEE International Conference on Communications. Piscataway:IEEE Press, 2019: 1-6.
[45]	ABADI M , CHU A , GOODFELLOW I ,et al. Deep learning with differential privacy[C]// Proceedings of 2016 ACM SIGSAC Conference on Computer and Communications Security. New York:ACM Press, 2016: 308-318.
[46]	BHOWMICK A , DUCHI J , FREUDIGER J ,et al. Protection against reconstruction and its applications in private federated learning[J]. arXiv preprint, 2018,arXiv:1812.00984.
[47]	LU Y L , HUANG X H , DAI Y Y ,et al. Differentially private asynchronous federated learning for mobile edge computing in urban informatics[J]. IEEE Transactions on Industrial Informatics, 2020,16(3): 2134-2143.
[48]	WANG Y S , TONG Y X , SHI D Y . Federated latent dirichlet allocation:a local differential privacy based framework[J]. Proceedings of the AAAI Conference on Artificial Intelligence.[S.l.:s.n.], 2020,34(4): 6283-6290.
[49]	MOHRI M , SIVEK G , SURESH A T . Agnostic federated learning[C]// Proceedings of the 36th International Conference on Machine Learning.[S.l.:s.n.], 2019, 4615-4625.
[50]	MANSOUR Y , MOHRI M , RO J ,et al. Three approaches for personalization with applications to federated learning[J]. arXiv preprint, 2020,arXiv:2002.10619.
[51]	DENG Y , KAMANI M M , MAHDAVI M . Adaptive personalized federated learning[J]. arXiv preprint, 2020,arXiv:2003.13461.
[52]	LI T , SAHU A K , ZAHEER M ,et al. Federated optimization in heterogeneous networks[J]. arXiv preprint, 2018,arXiv:1812.06127.
[53]	KARIMIREDDY S P , KALE S , MOHRI M ,et al. SCAFFOLD:stochastic controlled averaging for federated learning[C]// Proceeding of the International Conference on Machine Learning.[S.l.:s.n.], 2020: 5132-5143.
[54]	ROTHCHILD D , PANDA A , ULLAH E ,et al. FetchSGD:communication-efficient federated learning with sketching[C]// Proceedings of the 37th International Conference on Machine Learning.[S.l.:s.n.], 2020: 8253-8265.
[55]	HAMER J , MOHRI M , SURESH A T . FedBoost:a communication-efficient algorithm for federated learning[C]// Proceedings of the 37th International Conference on Machine Learning.[S.l.:s.n.], 2020: 3973-3983.
[56]	SURESH A T , YU F X , KUMAR S ,et al. Distributed mean estimation with limited communication[C]// Proceedings of 2017 International Conference on Machine Learning.[S.l.:s.n.], 2017: 3329-3337.
[57]	CALDAS S , KONE?NY J ,, MCMAHAN H B ,et al. Expanding the reach of federated learning by reducing client resource requirements[J]. arXiv preprint, 2018,arXiv:1812.07210.
[58]	XU J J , DU W L , JIN Y C ,et al. Ternary compression for communication-efficient federated learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022,33(3): 1162-1176.
[59]	HADDADPOUR F , KAMANI M M , MOKHTARI A ,et al. Federated learning with compression:unified analysis and sharp guarantees[C]// Proceedings of the International Conference on Artificial Intelligence and Statistics.[S.l.:s.n.], 2021: 2350-2358.
[60]	CUI L Z , SU X X , MING Z X ,et al. CREAT:blockchain-assisted compression algorithm of federated learning for content caching in edge computing[J]. IEEE Internet of Things Journal, 2020,4370(99): 1.
[61]	WEI S Y , TONG Y X , ZHOU Z M ,et al. Efficient and fair data valuation for horizontal federated learning[M]// Federated learning. Cham: Springer, 2020.
[62]	CHAI Z , ALI A , ZAWAD S ,et al. TiFL:a tier-based federated learning system[C]// Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing. New York:ACM Press, 2020: 125-136.2020.
[63]	HUANG T S , LIN W W , WU W T ,et al. An efficiency-boosting client selection scheme for federated learning with fairness guarantee[J]. IEEE Transactions on Parallel and Distributed Systems, 2021,32(7): 1552-1564.
[64]	LAI F , ZHU X F , MADHYASTHA H ,et al. Oort:informed participant selection for scalable federated learning[J]. arXiv preprint, 2020,arXiv:2010.06081.
[65]	WANG H , KAPLAN Z , NIU D ,et al. Optimizing federated learning on non-IID data with reinforcement learning[C]// Proceedings of 2020 IEEE Conference on Computer Communications. Piscataway:IEEE Press, 2020: 1698-1707.
[66]	SONG T S , TONG Y X , WEI S Y . Profit allocation for federated learning[C]// Proceedings of 2019 IEEE International Conference on Big Data. Piscataway:IEEE Press, 2019: 2577-2586.
[67]	HARDY S , HENECKA W , IVEY-LAW H , ,et al. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption[J]. arXiv preprint, 2017,arXiv:1711.10677.
[68]	KONE?NY J , MCMAHAN H B , RAMAGE D ,et al. Federated optimization:distributed machine learning for on-device intelligence[J]. arXiv preprint, 2016,arXiv:1610.02527.
[69]	TRUEX S , BARACALDO N , ANWAR A ,et al. A hybrid approach to privacy-preserving federated learning[C]// Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security. New York:ACM Press, 2019: 1-11.
[70]	WU Y C , CAI S F , XIAO X K ,et al. Privacy preserving vertical federated learning for tree-based models[J]. Proceedings of the VLDB Endowment, 2020,13(12): 2090-2103.
[71]	CHENG K W , FAN T , JIN Y L ,et al. SecureBoost:a lossless federated learning framework[J]. IEEE Intelligent Systems, 2021,36(6): 87-98.
[72]	LIU Y , LIU Y T , LIU Z J ,et al. Federated forest[J]. IEEE Transactions on Big Data, 2020,2755(99): 1.
[73]	LIU Y , MA Z , LIU X M ,et al. Revocable federated learning:a benchmark of federated forest[J]. arXiv preprint, 2019,arXiv:1911.03242.
[74]	SAHU A K , LI T , SANJABI M ,et al. On the convergence of federated optimization in heterogeneous networks[J]. arXiv preprint, 2018,arXiv:1812.06127.
[75]	YUROCHKIN M , AGARWAL M , GHOSH S ,et al. Bayesian nonparametric federated learning of neural networks[C]// Proceedings of the International Conference on Machine Learning.[S.l.:s.n.], 2019: 7252-7261.
[76]	WANG H Y , YUROCHKIN M , SUN Y K ,et al. Federated learning with matched averaging[J]. arXiv preprint, 2020,arXiv:2002.06440.
[77]	HE C Y , ANNAVARAM M , AVESTIMEHR S . Group knowledge transfer:federated learning of large CNNs at the edge[J]. arXiv preprint, 2020,arXiv:2007.14513.
[78]	BACCIU D , DI SARLI D , FARAJI P ,et al. Federated reservoir computing neural networks[C]// Proceedings of 2021 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2021: 1-7.
[79]	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017,60(6): 84-90.
[80]	SUNDERMEYER M , SCHLüTER R ,, NEY H . LSTM neural networks for language modeling[C]// Proceedings of the 13th Annual Conference of The International Speech Communication Association.[S.l.:s.n.], 2012.
[81]	GRAVES A , MOHAMED A R , HINTON G . Speech recognition with deep recurrent neural networks[C]// Proceedings of 2013 IEEE International Conference on Acoustics,Speech and Signal Processing. Piscataway:IEEE Press, 2013: 6645-6649.
[82]	SUTSKEVER I , VINYALS O , LE Q V . Sequence to sequence learning with neural networks[J]. Advances in Neural Information Processing Systems, 2014.
[83]	WANG R S , LI Z , CAO J ,et al. Recurrent convolutional neural networks for text classification[C]// Proceedings of 2019 International Joint Conference on Neural Networks.[S.l.:s.n.], 2015.
[84]	CHO K , VAN MERRIENBOER B , BAHDANAU D ,et al. On the properties of neural machine translation:encoder–decoder approaches[C]// Proceedings of the 8th Workshop on Syntax,Semantics and Structure in Statistical Translation. Stroudsburg:Association for Computational Linguistics, 2014.
[85]	SHUKLA S , SRIVASTAVA N . Federated matched averaging with information-gain based parameter sampling[C]// Proceedings of the 1st International Conference on AI-ML-Systems.[S.l.:s.n.], 2021: 1-7.
[86]	LIU Y , FAN T , CHEN T J ,et al. FATE:an industrial grade platform for collaborative learning with data protection[J]. Journal of Machine Learning Research, 2021,22(226): 1-6.
[87]	BONAWITZ K , EICHNER H , GRIESKAMP W ,et al. Towards federated learning at scale:system design[J]. Proceedings of Machine Learning and Systems, 2019,1: 374-388.
[88]	ZILLER A , TRASK A , LOPARDO A ,et al. PySyft:a library for easy federated learning[M]// Federated learning systems. Cham: Springer, 2021.
[89]	HE C Y , LI S Z , SO J ,et al. FedML:a research library and benchmark for federated machine learning[J]. arXiv preprint, 2020,arXiv:2007.13518.
[90]	BEUTEL D J , TOPAL T , MATHUR A ,et al. Flower:a friendly federated learning research framework[J]. arXiv preprint, 2020,arXiv:2007.14390.
[91]	LIU Y , YU J J Q , KANG J W ,et al. Privacy-preserving traffic flow prediction:a federated learning approach[J]. IEEE Internet of Things Journal, 2020,7(8): 7751-7763.
[92]	YU S , CHEN X , ZHOU Z ,et al. When deep reinforcement learning meets federated learning:intelligent multitimescale resource management for multiaccess edge computing in 5G ultradense network[J]. IEEE Internet of Things Journal, 2021,8(4): 2238-2251.
[93]	ZHAO N , WU H , YU F R ,et al. Deep-reinforcement-learning-based latency minimization in edge intelligence over vehicular networks[J]. IEEE Internet of Things Journal, 2022,9(2): 1300-1312.
[94]	SHI D Y , TONG Y X , ZHOU Z M ,et al. Learning to assign:towards fair task assignment in large-scale ride hailing[C]// Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery ＆ Data Mining.[S.l.:s.n.], 2021: 3549-3557.
[95]	ZHENG W B , YAN L , GOU C ,et al. Federated meta-learning for fraudulent credit card detection[C]// Proceedings of the 29th International Joint Conference on Artificial Intelligence. California:International Joint Conferences on Artificial Intelligence Organization, 2020: 4654-4660.
[96]	HAO M , LI H W , XU G W ,et al. Privacy-aware and resource-saving collaborative learning for healthcare in cloud computing[C]// Proceedings of 2020 IEEE International Conference on Communications. Piscataway:IEEE Press, 2020: 1-6.
[97]	BRISIMI T S , CHEN R D , MELA T ,et al. Federated learning of predictive models from federated electronic health records[J]. International Journal of Medical Informatics, 2018,112: 59-67.
[98]	ZHANG W S , ZHOU T , LU Q H ,et al. Dynamic fusion-based federated learning for COVID-19 detection[J]. IEEE Internet of Things Journal, 2021,8(21): 15884-15891.

Metrics

Recommended 0

No Suggested Reading articles found!

联邦学习的算法优化分类			参考文献
算法隐私优化	同态加密	优化安全性	[23-25]
		优化加密效率	[26-27]
	安全多方计算	基于混淆电路	[28-31]
		基于秘密共享	[32-36]
	差分隐私	基于中心化差分隐私	[37-43]
		基于本地化差分隐私	[44-48]
算法精度优化	泛化界优化		[2]、[49-51]
	收敛性优化		[17]、[52-55]
算法效率优化	模型压缩		[56-60]
	客户端选择		[2]、[61-65]

算法	数据划分	适用模型	模型类型	隐私保护机制
VFL-LM^[3]	纵向	线性模型	线性回归	同态加密
VFL-LR^[67]	纵向		逻辑回归	同态加密
IFCA^[19]	横向		线性回归	无
FSVRG^[68]	横向		线性回归/逻辑回归	无
PDTL^[69]	横向	树模型	决策树	多方安全计算/差分隐私
Pivot^[70]	纵向		决策树/随机森林	多方安全计算
SecureBoost^[71]	纵向		决策树	同态加密
FRF^[72]	纵向		随机森林	加密/混淆
RevFRF^[73]	纵向		随机森林	加密/混淆
FedAvg^[2]	横向	神经网络模型	多层感知机/卷积神经网络	无
FedProx^[74]	横向		多层感知机/卷积神经网络	无
PFNM^[75]	横向		多层感知机	无
FedMA^[76]	横向		多层感知机/循环神经网络	无
FedGKT^[77]	横向		卷积神经网络	无
IncFed^[78]	横向		循环神经网络	无

A survey on federated learning in crowd intelligence

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 98

Related Articles 2

Metrics

Recommended 0

[1]	Mingyang CHEN, Wen ZHANG, Xiangnan CHEN, Hongting ZHOU, Huajun CHEN. Collective knowledge graph: meta knowledge transfer and federated graph reasoning [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(1): 55-64.
[2]	Yating WEI,Zhiyong WANG,Shuyue ZHOU,Wei CHEN. Federated visualization:a new model for privacy-preserving visualization [J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(4): 415-420.