基于多智能体深度强化学习的多域协同抗干扰方法研究

doi:10.11959/j.issn.2096-3750.2022.00293

Abstract

Abstract:

Dynamic transmission requirements and the limited cache space bring great challenges to wireless data transmission in the malicious jamming environment.Aiming at the above problems, a collaborative anti-jamming channel selection and data scheduling joint decision method for distributed internet of things was studied from the perspective of frequency domain and time domain.A data transmission model based on multi-user Markov decision process was constructed and a collaborativeanti-jamming joint-channel-and-data decision algorithm based on multi-agent deep reinforcement learning was proposed.Simulation results show that the proposed algorithm can effectively avoid the malicious jamming and the co-channel interference.Compared with the comparison algorithm, the network throughput is significantly improved, and the number of packet dropout is significantly reduced.

Key words: collaborative anti-jamming, channel selection, data scheduling, multi-agent reinforcement learning, deep learning

CLC Number:

Biao ZHANG, Ximing WANG, Yifan XU, Wen LI, Hao HAN, Songyi LIU, Xueqiang CHEN. Multi-domain collaborative anti-jamming based on multi-agent deep reinforcement learning[J]. Chinese Journal on Internet of Things, 2022, 6(4): 104-116.

Figures/Tables 16

References 30

[1]	CHOWDHURY M Z , SHAHJALAL M , AHMED S ,et al. 6G wireless communication systems:applications,requirements,technologies,challenges,and research directions[J]. IEEE Open Journal of the Communications Society, 2020(1): 957-975.
[2]	ZHANG L , LIANG Y C , NIYATO D . 6G Visions:mobile ultra-broadband,super internet of things,and artificial intelligence[J]. China Communications, 2019,16(8): 1-14.
[3]	AL-FUQAHA A , GUIZANI M , MOHAMMADI M ,et al. Internet of things:a survey on enabling technologies,protocols,and applications[J]. IEEE Communications Surveys ＆Tutorials, 2015,17(4): 2347-2376.
[4]	PIRAYESH H , ZENG H C . Jamming attacks and anti-jamming strategies in wireless networks:a comprehensive survey[J]. IEEE Communications Surveys ＆ Tutorials, 2022,24(2): 767-809.
[5]	KARAGIANNIS D , ARGYRIOU A . Jamming attack detection in a pair of RF communicating vehicles using unsupervised machine learning[J]. Vehicular Communications, 2018,13: 56-63.
[6]	王海超, 王金龙, 丁国如 ,等. 空天地一体化网络中智能协同抗干扰技术[J]. 指挥与控制学报, 2020,6(3): 185-191.
	WANG H C , WANG J L , DING G R ,et al. Intelligent cooperative anti-jamming technology in space-air-ground integrated networks[J]. Journal of Command and Control, 2020,6(3): 185-191.
[7]	冉雨, 程郁凡, 陈大勇 ,等. 采用BP神经网络的智能抗干扰决策引擎研究[J]. 信号处理, 2019,35(8): 1350-1357.
	RAN Y , CHENG Y F , CHEN D Y ,et al. Intelligent anti-jamming decision engine based on BP neural network[J]. Journal of Signal Processing, 2019,35(8): 1350-1357.
[8]	KONG L J , XU Y H , ZHANG Y L ,et al. A reinforcement learning approach for dynamic spectrum anti-jamming in fading environment[C]// Proceedings of 2018 IEEE 18th International Conference on Communication Technology. Piscataway:IEEE Press, 2018: 51-58.
[9]	PEI X F , WANG X M , YAO J N ,et al. Joint time-frequency anti-jamming communications:a reinforcement learning approach[C]// Proceedings of 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP). Piscataway:IEEE Press, 2019: 1-6.
[10]	HAN H , WANG X M , GU F L ,et al. Better late than never:GAN-enhanced dynamic anti-jamming spectrum access with incomplete sensing information[J]. IEEE Wireless Communications Letters, 2021,10(8): 1800-1804.
[11]	XIAO L , WAN X Y , LU X Z ,et al. IoT security techniques based on machine learning:how do IoT devices use AI to enhance security?[J]. IEEE Signal Processing Magazine, 2018,35(5): 41-49.
[12]	LIU X , XU Y H , JIA L L ,et al. Anti-jamming communications using spectrum waterfall:a deep reinforcement learning approach[J]. IEEE Communications Letters, 2018,22(5): 998-1001.
[13]	XU Y F , XU Y H , REN G C ,et al. Play it by ear:context-aware distributed coordinated anti-jamming channel access[J]. IEEE Transactions on Information Forensics and Security, 2021,16: 5279-5293.
[14]	XU Y F , XU Y H , DONG X ,et al. Convert harm into benefit:a coordination-learning based dynamic spectrum anti-jamming approach[J]. IEEE Transactions on Vehicular Technology, 2020,69(11): 13018-13032.
[15]	XU Y F , REN G C , CHEN J ,et al. A one-leader multi-follower Bayesian-stackelberg game for anti-jamming transmission in UAV communication networks[J]. IEEE Access, 2018(6): 21697-21709.
[16]	YAO F Q , JIAL L . A collaborative multi-agent reinforcement learning anti-jamming algorithm in wireless networks[J]. IEEE Wireless Communications Letters, 2019,8(4): 1024-1027.
[17]	WANG X M , XU Y H , CHEN J ,et al. Mean field reinforcement learning based anti-jamming communications for ultra-dense Internet of Things in 6G[C]// Proceedings of 2020 International Conference on Wireless Communications and Signal Processing (WCSP). Piscataway:IEEE Press, 2020: 195-200.
[18]	ELLEUCH I , POURRANJBAR A , KADDOUM G . A novel distributed multi-agent reinforcement learning algorithm against jamming attacks[J]. IEEE Communications Letters, 2021,25(10): 3204-3208.
[19]	LI W , XU Y H , GUO Q J ,et al. A Q-learning-based channel selection and data scheduling approach for high-frequency communications in jamming environment[C]// Machine Learning and Intelligent Communications, 2019: 145-160.
[20]	WANG X M , CHEN X Q , WANG M ,et al. Decentralized reinforcement learning based anti-jamming communication for self-organizing networks[C]// Proceedings of 2021 IEEE Wireless Communications and Networking Conference. Piscataway:IEEE Press, 2021: 1-6.
[21]	PEI X F , WANG X M , RUAN L ,et al. Joint power and channel selection for anti-jamming communications:a reinforcement learning approach[C]// Machine Learning and Intelligent Communications, 2019: 551-562.
[22]	XUE C J , . Anti-interference performance of multi-path direct sequence spread spectrum wireless communication system[C]// Proceedings of 2010 International Conference on E-Health Networking Digital Ecosystems and Technologies (EDT). Piscataway:IEEE Press, 2010(1): 461-464.
[23]	ORORBIA M E , WARN G P . Design synthesis through a Markov decision process and reinforcement learning framework[J]. Journal of Computing and Information Science in Engineering, 2022,22(2): 021002.
[24]	FEINBERG V , WAN A , STOICA I ,et al. Model-based value estimation for efficient model-free reinforcement learning[EB]. 2018.
[25]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[26]	HE K M , SUN J . Convolutional neural networks at constrained time cost[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2015: 5353-5360.
[27]	ZHANG X B , WANG H , RUAN L ,et al. Joint channel,power and bandwidth optimization for anti-jamming communications:amulti-agent Q-learning approach[C]// Proceedings of 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP). Piscataway:IEEE Press, 2021: 1-6.
[28]	陈昕, 徐彤, 向旭东 ,等. 具有并行信道的认知无线网络性能评价研究[J]. 计算机研究与发展, 2013,50(10): 2126-2132.
	CHEN X , XU T , XIANG X D ,et al. Performance evaluation of cognitive radio networks with parallel channels[J]. Journal of Computer Research and Development, 2013,50(10): 2126-2132.
[29]	LI J , HAN Y . Optimal resource allocation for packet delay minimization in multi-layer UAV networks[J]. IEEE Communications Letters, 2017,21(3): 580-583.
[30]	KAWABATA A , CHATTERJEE B C , BA S ,et al. A real-time delay-sensitive communication approach based on distributed processing[J]. IEEE Access, 2017(5): 20235-20248.

Metrics

Recommended 0

No Suggested Reading articles found!

名称	参数值
用户数量	N= 5
信道数量	M =10
缓冲区长度	L=10个
历史时长	Φ=200 ms
数据包到达速率	λ=[1,2,3,4,5,6,7]个/时隙
单时隙最大传输数据量	H_max=5个
发送功率	0 dBm
最低速率门限	V_th=7 Mbit/s
通信服务质量门限	β_th=10 dB
成功传输奖励因子	ε= 0.5
失败传输惩罚因子	?=- 0.1、η=-0.2

名称	参数值
干扰信道数量	4
干扰机功率	30 dBm
背景噪声功率	- 90 dBm/Hz
干扰切换速度	15 ms

名称	参数值
迭代次数	2 000
折扣因子	γ= 0.9
学习率	α=0.02
采样批次大小	B=64个

Multi-domain collaborative anti-jamming based on multi-agent deep reinforcement learning

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 30

Related Articles 9

Metrics

Recommended 0

[1]	Zhifei ZHANG, Feng LIU, Yiyang GE, Shuo LI, Yu ZHANG, Ke XIONG. An intrusion detection method based on depthwise separable convolution and attention mechanism [J]. Chinese Journal on Internet of Things, 2023, 7(1): 49-59.
[2]	Rui JIANG, Liuting SUN, Xiaoming WANG, Dapeng LI, Youyun XU. Research on EEG signal classification of motor imagery based on AE and Transformer [J]. Chinese Journal on Internet of Things, 2023, 7(1): 118-128.
[3]	Dan LUO, Ruzhi XU, Zhitao GUAN. Differential privacy budget optimization based on deep learning in IoT [J]. Chinese Journal on Internet of Things, 2022, 6(2): 65-76.
[4]	Xuanzhe XU, Ke NING, Xuemin ZHENG, Mingxin ZHAO, Mengmeng XU, Nanjian WU, Liyuan LIU. Verification of an artificial intelligence vision chip design for edge computing based on hardware simulation system [J]. Chinese Journal on Internet of Things, 2022, 6(1): 20-28.
[5]	Guoquan LI, Yonghai XU, Jinzhao LIN, Zhengwen HUANG. Res-DNN based signal detection algorithm for end-to-end MIMO systems [J]. Chinese Journal on Internet of Things, 2022, 6(1): 65-72.
[6]	Ling TAN, Shanshan RONG, Jingming XIA, Sarker SAJIB, Wenjie MA. Real-time diagnosis of multi-category skin diseases based on IR-VGG [J]. Chinese Journal on Internet of Things, 2021, 5(3): 115-125.
[7]	Chunmin LIN, Liekang ZENG, Xu CHEN. Research on power efficient autonomous UAV navigation algorithm: an edge intelligence driven approach [J]. Chinese Journal on Internet of Things, 2021, 5(2): 87-96.
[8]	Muhan CHEN,Jiajia GUO,Xiao LI,Shi JIN. An overview of the CSI feedback based on deep learning for massive MIMO systems [J]. Chinese Journal on Internet of Things, 2020, 4(1): 33-44.
[9]	Yong LIAO,Haimei YAO,Yuanxiao HUA. Channel state information acquisition algorithm based on deep learning for IoT [J]. Chinese Journal on Internet of Things, 2019, 3(1): 8-13.