智能电网中基于多智能体强化学习的频谱分配算法

doi:10.11959/j.issn.1000-436x.2023179

Abstract

Abstract:

In view of the fact that 5G networks are used to meet the service requirements of various power terminals in smart grid, a spectrum allocation algorithm based on multi-agent reinforcement learning was proposed.Firstly, for the integrated access backhaul system deployed in smart grid, considering the different communication requirements of services in lightweight and non-lightweight terminal, the spectrum allocation problem was formulated as a non-convex mixed-integer programming aiming to maximize the overall energy efficiency.Secondly, the above problem was modeled as a partially observable Markov decision process and transformed into a fully cooperative multi-agent problem, then a spectrum allocation algorithm was proposed which was based on multi-agent proximal policy optimization under the framework of centralized training and distributed execution.Finally, the performance of the proposed algorithm was verified by simulation.The results show that the proposed algorithm has a faster convergence speed and can increase the overall transmission rate by 25.2% through effectively reducing intra-layer and inter-layer interference and balancing the access and backhaul link rates.

Key words: smart grid, integrated access and backhaul, spectrum allocation, multi-agent reinforcement learning

CLC Number:

TN92

Feng YAN, Xiaowei LIN, Zhenghao LI, Xia XU, Weiwei XIA, Lianfeng SHEN. Spectrum allocation algorithm based on multi-agent reinforcement learning in smart grid[J]. Journal on Communications, 2023, 44(9): 12-24.

Figures/Tables 11

References 30

[1]	DILEEP G . A survey on smart grid technologies and applications[J]. Renewable Energy, 2020,146: 2589-2625.
[2]	CHI Y Y , ZHANG Y , LIU Y ,et al. Deep reinforcement learning based edge computing network aided resource allocation algorithm for smart grid[J]. IEEE Access, 2022,11: 6541-6550.
[3]	ANDREWS J G , BUZZI S , CHOI W ,et al. What will 5G be？[J]. IEEE Journal on Selected Areas in Communications, 2014,32(6): 1065-1082.
[4]	KONG P Y . Radio resource allocation scheme for reliable demand response management using D2D communications in smart grid[J]. IEEE Transactions on Smart Grid, 2020,11(3): 2417-2426.
[5]	YANG J J , LIU G , REN J ,et al. Resource allocation for intelligent reflecting surface-assisted cooperative NOMA-URLLC networks in smart grid[C]// Proceedings of 2022 IEEE International Conference on Communications,Control,and Computing Technologies for Smart Grids. Piscataway:IEEE Press, 2022: 83-89.
[6]	CAO Z J , LIN J , WAN C ,et al. Optimal cloud computing resource allocation for demand side management in smart grid[J]. IEEE Transactions on Smart Grid, 2017,8(4): 1943-1955.
[7]	SUN M Y , YUAN Y Z , MA K ,et al. Spectrum allocation and computing resources optimization for demand-side cooperative communications in smart grid[J]. IEEE Transactions on Smart Grid, 2022,13(3): 1967-1975.
[8]	LI Z , LIANG Q L . Capacity optimization in heterogeneous home area networks with application to smart grid[J]. IEEE Transactions on Vehicular Technology, 2016,65(2): 699-706.
[9]	YIN F F , ZENG M Y , ZHANG Z L ,et al. Coded caching for smart grid enabled HetNets with resource allocation and energy cooperation[J]. IEEE Transactions on Vehicular Technology, 2020,69(10): 12058-12071.
[10]	LIU L L , ZHANG Z Z , WANG N ,et al. Online resource management of heterogeneous cellular networks powered by grid-connected smart micro grids[J]. IEEE Transactions on Wireless Communications, 2022,21(10): 8416-8430.
[11]	GUNGOR V C , SAHIN D , KOCAK T ,et al. A survey on smart grid potential applications and communication requirements[J]. IEEE Transactions on Industrial Informatics, 2013,9(1): 28-42.
[12]	3GPP. Study on integrated access and backhaul (release 16):TR 38.874[S]. 2018.
[13]	BELAID M N , AUDEBERT V , DENEUVILLE B ,et al. Smart grid critical traffic routing and link scheduling in 5G IAB networks[C]// Proceedings of 2022 IEEE International Conference on Communications,Control,and Computing Technologies for Smart Grid. Piscataway:IEEE Press, 2022: 76-82.
[14]	YASHIMA T , NISHIYAMA H . Analysis of optimal bandwidth partitioning ratio in full-duplex integrated access and backhaul[C]// Proceedings of 2022 IEEE International Conference on Communications Workshops. Piscataway:IEEE Press, 2022: 1-6.
[15]	ZHANG S M , XU X D , SUN M Y ,et al. Joint spectrum and power allocation in 5G integrated access and backhaul networks at mmWave band[C]// Proceedings of 2020 IEEE 31st Annual International Symposium on Personal,Indoor and Mobile Radio Communications. Piscataway:IEEE Press, 2020: 1-7.
[16]	PAGIN M , ZUGNO T , POLESE M ,et al. Resource management for 5G NR integrated access and backhaul:a semi-centralized approach[J]. IEEE Transactions on Wireless Communications, 2022,21(2): 753-767.
[17]	WANG X M , ZHANG Y H , SHEN R J ,et al. DRL-based energy-efficient resource allocation frameworks for uplink NOMA systems[J]. IEEE Internet of Things Journal, 2020,7(8): 7279-7294.
[18]	喻鹏, 张俊也, 李文璟 ,等. 移动边缘网络中基于双深度 Q 学习的高能效资源分配方法[J]. 通信学报, 2020,41(12): 148-161.
	YU P , ZHANG J Y , LI W J ,et al. Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning[J]. Journal on Communications, 2020,41(12): 148-161.
[19]	ZHU X Y , WANG J , LI J M ,et al. A scheme for uplink NOMA communication with intelligent resource allocation for mMTC traffic over eMBB traffic[C]// Proceedings of 2022 IEEE 95th Vehicular Technology Conference. Piscataway:IEEE Press, 2022: 1-5.
[20]	徐思雅, 邢逸斐, 郭少勇 ,等. 基于深度强化学习的能源互联网智能巡检任务分配机制[J]. 通信学报, 2021,42(5): 191-204.
	XU S Y , XING Y F , GUO S Y ,et al. Deep reinforcement learning based task allocation mechanism for intelligent inspection in energy Internet[J]. Journal on Communications, 2021,42(5): 191-204.
[21]	LEI W L , YE Y , XIAO M . Deep reinforcement learning-based spectrum allocation in integrated access and backhaul networks[J]. IEEE Transactions on Cognitive Communications and Networking, 2020,6(3): 970-979.
[22]	CHENG Q Q , WEI Z Q , YUAN J H . Deep reinforcement learning-based spectrum allocation and power management for IAB networks[C]// Proceedings of 2021 IEEE International Conference on Communications Workshops (ICC Workshops). Piscataway:IEEE Press, 2021: 1-6.
[23]	CHENG Z P , MIN M H , GAO Z B ,et al. Joint task offloading and resource allocation for mobile edge computing in ultra-dense network[C]// Proceedings of 2021 IEEE Global Communications Conference. Piscataway:IEEE Press, 2021: 1-6.
[24]	MENG F , CHEN P , WU L N ,et al. Power allocation in multi-user cellular networks:deep reinforcement learning approaches[J]. IEEE Transactions on Wireless Communications, 2020,19(10): 6255-6267.
[25]	周凡, 王鸿, 宋荣方 . 密集异构蜂窝网络中基于深度强化学习的下行链路功率分配算法[J]. 南京邮电大学学报(自然科学版), 2021,41(2): 12-19.
	ZHOU F , WANG H , SONG R F . Deep reinforcement learning based downlink power allocation algorithm in dense heterogeneous cellular networks[J]. Journal of Nanjing University of Posts and Telecommunications (Natural Science), 2021,41(2): 12-19.
[26]	KIM Y , LIM H . Multi-agent reinforcement learning-based resource management for end-to-end network slicing[J]. IEEE Access, 2021,9: 56178-56190.
[27]	GUO D L , TANG L , ZHANG X G ,et al. Joint optimization of handover control and power allocation based on multi-agent deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2020,69(11): 13124-13138.
[28]	YU C , VELU A , VINITSKY E ,et al. The surprising effectiveness of ppo in cooperative multi-agent games[J]. Advances in Neural Information Processing Systems, 2022,35: 24611-24624.
[29]	LOWE R , WU Y , TAMAR A ,et al. Multi-agent actor-critic for mixed cooperative-competitive environments[J]. arXiv Preprint,arXiv:1706.02275, 2017.
[30]	HOU W J , WEN H , SONG H H ,et al. Multiagent deep reinforcement learning for task offloading and resource allocation in cybertwin-based networks[J]. IEEE Internet of Things Journal, 2021,8(22): 16256-16268.

Metrics

Recommended 0

No Suggested Reading articles found!

算法	时间复杂度
平均分配	O(N(L+H)T _AA )
PSO	O(XNMT _PSO )
MAPPO、MADDPG	O(T _step N)

参数	设置值
每个基站信道总带宽/MHz	50
IAB节点路径损耗/dB	28+22lg(d)+20lg(f_c )
电力终端路径损耗/dB	32.4+211g(d)+20lg(f_c )
载波频率 f_c/GHz	2.4
IAB节点发射功率/dBm	33
电力终端发射功率/dBm	20
噪声功率谱密度/ (dBm·Hz^-1)	-174
自干扰/dB	-70
子信道衰落模型	瑞利衰落

Spectrum allocation algorithm based on multi-agent reinforcement learning in smart grid

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 30

Related Articles 15

Metrics

Recommended 0

[1]	Shengbao WANG, Xin ZHOU, Kang WEN, Bosen WENG. Tripartite authenticated key exchange protocol for smart grid [J]. Journal on Communications, 2023, 44(2): 210-218.
[2]	Wenjun XU, Silei WU, Fengyu WANG, Lan LIN, Guojun LI, Zhi ZHANG. Large-scale post-disaster user distributed coverage optimization based on multi-agent reinforcement learning [J]. Journal on Communications, 2022, 43(8): 1-16.
[3]	Xiaoqing DONG,Lianglun CHENG,Gengzhong ZHENG,Tao WANG. Spectrum resource allocation method of maximizing transmission rate in cognitive heterogeneous wireless networks [J]. Journal on Communications, 2019, 40(9): 124-135.
[4]	Min JIA,Xiaoye JING,Xiaofeng LIU,Feng LIU,Qing GUO,Xuemai GU. Spectrum allocation method for cognitive satellite network based on service priorities [J]. Journal on Communications, 2019, 40(4): 140-148.
[5]	Ya-nan JIA,Dian-wu YUE. Energy efficiency-based downlink resource allocation in cognitive small cell networks [J]. Journal on Communications, 2016, 37(4): 116-127.
[6]	Yue SHI,song QIUXue,yong GUOShao,Feng QI. Optimal planning of optical transmission network using improved genetic algorithm [J]. Journal on Communications, 2016, 37(1): 116-122.
[7]	Chi ZHANG,Bi-qing ZENG,Jin-song YANG,Xiao-hong XIE. Requirements-oriented spectrum sharing for OFDMA cognitive radio networks [J]. Journal on Communications, 2015, 36(8): 192-206.
[8]	Shun-fu JIN,Shi-ying GE,Zhan-qiang HUO. Dynamic spectrum allocation strategy with probabilistic feedback mechanism and performance analysis [J]. Journal on Communications, 2015, 36(7): 10-17.
[9]	Zhu XIAO,Ren-fa LI,Ke-chu YI,Jie ZHANG. Development and prospect of research on femtocell in two-tier heterogeneous networks [J]. Journal on Communications, 2013, 34(2): 156-169.
[10]	Shun-fu JIN,Hong-ting XIE,Yuan ZHAO. Hybrid channel allocation strategy with group switching and performance evaluation in cognitive radio network [J]. Journal on Communications, 2013, 34(12): 11-19.
[11]	Hua SHI,Jian-dong LI,Zhao LI. Dynamic spectrum allocation based on clone selection algorithm in cognitive heterogeneous wireless networks [J]. Journal on Communications, 2012, 33(7): 59-66.
[12]	Jun LIU,Xiu-feng XIE. Cognitive network channel allocation method based on the queuing delay and game analysis [J]. Journal on Communications, 2012, 33(6): 73-81.
[13]	Wen-feng DU,Ya-tao LIU,Zhong MING,Yin-xue SUI. Interference elimination based spectrum allocation algorithm for cognitive radio [J]. Journal on Communications, 2012, 33(5): 106-114.
[14]	Wen-zhu ZHANG,Ling-yun WANG. Dynamic spectrum allocation based on one-band multi-winner auction [J]. Journal on Communications, 2012, 33(2): 1-6.
[15]	Liang ZHAO,Liang JIN,Kai-zhi HUANG,Mei-yue YANG. Shared carrier vertical network transformation algorithm in heterogeneous wireless networks [J]. Journal on Communications, 2012, 33(1): 79-88.