基于多智能体强化学习的异构网络CRE偏置动态优化算法

doi:10.11959/j.issn.1000-436x.2023235

Abstract

Abstract:

To cope with the high throughput demand caused by the proliferation of wireless network users, a multi-agent reinforcement learning based dynamic optimization algorithm of cell range expansion (CRE) offset was proposed for interference scenarios in macro-pico heterogeneous networks.Based on the value decomposition network framework of collaborative multi-agent reinforcement learning, a personalized online local decision of CRE offset for all pico-base stations was achieved by reasonably utilizing and interacting the intra-system user distribution and their interference levels among pico-base stations.Simulation results show that the proposed algorithm has significant advantages in increasing system throughput, balancing the throughput of each base station and improving edge-user throughput, compared to CRE=5 dB and distributed Q-learning algorithms.

Key words: heterogeneous network, cell range expansion, multi-agent reinforcement learning, value decomposition network algorithm

CLC Number:

TN92

Cheng ZHANG, Jiaye ZHU, Zening LIU, Yongming HUANG. Multi-agent reinforcement learning based dynamic optimization algorithm of CRE offset for heterogeneous networks[J]. Journal on Communications, 2023, 44(12): 86-98.

Figures/Tables 12

References 28

[1]	CHUANG K , YEKTAII H , OUTALEB N ,et al. Towards sustainable networks:attacking energy consumption in wireless infrastructure with novel technologies[J]. IEEE Microwave Magazine, 2023,24(12): 44-59.
[2]	ELHOUSHY S , IBRAHIM M , HAMOUDA W . Cell-free massive MIMO:a survey[J]. IEEE Communications Surveys ＆ Tutorials, 2022,24(1): 492-523.
[3]	XU Y J , GUI G , GACANIN H ,et al. A survey on resource allocation for 5G heterogeneous networks:current research,future trends,and challenges[J]. IEEE Communications Surveys ＆ Tutorials, 2021,23(2): 668-695.
[4]	3GPP. Requirements for further advancements for evolved universal terrestrial radio access (E-UTRA) (LTE-advanced):TR 36.913[S]. 2011.
[5]	BIANZINO A P , CHAUDET C , ROSSI D ,et al. A survey of green networking research[J]. IEEE Communications Surveys ＆ Tutorials, 2012,14(1): 3-20.
[6]	JAMIL S , ABBAS M S , UMAIR M ,et al. A review of techniques and challenges in green communication[C]// Proceedings of International Conference on Information Science and Communication Technology (ICISCT). Piscataway:IEEE Press, 2020: 1-6.
[7]	DAMNJANOVIC A , MONTOJO J , WEI Y B ,et al. A survey on 3GPP heterogeneous networks[J]. IEEE Wireless Communications, 2011,18(3): 10-21.
[8]	ABBAS Z H , HAROON M S , MUHAMMAD F ,et al. Enabling soft frequency reuse and stienen’s cell partition in two-tier heterogeneous networks:cell deployment and coverage analysis[J]. IEEE Transactions on Vehicular Technology, 2021,70(1): 613-626.
[9]	LI J , WANG X M , LI Z Q ,et al. Energy efficiency optimization based on eICIC for wireless heterogeneous networks[J]. IEEE Internet of Things Journal, 2019,6(6): 10166-10176.
[10]	MICHEL D D E , ROGER F B A , GUTENBERT K W J . Performance evaluation of the eICIC technique applied to a heterogeneous 4G mobile network[J]. European Journal of Applied Sciences, 2022,10(2): 540-560.
[11]	TORRES-CRUZ N , VILLORDO-JIMENEZ I , MONTIEL-SAAVEDRA A . Analysis of the geographical-information impact on the performance of ABS-CRE HetNets[J]. IEEE Latin America Transactions, 2020,18(3): 613-622.
[12]	JUNG T , SONG I , LEE S ,et al. Cell range expansion with geometric information of pico-cell in heterogeneous networks[C]// Proceedings of IEEE 87th Vehicular Technology Conference (VTC Spring). Piscataway:IEEE Press, 2018: 1-5.
[13]	LEE C N , LIN J H , WU C F ,et al. A dynamic CRE and ABS scheme for enhancing network capacity in LTE-advanced heterogeneous networks[J]. Wireless Networks, 2019,25(6): 3307-3322.
[14]	成思玥, 李浩然, 白卫岗 ,等. 基于多智能体深度强化学习的测运控一体化资源调度方法[J]. 天地一体化信息网络, 2023,4(1): 12-22.
	CHENG S Y , LI H R , BAI W G ,et al. Resource scheduling method for integration of TT＆C and observation based on multi-agent deep reinforcement learning[J]. Space-Integrated-Ground Information Networks, 2023,4(1): 12-22.
[15]	张彪, 汪西明, 徐逸凡 ,等. 基于多智能体深度强化学习的多域协同抗干扰方法研究[J]. 物联网学报, 2022,6(4): 104-116.
	ZHANG B , WANG X M , XU Y F ,et al. Multi-domain collaborative anti-jamming based on multi-agent deep reinforcement learning[J]. Chinese Journal on Internet of Things, 2022,6(4): 104-116.
[16]	丁雨, 李晨凯, 韩会梅 ,等. 基于5G无人机通信的多智能体异构网络选择方法[J]. 电信科学, 2022,38(8): 28-36.
	DING Y , LI C K , HAN H M ,et al. Multi-agent heterogeneous network selection method based on 5G UAV communication[J]. Telecommunications Science, 2022,38(8): 28-36.
[17]	CHOI H , KIM T , PARK H S ,et al. A cooperative online learning-based load balancing scheme for maximizing QoS satisfaction in dense HetNets[J]. IEEE Access, 2021,9: 92345-92357.
[18]	ALSUHLI G , BANAWAN K , ATTIAH K ,et al. Mobility load management in cellular networks:a deep reinforcement learning approach[J]. IEEE Transactions on Mobile Computing, 2023,22(3): 1581-1598.
[19]	KUDO T , OHTSUKI T . Cell range expansion using distributed Q-learning in heterogeneous networks[J]. EURASIP Journal on Wireless Communications and Networking, 2013(1): 1-10.
[20]	ASGHARI M Z , OZTURK M , HAMALAINEN J . Reinforcement learning based mobility load balancing with the cell individual offset[C]// Proceedings of IEEE 93rd Vehicular Technology Conference (VTC2021-Spring). Piscataway:IEEE Press, 2021: 1-5.
[21]	TABUCHI S , MAKINO I , MIKI N . Combined usage of convex optimization and neural network for resource allocation[C]// Proceedings of 14th International Conference on Signal Processing and Communication Systems (ICSPCS). Piscataway:IEEE Press, 2020: 1-6.
[22]	MATIGNON L , LAURENT G J , LE FORT-PIAT N . Independent reinforcement learners in cooperative Markov games:a survey regarding coordination problems[J]. The Knowledge Engineering Review, 2012,27(1): 1-31.
[23]	SUNEHAG P , LEVER G , GRUSLYS A ,et al. Value-decomposition networks for cooperative multi-agent learning based on team reward[C]// Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. New York:ACM Press, 2018: 2085-2087.
[24]	DAI Y Y , ZHANG K , MAHARJAN S ,et al. Deep reinforcement learning for stochastic computation offloading in digital twin networks[J]. IEEE Transactions on Industrial Informatics, 2021,17(7): 4968-4977.
[25]	FERIANI A , HOSSAIN E . Single and multi-agent deep reinforcement learning for AI-enabled wireless networks:a tutorial[J]. IEEE Communications Surveys ＆ Tutorials, 2021,23(2): 1226-1252.
[26]	WANG H N , LIU N , ZHANG Y Y ,et al. Deep reinforcement learning:a survey[J]. Frontiers of Information Technology ＆ Electronic Engineering, 2020,21(12): 1726-1744.
[27]	TABISH R , MIKAYEL S , SCHROEDER D W C ,et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Journal of Machine Learning Research, 2020,21(1): 7234-7284.
[28]	CASTELLINI J , OLIEHOEK F A , SAVANI R ,et al. The representational capacity of action-value networks for multi-agent reinforcement learning[C]// Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. New York:ACM Press, 2019: 1862-1864.

Metrics

Recommended 0

No Suggested Reading articles found!

参数	取值
参数	MBS	PBS
小区半径/m	289	40
发射功率/dBm	46	30
距离相关路径损耗/dB	140.7+36.7lgR	128.1+36.7lgR
载波频率/GHz	2
带宽/MHz	20
热噪声密度/(dBm·Hz^-1)	-174
信道衰落	瑞利衰落
偏置值范围/dB	{1,2,3,4,5,6,7,8,9}
SINR阈值/dB	15
MBS与PBS之间最小距离/m	75
PBS与PBS之间最小距离/m	40
MBS与用户之间最小距离/m	35
PBS与用户之间最小距离/m	10

参数	取值
学习率r_a	0.000 5
折扣因子r_d	0.88
奖励调节因子μ	65
贪婪策略初始值ε₀	0.5
每轮最大相同步骤数	30
最大训练轮数	500
最大训练步骤数	100
经验池容量	10 000
小批量数据	32
隐藏层1神经元数量	32
隐藏层2神经元数量	32

Multi-agent reinforcement learning based dynamic optimization algorithm of CRE offset for heterogeneous networks

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 28

Related Articles 15

Metrics

Recommended 0

[1]	Feng YAN, Xiaowei LIN, Zhenghao LI, Xia XU, Weiwei XIA, Lianfeng SHEN. Spectrum allocation algorithm based on multi-agent reinforcement learning in smart grid [J]. Journal on Communications, 2023, 44(9): 12-24.
[2]	Yutong ZHANG, Yuming PENG, Boya DI, Lingyang SONG. Heterogeneous resource cooperative game in space-ground computing power network [J]. Journal on Communications, 2023, 44(12): 15-27.
[3]	Bingyi LIU, Yuhao LIU, Weizhen HAN, Zhenchang XIA, Libing WU, Shengwu XIONG. Edge intelligence-assisted routing protocol for Internet of vehicles via reinforcement learning [J]. Journal on Communications, 2023, 44(11): 110-119.
[4]	Wenjun XU, Silei WU, Fengyu WANG, Lan LIN, Guojun LI, Zhi ZHANG. Large-scale post-disaster user distributed coverage optimization based on multi-agent reinforcement learning [J]. Journal on Communications, 2022, 43(8): 1-16.
[5]	Rong QIAN, Jianting XU, Kejun ZHANG, Hongyu DONG, Fangyuan XING. Research on HMM based link prediction method in heterogeneous network [J]. Journal on Communications, 2022, 43(5): 214-225.
[6]	Hongbin LUO, Shan ZHANG, Zhiyuan WANG. Architecture and mechanisms for secure and efficient internetworking of heterogeneous network [J]. Journal on Communications, 2022, 43(4): 36-49.
[7]	Li YANG, Chengsheng PAN, Xiangguang KONG, Qilong HUANG, Yaowen QI. Review on 5G-satellite integrated network [J]. Journal on Communications, 2022, 43(4): 202-215.
[8]	Yang CAO, Ye ZHONG, Chunling PENG, Xiaofeng PENG. Energy efficiency optimization algorithm of heterogeneous networks based on hybrid energy supply and energy cooperation [J]. Journal on Communications, 2022, 43(3): 135-147.
[9]	Xianhao SHEN, Ziling ZENG, Shaohua NIU. Reconfigurable intelligent surface assisted resource optimization method for heterogeneous network [J]. Journal on Communications, 2022, 43(11): 171-182.
[10]	Xue WANG, Jing LIU, Jiani SUN, Jizhen ZHANG, Zhihong QIAN. Spectral clustering-based energy-efficient resource allocation algorithm in heterogeneous cellular ultra-dense network [J]. Journal on Communications, 2021, 42(7): 162-175.
[11]	Yongjun XU,Hao XIE,Qianbin CHEN,Jinzhao LIN,Qilie LIU. Energy efficiency optimization algorithm for heterogeneous NOMA network based on imperfect CSI [J]. Journal on Communications, 2020, 41(7): 131-140.
[12]	Haijun ZHANG,Zizheng ZHANG,Keping LONG. Resource allocation in NOMA heterogeneous network based on MEC [J]. Journal on Communications, 2020, 41(4): 27-33.
[13]	Yaqiong QIAO, Xiangyang LUO, Jiangtao MA, Chenliang LI, Meng ZHANG, Ruixiang LI. Social media user geolocalization based on multiple mention relationships [J]. Journal on Communications, 2020, 41(12): 72-81.
[14]	Haitao ZHAO,Yue CHEN,Tangwei ZHANG,Qixing ZHU,Hongbo ZHU. Optimization of multipath transmission path scheduling based on forward delay in vehicular heterogeneous network [J]. Journal on Communications, 2020, 41(11): 99-107.
[15]	Bo ZHANG,Kaizhi HUANG,Zhou ZHONG,Yajun CHEN. Artificial noise-aided robust secure information and power transmission scheme in heterogeneous networks with simultaneous wireless information and power transfer [J]. Journal on Communications, 2019, 40(3): 60-72.