基于CBR与合作Q学习的分布式CRN资源分配算法

doi:10.11959/j.issn.1000-0801.2019005

电信科学 ›› 2019, Vol. 35 ›› Issue (2): 35-42.doi: 10.11959/j.issn.1000-0801.2019005

基于CBR与合作Q学习的分布式CRN资源分配算法

徐琳,赵知劲

杭州电子科技大学通信工程学院，浙江杭州 310018

修回日期:2018-11-30 出版日期:2019-02-01 发布日期:2019-02-23
作者简介:徐琳（1994- ），女，杭州电子科技大学通信工程学院硕士生，主要研究方向为认知无线电、信号处理。|赵知劲（1959- ），女，博士，杭州电子科技大学教授、博士生导师，主要研究方向为认知无线电、通信信号处理和自适应信号处理等。
基金资助:
“十二五”国防预研项目(41001010401)

A distributed CRN resource allocation algorithm based on CBR and cooperative Q-learning

Lin XU,Zhijin ZHAO

School of Communication Engineering,Hangzhou Dianzi University,Hangzhou 310018,China

Revised:2018-11-30 Online:2019-02-01 Published:2019-02-23
Supported by:
The National Defense Pre-Research Project During the Twelfth Five-year Plan Period(41001010401)

摘要/Abstract

摘要：

针对分布式认知无线电网络的信道和功率分配问题，提出一种基于案例推理与合作的Q学习算法。为了优化Q学习算法的Q初始化，将当前问题和历史案例依据相似度函数进行匹配，提取匹配案例的Q值并在归一化后作为初始值，进行合作Q学习。合作Q学习是基于总奖赏值进行的，各Agent以不同权值融合其他具有更高奖赏值的Agent的Q值来获取学习经验，以减少不必要的探索。仿真结果表明，该算法提高了认知系统信道和功率分配的能量效率，加快了系统的收敛速度。

关键词: 认知无线电, 合作Q学习, 案例推理, 信道和功率分配, 能量效率, 收敛速度

Abstract:

In order to solve the problem of channel and power allocation in distributed cognitive radio networks (CRN),a case-based reasoning (CBR) and cooperative Q-learning algorithm was proposed.In order to optimize the Q initialization of Q-learning algorithm,the current problem and the historical case were matched according to the similarity function,the Q value of the matching case was extracted and normalized as the initial value.Cooperative Q-learning was based on the total reward value,and each agent integrates the Q values of other agents with higher reward values with different weights to gain learning experience to reduce unnecessary exploration.Simulations show that the proposed algorithm can improve the energy efficiency of the cognitive system’s channel and power allocation,and accelerate the convergence speed of the system.

Key words: cognitive radio, cooperative Q-learning, case-based reasoning, channel and power allocation, energy efficiency, convergence speed

中图分类号:

TN925

徐琳,赵知劲. 基于CBR与合作Q学习的分布式CRN资源分配算法[J]. 电信科学, 2019, 35(2): 35-42.

Lin XU,Zhijin ZHAO. A distributed CRN resource allocation algorithm based on CBR and cooperative Q-learning[J]. Telecommunications Science, 2019, 35(2): 35-42.

图/表 10

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

参考文献 13

[1]	罗骥 . 基于自适应中继的认知协作网络功率分配[J]. 计算机工程, 2017,43(7): 151-155.
	LUO J . Cognitive cooperative network power allocation based on adaptive relay[J]. Computer Engineering, 2017,43(7): 151-155.
[2]	HE J , PENG J , JIANG F ,et al. A distributed Q learning spectrum decision scheme for cognitive radio sensor network[J]. International Journal of Distributed Sensor Networks, 2015: 1-10.
[3]	MOROZS N , CLARKE T , GRACE D . Distributed heuristically accelerated Q-Learning for robust cognitive spectrum management in LTE cellular systems[J]. IEEE Transactions on Mobile Computing, 2016,15(4): 817-825.
[4]	李晓静 . 基于强化学习的动态频谱分配算法研究[D]. 南京:南京邮电大学, 2011.
	LI X J . The study of the dynamic spectrum allocation algorithm based on reinforcement learning[D]. Nanjing:Nanjing University of Posts and Telecommunications, 2011.
[5]	BOUMEDIENE L , ZGEN G G , LIU S . Distributed multi-agent Q-learning for joint channel allocation and power control in cognitive radio networks[J]. Journal of Computational Information Systems, 2012,8(17): 7071-7078.
[6]	MOROZS N , CLARKE T , GRACE D . Distributed Q-learning based dynamic spectrum management in cognitive cellular systems:choosing the right learning rate[C]// 2014 IEEE Symposium on Computers and Communication,June 23-26,2014,Funchal,Portugal. Piscataway:IEEE Press, 2014: 1-10.
[7]	CHO B , KIM K J , CHUNG J W . CBR-based network performance management with multi-agent approach[J]. Cluster Computing, 2017,20(1): 1-11.
[8]	张文柱, 周雪婷, 刘玉琦 . 认知无线电 NC-OFDM 中基于案例推理的无线资源分配[J]. 移动通信, 2017,41(14): 82-88.
	ZHANG W Z , ZHOU X T , LIU Y Q . Radio resource allocation based on case-reasoning in NC-OFDM systems for cognitive radio[J]. Mobile Communication, 2017,41(14): 82-88.
[9]	冯陈伟, 袁江南 . 基于强化学习的异构无线网络资源管理算法[J]. 电信科学, 2015,31(8): 99-106.
	FENG C W , YUAN J N . Heterogeneous wireless network resource management algorithm based on reinforcement learning[J]. Telecommunications Science, 2015,31(8): 99-106.
[10]	CHI C , ZHANG Q Z , BO X et al . A multi-agent reinforcement learning algorithm based on Stackelberg game[C]// IEEE 6th Data Driven Control and Learning Systems Conference,May 26-27,2017,Chongqing,China. Piscataway:IEEE Press, 2017: 727-732.
[11]	赖海超, 赵知劲, 郑仕链 . 应用案例推理技术的快速认知引擎[J]. 信号处理, 2012,28(12): 1700-1705.
	LAI H C , ZHAO Z J , ZHENG S L . Fast cognitive engine using case-based reasoning[J]. Signal Processing, 2012,28(12): 1700-1705.
[12]	LALL S , SADHU A K , AMIT K . Multi-agent reinforcement learning for stochastic power management in cognitive radio network[C]// International Conference on Microelectronics,Computing and Communications,Jan 23-25,2016,Durgapur,India. Piscataway:IEEE Press, 2016: 1-6.
[13]	曾庆瑾 . 认知无线电动态频谱的分配和接入问题研究[D]. 南宁:广西大学, 2015.
	ZENG Q J . Study on the allocation and access of the dynamic frequency spectrum for cognitive radio[D]. Nanning:Guangxi University, 2015.

[1]	蒋跃宇, 承昊新, 王康, 戴文骏, 刘笑宇. PLC-RF无线传输网络中的能效最优设计方法[J]. 电信科学, 2023, 39(4): 111-119.
[2]	马巧巧, 董黎刚, 蒋献. 面向SDWSN的分布式高效熵节能分簇路由算法[J]. 电信科学, 2023, 39(3): 100-114.
[3]	葛海江, 贾宁, 池凯凯, 陈云志. 非线性能量收集认知无线电网络的次用户吞吐量最大化方案[J]. 电信科学, 2023, 39(2): 103-117.
[4]	贾忠杰, 金明, 宋晓群. 一种基于黏性隐马尔可夫模型的多频带频谱感知方法[J]. 电信科学, 2021, 37(1): 48-57.
[5]	葛海江, 俞湛威, 池凯凯. 无线供能网络中数据收集的能量效率最大化方案[J]. 电信科学, 2019, 35(12): 49-56.
[6]	肖洁,陈跃斌,陈楚天,郑婷,钱继武. 基于信任度的可变门限能量检测算法[J]. 电信科学, 2018, 34(8): 129-135.
[7]	邓宏宇,汪一鸣,吴澄. 基于MRR调度的认知LTE-R基站和服务类型值优先队列管理[J]. 电信科学, 2018, 34(6): 80-90.
[8]	王云川,许晓荣,姚英彪,伍伟伟. 一种能效优先的认知无线电模仿主用户攻击防御策略设计与性能分析[J]. 电信科学, 2017, 33(8): 100-106.
[9]	葛亮,王志旭. 能量高效超密集网络的最优部署[J]. 电信科学, 2017, 33(3): 44-51.
[10]	李汀,仇林杰,季薇. 3D MIMO-OFDMA系统中基于垂直波束成形的能效优化算法[J]. 电信科学, 2017, 33(11): 17-26.
[11]	李晓宇,杨守义,赵晓娟,张瑞哲. 基于多主用户活跃性的多功率认知接入策略[J]. 电信科学, 2017, 33(10): 50-57.
[12]	卢光跃,苏杭. 分布式协作认知无线电SSDF攻击的防御策略综述[J]. 电信科学, 2017, 33(1): 95-105.
[13]	许蒙迪,金明,童景文. 一种基于切换失败概率和认知用户信道容量联合优化的访问策略[J]. 电信科学, 2016, 32(9): 82-88.
[14]	赵知劲,吕曦,郑仕链. 基于功率谱密度中段平均的频谱感知算法[J]. 电信科学, 2016, 32(7): 53-60.
[15]	卢光跃,徐偲,叶迎晖,弥寅. 基于拟合优度的频谱感知技术研究[J]. 电信科学, 2016, 32(5): 52-61.

基于CBR与合作Q学习的分布式CRN资源分配算法

A distributed CRN resource allocation algorithm based on CBR and cooperative Q-learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 13

相关文章 15

Metrics

推荐阅读 0