天地算力网络中的异构资源协同博弈

doi:10.11959/j.issn.1000-436x.2023214

通信学报 ›› 2023, Vol. 44 ›› Issue (12): 15-27.doi: 10.11959/j.issn.1000-436x.2023214

• 专题：空天地一体化网络频谱博弈关键技术 • 上一篇

天地算力网络中的异构资源协同博弈

张雨童¹, 彭煜明¹, 邸博雅¹, 宋令阳¹^,²

¹ 北京大学区域光纤通信网与新型光通信系统国家重点实验室，北京 100871
² 北京大学深圳研究生院信息工程学院，广东深圳 518055

修回日期:2023-12-12 出版日期:2023-12-01 发布日期:2023-12-01
作者简介:张雨童（1995- ），女，山东德州人，北京大学博士生，主要研究方向为可重构智能超表面码本设计等
彭煜明（1999- ），男，湖南株洲人，北京大学博士生，主要研究方向为超材料传感器设计等
邸博雅（1992- ），女，黑龙江大庆人，博士，北京大学助理教授，主要研究方向为无线通信、边缘计算、车载网络、智能反射面和非正交多址接入等
宋令阳（1979- ），男，辽宁抚顺人，博士，北京大学教授，主要研究方向为无线通信和网络、MIMO、OFDMA以及信号处理和机器学习等
基金资助:
国家重点研发计划基金资助项目(2022YFE0111900);湖南省科技创新计划基金资助项目(2022RC4024);国家自然科学基金资助项目(62227809);国家自然科学基金资助项目(61931019);国家自然科学基金资助项目(62271012);北京市自然科学基金资助项目(L212027);北京市自然科学基金资助项目(4222005)

Heterogeneous resource cooperative game in space-ground computing power network

Yutong ZHANG¹, Yuming PENG¹, Boya DI¹, Lingyang SONG¹^,²

¹ State Key Laboratory of Advanced Optical Communication Systems and Networks, Peking University, Beijing 100871, China
² School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School, Shenzhen 518055, China

Revised:2023-12-12 Online:2023-12-01 Published:2023-12-01
Supported by:
The National Key Research and Development Program of China(2022YFE0111900);The Science and Technology Innovation Program of Hunan Province(2022RC4024);The National Natural Science Foundation of China(62227809);The National Natural Science Foundation of China(61931019);The National Natural Science Foundation of China(62271012);The Beijing Natural Science Foundation(L212027);The Beijing Natural Science Foundation(4222005)

摘要/Abstract

摘要：

为解决多卫星天地算力网络中的星间资源博弈，围绕计算、频谱域资源管理问题，设计了一种天地异构资源协同博弈机制。每颗卫星搭载一项计算任务，各任务间彼此独立，依赖用户设备从环境中获取原始数据，通过竞争网络中的计算/频谱资源实现数据卸载与计算。为提供高速数据服务，提出基于多智能体强化学习的分布式算法，以协调星间异构资源竞争，实现系统时延最小化。仿真表明，与现有方案相比，所提算法可获得更低的系统时延。

关键词: 天地算力网络, 异构资源协同博弈, 多智能体强化学习

Abstract:

To deal with the resource competition among satellites in the multi-satellite space-ground computing network, a space-ground heterogeneous resource cooperative game mechanism was designed in terms of the computing and spectrum domains.Each satellite published a computing task which was independent of other tasks and relied on UE to generate raw data.By competing the resources of user terminals and UE, the task offloading and processing was achieved.To provide real-time data services, a distributed scheme was proposed based on multi-agent reinforcement learning to coordinate the computing and spectrum resource competition among satellites, thereby minimizing the system latency.Simulation results indicated that, compared with the existing schemes, the proposed algorithm achieves a lower system latency by fully utilizing the computing and spectrum resources and coordinating the resource competition.

Key words: space-ground computing power network, heterogeneous resource cooperative game, multi-agent reinforcement learning

中图分类号:

TN927

张雨童, 彭煜明, 邸博雅, 宋令阳. 天地算力网络中的异构资源协同博弈[J]. 通信学报, 2023, 44(12): 15-27.

Yutong ZHANG, Yuming PENG, Boya DI, Lingyang SONG. Heterogeneous resource cooperative game in space-ground computing power network[J]. Journal on Communications, 2023, 44(12): 15-27.

图/表 16

表1

符号定义"

符号	定义
L	卫星总数
N	TST总数
M	每个TST下的UE总数
l^n,m	UE m-TST n-卫星l链路上的子任务
$λ_{UE}^{n, m} (l)$	子任务l^n,m的数据产生速率
$x_{TST}^{n, m} (l)$	对于子任务l^n,m，在TST n上计算的原始数据占其接收到的原始数据总数的百分比
$x_{UE}^{n, m} (l)$	对于子任务l^n,m，在UE m上计算的原始数据占其产生的原始数据总数的百分比
$ϕ_{TST}^{n, m} (l)$	子任务l^n,m的卫星-TST传输资源
$ϕ_{UE}^{n, m} (l)$	子任务l^n,m的TST-UE传输资源
$θ_{SAT}^{n, m} (l)$	子任务l^n,m的卫星计算资源
$θ_{TST}^{n, m} (l)$	子任务l^n,m的TST计算资源
$θ_{UE}^{n, m} (l)$	子任务l^n,m的UE计算资源
$t_{SAT}^{n, m} (l)$	子任务l^n,m在卫星上的数据计算时间
$t_{TST}^{n, m} (l)$	子任务l^n,m在TST上的数据计算时间
$t_{UE}^{n, m} (l)$	子任务l^n,m在UE上的数据计算时间
$τ_{TST}^{n, m} (l)$	子任务l^n,m从TST到卫星的数据传输时间
$τ_{UE}^{n, m} (l)$	子任务l^n,m从UE到TST的数据传输时间
$T^{n, m} (l)$	子任务l^n,m的时延
T (l)	任务l的时延
T	系统时延
ρ	数据压缩率

表1

图1

图2

图3

表2

图4

图5

图6

表3

表4

表5

表6

图7

图8

表7

动作空间缩减"

缩减原则	占 $\| A \|$ 的比例
x_UE=1且x_TST≠0	$\frac{1}{d} (1 - \frac{1}{d})$
x_UE=0且θ_UE≠0	$\frac{1}{d} (1 - \frac{1}{c})$
x_TST=0且θ_TST≠0	$\frac{1}{d} (1 - \frac{1}{c})$
x_TST=1且θ_SAT≠0	$\frac{1}{d} (1 - \frac{1}{c})$

表7

表8

缩减原则重叠部分"

重叠部分	占 $\| A \|$ 的比例
x_UE=1，x_TST=1且θ_SAT≠0	$\frac{1}{d} \frac{1}{d} (1 - \frac{1}{c})$
x_UE=0，x_TST=0，θ_UE≠0且θ_TST≠0	${[\frac{1}{d} (1 - \frac{1}{c})]}^{2}$
x_UE=0，x_TST=1，θ_UE=0且θ_TST≠0	${[\frac{1}{d} (1 - \frac{1}{c})]}^{2}$

表8

参考文献 30

[1]	DI B Y , ZHANG H L , SONG L Y ,et al. Ultra-dense LEO:integrating terrestrial-satellite networks into 5G and beyond for data offloading[J]. IEEE Transactions on Wireless Communications, 2019,18(1): 47-62.
[2]	SU Y T , LIU Y Q , ZHOU Y Q ,et al. Broadband LEO satellite communications:architectures and key technologies[J]. IEEE Wireless Communications, 2019,26(2): 55-61.
[3]	胡馨元, 邓若琪, 邸博雅 ,等. 可重构全息超表面辅助卫星通信关键技术[J]. 电信科学, 2022,38(10): 46-56.
	HU X Y , DENG R Q , DI B Y ,et al. Key technologies of satellite communications aided by reconfigurable holographic surfaces[J]. Telecommunications Science, 2022,38(10): 46-56.
[4]	VOIGT S , GIULIO-TONOLO F , LYONS J ,et al. Global trends in satellite-based emergency mapping[J]. Science, 2016,353(6296): 247-252.
[5]	王鹏飞, 邸博雅, 唐斌 ,等. 面向广域海洋覆盖的密集低轨卫星星座[J]. 无线电通信技术, 2021,47(4): 402-409.
	WANG P F , DI B Y , TANG B ,et al. Ultra?dense LEO satellite constellation design for wide ocean coverage area[J]. Radio Communications Technology, 2021,47(4): 402-409.
[6]	ZHOU Y , LIU L , WANG L ,et al. Service-aware 6G:an intelligent and open network based on the convergence of communication,computing and caching[J]. Digital Communications and Networks, 2020,6(3): 253-260.
[7]	ZHOU Y Q , TIAN L , LIU L ,et al. Fog computing enabled future mobile communication networks:a convergence of communication and computing[J]. IEEE Communications Magazine, 2019,57(5): 20-27.
[8]	LIAO X L , HU X , LIU Z J ,et al. Distributed intelligence:a verification for multi-agent DRL-based multibeam satellite resource allocation[J]. IEEE Communications Letters, 2020,24(12): 2785-2789.
[9]	LIU R , GUO K F , AN K ,et al. Resource allocation for NOMA-enabled cognitive satellite-UAV-terrestrial networks with imperfect CSI[J]. IEEE Transactions on Cognitive Communications and Networking, 2023,9(4): 963-976.
[10]	WANG W L , WEI J , ZHAO S H ,et al. Energy efficiency resource allocation based on spectrum-power tradeoff in distributed satellite cluster network[J]. Wireless Networks, 2020,26(6): 4389-4402.
[11]	ZHANG Y D , YIN L G , JIANG C X ,et al. Joint beamforming design and resource allocation for terrestrial-satellite cooperation system[J]. IEEE Transactions on Communications, 2019,68(2): 778-791.
[12]	ZHANG S H , CUI G F , LONG Y T ,et al. Joint computing and communication resource allocation for satellite communication networks with edge computing[J]. China Communications, 2021,18(7): 236-252.
[13]	FU S , GAO J , ZHAO L . Collaborative multi-resource allocation in terrestrial-satellite network towards 6G[J]. IEEE Transactions on Wireless Communications, 2021,20(11): 7057-7071.
[14]	DING C F , WANG J B , CHENG M ,et al. Dynamic transmission and computation resource optimization for dense LEO satellite assisted mobile-edge computing[J]. IEEE Transactions on Communications, 2023,71(5): 3087-3102.
[15]	DENG R Q , DI B Y , CHEN S Z ,et al. Ultra-dense LEO satellite offloading for terrestrial networks:how much to pay the satellite operator?[J]. IEEE Transactions on Wireless Communications, 2020,19(10): 6240-6254.
[16]	东方星 . 我国高分卫星与应用简析[J]. 卫星应用, 2015(3): 44-48.
	DONGFANG X . Analysis of high score satellite and its application in China[J]. Satellite Application, 2015(3): 44-48.
[17]	白徐祥, 官山林 . 动中通卫星通信天线[J]. 无线通信技术, 2004,13(1): 26-28,32.
	BAI X X , GUAN S L . Antenna design for satellite communication in motion[J]. Wireless Communication Technology, 2004,13(1): 26-28,32.
[18]	GAO X Q , LIU R K , KAUSHIK A ,et al. Dynamic resource allocation for virtual network function placement in satellite edge clouds[J]. IEEE Transactions on Network Science and Engineering, 2022,9(4): 2252-2265.
[19]	REN J K , YU G D , CAI Y L ,et al. Latency optimization for resource allocation in mobile-edge computation offloading[J]. IEEE Transactions on Wireless Communications, 2018,17(8): 5506-5519.
[20]	ZHANG Y T , DI B Y , ZHENG Z J ,et al. Distributed multi-cloud multi-access edge computing by multi-agent reinforcement learning[J]. IEEE Transactions on Wireless Communications, 2021,20(4): 2565-2578.
[21]	ZHAO N , YE Z Y , PEI Y Y ,et al. Multi-agent deep reinforcement learning for task offloading in UAV-assisted mobile edge computing[J]. IEEE Transactions on Wireless Communications, 2022,21(9): 6949-6960.
[22]	XU J , CHEN L X , REN S L . Online learning for offloading and autoscaling in energy harvesting mobile edge computing[J]. IEEE Transactions on Cognitive Communications and Networking, 2017,3(3): 361-373.
[23]	CLAUS C , BOUTILIER C . The dynamics of reinforcement learning in cooperative multiagent systems[C]// Proceedings of the Fifteenth National/Tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence. New York:ACM Press, 1998: 746-752.
[24]	TEYMOORI P , BOUKERCHE A . Dynamic multi-user computation offloading for mobile edge computing using game theory and deep reinforcement learning[C]// Proceedings of IEEE International Conference on Communications. Piscataway:IEEE Press, 2022: 1930-1935.
[25]	GOMES E R , KOWALCZYK R . Dynamic analysis of multiagent Q-learning with ε-greedy exploration[C]// Proceedings of the 26th Annual International Conference on Machine Learning. New York:ACM Press, 2009: 369-376.
[26]	SADHU A K , KONAR A . An efficient computing of correlated equilibrium for cooperative Q-learning-based multi-robot planning[J]. IEEE Transactions on Systems,Man,and Cybernetics:Systems, 2020,50(8): 2779-2794.
[27]	JU Y , CHEN Y C , CAO Z W ,et al. Joint secure offloading and resource allocation for vehicular edge computing network:a multi-agent deep reinforcement learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2023,24(5): 5555-5569.
[28]	WAQAR N , HASSAN S A , MAHMOOD A ,et al. Computation offloading and resource allocation in MEC-enabled integrated aerial-terrestrial vehicular networks:a reinforcement learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2022,23(11): 21478-21491.
[29]	DENG R Q , DI B Y , ZHANG H L ,et al. Ultra-dense LEO satellite constellations:how many LEO satellites do we need[J]. IEEE Transactions on Wireless Communications, 2021,20(8): 4843-4857.
[30]	WANG P F , DI B Y , SONG L Y . Mega-constellation design for integrated satellite-terrestrial networks for global seamless connectivity[J]. IEEE Wireless Communications Letters, 2022,11(8): 1669-1673.

参数	值
每UE CPU频率/(cycle·s^-1)	3×10¹
每TST CPU频率/(cycle·s^-1)	8×10¹⁰
每卫星 CPU频率/(cycle·s^-1)	2.4×10¹⁰
处理每比特数据所需CPU周期/(cycle·bit^-1)	100
TST-UE链路带宽/MHz	20
TST-UE链路子信道数量	500
TST-UE链路发射功率/dBm	43
TST-UE链路发射天线增益/dBi	31.2
TST-UE链路噪声/(dBm·Hz^-1)	-120
卫星-TST链路带宽/MHz	100
卫星-TST链路子信道数量	16
卫星-TST链路发射功率/W	2
卫星-TST链路发射天线增益/dBi	37.1
卫星-TST链路噪声/(dBm·Hz^-1)	-174
数据压缩率	10%
折扣率	0.9
初始学习率α₀	0.15
初始探索率ε₀	0.1

卫星数量/颗	中心式算法/s	SHRCG/s	卫星计算/s	本地计算/s
1	1.171 8	1.178 5	1.210 0	1.502 6
2	1.336 4	1.339 3	1.410 0	2.004 6
3	1.492 6	1.503 5	1.610 0	2.506 6

TST数量/个	中心式算法/s	SHRCG/s	卫星计算/s	本地计算/s
1	1.436 4	1.439 3	1.510 0	2.104 6
2	1.443 6	1.451 0	1.520 0	2.105 2
3	1.450 7	1.459 3	1.530 0	2.105 8

UE数量/个	中心式算法/s	SHRCG/s	卫星计算/s	本地计算/s
1	1.336 4	1.339 3	1.410 0	2.004 6
2	1.554 6	1.574 4	1.820 0	2.009 2
3	1.663 9	1.669 6	2.230 0	2.013 8

设备数量	卫星/次	TST/次	UE/次
1	3.1×10⁴	4.9×10⁴	4.9×10⁴
2	4.9×10⁴	5.3×10⁴	5.8×10⁴
3	5.3×10⁵	6.1×10⁴	5.5×10⁴

天地算力网络中的异构资源协同博弈

Heterogeneous resource cooperative game in space-ground computing power network

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 30

相关文章 3

Metrics

推荐阅读 0

[1]	燕锋, 林晓薇, 李正浩, 徐霞, 夏玮玮, 沈连丰. 智能电网中基于多智能体强化学习的频谱分配算法[J]. 通信学报, 2023, 44(9): 12-24.
[2]	刘冰艺, 刘煜昊, 韩玮祯, 夏振厂, 吴黎兵, 熊盛武. 边缘智能下基于强化学习的车联网路由协议[J]. 通信学报, 2023, 44(11): 110-119.
[3]	许文俊, 吴思雷, 王凤玉, 林兰, 李国军, 张治. 基于多智能体强化学习的大规模灾后用户分布式覆盖优化[J]. 通信学报, 2022, 43(8): 1-16.