基于Q-learning的HTTP自适应流码率控制方法研究

doi:10.11959/j.issn.1000-436x.2017178

通信学报 ›› 2017, Vol. 38 ›› Issue (9): 18-24.doi: 10.11959/j.issn.1000-436x.2017178

基于Q-learning的HTTP自适应流码率控制方法研究

熊丽荣,雷静之,金鑫

浙江工业大学计算机科学与技术学院，浙江杭州 310023

修回日期:2017-05-22 出版日期:2017-09-01 发布日期:2017-10-18
作者简介:熊丽荣（1973-），女，湖北崇阳人，浙江工业大学副教授，主要研究方向为服务计算和计算机网络。|雷静之（1992-），女，江西抚州人，浙江工业大学硕士生，主要研究方向为服务计算、计算机网络。|金鑫（1991-），男，浙江台州人，浙江工业大学硕士生，主要研究方向为服务计算。
基金资助:
浙江省重大科技专项重大工业基金资助项目(2012C11026-2);杭州市重大科技创新专项基金资助项目(20132011A16)

Research on Q-learning based rate control approach for HTTP adaptive streaming

Li-rong XIONG,Jing-zhi LEI,Xin JIN

College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China

Revised:2017-05-22 Online:2017-09-01 Published:2017-10-18
Supported by:
The Key Project of Science and Technology Planning Project of Zhejiang Province(2012C11026-2);The Special Found for Science and Technology Innovation Program of Hangzhou(20132011A16)

摘要/Abstract

摘要：

基于HTTP的自适应流HAS已经成为自适应视频流服务的标准。在HAS客户端网络状态多变的情况下，硬编码形式的码率决策方法灵活性偏低，对用户体验考虑不足。为了优化用户体验质量（QoE），提出一种基于Q-Learning 的码率控制算法，结合 HTTP 自适应视频流客户端环境进行建模并定义状态转移规则；量化与用户QoE相关的参数，构建新的回报函数；实验表明引入Q-Learning进行码率调整的自适应算法在码率切换的稳定性方面表现较好。

关键词: HTTP自适应流, 硬编码, Q学习, 码率控制, 稳定性

Abstract:

HTTP adaptive streaming (HAS) has become the standard for adaptive video streaming service.In changing network environments,current hardcoded-based rate adaptation algorithm was less flexible,and it is insufficient to consider the quality of experience (QoE).To optimize the QoE of users,a rate control approach based on Q-learning strategy was proposed.the client environments of HTTP adaptive video streaming was modeled and the state transition rule was defined.Three parameters related to QoE were quantified and a novel reward function was constructed.The experiments were employed by the Q-learning rate control approach in two typical HAS algorithms.The experiments show the rate control approach can enhance the stability of rate switching in HAS clients.

Key words: HTTP adaptive streaming, hardcoded, Q-learning, rate control, stability

中图分类号:

TN919.85

熊丽荣,雷静之,金鑫. 基于Q-learning的HTTP自适应流码率控制方法研究[J]. 通信学报, 2017, 38(9): 18-24.

Li-rong XIONG,Jing-zhi LEI,Xin JIN. Research on Q-learning based rate control approach for HTTP adaptive streaming[J]. Journal on Communications, 2017, 38(9): 18-24.

图/表 13

表1

图1

表2

表3

图2

图3

图4

图5

图6

图7

图8

图9

图10

参考文献 22

[1]	LIU C , BOUAZIZI I , GABBOUJ M . Rate adaptation for adaptive HTTP streaming[C]// The Second Annual ACM conference on Multimedia systems. 2011: 169-174.
[2]	THANG T C , HO Q D , KANG J W ,et al. Adaptive streaming of audiovisual content using MPEG DASH[J]. IEEE Transactions on Consumer Electronics, 2012,58(1): 78-85.
[3]	ROMERO L R . A dynamic adaptive http streaming video service for google android[J]. Communication Systems Cos, 2011: 33-51.
[4]	LIU C , BOUAZIZI I , HANNUKSELA M M ,et al. Rate adaptation for dynamic adaptive streaming over HTTP in content distribution network[J]. Signal Processing Image Communication, 2012,27(4): 288-311.
[5]	CLAEYS M , LATRé S , FAMAEY J ,et al. Design of a Q-learningbased client quality selection algorithm for HTTP adaptive video streaming[C]// Adaptive and Learning Agents Workshop. 2013: 30-37.
[6]	CLAEYS M , LATRé S , FAMAEY J ,et al. Design and evaluation of a self-learning HTTP adaptive video streaming client[J]. IEEE Communications Letters, 2014,18(4): 716-719.
[7]	MART V , GARC N . Q-learning based control algorithm for HTTP adaptive streaming[C]// 2015 Visual Communications and Image Processing (VCIP). 2015: 1-4.
[8]	MOK R K P , CHAN E W W , CHANG R K C . Measuring the quality of experience of HTTP video streaming[C]// The 2011 IFIP/IEEE International Symposium on Integrated Network Management. 2011: 485-492.
[9]	LI Z , ZHU X , GAHM J ,et al. Probe and adapt:rate adaptation for http video streaming at scale[J]. IEEE Journal on Selected Areas in Communications, 2014,32(4): 719-733.
[10]	HESSE S , . Design of scheduling and rate-adaptation algorithms for adaptive HTTP streaming[C]// Applications of Digital Image Processing. 2013: 219-224.
[11]	GARCIA S , CABRERA J , GARCIA N . Quality-control algorithm for adaptive streaming services over wireless channels[J]. IEEE Journal of Selected Topics in Signal Processing, 2015,9(1): 50-59.
[12]	MARTIN V , CABRERA J , GARCIA N . Evaluation of Q-Learning approach for HTTP adaptive streaming[C]// IEEE International Conference on Consumer Electronics. 2016: 293-294.
[13]	ZHOU C , LIN C W , GUO Z . mDASH:a Markov decision-based rate adaptation approach for dynamic HTTP streaming[J]. IEEE Transactions on Multimedia, 2016,18(4): 738-751.
[14]	ISTEPANIAN R S H , PHILIP N Y , MARTINI M G . Medical QoS provision based on reinforcement learning in ultrasound streaming over 3.5 G wireless systems[J]. IEEE Journal on Selected Areas in Communications, 2009,27(4): 566-574.
[15]	CLAEYS M , LATRé S , FAMAEY J ,et al. Design and optimisation of a (FA) Q-learning-based HTTP adaptive streaming client[J]. Connection Science, 2014,26(1): 25-43.
[16]	MENKOVSKI V , LIOTTA A . Intelligent control for adaptive video streaming[C]// International Conference on Consumer Electronics. 2013: 127-128.
[17]	CHIARIOTTI F . Reinforcement learning algorithms for DASH video streaming[D]. Italian:University of Padova, 2015.
[18]	CHIARIOTTI F , D'ARONCO S L ,et al. Online learning adaptation strategy for DASH clients[C]// International Conference on Multimedia Systems. 2016:8.
[19]	SUTTON R S , BARTO A G . Reinforcement learning:an introduction[M]. MA: MIT Press, 1998.
[20]	TOKIC M , . Adaptive ε-greedy exploration in reinforcement learning based on value differences[C]// Advances in Artificial Intelligence,German Conference on Ai,Karlsruhe,Germany,September 21-24,2010.Proceedings. DBLP, 2010: 203-210.
[21]	TOKIC M , PALM G . Value-difference based exploration:adaptive control between epsilon-greedy and softmax[C]// The 34th Annual German Conference on Advances in Artificial Intelligence. 2011: 335-346.
[22]	MUELLER C , LEDERER S , POECHER J ,et al. Demo paper:libdash-an open source software library for the MPEG-DASH standard[C]// The 2013 IEEE International Conference on Multimedia and Expo Workshops. 2013: 1-2.

状态元素	范围	级别
Bandwidth	0;BW_max ]bit ·s^?1	N+ 1
Buffer filling	0;τB_max ]s	B_max+1

ε	收敛时的迭代次数
ε	γ=0.1	γ=0.5	γ=0.7	γ=0.9
0	65	95	169	493
0.5	109	179	312	900
1	7 472	12 218	20 131	56 735

ε	收敛时的平均回报值
ε	γ=0.1	γ=0.5	γ=0.7	γ=0.9
0	-3.805 95	-0.698 535	7.238 54	52.5
0.5	-3.805 91	-0.698 533	7.238 56	52.500 1
1	-3.804 86	-0.693 75	7.239 58	52.506 3

基于Q-learning的HTTP自适应流码率控制方法研究

Research on Q-learning based rate control approach for HTTP adaptive streaming

在线阅读

PDF下载

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 22

相关文章 15

Metrics

推荐阅读 0

[1]	公衍超, 王玲, 刘颖, 杨楷芳, 林庆帆, 王富平. 视频主观观测实验启发的HEVC感知帧内码率控制[J]. 通信学报, 2021, 42(8): 90-102.
[2]	喻鹏, 张俊也, 李文璟, 周凡钦, 丰雷, 付澍, 邱雪松. 移动边缘网络中基于双深度Q学习的高能效资源分配方法[J]. 通信学报, 2020, 41(12): 148-161.
[3]	朱江,王婷婷,宋永辉,刘亚利. 无线网络中基于深度Q学习的传输调度方案[J]. 通信学报, 2018, 39(4): 35-44.
[4]	苏成龙,金光,江先亮,钮俊. 公平和低时延的主动队列管理算法研究[J]. 通信学报, 2017, 38(5): 199-206.
[5]	李智楠,杨晓冬. 基于可靠路径稳定性估计的MANET路由发现算法研究[J]. 通信学报, 2016, 37(8): 119-128.
[6]	刘舒祺,汪一鸣,崔翠梅. 基于局部拓扑控制的认知网络路由方法[J]. 通信学报, 2016, 37(5): 106-114.
[7]	熊丹妮,李屹. 星地融合网络中基于Q学习的切换算法研究[J]. 通信学报, 2015, 36(9): 252-258.
[8]	杜白,李红艳,龙彦. 最小最大剩余服务时间的异构网络选择算法[J]. 通信学报, 2015, 36(8): 104-109.
[9]	李晓辉，张金钊，黑永强. 基于Markov模型的分布式队列稳定频谱接入算法[J]. 通信学报, 2014, 35(3): 3-29.
[10]	李晓辉,张金钊,黑永强. 基于Markov模型的分布式队列稳定频谱接入算法[J]. 通信学报, 2014, 35(3): 22-29.
[11]	李方伟，唐永川，朱江. 未知环境中基于图型博弈和Multi-Q学习的动态信道选择算法[J]. 通信学报, 2013, 34(11): 1-7.
[12]	于俊1，刘全1,2，傅启明1，孙洪坤1，陈桂兴1. 基于优先级扫描Dyna结构的贝叶斯Q学习方法[J]. 通信学报, 2013, 34(11): 15-139.
[13]	李方伟,唐永川,朱江. 未知环境中基于图型博弈和multi-Q学习的动态信道选择算法[J]. 通信学报, 2013, 34(11): 1-7.
[14]	于俊,刘全,傅启明,孙洪坤,陈桂兴. 基于优先级扫描Dyna结构的贝叶斯Q学习方法[J]. 通信学报, 2013, 34(11): 129-139.
[15]	李动节,朱仲杰,王玉儿. 结合视觉感知的对象基嵌入式图像编码中的码率控制方法[J]. 通信学报, 2012, 33(4): 115-120.