基于Q-learning的HTTP自适应流码率控制方法研究

doi:10.11959/j.issn.1000-436x.2017178

Abstract

Abstract:

HTTP adaptive streaming (HAS) has become the standard for adaptive video streaming service.In changing network environments,current hardcoded-based rate adaptation algorithm was less flexible,and it is insufficient to consider the quality of experience (QoE).To optimize the QoE of users,a rate control approach based on Q-learning strategy was proposed.the client environments of HTTP adaptive video streaming was modeled and the state transition rule was defined.Three parameters related to QoE were quantified and a novel reward function was constructed.The experiments were employed by the Q-learning rate control approach in two typical HAS algorithms.The experiments show the rate control approach can enhance the stability of rate switching in HAS clients.

Key words: HTTP adaptive streaming, hardcoded, Q-learning, rate control, stability

CLC Number:

TN919.85

Li-rong XIONG,Jing-zhi LEI,Xin JIN. Research on Q-learning based rate control approach for HTTP adaptive streaming[J]. Journal on Communications, 2017, 38(9): 18-24.

Figures/Tables 13

References 22

[1]	LIU C , BOUAZIZI I , GABBOUJ M . Rate adaptation for adaptive HTTP streaming[C]// The Second Annual ACM conference on Multimedia systems. 2011: 169-174.
[2]	THANG T C , HO Q D , KANG J W ,et al. Adaptive streaming of audiovisual content using MPEG DASH[J]. IEEE Transactions on Consumer Electronics, 2012,58(1): 78-85.
[3]	ROMERO L R . A dynamic adaptive http streaming video service for google android[J]. Communication Systems Cos, 2011: 33-51.
[4]	LIU C , BOUAZIZI I , HANNUKSELA M M ,et al. Rate adaptation for dynamic adaptive streaming over HTTP in content distribution network[J]. Signal Processing Image Communication, 2012,27(4): 288-311.
[5]	CLAEYS M , LATRé S , FAMAEY J ,et al. Design of a Q-learningbased client quality selection algorithm for HTTP adaptive video streaming[C]// Adaptive and Learning Agents Workshop. 2013: 30-37.
[6]	CLAEYS M , LATRé S , FAMAEY J ,et al. Design and evaluation of a self-learning HTTP adaptive video streaming client[J]. IEEE Communications Letters, 2014,18(4): 716-719.
[7]	MART V , GARC N . Q-learning based control algorithm for HTTP adaptive streaming[C]// 2015 Visual Communications and Image Processing (VCIP). 2015: 1-4.
[8]	MOK R K P , CHAN E W W , CHANG R K C . Measuring the quality of experience of HTTP video streaming[C]// The 2011 IFIP/IEEE International Symposium on Integrated Network Management. 2011: 485-492.
[9]	LI Z , ZHU X , GAHM J ,et al. Probe and adapt:rate adaptation for http video streaming at scale[J]. IEEE Journal on Selected Areas in Communications, 2014,32(4): 719-733.
[10]	HESSE S , . Design of scheduling and rate-adaptation algorithms for adaptive HTTP streaming[C]// Applications of Digital Image Processing. 2013: 219-224.
[11]	GARCIA S , CABRERA J , GARCIA N . Quality-control algorithm for adaptive streaming services over wireless channels[J]. IEEE Journal of Selected Topics in Signal Processing, 2015,9(1): 50-59.
[12]	MARTIN V , CABRERA J , GARCIA N . Evaluation of Q-Learning approach for HTTP adaptive streaming[C]// IEEE International Conference on Consumer Electronics. 2016: 293-294.
[13]	ZHOU C , LIN C W , GUO Z . mDASH:a Markov decision-based rate adaptation approach for dynamic HTTP streaming[J]. IEEE Transactions on Multimedia, 2016,18(4): 738-751.
[14]	ISTEPANIAN R S H , PHILIP N Y , MARTINI M G . Medical QoS provision based on reinforcement learning in ultrasound streaming over 3.5 G wireless systems[J]. IEEE Journal on Selected Areas in Communications, 2009,27(4): 566-574.
[15]	CLAEYS M , LATRé S , FAMAEY J ,et al. Design and optimisation of a (FA) Q-learning-based HTTP adaptive streaming client[J]. Connection Science, 2014,26(1): 25-43.
[16]	MENKOVSKI V , LIOTTA A . Intelligent control for adaptive video streaming[C]// International Conference on Consumer Electronics. 2013: 127-128.
[17]	CHIARIOTTI F . Reinforcement learning algorithms for DASH video streaming[D]. Italian:University of Padova, 2015.
[18]	CHIARIOTTI F , D'ARONCO S L ,et al. Online learning adaptation strategy for DASH clients[C]// International Conference on Multimedia Systems. 2016:8.
[19]	SUTTON R S , BARTO A G . Reinforcement learning:an introduction[M]. MA: MIT Press, 1998.
[20]	TOKIC M , . Adaptive ε-greedy exploration in reinforcement learning based on value differences[C]// Advances in Artificial Intelligence,German Conference on Ai,Karlsruhe,Germany,September 21-24,2010.Proceedings. DBLP, 2010: 203-210.
[21]	TOKIC M , PALM G . Value-difference based exploration:adaptive control between epsilon-greedy and softmax[C]// The 34th Annual German Conference on Advances in Artificial Intelligence. 2011: 335-346.
[22]	MUELLER C , LEDERER S , POECHER J ,et al. Demo paper:libdash-an open source software library for the MPEG-DASH standard[C]// The 2013 IEEE International Conference on Multimedia and Expo Workshops. 2013: 1-2.

Metrics

Recommended 0

No Suggested Reading articles found!

状态元素	范围	级别
Bandwidth	0;BW_max ]bit ·s^?1	N+ 1
Buffer filling	0;τB_max ]s	B_max+1

ε	收敛时的迭代次数
ε	γ=0.1	γ=0.5	γ=0.7	γ=0.9
0	65	95	169	493
0.5	109	179	312	900
1	7 472	12 218	20 131	56 735

ε	收敛时的平均回报值
ε	γ=0.1	γ=0.5	γ=0.7	γ=0.9
0	-3.805 95	-0.698 535	7.238 54	52.5
0.5	-3.805 91	-0.698 533	7.238 56	52.500 1
1	-3.804 86	-0.693 75	7.239 58	52.506 3

Research on Q-learning based rate control approach for HTTP adaptive streaming

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 22

Related Articles 15

Metrics

Recommended 0

[1]	Junfeng TIAN, Qianyu YANG, Xiao Jitian. Distributed audit causal consistency model based on biased stability [J]. Journal on Communications, 2023, 44(3): 164-177.
[2]	Yanchao GONG, Ling WANG, Ying LIU, Kaifang YANG, Qingfan LIN, Fuping WANG. HEVC perceptual intra-frame rate control inspired by video subjective observation experiment [J]. Journal on Communications, 2021, 42(8): 90-102.
[3]	Hongqi ZHANG,Junnan YANG,Chuanfu ZHANG. Defense decision-making method based on incomplete information stochastic game and Q-learning [J]. Journal on Communications, 2018, 39(8): 56-68.
[4]	Cheng-long SU,Guang JIN,Xian-liang JIANG,Jun NIU. Active queue management algorithm focusing on fairness and low delay [J]. Journal on Communications, 2017, 38(5): 199-206.
[5]	Zhi-nan LI,Xiao-dong YANG. Routing discovery algorithm based on reliable path stability estimation in MANET [J]. Journal on Communications, 2016, 37(8): 119-128.
[6]	Ye SHEN,Jing FENG,Zhan-feng WANG. Scalable overlay multicast algorithm with high stability [J]. Journal on Communications, 2016, 37(5): 73-80.
[7]	Shu-qi LIU,Yi-ming WANG,Cui-mei CUI. Routing approach based on local topology control in cognitive radio networks [J]. Journal on Communications, 2016, 37(5): 106-114.
[8]	Hong-yu WANG,Tian-shuang QIU. Unified study on the decomposition for deterministic signals and stationary random signals [J]. Journal on Communications, 2016, 37(10): 1-8.
[9]	Ming-jian LIU,Guo-zhen TAN,Shuai-bing LI,Nan DING,Cai-xia SONG. Adaptive message sending rate control method based on channel congestion cost calculation in VANET [J]. Journal on Communications, 2016, 37(10): 108-116.
[10]	Wei LI,Fu-zheng YANG,Peng REN. H.265/HEVC frame-level bitrate allocation algorithm considering video content [J]. Journal on Communications, 2015, 36(9): 76-81.
[11]	Dan-ni XIONG,Yi LI. Q-learning based handoff algorithm for satellite system with ancillary terrestrial component [J]. Journal on Communications, 2015, 36(9): 252-258.
[12]	Bai DU,Hong-yan LI,Yan LONG. Network selection algorithm in heterogeneous wireless networks to minimize the maximum residual service time [J]. Journal on Communications, 2015, 36(8): 104-109.
[13]	. Rate control algorithm for no-feedback distributed video coding [J]. Journal on Communications, 2014, 35(6): 5-38.
[14]	HENJian C,Chao HUI,Yong-hong KUO. Rate control algorithm for no-feedback distributed video coding [J]. Journal on Communications, 2014, 35(6): 32-38.
[15]	Xiao-hui LI,Jin-zhao ZHANG,Yong-qiang HEI. Queue stability distributed spectrum access algorithm based on Markov model for cognitive radio network [J]. Journal on Communications, 2014, 35(3): 22-29.