基于最邻近帧质量增强的视频编码参考帧列表优化算法

doi:10.11959/j.issn.1000-436x.2022185

摘要/Abstract

摘要：

帧间预测是视频编码的核心模块，其利用参考帧的重建样本来预测当前图像样本，从而通过传输少量预测残差数据表示复杂视频内容。在有损视频编码中，参考帧质量受到量化失真的影响，导致预测精度较差，影响编码性能。针对低时延视频业务，提出一种基于最邻近帧质量增强的参考帧列表优化算法，通过基于深度学习的卷积神经网络增强与当前帧最邻近参考帧的质量，并将增强后的高质量帧整合到当前帧的参考帧列表中，提高了帧间预测精度。以高效视频编码 H.265/HEVC 参考软件平台 HM16.22 为参考基准，所提算法在 Y、Cb、Cr这3个分量上可分别节省9.06%、14.92%、13.19%的编码码率。

关键词: H.265/HEVC, 深度学习, 帧间预测, 参考帧列表

Abstract:

Interframe prediction is a key module in video coding, which uses the samples in the reference frames to predict those in the current picture, thus helps to represent the complex video by transmitting a small amount of the prediction residual.In lossy video coding, the qualities of reference frames are affected by the quantization distortion, which lead to poor prediction accuracy and performance degradation.Targeted at the low latency video services, a reference frame list optimization algorithm was proposed, which enhanced the quality of the nearest reference frame by a deep learning-based convolutional neural network, and integrated the enhanced reference frame into the reference frame list to improve the accuracy of interframe prediction.Compared with H.265/HEVC reference software HM16.22, the proposed algorithm provides BD-rate savings of 9.06%, 14.92% and 13.19% for Y, Cb and Cr components, respectively.

Key words: H.265/HEVC, deep learning, interframe prediction, reference frame list

中图分类号:

TN911.7

霍俊彦, 邱瑞鹏, 马彦卓, 杨付正. 基于最邻近帧质量增强的视频编码参考帧列表优化算法[J]. 通信学报, 2022, 43(11): 136-147.

Junyan HUO, Ruipeng QIU, Yanzhuo MA, Fuzheng YANG. Reference frame list optimization algorithm in video coding by quality enhancement of the nearest picture[J]. Journal on Communications, 2022, 43(11): 136-147.

图/表 12

图1

图2

图3

图4

图5

图6

图7

图8

图9

表1

LDP配置编码下引入增强参考帧在HM16.22下的测试结果"

							BD-rate
类别	测试序列名称		第一位			第二位			最后位			替换第一位
		Y	Cb	Cr	Y	Cb	Cr	Y	Cb	Cr	Y	Cb	Cr
	PeopleOnStreet	-5.29%	-11.13%	-4.14%	-6.90%	-13.56%	-7.46%	-6.11%	-14.09%	-8.64%	-3.98%	-9.01%	-1.14%
A	Traffic	-6.26%	-8.92%	-5.30%	-8.53%	-11.25%	-8.39%	-8.89%	-12.11%	-9.47%	-2.46%	-4.55%	-0.28%
	平均	-5.78%	-10.02%	-4.72%	-7.72%	-12.40%	-7.92%	-7.50%	-13.10%	-9.06%	-3.22%	-6.78%	-0.71%
	Kimono	-8.28%	-15.60%	-3.92%	-9.12%	-16.32%	-5.21%	-8.57%	-15.87%	-5.89%	-7.34%	-14.45%	-1.97%
	ParkScene	-3.35%	-8.66%	-2.57%	-5.13%	-10.56%	-4.93%	-5.19%	-11.16%	-5.85%	-1.20%	-5.26%	0.38%
B	Cactus	-9.62%	-18.73%	-14.65%	-11.25%	-20.21%	-16.45%	-11.13%	-20.24%	-16.82%	-5.34%	-14.80%	-10.32%
	BasketballDrive	-5.40%	-17.14%	-13.80%	-7.36%	-19.09%	-15.84%	-6.92%	-18.27%	-15.46%	-3.41%	-15.66%	-10.96%
	BQTerrace	-13.13%	-24.33%	-22.21%	-14.21%	-25.09%	-23.51%	-14.13%	-25.60%	-23.82%	-11.60%	-23.68%	-22.05%
	平均	-7.96%	-16.89%	-11.43%	-9.41%	-18.25%	-13.19%	-9.19%	-18.23%	-13.57%	-5.78%	-14.77%	-8.98%
	BasketballDrill	-6.93%	-15.53%	-16.37%	-8.29%	-17.03%	-18.21%	-8.12%	-16.81%	-17.27%	-5.20%	-14.03%	-14.09%
	BQMall	-5.87%	-13.16%	-12.40%	-7.93%	-15.06%	-14.24%	-7.58%	-15.07%	-14.32%	-2.92%	-10.03%	-8.53%
C	PartyScene	-6.17%	-13.32%	-10.87%	-7.25%	-14.73%	-12.26%	-7.02%	-14.74%	-12.47%	-5.80%	-12.72%	-9.59%
	RaceHorses	-3.09%	-7.08%	-9.80%	-5.02%	-9.67%	-13.12%	-4.37%	-9.21%	-12.64%	-0.55%	-3.19%	-4.84%
	平均	-5.52%	-12.27%	-12.36%	-7.13%	-14.12%	-14.46%	-6.77%	-13.96%	-14.18%	-3.62%	-9.99%	-9.26%
	BasketballPass	-6.75%	-17.39%	-14.50%	-8.23%	-19.56%	-15.68%	-7.00%	-18.61%	-15.16%	-4.73%	-16.29%	-11.93%
	BQSquare	-9.48%	-12.02%	-15.88%	-11.01%	-12.61%	-17.53%	-10.83%	-13.07%	-18.59%	-9.64%	-9.59%	-15.09%
D	BlowingBubbles	-6.31%	-14.14%	-10.94%	-7.74%	-15.50%	-13.47%	-7.67%	-15.46%	-14.29%	-4.21%	-12.10%	-8.28%
	RaceHorses	-4.15%	-7.01%	-10.07%	-6.40%	-9.85%	-12.13%	-5.54%	-9.00%	-12.37%	-1.19%	-2.52%	-6.97%
	平均	-6.67%	-12.64%	-12.85%	-8.35%	-14.38%	-14.70%	-7.76%	-14.03%	-15.10%	-4.94%	-10.12%	-10.57%
	FourPeople	-10.45%	-14.50%	-14.32%	-11.99%	-15.91%	-16.12%	-12.09%	-15.89%	-16.08%	-6.29%	-10.51%	-9.77%
E	Johnny	-12.65%	-8.54%	-9.01%	-15.21%	-12.55%	-11.35%	-15.02%	-13.76%	-12.08%	-8.11%	-3.33%	-3.34%
	KristenAndSara	-8.95%	-6.43%	-8.90%	-11.48%	-10.03%	-11.46%	-11.74%	-11.25%	-12.03%	-4.52%	-0.21%	-2.42%
	平均	-10.68%	-9.82%	-10.75%	-12.89%	-12.83%	-12.98%	-12.95%	-13.63%	-13.40%	-6.31%	-4.68%	-5.17%
	总平均	-7.34%	-12.98%	-11.09%	-9.06%	-14.92%	-13.19%	-8.77%	-15.01%	-13.51%	-4.92%	-10.11%	-7.84%

表1

表2

表3

与现有方法的编码性能比较"

类别	测试序列名称					BD-rate
			文献[39]方法			文献[44]方法			本文算法
		Y	Cb	Cr	Y	Cb	Cr	Y	Cb	Cr
	PeopleOnStreet	-5.8%	-4.0%	-4.4%	-8.2%	-10.3%	-11.8%	-7.2%	-13.8%	-7.5%
A	Traffic	-3.6%	-3.8%	-3.1%	-5.8%	-11.0%	-11.8%	-8.3%	-8.7%	-6.3%
	平均	-4.7%	-3.9%	-3.8%	-7.0%	-10.7%	-11.8%	-7.8%	-11.2%	-6.9%
	Kimono	-5.1%	-7.3%	-3.7%	-8.9%	-22.2%	-4.6%	-8.5%	-12.5%	-3.7%
	ParkScene	-3.3%	-3.9%	-3.8%	-4.7%	-17.4%	-5.5%	-4.7%	-9.4%	-2.8%
B	Cactus	-6.9%	-10.3%	-6.0%	-10.7%	-19.5%	-9.4%	-10.4%	-17.1%	-11.7%
	BasketballDrive	-2.8%	-5.6%	-3.7%	-4.3%	-12.3%	-12.0%	-7.1%	-18.1%	-16.1%
	BQTerrace	-4.2%	-3.6%	-1.4%	-9.6%	-9.1%	-16.3%	-14.6%	-16.4%	-12.6%
	平均	-4.5%	-6.4%	-3.7%	-7.6%	-16.1%	-9.6%	-9.0%	-14.7%	-9.4%
	BasketballDrill	-2.9%	-5.2%	-3.0%	-5.5%	-9.3%	-9.3%	-9.5%	-19.4%	-16.9%
	BQMall	-6.0%	-7.3%	-6.7%	-9.3%	-19.8%	-20.9%	-8.0%	-14.6%	-13.0%
C	PartyScene	-3.3%	-3.9%	-3.8%	-4.9%	-12.2%	-11.7%	-6.4%	-11.9%	-8.7%
	RaceHorses	-1.1%	-1.5%	-1.9%	-1.9%	-4.3%	-5.8%	-5.3%	-9.0%	-11.7%
	平均	-3.3%	-4.5%	-3.9%	-5.4%	-11.4%	-11.9%	-7.3%	-13.7%	-12.6%
	BasketballPass	-4.4%	-5.7%	-4.6%	-5.9%	-10.5%	-8.6%	-6.9%	-16.8%	-13.4%
	BQSquare	-3.2%	-0.6%	1.5%	-6.8%	-4.1%	-11.3%	-9.8%	-9.4%	-10.9%
D	BlowingBubbles	-4.1%	-5.2%	-5.3%	-5.9%	-13.0%	-12.5%	-7.5%	-13.9%	-10.6%
	RaceHorses	-1.8%	-2.7%	-3.2%	-2.8%	-5.6%	-6.9%	-6.3%	-9.6%	-11.8%
	平均	-3.4%	-3.6%	-2.9%	-5.4%	-8.3%	-9.8%	-7.6%	-12.4%	-11.7%
	FourPeople	-10.1%	-11.9%	-11.6%	-12.2%	-13.1%	-17.2%	-12.6%	-12.6%	-12.4%
E	Johnny	-8.6%	-9.1%	-7.5%	-11.3%	-9.2%	-6.9%	-15.5%	-5.1%	-8.1%
	KristenAndSara	-8.7%	-22.2%	-9.1%	-12.4%	-12.0%	-14.4%	-11.9%	-6.3%	-9.2%
	平均	-9.1%	-14.4%	-9.4%	-12.0%	-11.4%	-12.8%	-13.3%	-8.0%	-9.9%
	总平均	-4.8%	-5.7%	-4.5%	-7.3%	-12.0%	-10.9%	-8.9%	-12.5%	-10.4%

表3

参考文献 48

[1]	SULLIVAN G J , OHM J R , HAN W J ,et al. Overview of the high efficiency video coding (HEVC) standard[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012,22(12): 1649-1668.
[2]	毕厚杰 . 新一代视频压缩编码标准—H.264/AVC[M]. 北京: 人民邮电出版社, 2005.
	BI H J . A new generation of video compression and coding standards-H.264 /AVC[M]. Beijing: Posts and Telecommunications Press, 2005.
[3]	DONG C , DENG Y B , LOY C C ,et al. Compression artifacts reduction by a deep convolutional network[C]// Proceedings of 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE Press, 2015: 576-584.
[4]	PARK W S , KIM M . CNN-based in-loop filtering for coding efficiency improvement[C]// Proceedings of 2016 IEEE 12th Image,Video,and Multidimensional Signal Processing Workshop. Piscataway:IEEE Press, 2016: 1-5.
[5]	ZHANG Y B , SHEN T , JI X Y ,et al. Residual highway convolutional neural networks for in-loop filtering in HEVC[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2018,27(8): 3827-3841.
[6]	XIA J Y , WEN J T . Asymmetric convolutional residual network for AV1 intra in-loop filtering[C]// Proceedings of 2020 IEEE International Conference on Image Processing. Piscataway:IEEE Press, 2020: 1291-1295.
[7]	ZHANG S F , FAN Z H , LING N ,et al. Recursive residual convolutional neural network- based in-loop filtering for intra frames[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020,30(7): 1888-1900.
[8]	DAI Y Y , LIU D , WU F . A convolutional neural network approach for post-processing in HEVC intra coding[C]// IEEE International Conference on Multimedia Modeling. Piscataway:IEEE Press, 2017: 28-39.
[9]	LAI P R , WANG J S . Multi-stage attention convolutional neural networks for HEVC in-loop filtering[C]// Proceedings of 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems. Piscataway:IEEE Press, 2020: 173-177.
[10]	DING D D , KONG L Y , CHEN G Y ,et al. A switchable deep learning approach for in-loop filtering in video coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020,30(7): 1871-1887.
[11]	HU J , SHEN L , ALBANIE S ,et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020,42(8): 2011-2023.
[12]	KANG J H , KIM S , LEE K M . Multi-modal/multi-scale convolutional neural network based in-loop filter design for next generation video codec[C]// Proceedings of 2017 IEEE International Conference on Image Processing. Piscataway:IEEE Press, 2017: 26-30.
[13]	HE X Y , HU Q , ZHANG X Y ,et al. Enhancing HEVC compressed videos with a partition-masked convolutional neural network[C]// Proceedings of 2018 25th IEEE International Conference on Image Processing. Piscataway:IEEE Press, 2018: 216-220.
[14]	LI D W , YU L . An in-loop filter based on low-complexity CNN using residuals in intra video coding[C]// Proceedings of 2019 IEEE International Symposium on Circuits and Systems. Piscataway:IEEE Press, 2019: 1-5.
[15]	ZHU H , XU X Z , LIU S . Residual convolutional neural network based in-loop filter with intra and inter frames processed respectively for Avs3[C]// Proceedings of 2020 IEEE International Conference on Multimedia ＆ Expo Workshops. Piscataway:IEEE Press, 2020: 1-6.
[16]	JIA C M , WANG S Q , ZHANG X F ,et al. Spatial-temporal residue network based in-loop filter for video coding[C]// Proceedings of 2017 IEEE Visual Communications and Image Processing. Piscataway:IEEE Press, 2017: 1-4.
[17]	LU G , OUYANG W L , XU D ,et al. Deep Kalman filtering network for video compression artifact reduction[C]// Proceedings of the European Conference on Computer Vision. Berlin:Springer, 2018: 568-584.
[18]	YANG R , SUN X Y , XU M ,et al. Quality-gated convolutional LSTM for enhancing compressed video[C]// Proceedings of 2019 IEEE International Conference on Multimedia and Expo. Piscataway:IEEE Press, 2019: 532-537.
[19]	LU M , CHENG M , XU Y L ,et al. Learned quality enhancement via multi-frame priors for HEVC compliant low-delay applications[C]// Proceedings of IEEE International Conference on Image Processing. Piscataway:IEEE Press, 2019: 934-938.
[20]	TONG J C , WU X L , DING D D ,et al. Learning-based multi-frame video quality enhancement[C]// Proceedings of 2019 IEEE International Conference on Image Processing. Piscataway:IEEE Press, 2019: 929-933.
[21]	MENG X D , DENG X , ZHU S Y ,et al. A robust quality enhancement method based on joint spatial-temporal priors for video coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021,31(6): 2401-2414.
[22]	DOSOVITSKIY A , FISCHER P , ILG E ,et al. FlowNet:learning optical flow with convolutional networks[C]// Proceedings of 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE Press, 2016: 2758-2766.
[23]	SUN D Q , YANG X D , LIU M Y ,et al. PWC-net:CNNs for optical flow using pyramid,warping,and cost volume[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 8934-8943.
[24]	YANG R , XU M , WANG Z L ,et al. Multi-frame quality enhancement for compressed video[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 6664-6673.
[25]	GUAN Z Y , XING Q L , XU M ,et al. MFQE 2.0:a new approach for multi-frame quality enhancement on compressed video[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021,43(3): 949-963.
[26]	DAI J F , QI H Z , XIONG Y W ,et al. Deformable convolutional networks[C]// Proceedings of 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE Press, 2017: 764-773.
[27]	ZHU X Z , HU H , LIN S ,et al. Deformable ConvNets V2:more deformable,better results[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2019: 9300-9308.
[28]	DENG J N , WANG L , PU S L ,et al. Spatio-temporal deformable convolution for compressed video quality enhancement[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto:AAAI Press, 2020: 10696-10703.
[29]	ZHAO M Y , XU Y , ZHOU S G . Recursive fusion and deformable spatiotemporal attention for video compression artifact reduction[C]// Proceedings of the 29th ACM International Conference on Multimedia. New York:ACM Press, 2021: 5646-5654.
[30]	LIN J Y , HUANG Y , WANG L . FDAN:flow-guided deformable alignment network for video super-resolution[C]// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2021: 1-10.
[31]	CHAN K C K , ZHOU S C , XU X Y ,et al. BasicVSR++:improving video super-resolution with enhanced propagation and alignment[C]// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2021: 5972-5981.
[32]	DING D D , WANG W Y , TONG J C ,et al. Biprediction-based video quality enhancement via learning[J]. IEEE Transactions on Cybernetics, 2022,52(2): 1207-1220.
[33]	WANG X T , CHAN K C K , YU K ,et al. EDVR:video restoration with enhanced deformable convolutional networks[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway:IEEE Press, 2019: 1954-1963.
[34]	YAN N , LIU D , LI H Q ,et al. A convolutional neural network approach for half-pel interpolation in video coding[C]// Proceedings of 2017 IEEE International Symposium on Circuits and Systems. Piscataway:IEEE Press, 2017: 1-4.
[35]	ZHANG H , SONG L , LUO Z Y ,et al. Learning a convolutional neural network for fractional interpolation in HEVC inter coding[C]// Proceedings of 2017 IEEE Visual Communications and Image Processing. Piscataway:IEEE Press, 2017: 1-4.
[36]	YAN N , LIU D , LI H Q ,et al. Convolutional neural network-based fractional-pixel motion compensation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019,29(3): 840-853.
[37]	YAN N , LIU D , LI H Q ,et al. Convolutional neural network-based invertible half-pixel interpolation filter for video coding[C]// Proceedings of 2018 25th IEEE International Conference on Image Processing. Piscataway:IEEE Press, 2018: 201-205.
[38]	ZHAO Z H , WANG S Q , WANG S S ,et al. Enhanced Bi-prediction with convolutional neural network for high-efficiency video coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019,29(11): 3291-3301.
[39]	CHOI H , BAJI? I V , . Deep frame prediction for video coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020,30(7): 1843-1855.
[40]	LIN J P , LIU D , LI H Q ,et al. Generative adversarial network-based frame extrapolation for video coding[C]// Proceedings of 2018 IEEE Visual Communications and Image Processing. Piscataway:IEEE Press, 2018: 1-4.
[41]	ZHAO L , WANG S Q , ZHANG X F ,et al. Enhanced CTU-level inter prediction with deep frame rate up-conversion for high efficiency video coding[C]// Proceedings of 2018 25th IEEE International Conference on Image Processing. Piscataway:IEEE Press, 2018: 206-210.
[42]	ZHAO L , WANG S Q , ZHANG X F ,et al. Enhanced motion-compensated video coding with deep virtual reference frame generation[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2019,28(10): 4832-4844.
[43]	LEE J K , KIM N , CHO S ,et al. Deep video prediction network-based inter-frame coding in HEVC[J]. IEEE Access, 2020,8: 95906-95917.
[44]	CHOI H , BAJI? I V , . Affine transformation-based deep frame prediction[J]. IEEE Transactions on Image Processing, 2021,30: 3321-3334.
[45]	KIM J , LEE J K , LEE K M . Accurate image super-resolution using very deep convolutional networks[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2016: 1646-1654.
[46]	RANJAN A , BLACK M J . Optical flow estimation using a spatial pyramid network[C]// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017: 4161-4170.
[47]	BOSSEN F . Common HM test conditions and software reference configuration[R]. 2013.
[48]	BJ?NTEGAARD G . Calculation of average PSNR differences between RD-curves[R]. 2001.

时间比	第一位	第二位	最后位	替换第一位
编码时间比	125%	124%	127%	117%
解码时间比	5 464%	5 464%	5 465%	5 500%