基于球域失真空-时依赖的全景视频编码

doi:10.11959/j.issn.1000-436x.2023197

通信学报 ›› 2023, Vol. 44 ›› Issue (10): 58-71.doi: 10.11959/j.issn.1000-436x.2023197

• 学术论文 • 上一篇

基于球域失真空-时依赖的全景视频编码

杨栩¹^,², 朱策³, 郭红伟³^,⁴, 罗雷¹^,³

¹ 重庆邮电大学通信与信息工程学院，重庆 400065
² 成都师范学院物理与工程技术学院，四川成都 611130
³ 电子科技大学信息与通信工程学院，四川成都 611731
⁴ 红河学院工学院，云南蒙自 661100

修回日期:2023-09-28 出版日期:2023-10-01 发布日期:2023-10-01
作者简介:杨栩（1983− ），男，四川苍溪人，重庆邮电大学博士生，成都师范学院讲师，主要研究方向为全景视频编码与通信
朱策（1969− ），男，四川自贡人，博士，电子科技大学教授、博士生导师，主要研究方向为视频编码与通信、视频分析和处理
郭红伟（1980− ），男，彝族，云南金平人，博士，红河学院教授，电子科技大学在站博士后，主要研究方向为视频编码与通信
罗雷（1986- ），男，重庆人，博士，重庆邮电大学副教授，电子科技大学在站博士后，主要研究方向为视频编码与通信
基金资助:
国家自然科学基金资助项目(62020106011);国家自然科学基金资助项目(U19A2052);国家自然科学基金资助项目(62061015);四川省科技厅基金资助项目(2022ZHCG0116);重庆市自然科学基金资助项目(2023NSCQ-MSX2930);重庆邮电大学青创基金资助项目(SCIE-QN-2022-05)

Panoramic video coding based on spherical distortion with spatio-temporal dependency

Xu YANG¹^,², Ce ZHU³, Hongwei GUO³^,⁴, Lei LUO¹^,³

¹ School of Information and Communication Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
² School of Physics and Engineering Technology, Chengdu Normal University, Chengdu 611130, China
³ School of Communication and Information Engineering, University of Electronic Science and Technology, Chengdu 611731, China
⁴ School of Engineering, Honghe University, Mengzi 661100, China

Revised:2023-09-28 Online:2023-10-01 Published:2023-10-01
Supported by:
The National Natural Science Foundation of China(62020106011);The National Natural Science Foundation of China(U19A2052);The National Natural Science Foundation of China(62061015);The Project of the Science and Technology Department in Sichuan Province(2022ZHCG0116);The Chongqing Natural Science Foundation(2023NSCQ-MSX2930);Youth Innovation Group Support Program of ICE Discipline of CQUPT(SCIE-QN-2022-05)

摘要/Abstract

摘要：

全景视频平面编码失真和球域感知失真不同域使主客观质量评价不一致，进而损失编码性能。此外，独立率失真优化技术没有考虑球域失真的时域依赖性对编码的影响，编码性能还有提升空间。针对上述问题，提出一种空-时域依赖的球域失真模型以优化全景视频编码。首先，提出一种球域失真到编码失真的空域映射模型，使主客观质量评价趋近一致；然后，提出一种球域失真时域传播模型，以提升传播链上所有编码单元的整体编码性能；最后，计算球域失真空域映射权重和时域传播权重来调整编码参数。实验结果表明，在低延时编码配置下，相较于通用视频编码基准VTM14.0，所提算法有平均7.4%（最高达22.1%）的码率节省和9%的编码时间节省。

关键词: 全景视频编码, 时域依赖率失真优化, 投影, 球域失真, 编码失真

Abstract:

The planar coding distortion, which affects objective quality, and spherical distortion, which affects subjective quality, as well as existing independent rate-distortion optimization techniques that fail to consider the temporal propagation of spherical distortion and its impact on coding performance, result in coding performance degradation.To address these issues, a spatio-temporal dependent spherical distortion model was proposed for optimizing panoramic video coding.Firstly, a spatial mapping model was proposed to map spherical distortions to coding distortions, aiming to align subjective and objective quality assessments.Secondly, a temporal propagation model for spherical distortions was introduced to enhance the overall coding performance of all coding units in the propagation chain.Finally, the coding parameters were adjusted by computing the weights for spatial mapping and temporal propagation of spherical distortions.Experimental results demonstrate that, under low-delay encoding configurations, compared to the VTM14.0, a state-of-the-art video coding benchmark, the proposed algorithm achieves an average bitrate savings of 7.4% (up to 22.1%) and reduction in coding time of 9%.

Key words: panoramic video coding, temporal dependent rate-distortion optimization, projection, spherical distortion, coding distortion

中图分类号:

TN919.81

杨栩, 朱策, 郭红伟, 罗雷. 基于球域失真空-时依赖的全景视频编码[J]. 通信学报, 2023, 44(10): 58-71.

Xu YANG, Ce ZHU, Hongwei GUO, Lei LUO. Panoramic video coding based on spherical distortion with spatio-temporal dependency[J]. Journal on Communications, 2023, 44(10): 58-71.

图/表 17

图1

图2

图3

图4

图5

图6

图7

图8

表1

图9

表2

表3

表4

表5

表6

图10

表7

参考文献 24

[1]	VISHWANATH B , NANJUNDASWAMY T , ROSE K . Rotational motion model for temporal prediction in 360 video coding[C]// Proceedings of 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP). Piscataway:IEEE Press, 2017: 1-6.
[2]	CHIARIOTTI F . A survey on 360-degree video:coding,quality of experience and streaming[J]. Computer Communications, 2021,177: 133-155.
[3]	XU M , LI C , ZHANG S Y ,et al. State-of-the-art in 360° video/image processing:perception,assessment and compression[J]. IEEE Journal of Selected Topics in Signal Processing, 2020,14(1): 5-26.
[4]	YOUVALARI R G , AMINLOU A , HANNUKSELA M M ,et al. Efficient coding of 360-degree pseudo-cylindrical panoramic video for virtual reality applications[C]// Proceedings of IEEE International Symposium on Multimedia (ISM). Piscataway:IEEE Press, 2017: 525-528.
[5]	SULLIVAN G J , OHM J R , HAN W J ,et al. Overview of the high efficiency video coding (HEVC) standard[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012,22(12): 1649-1668.
[6]	BROSS B , WANG Y K , YE Y ,et al. Overview of the versatile video coding (VVC) standard and its applications[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021,31(10): 3736-3764.
[7]	CHEN Z , LI Y , ZHANG Y . Recent advances in omnidirectional video coding for virtual reality:projection and evaluation[J]. Signal Processing, 2018,146: 66-78.
[8]	SAUER J , SCHNEIDER J , WIEN M . Improved motion compensation for 360° video projected to polytopes[C]// Proceedings of 2017 IEEE International Conference on Multimedia and Expo (ICME). Piscataway:IEEE Press, 2017: 61-66.
[9]	XU M , SONG Y H , WANG J Y ,et al. Predicting head movement in panoramic video:a deep reinforcement learning approach[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019,41(11): 2693-2708.
[10]	LI L , LI Z , MA X ,et al. Co-projection-plane based 3-D padding for polyhedron projection for 360-degree video[C]// Proceedings of 2017 IEEE International Conference on Multimedia and Expo (ICME). Piscataway:IEEE Press, 2017: 55-60.
[11]	LI L , LI Z , MA X ,et al. Advanced spherical motion model and local padding for 360° video compression[J]. IEEE Transactions on Image Processing, 2019,28(5): 2342-2356.
[12]	ZHOU Y M , TIAN L , ZHU C ,et al. Video coding optimization for virtual reality 360-degree source[J]. IEEE Journal of Selected Topics in Signal Processing, 2020,14(1): 118-129.
[13]	SUN Y L , YU L . Coding optimization based on weighted-to- spherically-uniform quality metric for 360 video[C]// Proceedings of IEEE Visual Communications and Image Processing (VCIP). Piscataway:IEEE Press, 2018: 1-4.
[14]	LI Y M , XU J Z , CHEN Z Z . Spherical domain rate-distortion optimization for omnidirectional video coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019,29(6): 1767-1780.
[15]	GAO Y B , ZHU C , LI S ,et al. Temporally dependent rate-distortion optimization for low-delay hierarchical video coding[J]. IEEE Transactions on Image Processing, 2017,26(9): 4457-4470.
[16]	GAO Y B , ZHU C , LI S ,et al. Source distortion temporal propagation analysis for random-access hierarchical video coding optimization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019,29(2): 546-559.
[17]	LI S , ZHU C , GAO Y B ,et al. Lagrangian multiplier adaptation for rate-distortion optimization with inter-frame dependency[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016,26(1): 117-129.
[18]	YANG T W , ZHU C , FAN X J ,et al. Source distortion temporal propagation model for motion compensated video coding optimization[C]// Proceedings of 2012 IEEE International Conference on Multimedia and Expo. Piscataway:IEEE Press, 2012: 85-90.
[19]	郭红伟, 朱策, 杨栩 ,等. 基于失真反向传播的时域依赖率失真优化[J]. 通信学报, 2022,43(12): 222-232.
	GUO H W , ZHU C , YANG X ,et al. Temporal dependent rate-distortion optimization based on distortion backward propagation[J]. Journal on Communications, 2022,43(12): 222-232.
[20]	GUO H , ZHU C , YE M ,et al. Pre-encoding based temporal dependent rate-distortion optimization for HEVC[J]. Signal Processing:Image Communication, 2023,115:116957.
[21]	ZHAO T S , WANG Z , CHEN C W . Adaptive quantization parameter cascading in HEVC hierarchical coding[J]. IEEE Transactions on Image Processing, 2016,25(7): 2997-3009.
[22]	LI B , XU J Z , ZHANG D ,et al. QP refinement according to lagrange multiplier for high efficiency video coding[C]// Proceedings of 2013 IEEE International Symposium on Circuits and Systems (ISCAS). Piscataway:IEEE Press, 2013: 477-480.
[23]	ALSHINA E , BOYCE J , ABBAS A ,et al. JVET common test conditions and evaluation procedures for 360 degree video:JVET document,JVET-G1030[S]. 2017.
[24]	JIANG L , XU M , LIU T ,et al. DeepVS:a deep learning based video saliency prediction approach[C]// European Conference on Computer Vision. Cham:Springer, 2018: 625-642.

类别	序列	帧宽/像素	帧高/像素	帧数/帧	帧率/(frame·s^-1)	比特深度/bit
	Trolley	8 192	4 096	300	30	10
	GasLamp	8 192	4 096	300	30	10
	Skateboardinginlot	8 192	4 096	300	30	8
8K	ChairliftRide	8 192	4 096	300	30	8
	KiteFlite	8 192	4 096	300	30	8
	Harbor	8 192	4 096	300	30	8
	SkateboardTrick	8 192	4 096	600	60	8
	Train	8 192	4 096	600	60	8
	Balboa	6 144	3 072	600	60	8
6K	Broadway	6 144	3 072	600	60	8
	Landing	6 144	3 072	300	60	8
	BranCastle	6 144	3 072	300	60	8
	PoleVault	3 840	1 920	300	30	8
4K	AerialCity	3 840	1 920	300	30	8
	DrivingInCity	3 840	1 920	300	30	8
	DrivingInCountry	3 840	1 920	300	30	8

类别	序列		SPSNR			CPP-PSNR			WS-PSNR
类别	序列	SPSNR-Y	SPSNR-U	SPSNR-V	CPPPSNR-Y	CPPPSNR-U	CPPPSNR-V	WSPSNR-Y	WSPSNR-U	WSPSNR-V
	Trolley	-2.7%	-4.0%	-3.6%	-2.8%	-4.1%	-3.7%	-3.0%	-4.0%	-3.7%
	GasLamp	-4.8%	-6.0%	-6.1%	-4.8%	-6.0%	-6.1%	-4.9%	-6.0%	-6.1%
	Skateboardinginlot	-1.2%	-2.1%	-0.8%	-1.2%	-2.1%	-0.8%	-1.6%	-2.2%	-1.0%
	ChairliftRide	-1.5%	-0.5%	-0.5%	-1.5%	-0.4%	-0.5%	-1.9%	-0.4%	-0.5%
8K	KiteFlite	-2.9%	-4.0%	-3.5%	-2.9%	-4.0%	-3.4%	-3.3%	-4.1%	-3.4%
	Harbor	-3.1%	-5.4%	-5.9%	-3.2%	-5.5%	-5.9%	-3.5%	-5.6%	-5.8%
	SkateboardTrick	-1.6%	-0.8%	-0.7%	-1.3%	-0.6%	-0.6%	-1.9%	-0.8%	-0.6%
	Train	-3.4%	-3.4%	-4.2%	-3.4%	-3.3%	-4.2%	-3.4%	-3.5%	-4.2%
	平均	-2.7%	-3.3%	-3.2%	-2.6%	-3.3%	-3.2%	-2.9%	-3.3%	-3.2%
	Balboa	-0.9%	0.3%	0.8%	-0.8%	0.3%	1.0%	-1.6%	0.3%	1.0%
	Broadway	-0.4%	-0.3%	-0.4%	-0.5%	-0.2%	-0.4%	-1.5%	-0.3%	-0.4%
6K	Landing	-1.3%	-0.5%	-0.6%	-1.3%	-0.4%	-0.5%	-1.6%	-0.6%	-0.6%
	BranCastle	-1.3%	-0.8%	0.4%	-1.3%	-0.8%	0.5%	-1.8%	-0.8%	0.5%
	平均	-1.0%	-0.3%	0.1%	-1.0%	-0.3%	0.2%	-1.6%	-0.4%	0.1%
	PoleVault	-4.7%	-3.5%	-1.9%	-4.6%	-3.5%	-1.6%	-5.4%	-3.8%	-1.8%
	AerialCity	-1.8%	-9.4%	-8.4%	-1.5%	-9.3%	-8.1%	-2.0%	-9.3%	-8.1%
4K	DrivingInCity	-2.5%	-5.2%	-4.7%	-2.3%	-5.3%	-4.6%	-2.6%	-5.3%	-4.7%
	DirvingInCountry	-1.3%	-4.4%	-3.5%	-1.0%	-4.2%	-3.1%	-1.7%	-4.3%	-3.2%
	平均	-2.6%	-5.6%	-4.6%	-2.4%	-5.6%	-4.4%	-2.9%	-5.7%	-4.5%
	总体平均	-2.2%	-3.2%	-2.8%	-2.1%	-3.1%	-2.7%	-2.6%	-3.2%	-2.6%

类别	序列		SPSNR			CPP-PSNR			WS-PSNR
类别	序列	SPSNR-Y	SPSNR-U	SPSNR-V	CPPPSNR-Y	CPPPSNR-U	CPPPSNR-V	WSPSNR-Y	WSPSNR-U	WSPSNR-V
	Trolley	-18.9%	-21.4%	-15.2%	-18.3%	-21.3%	-15.3%	-18.7%	-21.4%	-15.4%
	GasLamp	-17.1%	-21.6%	-21.5%	-17.1%	-21.6%	-21.5%	-18.1%	-21.8%	-21.5%
	Skateboardinginlot	-1.7%	-4.5%	-6.2%	-1.7%	-4.4%	-6.2%	-2.0%	-4.6%	-6.5%
	ChairliftRide	-1.5%	-5.8%	-5.7%	-1.6%	-5.7%	-5.6%	-2.0%	-5.6%	-5.6%
8K	KiteFlite	-8.6%	-8.0%	-5.6%	-8.6%	-8.0%	-5.5%	-9.0%	-8.1%	-5.6%
	Harbor	-19.7%	-30.6%	-31.2%	-19.3%	-30.6%	-31.0%	-21.4%	-30.9%	-31.1%
	SkateboardTrick	-7.6%	-17.0%	-17.5%	-8.2%	-16.9%	-17.4%	-8.3%	-17.2%	-17.6%
	Train	-22.1%	-23.4%	-27.8%	-22.2%	-23.5%	-28.0%	-22.3%	-23.7%	-28.1%
	平均	-12.1%	-16.5%	-16.3%	-12.1%	-16.5%	-16.3%	-12.7%	-16.7%	-16.4%
	Balboa	-1.4%	-3.7%	-4.2%	-1.4%	-3.6%	-4.0%	-2.0%	-3.6%	-4.0%
	Broadway	-0.9%	-3.7%	-5.0%	-1.0%	-3.6%	-5.0%	-1.8%	-3.7%	-5.0%
6K	Landing	-3.2%	-4.1%	-4.5%	-3.4%	-3.9%	-4.4%	-3.8%	-4.1%	-4.5%
	BranCastle	-2.3%	-0.8%	-1.4%	-2.5%	-0.8%	-1.4%	-2.8%	-0.8%	-1.4%
	平均	-1.9%	-3.1%	-3.8%	-2.1%	-3.0%	-3.7%	-2.6%	-3.0%	-3.7%
	PoleVault	-5.4%	-10.7%	-8.9%	-5.2%	-10.4%	-8.5%	-5.9%	-10.9%	-8.7%
	AerialCity	-1.3%	-14.0%	-15.4%	-1.0%	-13.7%	-15.2%	-1.1%	-13.9%	-15.3%
4K	DrivingInCity	-4.4%	-8.8%	-8.6%	-4.4%	-8.8%	-8.5%	-4.8%	-8.8%	-8.6%
	DirvingInCountry	-2.6%	-1.8%	-3.4%	-2.4%	-1.6%	-3.2%	-2.8%	-1.7%	-3.3%
	平均	-3.4%	-8.8%	-9.1%	-3.2%	-8.6%	-8.8%	-3.7%	-8.8	-9.0%
	总体平均	-7.4%	-11.3%	-11.4%	-7.4%	-11.2%	-11.3%	-7.9%	-11.3%	-11.4%

类别	序列	SPSNR	CPP-PSNR	WS-PSNR
	Trolley	-18.2%	-17.6%	-18.1%
	GasLamp	-16.7%	-16.7%	-17.7%
	Skateboardinginlot	-1.6%	-1.7%	-1.9%
8K	ChairliftRide	-1.2%	-1.2%	-1.6%
	KiteFlite	-8.7%	-8.7%	-9.0%
	Harbor	-18.6%	-18.2%	-20.3%
	SkateboardTrick	-6.5%	-7.5%	-7.4%
	Train	-21.9%	-22.1%	-22.3%
	Balboa	-0.9%	-0.9%	-1.6%
6K	Broadway	-0.6%	-0.7%	-1.6%
	Landing	-3.2%	-3.4%	-3.8%
	BranCastle	-2.3%	-2.5%	-2.8%
	PoleVault	-5.4%	-5.2%	-5.9%
4K	AerialCity	-0.2%	0.3%	0.1%
	DrivingInCity	-4.3%	-4.2%	-4.7%
	DirvingInCountry	-2.4%	-2.4%	-2.8%
	总体平均	-7.0%	-7.0%	-7.6%

类别	序列	文献[8]	文献[10]	文献[13]	本文算法DTD-SRDO
类别	序列	WS	WS	WS	S	CPP	WS
	Trolley	0.2%	-0.1%	-2.6%	-18.9%	-18.3%	-18.7%
	GasLamp	0.0%	0.0%	-2.4%	-17.1%	-17.1%	-18.1%
8K	Skateboardinginlot	1.2%	-1.8%	-4.0%	-1.7%	-1.7%	-2.0%
	ChairliftRide	-2.4%	-2.9%	-3.8%	-1.5%	-1.6%	-2.0%
	KiteFlite	0.2%	-0.1%	-1.6%	-8.6%	-8.6%	-9.0%
	Harbor	1.5%	0.0%	-2.4%	-19.7%	-19.3%	-21.4%
	PoleVault	-0.2%	-0.1%	-3.8%	-5.4%	-5.2%	-5.9%
4K	AerialCity	-1.6%	-2.5%	-3.8%	-1.3%	-1.0%	-1.1%
	DrivingInCity	-0.8%	-0.7%	-3.0%	-4.4%	-4.4%	-4.8%
	DirvingInCountry	-3.1%	-3.3%	-4.8%	-2.6%	-2.4%	-2.8%
	总体平均	-0.5%	-1.2%	-3.2%	-8.1%	-8.0%	-8.6%

基于球域失真空-时依赖的全景视频编码

Panoramic video coding based on spherical distortion with spatio-temporal dependency

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 24

相关文章 15

Metrics

推荐阅读 0

类别	序列	基准/h	TD-SRDO		STD-SRDO
类别	序列	基准/h	编码时间/h	比率	编码时间/h	比率
	Trolley	135	135	100%	114	84%
	GasLamp	67	62	93%	55	82%
	Skateboardinginlot	401	410	102%	381	95%
8K	ChairliftRide	243	256	105%	235	97%
	KiteFlite	167	165	98%	152	91%
	Harbor	129	122	95%	98	76%
	SkateboardTrick	393	371	94%	316	80%
	Train	269	256	95%	219	82%
	Balboa	584	571	98%	546	93%
6K	Broadway	615	615	100%	576	94%
	Landing	489	490	101%	467	95%
	BranCastle	593	595	101%	567	96%
	PoleVault	335	337	101%	302	90%
4K	AerialCity	247	257	104%	215	87%
	DrivingInCity	320	321	101%	293	92%
	DirvingInCountry	376	381	101%	350	93%
	总计	5 370	5 352	99%	4 894	91%

[1]	仪双燕, 梁永生, 陆晶晶, 柳伟, 胡涛, 何震宇. 联合低秩重构和投影重构的稳健特征选择方法[J]. 通信学报, 2023, 44(3): 209-219.
[2]	尹安琪, 郭渊博, 汪定, 曲彤洲, 陈琳. 可证明安全的抗量子两服务器口令认证密钥交换协议[J]. 通信学报, 2022, 43(3): 14-29.
[3]	于斌, 张南, 陆旭, 段振华, 田聪. 基于运行时验证的边缘服务器DoS攻击检测方法[J]. 通信学报, 2021, 42(9): 75-86.
[4]	郭方方, 吕宏武, 任威霖, 王瑞妮. 基于有监督判别投影的网络安全数据降维算法[J]. 通信学报, 2021, 42(6): 84-93.
[5]	王宏禹,邱天爽. 信号特征谱表示与平稳随机信号谱分解统一的研究[J]. 通信学报, 2018, 39(12): 1-9.
[6]	王宏禹,邱天爽. 确定性信号分解与平稳随机信号分解的统一研究[J]. 通信学报, 2016, 37(10): 1-8.
[7]	王超,杨静,张健沛,吕刚. 基于投影区域密度划分的k匿名算法[J]. 通信学报, 2015, 36(8): 125-134.
[8]	杨吉松,刘皓,孙辰. 基于空间投影的全双工MIMO中继站自反馈干扰抑制[J]. 通信学报, 2014, 35(9): 156-163.
[9]	王树鹏,陈明,吴广君. 面向互联网的大规模重复图像检索技术研究[J]. 通信学报, 2014, 35(12): 196-202.
[10]	王树鹏，陈明，吴广君. 面向互联网的大规模重复图像检索技术研究[J]. 通信学报, 2014, 35(12): 23-202.
[11]	张鹏,王新成,周庆. 基于投影寻踪分析的芯片硬件木马检测[J]. 通信学报, 2013, 34(4): 122-126.
[12]	张鹏，王新成，周庆. 基于投影寻踪分析的芯片硬件木马检测[J]. 通信学报, 2013, 34(4): 14-126.
[13]	李钊,李建东,肖丽媛. 认知MIMO系统基于授权通信模式信息的空频域机会接入[J]. 通信学报, 2012, 33(1): 38-44.
[14]	刘修生. 形式幂级数环中的循环码[J]. 通信学报, 2011, 32(2): 68-71.
[15]	王鼎,王超,吴瑛. 幅相误差对MUSIC算法空域谱及分辨性能影响的分析[J]. 通信学报, 2010, 31(4): 55-63.