基于少量图像的三维重建综述

doi:10.11959/j.issn.2096-6652.202240

摘要/Abstract

摘要：

基于少量图像的三维重建被认为是第三代人工智能的经典应用之一。在计算机图形学和计算机视觉领域，基于少量图像的三维重建任务因具有广泛的应用场景和很高的研究价值，长期以来吸引着众多学者的目光。引入深度学习方法后，该领域于近年来得到了长足发展。对此类基于少量图像的三维重建任务进行了全面阐述，并介绍了本研究组在该方面的系列工作，对其中涉及的数据类型进行分析，阐明其适用性和一般处理方法。此外，对常见的数据集进行分析、整理，针对不同重建方法，归纳出其基本框架、思路。最后，展示了一些常见三维重建的代表性实验结果，并提出了未来可能的研究方向。

关键词: 三维重建, 小样本学习, 人体重建

Abstract:

Few-shot 3D reconstruction is considered one of the classic applications of the third generation of artificial intelligence.In the area of computer graphics and computer vision, few-shot 3D reconstruction has attracted the attention of many researchers during the past several decades because of its wide application scenarios and high research value.The area has grown significantly in recent years after the introduction of deep learning methods.The state-of-the-art methods in image-based few-shot 3D reconstruction were reviewed comprehensively and the series of works of our research group were introduced.The various 3D data types were introduced, and their applicability and general processing procedures in 3D reconstruction were discussed.Furthermore, the most widely used datasets were categorized.Finally, some representative experimental results of common 3D reconstructions were presented, and potential future research directions were proposed.

Key words: 3D reconstruction, few-shot learning, human reconstruction

中图分类号:

TP391.41

于航, 付彦伟, 姜柏言, 等. 基于少量图像的三维重建综述[J]. 智能科学与技术学报, 2022, 4(4): 544-559.

Hang YU, Yanwei FU, Boyan JIANG, et al. A survey of image-based few-shot 3D reconstruction[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 544-559.

图/表 14

图1

图2

图3

图4

图5

表1

图6

图7

图8

图9

图10

图11

图12

表2

参考文献 39

[1]	CHANG A X , FUNKHOUSER T , GUIBAS L ,et al. ShapeNet:an information-rich 3D model repository[J]. arXiv preprint, 2015,arXiv:1512.03012.
[2]	TATARCHENKO M , RICHTER S R , RANFTL R ,et al. What do single-view 3D reconstruction networks learn?[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019: 3400-3409.
[3]	DENG J , DONG W , SOCHER R ,et al. ImageNet:a large-scale hierarchical image database[C]// Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2009: 248-255.
[4]	WANG W , YANG Y J , WANG X ,et al. Development of convolutional neural network and its application in image classification:a survey[J]. Optical Engineering, 2019,58:040901.
[5]	BARRON J L , FLEET D J , BEAUCHEMIN S S . Performance of optical flow techniques[J]. International Journal of Computer Vision, 1994,12(1): 43-77.
[6]	ULLMAN S . The interpretation of structure from motion[J]. The Interpretation of Visual Motion, 1979: 133-175.
[7]	LORENSEN W E , CLINE H E . Marching cubes:a high resolution 3D surface construction algorithm[J]. ACM SIGGRAPH Computer Graphics, 1987,21(4): 163-169.
[8]	LEWINER T , LOPES H , VIEIRA A W ,et al. Efficient implementation of marching cubes’ cases with topological guarantees[J]. Journal of Graphics Tools, 2003,8(2): 1-15.
[9]	PARK J J , FLORENCE P , STRAUB J ,et al. DeepSDF:learning continuous signed distance functions for shape representation[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019: 165-174.
[10]	CHOY C B , XU D F , GWAK J ,et al. 3D-R2N2:an unified approach for single and multi-view 3D object reconstruction[C]// Proceedings of 2016 European Conference on Computer Vision - ECCV 2016. Cham:Springer, 2016: 628-644.
[11]	FAN H Q , SU H , GUIBAS L . A point set generation network for 3D object reconstruction from a single image[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017: 2463-2471.
[12]	CHARLES R Q , HAO S , MO K C ,et al. PointNet:deep learning on point sets for 3D classification and segmentation[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017: 77-85.
[13]	QI C R , YI L , SU H ,et al. PointNet++:deep hierarchical feature learning on point sets in a metric space[J]. arXiv preprint, 2017,arXiv:1706
[14]	WANG N Y , ZHANG Y D , LI Z W ,et al. Pixel2Mesh:generating 3D mesh models from single RGB images[C]// Proceedings of 2018 European Conference on Computer Vision - ECCV 2018. Cham:Springer, 2018: 55-71.
[15]	WEN C , ZHANG Y D , LI Z W ,et al. Pixel2Mesh++:multi-view 3D mesh generation via deformation[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE Press, 2019: 1042-1051.
[16]	KATO H , USHIKU Y , HARADA T . Neural 3D mesh renderer[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 3907-3916.
[17]	GROUEIX T , FISHER M , KIM V G ,et al. A papier-mache approach to learning 3D surface generation[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 216-224.
[18]	MESCHEDER L , OECHSLE M , NIEMEYER M ,et al. Occupancy networks:learning 3D reconstruction in function space[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019: 4455-4465.
[19]	XIANG Y , MOTTAGHI R , SAVARESE S . Beyond PASCAL:a benchmark for 3D object detection in the wild[C]// Proceedings of IEEE Winter Conference on Applications of Computer Vision. Piscataway:IEEE Press, 2014: 75-82.
[20]	SUN X Y , WU J J , ZHANG X M ,et al. Pix3D:dataset and methods for single-image 3D shape modeling[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 2974-2983.
[21]	XIANG Y , KIM W , CHEN W ,et al. ObjectNet3D:a large scale database for 3D object recognition[C]// Proceedings of 2016 European Conference on Computer Vision - ECCV 2016. Cham:Springer, 2016: 160-176.
[22]	GEIGER A , LENZ P , URTASUN R . Are we ready for autonomous driving? The KITTI vision benchmark suite[C]// Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2012: 3354-3361.
[23]	KANAZAWA A , BLACK M J , JACOBS D W ,et al. End-to-end recovery of human shape and pose[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018: 7122-7131.
[24]	ZHU H , ZUO X X , WANG S ,et al. Detailed human shape estimation from a single image by hierarchical mesh deformation[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019: 4486-4495.
[25]	KOLOTOUROS N , PAVLAKOS G , DANIILIDIS K . Convolutional mesh regression for single-image human shape reconstruction[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019: 4496-4505.
[26]	SAITO S , HUANG Z , NATSUME R ,et al. PIFu:pixel-aligned implicit function for high-resolution clothed human digitization[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE Press, 2019: 2304-2314.
[27]	LIANG J B , LIN M . Shape-aware human pose and shape reconstruction using multi-view images[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE Press, 2019: 4351-4361.
[28]	YU H , CHEANG C , FU Y W ,et al. Multi-view shape generation for 3D human-like body[J]. ACM Transactions on Multimedia Computing,Communications,and Applications, 2022.
[29]	WANG J S , WEN C , FU Y W ,et al. Neural pose transfer by spatially adaptive instance normalization[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2020: 5830-5838.
[30]	XU C M , FU Y W , WEN C ,et al. Pose-guided person image synthesis in the non-iconic views[J]. IEEE Transactions on Image Processing, 2020,29: 9060-9072.
[31]	JIANG B Y , ZHANG Y D , WEI X K ,et al. Learning compositional representation for 4D captures with neural ODE[C]// Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2021: 5336-5346.
[32]	CHEN R T Q , RUBANOVA Y , BETTENCOURT J ,et al. Neural ordinary differential equations[J]. arXiv preprint, 2018,arxiv:18066.
[33]	JIANG B Y , ZHANG Y D , WEI X K ,et al. H4D:human 4D modeling by learning neural compositional representation[J]. arXiv preprint, 2022,arXiv:2203.01247.
[34]	CHO K , MERRIENBOER B V , BAHDANAU D ,et al. On the properties of neural machine translation:encoder-decoder approaches[C]// Proceedings of the 8th Workshop on Syntax,Semantics and Structure in Statistical Translation.[S.l.:s.n.], 2014: 103-111.
[35]	JOHNSON S , EVERINGHAM M . Clustered pose and nonlinear appearance models for human pose estimation[C]// Proceedings of the British Machine Vision Conference 2010.[S.l.:s.n.], 2010.
[36]	SIGAL L , BALAN A O , BLACK M J . HumanEva:synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion[J]. International Journal of Computer Vision, 2009,87(1/2): 4-27.
[37]	IONESCU C , PAPAVA D , OLARU V ,et al. Human3.6M:large scale datasets and predictive methods for 3D human sensing in natural environments[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014,36(7): 1325-1339.
[38]	GUO K W , XU F , YU T ,et al. Real-time geometry,albedo and motion reconstruction using a single RGBD camera[J]. ACM Transactions on Graphics, 2017,36(4).
[39]	XIAO C W , YANG D W , LI B ,et al. MeshAdv:adversarial meshes for visual recognition[C]// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019: 6891-6900.

类别			CD					EMD
类别	3D-R2N2	PSG	N3MR	AtlasNet	P2M	3D-R2N2	PSG	N3MR	AtlasNet	P2M
飞机	0.895	0.430	0.450	0.468	0.477	0.606	0.396	7.498	0.659	0.579
长凳	1.891	0.629	2.268	0.703	0.624	1.136	1.113	11.766	1.204	0.965
衣橱	0.735	0.439	2.555	0.433	0.381	2.520	2.986	17.062	2.503	2.563
汽车	0.845	0.333	2.298	0.340	0.268	1.670	1.747	11.641	1.407	1.297
椅子	1.432	0.645	2.084	0.724	0.610	1.466	1.946	11.809	1.534	1.399
显示器	1.707	0.722	3.111	0.848	0.755	1.667	1.891	14.097	1.641	1.536
台灯	4.009	1.193	3.013	1.575	1.295	1.424	1.222	14.741	1.350	1.314
音响	1.507	0.756	3.343	0.812	0.739	2.732	3.490	16.720	3.108	2.951
枪支	0.993	0.423	2.641	0.461	0.453	0.688	0.397	11.889	0.735	0.667
沙发	1.135	0.549	3.512	0.621	0.490	2.114	2.207	14.876	1.910	1.642
桌子	1.116	0.517	2.383	0.577	0.498	1.641	2.121	12.842	1.673	1.480
手机	1.137	0.438	4.366	0.443	0.421	0.912	1.019	17.649	0.868	0.724
船只	1.215	0.633	2.154	0.839	0.670	0.935	0.945	11.425	0.907	0.814
平均	1.432	0.593	2.629	0.680	0.591	1.501	1.652	13.386	1.500	1.380

数据集（缩写）	类型	数量	模特数	姿态数/场景数	提出机构	开源性
LSP	真实	2K	2K	2K	利兹大学	公开
LSP-extended	真实	10K	10K	10K	利兹大学	公开
HumanEva-I	真实	37K/6.8K/24K	4	76	MPI/布朗大学	注册
HumanEva-II	真实	2 460	2	2	MPI/布朗大学	注册
Human3.6M	真实	3.6M	11/3/4	15	IMAR	申请
TOSCA	人工	39	27/12	5	Technion	公开
SHREC	人工	1 184	1 184	5	Technion	公开
FAUST	真实	200/100	10	1	MPI	注册
DFAUST	真实	40K	10	1	MPI	注册
BUFF	真实	11 054	6	11 054	MPI	注册
MPII-Pose	真实	25K	40K	25K/410	MPI	注册
MS-COCO	真实	250K	250K	250K	MPI	公开
SURREAL	人工	5 901/5 281/2 118	115/30	1 964/703	MPI	申请
UP-3D	人工	8 515	5 703/1 423	8 515	MPI	公开
UP-S31	人工	8 515	8 515	8 515	MPI	公开
UP-P14	人工	8 128	8 128	8 128	MPI	公开
UP-P91	人工	8 128	8 128	8 128	MPI	公开
AMASS	真实	11K	300	11K	MPI	注册
FHB	真实	100K	26	45	MPI	申请
ObMan	人工	21K	2 772	21K	MPI	申请
RECON	人工	150	150	150	南京大学	公开
SYN	人工	300	300	300	南京大学	公开
Lifescan	真实-人工	-	2 043	2 043	MPI	私有
3DPeople	真实	2M	80	70	IRI.UPC-CSIC	公开
People Snapshot	人工	自提取帧	24	11	MPI	申请
MINI	人工	12K	12	12K	MPI	申请
Multi Garment	人工	96	96	96	MPI	注册
THUman	真实	7K	7K	7K	清华大学	申请
THUman2.0	真实	500	50	500	清华大学	申请
PROX	真实	100K	20	12	MPI	注册
GRAB	真实	1.6M	5/5/51	1 334	MPI	注册
HUMBI	真实	1.5K	772	772	MPI	注册