基于典型相关分析的多视图学习方法综述

doi:10.11959/j.issn.2096-6652.202206

摘要/Abstract

摘要：

多视图学习是将不同来源的特征子集加以融合的策略。典型相关分析是多视图学习中的重要方法，旨在最大化不同视图之间的相关性。传统的典型相关分析仅能计算两个视图之间的线性相关性，无法应用于包含多个视图或包含非线性相关性的数据集。此外，如果将典型相关分析应用于有监督任务，其作为一种无监督方法将导致标签信息的浪费。针对上述问题，提出大量非线性的、针对多个视图的、有监督的基于典型相关分析的多视图学习方法。首先，概述经典的基于典型相关分析的多视图方法；然后介绍这些方法在模式识别、跨模态检索和分类，以及多视图嵌入中的典型应用；最后，对基于典型相关分析的多视图学习方法面临的挑战和未来研究方向进行了总结和展望。

关键词: 典型相关分析, 多视图学习, 多模态检索, 多视图嵌入

Abstract:

Multi-view learning (MVL) is a strategy for fusing data from different sources or subsets.Canonical correlation analysis (CCA) is very important in MVL, whose main idea is to maximize the correlation of different views.The traditional CCA can only calculate the linear correlation between two views.Moreover, it is unsupervised, and the label information is wasted in supervised learning tasks.Many nonlinear, supervised, or generalized extensions have been proposed to accommodate these limitations.Firstly, a comprehensive overview of representative CCA approaches was provided.Then their classical applications in pattern recognition, cross-modal retrieval and classification, and multi-view embedding were described.Finally, the challenges and future research directions of CCA-based MVL approaches were pointed out.

Key words: canonical correlation analysis, multi-view learning, multi-modal retrieval, multi-view embedding

中图分类号:

TP181

郭陈凤,伍冬睿. 基于典型相关分析的多视图学习方法综述[J]. 智能科学与技术学报, 2022, 4(1): 14-26.

Chenfeng GUO,Dongrui WU. A survey on canonical correlation analysis based multi-view learning[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(1): 14-26.

图/表 6

图1

图2

表1

表2

本文使用的主要符号的含义"

符号	含义
$X, Y$	视图 $X$ 和视图 $Y$ 的特征矩阵
$(x_{i}, y_{i})$	视图 $X$ 和视图 $Y$ 的第i对样本
N	各视图的样本数
c	类别数
n_i	类别i的样本数
J	视图数
K	典型向量的个数
d_x	视图 $X$ 的特征维度
$w_{x}$	视图X 的一个典型向量
$W_{x}$	视图 $X$ 的典型矩阵
$S_{X Y}$	视图 $X$ 和视图 $Y$ 的协方差矩阵
$\cdot^{⊤}$	矩阵的转置
I	单位矩阵
${‖ \cdot ‖}_{2}$	向量的2范数
${‖ \cdot ‖}_{F}$	矩阵的Frobenius范数
$tr (\cdot)$	矩阵的迹

表2

图3

图4

参考文献 76

[1]	王建民 . 工业大数据技术综述[J]. 大数据, 2017,3(6): 3-14.
	WANG J M . Survey on industrial big data[J]. Big Data Research, 2017,3(6): 3-14.
[2]	郭斌, 翟书颖, 於志文 ,等. 群智大数据:感知、优选与理解[J]. 大数据, 2017,3(5): 57-69.
	GUO B , ZHAI S Y , YU Z W ,et al. Crowdsensing big data:sensing,data selection,and understanding[J]. Big Data Research, 2017,3(5): 57-69.
[3]	李姗, 李永超, 邹颖 ,等. 基于多模态影像下的抑郁症大脑异常[J]. 智能科学与技术学报, 2020,2(2): 116-125.
	LI S , LI Y C , ZOU Y ,et al. Brain abnormalities in depression based on multimodal imaging[J]. Chinese Journal of Intelligent Science and Technology, 2020,2(2): 116-125.
[4]	梁吉业, 钱宇华, 李德玉 ,等. 面向大数据的粒计算理论与方法研究进展[J]. 大数据, 2016,2(4): 13-23.
	LIANG J Y , QIAN Y H , LI D Y ,et al. Research development on granular computing theory and method for big data[J]. Big Data Research, 2016,2(4): 13-23.
[5]	TANG J J , TIAN Y J , ZHANG P ,et al. Multiview privileged support vector machines[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018,29(8): 3463-3477.
[6]	XU C , TAO D C , XU C . A survey on multi-view learning[J]. arXiv preprint, 2013,arXiv:1304.5634.
[7]	XUE X , NIE F , WANG S ,et al. Multi-view correlated feature learning by uncovering shared component[C]// Proceedings of the 31th AAAI Conference on Artificial Intelligence. CA:AAAI, 2017: 2810-2816.
[8]	BALTRU?AITIS T , AHUJA C , MORENCY L P . Multimodal machine learning:a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019,41(2): 423-443.
[9]	BLUM A , MITCHELL T . Combining labeled and unlabeled data with co-training[C]// Proceedings of the 11th Annual Conference on Computational Learning Theory. New York:ACM Press, 1998: 92-100.
[10]	ABNEY S , . Bootstrapping[C]// Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. PA:ACL, 2002: 360-367.
[11]	GEHLER P , NOWOZIN S . On feature combination for multiclass object classification[C]// Proceedings of 2009 IEEE 12th International Conference on Computer Vision. Piscataway:IEEE Press, 2009: 221-228.
[12]	GONEN M , ALPAYDIN E . Multiple kernel learning algorithms[J]. Journal of Machine Learning Research, 2011,12: 2211-2268.
[13]	HOTELLING H . Relations between two sets of variates[J]. Biometrika, 1936,28(3/4): 321-377.
[14]	DIETHE T , HARDOON D R , SHAWE-TAYLOR J , . Multiview fisher discriminant analysis[C]// Proceedings of the Neural Information Processing Systems Workshop on Learning from Multiple Sources. Cambridge:MIT Press, 2008: 1-8.
[15]	XIA T , TAO D C , MEI T ,et al. Multiview spectral embedding[J]. IEEE Transactions on Systems,Man,and Cybernetics,Part B (Cybernetics), 2010,40(6): 1438-1446.
[16]	ZHAO J , XIE X J , XU X ,et al. Multi-view learning overview:recent progress and new challenges[J]. Information Fusion, 2017,38: 43-54.
[17]	SUN S L . A survey of multi-view machine learning[J]. Neural Computing and Applications, 2013,23(7): 2031-2038.
[18]	ZHANG T , DENG Z H , WU D R ,et al. Multiview fuzzy logic system with the cooperation between visible and hidden views[J]. IEEE Transactions on Fuzzy Systems, 2019,27(6): 1162-1173.
[19]	CHAUDHURI K , KAKADE S M , LIVESCU K ,et al. Multi-view clustering via canonical correlation analysis[C]// Proceedings of the 26th Annual International Conference on Machine Learning. New York:ACM Press, 2009: 129-136.
[20]	BLASCHKO M B , LAMPERT C H . Correlational spectral clustering[C]// Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2008: 1-8.
[21]	KAKADE S M , FOSTER D P . Multi-view regression via canonical correlation analysis[M]// Learning theory. Heidelberg: Springer, 2007: 82-96.
[22]	ZHANG Y Y , ZHANG J C , PAN Z S ,et al. Multi-view dimensionality reduction via canonical random correlation analysis[J]. Frontiers of Computer Science, 2016,10(5): 856-869.
[23]	WU G , LI F . A randomized exponential canonical correlation analysis method for data analysis and dimensionality reduction[J]. Applied Numerical Mathematics, 2021,164: 101-124.
[24]	LAI P L , FYFE C . Kernel and nonlinear canonical correlation analysis[J]. International Journal of Neural Systems, 2000,10(5): 365-377.
[25]	SUN T K , CHEN S C . Locality preserving CCA with applications to data visualization and pose estimation[J]. Image and Vision Computing, 2007,25(5): 531-543.
[26]	WANG W R , ARORA R , LIVESCU K ,et al. On deep multi-view representation learning[C]// Proceedings of the 32th International Conference on Machine Learning. Cambridge:JMLR, 2015: 1083-1092.
[27]	ANDREW G , ARORA R , BILMES J ,et al. Deep canonical correlation analysis[C]// Proceedings of the 30th International Conference on Machine Learning. Cambridge:JMLR, 2013: 1247-1255.
[28]	WANG W R , YAN X C , LEE H ,et al. Deep variational canonical correlation analysis[J]. arXiv preprint, 2016,arXiv:1610.03454.
[29]	BENTON A , ARORA R , DREDZE M . Learning multiview embeddings of twitter users[C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2016: 14-19.
[30]	EL DIN ELMADANY N , HE Y F , GUAN L . Multiview learning via deep discriminative canonical correlation analysis[C]// Proceedings of 2016 IEEE International Conference on Acoustics,Speech and Signal Processing. Piscataway:IEEE Press, 2016: 2409-2413.
[31]	LUO Y , TAO D C , RAMAMOHANARAO K ,et al. Tensor canonical correlation analysis for multi-view dimension reduction[J]. IEEE Transactions on Knowledge and Data Engineering, 2015,27(11): 3111-3124.
[32]	VíA J , SANTAMARíA I , PéREZ J , . A learning algorithm for adaptive canonical correlation analysis of several data sets[J]. Neural Networks, 2007,20(1): 139-152.
[33]	CARROLL J D , . Generalization of canonical correlation analysis to three or more sets of variables[C]// Proceedings of the 76th Annual Convention of the American Psychological Association.[S.l.:s.n.], 1968: 227-228.
[34]	SUN T K , CHEN S C , YANG J Y ,et al. A novel method of combined feature extraction for recognition[C]// Proceedings of 2008 8th IEEE International Conference on Data Mining. Piscataway:IEEE Press, 2008: 1043-1048.
[35]	SUN S L , XIE X J , YANG M . Multiview uncorrelated discriminant analysis[J]. IEEE Transactions on Cybernetics, 2016,46(12): 3272-3284.
[36]	FISHER R A . The use of multiple measurements in taxonomic problems[J]. Annals of Eugenics, 1936,7(2): 179-188.
[37]	HARDOON D R , SZEDMAK S , SHAWE-TAYLOR J , . Canonical correlation analysis:an overview with application to learning methods[J]. Neural Computation, 2004,16(12): 2639-2664.
[38]	WITTEN D M , TIBSHIRANI R J . Extensions of sparse canonical correlation analysis with applications to genomic data[J]. Statistical Applications in Genetics and Molecular Biology, 2009,8(1): 1-27.
[39]	WITTEN D M , TIBSHIRANI R , HASTIE T . A penalized matrix decomposition,with applications to sparse principal components and canonical correlation analysis[J]. Biostatistics, 2009,10(3): 515-534.
[40]	CHEN X , . Structured sparse canonical correlation analysis[C]// Proceedings of the 15th International Conference Artificial Intelligence and Statistics. New York:ACM Press, 2012: 199-207.
[41]	LIN D , ZHANG J , LI J ,et al. Group sparse canonical correlation analysis for genomic data integration[J]. BMC Bioinformatics, 2013,14:245.
[42]	LIN D D , CALHOUN V D , WANG Y P . Correspondence between fMRI and SNP data by group sparse canonical correlation analysis[J]. Medical Image Analysis, 2014,18(6): 891-902.
[43]	LOPEZ-PAZ D , SRA S , SMOLA A ,et al. Randomized nonlinear component analysis[C]// Proceedings of the 31st International Conference on Machine Learning. Cambridge:JMLR, 2014: 1359-1367.
[44]	HOEGAERTS L , SUYKENS J A K , VANDEWALLE J ,et al. Subset based least squares subspace regression in RKHS[J]. Neurocomputing, 2005,63: 293-323.
[45]	YANG J F , ZHANG X . Feature-level fusion of fingerprint and finger-vein for personal identification[J]. Pattern Recognition Letters, 2012,33(5): 623-628.
[46]	WANG F S , ZHANG D Q . A new locality-preserving canonical correlation analysis algorithm for multi-view dimensionality reduction[J]. Neural Processing Letters, 2013,37(2): 135-146.
[47]	TIAN Y , SIGAL L , DE LA TORRE F ,et al. Canonical locality preserving latent variable model for discriminative pose inference[J]. Image and Vision Computing, 2013,31(3): 223-230.
[48]	WANG W R , ARORA R , LIVESCU K ,et al. Unsupervised learning of acoustic features via deep canonical correlation analysis[C]// Proceedings of 2015 IEEE International Conference on Acoustics,Speech and Signal Processing. Piscataway:IEEE Press, 2015: 4590-4594.
[49]	BACH F R , JORDAN M I . A probabilistic interpretation of canonical correlation analysis[R]. 2005.
[50]	KINGMA D P , WELLING M . Auto-encoding variational bayes[J]. arXiv preprint, 2013,arXiv:1312.6114.
[51]	KLAMI A , VIRTANEN S , KASKI S . Bayesian canonical correlation analysis[J]. Journal of Machine Learning Research, 2013,14(1): 965-1003.
[52]	WANG C . Variational Bayesian approach to canonical correlation analysis[J]. IEEE Transactions on Neural Networks, 2007,18(3): 905-910.
[53]	VIRTANEN S , KLAMI A , KASKI S . Bayesian CCA via group sparsity[C]// Proceedings of the 28th International Conference on Machine Learning. Madison:Omnipress, 2011: 457-464.
[54]	HORST P . Generalized canonical correlations and their applications to experimental data[J]. Journal of Clinical Psychology, 1961,17(4): 331-347.
[55]	BOUMAL N , MISHRA B , ABSIL P A ,et al. Manopt,a Matlab toolbox for optimization on manifolds[J]. Journal of Machine Learning Research, 2014,15(1): 1455-1459.
[56]	ASENDORF N A . Informative data fusion:beyond canonical correlation analysis[D]. Ann Arbor:The University of Michigan, 2015.
[57]	BENTON A , KHAYRALLAH H , GUJRAL B ,et al. Deep generalized canonical correlation analysis[C]// Proceedings of the 4th Workshop on Representation Learning for NLP. Stroudsburg:Association for Computational Linguistics, 2019: 1-6.
[58]	ZHENG W M , ZHOU X Y , ZOU C R ,et al. Facial expression recognition using kernel canonical correlation analysis (KCCA)[J]. IEEE Transactions on Neural Networks, 2006,17(1): 233-238.
[59]	SHAN C F , GONG S G , MCOWAN P W . Fusing gait and face cues for human gender recognition[J]. Neurocomputing, 2008,71(10/11/12): 1931-1938.
[60]	XING X L , WANG K J , YAN T ,et al. Complete canonical correlation analysis with application to multi-view gait recognition[J]. Pattern Recognition, 2016,50: 107-117.
[61]	ARORA R , LIVESCU K . Multi-view CCA-based acoustic features for phonetic recognition across speakers and domains[C]// Proceedings of 2013 IEEE International Conference on Acoustics,Speech and Signal Processing. Piscataway:IEEE Press, 2013: 7135-7139.
[62]	ZHUANG X W , YANG Z S , CORDES D . A technical review of canonical correlation analysis for neuroscience applications[J]. Human Brain Mapping, 2020,41(13): 3807-3833.
[63]	YONG S , CHEONG P . Analysis of correlation based dimension reduction methods[J]. International Journal of Applied Mathematics and Computer Science, 2011,21(3): 549-558.
[64]	ELMADANY N E D , HE Y F , GUAN L . Information fusion for human action recognition via biset/multiset globality locality preserving canonical correlation analysis[J]. IEEE Transactions on Image Processing, 2018,27(11): 5275-5287.
[65]	CHI J , HUANG X , PENG Y . Zero-shot cross-media retrieval with external knowledge[C]// Proceedings of the 9th International Conference on Internet Multimedia Computing and Service. Singapore:Springer, 2017: 200-211.
[66]	RASIWASIA N , PEREIRA J C , COVIELLO E ,et al. A new approach to cross-modal multimedia retrieval[C]// Proceedings of the 18th ACM International Conference on Multimedia. New York:ACM Press, 2010: 251-260.
[67]	PENG Y X , HUANG X , ZHAO Y Z . An overview of cross-media retrieval:concepts,methodologies,benchmarks,and challenges[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018,28(9): 2372-2385.
[68]	WU F , ZHANG H , ZHUANG Y T . Learning semantic correlations for cross-media retrieval[C]// Proceedings of 2006 International Conference on Image Processing. Piscataway:IEEE Press, 2006: 1465-1468.
[69]	CHANDAR S , KHAPRA M M , LAROCHELLE H ,et al. Correlational neural networks[J]. Neural Computation, 2016,28(2): 257-285.
[70]	DHILLON P , FOSTER D , UNGAR L . Multi-view learning of word embeddings via CCA[C]// Proceedings of the 25th Conference on Neural Information Processing Systems. New York:Curran Associates Inc, 2011: 199-207.
[71]	FARUQUI M , DYER C . Improving vector space word representations using multilingual correlation[C]// Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Stroudsburg:Association for Computational Linguistics, 2014: 462-471.
[72]	CHEN Z W , YANG C H , PENG T ,et al. A cumulative canonical correlation analysis-based sensor precision degradation detection method[J]. IEEE Transactions on Industrial Electronics, 2019,66(8): 6321-6330.
[73]	HUANG H , HE H T , FAN X ,et al. Super-resolution of human face image using canonical correlation analysis[J]. Pattern Recognition, 2010,43(7): 2532-2543.
[74]	AN L , BHANU B . Face image super-resolution using 2D CCA[J]. Signal Processing, 2014,103: 184-194.
[75]	JIANG Q C , YAN X F . Multimode process monitoring using variational Bayesian inference and canonical correlation analysis[J]. IEEE Transactions on Automation Science and Engineering, 2019,16(4): 1814-1824.
[76]	SONG B , SHI H , TAN S ,et al. Multi-subspace orthogonal canonical correlation analysis for quality related plant wide process monitoring[J]. IEEE Transactions on Industrial Informatics, 2020.

方法	多个视图	有监督	非线性	基于DNN
CCA
sCCA
KCCA			3
RCCA			3
LPCCA			3
DisCCA		3
MLDA		3
MULDA		3
DCCA			3	3
DisDCCA		3	3	3
DCCAE			3	3
DisDCCAE		3	3	3
DVCCA			3	3
SUMCOR-CCA	3
LS-CCA	3
GCCA	3
TCCA	3
DGCCA	3	3	3	3