基于3D分层卷积融合的多模态生理信号情绪识别

doi:10.11959/j.issn.2096-6652.202108

Abstract

Abstract:

In recent years, physiological signals such as electroencephalograhpy (EEG) have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However, the single-modal EEG signal has the problem of incomplete emotional information representation, and the multi-modal physiological signal has the problem of insufficient emotional information interaction.Therefore, a 3D hierarchical convolutional fusion model was proposed, which aimed to fully explore multi-modal interaction relationships and more accurately describe emotional information.The method first extracted the primary emotional representation information of EEG , electro-oculogram (EOG) and electromyography (EMG) by depthwise separable convolution network, and then performed 3D convolution fusion operation on the obtained multi-modal primary emotional representation information to realize the pairwise mode local interactions between states and global interactions among all modalities, so as to obtain multi-modal fusion representations containing emotional characteristics of different physiological signals.The results show that the accuracy in the valence and arousal of the two-class and four-class tasks on DEAP dataset are both 98% by the proposed model.

Key words: physiological signal, emotion recognition, 3D hierarchical convolutional, multi-modal interaction

CLC Number:

TP18，TN911.7，R318

Wenfen LING, Sihan CHEN, Yong PENG, et al. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 76-84.

Figures/Tables 14

特征	效价	唤醒度	效价-唤醒度
单模态脑电信号情绪特征	0.940 2± 0.057 1	0.933 9± 0.085 8	0.891 5± 0.059 5
多模态生理信号融合情绪特征	$0 . 9831 \pm 0 . 0145$	$0 . 9847 \pm 0 . 0178$	$0 . 9821 \pm 0 . 0168$

方法	模态	效价	唤醒度	效价-唤醒度
CNN^[19]	EEG+全部外围生理信号	85.5	87.3	—
DCCA^[11]	EEG+全部外围生理信号	85.62	84.33	85.51
MM-ResLSTM^[20]	EEG+全部外围生理信号	92.3±1.55	92.87±2.11	—
CNN+LSTM^[10]	EEG+全部外围生理信号	91.95	93.06	—
本文方法	EEG+EOG+EMG	$98 . 55 \pm 1 . 33$	$98 . 77 \pm 1 . 57$	$98 . 67 \pm 1 . 38$

References 20

[1]	DORNAIKA F , RADUCANU B . Efficient facial expression recognition for human robot interaction[C]// Proceedings of the 9th International Work-Conference on Artificial Neural Networks. Berlin:Springer, 2007: 700-708.
[2]	BARTNECK C , LYONS M J . HCI and the face:towards an art of the soluble[C]// Proceedings of the 21th International Conference on Human-Computer Interaction. Berlin:Springer, 2007: 20-29.
[3]	DE NADAI S , D'INCà M , PARODI F ,et al. Enhancing safety of transport by road by on-line monitoring of driver emotions[C]// Proceedings of the 2016 11th System of Systems Engineering Conference. Piscataway:IEEE Press, 2016: 1-4.
[4]	GUO R , LI S J , HE L ,et al. Pervasive and unobtrusive emotion sensing for human mental health[C]// Proceedings of the 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops. Piscataway:IEEE Press, 2013: 436-439.
[5]	ZHAN C , LI W , OGUNBONA P ,et al. A real-time facial expression recognition system for online games[J]. International Journal of Computer Games Technology, 2008: 542918.
[6]	LIN S , XIE J Y , YANG M Y ,et al. A review of emotion recognition using physiological signals[J]. Sensors, 2018,18(7): 2074.
[7]	SNOEK C G M , WORRING M , SMEULDERS A W M . Early versus late fusion in semantic video analysis[C]// Proceedings of the 13th annual ACM International Conference on Multimedia. New York:ACM Press, 2005: 399-402.
[8]	HASSAN M M , ALAM M G R , UDDIN M Z ,et al. Human emotion recognition using deep belief network architecture[J]. Information Fusion, 2019,51: 10-18.
[9]	KWON Y H , SHIN S B , KIM S D . Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system[J]. Sensors, 2018,18(5): 1383.
[10]	LIAO J X , ZHONG Q H , ZHU Y S ,et al. Multimodal physiological signal emotion recognition based on convolutional recurrent neural network[J]. IOP Conference Series Materials Science and Engineering, 2020,782: 032005.
[11]	QIU L , LIU W , LYU B L . Multi-view emotion recognition using deep canonical correlation analysis[C]// Proceedings of the 25th International Conference on Neural Information Processing. Cham:Springer, 2018: 221-231.
[12]	ZHAO Y X , CAO X Y , LIN J L ,et al. Multimodal emotion recognition model using physiological signals[J]. arXiv preprint, 2019,arXiv:1911.12918.
[13]	HUANG H P , HU Z C , WANG W M ,et al. Multimodal emotion recognition based on ensemble convolutional neural network[J]. IEEE Access, 2019,8: 3265-3271.
[14]	HOWARD A G , ZHU M L , CHEN B ,et al. MobileNets:efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint, 2017,arXiv:1704.04861.
[15]	CHOLLET F , . Xception:deep learning with depthwise separable convolutions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017: 1251-1258.
[16]	LAWHERN V J , SOLON A J , WAYTOWICH N R ,et al. EEGNet:a compact convolutional neural network for EEG-based brain-computer interfaces[J]. Journal of Neural Engineering, 2018,15(5): 056013.
[17]	KOELSTRA S , MUHL C , SOLEYMANI M ,et al. DEAP:a database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2011,3(1): 18-31.
[18]	YANG Y L , WU Q F , QIU M ,et al. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network[C]// Proceedings of the 2018 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2018: 1-7.
[19]	LIN W Q , LI C , SUN S Q . Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal[C]// Proceedings of the International Conference on Image and Graphics. Cham:Springer, 2017: 385-394.
[20]	MA J X , TANG H , ZHENG W L ,et al. Emotion recognition using multimodal residual LSTM network[C]// Proceedings of the 27th ACM International Conference on Multimedia. New York:ACM Press, 2019: 176-183.

Metrics

Recommended 0

No Suggested Reading articles found!

模型	网络层
特征提取模型	N=8、M=2、S=16
基于3D分层卷积的多模态特征融合模型	H₁=32、H₂=64

Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 20

Related Articles 2

Metrics

Recommended 0

[1]	Bao-Liang LU, Yaqian ZHANG, Wei-Long ZHENG. A survey of affective brain-computer interface [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 36-48.
[2]	Dongjun LIU, Yuhan WANG, Wenfen LING, Yong PENG, Wanzeng KONG. Emotion recognition based on brain and machine collaborative intelligence [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 65-75.