基于3D分层卷积融合的多模态生理信号情绪识别

doi:10.11959/j.issn.2096-6652.202108

智能科学与技术学报 ›› 2021, Vol. 3 ›› Issue (1): 76-84.doi: 10.11959/j.issn.2096-6652.202108

基于3D分层卷积融合的多模态生理信号情绪识别

凌文芬¹^,², 陈思含¹^,², 彭勇¹^,², 孔万增¹^,²

¹ 杭州电子科技大学计算机学院，浙江杭州 310018
² 浙江省脑机协同智能重点实验室，浙江杭州 310018

修回日期:2021-02-05 出版日期:2021-03-15 发布日期:2021-03-01
作者简介:凌文芬（1995- ），女，杭州电子科技大学计算机学院硕士生，主要研究方向为情绪生理信号、特征提取、情绪识别。
陈思含（1997- ），女，杭州电子科技大学计算机学院硕士生，主要研究方向为情绪生理信号、特征提取、情绪识别。
彭勇（1985- ），男，博士，杭州电子科技大学计算机学院副教授，主要研究方向为机器学习、模式识别与脑机交互算法及应用。
孔万增（1980- ），男，博士，杭州电子科技大学计算机学院认知与智能计算研究所教授、博士生导师，杭州电子科技大学研究生院副院长，浙江省脑机协同智能重点实验室主任，主要研究方向为人工智能与模式识别、嵌入式可穿戴计算、脑机交互与认知计算等。
基金资助:
国家重点研发计划基金资助项目(2017YFE0116800);国家自然科学基金资助项目(U1909202);浙江省科技计划项目(2018C04012);浙江省“脑机协同智能”重点实验室开放基金项目(20200E10010)

Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion

Wenfen LING¹^,², Sihan CHEN¹^,², Yong PENG¹^,², Wanzeng KONG¹^,²

¹ College of Computer Science and Techonology, Hangzhou Dianzi University, Hangzhou 310018, China
² Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou 310018, China

Revised:2021-02-05 Online:2021-03-15 Published:2021-03-01
Supported by:
The National Key Research and Development Program of China(2017YFE0116800);The National Natural Science Foundation of China(U1909202);Science and Technology Program of Zhejiang Province(2018C04012);Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province(20200E10010)

摘要/Abstract

摘要：

近年来，脑电等生理信号由于能客观体现真实情绪已逐渐成为情绪识别研究的热门对象。然而，单模态的脑电信号存在情绪信息特征不完备问题，多模态生理信号存在情绪信息交互不充分问题。针对这些问题，提出基于3D分层卷积的多模态特征融合模型，旨在充分挖掘多模态交互关系，更准确地刻画情感信息。首先分别通过深度可分离卷积网络提取脑电、眼电和肌电3种模态的生理信号的多模态初级情绪特征信息，再对得到的多模态初级情绪特征信息进行3D卷积融合操作，实现两两模态间的局部交互以及所有模态间的全局交互，获取包含不同生理信号情绪特征的多模态融合特征。实验结果表明，提出的模型在DEAP数据集的效价、唤醒度的二分类和四分类任务中达到了98%的平均准确率。

关键词: 生理信号, 情绪识别, 3D分层卷积, 多模态交互

Abstract:

In recent years, physiological signals such as electroencephalograhpy (EEG) have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However, the single-modal EEG signal has the problem of incomplete emotional information representation, and the multi-modal physiological signal has the problem of insufficient emotional information interaction.Therefore, a 3D hierarchical convolutional fusion model was proposed, which aimed to fully explore multi-modal interaction relationships and more accurately describe emotional information.The method first extracted the primary emotional representation information of EEG , electro-oculogram (EOG) and electromyography (EMG) by depthwise separable convolution network, and then performed 3D convolution fusion operation on the obtained multi-modal primary emotional representation information to realize the pairwise mode local interactions between states and global interactions among all modalities, so as to obtain multi-modal fusion representations containing emotional characteristics of different physiological signals.The results show that the accuracy in the valence and arousal of the two-class and four-class tasks on DEAP dataset are both 98% by the proposed model.

Key words: physiological signal, emotion recognition, 3D hierarchical convolutional, multi-modal interaction

中图分类号:

TP18，TN911.7，R318

凌文芬, 陈思含, 彭勇, 等. 基于3D分层卷积融合的多模态生理信号情绪识别[J]. 智能科学与技术学报, 2021, 3(1): 76-84.

Wenfen LING, Sihan CHEN, Yong PENG, et al. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 76-84.

图/表 14

图1

图2

图3

图4

图5

图6

表1

表2

单模态脑电信号情绪特征和多模态生理信号融合情绪特征在不同分类任务中的平均分类准确率和标准差"

特征	效价	唤醒度	效价-唤醒度
单模态脑电信号情绪特征	0.940 2± 0.057 1	0.933 9± 0.085 8	0.891 5± 0.059 5
多模态生理信号融合情绪特征	$0 . 9831 \pm 0 . 0145$	$0 . 9847 \pm 0 . 0178$	$0 . 9821 \pm 0 . 0168$

表2

图7

图8

图9

图10

图11

表3

不同多模态特征融合方法在DEAP数据集上的分类准确率"

方法	模态	效价	唤醒度	效价-唤醒度
CNN^[19]	EEG+全部外围生理信号	85.5	87.3	—
DCCA^[11]	EEG+全部外围生理信号	85.62	84.33	85.51
MM-ResLSTM^[20]	EEG+全部外围生理信号	92.3±1.55	92.87±2.11	—
CNN+LSTM^[10]	EEG+全部外围生理信号	91.95	93.06	—
本文方法	EEG+EOG+EMG	$98 . 55 \pm 1 . 33$	$98 . 77 \pm 1 . 57$	$98 . 67 \pm 1 . 38$

表3

参考文献 20

[1]	DORNAIKA F , RADUCANU B . Efficient facial expression recognition for human robot interaction[C]// Proceedings of the 9th International Work-Conference on Artificial Neural Networks. Berlin:Springer, 2007: 700-708.
[2]	BARTNECK C , LYONS M J . HCI and the face:towards an art of the soluble[C]// Proceedings of the 21th International Conference on Human-Computer Interaction. Berlin:Springer, 2007: 20-29.
[3]	DE NADAI S , D'INCà M , PARODI F ,et al. Enhancing safety of transport by road by on-line monitoring of driver emotions[C]// Proceedings of the 2016 11th System of Systems Engineering Conference. Piscataway:IEEE Press, 2016: 1-4.
[4]	GUO R , LI S J , HE L ,et al. Pervasive and unobtrusive emotion sensing for human mental health[C]// Proceedings of the 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops. Piscataway:IEEE Press, 2013: 436-439.
[5]	ZHAN C , LI W , OGUNBONA P ,et al. A real-time facial expression recognition system for online games[J]. International Journal of Computer Games Technology, 2008: 542918.
[6]	LIN S , XIE J Y , YANG M Y ,et al. A review of emotion recognition using physiological signals[J]. Sensors, 2018,18(7): 2074.
[7]	SNOEK C G M , WORRING M , SMEULDERS A W M . Early versus late fusion in semantic video analysis[C]// Proceedings of the 13th annual ACM International Conference on Multimedia. New York:ACM Press, 2005: 399-402.
[8]	HASSAN M M , ALAM M G R , UDDIN M Z ,et al. Human emotion recognition using deep belief network architecture[J]. Information Fusion, 2019,51: 10-18.
[9]	KWON Y H , SHIN S B , KIM S D . Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system[J]. Sensors, 2018,18(5): 1383.
[10]	LIAO J X , ZHONG Q H , ZHU Y S ,et al. Multimodal physiological signal emotion recognition based on convolutional recurrent neural network[J]. IOP Conference Series Materials Science and Engineering, 2020,782: 032005.
[11]	QIU L , LIU W , LYU B L . Multi-view emotion recognition using deep canonical correlation analysis[C]// Proceedings of the 25th International Conference on Neural Information Processing. Cham:Springer, 2018: 221-231.
[12]	ZHAO Y X , CAO X Y , LIN J L ,et al. Multimodal emotion recognition model using physiological signals[J]. arXiv preprint, 2019,arXiv:1911.12918.
[13]	HUANG H P , HU Z C , WANG W M ,et al. Multimodal emotion recognition based on ensemble convolutional neural network[J]. IEEE Access, 2019,8: 3265-3271.
[14]	HOWARD A G , ZHU M L , CHEN B ,et al. MobileNets:efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint, 2017,arXiv:1704.04861.
[15]	CHOLLET F , . Xception:deep learning with depthwise separable convolutions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017: 1251-1258.
[16]	LAWHERN V J , SOLON A J , WAYTOWICH N R ,et al. EEGNet:a compact convolutional neural network for EEG-based brain-computer interfaces[J]. Journal of Neural Engineering, 2018,15(5): 056013.
[17]	KOELSTRA S , MUHL C , SOLEYMANI M ,et al. DEAP:a database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2011,3(1): 18-31.
[18]	YANG Y L , WU Q F , QIU M ,et al. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network[C]// Proceedings of the 2018 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2018: 1-7.
[19]	LIN W Q , LI C , SUN S Q . Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal[C]// Proceedings of the International Conference on Image and Graphics. Cham:Springer, 2017: 385-394.
[20]	MA J X , TANG H , ZHENG W L ,et al. Emotion recognition using multimodal residual LSTM network[C]// Proceedings of the 27th ACM International Conference on Multimedia. New York:ACM Press, 2019: 176-183.

模型	网络层
特征提取模型	N=8、M=2、S=16
基于3D分层卷积的多模态特征融合模型	H₁=32、H₂=64

基于3D分层卷积融合的多模态生理信号情绪识别

Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 20

相关文章 2

Metrics

推荐阅读 0

[1]	吕宝粮, 张亚倩, 郑伟龙. 情感脑机接口研究综述[J]. 智能科学与技术学报, 2021, 3(1): 36-48.
[2]	刘栋军, 王宇涵, 凌文芬, 彭勇, 孔万增. 基于脑机协同智能的情绪识别[J]. 智能科学与技术学报, 2021, 3(1): 65-75.