基于并行可分离卷积和标签平滑正则化的脑电情感识别

doi:10.11959/j.issn.1000-0801.2023112

摘要/Abstract

摘要：

近年来，基于深度学习和脑电图（EEG）的情感识别方法取得了较好的效果。然而，现有方法依然存在脑电情感特征提取不够全面、受人工错误标注的情感标签影响较大等问题。对此，提出了并行可分离卷积和标签平滑正则化（PSC-LSR）网络模型。首先，通过注意力机制，赋予EEG重要时间点和重要通道更大的权重，得到 EEG 的浅层情感特征；其次，采用并行可分离卷积模块全面提取 EEG 情感信息，得到深层情感特征；最后，在优化模型参数时采用了情感标签平滑正则化方法，使模型对错误标签有更大的容错概率，增强了网络模型的泛化性和鲁棒性，提高了脑电情感识别的准确率。提出的方法在两个数据集进行了验证，其中，在DEAP数据集中，唤醒和效价两个维度的平均准确率分别达到了99.23%和99.13%；在Dreamer数据集中，唤醒和效价两个维度的平均准确率分别达到了97.33%和97.25%。

关键词: 脑电信号, 情感识别, 标签平滑正则化, 注意力机制

Abstract:

In recent years, emotion recognition methods based on deep learning and electroencephalogram (EEG) have achieved good results.However, existing methods still have issues such as incomplete extraction of emotional features from EEG and significant impact from artificially mislabeled emotional labels.A parallel separable convolution and label smoothing regularization (PSC-LSR) network model was proposed.Firstly, through the attention mechanism, EEG important time points and important channels were given greater weight to obtain shallow emotional features of EEG.Secondly, a parallel separable convolution module was used to comprehensively extract EEG emotional information and obtain deep emotional features.Finally, the emotion label smoothing regularization method was used to optimize the model parameters, which increased model’s fault tolerance probability for incorrect labels, enhanced the generalization and robustness of the network model, and improved accuracy of EEG emotion recognition.The proposed method has been validated in two datasets, in which the average accuracy rates of arousal and valence dimensions in the DEAP dataset reaches 99.23% and 99.13%, respectively.In the Dreamer dataset, the average accuracy rates for both arousal and valence dimensions reaches 97.33% and 97.25%.

Key words: EEG, emotional recognition, label smoothing regularization, attention mechanism

中图分类号:

TP391

张永, 刘纪奎, 柯文龙. 基于并行可分离卷积和标签平滑正则化的脑电情感识别[J]. 电信科学, 2023, 39(5): 116-128.

Yong ZHANG, Jikui LIU, Wenlong KE. EEG emotion recognition based on parallel separable convolution and label smoothing regularization[J]. Telecommunications Science, 2023, 39(5): 116-128.

图/表 15

图1

图2

图3

图4

表1

图5

表2

表3

表4

表5

表6

表7

图6

图7

图8

参考文献 42

[1]	GAO Z K , LI Y L , YANG Y X ,et al. A GPSO-optimized convolutional neural networks for EEG-based emotion recognition[J]. Neurocomputing, 2020(380): 225-235.
[2]	CHEN G J , ZHANG X Y , SUN Y ,et al. Emotion feature analysis and recognition based on reconstructed EEG sources[J]. IEEE Access, 2020(8): 11907-11916.
[3]	WANG Z M , ZHOU X X , WANG W L ,et al. Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video[J]. International Journal of Machine Learning and Cybernetics, 2020,11(4): 923-934.
[4]	KESSOUS L , CASTELLANO G , CARIDAKIS G . Multimodal emotion recognition in speech-based interaction using facial expression,body gesture and acoustic analysis[J]. Journal on Multimodal User Interfaces, 2010,3(1): 33-48.
[5]	ANDERSON K , MCOWAN P W . A real-time automated system for the recognition of human facial expressions[J]. IEEE Transactions on Systems,Man,and Cybernetics Part B,Cybernetics:a Publication of the IEEE Systems,Man,and Cybernetics Society, 2006,36(1): 96-105.
[6]	刘晓旻, 谭华春, 章毓晋 . 人脸表情识别研究的新进展[J]. 中国图象图形学报, 2006,11(10): 1359-1368.
	LIU X M , TAN H C , ZHANG Y J . New research advances in facial expression recognition[J]. Journal of Image and Graphics, 2006,11(10): 1359-1368.
[7]	刘玉娟, 方富熹 . 情绪的语音交流[J]. 中国行为医学科学, 2007,16(4): 374-376.
	LIU J Y , FANG F X . Voice communication of emotions[J]. Chinese Journal of Behavioral Medical and Brain Science, 2007,16(4): 374-376.
[8]	ANG J , DHILLON R , KRUPSKI A ,et al. Prosody-based automatic detection of annoyance and frustration in human-computer dialog[C]// Proceedings of 7th International Conference on Spoken Language Processing (ICSLP 2002).[S.l.:s.n.], 2002: 2037-2040.
[9]	EKMAN P . An argument for basic emotions[J]. Cognition and Emotion, 1992,6(3/4): 169-200.
[10]	PLUTCHIK R . The nature of emotions:human emotions have deep evolutionary roots,a fact that may explain their complexity and provide tools for clinical practice[J]. American Scientist, 2001,89(4): 344-350.
[11]	RUSSELL J A . A circumplex model of affect[J]. Journal of Personality and Social Psychology, 1980,39(6): 1161-1178.
[12]	MEHRABIAN A . Comparison of the PAD and PANAS as models for describing emotions and for differentiating anxiety from depression[J]. Journal of Psychopathology and Behavioral Assessment, 1997,19(4): 331-357.
[13]	SHU L , XIE J Y , YANG M Y ,et al. A review of emotion recognition using physiological signals[J]. Sensors (Basel,Switzerland), 2018,18(7): 2074.
[14]	DAI Y X , WANG X , ZHANG P B ,et al. Sparsity constrained differential evolution enabled feature-channel-sample hybrid selection for daily-life EEG emotion recognition[J]. Multimedia Tools and Applications, 2018,77(17): 21967-21994.
[15]	YU X Y , CHUM P , SIM K B . Analysis the effect of PCA for feature reduction in non-stationary EEG based motor imagery of BCI system[J]. Optik, 2014,125(3): 1498-1502.
[16]	FAYEK H M , LECH M , CAVEDON L . Modeling subjectiveness in emotion recognition with deep neural networks:ensembles vs soft labels[C]// Proceedings of 2016 International Joint Conference on Neural Networks (IJCNN). Piscataway:IEEE Press, 2016: 566-570.
[17]	RINGEVAL F , EYBEN F , KROUPI E ,et al. Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data[J]. Pattern Recognition Letters, 2015(66): 22-30.
[18]	ZHONG P X , WANG D , MIAO C Y . EEG-based emotion recognition using regularized graph neural networks[J]. IEEE Transactions on Affective Computing, 2022,13(3): 1290-1301.
[19]	PORBADNIGK A K , G?RNITZ N , SANNELLI C ,et al. When brain and behavior disagree:tackling systematic label noise in EEG data with machine learning[C]// Proceedings of 2014 International Winter Workshop on Brain-Computer Interface (BCI). Piscataway:IEEE Press, 2014: 1-4.
[20]	SZEGEDY C , VANHOUCKE V , IOFFE S ,et al. Rethinking the inception architecture for computer vision[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016: 2818-2826.
[21]	KHALILI Z , MORADI M H . Emotion recognition system using brain and peripheral signals:using correlation dimension to improve the results of EEG[C]// Proceedings of 2009 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2009: 1571-1575.
[22]	LIN Y P , WANG C H , JUNG T P ,et al. EEG-based emotion recognition in music listening[J]. IEEE Transactions on Biomedical Engineering, 2010,57(7): 1798-1806.
[23]	MURUGAPPAN M , RIZON M , NAGARAJAN R ,et al. Time-frequency analysis of EEG signals for human emotion detection[C]// Proceedings of 4th Kuala Lumpur International Conference on Biomedical Engineering 2008. Heidelberg:Springer, 2008: 262-265.
[24]	KHOSROWABADI R , QUEK H C , WAHAB A ,et al. EEG-based emotion recognition using self-organizing map for boundary detection[C]// Proceedings of 2010 20th International Conference on Pattern Recognition. Piscataway:IEEE Press, 2010: 4242-4245.
[25]	LIU W , ZHENG W L , LU B L . Emotion recognition using multimodal deep learning[C]// Proceedings of the 23rd International Conference on Neural Information Processing Springer International Publishing. Heidelberg:Springer, 2016: 521-529.
[26]	HASSAN M M , ALAM M G R , UDDIN M Z ,et al. Human emotion recognition using deep belief network architecture[J]. Information Fusion, 2019(51): 10-18.
[27]	VIJAYAKUMAR S , FLYNN R , MURRAY N . A comparative study of machine learning techniques for emotion recognition from peripheral physiological signals[C]// Proceedings of 2020 31st Irish Signals and Systems Conference (ISSC). Piscataway:IEEE Press, 2020: 1-6.
[28]	MERT A , AKAN A . Emotion recognition from EEG signals by using multivariate empirical mode decomposition[J]. Pattern Analysis and Applications, 2018,21(1): 81-89.
[29]	WU S Y , XU X M , SHU L ,et al. Estimation of valence of emotion using two frontal EEG channels[C]// Proceedings of 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Piscataway:IEEE Press, 2017: 1127-1130.
[30]	CHENG C L , WEI X W , JIAN Z . Emotion recognition algorithm based on convolution neural network[C]// Proceedings of 2017 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE). Piscataway:IEEE Press, 2018: 1-5.
[31]	KWON Y H , SHIN S B , KIM S D . Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system[J]. Sensors, 2018,18(5): 1383.
[32]	CUI H , LIU A P , ZHANG X ,et al. EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network[J]. Knowledge-Based Systems, 2020(205): 106243.
[33]	LI D D , CHAI B , WANG Z ,et al. EEG emotion recognition based on 3-D feature representation and dilated fully convolutional networks[J]. IEEE Transactions on Cognitive and Developmental Systems, 2021,13(4): 885-897.
[34]	LIU W , QIU J L , ZHENG W L ,et al. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition[J]. IEEE Transactions on Cognitive and Developmental Systems, 2022,14(2): 715-729.
[35]	LI C , LIN X J , LIU Y ,et al. EEG-based emotion recognition via efficient convolutional neural network and contrastive learning[J]. IEEE Sensors Journal, 2022,22(20): 19608-19619.
[36]	VASWANI A , SHAZEER N , PARMAR N ,et al. Attention is all you need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. New York:ACM Press, 2017: 6000-6010.
[37]	HOWARD A G , ZHU M L , CHEN B ,et al. MobileNets:efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint, 2017,arXiv:1704.04861.
[38]	KOELSTRA S , MUHL C , SOLEYMANI M ,et al. DEAP:a database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2012,3(1): 18-31.
[39]	KATSIGIANNIS S , RAMZAN N . DREAMER:a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices[J]. IEEE Journal of Biomedical and Health Informatics, 2018,22(1): 98-107.
[40]	SONG T F , ZHENG W M , SONG P ,et al. EEG emotion recognition using dynamical graph convolutional neural networks[J]. IEEE Transactions on Affective Computing, 2020,11(3): 532-541.
[41]	CHENG J , CHEN X . Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism[J]. Computers in Biology and Medicine, 2022(143): 105303.
[42]	CHENG J , CHEN M Y , LI C ,et al. Emotion recognition from multi-channel EEG via deep forest[J]. IEEE Journal of Biomedical and Health Informatics, 2021,25(2): 453-464.

数据集	数组名称	数组形状	数组内容
DEAP	数据	40×40×8 064	实验次数×通道×数据
	标签	40×4	实验次数×标签
Dreamer	数据	18×14×25 472 (M)	实验次数×通道×数据
	标签	18×3	实验次数×标签

名称	DEAP 输出维度（卷积核）	Dreamer 输出维度（卷积核）
输入	128×32	128×14
空间注意力模块	128×32	128×14
卷积	64×128×32 (3)	64×128×14 (3)
并行层1	128×64×16 (3,5)	128×64×7 (3,5)
并行层2	256×32×8 (3,5)	128×32×3 (3,5)
并行层3	512×16×4 (3,5)	512×16×1 (3,5)
全连接网络	32 768	8 192

层数	唤醒	效价
0	98.72%±1.52%	98.96%±1.05%
1	99.14%±0.83%	98.86%±1.32%
2	99.01%±1.31%	98.98%±1.13%
3	99.23%±0.91%	99.13%±0.76%

μ	唤醒	效价
0	98.93%±1.14%	98.62%±1.24%
0.1	99.06%±1.04%	99.00%±0.87%
0.2	99.23%±0.91%	99.13%±0.76%
0.3	98.97%±1.14%	98.94%±0.88%

对比项	唤醒	效价
使用SAM	99.23%±0.91%	99.13%±0.76%
不使用SAM	98.75%±1.36%	98.88%±1.01%