基于脑机协同智能的情绪识别

doi:10.11959/j.issn.2096-6652.202107

智能科学与技术学报 ›› 2021, Vol. 3 ›› Issue (1): 65-75.doi: 10.11959/j.issn.2096-6652.202107

基于脑机协同智能的情绪识别

刘栋军¹^,², 王宇涵¹^,², 凌文芬¹^,², 彭勇¹^,², 孔万增¹^,²

¹ 杭州电子科技大学计算机学院，浙江杭州 310018
² 浙江省脑机协同智能重点实验室，浙江杭州 310018

修回日期:2021-02-08 出版日期:2021-03-15 发布日期:2021-03-01
作者简介:刘栋军（1997- ），男，杭州电子科技大学计算机学院硕士生，主要研究方向为脑机协同智能、情绪识别。
王宇涵（1997- ），女，杭州电子科技大学计算机学院硕士生，主要研究方向为脑机协同智能、情绪识别。
凌文芬（1995- ），女，杭州电子科技大学计算机学院硕士生，主要研究方向为情绪生理信号、特征提取、情绪识别。
彭勇（1985- ），男，博士，杭州电子科技大学认知与智能计算研究所副教授，主要研究方向为机器学习、模式识别与脑机交互算法及应用。
孔万增（1980- ），男，博士，杭州电子科技大学计算机学院认知与智能计算研究所教授、博士生导师，杭州电子科技大学研究生院副院长、浙江省脑机协同智能重点实验室主任，主要研究方向为人工智能与模式识别、嵌入式可穿戴计算、脑机交互与认知计算等。
基金资助:
国家重点研发计划基金资助项目(2017YFE0116800);国家自然科学基金资助项目(U20B2074);浙江省科技计划项目(2018C04012);浙江省“脑机协同智能”重点实验室开放基金项目(20200E10010)

Emotion recognition based on brain and machine collaborative intelligence

Dongjun LIU¹^,², Yuhan WANG¹^,², Wenfen LING¹^,², Yong PENG¹^,², Wanzeng KONG¹^,²

¹ College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
² Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou 310018, China

Revised:2021-02-08 Online:2021-03-15 Published:2021-03-01
Supported by:
The National Key Research and Development Program of China(2017YFE0116800);The National Natural Science Foundation of China(U20B2074);Science and Technology Program of Zhejiang Province(2018C04012);Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province(20200E10010)

摘要/Abstract

摘要：

面部表情识别是一种直接、有效的情绪识别模式。机器学习可以对图像表情进行形式化表征，但由于缺乏大脑的认知表征能力，在小样本数据集或复杂表情（伪装）数据集上的识别性能并不理想。针对此问题，将机器人工智能的形式化表征与人脑通用智能的情感认知能力结合，提出一种基于脑机协同智能的情绪识别方法。首先，从脑电图信号中提取脑电情感特征，以获取大脑对情绪的认知表征。其次，从情感图像中提取图像的视觉特征，以获取机器对情绪的形式化表征。为了增强机器模型的泛化能力，在特征学习中引入样本间的迁移适配。在得到图像视觉特征和脑电情感特征后，采用随机森林回归模型训练得到图像视觉特征与脑电情感特征之间的脑机映射关系。测试图像的图像视觉特征经过脑机映射关系产生虚拟脑电情感特征，然后将虚拟脑电情感特征与图像视觉特征进行融合，用于情绪识别。该方法已经在中国面孔表情图片系统上进行了验证，发现对7种情绪的平均识别准确率为88.51%，相比单纯基于图像的方法，提升了3%～5%。

关键词: 情绪识别, 脑电图信号, 脑机协同智能, 深度学习

Abstract:

Emotion recognition is a direct and effective mode of emotion recognition.Machine learning relies on the formal representation of image expressions, lacks the cognitive representation ability of the brain, and has poor recognition performance on small sample data sets or complex expression (camouflage) data sets.To this end, the formal representation of machine artificial intelligence was combined with the emotional cognitive ability of human brain general intelligence, and a brain-machine collaborative intelligence emotion recognition method was proposed.Firstly, electroencephalogram (EEG) emotional features were extracted from EEG to obtain the brain’s cognitive representation of emotions.Secondly, the visual features of the image were extracted from the emotional image to obtain the machine’s formal representation of the emotion.In order to enhance the generalization ability of the machine model, the transfer adaptation between samples was introduced in the feature learning.After obtaining image visual features and EEG emotional features, the random forest regression model was trained to obtain the brain-machine mapping relationship between image visual features and EEG emotional features.The visual features of the test image were generated through the brain-machine mapping relationship to generate virtual EEG emotional features, and then the virtual EEG emotional features and image visual features were fused for emotion recognition.This method has been verified on the Chinese facial affective picture system (CFAPS) and found that the average recognition accuracy of the seven emotions is 88.51%, which is 3%～5% higher than the image-based method.

Key words: emotion recognition, EEG signal, brain-machine collaborative intelligence, deep learning

中图分类号:

TP18，TN911.7，R318

刘栋军, 王宇涵, 凌文芬, 等. 基于脑机协同智能的情绪识别[J]. 智能科学与技术学报, 2021, 3(1): 65-75.

Dongjun LIU, Yuhan WANG, Wenfen LING, et al. Emotion recognition based on brain and machine collaborative intelligence[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 65-75.

图/表 17

图1

图2

图3

图4

图5

图6

表1

表2

3种回归模型的R2系数"

回归对象	随机森林	极端随机树	岭核回归
平均脑电情感特征	0.227 0	-0.297 3	0.215 1
最佳脑电情感特征	$0 . 4514$	0.101 6	0.447 5

表2

表3

加入迁移学习前后虚拟脑电情感特征识别准确率"

样本迁移	平均脑电情感特征	最佳脑电情感特征
×	83.91%	86.21%
√	87.36%	$87 . 36 %$

表3

表4

表5

虚拟脑电情感特征与图像视觉特征融合和不融合的识别准确率"

特征融合	平均脑电情感特征	最佳脑电情感特征
×	87.36%	87.36%
√	88.51%	$88 . 51 %$

表5

表6

不同样本划分比例对应的识别准确率"

样本划分比例（训练样本:测试样本）	图像视觉特征	虚拟脑电情感特征
5:5	72.90%	74.71%
7:3	75.86%	79.31%
9:1	$83 . 91 %$	$87 . 36 %$

表6

图7

图8

图9

图10

图11

参考文献 24

[1]	聂聃, 王晓韡, 段若男 ,等. 基于脑电的情绪识别研究综述[J]. 中国生物医学工程学报, 2012,31(4): 595-606.
	NIE D , WANG X W , DUAN R N ,et al. Research review of emotion recognition based on EEG[J]. Chinese Journal of Biomedical Engineering, 2012,31(4): 595-606.
[2]	DORNAIKA F , RADUCANU B . Efficient facial expression recognition for human robot interaction[C]// Proceedings of the 9th International Work-Conference on Artificial Neural Networks. Berlin:Springer, 2007: 700-708.
[3]	HARPER R G , WIENS A N , MATARAZZO J D . Nonverbal communication:the state of the art[M].[S.l.]: John Wiley ＆ Sons, 1978.
[4]	尹旺, 李惠媛 . 深度学习在脑电情感识别方面的应用研究进展[J]. 计算机时代, 2020(8): 14-17.
	YIN W , LI H Y . The application research progress of deep learning in EEG emotion recognition[J]. Computer Era, 2020(8): 14-17.
[5]	陈景霞, 郑茹, 张鹏伟 ,等. 基于数据空间自适应与共空间模式的脑电情感分类[J]. 计算机工程, 2019,45(4): 296-301.
	CHEN J X , ZHENG R , ZHANG P W ,et al. EEG sentiment classification based on data space adaptation and co-space mode[J]. Computer Engineering, 2019,45(4): 296-301.
[6]	NIE D , WANG X W , SHI L C ,et al. EEG-based emotion recognition during watching movies[C]// Proceedings of the 2011 5th International IEEE/EMBS Conference on Neural Engineering. Piscataway:IEEE Press, 2011: 667-670.
[7]	YANG Y , WU Q , QIU M ,et al. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network[C]// Proceedings of the 2018 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2018: 1-7.
[8]	LIPTON Z C , BERKOWITZ J , ELKAN C . A critical review of recurrent neural networks for sequence learning[J]. arXiv preprint, 2015,arXiv:1506.00019.
[9]	WALECKI R , PAVLOVIC V , SCHULLER B ,et al. Deep structured learning for facial action unit intensity estimation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017: 3405-3414.
[10]	LI S , DENG W . Deep facial expression recognition:a survey[J]. IEEE Transactions on Affective Computing, 2018.
[11]	BREUER R , KIMMEL R . A deep learning perspective on the origin of facial expressions[J]. arXiv preprint, 2017,arXiv:1705.01842.
[12]	MOLLAHOSSEINI A , CHAN D , MAHOOR M H . Going deeper in facial expression recognition using deep neural networks[C]// Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision. Piscataway:IEEE Press, 2016: 1-10.
[13]	王云 . 基于 EEG 与计算机视觉的人脸检索闭环系统[D]. 杭州:浙江大学, 2016.
	WANG Y . Face retrieval closed-loop system based on EEG and computer vision[D]. Hangzhou:Zhejiang University, 2016.
[14]	LUO Y J , HUANG Y X , LI X Y ,et al. Effects of emotion on cognitive processing:series of event-related potentials study[J]. Advances in Psychological Science, 2006,14(4): 505.
[15]	KHALILI Z , MORADI M H . Emotion recognition system using brain and peripheral signals:using correlation dimension to improve the results of EEG[C]// Proceedings of the 2009 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2009: 1571-1575.
[16]	ZOURIDAKIS G , PATIDAR U , PADHYE N S ,et al. Spectral power of brain activity associated with emotion—a pilot MEG study[C]// Proceedings of the 17th International Conference on Biomagnetism Advances in Biomagnetism-Biomag 2010. Berlin:Springer, 2010: 354-357.
[17]	LIN Y P , WANG C H , JUNG T P ,et al. EEG-based emotion recognition in music listening[J]. IEEE Transactions on Biomedical Engineering, 2010,57(7): 1798-1806.
[18]	LI X , SONG D W , ZHANG P ,et al. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network[C]// 2016 IEEE International Conference on Bioinformatics and Biomedicine. Piscataway:IEEE Press, 2016: 352-359.
[19]	SCHMIDHUBER J , HOCHREITER S . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[20]	CHO K , VAN MERRI?NBOER B , BAHDANAU D ,et al. On the properties of neural machine translation:encoder-decoder approaches[J]. arXiv preprint, 2014arXiv:1409.1259.
[21]	LONG M , CAO Y , WANG J ,et al. Learning transferable features with deep adaptation networks[C]// Proceedings of the 32nd International Conference on Machine Learning.[S.l.:s.n.], 2015: 97-105.
[22]	GRETTON A , BORGWARDT K M , RASCH M J ,et al. A kernel two-sample test[J]. The Journal of Machine Learning Research, 2012,13(1): 723-773.
[23]	HE K , ZHANG X , REN S ,et al. Deep residual learning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2016: 770-778.
[24]	LING W F , KONG W Z , LONG Y F ,et al. Facial emotion recognition based on brain and machine collaborative intelligence[C]// Proceedings of the 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications. Piscataway:IEEE Press, 2019: 1-5.

基于脑机协同智能的情绪识别

Emotion recognition based on brain and machine collaborative intelligence

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 24

相关文章 15

Metrics

推荐阅读 0

指标	平均脑电情感特征	最佳脑电情感特征
accuracy	87.36%	87.36%
precision	88.48%	88.01%
recall	87.35%	87.36%

方法	图像视觉特征	虚拟脑电情感特征
方法	图像视觉特征	平均脑电情感特征	最佳脑电情感特征
LeNet	72.09%	75.86%	70.57%
AlexNet	82.76%	85.06%	85.06%
ResNet	83.91%	87.36%	87.36%

[1]	黄哲, 王永才, 李德英. 3D目标检测方法研究综述[J]. 智能科学与技术学报, 2023, 5(1): 7-31.
[2]	卢经纬, 程相, 王飞跃. 求解微分方程的人工智能与深度学习方法：现状及展望[J]. 智能科学与技术学报, 2022, 4(4): 461-476.
[3]	张俊, 许沛东, 陈思远, 高天露, 戴宇欣, 张科, 赵杭, 高杰迈, 白昱阳, 李金星, 张浩然, 李湘, 陈玖香. 物理-数据-知识混合驱动的人机混合增强智能系统管控方法[J]. 智能科学与技术学报, 2022, 4(4): 571-583.
[4]	陈妍, 罗雪琴, 梁伟, 谢永芳. 基于情感信息融合注意力机制的抑郁症识别[J]. 智能科学与技术学报, 2022, 4(4): 600-609.
[5]	栗仁武, 张凌霄, 高林, 李淳芃, 蒋浩. 基于点云的类级别物体姿态估计[J]. 智能科学与技术学报, 2022, 4(2): 246-254.
[6]	李琳辉, 周彬, 任威威, 连静. 行人轨迹预测方法综述[J]. 智能科学与技术学报, 2021, 3(4): 399-411.
[7]	李颖, 陈龙, 黄钊宏, 孙杨, 蔡国榕. 基于多尺度卷积神经网络特征融合的植株叶片检测技术[J]. 智能科学与技术学报, 2021, 3(3): 304-311.
[8]	田庆, 胡蓉, 李佐勇, 蔡远征, 余兆钗. 基于SE-YOLOv5s的绝缘子检测[J]. 智能科学与技术学报, 2021, 3(3): 312-321.
[9]	刘文, 胡琨林, 李岩, 刘钊. 移动目标轨迹预测方法研究综述[J]. 智能科学与技术学报, 2021, 3(2): 149-160.
[10]	张阳, 胡月, 辛东嵘. 一种考虑时空关联的深度学习短时交通流预测方法[J]. 智能科学与技术学报, 2021, 3(2): 172-178.
[11]	吕宝粮, 张亚倩, 郑伟龙. 情感脑机接口研究综述[J]. 智能科学与技术学报, 2021, 3(1): 36-48.
[12]	凌文芬, 陈思含, 彭勇, 孔万增. 基于3D分层卷积融合的多模态生理信号情绪识别[J]. 智能科学与技术学报, 2021, 3(1): 76-84.
[13]	赵亮, 谢志峰, 张坤鹏, 郑玉卿, 付园坤. 无线网络信号传输建模：一种区间二型模糊集成深度学习方法[J]. 智能科学与技术学报, 2020, 2(4): 401-411.
[14]	田思佳,顾强,胡蓉,李锐戈,何顶新. 一种基于深度学习的机械臂分拣方法[J]. 智能科学与技术学报, 2020, 2(3): 268-274.
[15]	王飞跃,曹东璞,魏庆来. 强化学习：迈向知行合一的智能机制与算法[J]. 智能科学与技术学报, 2020, 2(2): 101-106.