基于脑机协同智能的情绪识别

doi:10.11959/j.issn.2096-6652.202107

Abstract

Abstract:

Emotion recognition is a direct and effective mode of emotion recognition.Machine learning relies on the formal representation of image expressions, lacks the cognitive representation ability of the brain, and has poor recognition performance on small sample data sets or complex expression (camouflage) data sets.To this end, the formal representation of machine artificial intelligence was combined with the emotional cognitive ability of human brain general intelligence, and a brain-machine collaborative intelligence emotion recognition method was proposed.Firstly, electroencephalogram (EEG) emotional features were extracted from EEG to obtain the brain’s cognitive representation of emotions.Secondly, the visual features of the image were extracted from the emotional image to obtain the machine’s formal representation of the emotion.In order to enhance the generalization ability of the machine model, the transfer adaptation between samples was introduced in the feature learning.After obtaining image visual features and EEG emotional features, the random forest regression model was trained to obtain the brain-machine mapping relationship between image visual features and EEG emotional features.The visual features of the test image were generated through the brain-machine mapping relationship to generate virtual EEG emotional features, and then the virtual EEG emotional features and image visual features were fused for emotion recognition.This method has been verified on the Chinese facial affective picture system (CFAPS) and found that the average recognition accuracy of the seven emotions is 88.51%, which is 3%～5% higher than the image-based method.

Key words: emotion recognition, EEG signal, brain-machine collaborative intelligence, deep learning

CLC Number:

TP18，TN911.7，R318

Dongjun LIU, Yuhan WANG, Wenfen LING, et al. Emotion recognition based on brain and machine collaborative intelligence[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 65-75.

Figures/Tables 17

回归对象	随机森林	极端随机树	岭核回归
平均脑电情感特征	0.227 0	-0.297 3	0.215 1
最佳脑电情感特征	$0 . 4514$	0.101 6	0.447 5

样本迁移	平均脑电情感特征	最佳脑电情感特征
×	83.91%	86.21%
√	87.36%	$87 . 36 %$

特征融合	平均脑电情感特征	最佳脑电情感特征
×	87.36%	87.36%
√	88.51%	$88 . 51 %$

样本划分比例（训练样本:测试样本）	图像视觉特征	虚拟脑电情感特征
5:5	72.90%	74.71%
7:3	75.86%	79.31%
9:1	$83 . 91 %$	$87 . 36 %$

References 24

[1]	聂聃, 王晓韡, 段若男 ,等. 基于脑电的情绪识别研究综述[J]. 中国生物医学工程学报, 2012,31(4): 595-606.
	NIE D , WANG X W , DUAN R N ,et al. Research review of emotion recognition based on EEG[J]. Chinese Journal of Biomedical Engineering, 2012,31(4): 595-606.
[2]	DORNAIKA F , RADUCANU B . Efficient facial expression recognition for human robot interaction[C]// Proceedings of the 9th International Work-Conference on Artificial Neural Networks. Berlin:Springer, 2007: 700-708.
[3]	HARPER R G , WIENS A N , MATARAZZO J D . Nonverbal communication:the state of the art[M].[S.l.]: John Wiley ＆ Sons, 1978.
[4]	尹旺, 李惠媛 . 深度学习在脑电情感识别方面的应用研究进展[J]. 计算机时代, 2020(8): 14-17.
	YIN W , LI H Y . The application research progress of deep learning in EEG emotion recognition[J]. Computer Era, 2020(8): 14-17.
[5]	陈景霞, 郑茹, 张鹏伟 ,等. 基于数据空间自适应与共空间模式的脑电情感分类[J]. 计算机工程, 2019,45(4): 296-301.
	CHEN J X , ZHENG R , ZHANG P W ,et al. EEG sentiment classification based on data space adaptation and co-space mode[J]. Computer Engineering, 2019,45(4): 296-301.
[6]	NIE D , WANG X W , SHI L C ,et al. EEG-based emotion recognition during watching movies[C]// Proceedings of the 2011 5th International IEEE/EMBS Conference on Neural Engineering. Piscataway:IEEE Press, 2011: 667-670.
[7]	YANG Y , WU Q , QIU M ,et al. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network[C]// Proceedings of the 2018 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2018: 1-7.
[8]	LIPTON Z C , BERKOWITZ J , ELKAN C . A critical review of recurrent neural networks for sequence learning[J]. arXiv preprint, 2015,arXiv:1506.00019.
[9]	WALECKI R , PAVLOVIC V , SCHULLER B ,et al. Deep structured learning for facial action unit intensity estimation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017: 3405-3414.
[10]	LI S , DENG W . Deep facial expression recognition:a survey[J]. IEEE Transactions on Affective Computing, 2018.
[11]	BREUER R , KIMMEL R . A deep learning perspective on the origin of facial expressions[J]. arXiv preprint, 2017,arXiv:1705.01842.
[12]	MOLLAHOSSEINI A , CHAN D , MAHOOR M H . Going deeper in facial expression recognition using deep neural networks[C]// Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision. Piscataway:IEEE Press, 2016: 1-10.
[13]	王云 . 基于 EEG 与计算机视觉的人脸检索闭环系统[D]. 杭州:浙江大学, 2016.
	WANG Y . Face retrieval closed-loop system based on EEG and computer vision[D]. Hangzhou:Zhejiang University, 2016.
[14]	LUO Y J , HUANG Y X , LI X Y ,et al. Effects of emotion on cognitive processing:series of event-related potentials study[J]. Advances in Psychological Science, 2006,14(4): 505.
[15]	KHALILI Z , MORADI M H . Emotion recognition system using brain and peripheral signals:using correlation dimension to improve the results of EEG[C]// Proceedings of the 2009 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2009: 1571-1575.
[16]	ZOURIDAKIS G , PATIDAR U , PADHYE N S ,et al. Spectral power of brain activity associated with emotion—a pilot MEG study[C]// Proceedings of the 17th International Conference on Biomagnetism Advances in Biomagnetism-Biomag 2010. Berlin:Springer, 2010: 354-357.
[17]	LIN Y P , WANG C H , JUNG T P ,et al. EEG-based emotion recognition in music listening[J]. IEEE Transactions on Biomedical Engineering, 2010,57(7): 1798-1806.
[18]	LI X , SONG D W , ZHANG P ,et al. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network[C]// 2016 IEEE International Conference on Bioinformatics and Biomedicine. Piscataway:IEEE Press, 2016: 352-359.
[19]	SCHMIDHUBER J , HOCHREITER S . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[20]	CHO K , VAN MERRI?NBOER B , BAHDANAU D ,et al. On the properties of neural machine translation:encoder-decoder approaches[J]. arXiv preprint, 2014arXiv:1409.1259.
[21]	LONG M , CAO Y , WANG J ,et al. Learning transferable features with deep adaptation networks[C]// Proceedings of the 32nd International Conference on Machine Learning.[S.l.:s.n.], 2015: 97-105.
[22]	GRETTON A , BORGWARDT K M , RASCH M J ,et al. A kernel two-sample test[J]. The Journal of Machine Learning Research, 2012,13(1): 723-773.
[23]	HE K , ZHANG X , REN S ,et al. Deep residual learning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2016: 770-778.
[24]	LING W F , KONG W Z , LONG Y F ,et al. Facial emotion recognition based on brain and machine collaborative intelligence[C]// Proceedings of the 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications. Piscataway:IEEE Press, 2019: 1-5.

Metrics

Recommended 0

No Suggested Reading articles found!

方法	图像视觉特征	虚拟脑电情感特征
方法	图像视觉特征	平均脑电情感特征	最佳脑电情感特征
LeNet	72.09%	75.86%	70.57%
AlexNet	82.76%	85.06%	85.06%
ResNet	83.91%	87.36%	87.36%

Emotion recognition based on brain and machine collaborative intelligence

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 24

Related Articles 15

Metrics

Recommended 0

指标	平均脑电情感特征	最佳脑电情感特征
accuracy	87.36%	87.36%
precision	88.48%	88.01%
recall	87.35%	87.36%

[1]	Zhe HUANG, Yongcai WANG, Deying LI. A survey of 3D object detection algorithms [J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(1): 7-31.
[2]	Jingwei LU, Xiang CHENG, Fei-Yue WANG. Artificial intelligence and deep learning methods for solving differential equations: the state of the art and prospects [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 461-476.
[3]	Jun ZHANG, Peidong XU, Siyuan CHEN, Tianlu GAO, Yuxin DAI, Ke ZHANG, Hang ZHAO, Jiemai GAO, Yuyang BAI, Jinxing LI, Haoran ZHANG, Xiang LI, Jiuxiang CHEN. A hybrid physics-data-knowledge driven approach for human-machine hybrid-augmented intelligence-based system management and control [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 571-583.
[4]	Yan CHEN, Xueqin LUO, Wei LIANG, Yongfang XIE. Depression recognition based on emotional information fused with attentional mechanism [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 600-609.
[5]	Renwu LI, Lingxiao ZHANG, Lin GAO, Chunpeng LI, Hao JIANG. Category-level object pose estimation from depth point cloud [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 246-254.
[6]	Linhui LI, Bin ZHOU, Weiwei REN, Jing LIAN. Review of pedestrian trajectory prediction methods [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(4): 399-411.
[7]	Ying LI, Long CHEN, Zhaohong HUANG, Yang SUN, Guorong CAI. Plant leaf detection technology based on multi-scale CNN feature fusion [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(3): 304-311.
[8]	Qing TIAN, Rong HU, Zuoyong LI, Yuanzheng CAI, Zhaochai YU. Insulator detection based on SE-YOLOv5s [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(3): 312-321.
[9]	Wen LIU, Kunlin HU, Yan LI, Zhao LIU. A review of prediction methods for moving target trajectories [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(2): 149-160.
[10]	Yang ZHANG, Yue HU, Dongrong XIN. A deep learning short-term traffic flow prediction method considering spatial-temporal association [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(2): 172-178.
[11]	Bao-Liang LU, Yaqian ZHANG, Wei-Long ZHENG. A survey of affective brain-computer interface [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 36-48.
[12]	Hanbing GAO, Chi ZHANG, Mingyan JIN, Yang XIAO, Fengyu CONG. Cross-subject emotional EEG recognition based on multi-source domain adaptation [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 59-64.
[13]	Wenfen LING, Sihan CHEN, Yong PENG, Wanzeng KONG. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 76-84.
[14]	Liang ZHAO, Zhifeng XIE, Kunpeng ZHANG, Yuqing ZHENG, Yuankun FU. Modeling signal propagation in wireless network:an interval type-2 fuzzy ensemble deep learning approach [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 401-411.
[15]	Sijia TIAN,Qiang GU,Rong HU,Ruige LI,Dingxin HE. A robot sorting method based on deep learning [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(3): 268-274.