情感脑机接口研究综述

doi:10.11959/j.issn.2096-6652.202104

Abstract

Abstract:

An important research goal in emotion artificial intelligence is to make machines understand and recognize human emotions in real-time and facilitate human-computer interaction in a more natural and friendly way.Affective brain-computer interface (aBCI) is a type of BCI that can recognize and/or modulate human emotion.Thus, aBCI plays a critical role in promoting emotion artificial intelligence.The basic concepts and recent research development of aBCI were summarized, and the applications of aBCI in a wide range of domains were outlined.The roles that the aBCI can play in the development of artificial general intelligence and the challenges faced by the aBCI research community were discussed.

Key words: affective computing, affective brain-computer interface, emotion recognition, emotion regulation, multimodal affective brain-computer interface, artificial general intelligence

CLC Number:

TP18

Bao-Liang LU,Yaqian ZHANG,Wei-Long ZHENG. A survey of affective brain-computer interface[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 36-48.

Figures/Tables 2

References 98

[1]	WOLPAW J R , WOLPAW E W . 脑-机接口:原理与实践[M]. 伏云发,杨秋红,徐保磊,等,译. 北京: 国防工业出版社, 2017.
	WOLPAW J R , WOLPAW E W . Brain-computer interfaces:principles and practice[M]. Translated by FU Y F,YANG Q H,XU B L,et al. Beijing: National Defense Industry Press, 2017.
[2]	GRAIMANN B , ALLISON B Z , PFURTSCHELLER G . Brain- computer interfaces:revolutionizing human-computer interaction[M].[S.l.]: Springer Science ＆ Business Media, 2010.
[3]	SHANECHI M M . Brain-machine interfaces from motor to mood[J]. Nature Neuroscience, 2019,22: 1554-1564.
[4]	HU X , CHEN J J , WANG F ,et al. Ten challenges for EEG-based affective computing[J]. Brain Science Advances, 2019,5(1): 1-20.
[5]	吕宝粮, 郑伟龙 . 情感脑-机接口[M]// 脑-计算机交互研究前沿. 上海: 上海交通大学出版社, 2021.
	LYU B L , ZHENG W L . Affective brain-computer interface[M]// Frontiers of brain-computer interaction research. Shanghai: Shanghai Jiao Tong University Press, 2021.
[6]	EKMAN P . An argument for basic emotions[J]. Cognition and Emotion, 1992,6(3-4): 169-200.
[7]	RUSSELL J A . A circumplex model of affect[J]. Journal of Personality and Social Psychology, 1980,39(6): 1161-1178.
[8]	LARSEN J T , MCGRAW A P , CACIOPPO J T . Can people feel happy and sad at the same time?[J]. Journal of Personality and Social Psychology, 2001,81(4): 684-696.
[9]	FREDRICKSON B L . Positive emotions broaden and build[J]. Advances in Experimental Social Psycholog, 2013,47: 1-53.
[10]	BRANNAN S , LIOTTI M , EGAN G ,et al. Neuroimaging of cerebral activations and deactivations associated with hypercapnia and hunger for air[J]. Proceedings of the National Academy of Sciences, 2001,98(4): 2029-2034.
[11]	BüCHEL C , DOLAN R J . Classical fear conditioning in functional neuroimaging[J]. Current Opinion in Neurobiology, 2000,10(2): 219-223.
[12]	PHAN K L , WAGER T , TAYLOR S F ,et al. Functional neuroanatomy of emotion:a meta-analysis of emotion activation studies in PET and fMRI[J]. Neuroimage, 2002,16(2): 331-348.
[13]	LIOTTI M , PANKSEPP J . Imaging human emotions and affective feelings:implications for biological psychiatry[M]. Canada: Wiley-Liss, 2004.
[14]	DOLENSEK N , GEHRLACH D A , KLEIN A S ,et al. Facial expressions of emotion states and their neuronal correlates in mice[J]. Science, 2020,368(6486): 89-94.
[15]	宿云, 胡斌, 徐立新 ,等. 面向脑电数据的知识建模和情感识别[J]. 科学通报, 2015,60(11): 1002-1009.
	SU Y , HU B , XU L X ,et al. EEG-data-oriented knowledge modeling and emotion recognition[J]. Chinese Science Bulletin, 2015,60(11): 1002-1009.
[16]	权学良, 曾志刚, 蒋建华 ,等. 基于生理信号的情感计算研究综述[J]. 自动化学报, 2021,47: 1-16.
	QUAN X L , ZENG Z G , JIANG J H ,et al. Physiological signals based affective computing:a systematic review[J]. Acta Automatica Sinica, 2021,47: 1-16.
[17]	张冠华, 余旻婧, 陈果 ,等. 面向情绪识别的脑电特征研究综述[J]. 中国科学(信息科学), 2019,49(9): 1097-1118.
	ZHANG G H , YU M J , CHEN G ,et al. A review of EEG features for emotion recognition[J]. Science in China (Information Sciences), 2019,49(9): 1097-1118.
[18]	DIAZ-PIEDRA C , SEBASTIáN M V , DI STASI L L . EEG theta power activity reflects workload among army combat drivers:an experimental study[J]. Brain Sciences, 2020,10(199): 1-14.
[19]	BROUWER A-M , HOGERVORST M A , VAN ERP J B ,et al. Estimating workload using EEG spectral power and ERPs in the n-back task[J]. Journal of Neural Engineering, 2012,9(4): 045008.
[20]	PUTNAM K M , MCSWEENEY L B . Depressive symptoms and baseline prefrontal EEG alpha activity:a study utilizing ecological momentary assessment[J]. Biological Psychology, 2008,77(2): 237-240.
[21]	WHEELER R E , DAVIDSON R J , TOMARKEN A J . Frontal brain asymmetry and emotional reactivity:a biological substrate of affective style[J]. Psychophysiology, 1993,30(1): 82-89.
[22]	ZHENG W L , LYU B L . Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on Autonomous Mental Development, 2015,7(3): 162-175.
[23]	RAY W J , COLE H W . EEG alpha activity reflects attentional demands,and beta activity reflects emotional and cognitive processes[J]. Science, 1985,228(4700): 750-752.
[24]	AHERN G L , SCHWARTZ G E . Differential lateralization for positive and negative emotion in the human brain:EEG spectral analysis[J]. Neuropsychologia, 1985,23(6): 745-755.
[25]	MOHAMMADI Z , FROUNCHI J , AMIRI M . Wavelet-based emotion recognition system using EEG signal[J]. Neural Computing and Applications, 2017,28(8): 1985-1990.
[26]	JATUPAIBOON N , PAN-NGUM S , ISRASENA P . Emotion classification using minimal EEG channels and frequency bands[C]// Proceedings of The 10th International Joint Conference on Computer Science and Software Engineering.[S.l.:s.n. ], 2013: 21-24.
[27]	LI M , LYU B L . Emotion classification based on gamma-band EEG[C]// Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway:IEEE Press, 2009: 1223-1226.
[28]	YANG K , TONG L , SHU J ,et al. High gamma band EEG closely related to emotion:evidence from functional network[J]. Frontiers in Human Neuroscience, 2020,14: 89.
[29]	SHARMA K , CASTELLINI C , VAN DEN BROEK E L ,et al. A dataset of continuous affect annotations and physiological signals for emotion analysis[J]. Scientific Data, 2019,6(1): 1-13.
[30]	DING Y , HU X , XIA Z Y ,et al. Inter-brain EEG feature extraction and analysis for continuous implicit emotion tagging during video watching[J]. IEEE Transactions on Affective Computing, 2018,12(1): 92-102.
[31]	NUMMENMAA L , GLEREAN E , VIINIKAINEN M ,et al. Emotions promote social interaction by synchronizing brain activity across individuals[J]. Proceedings of the National Academy of Sciences, 2012,109(24): 9599-9604.
[32]	KOELSTRA S , MUHL C , SOLEYMANI M ,et al. DEAP:a database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2011,3(1): 18-31.
[33]	KATSIGIANNIS S , RAMZAN N . DREAMER:a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices[J]. IEEE Journal of Biomedical and Health Informatics, 2017,22(1): 98-107.
[34]	LIU Y J , YU M J , ZHAO G Z ,et al. Real-time movie-induced discrete emotion recognition from EEG signals[J]. IEEE Transactions on Affective Computing, 2017,9(4): 550-562.
[35]	SOLEYMANI M , LICHTENAUER J , PUN T ,et al. A multimodal database for affect recognition and implicit tagging[J]. IEEE Transactions on Affective Computing, 2011,3(1): 42-55.
[36]	SUBRAMANIAN R , WACHE J , ABADI M K ,et al. Emotion and personality recognition using commercial sensors[J]. IEEE Transactions on Affective Computing, 2016,9(2): 147-160.
[37]	CORREA J A M , ABADI M K , SEBE N ,et al. AMIGOS:a dataset for affect,personality and mood research on individuals and groups[J]. IEEE Transactions on Affective Computing, 2018: 1.
[38]	LIN Y P , JUNG T P . Improving EEG-based emotion classification using conditional transfer learning[J]. Frontiers in Human Neuroscience, 2017,11: 334.
[39]	LUO Y , ZHANG S Y , ZHENG W L ,et al. WGAN domain adaptation for EEG-based emotion recognition[C]// Proceedings of the International Conference on Neural Information Processing.[S.l.:s.n.], 2018: 275-286.
[40]	ZHENG W L , LYU B L . Personalizing EEG-based affective models with transfer learning[C]// Proceedings of The 25th International Joint Conference on Artificial Intelligence.[S.l.:s.n.], 2016: 2732-2738.
[41]	WAN Z , YANG R , HUANG M J ,et al. A review on transfer learning in EEG signal analysis[J]. Neurocomputing, 2020,421: 1-14.
[42]	WU D R , XU Y F , LYU B L . Transfer learning for EEG-based brain-computer interfaces:a review of progresses since 2016[J]. arXiv preprint,2020,arXiv:200406286.
[43]	LI J P , QIU S , SHEN Y Y ,et al. Multisource transfer learning for cross-subject EEG emotion recognition[J]. IEEE Transactions on Cybernetics, 2019,50(7): 3281-3293.
[44]	MA B Q , LI H , ZHENG W L ,et al. Reducing the subject variability of EEG signals with adversarial domain generalization[C]// The 26th International Conference on Neural Information Processing.[S.l.:s.n.], 2019: 30-42.
[45]	ZHAO L M , YAN X , LYU B L . Plug-and-play domain adaptation for cross-subject EEG-based emotion recognition[C]// Proceedings of the 35th AAAI Conference on Artificial Intelligence.[S.l.:s.n.], 2021.
[46]	JIANG Y Z , WU D R , DENG Z H ,et al. Seizure classification from EEG signals using transfer learning,semi-supervised learning and TSK fuzzy system[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017,25(12): 2270-2284.
[47]	郑伟龙, 石振锋, 吕宝粮 . 用异质迁移学习构建跨被试脑电情感模型[J]. 计算机学报, 2020,43(2): 177-189.
	ZHENG W L , SHI Z F , LYU B L . Building cross-subject EEG-based affective models using heterogeneous transfer learning[J]. Chinese Journal of Computers, 2020,43(2): 177-189.
[48]	ZHU Y , CHEN Y Q , LU Z Q ,et al. Heterogeneous transfer learning for image classification[C]// Proceedings of the 25th AAAI Conference on Artificial Intelligence.[S.l.:s.n.], 2011.
[49]	MARQUART G , CABRALL C , DE WINTER J . Review of eye-related measures of drivers’ mental workload[J]. Procedia Manufacturing, 2015,3: 2854-2861.
[50]	ZHU X M , ZHENG W L , LYU B L ,et al. EOG-based drowsiness detection using convolutional neural networks[C]// Proceedings of the 2014 International Joint Conference on Neural Networks.[S.l.:s.n.], 2014: 128-134.
[51]	CHANG K M , LIU S H , WU X H . A wireless sEMG recording system and its application to muscle fatigue detection[J]. Sensors, 2012,12(1): 489-499.
[52]	NOURBAKHSH N , WANG Y , CHEN F . GSR and blink features for cognitive load classification[C]// Proceedings of the IFIP Conference on Human- Computer Interaction.[S.l.:s.n.], 2013: 159-166.
[53]	ROSCOE A H . Assessing pilot workload.Why measure heart rate,HRV and respiration?[J]. Biological Psychology, 1992,34(2-3): 259-287.
[54]	SOLEYMANI M , ASGHARI-ESFEDEN S , PANTIC M ,et al. Continuous emotion detection using EEG signals and facial expressions[C]// Proceedings of the 2014 IEEE International Conference on Multimedia and Expo. Piscataway:IEEE Press, 2014: 1-6.
[55]	HUANG Y R , YANG J H , LIAO P K ,et al. Fusion of facial expressions and EEG for multimodal emotion recognition[J]. Computational Intelligence and Neuroscience, 2017: 2017451
[56]	HUANG X H , KORTELAINEN J , ZHAO G Y ,et al. Multi-modal emotion analysis from facial expressions and electroencephalogram[J]. Computer Vision and Image Understanding, 2016,147: 114-214.
[57]	LAN Y T , LIU W , LYU B L . Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism[C]// Proceedings of the 2020 International Joint Conference on Neural Networks.[S.l.:s.n.], 2020: 1-6.
[58]	ZHENG W L , LIU W , LU Y ,et al. Emotionmeter:a multimodal framework for recognizing human emotions[J]. IEEE Transactions on Cybernetics, 2018,49(3): 1110-1122.
[59]	LU Y F , ZHENG W L , LI B B ,et al. Combining eye movements and eeg to enhance emotion recognition[C]// Proceedings of the 24th International Conference on Artificial Intelligence.[S.l.:s.n.], 2015: 1170-1176.
[60]	EKMAN P . Facial expression and emotion[J]. American Psychologist, 1993,48(4): 384-392.
[61]	RUSSELL J A . Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies[J]. Psychological Bulletin, 1994,115(1): 102-114.
[62]	COWEN A S , KELTNER D , SCHROFF F ,et al. Sixteen facial expressions occur in similar contexts worldwide[J]. Nature, 2020(7841): 1-7.
[63]	SUN Y , AYAZ H , AKANSU A N . Multimodal affective state assessment using fNIRS+ EEG and spontaneous facial expression[J]. Brain Sciences, 2020,10(2): 85.
[64]	MINOTTO V P , JUNG C R , LEE B . Multimodal multi-channel on-line speaker diarization using sensor fusion through SVM[J]. IEEE Transactions on Multimedia, 2015,17(10): 1694-1705.
[65]	CAI H S , QU Z D , LI Z ,et al. Feature-level fusion approaches based on multimodal EEG data for depression recognition[J]. Information Fusion, 2020,59: 127-138.
[66]	GOSHVARPOUR A , GOSHVARPOUR A . Schizophrenia diagnosis using innovative EEG feature-level fusion schemes[J]. Physical and Engineering Sciences in Medicine, 2020,43(1): 227-238.
[67]	QIAN M , AGUILAR M , ZACHERY K N ,et al. Decision-level fusion of EEG and pupil features for single-trial visual detection analysis[J]. IEEE Transactions on Biomedical Engineering, 2009,56(7): 1929-1937.
[68]	ZHENG W L , DONG B N , LYU B L . Multimodal emotion recognition using EEG and eye tracking data[C]// Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway:IEEE Press, 2014: 5040-5043.
[69]	NGIAM J , KHOSLA A , KIM M ,et al. Multimodal deep learning[C]// Proceedings of the 26th International Conference on Machine Learning.[S.l.:s.n.], 2009.
[70]	LIU W , ZHENG W L , LYU B L . Multimodal emotion recognition using multimodal deep learning[J]. arXiv preprint, 2016,arXiv:160208225.
[71]	SRIVASTAVA N , SALAKHUTDINOV R . Multimodal learning with deep boltzmann machines[J]. The Journal of Machine Learning Research, 2014,15(1): 2949-2980.
[72]	VOS T , ABAJOBIR A A , ABATE K H ,et al. Global,regional,and national incidence,prevalence,and years lived with disability for 328 diseases and injuries for 195 countries,1990-2016:a systematic analysis for the Global Burden of Disease Study 2016[J]. The Lancet, 2017,390: 1211-1259.
[73]	HANCOCK P A , MESHKATI N . Human mental workload[M].[S.l.]: Elsevier, 1988.
[74]	NAKAMURA J , CSIKSZENTMIHALYI M . Flow theory and research[M].[S.l.]: Handbook of Positive Psychology, 2009.
[75]	LOFT S , SANDERSON P , NEAL A ,et al. Modeling and predicting mental workload in en route air traffic control:critical review and broader implications[J]. Human Factors, 2007,49(3): 376-399.
[76]	COLLET C , AVERTY P , DITTMAR A . Autonomic nervous system and subjective ratings of strain in air-traffic control[J]. Applied Ergonomics, 2009,40(1): 23-32.
[77]	BERKA C , LEVENDOWSKI D J , LUMICAO M N ,et al. EEG correlates of task engagement and mental workload in vigilance,learning,and memory tasks[J]. Aviation,Space,and Environmental Medicine, 2007,78(5): B231-44.
[78]	HOGERVORST M A , BROUWER A-M , VAN ERP J B . Combining and comparing EEG,peripheral physiology and eye-related measures for the assessment of mental workload[J]. Frontiers in Neuroscience, 2014,8: 322.
[79]	HANCOCK P A , MESHKATI N , ROBERTSON M . Physiological reflections of mental workload[J]. Aviation,Space,and Environmental Medicine, 1985,56(11): 1110-1114.
[80]	CHARLES R L , NIXON J . Measuring mental workload using physiological measures:a systematic review[J]. Applied Ergonomics, 2019,74: 221-232.
[81]	FALLAHI M , MOTAMEDZADE M , HEIDARIMOGHADAM R ,et al. Effects of mental workload on physiological and subjective responses during traffic density monitoring:a field study[J]. Applied Ergonomics, 2016,52: 95-103.
[82]	THIBODEAU R , JORGENSEN R S , KIM S . Depression,anxiety,and resting frontal EEG asymmetry:a meta-analytic review[J]. Journal of Abnormal Psychology, 2006,115(4): 715-729.
[83]	DAMBORSKá A , TOMESCU M I , HONZíRKOVá E ,et al. EEG resting-state large-scale brain network dynamics are related to depressive symptoms[J]. Frontiers in Psychiatry, 2019,10: 548.
[84]	CAI H S , ZHANG X Z , ZHANG Y H ,et al. A case-based reasoning model for depression based on three-electrode EEG data[J]. IEEE Transactions on Affective Computing, 2018,11(3): 383-392.
[85]	ZHANG Y , WU W , TOLL R T ,et al. Identification of psychiatric disorder subtypes from functional connectivity patterns in resting-state electroencephalography[J]. Nature Biomedical Engineering, 2020: 1-15.
[86]	HAMILTON M . The Hamilton rating scale for depression[M].[S.l.]: Springer, 1986.
[87]	MONTGOMERY S , ASBERG M . A new depression scale designed to be sensitive to change[J]. The British journal of psychiatry, 1979,134(4): 382-389.
[88]	KOHOUT F J , BERKMAN L F , EVANS D A ,et al. Two shorter forms of the CES-D depression symptoms index[J]. Journal of Aging and Health, 1993,5(2): 179-193.
[89]	ZUNG W W . A self-rating depression scale[J]. Archives of General Psychiatry, 1965,12(1): 63-70.
[90]	BECK A T , BECK R W . Screening depressed patients in family practice:a rapid technic[J]. Postgraduate Medicine, 1972,52(6): 81-85.
[91]	KROENKE K , SPITZER R L , WILLIAMS J B . The PHQ‐9:validity of a brief depression severity measure[J]. Journal of General Internal Medicine, 2001,16(9): 606-613.
[92]	ZIGMOND A S , SNAITH R P . The hospital anxiety and depression scale[J]. Acta Psychiatrica Scandinavica, 1983,67(6): 361-370.
[93]	CARNEY M , SHEFFIELD B . Depression and the newcastle scales their relationship to Hamilton’s scale[J]. The British Journal of Psychiatry, 1972,121(560): 35-40.
[94]	PERLMUTTER J S , MINK J W . Deep brain stimulation[J]. Annu Rev Neurosci, 2006,29: 229-257.
[95]	DENG Z D , LI D Y , ZHANG C C ,et al. Long-term follow-up of bilateral subthalamic deep brain stimulation for refractory tardive dystonia[J]. Parkinsonism and Related Disorders, 2017,41: 58-65.
[96]	YANG Y X , QIAO S Y , SANI O G ,et al. Modelling and prediction of the dynamic responses of large-scale brain networks during direct electrical stimulation[J]. Nature Biomedical Engineering, 2021: 1-22.
[97]	SCANGOS K W , MAKHOUL G S , SUGRUE L P ,et al. State- dependent responses to intracranial brain stimulation in a patient with depression[J]. Nature Medicine, 2021,27: 229-231.
[98]	RAO V R , SELLERS K K , WALLACE D L ,et al. Direct electrical stimulation of lateral orbitofrontal cortex acutely improves mood in individuals with symptoms of depression[J]. Current Biology, 2018,28(24): 3893-3902.

Metrics

Recommended 0

No Suggested Reading articles found!

A survey of affective brain-computer interface

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 2

References 98

Related Articles 3

Metrics

Recommended 0

[1]	Hanbing GAO, Chi ZHANG, Mingyan JIN, Yang XIAO, Fengyu CONG. Cross-subject emotional EEG recognition based on multi-source domain adaptation [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 59-64.
[2]	Dongjun LIU, Yuhan WANG, Wenfen LING, Yong PENG, Wanzeng KONG. Emotion recognition based on brain and machine collaborative intelligence [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 65-75.
[3]	Wenfen LING, Sihan CHEN, Yong PENG, Wanzeng KONG. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 76-84.