Chinese Journal of Intelligent Science and Technology ›› 2021, Vol. 3 ›› Issue (1): 36-48.doi: 10.11959/j.issn.2096-6652.202104
• Special topic:emotional brain computer interface • Previous Articles Next Articles
Bao-Liang LU1,2,3,4,5, Yaqian ZHANG1,2,3,5, Wei-Long ZHENG6
Revised:
2021-03-04
Online:
2021-03-15
Published:
2021-03-01
Supported by:
CLC Number:
Bao-Liang LU,Yaqian ZHANG,Wei-Long ZHENG. A survey of affective brain-computer interface[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 36-48.
[1] | WOLPAW J R , WOLPAW E W . 脑-机接口:原理与实践[M]. 伏云发,杨秋红,徐保磊,等,译. 北京: 国防工业出版社, 2017. |
WOLPAW J R , WOLPAW E W . Brain-computer interfaces:principles and practice[M]. Translated by FU Y F,YANG Q H,XU B L,et al. Beijing: National Defense Industry Press, 2017. | |
[2] | GRAIMANN B , ALLISON B Z , PFURTSCHELLER G . Brain- computer interfaces:revolutionizing human-computer interaction[M].[S.l.]: Springer Science & Business Media, 2010. |
[3] | SHANECHI M M . Brain-machine interfaces from motor to mood[J]. Nature Neuroscience, 2019,22: 1554-1564. |
[4] | HU X , CHEN J J , WANG F ,et al. Ten challenges for EEG-based affective computing[J]. Brain Science Advances, 2019,5(1): 1-20. |
[5] | 吕宝粮, 郑伟龙 . 情感脑-机接口[M]// 脑-计算机交互研究前沿. 上海: 上海交通大学出版社, 2021. |
LYU B L , ZHENG W L . Affective brain-computer interface[M]// Frontiers of brain-computer interaction research. Shanghai: Shanghai Jiao Tong University Press, 2021. | |
[6] | EKMAN P . An argument for basic emotions[J]. Cognition and Emotion, 1992,6(3-4): 169-200. |
[7] | RUSSELL J A . A circumplex model of affect[J]. Journal of Personality and Social Psychology, 1980,39(6): 1161-1178. |
[8] | LARSEN J T , MCGRAW A P , CACIOPPO J T . Can people feel happy and sad at the same time?[J]. Journal of Personality and Social Psychology, 2001,81(4): 684-696. |
[9] | FREDRICKSON B L . Positive emotions broaden and build[J]. Advances in Experimental Social Psycholog, 2013,47: 1-53. |
[10] | BRANNAN S , LIOTTI M , EGAN G ,et al. Neuroimaging of cerebral activations and deactivations associated with hypercapnia and hunger for air[J]. Proceedings of the National Academy of Sciences, 2001,98(4): 2029-2034. |
[11] | BüCHEL C , DOLAN R J . Classical fear conditioning in functional neuroimaging[J]. Current Opinion in Neurobiology, 2000,10(2): 219-223. |
[12] | PHAN K L , WAGER T , TAYLOR S F ,et al. Functional neuroanatomy of emotion:a meta-analysis of emotion activation studies in PET and fMRI[J]. Neuroimage, 2002,16(2): 331-348. |
[13] | LIOTTI M , PANKSEPP J . Imaging human emotions and affective feelings:implications for biological psychiatry[M]. Canada: Wiley-Liss, 2004. |
[14] | DOLENSEK N , GEHRLACH D A , KLEIN A S ,et al. Facial expressions of emotion states and their neuronal correlates in mice[J]. Science, 2020,368(6486): 89-94. |
[15] | 宿云, 胡斌, 徐立新 ,等. 面向脑电数据的知识建模和情感识别[J]. 科学通报, 2015,60(11): 1002-1009. |
SU Y , HU B , XU L X ,et al. EEG-data-oriented knowledge modeling and emotion recognition[J]. Chinese Science Bulletin, 2015,60(11): 1002-1009. | |
[16] | 权学良, 曾志刚, 蒋建华 ,等. 基于生理信号的情感计算研究综述[J]. 自动化学报, 2021,47: 1-16. |
QUAN X L , ZENG Z G , JIANG J H ,et al. Physiological signals based affective computing:a systematic review[J]. Acta Automatica Sinica, 2021,47: 1-16. | |
[17] | 张冠华, 余旻婧, 陈果 ,等. 面向情绪识别的脑电特征研究综述[J]. 中国科学(信息科学), 2019,49(9): 1097-1118. |
ZHANG G H , YU M J , CHEN G ,et al. A review of EEG features for emotion recognition[J]. Science in China (Information Sciences), 2019,49(9): 1097-1118. | |
[18] | DIAZ-PIEDRA C , SEBASTIáN M V , DI STASI L L . EEG theta power activity reflects workload among army combat drivers:an experimental study[J]. Brain Sciences, 2020,10(199): 1-14. |
[19] | BROUWER A-M , HOGERVORST M A , VAN ERP J B ,et al. Estimating workload using EEG spectral power and ERPs in the n-back task[J]. Journal of Neural Engineering, 2012,9(4): 045008. |
[20] | PUTNAM K M , MCSWEENEY L B . Depressive symptoms and baseline prefrontal EEG alpha activity:a study utilizing ecological momentary assessment[J]. Biological Psychology, 2008,77(2): 237-240. |
[21] | WHEELER R E , DAVIDSON R J , TOMARKEN A J . Frontal brain asymmetry and emotional reactivity:a biological substrate of affective style[J]. Psychophysiology, 1993,30(1): 82-89. |
[22] | ZHENG W L , LYU B L . Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on Autonomous Mental Development, 2015,7(3): 162-175. |
[23] | RAY W J , COLE H W . EEG alpha activity reflects attentional demands,and beta activity reflects emotional and cognitive processes[J]. Science, 1985,228(4700): 750-752. |
[24] | AHERN G L , SCHWARTZ G E . Differential lateralization for positive and negative emotion in the human brain:EEG spectral analysis[J]. Neuropsychologia, 1985,23(6): 745-755. |
[25] | MOHAMMADI Z , FROUNCHI J , AMIRI M . Wavelet-based emotion recognition system using EEG signal[J]. Neural Computing and Applications, 2017,28(8): 1985-1990. |
[26] | JATUPAIBOON N , PAN-NGUM S , ISRASENA P . Emotion classification using minimal EEG channels and frequency bands[C]// Proceedings of The 10th International Joint Conference on Computer Science and Software Engineering.[S.l.:s.n. ], 2013: 21-24. |
[27] | LI M , LYU B L . Emotion classification based on gamma-band EEG[C]// Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway:IEEE Press, 2009: 1223-1226. |
[28] | YANG K , TONG L , SHU J ,et al. High gamma band EEG closely related to emotion:evidence from functional network[J]. Frontiers in Human Neuroscience, 2020,14: 89. |
[29] | SHARMA K , CASTELLINI C , VAN DEN BROEK E L ,et al. A dataset of continuous affect annotations and physiological signals for emotion analysis[J]. Scientific Data, 2019,6(1): 1-13. |
[30] | DING Y , HU X , XIA Z Y ,et al. Inter-brain EEG feature extraction and analysis for continuous implicit emotion tagging during video watching[J]. IEEE Transactions on Affective Computing, 2018,12(1): 92-102. |
[31] | NUMMENMAA L , GLEREAN E , VIINIKAINEN M ,et al. Emotions promote social interaction by synchronizing brain activity across individuals[J]. Proceedings of the National Academy of Sciences, 2012,109(24): 9599-9604. |
[32] | KOELSTRA S , MUHL C , SOLEYMANI M ,et al. DEAP:a database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2011,3(1): 18-31. |
[33] | KATSIGIANNIS S , RAMZAN N . DREAMER:a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices[J]. IEEE Journal of Biomedical and Health Informatics, 2017,22(1): 98-107. |
[34] | LIU Y J , YU M J , ZHAO G Z ,et al. Real-time movie-induced discrete emotion recognition from EEG signals[J]. IEEE Transactions on Affective Computing, 2017,9(4): 550-562. |
[35] | SOLEYMANI M , LICHTENAUER J , PUN T ,et al. A multimodal database for affect recognition and implicit tagging[J]. IEEE Transactions on Affective Computing, 2011,3(1): 42-55. |
[36] | SUBRAMANIAN R , WACHE J , ABADI M K ,et al. Emotion and personality recognition using commercial sensors[J]. IEEE Transactions on Affective Computing, 2016,9(2): 147-160. |
[37] | CORREA J A M , ABADI M K , SEBE N ,et al. AMIGOS:a dataset for affect,personality and mood research on individuals and groups[J]. IEEE Transactions on Affective Computing, 2018: 1. |
[38] | LIN Y P , JUNG T P . Improving EEG-based emotion classification using conditional transfer learning[J]. Frontiers in Human Neuroscience, 2017,11: 334. |
[39] | LUO Y , ZHANG S Y , ZHENG W L ,et al. WGAN domain adaptation for EEG-based emotion recognition[C]// Proceedings of the International Conference on Neural Information Processing.[S.l.:s.n.], 2018: 275-286. |
[40] | ZHENG W L , LYU B L . Personalizing EEG-based affective models with transfer learning[C]// Proceedings of The 25th International Joint Conference on Artificial Intelligence.[S.l.:s.n.], 2016: 2732-2738. |
[41] | WAN Z , YANG R , HUANG M J ,et al. A review on transfer learning in EEG signal analysis[J]. Neurocomputing, 2020,421: 1-14. |
[42] | WU D R , XU Y F , LYU B L . Transfer learning for EEG-based brain-computer interfaces:a review of progresses since 2016[J]. arXiv preprint,2020,arXiv:200406286. |
[43] | LI J P , QIU S , SHEN Y Y ,et al. Multisource transfer learning for cross-subject EEG emotion recognition[J]. IEEE Transactions on Cybernetics, 2019,50(7): 3281-3293. |
[44] | MA B Q , LI H , ZHENG W L ,et al. Reducing the subject variability of EEG signals with adversarial domain generalization[C]// The 26th International Conference on Neural Information Processing.[S.l.:s.n.], 2019: 30-42. |
[45] | ZHAO L M , YAN X , LYU B L . Plug-and-play domain adaptation for cross-subject EEG-based emotion recognition[C]// Proceedings of the 35th AAAI Conference on Artificial Intelligence.[S.l.:s.n.], 2021. |
[46] | JIANG Y Z , WU D R , DENG Z H ,et al. Seizure classification from EEG signals using transfer learning,semi-supervised learning and TSK fuzzy system[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017,25(12): 2270-2284. |
[47] | 郑伟龙, 石振锋, 吕宝粮 . 用异质迁移学习构建跨被试脑电情感模型[J]. 计算机学报, 2020,43(2): 177-189. |
ZHENG W L , SHI Z F , LYU B L . Building cross-subject EEG-based affective models using heterogeneous transfer learning[J]. Chinese Journal of Computers, 2020,43(2): 177-189. | |
[48] | ZHU Y , CHEN Y Q , LU Z Q ,et al. Heterogeneous transfer learning for image classification[C]// Proceedings of the 25th AAAI Conference on Artificial Intelligence.[S.l.:s.n.], 2011. |
[49] | MARQUART G , CABRALL C , DE WINTER J . Review of eye-related measures of drivers’ mental workload[J]. Procedia Manufacturing, 2015,3: 2854-2861. |
[50] | ZHU X M , ZHENG W L , LYU B L ,et al. EOG-based drowsiness detection using convolutional neural networks[C]// Proceedings of the 2014 International Joint Conference on Neural Networks.[S.l.:s.n.], 2014: 128-134. |
[51] | CHANG K M , LIU S H , WU X H . A wireless sEMG recording system and its application to muscle fatigue detection[J]. Sensors, 2012,12(1): 489-499. |
[52] | NOURBAKHSH N , WANG Y , CHEN F . GSR and blink features for cognitive load classification[C]// Proceedings of the IFIP Conference on Human- Computer Interaction.[S.l.:s.n.], 2013: 159-166. |
[53] | ROSCOE A H . Assessing pilot workload.Why measure heart rate,HRV and respiration?[J]. Biological Psychology, 1992,34(2-3): 259-287. |
[54] | SOLEYMANI M , ASGHARI-ESFEDEN S , PANTIC M ,et al. Continuous emotion detection using EEG signals and facial expressions[C]// Proceedings of the 2014 IEEE International Conference on Multimedia and Expo. Piscataway:IEEE Press, 2014: 1-6. |
[55] | HUANG Y R , YANG J H , LIAO P K ,et al. Fusion of facial expressions and EEG for multimodal emotion recognition[J]. Computational Intelligence and Neuroscience, 2017: 2017451 |
[56] | HUANG X H , KORTELAINEN J , ZHAO G Y ,et al. Multi-modal emotion analysis from facial expressions and electroencephalogram[J]. Computer Vision and Image Understanding, 2016,147: 114-214. |
[57] | LAN Y T , LIU W , LYU B L . Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism[C]// Proceedings of the 2020 International Joint Conference on Neural Networks.[S.l.:s.n.], 2020: 1-6. |
[58] | ZHENG W L , LIU W , LU Y ,et al. Emotionmeter:a multimodal framework for recognizing human emotions[J]. IEEE Transactions on Cybernetics, 2018,49(3): 1110-1122. |
[59] | LU Y F , ZHENG W L , LI B B ,et al. Combining eye movements and eeg to enhance emotion recognition[C]// Proceedings of the 24th International Conference on Artificial Intelligence.[S.l.:s.n.], 2015: 1170-1176. |
[60] | EKMAN P . Facial expression and emotion[J]. American Psychologist, 1993,48(4): 384-392. |
[61] | RUSSELL J A . Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies[J]. Psychological Bulletin, 1994,115(1): 102-114. |
[62] | COWEN A S , KELTNER D , SCHROFF F ,et al. Sixteen facial expressions occur in similar contexts worldwide[J]. Nature, 2020(7841): 1-7. |
[63] | SUN Y , AYAZ H , AKANSU A N . Multimodal affective state assessment using fNIRS+ EEG and spontaneous facial expression[J]. Brain Sciences, 2020,10(2): 85. |
[64] | MINOTTO V P , JUNG C R , LEE B . Multimodal multi-channel on-line speaker diarization using sensor fusion through SVM[J]. IEEE Transactions on Multimedia, 2015,17(10): 1694-1705. |
[65] | CAI H S , QU Z D , LI Z ,et al. Feature-level fusion approaches based on multimodal EEG data for depression recognition[J]. Information Fusion, 2020,59: 127-138. |
[66] | GOSHVARPOUR A , GOSHVARPOUR A . Schizophrenia diagnosis using innovative EEG feature-level fusion schemes[J]. Physical and Engineering Sciences in Medicine, 2020,43(1): 227-238. |
[67] | QIAN M , AGUILAR M , ZACHERY K N ,et al. Decision-level fusion of EEG and pupil features for single-trial visual detection analysis[J]. IEEE Transactions on Biomedical Engineering, 2009,56(7): 1929-1937. |
[68] | ZHENG W L , DONG B N , LYU B L . Multimodal emotion recognition using EEG and eye tracking data[C]// Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway:IEEE Press, 2014: 5040-5043. |
[69] | NGIAM J , KHOSLA A , KIM M ,et al. Multimodal deep learning[C]// Proceedings of the 26th International Conference on Machine Learning.[S.l.:s.n.], 2009. |
[70] | LIU W , ZHENG W L , LYU B L . Multimodal emotion recognition using multimodal deep learning[J]. arXiv preprint, 2016,arXiv:160208225. |
[71] | SRIVASTAVA N , SALAKHUTDINOV R . Multimodal learning with deep boltzmann machines[J]. The Journal of Machine Learning Research, 2014,15(1): 2949-2980. |
[72] | VOS T , ABAJOBIR A A , ABATE K H ,et al. Global,regional,and national incidence,prevalence,and years lived with disability for 328 diseases and injuries for 195 countries,1990-2016:a systematic analysis for the Global Burden of Disease Study 2016[J]. The Lancet, 2017,390: 1211-1259. |
[73] | HANCOCK P A , MESHKATI N . Human mental workload[M].[S.l.]: Elsevier, 1988. |
[74] | NAKAMURA J , CSIKSZENTMIHALYI M . Flow theory and research[M].[S.l.]: Handbook of Positive Psychology, 2009. |
[75] | LOFT S , SANDERSON P , NEAL A ,et al. Modeling and predicting mental workload in en route air traffic control:critical review and broader implications[J]. Human Factors, 2007,49(3): 376-399. |
[76] | COLLET C , AVERTY P , DITTMAR A . Autonomic nervous system and subjective ratings of strain in air-traffic control[J]. Applied Ergonomics, 2009,40(1): 23-32. |
[77] | BERKA C , LEVENDOWSKI D J , LUMICAO M N ,et al. EEG correlates of task engagement and mental workload in vigilance,learning,and memory tasks[J]. Aviation,Space,and Environmental Medicine, 2007,78(5): B231-44. |
[78] | HOGERVORST M A , BROUWER A-M , VAN ERP J B . Combining and comparing EEG,peripheral physiology and eye-related measures for the assessment of mental workload[J]. Frontiers in Neuroscience, 2014,8: 322. |
[79] | HANCOCK P A , MESHKATI N , ROBERTSON M . Physiological reflections of mental workload[J]. Aviation,Space,and Environmental Medicine, 1985,56(11): 1110-1114. |
[80] | CHARLES R L , NIXON J . Measuring mental workload using physiological measures:a systematic review[J]. Applied Ergonomics, 2019,74: 221-232. |
[81] | FALLAHI M , MOTAMEDZADE M , HEIDARIMOGHADAM R ,et al. Effects of mental workload on physiological and subjective responses during traffic density monitoring:a field study[J]. Applied Ergonomics, 2016,52: 95-103. |
[82] | THIBODEAU R , JORGENSEN R S , KIM S . Depression,anxiety,and resting frontal EEG asymmetry:a meta-analytic review[J]. Journal of Abnormal Psychology, 2006,115(4): 715-729. |
[83] | DAMBORSKá A , TOMESCU M I , HONZíRKOVá E ,et al. EEG resting-state large-scale brain network dynamics are related to depressive symptoms[J]. Frontiers in Psychiatry, 2019,10: 548. |
[84] | CAI H S , ZHANG X Z , ZHANG Y H ,et al. A case-based reasoning model for depression based on three-electrode EEG data[J]. IEEE Transactions on Affective Computing, 2018,11(3): 383-392. |
[85] | ZHANG Y , WU W , TOLL R T ,et al. Identification of psychiatric disorder subtypes from functional connectivity patterns in resting-state electroencephalography[J]. Nature Biomedical Engineering, 2020: 1-15. |
[86] | HAMILTON M . The Hamilton rating scale for depression[M].[S.l.]: Springer, 1986. |
[87] | MONTGOMERY S , ASBERG M . A new depression scale designed to be sensitive to change[J]. The British journal of psychiatry, 1979,134(4): 382-389. |
[88] | KOHOUT F J , BERKMAN L F , EVANS D A ,et al. Two shorter forms of the CES-D depression symptoms index[J]. Journal of Aging and Health, 1993,5(2): 179-193. |
[89] | ZUNG W W . A self-rating depression scale[J]. Archives of General Psychiatry, 1965,12(1): 63-70. |
[90] | BECK A T , BECK R W . Screening depressed patients in family practice:a rapid technic[J]. Postgraduate Medicine, 1972,52(6): 81-85. |
[91] | KROENKE K , SPITZER R L , WILLIAMS J B . The PHQ‐9:validity of a brief depression severity measure[J]. Journal of General Internal Medicine, 2001,16(9): 606-613. |
[92] | ZIGMOND A S , SNAITH R P . The hospital anxiety and depression scale[J]. Acta Psychiatrica Scandinavica, 1983,67(6): 361-370. |
[93] | CARNEY M , SHEFFIELD B . Depression and the newcastle scales their relationship to Hamilton’s scale[J]. The British Journal of Psychiatry, 1972,121(560): 35-40. |
[94] | PERLMUTTER J S , MINK J W . Deep brain stimulation[J]. Annu Rev Neurosci, 2006,29: 229-257. |
[95] | DENG Z D , LI D Y , ZHANG C C ,et al. Long-term follow-up of bilateral subthalamic deep brain stimulation for refractory tardive dystonia[J]. Parkinsonism and Related Disorders, 2017,41: 58-65. |
[96] | YANG Y X , QIAO S Y , SANI O G ,et al. Modelling and prediction of the dynamic responses of large-scale brain networks during direct electrical stimulation[J]. Nature Biomedical Engineering, 2021: 1-22. |
[97] | SCANGOS K W , MAKHOUL G S , SUGRUE L P ,et al. State- dependent responses to intracranial brain stimulation in a patient with depression[J]. Nature Medicine, 2021,27: 229-231. |
[98] | RAO V R , SELLERS K K , WALLACE D L ,et al. Direct electrical stimulation of lateral orbitofrontal cortex acutely improves mood in individuals with symptoms of depression[J]. Current Biology, 2018,28(24): 3893-3902. |
[1] | Hanbing GAO, Chi ZHANG, Mingyan JIN, Yang XIAO, Fengyu CONG. Cross-subject emotional EEG recognition based on multi-source domain adaptation [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 59-64. |
[2] | Dongjun LIU, Yuhan WANG, Wenfen LING, Yong PENG, Wanzeng KONG. Emotion recognition based on brain and machine collaborative intelligence [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 65-75. |
[3] | Wenfen LING, Sihan CHEN, Yong PENG, Wanzeng KONG. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 76-84. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||
|