基于SAE和LSTM RNN的多模态生理信号融合和情感识别研究

doi:10.11959/j.issn.1000-436x.2017294

Abstract

Abstract:

In order to achieve more accurate emotion recognition accuracy from multi-modal bio-signal features，a novel method to extract and fuse the signal with the stacked auto-encoder and LSTM recurrent neural networks was proposed.The stacked auto-encoder neural network was used to compress and fuse the features.The deep LSTM recurrent neural network was employed to classify the emotion states.The results present that the fused multi-modal features provide more useful information than single-modal features.The deep LSTM recurrent neural network achieves more accurate emotion classification results than other method.The highest accuracy rate is 0.792 6

Key words: multi-modal bio-signal emotion recognition, stacked auto-encoder neural network, LSTM recurrent neural network, multi-modal bio-signals fusion

CLC Number:

TP181，TP183

You-jun LI,Jia-jin HUANG,Hai-yuan WANG,Ning ZHONG. Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network[J]. Journal on Communications, 2017, 38(12): 109-120.

Figures/Tables 14

References 25

[1]	聂聃, 王晓韡, 段若男 ,等. 基于脑电的情绪识别研究综述[J]. 中国生物医学工程学报, 2012,31(4): 595-606.
	NIE D , WANG X H , DUAN R N ,et al. A survey on EEG based emotion recognition[J]. Journal of Biomedical Engineering, 2012,31(4): 595-606.
[2]	JONGHWA K , ANDRE E . Emotion recognition based on physiological changes in music listening[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008,30: 2067-2083.
[3]	赵力, 钱向民, 邹采荣 ,等. 语音信号中的情感识别研究[J]. 软件学报, 2001,12(7): 1050-1055.
	ZHAO L , QIAN X M , ZOU C R ,et al. A study on emotional recognition in speech signal[J]. Journal of Software, 2001,12(7): 1050-1055.
[4]	林奕琳, 韦岗, 杨康才 . 语音情感识别的研究进展[J]. 新能源进展, 2007,12(1): 90-98.
	LIN Y L , WEI G , YANG K C . A survey of emotion recognition in speech[J]. Journal of Circuits and Systems, 2007,12(1): 90-98.
[5]	赵腊生, 张强, 魏小鹏 . 语音情感识别研究进展[J]. 计算机应用研究, 2009,26(2): 34-38.
	ZHAO L S , ZHANG Q , WEI X P . Survey on speech emotion recognition[J]. Application Research of Computers, 2009,26(2): 34-38.
[6]	OTHMAN M , WAHAB A , KARIM I ,et al. EEG emotion recognition based on the dimensional models of emotions[J]. Procedia-Social and Behavioral Sciences, 2013,97(2): 30-37.
[7]	陈曾, 刘光远 . 脑电信号在情感识别中的应用[J]. 计算机工程, 2010,36(9): 168-170.
	CHEN Z , LIU G Y . Application of EEG signal in emotion recognition[J]. Computer Engineering, 2010,36(9): 168-170.
[8]	张栋, 陈东伟, 游雅 ,等. 基于自适应Lempel-Ziv复杂度的情感脑电信号特征分析[J]. 计算机应用与软件, 2014(9): 162-165.
	ZHANG D , CHEN D W , YOU Y ,et al. Analyzing emotional EEG signals feature based on adaptive LEMPEL-ZIV complexity[J]. Computer Applications and Software, 2014(9): 162-165.
[9]	UPASANA T , SHYAMANTA M H . Estimation of mental fatigue during EEG based motor imagery[C]// IHCI 2016:Intelligent Human Computer Interaction. 2016: 122-132.
[10]	BAJAJ V , PACHORI R B . Detection of human emotions using features based on the multiwavelet transform of EEG signals[M]. Springer International Publishing, 2015: 215-240.
[11]	HOSSEINI S A , NAGHIBISISTAN M B . Emotion recognition method using entropy analysis of EEG signals[J]. International Journal of Image Graphics＆Signal Processing, 2011,3(5): 30-36.
[12]	王凯明, 钟宁, 周海燕 . 基于改进功率谱熵的抑郁症脑电信号活跃性研究[J]. 物理学报, 2014,63(17): 178701-178701.
	WANG K M , ZHONG N , ZHOU H Y . Activity analysis of depression electroencephalogram based on modified power spectral entropy[J]. Acta Phys Sin, 2014,63(17): 178701-178701.
[13]	KREIBIG S D . Autonomic nervous system activity in emotion:a review[J]. Biological Psychology, 2010,84(3): 394-421.
[14]	KOELSTRA S , MUHL C , SOLEYMANI M ,et al. DEAP:a database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2012,3(1): 18-31.
[15]	PAUL E . An argument for basic emotions[J]. Cognition and emotion, 1992,6(3/4): 169-200.
[16]	POSNER J , RUSSELL J A , PETERSON B S . The circumplex model of affect:an integrative approach to affective neuroscience,cognitive development,and psychopathology[J]. Development and Psychopathology, 2005,17(3): 715-734.
[17]	ZHANG P , MA X , ZHANG W ,et al. Multimodal fusion for sensor data using stacked autoencoders[C]// IEEE Tenth International Conference on Intelligent Sensors,Sensor Networks and Information Processing. 2015.
[18]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[19]	HUANG N E , SHEN Z , LONG S R ,et al. The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis[J]. The Royal Society, 1998,454(1971): 903-995.
[20]	AGRAFIOTI F , HATZINAKOS D , ANDERSON A K . ECG pattern analysis for emotion detection[J]. IEEE Transactions on Affective Computing, 2012,3(1): 102-115.
[21]	BAILENSON J N , PONTIKAKIS E D , MAUSS I B ,et al. Real-time classification of evoked emotions using facial feature tracking and physiological responses[J]. International Journal of Human-Computer Studies, 2008,66(5): 303-317.
[22]	LISETTI C L , NASOZ F . Using noninvasive wearable computers to recognize human emotions from physiological signals[J]. Eurasip Journal on Advances in Signal Processing, 2004,2004(11): 1672-1687.
[23]	FLEUREAU J , GUILLOTEL P , QUAN H T . Physiological-based affect event detector for entertainment video applications[J]. IEEE Transactions on Affective Computing, 2012,3(3): 379-385.
[24]	CHUNG S Y , YOON H J . Affective classification using Bayesian classifier and supervised learning[C]// International Conference on Control,Automation and Systems. 2012: 1768-1771.
[25]	LI X , SONG D , ZHANG P ,et al. Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network[C]// IEEE International Conference on Bioinformatics and Biomedicine. 2017: 352-359.

Metrics

Recommended 0

No Suggested Reading articles found!

提出者	情感种类	基本情感
Paul	6种	生气、厌恶、害怕、高兴、悲伤、吃惊
Parrot	6种	生气、害怕、高兴、喜爱、悲伤、吃惊
Frijda	6种	渴望、高兴、感兴趣、吃惊、惊奇、悲伤
Plutchik	8种	接纳、生气、期望、厌恶、快乐、害怕、悲伤、吃惊
Tomkins	9种	生气、感兴趣、轻蔑、厌恶、悲痛、害怕、高兴、害羞、吃惊

情感分类标签	样本个数
HVHA	348
LVHA	298
LVLA	282
HVLA	352
合计	1 280

分类	HVHA	LVHA	LVLA	HVLA
效价	1.634 35	?1.108 59	?2.324 97	1.185 36
唤醒度	2.127 02	1.267 37	?1.596 66	?1.896 65

生理信号类型（维数）	特征值及描述
脑电信号（EEG）特征值（32×5×60）	32导脑电数据× 60 s×5层IMF提取的PSD 特征值
眼电信号（EOG）特征值（4×60）	4个测量点×60信号能量特征值
肌电信号（EMG）特征值（2×60）	斜方肌肌电1×60、颧肌肌电1×60信号能量特征值
皮肤电信号（GSR）特征值（1×60）	1×60 0～2.4 Hz功率谱能量特征值

生物信号类别	输入层宽度	第一隐藏层宽度	第二隐藏层宽度
脑电信号	160	80
眼电信号	4	2	64
肌电信号	2	1	64
皮肤电信号	1	1

Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 25

Related Articles 15

Metrics

Recommended 0

信号类型	准确率	F1得分
皮肤电信号、眼电信号和肌电信号特征	0.543 7	0.528 7
脑电信号特征	0.762 3	0.744 8
多模态融合信号	0.792 6	0.768 2

分类方法	特征提取时间窗口时长
分类方法	1s	2s	3s	4s	5s	10 s	60 s
Complex Tree	0.413 2	0.453 4	0.432 3	0.477 2	0.519 1	0.512 3	0.511 2
KNN	0.423 2	0.463 4	0.482 3	0.527 2	0.549 1	0.572 3	0.531 2
SVM	0.637 2	0.651 0	0.672 1	0.641 0	0.628 1	0.665 7	0.626 0
RNN	0.572 1	0.558 2	0.543 9	0.551 3	0.533 7	0.521 7	0.501 0
LSTM RNN	0.594 3	0.648 9	0.792 6	0.755 6	0.683 4	0.678 1	0.507 5

主要研究者	情感刺激媒体	识别情感类型	被测试个数	生理信号种类	数据分析方法	正确率
Agrafioti	视频，游戏	正负唤醒度和效价	44	心电信号	EMD	正向唤醒度：0.784 3负向唤醒度：0.524 1留一交叉验证
Bailenson	视频	有趣、悲伤、平静	41	脸部表情、心电、皮肤电传导、体细胞活性	WEKA 数据分析平台中的相关分析方法	对于有趣情感类型，结合脸部表情和生理信号识别率达到0.90二折交叉验证
Lisetti	电影片段数学难题	悲伤、气氛、害怕、吃惊、挫败、有趣	29	皮肤电反应、心率、体温	KNN	对于不同的情感类型识别率0.704～0.809留一交叉验证
Fleureau	视频片段声音片段	事件判定、效价判定	10	皮肤电反应、肌电	高斯函数	对效价的最好分类达到0.854 1二折交叉验证
Chung	视频片段	唤醒度、效价、喜欢程度	32	脑电、肌电、皮肤电、血压、体温、呼吸	Bayes classifier	0.666效价0.664唤醒度
Li	视频片段	唤醒度、效价、喜欢程度	32	脑电、肌电、皮肤电、血压、体温、呼吸	CNN+RNN	0.720 6效价0.741 2唤醒度
Koelstra	视频片段	唤醒度、效价、喜欢程度	32	脑电、肌电、皮肤电、血压、体温、呼吸	Single-trial Classification	0.616～0.647

[1]	Rong QIAN, Jianting XU, Kejun ZHANG, Hongyu DONG, Fangyuan XING. Research on HMM based link prediction method in heterogeneous network [J]. Journal on Communications, 2022, 43(5): 214-225.
[2]	Yiran GU, Zhupeng YAO, Haigen YANG. Deep factorization machine model based on attention capsule [J]. Journal on Communications, 2021, 42(10): 130-139.
[3]	Qiuyang GU, Bao WU, Zhaoyang SUN, Renyong CHI. Key node identification algorithm for complex network based on improved grey wolf optimization [J]. Journal on Communications, 2021, 42(6): 72-83.
[4]	Xiaohui YANG,Xiaoming LIU. Local outlier factor algorithm based on correction of bidirectional neighbor [J]. Journal on Communications, 2020, 41(8): 130-140.
[5]	Fengli XU,Yong LI. Survey on user’s mobility behavior modelling in urban environment [J]. Journal on Communications, 2020, 41(7): 18-28.
[6]	Chunxiang GU,Weisen WU,Ya’nan SHI,Guangsong LI. Method of unknown protocol classification based on autoencoder [J]. Journal on Communications, 2020, 41(6): 88-97.
[7]	Feiyue QIU,Bowen CHEN,Tieming CHEN,Guodao ZHANG. Sparsity induced convex nonnegative matrix factorization algorithm with manifold regularization [J]. Journal on Communications, 2020, 41(5): 84-95.
[8]	Guanghui YAN, Meng ZHANG, Hao LUO, Shikui LI, Ting LIU. Identifying vital nodes algorithm in social networks fusing higher-order information [J]. Journal on Communications, 2019, 40(10): 109-118.
[9]	Bin WU,Yun CHEN,Zhongchuan SUN,Yangdong YE. Co-pairwise ranking model for item recommendation [J]. Journal on Communications, 2019, 40(9): 193-206.
[10]	Xiaohui YANG,Shengchang ZHANG. Anomaly detection model based on multi-grained cascade isolation forest algorithm [J]. Journal on Communications, 2019, 40(8): 133-142.
[11]	Xiaoling YIN,Xiaojiang CHEN,Qishou XIA,Juan HE,Pengyan ZHANG,Feng CHEN. Human motion state recognition based on smart phone built-in sensor [J]. Journal on Communications, 2019, 40(3): 157-169.
[12]	Haoran LIU,Pan DING,Changjiang GUO,Jinfeng CHANG,Jingchuang CUI. Study on Chinese spam filtering system based on Bayes algorithm [J]. Journal on Communications, 2018, 39(12): 151-159.
[13]	Li ZHANG,Cong WANG. Multi-label feature selection algorithm based on joint mutual information of max-relevance and min-redundancy [J]. Journal on Communications, 2018, 39(5): 111-122.
[14]	Lan-lan KANG,Wen-yong DONG,Wan-juan SONG,Kang-shun LI. Non-inertial opposition-based particle swarm optimization with adaptive elite mutation [J]. Journal on Communications, 2017, 38(8): 66-78.
[15]	Peng ZHANG,Quan LIU,Shan ZHONG,Jian-wei ZHAI,Wei-sheng QIAN. Actor-critic algorithm with incremental dual natural policy gradient [J]. Journal on Communications, 2017, 38(4): 166-177.