基于SDBN和BLSTM注意力融合的端到端视听双模态语音识别

doi:10.11959/j.issn.1000-0801.2019290

电信科学 ›› 2019, Vol. 35 ›› Issue (12): 79-89.doi: 10.11959/j.issn.1000-0801.2019290

基于SDBN和BLSTM注意力融合的端到端视听双模态语音识别

王一鸣,陈恳,萨阿卜杜萨拉木·艾海提拉木

宁波大学信息科学与工程学院，浙江宁波 315211

修回日期:2019-12-10 出版日期:2019-12-20 发布日期:2020-01-15
作者简介:王一鸣（1993- ），男，宁波大学信息科学与工程学院硕士生，主要研究方向为音视频信息处理、视听觉语音识别|陈恳（1962- ），男，宁波大学信息科学与工程学院副教授、硕士生导师，在核心期刊和重要国际会议发表论文共100余篇，参与和主持国家级、省部级、市厅级和校级科研项目共16项；获得相关科研相关奖项3项。主要研究方向为图像及视频信息处理、多媒体通信、智能控制|阿卜杜萨拉木·艾海提（1995- ），男，宁波大学信息科学与工程学院硕士生，主要研究方向为机器翻译、智能语音翻译
基金资助:
国家自然科学基金资助项目(60972063);宁波市自然科学基金资助项目(2014A610065);宁波大学科研基金（理）/学科资助项目(XKXL1308)

End-to-end audiovisual speech recognition based on attention fusion of SDBN and BLSTM

Yiming WANG,Ken CHEN,Aihaiti ABUDUSALAMU

Institute of Communication Technology,Ningbo University,Ningbo 315211,China

Revised:2019-12-10 Online:2019-12-20 Published:2020-01-15
Supported by:
The National Natural Science Foundation of China(60972063);The Natural Science Foundation of Ningbo of China(2014A610065);Scientific Research Foundation of Ningbo University(XKXL1308)

摘要/Abstract

摘要：

提出一种端到端的视听语音识别算法。在该算法中，通过具有瓶颈结构的深度信念网络（deep belief network，DBN）中引入混合的l1/2范数和l1范数构建一种稀疏DBN（sparse DBN，SDBN）来提取稀疏瓶颈特征，从而实现对数据的特征降维，然后用双向长短期记忆网络（bidirectional long short-term memory，BLSTM）在时序上对特征进行模态处理，之后利用一种注意力机制将经过模态处理的唇部视觉信息和音频听觉信息进行自动对齐、融合，最后将融合的视听觉信息通过一个附加了Softmax层的BLSTM进行分类识别。实验表明，该算法能有效地识别视听觉信息，在同类算法中有很好的识别率和顽健性。

关键词: 端到端, 视听语音识别, 稀疏瓶颈特征, 注意力机制

Abstract:

An end-to-end audiovisual speech recognition algorithm was proposed.In algorithm,a sparse DBN was constructed by introducing mixed l1/2norm and l1norm into Deep Belief Network with bottleneck structure to extract the sparse bottleneck features,so as to reduce the dimension of data features,and then a BLSTM was used to model the feature in time series.Then,a attention mechanism was used to align and fuse the lip visual information and audio auditory information automatically.Finally,the fused audiovisual information was classified and identified by a BLSTM with a Softmax layer attached.Experiments show that the algorithm can effectively identify visual and auditory information,and has good recognition rate and robustness in similar algorithms.

Key words: end-to-end, audiovisual speech recognition, sparse bottleneck features, attention mechanism

中图分类号:

TP391

王一鸣,陈恳,萨阿卜杜萨拉木·艾海提拉木. 基于SDBN和BLSTM注意力融合的端到端视听双模态语音识别[J]. 电信科学, 2019, 35(12): 79-89.

Yiming WANG,Ken CHEN,Aihaiti ABUDUSALAMU. End-to-end audiovisual speech recognition based on attention fusion of SDBN and BLSTM[J]. Telecommunications Science, 2019, 35(12): 79-89.

图/表 11

图1

图2

图3

图4

表1

图5

表2

图6

表3

图7

表4

参考文献 27

[1]	王海坤, 潘嘉, 刘聪 . 语音识别技术的研究进展与展望[J]. 电信科学, 2018,34(2): 1-11.
	WANG H K , PAN J , LIU C . Research development and forecast of automatic speech recognition technologies[J]. Telecommunications Science, 2018,34(2): 1-11.
[2]	CHIU C C , SAINATH T N , WU Y ,et al. State-of-the-art speech recognition with sequence-to-sequence models[C]// ICASSP 2018-2018 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP),April 15-20,2018,Calgary,AB,Canada. Piscataway:IEEE Press, 2018.
[3]	王志宏, 杨震 . 人工智能技术研究及未来智能化信息服务体系的思考[J]. 电信科学, 2017,33(5): 1-11.
	WANG Z H , YANG Z . Research on artificial intelligence technology and the future intelligent information service architecture[J]. Telecommunications Science, 2017,33(5): 1-11.
[4]	NODA K , YAMAGUCHI Y , NAKADAI K ,et al. Audio-visual speech recognition using deep learning[J]. Applied Intelligence, 2015,42(4): 722-737.
[5]	HU D , LI X , LU X . Temporal multimodal learning in audiovisual speech recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),April 19-24,2016,Brisbane,QLD,Australia. Piscataway:IEEE Press, 2016: 3574-3582.
[6]	TAKASHIMA Y , AIHARA R , TAKIGUCHI T ,et al. Audio-visual speech recognition using bimodal-trained bottleneck features for a person with severe hearing loss[J]. 2016:
[7]	CHUNG J S , SENIOR A , VINYALS O ,et al. Lip reading sentences in the wild[J]. arXiv:1611.05358, 2016.
[8]	PETRIDIS S , STAFYLAKIS T , MA P ,et al. End-to-end audiovisual speech recognition[C]// 2018 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP),April 15-20,2018,Calgary,AB,Canada. Piscataway:IEEE Press, 2018: 6548-6552.
[9]	WAND M , SCHMIDHUBER J , VU N T . Investigations on End-to-End Audiovisual Fusion[C]// 2018 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP),April 15-20,2018,Calgary,AB,Canada. Piscataway:IEEE Press, 2018: 3041-3045.
[10]	TAO F , BUSSO C . Gating neural network for large vocabulary audiovisual speech recognition[J]. IEEE/ACM Transactions on Audio,Speech and Language Processing (TASLP), 2018,26(7): 1286-1298.
[11]	HINTON G E , SALAKHUTDINOV R R . Reducing the dimensionality of data with neural networks[J]. Science, 2006,313(578): 504-507.
[12]	GREZL F , KARAFIAT M , KONTAR S ,et al. Probabilistic and bottle-neck features for LVCSR of meetings[C]// IEEE International Conference on Acoustics,April 15-20,2007,Honolulu,HI,USA. Piscataway:IEEE Press, 2007.
[13]	JI N N , ZHANG J S , ZHANG C X . A sparse-response deep belief network based on rate distortion theory[J]. Pattern Recognition, 2014,47(9): 3179-3191.
[14]	LEE H , EKANADHAM C , NG A Y . Sparse deep belief net model for visual area V2[C]// Advances in Neural Information Processing Systems,Dec 3-6,2007,Vancouver,British Columbia,Canada.[S.l:s.n. ], 2007: 873-880.
[15]	KIAEE F , CHRISTIAN G , ABBASI M . Alternating direction method of multipliers for sparse convolutional neural networks[J]. 2016:
[16]	KHALID F , FANANY M I . Combining normal sparse into discriminative deep belief networks[C]// International Conference on Advanced Computer Science ＆ Information Systems,Oct 15-16,2016,Malang,Indonesia. Piscataway:IEEE Press, 2016.
[17]	GRAVES A , SCHMIDHUBER J . Framewise phoneme classification with bidirectional LSTM and other neural network architectures[J]. Neural Networks, 2005,18(5-6): 602-610.
[18]	LUONG M T , PHAM H , MANNING C D . Effective approaches to attention-based neural machine translation[J].,2015. arXiv preprint arXiv:1508.04025, 2015.
[19]	ANINA I , ZHOU Z , ZHAO G ,et al. OuluVS2:a multi-view audiovisual database for non-rigid mouth motion analysis[C]// IEEE International Conference ＆ Workshops on Automatic Face ＆ Gesture Recognition,May 4-8,2015,Ljubljana,Slovenia. Piscataway:IEEE Press, 2015.
[20]	HINTON G E . A practical guide to training restricted Boltzmann machines[Z].2012. 2012.
[21]	GLOROT X , BENGIO Y . Understanding the difficulty of training deep feedforward neural networks[C]// The Thirteenth International Conference on Artificial Intelligence and Statistics,May 13-15,2010,Chia Laguna Resort,Sardinia,Italy.[S.l.:s.n]. 2010: 249-256.
[22]	PETRIDIS S , WANG Y , LI Z ,et al. End-to-end audiovisual fusion with LSTM[J]. arXiv preprint arXiv:1709.04343, 2017.
[23]	KOUMPAROULIS A , POTAMIANOS G . Deep View2View mapping for view-invariant lipreading[C]// 2018 IEEE Spoken Language Technology Workshop (SLT),Dec 18-21,2018,Athens,Greece. Piscataway:IEEE Press, 2018: 588-594.
[24]	SAITOH T , ZHOU Z , ZHAO G ,et al. Concatenated frame image based CNN for visual speech recognition[C]// Asian Conference on Computer Vision,Nov 20-24,2016,Taipei,China. Berlin:Springer, 2016: 277-289.
[25]	FUNG I , MAK B . End-to-end low-resource lip-reading with maxout CNN and LSTM[C]// 2018 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP),April 15-20,2018,Calgary,AB,Canada. Piscataway:IEEE Press, 2018: 2511-2515.
[26]	LEE D , LEE J , KIM K E . Multi-view automatic lip-reading using neural network[C]// Asian Conference on Computer Vision,Nov 20-24,2016,Taipei,China. Berlin:Springer, 2016: 290-302.
[27]	PETRIDIS S , WANG Y , LI Z ,et al. End-to-end multi-view lipreading[J]. arXiv preprint arXiv:1709.00443, 2017.

编号	短语
1	Excuse me
2	Good bye
3	Hello
4	How are you?
5	Nice to meet you
6	See you
7	I am sorry
8	Thank you
9	Have a good time
10	·ou are welcome

数据集	受试者编号
训练集	1、2、3、5、7、10、11、12、14、16、17、18、19、20、21、23、24、25、27、28、31、32、33、35、36、37、39、40、41、42、45、46、47、48、53
验证集	4、13、22、38、50
测试集	6、8、9、15、26、30、34、43、44、49、51、52

算法	0°	30°	45°	60°	90°%	A	A+0°	A+30°	A+45°	A+60°	A+90°
参考文献[22]算法	94.7%	89.7%	90.6%	87.5%	93.1%	98.5%	98.6%	98.7%	98.3%	98.6%	98.9%
参考文献[23]算法	90.0%	89.2%	89.7%	87.2%	84.2%	86.39%（-5 dB）	89.7%	90.3%	89.4%	86.7%	85.3%
本文算法	95.1%	90.0%	90.3%	87.8%	93.7%	98.4%	99.1%	98.9%	98.6%	98.8%	98.9%

算法	0°	30°	45°	60°	90°
参考文献[24]算法	85.6%	82.5%	82.5%	83.3%	80.3%
参考文献[25]算法	87.6%	—	—	—	—
参考文献[26]算法	81.1%	80.0%	76.9%	69.2%	82.2%
参考文献[27]算法	94.7%	89.7%	90.6%	87.5%	93.1%
本文算法	95.1%	90.0%	90.3%	87.8%	93.7%

基于SDBN和BLSTM注意力融合的端到端视听双模态语音识别

End-to-end audiovisual speech recognition based on attention fusion of SDBN and BLSTM

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 27

相关文章 15

Metrics

推荐阅读 0

[1]	张永, 刘纪奎, 柯文龙. 基于并行可分离卷积和标签平滑正则化的脑电情感识别[J]. 电信科学, 2023, 39(5): 116-128.
[2]	谭艳霞, 王光全, 王泽林, 郑滟雷, 张贺, 张晨芳, 韩赛, 沈世奎. SDN智能管控编排系统技术方案研究[J]. 电信科学, 2023, 39(3): 143-152.
[3]	陈佳, 章坚武, 张浙亮. 基于上下文信息与注意力特征的欺骗语音检测[J]. 电信科学, 2023, 39(2): 92-102.
[4]	戴剑锋, 陈星妤, 董黎刚, 蒋献. 融合混合嵌入与关系标签嵌入的三元组联合抽取方法[J]. 电信科学, 2023, 39(2): 132-144.
[5]	陈凯. 基于云网融合的5G切片专网技术方案及应用[J]. 电信科学, 2022, 38(7): 166-174.
[6]	宋勇, 严志伟, 秦玉坤, 赵东明, 叶晓舟, 柴园园, 欧阳晔. 基于矩阵分解和注意力多任务学习的客服投诉工单分类[J]. 电信科学, 2022, 38(2): 103-110.
[7]	张振华, 孙思月, 刘高赛, 王龙, 姜兴龙, 董琳, 梁广. 卫星互联网光电混合交换技术综述[J]. 电信科学, 2022, 38(11): 1-10.
[8]	张国新. 5G“云网边端”一体化纵深安全防护体系研究及应用[J]. 电信科学, 2022, 38(10): 173-179.
[9]	金楠, 王瑞琴, 陆悦聪. 基于艾宾浩斯遗忘曲线和注意力机制的推荐算法[J]. 电信科学, 2022, 38(10): 89-97.
[10]	谭艳霞, 郑滟雷, 王光全, 张贺. OTN控制器北向统一接口模型研究与实践[J]. 电信科学, 2022, 38(10): 163-172.
[11]	刘鹏, 杜宗鹏, 李永竞, 陆璐, 段晓东. 端到端确定性网络架构和关键技术[J]. 电信科学, 2021, 37(9): 64-73.
[12]	孙金杨, 刘柏嵩, 任豪, 钱江波. TAGAN：一种融合细粒度语义特征的学术论文对抗推荐算法[J]. 电信科学, 2021, 37(8): 57-65.
[13]	张艳. 基于5G切片专线的组网技术和实现方法[J]. 电信科学, 2021, 37(10): 143-151.
[14]	刘义亮,李鑫,薄开涛. 5G端到端网络协同关键技术[J]. 电信科学, 2020, 36(3): 144-155.
[15]	穆天杰,陈晓辉,汪逸云,马陆鹏,刘东,周晶,张文逸. 基于深度学习的信源信道联合编码方法综述[J]. 电信科学, 2020, 36(10): 56-66.