基于可见光的环境自适应手势识别系统

doi:10.11959/j.issn.2096-3750.2023.00344

物联网学报 ›› 2023, Vol. 7 ›› Issue (2): 15-25.doi: 10.11959/j.issn.2096-3750.2023.00344

基于可见光的环境自适应手势识别系统

王柱¹, 张化磊¹, 胡千红², 於志文¹

¹ 西北工业大学计算机学院，陕西西安 710072
² 湖南文理学院计算机与电气工程学院，湖南常德 415006

修回日期:2023-04-17 出版日期:2023-06-30 发布日期:2023-06-01
作者简介:王柱（1983- ），男，博士，西北工业大学教授，主要研究方向为智能感知与普适计算
张化磊（1998- ），男，西北工业大学计算机学院硕士生，主要研究方向为无线感知
胡千红（1993- ），女，湖南文理学院计算机与电气工程学院讲师，主要研究方向为移动开发、无线感知
於志文（1977- ），男，博士，西北工业大学教授，主要研究方向为智能感知、群智计算、人机计算
基金资助:
国家自然科学基金资助项目(61960206008);国家自然科学基金资助项目(62072375);湖南文理学院校级科研项目(E06020042)

An environment adaptive gesture recognition system based on visible light

Zhu WANG¹, Hualei ZHANG¹, Qianhong HU², Zhiwen YU¹

¹ School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China
² School of Computer Science and Electrical Engineering, Hunan University of Arts and Science, Changde 415006, China

Revised:2023-04-17 Online:2023-06-30 Published:2023-06-01
Supported by:
The National Natural Science Foundation of China(61960206008);The National Natural Science Foundation of China(62072375);The Research Project of Hunan University of Arts and Science(E06020042)

摘要/Abstract

摘要：

手势日益成为一种重要的人机交互方式，可在电子游戏、虚拟现实等场景中为用户提供更优质的体验。近年来，研究者探索利用不同感知技术实现手势识别，如射频信号、声学信号等。与之相比，利用可见光识别手势具有更强普适性。基本原理为：不同手势遮挡可见光会产生独特的阴影模式，通过光电传感器捕捉阴影变化即可实现手势识别。针对可见光手势识别面临的环境依赖难题，设计了一种基于光电传感器阵列的数字手势识别系统，提出了基于图像的阵列感知数据抽象表示模型，结合图像固有特性发掘不同传感器数据之间的时间和空间关联性，利用时空特征设计了基于CNN-RNN的环境自适应手势识别方法。为了验证所提方法的有效性，设计了环境自适应手势识别系统Vi-Gesture，准确率相比基线方法提升10%以上。

关键词: 可见光感知, 手势识别, 环境自适应, 时空特征, CNN-RNN

Abstract:

Gesture-based human-machine interaction is becoming more and more important, which can provide users with a better experience in scenarios such as video games and virtual reality.In recent years, researchers have explored different sensing technologies to facilitate gesture recognition, including RF signal, acoustic signal, etc.Compared with these approaches, visible light-based gesture recognition is a more pervasive option.The basic principle is that different gestures will produce unique shadow patterns as they block the visible light, and gesture recognition can be achieved by capturing shadow changes through photoelectric sensors.To address the environment-dependent problem faced by existing solutions, a digit gesture recognition system was designed based on the photoelectric sensor array.In particular, by modeling recordings of the sensor array as images, the temporal and spatial correlation between different sensor recordings was discovered.An environment adaptive gesture recognition method was designed based on CNN-RNN by fusing the spatio-temporal features.To verify the effectiveness of the proposed method, a prototype gesture recognition system was designed, named Vi-Gesture.Experimental results show that the proposed method outperforms baselines by more than 10% in recognition accuracy.

Key words: visible light sensing, gesture recognition, environment adaptive, spatio-temporal feature, CNN-RNN

中图分类号:

TP391

王柱, 张化磊, 胡千红, 於志文. 基于可见光的环境自适应手势识别系统[J]. 物联网学报, 2023, 7(2): 15-25.

Zhu WANG, Hualei ZHANG, Qianhong HU, Zhiwen YU. An environment adaptive gesture recognition system based on visible light[J]. Chinese Journal on Internet of Things, 2023, 7(2): 15-25.

图/表 11

图1

图2

图3

图4

图5

图6

图7

图8

图9

表1

表2

参考文献 30

[1]	SHEN S , GU K , CHEN X R ,et al. Gesture recognition through sEMG with wearable device based on deep learning[J]. Mobile Networksand Applications, 2020,25(6): 2447-2458.
[2]	CHAKRABORTY B K , SARMA D , BHUYAN M K ,et al. Review of constraints on vision-based gesture recognition for human-computer interaction[J]. IET Computer Vision, 2018,12(1): 3-15.
[3]	WANG Y W , SHEN J X , ZHENG Y Q . Push the limit of acoustic gesture recognition[C]// Proceedings of IEEE INFOCOM 2020 - IEEE Conference on Computer Communications. Piscataway:IEEE Press, 2020: 566-575.
[4]	GAO R Y , ZHANG M , ZHANG J ,et al. Towards position-independent sensing for gesture recognition with Wi-Fi[J]. Proceedings of the AC Mon Interactive,Mobile,Wearable and Ubiquitous Technologies, 2021,5(2): 1-28.
[5]	ZHANG C , TABOR J , ZHANG J L ,et al. Extending mobile interaction through near-field visible light sensing[C]// Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. New York:ACM, 2015: 345-357.
[6]	LIT X , ANC K , TIAN Z ,et al. Human sensing using visible light communication[C]// Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. New York:ACM, 2015: 331-344.
[7]	KAHOLOKULA D . Reusing ambient light to recognize hand gestures[EB]. 2016.
[8]	HU W J , GU H , PU Q F . Light sync:unsynchronized visual communication over screen-camera links[C]// Proceedings of the 19th Annual International Conference on Mobile Computing ＆ Networking. New York:ACM Press, 2013: 15-26.
[9]	KIMH S , KIMD R , YANG S H ,et al. An indoor visible light communication positioning system using a RF carrier allocation technique[J]. Journal of Light Wave Technology, 2013,31(1): 134-144.
[10]	ELAMASSIE M , KARBALAYGHAREH M , MIRAMIRKHANI F ,et al. Effect of fog and rain on the performance of vehicular visible light communications[C]// Proceedings of 2018 IEEE 87th Vehicular Technology Conference (VTC Spring). Piscataway:IEEE Press, 2018: 1-6.
[11]	SUN S Y , YANG F , SONG J . Sum rate maximization for intelligent reflecting surface-aided visible light communications[J]. IEEE Communications Letters, 2021,25(11): 3619-3623.
[12]	KUOY S , PANNUTO P , HSIAOK J ,et al. Luxapose:indoor positioning with mobile phones and visible light[C]// Proceedings of the 20th Annual International Conference on Mobile Computing and Networking. New York:ACM Press, 2014: 447-458.
[13]	LI L , HU P , PENG C ,et al. Epsilon:a visible light-based positioning system[C]// Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI’ 14),331-343, 2014.
[14]	ALAM F , CHEW M T , WENGE T ,et al. An accurate visible light positioning system using regenerated fingerprint database based on calibrated propagation model[J]. IEEE Transactions on Instrumentation and Measurement, 2019,68(8): 2714-2723.
[15]	MAJEED K , HRANILOVIC S . Passive indoor visible light positioning system using deep learning[J]. IEEE Internet of Things Journal, 2021,8(19): 14810-14821.
[16]	YANG Y B , HAO J , LUO J ,et al. Ceiling see:device-free occupancy inference through lighting infrastructure based LED sensing[C]// Proceedings of 2017 IEEE International Conference on Pervasive Computing and Communications (Per Com). Piscataway:IEEE Press, 2017: 247-256.
[17]	LI T X , LIU Q , ZHOU X . Practical human sensing in the light[C]// Proceedings of the 14th Annual International Conference on Mobile Systems,Applications,and Services. New York:ACM Press, 2016: 71-84.
[18]	LIT X , XIONG X , XIE Y F ,et al. Reconstructing hand poses using visible light[J]. Proceedings of the ACM on Interactive,Mobile,Wearable and Ubiquitous Technologies, 2017,1(3): 1-20.
[19]	VENKATNARAYANR H , SHAHZAD M . Gesture recognition using ambient light[J]. Proceedings of the ACM on Interactive,Mobile,Wearable and Ubiquitous Technologies, 2018,2(1): 1-28.
[20]	HU Q , YU Z , WANG Z ,et al. Vi hand:gesture recognition with ambient light[C]// Proceedings of the IEEE International Conference on Ubiquitous Intelligence and Computing,[S.l.:s.n.], 2019.
[21]	LAI K , YANUSHKEVICHS N . CNN RNN depth and skeleton based dynamic hand gesture recognition[C]// Proceedings of 2018 24th International Conference on Pattern Recognition (ICPR). Piscataway:IEEE Press, 2018: 3451-3456.
[22]	WEBBER J , MEHBODNIYA A , TENG R ,et al. Human-machine interaction using probabilistic neural network for light communication systems[J]. Electronics, 2022,11(6): 932.
[23]	LEYS C , LEY C , KLEIN C ,et al. Detecting outliers:do not use standard deviation around the mean,use absolute deviation around the Median[J]. Journal of Experimental Social Psychology, 2013,49(4): 764-766.
[24]	MA D , LAN G H , HU C S ,et al. Recognizing hand gestures using solar cells[J]. IEEE Transactions on Mobile Computing, 2022,PP(99): 1.
[25]	WEBBER J , MEHBODNIYA A , ARAFA A ,et al. Gesture recognition using machine learning for light communication systems[C]// Proceedings of 2022 International Mobile and Embedded Technology Conference (MECON). Piscataway:IEEE Press, 2022: 52-56.
[26]	ZHANG S , LIU K H , ZHANG Y L ,et al. Robust multi target device-free localization and tracking via visible light sensing[J]. IEEE Internet of Things Journal, 2022,9(17): 16446-16462.
[27]	ZHANG S , LIU K H , ZHANG Y L ,et al. A coarse fingerprint-assisted multiple target indoor device-free localization with visible light sensing[J]. IEEE Sensors Journal, 2022,22(2): 1461-1473.
[28]	YU L , ABUELLA H , ISLAM M Z ,et al. Gesture recognition using reflected visible and infrared light wave signals[J]. IEEE Transactions on Human-Machine Systems, 2021,51(1): 44-55.
[29]	MA D , LAN G H , HASSAN M ,et al. Solar gest:ubiquitous and battery-free gesture recognition using solar cells[C]// Proceedings of Mobi Com’ 19:The 25th Annual International Conference on Mobile Computing and Networking. New York:ACM Press, 2019: 1-15.
[30]	ZHANG D , PARK J W , ZHANG Y ,et al. Op to sense:towards ubiquitous self-powered ambient light sensing surfaces[J]. Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies,New York:ACM Press, 2020,4(3): 1-27.

训练样本规模	识别准确率
500	61.4%
1 000	67.6%
2 000	75.8%
4 000	82.6%
8 000	88.7%
10 000	91.7%
20 000	93.5%

阵列规模		平均准确率
阵列规模	Vi-Hand	LeNet-5	CNN-RNN
4×4	81.5%	85.7%	90.4%
5×5	82.4%	88.6%	92.1%
6×6	83.5%	89.5%	92.6%
7×7	84.2%	90.2%	93.4%
8×8	84.8%	92.0%	95.8%
9×9	85.5%	93.5%	96.5%
10×10	86.0%	93.6%	96.1%

基于可见光的环境自适应手势识别系统

An environment adaptive gesture recognition system based on visible light

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 30

相关文章 1

Metrics

推荐阅读 0