面向机器人操作任务的视触觉传感技术综述

doi:10.11959/j.issn.2096-6652.202222

Abstract

Abstract:

Thanks to the high spatial resolution and multi-mode tactile sensing, visuotactile sensing technology has been widely applied to various robotic manipulation tasks, such as robotic active perception, pose estimation, and in-hand manipulation.Firstly, the current mainstream visuotactile sensing technologies based on sensing principles were summarized, which could be mainly divided into three categories: GelSight-type visuotactile sensors, binocular (multi-view) visuotactile sensors, and other types.Meanwhile, the sensing methods of different tactile sensing modes were further summarized, including contact surface 3D geometry, force/torque, and sliding.Furthermore, focusing on the field of robot operation, the specific application scenarios of visuotactile sensors were discussed.Finally, future work of the visuotactile sensing technology and how they can be further applied to robotic dexterous manipulation tasks were given.

Key words: visuotactile sensor, robotic tactile perception, robotic dexterous manipulation

CLC Number:

TP242.6

Shaowei CUI, Shuo WANG, Jingyi HU, et al. A survey of visuotactile sensing technologies for robotic manipulation[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 186-199.

Figures/Tables 16

References 64

[6]	LI Q , KROEMER O , SU Z ,et al. A review of tactile information:perception and action through touch[J]. IEEE Transactions on Robotics, 2020,36(6): 1619-1634.
[7]	CHI C , SUN X G , XUE N ,et al. Recent progress in technologies for tactile sensors[J]. Sensors (Basel,Switzerland), 2018,18(4): 948.
[8]	ABAD A C , RANASINGHE A . Visuotactile sensors with emphasis on GelSight sensor:a review[J]. IEEE Sensors Journal, 2020,20(14): 7628-7638.
[9]	YUAN W Z , DONG S Y , ADELSON E H . GelSight:high-resolution robot tactile sensors for estimating geometry and force[J]. Sensors (Basel,Switzerland), 2017,17(12): 2762.
[10]	DONG S Y , YUAN W Z , ADELSON E H . Improved GelSight tactile sensor for measuring geometry and slip[C]// Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2017: 137-144.
[11]	DONLON E , DONG S Y , LIU M ,et al. GelSlim:a high-resolution,compact,robust,and calibrated tactile-sensing finger[C]// Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2018: 1927-1934.
[12]	TAYLOR I , DONG S Y , RODRIGUEZ A . GelSlim3.0:high-resolution measurement of shape,force and slip in a compact tactile-sensing finger[J]. arXiv preprint,2021,arXiv:2103.12269.
[13]	WARD-CHERRIER B , PESTELL N , CRAMPHORN L ,et al. The TacTip family:soft optical tactile sensors with 3D-printed biomimetic morphologies[J]. Soft Robotics, 2018,5(2): 216-227.
[1]	MYERS D . Sensation and perception.Psychology[M]. New York: Worth, 2010.
[2]	ANSHEL J R . Visual ergonomics in the workplace[J]. AAOHN Journal, 2007,55(10): 414-420.
[14]	ALSPACH A , HASHIMOTO K , KUPPUSWAMY N ,et al. Soft-bubble:a highly compliant dense geometry tactile sensor for robot manipulation[C]// Proceedings of 2019 2nd IEEE International Conference on Soft Robotics. Piscataway:IEEE Press, 2019: 597-604.
[15]	YAMAGUCHI A , ATKESON C G . Combining finger vision and optical tactile sensing:reducing and handling errors while cutting vegetables[C]// Proceedings of 2016 IEEE-RAS 16th International Conference on Humanoid Robots. Piscataway:IEEE Press, 2016: 1045-1051.
[3]	RANASINGHE A , SORNKARN N , DASGUPTA P ,et al. Salient feature of haptic-based guidance of people in low visibility environments using hard reins[J]. IEEE Transactions on Cybernetics, 2016,46(2): 568-579.
[4]	AKHTAR N , MIAN A . Threat of adversarial attacks on deep learning in computer vision:a survey[J]. IEEE Access, 2018,6: 14410-14430.
[16]	YAMAGUCHI A , ATKESON C G . Implementing tactile behaviors using FingerVision[C]// Proceedings of 2017 IEEE-RAS 17th International Conference on Humanoid Robotics. Piscataway:IEEE Press, 2017: 241-248.
[17]	LAMBETA M , CHOU P W , TIAN S ,et al. DIGIT:a novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation[J]. IEEE Robotics and Automation Letters, 2020,5(3): 3838-3845.
[18]	WANG S X , SHE Y , ROMERO B ,et al. GelSight Wedge:measuring high-resolution 3D contact geometry with a compact robot finger[C]// Proceedings of 2021 IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 2021: 6468-6475.
[19]	DO W K , KENNEDY III M . DenseTact:optical tactile sensor for dense shape reconstruction[J]. arXiv preprint,2022,arXiv:2201.01367.
[20]	GOMES D F , LIN Z L , LUO S . GelTip:a finger-shaped optical tactile sensor for robotic manipulation[C]// Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2021: 9903-9909.
[21]	SUN H , KUCHENBECKER K J , MARTIUS G . A soft thumb-sized vision-based sensor with accurate all-round force perception[J]. Nature Machine Intelligence, 2022,4(2): 135-145.
[22]	KUPPUSWAMY N , ALSPACH A , UTTAMCHANDANI A ,et al. Soft-bubble grippers for robust and perceptive manipulation[C]// Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2020: 9917-9924.
[23]	PADMANABHA A , EBERT F , TIAN S ,et al. OmniTact:a multi-directional high-resolution touch sensor[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 2020: 618-624.
[24]	ZHANG T , CONG Y , LI X M ,et al. Robot tactile sensing:vision based tactile sensor for force perception[C]// Proceedings of 2018 IEEE 8th Annual International Conference on CYBER Technology in Automation,Control,and Intelligent Systems. Piscataway:IEEE Press, 2018: 1360-1365.
[25]	CUI S W , WANG R , HU J Y ,et al. In-hand object localization using a novel high-resolution visuotactile sensor[J]. IEEE Transactions on Industrial Electronics, 2022,69(6): 6015-6025.
[26]	KAKANI V , CUI X N , MA M J ,et al. Vision-based tactile sensor mechanism for the estimation of contact position and force distribution using deep learning[J]. Sensors (Basel,Switzerland), 2021,21(5): 1920.
[27]	ZHANG L W , WANG Y , JIANG Y . Tac3D:a novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution[J]. arXiv preprint,2022,arXiv:2202.06211.
[28]	LEPORA N F . Soft biomimetic optical tactile sensing with the TacTip:a review[J]. IEEE Sensors Journal, 2021,21(19): 21131-21143.
[29]	VLACK K , MIZOTA T , KAWAKAMI N ,et al. GelForce:a vision-based traction field computer interface[C]// Proceedings of CHI’05 Extended Abstracts on Human Factors in Computing Systems. New York:ACM Press, 2005: 1154-1155.
[30]	SATO K , KAMIYAMA K , KAWAKAMI N ,et al. Finger-shaped GelForce:sensor for measuring surface traction fields for robotic hand[J]. IEEE Transactions on Haptics, 2010,3(1): 37-47.
[31]	SFERRAZZA C , D'ANDREA R , . Design,motivation and evaluation of a full-resolution optical tactile sensor[J]. Sensors (Basel,Switzerland), 2019,19(4): 928.
[32]	SFERRAZZA C , WAHLSTEN A , TRUEEB C ,et al. Ground truth force distribution for learning-based tactile sensing:a finite element approach[J]. IEEE Access, 2019,7: 173438-173449.
[33]	SFERRAZZA C , D’ANDREA R , . Sim-to-real for high-resolution optical tactile sensing:from images to 3D contact force distributions[J]. arXiv preprint,2020,arXiv:2012.11295.
[34]	ZHANG Y Z , YUAN W H , KAN Z C ,et al. Towards learning to detect and predict contact events on vision-based tactile sensors[J]. arXiv preprint,2019,arXiv:1910.03973.
[35]	DU Y P , ZHANG G L , ZHANG Y Z ,et al. High-resolution 3-dimensional contact deformation tracking for FingerVision sensor with dense random color pattern[J]. IEEE Robotics and Automation Letters, 2021,6(2): 2147-2154.
[36]	ZHANG G L , DU Y P , YU H Y ,et al. DelTact:a vision-based tactile sensor using dense color pattern[J]. arXiv preprint,2022,arXiv:2202.02179.
[37]	JOHNSON M K , ADELSON E H . Retrographic sensing for the measurement of surface texture and shape[C]// Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2009: 1070-1077.
[38]	ROMERO B , VEIGA F , ADELSON E . Soft,round,high resolution tactile fingertip sensors for dexterous robotic manipulation[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 2020: 4796-4802.
[39]	CUI S W , WANG R , HU J Y ,et al. Self-supervised contact geometry learning by GelStereo visuotactile sensing[J]. IEEE Transactions on Instrumentation and Measurement, 2022,71: 1-9.
[40]	KAMIYAMA K , KAJIMOTO H , KAWAKAMI N ,et al. Evaluation of a vision-based tactile sensor[C]// Proceedings of IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 2004: 1542-1547.
[41]	MA D L , DONLON E , DONG S Y ,et al. Dense tactile force estimation using GelSlim and inverse FEM[C]// Proceedings of 2019 International Conference on Robotics and Automation. Piscataway:IEEE Press, 2019: 5418-5424.
[42]	ZHANG Y Z , KAN Z C , YANG Y ,et al. Effective estimation of contact force and torque for vision-based tactile sensors with Helmholtz-Hodge decomposition[J]. IEEE Robotics and Automation Letters, 2019,4(4): 4094-4101.
[43]	DONG S Y , MA D L , DONLON E ,et al. Maintaining grasps within slipping bounds by monitoring incipient slip[C]// Proceedings of 2019 International Conference on Robotics and Automation. Piscataway:IEEE Press, 2019: 3818-3824.
[44]	SUI R M , ZHANG L W , LI T M ,et al. Incipient slip detection method with vision-based tactile sensor based on distribution force and deformation[J]. IEEE Sensors Journal, 2021,21(22): 25973-25985.
[45]	LI J H , DONG S Y , ADELSON E . Slip detection with combined tactile and visual information[C]// Proceedings of 2018 IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 2018: 7772-7777.
[46]	ZHANG Y Z , KAN Z C , TSE Y A ,et al. Fingervision tactile sensor design and slip detection using convolutional LSTM network[J]. arXiv preprint,2018,arXiv:1810.02653.
[47]	JAMES J W , LEPORA N F . Slip detection for grasp stabilization with a multifingered tactile robot hand[J]. IEEE Transactions on Robotics, 2021,37(2): 506-519.
[48]	ABAD A C , REID D , RANASINGHE A . HaptiTemp:a next-generation thermosensitive GelSight-like visuotactile sensor[J]. IEEE Sensors Journal, 2022,22(3): 2722-2734.
[49]	YUAN W Z , SRINIVASAN M A , ADELSON E H . Estimating object hardness with a GelSight touch sensor[C]// Proceedings of 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2016: 208-215.
[50]	CALANDRA R , OWENS A , UPADHYAYA M ,et al. The feeling of success:does touch sensing help predict grasp outcomes?[J]. arXiv preprint,2017,arXiv:1710.05512.
[51]	CALANDRA R , OWENS A , JAYARAMAN D ,et al. More than a feeling:learning to grasp and regrasp using vision and touch[J]. IEEE Robotics and Automation Letters, 2018,3(4): 3300-3307.
[52]	HOGAN F R , BAUZA M , CANAL O ,et al. Tactile regrasp:grasp adjustments via simulated tactile transformations[C]// Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2018: 2963-2970.
[53]	DONG S Y , JHA D K , ROMERES D ,et al. Tactile-RL for insertion:generalization to objects of unknown geometry[C]// Proceedings of 2021 IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 2021: 6437-6443.
[54]	WANG C , WANG S X , ROMERO B ,et al. SwingBot:learning physical features from in-hand tactile exploration for dynamic swing-up manipulation[C]// Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2021: 5633-5640.
[55]	ANZAI T , TAKAHASHI K . Deep gated multi-modal learning:in-hand object pose changes estimation using tactile and image data[C]// Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2021: 9361-9368.
[56]	LI R , PLATT R , YUAN W Z ,et al. Localization and manipulation of small parts using GelSight tactile sensing[C]// Proceedings of 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway:IEEE Press, 2014: 3988-3993.
[57]	BAUZA M , VALLS E , LIM B ,et al. Tactile object pose estimation from the first touch with geometric contact rendering[J]. arXiv preprint,2020,arXiv:2012.05205.
[58]	BAUZA M , CANAL O , RODRIGUEZ A . Tactile mapping and localization from high-resolution tactile imprints[C]// Proceedings of 2019 International Conference on Robotics and Automation. Piscataway:IEEE Press, 2019: 3811-3817.
[59]	LEPORA N F , LLOYD J . Optimal deep learning for robot touch:training accurate pose models of 3D surfaces and edges[J]. IEEE Robotics ＆ Automation Magazine, 2020,27(2): 66-77.
[60]	SHE Y , WANG S X , DONG S Y ,et al. Cable manipulation with a tactile-reactive gripper[J]. The International Journal of Robotics Research, 2021,40(12/13/14): 1385-1401.
[61]	KIM S , RODRIGUEZ A . Active extrinsic contact sensing:application to general peg-in-hole insertion[J]. arXiv preprint,2021,arXiv:2110.03555.
[62]	LAMBETA M , XU H Z , XU J W ,et al. PyTouch:a machine learning library for touch processing[C]// Proceedings of 2021 IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 2021: 13208-13214.
[63]	SORENSEN S , KOLAGUNDA A , SAPONARO P ,et al. Refractive stereo ray tracing for reconstructing underwater structures[C]// Proceedings of 2015 IEEE International Conference on Image Processing. Piscataway:IEEE Press, 2015: 1712-1716.
[5]	BILLARD A , KRAGIC D . Trends and challenges in robot manipulation[J]. Science, 2019,364(6446): eaat8414.
[64]	FISHEL J A , LOEB G E . Sensing tactile microvibrations with the BioTac—comparison with human sensitivity[C]// Proceedings of 2012 4th IEEE RAS ＆ EMBS International Conference on Biomedical Robotics and Biomechatronics. Piscataway:IEEE Press, 2012: 1122-1127.

Metrics

Recommended 0

No Suggested Reading articles found!

传感器	结构形态	接触胶体层	光源	相机系统	几何重建	力估计	滑动感知
GelSight^[9]	指尖	阵列标记点	RGB	单目	光度立体	神经网络	接触模型
GelSlim^[11]	指尖	阵列标记点	RGB	单目	光度立体	有限元	接触模型
DIGIT^[17]	指尖	无标记点	RGB	单目	待开发	无	无
GelTip^[20]	手指	无标记点	RGB	单目	光度立体	无	无
InSight^[21]	手指	无标记点	RGB	单目	光度立体	神经网络	无
DenseTact^[19]	指尖	无标记点	RGB	单目	神经网络	无	无
Soft-bubble^[14]	手掌	稠密标记点	单色	单目+深度	深度相机	无	无
GelStereo^[25]	指尖	阵列标记点	单色	双目	双目立体	神经网络	神经网络
OmniTact^[23]	指尖	无标记点	RGB	多目	光度立体	无	无
Tac3D^[27]	手掌	阵列标记点	单色	虚拟双目	双目立体	有限元	接触模型
TacTip^[13]	指尖	阵列标记探针	单色	单目	无	无	神经网络
GelForce^[29]	指尖	双层标记点	单色	单目	无	有限元	无
Sf.＆ D’ A.^[31]	手掌	稠密标记点	单色	单目	无	有限元	无
FingerVision^[16]	指尖	阵列标记点	无	单目	无	无	接触模型
DelTact^[36]	指尖	稠密像素图层	单色	单目	光流估计	有限元	神经网络

A survey of visuotactile sensing technologies for robotic manipulation

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 64

Related Articles 15

Metrics

Recommended 0

[1]	Xiang CAO, Changyin SUN. UUV mission re-planning based on threat assessment of uncertain events [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 493-502.
[2]	Jiaxin ZHANG, Senlin ZHANG, Meiqin LIU, Shanling DONG, Ronghao ZHENG. Multi-AUV cooperative localization in adaptive sampling for marine environmental monitoring [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 503-512.
[3]	Yinghao CAI, Hua YANG, Xuan AN, Wenshuo WANG, Yidong DU, Jiatao ZHANG, Zhigang WANG. Study on NeuroSymbolic learning and its applications [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 560-570.
[4]	Chao ZHAO, Jie XU, Xingyu CHEN, Kuizhi MEI, Xuguang LAN. A review of continual learning for robotics [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(3): 308-323.
[5]	De XU, Fangbo QIN. Research development on automated robotic peg-in-hole assembly [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 200-211.
[6]	Yueguang GE, Shaolin ZHANG, Yinghao CAI, Tao LU, Dayong WEN, Haitao WANG, Shuo WANG. A survey on applications of ontology knowledge representation in robotics [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 212-222.
[7]	Jingyi HU, Shaowei CUI, Chaofan ZHANG, Boyue ZHANG, Shuo WANG. 3D edge reconstruction method based on tactile sensing and servo [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 233-245.
[8]	Lu DONG, Ailing XIONG. Path planning for unmanned surface vehicle in complex dynamic environment based on improved RRT*-Smart [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 264-276.
[9]	Teng WANG, Jing PAN, Lu DONG, Changyin SUN. Key technologies and applications of intelligent guiding robots for epidemic prevention [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(2): 187-194.
[10]	Rizhong WANG, Huiping LI, Di CUI, Demin XU. Depth control of autonomous underwater vehicle using deep reinforcement learning [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 354-360.
[11]	Huiqiao FU, Kaiqiang TANG, Guizhou DENG, Xinpeng WANG, Chunlin CHEN. Motion planning for hexapod robot using deep reinforcement learning [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 361-371.
[12]	Dawei WANG,Zhuo WANG,Peng WANG,Zhigang WANG,Haitao WANG. Cloud native robot system based on edge computing [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(3): 275-283.
[13]	Haijun FU,Shichao CHEN,Yilun LIN,Gang XIONG,Bin HU. Research and validation of human-in-the-loop hybrid-augmented intelligence in Sawyer [J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(3): 280-286.
[14]	Xiang ZHANG,Hongwei LIU,Zhuoqun LIU,Zhenguo YAN,Xiaoqian CHEN,Yiyong HUANG. Dynamical modeling and intelligent control of space soft manipulator [J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(1): 52-61.
[15]	Yajun ZHENG,Lei XUE,Lu DONG,Qingling WANG. Formation control of mobile robots with UWB localization technology [J]. Chinese Journal of Intelligent Science and Technology, 2019, 1(1): 83-87.