基于AR虚实图像注意力机制的电缆装配质量检测方法

doi:10.11959/j.issn.2096-3750.2021.00236

物联网学报 ›› 2021, Vol. 5 ›› Issue (3): 27-38.doi: 10.11959/j.issn.2096-3750.2021.00236

• 专题：工业互联网与智能制造 • 上一篇下一篇

基于AR虚实图像注意力机制的电缆装配质量检测方法

赵甘霖¹, 余畅¹, 张建富¹, 杨建新², 冯平法¹^,³, 沈群⁴

¹ 清华大学机械工程系，北京100084
² 清华大学基础工业训练中心，北京 100084
³ 清华大学深圳国际研究生院先进制造学部，广东深圳 518055；4 航天东方红卫星有限公司，北京 100094

修回日期:2021-04-20 出版日期:2021-09-30 发布日期:2021-09-01
作者简介:赵甘霖（1995- ），男，清华大学机械工程系博士生，主要研究方向为机器视觉、增强现实辅助装配技术等
余畅（1998- ），男，清华大学机械工程系博士生，主要研究方向为机器视觉、增强现实、智能制造等
张建富（1975- ），男，博士，清华大学机械工程系副教授、特别研究员，主要研究方向为精密加工技术、超声加工技术及系统、先进制造装备、智能制造系统等
杨建新（1977- ），男，博士，清华大学基础工业训练中心副教授，主要研究方向为增强现实、智能制造、精密测量等
冯平法（1966- ），男，博士，清华大学机械工程系教授，主要研究方向为智能制造与精密加工等
沈群（1989- ），男，航天东方红卫星有限公司工程师，主要研究方向为卫星制造智能装备等
基金资助:
清华-东方红合作项目：基于AR的电缆网总装铺设引导系统软件开发

Inspection method for cable assembly quality based on AR virtual-real image attention mechanism

Ganlin ZHAO¹, Chang YU¹, Jianfu ZHANG¹, Jianxin YANG², Pingfa FENG¹^,³, Qun SHEN⁴

¹ Department of Mechanical Engineering, Tsinghua University, Beijing 100084, China
² Fundamental Industry Training Center, Tsinghua University, Beijing 100084, China
³ Division of Advanced Manufacturing, Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China;4.DFH Satellite Co., Ltd., Beijing 100094, China

Revised:2021-04-20 Online:2021-09-30 Published:2021-09-01
Supported by:
Tsinghua-DFH Cooperation Project: AR-based software development for cable assembly guidance system

摘要/Abstract

摘要：

航天器中电缆装配在总装环节占据大量比重，电缆装配质量直接影响着产品整机的工作性能。现有的增强现实辅助电缆装配系统缺乏对装配结果实时判断和反馈的能力，无法识别当前装配的电缆是否符合装配质量要求。为了解决上述问题，提出了一种基于AR虚实图像注意力机制的电缆装配质量检测方法，通过构建虚拟辅助模型，在拍摄的装配结果图像中定位检测关键区域，过滤冗余图像信息；进而基于 YOLOv4-Tiny 图像检测模型对电缆装配关键节点位置进行了判断，基于计算电缆邻域平均重合度的方法求得了敷设路径的重合度，基于相机逆投影的方法得到了电缆的弯曲半径。最后，利用HoloLens2开发了基于增强现实技术的电缆装配质量检测系统，并对设计的4个电缆装配案例进行了质量检测，实验验证了本文方法及系统的可行性和有效性。

关键词: 电缆装配, 质量检测, 增强现实辅助装配, AR虚实图像, 注意力机制

Abstract:

Cable assembly in spacecraft occupies a large proportion in the final assembly process, and the quality of cable assembly directly affects the working performance of the whole product.The existing augmented reality-assisted cable assembly system lacks the ability to make real-time judgment and feedback on the assembly results, and cannot identify whether the currently assembled cable meets the quality requirements.In order to solve the above problems, a cable assembly quality inspection method based on AR virtual-real image attention mechanism was proposed.Critical areas in the captured images were located and detected by constructing virtual assist models to filter redundant image information.Based on YOLOv4-Tiny image detection model, the key node location of assembled cable was judged.The overlap of the laying path was obtained based on the method of calculating the average overlap of the neighborhood, and the bending radius of the cable was derived based on the method of camera inverse projection.Finally, an augmented reality-based cable assembly quality inspection system was developed using HoloLens2.The quality inspection was performed on the four designed cable assembly cases, and the feasibility and effectiveness of the method and system were experimentally verified.

Key words: cable assembly, quality inspection, augmented reality assisted assembly, AR virtual-real image, attention mechanism

中图分类号:

TP391.9
V465

赵甘霖, 余畅, 张建富, 杨建新, 冯平法, 沈群. 基于AR虚实图像注意力机制的电缆装配质量检测方法[J]. 物联网学报, 2021, 5(3): 27-38.

Ganlin ZHAO, Chang YU, Jianfu ZHANG, Jianxin YANG, Pingfa FENG, Qun SHEN. Inspection method for cable assembly quality based on AR virtual-real image attention mechanism[J]. Chinese Journal on Internet of Things, 2021, 5(3): 27-38.

图/表 17

表1

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

表2

图11

图12

表3

表4

表5

参考文献 20

[1]	GOODDING J C , ARDELEAN E V , BABUSKA V ,et al. Experimental techniques and structural parameter estimation studies of spacecraft cables[J]. Journal of Spacecraft and Rockets, 2011,48(6): 942-957.
[2]	ARDELEAN E V , BABU?KA V , GOODDING J C ,et al. Cable effects study:tangents,rabbit holes,dead ends,and valuable results[J]. Journal of Spacecraft and Rockets, 2014,52(2): 569-583.
[3]	WANG X , ONG S K , NEE A Y C . A comprehensive survey of augmented reality assembly research[J]. Advances in Manufacturing, 2016,4(1): 1-22.
[4]	CAUDELL T P , MIZELL D W . Augmented reality:an application of heads-up display technology to manual manufacturing processes[C]// Proceedings of the 25th Hawaii International Conference on System Sciences.[S.l.:s.n.], 1992,2: 659-669.
[5]	MIZELL D . Boeing's wire bundle assembly project[M]. Fundamentals of Wearable Computers and Augmented Reality. Boca Raton: CRC Press, 2001: 462-482.
[6]	SERVáN J , MAS F , MENéNDEZ J L ,et al. Using augmented reality in AIRBUS A400M shop floor assembly work instructions[C]// AIP Conference Proceedings. Melville:American Institute of Physics,2012, 1431（1）： 633-640.
[7]	ZHENG L Y , LIU X , AN Z W ,et al. A smart assistance system for cable assembly by combining wearable augmented reality with portable visual inspection[J]. Virtual Reality ＆ Intelligent Hardware, 2020,2(1): 12-27.
[8]	ZHANG W , CHEN C Y , CHEN H J ,et al. Application of augmented reality in satellite cable network assembly[C]// IOP Conference Series:Materials Science and Engineering. Bristol:IOP Publishing, 2019,54(1): 012034.
[9]	陈华俊, 陈畅宇, 孙刚 ,等. 增强现实在航天器电缆装配中的应用技术研究[J]. 科技与创新, 2019(8): 40-44.
	CHEN H J , CHEN C Y , SUN G ,et al. Research on the application technology of augmented reality in spacecraft cable assembly[J]. Science and Technology ＆ Innovation, 2019(8): 40-44.
[10]	PARMAR P , . Use of computer vision to detect tangles in tangled objects[C]// Proceedings of 2013 IEEE Second International Conference on Image Information Processing (ICIIP-2013). Piscataway:IEEE Press, 2013: 39-44.
[11]	SUN F J , WANG H , FAN J Q . 2D otsu segmentation algorithm based on simulated annealing genetic algorithm for iced-cable images[C]// Proceedings of 2009 International Forum on Information Technology and Applications. Piscataway:IEEE Press, 2009: 600-602.
[12]	温沛涵, 吴梦杰, 易树平 ,等. 基于目标识别的软管装配视觉检测系统[J]. 计算机集成制造系统, 2015,21(6): 1486-1496.
	WEN P H , WU M J , YI S P ,et al. Visual inspection system for hose assembly based on object recognition[J]. Computer Integrated Manufacturing Systems, 2015,21(6): 1486-1496.
[13]	焦晨航 . 约束空间中电缆线安装方法研究[D]. 哈尔滨:哈尔滨工业大学, 2019.
	JIAO C H . Research on cable installation method in constrained space[D]. Harbin:Harbin Institute of Technology, 2019.
[14]	刘睿, 尹旭悦, 范秀敏 ,等. 面向 AR 辅助引导的电缆敷设质量一致性视觉检测方法[J]. 计算机集成制造系统, 2021,27(1): 53-62.
	LIU R , YIN X Y , FAN X M ,et al. AR guidance-based visual inspection method for cable laying quality consistency check[J]. Computer Integrated Manufacturing Systems, 2021,27(1): 53-62.
[15]	魏煌涛 . 基于视觉的电缆安装方法研究[D]. 哈尔滨:哈尔滨工业大学, 2020.
	WEI H T . Research on installation method of cable based on vision[D]. Harbin:Harbin Institute of Technology, 2020.
[16]	冒文彦 . 基于卷积神经网络的高铁电缆线夹装配质量检测技术研究[D]. 镇江:江苏大学, 2020.
	MAO W Y . Research on assembly quality inspection technology of high-speed rail cable clamps based on convolutional neural network[D]. Zhenjiang,China:Jiangsu University, 2020.
[17]	LI B , CHEN C . Transmission line detection based on a hierarchical and contextual model for aerial images[J]. Journal of Electronic Imaging, 2018,27(4): 1:043054.
[18]	LI S F , ZHENG P , ZHENG L Y . An AR-assisted deep learning-based approach for automatic inspection of aviation connectors[J]. IEEE Transactions on Industrial Informatics, 2021,17(3): 1721-1731.
[19]	BOCHKOVSKIY A , WANG C Y , LIAO H Y MARK . YOLOv4:optimal speed and accuracy of object detection[J]. arXiv:2004.10934, 2020.
[20]	EVERINGHAM M , WINN J . The PASCAL visual object classes challenge 2007 (voc2007) development kit[J]. International Journal of Computer Vision, 2006,111(1): 98-136.

标准类型	评价内容	量化指标
装配位置标准	电缆关键节点装配位置与设计的误差	≤25 mm
装配几何标准	电缆敷设路径与设计路径重合度	≥85%
	电缆最小弯曲半径（电缆外径为d）	≥10 d

编号					检测结果
	节点	E1.1	M1.2	M1.3	M1.4	M1.5	E1.6
电缆1	检测物体	connector	clip	clip	clip	clip	connector
	置信度	1.00	0.68	1.00	0.81	0.98	1.00
	节点	M2.1	M2.2	E2.3
电缆2	检测物体	未检出	clip	未检出
	置信度	无	1.00	无
	节点	M3.1	M3.2	M3.3	M3.4	M3.5	M3.6	E3.7
电缆3	检测物体	clip	clip	clip	未检出	clip	clip	connector
	置信度	0.99	1.00	0.98	无	0.94	1.00	1.00
	节点	E4.1	M4.2	M4.3	M4.4
电缆4	检测物体	connector	clip	clip	clip
	置信度	1.00	0.96	1.00	0.91

电缆编号	扩倍模型像素/个	重叠区域像素/个	邻域平均重合度	实际重合度
电缆1	162 259	47 111	87.10%	91.14%
电缆2	100 861	20 045	59.62%	54.43%
电缆3	168 389	27 395	48.81%	44.76%
电缆4	147 183	45 735	93.22%	86.74%

区域编号	电缆1-1	电缆1-2	电缆2-1	电缆3-1	电缆4-1	电缆4-2
区域图像
检测半径/mm	104.79	88.32	124.30	40.65	139.11	94.55
实际半径/mm	106.65	90.53	122.17	42.64	133.99	93.06
检测误差率	1.74%	2.44%	1.74%	4.66%	3.82%	1.60%

电缆编号	关键节点检测	敷设路径检测	弯曲半径检测	质量检测结果
电缆1	合格	合格	合格	合格
电缆2	不合格	不合格	合格	不合格
电缆3	不合格	不合格	不合格	不合格
电缆4	合格	合格	合格	合格

基于AR虚实图像注意力机制的电缆装配质量检测方法

Inspection method for cable assembly quality based on AR virtual-real image attention mechanism

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 20

相关文章 2

Metrics

推荐阅读 0

[1]	张志飞, 刘峰, 葛祎阳, 李烁, 张煜, 熊轲. 一种基于深度可分离卷积和注意力机制的入侵检测方法[J]. 物联网学报, 2023, 7(1): 49-59.
[2]	金驰,李志军,孙大洋,胡封晔. 基于空间特征的无线体域网人体姿态识别算法[J]. 物联网学报, 2019, 3(3): 70-75.