融合多尺度深度卷积的轻量级Transformer交通场景语义分割算法

doi:10.11959/j.issn.1000-436x.2023194

通信学报 ›› 2023, Vol. 44 ›› Issue (10): 213-225.doi: 10.11959/j.issn.1000-436x.2023194

• 学术通信 • 上一篇

融合多尺度深度卷积的轻量级Transformer交通场景语义分割算法

谢刚¹^,², 王荃毅¹^,², 谢新林¹^,², 王健安¹^,²

¹ 太原科技大学电子信息工程学院，山西太原 030024
² 先进控制与装备智能化山西省重点实验室，山西太原 030024

修回日期:2023-09-20 出版日期:2023-10-01 发布日期:2023-10-01
作者简介:谢刚（1972− ），男，山西五台人，博士，太原科技大学教授、博士生导师，主要研究方向为计算机视觉、智能控制等
王荃毅（1999− ），男，山西长治人，太原科技大学硕士生，主要研究方向为语义分割、深度学习等
谢新林（1990− ），男，山西运城人，博士，太原科技大学副教授、硕士生导师，主要研究方向为语义分割、深度学习等
王健安（1984− ），男，江西九江人，博士，太原科技大学教授、硕士生导师，主要研究方向为智能信息系统、复杂网络控制等
基金资助:
国家自然科学基金资助项目(62006169);山西省重点研发计划基金资助项目(202202010101005);太原科技大学博士科研启动基金资助项目(20192047)

Lightweight Transformer traffic scene semantic segmentation algorithm integrating multi-scale depth convolution

Gang XIE¹^,², Quanyi WANG¹^,², Xinlin XIE¹^,², Jian’an WANG¹^,²

¹ School of Electronic and Information Engineering, Taiyuan University of Science and Technology, Taiyuan 030024, China
² Shanxi Key Laboratory of Advanced Control and Equipment Intelligence, Taiyuan 030024，China

Revised:2023-09-20 Online:2023-10-01 Published:2023-10-01
Supported by:
The National Natural Science Foundation of China(62006169);Key Research and Development Plan of Shanxi Province(202202010101005);Taiyuan University of Science and Technology Scientific Research Initial Funding(20192047)

摘要/Abstract

摘要：

针对交通场景语义分割算法中存在的易融入周围背景的纤细条状目标分割不连续、模型参数量大等问题，提出一种融合多尺度深度卷积的轻量级 Transformer 交通场景语义分割算法。首先，基于深度卷积构建多尺度条形特征提取模块，在不同尺度下增强对纤细条状目标特征的表示能力。其次，在浅层网络中利用卷积归纳偏置特性设计空间细节辅助模块，以弥补深层空间细节信息的丢失来优化目标边缘分割。最后，提出基于Transformer-CNN框架的非对称编解码网络，编码器结合Transformer与CNN减少细节信息丢失并降低模型参数量；而解码器采用轻量级的多级特征融合设计来进一步建模全局上下文。所提算法在Cityscapes和CamVid交通场景公开数据集上分别取得的平均交并比为 78.63%和 81.06%，能够在交通场景语义分割中实现分割精度和模型大小之间的权衡，具备良好的应用前景。

关键词: 语义分割, 深度学习, 注意力机制, 轻量级, 交通场景

Abstract:

Aiming at the problems of discontinuous segmentation of thin strip objects that were easy to blend into the surrounding background and a large number of model parameters in the semantic segmentation algorithm of traffic scenes, a lightweight Transformer traffic scene semantic segmentation algorithm integrating multi-scale depth convolution was proposed.First, a multi-scale strip feature extraction module (MSEM) was constructed based on deep convolution to enhance the representation ability of thin strip target features at different scales.Secondly, a spatial detail auxiliary module (SDAM) was designed using the convolutional inductive bias feature in the shallow network to compensate for the loss of deep spatial detail information to optimize object edge segmentation.Finally, an asymmetric encoding-decoding network based on the Transformer-CNN framework (TC-AEDNet) was proposed.The encoder combined Transformer and CNN to alleviate the loss of detail information and reduce the amount of model parameters; while the decoder adopted a lightweight multi-level feature fusion design to further model the global context.The proposed algorithm achieves the mean intersection over union (mIoU) of 78.63% and 81.06% respectively on the Cityscapes and CamVid traffic scene public datasets.It can achieve a trade-off between segmentation accuracy and model size in traffic scene semantic segmentation and has a good application prospect.

Key words: semantic segmentation, deep learning, attention mechanism, lightweight, traffic scene

中图分类号:

TP391.4

谢刚, 王荃毅, 谢新林, 王健安. 融合多尺度深度卷积的轻量级Transformer交通场景语义分割算法[J]. 通信学报, 2023, 44(10): 213-225.

Gang XIE, Quanyi WANG, Xinlin XIE, Jian’an WANG. Lightweight Transformer traffic scene semantic segmentation algorithm integrating multi-scale depth convolution[J]. Journal on Communications, 2023, 44(10): 213-225.

图/表 14

图1

表1

图2

图3

图4

表2

表3

表4

表5

表6

表7

图5

表8

图6

参考文献 27

[1]	周鑫, 何晓新, 郑昌文 . 基于图像深度学习的无线电信号识别[J]. 通信学报, 2019,40(7): 114-125.
	ZHOU X , HE X X , ZHENG C W . Radio signal recognition based on image deep learning[J]. Journal on Communications, 2019,40(7): 114-125.
[2]	LV Q X , SUN X , CHEN C R ,et al. Parallel complement network for real-time semantic segmentation of road scenes[J]. IEEE Transactions on Intelligent Transportation Systems, 2022,23(5): 4432-4444.
[3]	李琳辉, 钱波, 连静 ,等. 基于卷积神经网络的交通场景语义分割方法研究[J]. 通信学报, 2018,39(4): 123-130.
	LI L H , QIAN B , LIAN J ,et al. Study on traffic scene semantic segmentation method based on convolutional neural network[J]. Journal on Communications, 2018,39(4): 123-130.
[4]	杨军, 党吉圣 . 基于上下文注意力 CNN 的三维点云语义分割[J]. 通信学报, 2020,41(7): 195-203.
	YANG J , DANG J S . Semantic segmentation of 3D point cloud based on contextual attention CNN[J]. Journal on Communications, 2020,41(7): 195-203.
[5]	CHEN B K , GONG C , YANG J . Importance-aware semantic segmentation for autonomous vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2019,20(1): 137-148.
[6]	CHEN L C , PAPANDREOU G , KOKKINOS I ,et al. DeepLab:semantic image segmentation with deep convolutional nets,atrous convolution,and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018,40(4): 834-848.
[7]	DONG G S , YAN Y , SHEN C H ,et al. Real-time high-performance semantic image segmentation of urban street scenes[J]. IEEE Transactions on Intelligent Transportation Systems, 2021,22(6): 3258-3274.
[8]	CHEN L C , ZHU Y , PAPANDREOU G ,et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]// Proceedings of the European Conference on Computer Vision (ECCV). Berlin:Springer, 2018: 801-818.
[9]	HUYNH C , TRAN A T , LUU K ,et al. Progressive semantic segmentation[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2021: 16750-16759.
[10]	WENG X , YAN Y , CHEN S ,et al. Stage-aware feature alignment network for real-time semantic segmentation of street scenes[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022,32(7): 4444-4459.
[11]	DOSOVITSKIY A , BEYER L , KOLESNIKOV A ,et al. An image is worth 16×16 words:transformers for image recognition at scale[J]. arXiv Preprint,arXiv:2010.11929, 2020.
[12]	LIU Z , LIN Y T , CAO Y ,et al. Swin transformer:hierarchical vision transformer using shifted windows[C]// Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2022: 9992-10002.
[13]	WANG W H , XIE E Z , LI X ,et al. Pyramid vision transformer:a versatile backbone for dense prediction without convolutions[C]// Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2022: 548-558.
[14]	PENG Z L , HUANG W , GU S Z ,et al. Conformer:local features coupling global representations for visual recognition[C]// Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2022: 357-366.
[15]	HU X G , JING L Y , SEHAR U . Joint pyramid attention network for real-time semantic segmentation of urban scenes[J]. Applied Intelligence, 2022,52(1): 580-594.
[16]	XIAO X , ZHAO Y , ZHANG F ,et al. BASeg:Boundary aware semantic segmentation for autonomous driving[J]. Neural Networks, 2023,157: 460-470.
[17]	RONNEBERGER O , FISCHER P , BROX T . U-Net:convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin:Springer, 2015: 234-241.
[18]	BADRINARAYANAN V , KENDALL A , CIPOLLA R . SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(12): 2481-2495.
[19]	BAI W . An ENet semantic segmentation method combined with attention mechanism[J]. Computational Intelligence and Neuroscience, 2023,2023: 1-9.
[20]	PAN H H , HONG Y D , SUN W C ,et al. Deep dual-resolution networks for real-time and accurate semantic segmentation of traffic scenes[J]. IEEE Transactions on Intelligent Transportation Systems, 2023,24(3): 3448-3460.
[21]	LIU Z , MAO H Z , WU C Y ,et al. A ConvNet for the 2020s[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2022: 11966-11976.
[22]	WOO S , PARK J , LEE J Y ,et al. CBAM:convolutional block attention module[C]// Proceedings of Computer Vision - ECCV 2018:15th European Conference. New York:ACM Press, 2018: 3-19.
[23]	WU Z F , SHEN C H , VAN DEN HENGEL A . Wider or deeper:revisiting the ResNet model for visual recognition[J]. Pattern Recognition, 2019,90(C): 119-133.
[24]	XIE E , WANG W , YU Z ,et al. SegFormer:simple and efficient design for semantic segmentation with transformers[J]. Advances in Neural Information Processing Systems, 2021,34: 12077-12090.
[25]	BROSTOW G J , FAUQUEUR J , CIPOLLA R . Semantic object classes in video:a high-definition ground truth database[J]. Pattern Recognition Letters, 2009,30(2): 88-97.
[26]	CORDTS M , OMRAN M , RAMOS S ,et al. The cityscapes dataset for semantic urban scene understanding[C]// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016: 3213-3223.
[27]	LONG J , SHELHAMER E , DARRELL T . Fully convolutional networks for semantic segmentation[C]// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2015: 3431-3440.

MDC	Scale_i	Input	s	k	p
MDC₁	i=1	320×64×64	1	1×7	0×3
				7×1	3×0
	i=2	320×64×64	1	1×9	0×4
				9×1	4×0
	i=3	320×64×64	1	1×11	0×5
				11×1	5×0
MDC₂	i=1	320×64×64	1	1×7	0×3
				7×1	3×0
	i=2	320×64×64	1	1×9	0×4
				9×1	4×0
	i=3	512×64×64	1	1×11	0×5
				11×1	5×0

数据集	图像大小	类别	训练集/幅	测试集/幅	验证集/幅
CamVid	960像素×720像素	11	367	100	233
Cityscapcs	2 048像素×1 024像素	19	2 975	500	1 525

算法	编码器	参数量	PA	mPA	mIoU
SegFormer-B2	Stage1～Stage4	27.36×10⁶	96.06%	86.15%	77.39%
	Stage4	24.56×10⁶	95.74%	81.83%	73.46%
TC-AEDNet	Stage1～Stage3+ASPP	15.31×10⁶	95.83%	82.70%	74.40%
	Stage1～Stage3+MSEM	18.55×10⁶	96.18%	84.53%	77.49%

实验序号	注意力机制	参数量	PA	mPA	mIoU
1	无	18.15×10⁶	96.00%	82.58%	75.10%
2	SAM	18.53×10⁶	95.17%	84.05%	75.57%
3	CAM	18.15×10⁶	96.06%	84.11%	76.37%
4	CBAM	18.55×10⁶	96.18%	84.53%	77.49%

算法	参数量	PA	mPA	mIoU
SegFormer-B2	27.36×10⁶	96.06%	86.15%	77.39%
SegFormer-B2+SDAM	27.45×10⁶	96.14% (↑0.08%)	86.64% (↑0.49%)	78.43% (↑1.04%)
TC-AEDNet	18.56×10⁶	96.05%	86.09%	77.91%
TC-AEDNet+SDAM	18.65×10⁶	96.16% (↑0.11%)	87.21% (↑1.12%)	78.63% (↑0.72%)

融合多尺度深度卷积的轻量级Transformer交通场景语义分割算法

Lightweight Transformer traffic scene semantic segmentation algorithm integrating multi-scale depth convolution

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 27

相关文章 15

Metrics

推荐阅读 0

特征融合	参数量	PA	mPA	mIoU
Stage1～Stage4	18.55×10⁶	96.18%	84.53%	77.49%
Stage[1, 2, 3-4]	18.56×10⁶	96.05%	86.09%	77.91%

序号	算法	参数量	PA	mPA	mIoU
1	FCN	18.64×10⁶	94.15%	63.12%	55.32%
2	SegNet	29.45×10⁶	93.53%	64.49%	57.01%
3	MagNet	6.37×10⁶	94.23%	69.71%	68.20%
4	RtHp	6.20×10⁶	94.95%	78.82%	73.67%
5	JPANet	3.49×10⁶	94.53%	78.42%	72.43%
6	DeepLab-V3+	58.75×10⁶	95.67%	83.49%	75.04%
7	Swin-B	88.23×10⁶	96.30%	85.74%	78.54%
8	SegFormer-B2	27.36×10⁶	96.06%	86.15%	77.91%
9	TC-AEDNet+SDAM	18.65×10⁶	96.16%	87.21%	78.63%

[1]	张学军, 张奉鹤, 盖继扬, 杜晓刚, 周文杰, 蔡特立, 赵博. mVulSniffer：一种多类型源代码漏洞检测方法[J]. 通信学报, 2023, 44(9): 149-160.
[2]	李致远, 徐丙磊, 周颖仪. 基于图神经网络的账户余额模型区块链地址分类方法[J]. 通信学报, 2023, 44(9): 115-126.
[3]	李勉, 李洋, 张纵辉, 史清江. Massive MIMO中通信高效的分布式预编码设计[J]. 通信学报, 2023, 44(8): 37-48.
[4]	王慧娇, 张鑫, 韦永壮, 李灵琛. 基于深度学习的SM4密码算法新型区分器[J]. 通信学报, 2023, 44(7): 171-184.
[5]	李荣鹏, 汪丙炎, 张宏纲, 赵志峰. 知识增强的语义通信接收端设计[J]. 通信学报, 2023, 44(6): 70-76.
[6]	陈东昱, 陈华, 范丽敏, 付一方, 王舰. 基于深度学习的随机性检验策略研究[J]. 通信学报, 2023, 44(6): 23-33.
[7]	马帅, 裴科, 祁华艳, 李航, 曹雯, 王洪梅, 熊海良, 李世银. 基于生成模型的地磁室内高精度定位算法研究[J]. 通信学报, 2023, 44(6): 211-222.
[8]	鲁斌, 孙洋, 杨振宇. 基于原始点云网格自注意力机制的三维目标检测方法[J]. 通信学报, 2023, 44(10): 72-84.
[9]	刘帅, 关杰, 胡斌, 马宿东. 基于MILP的轻量级密码算法ACE的差分分析[J]. 通信学报, 2023, 44(1): 39-48.
[10]	杨洁, 董标, 付雪, 王禹, 桂冠. 基于轻量化分布式学习的自动调制分类方法[J]. 通信学报, 2022, 43(7): 134-142.
[11]	王振宇, 郭阳, 李少青, 侯申, 邓丁. 面向轻量级物联网设备的高效匿名身份认证协议设计[J]. 通信学报, 2022, 43(7): 49-61.
[12]	杨秀璋, 彭国军, 李子川, 吕杨琦, 刘思德, 李晨光. 基于Bert和BiLSTM-CRF的APT攻击实体识别及对齐研究[J]. 通信学报, 2022, 43(6): 58-70.
[13]	廖育荣, 王海宁, 林存宝, 李阳, 方宇强, 倪淑燕. 基于深度学习的光学遥感图像目标检测研究进展[J]. 通信学报, 2022, 43(5): 190-203.
[14]	廖勇, 王世义. 高速移动环境下基于RM-Net的大规模MIMO CSI反馈算法[J]. 通信学报, 2022, 43(5): 166-176.
[15]	殷新春, 王梦宇, 宁建廷. 轻量级可搜索医疗数据共享方案[J]. 通信学报, 2022, 43(5): 110-122.

序号	算法	参数量	PA	mPA	mIoU
1	FCN^[13]	18.64×10⁶	91.89%	74.05%	64.75%
2	SegNet^[15]	29.45×10⁶	89.38%	74.25%	65.60%
3	RtHp^[7]	6.2×10⁶	93.86%	78.87%	68.14%
4	JPANet^[25]	3.49×10⁶	93.44%	78.27%	67.45%
5	DeepLab-V3+^[8]	58.75×10⁶	94.22%	85.16%	78.03%
6	SegFormer-B2^[18]	27.36×10⁶	95.11%	87.34%	80.55%
7	TC-AEDNet	18.65×10⁶	96.47%	87.64%	81.06%