针对可逆神经网络的可视化解释方法

doi:10.11959/j.issn.2096-109x.2023090

Abstract

Abstract:

The issue of model explainability has gained significant attention in understanding the vulnerabilities and anonymous decision-making processes inherent in deep neural networks (DNN).While there has been considerable research on explainability for traditional DNN, there is a lack of exploration on the operation mechanism and explainability of reversible neural networks (RevNN).Additionally, the existing explanation methods for traditional DNN are not suitable for RevNN and suffer from issues such as excessive noise and gradient saturation.To address these limitations, a visual explanation method called visual explanation method for reversible neural network (VERN) was proposed for RevNN.VERN leverages the reversible property of RevNN and is based on the class-activation mapping mechanism.The correspondence between the feature map and the input image was explored by VERN, allowing for the mapping of classification weights of regional feature maps to the corresponding regions of the input image.The importance of each region for model decision-making was revealed through this process, which generates a basis for model decision-making.Experimental comparisons with other explanation methods on generalized datasets demonstrate that VERN achieves a more focused visual effect, surpassing suboptimal methods with up to 7.80% improvement in average drop (AD) metrics and up to 6.05% improvement in average increase (AI) metrics in recognition tasks.VERN also exhibits an 82.00% level of localization for the maximum point of the heat value.Furthermore, VERN can be applied to explain traditional DNN and exhibits good scalability, improving the performance of other methods in explaining RevNN.Furthermore, through adversarial attack analysis experiments, it is observed that adversarial attacks alter the decision basis of the model.This is reflected in the misalignment of the model’s attention regions, thereby aiding in the exploration of the operation mechanism of adversarial attacks.

Key words: model explainability, reversible neural network, visualization, class activation mapping, artificial intelligence security

CLC Number:

TP393

Xinying MU, Bingbing SONG, Fanxiao LI, Yisen ZHENG, Wei ZHOU, Yunyun DONG. Visual explanation method for reversible neural networks[J]. Chinese Journal of Network and Information Security, 2023, 9(6): 154-165.

Figures/Tables 23

References 29

[1]	谭清尹, 曾颖明, 韩叶 ,等. 神经网络后门攻击研究[J]. 网络与信息安全学报, 2021,7(3): 46-58.
	TAN Q Y , ZENG Y M , HAN Y ,et al. Survey on backdoor attacks targeted on neural network[J]. Chinese Journal of Network and Information Security, 2021,7(3): 46-58.
[2]	杨朋波, 桑基韬, 张彪 ,等. 面向图像分类的深度模型可解释性研究综述[J]. 软件学报, 2023,34(1): 230-254.
	YANG P B , SANG J T , ZHANG B ,et al. Survey on interpretability of deep models for image classification[J]. Journal of Software, 2023,34(1): 230-254.
[3]	FANG Z , KUANG K , LIN Y ,et al. Concept-based explanation for fine-grained images and its application in infectious keratitis classification[C]// Proceedings of the 28th ACM International Conference on Multimedia. 2020: 700-708.
[4]	化盈盈, 张岱墀, 葛仕明 . 深度学习模型可解释性的研究进展[J]. 信息安全学报, 2020,5(3): 1-12.
	HUA Y Y , ZHANG D X , GE S M ,et al. Research progress in the interpretability of deep learning models[J]. Journal of Cyber Security, 2020,5(3): 1-12.
[5]	纪守领, 李进锋, 杜天宇 ,等. 机器学习模型可解释性方法、应用与安全研究综述[J]. 计算机研究与发展, 2019,56(10): 2071-2096.
	JI S L , LI J F , DU T Y ,et al. Survey on the techniques,applications and security of machine learning interpretablity[J]. Journal of Computer Research and Development, 2019,56(10): 2071-2096.
[6]	ZEILER M D , FERGUS R . Visualizing and understanding convolutional networks[C]// Proceedings of European Conference on Computer Vision–ECCV 2014. 2014: 818-833.
[7]	AGARWAL C , NGUYEN A . Explaining image classifiers by removing input features using generative models[C]// Proceedings of the Asian Conference on Computer Vision. 2020.
[8]	FONG R C , VEDALDI A . Interpretable explanations of black boxes by meaningful perturbation[C]// Proceedings of the IEEE International Conference on Computer Vision. 2017: 3429-3437.
[9]	SELVARAJU R R , COGSWELL M , DAS A ,et al. Grad-CAM:visual explanations from deep networks via gradient-based localization[C]// Proceedings of the IEEE International Conference on Computer Vision. 2017: 618-626.
[10]	CHATTOPADHAY A , SARKAR A , HOWLADER P ,et al. Grad-CAM++:generalized gradient-based visual explanations for deep convolutional networks[C]// 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). 2018: 839-847.
[11]	WANG H , WANG Z , DU M ,et al. Score-CAM:score-weighted visual explanations for convolutional neural networks[C]// Proceedings of the IEEE/CVF Conference on Computer vision and Pattern Recognition Workshops. 2020: 24-25.
[12]	JACOBSEN J H , ARNOLD W M S , EDOUARD O . I-RevNet:deep invertible networks[C]// Proceedings of International Conference on Learning Representations. 2018.
[13]	JING J , DENG X , XU M ,et al. Hinet:deep image hiding by invertible network[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4733-4742.
[14]	LUGMAYR A , DANELLJAN M , VAN GOOL L ,et al. Srflow:learning the super-resolution space with normalizing flow[C]// Proceedings of Computer Vision–ECCV 2020. 2020: 715-732.
[15]	ZHOU B , KHOSLA A , LAPEDRIZA A ,et al. Learning deep features for discriminative localization[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 2921-2929.
[16]	SIMONYAN K , VEDALDI A , ZISSERMAN A . Deep inside convolutional networks:visualising image classification models and saliency maps[J]. arXiv Preprint arXiv:1312.6034, 2013.
[17]	SMILKOV D , THORAT N , KIM B ,et al. Smoothgrad:removing noise by adding noise[J]. arXiv Preprint arXiv:1706.03825, 2017.
[18]	SPRINGENBERG J T , DOSOVITSKIY A , BROX T ,et al. Striving for simplicity:the all convolutional net[J]. arXiv Preprint arXiv:1412.6806, 2014.
[19]	SUNDARARAJAN M , TALY A , YAN Q . Axiomatic attribution for deep networks[C]// International Conference on Machine Learning. 2017: 3319-3328.
[20]	PETSIUK V , DAS A , SAENKO K . Rise:randomized input sampling for explanation of black-box models[J]. arXiv Preprint arXiv:1806.07421, 2018.
[21]	GOMEZ A N , REN M , URTASUN R ,et al. The reversible residual network:backpropagation without storing activations[J]. Advances in Neural Information Processing Systems, 2017,30.
[22]	RAMASWAMY H G . Ablation-cam:visual explanations for deep convolutional network via gradient-free localization[C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2020: 983-991.
[23]	KRIZHEVSKY A , HINTON G . Learning multiple layers of features from tiny images[R]. 2009.
[24]	DENG J , DONG W , SOCHER R ,et al. ImageNet:a large-scale hierarchical image database[C]// Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009: 248-255.
[25]	MADRY A , MAKELOV A , SCHMIDT L ,et al. Towards deep learning models resistant to adversarial attacks[J]. arXiv Preprint arXiv:1706.06083, 2017.
[26]	DONG Y , LIAO F , PANG T ,et al. Boosting adversarial attacks with momentum[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 9185-9193.
[27]	TRAMèR F , KURAKIN A , PAPERNOT N ,et al. Ensemble adversarial training:Attacks and defenses[J]. arXiv Preprint arXiv:1705.07204, 2017.
[28]	KHAKZAR A , KHORSANDI P , NOBAHARI R ,et al. Do explanations explain? Model knows best[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10244-10253.
[29]	DABKOWSKI P , GAL Y . Real time image saliency for black box classifiers[C]// Advances in Neural Information Processing Systems. 2017,30.

Metrics

Recommended 0

No Suggested Reading articles found!

解释方法	生成一幅解释图所用时间/ms
解释方法	i-RevNet	VGG16	AlexNet	ResNet18
Grad-CAM	58.4	19.7	7.9	18.3
Grad-CAM++	61.9	23.8	9.3	18.8
Score-CAM	—	1758.5	301.7	1543.0
VERN	37.3	223.7	154.0	29.7

解释方法	AD指标	AI指标
Gradient	72.59%	5.40%
Occlusion	85.48%	4.83%
Mask	67.11%	6.0%3
RISE	31.74%	24.00%
Grad-CAM	26.79%	23.77%
Grad-CAM++	27.52%	21.93%
Score-CAM	89.61%	2.70%
VERN	26.57%	24.23%

解释方法	AD指标	AI指标
Gradient	74.56%	3.37%
Occlusion	50.80%	10.13%
Mask	57.05%	8.17%
RISE	46.15%	10.87%
Grad-CAM	30.85%	11.40%
Grad-CAM++	34.27%	10.10%
Score-CAM	44.14%	0.80%
VERN	23.4%	14.97%

解释方法	VGG16		AlexNet		ResNet18
解释方法	AD指标	AI指标	AD指标	AI指标	AD指标	AI指标
Gradient	76.69%	4.65%	79.62%	4.10%	75.98%	4.10%
Occlusion	94.74%	0.60%	89.16%	2.60%	94.81%	0.90%
Mask	52.75%	8.95%	67.35%	5.30%	57.75%	8.30%
RISE	34.60%	17.60%	52.94%	14.60%	34.55%	20.10%
Grad-CAM	34.04%	19.70%	60.69%	10.35%	26.30%	21.40%
Grad-CAM++	33.44%	16.70%	66.88%	6.90%	27.57%	19.7%
Score-CAM	25.59%	19.70%	54.57%	10.40%	28.44%	18.85%
VERN	18.16%	25.75%	45.14%	15.50%	26.11%	21.70%

数据集		Deletion指标			Insertion指标
数据集	Grad-CAM	Grad-CAM++	VERN	Grad-CAM	Grad-CAM++	VERN
ImageNet	0.159	0.163	0.159	0.557	0.554	0.560
CIFAR-10	0.346	0.365	0.333	0.610	0.594	0.644

Visual explanation method for reversible neural networks

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 23

References 29

Related Articles 10

Metrics

Recommended 0

传统深度神经网络待解释模型	Deletion指标				Insertion指标
传统深度神经网络待解释模型	Grad-CAM	Grad-CAM++	Score-CAM	VERN	Grad-CAM	Grad-CAM++	Score-CAM	VERN
VGG16	0.120	0.121	0.118	0.157	0.540	0.540	0.581	0.622
AlexNet	0.101	0.118	0.101	0.114	0.345	0.311	0.360	0.408
ResNet18	0.134	0.137	0.139	0.134	0.549	0.540	0.539	0.549

解释方法	AD指标	Improved AD指标	AI指标	Improved AI指标
Gradient	72.59%	53.15%	5.4%	13.2%
Occlusion	85.48%	60.33%	4.83%	25.83%
Mask	67.11%	43.87%	6.03%	10.13%
RISE	31.74%	24.39%	24%	28.27%
Grad-CAM	26.79%	26.11%	23.77%	24.43%
Grad-CAM++	27.52%	26.56%	21.93%	23.23%
Score-CAM	89.61%	88.72%	2.7%	3.97%

解释方法	落入比例
Grad-CAM	79.2 %
Grad-CAM++	77.8 %
VERN	82.00%

模型	解释方法	AD指标	AI指标
	Grad-CAM	27.22%	12.20%
VGG16	Grad-CAM++	27.23%	11.40%
	VERN	26.39%	12.20%
	Grad-CAM	52.69%	10.40%
AlexNet	Grad-CAM++	52.50%	10.20%
	VERN	52.00%	11.20%
	Grad-CAM	25.98%	15.40%
ResNet18	Grad-CAM++	26.04%	16.00%
	VERN	25.00%	17.40%

[1]	Jinwei WANG, Zhengjia CHEN, Xue XIE, Xiangyang LUO, Bin MA. Review of malware detection and classification visualization techniques [J]. Chinese Journal of Network and Information Security, 2023, 9(5): 1-20.
[2]	Yifan HE, Jie ZHANG, Weiming ZHANG, Nenghai YU. Privacy leakage risk assessment for reversible neural network [J]. Chinese Journal of Network and Information Security, 2023, 9(4): 29-39.
[3]	Xiaochen SHEN, Yinhui GE, Bo CHEN, Ling YU. Research on construction technology of artificial intelligence security knowledge graph [J]. Chinese Journal of Network and Information Security, 2023, 9(2): 164-174.
[4]	Qingyin TAN, Yingming ZENG, Ye HAN, Yijing LIU, Zheli LIU. Survey on backdoor attacks targeted on neural network [J]. Chinese Journal of Network and Information Security, 2021, 7(3): 46-58.
[5]	Ying ZHAO,Zhuo ZHANG,Xiaoru YUAN. ChinaVis Data Challenge from 2015 to 2017 [J]. Chinese Journal of Network and Information Security, 2018, 4(2): 55-61.
[6]	Haocheng ZHANG,Xiaojie WU,Xiang TANG,Runxuan SHU,Tianchen DING,Xiaoju DONG. System detecting network anomaly with visualization techniques [J]. Chinese Journal of Network and Information Security, 2018, 4(2): 40-54.
[7]	Islam Junayed,Xu Kai,Wong William. [J]. Chinese Journal of Network and Information Security, 2018, 4(2): 18-33.
[8]	Guo-jun MA,Shui-bo WANG,Qing-qi PEI,Yang ZHAN. Research on parallel coordinate visualization technology based on principal component analysis and K-means clustering [J]. Chinese Journal of Network and Information Security, 2017, 3(8): 18-27.
[9]	Yi-fan ZHANG,Xiao-ju DONG. Visualization analysis and design of DDoS attack [J]. Chinese Journal of Network and Information Security, 2017, 3(2): 53-65.
[10]	Wei-ming LI,De-qing ZOU,Guo-zhong SUN. Successive memory image analysis method for malicious codes [J]. Chinese Journal of Network and Information Security, 2017, 3(2): 20-30.

解释方法	IoU
Grad-CAM	0.277
Grad-CAM++	0.293
VERN	0.438