增强型虚拟寄存器轮转算法

doi:10.11959/j.issn.2096-109x.2018038

摘要/Abstract

摘要：

为了对抗基于数据流逆向分析的语义攻击，以虚拟寄存器为切入点，提出了一种增强型虚拟寄存器轮转算法。该算法通过在解释执行中随机打乱部分虚拟寄存器与操作数的虚拟编译映射关系，有效地增加了虚拟机在解释执行过程中的数据流复杂度；同时，随机采用3种机制对轮转长度进行设定，增强了虚拟机代码保护系统的多样性。最后，设计实现了采用增强型虚拟寄存器轮转算法的虚拟机代码保护原型系统，验证了该算法的有效性。

关键词: 虚拟机代码保护, 寄存器轮转, 数据流分析, 语义攻击

Abstract:

Sematic attacks based on the data flow analysis bring big challenges to the code obfuscation.Concerning the data flow of virtual machine based (VM-based) code protection,the method transfers the mapping relation between the virtual registers and the op-code of the bytecode during executing,which means the uncertainty and complexity of the data flow during interpretive execution of the bytecode.In addition,three policies are proposed to address the problem that how to choose the length of rotation for each bytecode,which grows complexity of the protection.Finally,a prototype of VRR-VM (virtual machine protection system based on virtual registers rotation) was implemented.Experiment results show that the method is effective and applicable for anti-reversing.

Key words: VM-based code protection, virtual registers rotation, data flow analysis, sematic attacks

中图分类号:

TP311

潘雁,林伟. 增强型虚拟寄存器轮转算法[J]. 网络与信息安全学报, 2018, 4(5): 47-54.

Yan PAN,Wei LIN. Enhanced method based on virtual registers rotation[J]. Chinese Journal of Network and Information Security, 2018, 4(5): 47-54.

图/表 16

图1

图2

图3

图4

表1

图5

图6

图7

表2

表3

图8

表4

表5

表6

图9

图10

参考文献 22

[1]	NEWSOME J , SONG D . Dynamic taint analysis for automatic detection,analysis,and signature generation of exploits on commodity software[J]. Chinese Journal of Engineering Mathematics, 2005,29(5): 720-724.
[2]	徐欣 . 动态数据流分析技术在恶意软件分析中的应用研究[D]. 合肥:中国科学技术大学, 2016.
	XU X . Research of dynamic data flow analysis technology application in malware analysis[D]. Hefei:University of Science and Technology of China, 2016.
[3]	REDDI JANAPA V ,et al. PIN:a binary instrumentation tool for computer architecture research and education[C]// The Workshop on Computer Architecture Education, 2004:22.
[4]	CHOW J , PFAFF B , GARFINKEL T ,et al. Understanding data lifetime via whole system simulation[C]// Usenix Security Symposium, 2004: 321-336.
[5]	SHARIF M , LANZI A , GIFFIN J ,et al. Automatic reverse engineering of malware emulators[C]// 2009 30th IEEE Symposium on Security and Privacy. 2009: 94-109.
[6]	黄荷洁, 康绯, 舒辉 ,等. 基于动态数据流分析的虚拟机保护破解技术[J]. 计算机工程, 2014,40(9): 59-65.
	HUANG H J , KANG F , SHU H ,et al. Reverse technology of virtual machine protection based on dynamic dataflow analysis[J]. Computer Engineering, 2014,40(9): 59-65.
[7]	徐方华 . 基于虚拟堆的虚拟保护技术的研究[D]. 昆明:云南大学, 2013.
	XU F H . Research on virtual protection technology based on virtual heap[D]. Kunming:Yunnan University, 2013.
[8]	WANG H , FANG D , LI G ,et al. NISLVMP:improved virtual machine-based software protection[C]// The Ninth International Conference on Computational Intelligence and Security. 2013: 479-483.
[9]	舒柏程, 李毅超, 曹跃 . 基于虚拟机的软件保护技术研究[J]. 计算机工程与科学, 2008,30(A1): 25-28.
	SHU B C , LI Y C , CAO Y . Research on software protection based on virtual machine[J]. Computer Engineering ＆ Science, 2008,30(A1): 25-28.
[10]	GHOSH S , HISER J , DAVIDSON J W . Replacement attacks against VM-protected applications[C]// ACM Sigplan/sigops Conference on Virtual Execution Environments. 2012: 203-214.
[11]	COOGAN K P . Deobfuscation of packed and virtualization-obfuscation protected binaries[D]. Arizona:University of Arizona, 2011.
[12]	COOGAN K , LU G , DEBRAY S . Deobfuscation of virtualization-obfuscated software:a semantics-based approach[C]// ACM Conference on Computer and Communications Security. 2011: 275-284.
[13]	房鼎益, 张恒, 汤战勇 ,等. 一种抗语义攻击的虚拟化软件保护方法[J]. 四川大学学报(工程科学版), 2017,49(1): 159-168.
	FANG D Y , ZHANG H , TANG Z Y ,et al. DAS-VMP:a virtual machine-based software protection method for defending against semantic attacks[J]. Journal of Sichuan University (Advanced Engineering Sciences), 2017,49(1): 159-168.
[14]	谢鑫, 刘粉林, 芦斌 ,等. Handler混淆增强的虚拟机保护方法[J]. 计算机工程与应用, 2016,52(15): 146-152.
	XIE X , LIU F L , LU B ,et al. Virtual machine protection based on Handler obfuscation enhancement[J]. Computer Engineering and Applications, 2016,52(15): 146-152.
[15]	谢鑫, 刘粉林, 芦斌 ,等. 一种基于代码并行化和虚拟机多样化的软件保护方法[J]. 小型微型计算机系统, 2015,36(11): 2588-2593.
	XIE X , LIU F L , LU B ,et al. Software protection scheme based on code parallelization and virtual machine diversity[J]. Journal of Chinese Computer Systems, 2015,36(11): 2588-2593.
[16]	房鼎益, 赵媛, 王怀军 ,等. 一种具有时间多样性的虚拟机软件保护方法[J]. 软件学报, 2015,26(6): 1322-1339.
	FANG D Y , ZHAO Y , WANG H J ,et al. Software protection based on virtual machine with time diversity[J]. Journal of Software, 2015,26(6): 1322-1339.
[17]	WANG H J , FANG D Y , LI G ,et al. TDVMP:Improved virtual machine-based software protection with time diversity[C]// ACM Sigplan on Program Protection and Reverse Engineering Workshop. 2014: 1-9.
[18]	WANG H J , FANG D Y , LI G ,et al. NISLVMP:improved virtual machine-based software protection[C]// The Ninth International Conference on Computational Intelligence and Security. 2013: 479-483.
[19]	KUANG K , TANG Z , GONG X ,et al. Exploiting dynamic scheduling for vm-based code obfuscation[C]// IEEE Trustcom/bigdatase/ispa. 2017: 489-496.
[20]	KUANG K , TANG Z , GONG X ,et al. Enhance virtual-machine-based code obfuscation security through dynamic bytecode scheduling[J]. Computers ＆ Security, 2018,74: 202-220.
[21]	TANG Z , LI G , FANG D ,et al. Code virtualized protection system with instruction set randomization[J]. Journal of Huazhong University of Science ＆ Technology, 2016.
[22]	XU J F , ZHANG W , SUN B . Research on software protection based on virtual machine[J]. Journal of China Universities of Posts ＆Telecommunications, 2012,19(S1): 122-126.

明文字节码序列	助记符指令序列差错识别	助记符指令序列正确识别
A2 03	V_Pop VR7	V_Pop VR7
2D 01 06	V_Mov VR6,VR1	V_Mov VR1,VR7

被保护软件	保护前文件大小/kB	保护后文件大小/kB
被保护软件	保护前文件大小/kB	无轮转算法	有轮转算法
ShellSort.exe	19	34	36
InsertionSort.exe	19	33	35
BubbleSort.exe	19	33	36
QuickSort.exe	19	34	36

被保护软件	保护前目标代码执行时间/ms	保护后目标代码执行时间/ms		增长比例
被保护软件	保护前目标代码执行时间/ms	无轮转算法	有轮转算法	增长比例
ShellSort.exe	0.39	611	1 391	1.28
InsertionSort.exe	1.51	3 457	8 525	1.47
BubbleSort.exe	3.84	93 456	20 480	1.19
QuickSort.exe	0.37	339	726	1.14

测试程序	目标代码指令数	Key-Code执行时间/ms
ShellSort.exe	58	0.39
InsertionSort.exe	38	1.51
BubbleSort.exe	41	3.84
QuickSort.exe	58	0.37

被保护软件	保护前文件大小/kB	保护后文件大小/kB
被保护软件	保护前文件大小/kB	CV2.2.1.0	VMP2.13.8	X86VM
ShellSort.exe	19	725	31	36
InsertionSort.exe	19	727	30	35
BubbleSort.exe	19	709	30	36
QuickSort.exe	19	722	30	36