基于MapReduce的HDFS数据窃取随机检测算法

doi:10.11959/j.issn.1000-436x.2018222

Abstract

Abstract:

To address the problems of big data efficient analysis and insider theft detection in the data theft detection of distributed cloud computing storage,taking HDFS (hadoop distributed file system) as a case study,a stochastic algorithm for HDFS data theft detection based on MapReduce was proposed.By analyzing the MAC timestamp features of HDFS generated by folder replication,the replication behavior’s detection and measurement method was established to detect all data theft modes including insider theft.The data set which is suitable for MapReduce task partition and maintains the HDFS hierarchy was designed to achieve efficient analysis of large-volume timestamps.The experimental results show that the missed rate and the number of mislabeled folders could be kept at a low level by adopting segment detection strategy.The algorithm was proved to be efficient and had good scalability under the MapReduce framework.

Key words: stochastic detection algorithm, HDFS, MapReduce, MAC timestamp, cloud computing storage

CLC Number:

TP311.1

Yuanzhao GAO,Binglong LI,Xingyuan CHEN. Stochastic algorithm for HDFS data theft detection based on MapReduce[J]. Journal on Communications, 2018, 39(10): 11-21.

Figures/Tables 12

References 22

[1]	中国信息通信研究院.云计算白皮书[R]. 北京:中国信息通信研究院, 2016.
	China Academy of Information and Communication Technology.>云计算白皮书>[R]. White Papers of Cloud Computing[R]. Beijing:China Academy of Information and Communication Technology, 2016.
[2]	张玉清, 王晓菲, 刘雪峰 ,等. 云计算环境安全综述[J]. 软件学报, 2016,27(6): 1328-1348.
	ZHANG Y Q , WANG X F , LIU X F ,et al. Survey on cloud computing security[J]. Journal of Software, 2016,27(6): 1328-1348.
[3]	CHANG V , RAMACHANDRAN M . Towards achieving data security with the cloud computing adoption framework[J]. IEEE Trans Services Computing, 2016,9(1): 138-151.
[4]	MARTINI B , CHOO K K R . Cloud storage forensics:owncloud as a case study[J]. Digital Investigation, 2013,10(4): 287-299.
[5]	LI Y , GAI K , QIU L ,et al. Intelligent cryptography approach for secure distributed big data storage in cloud computing[J]. Information Sciences, 2017,387: 103-115.
[6]	ALVA A , CALEFF O , ELKINS G ,et al. The Notorious nine:cloud computing top threats in 2013[R]. Cloud Security Alliance, 2013.
[7]	BROOK J M , FIELD S , SHACKLEFORD D ,et al. The treacherous 12 - cloud computing top threats in 2016[R]. Seattle:Cloud Security Alliance, 2016.
[8]	中国云计算安全政策与法律工作组.中国云计算安全政策与法律蓝皮书（2016）[R]. 上海:中国云计算安全政策与法律工作组, 2016.
	Cloud Computing Security Policies and Laws Group. Cloud computing security policies and laws blue book(2016)[R]. Shanghai:Cloud Computing Security Policies and Laws Group, 2016.
[9]	STOLFO S J , SALEM M B , KEROMYTIS A D . Fog computing:mitigating insider data theft attacks in the cloud[C]// IEEE Symposium on Security and Privacy Workshops. 2012: 125-128.
[10]	SRIRAM M , PATEL V , HARISHMA D ,et al. A hybrid protocol to secure the cloud from insider threats[C]// IEEE International Conference on Cloud Computing in Emerging Markets. 2014: 1-5.
[11]	NIKOLAI J , WANG Y . A system for detecting malicious insider data theft in IaaS cloud environments[C]// 2016 IEEE Global Communications Conference (GLOBECOM). 2016: 1-6.
[12]	SUBASHINI S , KAVITHA V . A survey on security issues in service delivery models of cloud computing[J]. Journal of Network ＆ Computer Applications, 2011,34(1): 1-11.
[13]	ROCHA F , CORREIA M . Lucy in the sky without diamonds:Stealing confidential data in the cloud[C]// IEEE/IFIP International Conference on Dependable Systems and Networks Workshops. 2011: 129-134.
[14]	PITROPAKIS N , LYVAS C , LAMBRINOUDAKIS C . The greater the power,the more dangerous the abuse:facing malicious insiders in the cloud[J]. Cloud Computing 2017, 2017: 156-161.
[15]	MARTINI B , CHOO K K R . Distributed filesystem forensics:XtreemFS as a case study[J]. Digital Investigation, 2014,11(4): 295-313.
[16]	GRIER J . Detecting data theft using stochastic forensics[J]. Digital Investigation, 2011,8(8): 71-77.
[17]	PATEL P C , SINGH U . A novel classification model for data theft detection using advanced pattern mining[J]. Digital Investigation, 2013,10(4): 385-397.
[18]	SHVACHKO K , KUANG H , RADIA S ,et al. The hadoop distributed file system[C]// 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST). 2010: 1-10.
[19]	GAO Y , LI B . A forensic method of efficient file extraction in HDFS based on three-level mapping[J]. Wuhan University Journal of Natural Sciences, 2017,22(2): 114-126.
[20]	WHITE T . hadoop:The definitive guide(3th Edition)[M]. Sebastopol: O'Reilly Media,IncPress, 2015.
[21]	BARHAM P , DRAGOVIC B , FRASER K ,et al. Xen and the art of virtualization[C]// ACM SIGOPS Operating Systems Review. 2003: 164-177.
[22]	MACKAY E B , CHALLENOR P G , BAHAJ A S . A comparison of estimators for the generalised Pareto distribution[J]. Ocean Engineering, 2011,38(11): 1338-1346.

Metrics

Recommended 0

No Suggested Reading articles found!

行号	inode_id	atime	ctime	inodeP₁_id	inodeP₂_id	…	inodeP_n_id
0	19 449	1 490 006 411 841	1 488 326 413 425	19 448	19 028	…	16 385
1	19 450	1 490 006 412 091	1 488 326 413 464	19 448	19 028	…	16 385
2	19 479	1 490 006 680 113	1 488 326 413 506	19 478	19 028	…	16 385
3	19 480	1 490 006 680 213	1 488 326 413 522	19 478	19 028	…	16 385

M_t(f)取值范围	C_f=0.1	C_f=0.3	C_f= 0.5	C_f=0.7	C_f=0.9
50≤M_t(f)<100	6.50%	13.4%	26.4%	40.1%	57.0%
100≤ M_t(f)<200	1.44%	3.35%	8.61%	19.1%	34.9%
200≤ M_t(f)<500	0	1.60%	4.00%	11.2%	25.6%
500≤ M_t(f)<1 000	0	0	0	2.60%	3.90%
1 000≤ M_t(f)<2 000	0	0	0	0	7.14%
2 000≤ M_t(f)<5 000	0	0	0	0	6.67%
M_t(f)≥5 000	0	0	0	3.57%	10.7%

M_t(f) 取值范围	C_f=0.1	C_f=0.3	C_f= 0.5	C_f=0.7	C_f=0.9
50≤M_t(f)<100	129	87	43	17	13
100≤M_t(f)<200	23	9	1	1	1
200≤M_t(f)<500	3	2	0	0	0
500≤M_t(f)<1 000	0	0	0	0	0
1 000≤M_t(f)<2 000	0	0	0	0	0
2 000≤M_t(f)<5 000	0	0	0	0	0
M_t(f)≥5 000	0	0	0	0	0

N值	Q=S	Q=2S	Q=4S	Q=8S
N=1	88	182	396	783
N=2	44	84	166	317
N=4	46	49	90	167
N=8	47.5	54.5	59	106

Stochastic algorithm for HDFS data theft detection based on MapReduce

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 22

Related Articles 12

Metrics

Recommended 0

M_f	A	B	C	D-1	D-2	E	F	G	H	I	J
50	13	17	31	1	1	61	4	317	140	107	53
100	13	17	31	1	1	11	4	179	85	107	19
200	13	17	31	1	1	1	4	55	51	84	1
500	13	17	16	0	1	1	4	1	30	50	1
1 000	13	17	1	0	1	1	4	1	17	1	1
2 000	10	14	1	0	1	1	4	1	9	1	1
5 000	7	9	1	0	0	1	4	1	4	1	0

[1]	Yihan YU,Yu FU,Xiaoping WU. Stochastic gradient descent algorithm preserving differential privacy in MapReduce framework [J]. Journal on Communications, 2018, 39(1): 70-77.
[2]	Hong-cheng LI,Xiao-ping WU,Yan CHEN. k-means clustering method preserving differential privacy in MapReduce framework [J]. Journal on Communications, 2016, 37(2): 125-131.
[3]	Hai-yan WANG,Pan CAO. Information extraction from massive Web pages based on node property and text content [J]. Journal on Communications, 2016, 37(10): 9-17.
[4]	Bin LIAO,Tao ZHANG,Jiong YU,Ji LIU,tong YINLu,Gang GUO. Temperature aware energy-efficient task scheduling strategies for mapreduce [J]. Journal on Communications, 2016, 37(1): 61-75.
[5]	. Research on distributed genetic k-means for anomaly detection in MANET [J]. Journal on Communications, 2015, 36(11): 167-173.
[6]	Qing-ni SHEN,Si-han QING,Zhong-hai WU,Li-zhe ZHANG,Ya-hui YANG. Securely redundant scheduling policy for MapReduce based on dynamic domains partition [J]. Journal on Communications, 2014, 35(1): 34-46.
[7]	. Design and application research on data service platform for big data [J]. Journal on Communications, 2013, 34(Z2): 32-174.
[8]	Yun-feng LIU,Li LI,Su-mei WANG,Qian-yi WANG,Xu YANG,Rong-bin OUYANG,Xin-zheng LONG. Design and application research on data service platform for big data [J]. Journal on Communications, 2013, 34(Z2): 170-174.
[9]	Chang-hai ZHAO,Hai-hua YAN,Xiao-peng LIU,Deng XIONG,Xiao-hua SHI. Evaluating MapReduce for seismic data processing using a practical application [J]. Journal on Communications, 2012, 33(Z2): 81-89.
[10]	Chun-ling CHENG,Qian CHAI,Xiao-long XU. Cloud computing virus detection algorithm based on co-evolution immune networ k [J]. Journal on Communications, 2011, 32(9A): 1-7.
[11]	Zhong-gang WANG,Tian-rui LI,Jun-bo ZHANG,Cheng-bing ZHAO,Zi-zhe GAO. Parallel method for preprocessing high-speed railway noise data based on MapReduce [J]. Journal on Communications, 2011, 32(9A): 263-269.
[12]	Jie XIN,Zhi-ming CUI,Peng-peng ZHAO,Guang-ming ZHANG,Xue-feng XIAN. Applying MapReduce frameworks to a virtualization platform for Deep Web data source discovery [J]. Journal on Communications, 2011, 32(7): 189-195.