一种基于Spark的大数据匿名化系统实现

doi:10.11959/j.issn.1000-0801.2018133

Abstract

Abstract:

Group based anonymization is a classical data anonymization framework,which achieves the effect of privacy protection by constructing groups of anonymized data records ensuring that records in the same group cannot be distinguished with each other.The electric power industry big data analysis involves the core data of the power enterprises and the user privacy data,the data sensitivity is stronger,traditional data anonymization systems are unable to meet the needs of big data business applications and safety protection of electric power industry.A new big data anonymization system based on Spark was designed and implemented,which could provide the support for multiple data formats stored on Hadoop and substantially improve the processing efficiency for big data.

Key words: data anonymization, privacy, electric power industry, safety protection, Spark

CLC Number:

TP391

Chaoyi BIAN,Shaomin ZHU,Tao ZHOU. Implementation of a big data anonymization system based on Spark[J]. Telecommunications Science, 2018, 34(4): 156-161.

Figures/Tables 7

References 9

[1]	Apache Software Foundation. Apache HadoopEB]. 2011.
[2]	Apache Software Foundation. Apache SparkEB]. 2014.
[3]	SWEENEY L . K-anonymity:a model for protecting privacy[J]. International Journal of Uncertainty,Fuzziness and Knowledge-Based Systems, 2002,10(5): 557-570.
[4]	MACHANAVAJJHALA A , KIFER D , GEHRKE J ,et al. L-diversity:privacy beyondk-anonymity[J]. ACM Transactions on Knowledge Discovery from Data (TKDD), 2007,1(1): 3.
[5]	FUNG B , WANG K , CHEN R ,et al. Privacy-preserving data publishing:a survey of recent developments[J]. ACM Computing Surveys (Csur), 2010,42(4): 14.
[6]	PRASSER F , KOHLMAYER F . Putting statistical disclosure control into practice:the ARX data anonymization tool[M]. Berlin: Springer International Publishing, 2015.
[7]	JADHAV R H , INGLE R B . A survey on data anonymization approaches using MapReduce framework on cloud[J]. Digital Image Processing, 2015,7(2): 48-49.
[8]	LEFEVRE K , DEWITT D J , RAMAKRISHNAN R . Mondrian multidimensional k-anonymity[C]// International Conference on Data Engineering,April 3-7,2006,Atlanta,GA,USA. Piscataway:IEEE Press, 2006:25.
[9]	QIYUAN G . A survey on data anonymization approaches using MapReduce framework on cloudEB]. 2015.

Metrics

Recommended 0

No Suggested Reading articles found!

序号	非敏感信息			敏感信息
序号	邮编	年龄/岁	国籍	用电量
1	15072	29	英国	较大
2	15041	27	美国	较大
3	15041	22	日本	一般
4	15072	25	美国	一般
5	12904	51	印度	很小
6	12904	53	英国	较大
7	12907	43	美国	一般
8	12907	41	美国	一般
9	15072	33	美国	很小
10	15072	39	印度	很小
11	15041	38	日本	很小
12	15041	37	美国	很小

序号	非敏感信息			敏感信息
序号	邮编	年龄/岁	国籍	用电量
1	150**	<30	*	较大
2	150**	<30	*	较大
3	150**	<30	*	一般
4	150**	<30	*	一般
5	1290*	≥ 40	*	很小
6	1290*	≥ 40	*	较大
7	1290*	≥ 40	*	一般
8	1290*	≥ 40	*	一般
9	150**	3*	*	很小
10	150**	3*	*	很小
11	150**	3*	*	很小
12	150**	3*	*	很小

序号	非敏感信息			用电量敏感信息
序号	邮编	年龄/岁	国籍	用电量敏感信息
1	1507*	≤ 40	*	较大
2	1507*	≤ 40	*	一般
3	1507*	≤ 40	*	很小
4	1507*	≤ 40	*	很小
5	1290*	>40	*	很小
6	1290*	>40	*	较大
7	1290*	>40	*	一般
8	1290*	>40	*	一般
9	1504*	≤ 40	*	较大
10	1504*	≤ 40	*	一般
11	1504*	≤ 40	*	很小
12	1504*	≤ 40	*	很小

[1]	Chi ZHANG, Ye LU, Yuping LUO, Xiaokai SUN, Hanke ZHU. A novel approach for face privacy protection based on surveillance video in complex scene [J]. Telecommunications Science, 2021, 37(1): 94-101.
[2]	Shaoliang PENG,Liang BAI,Li WANG,Minxia CHENG,Shulin WANG. Trusted edge computing for smart healthcare [J]. Telecommunications Science, 2020, 36(6): 56-63.
[3]	Zishan LIU,Qiang CHENG,Bo LV. A survey on key technologies of privacy protection for machine learning [J]. Telecommunications Science, 2020, 36(11): 18-27.
[4]	Weiyu JIANG, Bingyang LIU, Chuang WANG. Network architecture with intrinsic security [J]. Telecommunications Science, 2019, 35(9): 20-28.
[5]	Ming XU,Xue YANG,Jianwu ZHANG. A review of network traffic analysis targeting mobile devices [J]. Telecommunications Science, 2018, 34(4): 98-108.
[6]	Ye CHEN,Dongjin XU,Liang XIAO. Survey on network security based on blockchain [J]. Telecommunications Science, 2018, 34(3): 10-16.
[7]	Shengli ZHOU,Bin CHEN,Lifa WU. Data security protection schema of telecom operators under the environment of big data [J]. Telecommunications Science, 2017, 33(5): 119-125.
[8]	Zhuohang LI. An improved CLTree algorithm [J]. Telecommunications Science, 2016, 32(8): 124-127.
[9]	Ying ZOU,Yirong WANG,Yanru WANG,Sining WANG. Wireless VPN network structure and safety protection in the north and south call center of SGCC [J]. Telecommunications Science, 2016, 32(2): 175-181.
[10]	Jianxin Liao. Big Data Technology：Current Applications and Prospects [J]. Telecommunications Science, 2015, 31(7): 1-12.
[11]	Ai Fang,Xiong Xu,Bing Liang,Yuzhong Zhang,Yiping Yang. Comparison of Open-Source Distributed Computing Framework for Big Data [J]. Telecommunications Science, 2015, 31(7): 152-157.
[12]	Yirong Wang,Sining Wang,Ying Zou,Botao Yu. Research on Business Requirement Analysis and Network Architecture of CDN for Power Grid [J]. Telecommunications Science, 2015, 31(5): 150-757.
[13]	Wenyu Pan,Xin Cai,Yilong Ruan,Xiaohui Yao. Research on Sensitive Information Protection of Carriers' Big-Data Platform [J]. Telecommunications Science, 2014, 30(11): 125-130.
[14]	Shengyong Ding,Shiwu Min,Yongbing Fan. A Large Scale NetFlow Analysis System Based on Spark [J]. Telecommunications Science, 2014, 30(10): 48-51.
[15]	Feng Gao,Feng Zhang,Wei Zhou. Study on Standardization of Privacy Protection in Network Environment [J]. Telecommunications Science, 2013, 29(4): 112-117.

Implementation of a big data anonymization system based on Spark

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 9

Related Articles 15

Metrics

Recommended 0