数据整理——大数据治理的关键技术

doi:10.11959/j.issn.2096-0271.2019020

Abstract

Abstract:

Data is an important resource for governments,businesses and institutions.Data governance focuses on many aspects of effective use of data resources,such as data asset,data management,data sharing,and data privacy.A key techniquedata wrangling-in data governance from the perspective of data management was explored.The key technologies of data wrangling based on data owners and direct users-industry users were emphasized,including data structure processing,data quality assessment and data cleaning,data normalization,data fusion and extraction,data publishing and sharing,etc.Finally,some thoughts on strengthening the research on data organization were put forward.

Key words: data wrangling, data preparation, data governance, data management

CLC Number:

TP315

Xiaoyong DU, Yueguo CHEN, Ju FAN, Wei LU. Data wrangling:a key technique of data governance[J]. Big Data Research, 2019, 5(3): 13-22.

Figures/Tables 3

References 11

[1]	HELLERSTEIN J M , HEER J , KANDEL S . Self-service data preparation:research to practice[J]. IEEE Data Engineering Bulletin, 2018,41(2): 23-34.
[2]	HEER J , HELLERSTEIN J M , KANDEL S . Data wrangling[M]// Encyclopedia of big data technologies 2019.[S.l.:s.n]. 2019.
[3]	LI G L , ZHENG Y D , FAN J ,et al. Crowdsourced data management:overview and challenges[C]// The 2017 ACM International Conference on Management of Data,May 1419,2017,Chicago,USA. New York:ACM Press, 2017: 1711-1716.
[4]	DOAN A H , ARDALAN A , BALLARD J R ,et al. Toward a system building agenda for data integration[J]. IEEE Data Engineering Bulletin, 2018,41(2): 35-46.
[5]	宋晓宇, 王永会 . 数据集成与应用集成[M]. 北京: 中国水利水电出版社, 2008.
	SONG X Y , WANG Y H . Data integration and application integration[M]. Beijing: China Water and Power PressPress, 2008.
[6]	ABEDJAN Z , CHU X , DENG D ,et al. Detecting data errors:where are we and what needs to be done[J]. Proceedings of the VLDB Endowment, 2016,9(12): 993-1004.
[7]	BOHANNON P , FAN W F , GEERTS F ,et al. Conditional functional dependencies for data cleaning[C]// 2007 IEEE 23rd International Conference on Data Engineering,April 15-20,2007,Istanbul,Turkey. Piscataway:IEEE Press, 2007: 746-755.
[8]	CHU X , ILYAS I F , PAPOTTI P . Holistic data cleaning:putting violations into context[C]// 2013 IEEE 29th International Conference on Data Engineering (ICDE),April 8-12,2013,Brisbane,Australia. Piscataway:IEEE Press, 2013: 458-469.
[9]	CHU X , MORCOS J , ILYAS I F ,et al. KATARA:a data cleaning system powered by knowledge bases and crowdsourcing[C]// The 2015 ACM SIGMOD International Conference on Management of Data,May 31-June 4,2015,Melbourne,Australia. New York:ACM Press, 2015: 1247-1261.
[10]	YAKOUT M , BERTI-éQUILLE L , ELMAGARMID A K . Don’t be scared:use scalable automatic repairing with maximal likelihood and bounded changes[C]// The 2013 ACM SIGMOD International Conference on Management of Data,June 22-27,2013,New York,USA. New York:ACM Press, 2013: 553-564.
[11]	REKATSINAS T , CHU X , ILYAS I F ,et al. HoloClean:holistic data repairs with probabilistic inference[J]. Proceedings of the VLDB Endowment, 2017,10(11): 1190-1201.

Metrics

Recommended 0

No Suggested Reading articles found!

[1]	Hong MEI, Xiaoyong DU, Hai JIN, Xueqi CHENG, Yunpeng CHAI, Xuanhua SHI, Xiaolong JIN, Yasha WANG, Chi LIU. Big data technologies forward-looking [J]. Big Data Research, 2023, 9(1): 1-20.
[2]	Rui ZHU, Hongzhi WANG, Shuangshuang CUI, Kaixin ZHANG, Yu YAN. Cloud-edge-end collaborative big data management for metaverse [J]. Big Data Research, 2023, 9(1): 63-77.
[3]	Keman HUANG, Xiaoyong DU. Value chain model of data governance and its application on data governance regulation analysis [J]. Big Data Research, 2022, 8(4): 3-16.
[4]	Wenlong LI, Yuan YUAN, Xiaopeng AN. Modus operandi of big data governance: some preliminary observations [J]. Big Data Research, 2022, 8(4): 34-45.
[5]	Hao HU, Fazhi QI, Xiaokang SUN, Qi LUO. HEPS scientific data management policy research and applications [J]. Big Data Research, 2022, 8(1): 5-14.
[6]	Qi XU, Ziming ZOU, Yaqin YUAN, Xiaoyan HU, Jizhou TONG, Wenzhen MA. Data management process model for the science and technology programs [J]. Big Data Research, 2022, 8(1): 15-23.
[7]	Feng ZHI, Feng TIAN, Ruofan ZHAO. Classification of big data in metrology [J]. Big Data Research, 2022, 8(1): 60-72.
[8]	Huanyou CHAI, Sannyuya LIU, Lingyun KANG, Yaxian ZHANG, Qing LI, Zhi LIU. Research on the mechanism and key technologies for big data collection in education [J]. Big Data Research, 2020, 6(6): 14-25.
[9]	Hong DAI, Qun ZHANG, Haolin LU, Junzhi BIN. Comparative analysis between bank industry data governance guidelines and DCMM [J]. Big Data Research, 2020, 6(5): 118-128.
[10]	Menghui YANG, Xiaoyong DU. Big data governance in governments:a new form of the government administration [J]. Big Data Research, 2020, 6(2): 3-18.
[11]	Ju FAN, Yueguo CHEN, Xiaoyong DU. Progress on human-in-the-loop data preparation [J]. Big Data Research, 2019, 5(6): 3-18.
[12]	Minghe YU, Tiezheng NIE, Guoliang LI. Data curation technologies and applications [J]. Big Data Research, 2019, 5(6): 30-46.
[13]	Xiaomi AN, Mingjun GUO, Xuehai HONG, Wei WEI. Framework of government big data governance system and effective way of implementation [J]. Big Data Research, 2019, 5(3): 3-12.
[14]	Dongxing JIANG, Ruonan GAO, Haoyu WANG. Research on supervising big data governance method for securities and futures industry [J]. Big Data Research, 2019, 5(3): 23-34.
[15]	Hong DAI, Qun ZHANG, Zhuo YIN. Study on big data governance standard system [J]. Big Data Research, 2019, 5(3): 47-54.

Data wrangling:a key technique of data governance

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 3

References 11

Related Articles 15

Metrics

Recommended 0