一种软硬件结合的大数据访存踪迹收集分析工具集

doi:10.11959/j.issn.2096-0271.2019031

Abstract

Abstract:

The rise of in-memory computing framework represented by Spark,the gradual deepening of new non-volatile memory research and the increasingly severe data security situation made the existing memory behavior analysis tools unable to meet the demand for big data applications.A software-hardware hybrid memory trace collection and analysis toolkit for big data applications was proposed.Based on the basic memory trace collected by hardware,the memory behavior information with rich semantic information can be obtained quickly,accurately and undistorted by combining software information synchronization and offline annotation.It also provides an implementation method for real-time security monitoring of large data access.Finally,a group of real big data applications were analyzed by this toolkit.

Key words: memory trace, memory behavior, big data

CLC Number:

TP391

Zuojun LI,Haiyang PAN,Mingyu CHEN,Yungang BAO. A hybrid memory trace collection and analysis toolkit for big data applications[J]. Big Data Research, 2019, 5(4): 41-49.

Figures/Tables 6

References 33

[1]	ZAHARIA M , CHOWDHURY M , FRANKLIN M J ,et al. Spark:cluster computing with working sets[C]// The 2nd USENIX Conference on Hot Topics in Cloud Computing,June 22-25,2010,Boston,USA. Berkeley:USENIX Association, 2010:10.
[2]	BEZ R , PIROVANO A . Non-volatile memory technologies:emerging concepts and new materials[J]. Materials Science in Semiconductor Processing, 2004,7(4-6): 349-355.
[3]	ROSENFELD P , COOPERBALIS E , JACOB B . DRAMSim2:a cycle accurate memory system simulator[J]. IEEE Computer Architecture Letters, 2011,10(1): 16-19.
[4]	MAGNUSSON P , CHRISTENSSON M , ESKILSON J ,et al. Simics:a full system simulation platform[J]. Computer, 2002,35(2): 50-58.
[5]	BINKERT N , BECKMANN B , BLACK G ,et al. The Gem5 simulator[J]. ACM Sigarch Computer Architecture News, 2011,39(2): 1-7.
[6]	JALEEL A , COHN R , LUK C K ,et al. CMP$im:a pin-based on-the-fly multicore cache simulator[C]// The 4th Annual Workshop on Modeling,Benchmarking and Simulation,June 22,2008,Beijing,China.[S.l.:s.n. ], 2008: 28-36.
[7]	PATEL A , AFRAM F , CHEN S F ,et al. MARSSx86:a full system simulator for multicore x86 CPUs[C]// The 48th ACM/EDAC/IEEE Design Automation Conference,June 5-9,2011,New York,USA. Piscataway:IEEE Press, 2011: 1050-1055.
[8]	LUK C K , COHN R , MUTH R ,et al. Pin:building customized program analysis tools with dynamic instrumentation[C]// The 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation,June 12-15,2005,Chicago,USA. New York:ACM Press, 2005: 190-200.
[9]	NETHERCOTE N , SEWARD J . Valgrind:a framework for heavyweight dynamic binary instrumentation[C]// The 28th ACM SIGPLAN Conference on Programming Language Design and Implementation,June 10-13,2007,San Diego,USA. New York:ACM Press, 2007: 89-100.
[10]	SRIVASTAVA A , EUSTACE A . ATOM:a system for building customized program analysis tools[C]// The ACM SIGPLAN 1994 Conference on Programming Language Design and Implementation,June 20-24,1994,Orlando,USA. New York:ACM Press, 1994: 196-205.
[11]	LAURENZANO M , TIKIR M , CARRINGTON L ,et al. PEBIL:efficient static binary instrumentation for Linux[C]// IEEE International Symposium on Performance Analysis of Systems ＆Software,March 28-30,2010,White Plains,USA. Piscataway:IEEE Press, 2010: 175-183.
[12]	REINDERS J . VTune performance analyzer essentials:measurement and tuning techniques for software developers[M]. Santa Clara: Intel PressPress, 2004: 1-500.
[13]	GREGG B , MAURO J . DTrace:dynamictracing in Oracle Solaris,Mac OS X,and free BSD[M]. Upper Saddle River: Prentice Hall ProfessionalPress, 2011: 1-1152.
[14]	TAN Z X , WATERMAN A , AVIZIENIS R ,et al. RAMP gold:an FPGAbased architecture simulator for multiprocessors[C]// The 47th Design Automation Conference,June 13-18,2010,Anaheim,USA. New York:ACM Press, 2010: 463-468.
[15]	ROTHMAN J , CHEN C . BEE technology overview[C]// International Conference on Embedded Computer Systems,July 16-19,2012,Samos,Greece. Piscataway:IEEE Press, 2012:277.
[16]	NANDA A , MAK K K , SUGAVANAM K ,et al. MemorIES:a programmable,real-time hardware emulation tool for multiprocessor server design[J]. ACM SIGPLAN Notices, 2000,35(11): 37-48.
[17]	CHALAINANONT N , NURVITADHI E , MORRISON R ,et al. Real-time L3 cache simulations using the programmable hardware-assisted cache emulator (PHA$E)[C]// IEEE International Conference on Communications,October 27,2003,Austin,USA. Piscataway:IEEE Press, 2003: 86-95.
[18]	HONG J , NURVITADHI E , LU S L . Design,implementation,and verification of active cache emulator (ACE)[C]// The 2006 ACM/SIGDA 14th International Symposium on Field Programmable Gate Arrays,February 22-24,2006,Monterey,USA. New York:ACM Press, 2006: 63-72.
[19]	YOON H M,PARK G H , LEE K W , et a l . Reconfigurable address collector and flying cache simulator[C]// The HighPerformance Computing on the Information Superhighway,April 28-May 2,1997,Seoul,Korea. Piscataway:IEEE Press, 1997:552.
[20]	阮元, 陈明宇, 包云岗 ,等. 基于硬件的内存trace工具——MTT的设计与实现[J]. 电子学报, 2008,36(8): 1519-1525.
	RUAN Y , CHEN M Y , BAO Y G ,et al. The design and implementation of MIT a hardware-based memory trace tool[J]. Acta Electronica Sinica, 2008,36(8): 1519-1525.
[21]	朱晏 . 内存行为、功耗监控平台的硬件设计[D]. 北京:中国科学院大学, 2011.
	ZHU Y . Hardware design of memory behavior and power monitoring platform[D]. Beijing:University of Chinese Academy of Sciences, 2011.
[22]	李作骏 . 访存踪迹收集工具的逻辑设计与实现[D]. 北京:中国科学院大学, 2016.
	LI Z J . Logical design and implementation of memory trace collection tool[D]. Beijing:University of Chinese Academy of Sciences, 2016.
[23]	李作骏, 陈明宇 . 一种支持DDR4的软硬件结合的访存踪迹收集分析工具集[C]// 全国高性能计算学术年会,2018年10月18-20日,青岛,中国. 北京 :中国计算机学会, 2018: 224-231.
	LI Z J , CHEN M Y . Hybrid memory trace collection and analysis toolkit for DDR4[C]// National Annual Conference on High Performance Computing,October 18-20,2018,Qingdao,China. Beijing:China Computer Federation, 2018: 224-231.
[24]	TANG D , BAO Y G , HU W W ,et al. DMA cache:using on-chip storage to architecturally separate I/O data from CPU data for improving I/O p erformance[C]// The 16th International Symposium on High-Performance Computer Architecture,January 9-14,2010,Bangalore,India. Piscataway:IEEE Press, 2010.
[25]	CHEN L C , CUI Z H , BAO Y G ,et al. A lightweight hybrid hardware/software approach for object-relative memory profiling[C]// The 2012 IE EE International Symposium on Performance Analysis of Systems ＆ Software,April 1-3,2012,New Brunswick,USA. Piscataway:IEEE Press, 2012: 46-57.
[26]	ZHANG J T , LIU Y H , LI H F ,et al. PTAT:an efficient and precise tool for tracing and profiling detailed TLB misses[C]// ACM Transactions on Embedded Computing Systems (TECS),April 24-25,2017,Santa Rosa,USA. Piscataway:IEEE Press, 2017.
[27]	HUANG Y B , CUI Z H , CHEN L C ,et al. HaLock:hardware-assisted lock contention detection in multithreaded applications[C]// The 21st Int ernational Conference on Parallel Architectures and Compilation Techniques,September 19-23,2012,Minneapolis,USA. New York:ACM Press, 2012.
[28]	BAO Y G , CHEN M Y , RUAN Y ,et al. HMTT:a platform independent fullsystem memory trace monitoring system[C]// The 2008 ACM Internationa l Conference on Measurement and Modeling of Computer Systems,June 2-6,2008,Annapolis,USA. New York:ACM Press, 2008: 229-240.
[29]	CUI Z H , ZHU Y , BAO Y G ,et al. A fine-grained component-level power measurement method[C]// 2011 International Green Computing Conferen ce and Workshops,July 25-28,2011,Orlando,USA. Piscataway:IEEE Press, 2011.
[30]	LU T Y , LIU Y H , PAN H Y ,et al. TDV cache:organizing off-chip DRAM cache of NVMM from a fusion perspective[C]// The 35th IEEE International Conference on Computer Design (ICCD),November 5-8,2017,Boston,USA. Piscataway:IEEE Press, 2017.
[31]	HENNING J . SPEC CPU2006 benchmark descriptions[J]. ACM SIGARCH Computer Architecture News, 2006,34(4): 1-17.
[32]	BIENIA C , KUMAR S , SINGH J ,et al. The PARSEC benchmark suite:characterization and architectural implications[C]// The 17th International Conference on Parallel Architectures and Compilation Techniques,October 25-29,2008,Toronto,Canada. New York:ACM Press, 2008.
[33]	WANG L , ZHAN J F , LUO C J ,et al. BigDataBench:a big data benchmark suite from Internet services[C]// IEEE20th International Symposium on High Performance Computer Architecture,February 15-19,2014,Orlando,USA. Piscataway:IEEE Press, 2014.

Metrics

Recommended 0

No Suggested Reading articles found!

方法	完整	快速	无失真	丰富语义信息	低成本
软件模拟	√-	×	√	√	√
代码插入	√-	×	×	√-	√
硬件计数器	×	√	√	×	√
硬件仿真	√	√	×	√	×
硬件侦听	√	√	√	×	√-
HMTT	√	√	√	√	√

版本	内存速率	传输协议	传输带宽	其他特性
HMTT 1	DDR-200	千兆以太网	125 MB/s	旁路监听、行为标记
HMTT 2	DDR2-667	千兆以太网	375 MB/s	功耗监控、踪迹压缩、页表信息
HMTT 3	DDR3-800	PCIe Gen1	1 GB/s	内存集成、多板同步、函数、对象及I/O信息
HMTT 4	DDR4-1600	PCIe Gen2	2 GB/s	稳定性、可移植性提升

实验平台	组件	参数
发送机	中央处理器	两个Intel E5-2620V2,，6核心12线程，主频2.1 GHz，三级缓存15 MB
	内存	8条32 GB内存，共256 GB，DDR3-800， 8路交错
	操作系统	CentOS 7
接收机	磁盘阵列	4个1 TB固态硬盘组成4 TB大小的磁盘阵列，最高写入带宽为2 000 MB/s

工作集	使用内存大小/GB	访存踪迹大小/GB	描述
Quicksort	247.9	195	排序
SAP HANA	111.9	362	内存计算数据库
PageRank	253.4	553	搜索引擎
TeraSort	253.2	235	排序
Bayes	206.8	96	分类
Kmeans	95.2	425	聚类

A hybrid memory trace collection and analysis toolkit for big data applications

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 33

Related Articles 15

Metrics

Recommended 0

[1]	Haihong QIAN, Maoyi WANG, Yun XIONG. Digital transformation in higher education:a systematic review [J]. Big Data Research, 2023, 9(3): 56-70.
[2]	Hong MEI, Xiaoyong DU, Hai JIN, Xueqi CHENG, Yunpeng CHAI, Xuanhua SHI, Xiaolong JIN, Yasha WANG, Chi LIU. Big data technologies forward-looking [J]. Big Data Research, 2023, 9(1): 1-20.
[3]	Yang SHEN, Menglong YU. Metaverse and big data: data insight and value connection in spatio-temporal intelligence [J]. Big Data Research, 2023, 9(1): 103-110.
[4]	Jing CHEN. Humanities big data and its application in the field of digital humanities [J]. Big Data Research, 2022, 8(6): 3-14.
[5]	Yuchu LUO, Hao WU, Yuhan GUO, Shaocong TAN, Can LIU, Ruike JIANG, Xiaoru YUAN. Visualization in digital humanities [J]. Big Data Research, 2022, 8(6): 74-93.
[6]	Tongzheheng ZHENG, Bin LI, Minxuan FENG, Bolin CHANG, Dongbo WANG. Explore the structuration of historical books:the construction and quantitative analysis of digital humanities database of the Biographies of the Shiji [J]. Big Data Research, 2022, 8(6): 40-55.
[7]	Wenlong LI, Yuan YUAN, Xiaopeng AN. Modus operandi of big data governance: some preliminary observations [J]. Big Data Research, 2022, 8(4): 34-45.
[8]	Qifeng TANG, Zhiqing SHAO, Yazhen YE. Authenticating and licensing architecture of data rights in data trade [J]. Big Data Research, 2022, 8(3): 40-53.
[9]	Chenhuizi WANG, Wei CAI. Digital economics in metaverse: state-of-the-art, characteristics, and vision [J]. Big Data Research, 2022, 8(3): 140-150.
[10]	Mei YANG, Wei LI, Siyuan QIAO, Wei LIU. Research on calculation method of China’s big data industry output value [J]. Big Data Research, 2022, 8(3): 151-160.
[11]	Deren LI, Guo ZHANG, Yonghua JIANG, Xin SHEN, Weiling LIU. Opportunities and challenges of geo-spatial information science from the perspective of big data [J]. Big Data Research, 2022, 8(2): 3-14.
[12]	Xiaolan QIU, Yuxin HU, Songtao SHANGGUAN, Kun FU. Remote sensing satellite big data high-recision integration processing technology [J]. Big Data Research, 2022, 8(2): 15-27.
[13]	Weiquan LIU, Cheng WANG, Yu ZANG, Qian HU, Shangshu YU, Baiqi LAI. A survey on information extraction technology based on remote sensing big data [J]. Big Data Research, 2022, 8(2): 28-57.
[14]	Jianqiang LIU, Xiaomin YE, Youguo LAN. Remote sensing big data from Chinese ocean satellites and its application service [J]. Big Data Research, 2022, 8(2): 75-88.
[15]	Hequn YANG, Xiaofeng WANG, Yanqing GAO, Yiwen LU, Bingxin MA, Xinyao WANG. Analysis of satellite big data requirements in numerical weather prediction [J]. Big Data Research, 2022, 8(2): 89-102.