Journal on Communications
Previous Articles Next Articles
Online:
Published:
Abstract: With the rapid expansion of network bandwidth, devices and applications, log management is facing the challenge of exploding data volumes. Log analysis platform built on SQL-on-Hadoop is capable of storing and querying hundreds of billions of log entries effectively. Columnar and compressed data formats for Hadoop are benchmarked with real-world multi-TB dataset. Conditional and statistical querying efficiency of Hive and Impala is tested. With gzipped parquet format, log data can be compressed by 80%, and querying with impala is 5 times faster. On this platform, six security incident analysis and detection applications are already deployed.
0 / / Recommend
Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks
URL: https://www.infocomm-journal.com/txxb/EN/
https://www.infocomm-journal.com/txxb/EN/Y2014/V35/IZ1/4