System Anomaly Detection in Distributed Systems through MapReduce-Based Log Analysis
System anomaly detection is very important for development, maintenance and performance refinement in large scale distributed systems. It’s a good way to obtain the troubleshooting and problem diagnosis by analyzing system logs produced by distributed systems. However, due to the increasing scale and complexity of distributed systems, the size of logs must be very large. Thus, it’s inefficient for common methods to analyze system logs on single node. Therefore, there is a great demand to adopt a distributed method for anomaly detection techniques based on log analysis. In this paper, a MapReduce-Based Framework is implemented to analyze the distributed log for detecting anomaly. The framework is built on top of Hadoop, an open source distributed file system and MapReduce implementation. We first make use of Random Access File to realize an incremental way for aggregating system logs from each node of the monitored cluster, and collect them to the analysis cluster. Then, we apply the K-means clustering algorithm to integrate the collected logs. After that, we implement a MapReduce-Based algorithm to parser these clustered log files. Furthermore, in order to make the best use of this collected data, a flexible and powerful way is utilized to display monitoring and analysis results. Thus, we can monitor system status of large distributed cluster and detect its anomalies.
MapReduce log analysis anomaly detection distributed system K-means
Yan Liu Wei Pan Ning Cao Guangwei Qiao
Ideal Institute of Information and TechnologyNortheast Normal UniversityEngineering & Research Cente College of Information Science and EngineeringNortheastern UniversityShenyang, China Ideal Institute of Information and Technology Northeast Normal University Engineering & Research Cen
国际会议
成都
英文
1-4
2010-08-20（万方平台首次上网日期，不代表论文的发表时间）