Analysis of Resource Usage Profile for MapReduce Applications Using Hadoop on Cloud

摘要：

　　In this paper we present a study of resource consumption profiles for MapReduce applications using Hadoop on Amazon EC2.We selected three applications and measured their resource usage in terms of CPU and memory footprint.Specifically,we study Grep,Word Count and Sort applications while altering Hadoop’s configuration parameters corresponding to I/O buffer.Our study brings up 3 key points.Firstly,effect of I/O parameters on total running time of the application;secondly,invalid assumptions of Hadoop scheduler that three phases (copy,sort and reduce) of a Reduce task are equal;finally,an insight supported by the results from the experiments on ways to improve the Hadoop scheduler for running multiple jobs by capturing the resource consumption information of different applications.To the best of our knowledge this is the first work that presents resource usage study.

关键词： map reduce resource usage hadoop scheduler

作者: Zheyuan Liu Dejun Mu

作者单位: Control and Network Institute Northwestern Polytechnical University Xian, China

会议类型: 国际会议

会议名称: 2012 International Conference on Quality, Reliability, Risk, Maintenance, and Safety Engineering & The 3rd International Conference on Maintenance Engineering (2012质量,可靠性,风险,维修性及安全性工程国际会议(QR2MSE 2012 & ICME 2012))

会议地点: 成都

会议语种:英文

页码: 1506-1510

在线出版日期: 2012-06-15（万方平台首次上网日期，不代表论文的发表时间）