A Mapreduce Task Scheduling Algorithm for Deadline-constraint in Homogeneous Environment

摘要：

　　The current works about task scheduling with deadline-constraint in homogeneous environment rarely take the differences of Map and Reduce task and data locality into account in the same scheduler.To address this problem,we introduce a scheduling algorithm that Map and Reduce are regarded as two separated stages of scheduling problem in homogeneous environment.For the sake of realizing this algorithm,five aspects that are average execution time of map/reduce tasks,map/reduce stage deadline,remaining time of map/reduce stage,jobs priority and data locality must be taken into consider.Compared with other real-time scheduling algorithm,we propose several methods which are one-to-one sampling,estimating requirements of resource and compromised task-data matching strategy to solve above five aspects.The experimental results show the sampling method can get accurate map/reduce task execution time and the proposed scheduling algorithm not only satisfies the jobs real-time requirement but also improves the throughput of cluster.

关键词： mapreduce scheduling algorithm data locality deadline-constraint hadoop

作者: Yi Yang Jiao Xu Fei Wang Zhaocai Ma Jingshan Wang Lian Li

作者单位: School of information science and engineering Lanzhou University Lanzhou,Gansu,China 730000

会议类型: 国际会议

会议名称: 2014 2nd International Conference on Advanced Cloud and Big Data (CBD 2014)(2014年先进云计算和大数据国际会议)

会议地点: 安徽黄山

会议语种:英文

页码: 208-212

在线出版日期: 2014-11-20（万方平台首次上网日期，不代表论文的发表时间）