Study on re-implement mechanism based on node-ability in Hadoop
Face to massive web log data, single log analysis has been unable to meet peoples pursuit of high efficiency, the traditional log analysis based on Hadoop has some drawbacks in resource utilization, etc. Therefore, based on traditional Hadoop task scheduler, a new tasks pre-scheduling mechanism based on node-ability is proposed, the size of the task block is determined according to the ability of each processing node. Thus, the conflicts between job execution time and system resource utilization is balanced , which can protect the jobs of the response time and effective enforcement mechanism to reduce the weight of the additional overhead and improve overall resource utilization.
web log analysis Hadoop scheduler mechanism
Yaohui WANG Tao YANG
School of Information Engineering Northeast Dianli University JilinJilin 132012, China
国际会议
哈尔滨
英文
1220-1223
2011-12-24（万方平台首次上网日期，不代表论文的发表时间）