High Fault-Tolerance for MapReduce Application through Hierarchical Master-Worker Framework

摘要：

Over the past few years, the top web companies like Google, Yahoo etc have come up with several special techniques that process large amounts of raw data. The issue is how to handle large computations and large data. MapReduce is a programming framework developed by Google for parallel processing on largescale datasets. Hadoop is designed on Master-worker framework. The key problem of master-worker architectural is that master node is in a very important location. Once the master crushes down, the whole framework will stop working. In this paper, we focus on a specific type of optimization: fault tolerance and master recovery of Map-Reduce jobs. Finally, we propose a solution using hierarchical master-worker framework. The experiments demonstrate the feasibility and efficiency of our solution.

关键词： MapReduce Hierarchical Master-Worker Framework Fault-Tolerance

作者: Zhou Yike Wang Dong

作者单位: E-Logistics Information and RFID Research Institute, Shanghai Jiao Tong University, Shanghai 200240, China

会议类型: 国际会议

会议名称: 2011 International Conference on Information and Industrial Electronics(2011年信息与工业电子国际会议 ICIIE 2011)

会议地点: 成都

会议语种:英文

页码: 474-477

在线出版日期: 2011-01-14（万方平台首次上网日期，不代表论文的发表时间）