Application of VM-Based Computations to Speedup the Web Crawling Process on Multi-Core Processors
A Web crawler is an important component of the Web search engine.It demands large amount of hardware resources to crawl data from the rapidly growing and changing Web.The crawling process should be performed continuously to maintain up-to-date data.This paper develops a new approach to speed up the crawling process on a multi-core processor by utilizing the concept of virtualization.In this approach,the multi-core processor is divided into a number of virtual-machines(VMs),which can concurrently perform different crawling tasks on different initial data.It presents a description,implementation,and evaluation of a VM-based distributed Web crawler.The speedup factor achieved by the VM-based crawler over no virtualization crawler,for crawling various numbers of documents,is estimated.Also,the effect of number of VMs on the speedup factor is investigated.
Web search engine Web crawler virtualization virtual machines distributed crawling multi-core processor processor-farm methodology
Hussein Al-Bahadili Hamzah Qtishat
Faculty of Information Technology University of Petra Amman,Jordan Faculty of Information Technology Middle East University Amman,Jordan
国际会议
英国伦敦
英文
157-161
2013-09-02（万方平台首次上网日期，不代表论文的发表时间）