A Solution of Web Crawling Bottlenecks Based on Virtualization Technology
This paper investigates the feasibility of applying virtualization technology to web crawlers in order to solve crawling bottlenecks. Different from some mainstream solutions that focus on reforming crawling strategies or optimizing the crawler structure, we introduce virtualization technology which could consolidate multiple environments into a single server or PC to maximum hardware utilizations. After analyzing the advantages of adopting virtualization to deal with crawling bottlenecks, we simulate three servers with specialized function in one IBM server to measure the efficiency. Experiment indicates that this Server virtualization technology could improve server performance, increase crawling efficiency as well as save implementation costs.
virtualization technology web crawler crawling bottlenecks crawling efficiency
Jie Xin Zhiming Cui Xuefeng Xian Zhiping Zhang
The Institute of Intelligent Information Processing and Application Soochow University Suzhou,Jiangs Networking Centre Soochow University Suzhou,Jiangsu
国际会议
太原
英文
525-528
2011-02-26（万方平台首次上网日期，不代表论文的发表时间）