Low-Load Server Crawler: Design and Evaluation

摘要：

This paper proposes a method of crawling Web servers connected to the Internet without imposing a high processing load. We are using the crawler for a field survey of the digital divide, including the ability to connect to the network. Rather than employing normal Web page crawling algorithm, which usually collect all pages found on the target server, we have developed server crawling algorithm, which collect only minimum pages from the same server and achieved low-load and high-speed crawling of servers.

关键词： Global Digital Divide Server crawler

作者: Katsuko T. Nakahira Tetsuya Hoshino Yoshiki Mikami

作者单位: Nagaoka University of Technology 16031 Kamitomiokamachi,Nagaoka Niigata, Japan Nagaoka University ofTechnology 16031 Kamitomiokamachi,Nagaoka Niigata, Japan

会议类型: 国际会议

会议名称: 第十七届国际万维网大会(the 17th International World Wide Web Conference)(WWW08)

会议地点: 北京

会议语种:英文

在线出版日期: 2008-04-21（万方平台首次上网日期，不代表论文的发表时间）