A Web Crawler Detection Algorithm Based on Web Page Member List

摘要：

Following the widely use of search engines, the impact Web crawlers have on the Web sites should not be ignored. After analyzing the navigational patterns of Web crawlers from Web logs, a new algorithm based on Web page member list is proposed. The algorithm constructs one member list for every Web page and one show table for every visitor. The experiment shows that the new algorithm can detect the unknown crawlers and unfriendly crawlers who do not obey the Standard for Robot Exclusion.

关键词： Search engine Web crawler detection Web page member list Web log

作者: Weigang Guo Yong Zhong Jianqin Xie

作者单位: School of Electronic and Information Engineering Foshan University Foshan, Guangdong Province, China

会议类型: 国际会议

会议名称: 2012 4th International Conference on Intelligent Human-Machine Systems and Cybernetics 第4届智能人机系统与控制论国际会议 IHMSC 2012

会议地点: 南昌

会议语种:英文

页码: 189-192

在线出版日期: 2012-08-26（万方平台首次上网日期，不代表论文的发表时间）