Objective Evaluation of Spider Detection Techniques

摘要：

Spider is a program for harvesting internet resources. Spiders Detection Techniques(SDT) are used to regulate and monitor behaviors of spiders visiting website. In this paper, an Evaluation Method based on Trap technique(EMT) is proposed to calculate the recall rate and precision rate of SDT. Without relying on manual analysis, it is more objective and more adaptive to the development of SDT. The principles of EMT bases on the statistical hypothesis that the distribution of users captured by trap obeys binomial distribution theory. The experiment of EMT indicates three conclusions: (1)EMT has the consistent conclusion with the manual analysis result. (2)EMT is little impacted by time span of analysis.(3)EMT is little impacted by the traps layout rate when it changes in ±10％.

关键词： spider detection binomial distribution trap layout rate evaluation

作者: Fan Chunlong Yu Zhouhua Xu Lei

作者单位: School of Computer, University of Shenyang Aerospace, City of Shenyang

会议类型: 国际会议

会议名称: 2010 IEEE International Conference Conferenhce on Wireless Communications,Networking and Information Security(2010 IEEE 无线通信、网络技术与信息安全国际会议 WCNIS)

会议地点: 北京

会议语种:英文

页码: 1-5

在线出版日期: 2010-06-25（万方平台首次上网日期，不代表论文的发表时间）