A Crawler Guard for Quickly Blocking Unauthorized Web Robot
Nowadays Web robots can be used to perform a number of useful navigational goals, such as statistical analysis, link check, and resource collection.On one hand, Web crawler is a particular group of users whose traverse should not make part of regular analysis.Such disturbance affects site decision making in every possible way: marketing campaigns, site re-structuring, site personalization or server balancing, just to name a few.Therefore, it is necessary to correctly detect various robots as soon as possible so as to let the robots to be used under the security policy.In this paper, we come up with a crawler guard to detect and block unauthorized robots under the security policy.It can immediately differentiate various robots based on their functions (navigational goals) to ensure that only the welcome robots which obey the security policy are allowed to view the protected Web pages.Our experiment focuses on how the crawler guard could identify precisely the viewing goal of the robots under certain limits of Web page hits.The experimental results show that the request count is smaller than 8 while the accuracy of detection is 100%.
Web Crawler Web Robot Navigational Behavior
Jan-Min Chen
The Dept.of Information Management,Yu Da University of Science and Technology,Miaoli 36143,Taiwan
国际会议
The 5th International Symposium on Cyberspace Safety and Security ( CSS2013)(第五届国际网络空间安全和安保研讨会)
张家界
英文
1-13
2013-11-13（万方平台首次上网日期，不代表论文的发表时间）