Fixing the Threshold for Effective Detection of Near Duplicate Web Documents in Web Crawling
The drastic development of the WWW in recent times has made the concept of Web Crawling receive remarkable significance. The voluminous amounts of web documents swarming the web have posed huge challenges to web search engines making their results less relevant to the users. The presence of duplicate and near duplicate web documents in abundance has created additional overheads for the search engines critically affecting their performance and quality which have to be removed to provide users with the relevant results for their queries. In this paper, we have presented a novel and efficient approach for the detection of near duplicate web pages in web crawling where the keywords are extracted from the crawled pages and the similarity score between two pages is calculated. The documents having similarity score greater than a threshold value are considered as near duplicates. In this paper we have fixed the threshold value.
Fingerprint Similarity score Near-duplicate Web `crawling and Threshold
V.A. Narayana P. Premchand A. Govardhan
Department of Computer Science & Engineering,CMR College of Engineering & Technology,Hyderabad India Department of Computer Science & Engineering University College of Engineering Osmania University Hy JNTUH College of Engineering Nachupally (Kondagattu) Karimnagar Dist AP India
国际会议
6th International Conference on Advanced Data Mining and Applications(第六届先进数据挖掘及应用国际会议 ADMA 2010)
重庆
英文
169-180
2010-11-19（万方平台首次上网日期，不代表论文的发表时间）