A NOVEL WEB PAGE DUPLICATION DETECTION FRAMEWORK

摘要：

There are a lot of redundant web pages on Internet. Based on tag statistic and text similarity comparison, we present a novel multilayer framework for detecting duplicated web pages in this paper. We propose two similarity text paragraphs detection algorithms and implement our framework. The experimental results show that our approach achieves high performance, which means that duplicated web pages can be efficiently detected simply by tag statistic and text comparison.

关键词： Duplication Detection Web Page Framework

作者: Zhongming Han Dagao Duan Hongzhi Liu Jianzhi Sun

作者单位: School of computer science and information engineering,Beijing Technology and Business University, Beijing, China

会议类型: 国际会议

会议名称: 2009 IEEE International Conference on Network Infrastructure and Digital Content(2009年IEEE网络基础设施与数字内容国际会议 IEEE IC-NIDC2009)

会议地点: 北京

会议语种:英文

页码: 374-378

在线出版日期: 2009-11-06（万方平台首次上网日期，不代表论文的发表时间）