摘要
为了减少重复网页对用户的干扰,提高去重效率,提出一种新的大规模网页去重算法。首先利用预定义网页标签值建立网页正文结构树,实现了层次计算指纹相似度;其次,提取网页中高频标点字符所在句子中的首尾汉字作为特征码;最后,利用Bloom Filter算法对获取的特征指纹进行网页相似度判别。实验表明,该算法将召回率提高到了90%以上,时间复杂度降低到了O(n)。
In order to reduce the interference of the duplicated Web pages, and improve the efficiency of detection and elimination of similar Web pages, a new kind of large-scale Web page detection algorithm was proposed. Firstly, adopting the Web label values, the algorithm created the text structure trees to realize the fingerprint similarity calculation layer by layer. Secondly, the head and tail words of a certain sentence, in which high frequency punctuations occur, were extracted out as the feature code. Lastly, the fingerprint similarity of Web page features was discriminated with Bloom filter algorithm. The experimental results show that the algorithm can improve the recall rate up to more than 90%, and reduce the time complexity to O（n）.
出处
《计算机应用》
CSCD
北大核心
2013年第2期554-557,共4页
journal of Computer Applications
关键词
网页去重
网页标签值
高频标点
特征码
网页指纹相似度
detection and elimination of similar Web pages
Web label value
high frequency punctuation
feature code
fingerprint similarity of Web page
