摘要
分析了现有搜索引擎采集器过滤互联网上重复URL的技术,指出了它们在速度和效率方面存在的不足,借鉴了操作系统请求页式存储管理中的页面置换算法,提出了页置换技术在搜索引擎采集器中高数量级过滤重复地址的应用。使用了虚拟存储技术,使得容量不大的内存可以为众多的进程提供资源,该技术较好地解决了在有限系统资源下进行亿量级URL过滤的速度和效率问题。
The technology of filter repeat Intemet URL for the existing search engine collector is analyzed. The shortcomings of speed and efficiency is pointed out, virtual memory page replacement algorithm of virtual memory manager is applied. The page-replacement technology in the search engine collector of high-volume applications address filtering repeat is brought in. The use of the virtual storage technology, make small memory capacity is provided for a large number of process resources. The technology can solve the speed and efficiency problem for billion URL filtering in the limited system resource.
出处
《计算机工程与设计》
CSCD
北大核心
2009年第5期1137-1139,共3页
Computer Engineering and Design
关键词
搜索引擎
采集器
页置换
过滤
指纹
URL
search engine
collector
page-replacement
filter
finger mark
URL
