摘要
随着新闻网信息的急剧增加,研究面向新闻网的智能抓取技术变得十分必要。在基于主题网络爬虫的基础上,改进了网页与主题的相关度算法,设计出面向新闻网的智能抓取技术。按照预先确定的主题,分析超链接和刚刚抓取的网页内容,获取下一个要爬行的URL。有选择的访问新闻网上的网页和相关的链接,尽可能保证多爬行与主题相关的网页。实验结果表明,改进后的抓取技术比以往的新闻网抓取技术在性能上有较大提升,抓取准确率显著提高。
With the rapid increase of network information, it becomes necessary to research intelligent fetching technology for news. In this paper, on the basis of topic web crawler, it improved web page with the theme of correlation algorithm, designed news oriented in- telligent fetching. According to the predetermined theme, it should be analyzed hyperlinks and grab web content to obtain the next craw- ling URL. Web pages online and related links are selective to access, the related subject web pages are crawled as much as possible. The experiment results show that the improved technology was highly promoted in the performance and the grab significantly accuracy compared to the previous technology.
出处
《控制工程》
CSCD
北大核心
2013年第4期707-710,共4页
Control Engineering of China
基金
国家自然科学基金(61272179)
中央高校基本科研业务费专项资金资助(N110316001)
辽宁省自然科学基金(20092005)
关键词
主题提取
网络爬虫
页面相关度
topic distillation
web crawler
page relevance
