摘要
面向互联网的主题采集是情报获取的重要手段,面对爆发式增长的互联网信息资源,设计并实现一套由采集准备、URL分析及提取、模板学习、正文抽取等几阶段组成的主题采集工具,其中URL分析与提取采用基于链接类型的URL筛选方法,实现正文网页URL的筛选;模板学习和正文抽取部分采用基于DOM树的节点比对方法,完成模板的构建与正文抽取。实验结果表明,本文所提出的主题采集工具采集准确率较高,能够适应目前情报信息采集的需求。
Topic information collection based on the Internet is an important means of acquiring intelligence. A topic information crawler is designed and realized to deal with the explosive growth of Internet information resources. The crawler comprises stages of acquisition preparation, URL analysis and extraction, template learning, and text extraction. A URL filtering method based on link types is used in the URL analysis and extraction stage to filter the URLs of text - containing Web pages. A node comparison method based on the DOM tree is used in the template learning and text extraction stages to construct templates and extract text. Test results show that the topic information crawler has a high accuracy in gathering information, and thus can meet the current need for information acquisition.
出处
《图书情报工作》
CSSCI
北大核心
2014年第20期91-99,共9页
Library and Information Service
基金
上海市科技发展基金软科学研究项目"大数据环境下基于领域本体的情报处理分析方法研究--以钢铁行业为例"(项目编号:14692107100)研究成果之一
关键词
网络爬虫
主题采集
链接筛选
DOM树
Web crawler topic information acquisition link filtering DOM tree
