摘要
网络技术快速发展,信息资源多种多样,面对具有范围广、信息量大等特点的地震数据共享平台,如何快速又准确地抓取有用的相关信息十分重要,因此爬虫为该地震数据共享平台的重要技术之一。构建使用Docker部署的基于Golang的Crawlab的爬虫框架进行数据爬取及数据处理的地震数据共享平台,解决传统爬虫技术在面对庞大数据量时会出现的效率低下问题。
With the rapid development of network technology and a variety of information resources,the seismic data sharing platform with wide range and large amount of information is facing,it is very important to grasp useful information quickly and accurately in the face of the seismic data sharing platform with wide range and large amount of information.Therefore,crawler is one of the important technologies of the seismic data sharing platform.A crawler framework of Crawler based on Golang deployed by Docker is constructed to share seismic data for data crawling and data processing,so as to solve the inefficient problem of traditional crawler technology in the face of huge data volume.
作者
黄诗涵
韦廷岳
靳一飞
黄猛
HUANG Shihan;WEI Tingyue;JIN Yifei;HUANG Meng(Institute of Disaster Prevention,Sanhe 065201,China)
出处
《现代信息科技》
2020年第2期28-29,共2页
Modern Information Technology
基金
地震震害防御专项项目
中央高校基本科研业务费专项项目(ZY20180124)。
