摘要
交通流数据具有数据海量、存储和交互速率快等特征,因此其数据的采集、存储及检索成为了车辆远程监控平台中的关键问题。采用LVS集群技术进行数据采集负载均衡,队列缓存处理I/O时延,HBase进行分布式数据存储;针对Hadoop实时在线数据处理不足的问题,整合Elastic Search并构建了分层索引。通过关键技术的设计和实现,车辆监控由400辆扩展到上万辆,PB级数据在线查询速度提升了10~20倍,验证了方案的高效性。
Traffic data has the characteristics of massive and real-time, and its massive data acquisition, storage and retrieval has become a key issue in the vehicle remote monitoring platform. According to the study of these problems, the cluster technology of LVS was used to solve the data acquisition load balance, the queue cache model was used to solve I/O delay, and HBase distributed data storage scheme was used to solve the massive data storage. HBase integration ElasticS earch, which was aimed to solve the real-time online data processing problems of Hadoop, was designed to build a hierarchical index. Through the design and implementation of the key technologies, the number of vehicle monitoring had been promoted from 400 to 1 million, online query speed increased about 10 to 20 times based on PB level data. The results verified the efficiency of the scheme.
出处
《大数据》
2017年第1期80-89,共10页
Big Data Research
