摘要
大数据量的实时数据分析系统,需要快速的处理和响应。为了实现实时数据分析,本文设计了基于Spark Streaming的实时数据采集分析系统,并对有状态计算操作的基数计算的精确计算方法和估算方法进行了比较分析。实验表明,基于HyperLogLog++的基数估算方法在处理时间和存储占用空间上有明显优势,而计算偏差基本可以忽略不计,更适于大数据的基数估算。
Real - time data analysis systems with large amounts of data require fast processing and response. In order to realize real - time data analysis, this article designs a real - time data collection and analysis system based on Spark Streaming and analyzes the per- formance between exactly and approximately counting in cardinality counting for the stateful operation. Experiments show that the cardi- nality estimation method based on HyperLogLog ＋ ＋ has obvious advantages in processing time and storage space, and the impact caused by relative deviation is negligible, which is more suitable for the cardinality counting in real - time large data analysis.
出处
《网络新媒体技术》
2017年第5期48-53,共6页
Network New Media Technology
基金
中国科学院战略性先导科技专项:新型传播技术研究与系统研制(XDA06040602)
