摘要
分析了Cloudera公司推出的Impala实时查询引擎原理与架构,并深入比较Impala与传统MapReduce的性能与特点,针对Impala进行复杂大数据处理方面的不足,提出了MapReduce与Impala结合的大数据处理方法,通过使用MapReduce对Impala的输入数据进行预处理,利用MapReduce在复杂作业处理方面的长处弥补了Impala在这方面的不足。最后对电信手机上网日志进行大数据查询和分析计算实验,实验结果表明,在大数据查询性能方面,基于MapReduce与Impala结合的大数据处理速度比传统MapReduce快了一倍。特别地,在迭代查询实验中,基于MapReduce与Impala结合的处理方法超过传统MapReduce方法八倍以上。基于MapReduce与Impala结合的处理方法在单次查询中的效率仍然高于传统MapReduce;而在迭代查询中,MapReduce与Impala结合的处理方法远远地超过了MapReduce。因此,MapReduce与Impala结合的处理方法能够发挥Impala和Hadoop各自的优点,让处理效率远超传统MapReduce,对于复杂的大数据处理的能力高于Impala。
First of all,this paper analyzed the elements and architecture of Impala the big data real-time query engine released by Cloudera recently. Then it compared the feature and efficiency between traditional MapReduce and Impala. Based on the comparison,it discovered the disadvantages of Impala. After that it proposed a method to process data with both MapReduce and Impala： using MapReduce to preprocess incoming data of Impala. This method utilized the flexibility of MapReduce to cover the disadvantages of Impala. Comparative experiments on the access log generated by China telecom＇s daily wap traffice have proved that Impala is evidently faster than traditional MapReduce and the combination of MapReduce and Impala will run twice faster than the traditional MapReduce. Especially in iterative analysis,the combination of MapReduce and Impala shows its overwhelming superiority towards traditional MapReduce. Hence,it is concluded that the combination of MapReduce and Impala can adopt the advantage of each other. It outperform traditional MapReduce on Performance and Impala on flexibility on complex data processing.
出处
《计算机应用研究》
CSCD
北大核心
2015年第5期1330-1334,共5页
Application Research of Computers
基金
国家自然科学基金资助项目(61070015)
广东省自然科学基金资助项目(S2011010001754
S2012030006242)
广东省科技计划资助项目(2012B010100030)
