期刊文献+

基于ElasticSearch的大日志实时搜索的软件集成方案研究 被引量:24

The Design of Software Integration for Big Log Data Real Time Search based on ElasticSearch
下载PDF
导出
摘要 现代企业每天生成很多日志文件,如果能实时处理日志数据,企业能获取更大的商业价值,但管理这个大日志数据是一个巨大的挑战,因为传统的技术用来处理庞大的数据不够高效.Hadoop生态系统提供一种新的方式来处理大数据,ElasticSearch技术是基于云环境的实时搜索引擎.本文提出了基于ElasticSearch实时进行大日志数据搜索的软件集成方案,采用基于硬件创建虚拟机环境,根据搜索条件使用ElasticSearch得到需要的rowkey列表,Hbase用这些rowkey直接从数据库中得到数据.实验证明,随着日志事件搜索量的增加,搜索反应时间不线性增加,基于ElasticSearch的大日志实时搜索的软件集成方案设计具有可行性. Modern enterprise generates a lot of log files, the enterprise can obtain greater business value if it can be real-time processing of log data. But manage the big log data is a huge challenge because the traditional technology is not efficiently to deal with the huge data. Hadoop ecosystem provides a new way to deal with large data, the ElasticSearch technology is a real-time search engine based on the cloud. The integration scheme is proposed in this paper based on ElasticSearch real-time search software. The virtual machine environment is created base on the hardware, and the needed rowkey list is obtained by using ElasticSearch and the search conditions, then the Hbase use these rowkey to get data directly from the database. The experiments show that the search response time is not linear increase with the increment of log event search, and big log real-time search based on ElasticSearch is feasible for the integration of the software design.
作者 白俊 郭贺彬
出处 《吉林师范大学学报(自然科学版)》 2014年第1期85-87,共3页 Journal of Jilin Normal University:Natural Science Edition
基金 国家自然科学基金项目(60973011)
关键词 ElasticSearch 大数据 HBASE 实时搜索 软件集成方案 ElasticSearch big data Hbase real-time search software integration solutions
  • 相关文献

参考文献4

二级参考文献22

  • 1曾理,王以群.Hadoop集群和单机数据处理的耗时对比实验[J].硅谷,2009,2(19):55-56. 被引量:9
  • 2王学伟,马明栋.Dot Net框架下实现对象持久化的基本原理[J].电脑知识与技术(技术论坛),2005(12):109-110. 被引量:1
  • 3汪廷华,程从从.一种元规则指导的股票联动关联规则挖掘算法[J].计算机工程,2006,32(5):260-262. 被引量:3
  • 4Mikhail Garber, sunjune. Use search engine technology for object persistence, http ://www. javawodd. com/javawodd/jw-01-2005/jw-0103-search. html.
  • 5Jesse Liberty, Dan Hurwitz. Programming ASP. NET 3rd Edition: Publishing House of Electronics Industry,2006.
  • 6Murugesan S. Understanding Web 2.0[J]. IT Professional, 2007, 9(4): 34-41.
  • 7Oreilly T. What is Web 2.0: design patterns and business models for the next generation of software[J]. International Journal of Digital Economics, 2007, 65(3): 17-37.
  • 8Stonebraker M. SQL databases v. NoSQL databases[J]. Com- munications of the ACM, 2010, 53(4): 10-11.
  • 9Chang F, Dean J, Ghemawat S, et al. Bigtable: a distributed storage system for structured data[C]//Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI '06). Berkeley, CA, USA: USENIX Association, 2006:15.
  • 10White T. Hadoop: the definitive guide[M]. 2nd ed. [S.1.]: Yahoo Press, 2010.

共引文献29

同被引文献163

引证文献24

二级引证文献150

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部