摘要
随着国产数据库技术的发展,电网企业对相应数据库及中间件的应用逐步增多,业务中产生的大量的运维信息对国产数据库的自动化运维技术要求也随之提高。针对国产数据库和中间件产生的海量运维数据和日志,文中提出一种基于Ansible和Flume的高效数据采集系统。该系统通过调用Python脚本的方式定期从Ansible Inventory中拉取各主机的运行状态信息,同时由Flume配置采集agent到各数据库和中间件服务器中,进行日志数据的采集、预处理和下放,并将数据输出到数据库中,实现对数据库和中间件运行数据的自动化采集与管理。经系统测试,该系统可以高效完成多线程Web访问,并在数据采集方面表现出良好的性能。
With the development of domestic database technology,more and more domestic databases and middleware are used by power grid enterprises. The large amount of operation and maintenance information generated in power grid business demands more and more high for the automated operation and maintenance technology of domestic databases. Aiming at the massive operation and maintenance data and logs generated by domestic databases and middleware,an efficient data acquisition system based on Ansible and Flume is proposed in this paper. By calling Python script,the system regularly pulls the running status information of each host from Ansible Inventory. At the same time,Flume configures the acquisition agent to each database and middleware server,collects,preprocesses and downloads the log data,and outputs the data to the database. Realize the automatic collection and management of database and middleware operation data. Through system testing,the system can efficiently complete multi-threaded Web access and show good performance in data acquisition.
作者
赵创业
唐亮亮
郭威
王哲
黄晓波
ZHAO Chuang-ye;TANG Liang-liang;GUO Wei;WANG Zhe;HUANG Xiao-bo(Information Center,Guangdong Power Grid Co.,Ltd.,Guangzhou 510030,China)
出处
《电子设计工程》
2020年第3期47-51,共5页
Electronic Design Engineering
基金
广东电网科技项目(037800KK52170001)。
