摘要
以某省电网实时数据监测系统的数据中心建设项目为背景,提出了一种轻量级的,适合于电网实时数据迁移与同步更新的ETL系统解决方案。系统支持多种关系型数据库和数据文件,提供全量、增量、主细表等多种自定义模式数据迁移方案。系统利用JDBC数据库访问技术、JSR-166的Util.Concurrent并发线程包以及SWT技术,结合元数据的管理,解决了异构数据的快速抽取、清洗转换与加载、任务的调度和跨平台运行的问题,并在实际应用中表现出了实用性和稳定性。
Using the data center construction project for a power real-time monitoring system as the background, this paper proposes a novel lightweight ETL solution for the migration and synchronization of power real-time data. The solution not only supports a variety of relational database and text-based data sources, but also provides total, incremental, main-sub table and several other custom mode data migration schemes. By combining JDBC database access technology, Util.Concurrent package of JSR-166 and SWT technology with the management of metadata, our system solves the problems of heterogeneous data extraction, cleaning, transformation and loading, task scheduling and cross-platform operation. Practical applications demonstrate the practicality and stability of this system.
出处
《电力系统保护与控制》
EI
CSCD
北大核心
2010年第18期174-177,182,共5页
Power System Protection and Control
关键词
ETL系统
异构数据
增量
任务调度
跨平台
ETL system
heterogeneous data
incremental
task scheduling
cross-platform
