摘要
在ETL系统中,当数据生产速度遇到较大的波动时,为了使现有资源在有限的条件下极大程度地满足用户的使用需求,提出了基于资源紧迫度的ETL弹性调度机制。首先,构建基于历史数据的时间序列预测模型,用来预测数据生产速度;然后,根据各个ETL过程提交的资源清单和各个服务器负载情况,对单个用户的多个ETL过程的资源进行调整。该调度机制将资源调度问题转换为多维背包问题,提出了基于资源紧迫度的动态调度算法。实验结果表明,当用户资源对所有的ETL过程稀缺的时候,该调度算法会使资源得到更好的利用。通过实验验证了弹性调度比传统的调度方式在资源利用方面具有优势。
In the ETL system,when the data generation speed encounters large fluctuations,in order to make the existing resources meet the user’s needs to a great extent under limited conditions,this paper proposed an ETL flexible scheduling me-chanism based on resource urgency.First,it built a time series prediction model based on historical data to predict the speed of data generation.Then,according to the resource list each ETL process submits and the load of each server,it adjusted the resources of multiple ETL processes for a single user.This scheduling mechanism turned resource scheduling problem into a multi-dimensional knapsack problem,and proposed urgency based dynamic schedule algorithm.Experimental results show that when user’s resources are scarce for all ETL processes,the proposed scheduling algorithm makes better use.The experiments verify that the flexible scheduling has advantages over the traditional scheduling method in resource utilization.
作者
王萌
刘旋律
高峰
顾进广
Wang Meng;Liu Xuanlyu;Gao Feng;Gu Jinguang(College of Computer Science&Technology,Wuhan University of Science&Technology,Wuhan 430065,China;Hubei Key Laboratory of Intelligent Information Processing&Real Time Industrial System,Wuhan University of Science&Technology,Wuhan 430065,China;Big Data Science&Engineering Research Institute,Wuhan University of Science&Technology,Wuhan 430065,China;Key Laboratory of Rich Media Digital Publishing Content Organization&Knowledge Service,State Administration of Press&Publication,Beijing 100038,China)
出处
《计算机应用研究》
CSCD
北大核心
2021年第7期2118-2124,共7页
Application Research of Computers
基金
国家社会科学基金重大项目(11&ZD189)
国家自然科学基金资助项目(61673304)
国家自然科学基金通用联合基金资助项目(U1836118)。
关键词
实时ETL
弹性调度
资源调度
多维背包
real time ETL
elastic scheduling
resource scheduling
multidimensional backpack problem
