摘要
在大量变化着的数据中,数据分析师常常只关心预测结果为特定值的少量数据.然而,利用机器学习模型进行推理的工作流程中,由于机器学习算法库默认数据以单表方式组织,用户必须先通过SQL语句查询出全部数据,即使随后在模型推理过程中会将大量数据丢弃.指出了在这个过程中,如果可以预先从模型中提取信息,就有望能在数据获取阶段快速排除不需要的数据,从而降低数据获取过程中的多表连接代价、进程间通信代价以及模型预测代价,进而加速整个工作流程.以决策树模型为例,首先提出一种预筛选+验证的执行方法对查询过程进行优化,之后给出了从决策树中提取用于预筛选谓词的离线算法,最后在真实数据集上进行测试.实验结果表明,所提出的方法能够对借助决策树模型推理结果对数据进行筛选的应用场景起到较好的加速效果.
In a large number of changing data,data analysts often only care about a small amount of data with specific prediction results.However,users must query all the data by SQL before inference step,even if a large amount of data will be dropped,because the machine learning algorithm libraries always assume that the data is organized in a single table.This study points out that in this process,if some hints can be gotten from model in advance,it is expected that unnecessary data can be quickly eliminated in the data acquisition phase,thus reducing the cost of multi-table join,inter-process communication,and model prediction.This work takes a specific kind of machine learning model,i.e.,decision tree,as an example.Firstly,a pre-filtering and validation execution workflow is proposed.Then,an offline algorithm is used to extract pre-filtering predicates from the decision tree.Finally,the algorithm is tested on real world dataset.Experiments show that the method proposed in this study can accelerate the execution of SQL queries containing predicates on decision tree prediction result.
作者
钮泽平
李国良
NIU Ze-Ping;LI Guo-Liang(Department of Computer Science and Technology,Tsinghua University,Beijing 100084,China)
出处
《软件学报》
EI
CSCD
北大核心
2021年第3期622-635,共14页
Journal of Software
基金
国家自然科学基金(61925205,61632016)。
