摘要
试飞数据是民机飞行试验的重要产物,具有测量参数数量大,数据体量大,飞行试验数据与试飞任务信息关联性强等特征,支撑飞机型号取证与设计优化等任务;试飞数据平台数据架构对多源异构数据集成接入、多类形态数据存储管理、多种层次数据处理分析等技术进行了研究,采用湖仓一体的关键技术和方法打造试飞数据全集;试飞数据接入采用流批一体的数据处理技术,融合Spark和Flink主流数据处理引擎,具备试飞数据快速入库能力;提出按秒聚合方法,具备PB级多维度试飞数据压缩存储功能,存储性能提升近10倍;采用以秒为索引条件支持快速检索,强化数据湖查询能力;研究数据仓库技术,设计试飞数据多层数据模型,具备多维信息精细查询,多层数据灵活钻取,多功能自定义函数集成等功能,并成功应用在某型国产民机的飞行试验数据管理中,服务于试飞数据用户,提高了试飞数据管理效率与试飞数据应用价值。
Flight test data is an important output of civil aircraft flight tests,it has the characteristics of many measurement pa-rameters,large data volume,and strong correlation between flight test data and test flight mission information,supporting aircraft type certification and design optimization tasks.The data architecture of the flight test data platform integrates multiple heterogeneous data sources,manages various types of data in different formats,and provides multi-level data processing and analysis functions,crea-ting a unified data lakehouse for flight test data.The data processing technology used for flight test data integration adopts a hybrid approach of stream-batch integration,incorporating the mainstream data processing engines such as Spark and Flink,with the ability to quickly ingest the flight test data into the data platform.The platform proposes a method of aggregation at the second level,with PB-level multidimensional flight test data compression and storage capabilities,the storage performance is improved by nearly 1o times,supporting fast retrieval based on second-level indexing conditions and enhancing data lake query capabilities,Data warehousing technology is also studied,a multi-layer data model for flight test data is designed,supporting the fine-grained queries for multidimen-sional information,flexible drilling down into multiple layers of data,and integration of custom functions,which is successfully ap-plied in the management of flight test data for a certain type of domestic civil aircraft,serving flight test data users and improving the efficiency and value of flight test data management.
作者
邓国宝
查晓文
冯灿
张逸飞
薛博文
DENG Guobao;ZHA Xiaowen;FENG Can;ZHANG Yifei;XUE Bowen(Flight Test Center Flight-Test Instrumentation Dept.,COMAC,Shanghai 200232,China)
出处
《计算机测量与控制》
2023年第12期271-276,共6页
Computer Measurement &Control
关键词
试飞数据
数据架构
湖仓一体
流批一体
按秒聚合
数据仓库
flight data
data architecture
lake house
stream-batch integration
second-level aggregation
data warehouse
