摘要
地理国情监测数据具有精度高、数据量大等特性,随着新型硬件构架(单机多核、集群等)计算机资源能力的不断升级,传统的数据处理技术和串行计算技术难以满足高精细化地理国情大数据处理的需求。对此,本文提出了一种基于TPL和管道技术的多进程并行计算方法。通过实例验证,在同一台计算机上该方法比串行方法的计算效率提高了12倍;并且在此基础上,实现了计算节点的负载均衡及可扩展,满足了多核应用,有效地提高了地理国情监测大数据处理的实际效率。
High precision and large amount of data are the characteristics of national geographic conditions monitoring data.With the development of the new hardware architecture（multi-core computing, cluster computing,etc.）and constant upgrade of the computer resources,traditional data processing techniques and serial computing technology are difficult to meet the requirements of Big Data processing for high-precise national geographic conditions monitoring.To solve this problem,the paper proposed a multiprocess parallel computing method based on TPL and Named Pipes technology.The results showed that this approach could improve CPU utilization with computing speed increasing by 12times.Moreover,the method could realize the load balancing and extendibility of compute nodes and effectively improve the processing efficiency of national geographic conditions monitoring data.
出处
《测绘科学》
CSCD
北大核心
2014年第5期13-17,共5页
Science of Surveying and Mapping
基金
国家科技支撑计划项目(2012BAH28B03)
关键词
地理国情基本统计
并行计算
大数据
多进程
管道技术
basic statistics of national geographic conditions
parallel computing
Big Data
multiple process
pipeline technology
