Success Stories
Success Stories
SSAI Modernizes NASA's Earth Science Data and Information System (ESDIS) Metrics System by Optimizing Processing Time and Power Requirements
NASA's Earth Science Data and Information System (ESDIS) Metrics System (EMS) collects and organizes various data file metrics from Earth Observing System Data and Information System (EOSDIS) Data Providers. The metrics data and the analysis reports generated from EMS provides NASA managers with the information they need to determine how to best apply resources to support the science community. The SSAI-led EMS team successfully upgraded this major, multi-component system to meet the increasing demand for processing and power.
The SSAI-led EMS team developed and implemented a series of upgrades to meet current processing needs and drastically improve processing power; an effort that involved optimization of algorithms, as well as development and implementation of parallel processing and a dedicated prestaging system.
Overview
The ESDIS Metrics System (EMS) is a collection of servers, databases, Commercial Off-The-Shelf (COTS) software applications, and locally developed Oracle and Perl custom code designed to support ESDIS project management by collecting and organizing various metrics from EOSDIS Distributed Active Archive Centers (DAACs) and other Data Providers. SSAI-led teams are responsible for all EMS software, life-cycle processes, from development to production, including maintenance and upgrades.
The EMS processing system collects and processes raw logs from DAAC providers that include information from all ingest, archive, and distribution interfaces throughout EOSDIS. The logs are subsequently processed to become EMS metrics depicting the use of products and services stored in databases or delivered via the Internet. Since the EMS went live nearly two decades ago, there has been an exponential increase in the quantity of logs that are received and require processing. In response, the SSAI-led EMS team developed and implemented a series of upgrades to meet current processing needs and drastically improve processing power. This effort involved optimization of algorithms, as well as development and implementation of parallel processing and a dedicated prestaging system.
Results
Since parallel processing and prestaging were implemented in EMS, daily processing time has been reduced from 20+ hours, which is nearly full capacity, to ~ 10 hours. This 50% decrease in processing time has enabled massive reprocessing efforts that were considered impossible before these performance improvements were implemented. Moreover, these improvements not only upgraded system performance, but also provided early validation of raw records and improved error notification to log providers.
Due to these upgrade efforts by the SSAI-led EMS team, EMS processing capacity now effectively meets the demand. Furthermore, EMS can automatically handle large amounts of reprocessing, without affecting routine daily processing, which saves further time, effort, and resources.