A Unified Framework for IOOS Model Data Access

Project Team

Project Lead/Scientific PI: Sean C. Arms, UCAR/Unidata

Transition PI: Kyle Wilcox, Axiom Data Science

NOAA/NOS Technical Points of Contact: Gregory Seroka, John Kelley, Edward Myers, Aijun Zhang, Jiangtao Xu

Project Overview and Results

In the last decade, IOOS has made remarkable progress toward an interoperable framework for working with ocean forecast model data. Providers deliver data via community standard OPeNDAP services using Climate and Forecast (CF) metadata conventions, and developers then use standard libraries to build applications that will work with any structured grid model (e.g. ROMS, NCOM, HYCOM, Delft3D) without the need to write any model-specific code. An outstanding gap, however, is the ability to fully support unstructured grid (e.g. triangular mesh) models, which are now being used throughout IOOS for both inundation and full 3D operational ocean forecasts.

The need for better tools to work with unstructured grids has been apparent for some time: In 2006, NOAA and NSF co-sponsored a workshop, “Community Standards for Unstructured Grids,” which was attended by 30 representatives from academic, government, and commercial organizations. The consensus statement of the workshop was “The unstructured ocean model community should define a data model (in the language of OPeNDAP) and ensure the mapping of that model onto netCDF3 and HDF5 is functional”. The community subsequently developed the UGRID Conventions, which extend the CF metadata conventions to describe unstructured grid model output. It remains to embed these standards into the services most commonly used by the IOOS community.

While IOOS-supported services can be delivered and consumed by a variety of software components, Unidata’s THREDDS Data Server (TDS) is the most widely used in IOOS. The TDS not only provides OPeNDAP access, but provides the OGC Web Map Service to create maps and NetCDF Subset Service to extract datasets based on longitude/latitude and calendar date extents. The Unidata NetCDF-Java library, which underlies the TDS, also supports many client applications (e.g. Unidata IDV, Matlab NCTOOLBOX, University of Reading NcWMS, NASA Panoply).

This project aims to create a unified framework for IOOS model data by: a) extending Unidata’s NetCDF-Java library and THREDDS Data Server (TDS) to fully support the unstructured grid models, b) implementing support for Object Storage (such as Amazon S3 and Google Cloud Storage), and c) aiding in the installation and testing of the new services among data providers.

As of September 20th, 2021, TDS version 5.0 was released, the first version of the TDS to include Object Storage support. This was a critical component of the proposed framework and a significant milestone, enabling the TDS to serve model output from an AWS S3 bucket.

Development on native unstructured grid support in the NetCDF-Java library and TDS is nearing completion, but still requires extensive testing and evaluation, both internally and from end users and data providers.

The capstone goal of this project is to support users with deployment and implementation of the new technologies, which rely on the latest major release of the TDS. Throughout the transition, Unidata developers have received direct feedback and made iterative improvements to the framework. The next minor release of the TDS is now scheduled for May 2022 and will provide improved stability from the initial release. The following release, expected by the end of 2022, will contain feature-complete Object Store support as well as fully integrated unstructured grids into the technical stack, bringing long-needed functionality to the IOOS community.