Quick Start

Installation

Prerequisites

Build and Install

Option 1 (Manual Install)

git clone https://github.com/NOAA-EMC/bufr-query.git
mkdir bufr-query/build
cd bufr-query/build
ecbuild -DCMAKE_INSTALL_PREFIX=<my install dir> -DPython3_EXECUTABLE=`which python` ..
make install

# Extend the python path (may want to add to your shell configuration files)
export PYTHONPATH=<my install dir>/lib64/python<version>/site-packages:$PYTHONPATH

Note

You may also need to include other cmake arguments, such as -DCMAKE_PREFIX_PATH, if you installed any of the requirements manually.

Option 2 (via ObsForge)

Bufr-Query is installed as part of the NOAA EMC ObsForge project. Please follow their instructions.

Using It

There are many ways to run the BUFR-Query lib either on the command line, via your own python or C++ code, or via the JCSDA IODA framework to ingest data directly into a JEDI application.

Basically there are several steps involved in reading a BUFR file.

  1. Identify the data fields (by their queries) you want to read (see Query Path). It’s recommended to compile your set of queries into a description YAML file.

  2. Either use your YAML file directly, create an ObsBuilder module (recommended), or write a custom Python script to use your queries.

Command Line

bufr2netcdf.x

Executable that processes a BUFR file based on the provided yaml mapping file.

Examples:

# Single threaded
bufr2netcdf.x gdas.1bmhs.bufr_d mhs_mapping.yaml mhs.nc

# Using mpi
mpirun -n 4 bufr2netcdf.x gdas.1bmhs.bufr_d mhs_mapping.yaml mhs.nc

# Using slurm
srun -A <my account> -n 4 bufr2netcdf.x gdas.1bmhs.bufr_d mhs_mapping.yaml mhs.nc

# Splitting according to sat ID in yaml spec
bufr2netcdf.x gdas.1bmhs.bufr_d split_mhs_mapping.yaml mhs-{splits/satId}.nc

# Running WMO bufr file requires extra arguments
bufr2netcdf.x -t <BUFR table dir> wmo.bufr_d wmo_mapping.yaml wmo.nc

ObsBuilder

Obsbuilder modules can be executed from the command line directly by running them via python, by using IODA, or by importing the modules and using them directly (we do this in project Ocelot)

python my_obsbuilder.py  --input gdas.1bmhs.bufr_d  --output mhs.nc

The command line arguments will mirror those defined in the ObsBuilder method create_obs_file which you can optionally override (please see the ObsBuilder docs).

IODA

You can configure to load data directly from a BUFR file if the YAML file description is all you need.

If you have implemented an ObsBuilder module (or a custom python script that implement make_obs_group) you can use the IODA Script interface.

Examples

There are many examples available. Here is a list of a few places to look:

  • The test/testinput directory of bufr-query.

  • The NOAA EMC SPOC project (dump directory).

  • The NOAA EMC Ocelot project