Writing GRIB to an FDB target

This example demonstrates how to write earthkit-data GRIB fields into an FDB.

FDB (Fields DataBase) is a domain-specific object store developed at ECMWF for storing, indexing and retrieving GRIB data. For more information on FBD please consult the following pages:

• FDB

• pyfdb

Setting up the target FDB

In this example we will create an FDB in the current folder using the schema taken from the pyfdb test suite. To do so first we need to ensure the directory exists. Next, we have to specify the configuration.

import os

fdb_schema = "../default_fdb_schema"
fdb_dir = "./_fdb_target_demo"
os.makedirs(fdb_dir, exist_ok=True)

config = {
    "type": "local",
    "engine": "toc",
    "schema": fdb_schema,
    "spaces": [{"handler": "Default", "roots": [{"path": fdb_dir}]}],
}

Working with FDB requires pyfdb and fdb to be installed. The path to the fdb installation should also be set e.g. via the FDB5_DIR environment variable.

Getting the input data

import earthkit.data as ekd

ds = ekd.from_source("sample", "tuv_pl.grib").to_fieldlist()

Using to_target() on the data object

We use to_target() to write the GRIB fieldlist/field into the FDB. The encoder is automatically guessed from the input data.

# writing a field
ds[0].to_target("fdb", config=config)

# writing a whole fieldlist
ds.to_target("fdb", config=config)

# checking the result
request = {
    "class": "od",
    "expver": "0001",
    "stream": "oper",
    "date": "20180801",
    "time": 1200,
    "domain": "g",
    "type": "an",
    "levtype": "pl",
    "levelist": 500,
    "step": 0,
    "param": [131, 132],
}

ekd.from_source("fdb", request, config=config, stream=False).to_fieldlist().ls()

	parameter.variable	time.valid_datetime	time.base_datetime	time.step	vertical.level	vertical.level_type	ensemble.member	geography.grid_type
0	u	2018-08-01 12:00:00	2018-08-01 12:00:00	0 days	500	pressure	0	regular_ll
1	v	2018-08-01 12:00:00	2018-08-01 12:00:00	0 days	500	pressure	0	regular_ll

# setting GRIB keys for the output
ds.to_target("fdb", config=config, metadata={"date": 20250108})

# checking the result
request_1 = dict(**request)
request_1.update({"date": 20250108})
ekd.from_source("fdb", request_1, config=config, stream=False).to_fieldlist().ls()

	parameter.variable	time.valid_datetime	time.base_datetime	time.step	vertical.level	vertical.level_type	ensemble.member	geography.grid_type
0	u	2025-01-08 12:00:00	2025-01-08 12:00:00	0 days	500	pressure	0	regular_ll
1	v	2025-01-08 12:00:00	2025-01-08 12:00:00	0 days	500	pressure	0	regular_ll

Using a Target object

# basic usage
target = ekd.create_target("fdb", config=config)
target.write(ds)
target.flush()

# can be used as a context manager, no need to call flush() in the end
with ekd.create_target("fdb", config=config) as target:
    target.write(ds)

# a fieldlist can be written field by field into the target
with ekd.create_target("fdb", config=config) as target:
    for f in ds:
        target.write(f)