Retrieving data from FDB

import earthkit.data

FDB (Fields DataBase) is a domain-specific object store developed at ECMWF for storing, indexing and retrieving GRIB data. For more information on FBD please consult the following pages:

• FDB

• pyfdb

This example requires FDB access and the FDB_HOME environment variable has to be set correctly.

The following request was written to retrieve data from the operational FDB at ECMWF. Please note that the date must be adjusted since FDB at ECMWF only stores the most recent dates.

request = {
    'class': 'od',
    'expver': '0001',
    'stream': 'oper',
    'date': '20240421',
    'time': [0, 12],
    'domain': 'g',
    'type': 'an',
    'levtype': 'sfc',
    'step': 0,
    'param': [151, 167, 168]
}

Reading as a stream

By default we retrieve data from an FDB source with from_source() as a stream.

Iteration with one field at a time in memory

When we use the default arguments in from_source() the resulting object can only be used for iteration and only one field is kept in memory at a time. Fields created in the iteration get deleted when going out of scope.

ds = earthkit.data.from_source("fdb", request=request)
for f in ds:
    print(f)

GribField(msl,None,20240421,0,0,0)
GribField(2t,None,20240421,0,0,0)
GribField(2d,None,20240421,0,0,0)
GribField(msl,None,20240421,1200,0,0)
GribField(2t,None,20240421,1200,0,0)
GribField(2d,None,20240421,1200,0,0)

Once the iteration is completed, there is nothing left in ds.

sum([1 for _ in ds])

0

Iteration with group_by

When we use the group_by method we can iterate throught the stream in groups defined by metadata keys. Each iteration step results in a FieldList object, which is built by consuming GRIB messages from the stream until the values of the metadata keys change. The generated FieldList keeps GRIB messages in memory then gets deleted when going out of scope.

ds = earthkit.data.from_source("fdb", request=request)
for f in ds.group_by("time"):
    print(f"len={len(f)} {f.metadata(('param', 'level'))}")

len=3 [('msl', 0), ('2t', 0), ('2d', 0)]
len=3 [('msl', 0), ('2t', 0), ('2d', 0)]

Iteration with batched

When we use the batched method we can iterate throught the stream in batches of fixed size. In this example we create a stream and read 2 fields from it at a time.

ds = earthkit.data.from_source("fdb", request=request)
for f in ds.batched(2):
    print(f"len={len(f)} {f.metadata(('param', 'level'))}")

len=2 [('msl', 0), ('2t', 0)]
len=2 [('2d', 0), ('msl', 0)]
len=2 [('2t', 0), ('2d', 0)]

Storing all the fields in memory

We can load the whole stream into memory by using read_all=True in from_source(). The resulting object will be a FieldList.

ds = earthkit.data.from_source("fdb", request=request, read_all=True)

len(ds)

6

ds.ls()

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType
0	ecmf	msl	surface	0	20240421	0	0	an	0	reduced_gg
1	ecmf	2t	surface	0	20240421	0	0	an	0	reduced_gg
2	ecmf	2d	surface	0	20240421	0	0	an	0	reduced_gg
3	ecmf	msl	surface	0	20240421	1200	0	an	0	reduced_gg
4	ecmf	2t	surface	0	20240421	1200	0	an	0	reduced_gg
5	ecmf	2d	surface	0	20240421	1200	0	an	0	reduced_gg

ds.sel(param="2t").ls()

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType
0	ecmf	2t	surface	0	20240421	0	0	an	0	reduced_gg
1	ecmf	2t	surface	0	20240421	1200	0	an	0	reduced_gg

ds.to_xarray()

<xarray.Dataset>
Dimensions:     (number: 1, time: 2, step: 1, surface: 1, values: 6599680)
Coordinates:
  * number      (number) int64 0
  * time        (time) datetime64[ns] 2024-04-21 2024-04-21T12:00:00
  * step        (step) timedelta64[ns] 00:00:00
  * surface     (surface) float64 0.0
    latitude    (values) float64 ...
    longitude   (values) float64 ...
    valid_time  (time, step) datetime64[ns] ...
Dimensions without coordinates: values
Data variables:
    msl         (number, time, step, surface, values) float32 ...
    t2m         (number, time, step, surface, values) float32 ...
    d2m         (number, time, step, surface, values) float32 ...
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2024-04-22T11:01 GRIB to CDM+CF via cfgrib-0.9.1...

xarray.Dataset

• Dimensions:
  • number: 1
  • time: 2
  • step: 1
  • surface: 1
  • values: 6599680
• Coordinates: (7)
  • number
    (number)
    int64
    0

    long_name :

    ensemble member numerical id

    units :

    1

    standard_name :

    realization

    array([0])

  • time
    (time)
    datetime64[ns]
    2024-04-21 2024-04-21T12:00:00

    long_name :

    initial time of forecast

    standard_name :

    forecast_reference_time

    array(['2024-04-21T00:00:00.000000000', '2024-04-21T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • step
    (step)
    timedelta64[ns]
    00:00:00

    long_name :

    time since forecast_reference_time

    standard_name :

    forecast_period

    array([0], dtype='timedelta64[ns]')

  • surface
    (surface)
    float64
    0.0

    long_name :

    original GRIB coordinate for key: level(surface)

    units :

    1

    array([0.])

  • latitude
    (values)
    float64
    ...

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    [6599680 values with dtype=float64]

  • longitude
    (values)
    float64
    ...

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    [6599680 values with dtype=float64]

  • valid_time
    (time, step)
    datetime64[ns]
    ...

    standard_name :

    time

    long_name :

    time

    [2 values with dtype=datetime64[ns]]

• Data variables: (3)
  • msl
    (number, time, step, surface, values)
    float32
    ...

    GRIB_paramId :

    151

    GRIB_dataType :

    an

    GRIB_numberOfPoints :

    6599680

    GRIB_typeOfLevel :

    surface

    GRIB_stepUnits :

    1

    GRIB_stepType :

    instant

    GRIB_gridType :

    reduced_gg

    GRIB_N :

    1280

    GRIB_NV :

    0

    GRIB_cfName :

    air_pressure_at_mean_sea_level

    GRIB_cfVarName :

    msl

    GRIB_gridDefinitionDescription :

    Gaussian Latitude/Longitude Grid

    GRIB_missingValue :

    9999

    GRIB_name :

    Mean sea level pressure

    GRIB_pl :

    [20 24 28 ... 28 24 20]

    GRIB_shortName :

    msl

    GRIB_totalNumber :

    0

    GRIB_units :

    Pa

    long_name :

    Mean sea level pressure

    units :

    Pa

    standard_name :

    air_pressure_at_mean_sea_level

    [13199360 values with dtype=float32]

  • t2m
    (number, time, step, surface, values)
    float32
    ...

    GRIB_paramId :

    167

    GRIB_dataType :

    an

    GRIB_numberOfPoints :

    6599680

    GRIB_typeOfLevel :

    surface

    GRIB_stepUnits :

    1

    GRIB_stepType :

    instant

    GRIB_gridType :

    reduced_gg

    GRIB_N :

    1280

    GRIB_NV :

    0

    GRIB_cfName :

    unknown

    GRIB_cfVarName :

    t2m

    GRIB_gridDefinitionDescription :

    Gaussian Latitude/Longitude Grid

    GRIB_missingValue :

    9999

    GRIB_name :

    2 metre temperature

    GRIB_pl :

    [20 24 28 ... 28 24 20]

    GRIB_shortName :

    2t

    GRIB_totalNumber :

    0

    GRIB_units :

    K

    long_name :

    2 metre temperature

    units :

    K

    standard_name :

    unknown

    [13199360 values with dtype=float32]

  • d2m
    (number, time, step, surface, values)
    float32
    ...

    GRIB_paramId :

    168

    GRIB_dataType :

    an

    GRIB_numberOfPoints :

    6599680

    GRIB_typeOfLevel :

    surface

    GRIB_stepUnits :

    1

    GRIB_stepType :

    instant

    GRIB_gridType :

    reduced_gg

    GRIB_N :

    1280

    GRIB_NV :

    0

    GRIB_cfName :

    unknown

    GRIB_cfVarName :

    d2m

    GRIB_gridDefinitionDescription :

    Gaussian Latitude/Longitude Grid

    GRIB_missingValue :

    9999

    GRIB_name :

    2 metre dewpoint temperature

    GRIB_pl :

    [20 24 28 ... 28 24 20]

    GRIB_shortName :

    2d

    GRIB_totalNumber :

    0

    GRIB_units :

    K

    long_name :

    2 metre dewpoint temperature

    units :

    K

    standard_name :

    unknown

    [13199360 values with dtype=float32]

• Indexes: (4)
  • number
    PandasIndex

    PandasIndex(Int64Index([0], dtype='int64', name='number'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2024-04-21 00:00:00', '2024-04-21 12:00:00'], dtype='datetime64[ns]', name='time', freq=None))

  • step
    PandasIndex

    PandasIndex(TimedeltaIndex(['0 days'], dtype='timedelta64[ns]', name='step', freq=None))

  • surface
    PandasIndex

    PandasIndex(Float64Index([0.0], dtype='float64', name='surface'))

• Attributes: (7)

  GRIB_edition :

  1

  GRIB_centre :

  ecmf

  GRIB_centreDescription :

  European Centre for Medium-Range Weather Forecasts

  GRIB_subCentre :

  0

  Conventions :

  CF-1.7

  institution :

  European Centre for Medium-Range Weather Forecasts

  history :

  2024-04-22T11:01 GRIB to CDM+CF via cfgrib-0.9.10.3/ecCodes-2.33.0 with {"source": "N/A", "filter_by_keys": {}, "encode_cf": ["parameter", "time", "geography", "vertical"]}

Reading into a file

We can retrieve data from FDB into a file, which is located in the cache:

ds = earthkit.data.from_source("fdb", request=request, stream=False)

ds.ls()

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType
0	ecmf	msl	surface	0	20240421	0	0	an	0	reduced_gg
1	ecmf	2t	surface	0	20240421	0	0	an	0	reduced_gg
2	ecmf	2d	surface	0	20240421	0	0	an	0	reduced_gg
3	ecmf	msl	surface	0	20240421	1200	0	an	0	reduced_gg
4	ecmf	2t	surface	0	20240421	1200	0	an	0	reduced_gg
5	ecmf	2d	surface	0	20240421	1200	0	an	0	reduced_gg

The data is now cached. Subsequent retrievals will used the cached file directly.