Xarray engine: mono variable with remapping

This notebook demonstrates how to generate an Xarray with a single dataarray containing all the parameters from a GRIB fieldlist. This data structure is often needed for machine learning.

First, we get GRIB data containing multiple forecasts on the surface and pressure levels. We select a single forecast out of it.

import earthkit.data as ekd
ds_fl = ekd.from_source("sample", "mixed_pl_sfc.grib").sel(date=20240603, time=0)

Next, we convert the GRIB Fieldlist to Xarray with to_xarray(). The goal is to create a single variable in the dataset called “data”. Since we have both surface and pressure level parameters the input data does not form a full hypercube. To overcome this problem we use the remapping option to merge the “param” and “level” metadata keys into a single key. With fixed_dims we define the dimensions and their order to use and mono_variable=True ensures a single dataarray will be created.

ds = ds_fl.to_xarray(fixed_dims=["valid_time", "param", "number"],
                     mono_variable=True,
                     chunks={"valid_time": 1},
                     flatten_values=True,
                     add_earthkit_attrs=False,
                     remapping={"param": "{param}_{level}"}
                    )
ds

<xarray.Dataset> Size: 362kB
Dimensions:     (valid_time: 2, param: 32, number: 1, values: 684)
Coordinates:
  * valid_time  (valid_time) datetime64[ns] 16B 2024-06-03 2024-06-03T06:00:00
  * param       (param) <U6 768B '2t_0' 'msl_0' 'r_1000' ... 'z_700' 'z_850'
  * number      (number) int64 8B 0
    latitude    (values) float64 5kB dask.array<chunksize=(684,), meta=np.ndarray>
    longitude   (values) float64 5kB dask.array<chunksize=(684,), meta=np.ndarray>
Dimensions without coordinates: values
Data variables:
    data        (valid_time, param, number, values) float64 350kB dask.array<chunksize=(1, 32, 1, 684), meta=np.ndarray>
Attributes:
    paramId:      167
    class:        od
    stream:       oper
    levtype:      sfc
    type:         fc
    expver:       0001
    date:         20240603
    time:         0
    domain:       g
    Conventions:  CF-1.8
    institution:  ECMWF

xarray.Dataset

• Dimensions:
  • valid_time: 2
  • param: 32
  • number: 1
  • values: 684
• Coordinates: (5)
  • valid_time
    (valid_time)
    datetime64[ns]
    2024-06-03 2024-06-03T06:00:00

    standard_name :

    time

    long_name :

    valid_time

    array(['2024-06-03T00:00:00.000000000', '2024-06-03T06:00:00.000000000'],
          dtype='datetime64[ns]')

  • param
    (param)
    <U6
    '2t_0' 'msl_0' ... 'z_700' 'z_850'

    array(['2t_0', 'msl_0', 'r_1000', 'r_300', 'r_400', 'r_500', 'r_700', 'r_850',
           't_1000', 't_300', 't_400', 't_500', 't_700', 't_850', 'u_1000',
           'u_300', 'u_400', 'u_500', 'u_700', 'u_850', 'v_1000', 'v_300', 'v_400',
           'v_500', 'v_700', 'v_850', 'z_1000', 'z_300', 'z_400', 'z_500', 'z_700',
           'z_850'], dtype='<U6')

  • number
    (number)
    int64
    0

    array([0])

  • latitude
    (values)
    float64
    dask.array<chunksize=(684,), meta=np.ndarray>

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    Array Chunk Bytes 5.34 kiB 5.34 kiB Shape (684,) (684,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • longitude
    (values)
    float64
    dask.array<chunksize=(684,), meta=np.ndarray>

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    Array Chunk Bytes 5.34 kiB 5.34 kiB Shape (684,) (684,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Data variables: (1)
  • data
    (valid_time, param, number, values)
    float64
    dask.array<chunksize=(1, 32, 1, 684), meta=np.ndarray>

    standard_name :

    unknown

    long_name :

    2 metre temperature

    units :

    K

    Array Chunk Bytes 342.00 kiB 171.00 kiB Shape (2, 32, 1, 684) (1, 32, 1, 684) Dask graph 2 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • valid_time
    PandasIndex

    PandasIndex(DatetimeIndex(['2024-06-03 00:00:00', '2024-06-03 06:00:00'], dtype='datetime64[ns]', name='valid_time', freq=None))

  • param
    PandasIndex

    PandasIndex(Index(['2t_0', 'msl_0', 'r_1000', 'r_300', 'r_400', 'r_500', 'r_700', 'r_850',
           't_1000', 't_300', 't_400', 't_500', 't_700', 't_850', 'u_1000',
           'u_300', 'u_400', 'u_500', 'u_700', 'u_850', 'v_1000', 'v_300', 'v_400',
           'v_500', 'v_700', 'v_850', 'z_1000', 'z_300', 'z_400', 'z_500', 'z_700',
           'z_850'],
          dtype='object', name='param'))

  • number
    PandasIndex

    PandasIndex(Index([0], dtype='int64', name='number'))

• Attributes: (11)

  paramId :

  167

  class :

  od

  stream :

  oper

  levtype :

  sfc

  type :

  fc

  expver :

  0001

  date :

  20240603

  time :

  0

  domain :

  g

  Conventions :

  CF-1.8

  institution :

  ECMWF

When generating the Xarray we flattened the field values and chose the chunking so that one chunk would contain all the data belonging to a given valid time.

ds["data"]

<xarray.DataArray 'data' (valid_time: 2, param: 32, number: 1, values: 684)> Size: 350kB
dask.array<open_dataset-data, shape=(2, 32, 1, 684), dtype=float64, chunksize=(1, 32, 1, 684), chunktype=numpy.ndarray>
Coordinates:
  * valid_time  (valid_time) datetime64[ns] 16B 2024-06-03 2024-06-03T06:00:00
  * param       (param) <U6 768B '2t_0' 'msl_0' 'r_1000' ... 'z_700' 'z_850'
  * number      (number) int64 8B 0
    latitude    (values) float64 5kB dask.array<chunksize=(684,), meta=np.ndarray>
    longitude   (values) float64 5kB dask.array<chunksize=(684,), meta=np.ndarray>
Dimensions without coordinates: values
Attributes:
    standard_name:  unknown
    long_name:      2 metre temperature
    units:          K

xarray.DataArray

'data'

• valid_time: 2
• param: 32
• number: 1
• values: 684

• dask.array<chunksize=(1, 32, 1, 684), meta=np.ndarray>

  Array Chunk Bytes 342.00 kiB 171.00 kiB Shape (2, 32, 1, 684) (1, 32, 1, 684) Dask graph 2 chunks in 2 graph layers Data type float64 numpy.ndarray

• Coordinates: (5)
  • valid_time
    (valid_time)
    datetime64[ns]
    2024-06-03 2024-06-03T06:00:00

    standard_name :

    time

    long_name :

    valid_time

    array(['2024-06-03T00:00:00.000000000', '2024-06-03T06:00:00.000000000'],
          dtype='datetime64[ns]')

  • param
    (param)
    <U6
    '2t_0' 'msl_0' ... 'z_700' 'z_850'

    array(['2t_0', 'msl_0', 'r_1000', 'r_300', 'r_400', 'r_500', 'r_700', 'r_850',
           't_1000', 't_300', 't_400', 't_500', 't_700', 't_850', 'u_1000',
           'u_300', 'u_400', 'u_500', 'u_700', 'u_850', 'v_1000', 'v_300', 'v_400',
           'v_500', 'v_700', 'v_850', 'z_1000', 'z_300', 'z_400', 'z_500', 'z_700',
           'z_850'], dtype='<U6')

  • number
    (number)
    int64
    0

    array([0])

  • latitude
    (values)
    float64
    dask.array<chunksize=(684,), meta=np.ndarray>

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    Array Chunk Bytes 5.34 kiB 5.34 kiB Shape (684,) (684,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • longitude
    (values)
    float64
    dask.array<chunksize=(684,), meta=np.ndarray>

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    Array Chunk Bytes 5.34 kiB 5.34 kiB Shape (684,) (684,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • valid_time
    PandasIndex

    PandasIndex(DatetimeIndex(['2024-06-03 00:00:00', '2024-06-03 06:00:00'], dtype='datetime64[ns]', name='valid_time', freq=None))

  • param
    PandasIndex

    PandasIndex(Index(['2t_0', 'msl_0', 'r_1000', 'r_300', 'r_400', 'r_500', 'r_700', 'r_850',
           't_1000', 't_300', 't_400', 't_500', 't_700', 't_850', 'u_1000',
           'u_300', 'u_400', 'u_500', 'u_700', 'u_850', 'v_1000', 'v_300', 'v_400',
           'v_500', 'v_700', 'v_850', 'z_1000', 'z_300', 'z_400', 'z_500', 'z_700',
           'z_850'],
          dtype='object', name='param'))

  • number
    PandasIndex

    PandasIndex(Index([0], dtype='int64', name='number'))

• Attributes: (3)

  standard_name :

  unknown

  long_name :

  2 metre temperature

  units :

  K