Xarray engine: mono variable

This notebook demonstrates how to generate an Xarray with a single DataArray containing all the parameters from a GRIB fieldlist. This data structure is often needed for machine learning.

First, we get 2m temperature and dewpoint data for a whole year on a low resolution regular latitude-longitude grid. It contains 2 fields per day (at 0 and 12 UTC) per parameter.

import earthkit.data as ekd

ds_fl = ekd.from_source("sample", "t2_td2_1_year.grib").to_fieldlist()
len(ds_fl)

1464

Next, we convert the GRIB Fieldlist to Xarray with to_xarray(). There will be a single variable in the dataset called “data”.

ds = ds_fl.to_xarray(
    fixed_dims=["time.valid_datetime", "parameter.variable"],
    mono_variable=True,
    chunks={"valid_time": 1},
    flatten_values=True,
    add_earthkit_attrs=False,
)
ds

<xarray.Dataset> Size: 111kB
Dimensions:         (valid_datetime: 732, variable: 2, values: 9)
Coordinates:
  * valid_datetime  (valid_datetime) datetime64[ns] 6kB 2020-01-01 ... 2020-1...
  * variable        (variable) <U2 16B '2d' '2t'
    latitude        (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
    longitude       (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
Dimensions without coordinates: values
Data variables:
    data            (valid_datetime, variable, values) float64 105kB dask.array<chunksize=(732, 2, 9), meta=np.ndarray>
Attributes:
    Conventions:  CF-1.8
    institution:  ECMWF

xarray.Dataset

• Dimensions:
  • valid_datetime: 732
  • variable: 2
  • values: 9
• Coordinates: (4)
  • valid_datetime
    (valid_datetime)
    datetime64[ns]
    2020-01-01 ... 2020-12-31T12:00:00

    array(['2020-01-01T00:00:00.000000000', '2020-01-01T12:00:00.000000000',
           '2020-01-02T00:00:00.000000000', ..., '2020-12-30T12:00:00.000000000',
           '2020-12-31T00:00:00.000000000', '2020-12-31T12:00:00.000000000'],
          shape=(732,), dtype='datetime64[ns]')

  • variable
    (variable)
    <U2
    '2d' '2t'

    array(['2d', '2t'], dtype='<U2')

  • latitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • longitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Data variables: (1)
  • data
    (valid_datetime, variable, values)
    float64
    dask.array<chunksize=(732, 2, 9), meta=np.ndarray>

    standard_name :

    unknown

    long_name :

    2 metre dewpoint temperature

    units :

    kelvin

    level_type :

    height_above_ground_level

    Array Chunk Bytes 102.94 kiB 102.94 kiB Shape (732, 2, 9) (732, 2, 9) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Attributes: (2)

  Conventions :

  CF-1.8

  institution :

  ECMWF

When generating the Xarray we flattened the field values and chose the chunking so that one chunk would contain all the data belonging to a given valid time.

ds["data"]

<xarray.DataArray 'data' (valid_datetime: 732, variable: 2, values: 9)> Size: 105kB
dask.array<open_dataset-data, shape=(732, 2, 9), dtype=float64, chunksize=(732, 2, 9), chunktype=numpy.ndarray>
Coordinates:
  * valid_datetime  (valid_datetime) datetime64[ns] 6kB 2020-01-01 ... 2020-1...
  * variable        (variable) <U2 16B '2d' '2t'
    latitude        (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
    longitude       (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
Dimensions without coordinates: values
Attributes:
    standard_name:  unknown
    long_name:      2 metre dewpoint temperature
    units:          kelvin
    level_type:     height_above_ground_level

xarray.DataArray

'data'

• valid_datetime: 732
• variable: 2
• values: 9

• dask.array<chunksize=(732, 2, 9), meta=np.ndarray>

  Array Chunk Bytes 102.94 kiB 102.94 kiB Shape (732, 2, 9) (732, 2, 9) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Coordinates: (4)
  • valid_datetime
    (valid_datetime)
    datetime64[ns]
    2020-01-01 ... 2020-12-31T12:00:00

    array(['2020-01-01T00:00:00.000000000', '2020-01-01T12:00:00.000000000',
           '2020-01-02T00:00:00.000000000', ..., '2020-12-30T12:00:00.000000000',
           '2020-12-31T00:00:00.000000000', '2020-12-31T12:00:00.000000000'],
          shape=(732,), dtype='datetime64[ns]')

  • variable
    (variable)
    <U2
    '2d' '2t'

    array(['2d', '2t'], dtype='<U2')

  • latitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • longitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Attributes: (4)

  standard_name :

  unknown

  long_name :

  2 metre dewpoint temperature

  units :

  kelvin

  level_type :

  height_above_ground_level

Adding ensemble dimension

We add the ensemble member as an additional dimension to the generated Xarray. Because the input is not ensemble data the value of the ensemble.member can be None for some/all fields preventing us from forming this dimension. To overcome this problem we provide a meaningful default with the fill_metadata kwarg.

ds = ds_fl.to_xarray(
    fixed_dims=["time.valid_datetime", "parameter.variable", "ensemble.member"],
    mono_variable=True,
    chunks={"valid_time": 1},
    flatten_values=True,
    add_earthkit_attrs=False,
    fill_metadata={"ensemble.member": "0"},
)
ds

<xarray.Dataset> Size: 111kB
Dimensions:         (valid_datetime: 732, variable: 2, member: 1, values: 9)
Coordinates:
  * valid_datetime  (valid_datetime) datetime64[ns] 6kB 2020-01-01 ... 2020-1...
  * variable        (variable) <U2 16B '2d' '2t'
  * member          (member) <U1 4B '0'
    latitude        (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
    longitude       (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
Dimensions without coordinates: values
Data variables:
    data            (valid_datetime, variable, member, values) float64 105kB dask.array<chunksize=(732, 2, 1, 9), meta=np.ndarray>
Attributes:
    Conventions:  CF-1.8
    institution:  ECMWF

xarray.Dataset

• Dimensions:
  • valid_datetime: 732
  • variable: 2
  • member: 1
  • values: 9
• Coordinates: (5)
  • valid_datetime
    (valid_datetime)
    datetime64[ns]
    2020-01-01 ... 2020-12-31T12:00:00

    array(['2020-01-01T00:00:00.000000000', '2020-01-01T12:00:00.000000000',
           '2020-01-02T00:00:00.000000000', ..., '2020-12-30T12:00:00.000000000',
           '2020-12-31T00:00:00.000000000', '2020-12-31T12:00:00.000000000'],
          shape=(732,), dtype='datetime64[ns]')

  • variable
    (variable)
    <U2
    '2d' '2t'

    array(['2d', '2t'], dtype='<U2')

  • member
    (member)
    <U1
    '0'

    array(['0'], dtype='<U1')

  • latitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • longitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Data variables: (1)
  • data
    (valid_datetime, variable, member, values)
    float64
    dask.array<chunksize=(732, 2, 1, 9), meta=np.ndarray>

    standard_name :

    unknown

    long_name :

    2 metre dewpoint temperature

    units :

    kelvin

    level_type :

    height_above_ground_level

    Array Chunk Bytes 102.94 kiB 102.94 kiB Shape (732, 2, 1, 9) (732, 2, 1, 9) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Attributes: (2)

  Conventions :

  CF-1.8

  institution :

  ECMWF