Xarray engine: mono variable

This notebook demonstrates how to generate an Xarray with a single dataarray containing all the parameters from a GRIB fieldlist. This data structure is often needed for machine learning.

First, we get 2m temperature and dewpoint data for a whole year on a low resolution regular latitude-longitude grid. It contains 2 fields per day (at 0 and 12 UTC) per parameter.

import earthkit.data as ekd
ds_fl = ekd.from_source("sample", "t2_td2_1_year.grib")
len(ds_fl)

1464

Next, we convert the GRIB Fieldlist to Xarray with to_xarray(). There will be a single variable in the dataset called “data”.

ds = ds_fl.to_xarray(fixed_dims=["valid_time", "param"],
                     mono_variable=True,
                     chunks={"valid_time": 1},
                     flatten_values=True,
                     add_earthkit_attrs=False,
                    )
ds

<xarray.Dataset> Size: 111kB
Dimensions:     (valid_time: 732, param: 2, values: 9)
Coordinates:
  * valid_time  (valid_time) datetime64[ns] 6kB 2020-01-01 ... 2020-12-31T12:...
  * param       (param) <U2 16B '2d' '2t'
    latitude    (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
    longitude   (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
Dimensions without coordinates: values
Data variables:
    data        (valid_time, param, values) float64 105kB dask.array<chunksize=(1, 2, 9), meta=np.ndarray>
Attributes:
    paramId:      168
    class:        d1
    stream:       clte
    levtype:      sfc
    type:         fc
    expver:       0001
    date:         20200101
    time:         0
    domain:       g
    Conventions:  CF-1.8
    institution:  ECMWF

xarray.Dataset

• Dimensions:
  • valid_time: 732
  • param: 2
  • values: 9
• Coordinates: (4)
  • valid_time
    (valid_time)
    datetime64[ns]
    2020-01-01 ... 2020-12-31T12:00:00

    standard_name :

    time

    long_name :

    valid_time

    array(['2020-01-01T00:00:00.000000000', '2020-01-01T12:00:00.000000000',
           '2020-01-02T00:00:00.000000000', ..., '2020-12-30T12:00:00.000000000',
           '2020-12-31T00:00:00.000000000', '2020-12-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • param
    (param)
    <U2
    '2d' '2t'

    array(['2d', '2t'], dtype='<U2')

  • latitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • longitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Data variables: (1)
  • data
    (valid_time, param, values)
    float64
    dask.array<chunksize=(1, 2, 9), meta=np.ndarray>

    standard_name :

    unknown

    long_name :

    2 metre dewpoint temperature

    units :

    K

    Array Chunk Bytes 102.94 kiB 144 B Shape (732, 2, 9) (1, 2, 9) Dask graph 732 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (2)
  • valid_time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-01 00:00:00', '2020-01-01 12:00:00',
                   '2020-01-02 00:00:00', '2020-01-02 12:00:00',
                   '2020-01-03 00:00:00', '2020-01-03 12:00:00',
                   '2020-01-04 00:00:00', '2020-01-04 12:00:00',
                   '2020-01-05 00:00:00', '2020-01-05 12:00:00',
                   ...
                   '2020-12-27 00:00:00', '2020-12-27 12:00:00',
                   '2020-12-28 00:00:00', '2020-12-28 12:00:00',
                   '2020-12-29 00:00:00', '2020-12-29 12:00:00',
                   '2020-12-30 00:00:00', '2020-12-30 12:00:00',
                   '2020-12-31 00:00:00', '2020-12-31 12:00:00'],
                  dtype='datetime64[ns]', name='valid_time', length=732, freq=None))

  • param
    PandasIndex

    PandasIndex(Index(['2d', '2t'], dtype='object', name='param'))

• Attributes: (11)

  paramId :

  168

  class :

  d1

  stream :

  clte

  levtype :

  sfc

  type :

  fc

  expver :

  0001

  date :

  20200101

  time :

  0

  domain :

  g

  Conventions :

  CF-1.8

  institution :

  ECMWF

When generating the Xarray we flattened the field values and chose the chunking so that one chunk would contain all the data belonging to a given valid time.

ds["data"]

<xarray.DataArray 'data' (valid_time: 732, param: 2, values: 9)> Size: 105kB
dask.array<open_dataset-data, shape=(732, 2, 9), dtype=float64, chunksize=(1, 2, 9), chunktype=numpy.ndarray>
Coordinates:
  * valid_time  (valid_time) datetime64[ns] 6kB 2020-01-01 ... 2020-12-31T12:...
  * param       (param) <U2 16B '2d' '2t'
    latitude    (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
    longitude   (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
Dimensions without coordinates: values
Attributes:
    standard_name:  unknown
    long_name:      2 metre dewpoint temperature
    units:          K

xarray.DataArray

'data'

• valid_time: 732
• param: 2
• values: 9

• dask.array<chunksize=(1, 2, 9), meta=np.ndarray>

  Array Chunk Bytes 102.94 kiB 144 B Shape (732, 2, 9) (1, 2, 9) Dask graph 732 chunks in 2 graph layers Data type float64 numpy.ndarray

• Coordinates: (4)
  • valid_time
    (valid_time)
    datetime64[ns]
    2020-01-01 ... 2020-12-31T12:00:00

    standard_name :

    time

    long_name :

    valid_time

    array(['2020-01-01T00:00:00.000000000', '2020-01-01T12:00:00.000000000',
           '2020-01-02T00:00:00.000000000', ..., '2020-12-30T12:00:00.000000000',
           '2020-12-31T00:00:00.000000000', '2020-12-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • param
    (param)
    <U2
    '2d' '2t'

    array(['2d', '2t'], dtype='<U2')

  • latitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • longitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (2)
  • valid_time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-01 00:00:00', '2020-01-01 12:00:00',
                   '2020-01-02 00:00:00', '2020-01-02 12:00:00',
                   '2020-01-03 00:00:00', '2020-01-03 12:00:00',
                   '2020-01-04 00:00:00', '2020-01-04 12:00:00',
                   '2020-01-05 00:00:00', '2020-01-05 12:00:00',
                   ...
                   '2020-12-27 00:00:00', '2020-12-27 12:00:00',
                   '2020-12-28 00:00:00', '2020-12-28 12:00:00',
                   '2020-12-29 00:00:00', '2020-12-29 12:00:00',
                   '2020-12-30 00:00:00', '2020-12-30 12:00:00',
                   '2020-12-31 00:00:00', '2020-12-31 12:00:00'],
                  dtype='datetime64[ns]', name='valid_time', length=732, freq=None))

  • param
    PandasIndex

    PandasIndex(Index(['2d', '2t'], dtype='object', name='param'))

• Attributes: (3)

  standard_name :

  unknown

  long_name :

  2 metre dewpoint temperature

  units :

  K

Adding ensemble dimension

We add the ensemble member as an additional dimension to the generated Xarray. Because the input is not ensemble data the value of the “number” ecCodes key can be missing. So we need to provide a meaningful default with the fill_metadata kwarg to be able to build the “number” dimension.

ds = ds_fl.to_xarray(fixed_dims=["valid_time", "param", "number"],
                     mono_variable=True,
                     chunks={"valid_time": 1},
                     flatten_values=True,
                     add_earthkit_attrs=False,
                     fill_metadata={"number": 0},
                    )
ds

<xarray.Dataset> Size: 111kB
Dimensions:     (valid_time: 732, param: 2, number: 1, values: 9)
Coordinates:
  * valid_time  (valid_time) datetime64[ns] 6kB 2020-01-01 ... 2020-12-31T12:...
  * param       (param) <U2 16B '2d' '2t'
  * number      (number) int64 8B 0
    latitude    (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
    longitude   (values) float64 72B dask.array<chunksize=(9,), meta=np.ndarray>
Dimensions without coordinates: values
Data variables:
    data        (valid_time, param, number, values) float64 105kB dask.array<chunksize=(1, 2, 1, 9), meta=np.ndarray>
Attributes:
    paramId:      168
    class:        d1
    stream:       clte
    levtype:      sfc
    type:         fc
    expver:       0001
    date:         20200101
    time:         0
    domain:       g
    Conventions:  CF-1.8
    institution:  ECMWF

xarray.Dataset

• Dimensions:
  • valid_time: 732
  • param: 2
  • number: 1
  • values: 9
• Coordinates: (5)
  • valid_time
    (valid_time)
    datetime64[ns]
    2020-01-01 ... 2020-12-31T12:00:00

    standard_name :

    time

    long_name :

    valid_time

    array(['2020-01-01T00:00:00.000000000', '2020-01-01T12:00:00.000000000',
           '2020-01-02T00:00:00.000000000', ..., '2020-12-30T12:00:00.000000000',
           '2020-12-31T00:00:00.000000000', '2020-12-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • param
    (param)
    <U2
    '2d' '2t'

    array(['2d', '2t'], dtype='<U2')

  • number
    (number)
    int64
    0

    array([0])

  • latitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • longitude
    (values)
    float64
    dask.array<chunksize=(9,), meta=np.ndarray>

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    Array Chunk Bytes 72 B 72 B Shape (9,) (9,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Data variables: (1)
  • data
    (valid_time, param, number, values)
    float64
    dask.array<chunksize=(1, 2, 1, 9), meta=np.ndarray>

    standard_name :

    unknown

    long_name :

    2 metre dewpoint temperature

    units :

    K

    Array Chunk Bytes 102.94 kiB 144 B Shape (732, 2, 1, 9) (1, 2, 1, 9) Dask graph 732 chunks in 2 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • valid_time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-01 00:00:00', '2020-01-01 12:00:00',
                   '2020-01-02 00:00:00', '2020-01-02 12:00:00',
                   '2020-01-03 00:00:00', '2020-01-03 12:00:00',
                   '2020-01-04 00:00:00', '2020-01-04 12:00:00',
                   '2020-01-05 00:00:00', '2020-01-05 12:00:00',
                   ...
                   '2020-12-27 00:00:00', '2020-12-27 12:00:00',
                   '2020-12-28 00:00:00', '2020-12-28 12:00:00',
                   '2020-12-29 00:00:00', '2020-12-29 12:00:00',
                   '2020-12-30 00:00:00', '2020-12-30 12:00:00',
                   '2020-12-31 00:00:00', '2020-12-31 12:00:00'],
                  dtype='datetime64[ns]', name='valid_time', length=732, freq=None))

  • param
    PandasIndex

    PandasIndex(Index(['2d', '2t'], dtype='object', name='param'))

  • number
    PandasIndex

    PandasIndex(Index([0], dtype='int64', name='number'))

• Attributes: (11)

  paramId :

  168

  class :

  d1

  stream :

  clte

  levtype :

  sfc

  type :

  fc

  expver :

  0001

  date :

  20200101

  time :

  0

  domain :

  g

  Conventions :

  CF-1.8

  institution :

  ECMWF