Xarray engine: hypercube with holes

This notebook demonstrates how to work with input data not forming a full hypercube.

First, we get pressure level GRIB data with t, u and v variables, where u only contains the subset of the levels in t and v. This data forms a hypercube with holes.

import earthkit.data as ekd

ds_fl = ekd.from_source("sample", "tuv_pl_holes.grib").to_fieldlist()

Converting the input to Xarray with to_xarray() fails because holes are not allowed by default.

try:
    ds_fl.to_xarray()
except ValueError as e:
    import textwrap

    print(textwrap.fill(str(e), 100))

Variable "u" has inconsistent dimension "vertical.level" compared to other variables. Expected
values: (6) [300, 400, 500, 700, 850, 1000], got: (2) [500, 700]. Length mismatch: 6 != 2

We need to use the allow_holes=True option to make the conversion work.

ds = ds_fl.to_xarray(allow_holes=True)
ds

<xarray.Dataset> Size: 12kB
Dimensions:    (level: 6, latitude: 7, longitude: 12)
Coordinates:
  * level      (level) int64 48B 300 400 500 700 850 1000
  * latitude   (latitude) float64 56B 90.0 60.0 30.0 0.0 -30.0 -60.0 -90.0
  * longitude  (longitude) float64 96B 0.0 30.0 60.0 90.0 ... 270.0 300.0 330.0
Data variables:
    t          (level, latitude, longitude) float64 4kB ...
    u          (level, latitude, longitude) float64 4kB ...
    v          (level, latitude, longitude) float64 4kB ...
Attributes:
    Conventions:  CF-1.8
    institution:  ECMWF

xarray.Dataset

• Dimensions:
  • level: 6
  • latitude: 7
  • longitude: 12
• Coordinates: (3)
  • level
    (level)
    int64
    300 400 500 700 850 1000

    standard_name :

    air_pressure

    long_name :

    pressure

    units :

    hectopascal

    positive :

    down

    array([ 300,  400,  500,  700,  850, 1000])

  • latitude
    (latitude)
    float64
    90.0 60.0 30.0 ... -60.0 -90.0

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    array([ 90.,  60.,  30.,   0., -30., -60., -90.])

  • longitude
    (longitude)
    float64
    0.0 30.0 60.0 ... 270.0 300.0 330.0

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    array([  0.,  30.,  60.,  90., 120., 150., 180., 210., 240., 270., 300., 330.])

• Data variables: (3)
  • t
    (level, latitude, longitude)
    float64
    ...

    standard_name :

    air_temperature

    long_name :

    Temperature

    units :

    kelvin

    level_type :

    pressure

    _earthkit :

    {'message': b'GRIB\x00\x00l\x01\x00\x004\x80b\x95\xff\x80\x82d\x03\xe8\x12\x08\x01\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x02\x04\x010001\x00\x00\x00\x00\x00 \x00\xff\x00\x00\x0c\x00\x07\x01_\x90\x00\x00\x00\x80\x81_\x90\x05\t\x10u0u0\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x80\x02D\xb9}n\x00\x007777', 'bitsPerValue': 4}

    [504 values with dtype=float64]

  • u
    (level, latitude, longitude)
    float64
    ...

    standard_name :

    eastward_wind

    long_name :

    U component of wind

    units :

    meter / second

    level_type :

    pressure

    _earthkit :

    {'message': b'GRIB\x00\x00l\x01\x00\x004\x80b\x95\xff\x80\x83d\x02\xbc\x12\x08\x01\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x02\x04\x010001\x00\x00\x00\x00\x00 \x00\xff\x00\x00\x0c\x00\x07\x01_\x90\x00\x00\x00\x80\x81_\x90\x05\t\x10u0u0\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x80\x02D\xb9}n\x00\x007777', 'bitsPerValue': 4}

    [504 values with dtype=float64]

  • v
    (level, latitude, longitude)
    float64
    ...

    standard_name :

    northward_wind

    long_name :

    V component of wind

    units :

    meter / second

    level_type :

    pressure

    _earthkit :

    {'message': b'GRIB\x00\x00l\x01\x00\x004\x80b\x95\xff\x80\x84d\x03\xe8\x12\x08\x01\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x02\x04\x010001\x00\x00\x00\x00\x00 \x00\xff\x00\x00\x0c\x00\x07\x01_\x90\x00\x00\x00\x80\x81_\x90\x05\t\x10u0u0\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x80\x02D\xb9}n\x00\x007777', 'bitsPerValue': 4}

    [504 values with dtype=float64]

• Attributes: (2)

  Conventions :

  CF-1.8

  institution :

  ECMWF

The resulting dataset works as expected but when we slice into the holes NaNs are returned.

ds.u.sel(level=300).values

array([[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]])

ds.u.sel(level=[500, 300]).values

array([[[  4.74000549,   4.74000549,   4.74000549,   4.74000549,
           4.74000549,   4.74000549,   4.74000549,   4.74000549,
           4.74000549,   4.74000549,   4.74000549,   4.74000549],
        [ 12.74000549,   4.74000549,   4.74000549,  28.74000549,
          12.74000549,  -3.25999451,   4.74000549,   4.74000549,
           4.74000549,  12.74000549,  20.74000549,   4.74000549],
        [ -3.25999451,   4.74000549,  -3.25999451,   4.74000549,
         -11.25999451,   4.74000549,  -3.25999451,  -3.25999451,
          -3.25999451,   4.74000549, -11.25999451,  12.74000549],
        [  4.74000549,  -3.25999451,  -3.25999451,  12.74000549,
          -3.25999451, -11.25999451, -11.25999451,  -3.25999451,
           4.74000549,  -3.25999451, -19.25999451,   4.74000549],
        [ 20.74000549,  36.74000549,  20.74000549,  12.74000549,
          20.74000549,  12.74000549,  12.74000549,  36.74000549,
          12.74000549,   4.74000549,  20.74000549,  12.74000549],
        [ 36.74000549,  -3.25999451,   4.74000549,  -3.25999451,
           4.74000549,  28.74000549,  12.74000549,   4.74000549,
          12.74000549,  36.74000549,  28.74000549,  20.74000549],
        [  4.74000549,   4.74000549,   4.74000549,   4.74000549,
           4.74000549,   4.74000549,   4.74000549,   4.74000549,
           4.74000549,   4.74000549,   4.74000549,   4.74000549]],

       [[         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan]]])