Xarray engine: hypercube with holes

This notebook demonstrates how to work with input data not forming a full hypercube.

First, we get pressure level GRIB data with t, u and v variables, where u only contains the subset of the levels in t and v. This data forms a hypercube with holes.

import earthkit.data as ekd
ds_fl = ekd.from_source("sample", "tuv_pl_holes.grib")

Converting the input to Xarray with to_xarray() fails because holes are not allowed by default.

try:
    ds_fl.to_xarray()
except ValueError as e:
    import textwrap
    print(textwrap.fill(str(e), 100))

Variable "u" has inconsistent dimension "levelist" compared to other variables. Expected values: (6)
[300, 400, 500, 700, 850, 1000], got: (2) [500, 700]. Length mismatch: 6 != 2

We need to use the allow_holes=True option to make the conversion work.

ds = ds_fl.to_xarray(allow_holes=True)
ds

<xarray.Dataset> Size: 12kB
Dimensions:    (level: 6, latitude: 7, longitude: 12)
Coordinates:
  * level      (level) int64 48B 300 400 500 700 850 1000
  * latitude   (latitude) float64 56B 90.0 60.0 30.0 0.0 -30.0 -60.0 -90.0
  * longitude  (longitude) float64 96B 0.0 30.0 60.0 90.0 ... 270.0 300.0 330.0
Data variables:
    t          (level, latitude, longitude) float64 4kB ...
    u          (level, latitude, longitude) float64 4kB ...
    v          (level, latitude, longitude) float64 4kB ...
Attributes:
    class:        od
    stream:       oper
    levtype:      pl
    type:         an
    expver:       0001
    date:         20180801
    time:         1200
    domain:       g
    number:       0
    Conventions:  CF-1.8
    institution:  ECMWF

xarray.Dataset

• Dimensions:
  • level: 6
  • latitude: 7
  • longitude: 12
• Coordinates: (3)
  • level
    (level)
    int64
    300 400 500 700 850 1000

    units :

    hPa

    positive :

    down

    stored_direction :

    decreasing

    standard_name :

    air_pressure

    long_name :

    pressure

    array([ 300,  400,  500,  700,  850, 1000])

  • latitude
    (latitude)
    float64
    90.0 60.0 30.0 ... -60.0 -90.0

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    array([ 90.,  60.,  30.,   0., -30., -60., -90.])

  • longitude
    (longitude)
    float64
    0.0 30.0 60.0 ... 270.0 300.0 330.0

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    array([  0.,  30.,  60.,  90., 120., 150., 180., 210., 240., 270., 300., 330.])

• Data variables: (3)
  • t
    (level, latitude, longitude)
    float64
    ...

    param :

    t

    standard_name :

    air_temperature

    long_name :

    Temperature

    paramId :

    130

    units :

    K

    _earthkit :

    {'message': b'GRIB\x00\x00l\x01\x00\x004\x80b\x95\xff\x80\x82d\x03\xe8\x12\x08\x01\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x02\x04\x010001\x00\x00\x00\x00\x00 \x00\xff\x00\x00\x0c\x00\x07\x01_\x90\x00\x00\x00\x80\x81_\x90\x05\t\x10u0u0\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x80\x02D\xb9}n\x00\x007777', 'bitsPerValue': 4}

    [504 values with dtype=float64]

  • u
    (level, latitude, longitude)
    float64
    ...

    param :

    u

    standard_name :

    eastward_wind

    long_name :

    U component of wind

    paramId :

    131

    units :

    m s**-1

    _earthkit :

    {'message': b'GRIB\x00\x00l\x01\x00\x004\x80b\x95\xff\x80\x83d\x02\xbc\x12\x08\x01\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x02\x04\x010001\x00\x00\x00\x00\x00 \x00\xff\x00\x00\x0c\x00\x07\x01_\x90\x00\x00\x00\x80\x81_\x90\x05\t\x10u0u0\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x80\x02D\xb9}n\x00\x007777', 'bitsPerValue': 4}

    [504 values with dtype=float64]

  • v
    (level, latitude, longitude)
    float64
    ...

    param :

    v

    standard_name :

    northward_wind

    long_name :

    V component of wind

    paramId :

    132

    units :

    m s**-1

    _earthkit :

    {'message': b'GRIB\x00\x00l\x01\x00\x004\x80b\x95\xff\x80\x84d\x03\xe8\x12\x08\x01\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x02\x04\x010001\x00\x00\x00\x00\x00 \x00\xff\x00\x00\x0c\x00\x07\x01_\x90\x00\x00\x00\x80\x81_\x90\x05\t\x10u0u0\x00\x00\x00\x00\x00\x00\x00\x0c\x00\x80\x02D\xb9}n\x00\x007777', 'bitsPerValue': 4}

    [504 values with dtype=float64]

• Indexes: (3)
  • level
    PandasIndex

    PandasIndex(Index([300, 400, 500, 700, 850, 1000], dtype='int64', name='level'))

  • latitude
    PandasIndex

    PandasIndex(Index([90.0, 60.0, 30.0, 0.0, -30.0, -60.0, -90.0], dtype='float64', name='latitude'))

  • longitude
    PandasIndex

    PandasIndex(Index([0.0, 30.0, 60.0, 90.0, 120.0, 150.0, 180.0, 210.0, 240.0, 270.0, 300.0,
           330.0],
          dtype='float64', name='longitude'))

• Attributes: (11)

  class :

  od

  stream :

  oper

  levtype :

  pl

  type :

  an

  expver :

  0001

  date :

  20180801

  time :

  1200

  domain :

  g

  number :

  0

  Conventions :

  CF-1.8

  institution :

  ECMWF

The resulting dataset works as expected but when we slice into the holes nans are returned.

ds.u.sel(level=300).values

array([[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]])

ds.u.sel(level=[500,300]).values

array([[[  4.74000549,   4.74000549,   4.74000549,   4.74000549,
           4.74000549,   4.74000549,   4.74000549,   4.74000549,
           4.74000549,   4.74000549,   4.74000549,   4.74000549],
        [ 12.74000549,   4.74000549,   4.74000549,  28.74000549,
          12.74000549,  -3.25999451,   4.74000549,   4.74000549,
           4.74000549,  12.74000549,  20.74000549,   4.74000549],
        [ -3.25999451,   4.74000549,  -3.25999451,   4.74000549,
         -11.25999451,   4.74000549,  -3.25999451,  -3.25999451,
          -3.25999451,   4.74000549, -11.25999451,  12.74000549],
        [  4.74000549,  -3.25999451,  -3.25999451,  12.74000549,
          -3.25999451, -11.25999451, -11.25999451,  -3.25999451,
           4.74000549,  -3.25999451, -19.25999451,   4.74000549],
        [ 20.74000549,  36.74000549,  20.74000549,  12.74000549,
          20.74000549,  12.74000549,  12.74000549,  36.74000549,
          12.74000549,   4.74000549,  20.74000549,  12.74000549],
        [ 36.74000549,  -3.25999451,   4.74000549,  -3.25999451,
           4.74000549,  28.74000549,  12.74000549,   4.74000549,
          12.74000549,  36.74000549,  28.74000549,  20.74000549],
        [  4.74000549,   4.74000549,   4.74000549,   4.74000549,
           4.74000549,   4.74000549,   4.74000549,   4.74000549,
           4.74000549,   4.74000549,   4.74000549,   4.74000549]],

       [[         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan],
        [         nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan,
                  nan,          nan,          nan,          nan]]])