Writing GRIB data to Zarr

# get input GRIB data
import earthkit.data as ekd

ds = ekd.from_source("sample", "pl.grib").to_fieldlist()

This data contains 32 fields: several forecasts on pressure levels for 2 parameters.

Using to_target() on the data object

We use to_target() to write the GRIB fieldlist/field into a zarr store. First, the data is converted to Xarray then xarray.Dataset.to_zarr() is called to generate the zarr store. We need to set the kwargs accordingly.

# with these options each field will be a separate chunk
ds.to_target(
    "zarr",
    earthkit_to_xarray_kwargs={"chunks": {"forecast_reference_time": 1, "step": 1, "level": 1}},
    xarray_to_zarr_kwargs={"store": "_pl.zarr", "mode": "w"},
)

import zarr

root = zarr.group("_pl.zarr")
root.tree()

root["t"].info

Name	/t
Type	zarr.core.Array
Data type	float64
Shape	(4, 2, 2, 19, 36)
Chunk shape	(1, 1, 1, 19, 36)
Order	C
Read-only	False
Compressor	Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0)
Store type	zarr.storage.DirectoryStore
No. bytes	87552 (85.5K)
No. bytes stored	26897 (26.3K)
Storage ratio	3.3
Chunks initialized	16/16

The zarr store can be loaded to Xarray to check its content.

import xarray

xarray.open_dataset("_pl.zarr")

<xarray.Dataset> Size: 176kB
Dimensions:                  (forecast_reference_time: 4, step: 2, level: 2,
                              latitude: 19, longitude: 36)
Coordinates:
  * forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...
  * step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00
  * level                    (level) int64 16B 500 700
  * latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0
  * longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0
Data variables:
    r                        (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...
    t                        (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...
Attributes:
    Conventions:  CF-1.8
    institution:  ECMWF

xarray.Dataset

• Dimensions:
  • forecast_reference_time: 4
  • step: 2
  • level: 2
  • latitude: 19
  • longitude: 36
• Coordinates: (5)
  • forecast_reference_time
    (forecast_reference_time)
    datetime64[ns]
    2024-06-03 ... 2024-06-04T12:00:00

    array(['2024-06-03T00:00:00.000000000', '2024-06-03T12:00:00.000000000',
           '2024-06-04T00:00:00.000000000', '2024-06-04T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • step
    (step)
    timedelta64[ns]
    00:00:00 06:00:00

    long_name :

    time since forecast_reference_time

    standard_name :

    forecast_period

    array([             0, 21600000000000], dtype='timedelta64[ns]')

  • level
    (level)
    int64
    500 700

    long_name :

    pressure

    positive :

    down

    standard_name :

    air_pressure

    units :

    hectopascal

    array([500, 700])

  • latitude
    (latitude)
    float64
    90.0 80.0 70.0 ... -80.0 -90.0

    long_name :

    latitude

    standard_name :

    latitude

    units :

    degrees_north

    array([ 90.,  80.,  70.,  60.,  50.,  40.,  30.,  20.,  10.,   0., -10., -20.,
           -30., -40., -50., -60., -70., -80., -90.])

  • longitude
    (longitude)
    float64
    0.0 10.0 20.0 ... 330.0 340.0 350.0

    long_name :

    longitude

    standard_name :

    longitude

    units :

    degrees_east

    array([  0.,  10.,  20.,  30.,  40.,  50.,  60.,  70.,  80.,  90., 100., 110.,
           120., 130., 140., 150., 160., 170., 180., 190., 200., 210., 220., 230.,
           240., 250., 260., 270., 280., 290., 300., 310., 320., 330., 340., 350.])

• Data variables: (2)
  • r
    (forecast_reference_time, step, level, latitude, longitude)
    float64
    ...

    ek_grid_spec :

    {"grid": [10, 10]}

    level_type :

    pressure

    long_name :

    Relative humidity

    standard_name :

    relative_humidity

    units :

    percent

    [10944 values with dtype=float64]

  • t
    (forecast_reference_time, step, level, latitude, longitude)
    float64
    ...

    ek_grid_spec :

    {"grid": [10, 10]}

    level_type :

    pressure

    long_name :

    Temperature

    standard_name :

    air_temperature

    units :

    kelvin

    [10944 values with dtype=float64]

• Attributes: (2)

  Conventions :

  CF-1.8

  institution :

  ECMWF