Version 0.15 Updates

Version 0.15.2

Dependency updates

Ensure compatibility between earthkit components by restricting the following dependencies:

  • earthkit-utils<0.2

  • earthkit-meteo<0.6

Version 0.15.1

Fixes

  • Fixed issue when xarray.open_dataset() did not work with the “earthkit” engine when a pathlib.Path object specified the input data (#741).

Version 0.15.0

Deprecations

Xarray engine

Breaking xarray engine changes

  • Separated the dimension names from the metadata keys used to generate the dimensions. Dimensions associated with the dimension roles are now taking the name of the dimension role, irrespective of the metadata key the dimension role is mapped to. The example belows shows what this means for e.g. the “level” dimension role when the mars profile (the default) is used. In this case the “level” role is mapped to the “levelist” ecCodes GRIB key.

    import earthkit.data as ekd

ds_fl = ekd.from_source("sample", "pl.grib")
ds_fl.to_xarray().coords

    With the new code the dimension/coordinate name will be “level”. So the output is as follows:

    Coordinates:
* forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...
* step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00
* level                    (level) int64 16B 500 700
* latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0
* longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0

    However, using the previous version the output would be as follows:

    Coordinates:
* forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...
* step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00
* levelist                 (levelist) int64 16B 500 700
* latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0
* longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0

    The old behaviour can still be invoked by using the newly added dim_name_from_role_name=False option. See: to_xarray().

  • The step dimension role is now mapped to the step_timedelta metadata key, which is the datetime.timedelta representation of the "endStep" GRIB/metadata key. Previously, this role was mapped to the "step" key. This change was necessary for the following reasons:

    • allows handling cases when the step key contains ranges as a str in the form of e.g. “12-24”

    • allows proper sorting of the “step” dimension values when building the dataset

Please note that due to this change when dim_name_from_role_name=False is used the step dimension will be called “step_timedelta” instead of “step”. You can still reproduce the old behaviour by using: dim_roles={"step": "step"}. See the following notebook example: Xarray engine: step range.

Other Xarray engine changes

  • Allowed using mappings in the extra_dims and fixed_dims options to define both the name of the dimensions and the metadata keys to generate their values. Previously, these options only took a single/multiple metadata keys. E.g. both the options below will generate the “expver”, “mars_stream” and “mars_class” dimensions using the “expver”, “stream” and “class” metadata keys.

    extra_dims = ["expver", {"mars_stream": "stream"}, ("mars_class", "class")]
extra_dims = {
    "expver": "expver",
    "mars_stream": "stream",
    "mars_class": "class",
}

  • Improved the serialisation of GRIB fieldlists to reduce memory usage when Xarray is generated with chunks (#700). See the Xarray engine: chunks notebook example.

  • TensorBackendArray, which implements the lazy loading of DataArrays in the Xarray engine, now uses a dask.utils.SerializableLock when accessing the data (#700).

  • Enabled converting list-of-dicts fieldlists into Xarray (#701). See the list-of-dict: converting to Xarray notebook example.

  • Enabled converting Xarray generated with the earthkit engine into GRIB using to_target() (#730). See Converting Xarray to GRIB and the related Xarray engine: writing back to GRIB notebook example.

New Xarray engine notebooks

New features

  • Added the zarr source to read Zarr data (#675).

  • Added the zarr target (#716). See the Writing GRIB data to Zarr notebook example.

  • Added new config option grib-file-serialisation-policy to control how GRIB data on disk is pickled. The options are “path” and “memory”. The default is “path”. Previously, only “memory” was implemented (#700).

  • Added serialisation to GRIB fields (both on disk and in-memory) (#700)

  • Enabled specifying earthkit Xarray engine options via the earthkit_to_xarray_kwargs kwarg in to_target() when converting GRIB to NetCDF. See Converting GRIB to NetCDF and related the GRIB: converting to NetCDF notebook example. (#729) E.g.

    ds.to_target(
    "netcdf", "pl.nc", earthkit_to_xarray_kwargs={"flatten_values": True}
)

Fixes

  • Fixed issue when the forcings source did not handle time-zone aware datetimes correctly (#693).