Migration guide for 1.0¶

from_source¶

The returned object¶

The return type of the from_source() function was changed and now it returns a data object object. This object provides some basic information about the data but its primary goal is to convert it to a given representation for further work. The actual data loading is deferred as much as possible, until the data is converted into a given type.

For example, when we read GRIB data with from_source(), it returns a data object that can be converted to a fieldlist with to_fieldlist(). Previously, from_source() returned a fieldlist directly. E.g.:

Old way:

import earthkit.data as ekd

fl = ekd.from_source("file", "test6.grib")

New way:

import earthkit.data as ekd

fl = ekd.from_source("file", "test6.grib").to_fieldlist()

The list of available conversion types can be quickly get by calling Data.available_types on the returned object. E.g.:

import earthkit.data as ekd

data = ekd.from_source("file", "test6.grib")
print(data.available_types)
['fieldlist', 'xarray', 'pandas', 'numpy']

Then we can call any of the corresponding to_* methods to convert the data to the desired type. E.g. to convert to an Xarray Dataset we can do:

import earthkit.data as ekd

data = ekd.from_source("file", "test6.grib")
ds = data.to_xarray()

Examples:

The object returned by from_source

The `read_all` kwarg¶

Previously, we could use the read_all kwarg in from_source() when stream=True was also set (this latter is only available in sources supporting streams). This is now removed and the same functionality can be achieved by passing read_all as a kwarg to to_fieldlist(). E.g.:

Old way:

import earthkit.data as ekd

url = "https://sites.ecmwf.int/repository/earthkit-data/tutorials/test.grib"
fl = ekd.from_source("url", url, stream=True, read_all=True)

New way:

import earthkit.data as ekd

url = "https://sites.ecmwf.int/repository/earthkit-data/tutorials/test.grib"
fl = ekd.from_source("url", url, stream=True).to_fieldlist(read_all=True)

See more details in Reading all the data into memory.

Concatenation¶

Previously, fiedllists and some sources could be concatenated using the + operator. This has been replaced with the concat function:

from earthkit.data import concat

ds3 = concat(ds1, ds2)

Please note that + operator is used an arithmetic operator for Fields and Fieldlists, so it is still available but with a different meaning.

Field¶

The Field is now not polymorphic but is made up of polymorphic components using format independent high-level metadata. See the Field overview for more details.

Field components and metadata¶

So far the following field components are implemented:

Each component has its own set of high-level metadata keys and methods. There are two ways to access the related values from the components:

# use the get() method
f.get("time.base_datetime")

# use the key method on the component
f.time.base_datetime()

These metadata keys can be used in sel() and order_by() to select and sort fields. See the keys page for a complete list of all available component metadata keys.

# select all fields with a base datetime of 2020-01-01 00:00:00
fl.sel({"time.base_datetime": "2020-01-01 00:00:00"})

# order fields by valid datetime
fl.order_by("time.valid_datetime")

For the full list of available component (high-level) metadata keys, see Metadata key reference.

Field raw metadata¶

Raw metadata keys are still available but they are only accessible either by using the “metadata.” prefix in get() or through the metadata() method. E.g. if the Field was created from a GRIB message, we can access the “shortName” ecCodes GRIB key from the raw metadata like this:

f.get("metadata.shortName")
f.metadata("shortName")
f.metadata("metadata.shortName")

The metadata() method will raise a KeyError if the key is not found, while the get() method allows to specify a default value to return if the key is not found. For example, if the field does not have raw (GRIB) metadata:

f.get("metadata.shortName")
None

f.get("metadata.shortName", default="N/A")
'N/A'

f.metadata("shortName")
Traceback (most recent call last):
    ...
KeyError: 'shortName'

The raw metadata, when available, can also be used in methods such as sel() and order_by() by using the "metadata.<key>" prefix. For example, to select all fields with an ecCodes GRIB short name of "2t":

fl.sel({"metadata.shortName": "2t"})

Field modification¶

Fields can be modified using the set() method. This method allows to set new data values and/or change metadata keys. See the notebook examples:

Field arithmetic¶

Added Field arithmetic. The basic maths operators can now be used to perform arithmetic operations on fields. The operations are performed on the data arrays of the fields, and the resulting field has the same metadata as the left operand and will be entirely stored in memory. For example:

f3 = f1 + f2
f4 = f1 - f2
f5 = f1 * f2
f6 = f1 / f2

Notebook examples¶

Field overview

Changes in the Field API¶

The Field API has been redesigned and many methods have been removed or changed. The following table gives an overview of the changes in the Field API (the f in the table below is a Field object):

Old API	New API	Notes
to_numpy()	`to_numpy()`	New kwarg added: `copy=True`. By default returns a copy of the data array.
to_array()	`to_array()`	New kwarg added: `copy=True`. By default returns a copy of the data array.
to_latlon()	`f.geography.latlons()`	The new function returns a tuple of arrays (lats, lons)
to_points()	`f.geography.points()` or `f.geography.xys()`	The new functions return a tuple of arrays (x, y)
grid_points()	`f.geography.latlons()`	The new function returns a tuple of arrays (lats, lons).
projection()	`f.geography.projection()`
bounding_box()	`f.geography.bounding_box()`
clone()	N/A	Functionality not needed. Use `set()` instead
copy()	N/A	Functionality not needed. Use `set()` instead
as_namespace()	`get()` with the `collections` kwarg	Use e.g. `f.get(collections="metadata.mars")` to get the “mars” ecCodes namespace. Only available for GRIB data.
datetime()	N/A	Use `f.time.base_datetime()` and `f.time.valid_datetime()` instead.
valid_datetime()	`f.time.valid_datetime()`	The new function returns a datetime.datetime object.
base_datetime()	`f.time.base_datetime()`	The new function returns a datetime.datetime object.
metadata()	`metadata()`	Has limited scope now. Can only access keys in the raw metadata belonging to the object the field was created from. E.g. for GRIB this works: f.metadata("shortName") f.metadata("metadata.shortName") When the key does not exist in the raw metadata, it raises a KeyError.
MetaData object accessed by calling metadata() without args/kwargs	N/A	This object is no longer available. The field itself provides the same functionality through `get()` and `set()`.
dump()	N/A	Use `describe()` instead.
describe()	Still exists but functionality changed.
handle	N/A	The handle is no longer available. Use `get()` or `metadata()` to access the raw metadata keys.
mars_area	N/A	Use: `f.geography.area()`
mars_grid	N/A	Use: `f.geography.grid()`
resolution	N/A	This is no longer available.
rotation	N/A	This is no longer available.
grid_points_unrotated()	N/A	This is no longer available.
save()	`to_target()`
write()	`to_target()`

Fieldlist¶

FieldList arithmetic¶

Added FieldList arithmetic. The basic maths operators can now be used to perform arithmetic operations on fieldlists. The operations are performed on the data arrays of the fieldlists, and the resulting fieldlist has the same metadata as the left operand and will be entirely stored in memory. For example:

fl3 = fl1 + fl2
fl4 = fl1 - fl2
fl5 = fl1 * fl2
fl6 = fl1 / fl2

Changes in the FieldList API¶

The following table gives an overview of the changes in the Fieldlist API (the fl in the table below is a FieldList object):

Old API	New API	Notes
to_numpy()	`to_numpy()`	New kwarg added: `copy=True`. By default returns a copy of the data array.
to_array()	`to_array()`	New kwarg added: `copy=True`. By default returns a copy of the data array.
to_latlon()	`fl.geography.latlons()`	The new function returns a tuple of arrays (lats, lons)
to_points()	`fl.geography.points()` or `fl.geography.xys()`	These functions return a tuple of arrays (x, y)
projection()	`fl.geography.projection()`
bounding_box()	`fl.geography.bounding_box()`
datetime()	N/A	This method is no longer available. Use the following instead: f.get("time.base_datetime") f.get("time.valid_datetime")
metadata()	`metadata()`	Has limited scope now. Can only access keys in the raw metadata belonging to the object the field was created from. E.g. for GRIB this works: f.metadata("shortName") f.metadata("metadata.shortName") When the key does not exist in the raw metadata, it raises a KeyError.
sel()	`sel()`	Primarily, now works with high-level metadata keys. Raw metadata keys can also be used with the `"metadata."` prefix: fl.sel({"time.base_datetime": "2020-01-01 00:00:00"}) # high-level metadata key fl.sel({"metadata.shortName": "2t"}) # raw GRIB metadata key
order_by()	`order_by()`	Primarily, now works with high-level metadata keys. Raw metadata keys can also be used with the `"metadata."` prefix: fl.order_by("time.valid_datetime") # high-level metadata key fl.order_by("metadata.shortName") # raw GRIB metadata key
save()	`to_target()`
write()	`to_target()`

Indexed mode removed¶

The GRIB indexed mode, previously available via the indexing=True kwarg in from_source(), has been removed:

# No longer supported
fl = ekd.from_source("file", "data.grib", indexing=True)

In indexed mode the GRIB file was pre-scanned and all field metadata was stored in an in-memory index, making repeated sel() and order_by() calls faster for large files at the cost of upfront indexing time and additional memory. This feature will be reimplemented in a future release, potentially with a different interface.

Xarray engine¶

The Xarray engine has been refactored and many of the internal classes and methods have been changed. The following list gives an overview of the changes in the Xarray engine:

a new default profile earthkit has been added which is used when no profile is specified. This profile is designed to work with the new format independent metadata keys from Field to generate the Xarray dataset.
the old mars and grib profiles were kept but they are now using some of the new format independent metadata keys to generate the Xarray dataset.
the “number” dim_role was renamed to “member” in line with the new format independent metadata keys. See: Dimensions for more details.
the time_dim_mode kwarg in to_xarray() was replaced by time_dims and the meaning of some temporal dimensions in the dim_roles also changed. See Temporal dimensions for more details.