GRIB: modifying fields

This notebook explains how to use clone() and copy() to create new fields. The main difference between these methods can be summarised as follows:

• clone(): the new field keeps a reference to the original field and allows more flexible metadata modification than using copy()

• copy(): always creates a deep copy. The resulting field is always an ArrayField storing the all the data in memory

We will use the first field of the input data in the examples.

import earthkit.data as ekd

ds = ekd.from_source("file", "test4.grib")
f_ori = ds[0]
f_ori.ls()

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType
0	ecmf	t	isobaricInhPa	500	20070101	1200	0	an	0	regular_ll

Using clone()

Modifying metadata

f_new = f_ori.clone(level=700)
f_new.ls()

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType
0	ecmf	t	isobaricInhPa	700	20070101	1200	0	an	0	regular_ll

Please note that the metadata object in the new field is a wrapper around the original one and override() is not called to update it with the newly set keys. This means that there is no guarantee that the newly set keys are compatible with original metadata. So, arbitrary metadata keys can be set with no checks performed on their validity. This is generally not a problem for most of the field methods (e.g. sel()) but can be an issue we we try to write the modified fields into a GRIB file (see below).

To demonstrate what was said above we set “level” to an invalid value and add a custom (non GRIB) key:

f_new = f_ori.clone(level="abc", my_key="123")
f_new.ls(extra_keys="my_key")

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType	my_key
0	ecmf	t	isobaricInhPa	abc	20070101	1200	0	an	0	regular_ll	123

Metadata values passed to clone() can be callables.

def _f(field, key, original_metadata):
    return original_metadata["param"] + str(original_metadata["level"])

f_new = f_ori.clone(custom_name=_f)
f_new.ls(extra_keys="custom_name")

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType	custom_name
0	ecmf	t	isobaricInhPa	500	20070101	1200	0	an	0	regular_ll	t500

Modifying values

f_new = f_ori.clone(values=f_ori.values + 1)
f_new.values[0:3], f_ori.values[0:3]

(array([229.04600525, 229.04600525, 229.04600525]),
 array([228.04600525, 228.04600525, 228.04600525]))

Modifying both metadata and values

f_new = f_ori.clone(values=f_ori.values + 1, level=700)
f_new.ls()

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType
0	ecmf	t	isobaricInhPa	700	20070101	1200	0	an	0	regular_ll

f_new.values[0:3]

array([229.04600525, 229.04600525, 229.04600525])

Saving the modifield field into a GRIB file

f_new.save("_modified_field.grib")
ds_new = ekd.from_source("file", "_modified_field.grib")
ds_new[0].ls()

/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/ipykernel_19855/457062347.py:1: DeprecatedWarning: save is deprecated as of 0.13.0 and will be removed in 0.14.0. Use to_target() instead
  f_new.save("_modified_field.grib")

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType
0	ecmf	t	isobaricInhPa	700	20070101	1200	0	an	0	regular_ll

ds_new[0].values[0:3]

array([229.04600525, 229.04600525, 229.04600525])

Saving is not always possible

Saving the modified field into a GRIB is only possible if the modified metadata is GRIB compatible. To demonstrate this we add a custom metadata key called “_level”.

f_new = f_ori.clone(_level=700)
f_new.metadata("_level", "level")

(700, 500)

Writing to GRIB is not possible because “_level” is not a valid GRIB key and ecCodes raises an exception.

try:
    f_new.to_target("file", "_modified_field1.grib")
except Exception as e:
    print(e)

Error setting _level=700
Key/value not found
Traceback (most recent call last):
  File "/Users/cgr/git/earthkit-data/src/earthkit/data/utils/message.py", line 274, in set
    return eccodes.codes_set(self._handle, name, value)
  File "/Users/cgr/git/eccodes-python/gribapi/gribapi.py", line 2140, in grib_set
    grib_set_long(msgid, key, value)
  File "/Users/cgr/git/eccodes-python/gribapi/gribapi.py", line 1006, in grib_set_long
    GRIB_CHECK(lib.grib_set_long(h, key.encode(ENC), value))
  File "/Users/cgr/git/eccodes-python/gribapi/gribapi.py", line 226, in GRIB_CHECK
    errors.raise_grib_error(errid)
  File "/Users/cgr/git/eccodes-python/gribapi/errors.py", line 381, in raise_grib_error
    raise ERROR_MAP[errid](errid)
gribapi.errors.KeyValueNotFoundError: Key/value not found

Key/value not found

Using copy()

copy() performs a deep copy to generate an ArrayField storing all the all the data in memory.

When copy() is called without arguments both the values and the Metadata object of the original field are copied into the new field. The latter copy is created by calling override() on the original metadata object.

f_new = f_ori.copy()
f_new.values[:3], f_ori.values[:3]

(array([228.04600525, 228.04600525, 228.04600525]),
 array([228.04600525, 228.04600525, 228.04600525]))

We can pass new values to copy().

f_new = f_ori.copy(values=f_ori.values + 1)
f_new.values[:3], f_ori.values[:3]

(array([229.04600525, 229.04600525, 229.04600525]),
 array([228.04600525, 228.04600525, 228.04600525]))

We can also pass a new metadata object.

f_new = f_ori.copy(metadata=f_ori.metadata().override(level=300))
f_new.ls()

	centre	shortName	typeOfLevel	level	dataDate	dataTime	stepRange	dataType	number	gridType
0	ecmf	t	isobaricInhPa	300	20070101	1200	0	an	0	regular_ll