data.readers.grib.index ======================= .. py:module:: data.readers.grib.index Classes ------- .. autoapisummary:: data.readers.grib.index.GribFieldList Package Contents ---------------- .. py:class:: GribFieldList(*args, **kwargs) Bases: :py:obj:`earthkit.data.readers.grib.pandas.PandasMixIn`, :py:obj:`earthkit.data.readers.grib.xarray.XarrayMixIn`, :py:obj:`earthkit.data.core.fieldlist.FieldList` Represents a list of :obj:`GribField `\ s. We can **iterate** through the fields as follows: >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/test6.grib") >>> len(ds) 6 >>> for f in ds: ... print(f) ... GribField(t,1000,20180801,1200,0,0) GribField(u,1000,20180801,1200,0,0) GribField(v,1000,20180801,1200,0,0) GribField(t,850,20180801,1200,0,0) GribField(u,850,20180801,1200,0,0) GribField(v,850,20180801,1200,0,0) :obj:`Fieldset` objects can be **concatenated** with the + operator: >>> import earthkit.data >>> ds1 = earthkit.data.from_source("file", "docs/examples/test.grib") >>> len(ds1) 2 >>> ds2 = earthkit.data.from_source("file", "docs/examples/test6.grib") >>> len(ds2) 6 >>> ds = ds1 + ds2 >>> len(ds) 8 Standard Python slicing works: >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/test6.grib") >>> len(ds) 6 >>> ds[0] GribField(t,1000,20180801,1200,0,0) >>> for f in ds[0:3]: ... print(f) GribField(t,1000,20180801,1200,0,0) GribField(u,1000,20180801,1200,0,0) GribField(v,1000,20180801,1200,0,0) >>> for f in ds[0:4:2]: ... print(f) GribField(t,1000,20180801,1200,0,0) GribField(v,1000,20180801,1200,0,0) >>> ds[-1] GribField(v,850,20180801,1200,0,0) >>> for f in ds[-2:]: ... print(f) GribField(u,850,20180801,1200,0,0) GribField(v,850,20180801,1200,0,0) Slicing also works with a list or an ndarray: >>> for f in ds[[1, 3]]: ... print(f) ... GribField(u,1000,20180801,1200,0,0) GribField(t,850,20180801,1200,0,0) >>> for f in ds[np.array([1, 3])]: ... print(f) ... GribField(u,1000,20180801,1200,0,0) GribField(t,850,20180801,1200,0,0) .. py:property:: availability .. py:property:: availability_path .. py:method:: batched(n) Iterate through the object in batches of ``n``. :param n: Batch size. :type n: :class:`int` :returns: Returns an iterator yielding batches of ``n`` elements. Each batch is a new object containing a view to the data in the original object, so no data is copied. The last batch may contain fewer than ``n`` elements. :rtype: :class:`object` .. py:method:: bounding_box() Return the bounding box for each field. :returns: List with one :obj:`BoundingBox ` per :obj:`Field` :rtype: :class:`list` .. py:method:: config(name) .. py:method:: cube(*args, **kwargs) .. py:method:: data(keys=('lat', 'lon', 'value'), flatten=False, dtype=None, index=None) Return the values and/or the geographical coordinates. Only works when all the fields have the same grid geometry. :param keys: Specifies the type of data to be returned. Any combination of "lat", "lon" and "value" is allowed here. :type keys: :obj:`str`, :obj:`list` or :obj:`tuple` :param flatten: When it is True the "lat", "lon" arrays and the "value" arrays per field will all be flattened. Otherwise they will preserve the field's :obj:`shape`. :type flatten: :class:`bool` :param dtype: Typecode or data-type of the arrays. When it is :obj:`None` the default type used by the underlying data accessor is used. For GRIB it is ``float64``. :type dtype: :class:`str`, :class:`array.dtype` or :obj:`None` :param index: The index of the values to be extracted from each field. When it is None all the values are extracted. :type index: :class:`array indexing object`, *optional* :returns: The elements of the array (in the order of the ``keys``) are as follows: * the latitudes array from the first field when "lat" is in ``keys`` * the longitudes array from the first field when "lon" is in ``keys`` * a values array per field when "values" is in ``keys`` :rtype: :class:`array-like` :raises ValueError: When not all the fields have the same grid geometry. .. rubric:: Examples - :ref:`/examples/grib_lat_lon_value.ipynb` >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/test6.grib") >>> len(ds) 6 >>> d = ds.data() >>> d.shape (8, 7, 12) >>> d[0, 0, 0] # first latitude 90.0 >>> d[1, 0, 0] # first longitude 0.0 >>> d[2:, 0, 0] # first value per field array([272.56417847, -6.28688049, 7.83348083, 272.53916931, -4.89837646, 8.66096497]) >>> d = ds.data(keys="lon") >>> d.shape (1, 7, 12) >>> d[0, 0, 0] # first longitude 0.0 .. seealso:: :obj:`to_latlon`, :obj:`to_points`, :obj:`to_numpy`, :obj:`values` .. py:method:: datetime() Return the unique, sorted list of dates and times in the fieldlist. :returns: Dict with items "base_time" and "valid_time". :rtype: :class:`dict` of :class:`datatime.datetime` .. rubric:: Examples >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "tests/data/t_time_series.grib") >>> ds.datetime() {'base_time': [datetime.datetime(2020, 12, 21, 12, 0)], 'valid_time': [ datetime.datetime(2020, 12, 21, 12, 0), datetime.datetime(2020, 12, 21, 15, 0), datetime.datetime(2020, 12, 21, 18, 0), datetime.datetime(2020, 12, 21, 21, 0), datetime.datetime(2020, 12, 23, 12, 0)]} .. py:method:: default_encoder() .. py:method:: describe(*args, **kwargs) Generate a summary of the fieldlist. .. py:method:: from_array(array, metadata) :staticmethod: Create an :class:`SimpleFieldList`. :param array: The fields' values. When it is a list it must contain one array per field. :type array: :class:`array-like`, :class:`list` :param metadata: The fields' metadata. Must contain one :class:`Metadata` object per field. Or it can be a single :class:`Metadata` object when all the fields have the same metadata. :type metadata: :class:`list`, :class:`Metadata` :param In the generated :class:`SimpleFieldList`: :param each field is represented by an array: :param storing the field values and a :class:`MetaData` object holding: :param the field metadata. The shape and dtype of the array is controlled by the ``kwargs``.: .. py:method:: from_fields(fields) :staticmethod: Create a :class:`SimpleFieldList`. :param fields: Iterable of :obj:`Field` objects. :type fields: :class:`iterable` :rtype: :class:`SimpleFieldList` .. py:method:: from_numpy(array, metadata) :staticmethod: .. py:method:: group_by(*keys, sort=True) Iterate through the object in groups defined by metadata keys. :param \*keys: Positional arguments specifying the metadata keys to group by. Keys can be a single or multiple str, or a list or tuple of str. :type \*keys: :class:`tuple` :param sort: If ``True`` (default), the object is sorted by the metadata ``keys`` before grouping. Sorting is only applied if the object is supporting the sorting operation. :type sort: :class:`bool`, *optional* :returns: Returns an iterator yielding batches of elements grouped by the metadata ``keys``. Each batch is a new object containing a view to the data in the original object, so no data is copied. It generates a new group every time the value of the ``keys`` change. :rtype: :class:`object` .. py:method:: head(n=5, **kwargs) Generate a list like summary of the first ``n`` :obj:`Field`\ s. Same as calling :obj:`ls` with ``n``. :param n: The number of messages (``n`` > 0) to be printed from the front. :type n: :class:`int`, :obj:`None` :param \*\*kwargs: Other keyword arguments passed to :obj:`ls`. :type \*\*kwargs: :class:`dict`, *optional* :returns: See :obj:`ls`. :rtype: :class:`Pandas DataFrame` .. seealso:: :obj:`ls`, :obj:`tail` .. rubric:: Notes The following calls are equivalent: .. code-block:: python ds.head() ds.head(5) ds.head(n=5) ds.ls(5) ds.ls(n=5) .. py:method:: index(key) Return the unique, sorted values of the specified metadata ``key`` from all the fields. ``key`` will be automatically added to the keys returned by :obj:`indices`. :param key: Metadata key. :type key: :class:`str` :returns: Unique, sorted values of ``key`` from all the :obj:`Field`\ s. :rtype: :class:`list` .. seealso:: :obj:`index` .. rubric:: Examples >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/tuv_pl.grib") >>> ds.index("level") [300, 400, 500, 700, 850, 1000] .. py:method:: indices(squeeze=False) Return the unique, sorted values for a set of metadata keys (see below) from all the fields. Individual keys can be also queried by :obj:`index`. :param squeeze: When True only returns the metadata keys that have more than one values. :type squeeze: :obj:`False` :returns: Unique, sorted metadata values from all the :obj:`Field`\ s. :rtype: :class:`dict` .. seealso:: :obj:`index` .. rubric:: Examples >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/tuv_pl.grib") >>> ds.indices() {'class': ['od'], 'stream': ['oper'], 'levtype': ['pl'], 'type': ['an'], 'expver': ['0001'], 'date': [20180801], 'time': [1200], 'domain': ['g'], 'number': [0], 'levelist': [300, 400, 500, 700, 850, 1000], 'param': ['t', 'u', 'v']} >>> ds.indices(squeeze=True) {'levelist': [300, 400, 500, 700, 850, 1000], 'param': ['t', 'u', 'v']} >>> ds.index("level") [300, 400, 500, 700, 850, 1000] By default :obj:`indices` uses the keys from the "mars" :xref:`eccodes_namespace`. Keys with no valid values are not included. Keys that :obj:`index` is called with are automatically added to the original set of keys used in :obj:`indices`. .. py:method:: isel(*args, **kwargs) Uses metadata value indices to select a subset of the elements from a fieldlist-like object. :param \*args: Positional arguments specifying the filter conditions. (See below for details). :type \*args: :class:`tuple` :param \*\*kwargs: Other keyword arguments specifying the metadata keys to perform the filtering on. (See below for details). :type \*\*kwargs: :class:`dict`, *optional* :returns: Returns a new object with the filtered elements. It contains a view to the data in the original object, so no data is copied. :rtype: :class:`object` .. rubric:: Notes :obj:`isel` works similarly to :obj:`sel` but conditions are specified by indices of metadata keys. A metadata index stores the unique, **sorted** values of the corresponding metadata key from all the fields in the input data. If the object is a obj:`FieldList ` to list the indices that have more than one values use :meth:`FieldList.indices() `, or to find out the values of a specific index use :meth:`FieldList.index() `. Filter conditions are specified by a set of **metadata** keys either by a dictionary (in ``*args``) or a set of ``**kwargs``. Both single or multiple keys are allowed to use and each can specify the following type of filter values: - single index:: ds.sel(param=1) - list of indices:: ds.sel(param=[1, 3]) - **slice** of values (behaves like normal Python indexing, stop value not included):: # filter levels on level indices 1 and 2 ds.sel(level=slice(1,3)) .. rubric:: Examples >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/tuv_pl.grib") >>> len(ds) 18 >>> ds.indices {'levelist': (1000, 850, 700, 500, 400, 300), 'param': ('t', 'u', 'v')} >>> subset = ds.isel(param=0) >>> len(ds) 6 >>> for f in subset: ... print(f) ... GribField(t,1000,20180801,1200,0,0) GribField(t,850,20180801,1200,0,0) GribField(t,700,20180801,1200,0,0) GribField(t,500,20180801,1200,0,0) GribField(t,400,20180801,1200,0,0) GribField(t,300,20180801,1200,0,0) >>> subset = ds.isel(param=[1, 2], level=slice(2, 4)) >>> len(subset) 4 >>> for f in subset: ... print(f) ... GribField(u,700,20180801,1200,0,0) GribField(v,700,20180801,1200,0,0) GribField(u,500,20180801,1200,0,0) GribField(v,500,20180801,1200,0,0) .. py:method:: ls(n=None, keys=None, extra_keys=None, namespace=None) Generate a list like summary using a set of metadata keys. :param n: The number of :obj:`Field`\ s to be listed. None means all the messages, ``n > 0`` means fields from the front, while ``n < 0`` means fields from the back of the fieldlist. :type n: :class:`int`, :obj:`None` :param keys: Metadata keys. If it is None the following default set of keys will be used: "centre", "shortName", "typeOfLevel", "level", "dataDate", "dataTime", "stepRange", "dataType", "number", "gridType". To specify a column title for each key in the output use a dict. :type keys: :class:`list` of :class:`str`, :class:`dict`, :obj:`None` :param extra_keys: List of additional keys to ``keys``. To specify a column title for each key in the output use a dict. :type extra_keys: :class:`list` of :class:`str`, :class:`dict`, :obj:`None` :param namespace: The namespace to choose the ``keys`` from. When it is set ``keys`` and ``extra_keys`` are omitted. :type namespace: :class:`str`, :obj:`None` :returns: DataFrame with one row per :obj:`Field`. :rtype: :class:`Pandas DataFrame` .. seealso:: :obj:`head`, :obj:`tail` .. py:method:: metadata(*args, **kwargs) Return the metadata values for each field. :param \*args: Positional arguments defining the metadata keys. Passed to :obj:`GribField.metadata() ` :type \*args: :class:`tuple` :param \*\*kwargs: Keyword arguments passed to :obj:`GribField.metadata() ` :type \*\*kwargs: :class:`dict`, *optional* :returns: List with one item per :obj:`GribField ` :rtype: :class:`list` .. rubric:: Examples >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/test.grib") >>> ds.metadata("param") ['2t', 'msl'] >>> ds.metadata("param", "units") [('2t', 'K'), ('msl', 'Pa')] >>> ds.metadata(["param", "units"]) [['2t', 'K'], ['msl', 'Pa']] .. py:method:: order_by(*args, remapping=None, patches=None, **kwargs) Changes the order of the elements in a fieldlist-like object. :param \*args: Positional arguments specifying the metadata keys to perform the ordering on. (See below for details) :type \*args: :class:`tuple` :param remapping: Defines new metadata keys from existing ones that we can refer to in ``*args`` and ``**kwargs``. E.g. to define a new key "param_level" as the concatenated value of the "param" and "level" keys use:: remapping={"param_level": "{param}{level}"} See below for a more elaborate example. :type remapping: :class:`dict` :param \*\*kwargs: Other keyword arguments specifying the metadata keys to perform the ordering on. (See below for details) :type \*\*kwargs: :class:`dict`, *optional* :returns: Returns a new object with reordered elements. It contains a view to the data in the original object, so no data is copied. :rtype: :class:`object` .. rubric:: Examples Ordering by a single metadata key ("param"). The default ordering direction is ``ascending``: >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/test6.grib") >>> for f in ds.order_by("param"): ... print(f) ... GribField(t,850,20180801,1200,0,0) GribField(t,1000,20180801,1200,0,0) GribField(u,850,20180801,1200,0,0)s GribField(u,1000,20180801,1200,0,0) GribField(v,850,20180801,1200,0,0) GribField(v,1000,20180801,1200,0,0) Ordering by multiple keys (first by "level" then by "param"): >>> for f in ds.order_by(["level", "param"]): ... print(f) ... GribField(t,850,20180801,1200,0,0) GribField(u,850,20180801,1200,0,0) GribField(v,850,20180801,1200,0,0) GribField(t,1000,20180801,1200,0,0) GribField(u,1000,20180801,1200,0,0) GribField(v,1000,20180801,1200,0,0) Specifying the ordering direction: >>> for f in ds.order_by(param="ascending", level="descending"): ... print(f) ... GribField(t,1000,20180801,1200,0,0) GribField(t,850,20180801,1200,0,0) GribField(u,1000,20180801,1200,0,0) GribField(u,850,20180801,1200,0,0) GribField(v,1000,20180801,1200,0,0) GribField(v,850,20180801,1200,0,0) Using the list of all the values of a key ("param") to define the order: >>> for f in ds.order_by(param=["u", "t", "v"]): ... print(f) ... GribField(u,1000,20180801,1200,0,0) GribField(u,850,20180801,1200,0,0) GribField(t,1000,20180801,1200,0,0) GribField(t,850,20180801,1200,0,0) GribField(v,1000,20180801,1200,0,0) GribField(v,850,20180801,1200,0,0) Using ``remapping`` to specify the order by a key created from two other keys (we created key "param_level" from "param" and "levelist"): >>> ordering = ["t850", "t1000", "u1000", "v850", "v1000", "u850"] >>> remapping = {"param_level": "{param}{levelist}"} >>> for f in ds.order_by(param_level=ordering, remapping=remapping): ... print(f) ... GribField(t,850,20180801,1200,0,0) GribField(t,1000,20180801,1200,0,0) GribField(u,1000,20180801,1200,0,0) GribField(v,850,20180801,1200,0,0) GribField(v,1000,20180801,1200,0,0) GribField(u,850,20180801,1200,0,0) .. py:property:: parent .. py:method:: projection() Return the projection information shared by all the fields. :rtype: :obj:`Projection` :raises ValueError: When not all the fields have the same grid geometry .. rubric:: Examples >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/test.grib") >>> ds.projection() Name: unknown Axis Info [cartesian]: - E[east]: Easting (unknown) - N[north]: Northing (unknown) - h[up]: Ellipsoidal height (metre) Area of Use: - undefined Coordinate Operation: - name: unknown - method: Equidistant Cylindrical Datum: Unknown based on WGS 84 ellipsoid - Ellipsoid: WGS 84 - Prime Meridian: Greenwich >>> ds.projection().to_proj_string() '+proj=eqc +ellps=WGS84 +a=6378137.0 +lon_0=0.0 +to_meter=111319.4907932736 +no_defs +type=crs' .. py:method:: save(filename, append=False, bits_per_value=None) Write all the fields into a file. :param filename: The target file path. :type filename: :class:`str` :param append: When it is true append data to the target file. Otherwise the target file be overwritten if already exists. :type append: :class:`bool` :param bits_per_value: Set the ``bitsPerValue`` GRIB key for each message in the generated output. When None the ``bitsPerValue`` stored in the message metadata will be used. :type bits_per_value: :class:`int` or :obj:`None` .. seealso:: :obj:`write` .. py:method:: sel(*args, remapping=None, **kwargs) Uses metadata values to select a subset of the elements from a fieldlist-like object. :param \*args: Positional arguments specifying the filter condition as dict. (See below for details). :type \*args: :class:`tuple` :param remapping: Creates new metadata keys from existing ones that we can refer to in ``*args`` and ``**kwargs``. E.g. to define a new key "param_level" as the concatenated value of the "param" and "level" keys use:: remapping={"param_level": "{param}{level}"} See below for a more elaborate example. :type remapping: :class:`dict` :param \*\*kwargs: Other keyword arguments specifying the filter conditions. (See below for details). :type \*\*kwargs: :class:`dict`, *optional* :returns: Returns a new object with the filtered elements. It contains a view to the data in the original object, so no data is copied. :rtype: :class:`object` .. rubric:: Notes Filter conditions are specified by a set of **metadata** keys either by a dictionary (in ``*args``) or a set of ``**kwargs``. Both single or multiple keys are allowed to use and each can specify the following type of filter values: - single value:: ds.sel(param="t") - list of values:: ds.sel(param=["u", "v"]) - **slice** of values (defines a **closed interval**, so treated as inclusive of both the start and stop values, unlike normal Python indexing):: # filter levels between 300 and 500 inclusively ds.sel(level=slice(300, 500)) .. rubric:: Examples >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/tuv_pl.grib") >>> len(ds) 18 Selecting by a single key ("param"): >>> subset = ds.sel(param="t") >>> for f in subset: ... print(f) ... GribField(t,1000,20180801,1200,0,0) GribField(t,850,20180801,1200,0,0) GribField(t,700,20180801,1200,0,0) GribField(t,500,20180801,1200,0,0) GribField(t,400,20180801,1200,0,0) GribField(t,300,20180801,1200,0,0) Selecting by multiple keys ("param", "level") with a list and slice of values: >>> subset = ds.sel(param=["u", "v"], level=slice(400, 700)) >>> for f in subset: ... print(f) ... GribField(u,700,20180801,1200,0,0) GribField(v,700,20180801,1200,0,0) GribField(u,500,20180801,1200,0,0) GribField(v,500,20180801,1200,0,0) GribField(u,400,20180801,1200,0,0) GribField(v,400,20180801, 1200,0,0) Using ``remapping`` to specify the selection by a key created from two other keys (we created key "param_level" from "param" and "levelist"): >>> subset = ds.sel( ... param_level=["t850", "u1000"], ... remapping={"param_level": "{param}{levelist}"}, ... ) >>> for f in subset: ... print(f) ... GribField(u,1000,20180801,1200,0,0) GribField(t,850,20180801,1200,0,0) .. py:method:: tail(n=5, **kwargs) Generate a list like summary of the last ``n`` :obj:`Field`\ s. Same as calling :obj:`ls` with ``-n``. :param n: The number of messages (``n`` > 0) to be printed from the back. :type n: :class:`int`, :obj:`None` :param \*\*kwargs: Other keyword arguments passed to :obj:`ls`. :type \*\*kwargs: :class:`dict`, *optional* :returns: See :obj:`ls`. :rtype: :class:`Pandas DataFrame` .. seealso:: :obj:`head`, :obj:`ls` .. rubric:: Notes The following calls are equivalent: .. code-block:: python ds.tail() ds.tail(5) ds.tail(n=5) ds.ls(-5) ds.ls(n=-5) .. py:method:: to_array(**kwargs) Return all the fields' values as an array. It is formed as the array of the :obj:`data.core.fieldlist.Field.to_array` values per field. :param \*\*kwargs: Keyword arguments passed to :obj:`data.core.fieldlist.Field.to_array` :type \*\*kwargs: :class:`dict`, *optional* :returns: Array containing the field values. :rtype: :class:`array-like` .. seealso:: :obj:`values`, :obj:`to_numpy` .. py:method:: to_fieldlist(array_backend=None, array_namespace=None, device=None, **kwargs) Convert to a new :class:`FieldList`. :param array_backend: Specify the array namespace for the generated :class:`FieldList`. **Deprecated in version 0.19.0**. Use ``array_namespace`` instead. In versions before 0.19.0 an :obj:`ArrayBackend` was also accepted here, which is no longer the case. :type array_backend: :class:`str`, :class:`array_namespace` or :obj:`None` :param array_namespace: The array namespace to be used. **New in version 0.19.0**. :type array_namespace: :class:`str`, :class:`array_namespace` or :obj:`None` :param device: The device where the array will be allocated. When it is :obj:`None` the default device is used. **New in version 0.19.0**. :type device: :class:`str` or :obj:`None` :param \*\*kwargs: ``kwargs`` are passed to :obj:`to_array` to extract the field values the resulting object will store. :type \*\*kwargs: :class:`dict`, *optional* :returns: - a new fieldlist containing :class`ArrayField` fields :rtype: :class:`SimpleFieldList` .. rubric:: Examples The following example will convert a fieldlist read from a file into a :class:`SimpleFieldList` storing single precision field values. >>> import numpy as np >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/tuv_pl.grib") >>> ds.path 'docs/examples/tuv_pl.grib' >>> r = ds.to_fieldlist(array_namespace="numpy", dtype=np.float32) >>> r SimpleFieldList(fields=18) >>> hasattr(r, "path") False >>> r.to_numpy().dtype dtype('float32') .. py:method:: to_latlon(index=None, **kwargs) Return the latitudes/longitudes shared by all the fields. :param index: The index of the latitudes/longitudes to be extracted. When it is None all the values are extracted. :type index: :class:`array indexing object`, *optional* :param \*\*kwargs: Keyword arguments passed to :meth:`Field.to_latlon() ` :type \*\*kwargs: :class:`dict`, *optional* :returns: Dictionary with items "lat" and "lon", containing the arrays of the latitudes and longitudes, respectively. :rtype: :class:`dict` :raises ValueError: When not all the fields have the same grid geometry .. rubric:: Examples >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/test.grib") >>> for f in ds: ... print(f.shape) ... (11, 19) (11, 19) >>> r = ds.to_latlon() >>> for k, v in r.items(): ... print(f"{k}: shape={v.shape}") ... lat: shape=(11, 19) lon: shape=(11, 19) >>> r["lon"][:2] array([[-27., -23., -19., -15., -11., -7., -3., 1., 5., 9., 13., 17., 21., 25., 29., 33., 37., 41., 45.], [-27., -23., -19., -15., -11., -7., -3., 1., 5., 9., 13., 17., 21., 25., 29., 33., 37., 41., 45.]]) .. py:method:: to_numpy(**kwargs) Return all the fields' values as an ndarray. It is formed as the array of the :obj:`data.core.fieldlist.Field.to_numpy` values per field. :param \*\*kwargs: Keyword arguments passed to :obj:`data.core.fieldlist.Field.to_numpy` :type \*\*kwargs: :class:`dict`, *optional* :returns: Array containing the field values. :rtype: :class:`ndarray` .. seealso:: :obj:`to_array`, :obj:`values` .. py:method:: to_pandas(latitude=None, longitude=None, **kwargs) Convert into a pandas dataframe .. py:method:: to_points(**kwargs) Return the geographical coordinates shared by all the fields in the data's original Coordinate Reference System (CRS). :param \*\*kwargs: Keyword arguments passed to :obj:`Field.to_points() ` :type \*\*kwargs: :class:`dict`, *optional* :returns: Dictionary with items "x" and "y", containing the arrays of the x and y coordinates, respectively. :rtype: :class:`dict` :raises ValueError: When not all the fields have the same grid geometry. .. py:method:: to_target(target, *args, **kwargs) Write data into the specified target. .. py:method:: to_tensor(*args, **kwargs) .. py:method:: to_xarray(engine='earthkit', xarray_open_dataset_kwargs=None, **kwargs) Convert the FieldList into an Xarray Dataset. :param engine: The Xarray engine to use for generating the dataset. Default value is ``"earthkit"``. If set to ``cfgrib``, the :xref:`cfgrib` engine is used. No other values are supported. :type engine: :class:`str`, *optional* :param split_dims: Dimension or list of dimensions to use for splitting the data into multiple hypercubes. Default is None. Only used when ``engine="earthkit"``. Please note that ``split_dims`` is not a valid option when the Xarray is directly generated via :py:meth:`xarray.open_dataset`. :type split_dims: :class:`str`, or :class:`iterable` of :class:`str`, :obj:`None` :param xarray_open_dataset_kwargs: Keyword arguments passed to :py:func:`xarray.open_dataset`. Either this or ``**kwargs`` can be used, but not both. :type xarray_open_dataset_kwargs: :class:`dict`, *optional* :param \*\*kwargs: Any keyword arguments that can be passed to :py:func:`xarray.open_dataset`. Engine specific keywords are automatically grouped and passed as ``backend_kwargs``. Either ``**kwargs`` or ``xarray_open_dataset_kwargs`` can be used, but not both. When ``engine="earthkit"`` the following engine specific kwargs are supported: * profile: str, dict or None Provide custom default values for most of the kwargs. Currently, the "mars" and "grid" built-in profiles are available, otherwise an explicit dict can be used. None is equivalent to an empty dict. When a kwarg is specified it will update a default value if it is a dict otherwise it will overwrite it. See: :ref:`xr_profile` for more information. * variable_key: str, None Metadata key to specify the dataset variables. It cannot be defined as a dimension. Default is "param" (in earthkit-data this is the same as "shortName"). Only enabled when ``mono_variable`` is False or None. * drop_variables: str, or iterable of str, None A variable or list of variables to drop from the dataset. Default is None. Only used when ``variable_key`` is enabled. * rename_variables: dict, None Mapping to rename variables. Default is None. Only used when ``variable_key`` is enabled. * mono_variable: bool, str, None If True or str, the dataset will contain a single variable called "data" (or the value of the ``mono_variable`` kwarg when it is a str). If False, the dataset will contain one variable for each distinct value of ``variable_key`` metadata key. The default value (None) expands to False unless the ``profile`` overwrites it. * extra_dims: str, or iterable of str, None Define additional dimensions on top of the predefined dimensions. Only enabled when no ``fixed_dims`` is specified. Default is None. It can be a single item or a list. Each item is either a metadata key, or a dict/tuple defining mapping between the dimension name and the metadata key. The whole option can be a dict. E.g. .. code-block:: python # use key "expver" as a dimension extra_dims = "expver" # use keys "expver" and "steam" as a dimension extra_dims = ["expver", "stream"] # define dimensions "expver", mars_stream" and "mars_type" from # metadata keys "expver", "stream" and "type" extra_dims = [ "expver", {"mars_stream": "stream"}, ("mars_type", "type"), ] extra_dims = [ { "expver": "expver", "mars_stream": "stream", "mars_type": "type", } ] * drop_dims: str, or iterable of str, None Single or multiple dimensions to be ignored. Default is None. Default is None. * ensure_dims: str, or iterable of str, None Dimension or dimensions that should be kept even when ``squeeze=True`` and their size is only 1. Default is None. * fixed_dims: str, or iterable of str, None Define all the dimensions to be generated. When used no other dimensions will be created. Might be incompatible with other settings. Default is None. It can be a single item or a list. Each item is either a metadata key, or a dict/tuple defining mapping between the dimension name and the metadata key. The whole option can be a dict. E.g.: .. code-block:: python # use key "step" as a dimension fixed_dims = "step" # use keys "step" and "levelist" as a dimension extra_dims = ["step", "levelist"] # define dimensions "step", level" and "level_type" from # metadata keys "step", "levelist" and "levtype" extra_dims = [ "step", {"level": "levelist"}, ("level_type", "levtype"), ] extra_dims = [ {"step": "step", "level": "levelist", "level_type": "levtype"} ] * dim_roles: dict, None Specify the "roles" used to form the predefined dimensions. The predefined dimensions are automatically generated when no ``fixed_dims`` specified and comprise the following (in a fixed order): - ensemble forecast member dimension - temporal dimensions (controlled by ``time_dim_mode``) - vertical dimensions (controlled by ``level_dim_mode``) ``dim_roles`` is a mapping between the "roles" and the metadata keys representing the roles. The possible roles are as follows: - "number": metadata key interpreted as ensemble forecast members - "date": metadata key interpreted as date part of the "forecast_reference_time" - "time": metadata key interpreted as time part of the "forecast_reference_time" - "step": metadata key interpreted as forecast step - "forecast_reference_time": if not specified or None or empty the forecast reference time is built using the "date" and "time" roles - "valid_time": if not specified or None or empty the valid time is built using the "validityDate" and "validityTime" metadata keys - "level": metadata key interpreted as level - "level_type": metadata key interpreted as level type The default values are as follows: .. code-block:: python { "number": "number", "date": "dataDate", "time": "dataTime", "step": "step", "forecast_reference_time": None, "valid_date": None, "level": "level", "level_type": "typeOfLevel", } ``dims_roles`` behaves differently to the other kwargs in the sense that it does not override but update the default values. So e.g. to change only "number" in the defaults it is enough to specify: "dim_roles={"number": "perturbationNumber"}. * dim_name_from_role_name: bool, None If True, the dimension names are formed from the role names. Otherwise the dimension names are formed from the metadata keys specified in ``dim_roles``. Its default value (None) expands to True unless the ``profile`` overwrites it. Only used when no `fixed_dims`` are specified. *New in version 0.15.0*. * rename_dims: dict, None Mapping to rename dimensions. Default is None. * dims_as_attrs: str, or iterable of str, None Dimension or list of dimensions which should be turned to variable attributes if they have only one value for the given variable. Default is None. * time_dim_mode: str, None Define how predefined temporal dimensions are formed. The default is "forecast". The possible values are as follows: - "forecast": adds two dimensions: - "forecast_reference_time": built from the "date" and "time" roles (see ``dim_roles``) as np.datetime64 values - "step": built from the "step" role. When ``decode_times=True`` the values are np.timedelta64 - "valid_time": adds a dimension called "valid_time" as described by the "valid_time" role (see ``dim_roles``). Will contain np.datetime64 values. - "raw": the "date", "time" and "step" roles are turned into 3 separate dimensions * level_dim_mode: str, None Define how predefined vertical dimensions are formed. The default is "level". The possible values are: - "level": adds a single dimension according to the "level" role (see ``dim_roles``) - "level_per_type": adds a separate dimensions for each level type based on the "level" and "level_type" roles. - "level_and_type": Use a single dimension for combined level and type of level. * squeeze: bool, None Remove dimensions which have only one valid value. Not applies to dimensions in ``ensure_dims``. Its default value (None) expands to True unless the ``profile`` overwrites it. * add_valid_time_coord: bool, None If True, add the `valid_time` coordinate containing np.datetime64 values to the dataset. Only makes effect when ``time_dim_mode`` is not "valid_time". Its default value (None) expands to False unless the ``profile`` overwrites it. * decode_times: bool, None If True, decode date and datetime coordinates into datetime64 values. If False, leave coordinates representing date-like GRIB keys (e.g. "date", "validityDate") encoded as native int values. The default value (None) expands to True unless the ``profile`` overwrites it. * decode_timedelta: bool, None If True, decode coordinates representing time-like or duration-like GRIB keys (e.g. "time", "validityTime", "step") into timedelta64 values. If False, leave time-like coordinates encoded as native int values, while duration-like coordinates will be encoded as int with the units attached to the coordinate as the "units" attribute. If None (default), assume the same value of ``decode_times`` unless the ``profile`` overwrites it. * add_geo_coords: bool, None Add geographic coordinates to the dataset when field values are represented by a single "values" dimension. Its default value (None) expands to True unless the ``profile`` overwrites it. * flatten_values: bool, None If True, flatten the values per field resulting in a single dimension called "values" representing a field. If False, the field shape is used to form the field dimensions. When the fields are defined on an unstructured grid (e.g. reduced Gaussian) or are spectral (e.g. spherical harmonics) this option is ignored and the field values are always represented by a single "values" dimension. Its default value (None) expands to False unless the ``profile`` overwrites it. * attrs_mode: str, None Define how attributes are generated. Default is "fixed". The possible values are: - "fixed": Use the attributes defined in ``variable_attrs`` as variables attributes and ``global_attrs`` as global attributes. - "unique": Use all the attributes defined in ``attrs``, ``variable_attrs`` and ``global_attrs``. When an attribute has unique value for a dataset it will be a global attribute, otherwise it will be a variable attribute. However, this logic is only applied if a unique variable attribute can be a global attribute according to the CF conventions Appendix A. (e.g. "units" cannot be a global attribute). Additionally, keys in ``variable_attrs`` are always used as variable attributes, while keys in ``global_attrs`` are always used as global attributes. * attrs: str, number, callable, dict or list of these, None Attribute or list of attributes. Only used when ``attrs_mode`` is ``unique``. Its default value (None) expands to [] unless the ``profile`` overwrites it. The following attributes are supported: - str: Name of the attribute used as a metadata key to generate the value of the attribute. Can also be specified by prefixing with "key=" (e.g. "key=level"). When prefixed with "namespace=" it specifies a metadata namespace (e.g. "namespace=parameter"), which will be added as a dict to the attribute. - callable: A callable that takes a Metadata object and returns a dict of attributes - dict: A dictionary of attributes with the keys as the attribute names. If the value is a callable it takes the attribute name and a Metadata object and returns the value of the attribute. A str value prefixed with "key=" or "namespace=" is interpreted as explained above. Any other values are used as the pre-defined value for the attribute. * variable_attrs: str, number, callable, dict or list of these, None Variable attribute or attributes. For the allowed values see ``attrs``. Its default value (None) expands to [] unless the ``profile`` overwrites it. * global_attrs: str, number, dict or list of these, None Global attribute or attributes. For the allowed values see ``attrs``. Its default value (None) expands to [] unless the ``profile`` overwrites it. * coord_attrs: dict, None To be documented. Default is None. * add_earthkit_attrs: bool, None If True, add earthkit specific attributes to the dataset. Its default value (None) expands to True unless the ``profile`` overwrites it. * rename_attrs: dict, None A dictionary of attribute to rename. Default is None. * fill_metadata: dict, None Define fill_metadata values to metadata keys. Default is None. * remapping: dict, None Define new metadata keys for indexing. Default is None. * lazy_load: bool, None If True, the resulting Dataset will load data lazily from the underlying data source. If False, a Dataset holding all the data in memory and decoupled from the backend source will be created. Using ``lazy_load=False`` with ``release_source=True`` can provide optimised memory usage in certain cases. The default value of ``lazy_load`` (None) expands to True unless the ``profile`` overwrites it. * release_source: bool, None Only used when ``lazy_load=False``. If True, memory held in the input fields are released as soon as their values are copied into the resulting Dataset. This is done per field to avoid memory spikes. The release operation is currently only supported for GRIB fields stored entirely in memory, e.g. when read from a :ref:`stream `. When a field does not support the release operation, this option is ignored. Having run :obj:`to_xarray` the input data becomes unusable, so use this option carefully. The default value of ``release_source`` (None) expands to False unless the ``profile`` overwrites it. * allow_holes: bool, None If False, GRIB fields must form a full hypercube (without holes). If True, a dataset will be created from any GRIB fields and its coordinates will be a union of coordinates of the fields (outer join). Values corresponding to missing GRIB fields will be filled with NaN. The default value of ``allow_holes`` (None) expands to False unless the ``profile`` overwrites it. * strict: bool, None If True, perform stricter checks on hypercube consistency. Its default value (None) expands to False unless the ``profile`` overwrites it. * dtype: str, numpy.dtype or None Typecode or data-type of the array data. * array_backend: str, array namespace, None The array namespace to use for array operations. The default value (None) is expanded to "numpy". **Deprecated since version 0.19.0**. Please use ``array_namespace`` instead. In versions before 0.19.0 an :obj:`ArrayBackend` was also accepted here, which is no longer the case. * array_namespace: str, array namespace,None The array namespace to use for array operations. The default value (None) is expanded to "numpy". ** New in version 0.19.0**. * direct_backend: bool, None If True, the backend is used directly bypassing :py:meth:`xarray.open_dataset` and ignoring all non-backend related kwargs. If False, the data is read via :py:meth:`xarray.open_dataset`. Its default value (None) expands to False unless the ``profile`` overwrites it. When ``engine="cfgrib"`` the following engine specific kwargs are supported: * ignore_keys: list, None It specifies the metadata keys that should be ignored when reading the GRIB messages in the backend. Please note that is not supported by :xref:`cfgrib`, but implemented in earthkit-data. * For the rest of the supported keyword arguments, please refer to the :xref:`cfgrib` documentation. :type \*\*kwargs: :class:`dict`, *optional* :returns: When ``split_dims`` is unset a Dataset is returned. When ``engine="earthkit"`` and ``split_dims`` is set a tuple is returned. The first element of the tuple is the list of Datasets and the second element is the list of corresponding dictionaries with the spitting keys/values (one dictionary per Dataset). :rtype: :class:`Xarray Dataset` or :class:`tuple` .. rubric:: Notes The default values of ``xarray_open_dataset_kwargs`` or ``**kwargs`` passed to :py:func:`xarray.open_dataset` are as follows: - when ``engine="earthkit"``:: {"cache": True, "chunks": None, "engine": "earthkit"} - when ``engine="cfgrib"``:: {"backend_kwargs": {"errors": "raise", "ignore_keys": [], "squeeze": False,}, "cache": True, "chunks": None, "engine": "cfgrib"} Please note that settings ``errors="raise"`` and ``engine`` are always enforced and cannot be changed. .. rubric:: Examples >>> import earthkit.data >>> fs = earthkit.data.from_source("file", "test6.grib") >>> ds = fs.to_xarray(time_dim_mode="forecast") >>> # also possible to use the xarray_open_dataset_kwargs >>> ds = fs.to_xarray( ... xarray_open_dataset_kwargs={ ... "backend_kwargs": {"time_dim_mode": "forecast"} ... } ... ) .. py:method:: to_xarray_cfgrib(user_kwargs) .. py:method:: to_xarray_earthkit(user_kwargs) .. py:method:: unique_values(*coords, remapping=None, patches=None, progress_bar=False) Given a list of metadata attributes, such as date, param, levels, returns the list of unique values for each attributes .. py:property:: values Get all the fields' values as a 2D array. It is formed as the array of :obj:`GribField.values ` per field. .. seealso:: to_array >>> import earthkit.data >>> ds = earthkit.data.from_source("file", "docs/examples/test.grib") >>> for f in ds: ... print(f.values.shape) ... (209,) (209,) >>> v = ds.values >>> v.shape (2, 209) >>> v[0][:3] array([262.78027344, 267.44726562, 268.61230469]) :type: array-like .. py:method:: write(f, **kwargs) Write all the fields to a file object. :param f: The target file object. :type f: :class:`file object` :param \*\*kwargs: Other keyword arguments passed to the underlying field implementation. :type \*\*kwargs: :class:`dict`, *optional* .. seealso:: :obj:`read`