{ "cells": [ { "cell_type": "markdown", "id": "7b1f3038-fbf6-424e-bfc1-14f23e5559d1", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Xarray engine: overview" ] }, { "cell_type": "markdown", "id": "17eec2d0-2841-4f3f-8786-4e5d8d12a83b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Earthkit-data comes with its own Xarray engine called \"earthkit\" to perform conversions between GRIB and Xarray data." ] }, { "cell_type": "markdown", "id": "34202724-dcba-479c-b3ee-643326a73e0b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "To start with, we get the example data will use in this notebook and read it into a GRIB fieldlist." ] }, { "cell_type": "code", "execution_count": 1, "id": "6c4a57a5-8693-4886-957e-669c31fa9cd3", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "8b2a598b3f264e2aa75a0cddab1650d2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "pl.grib: 0%| | 0.00/48.8k [00:00\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 176kB\n",
       "Dimensions:                  (forecast_reference_time: 4, step: 2, level: 2,\n",
       "                              latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...\n",
       "  * step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * level                    (level) int64 16B 500 700\n",
       "  * latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n",
       "  * longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n",
       "Data variables:\n",
       "    r                        (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n",
       "    t                        (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    levtype:      pl\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    date:         20240603\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 176kB\n", "Dimensions: (forecast_reference_time: 4, step: 2, level: 2,\n", " latitude: 19, longitude: 36)\n", "Coordinates:\n", " * forecast_reference_time (forecast_reference_time) datetime64[ns] 32B 202...\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * level (level) int64 16B 500 700\n", " * latitude (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n", "Data variables:\n", " r (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n", " t (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " levtype: pl\n", " type: fc\n", " expver: 0001\n", " date: 20240603\n", " time: 0\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = ds_fl.to_xarray()\n", "ds" ] }, { "cell_type": "raw", "id": "30df618b-90f1-41c5-a6eb-da3f451d4c3e", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ ":py:meth:`~data.readers.grib.index.GribFieldList.to_xarray` has a large number of keyword arguments to control how the Xarray dataset is generated. To simplify the usage we can define :ref:`profiles ` providing custom defaults for most of the keyword arguments. At the moment, there are 2 pre-defined profiles available: \"mars\" (the default) and \"grib\". We can pass them :py:meth:`~data.readers.grib.index.GribFieldList.to_xarray` via the ``profile`` kwarg." ] }, { "cell_type": "markdown", "id": "6f7a1692-f382-4c14-88dc-df28a7522f9f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "### Writing back to GRIB" ] }, { "cell_type": "markdown", "id": "f9324c45-127b-4971-92f0-553fa6ac8363", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "This is an **experimental** feature!\n", "\n", "In order to write back the Xarray into a GRIB it has to keep the original variable attributes that the eartkit engine generated. By default, variable attributes are not kept in Xarray computations so we need to set the global Xarray ``keep_attrs`` option to enable it as shown in the following cell: " ] }, { "cell_type": "code", "execution_count": 3, "id": "e8e5d214-bf09-4a8a-a944-26f7e8fc5b8d", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "import xarray as xr\n", "xr.set_options(keep_attrs=True)\n", "\n", "ds = ds_fl.to_xarray()\n", "ds += 1" ] }, { "cell_type": "markdown", "id": "04d442d9-27f8-4f46-ae35-41a85d765f8d", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Generating a fieldlist" ] }, { "cell_type": "raw", "id": "80eab6d7-bb06-4534-ab83-4c2e4c49fab6", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "To create GRIB fieldlist we need to call :py:meth:`to_fieldlist()` on the \"earthkit\" accessor. The result is an array fieldlist holding all the data in memory." ] }, { "cell_type": "code", "execution_count": 4, "id": "8c7856fb-c03a-4b27-976c-76724635bea7", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmfrisobaricInhPa5002024060300fc0regular_ll
1ecmfrisobaricInhPa7002024060300fc0regular_ll
2ecmfrisobaricInhPa5002024060306fc0regular_ll
3ecmfrisobaricInhPa7002024060306fc0regular_ll
4ecmfrisobaricInhPa5002024060312000fc0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange \\\n", "0 ecmf r isobaricInhPa 500 20240603 0 0 \n", "1 ecmf r isobaricInhPa 700 20240603 0 0 \n", "2 ecmf r isobaricInhPa 500 20240603 0 6 \n", "3 ecmf r isobaricInhPa 700 20240603 0 6 \n", "4 ecmf r isobaricInhPa 500 20240603 1200 0 \n", "\n", " dataType number gridType \n", "0 fc 0 regular_ll \n", "1 fc 0 regular_ll \n", "2 fc 0 regular_ll \n", "3 fc 0 regular_ll \n", "4 fc 0 regular_ll " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds_fl1 = ds.earthkit.to_fieldlist()\n", "ds_fl1.head()" ] }, { "cell_type": "markdown", "id": "8a7c0b90-9b87-4ec8-a44e-90eb4f5ee089", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "We can see that the GRIB field values changed as expected if we compare the original and resulting fieldlists." ] }, { "cell_type": "code", "execution_count": 8, "id": "5ad32a3e-2f48-49f5-b207-15e89c397fba", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "(254.25649845948692, 255.25649845948692)" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m_0 = ds_fl.sel(param=\"t\", step=6, level=500)[0].values.mean() \n", "m_1 = ds_fl1.sel(param=\"t\", step=6, level=500)[0].values.mean()\n", "m_0, m_1" ] }, { "cell_type": "markdown", "id": "39237f5c-c02d-49d7-88eb-1839875e2ecf", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Generating a GRIB file" ] }, { "cell_type": "raw", "id": "08f6dbd3-e4e2-447d-a85f-e2948f157ca1", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Once we have the GRIB fieldlist it can be saved to disk using :py:meth:`~data.readers.grib.index.GribFieldList.save` method." ] }, { "cell_type": "code", "execution_count": 6, "id": "e5b9c5ab-cc08-450a-aa30-691b405b3230", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmfrisobaricInhPa5002024060300fc0regular_ll
1ecmfrisobaricInhPa7002024060300fc0regular_ll
2ecmfrisobaricInhPa5002024060306fc0regular_ll
3ecmfrisobaricInhPa7002024060306fc0regular_ll
4ecmfrisobaricInhPa5002024060312000fc0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange \\\n", "0 ecmf r isobaricInhPa 500 20240603 0 0 \n", "1 ecmf r isobaricInhPa 700 20240603 0 0 \n", "2 ecmf r isobaricInhPa 500 20240603 0 6 \n", "3 ecmf r isobaricInhPa 700 20240603 0 6 \n", "4 ecmf r isobaricInhPa 500 20240603 1200 0 \n", "\n", " dataType number gridType \n", "0 fc 0 regular_ll \n", "1 fc 0 regular_ll \n", "2 fc 0 regular_ll \n", "3 fc 0 regular_ll \n", "4 fc 0 regular_ll " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds_fl1.to_target(\"file\", \"_from_xr_1.grib\")\n", "ekd.from_source(\"file\", \"_from_xr_1.grib\").head()" ] }, { "cell_type": "raw", "id": "bb3a8f0a-d1ea-461b-ae4e-8b3fa312226e", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "It is also possible to directly write the Xarray into a GRIB file when calling :py:meth:`~data.utils.xarray.engine.XarrayEarthkit.to_grib` on the ``earthkit`` accessor. This will be a more memory efficient way to write the data to disk than generating a fieldlist first." ] }, { "cell_type": "code", "execution_count": 7, "id": "d12b5452-822e-4d96-8ddd-0e9f7b7c982c", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmfrisobaricInhPa5002024060300fc0regular_ll
1ecmfrisobaricInhPa7002024060300fc0regular_ll
2ecmfrisobaricInhPa5002024060306fc0regular_ll
3ecmfrisobaricInhPa7002024060306fc0regular_ll
4ecmfrisobaricInhPa5002024060312000fc0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange \\\n", "0 ecmf r isobaricInhPa 500 20240603 0 0 \n", "1 ecmf r isobaricInhPa 700 20240603 0 0 \n", "2 ecmf r isobaricInhPa 500 20240603 0 6 \n", "3 ecmf r isobaricInhPa 700 20240603 0 6 \n", "4 ecmf r isobaricInhPa 500 20240603 1200 0 \n", "\n", " dataType number gridType \n", "0 fc 0 regular_ll \n", "1 fc 0 regular_ll \n", "2 fc 0 regular_ll \n", "3 fc 0 regular_ll \n", "4 fc 0 regular_ll " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds.earthkit.to_grib(\"_from_xr_2.grib\")\n", "ekd.from_source(\"file\", \"_from_xr_2.grib\").head()" ] }, { "cell_type": "code", "execution_count": null, "id": "3ee920a2-24ec-4462-975f-d2b1f7a7b293", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "dev", "language": "python", "name": "dev" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.12" } }, "nbformat": 4, "nbformat_minor": 5 }