{ "cells": [ { "cell_type": "markdown", "id": "bc9477f3-1aa0-4539-acef-0bd2b43191ba", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Xarray engine: variable key" ] }, { "cell_type": "raw", "id": "236700a8-2212-4675-bd0d-d91159219a6a", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "The ``variable_key`` option in :py:meth:`~data.readers.grib.index.GribFieldList.to_xarray` controls what metadata key should be used to form the DataArray variables. By default it is \"param\", which is an alias to \"shortName\" for GRIB in earthkit-data.\n", "\n", "Please note it is also possible to generate an Xarray with a single dataarray containing all the parameters from a GRIB fieldlist. See e.g. the :ref:`/examples/xarray_engine_mono_variable.ipynb` notebook for details." ] }, { "cell_type": "code", "execution_count": 1, "id": "08b75c56-0b2f-4cc6-9637-b28ad2aa4455", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4b267deb657948f6a4992cde6a710f41", "version_major": 2, "version_minor": 0 }, "text/plain": [ "pl.grib: 0%| | 0.00/48.8k [00:00\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 176kB\n",
       "Dimensions:                  (forecast_reference_time: 4, step: 2, level: 2,\n",
       "                              latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...\n",
       "  * step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * level                    (level) int64 16B 500 700\n",
       "  * latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n",
       "  * longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n",
       "Data variables:\n",
       "    r                        (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n",
       "    t                        (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    levtype:      pl\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    date:         20240603\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 176kB\n", "Dimensions: (forecast_reference_time: 4, step: 2, level: 2,\n", " latitude: 19, longitude: 36)\n", "Coordinates:\n", " * forecast_reference_time (forecast_reference_time) datetime64[ns] 32B 202...\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * level (level) int64 16B 500 700\n", " * latitude (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n", "Data variables:\n", " r (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n", " t (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " levtype: pl\n", " type: fc\n", " expver: 0001\n", " date: 20240603\n", " time: 0\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import earthkit.data as ekd\n", "ds_fl = ekd.from_source(\"sample\", \"pl.grib\")\n", "ds = ds_fl.to_xarray()\n", "ds" ] }, { "cell_type": "markdown", "id": "7c2697d5-882e-4b47-977d-f45a91c222c2", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### The param_level key" ] }, { "cell_type": "markdown", "id": "08015f0b-65de-44f0-beda-c75390615352", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "The built-in \"param_level\" metadata key combines the values of the \"param\" and \"level\" metadata keys as a str. We can use it as ``variable_key`` to build our Xarray." ] }, { "cell_type": "code", "execution_count": 2, "id": "4898ceb3-4657-4397-b1d8-3bc1110b86eb", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 176kB\n",
       "Dimensions:                  (forecast_reference_time: 4, step: 2,\n",
       "                              latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...\n",
       "  * step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n",
       "  * longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n",
       "Data variables:\n",
       "    r500                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r700                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    t500                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    t700                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    levtype:      pl\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    date:         20240603\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 176kB\n", "Dimensions: (forecast_reference_time: 4, step: 2,\n", " latitude: 19, longitude: 36)\n", "Coordinates:\n", " * forecast_reference_time (forecast_reference_time) datetime64[ns] 32B 202...\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * latitude (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n", "Data variables:\n", " r500 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r700 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " t500 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " t700 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " levtype: pl\n", " type: fc\n", " expver: 0001\n", " date: 20240603\n", " time: 0\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = ds_fl.to_xarray(variable_key=\"param_level\")\n", "ds" ] }, { "cell_type": "markdown", "id": "0921516f-74b2-417a-a85a-378525090348", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "This technique can come in handy when there are parameters with different level types in the input data." ] }, { "cell_type": "code", "execution_count": 3, "id": "f55fe598-7f80-4e9a-87b3-43131f78f2e1", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d314ccebbb2d4072b01a5774f4bfd637", "version_major": 2, "version_minor": 0 }, "text/plain": [ "mixed_pl_sfc.grib: 0%| | 0.00/390k [00:00\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 1MB\n",
       "Dimensions:                  (forecast_reference_time: 4, step: 2,\n",
       "                              latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...\n",
       "  * step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n",
       "  * longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n",
       "Data variables: (12/32)\n",
       "    2t0                      (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    msl0                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r1000                    (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r300                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r400                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r500                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    ...                       ...\n",
       "    z1000                    (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z300                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z400                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z500                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z700                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z850                     (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    date:         20240603\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 1MB\n", "Dimensions: (forecast_reference_time: 4, step: 2,\n", " latitude: 19, longitude: 36)\n", "Coordinates:\n", " * forecast_reference_time (forecast_reference_time) datetime64[ns] 32B 202...\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * latitude (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n", "Data variables: (12/32)\n", " 2t0 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " msl0 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r1000 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r300 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r400 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r500 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " ... ...\n", " z1000 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z300 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z400 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z500 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z700 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z850 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " type: fc\n", " expver: 0001\n", " date: 20240603\n", " time: 0\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds_fl = ekd.from_source(\"sample\", \"mixed_pl_sfc.grib\")\n", "ds = ds_fl.to_xarray(variable_key=\"param_level\")\n", "ds" ] }, { "cell_type": "markdown", "id": "1939ee80-8c1e-4498-8a80-d5e6ac65f8e7", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Using remapping" ] }, { "cell_type": "markdown", "id": "7e22e364-2518-4460-ab35-64230d49a703", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "We can take it one step further and define a metadata key that combines the param, the level and the level type into a single key. We can achieve it by using the ``remapping`` option." ] }, { "cell_type": "code", "execution_count": 4, "id": "e1bc4e80-2efc-460c-96b0-26553bed7591", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2fb6d5cb8cf44e839b0ec741cb1f041b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "mixed_pl_sfc.grib: 0%| | 0.00/390k [00:00\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 1MB\n",
       "Dimensions:                  (forecast_reference_time: 4, step: 2,\n",
       "                              latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...\n",
       "  * step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n",
       "  * longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n",
       "Data variables: (12/32)\n",
       "    2t_sfc                   (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    msl_sfc                  (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r_1000_pl                (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r_300_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r_400_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    r_500_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    ...                       ...\n",
       "    z_1000_pl                (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z_300_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z_400_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z_500_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z_700_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    z_850_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    date:         20240603\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 1MB\n", "Dimensions: (forecast_reference_time: 4, step: 2,\n", " latitude: 19, longitude: 36)\n", "Coordinates:\n", " * forecast_reference_time (forecast_reference_time) datetime64[ns] 32B 202...\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * latitude (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n", "Data variables: (12/32)\n", " 2t_sfc (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " msl_sfc (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r_1000_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r_300_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r_400_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " r_500_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " ... ...\n", " z_1000_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z_300_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z_400_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z_500_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z_700_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " z_850_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " type: fc\n", " expver: 0001\n", " date: 20240603\n", " time: 0\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds_fl = ekd.from_source(\"sample\", \"mixed_pl_sfc.grib\")\n", "ds = ds_fl.to_xarray(variable_key=\"p_l_t\", remapping={\"p_l_t\": \"{param}_{levelist}_{levtype}\"})\n", "ds" ] }, { "cell_type": "markdown", "id": "374ea92f-5b3f-4134-adfb-1c93dc1258f7", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "This technique is particularly useful when the same parameter is available on multiple level types in the input data. In this case using \"param_level\" does not result in a full hypercube, however the same ``remapping`` that we used above does." ] }, { "cell_type": "code", "execution_count": 5, "id": "39cbb360-43c4-416e-956b-8b6cfadda26c", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b69825ffceb9401da6c4b0de9c92ed99", "version_major": 2, "version_minor": 0 }, "text/plain": [ "mixed_pl_ml.grib: 0%| | 0.00/176k [00:00\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 351kB\n",
       "Dimensions:                  (forecast_reference_time: 4, step: 2,\n",
       "                              latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...\n",
       "  * step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n",
       "  * longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n",
       "Data variables:\n",
       "    t_137_ml                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    t_500_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    t_700_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    t_90_ml                  (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    u_137_ml                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    u_500_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    u_700_pl                 (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "    u_90_ml                  (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    date:         20240603\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 351kB\n", "Dimensions: (forecast_reference_time: 4, step: 2,\n", " latitude: 19, longitude: 36)\n", "Coordinates:\n", " * forecast_reference_time (forecast_reference_time) datetime64[ns] 32B 202...\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * latitude (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n", "Data variables:\n", " t_137_ml (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " t_500_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " t_700_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " t_90_ml (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " u_137_ml (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " u_500_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " u_700_pl (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", " u_90_ml (forecast_reference_time, step, latitude, longitude) float64 44kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " type: fc\n", " expver: 0001\n", " date: 20240603\n", " time: 0\n", " domain: g\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds_fl = ekd.from_source(\"sample\", \"mixed_pl_ml.grib\")\n", "ds = ds_fl.to_xarray(variable_key=\"p_l_t\", remapping={\"p_l_t\": \"{param}_{levelist}_{levtype}\"})\n", "ds" ] }, { "cell_type": "code", "execution_count": null, "id": "4be3d1c2-873b-42a7-9a92-468d38ee64e2", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "dev", "language": "python", "name": "dev" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.12" } }, "nbformat": 4, "nbformat_minor": 5 }