{ "cells": [ { "cell_type": "markdown", "id": "7b1f3038-fbf6-424e-bfc1-14f23e5559d1", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Xarray engine: temporal options" ] }, { "cell_type": "raw", "id": "750aff09-ea6a-4b00-ae64-4d36a23079bb", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "We can convert a GRIB fieldlist to Xarray with :py:meth:`~data.readers.grib.index.GribFieldList.to_xarray`. This notebook discusses the **temporal** options used by this method." ] }, { "cell_type": "markdown", "id": "34202724-dcba-479c-b3ee-643326a73e0b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "First, we get some forecast GRIB data defined on pressure levels and read it into a GRIB fieldlist." ] }, { "cell_type": "code", "execution_count": 1, "id": "6c4a57a5-8693-4886-957e-669c31fa9cd3", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "8dca45f059a04a48898e26443d3b1a64", "version_major": 2, "version_minor": 0 }, "text/plain": [ "pl.grib: 0%| | 0.00/48.8k [00:00`)." ] }, { "cell_type": "markdown", "id": "12108792-3fd6-45fc-8dd2-b39ced33225f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### time_dim_mode=raw" ] }, { "cell_type": "markdown", "id": "7299a832-5fff-486f-8e75-a970a36cb89a", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "When ``time_dim_mode=\"raw\"`` the \"date\", \"time\" and \"step\" roles are used to form the temporal dimensions." ] }, { "cell_type": "code", "execution_count": 2, "id": "f4dd2a41-f073-411b-82f8-b6d7366963c4", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 176kB\n",
       "Dimensions:    (date: 2, time: 2, step: 2, level: 2, latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * date       (date) datetime64[ns] 16B 2024-06-03 2024-06-04\n",
       "  * time       (time) timedelta64[ns] 16B 00:00:00 12:00:00\n",
       "  * step       (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * level      (level) int64 16B 500 700\n",
       "  * latitude   (latitude) float64 152B 90.0 80.0 70.0 60.0 ... -70.0 -80.0 -90.0\n",
       "  * longitude  (longitude) float64 288B 0.0 10.0 20.0 30.0 ... 330.0 340.0 350.0\n",
       "Data variables:\n",
       "    r          (date, time, step, level, latitude, longitude) float64 88kB ...\n",
       "    t          (date, time, step, level, latitude, longitude) float64 88kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    levtype:      pl\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 176kB\n", "Dimensions: (date: 2, time: 2, step: 2, level: 2, latitude: 19, longitude: 36)\n", "Coordinates:\n", " * date (date) datetime64[ns] 16B 2024-06-03 2024-06-04\n", " * time (time) timedelta64[ns] 16B 00:00:00 12:00:00\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * level (level) int64 16B 500 700\n", " * latitude (latitude) float64 152B 90.0 80.0 70.0 60.0 ... -70.0 -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 20.0 30.0 ... 330.0 340.0 350.0\n", "Data variables:\n", " r (date, time, step, level, latitude, longitude) float64 88kB ...\n", " t (date, time, step, level, latitude, longitude) float64 88kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " levtype: pl\n", " type: fc\n", " expver: 0001\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = ds_fl.to_xarray(time_dim_mode=\"raw\")\n", "ds" ] }, { "cell_type": "markdown", "id": "12f84d47-37b2-4744-ae9d-3825d2ac6880", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### time_dim_mode=forecast" ] }, { "cell_type": "markdown", "id": "90719e23-17cc-47f5-a7e4-24494484520d", "metadata": { "editable": true, "raw_mimetype": "", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "When ``time_dim_mode=\"forecast\"`` the \"date\" and \"time\" roles are merged to form the dimension \"forecats_reference_time\". It also adds the \"step\" dimension." ] }, { "cell_type": "code", "execution_count": 3, "id": "1aed2676-9b6d-4491-90e1-ce5896fa064b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 176kB\n",
       "Dimensions:                  (forecast_reference_time: 4, step: 2, level: 2,\n",
       "                              latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * forecast_reference_time  (forecast_reference_time) datetime64[ns] 32B 202...\n",
       "  * step                     (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * level                    (level) int64 16B 500 700\n",
       "  * latitude                 (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n",
       "  * longitude                (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n",
       "Data variables:\n",
       "    r                        (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n",
       "    t                        (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    levtype:      pl\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    date:         20240603\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 176kB\n", "Dimensions: (forecast_reference_time: 4, step: 2, level: 2,\n", " latitude: 19, longitude: 36)\n", "Coordinates:\n", " * forecast_reference_time (forecast_reference_time) datetime64[ns] 32B 202...\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * level (level) int64 16B 500 700\n", " * latitude (latitude) float64 152B 90.0 80.0 ... -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 ... 340.0 350.0\n", "Data variables:\n", " r (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n", " t (forecast_reference_time, step, level, latitude, longitude) float64 88kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " levtype: pl\n", " type: fc\n", " expver: 0001\n", " date: 20240603\n", " time: 0\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# default mode\n", "ds = ds_fl.to_xarray(time_dim_mode=\"forecast\")\n", "ds" ] }, { "cell_type": "markdown", "id": "114440f2-e8a0-4348-b1ac-21674453f553", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### time_dim_mode=valid_time" ] }, { "cell_type": "markdown", "id": "b72d1c6b-259e-411d-acbd-613dcd9ecc5a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "When ``time_dim_mode=\"valid_time\"`` the only temporal dimension is \"valid_time\". By default, it is built from the values of the \"validityDate\" and \"validityTime\" ecCodes GRIB keys. This dimension can only be generated if each GRIB field has a distinct valid time, so it typically fits for analysis/climate data." ] }, { "cell_type": "code", "execution_count": 4, "id": "5728c723-2eb7-4bc2-9768-3263b6ba22b8", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "98fb8660778641739b5bdba816ad80ef", "version_major": 2, "version_minor": 0 }, "text/plain": [ "msl_analysis.grib: 0%| | 0.00/1.88k [00:00\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 6kB\n",
       "Dimensions:     (valid_time: 8, latitude: 7, longitude: 12)\n",
       "Coordinates:\n",
       "  * valid_time  (valid_time) datetime64[ns] 64B 2016-09-25 ... 2016-09-26T18:...\n",
       "  * latitude    (latitude) float64 56B 80.0 70.0 60.0 50.0 40.0 30.0 20.0\n",
       "  * longitude   (longitude) float64 96B -70.0 -60.0 -50.0 ... 20.0 30.0 40.0\n",
       "Data variables:\n",
       "    msl         (valid_time, latitude, longitude) float64 5kB ...\n",
       "Attributes: (12/13)\n",
       "    param:        msl\n",
       "    paramId:      151\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    levtype:      sfc\n",
       "    type:         an\n",
       "    ...           ...\n",
       "    date:         20160925\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 6kB\n", "Dimensions: (valid_time: 8, latitude: 7, longitude: 12)\n", "Coordinates:\n", " * valid_time (valid_time) datetime64[ns] 64B 2016-09-25 ... 2016-09-26T18:...\n", " * latitude (latitude) float64 56B 80.0 70.0 60.0 50.0 40.0 30.0 20.0\n", " * longitude (longitude) float64 96B -70.0 -60.0 -50.0 ... 20.0 30.0 40.0\n", "Data variables:\n", " msl (valid_time, latitude, longitude) float64 5kB ...\n", "Attributes: (12/13)\n", " param: msl\n", " paramId: 151\n", " class: od\n", " stream: oper\n", " levtype: sfc\n", " type: an\n", " ... ...\n", " date: 20160925\n", " time: 0\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds_fl_an = ekd.from_source(\"sample\", \"msl_analysis.grib\")\n", "ds_xr = ds_fl_an.to_xarray(time_dim_mode=\"valid_time\")\n", "ds_xr" ] }, { "cell_type": "markdown", "id": "f2001e5c-1fd4-4f86-b024-68bfb8c4a11a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "This mode can also be used for suitable forecasts data. To use it for the original forecast data first we need to filter it." ] }, { "cell_type": "code", "execution_count": 5, "id": "38a38b2d-a5be-4fba-9bcd-b98ee05ca94b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 44kB\n",
       "Dimensions:     (valid_time: 2, level: 2, latitude: 19, longitude: 36)\n",
       "Coordinates:\n",
       "  * valid_time  (valid_time) datetime64[ns] 16B 2024-06-03 2024-06-03T06:00:00\n",
       "  * level       (level) int64 16B 500 700\n",
       "  * latitude    (latitude) float64 152B 90.0 80.0 70.0 ... -70.0 -80.0 -90.0\n",
       "  * longitude   (longitude) float64 288B 0.0 10.0 20.0 ... 330.0 340.0 350.0\n",
       "Data variables:\n",
       "    r           (valid_time, level, latitude, longitude) float64 22kB ...\n",
       "    t           (valid_time, level, latitude, longitude) float64 22kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    levtype:      pl\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    date:         20240603\n",
       "    time:         0\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 44kB\n", "Dimensions: (valid_time: 2, level: 2, latitude: 19, longitude: 36)\n", "Coordinates:\n", " * valid_time (valid_time) datetime64[ns] 16B 2024-06-03 2024-06-03T06:00:00\n", " * level (level) int64 16B 500 700\n", " * latitude (latitude) float64 152B 90.0 80.0 70.0 ... -70.0 -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 20.0 ... 330.0 340.0 350.0\n", "Data variables:\n", " r (valid_time, level, latitude, longitude) float64 22kB ...\n", " t (valid_time, level, latitude, longitude) float64 22kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " levtype: pl\n", " type: fc\n", " expver: 0001\n", " date: 20240603\n", " time: 0\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = ds_fl.sel(date=20240603, time=0).to_xarray(time_dim_mode=\"valid_time\")\n", "ds" ] }, { "cell_type": "markdown", "id": "0c2ecf65-97e7-4379-a193-db056ad73098", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "### Adding valid_time coord" ] }, { "cell_type": "markdown", "id": "0e2ce07d-82cf-4474-ac99-58ecd890099b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "When ``add_valid_time_coord=True`` it adds the coordinate `valid_time` containing the valid times for all the different temporal dimensions as datetime64. When ``time_dim_mode=\"valid_time\"`` this coordinate is always added irrespective of the value of ``add_valid_time_coord``." ] }, { "cell_type": "code", "execution_count": 6, "id": "a6f280b9-83c5-41ce-9419-b52dad7c173a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 176kB\n",
       "Dimensions:     (date: 2, time: 2, step: 2, level: 2, latitude: 19,\n",
       "                 longitude: 36)\n",
       "Coordinates:\n",
       "  * date        (date) datetime64[ns] 16B 2024-06-03 2024-06-04\n",
       "  * time        (time) timedelta64[ns] 16B 00:00:00 12:00:00\n",
       "  * step        (step) timedelta64[ns] 16B 00:00:00 06:00:00\n",
       "  * level       (level) int64 16B 500 700\n",
       "    valid_time  (date, time, step) datetime64[ns] 64B ...\n",
       "  * latitude    (latitude) float64 152B 90.0 80.0 70.0 ... -70.0 -80.0 -90.0\n",
       "  * longitude   (longitude) float64 288B 0.0 10.0 20.0 ... 330.0 340.0 350.0\n",
       "Data variables:\n",
       "    r           (date, time, step, level, latitude, longitude) float64 88kB ...\n",
       "    t           (date, time, step, level, latitude, longitude) float64 88kB ...\n",
       "Attributes:\n",
       "    class:        od\n",
       "    stream:       oper\n",
       "    levtype:      pl\n",
       "    type:         fc\n",
       "    expver:       0001\n",
       "    domain:       g\n",
       "    number:       0\n",
       "    Conventions:  CF-1.8\n",
       "    institution:  ECMWF
" ], "text/plain": [ " Size: 176kB\n", "Dimensions: (date: 2, time: 2, step: 2, level: 2, latitude: 19,\n", " longitude: 36)\n", "Coordinates:\n", " * date (date) datetime64[ns] 16B 2024-06-03 2024-06-04\n", " * time (time) timedelta64[ns] 16B 00:00:00 12:00:00\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * level (level) int64 16B 500 700\n", " valid_time (date, time, step) datetime64[ns] 64B ...\n", " * latitude (latitude) float64 152B 90.0 80.0 70.0 ... -70.0 -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 20.0 ... 330.0 340.0 350.0\n", "Data variables:\n", " r (date, time, step, level, latitude, longitude) float64 88kB ...\n", " t (date, time, step, level, latitude, longitude) float64 88kB ...\n", "Attributes:\n", " class: od\n", " stream: oper\n", " levtype: pl\n", " type: fc\n", " expver: 0001\n", " domain: g\n", " number: 0\n", " Conventions: CF-1.8\n", " institution: ECMWF" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = ds_fl.to_xarray(time_dim_mode=\"raw\", add_valid_time_coord=True)\n", "ds" ] }, { "cell_type": "markdown", "id": "68723782-0bf3-4b1d-a09a-aa966d004e30", "metadata": {}, "source": [ "### Decoding temporal coords" ] }, { "cell_type": "markdown", "id": "72c38ceb-2cd6-408a-827d-d3af1b0f8d4a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "When ``decode_times=True`` (the default) the follwing coordinates will be stored as datetime64:\n", "\n", "- coordinates representing the date-like roles or GRIB keys (e.g. \"date\", \"validityDate\" etc.)\n", "- datetime coordinates (e.g. \"forecast_reference_time\" etc.)\n", "\n", "When ``decode_timedelta=True`` (the default) the following coordinates will be stored as timedelta64:\n", "\n", "- coordinates representing the time-like roles or GRIB keys (e.g. \"time\", \"validityTime\" etc.)\n", "- duration-like coordinates (e.g. \"step\", \"endStep\")" ] }, { "cell_type": "code", "execution_count": 7, "id": "429337ba-5037-4cb3-8a09-276381649732", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Coordinates:\n", " * date (date) datetime64[ns] 16B 2024-06-03 2024-06-04\n", " * time (time) timedelta64[ns] 16B 00:00:00 12:00:00\n", " * step (step) timedelta64[ns] 16B 00:00:00 06:00:00\n", " * level (level) int64 16B 500 700\n", " * latitude (latitude) float64 152B 90.0 80.0 70.0 60.0 ... -70.0 -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 20.0 30.0 ... 330.0 340.0 350.0" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = ds_fl.to_xarray(time_dim_mode=\"raw\", decode_times=True, decode_timedelta=True)\n", "ds.coords" ] }, { "cell_type": "markdown", "id": "50329cff-8588-4b3e-a835-f927d4e5ca06", "metadata": {}, "source": [ "When ``decode_times=False`` the following rules apply:\n", "\n", "- coordinates representing date-like GRIB keys (e.g. \"date\", \"validityDate\" etc.) will store the native GRIB int values (as yyyymmdd)\n", "- datetime coordinates (e.g. \"forecast_reference_time\" etc.) will store datetime64 values\n", "\n", "When ``decode_timedelta=False`` the following rules apply:\n", "\n", "- coordinates representing the time-like GRIB keys (e.g. \"time\", \"validityTime\" etc.) will store the native GRIB int values (as 100*hours + minutes)\n", "- duration-like (e.g. \"step\") coordinates will store int values with units indicated by the coordinate attribute \"units\"" ] }, { "cell_type": "code", "execution_count": 8, "id": "65b9016b-0c6a-4199-9601-fa4d58992372", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "Coordinates:\n", " * date (date) int64 16B 20240603 20240604\n", " * time (time) int64 16B 0 1200\n", " * step (step) int64 16B 0 6\n", " * level (level) int64 16B 500 700\n", " * latitude (latitude) float64 152B 90.0 80.0 70.0 60.0 ... -70.0 -80.0 -90.0\n", " * longitude (longitude) float64 288B 0.0 10.0 20.0 30.0 ... 330.0 340.0 350.0" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = ds_fl.to_xarray(time_dim_mode=\"raw\", decode_times=False, decode_timedelta=False)\n", "ds.coords" ] }, { "cell_type": "code", "execution_count": 9, "id": "09e9b62d-dcb5-4503-9a49-1f3b84fe2618", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "{'units': 'hours'}" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds.coords[\"step\"].attrs" ] }, { "cell_type": "code", "execution_count": null, "id": "1454d128-8915-470d-9100-89796fe32c78", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "dev", "language": "python", "name": "dev" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.12" } }, "nbformat": 4, "nbformat_minor": 5 }