{ "cells": [ { "cell_type": "markdown", "id": "e5dd8e8d-437e-4ef4-9cf5-ef2a4cd09e81", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## GRIB encoder" ] }, { "cell_type": "raw", "id": "b7ff9549-beae-4ef9-8076-a7a73263d653", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "An :ref:`encoder ` is used to generate data in a suitable format that can be written/added to a :ref:`target `. Encoders are typically used implicitly via :func:`to_target` but we can also instantiate an object and work with it directly." ] }, { "cell_type": "code", "execution_count": 1, "id": "9d9760b5-fb39-4e0f-9f3f-e6bf2c15e1a2", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "import earthkit.data as ekd" ] }, { "cell_type": "code", "execution_count": 2, "id": "20bc3e5a-0e84-419f-8d3c-1887b2cd2cac", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmf2tsurface02020051312000an0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange dataType \\\n", "0 ecmf 2t surface 0 20200513 1200 0 an \n", "\n", " number gridType \n", "0 0 regular_ll " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# get some input GRIB data\n", "ds = ekd.from_source(\"file\", \"test.grib\")\n", "ds[0].ls()" ] }, { "cell_type": "code", "execution_count": 3, "id": "a375d5a3-da37-40ab-a851-92c40e3f4b32", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# create a GribEncoder\n", "encoder = ekd.create_encoder(\"grib\")\n", "encoder" ] }, { "cell_type": "raw", "id": "9815a232-4848-4950-a9ee-6e07d593c6fe", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "The method to call is :meth:`GribEncoder.encode`. The ``template`` argument can be a GRIB field. In the example below :meth:`GribEncoder.encode` will simply create a copy (clone) of the underlying GRIB handle in the field." ] }, { "cell_type": "code", "execution_count": 4, "id": "77a3548b-cac3-4749-bc0d-a0a2f02e20b1", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r = encoder.encode(template=ds[0])\n", "r" ] }, { "cell_type": "markdown", "id": "17bf5946-71b7-40c3-a122-78d4660c9b84", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "The resulting object can be used in various ways." ] }, { "cell_type": "code", "execution_count": 5, "id": "e2b4d8df-1ab6-4c83-b65a-7ff510c09ec1", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "b'GRIB\\x00\\x02\\x0e\\x01\\x00\\x00'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r.to_bytes()[:10]" ] }, { "cell_type": "code", "execution_count": 6, "id": "8ae392c9-fa79-4ff6-9ef3-4f13bf69329f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmf2tsurface02020051312000an0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange dataType \\\n", "0 ecmf 2t surface 0 20200513 1200 0 an \n", "\n", " number gridType \n", "0 0 regular_ll " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "f = r.to_field()\n", "f.ls()" ] }, { "cell_type": "code", "execution_count": 7, "id": "8a9f93d9-d04e-48d9-9e7b-94851f270882", "metadata": {}, "outputs": [], "source": [ "with open(\"_res_encoded.grib\", \"wb\") as out:\n", " r.to_file(out)" ] }, { "cell_type": "markdown", "id": "9873e7b3-0bc2-4a7c-8bdc-3d16f9d89e67", "metadata": {}, "source": [ "#### Specifying metadata" ] }, { "cell_type": "markdown", "id": "e3db5691-9529-4a28-92a6-447189c24335", "metadata": {}, "source": [ "When we specify metadata it will be written into the resulting GRIB fields." ] }, { "cell_type": "code", "execution_count": 8, "id": "ac1e1525-c9b9-497b-841d-17f830b1a400", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmf2tsurface02021051412000an0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange dataType \\\n", "0 ecmf 2t surface 0 20210514 1200 0 an \n", "\n", " number gridType \n", "0 0 regular_ll " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r = encoder.encode(template=ds[0], metadata={\"date\": 20210514})\n", "r.to_field().ls()" ] }, { "cell_type": "code", "execution_count": 9, "id": "05028656-a641-4530-8bab-02b9ec97be76", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmf2tsurface02021051412000an0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange dataType \\\n", "0 ecmf 2t surface 0 20210514 1200 0 an \n", "\n", " number gridType \n", "0 0 regular_ll " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r = encoder.encode(template=ds[0], date=20210514)\n", "r.to_field().ls()" ] }, { "cell_type": "markdown", "id": "aaded4d0-0996-4064-8719-a480eb068409", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Specifying new values" ] }, { "cell_type": "markdown", "id": "aa339c80-7bb8-46c0-b6a0-08b170741a10", "metadata": {}, "source": [ "To replace the values in the resulting fields we need to use the ``values`` keyword argument." ] }, { "cell_type": "code", "execution_count": 10, "id": "83a519ba-d744-488e-bbb1-cf4fff5a15db", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "(315.4599609375, 316.4599609375)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vals = ds[0].values\n", "r = encoder.encode(values=vals + 1, template=ds[0])\n", "ds[0].values.max(), r.to_field().values.max()" ] }, { "cell_type": "markdown", "id": "986e8d46-fad4-4d1c-9009-5b6e7d260010", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Specifying a field" ] }, { "cell_type": "markdown", "id": "d6d3860a-c46c-4d30-8e2f-c517f0fb254d", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "When a field is specified as the ``data`` it is used as a template." ] }, { "cell_type": "code", "execution_count": 11, "id": "4b6c42c2-cf5c-4bf5-94a6-c5581e3cd665", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmfmslsurface02021051412000an0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange dataType \\\n", "0 ecmf msl surface 0 20210514 1200 0 an \n", "\n", " number gridType \n", "0 0 regular_ll " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r = encoder.encode(data=ds[1], metadata={\"date\": 20210514})\n", "r.to_field().ls()" ] }, { "cell_type": "markdown", "id": "667bdff9-d36a-4627-8849-38f17d3ff8bb", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "When both ``data`` and ``template`` are specified, the values from the field in ``data`` will be copied into the GRIB message created from the template." ] }, { "cell_type": "code", "execution_count": 12, "id": "05b95db7-f5b4-4c9f-8f44-b0a574bc5c7f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmf2tsurface02021051412000an0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange dataType \\\n", "0 ecmf 2t surface 0 20210514 1200 0 an \n", "\n", " number gridType \n", "0 0 regular_ll " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r = encoder.encode(data=ds[1], template=ds[0], metadata={\"date\": 20210514})\n", "r.to_field().ls()" ] }, { "cell_type": "markdown", "id": "77b56c59-b2ba-40bb-a6b0-18f94bef978b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "We cannot use ``data``, ``values`` and ``template`` together." ] }, { "cell_type": "code", "execution_count": 13, "id": "a8337dbc-1838-4383-9d1c-c4a4763b661b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cannot provide data, values and template together\n" ] } ], "source": [ "try:\n", " r = encoder.encode(data=ds[1], template=ds[0], values=vals, metadata={\"date\": 20210514})\n", "except Exception as e:\n", " print(e)" ] }, { "cell_type": "markdown", "id": "5638a8cf-0cb7-4ac0-b50d-fbdff299fc80", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Encoding without a template" ] }, { "cell_type": "markdown", "id": "41e869d3-1c30-41d2-b7d1-ff72a988c439", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "It is possible to encode GRIB data without providing a template using only values and metadata. This is an **experimental feature** and only works for certain metadata keys and the grid has to be either global lat-lon or reduced Gaussian grid. The geography is inferred from the shape of the specified values." ] }, { "cell_type": "code", "execution_count": 14, "id": "8951336c-382a-47be-8897-6df7c643c573", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmf2theightAboveGround220250108120024afNoneregular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange \\\n", "0 ecmf 2t heightAboveGround 2 20250108 1200 24 \n", "\n", " dataType number gridType \n", "0 af None regular_ll " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# global 1x1 degree data\n", "import numpy as np\n", "\n", "vals = np.random.normal(0, 1, (181, 360))\n", "r = encoder.encode(values=vals, date=20250108, param=\"2t\", time=12, step=24, edition=2)\n", "r.to_field().ls()" ] }, { "cell_type": "code", "execution_count": 15, "id": "94683efb-0ce7-4deb-916b-f2fbae6ea2a6", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "(4.697071658422638, 4.696983814239502)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vals.max(), r.to_field().values.max()" ] }, { "cell_type": "code", "execution_count": 16, "id": "cf755892-72b8-4560-8614-a623d0ca2fee", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmftisobaricInhPa70020250108120024afNoneregular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange \\\n", "0 ecmf t isobaricInhPa 700 20250108 1200 24 \n", "\n", " dataType number gridType \n", "0 af None regular_ll " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# encode as a pressure level field\n", "r = encoder.encode(values=vals, date=20250108, param=\"t\", level=700, levtype=\"pl\", time=12, step=24, edition=2)\n", "r.to_field().ls()" ] }, { "cell_type": "code", "execution_count": 17, "id": "6ea3f635-7aee-4563-8e9a-c2e1c8cc851a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridTypeisOctahedralN
0ecmf2theightAboveGround220250108120024afNonereduced_gg196
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange \\\n", "0 ecmf 2t heightAboveGround 2 20250108 1200 24 \n", "\n", " dataType number gridType isOctahedral N \n", "0 af None reduced_gg 1 96 " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# global O96 data\n", "vals = np.random.normal(0, 1, 40320)\n", "r = encoder.encode(values=vals, date=20250108, param=\"2t\", time=12, step=24, edition=2)\n", "r.to_field().ls(extra_keys=[\"isOctahedral\",\"N\"])" ] }, { "cell_type": "markdown", "id": "5fdab838-4890-4e71-95d1-5b15af13f3a1", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Using preset options" ] }, { "cell_type": "code", "execution_count": 18, "id": "9dd4b17d-0601-45e3-9423-56ae73c06fd6", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "# create a GribEncoder with preset options\n", "encoder = ekd.create_encoder(\"grib\", date=20250108, template=ds[0])\n", "\n", "d1 = encoder.encode(step=12)\n", "d2 = encoder.encode(step=24)" ] }, { "cell_type": "code", "execution_count": 19, "id": "12247fcb-98e9-46b5-821c-0660e5dfb788", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmf2tsurface020250108120012fc0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange dataType \\\n", "0 ecmf 2t surface 0 20250108 1200 12 fc \n", "\n", " number gridType \n", "0 0 regular_ll " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d1.to_field().ls()" ] }, { "cell_type": "code", "execution_count": 20, "id": "4457ac71-cfe3-40ea-8be1-298ace04ec58", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmf2tsurface020250108120024fc0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange dataType \\\n", "0 ecmf 2t surface 0 20250108 1200 24 fc \n", "\n", " number gridType \n", "0 0 regular_ll " ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d2.to_field().ls()" ] }, { "cell_type": "markdown", "id": "12871d90-a4ab-4da7-8ecc-87a286c0de72", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Working with fieldlists" ] }, { "cell_type": "code", "execution_count": 21, "id": "1c1bfb89-f504-4d6b-8402-aa05c426b47f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " shortName step\n", "0 2t 18\n", " shortName step\n", "0 msl 18\n" ] } ], "source": [ "# create a GribEncoder\n", "encoder = ekd.create_encoder(\"grib\")\n", "\n", "# encode now returns a generator\n", "for d in encoder.encode(data=ds, step=18):\n", " print(d.to_field().ls(keys=[\"shortName\", \"step\"]))" ] }, { "cell_type": "code", "execution_count": null, "id": "bd24efbb-9c27-47a0-870e-43e1e21b5cc3", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "dev_ecc", "language": "python", "name": "dev_ecc" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 5 }