{ "cells": [ { "cell_type": "markdown", "id": "50c3f565-f9a8-4100-a977-f9d3818853da", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Writing GRIB to an FDB target" ] }, { "cell_type": "markdown", "id": "d4878bcc-d223-46ba-8d1d-183b60e6085b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "This example demonstrates how to **write earthkit-data GRIB fields into an FDB**. \n", "\n", "FDB (Fields DataBase) is a domain-specific object store developed at ECMWF for storing, indexing and retrieving GRIB data. For more information on FBD please consult the following pages:\n", "\n", "- [FDB](https://fields-database.readthedocs.io/en/latest/)\n", "- [pyfdb](https://pyfdb.readthedocs.io/en/latest/)" ] }, { "cell_type": "markdown", "id": "5fc7ae99-bd9d-4e7b-b3e5-73f12e15d047", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Setting up the target FDB" ] }, { "cell_type": "markdown", "id": "fbbaf36f-a912-41e2-88db-3964e363fd3f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "In this example we will create an FDB in the current folder using the schema taken from the pyfdb test suite. To do so first we need to ensure the directory exists. Next, we have to specify the configuration." ] }, { "cell_type": "code", "execution_count": 1, "id": "e3f1ece1-a079-4b72-bafe-4c5f92a1bd26", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "import os\n", "fdb_schema = \"./default_fdb_schema\"\n", "fdb_dir = \"./_fdb_target_demo\"\n", "os.makedirs(fdb_dir, exist_ok=True)\n", "\n", "config = {\"type\":\"local\",\n", " \"engine\":\"toc\",\n", " \"schema\":fdb_schema,\n", " \"spaces\":[{\"handler\":\"Default\",\n", " \"roots\":[{\"path\":fdb_dir}]}]}" ] }, { "cell_type": "markdown", "id": "a319a355-ce2e-4c64-9d75-c786828049bf", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Working with FDB requires pyfdb and fdb to be installed. The path to the fdb installation should also be set e.g. via the FDB5_DIR environment variable." ] }, { "cell_type": "markdown", "id": "3a9d9260-ae51-4646-bb7e-8cb0120645fd", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Getting the input data" ] }, { "cell_type": "code", "execution_count": 2, "id": "8a7930df-cf89-4be4-a25e-bdb7f362352d", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "", "version_major": 2, "version_minor": 0 }, "text/plain": [ "tuv_pl.grib: 0%| | 0.00/4.22k [00:00\n", "#T_adfe4 th {\n", " text-align: left;\n", "}\n", "#T_adfe4_row0_col0, #T_adfe4_row0_col1, #T_adfe4_row0_col2, #T_adfe4_row0_col3, #T_adfe4_row0_col4, #T_adfe4_row0_col5, #T_adfe4_row0_col6, #T_adfe4_row0_col7, #T_adfe4_row0_col8, #T_adfe4_row1_col0, #T_adfe4_row1_col1, #T_adfe4_row1_col2, #T_adfe4_row1_col3, #T_adfe4_row1_col4, #T_adfe4_row1_col5, #T_adfe4_row1_col6, #T_adfe4_row1_col7, #T_adfe4_row1_col8, #T_adfe4_row2_col0, #T_adfe4_row2_col1, #T_adfe4_row2_col2, #T_adfe4_row2_col3, #T_adfe4_row2_col4, #T_adfe4_row2_col5, #T_adfe4_row2_col6, #T_adfe4_row2_col7, #T_adfe4_row2_col8 {\n", " text-align: left;\n", "}\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
  leveldatetimestepparamIdclassstreamtypeexperimentVersionNumber
shortNametypeOfLevel         
tisobaricInhPa1000,300,...2018080112000130odoperan0001
uisobaricInhPa1000,300,...2018080112000131odoperan0001
visobaricInhPa1000,300,...2018080112000132odoperan0001
\n" ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import earthkit.data as ekd\n", "ds = ekd.from_source(\"sample\", \"tuv_pl.grib\")\n", "ds.describe()" ] }, { "cell_type": "markdown", "id": "d088080d-9bc8-486c-9653-164470ab8f99", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Using to_target() on the data object" ] }, { "cell_type": "raw", "id": "c70d78f4-cfc1-4077-a75b-f1e0d6f4285d", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "We use :func:`to_target` to write the GRIB fieldlist/field into the FDB. The encoder is automatically guessed from the input data." ] }, { "cell_type": "code", "execution_count": 3, "id": "c1655485-bc68-4d99-a8c7-3b101c2ea6ee", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "# writing a field\n", "ds[0].to_target(\"fdb\", config=config)\n", "\n", "# writing a whole fieldlist\n", "ds.to_target(\"fdb\", config=config)" ] }, { "cell_type": "code", "execution_count": 4, "id": "a0c27132-ec2d-43fd-a96f-f23ae5d9bc01", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmfuisobaricInhPa5002018080112000an0regular_ll
1ecmfvisobaricInhPa5002018080112000an0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange \\\n", "0 ecmf u isobaricInhPa 500 20180801 1200 0 \n", "1 ecmf v isobaricInhPa 500 20180801 1200 0 \n", "\n", " dataType number gridType \n", "0 an 0 regular_ll \n", "1 an 0 regular_ll " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# checking the result\n", "request = {\n", " 'class': 'od',\n", " 'expver': '0001',\n", " 'stream': 'oper',\n", " 'date': '20180801',\n", " 'time': 1200,\n", " 'domain': 'g',\n", " 'type': 'an',\n", " 'levtype': 'pl',\n", " 'levelist': 500,\n", " 'step': 0,\n", " 'param': [131,132]\n", "}\n", "\n", "ekd.from_source(\"fdb\", request, config=config, stream=False).ls()" ] }, { "cell_type": "code", "execution_count": 5, "id": "0d115cb7-c189-4b9d-a945-ad4eb8c6f052", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "# setting GRIB keys for the output\n", "ds.to_target(\"fdb\", config=config, metadata={\"date\": 20250108})" ] }, { "cell_type": "code", "execution_count": 6, "id": "59d005bc-a100-41a3-8cc6-f7c5c02950ec", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
centreshortNametypeOfLevelleveldataDatedataTimestepRangedataTypenumbergridType
0ecmfuisobaricInhPa5002025010812000an0regular_ll
1ecmfvisobaricInhPa5002025010812000an0regular_ll
\n", "
" ], "text/plain": [ " centre shortName typeOfLevel level dataDate dataTime stepRange \\\n", "0 ecmf u isobaricInhPa 500 20250108 1200 0 \n", "1 ecmf v isobaricInhPa 500 20250108 1200 0 \n", "\n", " dataType number gridType \n", "0 an 0 regular_ll \n", "1 an 0 regular_ll " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# checking the result\n", "request_1 = dict(**request)\n", "request_1.update({\"date\": 20250108})\n", "ekd.from_source(\"fdb\", request_1, config=config, stream=False).ls()" ] }, { "cell_type": "markdown", "id": "c51966f3-db61-43cf-8c46-43989dc61d0a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "#### Using a Target object" ] }, { "cell_type": "code", "execution_count": null, "id": "0916438e-ed09-47b7-9618-75e47f3f1589", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "# basic usage\n", "target = ekd.create_target(\"fdb\", config=config)\n", "target.write(ds)\n", "target.flush()\n", "\n", "# can be used as a context manager, no need to call flush() in the end\n", "with ekd.create_target(\"fdb\", config=config) as target:\n", " target.write(ds)\n", "\n", "# a fieldlist can be written field by field into the target\n", "with ekd.create_target(\"fdb\", config=config) as target:\n", " for f in ds:\n", " target.write(f)" ] } ], "metadata": { "kernelspec": { "display_name": "dev_ecc", "language": "python", "name": "dev_ecc" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 5 }