{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3c7855c9-6d7c-407c-91a3-47c5a2f63f8a",
   "metadata": {},
   "source": [
    "# Python package builder example\n",
    "The package `ckanapi_harvesters.builder` implements functions to manage a CKAN dataset (previously known as a package) with the help of an Excel workbook. This Excel file specifies the package metadata and information to upload/download the resources of the package. An illustration of these tasks is given in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dd831cd-402d-4522-9c07-1d6de39925e0",
   "metadata": {},
   "source": [
    "## Initialisation\n",
    "The package can be installed with the following command:\n",
    "```sh\n",
    "> pip install ckanapi_harvesters[extras]\n",
    "```\n",
    "\n",
    "The following cell refers to the code present in the Git directory with the option `use_git_package`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f94124e60fe21f5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# optionally, use the ckanapi_harvesters package present in the Git directory\n",
    "use_git_package = True\n",
    "if use_git_package:\n",
    "    import os\n",
    "    cwd = os.getcwd()\n",
    "    if not os.path.isdir(os.path.join(cwd, \"ckanapi_harvesters\")):\n",
    "        # we assume we are in the examples directory\n",
    "        cwd = os.path.join(cwd, r\"../../src\")  # aim for src directory\n",
    "        assert(os.path.isdir(os.path.join(cwd, \"ckanapi_harvesters\")))\n",
    "        os.chdir(cwd)\n",
    "        print(\"CWD changed to: \" + os.path.abspath(\"\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae3320c0c64822ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from ckanapi_harvesters import CkanApi, BuilderPackage, CkanCallbackLevel\n",
    "from ckanapi_harvesters.builder.example import example_package_xls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9a847c0-3ab2-49fe-af4e-ee7437848bf3",
   "metadata": {},
   "source": [
    "### Loading package metadata from Excel file\n",
    "The Excel workbook given in the example refers to an external code module for DataFrame upload/download functions. To activate this feature, a call to `unlock_external_code_execution` must be done.\n",
    "\n",
    "__Warning:__ Only enable this feature for code which comes from trusted sources as this executes the module referred in the Excel workbook (`Auxiliary functions file` field)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "500e2647-9f72-45d0-9090-09c2db0f91ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "BuilderPackage.unlock_external_code_execution()\n",
    "mdl = BuilderPackage.from_excel(example_package_xls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bd9e9fa-0343-40dc-8f1f-588c70e92e5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ckan = CkanApi(None)\n",
    "# you can specify the CKAN URL, owner organization, API key here or in the Excel workbook\n",
    "ckan = mdl.init_ckan(ckan)\n",
    "ckan.input_missing_info(input_args_if_necessary=True, input_owner_org=True, error_not_found=False)  # request user input to configure CKAN\n",
    "ckan.set_limits(10000)  # reduce if server hangs up\n",
    "ckan.set_requests_delay(0.1)  # increase if server errors 502\n",
    "ckan.set_verbosity(True)  # this displays all the steps performed by the script\n",
    "ckan.test_ckan_login(raise_error=True, verbose=True)  # test if you are correctly logged in"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4477553b-0d83-4645-846a-1a1f53c176af",
   "metadata": {},
   "source": [
    "### Displaying the package model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59c19cfa-abb9-42b2-801c-3981101fe15b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_dict = mdl.get_all_df()\n",
    "for tab, df in df_dict.items():\n",
    "    display(f\"Tab {tab}:\")\n",
    "    display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e533808-7d0a-4579-b755-55bd73ca25ea",
   "metadata": {},
   "source": [
    "#### Auxiliary function to display a progress bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e9459ea-68de-4b7f-9fe2-125c5104038c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import IntProgress\n",
    "from IPython.display import display\n",
    "f = IntProgress(min=0,max=100)\n",
    "\n",
    "def progress_callback(index:int, total:int, level, **kwargs):\n",
    "    if level == CkanCallbackLevel.ResourceChunks:\n",
    "        f.value = int(index/total*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe8e84f-0c98-43ea-a1a0-bb2f8800e393",
   "metadata": {},
   "source": [
    "## Initiating the package\n",
    "This call creates the package if no other package with the same name exists. If the package already exists, it is updated with information from the Excel workbook. The resources are initialized. Optionally, the resources data can be fully reuploaded (even if the resources already exist) to ensure the server side of the package represents the information specified in the Excel workbook. However, if there are large datasets, this resets them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f41c0a6d-fc98-48a4-a964-f673ef20790e",
   "metadata": {},
   "outputs": [],
   "source": [
    "reupload = True  # True: reuploads all documents and resets large datasets to the first document (not recommended if there is a large dataset)\n",
    "mdl.patch_request_full(ckan, reupload=reupload)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8275c0f4-1fd9-4787-b0e4-cd3441008c2a",
   "metadata": {},
   "source": [
    "### Uploading large datasets\n",
    "Large datasets are defined locally by a directory containing multiple CSV files. The first file found (in alphabetic order) is used to initialize the dataset. This function automates the concatenation of other files using the API `datastore_upsert` in a multi-threaded implementation. It can be executed multiple times without affecting the final result, as long as all the data has been transferred.\n",
    "The number of threads should be adjusted if there are too many request errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2cde8b9-6fb9-40ea-bacc-df14ff366939",
   "metadata": {},
   "outputs": [],
   "source": [
    "threads = 3  # > 1: multi-threading mode - reduce if HTTP 502 errors\n",
    "display(f)\n",
    "mdl.upload_large_datasets(ckan, threads=threads, progress_callback=progress_callback, only_missing=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06919a09-60b8-41c5-9460-a6b9f1a4e281",
   "metadata": {},
   "source": [
    "## Downloading the package\n",
    "This function downloads all the resources of a dataset. The multi-threaded implementation is reserved to download large datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "713ca54a01990fc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the destination directory\n",
    "example_package_download_dir = os.path.abspath(\"package_download\")\n",
    "print(\"Package will be downloaded in: \" + example_package_download_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd69d2e6ab81fe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "threads = 3  # > 1: number of threads to download large datasets\n",
    "display(f)\n",
    "mdl.download_request_full(ckan, example_package_download_dir, full_download=True, threads=threads,\n",
    "                              skip_existing=False, progress_callback=progress_callback)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba51545c-d667-4203-a296-94c91f176bdd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}