{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5aad9d7a",
   "metadata": {},
   "source": [
    "# Preprocessing and Imputation Demo\n",
    "\n",
    "This notebook shows how to use `mobts` to:\n",
    "\n",
    "1. preprocess mobility time series data and remove likely measurement errors and outliers;\n",
    "2. impute missing values.\n",
    "\n",
    "The typical workflow is:\n",
    "\n",
    "```text\n",
    "raw data → preprocessing → cleaned data → imputation → completed time series\n",
    "```\n",
    "\n",
    "Preprocessing is recommended before imputation because suspicious observations are replaced by `NaN`, and these missing values can then be filled by the imputation methods.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91d3ff75",
   "metadata": {},
   "source": [
    "## 1. Import packages\n",
    "\n",
    "We first import the standard Python packages and the two main `mobts` classes used in this notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3d22273",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from mobts.preprocessing import preprocess\n",
    "from mobts.imputation import impute"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1713b60",
   "metadata": {},
   "source": [
    "## 2. Load the data\n",
    "\n",
    "Replace the file path below with the path to your own dataset.\n",
    "\n",
    "The dataset should contain at least:\n",
    "\n",
    "- one column identifying the counter or station;\n",
    "- one timestamp column;\n",
    "- one count column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51b0e58a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example\n",
    "# df = pd.read_csv(\"../data/df_hourly.csv\")\n",
    "\n",
    "df = pd.read_csv('YOUR_DATA_FILE.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8689f5d1",
   "metadata": {},
   "source": [
    "## 3. Inspect the data\n",
    "\n",
    "Before running the package, check the column names in your dataset.\n",
    "\n",
    "You will need to tell `mobts` which column contains:\n",
    "\n",
    "- the observed count: count_col\n",
    "- the counter name: counter_col\n",
    "- the timestamp: timestamp_col\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9acb1173",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "749f205f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80e04c83-08d7-4e19-941f-12075aefde20",
   "metadata": {},
   "outputs": [],
   "source": [
    "count_col = 'Comptage horaire'\n",
    "counter_col = 'Nom du compteur'\n",
    "timestamp_col = 'Date et heure de comptage'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebe1feb1",
   "metadata": {},
   "source": [
    "## 4. Define the required input columns\n",
    "\n",
    "Update the values below so that they match your dataset.\n",
    "\n",
    "If your data is hourly:\n",
    "```python\n",
    "data_is_hourly = True\n",
    "```\n",
    "\n",
    "If your data is daily:\n",
    "```python\n",
    "data_is_hourly = False\n",
    "```\n",
    "\n",
    "In this example, the original data is hourly and we keep it hourly.\n",
    "\n",
    "If your data is hourly but you want to aggregate it to daily data, set:\n",
    "\n",
    "```python\n",
    "change_to_daily = True\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "288af6b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_is_hourly = True\n",
    "change_to_daily = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab745187",
   "metadata": {},
   "source": [
    "# Part A - Preprocessing\n",
    "\n",
    "The preprocessing step cleans the raw data.\n",
    "\n",
    "It detects likely measurement errors, including:\n",
    "\n",
    "- very low noisy observations;\n",
    "- unusually high records;\n",
    "\n",
    "These suspicious values are replaced with `NaN`.\n",
    "\n",
    "After preprocessing, missing or suspicious values can be handled by the imputation step.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d29f763",
   "metadata": {},
   "source": [
    "## 5. Create the preprocessing object\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cd27ed7",
   "metadata": {},
   "outputs": [],
   "source": [
    "pp = preprocess()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2ac137b",
   "metadata": {},
   "source": [
    "## 6. Run preprocessing\n",
    "\n",
    "The output `df_preprocessed` is the cleaned dataset.\n",
    "\n",
    "By default, the package uses an internal threshold for outlier detection, which has been optimized based on the initial analysis on Paris' bike loop detector counter dataset.\n",
    "\n",
    "It is set to 20 for daily data, and 45 for hourly data.\n",
    "\n",
    "You can also pass a custom threshold if the default one is too strict or too permissive for your data. A warning is given indicating that it is advised to observe the measurement detection quality via the plot function and tweak the threshold heuristically.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c952861",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_preprocessed = pp.run(\n",
    "    df_raw=df,\n",
    "    count_col=count_col,\n",
    "    counter_col=counter_col,\n",
    "    timestamp_col=timestamp_col,\n",
    "    data_is_hourly=data_is_hourly,\n",
    "    change_to_daily=change_to_daily,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fb8b6da",
   "metadata": {},
   "source": [
    "## 7. Inspect the preprocessed data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f91c93a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_preprocessed.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7961f03",
   "metadata": {},
   "source": [
    "## 8. Check the preprocessing report\n",
    "\n",
    "The report summarizes the details of the function in executing the task on the dataset.\n",
    "\n",
    "Use:\n",
    "\n",
    "- `print_output=True` to print the report in the notebook;\n",
    "- `save=True` to save the report as a text file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "190c576f",
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.report(print_output=True, save=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0fbff71",
   "metadata": {},
   "source": [
    "## 9. Plot detected outliers\n",
    "\n",
    "It is recommended to visually inspect detected outliers.\n",
    "\n",
    "If the preprocessing is too strict or too permissive, adjust the threshold and run preprocessing again.\n",
    "\n",
    "Arguments:\n",
    "\n",
    "- `counters`: list of counters to plot. Use `None` to let the function select counters automatically.\n",
    "- `max_counters`: maximum number of counters to plot.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc8357b3-6287-4871-a26e-daae0b74534d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.plot_outliers(\n",
    "    counters=None,\n",
    "    max_counters=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7384375a-0825-4a0e-9eb4-179e267ba2a9",
   "metadata": {},
   "source": [
    "After observing the detected measurement errors detected using the default threshold, you can tweak the threshold within the plotting function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6265571b-d6d9-453e-beaf-a67f14cc0ed6",
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold_to_check = 25\n",
    "\n",
    "pp.plot_outliers(\n",
    "    counters=None,\n",
    "    max_counters=5,\n",
    "    threshold=threshold_to_check,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfe0d6a7",
   "metadata": {},
   "source": [
    "## 10. Rerun preprocessing with a custom threshold\n",
    "\n",
    "Use this only if the default threshold does not fit your data well.\n",
    "\n",
    "A lower threshold usually detects more outliers.  \n",
    "A higher threshold usually detects fewer outliers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f6e2dac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example only — uncomment and adjust if needed\n",
    "# df_preprocessed = pp.run(\n",
    "#     df_raw=df,\n",
    "#     count_col=count_col,\n",
    "#     counter_col=counter_col,\n",
    "#     timestamp_col=timestamp_col,\n",
    "#     data_is_hourly=data_is_hourly,\n",
    "#     change_to_daily=change_to_daily,\n",
    "#     threshold=YOUR_THRESHOLD,\n",
    "# )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccba77e5",
   "metadata": {},
   "source": [
    "# Part B - Imputation\n",
    "\n",
    "The imputation step fills missing values.\n",
    "\n",
    "It is recommended to use the preprocessed dataset as input, so that suspicious observations have already been replaced with `NaN`.\n",
    "\n",
    "The imputation module can use:\n",
    "\n",
    "- donor-based regression;\n",
    "- donor-based scaled medians;\n",
    "- STL-based univariate imputation.\n",
    "\n",
    "The final output column is:\n",
    "\n",
    "```text\n",
    "count_imputed/clean\n",
    "```\n",
    "However, `uni-variate(STL)`, `multi-variate(scaled-medians)`, and `multi-variate(regression)`, corresponding to time series imputed by STL, Scaled Medians, and Regression imputation methods respectively, can be used as outputs by the user.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f4ebb39-3110-4a13-95a5-0e18572cc6a4",
   "metadata": {},
   "source": [
    "## 11. [Re]define column names "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcad0c80-c8b6-4080-acd2-dc754dc2d7b9",
   "metadata": {},
   "source": [
    "**Note**: If the Imputation function is used on already preprocessed data, and the Preprocessing function is not implemented on the data a priori, it is necessary to define column names to pass for the Imputation function.\n",
    "\n",
    "**Note**: If the Preprocessing function is already used on the data, this step is not necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42273d4b-d565-4d74-8e7c-d7cfbb0335f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# count_col = 'Comptage horaire'\n",
    "# counter_col = 'Nom du compteur'\n",
    "# timestamp_col = 'Date et heure de comptage'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06b37730",
   "metadata": {},
   "source": [
    "## 12. Create the imputation object\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83622268",
   "metadata": {},
   "outputs": [],
   "source": [
    "imp = impute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18125bdc",
   "metadata": {},
   "source": [
    "## 13. Run imputation\n",
    "\n",
    "The input should usually be the preprocessed dataframe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeb7a74d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_imputed = imp.run(df=df_preprocessed, counter_col=counter_col, timestamp_col=timestamp_col, count_col=count_col)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae67abbe",
   "metadata": {},
   "source": [
    "## 14. Inspect the imputed data\n",
    "\n",
    "The output contains the original/preprocessed data plus additional imputation columns.\n",
    "\n",
    "Depending on the configuration, you may see columns such as:\n",
    "\n",
    "- STL imputation output;\n",
    "- Scaled median imputation output;\n",
    "- Regression imputation output;\n",
    "- Final imputed or clean count;\n",
    "- Imputation method.\n",
    "\n",
    "The `count_imputed/clean` column contains the final complete time-series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6a654cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_imputed.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40082b4f",
   "metadata": {},
   "source": [
    "## 15. Check which methods were used\n",
    "\n",
    "The `imputation_method` column indicates how each value was obtained.\n",
    "\n",
    "Typical values are:\n",
    "\n",
    "- `observed`: the value was already observed and did not need imputation;\n",
    "- `multi-variate(regression)`: the value was imputed using donor-based regression;\n",
    "- `multi-variate(scaled-medians)`: the value was imputed using donor-based scaled medians;\n",
    "- `uni-variate(STL)`: the value was imputed using STL-based imputation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8c295e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_imputed['imputation_method'].value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7fa24d2",
   "metadata": {},
   "source": [
    "## 16. Imputation fallback logic\n",
    "\n",
    "The package checks whether each counter is eligible for donor-based methods.\n",
    "\n",
    "For hourly data:\n",
    "\n",
    "- If enough quality donors are available, Regression method will be used;\n",
    "- Otherwise, the method falls back to STL imputation for that counter.\n",
    "\n",
    "For daily data:\n",
    "\n",
    "- Regression imputation is preferred when possible;\n",
    "- If regression is not possible, the package tries scaled median imputation;\n",
    "- If scaled median imputation is also not possible, it falls back to STL imputation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2652a504",
   "metadata": {},
   "source": [
    "## 17. Check the imputation report\n",
    "\n",
    "The report summarizes what happened during imputation.\n",
    "\n",
    "Use:\n",
    "\n",
    "- `print_output=True` to print the report in the notebook;\n",
    "- `save=True` to save the report as a text file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9187bea2",
   "metadata": {},
   "outputs": [],
   "source": [
    "imp.report(print_output=True, save=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}