Preprocessing and Imputation Demo

This notebook shows how to use mobts to:

  1. preprocess mobility time series data and remove likely measurement errors and outliers;

  2. impute missing values.

The typical workflow is:

raw data → preprocessing → cleaned data → imputation → completed time series

Preprocessing is recommended before imputation because suspicious observations are replaced by NaN, and these missing values can then be filled by the imputation methods.

1. Import packages

We first import the standard Python packages and the two main mobts classes used in this notebook.

import pandas as pd
import numpy as np

from mobts.preprocessing import preprocess
from mobts.imputation import impute

2. Load the data

Replace the file path below with the path to your own dataset.

The dataset should contain at least:

  • one column identifying the counter or station;

  • one timestamp column;

  • one count column.

# Example
# df = pd.read_csv("../data/df_hourly.csv")

df = pd.read_csv('YOUR_DATA_FILE.csv')

3. Inspect the data

Before running the package, check the column names in your dataset.

You will need to tell mobts which column contains:

  • the observed count: count_col

  • the counter name: counter_col

  • the timestamp: timestamp_col

df.head()

df.columns

count_col = 'Comptage horaire'
counter_col = 'Nom du compteur'
timestamp_col = 'Date et heure de comptage'

4. Define the required input columns

Update the values below so that they match your dataset.

If your data is hourly:

data_is_hourly = True

If your data is daily:

data_is_hourly = False

In this example, the original data is hourly and we keep it hourly.

If your data is hourly but you want to aggregate it to daily data, set:

change_to_daily = True

data_is_hourly = True
change_to_daily = False

Part A - Preprocessing

The preprocessing step cleans the raw data.

It detects likely measurement errors, including:

  • very low noisy observations;

  • unusually high records;

These suspicious values are replaced with NaN.

After preprocessing, missing or suspicious values can be handled by the imputation step.

5. Create the preprocessing object

pp = preprocess()

6. Run preprocessing

The output df_preprocessed is the cleaned dataset.

By default, the package uses an internal threshold for outlier detection, which has been optimized based on the initial analysis on Paris’ bike loop detector counter dataset.

It is set to 20 for daily data, and 45 for hourly data.

You can also pass a custom threshold if the default one is too strict or too permissive for your data. A warning is given indicating that it is advised to observe the measurement detection quality via the plot function and tweak the threshold heuristically.

df_preprocessed = pp.run(
    df_raw=df,
    count_col=count_col,
    counter_col=counter_col,
    timestamp_col=timestamp_col,
    data_is_hourly=data_is_hourly,
    change_to_daily=change_to_daily,
)

7. Inspect the preprocessed data

df_preprocessed.head()

8. Check the preprocessing report

The report summarizes the details of the function in executing the task on the dataset.

Use:

  • print_output=True to print the report in the notebook;

  • save=True to save the report as a text file.

pp.report(print_output=True, save=False)

9. Plot detected outliers

It is recommended to visually inspect detected outliers.

If the preprocessing is too strict or too permissive, adjust the threshold and run preprocessing again.

Arguments:

  • counters: list of counters to plot. Use None to let the function select counters automatically.

  • max_counters: maximum number of counters to plot.

pp.plot_outliers(
    counters=None,
    max_counters=5,
)

After observing the detected measurement errors detected using the default threshold, you can tweak the threshold within the plotting function.

threshold_to_check = 25

pp.plot_outliers(
    counters=None,
    max_counters=5,
    threshold=threshold_to_check,
)

10. Rerun preprocessing with a custom threshold

Use this only if the default threshold does not fit your data well.

A lower threshold usually detects more outliers.
A higher threshold usually detects fewer outliers.

# Example only — uncomment and adjust if needed
# df_preprocessed = pp.run(
#     df_raw=df,
#     count_col=count_col,
#     counter_col=counter_col,
#     timestamp_col=timestamp_col,
#     data_is_hourly=data_is_hourly,
#     change_to_daily=change_to_daily,
#     threshold=YOUR_THRESHOLD,
# )

Part B - Imputation

The imputation step fills missing values.

It is recommended to use the preprocessed dataset as input, so that suspicious observations have already been replaced with NaN.

The imputation module can use:

  • donor-based regression;

  • donor-based scaled medians;

  • STL-based univariate imputation.

The final output column is:

count_imputed/clean

However, uni-variate(STL), multi-variate(scaled-medians), and multi-variate(regression), corresponding to time series imputed by STL, Scaled Medians, and Regression imputation methods respectively, can be used as outputs by the user.

11. [Re]define column names

Note: If the Imputation function is used on already preprocessed data, and the Preprocessing function is not implemented on the data a priori, it is necessary to define column names to pass for the Imputation function.

Note: If the Preprocessing function is already used on the data, this step is not necessary.

# count_col = 'Comptage horaire'
# counter_col = 'Nom du compteur'
# timestamp_col = 'Date et heure de comptage'

12. Create the imputation object

imp = impute()

13. Run imputation

The input should usually be the preprocessed dataframe.

df_imputed = imp.run(df=df_preprocessed, counter_col=counter_col, timestamp_col=timestamp_col, count_col=count_col)

14. Inspect the imputed data

The output contains the original/preprocessed data plus additional imputation columns.

Depending on the configuration, you may see columns such as:

  • STL imputation output;

  • Scaled median imputation output;

  • Regression imputation output;

  • Final imputed or clean count;

  • Imputation method.

The count_imputed/clean column contains the final complete time-series.

df_imputed.head()

15. Check which methods were used

The imputation_method column indicates how each value was obtained.

Typical values are:

  • observed: the value was already observed and did not need imputation;

  • multi-variate(regression): the value was imputed using donor-based regression;

  • multi-variate(scaled-medians): the value was imputed using donor-based scaled medians;

  • uni-variate(STL): the value was imputed using STL-based imputation.

df_imputed['imputation_method'].value_counts(dropna=False)

16. Imputation fallback logic

The package checks whether each counter is eligible for donor-based methods.

For hourly data:

  • If enough quality donors are available, Regression method will be used;

  • Otherwise, the method falls back to STL imputation for that counter.

For daily data:

  • Regression imputation is preferred when possible;

  • If regression is not possible, the package tries scaled median imputation;

  • If scaled median imputation is also not possible, it falls back to STL imputation.

17. Check the imputation report

The report summarizes what happened during imputation.

Use:

  • print_output=True to print the report in the notebook;

  • save=True to save the report as a text file.

imp.report(print_output=True, save=False)