# Preprocessing

Preprocessing is the step where raw mobility data is prepared before imputation or analysis.

Raw counter data is often imperfect. Counters may stop working, report suspicious zero values, or contain abnormal observations.

The goal of preprocessing is to make the dataset more reliable before missing values are filled.

Imputation works better when the input data has already been checked.

For example, a missing value and a true zero are not always the same thing.

In mobility data:

- A true zero may mean no traffic was observed
- A suspicious zero may mean the counter was not working

Preprocessing helps separate these cases before imputation.



## What preprocessing does

Preprocessing works in two stages.

**1. Measurement-error removal (always applied).** Using fixed, configurable rules, `preprocess` detects and nulls out observations that look like sensor errors rather than real counts, such as whole zero-days, sustained low-count runs, or short "islands" of real-looking data surrounded by long suspicious gaps. The exact rules differ between hourly and daily data (for example, night hours are treated differently for hourly data). These rules are controlled by `PreprocessConfig` (see [Configuration](configuration.md)).

**2. Outlier scoring and thresholding (tunable per run).** Each remaining observation gets an outlier score, based on residuals from an STL (Seasonal-Trend decomposition using LOESS) decomposition of the counter's time series. Observations scoring above a threshold are treated as outliers and replaced with `NaN`. The default threshold (20 for daily data, 45 for hourly data) was tuned on a specific dataset and may not suit yours as-is.



## Output

`preprocess.run()` returns the input data with:

- suspicious/unlikely observations replaced with `NaN`
- an `out_score` column, giving the outlier score computed for each observation

Call `pp.report()` afterward for a summary of what was changed during the run. See [Pipeline notes](pipeline_notes.md) for details on reading it.


## Usage

For the basic call, see [Quickstart](quickstart.md). Once you have a preprocessed result, you can inspect and re-tune it:

```python
pp.plot_outliers(counters=None, max_counters=5)

# adjust and re-run with a custom threshold after inspecting the outliers through the plot function, if the default does not fit
df_preprocessed = pp.run(
    df_raw=df,
    counter_col=counter_col,
    timestamp_col=timestamp_col,
    count_col=count_col,
    threshold=15,
)
```