# Pipeline notes

[Quickstart](quickstart.md) shows the shortest path from raw data to a completed time series. This page covers practical details of running the pipeline: preserving metadata, tuning before committing, and reading the run reports.



## The pipeline at a glance

1. Raw data is standardized and checked for structural issues (missing counter names, insufficient history). See [Data format](data_format.md).
2. **Preprocessing** removes likely measurement errors using fixed rules, then scores and flags remaining outliers against a tunable threshold. See [Preprocessing](preprocessing.md).
3. **Imputation** looks for correlated donor counters and fills gaps via regression or scaled medians where eligible, falling back to STL decomposition otherwise. See [Imputation](imputation.md).
4. Every threshold and eligibility rule in steps 2–3 is overridable - see [Configuration](configuration.md).



## Passing metadata columns through

If your dataset has extra columns you want preserved rather than treated as count data (e.g. a station name, or metro line), pass their names in a list via the `metadata_cols` argument to `run()`, on either stage.

It is important to note that `metadata_cols` is intended for attributes that are constant per counter (e.g. a station name or fixed coordinates). If a column's value actually varies across timestamps for the same counter, only one arbitrary value will be kept and applied to every row for that counter.

```python
df_preprocessed = pp.run(
    df_raw=df,
    counter_col=counter_col,
    timestamp_col=timestamp_col,
    count_col=count_col,
    metadata_cols=['station_name', 'zip_code'],
)
```



## Tuning before moving on

Do not feed a default-threshold preprocessing result straight into imputation without checking it first. See [Preprocessing](preprocessing.md#tuning-the-outlier-threshold) for the plot-and-adjust loop.



## Reading the reports

Both `pp.report()` and `imp.report()` summarize what happened during the run, print a readable breakdown, and return the same information as a dictionary for programmatic use:

```python
pp.report(print_output=True, save=True, filepath='preprocess_report.txt')
imp.report(print_output=True, save=True, filepath='imputation_report.txt')
```

A preprocessing report looks like this:

```text
=== PREPROCESS REPORT ===
Aggregate number of entries: 76222
Number of counters observed: 212
Number of entries with missing counter names: 0
Number of zero/low entries cleaned: 1831
Number of counters with cleaned zero/low entries: 104
Number of counters removed due to low number of entries: 0
Number of measurement errors flagged by threshold: 98
Number of ounters with measurement errors: 59
Is data hourly: False
Have hourly data been transformed to daily data: False
Threshold used: 10
```

And an imputation report looks like this:

```text
=== Imputation REPORT ===
Aggregate number of entries: 77305
Number of counters observed: 212
The temporal granularity of the project is: daily
Number of entries with missing counter names: 0
Number of counters removed due to low number of entries: 0
Number of holes in data: 1954
Number of ounters with missing information: 130
Number of holes imputed via regression: 1652
Number of holes imputed via scaled medians: 302
Number of holes imputed via STL: 0
```

Together, the two reports give a quick overview on how many observations were flagged and cleaned, how many gaps existed, and which methods filled them.