Pipeline notes

Quickstart shows the shortest path from raw data to a completed time series. This page covers practical details of running the pipeline: preserving metadata, tuning before committing, and reading the run reports.

The pipeline at a glance

  1. Raw data is standardized and checked for structural issues (missing counter names, insufficient history). See Data format.

  2. Preprocessing removes likely measurement errors using fixed rules, then scores and flags remaining outliers against a tunable threshold. See Preprocessing.

  3. Imputation looks for correlated donor counters and fills gaps via regression or scaled medians where eligible, falling back to STL decomposition otherwise. See Imputation.

  4. Every threshold and eligibility rule in steps 2–3 is overridable - see Configuration.

Passing metadata columns through

If your dataset has extra columns you want preserved rather than treated as count data (e.g. a station name, or metro line), pass their names in a list via the metadata_cols argument to run(), on either stage.

It is important to note that metadata_cols is intended for attributes that are constant per counter (e.g. a station name or fixed coordinates). If a column’s value actually varies across timestamps for the same counter, only one arbitrary value will be kept and applied to every row for that counter.

df_preprocessed = pp.run(
    df_raw=df,
    counter_col=counter_col,
    timestamp_col=timestamp_col,
    count_col=count_col,
    metadata_cols=['station_name', 'zip_code'],
)

Tuning before moving on

Do not feed a default-threshold preprocessing result straight into imputation without checking it first. See Preprocessing for the plot-and-adjust loop.

Reading the reports

Both pp.report() and imp.report() summarize what happened during the run, print a readable breakdown, and return the same information as a dictionary for programmatic use:

pp.report(print_output=True, save=True, filepath='preprocess_report.txt')
imp.report(print_output=True, save=True, filepath='imputation_report.txt')

A preprocessing report looks like this:

=== PREPROCESS REPORT ===
Aggregate number of entries: 76222
Number of counters observed: 212
Number of entries with missing counter names: 0
Number of zero/low entries cleaned: 1831
Number of counters with cleaned zero/low entries: 104
Number of counters removed due to low number of entries: 0
Number of measurement errors flagged by threshold: 98
Number of ounters with measurement errors: 59
Is data hourly: False
Have hourly data been transformed to daily data: False
Threshold used: 10

And an imputation report looks like this:

=== Imputation REPORT ===
Aggregate number of entries: 77305
Number of counters observed: 212
The temporal granularity of the project is: daily
Number of entries with missing counter names: 0
Number of counters removed due to low number of entries: 0
Number of holes in data: 1954
Number of ounters with missing information: 130
Number of holes imputed via regression: 1652
Number of holes imputed via scaled medians: 302
Number of holes imputed via STL: 0

Together, the two reports give a quick overview on how many observations were flagged and cleaned, how many gaps existed, and which methods filled them.