Preprocessing And Cohort QC

fyron.preprocessing provides small quality-control tables for clinical feature matrices before modeling or survival analysis. Use it to inspect missingness, duplicate patient identifiers, categorical levels, train/test balance, and column names that may leak outcomes.

If the feature matrix still needs ICD-10 groups, OPS categories, exact code indicators, age-at-baseline, or time-window flags, create those first with DataFrame Operations. Then use preprocessing for imputation, stable categorical encoding, and leakage checks.

Install

bash

uv add fyron

Imports

python

from fyron import preprocessing as fp

API Contract

Topic	Contract
Input shape	Cohort or feature-matrix DataFrames; encoders/imputers are plain dictionaries returned by fit helpers.
Required columns	ID, group, numeric, categorical, or feature columns named in the call must exist.
Return shape	QC tables, transformed DataFrames, or `(DataFrame, transformer_dict)` tuples.
Saved artifacts	Save QC reports, imputation plans, encoding plans, and learned transformer dictionaries when reproducibility matters.
Failure modes	Missing columns, unsupported strategies, nonnumeric columns passed as numeric, unseen categorical levels, or leakage flags requiring review.

Function Reference

Function	Required	Optional	Returns
`summarize_missingness(df, ...)`	`df`	`columns`, `group_col`	missingness DataFrame
`find_duplicate_ids(df, ...)`	`df`	`id_col="patient_id"`	duplicate rows with counts
`summarize_categorical_levels(df, ...)`	`df`	`columns`, `max_levels`, `include_missing`	level count table
`compare_train_test_balance(train, test, ...)`	`train`, `test`	`columns`	train/test balance DataFrame
`detect_potential_leakage(columns, ...)`	DataFrame or column list	outcome/time keywords	suspicious column table
`feature_matrix_report(X, ...)`	feature matrix	labels, rare threshold, leakage keywords	one-row-per-feature QC table
`summarize_imputation_plan(df, ...)`	`df`	numeric/categorical columns and strategies	imputation plan table
`fit_imputer(df, ...)`	training DataFrame	strategies, fill values, missing indicators	imputer dict
`apply_imputer(df, imputer)`	DataFrame and imputer	none	imputed DataFrame
`impute_dataframe(df, ...)`	DataFrame	same as `fit_imputer`	`(DataFrame, imputer)`
`summarize_encoding_plan(df, ...)`	`df`	columns, max levels	encoding plan table
`fit_categorical_encoder(df, ...)`	training DataFrame	columns, rare category settings, drop first	encoder dict
`apply_categorical_encoder(df, encoder)`	DataFrame and encoder	none	encoded DataFrame
`encode_categorical(df, ...)`	DataFrame	same as `fit_categorical_encoder`	`(DataFrame, encoder)`

Example

python

from fyron import preprocessing as fp

missing = fp.summarize_missingness(
    cohort,
    columns=["age", "sex", "l3_sma_cm2", "event"],
    group_col="risk_group",
)

duplicates = fp.find_duplicate_ids(cohort, id_col="patient_id")
levels = fp.summarize_categorical_levels(cohort, columns=["sex", "stage"])
leakage = fp.detect_potential_leakage(cohort)
feature_qc = fp.feature_matrix_report(cohort[feature_cols], cohort["event"])

Use these tables before dropping rows or fitting models. They are meant to make data-cleaning decisions visible in notebooks and analysis supplements.

Research Decision: Feature Matrix Readiness

Before a model is trained, save a feature matrix report. It flags constant columns, missingness, rare binary variables, and column names that look like outcomes or post-index information.

python

feature_qc = fp.feature_matrix_report(
    X_train,
    y_train,
    rare_threshold=0.01,
)

Treat leakage flags as review prompts, not automatic exclusions. A column named last_followup_days is usually suspicious in a baseline prediction model, but a column named baseline_event_count may be legitimate if it was measured before the index date.

Imputation

Fit imputation on training data and apply the learned fill values to validation or external cohorts.

python

imputer = fp.fit_imputer(
    train,
    numeric=["age", "l3_sma_cm2"],
    categorical=["sex", "stage"],
    strategy_numeric="median",
    strategy_categorical="most_frequent",
    add_missing_indicators=True,
)

train_imp = fp.apply_imputer(train, imputer)
external_imp = fp.apply_imputer(external, imputer)

The imputer is a plain dictionary with columns, strategies, fill_values, and metadata.

Encoding

Use Fyron’s encoder when you need stable one-hot columns across train/test splits without adding a pipeline object.

python

encoder = fp.fit_categorical_encoder(
    train_imp,
    columns=["sex", "stage"],
    rare_min_count=5,
)

X_train = fp.apply_categorical_encoder(train_imp, encoder)
X_external = fp.apply_categorical_encoder(external_imp, encoder)

Unseen levels map to __rare__ when rare-category handling is enabled.