Model Validation

The survival workflow distinguishes training, evaluation, and censoring-reference cohorts, risk scores, and event probabilities.

fyron.validation adds model reliability helpers that complement fyron.ml. Use it after training a classifier or survival model to inspect thresholds, calibration, subgroup performance, decision-curve net benefit, and external validation behavior.

Validation outputs are tables and dictionaries designed for review. They help you decide whether a model is clinically usable, not just whether it has a high headline score.

Calibration, threshold selection, and decision curves validate outcomes before encoding the positive class. Missing or multiclass outcomes, empty/misaligned arrays, and nonfinite or out-of-range probabilities raise ValueError. Resolve missing outcomes explicitly before analysis. Constant probabilities remain represented in the calibration bin table.

Threshold selection and binary metric bootstrapping require both target classes. Bootstrap draws containing only one class are excluded so macro and binary metrics are never mixed; inspect n_successful_rounds for each interval.

Install

Decision curves require nonempty, aligned one-dimensional label/probability vectors, nonmissing binary labels, finite probabilities in [0, 1], and finite thresholds strictly inside (0, 1). Column-vector arrays such as shape (n, 1) now raise an error instead of broadcasting to an incorrect count matrix. Explicitly select a Series or flatten the intended single column; recalculate curves previously generated from two-dimensional inputs.

uv add "fyron[validation]"

Survival validation additionally requires uv add "fyron[survival-ml]". These dependencies are loaded only when a survival metric is called.

Survival Inference Corrections

Fixed-horizon calibration uses Kaplan-Meier event probability, and Brier scores use inverse-probability-of-censoring weighting (IPCW). Early censoring is not a known non-event. Predictions must be event probabilities in [0, 1]; ranking scores belong in concordance/AUC calculations, not probability calibration.

survival_calibration_at_time, brier_score_at_time, and external_survival_validation accept keyword-only censoring_df=None. Supply a reference DataFrame with the same duration/event column names to estimate its censoring distribution; otherwise the evaluation cohort is used. IPCW assumes independent censoring and a censoring distribution applicable to the evaluated cohort. Required censoring probabilities must be positive. Unsupported horizons raise ValueError, including calibration bins ending in censoring before the horizon. Reduce the horizon or choose fewer bins for sparse calibration data.

Survival outcomes must be complete, finite, positive-time observations with binary events. Scores must be finite. Invalid outcomes are not silently coerced or dropped. Dynamic AUC preserves requested horizon order, including repeats. Concordance follows scikit-survival's event/censoring and risk-tie conventions. Bootstrap output adds valid_rounds and dropped_rounds; undefined replicates are excluded, and no valid replicates raises ValueError.

Recompute prior naive survival Brier/calibration results, concordance with tied follow-up times, and AUC tables requested with unsorted horizons.

Imports

from fyron import validation as fv

Core Concepts

API Contract

Function Guide

Classification Example

from fyron import validation as fv

threshold = fv.find_best_threshold(
    y_true=y_test,
    y_prob=rf["y_prob"],
    metric="youden",
)

calibration, calibration_bins = fv.calibration_summary(
    y_true=y_test,
    y_prob=rf["y_prob"],
    n_bins=10,
)

subgroups = fv.subgroup_metric_table(
    validation_df,
    group_col="sex",
    y_true_col="event",
    y_pred_col="prediction",
    y_prob_col="probability",
)

dca = fv.decision_curve_table(y_test, rf["y_prob"])

metric_ci = fv.bootstrap_classification_metrics(
    y_test,
    y_pred,
    rf["y_prob"],
    n_rounds=1000,
    random_state=42,
)

Research Decision: What To Report

Parameter Notes

Survival Validation

Use survival validation helpers after generating a risk score or fixed-horizon event-risk prediction.

c_index = fv.bootstrap_concordance_index(
    validation_df,
    duration_col="time",
    event_col="event",
    risk_col="risk_score",
    n_rounds=1000,
)

calibration, calibration_bins = fv.survival_calibration_at_time(
    validation_df,
    duration_col="time",
    event_col="event",
    prediction_col="risk_12_month",
    time_horizon=12,
)

brier = fv.brier_score_at_time(
    validation_df,
    "time",
    "event",
    "risk_12_month",
    12,
)

The fixed-horizon helpers treat prediction_col as event risk by the selected time horizon.

With fyron[survival-ml], dynamic AUC is available:

auc = fv.time_dependent_auc_table(
    train_df,
    validation_df,
    duration_col="time",
    event_col="event",
    risk_col="risk_score",
    times=[365, 730],
)

Common Pitfalls

• Do not report a threshold chosen on the test set as if it were pre-specified.
• Calibration can be poor even when AUC is high.
• Subgroup tables can become unstable when groups are small.
• Decision curves require clinically meaningful threshold probabilities.
• Ensure time_horizon uses the same unit as duration_col.

Related Modules

• Machine Learning for model training and classification metrics.
• Survival Analysis for risk scores and time-to-event models.
• Clinical Plotting for calibration, decision-curve, and validation figures.