Bits & Flames
bitsandflames/fyron

Model Validation

fyron.validation adds model reliability helpers that complement fyron.ml. Use it after training a classifier or survival model to inspect thresholds, calibration, subgroup performance, decision-curve net benefit, and external validation behavior.

Validation outputs are tables and dictionaries designed for review. They help you decide whether a model is clinically usable, not just whether it has a high headline score.

Install

bash
uv add "fyron[validation]"

Imports

python
from fyron import validation as fv

Core Concepts

ConceptMeaning
y_trueObserved labels, usually 0/1 for binary classification.
y_probPredicted probability for the positive class. Required for AUC, calibration, thresholds, and decision curves.
y_predHard predicted labels after applying a threshold.
pos_labelLabel treated as the event/positive class.
subgroup columnClinical group used to inspect performance heterogeneity.
fixed-horizon survival predictionPredicted event risk by a chosen time point, such as 12 months.

API Contract

TopicContract
Input shapeArrays for labels/predictions/probabilities, or DataFrames for subgroup and survival validation tables.
Required columnsNamed true label, prediction, probability, subgroup, time, event, or risk columns must exist.
Return shapeDictionaries with summary values plus DataFrames for bins/thresholds, or standalone validation DataFrames.
Saved artifactsThreshold tables, calibration bins, subgroup metrics, decision-curve tables, bootstrap CIs, and external validation summaries.
Failure modesMismatched lengths, missing probabilities, one-class validation sets, invalid thresholds, empty subgroups, or unavailable optional survival-ML dependencies.

Function Guide

FunctionUse it whenReturns
find_best_thresholdYou need a threshold table and exploratory best threshold.best-threshold dict with table
calibration_summaryYou need observed versus predicted risk by probability bin.summary dict and bin table
subgroup_metric_tableYou need metrics by sex, stage, center, scanner, or other subgroup.subgroup metric DataFrame
decision_curve_tableYou need net-benefit values over clinical risk thresholds.decision-curve DataFrame
bootstrap_classification_metricsYou need uncertainty intervals for classification metrics.metric CI DataFrame
bootstrap_concordance_indexYou need uncertainty for survival risk ranking.C-index summary dict
survival_calibration_at_timeYou need fixed-horizon survival calibration.summary dict and bin table
brier_score_at_timeYou need fixed-horizon prediction error.Brier score dict
external_survival_validationYou need a compact external validation summary.validation dict
time_dependent_auc_tableYou need dynamic AUC and have fyron[survival-ml].AUC DataFrame

Classification Example

python
from fyron import validation as fv

threshold = fv.find_best_threshold(
    y_true=y_test,
    y_prob=rf["y_prob"],
    metric="youden",
)

calibration, calibration_bins = fv.calibration_summary(
    y_true=y_test,
    y_prob=rf["y_prob"],
    n_bins=10,
)

subgroups = fv.subgroup_metric_table(
    validation_df,
    group_col="sex",
    y_true_col="event",
    y_pred_col="prediction",
    y_prob_col="probability",
)

dca = fv.decision_curve_table(y_test, rf["y_prob"])

metric_ci = fv.bootstrap_classification_metrics(
    y_test,
    y_pred,
    rf["y_prob"],
    n_rounds=1000,
    random_state=42,
)

Research Decision: What To Report

OutputWhy it matters
AUCRanking discrimination, not clinical usefulness by itself.
Sensitivity/specificityDepends on a threshold; state whether the threshold was locked.
CalibrationShows whether predicted probabilities match observed risk.
SubgroupsReveals performance heterogeneity across clinically relevant strata.
Decision curveConnects predictions to threshold-dependent net benefit.
Confidence intervalsMakes sampling uncertainty visible for metrics and C-index.

Parameter Notes

ParameterPractical guidance
metricThreshold scans often use youden, sensitivity, specificity, or balanced accuracy. Pre-specify this for confirmatory analyses.
n_binsCalibration bins should be large enough to contain meaningful patient counts.
group_colChoose clinically relevant subgroups, not arbitrary slices.
thresholdsUse clinically meaningful ranges for decision curves.
time_horizonMust use the same time unit as duration_col.
prediction_colFor fixed-horizon helpers, this should be event risk by time_horizon.

Survival Validation

Use survival validation helpers after generating a risk score or fixed-horizon event-risk prediction.

python
c_index = fv.bootstrap_concordance_index(
    validation_df,
    duration_col="time",
    event_col="event",
    risk_col="risk_score",
    n_rounds=1000,
)

calibration, calibration_bins = fv.survival_calibration_at_time(
    validation_df,
    duration_col="time",
    event_col="event",
    prediction_col="risk_12_month",
    time_horizon=12,
)

brier = fv.brier_score_at_time(
    validation_df,
    "time",
    "event",
    "risk_12_month",
    12,
)

The fixed-horizon helpers treat prediction_col as event risk by the selected time horizon.

With fyron[survival-ml], dynamic AUC is available:

python
auc = fv.time_dependent_auc_table(
    train_df,
    validation_df,
    duration_col="time",
    event_col="event",
    risk_col="risk_score",
    times=[365, 730],
)

Common Pitfalls

  • Do not report a threshold chosen on the test set as if it were pre-specified.
  • Calibration can be poor even when AUC is high.
  • Subgroup tables can become unstable when groups are small.
  • Decision curves require clinically meaningful threshold probabilities.
  • Ensure time_horizon uses the same unit as duration_col.