Bits & Flames
bitsandflames/fyron

Flow Explainability

fyron.flow records cohort and filter steps as structured, reviewer-facing artifacts. Use it when you need to explain why patients entered or left a study cohort, not when you need a general workflow engine.

The module complements model explainability. fyron.flow explains who entered the analysis; fyron.explainability explains which features influenced model behavior.

When To Use It

  • You apply sequential inclusion and exclusion filters.
  • You need a CONSORT/STROBE-style count trail.
  • You want a console summary during notebooks or scripts.
  • You want a Markdown/text flow chart for reports.
  • You want a table compatible with fyron.reporting.cohort_flow_table and fyron.plotting.plot_cohort_flow.

Imports

python
from fyron.flow import FlowTracker, track_filter

Top-level imports also work:

python
from fyron import FlowTracker, track_filter

Core Concepts

ConceptMeaning
FlowTrackerHolds ordered cohort/filter steps for one analysis.
stepOne recorded DataFrame state after a filter, join, review, or endpoint step.
nNumber of rows after the step. Usually patients or analysis units.
excludedDifference from the previous step.
retained_percentPercent retained relative to the first recorded step.
reasonHuman-readable exclusion reason.
metadataOptional key/value details such as query params or reviewer notes.

API Contract

TopicContract
Input shapeDataFrames passed manually to add_step or returned by decorated filter functions.
Required columnsNo fixed columns are required, but the row unit must remain stable and should be documented in metadata.
Return shapeFlowTracker exports DataFrame, Markdown, and text-flowchart representations.
Saved artifactsFlow table CSV, Markdown/text flow chart, plotted cohort flow, and provenance notes.
Failure modesDecorated functions returning non-DataFrames, unstable row units, duplicate IDs hidden by joins, or missing metadata for key criteria.

Research Decisions To Make Explicit

Flow explainability is only useful when the unit of counting is clear. Decide before tracking whether one row means one patient, one study, one lesion, one series, or one prediction. Keep that row unit stable across the flow or add a manual step that explains why the unit changed.

DecisionDocument in
Source population and extraction datefirst step metadata
Inclusion criteriastep labels and metadata
Exclusion reasonsreason
Row unitfirst step metadata and manuscript methods
Duplicate handlingmanual step after deduplication
Join behaviormanual step after merging tables
Endpoint constructionmanual step with event/censoring assumptions

Manual Tracking

Manual tracking is best for joins, merges, review exports, external tools, or any step that is not a simple function call.

python
from fyron.flow import FlowTracker

flow = FlowTracker("NSCLC CT cohort")

flow.add_step("Source FHIR cohort", raw)
flow.add_step(
    "Age >= 18",
    raw[raw["age"] >= 18],
    reason="age < 18",
    metadata={"criterion": "age_at_index >= 18"},
)
flow.add_step(
    "Has baseline CT",
    with_ct,
    reason="missing CT",
    metadata={"window": "index date -90 to +30 days"},
)
flow.add_step(
    "Has outcome follow-up",
    final,
    reason="missing follow-up",
    metadata={"minimum_followup_days": 90},
)

flow.print(color=False)

Decorated Filters

Use track_filter for DataFrame-in/DataFrame-out filters that should be recorded automatically.

python
from fyron.flow import FlowTracker, track_filter

flow = FlowTracker("NSCLC CT cohort")
flow.add_step("Source cohort", raw)

@track_filter(flow, label="Age >= 18", reason="age < 18")
def filter_adults(df):
    return df[df["age"] >= 18]

@track_filter(flow, label="Has baseline CT", reason="missing CT")
def filter_baseline_ct(df):
    return df[df["baseline_ct"].notna()]

cohort = filter_baseline_ct(filter_adults(raw))

Decorators deliberately require a pandas.DataFrame return value. Use manual add_step when a step returns several tables or needs local logic before recording.

Joins, Duplicates, And Unit Changes

Use manual steps for operations where counts can change for reasons other than a simple filter. This makes the flow auditable instead of pretending a join is an exclusion criterion.

python
flow.add_step("Source patients", patients, metadata={"row_unit": "patient"})

deduplicated = patients.drop_duplicates("patient_id")
flow.add_step(
    "Deduplicated patients",
    deduplicated,
    reason="duplicate patient_id",
    metadata={"id_col": "patient_id", "rule": "keep first reviewed row"},
)

with_imaging = deduplicated.merge(imaging, on="patient_id", how="inner")
flow.add_step(
    "Joined baseline imaging",
    with_imaging,
    reason="no matching baseline imaging row",
    metadata={"join": "inner", "on": "patient_id"},
)

If the row unit changes, say so in metadata. For example, going from one row per patient to one row per lesion should be a named manual step, not a hidden side effect.

Outputs

python
flow_table = flow.to_dataframe()
markdown = flow.to_markdown()
text_chart = flow.to_text_flowchart()

flow.to_dataframe() returns columns including step, n, excluded, retained_percent, excluded_percent, and reason. The step and n columns make it compatible with existing reporting and plotting helpers:

python
from fyron import plotting as fp
from fyron import reporting as fr

table = flow.to_dataframe()
flow_markdown = fr.dataframe_to_markdown(table)
fig, ax = fp.plot_cohort_flow(table)

What To Save

For manuscripts and reviewer responses, save all three forms when possible:

ArtifactUse
flow.to_dataframe()structured audit table and plotting input
flow.to_markdown()methods supplement or protocol notes
flow.to_text_flowchart()readable console/log output

Add those files to your provenance manifest together with source cohort hashes, filter parameters, and output paths.

Console Output

text
fyron cohort flow: NSCLC CT cohort

[1] Source cohort       n=1,240
[2] Age >= 18           n=1,210   excluded=30   reason=age < 18
[3] Has baseline CT     n=830     excluded=380  reason=missing CT

Final cohort: 830 / 1,240 retained (66.9%)

Common Pitfalls

  • Record the source cohort first so retention percentages have a meaningful denominator.
  • Use one row per patient or one row per analysis unit consistently.
  • Keep reasons short and reviewer-facing.
  • Do not hide clinical assumptions inside filter names; include key criteria in labels or metadata.
  • Use manual steps for joins and endpoint construction where the row count may change for reasons other than filtering.