Bits & Flames
bitsandflames/fyron

Documents

fyron.documents downloads clinical documents, PDFs, reports, text files, or other URL-addressable artifacts. It supports individual URLs, URL lists, and DataFrame-driven batch downloads.

Document downloads are often the bridge between structured cohorts and narrative evidence. Store the returned metadata with your cohort so each local file can be traced back to the source URL, row identifier, and download status. Avoid mixing document retrieval with LLM extraction in the same step; first collect the files, then analyze them.

Document Workflow

  1. Resolve or construct document URLs from your source system.
  2. Download documents with deterministic output paths.
  3. Store hashes, status, and local paths.
  4. Review failed downloads separately from content extraction.
  5. Pass local paths to downstream parsing or LLM workflows only after retrieval is complete.

Import

python
from fyron import DocumentDownloader

DocumentDownloader

python
loader = DocumentDownloader(
    base_url="https://example.org",
    output_dir="docs_out",
    skip_existing=True,
)

Constructor Parameters

ParameterRequiredTypeDefaultDescription
authno`Authrequests.SessionNone`NoneReuse Fyron auth/session.
base_urlno`strNone`env FHIR_BASE_URLPrefix for relative URLs.
output_dirno`strPath`"documents_out"Parent folder for downloads.
timeoutnoint30Request timeout in seconds.
skip_existingnoboolTrueSkip files already downloaded.
log_downloadsnoboolFalseLog each download.
max_workersnoint4Thread workers for URL lists/DataFrames.
retriesno`urllib3 RetryNone`default retry policyCustom retry behavior.
save_modenostr"auto"Save behavior: auto/text/PDF style handling.
force_extensionno`strNone`NoneForce a file extension.
basic_authno`(str, str)None`NoneStandalone basic auth tuple.

download_url

Downloads one URL into a deterministic folder based on URL hash.

python
result = loader.download_url("https://example.org/report_001.pdf")
ParameterRequiredDescription
urlyesAbsolute URL or relative URL when base_url is set.

Returns a DocumentResult with URL, local path, status, content type, hash, and error metadata.

download_urls

Downloads many URLs, optionally writing a results CSV.

python
results = loader.download_urls(
    [
        "https://example.org/report_001.pdf",
        "https://example.org/report_002.pdf",
    ],
    results_csv="document_downloads.csv",
)
ParameterRequiredDefaultDescription
urlsyesnoneIterable of URLs.
results_csvnoNoneOptional CSV path for results.

Returns a DataFrame of download results.

download_from_df

Downloads URLs from a DataFrame column.

python
results = loader.download_from_df(
    documents_df,
    url_col="document_url",
    results_csv="document_downloads.csv",
)
ParameterRequiredDefaultDescription
dfyesnoneInput DataFrame.
url_colno"document_url"Column containing URLs.
results_csvnoNoneOptional output CSV.

download_from_dataframe is the same method under the longer name.

Static Helper: download_id

python
download_id = DocumentDownloader.download_id("https://example.org/report.pdf")

Returns a deterministic SHA-256 ID for a URL. This ID is used for stable folder naming.

Context Manager

python
with DocumentDownloader(output_dir="docs") as loader:
    results = loader.download_urls(urls)

The context manager closes owned HTTP sessions.

Practical Tips

  • Use results_csv for long-running batches so failures can be inspected.
  • Use skip_existing=True to make download jobs restartable.
  • Use basic_auth or auth, not both.
  • Use force_extension=".pdf" only when the server does not provide useful filenames/content types.