Authoring Model Cards

A model card is the honest spec sheet for a model: what it's for, how well it
works, where it breaks, and who it might fail. For clinical models this is
governance-critical — an undocumented de-id model is one nobody can sign off on.
This skill fills a model card directly from OpenMed eval outputs so the numbers
are reproducible, not aspirational.

When to use this skill

• You're publishing or updating an OpenMed model and need its card.
• You have eval artifacts (GateReport, fairness_report, error_report) and

need to turn them into intended-use, metrics, and limitations sections.

• A clinical AI governance / model-risk review needs a transparency document.

Run the evals first (see evaluating-with-leakage-gates,
benchmarking-clinical-ner, auditing-subgroup-fairness); this skill documents
their results — it does not generate the numbers.

Card sections (Mitchell et al., + clinical extensions)

See references/model-card-sections.md for the full section-to-source map. The
load-bearing sections for an OpenMed model:

• Model details — repo id, family, tier, format, params, milestone, license

(Apache-2.0). Pull from the GateReport identity fields.

• Intended use — the clinical task and the deployment envelope.
• Out-of-scope / misuse — explicitly: not a medical device; not for autonomous

clinical decisions; de-id is verified, not assumed.

• Metrics — entity-level P/R/F1 and, for de-id, residual leakage + per-label

recall floors and the gate decision.

• Quantitative analysis (subgroups) — per-group leakage/recall from

fairness_report, including which groups lack data.

• Limitations — error patterns from error_report; calibration assumptions.
• Caveats & disclaimer — the medical-device disclaimer.

Quick start — fill the card from eval outputs

from openmed.eval import (
    run_suite, ReleaseGate, fairness_report, error_report,
)

report = run_suite("eval/gold/test.json", suite="golden",
                   model_name="OpenMed/Privacy-PII-Detection", device="cpu",
                   metadata={"family": "PII", "tier": "base",
                             "policy": "hipaa_safe_harbor"})

gate = ReleaseGate(milestone="v1.6", policy="hipaa_safe_harbor").evaluate(report)
fair = fairness_report("OpenMed/Privacy-PII-Detection", "golden")
errs = error_report("OpenMed/Privacy-PII-Detection", "eval/gold/test.json")

card = {
    "model_details": {
        "repo_id": gate.repo_id, "family": gate.family, "tier": gate.tier,
        "format": gate.format, "license": "Apache-2.0",
    },
    "metrics": {
        "exact_span_f1": report.metrics["exact_span_f1"]["f1"],
        "residual_leakage_rate": gate.residual_leakage_rate,
        "critical_leakage_count": gate.critical_leakage_count,
        "per_label_recall": dict(gate.per_label_recall),
        "release_decision": gate.decision,            # RELEASABLE / QUARANTINED
    },
    "subgroup_analysis": fair.to_dict(),              # per-group leakage/recall
    "limitations": errs.to_dict()["confusion_matrix"],
}
# Render `card` into Markdown front matter + body (or the HF card template).

error_report and fairness_report carry no plaintext PHI (offsets + hashes),
so their output is safe to paste into a public card.

Workflow

1. Gather artifacts. Gate report, fairness report, error report — all from a

pinned model + synthetic eval set.

1. Fill model details from the GateReport identity fields so the card,

models.jsonl, and the README cannot drift (the gate's manifest_coherence
and model_card checks enforce this).

1. Write intended use narrowly. Name the clinical task, language(s), and the

deployment envelope. Over-broad intended-use is the most common card failure.

1. State out-of-scope and the disclaimer plainly (see template below).
2. Report metrics with their floors. For de-id, lead with leakage and the

gate decision, not F1.

1. Report subgroups honestly, including the documentation gap: if race/

ethnicity isn't available, say so rather than implying parity.

1. List limitations from real errors, not boilerplate — cite the confusion

matrix's worst cells.

Disclaimer block (paste & adapt)

This model assists clinical text processing and is not a medical device.
It does not make autonomous clinical decisions. De-identification output must be
independently verified before any data is shared; residual PHI risk is never
zero. Validate on your own population before deployment.

Hand-off to / from OpenMed

• From evaluating-with-leakage-gates (GateReport),

benchmarking-clinical-ner (error_report), and auditing-subgroup-fairness
(fairness_report): these are the card's evidence.

• To building-with-openmed / models.jsonl: keep card front matter

(license, task, languages) coherent with the manifest — the gate checks it.

• Pairs with gating-deid-leakage: cite the green gate as the card's

release evidence.

Edge cases & gotchas

• Don't claim numbers you can't reproduce. Every metric in the card should

trace to an eval artifact and a pinned eval-set hash.

• Intended use ≠ capability. Document the supported envelope; mark everything

else out-of-scope.

• Subgroup silence is a finding. Omitting race because it wasn't collected is

itself a limitation to state — don't let absence read as equity.

• Card/manifest drift fails the gate. License/task/language mismatches between

the card and models.jsonl trip manifest_coherence.

• No raw PHI examples. Use the offset/hash examples from error_report; never

paste real patient strings as "qualitative examples".

• Quantized variants need their own line. Report INT8/INT4 recall deltas

(G4) per format; don't reuse the fp32 numbers.

Standards & references

• Mitchell et al., Model Cards for Model Reporting (FAT* 2019):

https://arxiv.org/abs/1810.03993

• Hugging Face model card spec & template:

https://huggingface.co/docs/hub/model-cards

• Sendak et al., *Presenting machine learning model information to clinical end

users* (clinical "model facts" label): https://doi.org/10.1038/s41746-020-0253-3

• FDA, Clinical Decision Support Software guidance:

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software

• Section-to-source map: references/model-card-sections.md.