技能 数据科学 临床数据文本去标识化

临床数据文本去标识化

v20260803
deidentify-a-dataset
该工具用于对本地的结构化数据集(如CSV, JSONL, Parquet)中的自由文本列进行去标识化处理。它利用OpenMed模型移除个人健康信息(PHI),生成一个完全脱敏的数据集和一个不含PHI的聚合摘要,从而实现数据的安全共享和分析,同时确保源数据的隐私不泄露。
获取技能
362 次下载
概览

De-identify a dataset

Keep the source local, name the free-text columns explicitly, and write to a
different destination. Never infer columns or print source and redacted cell
values.

Procedure

  1. Confirm that the input is CSV, JSONL/NDJSON, or Parquet.
  2. Confirm which columns contain free text. Do not scan or log values to guess.
  3. Choose a policy and language. Prefer strict_no_leak when recall is the

governing safety requirement.

  1. Write to a new path; never overwrite the input.
  2. Inspect only result.summary, which contains aggregate counts and rates.
  3. Validate recall and residual leakage on representative synthetic or

approved evaluation fixtures before releasing the output.

Runnable synthetic example

Install the model runtime first with python -m pip install "openmed[hf]".

import csv
from pathlib import Path

from openmed import redact_dataset

source = Path("synthetic-notes.csv")
destination = Path("synthetic-notes.redacted.csv")

with source.open("w", newline="", encoding="utf-8") as handle:
    writer = csv.DictWriter(handle, fieldnames=["record_id", "note"])
    writer.writeheader()
    writer.writerows(
        [
            {
                "record_id": "SYNTH-001",
                "note": (
                    "Taylor Example called 212-555-0198 about a "
                    "metformin refill."
                ),
            },
            {
                "record_id": "SYNTH-002",
                "note": (
                    "Send the synthetic follow-up to "
                    "demo.patient@example.test."
                ),
            },
        ]
    )

result = redact_dataset(
    source,
    text_columns=["note"],
    output_path=destination,
    policy="strict_no_leak",
    lang="en",
)

print(result.output_path)
print(result.summary.to_dict())  # Aggregate counts only; no cell contents.

Use the equivalent CLI for an existing dataset:

openmed redact-dataset notes.csv \
  --text-columns note,comment \
  --policy strict_no_leak \
  --output notes.redacted.csv

Safety checks

  • Keep model inference and files on infrastructure the user controls.
  • Do not print input rows, detected entity surfaces, reversible mappings, or

exception payloads that may contain source text.

  • Keep source and output paths separate and access-controlled.
  • Treat the aggregate summary as evidence, not as proof of compliance.
  • Never commit real clinical data or restricted evaluation corpora.

Repository example

Read and run
the offline dataset walkthrough when
you need a bundled synthetic fixture and first-run download controls.

信息
分类 数据科学
名称 deidentify-a-dataset
版本 v20260803
大小 2.97KB
更新时间 2026-08-04
语言