OpenEvidence Incident Runbook

Table of Contents

• Overview
• Prerequisites
• Instructions
• Output
• Error Handling
• Examples
• Resources

Overview

Rapid incident response procedures for OpenEvidence clinical AI integration outages in healthcare environments. Includes severity classification, triage steps, error-specific remediation, fallback procedures, and postmortem templates.

Prerequisites

• Access to OpenEvidence status page
• kubectl access to production cluster
• Prometheus/Grafana access
• PagerDuty or on-call system access

Severity Levels

Critical Note

OpenEvidence outages may affect clinical decision-making. Always communicate clearly with clinical staff, ensure fallback procedures are known, and document any clinical impact.

Instructions

Step 1: Initial Assessment (2 minutes)

Check OpenEvidence status page, your integration health endpoint, error rate metrics, and recent error logs.

Step 2: Follow Decision Tree

• API errors + OpenEvidence status incident -> Enable fallback, wait for resolution
• API errors + no status incident -> Check credentials, config, network
• No API errors + unhealthy service -> Infrastructure issue (pods, memory, network)

Step 3: Apply Error-Specific Remediation

• 401/403: Verify API key, rotate if needed, restart pods
• 429: Enable request queuing, contact OpenEvidence for limit increase
• 500/503: Enable graceful degradation, notify clinical staff
• Timeout: Increase timeout temporarily, check network latency

Step 4: Enable Fallback

Return helpful message directing to UpToDate, DynaMed, or clinical guidelines directly.

Step 5: Communicate

Notify clinical staff via Slack/Teams and email. Update status page.

Step 6: Post-Incident

Collect evidence (logs, metrics, alerts), run postmortem with clinical impact assessment, create action items.

Output

• Quick triage procedure completed
• Issue identified and categorized
• Remediation applied
• Clinical staff notified
• Evidence collected for postmortem

Error Handling

Examples

One-Line Health Check

set -euo pipefail
curl -sf https://api.yourhealthcare.com/health/openevidence | jq '.status' || echo "UNHEALTHY"

Enable/Disable Fallback

set -euo pipefail
kubectl set env deployment/clinical-evidence-api OPENEVIDENCE_FALLBACK=true   # Enable
kubectl set env deployment/clinical-evidence-api OPENEVIDENCE_FALLBACK=false  # Disable

See detailed implementation for advanced patterns.

Resources

• OpenEvidence Status
• OpenEvidence Support