OpenEvidence Production Checklist

Overview

Comprehensive checklist for deploying OpenEvidence clinical AI integrations to production healthcare environments.

Prerequisites

• Completed development and testing phases
• Signed BAA with OpenEvidence
• Security review completed
• Compliance team sign-off

Production Readiness Checklist

1. Legal & Compliance

• Business Associate Agreement (BAA) signed

  • Contact: compliance@openevidence.com
  • Required for any PHI handling

• API Terms of Service accepted

  • Review: https://www.openevidence.com/policies/api

• Clinical use disclaimers in place

  • All clinical outputs must include appropriate disclaimers
  • "For informational purposes only. Verify with current guidelines."

• Liability and malpractice considerations reviewed

  • Legal review of clinical decision support usage

• Data processing agreement documented

  • What data is sent, stored, processed

2. Security

• Production API keys generated

  • Separate keys from development/staging
  • Stored in secret manager (not environment variables)

• Key rotation schedule established

  • Recommend: 90-day rotation cycle

• IP allowlist configured (if available)

  • Restrict API access to known IP ranges

• TLS 1.2+ enforced

  • No fallback to older protocols

• PHI sanitization implemented

  • Review openevidence-security-basics

• Webhook signature verification active

  • All webhook endpoints validate signatures

• Audit logging enabled

  • HIPAA-compliant access logging
  • Minimum 6-year retention

3. Infrastructure

• High availability configuration

  // Multiple API endpoints for failover
  const config = {
    primaryEndpoint: 'https://api.openevidence.com',
    fallbackEndpoint: 'https://api-fallback.openevidence.com',
    timeout: 30000,
    retries: 3,
  };
  

• Circuit breaker implemented

  import CircuitBreaker from 'opossum';
  
  const breaker = new CircuitBreaker(clinicalQuery, {
    timeout: 30000,
    errorThresholdPercentage: 50,
    resetTimeout: 30000,
  });
  

• Rate limit handling configured

  • Review openevidence-rate-limits
  • Request queuing for traffic spikes

• Timeout configuration appropriate

  • Clinical queries: 15-30s
  • DeepConsult: 5-10 minutes

• Health check endpoint implemented

  app.get('/health/openevidence', async (req, res) => {
    try {
      await client.health.check();
      res.json({ status: 'healthy' });
    } catch (error) {
      res.status(503).json({ status: 'unhealthy', error: error.message });
    }
  });
  

4. Monitoring & Alerting

• Metrics collection active

  • Request count, latency, error rate
  • Review openevidence-observability

• Alert rules configured

  Alert	Threshold	Severity
  Error rate > 5%	5 min window	Warning
  Error rate > 20%	5 min window	Critical
  P95 latency > 15s	5 min window	Warning
  Health check failed	2 consecutive	Critical

• Dashboard created

  • Real-time visibility into OpenEvidence integration health

• Log aggregation configured

  • Centralized logging for debugging
  • PHI excluded from logs

5. Error Handling & Resilience

• Graceful degradation implemented

  async function clinicalQueryWithFallback(question: string) {
    try {
      return await openEvidenceQuery(question);
    } catch (error) {
      // Log error, alert, return cached or default response
      await alertService.send('OpenEvidence unavailable');
      return {
        answer: 'Clinical evidence service temporarily unavailable. Please consult current guidelines directly.',
        fallback: true,
      };
    }
  }
  

• Retry logic tested

  • Exponential backoff with jitter
  • Respects Retry-After headers

• Error classification implemented

  • Review openevidence-common-errors

6. Performance

• Response caching strategy

  • Cache frequently queried clinical information
  • Cache TTL appropriate for clinical data freshness

• Connection pooling configured

  • Reuse HTTP connections

• Query optimization reviewed

  • Clear, specific clinical questions
  • Appropriate context provided

• Load testing completed

  • Test at 2x expected peak traffic
  • Verify rate limits not exceeded

7. Documentation & Training

• Runbook created

  • Incident response procedures
  • Review openevidence-incident-runbook

• On-call training completed

  • Team knows how to triage OpenEvidence issues

• User documentation updated

  • How to use clinical decision support
  • Limitations and appropriate use

• Clinical staff training

  • Understanding AI-assisted decision support
  • When to rely on vs. question AI recommendations

8. Testing & Validation

• Integration tests passing

  • Against sandbox environment

• End-to-end tests passing

  • Full workflow validation

• Clinical validation completed

  • Sample queries reviewed by clinical team
  • Responses appropriate and accurate

• Performance benchmarks met

  • P50 < 5s, P95 < 15s for clinical queries
  • P50 < 3min, P95 < 5min for DeepConsult

9. Deployment

• Blue-green or canary deployment planned

  • Gradual rollout with monitoring

• Rollback procedure documented

  • Quick revert if issues detected

• Feature flags configured

  • Ability to disable OpenEvidence quickly

• Database migrations completed

  • Audit log tables, cache tables ready

10. Post-Launch

• Monitoring dashboard reviewed daily (first week)

• Error reports triaged immediately

• User feedback collection active

• Performance baseline established

Production Configuration Example

// config/openevidence.production.ts
export const productionConfig = {
  // Credentials from secret manager
  credentials: {
    source: 'secret-manager',
    apiKeyPath: 'projects/prod/secrets/openevidence-api-key',
    orgIdPath: 'projects/prod/secrets/openevidence-org-id',
  },

  // Endpoints
  api: {
    baseUrl: 'https://api.openevidence.com',
    timeout: 30000,  # 30000: 30 seconds in ms
    retries: 3,
  },

  // Rate limiting
  rateLimits: {
    enabled: true,
    maxConcurrent: 10,
    requestsPerMinute: 100,
  },

  // Circuit breaker
  circuitBreaker: {
    enabled: true,
    errorThreshold: 50,
    resetTimeout: 30000,  # 30 seconds in ms
  },

  // Caching
  cache: {
    enabled: true,
    ttlSeconds: 3600,  # 3600: timeout: 1 hour
    maxEntries: 10000,  # 10000: 10 seconds in ms
  },

  // Monitoring
  monitoring: {
    metricsEnabled: true,
    tracingEnabled: true,
    logLevel: 'info',
  },
};

Output

• Completed production readiness checklist
• Configuration validated
• All stakeholders signed off
• Go-live approved

Resources

• OpenEvidence
• OpenEvidence Security

Next Steps

For version upgrades, see openevidence-upgrade-migration.

Instructions

1. Assess the current state of the deployment configuration
2. Identify the specific requirements and constraints
3. Apply the recommended patterns from this skill
4. Validate the changes against expected behavior
5. Document the configuration for team reference

Error Handling

Examples

Basic usage: Apply openevidence prod checklist to a standard project setup with default configuration options.

Advanced scenario: Customize openevidence prod checklist for production environments with multiple constraints and team-specific requirements.