OpenEvidence Observability

Table of Contents

• Overview
• Prerequisites
• Instructions
• Output
• Error Handling
• Examples
• Resources

Overview

Set up comprehensive observability for OpenEvidence clinical AI integrations with Prometheus metrics, OpenTelemetry tracing, structured logging with PHI redaction, and Grafana dashboards.

Prerequisites

• Prometheus or compatible metrics backend
• OpenTelemetry SDK installed
• Grafana or similar dashboarding tool
• AlertManager or PagerDuty configured

Key Metrics

Instructions

Step 1: Set Up Prometheus Metrics

Register counters (requests, errors, cache hits/misses), histograms (request duration, DeepConsult duration), gauges (rate limit remaining), and summaries (confidence scores) with appropriate labels.

Step 2: Instrument Client Wrapper

Wrap OpenEvidence client with metrics collection, cache hit/miss tracking, rate limit monitoring from response headers, and confidence score recording.

Step 3: Configure Distributed Tracing

Initialize OpenTelemetry with Google Cloud Trace exporter, HTTP and Express instrumentations, and service metadata.

Step 4: Set Up Structured Logging

Use pino with PHI redaction (patient.*, patientId, mrn, *.ssn) and OpenEvidence-specific child logger for clinical queries and DeepConsult events.

Step 5: Define Alert Rules

Create Prometheus alerts for: high error rate (>5% warning, >20% critical), high P95 latency (>15s), low cache hit rate (<50%), rate limit warning (<10 remaining), service down.

Step 6: Build Grafana Dashboard

Create panels for request rate, error rate gauge, latency heatmap, cache hit rate timeseries, rate limit gauge, and confidence score distribution.

Output

• Prometheus metrics collection with /metrics endpoint
• Instrumented client wrapper with tracing
• Distributed tracing (OpenTelemetry + Cloud Trace)
• Structured logging with PHI redaction
• Alert rules for critical conditions
• Grafana dashboard JSON

Error Handling

Examples

Metrics Endpoint

router.get('/metrics', async (req, res) => {
  res.set('Content-Type', registry.contentType);
  res.send(await registry.metrics());
});

See detailed implementation for advanced patterns.

Resources

• Prometheus Best Practices
• OpenTelemetry Documentation
• Grafana Dashboards