Deepgram Incident Runbook

Contents

• Overview
• Prerequisites
• Instructions
• Output
• Error Handling
• Examples
• Resources

Overview

Standardized procedures for responding to Deepgram-related incidents with initial triage script, severity-based response (SEV1-SEV4), fallback activation, degradation investigation, and post-incident review templates.

Prerequisites

• Monitoring and alerting configured
• On-call rotation established
• Fallback/queueing system available
• Communication channels defined

Instructions

Step 1: Run Initial Triage (First 5 Minutes)

Execute triage script: check Deepgram status page, query error rate from Prometheus, check P95 latency, and test API connectivity with curl.

Step 2: Classify Severity

SEV1 (immediate): 100% failure, 5xx errors. SEV2 (<15min): 50%+ error rate. SEV3 (<1hr): elevated latency. SEV4 (<24hr): single feature affected.

Step 3: Respond to SEV1 (Complete Outage)

Acknowledge in PagerDuty/Slack. Verify API key validity. Check network. Activate fallback: queue requests for later replay, or switch to backup STT provider. Notify affected customers.

Step 4: Respond to SEV2 (Major Degradation)

Test transcription across multiple samples and models. Identify if specific model, feature, or audio type is affected. Mitigate: reduce request rate, disable non-critical features, switch models, enable retries.

Step 5: Respond to SEV3 (Minor Degradation)

Increase timeouts to 60s, enable aggressive retry (5 attempts), switch to simpler model (Nova), disable diarization. Monitor for improvement.

Step 6: Conduct Post-Incident Review

Document timeline, root cause, impact (duration, failed requests, revenue). List what went well and areas for improvement. Create action items with owners and due dates.

See detailed implementation for advanced patterns.

Output

• Automated triage script
• Severity classification guide
• Fallback activation procedures
• Degradation investigation playbook
• Post-incident review template

Error Handling

Examples

Quick Reference

Severity Levels

Escalation Contacts

Resources

• Deepgram Status Page
• Deepgram Support
• Internal Runbooks