Firecrawl Multi-Environment Setup

Overview

Firecrawl's credit-based pricing model makes environment separation critical. Development should use conservative crawl limits to avoid burning production credits during testing.

Prerequisites

• Firecrawl account with API key from firecrawl.dev
• Optional: self-hosted Firecrawl instance for development
• Understanding of credit consumption model (1 credit = 1 page scraped)

Environment Strategy

Instructions

Step 1: Configuration Structure

// config/firecrawl.ts
import FirecrawlApp from "@mendable/firecrawl-js";

type Env = "development" | "staging" | "production";

interface FirecrawlConfig {
  apiKey: string;
  apiUrl?: string;              // override for self-hosted
  maxPagesPerCrawl: number;     // hard limit per crawl job
  maxDepth: number;
  concurrency: number;
  waitFor: number;              // ms to wait for JS rendering
}

const configs: Record<Env, FirecrawlConfig> = {
  development: {
    apiKey: process.env.FIRECRAWL_API_KEY_DEV || "fc-localdev",
    apiUrl: process.env.FIRECRAWL_API_URL_DEV, // point to self-hosted if available
    maxPagesPerCrawl: 10,         // strict limit in dev to protect credits
    maxDepth: 2,
    concurrency: 1,
    waitFor: 2000,  # 2000: 2 seconds in ms
  },
  staging: {
    apiKey: process.env.FIRECRAWL_API_KEY_STAGING!,
    maxPagesPerCrawl: 100,
    maxDepth: 3,
    concurrency: 2,
    waitFor: 3000,  # 3000: 3 seconds in ms
  },
  production: {
    apiKey: process.env.FIRECRAWL_API_KEY_PROD!,
    maxPagesPerCrawl: 500,        // per-task limit, set lower for specific jobs  # HTTP 500 Internal Server Error
    maxDepth: 5,
    concurrency: 5,
    waitFor: 3000,  # 3 seconds in ms
  },
};

export function getFirecrawlConfig(): FirecrawlConfig {
  const env = (process.env.NODE_ENV || "development") as Env;
  return configs[env] || configs.development;
}

export function getFirecrawlClient(): FirecrawlApp {
  const cfg = getFirecrawlConfig();
  return new FirecrawlApp({
    apiKey: cfg.apiKey,
    ...(cfg.apiUrl ? { apiUrl: cfg.apiUrl } : {}),
  });
}

Step 2: Self-Hosted Firecrawl for Development

# docker-compose.dev.yml - Run Firecrawl locally in dev
version: "3.8"
services:
  firecrawl:
    image: mendableai/firecrawl:latest
    ports:
      - "3002:3002"  # 3002 = configured value
    environment:
      - USE_DB_AUTHENTICATION=false
      - PORT=3002

Set FIRECRAWL_API_URL_DEV=http://localhost:3002 and any API key in .env.local to use local instance with no credit consumption.

Step 3: Credit-Aware Scraping Wrapper

// lib/firecrawl-service.ts
import { getFirecrawlClient, getFirecrawlConfig } from "../config/firecrawl";

export async function safeScrape(url: string, options?: any) {
  const firecrawl = getFirecrawlClient();
  const cfg = getFirecrawlConfig();

  return firecrawl.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
    waitFor: cfg.waitFor,
    ...options,
  });
}

export async function safeCrawl(url: string, customLimit?: number) {
  const firecrawl = getFirecrawlClient();
  const cfg = getFirecrawlConfig();

  const limit = Math.min(customLimit ?? cfg.maxPagesPerCrawl, cfg.maxPagesPerCrawl);

  return firecrawl.asyncCrawlUrl(url, {
    limit,
    maxDepth: cfg.maxDepth,
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });
}

Step 4: Environment Variable Setup

# .env.local (development)
FIRECRAWL_API_KEY_DEV=fc-dev-abc123
FIRECRAWL_API_URL_DEV=http://localhost:3002  # 3002: optional self-hosted

# GitHub Actions - Staging environment
FIRECRAWL_API_KEY_STAGING=fc-staging-def456

# GitHub Actions - Production environment
FIRECRAWL_API_KEY_PROD=fc-prod-xyz789

Step 5: CI/CD Pipeline Integration

# .github/workflows/deploy.yml
jobs:
  test-scraping:
    environment: staging
    env:
      FIRECRAWL_API_KEY_STAGING: ${{ secrets.FIRECRAWL_API_KEY_STAGING }}
      NODE_ENV: staging
    steps:
      - run: node scripts/test-firecrawl.js
        # Uses staging config: 100-page limit, staging API key

  deploy:
    environment: production
    env:
      FIRECRAWL_API_KEY_PROD: ${{ secrets.FIRECRAWL_API_KEY_PROD }}
      NODE_ENV: production

Error Handling

Examples

Check Active Configuration

import { getFirecrawlConfig } from "./config/firecrawl";

const cfg = getFirecrawlConfig();
console.log(`Max pages: ${cfg.maxPagesPerCrawl}, depth: ${cfg.maxDepth}`);
console.log(`API URL: ${cfg.apiUrl || "https://api.firecrawl.dev"}`);

Budget Check Before Large Crawl

set -euo pipefail
# Check current credit balance
curl -s "https://api.firecrawl.dev/v1/team/credits" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" | jq .credits_remaining

Resources

• Firecrawl API Documentation
• Firecrawl Self-Hosting Guide
• Firecrawl Pricing

Next Steps

For deployment configuration, see firecrawl-deploy-integration.

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale