FireCrawl Reference Architecture

Overview

Production architecture for web scraping and data extraction with FireCrawl. Covers crawl job orchestration, content extraction pipelines, structured data output, and site mapping workflows.

Prerequisites

• FireCrawl API key
• @mendable/firecrawl-js SDK installed
• Queue system for large crawl jobs (optional)
• Storage for extracted content

Architecture Diagram

┌─────────────────────────────────────────────────────┐
│                 Crawl Orchestrator                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │
│  │ Scrape   │  │ Crawl    │  │ Map              │   │
│  │ (single) │  │ (multi)  │  │ (discovery)      │   │
│  └────┬─────┘  └────┬─────┘  └────────┬─────────┘   │
│       │              │                 │             │
│       ▼              ▼                 ▼             │
│  ┌──────────────────────────────────────────────┐    │
│  │         Content Processing Pipeline           │    │
│  │  Markdown │ HTML │ Screenshots │ Structured  │    │
│  └──────────────────────┬───────────────────────┘    │
│                         │                            │
│  ┌──────────────────────┴───────────────────────┐    │
│  │         Output & Storage                      │    │
│  │  JSON Files │ Database │ Vector Store │ S3    │    │
│  └──────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────┘

Instructions

Step 1: FireCrawl Service Layer

import FirecrawlApp from '@mendable/firecrawl-js';

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// Single page scrape with markdown output
async function scrapePage(url: string) {
  return firecrawl.scrapeUrl(url, {
    formats: ['markdown', 'html'],
    onlyMainContent: true,
    waitFor: 2000, // Wait for dynamic content  # 2000: 2 seconds in ms
  });
}

// Structured data extraction with schema
async function extractStructured(url: string, schema: any) {
  return firecrawl.scrapeUrl(url, {
    formats: ['extract'],
    extract: {
      schema,
      systemPrompt: 'Extract data precisely according to the schema.',
    },
  });
}

Step 2: Multi-Page Crawl Pipeline

async function crawlSite(baseUrl: string, options?: {
  maxPages?: number;
  includePaths?: string[];
  excludePaths?: string[];
}) {
  const crawlResult = await firecrawl.crawlUrl(baseUrl, {
    limit: options?.maxPages || 50,
    includePaths: options?.includePaths,
    excludePaths: options?.excludePaths || ['/blog/*', '/news/*'],
    scrapeOptions: {
      formats: ['markdown'],
      onlyMainContent: true,
    },
  });

  return crawlResult;
}

// Async crawl for large sites
async function asyncCrawl(baseUrl: string) {
  const job = await firecrawl.asyncCrawlUrl(baseUrl, {
    limit: 500,  # HTTP 500 Internal Server Error
    scrapeOptions: { formats: ['markdown'] },
  });

  // Poll for completion
  let status = await firecrawl.checkCrawlStatus(job.id);
  while (status.status === 'scraping') {
    await new Promise(r => setTimeout(r, 5000));  # 5000: 5 seconds in ms
    status = await firecrawl.checkCrawlStatus(job.id);
  }

  return status;
}

Step 3: Site Map Discovery

async function discoverSiteStructure(url: string) {
  const mapResult = await firecrawl.mapUrl(url);

  // Categorize discovered URLs
  const structure = {
    pages: mapResult.links?.filter(l => !l.includes('/api/')) || [],
    apiDocs: mapResult.links?.filter(l => l.includes('/api/') || l.includes('/docs/')) || [],
    blog: mapResult.links?.filter(l => l.includes('/blog/')) || [],
    total: mapResult.links?.length || 0,
  };

  return structure;
}

Step 4: Structured Extraction Pipeline

import { z } from 'zod';

const ProductSchema = z.object({
  name: z.string(),
  price: z.number(),
  description: z.string(),
  features: z.array(z.string()),
  availability: z.enum(['in_stock', 'out_of_stock', 'preorder']),
});

async function extractProducts(urls: string[]) {
  const results = [];
  for (const url of urls) {
    const data = await extractStructured(url, {
      type: 'object',
      properties: {
        name: { type: 'string' },
        price: { type: 'number' },
        description: { type: 'string' },
        features: { type: 'array', items: { type: 'string' } },
        availability: { type: 'string', enum: ['in_stock', 'out_of_stock', 'preorder'] },
      },
      required: ['name', 'price'],
    });
    results.push(data);
  }
  return results;
}

Error Handling

Examples

Documentation Scraper

async function scrapeDocumentation(docsUrl: string) {
  const sitemap = await discoverSiteStructure(docsUrl);
  const docPages = sitemap.apiDocs.slice(0, 100);

  const crawl = await crawlSite(docsUrl, {
    maxPages: 100,
    includePaths: ['/docs/*', '/api/*', '/guide/*'],
  });

  return crawl;
}

Resources

• FireCrawl Documentation
• FireCrawl Scraping Guide
• FireCrawl Extraction

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale