Firecrawl Architecture Variants

Overview

Deployment architectures for Firecrawl web scraping at different scales. Firecrawl's async crawl model, credit billing, and JavaScript rendering support different architectures from simple page scraping to enterprise content ingestion pipelines.

Prerequisites

• Firecrawl API configured
• Clear scraping use case defined
• Infrastructure for async job processing

Instructions

Step 1: On-Demand Scraping (Simple)

Best for: Single-page scraping, < 500 pages/day, content extraction.

User Request -> Backend -> Firecrawl scrapeUrl -> Parse Content -> Response

app.post('/extract', async (req, res) => {
  const result = await firecrawl.scrapeUrl(req.body.url, {
    formats: ['markdown'], onlyMainContent: true
  });
  res.json({ content: result.markdown, title: result.metadata.title });
});

Step 2: Scheduled Crawl Pipeline (Moderate)

Best for: Content monitoring, 500-10K pages/day, documentation indexing.

Scheduler (cron) -> Crawl Queue -> Firecrawl crawlUrl -> Result Store
                                                              |
                                                              v
                                                    Content Processor -> Search Index

// Scheduled crawler
cron.schedule('0 2 * * *', async () => {  // Daily at 2 AM
  const sites = await db.getCrawlTargets();
  for (const site of sites) {
    const crawl = await firecrawl.asyncCrawlUrl(site.url, {
      limit: site.maxPages, includePaths: site.paths
    });
    await db.saveCrawlJob({ siteId: site.id, jobId: crawl.id });
  }
});

// Separate worker polls for results
async function processCrawlResults() {
  const pending = await db.getPendingCrawlJobs();
  for (const job of pending) {
    const status = await firecrawl.checkCrawlStatus(job.jobId);
    if (status.status === 'completed') {
      await indexPages(status.data);
      await db.markComplete(job.id);
    }
  }
}

Step 3: Real-Time Content Pipeline (Scale)

Best for: Enterprise, 10K+ pages/day, AI training data, knowledge base.

URL Sources -> Priority Queue -> Firecrawl Workers -> Content Validation
                                                            |
                                                            v
                                                     Vector DB + Search Index
                                                            |
                                                            v
                                                      RAG / AI Pipeline

class ContentPipeline {
  async ingest(urls: string[], priority: 'high' | 'normal' | 'low') {
    const budget = this.creditTracker.canAfford(urls.length);
    if (!budget) throw new Error('Daily credit budget exceeded');

    const results = await firecrawl.batchScrapeUrls(urls, {
      formats: ['markdown'], onlyMainContent: true
    });

    const validated = results.filter(r => this.validateContent(r));
    await this.vectorStore.upsert(validated);
    this.creditTracker.record(urls.length);
    return { ingested: validated.length, rejected: urls.length - validated.length };
  }
}

Decision Matrix

Error Handling

Examples

Architecture Selection

< 500 pages/day, user-facing: On-Demand  # HTTP 500 Internal Server Error
500-10K pages, batch processing: Scheduled Pipeline    # HTTP 500 Internal Server Error
10K+, AI/ML ingestion: Real-Time Pipeline

Resources

• Firecrawl API Docs

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale