技能 编程开发 Firecrawl企业级访问控制

Firecrawl企业级访问控制

v20260423
firecrawl-enterprise-rbac
本指南介绍了如何在企业环境中为Firecrawl爬取资源实施强大的基于角色的访问控制(RBAC)。它允许企业管理多个API密钥,为每个团队设定信用额度限制,并限制爬取目标到指定的白名单域名。适用于需要精细资源治理和预算跟踪的大规模爬取运营场景。
获取技能
382 次下载
概览

Firecrawl Enterprise RBAC

Overview

Control access to Firecrawl scraping resources through API key management, domain allowlists, and credit budgets per team. Firecrawl's credit-based pricing means access control is primarily about limiting credit consumption and restricting scrape targets per consumer.

Prerequisites

  • Firecrawl Team or Scale plan
  • Dashboard access at firecrawl.dev/app
  • Understanding of credit-per-page billing

Instructions

Step 1: Separate API Keys per Consumer

set -euo pipefail
# Create dedicated keys at firecrawl.dev/app for each team/service

# Content indexing pipeline — high volume
# Key: fc-content-indexer-prod (monthly credit limit: 50,000)

# Sales team prospect research — scrape only
# Key: fc-sales-research (monthly credit limit: 5,000)

# Dev/testing — minimal
# Key: fc-dev-testing (monthly credit limit: 500)

Step 2: Gateway Proxy with Domain Allowlists

import FirecrawlApp from "@mendable/firecrawl-js";

const TEAM_POLICIES: Record<string, {
  apiKey: string;
  allowedDomains: string[];
  maxPagesPerCrawl: number;
  dailyCreditLimit: number;
}> = {
  "content-team": {
    apiKey: process.env.FIRECRAWL_KEY_CONTENT!,
    allowedDomains: ["docs.*", "*.readthedocs.io", "medium.com"],
    maxPagesPerCrawl: 200,
    dailyCreditLimit: 2000,
  },
  "sales-team": {
    apiKey: process.env.FIRECRAWL_KEY_SALES!,
    allowedDomains: ["linkedin.com", "crunchbase.com", "g2.com"],
    maxPagesPerCrawl: 20,
    dailyCreditLimit: 500,
  },
  "engineering": {
    apiKey: process.env.FIRECRAWL_KEY_ENGINEERING!,
    allowedDomains: ["*"],  // unrestricted
    maxPagesPerCrawl: 100,
    dailyCreditLimit: 1000,
  },
};

function isDomainAllowed(team: string, url: string): boolean {
  const policy = TEAM_POLICIES[team];
  if (!policy) return false;
  const domain = new URL(url).hostname;
  return policy.allowedDomains.some(pattern =>
    pattern === "*" || domain.endsWith(pattern.replace("*.", "").replace("*", ""))
  );
}

function getTeamClient(team: string): FirecrawlApp {
  const policy = TEAM_POLICIES[team];
  if (!policy) throw new Error(`Unknown team: ${team}`);
  return new FirecrawlApp({ apiKey: policy.apiKey });
}

Step 3: Credit Budget Enforcement

class TeamBudget {
  private usage = new Map<string, Map<string, number>>(); // team -> date -> credits

  record(team: string, credits: number) {
    const today = new Date().toISOString().split("T")[0];
    if (!this.usage.has(team)) this.usage.set(team, new Map());
    const teamUsage = this.usage.get(team)!;
    teamUsage.set(today, (teamUsage.get(today) || 0) + credits);
  }

  canAfford(team: string, credits: number): boolean {
    const policy = TEAM_POLICIES[team];
    if (!policy) return false;
    const today = new Date().toISOString().split("T")[0];
    const used = this.usage.get(team)?.get(today) || 0;
    return used + credits <= policy.dailyCreditLimit;
  }

  getUsage(team: string): number {
    const today = new Date().toISOString().split("T")[0];
    return this.usage.get(team)?.get(today) || 0;
  }
}

const budget = new TeamBudget();

Step 4: Policy-Enforced Scraping

export async function teamScrape(team: string, url: string) {
  // Check domain policy
  if (!isDomainAllowed(team, url)) {
    throw new Error(`Team "${team}" is not allowed to scrape ${new URL(url).hostname}`);
  }

  // Check credit budget
  if (!budget.canAfford(team, 1)) {
    throw new Error(`Team "${team}" has exceeded daily credit limit`);
  }

  // Scrape with team's API key
  const client = getTeamClient(team);
  const result = await client.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
  });

  budget.record(team, 1);
  return result;
}

export async function teamCrawl(team: string, url: string, pages: number) {
  const policy = TEAM_POLICIES[team];
  if (!policy) throw new Error(`Unknown team: ${team}`);

  if (!isDomainAllowed(team, url)) {
    throw new Error(`Domain not allowed for team "${team}"`);
  }

  const limit = Math.min(pages, policy.maxPagesPerCrawl);
  if (!budget.canAfford(team, limit)) {
    throw new Error(`Crawl of ${limit} pages exceeds "${team}" daily budget`);
  }

  const client = getTeamClient(team);
  const result = await client.crawlUrl(url, {
    limit,
    maxDepth: 3,
    scrapeOptions: { formats: ["markdown"] },
  });

  budget.record(team, result.data?.length || 0);
  return result;
}

Step 5: Key Rotation Schedule

set -euo pipefail
# Rotate keys quarterly:
# 1. Create new key at firecrawl.dev/app
# 2. Deploy new key alongside old (both valid)
# 3. Verify new key works
curl -s https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $NEW_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}' | jq .success
# 4. Remove old key from all services
# 5. Delete old key in dashboard after 48-hour overlap

Error Handling

IssueCauseSolution
402 Payment RequiredTeam credit limit reachedIncrease limit or wait for reset
403 on domainDomain not in allowlistAdd domain to team policy
Unexpected credit burnNo crawl limit enforcedUse maxPagesPerCrawl from policy
Wrong team key usedConfig errorVerify key-to-team mapping

Examples

Audit Team Usage

for (const team of Object.keys(TEAM_POLICIES)) {
  console.log(`${team}: ${budget.getUsage(team)} credits today`);
}

Resources

Next Steps

For migration strategies, see firecrawl-migration-deep-dive.

信息
分类 编程开发
名称 firecrawl-enterprise-rbac
版本 v20260423
大小 6.31KB
更新时间 2026-07-13
语言