Skill UI

Firecrawl provides a comprehensive API for web data harvesting. It enables users to perform single-page scraping, multi-page crawling, site mapping (URL discovery), and advanced structured data extraction using Large Language Models (LLMs). Ideal for integrating data pipelines, building web monitoring tools, or conducting market research.

A comprehensive runbook providing systematic procedures for responding to Firecrawl API outages and integration failures. It guides technical teams through triage, diagnosis, and mitigation for common issues (e.g., 401, 429, 5xx errors). Includes steps for fallback implementation, evidence collection, and postmortem documentation to ensure rapid service recovery.

This guide provides comprehensive instructions for setting up the Firecrawl SDK, covering both Node.js and Python environments. It details how to install the necessary packages via npm or pip, and how to securely configure the API key using environment variables. Use this guide when initializing web scraping functionality or integrating Firecrawl into a new development project.

This comprehensive guide identifies common anti-patterns and integration mistakes when using Firecrawl for web scraping and content extraction. It serves as a vital code review checklist, helping developers avoid pitfalls such as unbounded crawling, unspecified output formats, or aggressive API polling, ensuring robust and efficient data pipelines.

This skill provides advanced methods for load testing and scaling web scraping pipelines using Firecrawl. It manages rate limits and optimizes throughput by implementing concurrency control, batch processing, and queue-based request throttling. Essential for capacity planning, stress-testing scraping performance, or maximizing data ingestion efficiency across various service plans.

This comprehensive guide details how to set up a robust, local development environment for Firecrawl integrations. It utilizes self-hosted Docker containers and Redis to eliminate reliance on paid API credits during development. The setup supports both unit testing (with mocked SDK) and full integration testing against the local instance, ensuring a seamless and efficient workflow for scraping and web data extraction projects.

A comprehensive guide for migrating web scraping pipelines from tools like Puppeteer, Playwright, and Cheerio to Firecrawl. This skill demonstrates how to replace complex, browser-intensive code with simple API calls, handling single-page scraping, structured data extraction using LLMs, and large-scale site crawling, while eliminating the need for manual browser management and anti-bot workarounds.

This guide details how to configure Firecrawl for reliable multi-environment operations (Development, Staging, Production). It addresses critical operational concerns, such as managing credit budgets and ensuring separation between environments. Users learn how to implement self-hosted Docker setups for development, manage environment-specific API keys, and integrate safe scraping wrappers into CI/CD pipelines.

Provides comprehensive observability patterns for Firecrawl web scraping operations. It covers key metrics such as scrape success rates, credit consumption tracking, content quality assessment, and latency monitoring. Use this skill for implementing robust monitoring dashboards, setting up proactive failure alerts, and optimizing the reliability of web crawling workflows.

This comprehensive guide details advanced techniques to significantly boost Firecrawl scraping speed and efficiency. Learn how to optimize performance by selecting the most efficient output formats (e.g., Markdown over HTML), implementing URL caching, utilizing batch scraping for multiple URLs, and tuning delay parameters for dynamic Single Page Applications (SPAs). Mastering these methods is crucial for building high-throughput, cost-effective web scraping pipelines.

This skill provides robust, automated guardrails for web scraping pipelines using Firecrawl. It enforces critical policies to ensure compliance and stability, including domain blocklists (preventing scraping prohibited sites), managing API credit budgets, filtering low-quality content (e.g., error pages, thin content), and implementing granular per-domain rate limiting. Use this when building highly reliable, legal, and cost-efficient scraping solutions.

A comprehensive checklist for securely deploying and validating applications using Firecrawl's core APIs (scrape, crawl, map, extract) in a production environment. It covers critical areas including secure credential management, implementing crawl safety limits, robust error handling (rate limits, authentication failures), establishing monitoring alerts, and executing necessary rollback procedures.