Skill UI

This comprehensive guide details how to set up a robust, local development environment for Firecrawl integrations. It utilizes self-hosted Docker containers and Redis to eliminate reliance on paid API credits during development. The setup supports both unit testing (with mocked SDK) and full integration testing against the local instance, ensuring a seamless and efficient workflow for scraping and web data extraction projects.

A comprehensive guide for migrating web scraping pipelines from tools like Puppeteer, Playwright, and Cheerio to Firecrawl. This skill demonstrates how to replace complex, browser-intensive code with simple API calls, handling single-page scraping, structured data extraction using LLMs, and large-scale site crawling, while eliminating the need for manual browser management and anti-bot workarounds.

Provides comprehensive observability patterns for Firecrawl web scraping operations. It covers key metrics such as scrape success rates, credit consumption tracking, content quality assessment, and latency monitoring. Use this skill for implementing robust monitoring dashboards, setting up proactive failure alerts, and optimizing the reliability of web crawling workflows.

This comprehensive guide details advanced techniques to significantly boost Firecrawl scraping speed and efficiency. Learn how to optimize performance by selecting the most efficient output formats (e.g., Markdown over HTML), implementing URL caching, utilizing batch scraping for multiple URLs, and tuning delay parameters for dynamic Single Page Applications (SPAs). Mastering these methods is crucial for building high-throughput, cost-effective web scraping pipelines.

This skill provides robust, automated guardrails for web scraping pipelines using Firecrawl. It enforces critical policies to ensure compliance and stability, including domain blocklists (preventing scraping prohibited sites), managing API credit budgets, filtering low-quality content (e.g., error pages, thin content), and implementing granular per-domain rate limiting. Use this when building highly reliable, legal, and cost-efficient scraping solutions.

A comprehensive checklist for securely deploying and validating applications using Firecrawl's core APIs (scrape, crawl, map, extract) in a production environment. It covers critical areas including secure credential management, implementing crawl safety limits, robust error handling (rate limits, authentication failures), establishing monitoring alerts, and executing necessary rollback procedures.

A comprehensive guide to handling external API rate limits (429 errors) when using Firecrawl. This skill demonstrates advanced techniques like exponential backoff with jitter, request queuing using p-queue, and proactive throttling methods. It ensures resilient and efficient API integration for large-scale scraping projects.

This comprehensive reference architecture details a multi-stage pipeline for web scraping and content ingestion using Firecrawl. It seamlessly integrates single-page scraping, site-wide crawling, URL mapping, and structured data extraction. The subsequent processing layer handles cleaning, deduplication, and chunking, making this blueprint ideal for designing robust knowledge bases, RAG systems, and complex AI data pipelines.

A comprehensive guide to implementing production-grade reliability patterns for web scraping pipelines using Firecrawl. Learn techniques such as circuit breakers, crawl fallbacks, content quality validation, and robust backoff strategies to handle timeouts, API failures, and inconsistent data quality, ensuring maximum uptime and data integrity.

This skill provides deep web scraping capabilities, allowing users to extract complex content, take screenshots, and parse PDFs from web pages. It handles advanced interactions like clicking and scrolling, making it ideal for detailed data gathering, batch URL processing, and complex web content analysis.

Comprehensive guide for upgrading the Firecrawl SDK and migrating between major API versions (v0 to v1/v2). It details breaking changes across key functions like `crawlUrl`, `scrapeUrl`, and the new extract schema format. Use this when updating dependencies or transitioning from legacy API structures to the latest v2 standard.

Master the implementation of Firecrawl webhooks for robust, real-time handling of asynchronous crawling and batch scraping jobs. Instead of inefficiently polling the crawl status, configure webhook endpoints to receive instant POST events (e.g., `crawl.page`, `crawl.completed`). This guide covers event types, signature verification using HMAC-SHA256, and processing streamed data for building advanced data pipelines.