Skill UI

This comprehensive guide identifies common anti-patterns and integration mistakes when using Firecrawl for web scraping and content extraction. It serves as a vital code review checklist, helping developers avoid pitfalls such as unbounded crawling, unspecified output formats, or aggressive API polling, ensuring robust and efficient data pipelines.

This comprehensive guide details how to set up a robust, local development environment for Firecrawl integrations. It utilizes self-hosted Docker containers and Redis to eliminate reliance on paid API credits during development. The setup supports both unit testing (with mocked SDK) and full integration testing against the local instance, ensuring a seamless and efficient workflow for scraping and web data extraction projects.

A comprehensive guide for migrating web scraping pipelines from tools like Puppeteer, Playwright, and Cheerio to Firecrawl. This skill demonstrates how to replace complex, browser-intensive code with simple API calls, handling single-page scraping, structured data extraction using LLMs, and large-scale site crawling, while eliminating the need for manual browser management and anti-bot workarounds.

This guide details how to configure Firecrawl for reliable multi-environment operations (Development, Staging, Production). It addresses critical operational concerns, such as managing credit budgets and ensuring separation between environments. Users learn how to implement self-hosted Docker setups for development, manage environment-specific API keys, and integrate safe scraping wrappers into CI/CD pipelines.

This comprehensive guide details advanced techniques to significantly boost Firecrawl scraping speed and efficiency. Learn how to optimize performance by selecting the most efficient output formats (e.g., Markdown over HTML), implementing URL caching, utilizing batch scraping for multiple URLs, and tuning delay parameters for dynamic Single Page Applications (SPAs). Mastering these methods is crucial for building high-throughput, cost-effective web scraping pipelines.

A comprehensive guide to handling external API rate limits (429 errors) when using Firecrawl. This skill demonstrates advanced techniques like exponential backoff with jitter, request queuing using p-queue, and proactive throttling methods. It ensures resilient and efficient API integration for large-scale scraping projects.

A comprehensive guide to implementing production-grade reliability patterns for web scraping pipelines using Firecrawl. Learn techniques such as circuit breakers, crawl fallbacks, content quality validation, and robust backoff strategies to handle timeouts, API failures, and inconsistent data quality, ensuring maximum uptime and data integrity.

This guide provides production-ready patterns for integrating the Firecrawl SDK in TypeScript and Python. It covers essential architectural best practices, such as implementing singleton clients, typed service wrappers, automatic retries with exponential backoff, concurrency management, and robust response validation using Zod. Ideal for building scalable, reliable web scraping services.

This guide outlines essential security best practices for integrating Firecrawl. It covers secure API key storage (using environment variables and .gitignore), verifying webhook signatures with HMAC-SHA256 for data integrity, implementing key rotation strategies across different environments (dev/staging/prod), and sanitizing scraped content to mitigate XSS and PII risks. Essential for building secure, production-grade systems.

Comprehensive guide for upgrading the Firecrawl SDK and migrating between major API versions (v0 to v1/v2). It details breaking changes across key functions like `crawlUrl`, `scrapeUrl`, and the new extract schema format. Use this when updating dependencies or transitioning from legacy API structures to the latest v2 standard.

Master the implementation of Firecrawl webhooks for robust, real-time handling of asynchronous crawling and batch scraping jobs. Instead of inefficiently polling the crawl status, configure webhook endpoints to receive instant POST events (e.g., `crawl.page`, `crawl.completed`). This guide covers event types, signature verification using HMAC-SHA256, and processing streamed data for building advanced data pipelines.

This guide provides comprehensive setup for implementing CI/CD pipelines for Fireflies.ai integrations. It covers best practices for validating GraphQL queries, running mock-based unit tests (using Vitest), and executing live integration tests against the actual API. It integrates seamlessly with GitHub Actions, ensuring that code changes are automatically validated in a stable, automated testing environment before deployment. Essential for maintaining API stability and robustness.