Skill UI

Automate Fireberry operations via Composio’s toolkit by discovering Rube MCP tools, managing connections, and executing schema-driven workflows while enforcing session, memory, and pagination practices.

This guide details setting up automated Continuous Integration/Continuous Deployment (CI/CD) pipelines for Firecrawl scraping services. It covers configuring GitHub Actions, managing API keys, and implementing both mock unit tests and live integration tests. Use this when ensuring scraping behavior is validated automatically across pull requests and merges.

This advanced workflow executes sophisticated web data gathering tasks. It supports three primary functions: LLM-powered structured data extraction using custom JSON schemas or prompts; efficient batch scraping of multiple known URLs; and rapid site mapping to discover all reachable URLs on a domain. Use this when you require reliable, typed data structures rather than raw markdown content.

A comprehensive guide for deploying web scraping applications powered by Firecrawl to various production environments. Covers Vercel serverless functions, Google Cloud Run, and self-hosted Docker setups. Learn how to configure platform-specific secrets, implement API routes, and handle asynchronous crawl results via webhooks for scalable, reliable production service.

This guide outlines how to implement robust Role-Based Access Control (RBAC) for Firecrawl scraping resources. It allows enterprises to manage multiple API keys, enforce per-team credit consumption limits, and restrict scraping to specific allowed domains (allowlisting). Use this when scaling scraping operations across various departments or services that require granular resource governance and budget tracking.

This comprehensive guide identifies common anti-patterns and integration mistakes when using Firecrawl for web scraping and content extraction. It serves as a vital code review checklist, helping developers avoid pitfalls such as unbounded crawling, unspecified output formats, or aggressive API polling, ensuring robust and efficient data pipelines.

This skill provides advanced methods for load testing and scaling web scraping pipelines using Firecrawl. It manages rate limits and optimizes throughput by implementing concurrency control, batch processing, and queue-based request throttling. Essential for capacity planning, stress-testing scraping performance, or maximizing data ingestion efficiency across various service plans.

This comprehensive guide details how to set up a robust, local development environment for Firecrawl integrations. It utilizes self-hosted Docker containers and Redis to eliminate reliance on paid API credits during development. The setup supports both unit testing (with mocked SDK) and full integration testing against the local instance, ensuring a seamless and efficient workflow for scraping and web data extraction projects.

This guide details how to configure Firecrawl for reliable multi-environment operations (Development, Staging, Production). It addresses critical operational concerns, such as managing credit budgets and ensuring separation between environments. Users learn how to implement self-hosted Docker setups for development, manage environment-specific API keys, and integrate safe scraping wrappers into CI/CD pipelines.

This comprehensive guide details advanced techniques to significantly boost Firecrawl scraping speed and efficiency. Learn how to optimize performance by selecting the most efficient output formats (e.g., Markdown over HTML), implementing URL caching, utilizing batch scraping for multiple URLs, and tuning delay parameters for dynamic Single Page Applications (SPAs). Mastering these methods is crucial for building high-throughput, cost-effective web scraping pipelines.

This skill provides robust, automated guardrails for web scraping pipelines using Firecrawl. It enforces critical policies to ensure compliance and stability, including domain blocklists (preventing scraping prohibited sites), managing API credit budgets, filtering low-quality content (e.g., error pages, thin content), and implementing granular per-domain rate limiting. Use this when building highly reliable, legal, and cost-efficient scraping solutions.

A comprehensive guide to handling external API rate limits (429 errors) when using Firecrawl. This skill demonstrates advanced techniques like exponential backoff with jitter, request queuing using p-queue, and proactive throttling methods. It ensures resilient and efficient API integration for large-scale scraping projects.