Download

Skill UI

Browse and discover 18950+ curated skills

All Development Artificial Intelligence Design & Creative Product & Business Data Science Marketing Soft Skills Productivity Engineering Languages

Search Web Crawling , found 17 results

Default Newest Most Downloaded

Website Scraping and Crawling Pipeline

firecrawl-core-workflow-a

jeremylongshore/claude-code-plugins-plus-skills

This workflow provides the primary mechanism for converting complex, live websites into clean, LLM-ready markdown content. It supports single-page scraping using `scrapeUrl`, deep multi-page site crawling with `crawlUrl`, and scalable async crawling for large datasets. Ideal for building robust content ingestion pipelines and reliably extracting structured information from the web.

Web Scraping, Crawling, and Data Extraction API

firecrawl-hello-world

jeremylongshore/claude-code-plugins-plus-skills

Firecrawl provides a comprehensive API for web data harvesting. It enables users to perform single-page scraping, multi-page crawling, site mapping (URL discovery), and advanced structured data extraction using Large Language Models (LLMs). Ideal for integrating data pipelines, building web monitoring tools, or conducting market research.

Firecrawl Anti-Patterns and Best Practices

firecrawl-known-pitfalls

jeremylongshore/claude-code-plugins-plus-skills

This comprehensive guide identifies common anti-patterns and integration mistakes when using Firecrawl for web scraping and content extraction. It serves as a vital code review checklist, helping developers avoid pitfalls such as unbounded crawling, unspecified output formats, or aggressive API polling, ensuring robust and efficient data pipelines.

Scaling and Load Testing Web Crawling

firecrawl-load-scale

jeremylongshore/claude-code-plugins-plus-skills

This skill provides advanced methods for load testing and scaling web scraping pipelines using Firecrawl. It manages rate limits and optimizes throughput by implementing concurrency control, batch processing, and queue-based request throttling. Essential for capacity planning, stress-testing scraping performance, or maximizing data ingestion efficiency across various service plans.

Firecrawl Migration Deep Dive

firecrawl-migration-deep-dive

jeremylongshore/claude-code-plugins-plus-skills

A comprehensive guide for migrating web scraping pipelines from tools like Puppeteer, Playwright, and Cheerio to Firecrawl. This skill demonstrates how to replace complex, browser-intensive code with simple API calls, handling single-page scraping, structured data extraction using LLMs, and large-scale site crawling, while eliminating the need for manual browser management and anti-bot workarounds.

Observability for Web Scraping Pipelines

firecrawl-observability

jeremylongshore/claude-code-plugins-plus-skills

Provides comprehensive observability patterns for Firecrawl web scraping operations. It covers key metrics such as scrape success rates, credit consumption tracking, content quality assessment, and latency monitoring. Use this skill for implementing robust monitoring dashboards, setting up proactive failure alerts, and optimizing the reliability of web crawling workflows.

Firecrawl Content Ingestion Pipeline Architecture

firecrawl-reference-architecture

jeremylongshore/claude-code-plugins-plus-skills

This comprehensive reference architecture details a multi-stage pipeline for web scraping and content ingestion using Firecrawl. It seamlessly integrates single-page scraping, site-wide crawling, URL mapping, and structured data extraction. The subsequent processing layer handles cleaning, deduplication, and chunking, making this blueprint ideal for designing robust knowledge bases, RAG systems, and complex AI data pipelines.

Firecrawl Webhook Event Handling Guide

firecrawl-webhooks-events

jeremylongshore/claude-code-plugins-plus-skills

Master the implementation of Firecrawl webhooks for robust, real-time handling of asynchronous crawling and batch scraping jobs. Instead of inefficiently polling the crawl status, configure webhook endpoints to receive instant POST events (e.g., `crawl.page`, `crawl.completed`). This guide covers event types, signature verification using HMAC-SHA256, and processing streamed data for building advanced data pipelines.

Web Search, Crawling, and Content Extraction

sickn33/antigravity-awesome-skills

This skill provides comprehensive capabilities for web searching, content extraction, and web crawling using the Tavily API. It is ideal for tasks requiring up-to-date information retrieval, deep content scraping from specific URLs, or comprehensive website crawling for research purposes, making it a powerful research augmentation tool.

Generative Engine Audit

zubair-trabzada/geo-seo-claude

Performs a full GEO audit of any website by discovering business type, crawling prioritized pages, and delegating parallel subagents to evaluate AI visibility, platform readiness, technical infrastructure, EEAT content, and schema markup, then outputs a composite GEO score with prioritized actions.

Dark Web Threat Monitoring and Intelligence

performing-dark-web-monitoring-for-threats

mukul975/Anthropic-Cybersecurity-Skills

This skill provides a comprehensive guide to systematic dark web monitoring, involving scanning underground forums, paste sites, and marketplaces for organizational threats. It covers setting up Tor-based crawling, automating credential leak detection (e.g., HIBP integration), and analyzing threat intelligence (CTI) using Python. Essential for proactive incident response and security auditing.

Apify Client: Running Actors and Scraping Data

apify-hello-world

jeremylongshore/claude-code-plugins-plus-skills

A comprehensive guide to using the Apify Client SDK. Learn the fundamental pattern of executing remote web scraping Actors (like content crawling or general scraping), managing asynchronous run states, and retrieving structured data from resulting datasets and key-value stores. Essential for integrating powerful web scraping capabilities into applications.

Language