Download

Skill UI

Browse and discover 15857+ curated skills

All Development Artificial Intelligence Design & Creative Product & Business Data Science Marketing Soft Skills Productivity Engineering Languages

Search data-pipeline , found 400 results

Default Newest Most Downloaded

Generate AI Model Workflow Chains

sickn33/antigravity-awesome-skills

This skill provides guidance and patterns for generating structured JSON files that define complex workflow pipelines. It is essential for chaining multiple distinct AI models together, enabling multi-step, automated data processing. Use this when a single task requires a sequence of interconnected AI operations or services.

File Format Converter

file-format-converter

jeremylongshore/claude-code-plugins-plus-skills

Automates file format conversion guidance within data pipelines, covering ETL, workflow orchestration, data transformation, and streaming standards with production-ready examples.

Firecrawl Architecture Variants Guide

firecrawl-architecture-variants

jeremylongshore/claude-code-plugins-plus-skills

A comprehensive guide to selecting and implementing the optimal Firecrawl architecture pattern for various data ingestion needs. Learn to choose between On-Demand, Scheduled Crawl Pipelines, and Real-Time Ingestion Pipelines based on data volume, required latency, and complexity. This is essential for designing scalable, production-grade scraping infrastructure, including RAG and knowledge base applications.

Firecrawl Web Automation

firecrawl-automation

ComposioHQ/awesome-claude-skills

Firecrawl Automation lets you use Claude Code to scrape pages, crawl sites, batch URLs, extract structured data, and monitor jobs via the Composio integration, ideal for terminal-driven web data pipelines.

Website Scraping and Crawling Pipeline

firecrawl-core-workflow-a

jeremylongshore/claude-code-plugins-plus-skills

This workflow provides the primary mechanism for converting complex, live websites into clean, LLM-ready markdown content. It supports single-page scraping using `scrapeUrl`, deep multi-page site crawling with `crawlUrl`, and scalable async crawling for large datasets. Ideal for building robust content ingestion pipelines and reliably extracting structured information from the web.

Firecrawl Data Processing and Management

firecrawl-data-handling

jeremylongshore/claude-code-plugins-plus-skills

A comprehensive pipeline for processing, validating, and optimizing scraped web content from Firecrawl. This tool handles markdown cleaning, structured data extraction using Zod validation, content deduplication, and chunking tailored for Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems. It ensures crawled data is standardized, clean, and ready for downstream consumption or knowledge base indexing.

Web Scraping, Crawling, and Data Extraction API

firecrawl-hello-world

jeremylongshore/claude-code-plugins-plus-skills

Firecrawl provides a comprehensive API for web data harvesting. It enables users to perform single-page scraping, multi-page crawling, site mapping (URL discovery), and advanced structured data extraction using Large Language Models (LLMs). Ideal for integrating data pipelines, building web monitoring tools, or conducting market research.

Firecrawl Anti-Patterns and Best Practices

firecrawl-known-pitfalls

jeremylongshore/claude-code-plugins-plus-skills

This comprehensive guide identifies common anti-patterns and integration mistakes when using Firecrawl for web scraping and content extraction. It serves as a vital code review checklist, helping developers avoid pitfalls such as unbounded crawling, unspecified output formats, or aggressive API polling, ensuring robust and efficient data pipelines.

Scaling and Load Testing Web Crawling

firecrawl-load-scale

jeremylongshore/claude-code-plugins-plus-skills

This skill provides advanced methods for load testing and scaling web scraping pipelines using Firecrawl. It manages rate limits and optimizes throughput by implementing concurrency control, batch processing, and queue-based request throttling. Essential for capacity planning, stress-testing scraping performance, or maximizing data ingestion efficiency across various service plans.

Firecrawl Migration Deep Dive

firecrawl-migration-deep-dive

jeremylongshore/claude-code-plugins-plus-skills

A comprehensive guide for migrating web scraping pipelines from tools like Puppeteer, Playwright, and Cheerio to Firecrawl. This skill demonstrates how to replace complex, browser-intensive code with simple API calls, handling single-page scraping, structured data extraction using LLMs, and large-scale site crawling, while eliminating the need for manual browser management and anti-bot workarounds.

Firecrawl Content Ingestion Pipeline Architecture

firecrawl-reference-architecture

jeremylongshore/claude-code-plugins-plus-skills

This comprehensive reference architecture details a multi-stage pipeline for web scraping and content ingestion using Firecrawl. It seamlessly integrates single-page scraping, site-wide crawling, URL mapping, and structured data extraction. The subsequent processing layer handles cleaning, deduplication, and chunking, making this blueprint ideal for designing robust knowledge bases, RAG systems, and complex AI data pipelines.

Building Reliable Web Scraping Patterns

firecrawl-reliability-patterns

jeremylongshore/claude-code-plugins-plus-skills

A comprehensive guide to implementing production-grade reliability patterns for web scraping pipelines using Firecrawl. Learn techniques such as circuit breakers, crawl fallbacks, content quality validation, and robust backoff strategies to handle timeouts, API failures, and inconsistent data quality, ensuring maximum uptime and data integrity.

Prev 1 2 3 4 5 678 9 10...32 33 34 Next

Language