Automate web crawling and data extraction with Firecrawl -- scrape pages, crawl sites, extract structured data, batch scrape URLs, and map website structures through the Composio Firecrawl integration.
SKILL.md
Firecrawl Automation
Run Firecrawl web crawling and extraction directly from Claude Code. Scrape individual pages, crawl entire sites, extract structured data with AI, batch process URL lists, and map website structures without leaving your terminal.
FIRECRAWL_CRAWL_GET -- get status, progress, credits used, and crawled page data
FIRECRAWL_CANCEL_A_CRAWL_JOB -- stop an active or queued crawl
Both require the crawl job id (UUID) returned when the crawl was initiated.
Example prompt: "Check the status of crawl job 019b0806-b7a1-7652-94c1-e865b5d2e89a"
Known Pitfalls
Rate limiting: Firecrawl can trigger "Rate limit exceeded" errors (429). Prefer FIRECRAWL_BATCH_SCRAPE over many individual FIRECRAWL_SCRAPE calls, and implement backoff on 429/5xx responses.
Credit consumption:FIRECRAWL_EXTRACT can fail with "Insufficient credits." Scope tightly and avoid broad homepage URLs that yield sparse fields. Test on small URL sets first.
Nested error responses: Per-page failures may be nested in response.data.code (e.g., SCRAPE_DNS_RESOLUTION_ERROR) even when the outer API call succeeds. Always validate inner status/error fields.
JS-heavy pages: Non-rendered fetches may miss key content. Use waitFor (e.g., 1000-5000ms) for dynamic pages, or configure scrapeOptions_actions to interact with the page before scraping.
Extraction schema precision: Vague or shifting schemas/prompts produce noisy, inconsistent output. Freeze your schema and test on a small sample before scaling to many URLs.
Crawl jobs are async:FIRECRAWL_CRAWL_V2 returns immediately with a job ID. Use FIRECRAWL_CRAWL_GET to poll for results. Cancel stuck crawls with FIRECRAWL_CANCEL_A_CRAWL_JOB to avoid wasting credits.
Extract job polling:FIRECRAWL_EXTRACT is also async for larger jobs. Retrieve final output with FIRECRAWL_EXTRACT_GET.
URL batching for extract: Keep extract URL batches small (~10 URLs) to avoid 429 rate limit errors.
Deeply nested responses: Results are often nested under data.data or deeper. Inspect the returned shape rather than assuming flat keys.
