Firecrawl – Web Data API for AI Agents

TL;DR

TL;DR: Firecrawl is an open-source web scraping API that converts any URL into clean Markdown or structured JSON for AI agents — with search, crawl, map, and batch endpoints all in one library.

What Is Firecrawl?

Firecrawl is an open-source web data API built for AI applications. It scrapes websites and returns LLM-ready output — clean Markdown, structured JSON, screenshots, or HTML — without the usual headaches of proxy rotation, rate limiting, or JavaScriptblocked content.

The project has 139K GitHub stars and is actively maintained by Mendable, the same team behind the firecrawl-py and firecrawl npm packages.

Source and Accuracy Notes

• Project page: firecrawl.dev
• Source repository: github.com/mendableai/firecrawl
• License: AGPL-3.0 (verified via LICENSE file on main branch)
• HN launch thread: news.ycombinator.com/item?id=40055103
• Latest release: v2.11.0 (2026-06-19, verified via GitHub Releases API)
• Source last checked: 2026-06-26 (commit main branch, pushed 2026-06-26)

Setup

Install

Firecrawl is available as a Python package and npm package:

pip install firecrawl-py

npm install firecrawl

You will need an API key from firecrawl.dev — sign up for free at the project page. A hosted service is also available for teams who do not want to self-host.

Quick Start

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

# Search the web
search_result = app.search("firecrawl", limit=5)

# Scrape a single URL
result = app.scrape('firecrawl.dev')
print(result.markdown)

CLI

Install and initialize the CLI for agent integrations:

npx -y firecrawl-cli@latest init --all --browser

Restart your agent after installing. Works with Claude Code, Antigravity, and OpenCode.

Self-Hosting

Firecrawl can be self-hosted for teams with compliance or cost requirements. See the Self-Hosting Guide in the official docs.

Core Features

| Feature | Description | |---------|-------------| | Search | Search the web and get full page content from results | | Scrape | Convert any URL to markdown, HTML, screenshots, or structured JSON | | Crawl | Scrape all URLs of a website with a single request | | Map | Discover all URLs on a website instantly | | Interact | Click, scroll, write, wait, and press before extracting content | | Agent | Automated data gathering — describe what you need | | Batch Scrape | Scrape thousands of URLs asynchronously |

Output Formats

Firecrawl targets AI agent consumption:

• Markdown — stripped HTML, clean text ideal for LLM context windows
• Structured JSON — schema-matched data for programmatic use
• Screenshots — visual capture for vision-capable agents
• HTML — raw markup when needed
• Media parsing — PDFs, DOCX, and other web-hosted files

Practical Evaluation Checklist

• Covers 96 percent of the web including JavaScript-heavy SPAs
• P95 latency of 3.4 seconds across millions of pages (per official benchmarks)
• Handles rotating proxies, rate limits, and JS-blocked content automatically
• MCP client integration available (npx -y firecrawl-cli@latest init --all --browser)
• Available as hosted API (firecrawl.dev) or self-hosted deployment
• SDKs for Python and Node.js; REST API for other languages

Security Notes

• API key required for all hosted API calls — never commit keys to source control
• Self-hosting option available for teams with data residency requirements
• AGPL-3.0 license — source available for audit

FAQ

Q: Is there a free tier? A: Firecrawl offers a hosted service with tiered plans at firecrawl.dev. Self-hosting is also available.

Q: How does it compare to traditional scrapers like Scrapy or BeautifulSoup? A: Traditional scrapers require you to manage proxies, handle JavaScript rendering, and parse HTML yourself. Firecrawl handles all of that and returns LLM-ready output directly.

Q: Does it support JavaScript-rendered pages? A: Yes. Firecrawl handles JS-heavy pages natively — no separate headless browser setup required.

Q: Can I self-host Firecrawl? A: Yes. See the official self-hosting guide for deployment instructions.

Q: What programming languages are supported? A: Official SDKs for Python (firecrawl-py) and Node.js (firecrawl). A REST API covers any other language. A CLI tool is also available via npx.

Conclusion

Firecrawl solves the web data problem for AI agents end-to-end — from crawling to clean, LLM-ready output. With 139K stars, active development, and both cloud and self-hosted options, it is a solid choice for anyone building AIpowered data pipelines, research agents, or ingestion pipelines.