Firecrawl – Web Data API for AI Agents

TL;DR

TL;DR: Firecrawl is an open-source web scraping API that converts any URL into clean Markdown or structured JSON for AI agents — with search, crawl, map, and batch endpoints all in one library.

What Is Firecrawl?

Firecrawl is an open-source web data API built for AI applications. It scrapes websites and returns LLM-ready output — clean Markdown, structured JSON, screenshots, or HTML — without the usual headaches of proxy rotation, rate limiting, or JavaScriptblocked content.

The project has 139K GitHub stars and is actively maintained by Mendable, the same team behind the firecrawl-py and firecrawl npm packages.

Source and Accuracy Notes

• Project page: firecrawl.dev
• Source repository: github.com/mendableai/firecrawl
• License: AGPL-3.0 (verified via LICENSE file on main branch)
• HN launch thread: news.ycombinator.com/item?id=40055103
• Latest release: v2.11.0 (2026-06-19, verified via GitHub Releases API)

Setup

Install

Firecrawl is available as a Python package and npm package:

pip install firecrawl-py
```bash

```bash
npm install firecrawl
```bash

You will need an API key from [firecrawl.dev](https://firecrawl.dev) — sign up for free at the project page. A hosted service is also available for teams who do not want to self-host.

### Quick Start

```python
from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

# Search the web
search_result = app.search("firecrawl", limit=5)

# Scrape a single URL
result = app.scrape('firecrawl.dev')
print(result.markdown)
```bash

### CLI

Install and initialize the CLI for agent integrations:

```bash
npx -y firecrawl-cli@latest init --all --browser

Restart your agent after installing. Works with Claude Code, Antigravity, and OpenCode.

Self-Hosting

Firecrawl can be self-hosted for teams with compliance or cost requirements. See the Self-Hosting Guide in the official docs.

Core Features

Feature	Description
Search	Search the web and get full page content from results
Scrape	Convert any URL to markdown, HTML, screenshots, or structured JSON
Crawl	Scrape all URLs of a website with a single request
Map	Discover all URLs on a website instantly
Interact	Click, scroll, write, wait, and press before extracting content
Agent	Automated data gathering — describe what you need
Batch Scrape	Scrape thousands of URLs asynchronously

Output Formats

Firecrawl targets AI agent consumption:

• Markdown — stripped HTML, clean text ideal for LLM context windows
• Structured JSON — schema-matched data for programmatic use
• Screenshots — visual capture for vision-capable agents
• HTML — raw markup when needed
• Media parsing — PDFs, DOCX, and other web-hosted files

Practical Evaluation Checklist

• Covers 96 percent of the web including JavaScript-heavy SPAs
• P95 latency of 3.4 seconds across millions of pages (per official benchmarks)
• Handles rotating proxies, rate limits, and JS-blocked content automatically
• MCP client integration available (npx -y firecrawl-cli@latest init --all --browser)
• Available as hosted API (firecrawl.dev) or self-hosted deployment
• SDKs for Python and Node.js; REST API for other languages

Security Notes

• API key required for all hosted API calls — never commit keys to source control
• Self-hosting option available for teams with data residency requirements
• AGPL-3.0 license — source available for audit

FAQ

Q: Is there a free tier? A: Firecrawl offers a hosted service with tiered plans at firecrawl.dev. Self-hosting is also available.

Q: How does it compare to traditional scrapers like Scrapy or BeautifulSoup? A: Traditional scrapers require you to manage proxies, handle JavaScript rendering, and parse HTML yourself. Firecrawl handles all of that and returns LLM-ready output directly.

Q: Does it support JavaScript-rendered pages? A: Yes. Firecrawl handles JS-heavy pages natively — no separate headless browser setup required.

Q: Can I self-host Firecrawl? A: Yes. See the official self-hosting guide for deployment instructions.

Q: What programming languages are supported? A: Official SDKs for Python (firecrawl-py) and Node.js (firecrawl). A REST API covers any other language. A CLI tool is also available via npx.

Conclusion

Firecrawl solves the web data problem for AI agents end-to-end — from crawling to clean, LLM-ready output. With 139K stars, active development, and both cloud and self-hosted options, it is a solid choice for anyone building AIpowered data pipelines, research agents, or ingestion pipelines.