Crawlee Python - Web Scraping Library Built for Real Use

TL;DR

TL;DR: Crawlee is an open-source web scraping library for Python (and JS) that handles anti-bot detection, proxy rotation, and headless browser automation so you focus on data extraction.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: crawlee.dev ← verified
• Source repository (Python): github.com/apify/crawlee-python ← verified, read README
• License: Apache-2.0 ← verified
• HN launch thread: news.ycombinator.com/item?id=40913736 ← verified
• Source last checked: 2026-06-22 (commit from README confirmation)

What Is Crawlee?

Crawlee is an open-source web scraping and browser automation library originally built for JavaScript/TypeScript by Apify, the web scraping platform. The Python port, crawlee-python, launched on Hacker News in June 2026 and quickly reached 254 points.

The project’s own description:

Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast. Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration.

Unlike writing raw requests + BeautifulSoup scripts, Crawlee handles the infrastructure: request queuing, retry logic, proxy rotation, session management, and headless browser control — all behind a clean Python API.

Key facts verified from source:

• Python 3.10+ required
• Available on PyPI as crawlee
• License: Apache-2.0
• JS version: ~23.9k GitHub stars (Apache-2.0)
• Built by Apify (commercial platform behind the open-source library)

Setup Workflow

Step 1: Install

Crawlee is distributed as a pip package. The recommended install uses uv for speed:

# Using uv (recommended)
uvx 'crawlee[cli]' create my-crawler

# Or via pip
pip install 'crawlee[all]'

The [all] extra pulls in every optional dependency. You can also install only what you need:

pip install 'crawlee[beautifulsoup]'   # HTML-only scraping
pip install 'crawlee[playwright]'     # Headless browser automation

Step 2: Install Playwright dependencies

For browser automation (rendering JavaScript pages):

playwright install

Step 3: Verify installation

python -c 'import crawlee; print(crawlee.__version__)'

Step 4: Write your first crawler

HTML-only (fast, no browser):

import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

async def main() -> None:
    crawler = BeautifulSoupCrawler(
        max_requests_per_crawl=10,
    )

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        await context.push_data({
            'url': context.request.url,
            'title': context.page.title,
        })
        # Follow links on the page
        await context.enqueue_links()

    await crawler.run(['https://crawlee.dev'])

Headless browser (renders JavaScript):

import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext

async def main() -> None:
    crawler = PlaywrightCrawler(
        max_requests_per_crawl=10,
    )

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        await context.push_data({
            'url': context.request.url,
            'title': await context.page.title(),
        })
        await context.enqueue_links()

    await crawler.run(['https://crawlee.dev/python/'])

Step 5: Run

cd my-crawler
python main.py

By default, Crawlee stores results in ./storage/datasets/default/ as JSON files.

Deeper Analysis

Architecture

Crawlee’s crawler classes wrap different HTTP clients and browser engines:

| Crawler | Engine | Use case | |---|---|---| | BeautifulSoupCrawler | httpx + BeautifulSoup | Static HTML, fastest | | PlaywrightCrawler | Playwright headless browser | JavaScript-rendered pages | | HttpCrawler | Raw httpx | Custom HTTP logic |

All crawlers share the same request queue, retry logic, and storage layer.

Bot Detection Bypass

The headline feature is “flies under the radar of modern bot protections.” This is achieved through:

• Realistic browser fingerprints — Playwright’s headless mode is configured to mimic real browser TCP/TLS fingerprints
• Proxy rotation — Built-in proxy configuration with session stickiness
• Automatic retry with backoff — Handles rate limiting and temporary blocks
• Human-like request timing — Configurable delay between requests

For production scraping at scale, you still need a working proxy pool — Crawlee handles the integration, not the proxies themselves.

Data Storage

Crawlee uses a local ./storage/ directory by default:

storage/
├── datasets/default/      # JSON lines of scraped data
├── request_queues/default/ # Seen URLs, deduplicated
└── key_value_stores/default/ # Raw responses

You can swap this for cloud storage when running on Apify’s platform.

CLI Tool

The crawlee CLI creates a project from a template:

crawlee create my-crawler
# Choose from: hello-world, beautifulsoup, playwright, http etc.

Practical Evaluation Checklist

• ✅ Handles JavaScript-rendered pages via Playwright
• ✅ Fast HTML scraping via BeautifulSoup + httpx
• ✅ Bot detection bypass with headless browser fingerprinting
• ✅ Proxy rotation support
• ✅ Request queuing with deduplication
• ✅ Local storage (JSON, CSV export)
• ✅ Apache-2.0 license
• ✅ Python 3.10+, works on macOS/Linux/Windows
• ✅ Built by Apify (active maintenance, commercial backing)
• ⚠️ Not a no-code tool — requires Python coding

Security Notes

• Crawlee itself has no authentication or access control — it is an outbound HTTP client
• If crawling third-party sites, you are responsible for respecting robots.txt and site Terms of Service
• Do not use Crawlee to scrape personal data without legal basis
• The library stores all scraped data locally by default; secure the ./storage/ directory accordingly

FAQ

Q: How is this different from Scrapy? A: Scrapy is a mature, battle-tested framework with a different architecture. Crawlee’s Python version is newer and prioritizes JavaScript rendering (via Playwright) as a first-class feature rather than an afterthought. If you need pure speed on static HTML, Scrapy may be lighter; if you need headless browser automation built in, Crawlee is more integrated.

Q: Does it work without a browser installed? A: BeautifulSoupCrawler works with no browser (HTTP only). PlaywrightCrawler requires Playwright and a browser binary — run playwright install to set this up.

Q: Can I run this on a server? A: Yes. It runs on any machine with Python 3.10+ and network access. For large-scale crawling, Apify’s cloud platform runs Crawlee workflows at scale with managed infrastructure.

Q: Is the Python version production-ready? A: The Python port launched June 2026 and is under active development by Apify. The JavaScript version has 23.9k stars and years of production use. The Python API mirrors the JS version’s architecture.

Conclusion

Crawlee Python is a well-architected scraping library that removes the boilerplate of building reliable crawlers. Whether you need fast HTML extraction or full headless browser automation, it provides a consistent Python API backed by Apify’s years of scraping infrastructure experience. The Apache-2.0 license makes it free to use in commercial projects.

If you are building a data pipeline that needs to extract content from dynamic websites, Crawlee handles the hard parts — bot detection, proxy rotation, and browser automation — so you can focus on the data model.

• Crawlee.dev — documentation and guides
• crawlee-python on GitHub — source and issue tracker
• PyPI package — pip install