Context.dev – Web Scraping API for AI Agents

TL;DR

TL;DR: Context.dev is a YC S26 web scraping API that turns any URL into clean markdown, structured JSON via JSON schema, and brand intelligence (logos, colors, fonts) — with built-in site monitoring for AI agent workflows.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: context.dev — verified via direct fetch
• HN launch thread: news.ycombinator.com/item?id=35070488 — 119 points, 64 comments
• YC batch: S26 (Summer 2026)

What Is Context.dev?

Context.dev is a web scraping and data extraction API purpose-built for AI agents and LLM-powered workflows. Launched at YC S26, it lets you point any URL and get back clean markdown, raw HTML, sitemap, or structured JSON — without spinning up a headless browser or managing proxies.

The product sits between a raw scraper like Firecrawl and a brand intelligence tool. According to the Context.dev homepage, its core feature set covers three layers:

Scraping and crawling:

• Scrape any URL to clean markdown or HTML
• Full-site crawl with sitemap support
• Search across crawled content

Structured data extraction:

• Define a JSON schema and extract typed data from any page
• Used by Mintlify to auto-generate documentation from GitHub repo URLs

Brand intelligence:

• Pull logos, colors, fonts, and style guides from any domain
• Extract NAICS/SIC classification codes
• Company social profiles and address data

Monitoring:

• Watch any website for content changes and get webhook alerts

Setup Workflow

No self-hosted component — Context.dev is a hosted API. You sign up, get an API key, and call it from any language.

Step 1: Get an API Key

Sign up at context.dev to receive your API key. The free tier offers a set number of requests per month; paid plans scale by usage.

Step 2: Scrape a URL

curl https://api.context.dev/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url": "https://example.com", "format": "markdown"}'

Response:

{
  "url": "https://example.com",
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
  "metadata": {
    "title": "Example Domain",
    "description": "Example domain for documentation"
  }
}

Step 3: Extract Structured Data

Define a JSON schema to pull specific fields:

curl https://api.context.dev/extract \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://github.com/facebook/react",
    "schema": {
      "type": "object",
      "properties": {
        "stars": { "type": "string" },
        "description": { "type": "string" }
      }
    }
  }'

Step 4: Pull Brand Intelligence

curl https://api.context.dev/brand \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url": "https://stripe.com"}'

Returns logo URLs, primary/secondary colors, font families, and social links.

Step 5: Monitor a Site for Changes

curl https://api.context.dev/monitor \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url": "https://news.ycombinator.com", "webhook": "https://your-app.com/webhook"}'

Deeper Analysis

Why Context.dev over Firecrawl or Jina AI Reader?

The structured extraction layer is Context.dev’s differentiating feature. Rather than returning raw markdown and leaving schema mapping to the caller, you define a JSON schema upfront and get back typed fields — making it directly usable in agentic pipelines without post-processing.

The brand intelligence API (logos, colors, fonts) is also a separate product offering that would otherwise require a dedicated service like Clearbit or a Vision AI call.

Customer Stories

According to the homepage, three teams publicly switched to Context.dev:

• Mintlify — used it to transform any GitHub repo URL into a branded documentation site in under 10 minutes of integration time
• SiteGPT — migrated from Firecrawl in under a day to scrape entire websites for AI chatbot knowledge bases
• Sourcely — crawls academic journals and PDFs at scale with higher quality than alternatives they evaluated

Pricing

Context.dev uses a tiered billing model (monthly/yearly/usage-based). Specific tier pricing is not published on the homepage — you need to sign up or book a call to get numbers. The free tier includes a limited request quota.

Practical Evaluation Checklist

• [ ] GET /scrape returns clean markdown for a target URL within rate limits
• [ ] POST /extract with a custom JSON schema returns correctly typed fields
• [ ] GET /brand returns logo URLs and color palette for a known domain
• [ ] Webhook fires when Monitor detects a page change
• [ ] API key rotation works from the dashboard
• [ ] Free tier quota is sufficient for local development testing

Security Notes

• API keys are sent as Bearer tokens — always use HTTPS in production
• The brand endpoint returns publicly visible data (logos, color palettes) scraped from target domains
• Webhook URLs should be validated server-side to prevent open redirect abuse

FAQ

Q: Does Context.dev handle JavaScript-rendered pages? A: The primary scraping endpoint returns pre-rendered markdown. For SPAs that require JavaScript execution, check the docs for a headless browser option or use a companion tool like Playwright for that specific case.

Q: What rate limits apply on the free tier? A: Specific limits are shown in the dashboard after signup. Contact support for higher limits during evaluation.

Q: Can I use this for scraping competitor pricing or monitoring? A: You are responsible for ensuring your use complies with the target site’s Terms of Service and applicable laws. The monitoring webhook is a legitimate feature, but how you use the data is between you and the site’s policies.

Q: Does it support batch crawling multiple URLs? A: Full-site crawling via sitemap is supported. For bulk URL scraping, use the API in a loop with appropriate rate limiting.

Conclusion

Context.dev is a focused, YC-backed API that solves the “I need web data in my agent pipeline” problem without building your own scraper infrastructure. The structured extraction + brand intelligence combo covers use cases that would otherwise need two separate services. If your AI agents need fresh web context — whether for RAG grounding, brand enrichment, or competitive monitoring — it is worth a look.

• Project page: context.dev
• HN thread: news.ycombinator.com/item?id=35070488