Openbenchmarks - Verified API Benchmarks for AI Agents

TL;DR

TL;DR: Openbenchmarks is a free, independent hub that scores B2B data and AI-agent APIs against verified ground truth, with open methodology, an OpenAPI 3.1 interface, and an MCP server for agent-native discovery.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: openbenchmarks.com — verified by reading the live site (2026-07-18)
• Source repository: github.com/openbenchmarks-labs — org verified via GitHub API
• License: MIT (verified via GitHub API for openbenchmarks-labs/lookalikes repo; company-enrichment shows no license field)
• HN launch thread: news.ycombinator.com/item?id=48875730
• Source last checked: 2026-07-18

What Is Openbenchmarks?

Openbenchmarks is an independent, open benchmark hub that scores B2B data and AI-agent tooling APIs against verified ground truth. Every input in each benchmark dataset has a correct answer the Openbenchmarks team verified themselves. No vendor pays for inclusion, ranking, or removal.

The site is aimed at AI agents making build vs buy decisions on third-party APIs — a real pain point when evaluating whether to integrate an external API or build functionality in-house.

Current live benchmarks cover:

• Lookalike — 24 seed companies × 4 vendors, evaluated on Precision at 100
• Company Enrichment — 300 company domains × 7 APIs, scored on correct field yield, accuracy, coverage, resolution, and latency
• Company Funding — 300 company domains × 7 APIs, evaluated on correct funding stage yield
• Text-to-Speech (by Coval) — 28 TTS models on Time to First Audio and Word Error Rate
• Speech-to-Text (by Coval) — 29 STT models on Time to Final Segment and Word Error Rate

Top results as of the current benchmark pages:

| Benchmark | Top Score | |---|---| | Lookalike | 57% Precision at 100 | | Company Enrichment | 86.7% correct field yield | | Company Funding | 77.2% correct stage yield | | Text-to-Speech | 207 ms median TTFA | | Speech-to-Text | 60 ms median TTFS |

How the Benchmarks Work

Openbenchmarks publishes the dataset, scoring rules, and exact billing for each provider tested. Results include not just top-line scores but also per-provider breakdowns covering how often a provider got the right answer, how often it got it wrong, and what each correct answer cost.

For Lookalike specifically: each vendor returns its top 100 lookalikes per seed company. An LLM judge then labels each result as relevant or not. The cell value is Precision at 100 — of the 100 results you paid for, how many are actually usable.

The benchmark hub is transparent about what it does not cover. The site explicitly notes it is unrelated to OpenBenchmarking.org (Phoronix hardware benchmarks) or OpenBenchmark AI.

Developer Interface

Openbenchmarks ships two access paths designed for agent-native integration:

OpenAPI 3.1 REST API — query benchmark results programmatically. The API is listed at openbenchmarks.com/api/benchmarks.

MCP server — MCP server with OAuth 2.1 and dynamic client registration, available at https://mcp.openbenchmarks.com/mcp. This lets AI agents discover and query benchmarks through the Model Context Protocol.

Agent-readable documentation is also available: llms.txt, llms-full.txt, and agents.md are published on the site for agents to ingest benchmark methodology without needing to parse HTML.

Practical Evaluation Checklist

When evaluating a B2B data API for agent use, Openbenchmarks suggests checking:

• Correct field yield — how often does the API return the right data when it exists?
• Accuracy when present — when the API does return a field, is it correct?
• Fill rate — what percentage of requests return usable data?
• Resolution — can the API uniquely identify the entity?
• Latency — what is the time cost per query?
• Cost per correct answer — some providers are cheap per call but return more errors, making them more expensive per usable result

The benchmark pages for Company Enrichment and Company Funding normalize all 7 tested APIs into the same seven-field scoring contract, making direct comparison straightforward.

Security Notes

Openbenchmarks states that real data goes in and only aggregate provider scores come out. Individual query data is not exposed in benchmark results. The MCP server uses OAuth 2.1 with dynamic client registration for authentication.

As with any third-party API integration, evaluate the data handling practices of the underlying providers being benchmarked — Openbenchmarks evaluates provider API performance, not provider security posture.

FAQ

Q: Is Openbenchmarks free to use? A: Yes. Browsing benchmarks and accessing the OpenAPI is free. The site is independent and does not charge for inclusion or ranking.

Q: How is Openbenchmarks different from OpenBenchmarking.org? A: Openbenchmarks is unrelated to OpenBenchmarking.org (Phoronix hardware and Linux benchmarks) or OpenBenchmark AI. It focuses specifically on B2B data and AI-agent tooling APIs.

Q: Can I submit a new benchmark? A: The site does not appear to have a self-service submission flow. The public GitHub org (github.com/openbenchmarks-labs) hosts benchmark code and data mirrors for the lookalikes, company-enrichment, and company-funding benchmarks.

Q: Does Openbenchmarks benchmark providers it has a relationship with? A: The site explicitly states no vendor pays for inclusion, ranking, or removal. Methodology is public and reproducible.

Conclusion

Openbenchmarks solves a real problem for developers and AI agents evaluating B2B APIs: vendor marketing makes build vs buy decisions hard, and real performance data is scattered across sales decks and outdated blog posts. By publishing verified ground truth, scoring methodology, and per-provider cost-per-correct-answer data, it gives agents the information needed to make data-driven integration decisions.

The MCP server and agent-readable docs (llms.txt, agents.md) make it particularly suited for AI agent workflows — an agent can query benchmark results directly without human intervention.

If you are building an AI agent that relies on third-party data APIs, Openbenchmarks is worth bookmarking. Benchmark coverage is currently focused on company lookalikes, enrichment, and funding, with TTS and STT benchmarks contributed by Coval.