LLM API Test – Benchmark LLMs Including Local Models

TL;DR

TL;DR: LLM API Test is a free, browser-based tool for benchmarking OpenAI, Claude, Gemini, and local LLMs side-by-side — no install needed, instant metrics for response time, throughput, and cost.

What Is LLM API Test?

LLM API Test is a web-based benchmark tool that measures performance across multiple LLM providers. You paste in your API keys, pick the models you want to compare, and it runs standardized prompts against each one, collecting response time, tokens-per-second, success rate, and cost per query.

The tool supports OpenAI (GPT-3.5, GPT-4), Google Gemini (Gemini Pro, Gemini Pro Vision), and any endpoint that speaks the OpenAI API protocol — which means local models via Ollama, LM Studio, or llama.cpp are all fair game.

It was shared on Hacker News in mid-2025 and has accumulated 42 GitHub stars.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: llmapitest.com
• Source repository: github.com/qjr87/llm-api-test
• License: MIT (verified via GitHub API license.spdx_id)
• HN launch thread: news.ycombinator.com/item?id=44413921
• Source last checked: 2026-07-16 (commit main, release 1.0.0)

Key Features

API Support (verified from README):

• OpenAI: GPT-3.5, GPT-4, and latest models
• Google Gemini: Gemini Pro, Gemini Pro Vision
• Custom endpoints: any OpenAI-compatible API

Metrics collected:

• First token latency (response time to first token)
• Output speed (tokens per second)
• Success rate (reliability tracking)
• Cost per query

Deployment:

• Browser-only (no install required for basic use)
• Local development: npm install && npm start (or python -m http.server 8000)
• Static hosting compatible

Setup Workflow

Step 1: Try the Live Version

No install needed. Head to llmapitest.com and add your API keys in the settings panel.

Step 2: Run a Benchmark

1. Select the models you want to compare from the dropdown.
2. Enter your API keys (stored locally in your browser — never sent to a server).
3. Choose a prompt or write a custom one.
4. Click Run Benchmark and watch the live metrics populate.

Step 3: Self-Host or Run Locally

git clone https://github.com/qjr87/llm-api-test.git
cd llm-api-test
npm install
npm start
# Open http://localhost:8000

Alternative for a quick local server:

python3 -m http.server 8000
# Then open http://localhost:8000

Comparing Local vs Cloud Models

The standout use case for LLM API Test is comparing a local model (running via Ollama) against GPT-4 or Claude. Set up a local endpoint at http://localhost:11434/v1 with your Ollama models, add it as a custom provider, and run the same prompt set against both.

What it tells you:

• Whether your local model is fast enough for your latency budget
• Cost savings of local vs API providers at your usage volume
• Which cloud model gives the best speed/quality tradeoff for your workload

Deeper Analysis

What LLM API Test measures well

The tool is straightforward — it sends the same prompt to multiple providers and logs the raw numbers. For latency-sensitive applications (streaming UIs, coding assistants, real-time agents), the response time and tokens-per-second metrics directly map to user experience.

The cost-per-query metric is an estimate based on publicly listed API pricing — useful for ballparking but not accounting for volume discounts or context window variability.

What it does not measure

LLM API Test does not evaluate output quality, reasoning accuracy, or benchmark scores. Two models can return answers at identical speeds but differ dramatically in correctness. Treat the throughput numbers as one input to your model selection decision, not the whole answer.

Privacy considerations

API keys are entered in the browser UI and stored in localStorage. They are not transmitted to any third-party server beyond the LLM provider APIs themselves. If you are testing proprietary or sensitive workloads, use the self-hosted version to keep everything on your own infrastructure.

Practical Evaluation Checklist

• [ ] Add at least one cloud model (OpenAI or Gemini) and one local endpoint
• [ ] Run the same prompt set across both and record response time delta
• [ ] Verify that local Ollama models respond at the expected tokens/second for your hardware
• [ ] Check the cost estimate tab if you are running high-volume workloads
• [ ] Self-host if you need to keep API keys entirely offline

FAQ

Q: Does it support streaming responses? A: The benchmark tool measures time-to-first-token and total throughput. Streaming is handled by the underlying API but the UI aggregates total response time, not per-token latency.

Q: Can I use this with a model running on my MacBook? A: Yes, as long as your local model exposes an OpenAI-compatible endpoint (Ollama, LM Studio, and llama.cpp all do this by default).

Q: Is my API key sent to the llmapitest.com server? A: No — keys are stored in your browser localStorage and sent directly to the LLM provider APIs. The llmapitest.com site itself does not proxy your requests.

Q: What is the license? A: MIT, confirmed via the GitHub repository.

Conclusion

LLM API Test fills a specific niche: developers who want a quick, no-install comparison of how different LLM backends perform on the same workload. The browser-based version is the fastest path in; the self-hosted option keeps keys off shared infrastructure. For anyone evaluating local models against cloud providers, it is a practical first step before committing to a setup.

Source repository: github.com/qjr87/llm-api-test