yardstiq - Compare AI Models Side-by-Side in Your Terminal

TL;DR

TL;DR: yardstiq is an open-source npm CLI that runs a single prompt against multiple AI models in parallel, streams the outputs side-by-side in your terminal, and reports real performance stats (time, tokens/s, cost) with an optional AI judge.

Source and Accuracy Notes

All links verified from primary sources on 2026-06-30.

• Project page: yardstiq.sh
• GitHub repository: stanleycyang/yardstiq (7 stars, MIT license, verified via GitHub API)
• npm package: npmjs.com/package/yardstiq
• HN launch thread: news.ycombinator.com/item?id=46890417

What Is yardstiq?

Choosing between AI models usually means copy-pasting the same prompt into three browser tabs and manually comparing the results. yardstiq automates that workflow entirely in the terminal.

From the README:

Compare AI models side-by-side in your terminal. One prompt, multiple models, real-time streaming, performance stats, and an AI judge — all in a single command.

It is a Node.js CLI published on npm, installed globally via npm install -g yardstiq. The package description from npm reads:

Compare AI models side-by-side in your terminal. Side-by-side streaming, performance stats, AI judge, and 40+ models.

Key features (from the README)

• Side-by-side streaming — model outputs appear in parallel in real time
• 40+ models — Claude, GPT, Gemini, Llama, DeepSeek, Mistral, Grok, and more
• Performance stats — time to first token, throughput, token counts, and cost per model
• AI judge — an AI evaluates which response wins with scored verdicts and reasoning
• Multiple export formats — JSON for pipelines, Markdown for docs, self-contained HTML for sharing
• Benchmark suites — define prompt suites in YAML and run them across models with aggregate scoring
• Local models — compare Ollama models with zero API cost
• Flexible auth — Vercel AI Gateway for one-key access, or individual provider API keys

Setup Workflow

Prerequisites

• Node.js (package states Node.js compatibility)
• API keys for the providers you want to use (or Vercel AI Gateway for unified access)

Step 1: Install

# Install globally via npm
npm install -g yardstiq

# Or run without installing
npx yardstiq "your prompt here" -m claude-sonnet -m gpt-4o

Step 2: Configure API keys

# Interactive setup walks you through provider configuration
yardstiq setup

# Or configure a single provider directly
yardstiq setup --provider gateway

# Environment variables also work
export AI_GATEWAY_API_KEY=your_gateway_key
export ANTHROPIC_API_KEY=sk-ant-...

Step 3: Run your first comparison

# Compare two models
yardstiq "Explain quicksort in 3 sentences" -m claude-sonnet -m gpt-4o

# Three models with AI judge
yardstiq "Write a binary search function" -m claude-sonnet -m gpt-4o -m gemini-flash --judge

# Export results to JSON
yardstiq "Explain DNS" -m claude-sonnet -m gpt-4o -m gemini-flash --json > results.json

The output renders as a terminal table with columns for each model:

┌──────────────────────────────────┬──────────────────────────────────┐
│ Claude Sonnet ✓                  │ GPT-4o ✓                         │
│ Quicksort is a divide-and-       │ Quicksort works by selecting a   │
│ conquer sorting algorithm...      │ "pivot" element and partitioning │
└──────────────────────────────────┴──────────────────────────────────┘

Model              Time     TTFT     Tokens     Tok/sec   Cost
Claude Sonnet      1.24s    432ms    18→86      69.4 t/s  $0.0013
GPT-4o             1.89s    612ms    18→91      48.1 t/s  $0.0010

Step 4: Run local models (optional)

If you have Ollama running locally, yardstiq can compare local models with no API cost:

yardstiq "hello" -m local:llama3.2 -m local:mistral

Step 5: Define benchmark suites

Create a benchmark.yaml file:

name: model-showdown
prompts:
  - "Write a Python fibonacci with memoization"
  - "Explain quantum entanglement to a 10-year-old"
  - "Debug this async race condition: ..."
models:
  - claude-sonnet
  - gpt-4o
  - gemini-flash
judge: true

Run it:

yardstiq benchmark run benchmark.yaml --json

Deeper Analysis

Why this fills a real gap

Model comparison is currently a manual, tab-switching workflow for most developers. Web-based model explorers exist (glama.ai, modelcomparison tools) but require leaving the terminal.yardstiq brings the comparison loop into the environment where developers already live, and it captures measurable outputs — time, tokens, cost — rather than relying on subjective perception.

Export flexibility

The three export formats serve different workflows:

• JSON for CI pipelines, automated evaluations, or feeding results into other tools
• Markdown for documentation and pull request comments
• Self-contained HTML for sharing formatted results with teammates who do not have the CLI installed

Benchmark suites for systematic evaluation

The YAML-based benchmark feature is the most advanced capability. Instead of one-off comparisons, you define a suite of prompts and run them all across a model set, getting aggregate scores. This is useful for evaluating a new model against your actual workload before committing to it in production.

Privacy model

From the README: “all comparisons run locally; nothing is stored externally.” API calls go directly to providers — yardstiq itself does not proxy or log the content of prompts or responses.

Practical Evaluation Checklist

• Does it install cleanly with npm install -g yardstiq?
• Does yardstiq setup --provider gateway work interactively?
• Do API keys set via environment variables work without interactive setup?
• Does --judge flag produce a scored verdict?
• Does --json output valid JSON to stdout?
• Do local Ollama models work with local: prefix?
• Does yardstiq benchmark run parse a YAML suite and produce aggregate results?
• Does the history feature save and retrieve past comparisons?

Security Notes

• API keys are stored locally via the config system (yardstiq setup) — do not commit credentials to git
• No external logging or telemetry is mentioned in the README
• For CI use, pass keys via environment variables rather than interactive setup

FAQ

Q: Does yardstiq support Azure OpenAI or other provider variants? A: The README and npm description list Claude, GPT, Gemini, Llama, DeepSeek, Mistral, and Grok. Azure compatibility is not documented — check the GitHub issues if you need it.

Q: How does the AI judge work? A: Passing --judge runs the responses through an AI model to score and explain which response wins. The specific judge model is not documented in the README.

Q: Can I compare models from different providers in one command? A: Yes. The -m flag accepts model identifiers from any supported provider, and they stream in parallel.

Q: Does it work on Windows? A: It is a Node.js CLI so it runs on any platform with Node.js installed. Terminal rendering should work in Windows Terminal and similar.

Conclusion

yardstiq is a focused tool that automates one of the most manual parts of the AI development workflow. The combination of side-by-side streaming, measurable performance data, and an optional AI judge gives developers a data-driven basis for choosing which model to use — without leaving the terminal or opening a browser.

Install it with npm install -g yardstiq and run your first comparison with npx yardstiq "your prompt" -m claude-sonnet -m gpt-4o.