Agent Skills Eval: Test Your AI Skills Pipeline

TL;DR

TL;DR: Agent Skills Eval is a test runner for AI agent skills — define test cases in YAML, run them against any OpenAI-compatible model, and get structured JSONL results. Bring software engineering rigor to your AI agent skill development.

Source and Accuracy Notes

Based on the official darkrishabh/agent-skills-eval repository, MIT licensed. Test format and CLI behavior sourced from the repo’s source code and examples as of June 2026.

What Is Agent Skills Eval?

As the agent-skills ecosystem grows — with platforms like agentskills.io, Claude Code skills, and Codex skills — one problem becomes obvious: how do you know a skill actually works? Agent Skills Eval answers that question with a proper test runner for skill definitions.

It takes a skill defined in YAML (following the agentskills.io format) and runs it through a battery of test cases against any OpenAI-compatible LLM. The output is structured JSONL — machine-readable, diffable, and suitable for CI pipelines.

Why Testing Skills Matters

Skills without tests are prompt engineering guesses. You change one line in a skill definition and hope it still works. Agent Skills Eval gives you regression testing: define expected behaviors, run them automatically, and catch regressions before they reach users.

Repo-Specific Setup Workflow

Prerequisites

• Node.js 20+
• npm or yarn
• An OpenAI-compatible API key

Step 1: Install

npm install -g agent-skills-eval

Step 2: Create a Test Spec

Create a YAML file defining your test cases:

skill: path/to/skill.yaml
model: gpt-4o
tests:
  - name: "greeting response"
    input: "Hello"
    expected:
      contains: ["hello", "hi"]
      not_contains: ["error"]
  - name: "code generation"
    input: "Write a factorial function in Python"
    expected:
      contains: ["def factorial"]
      language: "python"

Step 3: Run Tests

agent-skills-eval run --spec tests.yaml --output results.jsonl

# With a different model
agent-skills-eval run --spec tests.yaml --model claude-sonnet-4-20250514 --provider anthropic

# Watch mode for development
agent-skills-eval run --spec tests.yaml --watch

Step 4: Review Results

# Summary statistics
agent-skills-eval summary results.jsonl

# Diff two runs to see what changed
agent-skills-eval diff results-v1.jsonl results-v2.jsonl

Deeper Analysis

Test Assertion Types

The test runner supports several assertion types beyond simple string matching:

• contains/not_contains: String presence checks
• language: Verify code blocks use the correct language
• schema: Validate structured output against a JSON schema
• tool_calls: Verify the agent invoked specific tools with expected parameters
• latency: Assert response time thresholds

These assertions map directly to what matters in skill behavior — correctness, format compliance, tool usage, and performance.

CI Integration

The JSONL output format makes CI integration straightforward:

# GitHub Actions example
- name: Test agent skills
  run: |
    agent-skills-eval run --spec skills/*.test.yaml --output results.jsonl
    agent-skills-eval summary results.jsonl --fail-on-regression

The --fail-on-regression flag compares against a baseline and exits non-zero if any skill regresses — perfect for PR checks.

Model-Agnostic Design

The runner supports any OpenAI-compatible endpoint, including Anthropic (via compatible proxy), local models via Ollama, or any custom provider. This means you can test skills against the same model you’ll use in production.

Practical Evaluation Checklist

• YAML-based test definitions — readable and version-controllable
• Multiple assertion types: string matching, schema validation, tool calls, latency
• JSONL output for CI pipelines and regression detection
• Model-agnostic: works with any OpenAI-compatible endpoint
• Watch mode for rapid skill development
• MIT licensed with no usage restrictions

Security Notes

• Test inputs and outputs are stored locally
• API keys handled via environment variables or config files
• No external telemetry or data collection
• Be mindful of sensitive test data in YAML files — use environment variable substitution where needed

The Growing Need for Skill Testing

As the agent skills ecosystem matures, skill quality will become a competitive differentiator. Skills with comprehensive test suites will be trusted and adopted; skills without tests will be treated as experimental. Agent Skills Eval is infrastructure for this transition — the equivalent of test runners like Jest or pytest, but for the emerging category of AI agent skills.

Consider the parallel with npm packages in the early 2010s. Initially, anyone could publish anything. Over time, the ecosystem developed testing standards, CI integration, and quality signals. Agent skills are at a similar inflection point, and Agent Skills Eval is one of the first purpose-built tools for this new category.

The JSONL output format also allows integration with data analysis tools. Pipe results through jq for quick filtering, load them into pandas for statistical analysis, or visualize pass/fail trends over time. This programmability makes Agent Skills Eval suitable not just for one-off testing but for ongoing quality monitoring of your skill portfolio.

FAQ

Q: What’s agentskills.io format? A: It’s an emerging standard for defining AI agent skills as YAML files with instructions, tool definitions, and metadata. Agent Skills Eval validates and tests skills in this format.

Q: Can I test skills that call external APIs? A: Yes, but be aware that test runs will make real API calls. Use mock endpoints or test-specific credentials for CI environments.

Q: How does this compare to LLM evaluation frameworks like Braintrust? A: Those are general-purpose LLM eval frameworks. Agent Skills Eval is purpose-built for agent skill testing — it understands skill structure, tool calls, and agent-specific behaviors that general eval tools don’t.

Q: Is there a hosted version? A: The project is CLI-only and self-hosted. Your test data never leaves your infrastructure.

undefined

Agent Skills Eval brings software engineering discipline to the wild west of AI agent skill development. As skills become the standard way to extend coding agents, the ability to test them rigorously — with regression detection, CI integration, and structured assertions — moves from nice-to-have to essential. For anyone building, sharing, or maintaining agent skills, this test runner should be in your toolkit.

Q: Can I use this to benchmark different models on the same skill? A: Yes. Run the same test suite against different models by changing the provider and model flags. Compare the JSONL output files to see which model produces better results. Useful for selecting the most cost-effective model for production deployment.

Conclusion

Agent Skills Eval brings software engineering discipline to AI agent skill development. As skills become the standard way to extend coding agents, the ability to test them rigorously — with regression detection, CI integration, and structured assertions — is essential. For anyone building or maintaining agent skills, this test runner belongs in your toolkit.