Armature – Analytics and Evals for Your MCP

TL;DR

TL;DR: Armature wraps your MCP server to capture full agent sessions, surfacing popular use cases, frequent errors, and regression risks — with client-side redaction so sensitive data never leaves your infrastructure.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: armature.tech — verified product description and features
• HN launch thread: news.ycombinator.com/item?id=49157807 — founders’ own description of the product, YC P26 batch
• YC page: ycombinator.com/companies/armature — YC P26
• Source last checked: 2026-08-18

What Is Armature?

Armature is an analytics and evaluation platform for MCP servers in production. When you expose an MCP to AI agents like Claude or ChatGPT, you have almost no visibility into what users actually ask the agent to do through it — the entire conversation lives in the AI client’s context. Armature changes that by wrapping your MCP and reconstructing the full session, including what the user asked and how the agent responded.

The founders describe it this way (from the HN launch):

“We reconstruct the entire session behind the MCP tool calls you receive, including what the user asked their agent to do and what the agent thought.”

Armature ships SDKs in TypeScript, Python, and Go. Setup is described as three lines of code. Once instrumented, your dashboard shows:

• All sessions reconstructed — read the full conversation the user had inside Claude or ChatGPT
• Popular use case ranking — clusters sessions to surface which MCP workflows users rely on most
• Frequent issue detection — identifies where agents encounter errors or unexpected behavior

The platform also generates evals automatically from session data, letting you test whether changes to your MCP actually improve outcomes across harnesses and models before shipping.

Setup Workflow

Step 1: Install the SDK

Choose your language:

# TypeScript / Node.js
npm install @armature/sdk

# Python
pip install armature-sdk

# Go
go get github.com/armature/sdk-go

Step 2: Wrap Your MCP

From the product description and HN thread, wrapping your MCP takes approximately three lines of code. The exact API depends on your SDK — consult the Armature documentation for language-specific examples. The core idea is to pass your existing MCP handler through the Armature instrumentation layer.

Step 3: View Sessions in Dashboard

After wrapping your MCP, open the Armature dashboard at app.armature.tech. Sessions begin appearing once your MCP handles real traffic. The free tier includes up to 1,000 credits per month.

Deeper Analysis

Privacy Design

Armature applies client-side redaction before data reaches their servers. The founders note this was a deliberate constraint from working with sensitive data:

“Then privacy was an obvious constraint so we applied the same methods we had learned from working with banking data or building sensitive data scanning in logs. Today, redaction runs client-side before reaching our servers.”

This matters for MCPs that handle credentials, personal data, or internal APIs — you get analytics without streaming PII externally.

Performance Impact

The HN thread reports that instrumentation reached “the exact same success rate as without our instrumentation (89.17% vs 89.15% pass rate out of 870 runs)” — meaning the overhead is negligible for most use cases.

Eval Generation

Armature is working toward closing the loop: identify top workflows and issues from session data, recommend fixes, test those fixes at scale on the same workflows, and open PRs to ship directly. The eval generation from sessions is the first step in that chain.

Practical Evaluation Checklist

• MCP server is accessible over network (not localhost-only) for the instrumentation to reach Armature’s infrastructure
• Free tier limited to 1,000 credits per month; beyond that requires a paid plan
• SDK requires Node.js 18+, Python 3.9+, or Go 1.21+ depending on your language choice
• Client-side redaction is applied by default — no additional configuration needed to redact sensitive fields
• Privacy-sensitive deployments should review Armature’s data handling terms before enabling

Security Notes

• Redaction runs client-side before data leaves your infrastructure — sensitive field values never reach Armature’s servers
• The founders explicitly mention experience with banking data and log scanning, suggesting familiarity with compliance-sensitive contexts
• As with any observability SDK, consider threat model before instrumenting MCPs that handle highly sensitive operations (payment processing, auth credentials, etc.)

FAQ

Q: Does Armature work with any MCP server? A: Armature wraps MCPs implemented in TypeScript, Python, or Go using their SDKs. If your MCP is in another language, you would need to check whether Armature has added support or exposes a generic HTTP-based integration.

Q: How does the free tier work? A: Armature offers free up to 1,000 credits per month. The HN thread describes it as a “generous free tier.” Pricing beyond that depends on volume — check armature.tech/pricing for current rates.

Q: Can I self-host Armature? A: The current product is cloud-only (app.armature.tech). Self-hosted deployment is not mentioned in the launch thread or product site as of August 2026.

Q: How does Armature avoid degrading MCP performance? A: The founders report iterating until they reached “the exact same success rate as without our instrumentation (89.17% vs 89.15% pass rate out of 870 runs)” — indicating the overhead is within measurement noise for most scenarios.

Conclusion

Armature solves a real gap in the MCP ecosystem: visibility into how agents actually use your server once it is deployed. By reconstructing sessions and surfacing popular workflows and frequent errors, it gives MCP operators the feedback loop they otherwise lack. The client-side redaction design is a thoughtful default for teams that cannot afford to pipe user conversations to a third party. The free tier makes it accessible for small projects, and the eval generation roadmap points toward a more automated improvement cycle. If you run an MCP in production and want to know whether agents and their users are getting what they need from it, Armature is worth evaluating.

URL: https://runany.dev/blog/armature-mcp-analytics/