Neuronpedia - Open Source AI Interpretability Platform

TL;DR

TL;DR: Neuronpedia is an open source platform for exploring, explaining, and scoring neural network features in models like Gemma 2B and Llama. It lets researchers and developers visualize what individual neurons and attention heads respond to, and ships with both a hosted service and a self-hosted option.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: neuronpedia.org — verified by direct fetch
• Source repository: github.com/hijohnnylin/neuronpedia — verified; MIT license, 928 stars as of June 2026
• License: MIT — verified via GitHub API license.spdx_id
• HN launch thread: news.ycombinator.com/item?id=43540427 — “Show HN: Neuronpedia, an open source platform for AI interpretability”
• Latest release: v1.0.796 — verified via GitHub API
• Source last checked: 2026-06-25 (commit main, pushed 2026-06-17)

What Is Neuronpedia?

Neuronpedia describes itself as an open source interpretability platform. The core idea: neural networks learn internal features — patterns in neurons and attention heads — that activate in response to specific concepts. Until recently, you needed heavy tooling to see what these features actually encode.

Neuronpedia provides a web interface and API to:

• Browse features across supported open models (Gemma 2B, Llama 3.3 70B, and more)
• Read natural language descriptions of what each feature responds to, generated by AI scorers
• Score features using automated scoring methods (including the EleutherAI sae-auto-interp scorer family)
• Run your own models via the self-hosted deployment
• Play the Interpretability Game — a GeoGuessr-like game that tests your ability to identify what features detect

The platform was co-developed with Anthropic as part of their Gemma Scope work, and is actively used by researchers at DeepMind, EleutherAI, and independent ML safety teams.

Supported Models and Resources

Neuronpedia hosts interpretability data for a growing set of open models. Publicly available resources include:

| Model | Type | Status | |---|---|---| | Gemma 2 2B | Sparse Autoencoders (SAEs) | Active | | Llama 3.3 70B | Features + Attention Heads | Active | | Llama 3.1 70B | Features + Attention Heads | Active | | Llama 3 8B | Features | Active |

Each resource page lists which layers have interpretability data, the number of features, and the scorer models available for evaluation.

Setup Options

Option 1: Use the Hosted Platform (Fastest)

No installation required. Visit app.neuronpedia.org and sign in with GitHub OAuth. You can immediately browse any public model resource, run scoring jobs, and use the Interpretability Game.

The hosted tier includes a free plan with limited API credits. Paid team plans add higher API rate limits and private projects.

Option 2: Self-Host with Docker

The Neuronpedia backend is open source and runs with Docker Compose. Requirements:

• Linux/macOS (no Windows native support)
• 8GB RAM minimum (16GB recommended)
• Docker and Docker Compose installed

# Clone the repository
git clone https://github.com/hijohnnylin/neuronpedia.git
cd neuronpedia

# Copy environment config
cp .env.example .env

# Edit .env with your API keys (OpenRouter recommended for scorer models)
# Then start the stack
docker-compose up -d

The full stack includes the Next.js frontend, API server, and a PostgreSQL database. First startup pulls pre-computed feature data for supported models.

Option 3: API-Only (Headless)

If you only want the API without the web UI:

git clone https://github.com/hijohnnylin/neuronpedia.git
cd neuronpedia/server
npm install
OPENROUTER_API_KEY=sk-or-v1-... npm run start:prod

The API gives you programmatic access to feature data, scoring jobs, and model management.

How Feature Scoring Works

Neuronpedia’s scoring pipeline uses a multi-step approach:

1. Activation extraction — For a given model layer, extract activation vectors from a large text corpus
2. Sparse Autoencoder (SAE) training — Train an SAE to decompose those activations into interpretable features
3. Automated explanation — Feed each feature’s top-activating contexts into a language model (via OpenRouter) with a structured prompt asking for a plain-English description
4. Verification scoring — Run a secondary scorer model to evaluate whether the explanation actually matches the feature’s behavior

This is the approach behind Anthropic’s Gemma Scope publication, adapted into a general-purpose platform.

Practical Evaluation Checklist

• [ ] Browse neuronpedia.org/gemma-scope for Gemma 2B feature maps
• [ ] Try the Interpretability Game — 3 rounds, no account needed
• [ ] Run a free scoring job via the API or web UI
• [ ] Read the GitHub README for self-hosting instructions
• [ ] Check available resources for the full list of supported models

Architecture Overview

Neuronpedia’s stack:

Frontend:     Next.js 14 (App Router, TypeScript)
API:          Node.js + Express
Database:     PostgreSQL
Scoring:      OpenRouter API (GPT-4o, Claude, DeepSeek-R1, etc.)
Deployment:   Vercel (frontend) + Railway/Render (API) or self-hosted Docker

The web app at neuronpedia.org is a Next.js application. The server/ directory contains the API and database layer. The packages/ directory holds shared TypeScript types.

Why This Matters for AI Safety

Mechanistic interpretability — understanding what neural networks actually compute internally — is one of the leading approaches to AI safety research. Projects like Neuraonpedia, TransformerLens, and the EleutherAI SAE pipeline are building the tooling to make this accessible to more researchers.

Gemma Scope (a collaboration between Neuronpedia and Anthropic) demonstrated that even small models like Gemma 2B have rich internal structure that can be meaningfully interpreted. The ability to browse this structure through a web interface lowers the barrier to entry significantly.

FAQ

Q: Does Neuronpedia work with closed models like GPT-4 or Claude? A: No — it only hosts interpretability data for open models (Gemma, Llama, etc.). The feature data must be computed from running the model, so only models you can run yourself can be analyzed.

Q: How is the feature data generated? Is it manual or automated? A: Primarily automated. Features are found by training Sparse Autoencoders on activation data, then explained using LLM-based prompting through the scorer pipeline. Human verification is valuable but not required for the data to be published.

Q: Can I contribute new model support? A: Yes. The project is MIT-licensed. You can run the feature extraction and SAE training pipeline on any model with a Transformers-compatible implementation, then submit the resulting data or contribute to the upstream repository.

Q: What is the Interpretability Game? A: A gamified quiz where you see top-activating contexts for a feature and must identify what concept the feature responds to. Inspired by GeoGuessr. Designed as a quick way to develop intuitions about how features map to concepts.

Q: Is there a Python library? A: Yes — hijohnnylin/neuronpedia-python on GitHub (7 stars) provides basic API bindings.

Conclusion

Neuronpedia fills a real gap in the AI developer tooling landscape: it makes mechanistic interpretability accessible without requiring you to set up a full SAE training pipeline from scratch. The combination of a polished web UI, an API, and a self-hosted option means you can integrate it into research workflows at whatever depth makes sense.

The platform is actively maintained (latest release v1.0.796, June 2026) with regular updates to supported models. If you are working on AI safety, model evaluation, or just want to understand what your models are actually computing, Neuronpedia is worth bookmarking.

Next steps:

• Browse app.neuronpedia.org — free account required for scoring jobs
• Read the GitHub README for self-hosting details
• Explore Gemma Scope — the most complete public resource on Neuronpedia