Libretto – Open-Source Browser Automation for AI Agents

TL;DR

TL;DR: Libretto is an open-source Node.js toolkit that gives AI coding agents a live browser and CLI to inspect pages, capture network traffic, record user actions, and replay them as deterministic Playwright scripts.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: libretto.sh — visited and verified
• Source repository: github.com/saffron-health/libretto — README read in full
• License: MIT — verified via LICENSE file
• HN launch thread: news.ycombinator.com/item?id=47780971 — Show HN, 134 points
• Source last checked: 2026-08-21

What Is Libretto?

Libretto is an open-source toolkit built by Saffron Health for building robust web integrations. It gives a coding agent — Claude Code, Codex, or any agent that can run CLI commands — a live browser with a token-efficient interface to:

• Inspect live pages with minimal context overhead
• Capture network traffic to reverse-engineer site APIs
• Record user actions and replay them as automation scripts
• Debug broken workflows interactively against the real site

The core insight: browser automation frameworks like Playwright are powerful but brittle when an AI agent drives them blindly. Libretto provides a structured CLI interface and skill layer so agents can observe, pause, inspect, and fix automation scripts in real time.

From the README:

“Libretto is a toolkit for building robust web integrations. It gives your coding agent a live browser and a token-efficient CLI to: inspect live pages, capture network traffic to reverse-engineer site APIs, record user actions and replay them as automation scripts, and debug broken workflows interactively against the real site.”

Setup Workflow

Step 1: Install

npm install libretto

Requires Node.js and npm.

Step 2: First-time setup

npx libretto setup

This creates the .libretto/ directory, installs agent skills, and downloads Chromium unless you pass --skip-browsers.

Step 3: Check readiness

npx libretto status

Core CLI Commands

npx libretto open <url>                    # launch browser and open a URL
npx libretto run ./integration.ts --headless  # run a workflow and close on success
npx libretto run ./integration.ts --headless --stay-open-on-success  # keep successful run inspectable
npx libretto snapshot --session <name>     # capture screenshot and compact accessibility tree
npx libretto exec "<code>"                 # execute Playwright TypeScript against the open page
npx libretto close                         # close the browser

Sessions run with --headless are inspectable through the same daemon-backed commands as open sessions. Failed or paused workflows keep the browser open so you can inspect the exact page state before fixing or resuming.

Use Cases

One-shot script generation

“Use the Libretto skill. Go on LinkedIn and scrape the first 10 posts for content, who posted it, the number of reactions, the first 25 comments, and the first 25 reposts.”

The agent opens a browser window for you to log in, then autonomously explores and generates the script.

Interactive script building

“I’m gonna show you a workflow in the eclinicalworks EHR to get a patient’s primary insurance ID. Use libretto skill to turn it into a Playwright script that takes patient name and dob as input.”

Libretto can read your actions in the browser, then use them to rebuild the workflow as a parameterized script.

Convert browser automation to direct API calls

“We have a browser script at ./integration.ts that automates going to Hacker News. Convert it to direct network scripts instead.”

Libretto reads network requests from the browser to reverse-engineer the underlying API, then generates a script that calls those requests directly — faster and more reliable than UI automation.

Fix broken integrations

“We have a browser script at ./integration.ts that is supposed to go to Availity and perform an eligibility check for a patient. But I’m getting a broken selector error when I run it. Fix it.”

Agents reproduce the failure, pause at the broken step, inspect the live page state, and fix the issue autonomously.

Deeper Analysis

Architecture

Libretto is a monorepo with the following layout:

• packages/libretto/src/cli/ — CLI commands
• packages/libretto/src/runtime/ — browser runtime (network, recovery, downloads)
• packages/libretto/src/shared/ — shared utilities (config, LLM client, logging, state)
• packages/libretto/skills/libretto/ — the agent skill definition

The project uses pnpm as its package manager.

Telemetry

Libretto records CLI telemetry (command name, install id, version, error boolean). It does not send command arguments, URLs, project paths, session cookies, API keys, error messages, or emails. To disable:

LIBRETTO_TELEMETRY_DISABLED=1
# or
DO_NOT_TRACK=1
# or run in CI
CI=1

Security Notes

• Telemetry does not capture URLs, cookies, API keys, or project paths.
• Browser sessions should be treated as having access to whatever the logged-in user can see — do not leave sessions open with elevated privileges.
• The libretto exec command runs arbitrary Playwright TypeScript against the open page — scope its use carefully.

FAQ

Q: Is Libretto production-ready? A: The README notes it is early-stage and APIs may change before version 1.0. Pin to specific versions in production.

Q: How is Libretto different from Playwright or Puppeteer? A: Playwright and Puppeteer are browser automation libraries. Libretto wraps them with a CLI and agent skill layer designed specifically for AI coding agents to use autonomously — with commands for inspection, action recording, network capture, and interactive debugging.

Q: Does Libretto work with any coding agent? A: Yes — any agent that can run CLI commands (Claude Code, Codex, etc.) can use the libretto CLI and skill. The skill is defined at packages/libretto/skills/libretto/.

Q: Can I convert a browser automation to direct API calls? A: Yes. Libretto captures network requests from the browser and can reverse-engineer the underlying API, generating a script that makes direct HTTP calls instead of automating the full browser UI — faster and more reliable.

Conclusion

Libretto addresses a real problem in AI-assisted development: coding agents are powerful but blindly run browser automations that break the moment a site changes. By giving the agent a live, inspectable browser with structured commands for capturing network traffic, recording actions, and pausing on failure, Libretto makes web automation deterministic and agent-debuggable.

If you are building healthcare integrations (Saffron Health’s original use case), scraping LinkedIn, or automating any browser-dependent workflow with an AI agent, Libretto is worth adding to your toolchain.