prompttools - Open-Source LLM and Vector DB Evaluation

TL;DR

TL;DR: prompttools is an open-source Python library from Hegel AI for evaluating LLMs and vector databases through code or a local Streamlit playground, with support for OpenAI, Anthropic, HuggingFace, Gemini, and more.

Source and Accuracy Notes

⚠️ This section is MANDATORY. All links must be verified from actual source, not guessed.

• Project page: hegel-ai.com (creator company)
• Source repository: github.com/hegelai/prompttools
• Live playground: prompttools.streamlit.app (Streamlit Community Cloud)
• Documentation: prompttools.readthedocs.io
• License: Apache-2.0 (verified via GitHub API license.spdx_id)
• HN launch thread: not applicable (organic community project, not a Show HN)

What Is prompttools?

prompttools is an open-source Python library created by Hegel AI for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. The core design philosophy is to let developers evaluate using familiar interfaces — Python code, Jupyter notebooks, and a local web playground — without requiring a hosted SaaS product.

The library ships with a Streamlit-based playground you can run locally with a single command.

Supported Integrations

LLMs

• OpenAI (Completion, ChatCompletion, Fine-tuned models)
• LLaMA.Cpp (LLaMA 1, LLaMA 2)
• HuggingFace (Hub API, Inference Endpoints)
• Anthropic
• Mistral AI
• Google Gemini
• Google PaLM (legacy)
• Google Vertex AI
• Azure OpenAI Service
• Replicate
• Ollama (in progress)

Vector Databases

• Chroma
• Weaviate
• Qdrant
• LanceDB
• Pinecone
• Milvus (exploratory)
• Epsilla (in progress)

Setup Workflow

Step 1: Install

pip install prompttools

Step 2: Run an experiment from Python

from prompttools.experiment import OpenAIChatExperiment

messages = [
    [{"role": "user", "content": "Tell me a joke."}],
    [{"role": "user", "content": "Is 17077 a prime number?"}],
]

models = ["gpt-3.5-turbo", "gpt-4"]
temperatures = [0.0]
openai_experiment = OpenAIChatExperiment(models, messages, temperature=temperatures)
openai_experiment.run()
openai_experiment.visualize()

Step 3: Launch the playground (optional)

pip install notebook
git clone https://github.com/hegelai/prompttools.git
cd prompttools && streamlit run prompttools/playground/playground.py

The playground provides a web UI for comparing model outputs side-by-side, adjusting parameters, and visualizing results without writing code.

Key Features

Side-by-side model comparison

Pass multiple models to a single experiment and compare outputs with visualize(). This is useful for prompt A/B testing across providers.

Vector retrieval evaluation

Test your RAG pipeline by comparing retrieval accuracy across Chroma, Weaviate, Qdrant, LanceDB, and Pinecone from the same interface.

Notebook-friendly

All Experiment classes are designed to run inside Jupyter or Colab notebooks. An example notebook is available for Google Colab.

Self-hosted, no API key required for local models

The library supports LLaMA.Cpp locally, meaning you can run evaluation entirely offline without routing traffic through OpenAI or Anthropic APIs.

Practical Evaluation Checklist

• Supports self-hosting with no usage limits
• Works with OpenAI, Anthropic, Google Gemini, Mistral, and local LLaMA.Cpp
• Vector DB comparison across 5+ backends (Chroma, Weaviate, Qdrant, LanceDB, Pinecone)
• Streamlit playground for non-code workflows
• Jupyter/Colab notebook integration
• Apache-2.0 license

FAQ

Q: Do I need API keys to use prompttools? A: It depends on the model. OpenAI, Anthropic, and other hosted providers require their respective API keys. LLaMA.Cpp runs locally without any API key.

Q: Can I evaluate prompts against local models? A: Yes. LLaMA.Cpp is supported, and Ollama integration is in progress.

Q: How does it compare to products like Braintrust or PromptLayer? A: prompttools is fully open-source and self-hostable. It focuses on local experimentation and comparison rather than hosted evaluation dashboards.

Q: What Python version is required? A: Python 3.9+ is recommended. Check pyproject.toml or setup.py in the repo for exact requirements.

Conclusion

prompttools fills a gap between purely manual prompt testing and full-featured SaaS evaluation platforms. By keeping evaluation local-first and notebook-native, it appeals to developers who want to iterate quickly without committing to a hosted vendor. With support for eight LLM providers and five vector databases, it is one of the broader open-source evaluation toolkits available today.

If you want to compare model outputs, stress-test retrieval pipelines, or just debug why your RAG chain is failing — and do it without sending data to a third party — prompttools is worth a look.