LiteLLM: One OpenAI-shaped gateway for 100+ LLM APIs

TL;DR: LiteLLM is an AI gateway that puts more than 100 model providers behind a single API shape. It was rewritten around a Rust core with a Python SDK on top, and the repository shows real production scale at 60,691 stars and 12,192 forks.

What it is

LiteLLM sits between your application and whichever model providers you use. The project describes itself as an AI gateway with a Rust core and a Python SDK, and its job is to let you call Bedrock, Azure, OpenAI, Anthropic, VertexAI, vLLM, or Nvidia NIM through one interface rather than writing a separate integration for each.

That interface is the part that matters. The README positions the gateway as OpenAI-format by default, with native provider formats available when you need them. If your code already speaks the OpenAI dialect, swapping a model name is often the whole migration.

The repository is primarily tagged Python, which tells you where the SDK and the surrounding tooling live. The Rust core is the part underneath it.

What it does well

Four capabilities show up directly in the project’s own description, and they are the ones that separate a gateway from a thin proxy.

• Cost tracking. Per-request cost accounting across providers that price tokens differently.
• Guardrails. Policy checks applied at the gateway layer rather than inside each service.
• Load balancing. Spreading traffic when one provider throttles or degrades.
• Logging. A single place to observe calls that would otherwise be scattered across provider dashboards.

Those four are the reason teams adopt gateways at all. Raw routing is the easy part. The operational layer is where the engineering time goes, and LiteLLM packages it as a default rather than a build-it-yourself project.

from openai import OpenAI

client = OpenAI(
    base_url="https://your-gateway.example/v1",
    api_key="sk-...",
)

response = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Summarize this changelog"}],
)

Because the surface is OpenAI-compatible, the change to your client code is usually a base URL and a model identifier, not a rewrite.

Provider coverage

The provider list named in the project description covers both hosted frontier services and the self-hosted serving stacks people run themselves.

Category	Providers named by the project
Hosted frontier	OpenAI, Anthropic, Azure OpenAI
Cloud platform	Bedrock, VertexAI
Self-hosted	vLLM, Nvidia NIM
Format	OpenAI-compatible, or native per provider

Azure OpenAI showing up as a distinct entry alongside plain OpenAI is a useful detail. They are separate endpoints with separate auth, and having both behind one gateway is the sort of thing that gets tedious to maintain by hand.

The numbers behind it

LiteLLM is one of the larger repositories in the AI infrastructure space, and the raw counts tell a fairly clear story.

Metric	Value
Stars	60,691
Forks	12,192
Open issues	5,408
Primary language	Python
Created	July 2023
License	See repository

The issue count deserves context rather than alarm. At this scale it reflects active development and heavy usage, not abandonment. The fork count, sitting close to a fifth of the star count, suggests people deploy their own copy rather than only calling a hosted endpoint.

GitHub reports the license field as NOASSERTION for this repository. That is not the same as no license. It means the project ships a license file that GitHub cannot map to a standard SPDX identifier, so check the license file in the repository directly before you build on it commercially.

When to use it

You want LiteLLM when you are past the first provider. If your application talks to exactly one model API and you have no plans to change, a gateway is overhead you do not need.

It earns its place once you have three or more of these problems at once:

• Multiple providers in rotation, whether for failover, cost, or capability routing
• Per-team or per-feature cost attribution you need to defend on a bill
• A policy layer (PII filtering, prompt restrictions) you want in one place
• Centralized logging across provider dashboards

Because the gateway is the single choke point, observability and guardrails are enforced by default rather than by remembering to add them in each service.

When not to use it

There is a real operational cost. You are now running a piece of infrastructure that every request depends on, and its configuration surface is bigger than a direct SDK call.

If you have a single provider and a single team, that complexity buys you nothing. If your team would rather debug request flow in a provider’s own tooling than maintain a self-hosted gateway, skip it.

Documentation lives at docs.litellm.ai/docs, which is linked as the project’s homepage. Start there before committing to an operating model.

Source and Accuracy Notes

• Project page: https://github.com/BerriAI/litellm
• Source repository: BerriAI/litellm (verified via GitHub API html_url)
• Documentation: https://docs.litellm.ai/docs/ (reported as homepage in GitHub API)
• License: NOASSERTION (verified via GitHub API license.spdx_id), meaning a license file exists but is not mapped to a standard SPDX identifier
• Stars at time of writing: 60,691
• Provider list, capabilities, and architecture claims are taken from the GitHub API repository description, not from independent testing