LiteLLM, one gateway to a hundred model providers

LiteLLM, one gateway to a hundred model providers

LiteLLM is an intermediary layer that reduces the interfaces of many language model providers to a single shape, the one familiar from OpenAI. It comes in two forms: a Python library you wire straight into your code, and a proxy server that exposes a gateway with keys, limits and budgets accounted for per team.

The current stable version of the litellm package on PyPI is 1.97.0, released on 16 August 2026, while the main branch already declares 1.99.0 and development builds ship every few days. The BerriAI/litellm repository was created in July 2023, holds roughly 56.9 thousand stars, 10.7 thousand forks and close to 5 thousand open issues. It is not archived, and the last change dates from the day this text was written.

What LiteLLM actually does

Two forms of the same project solve two different problems, and confusing them is the most common source of misunderstanding on first contact.

The first form is the library. You call litellm.completion() with a model name prefixed by its provider, and the library translates your parameters into the target interface's format, sends the request, and normalises the response into the shape of an OpenAI library object. Switching providers comes down to changing a string in the model field. The package has a modest dependency set, including openai, httpx, tiktoken, pydantic, tokenizers and boto3, and requires Python from 3.10 up to and including 3.14.

The second form is the proxy, called the gateway in the documentation. It is a FastAPI server, on port 4000 by default, exposing the same paths as the OpenAI interface, meaning /chat/completions, /embeddings and the rest. Your applications use an ordinary OpenAI client, and you swap only api_base and the key. Behind the gateway sits a configuration with a model list, virtual keys issued to teams, request and token limits per minute, monetary budgets, and spend recorded into a PostgreSQL database.

What LiteLLM does not do matters too. It does not run models, so local work still needs Ollama or an inference server, with LiteLLM only placing a shared interface in front of them. It is not an observability platform, though it can ship traces to Langfuse or Helicone. Nor is it an agent framework or a prompt orchestration layer.

The licence, or why GitHub shows NOASSERTION

The GitHub programming interface reports the licence for this repository as NOASSERTION, named "Other". This is neither a detector fault nor an oversight by the authors. The repository genuinely carries a mixed licence, and that is information you want before deployment rather than after an audit.

The LICENSE file in the root directory opens with a clause splitting the code in two. Everything inside the enterprise/ directory falls under a separate licence defined within that directory, and everything outside it is available under the MIT licence, copyright Berri AI, 2023. Because the file's first sentence does not match the pattern of plain MIT, the automatic detector declines to identify it and returns NOASSERTION. A small inconsistency: the clause points at a file called enterprise/LICENSE, while the file is in fact named enterprise/LICENSE.md.

The content of that second licence is explicit. It is called the BerriAI Enterprise License, with copyright held by Berrie AI Inc. from 2024 onwards. Software from that directory may be used in production only if you have agreed to BerriAI's subscription terms and hold a valid licence for the correct number of user seats. Copying and modifying it for development and testing is permitted without a subscription, but the rights to any modifications and patches you produce remain with BerriAI, and using them likewise requires a licence. Copying, merging, publishing, distributing, sublicensing and selling are forbidden outright.

There is also a detail that escapes a cursory check. The licence field of the litellm package on PyPI reads MIT, and that is true of the library itself. But the pyproject.toml file on the main branch defines an extra dependency set named proxy, and inside it sit two of the project's own packages: litellm-proxy-extras at version 0.4.88, licensed MIT, and litellm-enterprise, whose licence field reads LicenseRef-Proprietary. The stable 1.97.0 release pins it at 0.1.54 and the main branch at 0.1.58, but the number is beside the point: in either case pip install "litellm[proxy]" pulls a commercially licensed package onto disk even when you have no intention of using any paid feature.

The divergence between the two sources is in any case typical of projects with an open core and a paid rim. Package metadata describes what sits inside the published archive, not the terms governing packages pulled in alongside it as dependencies. A tool harvesting licences from the registry field alone will therefore report plain MIT, while something else lands in the environment.

The practical conclusion for a team comes in three steps. The library alone is plain MIT and poses no problem. Installing the proxy adds a proprietary package to the environment whose features only unlock with a licence key, so its mere presence is not a violation, but a company licence scanner will catch it and you need an answer ready. If, on the other hand, you plan to use features from the enterprise directory in production, you need a contract. The exception is single sign-on, which per the documentation is free for up to five users.

The library: one call, many providers

Installation and first run look like this, with the first command giving you the purely MIT part and the second adding the commercial package described above.

# the library alone, MIT licence
pip install litellm

# the gateway together with the commercially licensed litellm-enterprise package
pip install "litellm[proxy]"

# starting the gateway on port 4000
litellm --config config.yaml

# the same with full diagnostic logs
litellm --config config.yaml --detailed_debug

# the container variant from the GHCR registry
docker run -p 4000:4000 \
  -v "$(pwd)/config.yaml:/app/config.yaml" \
  ghcr.io/berriai/litellm:main-stable --config /app/config.yaml

Calling a model in code comes down to one function. The model name consists of a provider prefix and an identifier, and the library reads keys from environment variables.

import os
import litellm
from litellm import completion, completion_cost

os.environ["ANTHROPIC_API_KEY"] = "..."
os.environ["OPENAI_API_KEY"] = "..."

litellm.drop_params = True

response = completion(
    model="anthropic/claude-sonnet-5",
    messages=[{"role": "user", "content": "Summarise this report in five points."}],
    max_tokens=1024,
)

print(response.choices[0].message.content)
print(response.usage.total_tokens)
print(completion_cost(completion_response=response))
print(response._hidden_params["response_cost"])

Setting litellm.drop_params deserves a sentence of its own, because it decides whether portability works at all. Providers differ in the set of parameters they accept, and sending one of them a field it does not know ends in an error. With the switch on, the library silently discards parameters the target does not support. The convenience is real, and so is the price: you switch to another provider, your parameter vanishes without a message, and responses start looking different from what you expected. For production work it is better to keep this switch off and handle the differences deliberately.

The completion_cost function computes the cost of a call in dollars from counted tokens and the model's price list, while cost_per_token breaks that down into input and output. The price list comes from a community maintained source available at api.litellm.ai. That is convenient and simultaneously the weakest link in the whole spend tracking story, because the list can diverge from the provider's actual rates, especially right after a pricing change or for a freshly released model. Treat LiteLLM's numbers as an estimate for comparing teams rather than as a basis for invoicing.

The proxy: virtual keys, budgets and limits

The gateway configuration fits into a single YAML file with four main sections: model_list, router_settings, litellm_settings and general_settings. Below is a layout covering the typical case, meaning two deployments of the same model to spread traffic, a backup model at another provider, and one local model.

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-eu
      api_base: https://my-endpoint-europe.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY_EU
      rpm: 600
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 600
  - model_name: claude
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: local
    litellm_params:
      model: ollama/llama3
      api_base: http://127.0.0.1:11434

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  allowed_fails: 3
  cooldown_time: 30
  fallbacks: [{ "gpt-4o": ["claude"] }]

litellm_settings:
  drop_params: false
  success_callback: ["langfuse"]

general_settings:
  master_key: sk-1234
  database_url: "postgresql://user:password@host:5432/litellm"

Separating model_name from litellm_params.model is the heart of the whole construction. The first field is the name the client sees, the second is the actual model passed into the call. Two entries sharing the externally visible name form a group across which the gateway spreads traffic. Keys are supplied with the os.environ/NAME notation, which tells it to read an environment variable rather than keep a secret in the file. The master key is better set through the LITELLM_MASTER_KEY variable than written into the configuration, as in the example above left inline for readability.

Virtual keys are issued through the /key/generate endpoint, authorised with the master key. Each key carries its own list of permitted models, a team assignment, a budget with a renewal period, and throughput limits.

curl -X POST 'http://0.0.0.0:4000/key/generate' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "models": ["gpt-4o", "claude"],
    "team_id": "search-team",
    "max_budget": 50,
    "budget_duration": "30d",
    "tpm_limit": 200000,
    "rpm_limit": 300,
    "metadata": {"application": "internal-search"}
  }'

# current spend for a key
curl 'http://0.0.0.0:4000/key/info?key=sk-XXXX' \
  -H 'Authorization: Bearer sk-1234'

Spend lands in the LiteLLM_VerificationTokenTable table, and if the key has a user or a team attached, additionally in LiteLLM_UserTable and LiteLLM_TeamTable. You read the state through the /key/info, /user/info and /team/info endpoints, and create a new user through /user/new. The amount grows after each accounted call, so a budget does not block the single request that crosses the threshold, only the ones that follow it.

Permission inheritance holds a few surprises here. A key created by an administrator without an explicit user_id field has no owner and inherits nothing, because automatic identifier stamping applies only to calls made outside the administrator role. In the other direction: a key owned by an administrator bypasses management route restrictions and can create further keys, users and teams. Only an allowed_routes field set directly on the key constrains it.

Routing, fallbacks and cost accounting

The same load balancing mechanism is available in the library through the Router class, without standing up a server. That is often a sensible middle point when you want resilience against a provider outage but do not yet need a shared gateway for several teams.

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "gpt-4o",
            "litellm_params": {"model": "azure/gpt-4o-eu", "api_key": "..."},
        },
        {
            "model_name": "gpt-4o",
            "litellm_params": {"model": "openai/gpt-4o", "api_key": "..."},
        },
        {
            "model_name": "claude",
            "litellm_params": {"model": "anthropic/claude-sonnet-5", "api_key": "..."},
        },
    ],
    fallbacks=[{"gpt-4o": ["claude"]}],
    routing_strategy="latency-based-routing",
    num_retries=2,
    cooldown_time=30,
)

response = await router.acompletion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Routing strategies include simple-shuffle, least-busy, latency-based-routing and cost-based-routing. Fallbacks operate at the level of group names rather than individual deployments, and walk the list in the order given. A separate default_fallbacks field covers the case where an entire group is misconfigured. The allowed_fails and cooldown_time parameters decide after how many errors a deployment drops out of the pool and for how long.

One behavioural change can catch you out when writing tests. The mock_testing_fallbacks, mock_testing_context_fallbacks and mock_testing_content_policy_fallbacks flags have been stripped from requests arriving at the gateway since version 1.85.0 and have no effect there. They work only on direct calls to the Router class. Verifying fallbacks on the proxy therefore requires triggering a genuine provider error in a non-production environment.

What Enterprise adds and what it costs

The split between the open and the paid version is documented in reasonable detail and runs along the boundary between operation and governance. The open version covers an OpenAI-compatible gateway, virtual keys, users and teams, spend tracking, budgets, limits, fallbacks, request and response logging, and Prometheus metrics.

The paid layer adds single sign-on together with SCIM, authentication with OIDC and JWT tokens, audit logs with retention policies, role-based access control with organisations and team administrators, IP-based access lists, key rotation, and integrations with secret managers, among them AWS KMS, AWS Secrets Manager, Azure Key Vault, Google KMS, Google Secret Manager, HashiCorp Vault and CyberArk. On the cost control side come tag-based budgets, budgets set separately for each model within one key, time-boxed limit increases, and alerts before a threshold is crossed. On the observability side: routing each team's logs to its own project, disabling logging at team level, and export to Google Cloud Storage or Azure Blob.

One item is listed separately and is easy to miss when planning safeguards. The guardrail framework in the open version supports custom implementations plus Presidio for masking personal data, while seven ready-made integrations require a licence: llmguard_moderations, llamaguard_moderations, hide_secrets, openai_moderations, google_text_moderation, lakera_prompt_injection and aporia_prompt_injection. If your project starts with the sentence "we need content moderation and prompt injection detection", you are on the paid side of the boundary.

There is no public price list. The plans page gives the billing mechanism instead of figures: an annual contract, priced by your annual gateway request capacity, deployment architecture and support scope, explicitly never per token. A thirty day trial key is available and issued without a sales call. The support agreement states response targets: one hour for a total outage, six hours for a partial one, a day for configuration matters, and seventy two hours for a vulnerability. The vendor claims SOC 2 Type 2 and ISO 27001, and an air-gapped deployment is possible. The names of two tiers, Standard and SCALE, appear in the answers to questions, with no amounts attached.

LiteLLM against the alternatives

The choice turns on the question of where the model keys should sit. If they have to stay inside your infrastructure and several teams need accounted access, LiteLLM is the natural candidate and the open version is enough to start. If you would rather not maintain another server with a PostgreSQL database, a gateway as a service takes the operational work off you at the cost of a margin and of trusting an intermediary. If, on the other hand, you use a single provider and plan no change, an intermediary layer adds latency and a failure point with no clear gain, and calling OpenAI or Claude directly is simpler.

Common mistakes

The first is treating the whole repository as MIT on the strength of a PyPI field. The enterprise directory carries its own commercial licence, and installing with the proxy extra pulls in the litellm-enterprise package described as proprietary. Record that in your dependency register before an audit does it for you.

The second is drop_params switched on globally and forgotten. A parameter unknown to the provider disappears without a message, so after switching models behaviour changes quietly and diagnosis starts down the wrong path.

The third is treating the cost computed by LiteLLM as billing data. The price list comes from a community maintained source and can be out of date, particularly for recently released models. Compare it against the provider's invoice periodically.

The fourth is administrative keys with no allowed_routes field set. A key whose owner is an administrator skips management route restrictions and can issue further keys and alter teams.

The fifth is relying on the mock_testing_* flags when testing fallbacks on the gateway. Since version 1.85.0 they are stripped from requests, and the test passes while checking nothing.

The sixth is an unpinned image or package version. Development releases ship every few days and the main-stable tag moves along with them. In production, state the version number, as the documentation itself does for the Helm chart.

The seventh is starting the gateway without a database and being surprised that keys do not work. The /key/generate endpoint requires a configured database_url, because every key, budget and spend record lives in PostgreSQL rather than in the configuration file.

FAQ

Is LiteLLM free for commercial use?

The library is, under MIT. So is the gateway, as long as you do not reach for features from the enterprise directory, whose production use requires a subscription agreement and a licence for a number of seats. Single sign-on is free for up to five users.

Why does GitHub show NOASSERTION for this repository?

Because the LICENSE file is not plain MIT text. It opens with a clause carving the enterprise directory out of MIT and assigning it a separate commercial licence. The automatic detector cannot express such a split and returns the value "Other".

Do I need the proxy, or is the library enough?

For a single application the library is enough, and the Router class handles resilience against a provider outage. The proxy makes sense when several teams need separate keys with budgets and limits, or when provider keys should sit in one place instead of in every repository separately.

How accurate is the cost tracking?

It is computed from tokens and a price list fetched from a community maintained source at api.litellm.ai. That suffices for comparisons and budgets, but with new models or a fresh pricing change it can drift away from the invoice, so put the two numbers side by side every so often.

Does data pass through LiteLLM servers?

Not with self-hosting, which is the default and only deployment variant. Traffic goes from your infrastructure straight to the providers. What does leave for the network is the request for the model price list.

What happens when a provider returns an error?

The gateway retries the request according to num_retries, and once allowed_fails is exceeded it removes the deployment from the pool for the duration of cooldown_time. If the whole group is unavailable, traffic moves to the group named in fallbacks, in the order given on the list.

Documentation lives on the project site, the list of paid features in the Enterprise section, and the text of the commercial licence in the enterprise/LICENSE.md file.