What is LiteLLM? The gateway for many LLMs

Have AI infrastructure set up cleanly? WZ-IT builds and operates the open-source LLM stack on your own infrastructure - including gateway, inference and observability, GDPR-compliant from one team. See LLM hosting

As soon as more than one application uses a language model, the question of a central point of contact arises. LiteLLM is exactly that: a gateway that makes many models accessible through one interface and controls access. This article explains what LiteLLM does, why a gateway makes sense in the enterprise and how it sits in the open-source LLM stack. As of October 2026.

Table of contents

• The problem: many models, many APIs
• What LiteLLM does
• The LiteLLM Proxy (gateway)
• Why a gateway in the enterprise
• LiteLLM in the stack
• Security and operations

The problem: many models, many APIs

In practice it rarely stays with one model. A self-hosted LLM for sensitive data, a specialized model for code, perhaps an external service for exceptional cases - and each has its own interface, authentication and billing. If these details are built into every application, a patchwork emerges: every model change means changes in many places.

A gateway solves this by placing itself between applications and models and offering one unified interface to the front.

What LiteLLM does

LiteLLM provides a single, OpenAI-compatible interface across more than 100 language models. An application sends its request in the standard OpenAI format to LiteLLM, and LiteLLM translates it to the respective provider - with a consistent response format, regardless of which model actually answers.

The practical effect: applications only need to know the OpenAI format. Whether a self-hosted vLLM, a local Ollama or an external service works behind it is transparent to the application. The model becomes replaceable.

The LiteLLM Proxy (gateway)

Beyond pure translation, the LiteLLM proxy brings the functions that make up production operations (LiteLLM documentation):

• Authentication and authorization - a central API gateway with access control.
• Cost tracking and budgets - track spend and set budgets per project.
• Rate limits - cap usage per user.
• Retry and fallback - automatic failover across multiple deployments.
• Admin dashboard - a surface for monitoring and management.

LiteLLM is open source (by BerriAI) and can be fully self-hosted via Docker - the gateway thus runs on your own infrastructure.

Why a gateway in the enterprise

The value of a gateway shows as soon as several teams or applications use the same model infrastructure. Instead of solving access, cost and failure handling anew in every application, they sit in one place:

• Governance - who may use which model, with which limit.
• Cost control - transparent budgets and tracking instead of opaque usage.
• Decoupling - swap or extend models in the background without touching applications.
• Availability - fallback increases robustness without every application building it itself.

That is the difference between "a model runs" and "AI is operated in the enterprise".

LiteLLM in the stack

In the open-source LLM stack, LiteLLM sits as the gateway layer between the applications and the inference. Below it, an engine like vLLM serves the models; beside it, an observability layer like Langfuse records the requests. LiteLLM is thus the central control point that turns one or several running models into a controllable, auditable service - the prerequisite for a sovereign, production AI operation.

Security and operations

A gateway bundles not only access but also the API keys of every connected model. It should be operated accordingly. As of October 2026, these reports are among the relevant ones (LiteLLM security advisories):

• GHSA-7hp6-4w63-5g45 (critical, 30 September 2026): privilege escalation to proxy admin via cross-domain reuse of the salt key; fixed in 1.100.4, 1.101.3, 1.102.2 and 1.103.1, among others (advisory).
• GHSA-g5ff-637f-6q2m (high, 1 October 2026): a user with the internal_user_viewer role could read arbitrary local files via vertex_ai_credentials; fixed from 1.95.0 (advisory).
• Supply chain: on 24 March 2026, PyPI versions 1.82.7 and 1.82.8 were compromised and harvested credentials such as environment variables, SSH and cloud keys (Datadog Security Labs).

Four operating rules follow from this:

1. Pin versions. Use fixed package or image versions instead of latest and apply updates deliberately.
2. Track advisories. Check new advisories before each update and close critical flaws promptly; the release line moves fast (v1.105.0 as of October 2026).
3. Keep it internal. Make the proxy and admin UI reachable only on the internal network or via VPN, not publicly.
4. Separate keys. Keep the master key and salt key secret and separate, and grant user roles minimally.

The article LiteLLM vulnerability: hardening the gateway covers the September 2026 flaw in detail.