vLLM · Model provider
How do we put vLLM behind a gateway with budgets, guardrails and logging?
You serve a model with vLLM on your own GPUs and add it to the Katonic AI Gateway by giving its address. Agents and applications then call the model through the gateway, which applies your access rules, rate limits and budgets, checks each turn against your guardrail profile and records usage and cost. The model, the prompts and the responses stay on your infrastructure.
Where the model runs.
On your infrastructure.
How it is connected
- 01The base address of your vLLM server. The default port is 8000.
- 02No key is needed by default.
- 03The gateway reads the list of models from the server and calls its chat completions path.
What leaves your environment
Nothing.The model, the prompts and the responses stay on infrastructure you own or control. Nothing is sent to a public AI service.
Keys
Your key is kept encrypted and decrypted only for the call. It is not stored or cached in plain text.
Model access
A policy for each tenant and an entitlement for each team decide which models may be called.
Rate limits
Requests and tokens per minute, for each user, tenant and model.
Budgets
Checked before the call, at three levels: warn, soft and hard. A hard budget blocks the call.
Guardrails
The input and output of a chat turn are checked against your guardrail profile.
Failover
Health is tracked for each provider and model, with a circuit breaker. A call can move to the next member of its tier.
Cost and usage
Each call is priced when it is recorded, and usage and cost can be queried.
Who paid
Each usage record says whose credential paid for the call.
Air gap
In an install flagged as air-gapped, hosted providers start disabled.
The gateway does the same work in front of each of these.
Do prompts leave our infrastructure?
No. vLLM runs on your GPUs and the gateway runs in your cluster. Nothing is sent to a public AI service.
Can different teams have different budgets on the same model?
Yes. Budgets and rate limits are set for each user, team and tenant, and checked before the call.
Does the gateway change how vLLM serves the model?
No. You run vLLM as you do today. The gateway sits in front of it.