OpenAI-compatible endpoint · Model provider
How do we put our own OpenAI-compatible model server behind a gateway with budgets, guardrails and logging?
Any model server that speaks the OpenAI chat completions API can be added to the Katonic AI Gateway by giving its address and, where it needs one, a key. Agents and applications then call it through the gateway, which applies your access rules, rate limits and budgets, checks each turn against your guardrail profile and records usage and cost. Where you host the server, the model, the prompts and the responses stay on your infrastructure.
Where the model runs.
On your infrastructure.
How it is connected
- 01The base address of the server. The default port is 8000.
- 02A key, sent as a bearer token, where the server needs one.
- 03The gateway reads the list of models and calls the chat completions path.
What leaves your environment
Nothing.The model, the prompts and the responses stay on infrastructure you own or control. Nothing is sent to a public AI service.
Keys
Your key is kept encrypted and decrypted only for the call. It is not stored or cached in plain text.
Model access
A policy for each tenant and an entitlement for each team decide which models may be called.
Rate limits
Requests and tokens per minute, for each user, tenant and model.
Budgets
Checked before the call, at three levels: warn, soft and hard. A hard budget blocks the call.
Guardrails
The input and output of a chat turn are checked against your guardrail profile.
Failover
Health is tracked for each provider and model, with a circuit breaker. A call can move to the next member of its tier.
Cost and usage
Each call is priced when it is recorded, and usage and cost can be queried.
Who paid
Each usage record says whose credential paid for the call.
Air gap
In an install flagged as air-gapped, hosted providers start disabled.
The gateway does the same work in front of each of these.
Which servers does this cover?
Any server that offers the OpenAI chat completions API at an address you give. vLLM, Ollama, NVIDIA NIM and NVIDIA Dynamo have their own provider types.
Do prompts leave our infrastructure?
Not where you host the server. The gateway sends the call to the address you give it.
Can we add a model we fine-tuned ourselves?
Yes, once it is served at an address the gateway can reach.