Model providers: where the model runs,
and who holds the key.
The AI Gateway sits between your agents and the model. The model can be one you serve on your own GPUs, one in your cloud account or a hosted one. Each row says where the model runs. The gateway does the same work for a call to any of them.
Keys
Your key is kept encrypted and decrypted only for the call. It is not stored or cached in plain text.
Model access
A policy for each tenant and an entitlement for each team decide which models may be called.
Rate limits
Requests and tokens per minute, for each user, tenant and model.
Budgets
Checked before the call, at three levels: warn, soft and hard. A hard budget blocks the call.
Guardrails
The input and output of a chat turn are checked against your guardrail profile.
Failover
Health is tracked for each provider and model, with a circuit breaker. A call can move to the next member of its tier.
Which of these keep prompts on our infrastructure?
The runtimes you host: vLLM, Ollama, NVIDIA NIM in your own cluster, NVIDIA Dynamo and any OpenAI-compatible server you run. A call to any other provider sends the prompt and the response to that provider.
Who holds the provider keys?
You do. A key is kept encrypted in your Katonic install and decrypted only for the call.
What happens in an air-gapped install?
Hosted providers start disabled.
Can an administrator add a provider that is not listed?
Yes. Any server that speaks the OpenAI chat completions API can be added by its address.