NVIDIA NIM · Model provider
How do we put NVIDIA NIM behind a gateway with budgets, guardrails and logging?
You run NVIDIA NIM microservices in your own cluster and add them to the Katonic AI Gateway by giving their address. Agents and applications call the models through the gateway, which applies your access rules, rate limits and budgets, checks each turn against your guardrail profile and records usage and cost. The model, the prompts and the responses stay on your infrastructure.
Where the model runs.
On your infrastructure.
How it is connected
- 01The base address of the NIM service in your cluster. The default port is 8000.
- 02No key is needed by default.
- 03The gateway reads the list of models from the service and calls its chat completions path.
What leaves your environment
Nothing.The model, the prompts and the responses stay on infrastructure you own or control. Nothing is sent to a public AI service.
Keys
Your key is kept encrypted and decrypted only for the call. It is not stored or cached in plain text.
Model access
A policy for each tenant and an entitlement for each team decide which models may be called.
Rate limits
Requests and tokens per minute, for each user, tenant and model.
Budgets
Checked before the call, at three levels: warn, soft and hard. A hard budget blocks the call.
Guardrails
The input and output of a chat turn are checked against your guardrail profile.
Failover
Health is tracked for each provider and model, with a circuit breaker. A call can move to the next member of its tier.
Cost and usage
Each call is priced when it is recorded, and usage and cost can be queried.
Who paid
Each usage record says whose credential paid for the call.
Air gap
In an install flagged as air-gapped, hosted providers start disabled.
The gateway does the same work in front of each of these.
Do prompts leave our infrastructure?
No, where NIM runs in your own cluster. The gateway also has a separate provider for models NVIDIA hosts; calls to that one go to NVIDIA.
Is this the same as NVIDIA's hosted models?
No. This page is about NIM you run yourself. The hosted service is a separate provider in the gateway, and it needs an API key.
What does Katonic add to NIM?
Who may call which model, rate limits, budgets, guardrail checks, failover and a record of usage and cost.