AI gateway
How do you give departments access to LLMs with budgets and chargeback?
Put one AI gateway in front of every model you serve, and make it the only way in. Give each department its own access, a budget and a cap, meter every call against that department, and send a monthly statement that Group IT charges back.
The problem this solves
An organisation that buys GPU servers soon has scattered model endpoints, personal API keys and no view of which department uses which model at what cost. Group IT cannot allocate capacity, cap spend, apply guardrails or charge anything back.
The five steps
- Register every model you serve in one gateway. Close direct access to the model endpoints.
- Create access for each department, with its own keys or single sign-on group.
- Set a budget and a cap for each department. The budget is the plan. The cap is the hard stop.
- Meter every call as it happens: department, model, tokens in and tokens out.
- Produce a statement for each department every month, and charge the cost back.
A worked example
Take three departments on one GPU estate. The figures are illustrative.
| Department | Monthly budget | Cap | What happens at the cap |
|---|---|---|---|
| Customer service | 400 million tokens | 450 million | Calls are refused until the month ends or Group IT raises the cap |
| Legal | 60 million tokens | 60 million | Hard stop. Legal asked for no overrun |
| Engineering | 250 million tokens | 300 million | Group IT is alerted at 80 percent of budget |
At month end each department receives a statement of tokens by model. Group IT prices a token from the cost of the GPU estate and charges each department its share.
What the gateway must do
- Sit in front of every model, including models served by different tools.
- Attribute every call to a department and a user.
- Enforce budgets and caps itself, not report on them afterwards.
- Apply guardrails to what goes in and what comes out.
- Keep an audit log of every call.
- Run on your own infrastructure, with no outside service in the path.
Build it or use a platform
Open-source gateways such as LiteLLM give you routing, keys and spend tracking, and are a sound start for one team. The work grows when you add departments: single sign-on, caps that are enforced, guardrails, audit, statements and someone to run it all.
A platform puts those in one place. The Katonic AI Gateway registers the models you serve, gives each department access with a budget and a cap, applies guardrails and audit logging to every call, and meters usage per department for chargeback. It runs on your own GPU servers and can run air-gapped.
Read next
- Katonic AI Gateway →
- Use case: token factory for Group IT →
- Governance →
- The seven software layers above GPU servers →
Questions and answers
What is LLM chargeback?
Charging each department for the AI model usage it consumed, measured in tokens, so the cost of a shared GPU estate is recovered from the teams that use it.
What is the difference between a budget and a cap?
A budget is the planned amount for the period. A cap is the hard limit the gateway enforces. A department can be warned at its budget and stopped at its cap.
Can a department exceed its cap?
Not if the gateway enforces it. Group IT sets the cap, and calls past it are refused until the cap is raised or the period ends.
Does this replace personal API keys?
Yes. Access goes through the gateway, so every call is attributed, logged and subject to guardrails.
What is a token factory?
An internal service that turns GPU capacity into governed, metered access to AI models, so departments consume tokens against a budget instead of running their own endpoints.