NVIDIA Agent Toolkit
How do you run NVIDIA Agent Toolkit agents in production with governance and cost control?
To run an agent built with the NVIDIA Agent Toolkit in production, put six things around it: models served on your own GPUs, one gateway with budgets for every model call, an evaluation gate before release, guardrails on inputs and outputs, human approval on actions that matter, and a trace of every run. The toolkit supplies the building blocks for several of these. A platform applies them to every agent in the same way.
Update, November 2026
NVIDIA has deprecated the NeMo Agent Toolkit library as of October 2026, with end of life at the end of 2026, and names NeMo Relay, NeMo Fabric and NeMo Helix as its successors. The NVIDIA Agent Toolkit announced in March 2026 continues. Every agent built in Katonic is a declarative definition that compiles to NVIDIA's open agent software. Today that software is the NeMo Agent Toolkit. When NVIDIA changes the runtime, Katonic changes the compiler and your agents carry over.
What the toolkit gives you, and what you add
| Need | What the toolkit provides | What production adds |
|---|---|---|
| Define the agent | Configuration for the agent, its tools and its model | A way for people who are not developers to produce that configuration |
| Models | Open models such as Nemotron | Serving on your own GPUs, shared across teams |
| Quality | An evaluation harness | A rule that nothing is released without passing |
| Safety | NeMo Guardrails and policy runtimes | The same rails on every agent, set by the organisation |
| Visibility | Tracing and a profiler | One audit trail across every team, kept inside |
| Cost | Usage data per run | Budgets, caps and chargeback per department |
The six steps
- Serve the models on your own GPUs. Point the agent at those endpoints in place of a public model API.
- Put a gateway in front of the models. Every call is attributed to a department and an agent, and counted against a budget.
- Build a test set for each agent. Run the evaluation on every new version and block release on a failing score.
- Apply guardrails to what goes in and what comes out, and to the tools the agent may call.
- Add human approval where an agent acts on something that matters: a payment, a decision, a message to a customer.
- Keep a trace of every run: the prompt, each tool call and each model output. Store it inside your boundary.
Cost control in practice
- Set a budget and a cap for each department. The gateway enforces the cap.
- Track cost per agent, not only per team, so an expensive agent is visible.
- Use the profiler to find the steps that take the most tokens and time.
- Use a smaller model for the steps that do not need a large one.
- Charge usage back each month, so teams see what their agents cost.
Governance in practice
- One sign-in and one set of roles for builders, reviewers and users.
- A version history for every agent, and a named owner.
- An evaluation result attached to each released version.
- Guardrail policy set once by the organisation, not agent by agent.
- An audit trail that shows who ran what, with which data, and what the agent did.
How Katonic does it
Katonic adds a declarative layer that compiles to NVIDIA's open agent software. A business user declares the knowledge, tools, guardrails and approval points of an agent, and the platform produces the configuration. There are five ways to build and one output, so every agent runs the same way.
- Existing agents built on LangGraph, CrewAI, Google ADK or plain Python are brought in as they are.
- Evaluation tests are added, so each version is scored before release.
- NVIDIA NeMo Guardrails and access policies are applied.
- Approval steps use the human-in-the-loop support in the toolkit.
- Every model call goes through the Katonic AI Gateway, with budgets and audit logging.
- The agent, its models and its traces stay on your own GPU servers.
Read next
- NVIDIA Agent Toolkit for enterprise AI →
- What is the NVIDIA Agent Toolkit? →
- Budgets and chargeback for departments →
- Use case: moving agent pilots to production →
- Evaluation →
- Guardrails →
- NVIDIA NeMo Agent Toolkit is deprecated: what replaces it? →
- NVIDIA migration guide for the NeMo Agent Toolkit →
Questions and answers
Is the NVIDIA Agent Toolkit production ready by itself?
It provides open-source building blocks. Production also needs model serving, a gateway with budgets, access control, an audit trail and a place for staff to use the agents.
How do I control what agents cost?
Send every model call through one gateway, set a budget and a cap per department, track cost per agent, and charge usage back monthly.
How do I stop a bad agent version reaching users?
Keep a test set for each agent and block release unless the evaluation passes.
Can I keep agents I already built in LangGraph or CrewAI?
Yes. The NeMo Agent Toolkit works with those frameworks, and Katonic runs such agents under the same governance and audit trail.
Does Katonic replace the NVIDIA Agent Toolkit?
No. Katonic is built on it. Every agent built in Katonic compiles to NVIDIA's open agent software.