NVIDIA Agent Toolkit
Can you run the NVIDIA Agent Toolkit on-premises and air-gapped?
Yes. The NVIDIA Agent Toolkit is open source, and the NeMo Agent Toolkit library can run on your own servers with no outside connection. Two things have to change from a default setup: the models must be served on your own GPUs in place of a hosted endpoint, and every package, container and tool the agent uses must be available inside your boundary.
Update, November 2026
NVIDIA has deprecated the NeMo Agent Toolkit library as of October 2026, with end of life at the end of 2026, and names NeMo Relay, NeMo Fabric and NeMo Helix as its successors. The NVIDIA Agent Toolkit announced in March 2026 continues. Every agent built in Katonic is a declarative definition that compiles to NVIDIA's open agent software. Today that software is the NeMo Agent Toolkit. When NVIDIA changes the runtime, Katonic changes the compiler and your agents carry over.
What a default setup depends on
Many examples for the toolkit call models hosted by NVIDIA or another provider, and install packages from public registries. That is convenient for a first agent and does not work in a disconnected environment.
What has to change
| Dependency | Default | On-premises and air-gapped |
|---|---|---|
| Language model | A hosted endpoint | An open-weight model served on your own GPUs, for example with NVIDIA NIM |
| Embedding and reranking models | Often hosted | Served locally |
| Packages | Public package index | An internal mirror |
| Container images | Public registry | An internal registry |
| Tools the agent calls | May include web search and outside services | Only systems reachable inside the boundary |
| Traces and logs | May be sent to an outside service | Stored inside |
| Guardrails | May call a hosted service | NeMo Guardrails with a locally served model |
Steps
- Choose open-weight models whose licence allows local use, and serve them on your GPUs.
- Point the agent configuration at those model endpoints.
- Mirror the packages and container images you need, and pin their versions.
- Remove or replace any tool that needs the internet.
- Send traces and evaluation results to storage inside your boundary.
- Test with the network cable out: run a full agent task and check for any attempted outside connection.
What the library needs
The NeMo Agent Toolkit is a Python library. The library itself does not need a GPU. The models do. Size the GPUs for the models you serve and the number of people using them, not for the library.
Where Katonic fits
Katonic packages this for you. The platform runs on infrastructure you own and can run with no outside connection. Models are served on your own GPUs with NVIDIA NIM or NVIDIA Dynamo, every agent compiles to NVIDIA's open agent software, and the gateway, knowledge, guardrails, evaluation and audit all run inside the boundary.
Read next
- How to run AI agents fully on-premise or air-gapped →
- NVIDIA Agent Toolkit for enterprise AI →
- Running toolkit agents in production →
- Sovereign deployment with Katonic →
- NVIDIA NeMo Agent Toolkit is deprecated: what replaces it? →
- NVIDIA migration guide for the NeMo Agent Toolkit →
Questions and answers
Does the NVIDIA Agent Toolkit need an internet connection?
Not if you serve the models yourself and mirror the packages and images it needs. Default examples often call hosted models, which you replace.
Does the NeMo Agent Toolkit need a GPU?
The library does not. The models it calls do, when you serve them yourself.
Which models can I use on-premises?
Open-weight models whose licence allows local use, including the NVIDIA Nemotron family, served on your own GPUs.
Can a government agency run it in a classified environment?
The software can run disconnected. Whether it meets a given accreditation depends on the whole system and the rules of that agency.
Is there a platform that runs toolkit agents air-gapped?
Katonic AI runs on infrastructure you own, can run with no outside connection, and compiles every agent to NVIDIA's open agent software.