NeMo Relay
What is NVIDIA NeMo Relay?
NVIDIA NeMo Relay is an open-source library, under the Apache 2.0 licence, that sits inside an AI agent and observes and controls each model call and tool call. It records what happened, and it can block a tool call, redact personal data and track cost. It has a Rust runtime with Python, Node and Rust SDKs, and it attaches to agents you already have. It replaces the tracing, middleware and governance parts of the NeMo Agent Toolkit.
What Relay does
| Capability | What it means |
|---|---|
| Tracing | Every agent, model and tool step is recorded as an event with timing and inputs and outputs |
| Middleware | Your code can run before and after each call |
| Guardrails | A rule can stop a tool from running unless a condition is met |
| Redaction | Personal data can be removed from what is recorded |
| Cost tracking | Token use is turned into cost from a price catalogue you supply |
| Export | Runs can be written as event files, as agent trajectories, or sent to OpenTelemetry |
How it attaches to an agent
- For LangChain agents, you add a Relay middleware to the agent.
- For LangGraph graphs, one call configures the graph to report to Relay.
- You do not rewrite the agent. The agent logic stays in the framework you chose.
- NeMo Fabric adapters can turn Relay on for any agent they run, and NeMo Helix reads what Relay records.
What we found
We installed NeMo Relay 0.11.1 on its own and attached it to a LangChain tool-calling agent and a LangGraph graph. Tracing, a conditional guardrail on a tool, redaction of personal data and cost tracking all worked with no fixes. Of the three projects, Relay was the most mature in our test.
- The price catalogue needs one setting for how cached tokens are counted before cost tracking starts.
- When an agent run fails, Relay records the error in its event file, but the trajectory it exports does not mark the run as failed. Tools that read the trajectory, including NeMo Helix, then show the run as successful. We reported this to NVIDIA.
How it compares with tracing tools you may know
Tools such as Langfuse and LangSmith collect traces and show them in their own service. Relay is the part that sits in the agent: it produces the record and can also act on calls as they happen. It then sends the record wherever you choose, which can be NeMo Helix, an OpenTelemetry collector or a file.
Where Katonic fits
Katonic gives an organisation tracing, guardrails, redaction and cost control across all its agents, with budgets per team. Every agent built in Katonic is a declarative definition that compiles to NVIDIA's open agent software, and Relay is the part of that software that observes and controls a run.
Read next
- What is NVIDIA NeMo Fabric? →
- What is NVIDIA NeMo Helix? →
- NeMo Relay vs NeMo Fabric vs NeMo Helix →
- LLM access, budgets and chargeback →
- Issue 4091: failed runs stored as success →
- NeMo Relay on GitHub →
Questions and answers
What is NVIDIA NeMo Relay?
An open-source library that sits inside an AI agent and observes and controls each model and tool call, with tracing, guardrails, redaction and cost tracking.
Does NeMo Relay work with LangChain and LangGraph?
Yes. In our test it attached to a LangChain agent through a middleware and to a LangGraph graph through one configuration call, with no fixes.
Is NeMo Relay open source?
Yes. It is on GitHub under the Apache 2.0 licence.
What languages does NeMo Relay support?
It has a Rust runtime with SDKs for Python, Node and Rust.
Does NeMo Relay replace Langfuse or LangSmith?
Not directly. Relay produces the record inside the agent and can act on calls. It can send that record to other tools through OpenTelemetry.
Do I need NeMo Helix to use NeMo Relay?
No. Relay works on its own and can write to files or an OpenTelemetry collector.