NeMo Agent Toolkit
How to evaluate agents built with the NVIDIA NeMo Agent Toolkit
The NVIDIA NeMo Agent Toolkit includes an evaluation harness. You write a test set of questions with expected answers, name it in the agent configuration with the evaluators you want, and run the evaluation command. The toolkit runs the agent on every item, scores the answers and the steps the agent took, and writes the results to files. The work that decides quality is the test set and the pass mark, not the command.
Update, November 2026
NVIDIA has deprecated the NeMo Agent Toolkit library as of October 2026, with end of life at the end of 2026, and names NeMo Relay, NeMo Fabric and NeMo Helix as its successors. The NVIDIA Agent Toolkit announced in March 2026 continues. Every agent built in Katonic is a declarative definition that compiles to NVIDIA's open agent software. Today that software is the NeMo Agent Toolkit. When NVIDIA changes the runtime, Katonic changes the compiler and your agents carry over.
The steps
- Write a test set. Each item is a question a real user would ask and the answer you expect.
- Add an evaluation section to the agent configuration. Name the test set and the evaluators.
- Run the evaluation from the command line.
- Read the scores per item, not only the average. The failures tell you what to fix.
- Change the agent, run again, and compare with the last run.
What to measure
| Measure | Question it answers | When it matters most |
|---|---|---|
| Answer accuracy | Does the answer match the expected one? | Always |
| Groundedness | Is the answer supported by the retrieved material? | Agents that answer from documents |
| Retrieval relevance | Did the agent fetch the right material? | Agents that answer from documents |
| Trajectory | Did the agent take sensible steps and call the right tools? | Agents that act, not only answer |
| Latency and tokens | How long did it take and what did it cost? | Before wide release |
| A custom measure | Does it meet a rule specific to your work? | Regulated or specialised tasks |
The toolkit ships evaluators for answer quality and for the trajectory an agent took, supports evaluators you write yourself, and can profile time and tokens during the same run.
How to build a test set that means something
- Take questions from real requests, not from the documents the agent reads.
- Include questions the agent should refuse or pass to a person.
- Include questions whose answer is not in the knowledge source, to check the agent says so.
- Have someone who knows the subject write the expected answers.
- Keep the set under version control and add every failure found in use.
From a score to a release gate
A score is useful once it decides something. Set a pass mark for each measure before the run. An agent that falls below it is not released. An agent that passes is released with the score on record, so the next version can be compared with it. Run the same set again whenever the model, the prompt, a tool or the knowledge changes.
What the toolkit does not do for you
- Decide the pass mark.
- Stop an agent that failed from being released.
- Keep a record of who approved a release and on what evidence.
- Show evaluation history to people who do not use a command line.
Where Katonic fits
Katonic runs evaluation as a gate. Every agent built in Katonic compiles to NVIDIA's open agent software, test sets and evaluators are managed in the platform, an agent is evaluated before release, and the result is kept with the approval in one audit trail. The platform runs on infrastructure you own, so test data stays inside.
Read next
- Running toolkit agents in production →
- NeMo Agent Toolkit vs LangGraph, CrewAI and LangChain →
- Use case: moving agent pilots to production →
- NVIDIA Agent Toolkit for enterprise AI →
- NVIDIA NeMo Agent Toolkit is deprecated: what replaces it? →
- NVIDIA migration guide for the NeMo Agent Toolkit →
Questions and answers
Does the NeMo Agent Toolkit have built-in evaluation?
Yes. It includes an evaluation harness that runs an agent against a test set and scores the results, with built-in and custom evaluators.
How many test questions do I need?
Enough to cover the main kinds of request and the main ways the agent can fail. Start small with real questions and add every failure you find in use.
Can I evaluate an agent written in LangGraph or CrewAI?
Yes. The toolkit works with agents written in those frameworks and evaluates them the same way.
What is trajectory evaluation?
Scoring the steps an agent took, such as which tools it called and in what order, and not only its final answer.
How do I stop an agent that fails evaluation from going live?
With a release gate outside the toolkit. An agent platform such as Katonic evaluates before release and records the approval.