SignalTech/AI
Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes
NVIDIA NeMo guidance on converting agent failures into tests and fixes drives relevance.
Context
What happened: NVIDIA published a tutorial showing how to use NeMo Compass, NeMo Eval Author, and NeMo Relay to turn agent failures found in recorded traces into evaluations and harness fixes, using a Hermes Agent example in an OpenShell sandbox.
Key numbers: 42 traces analyzed; Ubuntu 24.04 with 8 vCPUs and 32 GB of RAM recommended;
Why it matters: The example agent called send_message without user confirmation, and the fix was a change to its SOUL.md instructions. NVIDIA says its internal benchmarks found NeMo Compass flagged anomalies that a coding agent on a similar task missed.
What changed
NVIDIA published a tutorial showing how to use NeMo Compass, NeMo Eval Author, and NeMo Relay to turn agent failures found in recorded traces into evaluations and harness fixes, using a Hermes Agent example in an OpenShell sandbox.
Why it matters
The example agent called send_message without user confirmation, and the fix was a change to its SOUL.md instructions. NVIDIA says its internal benchmarks found NeMo Compass flagged anomalies that a coding agent on a similar task missed.