Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

What happened: Researchers introduced TraceTrial-Bench, a benchmark that evaluates agent harness reliability separately from agent task performance, using mechanism-oriented tasks, trajectory tracing, and failure attribution. Key numbers: 95 tasks across seven components; 4 harnesses, 9 models, 3,420 runs; macro-averaged harness scores 59.8% to 72.0%; 31.1% of runs Why it matters: The authors state that task success does not always mean the harness worked reliably, and the benchmark is presented as an auditable, reproducible foundation for comparing harnesses and diagnosing failures.

Researchers introduced TraceTrial-Bench, a benchmark that evaluates agent harness reliability separately from agent task performance, using mechanism-oriented tasks, trajectory tracing, and failure attribution.

The authors state that task success does not always mean the harness worked reliably, and the benchmark is presented as an auditable, reproducible foundation for comparing harnesses and diagnosing failures.