SignalTech/AI
Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads
Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads
Context
What happened: A paper on agentic RAG evaluation measures how evaluation budgets are allocated across questions, search trajectories, and repeated reads, using a retrieval-feedback comparison on HotpotQA and MuSiQue.
Key numbers: 34.14 to 34.39M model tokens; broader question coverage lowers
Why it matters: The results indicate broader question coverage reduces standard error more than additional reads or trajectories at the same token budget, and that more questions beat more trajectories at search prices of $0 to $1 per 1,000 requests. Question-versus-read fee rankings remain unresolved.
What changed
A paper on agentic RAG evaluation measures how evaluation budgets are allocated across questions, search trajectories, and repeated reads, using a retrieval-feedback comparison on HotpotQA and MuSiQue.
Why it matters
The results indicate broader question coverage reduces standard error more than additional reads or trajectories at the same token budget, and that more questions beat more trajectories at search prices of $0 to $1 per 1,000 requests. Question-versus-read fee rankings remain unresolved.