SignalTech/AI
Retrieval Augmented Generation Compositionality
High-priority alert based on source significance and keyword relevance.
Context
What happened: LASAGNA, an open-source framework for controlled RAG evaluation, tested 450 configurations across eight benchmarks and corpora from 5K to 200K documents, finding that isolated technique gains do not reliably predict combined performance.
Key numbers: 450
Why it matters: The authors state that gains measured in isolation are not reliable predictors of realistic performance and call on researchers to test new RAG techniques with the full surrounding stack. They note the pre-declared H1 test was not met and that results use one English test bed and proprietary generators.
What changed
LASAGNA, an open-source framework for controlled RAG evaluation, tested 450 configurations across eight benchmarks and corpora from 5K to 200K documents, finding that isolated technique gains do not reliably predict combined performance.
Why it matters
The authors state that gains measured in isolation are not reliable predictors of realistic performance and call on researchers to test new RAG techniques with the full surrounding stack. They note the pre-declared H1 test was not met and that results use one English test bed and proprietary generators.