RAG evaluation · Research tooling
2026
RAGlign
Point it at your documents and it tells you which RAG configuration to use — and proves it.
Problem
Building RAG means choosing a chunking strategy, a retriever, and whether to rerank. Those choices interact, the right answer depends on your documents, and most teams pick by intuition. Measuring instead does not help as much as it should, because the standard measurement is biased: ground truth is recorded as a chunk ID, but chunk 57 only exists inside one chunking strategy. Score a different strategy against it and you are measuring how similar its cut points are, not how well it retrieves.
Contribution
I stored ground truth as a character span in the original document rather than a chunk ID, so every strategy is asked the same question — did you retrieve the text that answers this? — and none of them wrote the question. Around that I built a grid of 30 configurations, bootstrap confidence intervals on every metric, a per-question diagnosis naming the stage that lost it, and a dashboard that re-ranks live as you change what you value.
- configurations evaluated
- 0configurations evaluated
- hit@5 of measured bias
- 0.6hit@5 of measured bias
- retrieval ↔ answer quality
- +0.95retrieval ↔ answer quality
- Python
- sentence-transformers
- BM25
- NumPy
- Streamlit
- pytest
- bootstrap CIs
Architecture
Ground truth
- documents
- ~40 questions
- verbatim quotes
- quote resolved to character span
- offset invariant asserted
Evaluation
- 5 chunkers
- 3 retrievers
- reranker on / off
- span overlap
- hit@k
- MRR
- nDCG
- bootstrap CIs
Analysis
- chunker destroyed / degraded
- retriever missed
- reranker demoted
- quality / latency / context weights
- templated explanation
- Streamlit dashboard
- CLI scripts
