Retrieval and generation are evaluated separately
A document can fail to enter the index, the correct candidate can rank too low, fusion can suppress an exact match, context packing can omit supporting evidence, or generation can ignore evidence that was retrieved correctly. The system keeps those stages independently inspectable instead of collapsing them into a single answer-quality judgment.
Architecture keeps retrieval stages separable
OpenTelemetry spans from ingestion and QA can be inspected in Phoenix across the same pipeline.
OpenSearch and Qdrant outputs can be inspected before and after fusion, MMR, sibling expansion, reranking, and context packing, allowing each retrieval change to be compared against the same fixture.
Reproducible mixed benchmark
The retrieval stack is measured with a six-stage artifact pipeline: source acquisition, parsing, chunking, embedding, native OpenSearch/Qdrant indexing, and retrieval evaluation. Each stage records immutable lineage fingerprints and persists reusable artifacts privately, so a corpus or configuration change rebuilds only the stages it invalidates rather than starting from raw PDFs every time.
The mixed regression set combines Open RAGBench, OfficeQA, NFCorpus, and German/Arabic MIRACL subsets. composite-v2 is a controlled corpus-scale experiment: it keeps the same 80 positive Open RAGBench PDFs and the same 160 questions as v1, then expands that track from 200 to 1,000 PDFs by adding 800 deterministic non-gold distractors. Those additions were not selected using this system's retrieval scores or embedding similarity, so they are not presented as mined hard negatives.
Full composite-v2 retrieval result
| Track | Recall@1 | Recall@3 | Recall@5 | MRR | nDCG@5 | p95 ms |
|---|---|---|---|---|---|---|
| MIRACL Arabic | 0.3556 | 0.5933 | 0.7572 | 0.6957 | 0.6726 | 11.2 |
| MIRACL German | 0.1749 | 0.4251 | 0.5495 | 0.5857 | 0.4950 | 11.8 |
| NFCorpus | 0.0143 | 0.0356 | 0.0405 | 0.2650 | 0.1621 | 12.8 |
| OfficeQA | 0.0823 | 0.2347 | 0.2597 | 0.2473 | 0.2096 | 23.6 |
| Open RAGBench | 0.6875 | 0.9125 | 0.9437 | 0.7978 | 0.8350 | 35.0 |
| Macro average | 0.2629 | 0.4402 | 0.5101 | 0.5183 | 0.4749 | — |
What changed when the PDF corpus grew 5×?
| Open RAGBench metric | v1 · 200 PDFs | v2 · 1,000 PDFs | Delta |
|---|---|---|---|
| Recall@1 | 0.7688 | 0.6875 | -0.0813 |
| Recall@3 | 0.9563 | 0.9125 | -0.0438 |
| Recall@5 | 0.9563 | 0.9437 | -0.0126 |
| MRR | 0.8510 | 0.7978 | -0.0532 |
| nDCG@5 | 0.8780 | 0.8350 | -0.0430 |
| p95 latency | 26.3 ms | 35.0 ms | +8.7 ms |
The larger corpus primarily increased ranking pressure: Recall@1 fell materially, while Recall@5 remained high at 0.9437. In other words, the relevant Open RAGBench document was displaced from the first position more often but was still usually present in the top five. Because all five tracks share the combined index, the extra PDFs also act as distractors for OfficeQA, NFCorpus, and MIRACL queries.
These are project-regression results on deliberately subsetted and combined source benchmarks, not official upstream benchmark scores or a general claim about RAG accuracy.
Retrieval hit is not the same as usable context
The evaluation separates expected-document hits, supporting-document hits, and final-context support because a relevant candidate can be retrieved and still disappear during diversification, sibling expansion, ranking, or token-budget packing before generation.
Candidate retrieval
Lexical and dense candidates remain inspectable before later retrieval stages transform the ranked set.
Context handoff
Support labels test whether sufficient evidence survives into the final context rather than counting any retrieval hit as a grounded answer.
Generation
Source metadata remains attached so output can be checked against the evidence actually handed to the model.
Measured retrieval variants
Optional cross-encoder reranking was integrated and evaluated against the same checked-in retrieval fixture. It reduced retrieval quality on this corpus, so it remained disabled and subsequent experiments shifted to context-packing alternatives rather than tuning a stage that had not helped.
Current scope
The repository is an evolving local system rather than a production-hosted service. Production authorization, service exposure, and hosted multi-user security are outside the current scope.