Retrieval is part of answer quality
A plausible answer can still be wrong because the failure happened before generation: a document was not ingested, chunks lost useful context, the right candidate ranked too low, fusion suppressed an exact match, or the final context pack omitted supporting evidence. The project therefore keeps retrieval and QA handoff independently inspectable.
Architecture
OpenTelemetry spans from ingestion and QA can be inspected in Phoenix across the same pipeline.
Lexical retrieval remains useful for exact terms, identifiers, and rare phrases. Dense retrieval broadens semantic recall. The combination is configurable so changes can be measured against the same fixture rather than assumed to help.
Working interface
Evaluation changed the retrieval configuration
The repository separates retrieval evaluation from final QA handoff. Historical runs record expected-document Hit@k, supporting-document Hit@k, context support, and stage latency. These artifacts are tied to a fixed local fixture and are not presented as general benchmarks.
| 20-query historical fixture | Cross-encoder off | Cross-encoder on |
|---|---|---|
| Strict Hit@1 | 9/20 · 0.45 | 5/20 · 0.25 |
| Strict Hit@3 | 15/20 · 0.75 | 9/20 · 0.45 |
| Strict Hit@5 | 18/20 · 0.90 | 12/20 · 0.60 |
| Supporting Hit@3 | 16/20 · 0.80 | 10/20 · 0.50 |
| Final-context support | 16/20 · 0.80 | 10/20 · 0.50 |
| Average retrieval time | ~569 ms | ~650 ms |
The cross-encoder improved one query but regressed seven. On this corpus and configuration it reduced recall/support while adding reranking latency, so it was left disabled and the next experiments moved to context-budget and handoff policy instead.
Scope: one checked-in 20-positive-query evaluation fixture plus three control queries. The numbers describe that historical experiment only.
Failures are separated by stage
Ingestion
Checksums, paths, chunk metadata, and asynchronous worker state make missing or duplicated source material inspectable.
Candidate retrieval
Lexical and dense results can be compared before fusion, MMR, sibling expansion, or reranking changes the candidate set.
Context handoff
Support labels test whether enough evidence reaches the answer stage instead of treating a retrieval hit as equivalent to a grounded answer.
Generation and tracing
Source metadata is retained and OpenTelemetry/Phoenix traces expose the path through retrieval, prompt construction, and generation.
Local-first boundaries
Local deployment improves data control but does not make a system secure by default. Ports, logs, model services, host filesystems, optional third-party providers, and document-level authorization still require explicit decisions. The repository is therefore presented as an evolving local system, not as a production-ready hosted product.