Skip to content

0001 — Hybrid retrieval (dense + sparse) fused with Reciprocal Rank Fusion

Status

Accepted (v0.1.0)

Context

A single retrieval signal has known, complementary blind spots:

  • Dense (bi-encoder) retrieval (BAAI/bge-small-en-v1.5 embeddings compared by cosine distance in pgvector, retrieval/dense_search.py) is good at semantic/paraphrase matches but can miss exact keyword/identifier matches that don't embed close together (product codes, proper nouns, rare terms).
  • Sparse (BM25) retrieval (retrieval/sparse_search.py, backed by a per-tenant in-memory rank_bm25.BM25Okapi index in indexing/sparse_index.py) is good at exact lexical matches but has no notion of semantic similarity.

The PRD needs both without one signal silently overriding the other, and needs a fusion method that doesn't require normalizing two very differently-scaled score distributions (cosine distance vs. BM25 score) against each other.

Decision

Retrieve the top-100 candidates independently from each signal, then fuse with Reciprocal Rank Fusion (RRF), not a weighted blend of raw scores:

score(doc) = sum(1 / (k + rank_r(doc)) for r in rankers if doc appears in r)

with k=60 (the original RRF paper's default, Cormack et al., 2009). Implementation: retrieval/rrf.py::fuse, called from retrieval/hybrid_retriever.py::search, which runs the dense and sparse top-100 lookups concurrently (asyncio.gather) and fuses by Chunk.id before handing the fused top-25 to cross-encoder reranking (see ADR-0005 and the reranking module for that separate stage).

RRF was chosen over score-level fusion because: - It only needs each ranker's rank position, not a comparable score scale — cosine distance and BM25 score are never combined arithmetically, so there's no min-max normalization or score-calibration step to get wrong. - A doc that both rankers agree on (present in both top-100 lists) is rewarded — its two per-ranker RRF terms sum — without being double-counted as two separate hits in the output; fuse is keyed by doc identity and returns each doc exactly once. - A doc entirely absent from one ranker's list contributes zero from that ranker rather than being penalized, which is what lets two only-partially-overlapping top-100 sets (dense and sparse routinely disagree on much of their candidate sets) fuse cleanly.

Consequences

  • Every query pays for two independent top-100 lookups (run concurrently, not sequentially) plus a cheap in-memory fuse (rrf.fuse is O(n log n) over at most 200 candidates) before reranking.
  • k=60 is a fixed default threaded as a parameter, not hard-coded past rrf.fuse's signature — it can be tuned later without an interface change, but was not tuned for this release.
  • Recall depends on both stages' top-100 cutoffs. eval/retrieval_eval.py measures recall@5/10/25 on the hand-labeled qrels.json fixture and found recall of 1.0/1.0/1.0 at the eval corpus scale (see the README's "Measured performance" section and ADR-0006) — a real result on a small, hand-checkable fixture, not a claim that generalizes to every corpus shape.
  • The original p95 latency finding (751ms, ~6x over the <120ms PRD target) traced back to the underlying dense/sparse queries, not to the fusion step itself, which stays sub-millisecond; see ADR-0006 for the full root cause and fix.