Skip to content

0004 — In-process per-tenant BM25 vs. Postgres full-text search

Status

Accepted for v0.1.0, with a known, documented scaling limit (not solved here)

Context

Sparse (lexical) retrieval needs a BM25-style ranking over each tenant's chunk text. Two realistic options existed: 1. An in-process BM25 index (rank_bm25.BM25Okapi) built and cached per tenant inside the API process (indexing/sparse_index.py). 2. Postgres's built-in full-text search (tsvector/tsquery, ts_rank), queried like any other column.

Decision

Use in-process, per-tenant BM25Okapi indexes, one per tenant, cached in a module-level dict (indexing/sparse_index._registry) guarded by a threading.Lock, built lazily on first access per tenant (get_or_build) and rebuilt after every ingestion batch for that tenant (rebuild, called from ingestion/pipeline.py).

Rationale at this project's scale and timeline: - Ranking quality control. rank_bm25.BM25Okapi gives direct, well-understood BM25 semantics (k1=1.5, b=0.75 — the library defaults) that are easy to reason about and unit test in isolation (_rank_by_bm25_score in retrieval/sparse_search.py is a pure function with no DB dependency, tested directly against a hand-built index). Postgres FTS's ranking (ts_rank/ts_rank_cd) is a different, less-directly-BM25 scoring model that would need its own tuning and validation pass to reach comparable, well-understood behavior. - No new schema/index-type dependency for this release. Postgres FTS would need a tsvector column (generated or maintained), a GIN index, and language-specific configuration (stemming, stop words) — a real but separate scope of work not undertaken for v0.1.0. - Fits the current single-process deployment. docker-compose.yml runs exactly one api service instance; an in-memory-per-process index has no replica-consistency problem to solve at that scale.

Tenant scoping for this index is structural, matching the rest of src/: get_or_build's underlying Postgres read runs through get_tenant_session(tenant_id) (RLS-enforced — see ADR-0002), so the BM25 corpus itself can never be built from another tenant's rows, independent of the in-process cache being keyed correctly by tenant_id.

The known, documented limit — this does NOT solve horizontal scaling

This design has a real scaling ceiling, called out explicitly rather than discovered later in production: the in-process BM25 index does not horizontally scale across API replicas without a shared index store.

  • Each API process instance holds its own independent _registry in local memory. Running more than one replica of the api service means each replica builds and caches its own copy of every tenant's BM25 index — no memory is shared between replicas.
  • rebuild(tenant_id) (called after ingestion) only updates the calling process's own cached index. In a multi-replica deployment, a chunk newly ingested via replica A would not be reflected in replica B's BM25 index until replica B independently rebuilds it (which nothing currently triggers cross-process) — a genuine staleness/consistency gap under horizontal scaling, not merely a performance one.
  • Memory cost scales with the number of distinct tenants actively queried against each replica, multiplied by replica count, rather than being shared once across a fleet.

This is a known limit of the v0.1.0 architecture, not something this task solves. A real fix (a shared/external index store — e.g. Postgres FTS after all, or a dedicated search service, or a rebuild-broadcast mechanism) is future work, to be designed if/when horizontal scaling of the api service is actually needed.

Consequences

  • v0.1.0 is safe to run as a single api replica (exactly what docker-compose.yml ships). Scaling api replicas today would silently reintroduce cross-replica staleness in sparse retrieval results without any code change flagging it.
  • Revisiting this decision (most plausibly toward Postgres FTS, or a shared cache) is the natural next step if/when replica count > 1 becomes a real requirement — tracked as a known gap, not filed as a task in this release.