Skip to content

0003 — Embedding model: BAAI/bge-small-en-v1.5

Status

Accepted (v0.1.0)

Context

Dense retrieval (ADR-0001) needs a sentence-embedding model for both indexing (indexing/dense.py) and query-time encoding (retrieval/dense_search.py::_encode_query, which deliberately reuses indexing.dense's model singleton rather than loading a second copy — see that module's docstring). The choice affects: embedding quality (recall), vector dimension (storage + index cost), and inference latency/resource footprint (the whole app runs CPU-only in the shipped Docker image — docker/Dockerfile installs the CPU build of torch).

Decision

Use BAAI/bge-small-en-v1.5 (via sentence-transformers), loaded once per process behind a double-checked-lock singleton (indexing/dense.py::_load_model) rather than functools.lru_cache, specifically because it is invoked concurrently from a thread-pool executor during batch ingestion and lru_cache does not guarantee single-construction under a concurrent cold-cache race.

Rationale: - Small, CPU-friendly footprint. The "small" variant targets exactly the profile this stack needs — no GPU is assumed anywhere in the Docker image or docker-compose service — while still being a competitive general-purpose English embedding model on public retrieval benchmarks (MTEB) at release time. - Fixed, known output dimension (384). This matters concretely, not just conceptually: pgvector's HNSW/IVFFlat index types reject an undimensioned vector column outright (discovered in Task 7 — see ADR-0006), so the column had to be narrowed to vector(384) before it could be indexed at all. A model with a fixed, documented dimension makes that narrowing a one-time, confident migration step (ALTER COLUMN ... TYPE vector(384)) rather than something inferred at runtime. - English-only scope matches the PRD. No multilingual requirement was in scope for v0.1.0, so a general multilingual model's larger footprint and typically lower per-language quality on English-only corpora were not worth taking on.

Consequences

  • chunks.embedding is vector(384) at the database level (post-ADR-0006's migration); any future change of embedding model with a different output dimension requires a new migration (a dimension-changing ALTER COLUMN, an index rebuild, and a full re-embed of existing rows — not a drop-in swap).
  • ingestion/chunker.py separately loads a SentenceTransformer tokenizer (via its own load_tokenizer, for chunk-boundary decisions) for the same model name (EMBEDDING_MODEL_NAME) but does not share the loaded model instance with indexing/dense.py — indexing/dense.py's module docstring records why that sharing is not actually possible without a separate refactor (the tokenizer loader discards the model after pulling the tokenizer off it) and treats it as explicitly out of scope for this project.
  • Model weights are downloaded on first use and cached (docker-compose.yml's model_cache volume, mounted at /root/.cache/torch), so a fresh container start pays a one-time download + load cost before the app can serve dense retrieval.
  • No fine-tuning or domain adaptation was done — this is the off-the-shelf public checkpoint.