The default advice for retrieval is to reach for a purpose-built vector database. It is good advice for a particular shape of problem: enormous corpora, embeddings that are the whole product, recall that has to be state of the art. My problem is not that shape, and pretending otherwise would have cost me real correctness.
Look at the shape of the query
Every retrieval I do is scoped to a tenant and almost always joined against relational data. Not “find similar text”, but: find records belonging to this organisation, that this user is allowed to see, ranked by similarity, along with what they are connected to.
SELECT d.id, d.title, 1 - (d.embedding <=> $1) AS score
FROM documents d
WHERE d.organization_id = $2
ORDER BY d.embedding <=> $1
LIMIT 12;In a split architecture that becomes: query the vector store, get ids back, query the database with those ids, filter by tenant, discover that several results belong to someone else or were deleted last week, and go back for more. You have reimplemented a join across a network boundary — and moved the access check to the least reliable place in the system, which is after retrieval.
What one database buys
- One transaction. A record and its embedding are written together or not at all, so there is no window where the index describes a row that does not exist.
- One backup. A point-in-time restore brings back the index consistent with the data, because they were never separate things.
- One access model. The tenant filter is a WHERE clause the database enforces, reviewed like every other query.
- One system to operate. On a small team, the number of stateful services you run is a real constraint rather than a footnote.
The setup
A vector column indexed with HNSW on cosine distance. HNSW rather than IVFFlat because the workload is read-heavy with steady incremental writes, and IVFFlat's lists want rebuilding as the corpus grows.
One thing I would do again regardless of the storage choice: put the embedding provider behind an interface with a deterministic offline implementation. The test suite runs with no network and no API bill, and swapping providers is a single class. That indirection cost an hour and has already paid for itself twice.
Where this stops being right
I would not defend this at every scale. Once the vector workload starts competing with everything else for the database's resources, or recall genuinely becomes the product rather than a feature of it, extracting it is correct. The interface and the tenant-scoped query surface are what keep that migration contained rather than architectural.
Decide slowly about what you cannot undo, and keep everything else undoable. Retrieval infrastructure stays undoable if you build the seam first.