feat: embed on the worker, and stop batching the ONNX pass

The API embeds every query it serves, so it held its own copy of the model:
~890 MB it never needed. EMBEDDINGS_DELEGATE_TO_WORKER (on by default) sends
the text to the Celery worker instead and gets the vector back, taking an API
process from 1176 MB to 285 MB with no ONNX Runtime imported at all. The client
embeds locally when it finds itself inside a worker task, so the worker never
dispatches to itself -- the same self-deadlock DOCUMENT_PARSE_QUEUE avoids on
the parsing side. EMBEDDINGS_BASE_URL still wins over it, and remains the right
answer for production.

ensure_vector_schema was constructing the embeddings instance purely to read
.dimension off it, loading several hundred MB of ONNX into every API and worker
process at import. For a model the registry describes that is a lookup; only an
unregistered name now falls back to loading.

EMBEDDINGS_BATCH_SIZE was sizing two unrelated things: chunks per store
transaction (and per remote embed request) and documents per ONNX forward pass.
Each pass pads every input up to its longest, and that waste grows with the
square of chunk length, so at the 1250-token default a batch of 32 peaked at
6.6 GB and took 326s where a batch of 1 peaked at 2.9 GB and took 90s. The
forward pass is now sized by EMBEDDINGS_MODEL_BATCH_SIZE, defaulting to 1;
storage and remote batching are unchanged at 32.

reembed embeds in-process: a batch job that walks the whole index should not
round-trip every chunk through a broker, and loading the model there reports a
real failure instead of timing out against an empty queue.

Also drops the mpnet zip download from the docs and the devcontainer, which
pointed at a SentenceTransformers export with no ONNX graph and had been inert
since the FastEmbed swap; corrects the claim that any sentence-transformers
model works; and settles the Configuring/Settings pages on what the registry
and the repository metadata actually decide.
This commit is contained in:
Alex committed 2026-08-28 12:23:18 +01:00
1 parent ef51ee97b3
commit 47e53a71c4
20 files changed
+679 -266

No files matched your search

+11
View File
@@ -411,6 +411,17 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
args = build_parser().parse_args(argv)
_log_setup(args.verbose)
# Embed in this process. ``EMBEDDINGS_DELEGATE_TO_WORKER`` exists to keep a
# model out of the API, which serves one query at a time and holds the
# model for nothing in between. This is the opposite case: a batch job that
# embeds every chunk in the index, where a broker round trip per batch adds
# latency and a dependency on a worker running. Loading the model here also
# means the script reports a real failure for a model it cannot load,
# instead of timing out against an empty queue.
if getattr(settings, "EMBEDDINGS_DELEGATE_TO_WORKER", False):
logger.info("Embedding in-process; worker delegation does not apply here.")
settings.EMBEDDINGS_DELEGATE_TO_WORKER = False
store_type = (settings.VECTOR_STORE or "").lower()
if store_type not in SUPPORTED_STORES:
logger.error(