fix: make worker-delegated embedding survive the shipped deployments

Query embedding moved to the Celery worker, but nothing that ships was
updated to consume the queue it dispatches to.

- Add `embeddings` to every worker `-Q` list (compose x3, k8s, devcontainer,
  sandbox README). Without it a search blocked for EMBEDDINGS_DELEGATE_TIMEOUT
  and then answered with no retrieved context, because classic_rag swallows the
  dispatch error and skips the source -- bad answers, not an error.

- Skip the task_postrun heap reclaim for the embed task. The full gc.collect()
  was written for docling/torch parses; on a worker holding the ONNX model it
  measured ~86ms against ~8ms for the embed itself, a 9x slowdown of the round
  trip for a task that allocates a few kilobytes.

- Resolve the installation pin in the re-embed script. It never imports
  application.app, so an install pinned in app_metadata with no EMBEDDINGS_NAME
  set -- every stock k8s deployment, whose manifests carry no embedding config
  -- would rewrite its whole index with the legacy default and stamp
  sources.model to match, then be told by the boot warning to run it again.

- Fail fast for 30s after a failed dispatch. fanout.embed_questions falls back
  to letting each store embed its own query, so one dead-worker retrieval paid
  the timeout once in the fan-out and again per source.

- Forget the task result. Nothing reads it back: the key is per-dispatch UUID,
  not content-addressed, so a repeated query mints another. Left alone every
  search leaked ~17KB for result_expires (7 days) into the Redis the broker
  shares -- on the bundled k8s manifest (1Gi, no maxmemory policy) that is an
  OOMKill that takes the broker with it.

- Release the model ensure_vector_schema loads to read the width of an
  unregistered model, in a process that delegates and would never call it.
  The width still comes from the model, not the table, so the mismatch check
  the hook exists for keeps working.

- Correct the docs that said otherwise: embeddings.md claimed the standard
  deployment worked unchanged, upgrading.mdx said no action was needed, and
  the settings table listed none of the three delegation settings.
This commit is contained in:
Alex committed 2026-08-28 14:31:19 +01:00
1 parent cb62dea701
commit 00be2c05ad
17 files changed
+449 -50

No files matched your search

+11
View File
@@ -440,6 +440,17 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
args = build_parser().parse_args(argv)
_log_setup(args.verbose)
# Which model this installation uses may live in ``app_metadata`` rather
# than the environment -- ``application.app`` resolves it at boot, and this
# script never imports that. Without this, an install pinned to granite
# with no EMBEDDINGS_NAME set (every stock Kubernetes deployment: the
# manifests carry no embedding config at all) would re-embed its whole
# index with the *legacy* code default and stamp ``sources.model`` to
# match -- the silent cross-model index this script exists to repair.
from application.storage.db.embeddings_pin import resolve_embeddings_pin
resolve_embeddings_pin(logger)
# Embed in this process. ``EMBEDDINGS_DELEGATE_TO_WORKER`` exists to keep a
# model out of the API, which serves one query at a time and holds the
# model for nothing in between. This is the opposite case: a batch job that