fix: make worker-delegated embedding survive the shipped deployments

Query embedding moved to the Celery worker, but nothing that ships was
updated to consume the queue it dispatches to.

- Add `embeddings` to every worker `-Q` list (compose x3, k8s, devcontainer,
  sandbox README). Without it a search blocked for EMBEDDINGS_DELEGATE_TIMEOUT
  and then answered with no retrieved context, because classic_rag swallows the
  dispatch error and skips the source -- bad answers, not an error.

- Skip the task_postrun heap reclaim for the embed task. The full gc.collect()
  was written for docling/torch parses; on a worker holding the ONNX model it
  measured ~86ms against ~8ms for the embed itself, a 9x slowdown of the round
  trip for a task that allocates a few kilobytes.

- Resolve the installation pin in the re-embed script. It never imports
  application.app, so an install pinned in app_metadata with no EMBEDDINGS_NAME
  set -- every stock k8s deployment, whose manifests carry no embedding config
  -- would rewrite its whole index with the legacy default and stamp
  sources.model to match, then be told by the boot warning to run it again.

- Fail fast for 30s after a failed dispatch. fanout.embed_questions falls back
  to letting each store embed its own query, so one dead-worker retrieval paid
  the timeout once in the fan-out and again per source.

- Forget the task result. Nothing reads it back: the key is per-dispatch UUID,
  not content-addressed, so a repeated query mints another. Left alone every
  search leaked ~17KB for result_expires (7 days) into the Redis the broker
  shares -- on the bundled k8s manifest (1Gi, no maxmemory policy) that is an
  OOMKill that takes the broker with it.

- Release the model ensure_vector_schema loads to read the width of an
  unregistered model, in a process that delegates and would never call it.
  The width still comes from the model, not the table, so the mismatch check
  the hook exists for keeps working.

- Correct the docs that said otherwise: embeddings.md claimed the standard
  deployment worked unchanged, upgrading.mdx said no action was needed, and
  the settings table listed none of the three delegation settings.
This commit is contained in:
Alex committed 2026-08-28 14:31:19 +01:00
1 parent cb62dea701
commit 00be2c05ad
17 files changed
+449 -50

No files matched your search

+31
View File
@@ -428,3 +428,34 @@ class TestRecordsTheModel:
), patch.object(reembed, "reembed_pgvector", side_effect=RuntimeError("boom")):
assert reembed.run("pgvector", None, 64, False) == 1
record.assert_not_called()
class TestThePinIsResolved:
"""The script must embed with the model the installation is pinned to.
``resolve_embeddings_pin`` runs in ``application.app``, which this script
never imports. An install pinned in ``app_metadata`` with no
``EMBEDDINGS_NAME`` in the environment -- every stock Kubernetes
deployment, whose manifests carry no embedding config -- would otherwise
rewrite its whole index with the legacy code default and stamp
``sources.model`` to match, creating the cross-model index this script
exists to repair.
"""
def test_main_resolves_the_pin_before_reading_the_store(self):
order = []
with patch(
"application.storage.db.embeddings_pin.resolve_embeddings_pin",
side_effect=lambda *a, **k: order.append("pin"),
), patch.object(reembed.settings, "VECTOR_STORE", "pgvector", create=True), patch.object(
reembed, "run", side_effect=lambda *a, **k: (order.append("run"), 0)[1]
):
assert reembed.main([]) == 0
assert order == ["pin", "run"], "the pin must resolve before anything embeds"
def test_an_unsupported_store_still_resolved_the_pin_first(self):
with patch(
"application.storage.db.embeddings_pin.resolve_embeddings_pin"
) as pin, patch.object(reembed.settings, "VECTOR_STORE", "qdrant", create=True):
assert reembed.main([]) == 2
pin.assert_called_once()