mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-15 06:15:57 +00:00
fix: make worker-delegated embedding survive the shipped deployments
Query embedding moved to the Celery worker, but nothing that ships was updated to consume the queue it dispatches to. - Add `embeddings` to every worker `-Q` list (compose x3, k8s, devcontainer, sandbox README). Without it a search blocked for EMBEDDINGS_DELEGATE_TIMEOUT and then answered with no retrieved context, because classic_rag swallows the dispatch error and skips the source -- bad answers, not an error. - Skip the task_postrun heap reclaim for the embed task. The full gc.collect() was written for docling/torch parses; on a worker holding the ONNX model it measured ~86ms against ~8ms for the embed itself, a 9x slowdown of the round trip for a task that allocates a few kilobytes. - Resolve the installation pin in the re-embed script. It never imports application.app, so an install pinned in app_metadata with no EMBEDDINGS_NAME set -- every stock k8s deployment, whose manifests carry no embedding config -- would rewrite its whole index with the legacy default and stamp sources.model to match, then be told by the boot warning to run it again. - Fail fast for 30s after a failed dispatch. fanout.embed_questions falls back to letting each store embed its own query, so one dead-worker retrieval paid the timeout once in the fan-out and again per source. - Forget the task result. Nothing reads it back: the key is per-dispatch UUID, not content-addressed, so a repeated query mints another. Left alone every search leaked ~17KB for result_expires (7 days) into the Redis the broker shares -- on the bundled k8s manifest (1Gi, no maxmemory policy) that is an OOMKill that takes the broker with it. - Release the model ensure_vector_schema loads to read the width of an unregistered model, in a process that delegates and would never call it. The width still comes from the model, not the table, so the mismatch check the hook exists for keeps working. - Correct the docs that said otherwise: embeddings.md claimed the standard deployment worked unchanged, upgrading.mdx said no action was needed, and the settings table listed none of the three delegation settings.
This commit is contained in:
1 parent
cb62dea701
commit
00be2c05ad
17 files changed
+449
-50
No files matched your search
@@ -428,3 +428,34 @@ class TestRecordsTheModel:
|
||||
), patch.object(reembed, "reembed_pgvector", side_effect=RuntimeError("boom")):
|
||||
assert reembed.run("pgvector", None, 64, False) == 1
|
||||
record.assert_not_called()
|
||||
|
||||
|
||||
class TestThePinIsResolved:
|
||||
"""The script must embed with the model the installation is pinned to.
|
||||
|
||||
``resolve_embeddings_pin`` runs in ``application.app``, which this script
|
||||
never imports. An install pinned in ``app_metadata`` with no
|
||||
``EMBEDDINGS_NAME`` in the environment -- every stock Kubernetes
|
||||
deployment, whose manifests carry no embedding config -- would otherwise
|
||||
rewrite its whole index with the legacy code default and stamp
|
||||
``sources.model`` to match, creating the cross-model index this script
|
||||
exists to repair.
|
||||
"""
|
||||
|
||||
def test_main_resolves_the_pin_before_reading_the_store(self):
|
||||
order = []
|
||||
with patch(
|
||||
"application.storage.db.embeddings_pin.resolve_embeddings_pin",
|
||||
side_effect=lambda *a, **k: order.append("pin"),
|
||||
), patch.object(reembed.settings, "VECTOR_STORE", "pgvector", create=True), patch.object(
|
||||
reembed, "run", side_effect=lambda *a, **k: (order.append("run"), 0)[1]
|
||||
):
|
||||
assert reembed.main([]) == 0
|
||||
assert order == ["pin", "run"], "the pin must resolve before anything embeds"
|
||||
|
||||
def test_an_unsupported_store_still_resolved_the_pin_first(self):
|
||||
with patch(
|
||||
"application.storage.db.embeddings_pin.resolve_embeddings_pin"
|
||||
) as pin, patch.object(reembed.settings, "VECTOR_STORE", "qdrant", create=True):
|
||||
assert reembed.main([]) == 2
|
||||
pin.assert_called_once()
|
||||
Reference in new issue
Block a user