mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-15 12:13:38 +00:00
fix: stop the re-embed migration from destroying the index it rebuilds
Five defects from a review of the embeddings work, four of them silent. - Write local files atomically. `LocalStorage.save_file` streamed straight onto the destination, so an interrupted write left a truncated file. `reembed` rewrites every index it touches, and a half-written `index.faiss` loads at neither the old width nor the new one -- the source was unrecoverable, with no backup and no temp file left behind. Bytes now land beside the destination and move into place with `os.replace`. S3 was already safe (single PUT). - Read pgvector chunks a page at a time. `reembed_pgvector` materialised every `(id, text)` row for a source before embedding -- ~1.6 GB at 200k chunks and several times that for non-Latin scripts, with the `PGresult` held alongside until the cursor closed. Inside the shipped 4Gi limit, while also holding the model, that is an OOMKill -- which is exactly the SIGKILL the point above turned into a destroyed index. It now walks the source by keyset. - Bound the first wave of delegated embeds. The failure cooldown is only latched once the first `get()` returns, so every request already in flight paid the full EMBEDDINGS_DELEGATE_TIMEOUT: measured 64 threads all timing out together, and at the shipped 60s across a 96-thread WSGI pool that is an API serving nothing at all, health checks included. One caller now probes while the rest fail fast; after a single success the gate leaves the path entirely. - Ship EMBEDDINGS_NAME commented in .env-template. The comment directly above it says to leave it commented when upgrading, and the line shipped set. Any value reaching `.env` lands in `model_fields_set`, which makes `resolve_embeddings_pin` bail -- so a template-derived `.env` disabled the legacy pin outright and repointed a populated index at a different 768-dim model, where no width check fires. The pin already picks granite for a fresh install and mpnet for an existing one, so nothing needs to be set by hand. - Stamp `sources.model` on wiki sources. They were created with the column NULL and then embedded like any other source, and the boot check reads NULL as "pre-dates the column, therefore the legacy model" -- reporting a correctly embedded source as stale on every startup of every process. Stamped at creation, and again on each page re-embed so existing rows heal. The two docs that promised the FAISS index survives a failed run said so of the embed only; both now describe the write, and upgrading.mdx says to stop ingest for the duration.
This commit is contained in:
1 parent
00be2c05ad
commit
12dd7c7eab
14 files changed
+583
-135
No files matched your search
@@ -95,6 +95,12 @@ class TestCreateWikiSource:
|
||||
assert row["type"] == "wiki"
|
||||
assert row["config"]["kind"] == "wiki"
|
||||
assert int(row["tokens"]) == 0
|
||||
# Wiki pages get embedded like any other source, so the row has to name
|
||||
# the model that did it. NULL reads as "the legacy model" to the boot
|
||||
# mismatch check, which then reports the source as stale forever.
|
||||
from application.core.settings import settings
|
||||
|
||||
assert row["model"] == settings.EMBEDDINGS_NAME
|
||||
# No seed content → no re-embed, and never any ingest/reingest task.
|
||||
mock_reembed.assert_not_called()
|
||||
mock_ingest.assert_not_called()
|
||||
|
||||
Reference in new issue
Block a user