mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-15 10:14:32 +00:00
fix: stop the re-embed migration from destroying the index it rebuilds
Five defects from a review of the embeddings work, four of them silent. - Write local files atomically. `LocalStorage.save_file` streamed straight onto the destination, so an interrupted write left a truncated file. `reembed` rewrites every index it touches, and a half-written `index.faiss` loads at neither the old width nor the new one -- the source was unrecoverable, with no backup and no temp file left behind. Bytes now land beside the destination and move into place with `os.replace`. S3 was already safe (single PUT). - Read pgvector chunks a page at a time. `reembed_pgvector` materialised every `(id, text)` row for a source before embedding -- ~1.6 GB at 200k chunks and several times that for non-Latin scripts, with the `PGresult` held alongside until the cursor closed. Inside the shipped 4Gi limit, while also holding the model, that is an OOMKill -- which is exactly the SIGKILL the point above turned into a destroyed index. It now walks the source by keyset. - Bound the first wave of delegated embeds. The failure cooldown is only latched once the first `get()` returns, so every request already in flight paid the full EMBEDDINGS_DELEGATE_TIMEOUT: measured 64 threads all timing out together, and at the shipped 60s across a 96-thread WSGI pool that is an API serving nothing at all, health checks included. One caller now probes while the rest fail fast; after a single success the gate leaves the path entirely. - Ship EMBEDDINGS_NAME commented in .env-template. The comment directly above it says to leave it commented when upgrading, and the line shipped set. Any value reaching `.env` lands in `model_fields_set`, which makes `resolve_embeddings_pin` bail -- so a template-derived `.env` disabled the legacy pin outright and repointed a populated index at a different 768-dim model, where no width check fires. The pin already picks granite for a fresh install and mpnet for an existing one, so nothing needs to be set by hand. - Stamp `sources.model` on wiki sources. They were created with the column NULL and then embedded like any other source, and the boot check reads NULL as "pre-dates the column, therefore the legacy model" -- reporting a correctly embedded source as stale on every startup of every process. Stamped at creation, and again on each page re-embed so existing rows heal. The two docs that promised the FAISS index survives a failed run said so of the embed only; both now describe the write, and upgrading.mdx says to stop ingest for the duration.
This commit is contained in:
1 parent
00be2c05ad
commit
12dd7c7eab
14 files changed
+583
-135
No files matched your search
@@ -59,7 +59,7 @@ Here are some of the most fundamental settings you'll likely want to configure:
|
||||
|
||||
- **`EMBEDDINGS_NAME`**: This setting defines which embedding model DocsGPT will use to generate vector embeddings for your documents. Embeddings are numerical representations of text that allow DocsGPT to understand the semantic meaning of your documents for efficient search and retrieval.
|
||||
|
||||
- **Default value:** `huggingface_sentence-transformers/all-mpnet-base-v2`, kept so an existing index stays readable. New installs are pointed at `ibm-granite/granite-embedding-311m-multilingual-r2` (multilingual, 32k context, same 768 dimensions) by `.env-template` and `setup.sh`.
|
||||
- **Default value:** leave it unset and DocsGPT picks for you at first boot, recording the choice so it never changes underneath you: a fresh install is pinned to `ibm-granite/granite-embedding-311m-multilingual-r2` (multilingual, 32k context, same 768 dimensions), and an install that already has sources is pinned to `huggingface_sentence-transformers/all-mpnet-base-v2` so its index stays readable. Setting it here overrides that pin.
|
||||
- **Other options:** Any FastEmbed built-in model, or any Hugging Face repository shipping an ONNX export. See [Embeddings](/Models/embeddings).
|
||||
- **Changing it on an existing index requires re-embedding** — same-width models swap without any error and silently degrade retrieval. Run `python -m application.scripts.reembed`.
|
||||
|
||||
|
||||
@@ -55,7 +55,11 @@ Useful flags:
|
||||
| `--sources a,b` | Only these source ids — also how you retry a failed source |
|
||||
| `--batch-size N` | Chunks per embed call (default 64) |
|
||||
|
||||
The script processes sources independently: one failing source is reported and skipped rather than aborting the run, and the exit code is non-zero if any failed. For `pgvector` it updates rows in place, so an interrupted run simply re-does its last batch. For `faiss` it builds the replacement index in memory and only writes once the whole rebuild succeeds, leaving the existing index untouched on failure.
|
||||
The script processes sources independently: one failing source is reported and skipped rather than aborting the run, and the exit code is non-zero if any failed. For `pgvector` it reads a page of chunks at a time and updates rows in place, so memory stays flat on a large index and an interrupted run simply re-does its last batch. For `faiss` it builds the replacement index in memory, writes each file to a temporary path, and moves it into place — so an interrupt during either the rebuild or the write leaves the existing index intact rather than truncated.
|
||||
|
||||
<Callout type="warning">
|
||||
Stop ingest before you run this. It reads each source's chunks and writes the vectors back; anything ingested while it runs can be overwritten by the rebuild (`faiss`) or missed by it (`pgvector`).
|
||||
</Callout>
|
||||
|
||||
<Callout type="info">
|
||||
Running [GraphRAG](/Sources/GraphRAG)? The script also rewrites `graph_nodes.name_embedding`, which seeds every graph traversal. Those vectors are written once at extraction time and share the chunk vectors' width, so leaving them in the old model's space degrades graph retrieval just as silently as the chunk vectors would — and needs no LLM re-extraction to fix.
|
||||
|
||||
Reference in new issue
Block a user