mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-14 16:16:48 +00:00
fix: embeddings registry edge cases and chunk-size accounting
Follow-up to the embeddings work, from a review pass over the branch. - Route the OpenAI/Azure key handling through the model registry instead of matching the canonical name literally, so the `text-embedding-ada-002` alias the registry now accepts also reaches the Azure deployment name rather than failing every embed with DeploymentNotFound. - Fall back to a default width where the embeddings model reports no dimension. A model outside the registry returns None rather than no attribute, so `getattr` with a default did not catch it and the width reached the DDL as `vector(None)` / `list_size=None`. - Point HF_HUB_CACHE at the prefetch directory. Chunking loads the tokenizer through `tokenizers`, which reads the hub cache, so a fresh container fetched over the network on first ingest and an offline one silently fell back to cl100k. - Charge a token that collapses a long unbroken run by its character span. WordPiece emits one [UNK] for any word over its character limit, which made base64 and minified content count as near-zero tokens, so nothing split it and oversized chunks reached the embedding server. - Preserve chunk ids and honour --batch-size when rebuilding a FAISS index. Fresh uuids orphaned GraphRAG's graph_node_chunks rows, and the whole index went out in a single embed call on remote servers. - Document that granite runs an int8-quantised graph, and scope the SentenceTransformer parity claim to mpnet's fp32 graph, which is where it was measured. - Correct the embeddings docs: a matching dimension is not a matching model, so a same-width swap raises nothing and silently degrades retrieval.
This commit is contained in:
1 parent
869d87452b
commit
25e07f5cee
15 files changed
+296
-20
No files matched your search
@@ -107,6 +107,22 @@ class TestFaissRebuild:
|
||||
keys = [call.kwargs.get("embeddings_key") for call in factory.call_args_list]
|
||||
assert keys == ["sk-real", "sk-real"]
|
||||
|
||||
def test_chunk_ids_are_preserved(self, stores):
|
||||
"""Re-embedding must not renumber chunks.
|
||||
|
||||
Fresh ids orphan every GraphRAG ``graph_node_chunks`` row for the
|
||||
source and invalidate any id a client already holds.
|
||||
"""
|
||||
_, _, factory = stores
|
||||
reembed.reembed_faiss("s1", batch_size=8, dry_run=False)
|
||||
assert factory.call_args_list[1].kwargs["ids"] == ["1", "2"]
|
||||
|
||||
def test_batch_size_is_forwarded_to_the_rebuild(self, stores):
|
||||
"""On a remote embeddings server the whole index is otherwise one POST."""
|
||||
_, _, factory = stores
|
||||
reembed.reembed_faiss("s1", batch_size=8, dry_run=False)
|
||||
assert factory.call_args_list[1].kwargs["batch_size"] == 8
|
||||
|
||||
def test_existing_index_is_not_deleted(self, stores):
|
||||
"""The rebuild must not destroy the old index before the new one exists."""
|
||||
existing, rebuilt, _ = stores
|
||||
|
||||
Reference in new issue
Block a user