fix: embeddings registry edge cases and chunk-size accounting

Follow-up to the embeddings work, from a review pass over the branch.

- Route the OpenAI/Azure key handling through the model registry instead of
  matching the canonical name literally, so the `text-embedding-ada-002`
  alias the registry now accepts also reaches the Azure deployment name
  rather than failing every embed with DeploymentNotFound.
- Fall back to a default width where the embeddings model reports no
  dimension. A model outside the registry returns None rather than no
  attribute, so `getattr` with a default did not catch it and the width
  reached the DDL as `vector(None)` / `list_size=None`.
- Point HF_HUB_CACHE at the prefetch directory. Chunking loads the tokenizer
  through `tokenizers`, which reads the hub cache, so a fresh container
  fetched over the network on first ingest and an offline one silently fell
  back to cl100k.
- Charge a token that collapses a long unbroken run by its character span.
  WordPiece emits one [UNK] for any word over its character limit, which made
  base64 and minified content count as near-zero tokens, so nothing split it
  and oversized chunks reached the embedding server.
- Preserve chunk ids and honour --batch-size when rebuilding a FAISS index.
  Fresh uuids orphaned GraphRAG's graph_node_chunks rows, and the whole index
  went out in a single embed call on remote servers.
- Document that granite runs an int8-quantised graph, and scope the
  SentenceTransformer parity claim to mpnet's fp32 graph, which is where it
  was measured.
- Correct the embeddings docs: a matching dimension is not a matching model,
  so a same-width swap raises nothing and silently degrades retrieval.
This commit is contained in:
Alex committed 2026-08-27 13:59:51 +01:00
1 parent 869d87452b
commit 25e07f5cee
15 files changed
+296 -20

No files matched your search

+16
View File
@@ -107,6 +107,22 @@ class TestFaissRebuild:
keys = [call.kwargs.get("embeddings_key") for call in factory.call_args_list]
assert keys == ["sk-real", "sk-real"]
def test_chunk_ids_are_preserved(self, stores):
"""Re-embedding must not renumber chunks.
Fresh ids orphan every GraphRAG ``graph_node_chunks`` row for the
source and invalidate any id a client already holds.
"""
_, _, factory = stores
reembed.reembed_faiss("s1", batch_size=8, dry_run=False)
assert factory.call_args_list[1].kwargs["ids"] == ["1", "2"]
def test_batch_size_is_forwarded_to_the_rebuild(self, stores):
"""On a remote embeddings server the whole index is otherwise one POST."""
_, _, factory = stores
reembed.reembed_faiss("s1", batch_size=8, dry_run=False)
assert factory.call_args_list[1].kwargs["batch_size"] == 8
def test_existing_index_is_not_deleted(self, stores):
"""The rebuild must not destroy the old index before the new one exists."""
existing, rebuilt, _ = stores