feat: per-source RAG configuration (retrieval strategies, chunking, exposure, prescreen)

Introduces a per-source config contract that makes RAG behavior strategy-dispatched instead of a single hardcoded path. Every source gains a validated JSONB config; an empty/absent config reproduces current behavior byte-for-byte, and the whole path is gated by PER_SOURCE_RETRIEVAL_ENABLED.

Foundation: sources.config JSONB column + migration 0022_source_config; SourceConfig/ChunkingConfig/RetrievalConfig pydantic models (strict on write, lenient on read); ChunkerCreator and RetrieverCreator.register registries; config threaded through the upload routes, ingest/remote/connector workers, and reingest.

Retrieval: a Dispatcher groups sources by retriever key (all-classic collapses to today's single ClassicRAG under one shared token budget; non-classic retrievers get their own instance), removing the previous single-global-retriever collapse in stream_processor. Per-source chunks, score_threshold (honored for pgvector/mongodb, safely ignored elsewhere), and rephrase_query toggle. New PATCH /api/sources/<id>/config with team-aware (effective_write_owner) authz and a requires_reingest signal.

Chunking strategies: recursive, markdown, parent_child (selectable per source; re-ingest to apply). Search exposure: per-source prefetch vs agentic_tool for agentic/research agents. Map-reduce prescreen: optional LLM relevance pre-filter implemented as a composable post-retrieval stage that wraps any retriever.

Backend and frontend (shared Retrieval options panel + edit modal) with tests; backend suite and frontend vitest green. Excludes the wiki and GraphRAG flagships.
This commit is contained in:
Alex committed 2026-06-20 21:54:23 +01:00
1 parent cfda009be0
commit f6400cd736
53 files changed
+4313 -74

No files matched your search

+19
View File
@@ -111,6 +111,25 @@ class TestPGVectorStoreSearch:
assert len(results) == 1
assert results[0].metadata == {}
def test_score_threshold_filters_by_distance(self):
# similarity = 1 - distance; threshold 0.85 → keep distance <= 0.15.
store, _, mock_cursor, _ = _make_store()
mock_cursor.fetchall.return_value = [
("close", {}, 0.10), # sim 0.90 → kept
("far", {}, 0.40), # sim 0.60 → dropped
]
results = store.search("query", k=5, score_threshold=0.85)
assert [r.page_content for r in results] == ["close"]
def test_no_score_threshold_keeps_all(self):
store, _, mock_cursor, _ = _make_store()
mock_cursor.fetchall.return_value = [
("a", {}, 0.10),
("b", {}, 0.90),
]
results = store.search("query", k=5)
assert len(results) == 2
@pytest.mark.unit
class TestPGVectorStoreAddTexts: