mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-17 04:14:29 +00:00
feat: per-source RAG configuration (retrieval strategies, chunking, exposure, prescreen)
Introduces a per-source config contract that makes RAG behavior strategy-dispatched instead of a single hardcoded path. Every source gains a validated JSONB config; an empty/absent config reproduces current behavior byte-for-byte, and the whole path is gated by PER_SOURCE_RETRIEVAL_ENABLED. Foundation: sources.config JSONB column + migration 0022_source_config; SourceConfig/ChunkingConfig/RetrievalConfig pydantic models (strict on write, lenient on read); ChunkerCreator and RetrieverCreator.register registries; config threaded through the upload routes, ingest/remote/connector workers, and reingest. Retrieval: a Dispatcher groups sources by retriever key (all-classic collapses to today's single ClassicRAG under one shared token budget; non-classic retrievers get their own instance), removing the previous single-global-retriever collapse in stream_processor. Per-source chunks, score_threshold (honored for pgvector/mongodb, safely ignored elsewhere), and rephrase_query toggle. New PATCH /api/sources/<id>/config with team-aware (effective_write_owner) authz and a requires_reingest signal. Chunking strategies: recursive, markdown, parent_child (selectable per source; re-ingest to apply). Search exposure: per-source prefetch vs agentic_tool for agentic/research agents. Map-reduce prescreen: optional LLM relevance pre-filter implemented as a composable post-retrieval stage that wraps any retriever. Backend and frontend (shared Retrieval options panel + edit modal) with tests; backend suite and frontend vitest green. Excludes the wiki and GraphRAG flagships.
This commit is contained in:
1 parent
cfda009be0
commit
f6400cd736
53 files changed
+4313
-74
No files matched your search
@@ -111,6 +111,25 @@ class TestPGVectorStoreSearch:
|
||||
assert len(results) == 1
|
||||
assert results[0].metadata == {}
|
||||
|
||||
def test_score_threshold_filters_by_distance(self):
|
||||
# similarity = 1 - distance; threshold 0.85 → keep distance <= 0.15.
|
||||
store, _, mock_cursor, _ = _make_store()
|
||||
mock_cursor.fetchall.return_value = [
|
||||
("close", {}, 0.10), # sim 0.90 → kept
|
||||
("far", {}, 0.40), # sim 0.60 → dropped
|
||||
]
|
||||
results = store.search("query", k=5, score_threshold=0.85)
|
||||
assert [r.page_content for r in results] == ["close"]
|
||||
|
||||
def test_no_score_threshold_keeps_all(self):
|
||||
store, _, mock_cursor, _ = _make_store()
|
||||
mock_cursor.fetchall.return_value = [
|
||||
("a", {}, 0.10),
|
||||
("b", {}, 0.90),
|
||||
]
|
||||
results = store.search("query", k=5)
|
||||
assert len(results) == 2
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
class TestPGVectorStoreAddTexts:
|
||||
|
||||
Reference in new issue
Block a user