Six vendor-run MCP servers (Notion, Linear, Atlassian, Sentry, Asana,
Stripe) join the catalog as presets from docsgpt/connectors/presets/
mcp.yaml; existing connections to those servers show under them.
connector_policies lets an admin turn a connector off or force whose
account every share uses, and app_metadata's
connectors.allow_custom_mcp turns custom MCP servers off. Both are
enforced on the server: new connections, OAuth sign-ins, MCP test and
save, and the tools of a disabled connector stop resolving.
GET/PUT /api/admin/connectors reads and changes them.
A shared tool's owner picks owner or member credentials with
PUT /api/connections/tools/<id>/credential-mode, and a member running
the owner's account always confirms write actions.
Tool-call events and retrieved chunks name the connector they came
from (key and display name, never the account), and tool calls keep
those fields when the conversation is reloaded.
chunks is a total per request, split across the attached sources, so an
agent with two sources and the default of 2 got a single chunk from each,
and its answers changed with whichever chunk won. 6 gives three per
source for about 4-5k more input tokens per retrieval turn.
Every literal default moves from 2 to 6: the request default, the
retrievers, the internal search tool, workflow agent nodes, scheduled and
headless runs, agent create/update/import, the source retrieval config and
the frontend forms. Existing agents and sources keep what they store; a
source saved with chunks=2 now counts as configured at 2, which is pinned
by a test.
Headless runs also read chunks=0 as unset (`or 2`), so an agent with
retrieval switched off retrieved anyway on scheduled runs; 0 now stays 0.
The dispatcher and each retriever's search open retrieval spans recording
sources, chunk counts, top scores and a chunk preview. The shared query
embedding, every per-source vector search and the prescreen rerank get
their own spans; pool workers carry the trace so they nest correctly.
Only a raise routed a source to the ClassicRAG fallback, and every graph read
logs its own failure and returns empty. So a query that broke, a half-built
graph and a walk that genuinely found nothing were indistinguishable, and each
made the source contribute nothing at all to the answer -- no fallback, no
vector blend, which is skipped by the same early return.
A graph source that produces no documents now joins the classic batch, exactly
as a source with no graph already does.
The passage stage also rescanned every node's chunk list once per candidate.
It inverts the links once instead: 7.3 ms to 0.16 ms on a 400-node subgraph at
the candidate cap, with identical output.
The three per-source graph options are read from the retrieval config the
Dispatcher hands over, and it only hands one over for a source it considers
overridden -- which it decided from chunks, score_threshold, rephrase_query
and prescreen alone. A source that changed only its graph options was not
"overridden", so nothing was carried and every graph source ran the defaults:
the UI toggles did nothing at all.
They count as an override now, for graphrag sources only. They mean nothing to
any other retriever, and an override also hands the source its own chunk
budget, which a classic source must not pick up from a graph setting.
Graph retrieval tied plain vector search at best and never beat it. Measured
across five corpora, the bottleneck was seeding, not the graph: the walk
started from nodes whose embeddings were computed from bare entity names, and
a whole question shares almost nothing with a name like "Quill".
Extraction now embeds each node from "name (type): description" and each
relationship as the fact it asserts ("Alder streams_to Quill: ..."), stored on
a new nullable graph_edges.fact_embedding column that ensure_vector_schema adds
in place. Entity names are canonicalised (case, punctuation, word breaks and a
cautious plural) so "VECTOR_STORE" and "vector stores" land on one node. Extraction calls run
concurrently (GRAPHRAG_EXTRACTION_WORKERS, default 8) while embedding and graph
writes stay serial on the task thread, so ordering and idempotency are
unchanged; that measured 8.4x faster with identical output.
Retrieval gains per-source options, stored under retrieval.graph and read live
at query time:
- seed_strategy: start from matching entities (default) or matching
relationships, which can reach an entity the question never names;
- passage_nodes (on): walk the source's passages alongside entities, with
PageRank damping 0.5 instead of 0.85;
- blend_vector (on): fuse the graph ranking with the source's vector ranking
by reciprocal rank.
The defaults are the measured-best configuration. Through GraphRAGRetriever,
the new seeding moved recall@4 from 0.41 to 0.68 on a multi-hop corpus and
from 0.50 to 1.00 on the docs corpus, and regressed none of the corpora
measured. Existing graphs keep name-only embeddings until rebuilt.
Three findings from review, all on code this branch introduced.
apply_chunk was not replay-safe. commit() can report connection loss *after*
Postgres committed, and the reconnect retry then replays the write: _upsert_node
bumps doc_freq a second time and _add_edge inserts another row, since
graph_edges has no uniqueness constraint for a logical edge. The chunk's
graph_ingest_progress row is now written in the same transaction as the rows it
describes, and a replay that finds it already "done" returns (0, 0) without
touching the graph. Extraction drops its separate mark_chunk("done"): the
checkpoint and the graph can no longer disagree.
count_nodes swallows every query failure and answers 0, so extraction's
"fall back to the write count" handler could never run — a failed count after a
successful build reported an empty graph. count_nodes grows a strict mode that
re-raises; retrieval keeps the swallow, which is what routes a source to
ClassicRAG.
A zero edge weight was read as a full-strength link: `or 1.0` rewrote an
explicit 0 before the <= 0 filter. Only missing and null weights default now.
The same coercion sat in _ppr_scores, where it would have kept the ranker's
rule unreachable from the product path, so it is fixed there too.
Review feedback on _write_with_reconnect: an explicit return inside the loop
plus an implicit fall-through past it reads as a path that returns None. The
retry is exactly two attempts, so say so — first attempt, reconnect on
connection loss, second and final attempt — and every path now returns or
raises.
Also narrows the transitions hint to the (node, weight) tuples it holds, and
logs at debug when the power iteration hits its iteration cap instead of
returning the last iterate with no trace.
networkx.pagerank delegates to a scipy implementation, and scipy is not a
DocsGPT dependency — it only arrives transitively through the optional
docling extra. In a default install every graph retrieval raised
ModuleNotFoundError inside _ppr_scores, hit the per-source except, and
degraded to ClassicRAG: the graph was built and paid for, then never used,
with one ERROR line per source per query as the only signal.
Rank with a local power iteration over the same row-normalized transition
matrix: undirected edges normalized per endpoint, dangling nodes
redistributed along the restart vector, and the restart vector normalized
across the nodes the subgraph actually holds so seed mass cannot leak.
Parity with networkx is asserted while scipy happens to be installed in the
test env, and the retrieval path is exercised with the import blocked.
About 85 call sites read a setting as getattr(settings, "NAME", fallback),
each carrying its own copy of the default. Every one of those names is a
field with a default on the model, so the fallback could never apply to
the real settings object; it only masked drift. Two had drifted:
- OPENAI_PROMPT_CACHE_KEY defaults to True on the model but the reader
fell back to False, and two test stubs relied on that.
- SharePoint's MICROSOFT_AUTHORITY fallback to
https://login.microsoftonline.com/<tenant> never fired, because the
attribute always exists (as None), so MSAL got authority=None. The
connector now derives the tenant authority when the setting is unset,
as its test always assumed.
Four places read EMBEDDINGS_KEY straight from os.environ, skipping the
"None"/"" normalisation the model applies; they read the setting now.
Test stubs that replaced a module's settings with a SimpleNamespace list
every setting the code under test reads.
The backend import package is now docsgpt, the name it will carry on PyPI;
application was far too generic to install into anyone's site-packages.
git mv plus a mechanical rewrite of every import, dotted string and path
reference: 734 Python files, the compose files, Dockerfile, workflows, docs,
setup scripts, devcontainer, k8s manifests, vscode config, pytest and coverage
config, .gitignore. Behaviour is unchanged.
Kept for one release:
- A top-level application package whose meta-path finder resolves
application.x.y to the already-imported docsgpt.x.y object, so old imports
and entry points (celery -A application.app.celery,
uvicorn application.asgi:asgi_app) keep working with a FutureWarning.
- Celery registers every application.* task name as an alias of its
docsgpt.* task on start-up, so messages queued by the previous release still
run. The redbeat key prefix moves to redbeat:docsgpt:v2: so schedule entries
the previous release wrote are left unread instead of firing twice.
The backend image builds from the repository root (docker build -f
docsgpt/Dockerfile .) so it can ship the alias package; a root .dockerignore
allow-lists docsgpt/ and application/ and keeps caches, local data, .env
files, the sample index files and the Dockerfile out. Compose and the image
workflows point at the new context.
arc53-machine