The agent and workflow reads now return resource_states to people who may
edit them: every attached tool, source and prompt (and workflow node tool
and source) with active or stopped and the reason: deleted,
owner_lost_access, the sponsor reasons, connection_needs_reconnect,
connection_removed or connector_disabled. Each entry names the sponsor,
someone other than the reader who can fix it, the service for a connection
reason, and whether the reader may take it over or reconnect it. When
something can be taken over, sponsor_audience says who it would reach.
The state comes from the checks the run itself uses (ref_access,
resolve_holder_tool and the tool's connection as the run resolves it), so
the page and the run can't disagree. A run that leaves a resource out logs
resource_stopped with the holder, type, id and reason.
The workflow read gives sponsor details, run state and node resource names
only to people who may edit it, and names only resources the workflow runs,
someone sponsored, or the reader can see. Owner saves and new workflows
now refuse node tools and sources the owner can't use, like editor saves.
Research steps ran each tool call straight through the executor, never
asking check_pause, so approval-gated actions (including a public-link
visitor's wiki edits) and an outside caller's writes on the owner's
accounts ran unchecked. A step can't pause, so it now checks each call
first and answers any pause with a refusal the model reads, journaled
like a headless denial. Headless and research runs share the journaling.
A paused turn is found by the agent owner's id, so anyone holding one of
the owner's agent keys could resume the owner's own chat with its saved
wiki edit rights. A resume now counts as an API or widget caller when
either the saved state or the resuming request is one, cuts the wiki tool
to wiki_view unless the wiki allows outside edits, and gives the tool
executor the same flags. A request that names an agent, by key or id, may
only resume that agent's turn; otherwise the claim is released and the
request refused.
Public-link visitors run as themselves and reach only wikis they may edit,
so the wiki switch no longer applies to them. Instead every wiki write in
a public-link run waits for the visitor's approval, so the agent owner's
prompt or sources can't steer an edit to the visitor's wiki unasked.
A run from an agent's API key or widget acts as the agent's owner, so it
could rewrite any wiki the owner can edit. A new per-wiki setting,
wiki_outside_edits (off by default), decides whether such runs, and runs
from the agent's public link, get the wiki's edit actions. While it is off
they are offered only wiki_view, and the tool refuses writes itself after
reading the live setting. The owner changes it through the owner-only
/api/sources/<id>/wiki/settings route; tokens can read it but not change it.
A scheduled or webhook run acts as the agent's owner with no one to
approve, so a public-link user or API-key caller could have the agent
schedule a write and have it run on the owner's accounts. Runs now keep
the caller's rules: a schedule set by someone who reaches the agent only
by its public link runs as a public-link caller, one set through the API
(recorded as created_via 'api', migration 0042) and every webhook run as
an external caller, each with the agent's API write allowlist.
/api/update_tool now checks submitted actions against the stored ones the
same way /api/update_tool_actions does: nothing can be added, and only the
tool's owner can change a fixed value. The tool type (name) can no longer
be changed after creation by anyone.
For API tools, changing who fills an existing header, query or body
parameter, its value, or clearing a stored value is now owner-only on both
/api/update_tool and /api/update_tool_config. Before, an editor could hand
a stored secret query value to the model and read it back in the chat.
The chat now masks every value that came from the stored action rather
than from the model.
A connection-backed MCP tool can no longer be moved to another server or
sign-in method through /api/update_tool or /api/update_tool_config (400,
pointing to /api/mcp_server/save), and a new key saved through
/api/update_tool_config goes to its connection, owner only.
Main's access model (team editors and viewers, edit_credentials, owner-only
OAuth servers, per-user tool preferences, resource sponsors) now applies to
connection-backed tools and sources. Our migrations are renumbered to
0040_connections and 0041_connection_account_name, after main's
0038_resource_access_settings and 0039_resource_sponsors.
Where the two sides met: MCP tools save and load their connections as the
tool owner, editors may add a key (a new connection on the owner's account)
but never rewrite an existing connection's secret, fixed values stay
owner-only, and a member's own connection is used in member mode.
A value the owner fixed may be a secret, and tool-call events reach whoever
runs the agent, so query and body values the owner fixed now show as
(fixed); a value the connection sets, like Telegram's default chat, still
shows. sent_arguments is saved with the conversation, so a reopened chat
shows the same arguments as the live one.
The MCP SDK knows a token's expiry only when it obtained the token in the
same process. A worker or a restarted API loading tokens from the
connection sent an expired one, got a 401 and asked the owner to sign in
again. The expiry is now saved with the tokens and handed back to the SDK,
which then renews the token with its refresh token first; a token saved
before expiries were kept is renewed once. A sign-in that cannot be
renewed raises MCPReauthorizationRequired.
The approval card and a finished call showed what the model asked for,
even where a fixed value replaced it. Calls now carry sent_arguments for
the chat, leaving headers out since they may hold a fixed secret. The
model's own arguments are kept as they were, because they are what later
turns replay to it and it never sees fixed values.
A Telegram connection can now hold an optional default chat ID. When it
is set, both Telegram actions send there, the model is no longer asked
for a chat, and a chat it names anyway is ignored. In member mode each
member's own connection supplies their own chat. The same bot with a
different chat is a separate connection.
A parameter hidden from the model with a stored value is now always sent
with that value; a model that names the key anyway (by mistake, or because
a prompt told it to) no longer overrides it. 0 and false are real fixed
values; an empty value still means the parameter is left out.
The cache key held the first ten characters of a bearer token or API key.
Every fine-grained GitHub token starts with github_pat_, so two users of
the same server shared one cached client, and with it the first user's
token, for up to five minutes.
mcp 2.x reads code, state and the RFC 9207 iss off the callback handler's
result; the handler still returned a (code, state) tuple, so every MCP
sign-in failed after the user approved it. The callback route now keeps the
issuer too, since servers that advertise it (Linear) refuse a sign-in
without it.
A page opened on a plain-HTTP address registered its own origin as the
callback, which servers such as Linear refuse. MCP_OAUTH_REDIRECT_URI now
wins when set; the page's value is the fallback.
Migration 0038 moves every stored secret (OAuth tokens, MCP OAuth
tokens and client registrations, API keys) into the connection's
encrypted envelope, links API-key tools to one connection per distinct
credential, allows several accounts per provider, and adds
credential_mode to sources and tools. OAuth MCP tools keep resolving
each member's own token, as they did before.
docsgpt.connectors.service is now the only reader of OAuth tokens:
get_valid_token_info refreshes under a row lock and persists rotated
refresh tokens, and a revoked grant flags the connection, pauses its
sources and notifies the owner. Loaders build from a connection
(BaseConnectorLoader.from_connection), so scheduled sync covers Drive,
SharePoint and Confluence sources with no browser. S3 and Reddit keys
stay on the connection instead of in remote_data.
New endpoints: POST /api/connections, /setup, /reconnect,
/picker-token, /claim, DELETE /api/connections/<id>, per-action
permissions and MCP refresh-tools. Upload, file listing, sync and
validate-session take a connection_id; session tokens keep working for
this release. The tool executor reads credentials from the resolved
connection (owner or member mode) and pauses on a Connect card when a
connection needs signing in. docsgpt connectors reencrypt rewrites
stored credentials after a key rotation.
chunks is a total per request, split across the attached sources, so an
agent with two sources and the default of 2 got a single chunk from each,
and its answers changed with whichever chunk won. 6 gives three per
source for about 4-5k more input tokens per retrieval turn.
Every literal default moves from 2 to 6: the request default, the
retrievers, the internal search tool, workflow agent nodes, scheduled and
headless runs, agent create/update/import, the source retrieval config and
the frontend forms. Existing agents and sources keep what they store; a
source saved with chunks=2 now counts as configured at 2, which is pinned
by a test.
Headless runs also read chunks=0 as unset (`or 2`), so an agent with
retrieval switched off retrieved anyway on scheduled runs; 0 now stays 0.
gen_continuation is not wrapped by @log_activity, so the llm_* log lines
from a resumed turn carried no user, agent, conversation or endpoint. Every
OpenAI-compatible /v1 client-tool round landed anonymous in the logs.
Add agent_log_context, which binds the same keys log_activity binds while
keeping any enclosing activity_id, and wrap gen_continuation in it. Both
now derive the keys from one helper.
Guardrail evaluations are traced when they call a remote check or fire; a
firing guardrail drops every content preview from the stored trace. Each
workflow node and each research phase is a step span, and the workflow run
id links the trace to its Logs row.
sum_tokens_in_range now skips the scheduler's per-run rollup rows, whose
tokens are already on the run's per-call rows, so the per-agent 24h limit
and the admin total no longer count scheduled spend twice.
Workflow node LLMs carry the workflow agent's id, so their usage rows are
attributed to the agent instead of landing with a user id only.
usage_totals returns a user's tokens and cost since a window start, split
by interactive and agent-key traffic.
The graph tool records every page read_entity_pages returns in
retrieved_docs, but agents only ever collected internal_search's. A turn
answered from graph pages therefore emitted no sources: on the multi-hop e2e
run, a question answered from two read_entity_pages calls came back with none.
_search_tool_docs gathers both tools' documents the way the executor caches
them, and both the classic/agentic collector and the research agent's
per-step citations use it. Checked against the real executor and a real
graph: a graph-only turn now cites the pages it read.
Celery records the executing task on the thread that runs it, so a thread that
task starts sees none. The embeddings client and read_document both decided
"am I in a worker?" from that alone, and from any other thread took the
web-process branch: dispatch to the worker they were running in and block on
the result. Celery refuses that get() ("Never call result.get() within a
task!"), so the embed failed and latched the 30s dispatch cooldown for every
caller after it; with joins allowed, read_document would instead wait on a
parsing queue only its own busy process serves.
Threads inside tasks are not hypothetical: per-source retrieval fans out to a
pool, so a scheduled or webhook agent searching several sources embedded from
pool threads. Graph extraction did too, which failed every chunk of a build.
in_worker() in celery_init answers for the whole process. Celery's
task_join_will_block is process-wide and set for every blocking pool (prefork,
solo, threads) -- exactly the condition under which dispatch-and-wait goes
wrong; eventlet/gevent leave it unset, so the task's own thread still counts
through current_worker_task. Verified with real workers on each blocking pool:
from a thread a task started, the old check dispatched and hit the error, the
new one embedded locally.
The backend import package is now docsgpt, the name it will carry on PyPI;
application was far too generic to install into anyone's site-packages.
git mv plus a mechanical rewrite of every import, dotted string and path
reference: 734 Python files, the compose files, Dockerfile, workflows, docs,
setup scripts, devcontainer, k8s manifests, vscode config, pytest and coverage
config, .gitignore. Behaviour is unchanged.
Kept for one release:
- A top-level application package whose meta-path finder resolves
application.x.y to the already-imported docsgpt.x.y object, so old imports
and entry points (celery -A application.app.celery,
uvicorn application.asgi:asgi_app) keep working with a FutureWarning.
- Celery registers every application.* task name as an alias of its
docsgpt.* task on start-up, so messages queued by the previous release still
run. The redbeat key prefix moves to redbeat:docsgpt:v2: so schedule entries
the previous release wrote are left unread instead of firing twice.
The backend image builds from the repository root (docker build -f
docsgpt/Dockerfile .) so it can ship the alias package; a root .dockerignore
allow-lists docsgpt/ and application/ and keeps caches, local data, .env
files, the sample index files and the Dockerfile out. Compose and the image
workflows point at the new context.
Source access control
---------------------
`active_docs` is client-supplied and reached the retriever unchecked, and the
retriever queries `WHERE source_id = <id>` with no owner predicate — so any
caller could pass any source id to /stream or /api/answer and have another
tenant's documents quoted back, while /api/sources/<id>/search correctly
refused the same id. Gate it through `can_access`, the helper the guarded
endpoints already use, and filter `self.source` down to the authorized set.
Fails closed: no principal, or a check that errors, drops the source.
Three sibling paths had the same gap:
- workflow agent nodes: `AgentNodeConfig.sources` is written verbatim from
client JSON at save time and nothing validated it, so a node could name any
tenant's source. Gate against the workflow owner, so shared workflows keep
reading their owner's sources like shared agents do.
- /api/share: `_resolve_source_pg_id` resolved any id with no ownership
predicate and baked it into the agent the share creates; /api/search then
searched it. Authorize before attaching.
- search_service: re-resolve the ids stored on an agent row instead of
trusting them, so a row written by any future path with the same gap cannot
be read back.
Team grantees previously lost their source's retrieval config: the post-check
read was still owner-scoped, so it missed and fell back to defaults (an
`agentic_tool` source was bulk-prefetched for every grantee). Read unscoped
after `can_access` passes.
Retrieval
---------
`PGVectorStore._ensure_table_exists` created an IVFFlat index on the empty
table it had just created. IVFFlat computes centroids at build time, so those
centroids were random, and combined with the `source_id` post-filter a source
with hundreds of embedded chunks returned zero rows — retrieval reported no
documents, the model answered from memory, and nothing was logged. Stop
creating the index (exact search is correct and fast well past the sizes most
deployments reach); raise `ivfflat.probes` to sqrt(lists) where an index still
exists; and re-run a short indexed search exactly, since post-filtering means
no index setting can guarantee a full result. `graphrag` had the same
empty-table index with no fallback at all.
Also: bound `chunks` to 0-500 on both the request and agent paths (0 still
means "skip retrieval"), let a source's configured `retrieval.chunks` outrank
the request body, and cap ClassicRAG's per-source floor at
max(top_k, n_sources) so attaching sources cannot inflate the result set.
Silent failures
---------------
An empty retrieval was invisible to both the model and the client: the `source`
event was suppressed when the list was empty, so "searched and found nothing"
looked identical to "no source attached", and the prompt said nothing at all.
Emit the event always, and tell the model when a search ran and returned
nothing. A file that parses to nothing now fails ingest with a message naming
the cause instead of storing an embedding of the empty string. `score_threshold`
returns warnings when the active store or retriever cannot honour it.
Prompt structure
----------------
Retrieved documents move from the system prompt into the user turn, with the
injection guard restated next to them: they change every turn (defeating prefix
caching), they are third-party text that should not carry system authority, and
routing them through the query budget makes them truncatable rather than
silently crowding it out. Documents are shed lowest-ranked-first before the
question is touched.
The six chat presets (3 tones x 2 retrieval modes) differed only in their
Answering section; they are now composed from single-source fragments at load
time, not through Jinja inheritance, which would have opened a file-read
surface in the template sandbox and broken the tool-prefetch parser. Per-tool
guidance moves out of the prompt into tool schemas, so it travels with the tool
and cannot render when the tool is absent. A plain-text custom prompt is staged
as a persona value inside the skeleton instead of replacing it wholesale — it
used to silently lose the injection guard, platform block, memory and
attachments, and its braces are now inert.
Other fixes
-----------
- agents/base: an oversized system prompt drove the query budget negative and
dispatched a full-price request with an empty question; raise instead.
- llm/anthropic: migrate off the retired Text Completions API. It flattened
history to first+last message and ignored tools entirely. Adds the missing
Anthropic handler, without which every tool call was silently dropped.
- sources/upload: `sitemap` had no branch, so every sitemap ingest died on a
TypeError; `validate_url` now rejects a falsy URL cleanly.
- workflow nodes: retrieved documents never reached the node agent, so a
classic node with a source and an ordinary prompt answered "I have no
documents" while the run reported completed.
- parser/bulk: copy the metadata dict, or every chunk reports the last chunk's
token_count.
- crawler_loader: carry the page title, or citations render the whole chunk
body as the label.
Two follow-ups from PR review.
_fit_attachment_text tokenized the extraction twice on the oversized
path — once for the budget check and once for chars_per_token. That is a
full BPE pass over the whole document (~12ms per 250k chars) on the hot
path of every attachment turn. Compute it once.
The CEL error summarizer kept quoted fragments that looked like
identifiers, on the theory that those are variable names rather than
data. They are not distinguishable: celpy interpolates state values into
messages (`StringType('SECRET')`), and a one-word user query such as
`invoice` is a valid identifier, so it survived redaction and reached
both the log line and — since config errors now bypass
sanitize_api_error — the user. The earlier test passed only because its
sample value contained hyphens.
Quoted fragments are now redacted by default, allowing only the two
positions known to hold names rather than data: the variable in
"undeclared reference to 'x'" and the class in "<class 'ValueError'>".
Tests cover identifier-shaped values, that the exception class survives,
the truncation backstop, and the empty-expression guard.
Workflows ship two expression languages in adjacent fields. Agent
`prompt_template` and end `output_template` are Jinja2, so `{{query}}` is
correct there. State and condition `expression` fields are bare CEL,
where the same string is a parse error.
Nothing told users this. docs/content/Agents/nodes.mdx documented the
pre-CEL Set State node with `{{variable_name}}` examples and Set /
Increment / Append operations that no longer exist — "CEL" appeared
nowhere in the docs. The builder's own placeholder was `input.foo + 1`,
referencing an `input` namespace that is not in workflow state, while the
panel beside it advertised `{{ agent.variable }}`.
There was also no validation anywhere: neither validateWorkflow nor
validate_workflow_structure looked at state nodes at all, and conditions
were only checked for a non-empty string. A workflow containing
`{{query}}` saved and published clean, then aborted on the first message
with a bare caret dump that sanitize_api_error collapsed into "An error
occurred … please try again later" — advice that sends the user round the
same loop, which is what the 2026-08-01 report shows.
Adds validate_cel_expression (compile-only, since state is built at run
time and unresolved names are not knowable when saving) and wires it into
the workflow save path for both node types, plus the half-configured
state operations the engine silently skips. `{{ }}` gets a targeted hint,
raised only after compilation has already failed so valid CEL that
contains braces — `x == "{{y}}"`, or a nested map literal — is not
rejected.
celpy errors are now summarized: undeclared-reference messages embed a
repr of the entire activation, thousands of characters including the
user's own query, and others interpolate the offending state value.
Quoted fragments that are not bare identifiers are redacted, since
config errors now bypass sanitize_api_error to reach the user, and a
shared agent's runner is not its owner.
Docs rewritten with a table of which field takes which syntax, CEL
examples for Set State, and the Condition node section that basics.mdx
has been linking to all along.
/api/models reports `display_provider` when a catalog YAML sets one
(`foundry`, `azure_foundry`, `cloudflare`), and the builder persists that
string as a node's `llm_name`. The engine passed it straight to
LLMCreator, which only knows the names in PROVIDERS_BY_NAME and raises
`No LLM class found for type <label>`.
That fails the node before any LLM call, so the turn ends in ~60ms with
an empty answer and no tokens generated. It hits the *default* path: the
platform-default model's label is stamped into every newly dragged agent
node and validateWorkflow requires an agent node, so a new user's
untouched workflow could not produce a token regardless of what they
typed. Ordinary chat was unaffected because it resolves the provider from
the model registry and never reads the stored name.
Adds `resolve_dispatch_provider`, which prefers a stored name that is a
real dispatch provider, then the registry lookup, then the parent agent.
Nodes already saved with a label are repaired at run time, so no
migration is needed. The api_key now resolves from the normalized name
too — `get_api_key_for_provider` falls back to settings.API_KEY for names
it does not recognize, which would have sent the deployment key to
whatever endpoint the label happened to select.
arc53-machine