run_agent_headless records each unattended run under its endpoint; the
scheduler passes its run id and the webhook worker its task id so Logs rows
can find their trace, while the LLM's own request id stays untouched for
quota counts. /api/search and MCP search_docs record their retrieval, and a
graph build records every extraction call under one step.
One row per execution holding its span tree as JSONB, linked to Logs rows
by request, message, activity and workflow-run ids. message_id cascades so
deleting or truncating a conversation drops its traces; a daily beat task
enforces TRACES_RETENTION_DAYS.
`float("inf") > 0` is true, so a stored `"inf"` was preserved as if it
were a real count and reached the UI as "∞". Require a finite positive
value, which also covers `nan` explicitly rather than by accident.
Chunk cards in the source viewer read `metadata.token_count` and print a
bare "-" whenever it is absent. Ingestion records the count, but chunks
indexed before it was written -- and any path that rebuilds a chunk's
metadata from scratch -- reach the UI without it, so the whole file shows
"-" with no way to recover the number short of a re-ingest.
Fill the key in on the way out when it is missing or unusable (empty,
non-numeric, zero or negative), counting only the page being returned so
the cost is bounded by `per_page`. A count that is already recorded is
left untouched, including a numeric string, since stores round-trip
metadata types differently.
The fallback counts in cl100k rather than the embedding model's
tokenizer: loading that tokenizer can reach for a Hugging Face download,
which does not belong in a request path.
Correctness
- /api/remote never recorded source.created, so URL, GitHub and connector
sources had a source.deleted with no matching creation. All three creation
paths now go through one _audit_source_created helper.
- The prompt-cache rate divided cached tokens by a whole bucket's prompt
tokens. A bucket is a day and mixes calls whose provider reports a cache
breakdown with calls whose provider does not, so filtering buckets in the
client could not separate them and the rate was understated by however much
traffic ran on a non-reporting provider. The denominator is now computed in
SQL over the reporting rows.
- The outcome pill matched values nothing writes. Guardrails emit triggered /
not_evaluated and the device feed emits dispatched; the map had blocked /
denied / allowed, so a guardrail that fired rendered neutral grey -- the one
signal the merged feed exists to surface. Fixtures were seeding the
fictional values, so the tests passed on it too.
- Stream duration_ms timed the consumer. stream_token_usage is a generator,
so start-to-exhaustion includes the agent loop's tool handling and the SSE
client's pace; a slow browser recorded ~30s for a sub-second call. It now
accumulates only the time spent inside next().
Safety
- Activity filters failed open: an unknown facet or unparseable timestamp was
dropped, and no filter means every row, so a typo widened an audit view and
on the export streamed the full history. Both are now a 400.
- The search term was interpolated into an ILIKE pattern, so "100%" matched
everything and "q1_report" matched more than it should. Escaped.
- 0034 set actor_id NOT NULL with no default. A previous-release process
inserting mid-rollout would raise, and in admin/routes.py that insert shares
the request transaction, so a role grant beside it would roll back too.
Noise and dead code
- The per-user panel is a security panel: data-plane events file under the
actor, so an active account's routine deletes pushed a denied login out of
the 20-row window. It now excludes them; the Activity tab shows everything.
- device_audit_log had no created_at-leading index, so the merged feed
sequentially scanned that branch every page (migration 0036).
- conversation.deleted_all no longer records when nothing was deleted, and
agent.updated no longer records an empty field list.
- Dropped by_model from /admin/usage (no consumer; an extra aggregate per page
load), the duplicate filter surface on AuthEventsRepository that nothing
called, and the unreachable FLOW_LABELS.schedule entry.
- Type hints on record_event's conn and the remaining unannotated helpers.
Review pass over the preceding commits.
- The activity export paged by OFFSET. The journals are append-only and the
feed is newest-first, so rows written mid-export shift the window down and
repeat rows already emitted. Pages by keyset on (created_at, feed, id) now.
- top_token_users and the per-user breakdown counted run-level rollup rows.
A scheduled run already has a row per LLM call, so its spend was billed
twice -- invisible while the column was tokens, obvious once it was dollars.
Both now exclude them, matching every other spend query.
- total_cost was summed from the per-model split, which drops rows with no
model_id and so undercounted the figure printed above the series. Summed
from the series instead.
- The CSV export encoded its detail cell without the fallback the NDJSON
branch had, so a non-JSON-native value would have failed the stream.
- The activity view reset pagination in an effect, which fetched the stale
page against the new filters before fetching again. Reset in the setters.
- The audit taxonomy is no longer re-exported from the helper module; the one
caller that wanted it imports from where it lives.
Logins, role grants and provisioning were audited from the first release;
creating and deleting sources, agents, agent keys and conversations were not.
An operator reviewing the trail could see who signed in but not who deleted
the source they were asking about.
Adds docsgpt/api/audit.py: one helper that records an event inside the
caller's transaction, in a savepoint, swallowing failures -- an audit write
must never be able to fail the action it describes -- and tolerating the
absence of a Flask request so a Celery task can record too.
Events added: source.created (upload and wiki), source.deleted,
source.reingested, agent.created, agent.updated, agent.deleted,
agent.key_regenerated, conversation.deleted, conversation.deleted_all.
agent.updated records field names only, never values, which can carry prompts
and credentials.
The same module carries the event -> category map (identity / access / config
/ data) that the admin activity feed filters on.
auth_events.user_id was overloaded and meant something different per call
site: admin mutations filed the row under the target user and hid the acting
admin in metadata->>'by', team events filed it under the actor, and quota
events switched between the two depending on scope. The practical consequence
was that "show me everything admin X did" had no answer.
Adds actor_id (never NULL) and target_id (NULL when the event is not about a
user) alongside the existing column, which keeps its meaning as the per-user
feed key so nothing historical is rewritten. The backfill recovers the actor
from metadata for rows that predate the columns.
Also adds the indexes the cross-user admin feed needs. The only index was
(user_id, created_at DESC), so the global feed -- which orders by created_at
with no user predicate -- fell back to a sequential scan plus a sort on every
page.
The feed gains actor, multi-event, until and free-text search filters, plus a
distinct-event catalogue so the UI can offer what the instance recorded
instead of asking operators to type a name from memory.
Admins read and set the instance default, team allowances and user
overrides under /api/admin/quotas. A user's endpoint also returns the
limits those layers resolve to, the layer each came from and the usage
against them. The overview lists catalog models used this period that have
no price, since a cost limit cannot see them. Every write is audited.
GET /api/user/quota gives a user their own limited buckets, usage and reset
time without naming the policies behind them; any valid token may call it.
check_usage now checks the billable user's quota on every request, before
the per-agent 24h limits, which keep applying to traffic through an agent.
Until now a request without an agent key skipped every limit. A refusal is
a 429 with Retry-After and a body naming the budget, usage, limit, the
layer the limit came from and when it resets.
Headless runs check the agent owner's quota before starting. A refused
scheduled run is recorded as budget_exceeded; a refused webhook run returns
a quota_exceeded result instead of raising, so Celery does not retry it.
A resource restriction was checked on ids in the request, not on what the
addressed row pulls in or belongs to. Closed:
- Workflow writes for tokens restricted on sources, tools or prompts (a graph
names those inside its nodes), and attaching a workflow to an agent unless
the token is restricted on workflows too.
- Chat for tools-restricted tokens (chat executes tools; rejected at token
creation as well), and agent-less chat for tokens restricted on prompts or
workflows.
- conversation_id on chat: it must belong to the agent being run, or to no
agent for agent-less chat. Otherwise the server continued, appended to, or
resumed pending tool calls of another agent's conversation.
- Schedules for tokens restricted on anything but agents; schedule-id routes
for every restricted token.
- Conversations and analytics for every restricted token, not only
agent-restricted ones.
Also: create, first publish and adopt return the agent API key masked to a
token without agents:keys; token ids must be canonical UUIDs (urn:uuid: gave
a 500); an expired token is reported as expired; token creation takes a
per-user advisory lock so the cap cannot be raced; admin revoke-sessions
writes a pat_revoked event per token. The UI drops a row whose revoke returns
404 and does not offer a tools restriction next to chat:run.
Message tail and the ASGI reconnect stream cannot tie a message to an
allowlist, so any token with a resource filter is refused there. The tools
listing now applies the allowlist to default and builtin rows as well. Token
creation answers 400 instead of 500 for a JSON body that is not an object.
A central rule table maps each route and method to the scope a token needs;
a route that is not listed cannot be called with a token, and a test fails
when a registered route is left unclassified. Token management, admin, team
management, sign-in, device pairing and OAuth handshakes are never token
reachable, and a token never carries the admin role.
A token restricted to specific agents, sources, prompts, tools or workflows
is held to its allowlist: ids are checked wherever a route carries them,
listings are filtered, creation is refused, and routes whose rows cannot be
tied to the allowlist are closed. Agent import checks the resolved target.
handle_auth resolves a dgpt_pat_ bearer against the database instead of
decoding it as a JWT, for both the Flask and the ASGI routes. Scopes and the
resource filter always come from the token row, and the claims that mark a
PAT are stripped from decoded JWTs so a session token cannot pose as one.
ASGI routes reject tokens unless they name the scope that admits them, and
/api/user/me reports what the calling token may do.
Returning a deferred marker traded one wrong signal for a worse one. A
redelivery reuses the original task id — Context.as_execution_options carries
task_id into the retry — so the duplicate's return marked the very id the
client polls as SUCCESS. /api/task_status reports celery's state verbatim and
the UI maps SUCCESS to "done", so the GraphRAG enable modal would announce a
finished build, rendered from a payload with no counts, while the run holding
the lease was still extracting.
Raise Ignore instead: celery records no state for the duplicate, so the task
id keeps whatever the holder sets and the poller keeps waiting. The autoretry
wrapper re-raises Ignore ahead of autoretry_for, so the wider
autoretry_for=(Exception,) on these tasks cannot turn it back into a retry.
A task that runs longer than the broker's visibility timeout is redelivered
while its first run is still going. The idempotency lease correctly stops the
duplicate from doing the work, but the duplicate then re-queued itself once
per LEASE_TTL until celery ran out of retries and raised
MaxRetriesExceededError — so a perfectly healthy long task (a large graph
extraction is the one that found this) reported a task failure, with a
traceback, while the real run was still making progress next to it.
Catch the exhaustion and return a "deferred" result instead. A normal
deferral still re-queues: only the give-up path changes, and the lease
holder's dedup row is left untouched so its own completion still records.
About 85 call sites read a setting as getattr(settings, "NAME", fallback),
each carrying its own copy of the default. Every one of those names is a
field with a default on the model, so the fallback could never apply to
the real settings object; it only masked drift. Two had drifted:
- OPENAI_PROMPT_CACHE_KEY defaults to True on the model but the reader
fell back to False, and two test stubs relied on that.
- SharePoint's MICROSOFT_AUTHORITY fallback to
https://login.microsoftonline.com/<tenant> never fired, because the
attribute always exists (as None), so MSAL got authority=None. The
connector now derives the tenant authority when the setting is unset,
as its test always assumed.
Four places read EMBEDDINGS_KEY straight from os.environ, skipping the
"None"/"" normalisation the model applies; they read the setting now.
Test stubs that replaced a module's settings with a SimpleNamespace list
every setting the code under test reads.
- Ship tiktoken's cl100k_base inside the package and build the encoding
from it, so token counting never downloads anything.
- Default EMBEDDINGS_CACHE_DIR to <data home>/models instead of FastEmbed's
temp dir, and read tokenizer.json and repo metadata from that cache, so
a model downloads once and survives reboots.
- TTS_PROVIDER=none and STT_PROVIDER=none switch the speech features off:
the endpoints return 404, audio files fail to ingest with a clear
message, /api/config reports tts_available/stt_available, and the UI
hides the Speak and microphone buttons.
- Drop the Google Fonts Roboto import from the web UI.
- prefetch-models fills the cache the app reads; verify-offline checks the
packaged encoding.
- Docs: new Air-Gapped Deployment guide, settings and cache notes.
The connector OAuth popup posted the session token to window.opener with
a '*' target origin, so any page that opened the popup received it. An
attacker with an account on a multi-user deployment could start a flow for
their own pending session, get a victim to finish the provider consent, and
receive a token backed by the victim's Drive/SharePoint/Confluence tokens.
- Post popup results only to allowed frontend origins: the callback origin,
OIDC_FRONTEND_URL, the new CONNECTOR_ALLOWED_ORIGINS, and localhost:5173
when the callback runs on a loopback host.
- Render the success page from the callback itself so the token never
appears in a URL; callback-status ignores session_token/user_email params.
- ConnectorAuth accepts messages only from the popup it opened, on the
callback origin reported by /api/connectors/auth.
- /api/connectors/disconnect requires auth and only deletes the caller's
session.
- /api/connectors/sync and /api/remote reject session tokens the caller
does not own.
Fixes#2766
- Per-user SSE cap uses per-connection leases in a sorted set instead of
a shared INCR/DECR counter with a TTL. A stream that outlived the TTL
could let the counter expire, then decrement another stream's slot or
drive the count negative. Leases refresh while a stream sends frames
and age out when a stream dies without cleanup.
- Shielded cleanup is bounded per step: stream close and on_close in
ClosingStreamingResponse, unsubscribe and close in AsyncTopic, lease
release, and the replay-budget check, so a dead Redis connection can't
hold a request or a graceful shutdown.
- Artifact downloads send an ASCII filename plus an RFC 5987 filename*
for non-ASCII names, build headers before opening the file, and close
disk handles off the event loop.
- Type hints and docstrings on the new helpers.
Move GET /api/events, the remote-device command stream and artifact
downloads from Flask to Starlette routes mounted ahead of the Flask
catch-all. On Flask each held an a2wsgi threadpool slot for as long as
its response stayed open, and because uvicorn drops writes after a
client disconnects, a closed tab never released it.
- asgi_auth: one JWT/OIDC gate for Starlette routes; the chat reconnect
reader uses it too
- ClosingStreamingResponse closes the body iterator and releases the
SSE slot or file handle even when the client leaves before the first
frame
- AsyncTopic liveness probe replaces the sync client's socket_timeout
guard against half-open pub/sub sockets
- ASYNC_REDIS_MAX_CONNECTIONS sizes the async Redis pool; every open
stream holds a connection and redis-py defaults to 100
PinAgent looked agents up with no ownership, share or team predicate, and
PinnedAgents returned those rows through a hand-rolled dict that bypassed
the blanking _format_agent_output applies. Anyone holding an agent id --
a share-link recipient learns the real UUID -- could pin it and keep
reading its name, description, prompt, tools, type, status and masked key
after the share was revoked.
Add _user_may_pin: the owner, a team grantee, a recipient of a still-live
share link, or a system template. PinAgent checks it before pinning and
PinnedAgents re-checks every row on read, so a revoked grant drops out of
the list instead of persisting. Unpinning stays open regardless of current
access, or a revoked share would strand a pin the user cannot clear. The
masked key is now owner-only, matching _format_agent_output.
Adds the first cross-user pin tests: every existing one pinned the
caller's own agent.
Review follow-ups:
- The agents list and the pinned list dropped agents that had neither a
source nor a retriever. Publishing such an agent is now allowed, so it
vanished from the list right after publish. Both filters are gone; a
source-less agent skips retrieval and is still runnable. Tests cover
both listings.
- The agent form selected an uploaded source by diffing the source
catalog before and after the upload, which would also pick up any
source created or shared in the meantime. Upload now passes the id of
the source it created to onSuccessfulUpload, and the form selects only
that id.
The sources list used to start with a fake "Default" entry that had no id
and, at run time, meant "no source, skip retrieval". The agent form
pre-selected it, snapped back to it when the last source was deselected,
and refused to publish without it, so a new agent always looked like it
had a knowledge base when it had none.
Backend
- /api/sources returns only ingested sources; no placeholder row.
- Publishing an agent no longer requires a source on create or update.
The legacy "default" value is still accepted and maps to NULL.
Frontend
- The agent source picker starts empty, can be cleared, and shows a hint
that a source-less agent answers from the model and its tools only.
- The picker groups sources into "Your sources" and "Shared with team"
when any team-shared source exists, shows "N sources selected" for a
multi-selection, and gets the same "Go to Sources" / "Upload new"
footer as the chat picker. A source uploaded from the form is selected
when it lands.
- Source selection serialisation and the picker id live in one helper
shared with the chat picker; the four copies in the form are gone.
- The client no longer seeds a placeholder source in the store, and the
dead auto-select of a "default" document is removed.
pip install docsgpt (extras: docling, milvus) installs the backend with a
docsgpt command: api, worker, migrate, prefetch-models, verify-offline,
reembed. Second step of the PyPI work after the package rename.
- hatchling build; the version comes from docsgpt/version.py. The wheel is
the docsgpt package with the data it reads at runtime (prompts, model
catalogs, seed config, alembic.ini and migrations) and without the
Dockerfile, the exported requirements, the sample index and local runtime
data. The application import alias stays checkout-only. uv sync installs
the package editable now that [tool.uv] package = false is gone.
- docsgpt/cli.py: api (gunicorn + BoundedDrainUvicornWorker with the image's
flags, --reload for uvicorn), worker (Celery worker with beat embedded,
--no-beat/-Q/--concurrency/--pool, solo pool on macOS), migrate, and
argument pass-through to the maintenance scripts. --help imports no app.
- docsgpt/core/paths.py: runtime data lives in a data home (DOCSGPT_HOME,
else the checkout, else cwd); DOCSGPT_ENV_FILE overrides the env file.
Settings, the dotenv load, LocalStorage and the internal upload route use
it instead of "three directories above this file", which is site-packages
for an installed package. A checkout and the Docker image behave as before.
- [project] dependencies are compatible ranges so the package installs next
to other packages; uv.lock resolves to the same versions and the exported
requirements files are unchanged.
- package-build.yml builds and checks the wheel on PRs and installs it into
a clean venv; pypi-publish.yml publishes on a published release through
trusted publishing (environment pypi), or to TestPyPI on a manual run.
- Docs: Deploying -> Install with pip. AGENTS.md notes the package.
- CI installs the backend requirements from docsgpt/; the old cd into
application/ silently installed nothing.
- The root .dockerignore re-admits only application/__init__.py. An upgraded
checkout may still hold gitignored application/{inputs,indexes,vectors,.env}
from the old layout, and the directory rule shipped them into the image.
- The compose files keep the host bind mounts on application/{indexes,inputs,
vectors}, so an upgrade does not start with empty data. The move comes with
the packaging work, together with an upgrade note.
- The alias loader puts the real docsgpt spec back on the shared module object
after import (the import machinery stamped the alias spec on it, which made
importlib.reload rename the module and skip re-execution) and delegates
get_code/get_source/get_filename to the target loader, so
python -m application.<name> runs.
- Each legacy application.* task name is registered as its own task object,
a subclass carrying the old name. Registering the same object under two
keys made Celery's tracer log every run under whichever name it built last.
- The redbeat key prefix stays redbeat:docsgpt:; the three schedule_syncs
entries get stable names instead. redbeat tracks its static entries and
deletes the ones that vanish from beat_schedule at start-up, and rewrites the
task path of named entries in place, so neither a prefix bump nor a cleanup
pass is needed (checked against redbeat 2.4.2 with a seeded Redis).
The backend import package is now docsgpt, the name it will carry on PyPI;
application was far too generic to install into anyone's site-packages.
git mv plus a mechanical rewrite of every import, dotted string and path
reference: 734 Python files, the compose files, Dockerfile, workflows, docs,
setup scripts, devcontainer, k8s manifests, vscode config, pytest and coverage
config, .gitignore. Behaviour is unchanged.
Kept for one release:
- A top-level application package whose meta-path finder resolves
application.x.y to the already-imported docsgpt.x.y object, so old imports
and entry points (celery -A application.app.celery,
uvicorn application.asgi:asgi_app) keep working with a FutureWarning.
- Celery registers every application.* task name as an alias of its
docsgpt.* task on start-up, so messages queued by the previous release still
run. The redbeat key prefix moves to redbeat:docsgpt:v2: so schedule entries
the previous release wrote are left unread instead of firing twice.
The backend image builds from the repository root (docker build -f
docsgpt/Dockerfile .) so it can ship the alias package; a root .dockerignore
allow-lists docsgpt/ and application/ and keeps caches, local data, .env
files, the sample index files and the Dockerfile out. Compose and the image
workflows point at the new context.
arc53-machine