A failed trace-summary lookup now leaves the Logs page intact without
chips. Tool-call counts include only calls that ran, not their paused,
denied or skipped records. A local guardrail that fires unchanged on every
streamed segment is recorded once, so it cannot use up the span cap.
build_agent no longer takes request_id from the request body: it becomes
the primary LLM's usage request id, and quotas count distinct request ids,
so a client could make every call count as one. Requests refused after
setup started (unauthorized, over quota, resume conflict, setup error) now
write their trace, marked error, through an after-request hook; streaming
routes hand the trace to complete_stream instead.
The drawer is opened from a Logs row rather than a SheetTrigger, so Radix
had nothing to restore focus to. It now remembers what had focus when it
opened and returns there on close; the e2e spec checks it.
The drawer opens straight onto the trace. Its title stays as the dialog's
screen-reader name, and the unused subtitle string is removed from every
locale.
Colours come from theme tokens only: status pills and chips use muted,
primary and destructive; waterfall bars use chart-1..5, one per group of
span kinds, with destructive for failures and muted-foreground for
cancelled or pending steps. Rows select through ghost Buttons, the error box
is a destructive Alert and View trace is an outline Button. Bar geometry and
indentation move to CSS custom properties, scale ticks to static
positions, and the sheet takes layout classes only.
Rows open with Enter or Space and expose aria-expanded, so the View trace
action is reachable without a mouse. Durations round to whole seconds before
splitting, so 119.6 s reads 2m 00s instead of 1m 60s.
Tool results, denial comments and tool exception text now reach a span only
as a capture-gated preview; span.error is a fixed message, since it is
stored and exported regardless of content settings. A turn whose stream
yields an error event is recorded as failed. Search traces are listed by a
query copied into the small summary column, so the Logs timeline never
reads the spans JSONB. Adds GraphRAG span tests.
A tool-calling chat turn and a RAG turn are traced, linked from their Logs
rows and opened in the trace panel. The timeline scale starts at 0 and its
labels no longer wrap.
The trace lifecycle moves into a decorator so complete_stream keeps its
real parameters for introspection. GET /api/traces is readable with the
analytics:read scope, like the Logs endpoint it complements.
Covers what is recorded, where it is shown, the TRACES_* settings, the
gen_ai.* spans and metrics, content capture, and the limits of exporting
spans when the request ends.
Expanded Logs rows show the trace's duration, LLM calls, tokens, tool calls
and retrieval time, and open a side panel with a waterfall of every step:
retrieval, embeddings, per-source searches, model calls and tool calls,
with each step's details, previews and raw attributes. Chat turns paused
for approval show one waterfall per round. Searches and graph builds get
their own log types. Strings are translated in every locale.
GET /api/traces returns a Logs row's stored traces, scoped to the caller or
to an agent they own. get_user_logs rows now carry the ids that find their
trace and a merged summary (duration, LLM and tool calls, tokens, retrieval
time) from one batched lookup per page; searches, MCP searches and graph
builds, which have no log row of their own, are listed from their traces.
run_agent_headless records each unattended run under its endpoint; the
scheduler passes its run id and the webhook worker its task id so Logs rows
can find their trace, while the LLM's own request id stays untouched for
quota counts. /api/search and MCP search_docs record their retrieval, and a
graph build records every extraction call under one step.
StreamProcessor starts the trace and mints the request id before the
agent is built, so pre-fetch retrieval and compression are inside it and
side-channel LLM calls share the id. complete_stream activates it in the
SSE pump thread, binds the message and conversation, and writes it once
however the stream ends: paused, failed, abandoned or superseded (dropped).
user_logs rows now carry request_id and message_id.
Guardrail evaluations are traced when they call a remote check or fire; a
firing guardrail drops every content preview from the stored trace. Each
workflow node and each research phase is a step span, and the workflow run
id links the trace to its Logs row.
The dispatcher and each retriever's search open retrieval spans recording
sources, chunk counts, top scores and a chunk preview. The shared query
embedding, every per-source vector search and the prescreen rerank get
their own spans; pool workers carry the trace so they nest correctly.
@log_activity opens the invoke_agent span for every agent run and names
the trace's activity; gen_continuation opens its own. ToolExecutor.execute
records each executed call with redacted argument and result previews, and
paused, denied, skipped and client-executed calls are recorded where they
are decided.
The token-usage wrappers open one span per decorated invocation, so a
primary attempt, its retry and a fallback appear as siblings with the
provider that ran. Stream spans start on the first pull, carry tokens,
cost, time to first token and cache hits, and feed the GenAI metrics.
Finished traces are replayed with their recorded timestamps and explicit
parents under the request's server span, using gen_ai.* attribute names.
Content is exported only when OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
opts in. gen_ai.client token-usage and operation-duration histograms are
recorded per model call.
docsgpt.tracing records agent, LLM, tool, retrieval and embedding spans in
memory and writes the finished trace once. Container spans nest by a
per-thread stack so suspended generators cannot corrupt the tree; open
spans are cancelled at flush and previews are bounded and redacted.
One row per execution holding its span tree as JSONB, linked to Logs rows
by request, message, activity and workflow-run ids. message_id cascades so
deleting or truncating a conversation drops its traces; a daily beat task
enforces TRACES_RETENTION_DAYS.
TRACES_* settings for the per-request trace timeline and its GenAI OTel
export. opentelemetry-api was only transitive; the tracing package
imports it directly.
`float("inf") > 0` is true, so a stored `"inf"` was preserved as if it
were a real count and reached the UI as "∞". Require a finite positive
value, which also covers `nan` explicitly rather than by accident.
Both chunk headers inlined the same `token_count ? … : '-'` expression, and
`ChunkType` declared every metadata value a string even though the JSON-typed
stores return the count as a number.
Move the formatting into `formatChunkTokens`, which accepts either shape and
only falls back to a dash when the value is genuinely unusable, and type
`ChunkType.metadata` to match what the API actually returns.
`reembed_wiki_page_worker` built each chunk's metadata from scratch as
`{source, title, filename}`, discarding everything the chunker attached
to the document -- including the per-chunk `token_count` every other
ingest path records. Wiki chunks therefore reached the source viewer
with no count at all.
Carry `extra_info` over and let the page's own path and title win on top,
so a chunk that inherited a stale source from an earlier conversion is
still keyed and filtered correctly.
Chunk cards in the source viewer read `metadata.token_count` and print a
bare "-" whenever it is absent. Ingestion records the count, but chunks
indexed before it was written -- and any path that rebuilds a chunk's
metadata from scratch -- reach the UI without it, so the whole file shows
"-" with no way to recover the number short of a re-ingest.
Fill the key in on the way out when it is missing or unusable (empty,
non-numeric, zero or negative), counting only the page being returned so
the cost is bounded by `per_page`. A count that is already recorded is
left untouched, including a numeric string, since stores round-trip
metadata types differently.
The fallback counts in cl100k rather than the embedding model's
tokenizer: loading that tokenizer can reach for a Hugging Face download,
which does not belong in a request path.
The DEFAULT 'unknown' added in the last pass stopped a previous-release
process from raising NotNullViolation mid-rollout, but it threw the
attribution away to do it. The highest-volume auth_events writers are OIDC
login and silent renewal, where the actor is simply the user, so a rollout
window would have flattened exactly the rows that were trivially recoverable.
Lifts both derivation rules into SQL functions -- auth_events_derive_actor
and auth_events_derive_target -- so the backfill and a BEFORE INSERT trigger
share one definition rather than two copies of the same CASE drifting apart.
The trigger guards on NEW.actor_id IS NULL, which identifies a legacy insert
exactly because the repository always supplies one. That distinction matters
for target_id: a current writer sets it NULL deliberately for events with no
user target, and deriving there would name the actor as their own target.
Kept rather than scheduled for removal: it is a no-op on the path the
repository takes, and it keeps the column's contract true for any writer that
bypasses it.
Correctness
- /api/remote never recorded source.created, so URL, GitHub and connector
sources had a source.deleted with no matching creation. All three creation
paths now go through one _audit_source_created helper.
- The prompt-cache rate divided cached tokens by a whole bucket's prompt
tokens. A bucket is a day and mixes calls whose provider reports a cache
breakdown with calls whose provider does not, so filtering buckets in the
client could not separate them and the rate was understated by however much
traffic ran on a non-reporting provider. The denominator is now computed in
SQL over the reporting rows.
- The outcome pill matched values nothing writes. Guardrails emit triggered /
not_evaluated and the device feed emits dispatched; the map had blocked /
denied / allowed, so a guardrail that fired rendered neutral grey -- the one
signal the merged feed exists to surface. Fixtures were seeding the
fictional values, so the tests passed on it too.
- Stream duration_ms timed the consumer. stream_token_usage is a generator,
so start-to-exhaustion includes the agent loop's tool handling and the SSE
client's pace; a slow browser recorded ~30s for a sub-second call. It now
accumulates only the time spent inside next().
Safety
- Activity filters failed open: an unknown facet or unparseable timestamp was
dropped, and no filter means every row, so a typo widened an audit view and
on the export streamed the full history. Both are now a 400.
- The search term was interpolated into an ILIKE pattern, so "100%" matched
everything and "q1_report" matched more than it should. Escaped.
- 0034 set actor_id NOT NULL with no default. A previous-release process
inserting mid-rollout would raise, and in admin/routes.py that insert shares
the request transaction, so a role grant beside it would roll back too.
Noise and dead code
- The per-user panel is a security panel: data-plane events file under the
actor, so an active account's routine deletes pushed a denied login out of
the 20-row window. It now excludes them; the Activity tab shows everything.
- device_audit_log had no created_at-leading index, so the merged feed
sequentially scanned that branch every page (migration 0036).
- conversation.deleted_all no longer records when nothing was deleted, and
agent.updated no longer records an empty field list.
- Dropped by_model from /admin/usage (no consumer; an extra aggregate per page
load), the duplicate filter surface on AuthEventsRepository that nothing
called, and the unreachable FLOW_LABELS.schedule entry.
- Type hints on record_event's conn and the remaining unannotated helpers.
- 0034's backfill set target_id to the acting admin for instance- and
team-scoped quota policy changes, which are filed under the actor and have
no user target. It now returns NULL for those, matching what quotas.py
records going forward, with a test covering all three scopes.
- The CSV export wrote every cell verbatim. user_agent is an attacker's raw
header, recorded without authenticating on a denied login, and the export
is opened in a spreadsheet by an admin -- a cell starting "=" would be
evaluated there. Every cell now has a leading formula trigger neutralized.
- The activity feed applied every response it received, so a slow reply for
an old filter could overwrite the current one. Guarded by a request id,
the same way settings/Analytics already does.
- The activity detail expander and the top-user drill-down were row onClick
handlers, unreachable without a mouse. Both are real buttons now, the
expander carrying aria-expanded and a label naming its event.
- FLOW_LABELS was applied to both breakdowns in the spend modal, so a model
named "fallback" or "workflow" rendered as a flow description. Only the
flow table maps keys now.
- Changing the range in that modal left the previous range's totals and
chart on screen while the new request was in flight.
The comment claimed X-Export-Max-Rows says whether the cap was reached; it
reports the cap that was in effect, which is what lets a caller tell a
truncated export from a complete one.
SCIM provisioning is performed by the identity provider on a user, so the
rows now carry actor_id='system:scim' and the user as target_id. The
assertions pin both rather than ignoring the new columns.
_page_after decided whether to append " AND " or " WHERE " by looking for
"WHERE" in the SQL it had just been handed, which only worked because no
union branch contains one. The query builder now always emits a WHERE, so
adding a branch with its own predicate cannot silently break the cursor.
bucketed_totals now returns cost and cached_tokens, so the exact-equality
assertion in test_day_bucket_sums_per_day had to grow the two fields; it
pins that an unreported cache bin reads NULL rather than 0.
The per-user breakdown test was seeded with a source='schedule' row, which
is a run-level rollup and no longer counted. Reseeded on a real flow, and
both spend queries gained a test that a rollup is excluded.
Review pass over the preceding commits.
- The activity export paged by OFFSET. The journals are append-only and the
feed is newest-first, so rows written mid-export shift the window down and
repeat rows already emitted. Pages by keyset on (created_at, feed, id) now.
- top_token_users and the per-user breakdown counted run-level rollup rows.
A scheduled run already has a row per LLM call, so its spend was billed
twice -- invisible while the column was tokens, obvious once it was dollars.
Both now exclude them, matching every other spend query.
- total_cost was summed from the per-model split, which drops rows with no
model_id and so undercounted the figure printed above the series. Summed
from the series instead.
- The CSV export encoded its detail cell without the fallback the NDJSON
branch had, so a non-JSON-native value would have failed the stream.
- The activity view reset pagination in an effect, which fetched the stale
page against the new filters before fetching again. Reset in the setters.
- The audit taxonomy is no longer re-exported from the helper module; the one
caller that wanted it imports from where it lives.
Covers what the previous commits changed: the actor_id / target_id split and
the system actors, the data-plane events, the merged Activity feed and its
export, and the new usage and per-user spend endpoints.
Also corrects "there is no UI for these events yet" in the OIDC login
auditing section, which is no longer true.
Three gaps in one view.
Cost: token_usage.cost has been written on every call since quotas landed, and
tokens_by_model already returned it, but bucketed_totals and top_token_users
selected tokens only. An admin could set a USD quota in /admin/quotas and had
no way to see the spend it was capping. Buckets and top users now carry cost,
the endpoint reports a window total and a per-model split, and the chart takes
a Tokens/Cost toggle with a currency axis.
Group-by: the endpoint has supported group_by=model|agent|source from the
start and the UI only ever sent bucket=day. The selector is now wired, and a
grouped series pads missing buckets so a model that was idle on Tuesday plots
a zero instead of shifting its whole row one bar left.
Latency: surfaced from the columns the previous commit added, as p50/p95 with
a median time-to-first-token underneath. Unmeasured rows are excluded rather
than counted as zero, and the card says so when nothing was measured.
Also surfaces the prompt-cache hit rate, computed only over rows whose
provider reported a cache breakdown -- NULL means "not reported", and folding
those in as 0% would understate it.
Per-user drill-down: GET /api/admin/users/<id>/usage returns a daily series
plus splits by model and by flow, reachable from the top-users table and from
the user detail dialog. The detail dialog previously showed one tokens_30d
number, which answers neither "what is this person costing" nor "what is
driving it".
The wrappers in docsgpt/usage.py already measured how long each call took --
for the llm_gen_finished / llm_stream_finished log lines -- and then threw the
number away. Nothing in the schema recorded it, so an operator could see what
an instance spent but never how slow it was.
Adds token_usage.duration_ms and token_usage.ttft_ms, and threads the
measurements the wrappers already take into the insert. The stream wrapper now
also stamps the moment of the first yielded chunk.
Both columns are nullable, and deliberately so: ttft_ms is NULL for a
non-streaming call and for a stream that failed before yielding anything, and
every row written before this migration is NULL too. A 0 there would drag a
p50 toward an instant first token that never happened.
Replaces the Audit tab with a merged Activity view over all three journals.
The old table rendered four columns -- event, user, IP, when -- and dropped
everything else the row carried. auth_events captures a metadata object and a
user agent on every insert, and neither was ever shown, so "Admin revoked ·
user-42" could not tell you which admin. Rows now expand into a detail panel
carrying the actor, the affected user, the outcome, the user agent and every
metadata field.
Filters were an event box whose placeholder was the only hint that denied
logins are spelled "oidc_login_denied", plus a user-id box. The backend
supported a time window that the UI never sent. Now: a category facet, an
event picker fed by the instance's own catalogue, quick time ranges and a
search that reaches into the detail payload. Export buttons pull the filtered
feed down as CSV or NDJSON.
The deep link from Overview (?event=oidc_login_denied) still lands pre-
filtered, and the route stays /admin/audit so existing bookmarks work.
Drops the two single-journal service readers the merged feed supersedes --
getDeviceAudit had shipped months ago and never had a caller. The
/api/admin/audit endpoint stays for API consumers.
TableCell/TableHeader gain colSpan so the detail row can span the table
instead of a second table being invented for it.
DocsGPT keeps three append-only journals -- auth_events, device_audit_log and
guardrail_events -- each readable only on its own terms, and the second of
those had an admin endpoint that no UI ever called. An operator reviewing an
incident wants one timeline, not three.
Adds GET /api/admin/activity: all three journals projected onto a common row
shape and merged, ordered newest first, with facets for journal, category,
event name, actor, affected user, a time window and free-text search over the
detail payload. A category filter that only one journal can satisfy drops the
others from the union rather than scanning and discarding them.
GET /api/admin/activity/events returns the distinct (event, category) pairs
the instance has actually recorded, so the filter UI can offer real choices
instead of asking an operator to type "oidc_login_denied" from memory.
GET /api/admin/activity/export streams the same filtered feed as CSV or
NDJSON. Streamed rather than buffered, and capped, so a compliance export of a
busy instance is bounded work.
guardrail_events.api_key (a raw agent key) and matched_value (unredacted
source text) are excluded from the projection, mirroring the exclusion the
per-agent guardrail view already makes. Admin-gating is not a reason to widen
what a list response carries.
Both fail on main as of f5b02ed3, before any of this branch's changes —
verified by running them against main in a clean worktree, where they fail
with byte-identical errors. Neither is testing a broken feature; both hang
their assertion on a DOM landmark that #2819 moved.
C13 waited for AgentPageHeader's `Agent sub-navigation` nav on
/agents/manage/logs/:agentId. That page now renders CurrentSectionHeader +
SectionPills, and WorkflowBuilder is the only remaining caller of
AgentPageHeader, so the landmark never appears. C2 proved a locale switch
had taken effect by looking for the Settings entry as a `link` role, which
60532ec4 moved into the sidebar as something else.
The behaviour underneath both still works — the logs page renders its
panels, and switching locale swaps the strings and persists to
localStorage. Only the selectors are stale, so these could be repaired
rather than dropped if someone wants the coverage back.
Removed insertStubAgent with C13, its only caller, and the two imports
that went with it.
Logins, role grants and provisioning were audited from the first release;
creating and deleting sources, agents, agent keys and conversations were not.
An operator reviewing the trail could see who signed in but not who deleted
the source they were asking about.
Adds docsgpt/api/audit.py: one helper that records an event inside the
caller's transaction, in a savepoint, swallowing failures -- an audit write
must never be able to fail the action it describes -- and tolerating the
absence of a Flask request so a Celery task can record too.
Events added: source.created (upload and wiki), source.deleted,
source.reingested, agent.created, agent.updated, agent.deleted,
agent.key_regenerated, conversation.deleted, conversation.deleted_all.
agent.updated records field names only, never values, which can carry prompts
and credentials.
The same module carries the event -> category map (identity / access / config
/ data) that the admin activity feed filters on.
auth_events.user_id was overloaded and meant something different per call
site: admin mutations filed the row under the target user and hid the acting
admin in metadata->>'by', team events filed it under the actor, and quota
events switched between the two depending on scope. The practical consequence
was that "show me everything admin X did" had no answer.
Adds actor_id (never NULL) and target_id (NULL when the event is not about a
user) alongside the existing column, which keeps its meaning as the per-user
feed key so nothing historical is rewritten. The backfill recovers the actor
from metadata for rows that predate the columns.
Also adds the indexes the cross-user admin feed needs. The only index was
(user_id, created_at DESC), so the global feed -- which orders by created_at
with no user predicate -- fell back to a sequential scan plus a sort on every
page.
The feed gains actor, multi-event, until and free-text search filters, plus a
distinct-event catalogue so the UI can offer what the instance recorded
instead of asking operators to type a name from memory.
test_each_provider_has_expected_ids asserts the exact set of model ids per
provider against a snapshot. For the first-party catalogs that is what we
want: the snapshot exists so an upstream id typo fails before it silently
breaks every agent that references the old id.
openai_compatible is different. It is the zero-Python extension point —
the README's "Add an OpenAI-compatible provider" section and
examples/mistral.yaml.example both tell you to drop a YAML next to the
built-ins, and the loader globs the whole directory. So anyone who follows
those instructions gets a permanently red test on their fork, for having
done the documented thing. The _internal suffix carve-out only covers the
one filename we happen to gitignore.
Treat the snapshot as a floor for that provider: the built-in ids must all
still be present, extra ones are somebody else's provider. The rename this
guards against still drops an id, so it still fails, and the assertion
names the missing ids instead of printing two large sets to diff by eye.
Verified by copying examples/mistral.yaml.example into the catalog
directory: red before, green after, and still red when a built-in id is
renamed.
Three things the UI pass started but did not carry all the way.
The MCP server dialog's Basic Authentication branch was fixed to use the
label keys for its labels and the placeholder keys for its placeholders,
but the API Key and Bearer Token branches above it in the same
renderAuthFields were left reading placeholders.apiKey and
placeholders.bearerToken as their <Label>. Both fields showed the
placeholder text twice over: the label above the input read "Your secret
API key" where the input already hinted it. authTypes.apiKey and
authTypes.bearer already held "API Key" and "Bearer Token". (Note that
the Basic branch that prompted all this is unreachable — `basic` is
commented out of the authTypes list, so that form cannot be opened.)
Dropped a dead `t(...) || 'Scopes (comma separated)'` fallback in the same
file while there. t() returns the key string when a key is missing, never
a falsy value, so the right-hand side could not run; the branch would
render the raw key rather than the English it looks like it guards.
The plural fix covered English only. resultSummary, view_more,
actionsFound and needsSetup all took _one/_other in en.json, but de, es
and ru kept a single form, so "1 Chunks abgerufen", "1 fragmentos
recuperados" and "ещё 1 источников" still rendered — the exact bug the
change set out to remove, in every language but the one it was tested in.
Added _one/_other for de and es, and _one/_few/_many/_other for ru, which
needs the three-way split (1 источник, 2 источника, 5 источников).
Translated needsSetup while there; it was still English in all three.
jp, zh and zh-TW have a single plural category, so their base form is
already correct and stays as it is.
Replacing Spinner with a skeleton dropped the role="status" that Spinner
carried, and the skeleton divs announce nothing, so a screen reader got
no signal at all between navigating to Agents and the list appearing.
Put it on SkeletonLoader itself rather than at the call sites, so every
variant gets it. It is sr-only, which is position: absolute, so it never
becomes a flex or grid item in the containers these render into.
The branch forked before the agents navigation moved into the sidebar, so
main had since rewritten both files this PR touches. Two conflicts.
AgentsList: main split the filters into their own routes and dropped the
FILTER_TABS pill row and the page <h1> for CurrentSectionHeader +
SectionPills, so the page-level skeleton this branch added no longer has
a place to live — it keyed off `activeFilter === 'all'` and gated the
whole page on all four sections having loaded. Resolved to main's
structure, which already loads each section independently, and kept only
the part that was the point: the section's Spinner becomes the agentCards
skeleton. That also drops `isLoading && !isDataLoaded`, which would have
suppressed the indicator entirely on a refetch over cached data.
Keeping per-section loading is the better behaviour anyway. The page-level
loader hid the section headers and the New Folder / Import / New Agent
buttons for as long as the slowest of the four endpoints took, and hid the
sections that had already arrived along with them.
AgentLogs: main moved the page onto CurrentSectionHeader + SectionPills
and a max-w-6xl wrapper. Took that, with this branch's removal of the two
page-level spinners — Analytics, GuardrailEvents and Logs each have their
own skeleton, so the outer spinners were the second of the two loaders the
PR set out to remove. Gated on `agentId` alone rather than
`agentId && (loadingAgent || agent)`: the children only ever needed the id,
and the longer condition mounted all three, then unmounted them mid-flight
when the agent fetch failed, leaving the page blank with no error.
The hook-order crash this branch still carried in AgentSection is fixed on
main already (the breadcrumb useMemo sits above the early returns), and the
merge takes that.
arc53-machine