Commit Graph
5610 Commits
Author SHA1 Message Date
Alex 7b93e67fea Merge pull request #2829 from arc53/chore/bump-npm-v0.8.0
chore: bump npm libraries to v0.8.0
2026-09-25 10:46:38 +01:00
github-actions[bot] 013631c755 chore: bump npm libraries to v0.8.0 2026-09-25 09:39:16 +00:00
Alex f0aa6ae275 Merge pull request #2826 from ManishMadan2882/main
(fix:widget/ui): mobile keyboard, back button
2026-09-25 10:36:39 +01:00
GH Action - Upstream Sync 192bf40001 Merge branch 'main' of https://github.com/arc53/DocsGPT 2026-09-25 03:51:15 +00:00
Alex a17ebf2fc9 Merge pull request #2828 from arc53-machine/chore/banner-secure-oss-fund
Point the dev banner at the Secure Open Source Fund post
2026-09-24 16:46:55 +01:00
ManishMadan2882 d57bb00215 (fix:widget: Keep a back-button entry while any panel is open 2026-09-24 20:34:18 +05:30
arc53-machine 0a28f9249a Point the dev banner at the Secure Open Source Fund post
The banner still announced 0.17, two releases back, through the old blog
domain. It now links the GitHub Secure Open Source Fund post on
www.docsgpt.cloud directly, skipping the blog.docsgpt.cloud redirect.
2026-09-24 16:00:51 +01:00
Alex f2ad13be57 Merge pull request #2827 from arc53/fix/continuation-log-context
Bind the log context for tool continuations
2026-09-24 13:54:24 +01:00
Alex cf9a5e9cb8 Fall back to model_id for the logged model
Agents store their model only in model_id, so neither log_activity nor
agent_log_context bound a model key. start_agent_span already reads it.
2026-09-24 13:47:46 +01:00
Alex 9df90ab9df Bind the log context for tool continuations
gen_continuation is not wrapped by @log_activity, so the llm_* log lines
from a resumed turn carried no user, agent, conversation or endpoint. Every
OpenAI-compatible /v1 client-tool round landed anonymous in the logs.

Add agent_log_context, which binds the same keys log_activity binds while
keeping any enclosing activity_id, and wrap gen_continuation in it. Both
now derive the keys from one helper.
2026-09-24 13:26:49 +01:00
GH Action - Upstream Sync 2d1e940fe1 Merge branch 'main' of https://github.com/arc53/DocsGPT 2026-09-24 03:34:37 +00:00
ManishMadan2882 d40d344bd6 (fix:widget/ui): mobile keyboard, back button 2026-09-24 05:43:21 +05:30
Alex 91d670b7bb Merge pull request #2825 from arc53-machine/feat/genai-traces
Execution traces: GenAI OpenTelemetry spans and a trace waterfall in Logs
2026-09-24 00:12:12 +01:00
arc53-machine f2d92bf8df Log failed chat turns as chat entries
A turn whose agent raised wrote no user_logs row, so it only surfaced as
the agent's system error row. Every finished turn now writes its chat row,
at level error with the error when it failed, and linked to its trace; the
system row for the same traced activity is no longer listed twice.
2026-09-24 00:08:53 +01:00
arc53-machine 3156ee37a1 Name the provider a model call actually reached
DeepSeek and every other OpenAI-compatible API run through OpenAILLM, so
spans and metrics reported them as openai. The provider now comes from the
endpoint: known API hosts map to their gen_ai.provider.name (deepseek,
azure.ai.openai, mistral_ai, ...), and an OpenAI client pointed anywhere
else is openai_compatible. LLM spans also record server.address.
2026-09-24 00:08:53 +01:00
arc53-machine 6b145d7aad Wait for the background trace write in the /v1 replay test
The first request's trace is written on the trace-writer pool after the
response returns; assert once it has landed rather than racing it.
2026-09-23 23:23:41 +01:00
arc53-machine a74420ed2c Trace unexecuted tool calls in one place; tidy span lifecycles
Paused, denied, skipped and client-run tool calls all go through
trace_unexecuted_tool_call. The research synthesis, continuation and
workflow step spans use with-blocks instead of hand-rolled GeneratorExit
handling. Fixes blank lines and adds a missing hint and docstrings.
2026-09-23 22:42:05 +01:00
arc53-machine b3aa509d3a Delete all of a conversation's traces and index trace listing
Only message-linked traces cascaded, so scheduled runs, stateless /v1
rounds and unreserved turns kept their previews after the conversation was
deleted; they are now deleted by conversation_id. The Logs search and graph
branches filter by source, so the listing indexes lead with it.
2026-09-23 22:42:05 +01:00
arc53-machine 248ebc8050 Discard the setup trace of a replayed /v1 request
An Idempotency-Key retry returns the cached response after setup already
ran; its trace was then written as a failure no Logs row points to.
2026-09-23 22:42:05 +01:00
arc53-machine 778830208b Write chat traces off the stream's thread
The OTel replay and the trace INSERT ran in the stream's finally, so a slow
database held the SSE connection open after the last event. The trace is
still frozen when the stream ends, but written on a small writer pool.
2026-09-23 22:42:05 +01:00
arc53-machine 0591580014 Keep exception text out of stored and exported span errors
Span.fail stored the exception message in span.error, which is stored and
exported whatever the content settings; a provider's content-filter error
can quote the prompt. span.error now holds the exception type, and the
message is a capture-gated preview, like tool errors already were. A
yielded stream error on the agent span is treated the same way.
2026-09-23 22:42:05 +01:00
arc53-machine 0bbbac0eb2 Skip trace work when disabled; share the agent span builder
With TRACES_ENABLED off, LLM calls record no GenAI metrics, and a streamed
answer is joined into a preview only when a span keeps it. Agent runs and
continuations build their invoke_agent span in one place, which now reads
the model from model_id. The trace panel shows a stream's model time next
to how long it was open. Adds missing type hints.
2026-09-23 22:09:58 +01:00
arc53-machine 699906a015 Give a scheduled run's trace to the user who scheduled it
On a shared agent the run's Logs row belongs to the scheduling user, not
the agent owner, so the trace is stored under that user.
2026-09-23 22:09:58 +01:00
arc53-machine 9f7f0b2f1e Keep trace summaries from breaking Logs and fix their counts
A failed trace-summary lookup now leaves the Logs page intact without
chips. Tool-call counts include only calls that ran, not their paused,
denied or skipped records. A local guardrail that fires unchanged on every
streamed segment is recorded once, so it cannot use up the span cap.
2026-09-23 22:09:58 +01:00
arc53-machine 24796ae61f Mint request ids on the server and trace refused requests
build_agent no longer takes request_id from the request body: it becomes
the primary LLM's usage request id, and quotas count distinct request ids,
so a client could make every call count as one. Requests refused after
setup started (unauthorized, over quota, resume conflict, setup error) now
write their trace, marked error, through an after-request hook; streaming
routes hand the trace to complete_stream instead.
2026-09-23 22:09:58 +01:00
arc53-machine 82b31760ee Put View trace before the trace chips in a Logs row 2026-09-23 21:08:27 +01:00
arc53-machine 5272440a21 Return focus to View trace when the drawer closes
The drawer is opened from a Logs row rather than a SheetTrigger, so Radix
had nothing to restore focus to. It now remembers what had focus when it
opened and returns there on close; the e2e spec checks it.
2026-09-23 21:08:27 +01:00
arc53-machine 56e8437a83 Drop the visible heading from the trace drawer
The drawer opens straight onto the trace. Its title stays as the dialog's
screen-reader name, and the unused subtitle string is removed from every
locale.
2026-09-23 21:06:49 +01:00
arc53-machine d7319928d1 Build the trace UI on theme tokens and existing components
Colours come from theme tokens only: status pills and chips use muted,
primary and destructive; waterfall bars use chart-1..5, one per group of
span kinds, with destructive for failures and muted-foreground for
cancelled or pending steps. Rows select through ghost Buttons, the error box
is a destructive Alert and View trace is an outline Button. Bar geometry and
indentation move to CSS custom properties, scale ticks to static
positions, and the sheet takes layout classes only.
2026-09-23 20:53:18 +01:00
arc53-machine 8157bcf3bf Make Logs rows keyboard-operable and fix minute rounding
Rows open with Enter or Space and expose aria-expanded, so the View trace
action is reachable without a mouse. Durations round to whole seconds before
splitting, so 119.6 s reads 2m 00s instead of 1m 60s.
2026-09-23 18:11:19 +01:00
arc53-machine efec1a2f83 Keep tool output out of span errors and fix trace status and listing
Tool results, denial comments and tool exception text now reach a span only
as a capture-gated preview; span.error is a fixed message, since it is
stored and exported regardless of content settings. A turn whose stream
yields an error event is recorded as failed. Search traces are listed by a
query copied into the small summary column, so the Logs timeline never
reads the spans JSONB. Adds GraphRAG span tests.
2026-09-23 18:11:19 +01:00
arc53-machine 285fdc914c Add an e2e spec for execution traces; tidy the timeline scale
A tool-calling chat turn and a RAG turn are traced, linked from their Logs
rows and opened in the trace panel. The timeline scale starts at 0 and its
labels no longer wrap.
2026-09-23 17:55:30 +01:00
arc53-machine 3512685728 Keep complete_stream's signature and classify /api/traces for tokens
The trace lifecycle moves into a decorator so complete_stream keeps its
real parameters for introspection. GET /api/traces is readable with the
analytics:read scope, like the Logs endpoint it complements.
2026-09-23 17:50:19 +01:00
arc53-machine 4e1002f5e0 Document execution traces and the GenAI OpenTelemetry export
Covers what is recorded, where it is shown, the TRACES_* settings, the
gen_ai.* spans and metrics, content capture, and the limits of exporting
spans when the request ends.
2026-09-23 17:46:18 +01:00
arc53-machine 3310b7fa1b Show execution traces in the Logs UI
Expanded Logs rows show the trace's duration, LLM calls, tokens, tool calls
and retrieval time, and open a side panel with a waterfall of every step:
retrieval, embeddings, per-source searches, model calls and tool calls,
with each step's details, previews and raw attributes. Chat turns paused
for approval show one waterfall per round. Searches and graph builds get
their own log types. Strings are translated in every locale.
2026-09-23 17:45:19 +01:00
arc53-machine 26881ba66b Serve traces to the Logs UI
GET /api/traces returns a Logs row's stored traces, scoped to the caller or
to an agent they own. get_user_logs rows now carry the ids that find their
trace and a merged summary (duration, LLM and tool calls, tokens, retrieval
time) from one batched lookup per page; searches, MCP searches and graph
builds, which have no log row of their own, are listed from their traces.
2026-09-23 17:39:09 +01:00
arc53-machine 5d0992eef8 Trace scheduled, webhook, search, MCP and graph-extraction runs
run_agent_headless records each unattended run under its endpoint; the
scheduler passes its run id and the webhook worker its task id so Logs rows
can find their trace, while the LLM's own request id stays untouched for
quota counts. /api/search and MCP search_docs record their retrieval, and a
graph build records every extraction call under one step.
2026-09-23 17:37:24 +01:00
arc53-machine bae842d151 Trace every chat turn from agent setup to the last event
StreamProcessor starts the trace and mints the request id before the
agent is built, so pre-fetch retrieval and compression are inside it and
side-channel LLM calls share the id. complete_stream activates it in the
SSE pump thread, binds the message and conversation, and writes it once
however the stream ends: paused, failed, abandoned or superseded (dropped).
user_logs rows now carry request_id and message_id.
2026-09-23 17:34:54 +01:00
arc53-machine da1bc16eb6 Trace guardrails, workflow steps and research phases
Guardrail evaluations are traced when they call a remote check or fire; a
firing guardrail drops every content preview from the stored trace. Each
workflow node and each research phase is a step span, and the workflow run
id links the trace to its Logs row.
2026-09-23 17:29:21 +01:00
arc53-machine f9605e7794 Trace RAG retrieval, query embeddings and prescreen
The dispatcher and each retriever's search open retrieval spans recording
sources, chunk counts, top scores and a chunk preview. The shared query
embedding, every per-source vector search and the prescreen rerank get
their own spans; pool workers carry the trace so they nest correctly.
2026-09-23 17:27:43 +01:00
arc53-machine a15940c281 Trace agent runs and tool calls
@log_activity opens the invoke_agent span for every agent run and names
the trace's activity; gen_continuation opens its own. ToolExecutor.execute
records each executed call with redacted argument and result previews, and
paused, denied, skipped and client-executed calls are recorded where they
are decided.
2026-09-23 17:25:41 +01:00
arc53-machine e8166e6c29 Record a chat span for every LLM call
The token-usage wrappers open one span per decorated invocation, so a
primary attempt, its retry and a fallback appear as siblings with the
provider that ran. Stream spans start on the first pull, carry tokens,
cost, time to first token and cache hits, and feed the GenAI metrics.
2026-09-23 17:23:06 +01:00
arc53-machine b5e9257659 Export traces as OpenTelemetry GenAI spans and metrics
Finished traces are replayed with their recorded timestamps and explicit
parents under the request's server span, using gen_ai.* attribute names.
Content is exported only when OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
opts in. gen_ai.client token-usage and operation-duration histograms are
recorded per model call.
2026-09-23 17:21:25 +01:00
arc53-machine 3f45d019b0 Add the per-request trace recorder
docsgpt.tracing records agent, LLM, tool, retrieval and embedding spans in
memory and writes the finished trace once. Container spans nest by a
per-thread stack so suspended generators cannot corrupt the tree; open
spans are cancelled at flush and previews are bounded and redacted.
2026-09-23 17:19:54 +01:00
arc53-machine f3f817b1e8 Store execution traces in a new request_traces table
One row per execution holding its span tree as JSONB, linked to Logs rows
by request, message, activity and workflow-run ids. message_id cascades so
deleting or truncating a conversation drops its traces; a daily beat task
enforces TRACES_RETENTION_DAYS.
2026-09-23 17:19:54 +01:00
arc53-machine f6269c483f Add execution-trace settings and declare opentelemetry-api
TRACES_* settings for the per-request trace timeline and its GenAI OTel
export. opentelemetry-api was only transitive; the tracing package
imports it directly.
2026-09-23 17:14:51 +01:00
Alex 8a9f3a75d6 Merge pull request #2823 from arc53-machine/fix-chunk-token-counts
Show token counts on source chunks
2026-09-22 16:45:42 +01:00
arc53-machine db8db62c93 Treat a non-finite token count as unrecorded
`float("inf") > 0` is true, so a stored `"inf"` was preserved as if it
were a real count and reached the UI as "∞". Require a finite positive
value, which also covers `nan` explicitly rather than by accident.
2026-09-22 14:01:27 +01:00
arc53-machine 3954de9b9e Render chunk token counts through one tested helper
Both chunk headers inlined the same `token_count ? … : '-'` expression, and
`ChunkType` declared every metadata value a string even though the JSON-typed
stores return the count as a number.

Move the formatting into `formatChunkTokens`, which accepts either shape and
only falls back to a dash when the value is genuinely unusable, and type
`ChunkType.metadata` to match what the API actually returns.
2026-09-22 13:45:40 +01:00
arc53-machine d787f54c2d Keep the chunker's metadata when re-embedding a wiki page
`reembed_wiki_page_worker` built each chunk's metadata from scratch as
`{source, title, filename}`, discarding everything the chunker attached
to the document -- including the per-chunk `token_count` every other
ingest path records. Wiki chunks therefore reached the source viewer
with no count at all.

Carry `extra_info` over and let the page's own path and title win on top,
so a chunk that inherited a stale source from an earlier conversion is
still keyed and filtered correctly.
2026-09-22 13:44:24 +01:00