Rename the Artifact Generator display name to "Artifact" (the internal
artifact_generator key is unchanged) and tighten the user-facing one-line
descriptions for Code Executor, Artifact, and Read Document.
Enable code_executor and artifact_generator by default in chats (added to
DEFAULT_CHAT_TOOLS) — they load via the synthetic-id path user- and
conversation-scoped like scheduler, and persist artifacts (no user_tools FK).
Make read_document an internal agent-builtin flagged workflow_only so it appears
only in the workflow builder, not the classic agent picker or the Add-Tool
catalog (reuses the builtin synthetic-id path; still run-scoped and authz-gated).
Per decision, default-on code_executor runs sandboxed code without an approval
prompt (the sandbox is the trust boundary); this is documented in settings and
the threat model, and multi-tenant deployments should add per-tenant isolation
(Daytona / gVisor / egress policy).
Wrap a storage read in a context manager (close the file handle), add a comment
on the expected SandboxCapacityError in the churn test, and unify the
artifacts-routes test on a single import style. Also annotate the intentional
per-session sandbox workspace path with # nosec B108 (controlled dir inside the
runner container, not an insecure shared temp file).
When code_executor or read_document is given an input that doesn't resolve to an
existing artifact, it now falls back to the caller's own chat attachments,
bridges the referenced one into a conversation-scoped artifact (idempotent,
server-computed size/sha256, quota-respected), and stages it. This is lazy —
plain chat uploads are never bridged, only a file a tool actually references —
and conversation-scoped only (workflow nodes bridge attachments up front).
The match is confined to the request's own attachments and re-verified against
the user-scoped attachments repo, so a model-supplied name/id can only ever reach
the caller's own files. Only the current request's attachments are reachable (the
attachments table has no conversation column); prior-turn files are a follow-up.
Replace the sandbox Docling extractor with read_document, backed by the in-process
backend parser (the same one ingestion uses) and offloaded to a dedicated
'parsing' Celery queue so it can run on GPU-capable workers with predictable RAM.
The tool resolves the input ref under the run-scoped gate, enqueues the parse,
and awaits it with a timeout (degrading to an error rather than hanging); the
worker independently re-resolves the artifact through the same gate and never
trusts a raw path. Untrusted files get the upload path's safeguards (extension
whitelist, size cap, sanitized temp file, cleanup). Options: output
(markdown/text/structured/chunks), ocr, pages, engine, max_chars, include_tables,
persist, json_schema. The workflow native-file 'extract' fallback now uses the
same worker path, so document parsing no longer needs the sandbox and works on
every backend.
Also fixes the branch's periodic-task test (the sandbox reaper made it 12) and
points the dev and e2e Celery workers at the parsing queue.
An agent node can list input_documents (short refs like A1, a literal "*" for all
inputs, or state variables holding refs produced by an upstream node). The engine
resolves each through the run-scoped artifact gate and feeds it to the node's LLM
natively when the model supports the format (PDFs included via the provider's
image synthesis), otherwise extracts it to text with Docling. A file_passing
policy (auto/native/extract) and per-node count + per-file byte caps bound cost;
the native decision uses the same supported-types source the provider handler
uses, so a document is never silently dropped. Bytes never enter run state.
The classic prompt renderer now passes artifact_parent={conversation_id} so a
normal agent's prompt can resolve a prior-turn artifact with
{{ artifacts.artifact(id) }}, scoped to its own conversation (parent-derived
authz; a missing conversation_id safely yields an empty lookup). This makes the
artifacts variable surfaced in the agent builder functional, and is the parent
wiring an agent-as-tool / subagent feature would also rely on.
Run each kernel under a scrubbed environment so untrusted code can never read
the host's secrets. A custom 'docsgpt-python' kernelspec launches ipykernel
through a wrapper that keeps only what the kernel needs (PATH, HOME, LANG, and
the Jupyter runtime/data dirs), dropping API keys, tokens, the database URL, and
the gateway token. The app selects this kernel by name via SANDBOX_KERNEL_NAME,
so the distinct name is never shadowed by the stock python3 spec. Per-session
workspaces are created mode 0700 (defense in depth under the shared uid). The
README documents the runner as a single trust domain and points to the Daytona
backend for per-tenant isolation.
When an executed tool call raises, append the assistant message carrying the
tool_calls before the tool error result, mirroring the success path. Previously
only the tool result was appended, leaving a tool message with no preceding
tool_calls parent, which made the next provider completion reject the request.
Every tool_call id in a batch is now answered even when some calls fail.
Let workflow runs consume and produce documents end to end: bridge uploaded
attachments into run-scoped artifacts so nodes receive the input documents
(with a per-run cap and server-computed size/sha256, and the run row pre-created
so produced artifacts are authorized during the run); emit the run id to the
client and add a builder panel that lists, previews, and downloads a run's
artifacts; and allow attaching documents to a Preview run via the existing
upload flow.
Also fixes issues a compliance workflow surfaced: attachment ownership now keys
on the raw identity instead of a sanitized one (the sanitized form could not be
read back and could collide across users); workflow code nodes read prior state
from a state.json data file instead of templating it into the program, so
untrusted document content can never be interpolated into executed code;
structured node output wrapped in code fences is recovered; and the live
speech-to-text ownership check compares the raw identity.
Stamp the engine's workflow_run_id into a workflow agent node's tool config, so
artifact_generator, code_executor, and document_extractor address artifacts by
the workflow run. An artifact (and its short ref like A1) created by one agent
node is then visible to a later code node and editable by a later agent node in
the same run - enabling an 'agent generates, a code node transforms, a second
agent edits' flow. Workflow agent nodes carry no conversation id, so only the
run scope applies.
Add an html kind to the artifact generator: it renders a styled HTML report from
a stored JSON spec (title + heading/paragraph/list/table/code blocks), producing
text/html bytes the sidebar already previews inline in the sandboxed iframe,
while keeping the spec as the source of truth so edit and rewrite make real
versions. So a single artifact both renders inline and versions conversationally.
The renderer is a fixed program fed the spec as data, and every spec text value
is HTML-escaped into the output.
Give each produced artifact a short virtual handle - A1, A2, ... - the n-th
artifact in the conversation or workflow run (case-insensitive, not stored). The
tools return it in their results, and the edit, rewrite, and input parameters
accept either the handle or the full id. A handle resolves only within the
caller's own conversation or run, and the resolved id is still checked against
that parent before any read, so it cannot reach another tenant's artifact. This
fixes the model creating a duplicate instead of a new version when asked to
edit an artifact.
Expose a user's artifacts through the MCP server as readable resources: each is
listed under an artifact:// URI with its mime type and read on demand as inline
text or a base64 blob, bounded by a size cap and scoped strictly to the owning
principal resolved from the request's API key, so no artifact is served across
tenants. Ship the network-level egress controls the sandbox runner needs but
cannot self-apply: a Kubernetes NetworkPolicy that allows public egress while
denying RFC1918, link-local, and cloud-metadata ranges, an optional
docker-compose egress overlay, and runner network-hardening docs.
Add runtime governance for the sandbox and artifact store: a per-process
concurrent-session cap with least-recently-used eviction of idle sessions and a
periodic idle reaper (Celery beat), so sandbox kernels do not accumulate. The
session manager performs all backend start/stop outside its lock and tears down
the captured handle, so eviction never closes a concurrently re-opened session.
Add per-user artifact quotas (count, total bytes, and per-file size) enforced at
persistence time as a soft cap, and best-effort cleanup of per-render scratch
directories in the sandbox workspace.
Add a document extractor tool with an extract_document action that converts an
input artifact (PDF, docx, pptx, ...) into structured JSON by running a fixed
Docling program in the sandbox (the document and parameters travel as data, so
nothing in them can execute). Output is a compact payload bounded by an input
size cap, a head-and-tail markdown window, and per-table caps, can be validated
against a JSON schema, and is persisted as a data artifact by reference. Docling
is heavy, so it stays out of the base image and ships in an opt-in sandbox image
variant.
Add an artifact generator tool with create, edit, and rewrite actions that
render presentations, documents, spreadsheets, and PDFs from a validated JSON
spec by running a fixed program in the sandbox (the spec travels as data, so
its contents can never execute). The spec is stored as the source of truth on
each artifact version; edits apply a JSON merge-patch and re-render, appending
a new version while earlier versions stay intact.
Add a code workflow node that runs code in the run-scoped sandbox session and
writes produced files as artifact references into workflow state, passing them
by reference (only id and metadata, never bytes) so downstream nodes and CEL
conditions can branch on them. Add an artifacts.* templating namespace that
resolves those references to metadata via a run-scoped lookup, available to
both the workflow engine and the prompt renderer. Extract the sandbox-to-
artifact persistence into a shared helper reused by the code node and the
code_executor tool.
Add a code_executor agent tool with a run_code action that runs agent-provided
code in the per-conversation sandbox session and captures produced files as
artifacts. Inputs are materialized only from artifacts the caller can access
(parent-scoped); produced files are stored under the user's namespace with
server-computed size and sha256, and the storage write is ordered last in the
transaction so a failure cannot orphan bytes. Output is a compact payload with
no raw bytes, and the produced artifact lights up the existing tool artifact
rail. Execution honors a wall-clock timeout and an agent-selectable session TTL
clamped by the global cap, and the action can be gated behind approval.
Tool-call argument logging is redacted so code bodies are not written to logs.
Add a pluggable Daytona backend (SANDBOX_BACKEND=daytona) that runs code on
Daytona Cloud via the Apache-2.0 SDK and DAYTONA_API_KEY, conforming to the
CodeSandbox interface. Sessions reattach to a labelled cloud sandbox by id
(waking a stopped one) and are created crash-safe so a failed setup never
orphans a paid sandbox; a configurable ceiling caps concurrent sandboxes and
auto-delete is clamped on so orphans always self-reap. File transfer is
workspace-contained with a size guard. Includes mocked unit tests and an
opt-in live smoke test.
Introduce a pluggable CodeSandbox abstraction with a SandboxManager and a
Jupyter Kernel Gateway backend: the app is a client of a single always-on
runner that executes code in stateful in-process kernels (no child-container
spawning, no docker socket). Sessions bind to a conversation or workflow run
with an agent-selectable TTL clamped by a global cap; execution enforces a
wall-clock deadline with interrupt-on-timeout and capped output, and file
transfer is workspace-contained with size and integrity checks. Adds a
docsgpt-sandbox docker-compose service (resource-capped, read-only, internal
network) plus settings, with a real local-gateway integration test.
Serve artifact metadata and bytes over HTTP with parent-derived
authorization: conversation-parented artifacts inherit conversation
access (owner, shared_with, or a public share token whose conversation
matches the parent), workflow-run artifacts check run ownership, and
access fails closed when the parent is missing or deleted.
Adds list/get/versions/download/restore routes, an authenticated
storage-agnostic download (sanitized Content-Disposition, 302 to a
short-lived private S3 presigned URL when that strategy is configured),
a generate_presigned_url primitive on the storage base and S3 backend,
and generalizes the tools artifact endpoint for documents/files. Shared
authorization helpers live in a dedicated module used by both surfaces.
Introduce artifacts and artifact_versions tables: an append-only,
pass-by-reference store with a stable identity row and immutable
versions that atomically bump current_version. Access is parent-derived
(conversation or workflow run); a CHECK enforces at least one parent and
a nullable team_id is reserved for future team scoping.
Adds the 0025_artifacts migration, ArtifactsRepository with a
parent-scoped safe getter alongside the unscoped one, and repo tests.
GraphStore.get_graph_overview (top-N by degree, bounded) + get_node_detail (description + linked chunks). GET /api/sources/<id>/graph + /graph/node/<id> (read-access gated, node scoped to source_id; empty graph -> {nodes:[],edges:[]}). Frontend GraphView (react-force-graph-2d) wired to config.kind=graphrag via card-click/'View graph'; node-click -> description + chunks; tooltip renders untrusted names as text (no innerHTML XSS). All read-only GraphStore methods rollback their txn (no idle-in-transaction lock). Unit G7. (Also a pre-existing prettier fix in WorkflowPreview to keep lint green.)
graph_enabled() sets kind=graphrag + retriever=graphrag. extract_graph_worker fetches the source's pgvector chunks and runs G3 extraction (graphrag_available guard, empty no-op). extract_graph durable+idempotent task; key varies with source updated_at so re-ingest/re-enable re-run incrementally (G3 checkpoint skips done chunks) while concurrent same-state enqueues dedup. The 4 ingest paths enqueue after embed when kind=graphrag (isolated in try/except so a broker hiccup can't fail the ingest). POST /api/sources/<id>/graphrag/enable: pgvector+GRAPHRAG_ENABLED gated, owner/editor write-authz; PATCH config still can't flip kind->graphrag. Unit G4.
Per-chunk schema-constrained LLM extraction (gleanings off, one .gen() per chunk) into the GraphStore: model resolves config.graph.extraction_model -> GRAPHRAG_EXTRACTION_MODEL -> instance LLM_NAME; token_usage logged per call (_token_usage_source=graph_extraction, owner-attributed via decoded_token); chunk text fenced as untrusted; robust JSON parse (malformed/errors skip the chunk, never crash); resumable via graph_ingest_progress; max_chunks cap; entity names embedded + merged by normalized_name; chunk_id = doc_id so links match retrieval. Unit G3.
On-demand graph_nodes/graph_edges/graph_node_chunks/graph_ingest_progress in the same DB as pgvector documents (no Alembic, no cross-DB FK; Python uuids; parameterized SQL). upsert_node (merge by normalized_name, doc_freq, dedup description), add_edge (+degree), link_node_chunk, search_nodes_by_embedding (cosine), bounded get_subgraph, checkpoint, delete_by_source. Embedding dim derived from the configured model (768 fallback). Unit G2.
sources.tokens was only set at ingest, so wikis showed no/stale token count on the card (blank wikis showed nothing; converted ones showed the stale original count). rebuild_wiki_directory_structure (called after every wiki mutation) now also sets sources.tokens to the sum of wiki_pages.token_count; blank wiki create sets tokens=0. So the card reflects live wiki content.
convert_source_to_wiki now builds pages from the source's already-ingested vector-store chunks instead of re-parsing storage files, so it works for crawler/remote/connector sources (which have no stored files; file_path empty). Groups by metadata file_path/file_name (URL- and connector-aware; URLs normalized to virtual paths) rather than the raw source. Passes embeddings_key to create_vectorstore in BOTH convert and reembed_wiki_page (fixes a TypeError crash on faiss/elasticsearch). Conservative overlap-trim (min length). Deletes the original chunks after reassembling so retrieval has no duplicate/stale content.
Unit 6 of F-Wiki (D22 + D24). Migration 0024 adds wiki_pages.updated_via; set to 'agent' by WikiTool + convert, 'human' by the edit/seed endpoints (content-hash short-circuit preserves it). WikiViewer gains a markdown editor (Save via PUT with expected_version; 409 reloads latest while keeping the draft; 403 graceful) and a per-page provenance stamp (editor/when/version). Edit gated on write access client-side; backend remains the real authz.
Unit 5 of F-Wiki (D20-D23). convert_source_to_wiki task reuses reingest's storage file-load + parser to materialize files->wiki_pages (one page/file), skips/reports non-text, re-embeds per page, and flips kind=wiki + exposure=agentic_tool only when pages were created. POST /wiki/convert (explicit, write-authz; blank source enables inline, fileful enqueues the task; rejects mid-ingest). PUT /wiki/page for human edits (write-authz, optimistic version -> 409, re-embed). PATCH /config preserves kind (kind changes only via convert).
Unit 4 of F-Wiki. POST /api/sources/wiki creates a type=wiki/kind=wiki source with no ingest task (optional seed page enqueues reembed). GET /wiki/pages + /wiki/page serve the tree + fresh page content, read-access gated (owner or team grant), path-validated. Frontend: 'Create Wiki' ingestor entry, read-only WikiViewer (FileTree + react-markdown, no raw HTML), sync/reingest hidden for wiki sources.
Unit 3 of F-Wiki. WikiTool (view/create/str_replace/insert/delete/rename) over wiki_pages: exact-case unique str_replace (no silent multi-replace), optimistic version on edits (WikiPageConflict), reads served fresh from Postgres, untrusted-content fencing on reads, 1MB page cap. Injected via add_wiki_tool only for writable wiki sources (effective_write_owner: owner/team-editor; viewers get nothing), scoped to one source_id. Each mutation enqueues reembed_wiki_page (owner as user, per-page idempotency key) and rebuilds directory_structure. Shared validate_tool_path extracted from MemoryTool.
Per-page re-embed (Unit 2 of F-Wiki): targeted delete of the page's old chunks, re-chunk via the source's chunking config, add_chunk with reingest-matching metadata (source=path), set embed_status embedded/failed. Durable + idempotent (key=content_hash), mirroring reingest_source_task. Missing page => purge only.
Migration 0023_wiki_pages (source-scoped; version/content_hash/embed_status; FK source_id->sources ON DELETE CASCADE; UNIQUE(source_id,path) + prefix index). WikiPagesRepository: path-keyed CRUD with content-hash short-circuit, version bump, and move-with-reject. BaseVectorStore.delete_chunks_by_source_path (loop default) + parameterized pgvector override (DELETE WHERE source_id=%s AND metadata->>'source'=%s). Unit 1 of F-Wiki.
Alex