Commit Graph
5680 Commits
Author SHA1 Message Date
arc53-machine 473e8ec15c Validate tool edits before touching the connection; scope kept MCP connections
- Editing a connection-backed tool validates the config before writing
  new credentials, so a rejected edit no longer rotates the connection's
  key and resumes its sources.
- An MCP edit keeps its previous connection only when the caller owns it
  and it signs in the way the tool now does.
2026-09-28 19:31:02 +01:00
arc53-machine 4d533d24d6 Test connection-backed tools, MCP refresh and API failure paths
Covers creating and editing tools whose key lives on a connection, the
MCP action refresh, and the auth, ownership and error branches of the
connections API. Editing a tool's key now also recovers a connection
whose stored credentials can no longer be decrypted.
2026-09-28 19:11:54 +01:00
arc53-machine fff3e7c904 Never send an MCP connection's secret to a different server
Editing an MCP server's URL kept the previous connection when no new
one could be stored (default encryption key), so its key would reach the
new server. The save route now keeps a previous connection only for the
same server, and the executor refuses an MCP tool whose connection was
stored for another server.
2026-09-28 19:11:54 +01:00
arc53-machine 269aa13bba Address review: tool group order in the agent builder, saved S3/Reddit account
- Connection groups sit between the default and custom tools; a second
  sort that did not know connection labels moved them to the top.
- Upload picks the first saved account when connections load after the
  modal opened on S3 or Reddit, without overriding the user's choice.
2026-09-28 18:45:27 +01:00
arc53-machine 6b4220d401 Address review: MCP token scope, API-key identity, reconnect lock, setup retries
- An MCP connection's tokens only go to its own server, and a
  connection_id can no longer come from a client: the MCP test/save
  routes drop it and the tool executor uses only the resolved one.
- API-key connections are reused only when they hold the same
  credentials. Two keys that share a hint or a label get separate
  connections ("…abcd (2)"), at runtime and in the 0038 backfill.
- Reconnecting a connection whose secrets cannot be decrypted no longer
  flags it from a second transaction while holding its row lock.
- Connection setup validates the sync request before claiming its
  Idempotency-Key, so a corrected retry is queued.
- A request that names no connection resolves to none without opening
  a database connection; fixes two tests that reached a real database.
2026-09-28 18:45:27 +01:00
arc53-machine 3e11442a83 Merge remote-tracking branch 'origin/main' into connectors
# Conflicts:
#	frontend/DESIGN.md
#	frontend/src/locale/de.json
#	frontend/src/locale/en.json
#	frontend/src/locale/es.json
#	frontend/src/locale/jp.json
#	frontend/src/locale/ru.json
#	frontend/src/locale/zh-TW.json
#	frontend/src/locale/zh.json
#	frontend/src/upload/Upload.tsx
2026-09-28 18:07:12 +01:00
arc53-machine 8f00118afc Document connectors and the upgrade to encrypted connections
- Guides > Connectors: using connections, sharing modes, attribution,
  and admin setup (encryption key and rotation, redirect URIs, Google
  Drive, SharePoint, Confluence, S3, MCP presets).
- Integration pages register CONNECTOR_REDIRECT_BASE_URI as-is; the
  ?provider= suffix never matched what the backend sends.
  VITE_GOOGLE_CLIENT_ID is optional.
- Upgrading: set ENCRYPTION_SECRET_KEY before migrating multi-user installs.
2026-09-28 17:59:36 +01:00
arc53-machine d9f823d601 Stop syncing sources of a connector an admin turned off
Scheduled syncs checked the connection's status but not the admin
policy, so a disabled connector kept pulling data. Connector sources now
return "disabled" and S3/Reddit syncs get no credentials until the
connector is turned back on.
2026-09-28 17:59:36 +01:00
arc53-machine 2411118048 Show which service did what, and let owners pick whose account a share uses
- Tool chips and the streaming status name the service: "Searched
  Notion", "Used Linear: create issue". Citations from a connector
  source read "From Google Drive".
- The share dialog for a connection-backed tool asks whose account team
  members use. Owner mode warns which account they act as and needs an
  explicit OK when the tool can write; an admin-forced mode is locked.
- Admin > Connectors: enable or disable each connector, force a sharing
  mode, allow custom MCP servers, and a setup guide per OAuth connector
  with its redirect URI and the server settings still missing. Warns
  when credentials use the default encryption key.
- The composer and the agent builder group tools by connection, and the
  Sources and Tools popovers link to Connect more.
- MCP presets open the server form titled after the service.
2026-09-28 17:55:56 +01:00
arc53-machine cefb117277 Keep tool keys when rolling back 0038
Downgrade removes API-key connections and assumed every tool still had
its v1 copy. Tools added through the wizard after the upgrade never had
one, and a reconnect can change the key after the backfill, so write the
connection's current credentials back to each linked tool first.
2026-09-28 17:55:56 +01:00
arc53-machine b692eaf7ad Add MCP presets, admin connector policies and connector attribution
Six vendor-run MCP servers (Notion, Linear, Atlassian, Sentry, Asana,
Stripe) join the catalog as presets from docsgpt/connectors/presets/
mcp.yaml; existing connections to those servers show under them.

connector_policies lets an admin turn a connector off or force whose
account every share uses, and app_metadata's
connectors.allow_custom_mcp turns custom MCP servers off. Both are
enforced on the server: new connections, OAuth sign-ins, MCP test and
save, and the tools of a disabled connector stop resolving.
GET/PUT /api/admin/connectors reads and changes them.

A shared tool's owner picks owner or member credentials with
PUT /api/connections/tools/<id>/credential-mode, and a member running
the owner's account always confirms write actions.

Tool-call events and retrieved chunks name the connector they came
from (key and display name, never the account), and tool calls keep
those fields when the conversation is reloaded.
2026-09-28 17:30:53 +01:00
arc53-machine 6e2ee43944 Connect services from one wizard, with no tokens in the browser
Every entry point (the Connectors page and drawer, Add Source, Add
Tool, the chat) opens one connect wizard: sign in or paste the
credentials, choose what to sync (content services only, skippable),
then a summary with Try it in chat. Tool services create their tools on
sign-in with write actions needing approval, and each action's
permission (Always allow, Needs approval, Off) can be changed in the
wizard or the drawer. The drawer also reconnects, syncs more content,
refreshes an MCP server's tools and removes a connection, keeping or
deleting what it feeds.

The file pickers work from a connection id and fetch a short-lived
picker token when they need one; no session token is kept in
localStorage any more, and any left by an older build is claimed once
and removed. S3 and Reddit reuse the keys stored on a connection.

A tool whose account needs signing in pauses on a Connect card in the
chat; after connecting, Continue resumes the call. A connection whose
sign-in stops working raises a Reconnect toast. The Connectors page is
hidden when the backend does not report connectors_enabled.
2026-09-28 17:21:44 +01:00
arc53-machine 3511004f50 Connections own their credentials
Migration 0038 moves every stored secret (OAuth tokens, MCP OAuth
tokens and client registrations, API keys) into the connection's
encrypted envelope, links API-key tools to one connection per distinct
credential, allows several accounts per provider, and adds
credential_mode to sources and tools. OAuth MCP tools keep resolving
each member's own token, as they did before.

docsgpt.connectors.service is now the only reader of OAuth tokens:
get_valid_token_info refreshes under a row lock and persists rotated
refresh tokens, and a revoked grant flags the connection, pauses its
sources and notifies the owner. Loaders build from a connection
(BaseConnectorLoader.from_connection), so scheduled sync covers Drive,
SharePoint and Confluence sources with no browser. S3 and Reddit keys
stay on the connection instead of in remote_data.

New endpoints: POST /api/connections, /setup, /reconnect,
/picker-token, /claim, DELETE /api/connections/<id>, per-action
permissions and MCP refresh-tools. Upload, file listing, sync and
validate-session take a connection_id; session tokens keep working for
this release. The tool executor reads credentials from the resolved
connection (owner or member mode) and pauses on a Connect card when a
connection needs signing in. docsgpt connectors reencrypt rewrites
stored credentials after a key rotation.
2026-09-28 17:03:09 +01:00
arc53-machine 667d4bbab0 Encrypt connection credentials with an owner-bound AES-GCM envelope
Adds a v2 credential envelope next to the v1 tool-secret helpers:
AES-256-GCM, a master key derived once per process from
ENCRYPTION_SECRET_KEY, and a per-record key from HKDF over the owner's
id, which is also the associated data, so a blob moved onto another
user's row does not decrypt. The envelope names its key, so
ENCRYPTION_SECRET_KEY_PREVIOUS keeps old rows readable during a
rotation.

Log redaction now also covers token_info, tokens and client_info, and
the API warns at startup when the public default key is in use.
2026-09-28 17:03:02 +01:00
arc53-machine 45c4d928c4 Group Add Source and Add Tool by where things come from
Add Source splits into Upload and web, and From a connection. Connector
tiles are always shown and say whether an account is connected or the
server still needs setup, instead of disappearing when a VITE_ client
id is unset; without the Google Picker key, Drive falls back to the
server-side file browser. The name follows the first picked item and
the button reads Add source.

Add Tool splits into Built in and From a service. The MCP and API tool
cards move to Connectors > Add custom connector.
2026-09-28 16:30:56 +01:00
arc53-machine aad43c2fd0 Add the Connectors page
Settings > Workspace gets a Connectors page between Sources and Tools:
a searchable catalog filtered by category, one card per service with
its capabilities (Sync, Read, Write) and state (Connect, Connected,
Reconnect, Needs admin setup, Disabled by admin, Custom). A card opens
a drawer listing the accounts, the sources each syncs and the tools
each provides, with Reconnect and Disconnect.

Tools and Sources show which connection an item comes from and when
its sync is paused. Brand icons now draw in currentColor so they
follow the theme, and all new copy is translated in every locale.
2026-09-28 16:30:56 +01:00
arc53-machine 5321a94388 Add the connector catalog and read-only connections API
connector_sessions already holds one row per signed-in account or MCP
server. Migration 0038 names each row (connector_key, auth_kind,
display_name, account_label) and lets sources and tools point at the
row they use through connection_id, backfilling the links it can prove.

docsgpt/connectors/catalog.py declares every service DocsGPT connects
to, with the server settings it needs and what a connection provides.
New endpoints: GET /api/connectors/catalog, GET /api/connections,
GET /api/connections/<id> and POST /api/connections/<id>/disconnect.
/api/available_tools now groups tools into built_in, service and
custom, and source listings carry connectionId.
2026-09-28 16:30:48 +01:00
Pavel ac7bbd811f Merge pull request #2842 from arc53/Agent-menu-UI
Agent menu UI
2026-09-28 17:20:26 +02:00
Pavel b12f5b6c81 Chunk nav fixed 2026-09-28 19:02:50 +04:00
Pavel f4331cd3a3 rabbit fixes 2026-09-28 18:40:49 +04:00
Alex 335f2af70c Merge pull request #2841 from arc53-machine/default-chunks-6
Default retrieval to 6 chunks instead of 2
2026-09-28 15:19:02 +01:00
arc53-machine 56aebc882c Cover the chunks fallbacks on agent update and YAML import
A blank chunks on update and an unparsable one on import both fall back
to the default.
2026-09-28 14:55:43 +01:00
arc53-machine c40f06e10b Say that chunks can be exceeded when sources outnumber it
Every attached source gets at least one chunk, so eight sources at
chunks: 6 return eight.
2026-09-28 14:45:06 +01:00
Pavel cdbc4695ea main sync 2026-09-28 17:41:29 +04:00
Pavel 706a0cb2b2 Big source revamp 2026-09-28 17:35:16 +04:00
arc53-machine 527880d3b8 Document the 6-chunk default and that chunks is a total
The agent basics page still described the Chunks per query field the form
no longer has.
2026-09-28 14:34:50 +01:00
arc53-machine ab3dbfa1be Default retrieval to 6 chunks instead of 2
chunks is a total per request, split across the attached sources, so an
agent with two sources and the default of 2 got a single chunk from each,
and its answers changed with whichever chunk won. 6 gives three per
source for about 4-5k more input tokens per retrieval turn.

Every literal default moves from 2 to 6: the request default, the
retrievers, the internal search tool, workflow agent nodes, scheduled and
headless runs, agent create/update/import, the source retrieval config and
the frontend forms. Existing agents and sources keep what they store; a
source saved with chunks=2 now counts as configured at 2, which is pinned
by a test.

Headless runs also read chunks=0 as unset (`or 2`), so an agent with
retrieval switched off retrieved anyway on scheduled runs; 0 now stays 0.
2026-09-28 14:34:50 +01:00
Alex 1c19ed8ce2 Merge pull request #2840 from arc53-machine/math-rendering-parser-level
Render chat math in the markdown parser: \( \), \[ \], $x$, mhchem, streaming
2026-09-28 13:50:39 +01:00
arc53-machine 1c9cf7b693 Open the sources list from a citation that has no card
Only the first three sources are shown as cards, so a pill for [4] or
later scrolled to nothing; it now opens the full list.
2026-09-28 13:30:55 +01:00
arc53-machine c0c587de81 Keep answer math in its frame and its case
A display formula wider than the column scrolls in its own box, as code
and tables do, and a formula in a table header is not uppercased: n and N
are different variables.
2026-09-28 13:21:28 +01:00
arc53-machine 845e39969e Link only citations that have a source card to jump to
A pill for [N] beyond the answer's sources, or on an answer whose sources
are hidden, scrolled to nothing.
2026-09-28 13:21:28 +01:00
arc53-machine 2496374904 Parse chat math in the markdown parser, not with string rewrites
Inline \( \) math rendered as a literal $x$ (the rewrite produced single
dollars, which remark-math is told to ignore), a one-line \[ \] shrank to
inline math, model-written $x$ never rendered, and the citation pass
linked [1] inside formulas.

- \( \) and \[ \] are micromark constructs: inline anywhere, display when
  \[ starts a line and its \] ends one. A line such as "\[a\] and more"
  stays inline instead of swallowing the rest of the answer, which is why
  this does not alias micromark-extension-llm-math.
- $x$ follows Pandoc's rules (ported from LibreChat, MIT), so prices stay
  text.
- A paragraph holding only $$ or \[ math, and anything KaTeX can only set in
  display mode (\tag, align, ...), becomes display math.
- mhchem is loaded for \ce and \pu.
- Citations are linked on the syntax tree, so code, math, link text and
  escaped brackets are never touched.
- While streaming, open emphasis, code and $$ / \[ math are closed (remend
  plus a code-aware math pass) and a formula KaTeX cannot set yet shows in
  the muted colour. Each top-level block renders on its own and is
  memoised, so only the growing block is parsed and typeset again.
2026-09-28 13:21:28 +01:00
Manish Madan 0455f37a1d Merge pull request #2774 from arc53/dependabot/npm_and_yarn/frontend/vitest-5.0.0
build(deps-dev): bump vitest from 4.1.11 to 5.0.0 in /frontend
2026-09-28 17:35:16 +05:30
Alex aa0ce42fd7 Merge pull request #2839 from arc53-machine/failed-attachment-never-blocks-send
A failed attachment never blocks Send: drop it and send the question
2026-09-28 12:21:30 +01:00
arc53-machine 3a156ff21d Name the file in the remove button's failure description
The remove button's label is "Remove attachment" and doesn't say which
file. Its failure description now reads "<file>: <reason>", the same text
as the tooltip.
2026-09-28 10:47:30 +01:00
arc53-machine 7e990d4384 Name the file in a failed chip's tooltip and announce failures
Follow-ups to moving the failure reason into a tooltip:

- The tooltip now reads "<file>: <reason>". The name is truncated on the
  chip, and two long names can look the same.
- A polite role="status" region, always mounted and screen-reader only,
  lists the failed files and their reasons. The red line it replaces was
  role="alert", so failures were announced; a hover tooltip is not.
- The reason now describes the remove button (aria-describedby on a
  hidden span), the chip's only focusable control. Before, it was sr-only
  text on a chip that can't take focus.
2026-09-28 10:37:09 +01:00
arc53-machine b8a62cedf9 Show a failed attachment's reason on hover, not as a red line
The composer showed each failed file's reason as a red line of text under
the chips. A failed file no longer blocks the send, since it is dropped
then, so that line was louder than it needed to be. DESIGN.md also says
status text of a sentence or more must not be a coloured paragraph.

The chip's warning icon already marks the failure. The reason is now a
Tooltip on the failed chip. It falls back to "Failed" when no reason is
known. It is also kept as sr-only text so screen readers still read it.
The native title moves from the chip to the truncating file name, where
DESIGN.md keeps title=, and is left off failed chips so the tooltip
doesn't double up.
2026-09-28 10:27:12 +01:00
arc53-machine dc5ebf5bb9 e2e: a failed attachment no longer blocks the send
Aborts /api/store_attachment, attaches a file, waits for the failed
chip's reason, then sends. Checks that one /stream request goes out with
no attachment ids, that the question is in the thread and the chip is
gone, and that the stored message has no attachments.
2026-09-28 10:06:25 +01:00
arc53-machine 2acfa34bc5 Fail stalled attachment uploads and say why an upload failed
Uploads had no timeout, so a stalled XHR could leave a chip on
"uploading" forever, and a queued send waited with it. Network failures
also set no errorMessage, so the failed chip showed no reason.

guardUploadStall aborts an upload once it goes 120 s without progress.
It restarts on every upload progress event and again when the body
finishes sending. A large file on a slow link keeps going; only a silent
stall is cut off. That is why it is not xhr.timeout, which caps the whole
request against a 100 MB file limit. onerror, onabort and ontimeout now
mark the chip failed with "Upload failed. The file couldn't be read or
the connection dropped." (single and multi-file paths). A failed chip is
dropped at send, so a queued question always goes out in the end. The
5-minute processing cap is unchanged.
2026-09-28 10:06:21 +01:00
arc53-machine be6fe8d0c2 Hold a queued send while an answer is streaming
A queued send flushed by calling submitNow directly, which skipped the
loading check in canSubmit. submitNow clears the composer before the
consumer runs, and the consumer ignores a submit while status is
'loading'. If a retry or an edit was streaming when the attachments
settled, the question was cleared and never sent.

useArmedSend takes a canFlush flag. The composer passes whether it can
take a submit right now (not loading, not recording or transcribing).
The armed send stays queued until that holds, then goes out.
2026-09-28 10:05:39 +01:00
arc53-machine a9c0082e9a Send with a failed attachment instead of blocking
A failed attachment made getSendReadiness return 'blocked', so Send was a
no-op until the user found and removed the chip. On a phone the only hint
was a small red line after the tap, and a new user lost their question
that way. A failed file never recovers, so holding the send only costs
the question.

Readiness now ignores failed attachments: it waits only on uploading or
processing files. On submit the composer drops every failed attachment
and sends the text with the completed ones. The payloads already read
completed attachments only, so no failed id reaches the request or the
history. The hero-card path in Conversation also clears failed chips.
The failed chip still shows before Send and can be removed by hand.

Removes the send-blocked alert and its locale key.
2026-09-28 10:05:19 +01:00
Pavel 4070b03053 Workflow revamp 2026-09-27 21:31:01 +04:00
Pavel d4c97a49ce Agent menu in new style 2026-09-27 19:45:05 +04:00
Pavel 20bcd92942 updated drawer 2026-09-27 16:53:18 +04:00
Pavel ed804a14e9 Cards standartization 2026-09-27 16:17:59 +04:00
Alex d37e0bd1ad Merge pull request #2834 from arc53/iphone-modal-fixes
Iphone modal fixes
2026-09-27 10:42:34 +01:00
Pavel 231849c957 Design additions 2026-09-27 13:00:11 +04:00
Pavel a63234e7d9 theme switch fox 2026-09-27 12:54:51 +04:00
Pavel 4e03bc0388 bottom color on modal 2026-09-27 12:37:41 +04:00
Pavel 4e6f1bf9e4 focus fixes 2026-09-27 11:58:51 +04:00