Commit Graph
237 Commits
Author SHA1 Message Date
arc53-machine 54ba7c7b9b Apply retrieval settings to a source synced from the connect wizard
The connection setup endpoint takes sync.config, validated like an
upload's config, and passes it to the ingest task so the synced source
gets the chosen chunking and retrieval settings. An invalid config is
refused before the idempotency key is claimed.
2026-09-29 12:17:33 +01:00
arc53-machine f2349edb4c Keep a Linear sync going when one issue or document cannot be read
An issue deleted after it was listed, or comments the token cannot read,
now lose that detail with a warning instead of failing the whole sync; an
unreadable document is skipped.
2026-09-29 12:14:10 +01:00
arc53-machine 06de211e61 Download an embedding model whose cached snapshot lacks its graph
The chunker caches only a model's tokenizer.json in the embeddings cache.
FastEmbed counts any cached snapshot as the model, so it never downloaded
the ONNX graph and every load failed with NO_SUCHFILE. The loader now
checks the files FastEmbed needs and fetches them first when they are
missing; offline it leaves them for FastEmbed to report.
2026-09-29 12:12:28 +01:00
arc53-machine 39063d5e41 Mark fixed values in the chat instead of showing them, and keep them on reload
A value the owner fixed may be a secret, and tool-call events reach whoever
runs the agent, so query and body values the owner fixed now show as
(fixed); a value the connection sets, like Telegram's default chat, still
shows. sent_arguments is saved with the conversation, so a reopened chat
shows the same arguments as the live one.
2026-09-29 12:09:13 +01:00
arc53-machine 03d1b10897 Sync Linear issues and documents into Knowledge with the Linear sign-in
The Linear connector now syncs as well as giving agents its tools, from
one connection. Linear's MCP server is its own OAuth issuer, so its
tokens are read through the same MCP tools the agents use (list_issues,
get_issue, list_comments, list_documents) rather than Linear's GraphQL
API, and no OAuth app has to be registered.

A source picks teams and projects, with comments (on by default) and the
projects' documents. Each issue becomes one document with its state,
assignee, priority, labels, description and comments, filed under its
team and citing its Linear URL. Each sync reads up to 500 issues and 100
documents again. /api/connections/<id>/linear lists the teams and
projects to pick from. Sources sync on their schedule with the owner's
connection, and pause when the sign-in needs reconnecting.
2026-09-29 11:32:40 +01:00
arc53-machine 805e65eeb4 Call an MCP server's tools from the server with a connection's sign-in
run_connection_session opens one MCP session signed in with an MCP OAuth
connection's stored tokens and runs a batch of tool calls in it, so a
sync makes hundreds of calls over one sign-in. The tokens only go to the
server the connection signed in to. A sign-in that expired and cannot be
renewed flags the connection for reconnecting, which pauses its sources;
network trouble and rate limits are reported as worth retrying.
2026-09-29 11:32:40 +01:00
arc53-machine 6eda4a6738 Renew a stored MCP sign-in once its access token expires
The MCP SDK knows a token's expiry only when it obtained the token in the
same process. A worker or a restarted API loading tokens from the
connection sent an expired one, got a 401 and asked the owner to sign in
again. The expiry is now saved with the tokens and handed back to the SDK,
which then renews the token with its refresh token first; a token saved
before expiries were kept is renewed once. A sign-in that cannot be
renewed raises MCPReauthorizationRequired.
2026-09-29 11:32:21 +01:00
arc53-machine 12ad4f5e85 Let a GitHub connection opt into write tools
A GitHub connection's MCP tool can now point at GitHub's full endpoint
(/mcp/) instead of the read-only one when its owner opts in, at setup
(allow_writes) or later (PUT /api/connections/<id>/writes), which re-reads
the actions and keeps the choices for those on both endpoints. Actions from
the write endpoint are writes unless GitHub marks them read-only, so they
default to asking first.

Admins can forbid it per connector (allow_writes in Admin > Connectors,
kept in app_metadata). Then the option is refused, a refresh goes back to
read-only, and at run time the tool only ever calls the read-only endpoint
and write calls are denied with a reason.
2026-09-29 11:22:01 +01:00
arc53-machine dba2de590b Show the arguments a tool call sends, fixed values included
The approval card and a finished call showed what the model asked for,
even where a fixed value replaced it. Calls now carry sent_arguments for
the chat, leaving headers out since they may hold a fixed secret. The
model's own arguments are kept as they were, because they are what later
turns replay to it and it never sees fixed values.
2026-09-29 11:08:50 +01:00
arc53-machine bdc453eaab Show the model an account only where it tells tools apart
Actions that different services share are described with the service
alone; the account is named only when one service is connected twice.
2026-09-29 10:42:03 +01:00
arc53-machine bdcb88866a Let people name accounts and use the names to tell accounts apart
Connections get an account_name (migration 0039) that the owner sets with
PATCH /api/connections/<id>; the account label stays the account's
identity, so signing in again still finds it. When someone has more than
one account of a service, its tools are listed as "Telegram · <account>"
and the model sees each account's actions under the account's name
(telegram_send_message_alerts_bot) with the account in the description,
instead of _1 and _2. Actions shared by different services are named
after the service. Tools a user renamed keep their name.
2026-09-29 10:41:54 +01:00
arc53-machine 607c2296e3 Show the Telegram default chat in the connect form and the drawer
The connect form shows field hints; Telegram's Default chat ID says what
it does and how to find a chat's id. In the drawer, a chat set on the
account shows as set there instead of as a choice.
2026-09-29 10:41:54 +01:00
arc53-machine c3e83ff9dd Add a default chat to the Telegram connector
A Telegram connection can now hold an optional default chat ID. When it
is set, both Telegram actions send there, the model is no longer asked
for a chat, and a chat it names anyway is ignored. In member mode each
member's own connection supplies their own chat. The same bot with a
different chat is a separate connection.
2026-09-29 10:41:48 +01:00
arc53-machine 28b56e8b03 Add an endpoint to fix a connection tool's parameters
PUT /api/connections/<id>/tools/<tool_id>/parameters takes an action and
a map of parameter to value (always use it) or null (let the model
decide), checked against the action's schema; only the tool's owner can
call it. Connection tools now list each action's parameters and whether
they are fixed.
2026-09-29 10:41:20 +01:00
arc53-machine 54bea5a1d8 Keep fixed values when an MCP server's tools are refreshed or re-saved
Parameters that still exist after re-reading the server keep the value the
user fixed; parameters the server dropped go with the old schema.
2026-09-29 10:41:20 +01:00
arc53-machine f6bdd853d5 Validate tool action updates and keep fixed values owner-only
update_tool_actions now checks the submitted actions against the tool's
stored ones: no new actions or parameters, fixed values must fit their
parameter's type. A team editor can still switch actions on and off but
gets 403 for changing a fixed value.
2026-09-29 10:41:13 +01:00
arc53-machine 34038ae1c2 Keep fixed tool parameters fixed when the model sends them
A parameter hidden from the model with a stored value is now always sent
with that value; a model that names the key anyway (by mistake, or because
a prompt told it to) no longer overrides it. 0 and false are real fixed
values; an empty value still means the parameter is left out.
2026-09-29 10:41:13 +01:00
arc53-machine bbf83f02b2 Skip scheduled sync of a remote source paused for reconnect
S3 and GitHub sources synced from a connection kept being retried on
every schedule after their connection needed reconnecting, failing and
notifying the owner each time. They now wait, like Drive and Confluence
sources, and resume when the connection is reconnected.
2026-09-29 10:41:03 +01:00
arc53-machine 74cbf68ffc Show GitHub's optional GitHub App settings in Admin > Connectors
The admin connectors API lists the settings that add Sign in with GitHub
and whether they are complete, next to the required settings (none for
GitHub, which works with tokens alone).
2026-09-29 10:41:03 +01:00
arc53-machine 7b516ade82 Add a built-in GitHub connector for repository sync and read-only tools
One GitHub connection feeds both a Knowledge source and an agent tool.
Users connect with a personal access token, checked against GitHub and
named after the account, or, when an admin registers a GitHub App
(GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, GITHUB_APP_SLUG), with Sign in
with GitHub. App tokens expire after eight hours and are refreshed before
each sync or tool call; tokens with no expiry are never treated as expired.

The catalog gains an optional second sign-in method (oauth_settings,
exposed as sign_in_methods) without changing any other connector.

Sync lists the repositories the connection can read (the token's own, or
the App installations') and ingests one with the connection's token. The
tool is GitHub's read-only MCP server (api.githubcopilot.com/mcp/readonly):
setup discovers its actions and creates it bound to the connection, and
the executor sends the connection's token only to that server.
2026-09-29 10:41:03 +01:00
arc53-machine 3572d968ec Read private repositories only with the user's own GitHub token
GITHUB_ACCESS_TOKEN belongs to the server, but any user could ingest any
repository it can see, private ones included. It is now used only for
public repositories; the loader checks the repository's visibility first
and asks for a GitHub connection otherwise.

The loader also takes a token from the connection a source syncs from.
Merging a connection's keys into a plain repository URL no longer fails
on json.loads, manual Sync now passes the source's connection, and a
token GitHub rejects pauses the connection's sources for reconnect.
2026-09-29 10:41:03 +01:00
arc53-machine 35d6d80e87 Key cached MCP clients by a digest of the whole token
The cache key held the first ten characters of a bearer token or API key.
Every fine-grained GitHub token starts with github_pat_, so two users of
the same server shared one cached client, and with it the first user's
token, for up to five minutes.
2026-09-29 10:41:03 +01:00
arc53-machine bacab4cfe7 Check a preset's switch for a key-based connection to its server
A custom MCP connection at a preset's address is reported and run as that
preset, but saving it checked the custom MCP switch. Turning custom servers
off now leaves a preset on a key alone, and turning the preset off stops it.
2026-09-29 09:36:25 +01:00
arc53-machine 2cb23938f7 Regroup connector categories and give empty Knowledge a way in
Linear, Jira & Confluence and Asana move to Projects & issues, Stripe to
Payments & business and Reddit to Search & web. An empty Knowledge page
offers Add knowledge and Connect a service.
2026-09-29 00:54:07 +01:00
arc53-machine 8cf46f2e57 Give a different account its own connection when reconnecting 2026-09-29 00:43:48 +01:00
arc53-machine cdc5084e02 Refuse a disabled connector before falling back to a legacy tool secret 2026-09-29 00:41:57 +01:00
arc53-machine a260ca44f1 Check a tool's connection before writing its permissions 2026-09-29 00:40:33 +01:00
arc53-machine 33f29cb408 Authenticate MCP token requests one way under Basic auth
With client_secret_basic the MCP SDK sets the Basic header and still sends
client_id in the body; Linear refuses that as two authentication methods.
Token and refresh requests now drop the body client_id when the Basic
header is set, as RFC 6749 describes.
2026-09-28 23:43:49 +01:00
arc53-machine 0d42916d67 Hand the MCP SDK an authorization result with the issuer
mcp 2.x reads code, state and the RFC 9207 iss off the callback handler's
result; the handler still returned a (code, state) tuple, so every MCP
sign-in failed after the user approved it. The callback route now keeps the
issuer too, since servers that advertise it (Linear) refuse a sign-in
without it.
2026-09-28 23:32:21 +01:00
arc53-machine cdaaaaf427 Prefer the configured MCP OAuth callback over the page's origin
A page opened on a plain-HTTP address registered its own origin as the
callback, which servers such as Linear refuse. MCP_OAUTH_REDIRECT_URI now
wins when set; the page's value is the fallback.
2026-09-28 23:20:44 +01:00
arc53-machine 46928cba3d Return the OAuth task id from the MCP test route
The client follows an MCP sign-in by its task's events, but the test route
kept only success, requires_oauth and auth_url, so the wizard and the custom
MCP modal reported a failed sign-in before the popup could load.
2026-09-28 23:16:42 +01:00
arc53-machine a9390ec39e Fix MCP OAuth for servers that publish their endpoint as the resource
Linear and Sentry publish https://host/mcp as their protected resource.
DocsGPTOAuth gave the MCP SDK only the server's origin, so the SDK's
resource check (RFC 8707) failed before sign-in with "Protected resource
https://mcp.linear.app/mcp does not match expected
https://mcp.linear.app". The provider now gets the full MCP URL; servers
that publish their origin still pass, and tokens stay keyed by origin.
2026-09-28 22:57:54 +01:00
arc53-machine 3b1288097d One vocabulary: Knowledge, tools, and connectors in plain words
- Sources (the collection) are called Knowledge: the settings page (now
  /settings/knowledge; /settings/sources redirects), the composer and
  agent pickers, Add knowledge, and "syncs into Knowledge" on connectors.
  Citations keep "sources". All seven locales.
- Connector cards describe what they do ("Syncs into Knowledge", "Looks
  things up", "Takes actions") and drop publisher labels ("Built in",
  "Preset"). Filters read "Files & storage" and "Docs & wikis" so
  "knowledge" means one thing.
- Custom tools are tools: "Add custom tool". The agent picker no longer
  splits built-in tools into "Built-in" and "Default".
- Confluence is one card: syncing pages into Knowledge and the Jira &
  Confluence MCP actions (part_of on the preset) share a page. The part
  shows on its own when Confluence sync is not set up.
2026-09-28 22:31:26 +01:00
arc53-machine df70869dc7 Simplify Admin > Connectors
- One control for members' own MCP servers: the custom MCP server row's
  switch (it sets both the policy and the instance switch); the separate
  top-level switch is gone.
- Connectors that only sync content show "No tools" instead of a
  sharing policy, and the policies read "The sharer decides per share",
  "Always the sharer's account", "Always each person's own account".
- A connector that needs server settings can't be switched on until they
  exist; a tooltip says why.
- On phones the table becomes a list ("On · 2 connections · …") and each
  connector's controls open in a sheet.
2026-09-28 21:52:18 +01:00
arc53-machine 23229181f1 One connect flow and one name from every entry point
- Add Source hands a connection tile to the connect wizard: an existing
  account goes straight to choosing what to sync, otherwise it connects
  one. The legacy per-service forms stay only for installs without
  connectors, so S3 no longer has two different forms.
- Add Tool lists services from the connector catalog first, MCP presets
  included, each with its connection state; a connected one opens its
  drawer, the rest start the wizard. Built-in tools follow.
- Service tools carry the connector's name everywhere ("Telegram", not
  "Telegram Bot"), and the composer groups custom tools (API tool, MCP
  servers) as Custom, like the agent builder.
- OpenAPI opens the API tool editor as an unsaved draft; the tool is
  created on its first save, so exploring the menu leaves nothing behind.
2026-09-28 21:49:01 +01:00
arc53-machine bfcc4a9773 Sign in to MCP presets from the connect wizard
Notion, Linear, Atlassian, Sentry, Asana and Stripe no longer open the
MCP server form. The wizard shows one "Sign in to Notion" button: the
pop-up opens inside the click (so browsers allow it), follows the
worker's mcp.oauth events to the provider, and the tool is saved on
success with its actions on the done step. Reconnecting a preset uses
the same flow, as do the chat bar and the health toast.

Saving an OAuth MCP server without a new handshake now uses the stored
sign-in instead of failing, and the custom MCP form keeps scopes and
timeout under Show advanced.
2026-09-28 21:42:50 +01:00
arc53-machine f6c89503d8 Reconnect in place from the chat bar and the health toast
- The Connect card's payload carries connection_id for the caller's own
  connection (never an owner's), so "Reconnect" in chat signs in again
  right there instead of sending the user to Settings. The bar offers
  Continue as soon as the connection works again, also after leaving and
  coming back to the chat.
- The reconnect event carries source and tool counts, and the toast says
  what stopped: "3 sources are paused", "so agents can use it again",
  or both. Reconnect signs in again in place and the toast closes itself
  once the connection works; MCP servers still reconnect from their
  drawer.
2026-09-28 21:37:19 +01:00
arc53-machine a23ace3491 Refuse writes on the owner's accounts from API-key callers unless allowed
An agent called with its API key (widget, API) runs as its owner, and
nobody can approve an action there, so a write set to Always allow ran
on the owner's account for anyone holding the key. A caller is external
when the request carries the key and is not signed in as the owner (an
owner previewing their agent keeps full access).

For external callers, write actions on connection-backed tools are
refused unless listed in the agent config's new api_write_allowlist
(tool_id:action), and a missing connection is refused instead of pausing
on a Connect card the widget cannot show. The flag and list survive a
paused-and-resumed stream. Owners pick the allowed actions under Access
details; a team editor's update keeps the stored list.
2026-09-28 21:33:41 +01:00
arc53-machine 7c6154749a Address review: model metadata, serialized admin saves, member account choice
- models.py: connector_policies.enabled is the nullable column; the
  earlier edit had loosened user_custom_models.enabled instead. A test
  pins both against their migrations.
- Admin > Connectors saves one change at a time, so an older snapshot
  can no longer overwrite a newer one.
- In member mode, a member with several connected accounts of the same
  service uses the one they used (or connected) most recently, not the
  oldest.
2026-09-28 20:59:36 +01:00
arc53-machine a564dea401 Hide connectors that need admin setup; they start turned off
A connector's enabled switch is now optional: unset means on when the
connector has the server settings it needs, so Google Drive, SharePoint
and Confluence start off until their OAuth settings are present, and an
admin's explicit switch always wins. Changing only the sharing mode no
longer switches a connector on.

Members only see connectors they can use. The catalog, Add Source tiles
and Add Tool leave out anything turned off or still needing setup, except
a connector a member already has a connection to, which stays listed as
turned off so it can be managed or removed.
2026-09-28 20:16:40 +01:00
arc53-machine 324d54ea67 Validate configs on the existing-connection path; MCP policy fails closed
- Creating a tool from an existing connection validates its config too;
  the connection supplies the secrets the request leaves out.
- When the connector policies cannot be read, the MCP test and save
  routes answer 503 instead of contacting the server.
2026-09-28 19:47:28 +01:00
arc53-machine 473e8ec15c Validate tool edits before touching the connection; scope kept MCP connections
- Editing a connection-backed tool validates the config before writing
  new credentials, so a rejected edit no longer rotates the connection's
  key and resumes its sources.
- An MCP edit keeps its previous connection only when the caller owns it
  and it signs in the way the tool now does.
2026-09-28 19:31:02 +01:00
arc53-machine 4d533d24d6 Test connection-backed tools, MCP refresh and API failure paths
Covers creating and editing tools whose key lives on a connection, the
MCP action refresh, and the auth, ownership and error branches of the
connections API. Editing a tool's key now also recovers a connection
whose stored credentials can no longer be decrypted.
2026-09-28 19:11:54 +01:00
arc53-machine fff3e7c904 Never send an MCP connection's secret to a different server
Editing an MCP server's URL kept the previous connection when no new
one could be stored (default encryption key), so its key would reach the
new server. The save route now keeps a previous connection only for the
same server, and the executor refuses an MCP tool whose connection was
stored for another server.
2026-09-28 19:11:54 +01:00
arc53-machine 6b4220d401 Address review: MCP token scope, API-key identity, reconnect lock, setup retries
- An MCP connection's tokens only go to its own server, and a
  connection_id can no longer come from a client: the MCP test/save
  routes drop it and the tool executor uses only the resolved one.
- API-key connections are reused only when they hold the same
  credentials. Two keys that share a hint or a label get separate
  connections ("…abcd (2)"), at runtime and in the 0038 backfill.
- Reconnecting a connection whose secrets cannot be decrypted no longer
  flags it from a second transaction while holding its row lock.
- Connection setup validates the sync request before claiming its
  Idempotency-Key, so a corrected retry is queued.
- A request that names no connection resolves to none without opening
  a database connection; fixes two tests that reached a real database.
2026-09-28 18:45:27 +01:00
arc53-machine 3e11442a83 Merge remote-tracking branch 'origin/main' into connectors
# Conflicts:
#	frontend/DESIGN.md
#	frontend/src/locale/de.json
#	frontend/src/locale/en.json
#	frontend/src/locale/es.json
#	frontend/src/locale/jp.json
#	frontend/src/locale/ru.json
#	frontend/src/locale/zh-TW.json
#	frontend/src/locale/zh.json
#	frontend/src/upload/Upload.tsx
2026-09-28 18:07:12 +01:00
arc53-machine d9f823d601 Stop syncing sources of a connector an admin turned off
Scheduled syncs checked the connection's status but not the admin
policy, so a disabled connector kept pulling data. Connector sources now
return "disabled" and S3/Reddit syncs get no credentials until the
connector is turned back on.
2026-09-28 17:59:36 +01:00
arc53-machine cefb117277 Keep tool keys when rolling back 0038
Downgrade removes API-key connections and assumed every tool still had
its v1 copy. Tools added through the wizard after the upgrade never had
one, and a reconnect can change the key after the backfill, so write the
connection's current credentials back to each linked tool first.
2026-09-28 17:55:56 +01:00
arc53-machine b692eaf7ad Add MCP presets, admin connector policies and connector attribution
Six vendor-run MCP servers (Notion, Linear, Atlassian, Sentry, Asana,
Stripe) join the catalog as presets from docsgpt/connectors/presets/
mcp.yaml; existing connections to those servers show under them.

connector_policies lets an admin turn a connector off or force whose
account every share uses, and app_metadata's
connectors.allow_custom_mcp turns custom MCP servers off. Both are
enforced on the server: new connections, OAuth sign-ins, MCP test and
save, and the tools of a disabled connector stop resolving.
GET/PUT /api/admin/connectors reads and changes them.

A shared tool's owner picks owner or member credentials with
PUT /api/connections/tools/<id>/credential-mode, and a member running
the owner's account always confirms write actions.

Tool-call events and retrieved chunks name the connector they came
from (key and display name, never the account), and tool calls keep
those fields when the conversation is reloaded.
2026-09-28 17:30:53 +01:00
arc53-machine 6e2ee43944 Connect services from one wizard, with no tokens in the browser
Every entry point (the Connectors page and drawer, Add Source, Add
Tool, the chat) opens one connect wizard: sign in or paste the
credentials, choose what to sync (content services only, skippable),
then a summary with Try it in chat. Tool services create their tools on
sign-in with write actions needing approval, and each action's
permission (Always allow, Needs approval, Off) can be changed in the
wizard or the drawer. The drawer also reconnects, syncs more content,
refreshes an MCP server's tools and removes a connection, keeping or
deleting what it feeds.

The file pickers work from a connection id and fetch a short-lived
picker token when they need one; no session token is kept in
localStorage any more, and any left by an older build is claimed once
and removed. S3 and Reddit reuse the keys stored on a connection.

A tool whose account needs signing in pauses on a Connect card in the
chat; after connecting, Continue resumes the call. A connection whose
sign-in stops working raises a Reconnect toast. The Connectors page is
hidden when the backend does not report connectors_enabled.
2026-09-28 17:21:44 +01:00