diff --git a/docs/content/Guides/Connectors.mdx b/docs/content/Guides/Connectors.mdx new file mode 100644 index 00000000..26e6ffb9 --- /dev/null +++ b/docs/content/Guides/Connectors.mdx @@ -0,0 +1,137 @@ +--- +title: Connectors +description: Connect DocsGPT to the services your team uses. One connection can sync content into Sources and give agents tools, with credentials encrypted on the server. +--- + +import { Callout } from 'nextra/components' +import { Steps } from 'nextra/components' + +# Connectors + +A connector is a service DocsGPT can connect to: Google Drive, SharePoint, Confluence, Amazon S3, Reddit, Brave Search, Telegram, ntfy, PostgreSQL, curated MCP servers (Notion, Linear, Atlassian, Sentry, Asana, Stripe), and any MCP server or OpenAPI spec you add yourself. A **connection** is one signed-in account or one saved API key for a connector. + +Connections are the single place credentials live. A synced source and an agent tool both point at a connection, so you sign in once, reconnect once when a token expires, and disconnect in one place. + +## Using connectors + +Open **Settings > Connectors**. Filters narrow the list by category, and **Connected** shows what you already use. + +1. Pick a connector. Services with OAuth open the provider's sign-in page in a pop-up; API-key services ask for the key. +2. Choose what the connection sets up. File services offer a synced source (pick files or folders, then how often to sync: never, daily, weekly or monthly). Tool services create their tools right away. +3. Adjust what the tools may do. Each action is **Read** or **Write**, and each can be **Always allow**, **Needs approval** or **Off**. Writes default to needing approval. + +The **Add Source** and **Add Tool** dialogs, the composer's Sources and Tools menus, and the agent builder all start the same flow, and list tools grouped by the connection they use. + +Click a connected card to see its account, the sources and tools that use it, and to **Reconnect** or **Disconnect**. Disconnecting keeps a source's indexed content but stops its sync. + +### When a connection needs attention + +If a provider rejects a token that DocsGPT cannot refresh, the connection is marked **Reconnect needed**, its synced sources pause, and you get a notification. Reconnecting the same account resumes them. In a chat, a tool whose connection is missing or expired shows a **Connect** prompt instead of failing; the answer continues once you connect. + +### Sharing a tool with a team + +Sharing a tool that uses a connection asks whose account team members use: + +- **Team members use my account.** Everyone acts as your account on that service. When the tool can write, you confirm this before sharing, and a member's write actions always need approval. +- **Each member connects their own account.** Members see a **Connect** prompt the first time they use the tool. + +Tools from OAuth MCP servers default to each member's own account. An admin can force one mode for a connector. + +### What answers show + +Tool calls name the service that ran them: "Searched Notion", "Read from Google Drive", "Used Linear: create issue". Citations from a synced source read "From Google Drive". Answers never show which account was used. + +## Admin setup + +Admins manage connectors in **Admin > Connectors**. It lists every connector with its status, how many connections use it, and: + +- **Enabled** turns a connector off for everyone. Members see it as turned off and cannot connect it, its tools stop working, and its sources stop syncing until you turn it back on. Existing connections are kept. +- **Shared tools use** is **Let owners choose** (default), **Always the owner's account** or **Always each member's own account**. +- **Allow custom MCP servers** controls whether members can add their own MCP servers. Presets are enabled one by one. +- **Setup guide** on an OAuth connector shows the redirect URI to register and which server settings are still missing. + +A connector that needs server settings stays visible to members as **Needs admin setup** until the settings are present. + +### Encryption key + +Credentials are encrypted with a key derived from `ENCRYPTION_SECRET_KEY` and bound to the connection's owner. **Set your own value before anyone connects a service.** + +```env +ENCRYPTION_SECRET_KEY=a-long-random-value +``` + +When authentication is on (`AUTH_TYPE` set), DocsGPT refuses to store new credentials while the key is the public default, and Admin > Connectors shows a warning. A single-user local install keeps working and logs a warning at startup. + +To rotate the key, move the old value to `ENCRYPTION_SECRET_KEY_PREVIOUS`, set the new one, restart the API and worker, then run: + +```bash +docsgpt connectors reencrypt +``` + +The command prints how many connections it rewrote. Connections it cannot decrypt with either key are marked **Reconnect needed** for their owners. Once it has run, you can remove `ENCRYPTION_SECRET_KEY_PREVIOUS`. + +### Redirect URIs + +OAuth connectors return to one callback, set by `CONNECTOR_REDIRECT_BASE_URI` (default `http://127.0.0.1:7091/api/connectors/callback`). Register it with each provider exactly as set, without query parameters. MCP servers use `MCP_OAUTH_REDIRECT_URI`, which is derived from the same base when unset. Admin > Connectors shows both values with copy buttons. + +If the frontend runs on a different origin from the API, list it in `CONNECTOR_ALLOWED_ORIGINS` so the sign-in pop-up can hand the result back. + +### Google Drive + + + +### Create OAuth credentials + +In the [Google Cloud Console](https://console.cloud.google.com/), enable the **Google Drive API**, then create an **OAuth client ID** of type **Web application**. Add the redirect URI from Admin > Connectors under **Authorized redirect URIs**. + +### Set the server settings + +```env +GOOGLE_CLIENT_ID=your-client-id +GOOGLE_CLIENT_SECRET=your-client-secret +``` + +To offer Google's own file picker, also build the frontend with `VITE_GOOGLE_CLIENT_ID` (and optionally `VITE_GOOGLE_PICKER_API_KEY`). Without them, members browse files in DocsGPT's picker. + +### Publish the app + +Publish the OAuth consent screen, or make it an **Internal** Workspace app. + + +Apps left in **Testing** get refresh tokens that expire after seven days, which stops background sync. + + + + +### SharePoint and OneDrive + +Register an app in [Microsoft Entra ID](https://entra.microsoft.com/) with a **Web** redirect URI from Admin > Connectors, create a client secret, and grant the delegated Microsoft Graph permissions `Files.Read`, `Sites.Read.All` and `User.Read`. + +```env +MICROSOFT_CLIENT_ID=your-application-id +MICROSOFT_CLIENT_SECRET=your-client-secret +MICROSOFT_TENANT_ID=common # or your tenant id for a single-tenant app +``` + +See [SharePoint / OneDrive](/Guides/Integrations/sharepoint-connector) for tenant options. + +### Confluence + +Create an **OAuth 2.0 (3LO)** app in the [Atlassian developer console](https://developer.atlassian.com/console/myapps/), add the redirect URI from Admin > Connectors as its callback URL, and add the scopes `read:page:confluence`, `read:space:confluence`, `read:attachment:confluence` and `read:me`. See [Confluence](/Guides/Integrations/confluence-connector) for details. + +```env +CONFLUENCE_CLIENT_ID=your-client-id +CONFLUENCE_CLIENT_SECRET=your-client-secret +``` + +### Amazon S3 + +S3 needs no server settings. Each member connects with an access key that can list and read the bucket (`s3:ListBucket`, `s3:GetObject`), then picks a bucket and optional path prefix. A custom endpoint URL connects S3-compatible storage such as MinIO or Cloudflare R2. + +### MCP presets + +Notion, Linear, Atlassian, Sentry, Asana and Stripe are remote MCP servers that support OAuth with dynamic client registration, so they need no server settings. Members sign in with their own account. The presets ship in `docsgpt/connectors/presets/mcp.yaml`. + + +Background sync runs on the Celery worker and beat. Keep both running, as in the bundled Compose and Kubernetes files. + diff --git a/docs/content/Guides/Integrations/confluence-connector.mdx b/docs/content/Guides/Integrations/confluence-connector.mdx index d9702fd2..6b81f2a7 100644 --- a/docs/content/Guides/Integrations/confluence-connector.mdx +++ b/docs/content/Guides/Integrations/confluence-connector.mdx @@ -10,6 +10,10 @@ import { Steps } from 'nextra/components' Connect your Confluence Cloud workspace to upload and process pages directly as an external knowledge base. Supports page content and attachments (PDFs, Office files, text files, images, and more). Authentication is handled via Atlassian OAuth 2.0 with automatic token refresh. + +Members connect this service from **Settings > Connectors**, and one sign-in serves every source that uses it. Admins can check what is still missing in **Admin > Connectors**. See [Connectors](/Guides/Connectors). + + ## Setup @@ -18,8 +22,8 @@ Connect your Confluence Cloud workspace to upload and process pages directly as 1. Go to [developer.atlassian.com/console/myapps](https://developer.atlassian.com/console/myapps/) and click **Create** > **OAuth 2.0 integration** 2. Under **Authorization**, add a callback URL: - - Local: `http://localhost:7091/api/connectors/callback?provider=confluence` - - Production: `https://yourdomain.com/api/connectors/callback?provider=confluence` + - Local: `http://localhost:7091/api/connectors/callback` + - Production: `https://yourdomain.com/api/connectors/callback` (the value of `CONNECTOR_REDIRECT_BASE_URI`, registered as-is) ### Step 2: Configure Permissions @@ -56,7 +60,7 @@ VITE_CONFLUENCE_CLIENT_ID=your-atlassian-client-id ### Step 5: Restart and Use -Restart your application, then go to the upload section in DocsGPT and select **Confluence** as the source. You'll be redirected to Atlassian to sign in, then can browse spaces and select pages to process. +Restart your application, then go to **Settings > Connectors** and pick **Confluence**. You'll be redirected to Atlassian to sign in, then can browse spaces and select pages to process. @@ -64,6 +68,6 @@ Restart your application, then go to the upload section in DocsGPT and select ** - **Option not appearing** — Verify `VITE_CONFLUENCE_CLIENT_ID` is set in the frontend `.env`, then restart. - **Sign-in popup closes but the account never connects** — The frontend origin is not allowed to receive the result. Add it to `CONNECTOR_ALLOWED_ORIGINS` in the backend `.env`. -- **Authentication failed** — Check that the callback URL matches exactly, including `?provider=confluence`. +- **Authentication failed** — Check that the callback URL matches exactly and equals `CONNECTOR_REDIRECT_BASE_URI`, with no query parameters. - **No accessible sites** — Ensure the authenticating user has access to at least one Confluence Cloud site. - **Permission denied** — Verify that the Confluence API scopes are enabled in your Atlassian app settings. diff --git a/docs/content/Guides/Integrations/google-drive-connector.mdx b/docs/content/Guides/Integrations/google-drive-connector.mdx index 3014c784..53af5056 100644 --- a/docs/content/Guides/Integrations/google-drive-connector.mdx +++ b/docs/content/Guides/Integrations/google-drive-connector.mdx @@ -10,6 +10,10 @@ import { Steps } from 'nextra/components' Connect your Google Drive account to upload and process files directly as an external knowledge base. Supports Google Workspace files (Docs, Sheets, Slides), Office files, PDFs, text files, CSVs, images, and more. Authentication is handled via Google OAuth 2.0 with automatic token refresh. + +Members connect this service from **Settings > Connectors**, and one sign-in serves every source that uses it. Admins can check what is still missing in **Admin > Connectors**. See [Connectors](/Guides/Connectors). + + ## Setup @@ -26,8 +30,8 @@ Connect your Google Drive account to upload and process files directly as an ext 3. Select **Web application** as the application type 4. Add your DocsGPT URL to **Authorized JavaScript origins** (e.g. `http://localhost:3000`) 5. Add your callback URL to **Authorized redirect URIs**: - - Local: `http://localhost:7091/api/connectors/callback?provider=google_drive` - - Production: `https://yourdomain.com/api/connectors/callback?provider=google_drive` + - Local: `http://localhost:7091/api/connectors/callback` + - Production: `https://yourdomain.com/api/connectors/callback` (the value of `CONNECTOR_REDIRECT_BASE_URI`, registered as-is) 6. Click **Create** and copy the **Client ID** and **Client Secret** ### Step 3: Configure Environment Variables @@ -39,7 +43,7 @@ GOOGLE_CLIENT_ID=your-google-client-id GOOGLE_CLIENT_SECRET=your-google-client-secret ``` -Add to your frontend `.env` file: +Optionally, to use Google's own file picker instead of DocsGPT's, add to your frontend `.env` file: ```env VITE_GOOGLE_CLIENT_ID=your-google-client-id @@ -49,23 +53,23 @@ VITE_GOOGLE_CLIENT_ID=your-google-client-id |----------|-------------|----------| | `GOOGLE_CLIENT_ID` | OAuth Client ID from GCP Credentials | Yes | | `GOOGLE_CLIENT_SECRET` | OAuth Client Secret from GCP Credentials | Yes | -| `VITE_GOOGLE_CLIENT_ID` | Same Client ID, used by the frontend to show the Google Drive option | Yes | +| `VITE_GOOGLE_CLIENT_ID` | Same Client ID, used by the frontend for Google's file picker | No | | `CONNECTOR_ALLOWED_ORIGINS` | Comma-separated frontend origins allowed to receive the sign-in result, e.g. `https://docsgpt.example.com`. Not needed when the frontend shares the API origin, or in local dev when the callback is on `localhost`/`127.0.0.1` and the frontend runs on port 5173 | When the frontend is on its own origin | -Make sure to use the same Google Client ID in both backend and frontend configurations. +If you set `VITE_GOOGLE_CLIENT_ID`, use the same Client ID as the backend. Publish the OAuth consent screen (or use an internal Workspace app): apps left in Testing get refresh tokens that expire after seven days, which stops background sync. ### Step 4: Restart and Use -Restart your application, then go to the upload section in DocsGPT and select **Google Drive** as the source. You'll be redirected to Google to sign in, then can browse and select files to process. +Restart your application, then go to **Settings > Connectors** and pick **Google Drive**. You'll be redirected to Google to sign in, then can browse and select files to process. ## Troubleshooting -- **Option not appearing** — Verify `VITE_GOOGLE_CLIENT_ID` is set in the frontend `.env`, then restart. -- **Authentication failed** — Check that the redirect URI matches exactly, including `?provider=google_drive`. Ensure the Google Drive API is enabled. +- **Shows Needs admin setup** — `GOOGLE_CLIENT_ID` or `GOOGLE_CLIENT_SECRET` is missing from the backend `.env`. Admin > Connectors lists which one. +- **Authentication failed** — Check that the redirect URI matches exactly and equals `CONNECTOR_REDIRECT_BASE_URI`, with no query parameters. Ensure the Google Drive API is enabled. - **Sign-in popup closes but the account never connects** — The frontend origin is not allowed to receive the result. Add it to `CONNECTOR_ALLOWED_ORIGINS` in the backend `.env`. - **Permission denied** — Verify the OAuth consent screen is configured and the user has access to the target files. - **Files not processing** — Check backend logs and verify that backend environment variables are correctly set. diff --git a/docs/content/Guides/Integrations/mcp-tool-integration.mdx b/docs/content/Guides/Integrations/mcp-tool-integration.mdx index d33e1b94..4d89fafe 100644 --- a/docs/content/Guides/Integrations/mcp-tool-integration.mdx +++ b/docs/content/Guides/Integrations/mcp-tool-integration.mdx @@ -22,11 +22,11 @@ Only needed if your MCP servers use OAuth authentication: MCP_OAUTH_REDIRECT_URI=https://yourdomain.com/api/mcp_server/callback ``` -If not set, falls back to `API_URL/api/mcp_server/callback`. +If not set, it is derived from the host of `CONNECTOR_REDIRECT_BASE_URI`, then from `API_URL`. ### Step 2: Add an MCP Server -Go to **Settings** > **Tools** > **Add Tool** > **MCP Server**. Enter the server URL, select an auth type, and click **Test Connection** to verify, then **Save**. +Go to **Settings** > **Connectors** and pick a preset (Notion, Linear, Atlassian, Sentry, Asana, Stripe), or choose **Add custom connector** > **MCP server**. The same form opens from **Settings** > **Tools** > **Add Tool**. Enter the server URL, select an auth type, and click **Test Connection** to verify, then **Save**. ### Step 3: Enable for Your Agent @@ -34,6 +34,8 @@ In your agent configuration, enable the MCP tools you want the agent to use. +Presets fill in the server URL and sign in with OAuth. Admins can turn presets off one by one, and turn off custom MCP servers, in **Admin > Connectors**. See [Connectors](/Guides/Connectors). + ## Authentication Types | Auth Type | Config Fields | diff --git a/docs/content/Guides/Integrations/sharepoint-connector.mdx b/docs/content/Guides/Integrations/sharepoint-connector.mdx index 58829f38..626b9788 100644 --- a/docs/content/Guides/Integrations/sharepoint-connector.mdx +++ b/docs/content/Guides/Integrations/sharepoint-connector.mdx @@ -10,6 +10,10 @@ import { Steps } from 'nextra/components' Connect your SharePoint or OneDrive account to upload and process files directly as an external knowledge base. Supports Office files, PDFs, text files, CSVs, images, and more. Authentication is handled via Microsoft Entra ID (Azure AD) with automatic token refresh. + +Members connect this service from **Settings > Connectors**, and one sign-in serves every source that uses it. Admins can check what is still missing in **Admin > Connectors**. See [Connectors](/Guides/Connectors). + + ## Setup @@ -18,8 +22,8 @@ Connect your SharePoint or OneDrive account to upload and process files directly 1. Go to the [Azure Portal](https://portal.azure.com/) > **Microsoft Entra ID** > **App registrations** > **New registration** 2. Set **Redirect URI** (Web) to: - - Local: `http://localhost:7091/api/connectors/callback?provider=share_point` - - Production: `https://yourdomain.com/api/connectors/callback?provider=share_point` + - Local: `http://localhost:7091/api/connectors/callback` + - Production: `https://yourdomain.com/api/connectors/callback` (the value of `CONNECTOR_REDIRECT_BASE_URI`, registered as-is) ### Step 2: Configure API Permissions @@ -53,13 +57,13 @@ MICROSOFT_TENANT_ID=your-azure-ad-tenant-id ### Step 5: Restart and Use -Restart your application, then go to the upload section in DocsGPT and select **SharePoint / OneDrive** as the source. You'll be redirected to Microsoft to sign in, then can browse and select files to process. +Restart your application, then go to **Settings > Connectors** and pick **SharePoint**. You'll be redirected to Microsoft to sign in, then can browse and select files to process. ## Troubleshooting - **Option not appearing** — Verify `MICROSOFT_CLIENT_ID` and `MICROSOFT_CLIENT_SECRET` are set, then restart. -- **Authentication failed** — Check that the redirect URI matches exactly, including `?provider=share_point`. +- **Authentication failed** — Check that the redirect URI matches exactly and equals `CONNECTOR_REDIRECT_BASE_URI`, with no query parameters. - **Sign-in popup closes but the account never connects** — The frontend origin is not allowed to receive the result. Add it to `CONNECTOR_ALLOWED_ORIGINS` in the backend `.env`. - **Permission denied** — Ensure admin consent is granted and the user has access to the target files. diff --git a/docs/content/Guides/_meta.js b/docs/content/Guides/_meta.js index 33cca51f..a9122c32 100644 --- a/docs/content/Guides/_meta.js +++ b/docs/content/Guides/_meta.js @@ -1,4 +1,8 @@ export default { + "Connectors": { + "title": "🔌 Connectors", + "href": "/Guides/Connectors" + }, "Customising-prompts": { "title": "️💻 Customising Prompts", "href": "/Guides/Customising-prompts" diff --git a/docs/content/Sources/_meta.js b/docs/content/Sources/_meta.js index 06a1d99a..cc800b43 100644 --- a/docs/content/Sources/_meta.js +++ b/docs/content/Sources/_meta.js @@ -1,4 +1,8 @@ export default { + "Connectors": { + "title": "🔌 Synced Sources (Connectors)", + "href": "/Guides/Connectors" + }, "Per-source-configuration": { "title": "🎛️ Per-Source Configuration", "href": "/Sources/Per-source-configuration" diff --git a/docs/content/Tools/basics.mdx b/docs/content/Tools/basics.mdx index b578eb8c..03581680 100644 --- a/docs/content/Tools/basics.mdx +++ b/docs/content/Tools/basics.mdx @@ -121,7 +121,8 @@ Interacting with tools in DocsGPT is designed to be intuitive: 2. **Configuration in UI:** * Tools are generally managed and configured within the DocsGPT application's settings, found under a "Tools" section in the GUI. - * For tools that interact with external services (like Brave Search, Telegram, or any service via the API Tool), you might need to provide authentication credentials (e.g., API keys, tokens) or specific endpoint information during the tool's setup in the UI. + * Tools for external services (Brave Search, Telegram, ntfy, PostgreSQL, MCP servers) are set up from **Settings > Connectors**. You connect the service once; its credentials stay encrypted on the server and never reach the browser, and the tools it provides are grouped under that connection. Each action is marked **Read** or **Write** and can be set to **Always allow**, **Needs approval** or **Off**. See [Connectors](/Guides/Connectors). + * When a tool's connection is missing or expired, the chat shows a **Connect** prompt and continues once you connect. 3. **Prompt Engineering for Tools:** While the LLM aims to intelligently use tools, for more complex or reliable agent-like behaviors, you might need to customize the system prompts. Modifying the prompt can guide the LLM on when and how to prioritize or chain tools to achieve specific outcomes, especially if you're building an agent designed to perform a certain sequence of actions every time. For more on this, see [Customising Prompts](/Guides/Customising-prompts). diff --git a/docs/content/upgrading.mdx b/docs/content/upgrading.mdx index c5397390..2d7d1589 100644 --- a/docs/content/upgrading.mdx +++ b/docs/content/upgrading.mdx @@ -11,6 +11,23 @@ import { Callout } from 'nextra/components' **Upgrading from 0.16.x?** User data moved from MongoDB to Postgres in 0.17.0. Follow the [Postgres Migration guide](/Deploying/Postgres-Migration) before running `docker compose pull` or `git pull` — existing deployments will not start cleanly without it. +## Connectors: set ENCRYPTION_SECRET_KEY first + +Service credentials (OAuth tokens for Google Drive, SharePoint, Confluence and MCP servers, and API keys for tools) now live on **connections** and are encrypted with a key derived from `ENCRYPTION_SECRET_KEY`. Migration `0038_connections` runs on startup, encrypts the stored tokens and removes their plaintext copies. + + + **Multi-user installs** (any `AUTH_TYPE`): set `ENCRYPTION_SECRET_KEY` to your own value **before** you upgrade, in the environment of the API, the worker and anything that runs migrations. The migration encrypts with the key it sees; changing it afterwards makes every connection ask its owner to reconnect. While the key is still the public default, DocsGPT refuses to store new credentials and Admin > Connectors shows a warning. + + +If you already used a key and want to change it, see [rotating the key](/Guides/Connectors#encryption-key). Single-user local installs keep working with the default and log a warning at startup. + +Other changes: + +- Register `CONNECTOR_REDIRECT_BASE_URI` exactly as set (no `?provider=` query) as the redirect URI of each OAuth app. Admin > Connectors shows it. +- `VITE_GOOGLE_CLIENT_ID` is optional now; without it, members pick Drive files in DocsGPT's own picker. +- Synced sources run as their connection, without a browser session. Keep Celery beat running for scheduled syncs. +- Downgrading to `0037_request_traces` decrypts the tokens back and gives each tool its key again. + ## pip installs: data home moved An installed package (`pip install docsgpt`, pipx, `uv tool`) used to keep its data home, meaning `.env`, `inputs/`, `indexes/` and `models/`, in the directory you ran `docsgpt api` and `docsgpt worker` from. It is now `~/.docsgpt/server` (`/opt/docsgpt` for root on Linux). Either move those files there, or set `DOCSGPT_HOME` to the old directory in the environment of both commands. Both commands print the data home they use, and point out a `.env` in the working directory that they no longer read. Source checkouts and the Docker images are not affected.