Files
goclaw/internal/store/pg/mcp_export_queries.go
SYNITYandDangTinh311 3b8bc1bc4d feat(bitrix24): replace two MCP text inputs with a filtered dropdown (#1392)
* feat(bitrix24): replace two MCP text inputs with a filtered dropdown [B24:2794]

Bitrix24 channel creation used to demand two hand-typed strings —
mcp_server_name and mcp_base_url — plus zero indication of which MCP
servers can actually auto-onboard. Typos silently disabled provisioning
and admin had to know which servers implement /api/auto-onboard.

Ship a single dropdown backed by mcp_servers.require_user_credentials,
plus the machinery to make it work end-to-end.

Phase 1 — DB & store
* Promote require_user_credentials from settings JSONB to a top-level
  column on mcp_servers (PG migration 000089, SQLite migration v54).
* Backfill from existing settings blobs so no admin needs to re-tick.
* Add MCPServerData.RequireUserCredentials to the Go store layer, plumb
  through Create / Get / GetByName / List / Update on both stores,
  extend the export DTO, and add require_user_credentials to the HTTP
  allowlist.
* Bump RequiredSchemaVersion 87 -> 89 (jumping 88, which was on disk
  but not wired) and SchemaVersion 53 -> 54 with an
  idempotentColumnMigration guard.

Phase 2 — Bitrix24 channel factory
* Add MCPServerID (UUID string) to bitrix24 InstanceConfig, keep
  MCPServerName + MCPBaseURL as legacy fallback with a "deprecated"
  doc comment.
* Factory validation accepts either mcp_server_id alone or the legacy
  pair; half-config still fails fast.
* initMCPProvisioner prefers GetServer(id) and sources the base URL
  from mcp_servers.url when the id path is used. Legacy name path
  unchanged so pre-migration configs keep working.
* Log line now carries mcp_server_id + require_user_credentials so
  operators can eyeball the wiring.

Phase 3 — Frontend types & MCP form
* Add optional top-level require_user_credentials to MCPServerData /
  MCPServerInput in both ui/web and ui/desktop/frontend types.
* mcp-form-dialog reads the top-level flag first and falls back to
  settings.require_user_credentials so cached responses from
  pre-upgrade backends still render correctly.
* On submit send both the top-level flag AND the legacy settings
  entry so mid-rollout backends stay consistent.

Phase 4 — Bitrix24 channel form dropdown
* New mcp-select field type + MCPServerSelect component. Uses the
  shared useMCP() react-query cache and filters client-side to
  servers whose require_user_credentials is true (OR settings
  JSONB during the migration window).
* Explicit "None (disable MCP provisioning)" option so admins can
  clear the binding without editing config JSON.
* Legacy mcp_server_name / mcp_base_url text inputs kept in the
  Advanced panel, relabelled "(legacy)" with pointer help text.

Phase 5 — channel_instances.config backfill
* PG migration 000090 and SQLite migration v55 rewrite existing
  bitrix24 channel_instances.config to add mcp_server_id by
  resolving mcp_server_name against mcp_servers, tenant-scoped
  via agents.tenant_id (channel_instances doesn't carry tenant_id
  directly).
* Idempotent — only touches rows already carrying
  mcp_server_name that lack mcp_server_id. Legacy keys are left
  in place so provisioner.go can still fall back for unmigrated
  or future-created legacy configs.
* down.sql drops the mcp_server_id key. Provisioner immediately
  reverts to the legacy fallback path.

Tests
* provisioner_test.go: three new cases exercise the mcp_server_id
  path (invalid UUID string, valid UUID with missing row, valid
  UUID with a per-user row). fakeMCPStore gains a serversByID
  map and a real GetServer implementation.
* Existing legacy-config tests unchanged and still green.

Verification
* go build ./... && go build -tags sqliteonly ./...
* go vet ./internal/mcp/... ./internal/channels/bitrix24/...
    ./internal/store/... ./internal/http/...
* go test ./internal/mcp/... ./internal/channels/bitrix24/...  -> ok
* Live-tested against a local docker image on the goclaw-deploy
  postgres. Migrations 89 + 90 applied cleanly. Three existing
  bitrix24 channels (bitrix-sales / nguyen-dao-openline / tieu-vi)
  had their configs backfilled with the b24-syn-mcp UUID and the
  provisioner boots with require_user_credentials=true. UI dropdown
  correctly shows only b24-syn-mcp (the only server with the flag
  ticked) alongside a "None" clearer option.

Surface parity
* Gateway server: store + factory + provisioner + HTTP allowlist.
* API contract: adds require_user_credentials + mcp_server_id
  as optional fields on existing routes. No new endpoints.
* Web UI: MCP form + Bitrix24 channel form + shared types.
* CLI/runtime package: N/A because no CLI subcommand reads the
  mcp_server_id field.

* fix(bitrix24): derive auto-onboard base URL from mcp_servers.url origin [B24:2794]

The Phase 2 refactor swapped provisioner base-URL sourcing from the
legacy per-channel MCPBaseURL config field (which historically stored the
MCP server's ORIGIN, e.g. https://mcp.example.com) to mcp_servers.url,
which stores the JSON-RPC ENDPOINT the agent loop dials (e.g.
https://mcp.example.com/mcp). The two are semantically different but
share a single column.

mcp_client.newMCPClient then appends "/api/auto-onboard" to whatever
baseURL it receives, so the id-path started POSTing to
".../mcp/api/auto-onboard" — 404 for every per-user credential mint and
refresh. Existing users kept working only until their cached access
tokens expired.

Fix: derive the origin (scheme://host[:port]) from server.URL before
handing it to the auto-onboard client. The legacy path is untouched
because MCPBaseURL from channel config is already the origin.

  https://b24-mcp-dev.synity.so/mcp   ->  https://b24-mcp-dev.synity.so
  https://mcp.example.com/mcp/        ->  https://mcp.example.com
  https://mcp.example.com             ->  https://mcp.example.com

Table-driven test covers six shapes plus four error cases (empty,
whitespace-only, no scheme, no host). Updated the existing
TestInitMCPProvisioner_MCPServerID fixture to seed a URL with the /mcp
subpath so it regression-guards the same code path.

Verified live: user 614 sent a message that triggered the expired-cred
refresh branch; goclaw logged "self-refreshed user credentials
created=false" and the agent immediately reported
"mcp.user_tools_loaded user=614 tools=2". Before this fix the same event
logged 'auto-onboard failed: mcp auto-onboard: 404 Not Found'.

Surface parity:
  - Gateway server: provisioner + one new helper (deriveAutoOnboardBaseURL).
  - API contract: N/A because the wire shape hasn't changed.
  - Web UI: N/A because the UI still writes mcp_server_id verbatim.
  - CLI/runtime: N/A because no CLI reads the derived base URL.

---------

Co-authored-by: DangTinh311 <dangtinh31193@gmail.com>
2026-07-08 19:23:21 +07:00

221 lines
6.9 KiB
Go

package pg
import (
"context"
"database/sql"
"encoding/json"
"errors"
"log/slog"
"github.com/google/uuid"
)
// MCPServerExport holds portable MCP server metadata.
// Sensitive fields (api_key, env, headers) are intentionally excluded.
type MCPServerExport struct {
Name string `json:"name"`
DisplayName string `json:"display_name,omitempty"`
Transport string `json:"transport"`
Command string `json:"command,omitempty"`
Args json.RawMessage `json:"args,omitempty"`
URL string `json:"url,omitempty"`
ToolPrefix string `json:"tool_prefix,omitempty"`
TimeoutSec int `json:"timeout_sec"`
Settings json.RawMessage `json:"settings,omitempty"`
Enabled bool `json:"enabled"`
RequireUserCredentials bool `json:"require_user_credentials,omitempty"`
}
// MCPGrantWithKey references an MCP agent grant by server_name and agent_key (portable).
type MCPGrantWithKey struct {
ServerName string `json:"server_name"`
AgentKey string `json:"agent_key"`
Enabled bool `json:"enabled"`
ToolAllow json.RawMessage `json:"tool_allow,omitempty"`
ToolDeny json.RawMessage `json:"tool_deny,omitempty"`
ConfigOverrides json.RawMessage `json:"config_overrides,omitempty"`
}
// MCPExportPreview holds lightweight counts for the export preview endpoint.
type MCPExportPreview struct {
Servers int `json:"servers"`
AgentGrants int `json:"agent_grants"`
}
// ExportMCPServers returns all MCP servers scoped to the current tenant.
// Sensitive fields (api_key, headers, env) are excluded from the export.
func ExportMCPServers(ctx context.Context, db *sql.DB) ([]MCPServerExport, error) {
tc, tcArgs, _, err := scopeClause(ctx, 1)
if err != nil {
return nil, err
}
rows, err := db.QueryContext(ctx,
"SELECT name, COALESCE(display_name,''), transport,"+
" COALESCE(command,''), args, COALESCE(url,''),"+
" COALESCE(tool_prefix,''), timeout_sec, settings, enabled, require_user_credentials"+
" FROM mcp_servers WHERE 1=1"+tc+
" ORDER BY name",
tcArgs...,
)
if err != nil {
return nil, err
}
defer rows.Close()
var result []MCPServerExport
for rows.Next() {
var (
srv MCPServerExport
argsRaw []byte
settings []byte
)
if err := rows.Scan(
&srv.Name, &srv.DisplayName, &srv.Transport,
&srv.Command, &argsRaw, &srv.URL,
&srv.ToolPrefix, &srv.TimeoutSec, &settings, &srv.Enabled, &srv.RequireUserCredentials,
); err != nil {
slog.Warn("mcp_export.servers.scan", "error", err)
continue
}
if len(argsRaw) > 0 {
srv.Args = json.RawMessage(argsRaw)
}
if len(settings) > 0 {
srv.Settings = json.RawMessage(settings)
}
result = append(result, srv)
}
return result, rows.Err()
}
// ExportMCPGrantsWithKeys returns all MCP agent grants with server_name and agent_key resolved.
func ExportMCPGrantsWithKeys(ctx context.Context, db *sql.DB) ([]MCPGrantWithKey, error) {
tc, tcArgs, _, err := scopeClauseAlias(ctx, 1, "g")
if err != nil {
return nil, err
}
rows, err := db.QueryContext(ctx,
"SELECT s.name, a.agent_key, g.enabled, g.tool_allow, g.tool_deny, g.config_overrides"+
" FROM mcp_agent_grants g"+
" JOIN mcp_servers s ON s.id = g.server_id"+
" JOIN agents a ON a.id = g.agent_id"+
" WHERE 1=1"+tc+
" ORDER BY s.name, a.agent_key",
tcArgs...,
)
if err != nil {
return nil, err
}
defer rows.Close()
var result []MCPGrantWithKey
for rows.Next() {
var (
g MCPGrantWithKey
toolAllow []byte
toolDeny []byte
configOverrides []byte
)
if err := rows.Scan(
&g.ServerName, &g.AgentKey, &g.Enabled,
&toolAllow, &toolDeny, &configOverrides,
); err != nil {
slog.Warn("mcp_export.grants.scan", "error", err)
continue
}
if len(toolAllow) > 0 {
g.ToolAllow = json.RawMessage(toolAllow)
}
if len(toolDeny) > 0 {
g.ToolDeny = json.RawMessage(toolDeny)
}
if len(configOverrides) > 0 {
g.ConfigOverrides = json.RawMessage(configOverrides)
}
result = append(result, g)
}
return result, rows.Err()
}
// ExportMCPPreview returns aggregate counts for MCP export preview.
func ExportMCPPreview(ctx context.Context, db *sql.DB) (*MCPExportPreview, error) {
tc, tcArgs, _, err := scopeClause(ctx, 1)
if err != nil {
return nil, err
}
var p MCPExportPreview
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM mcp_servers WHERE 1=1"+tc,
tcArgs...,
).Scan(&p.Servers); err != nil {
return nil, err
}
tc2, tcArgs2, _, err := scopeClause(ctx, 1)
if err != nil {
return nil, err
}
if err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM mcp_agent_grants WHERE 1=1"+tc2,
tcArgs2...,
).Scan(&p.AgentGrants); err != nil {
return nil, err
}
return &p, nil
}
// ImportMCPServer inserts an MCP server from export data if no server with that name exists.
// Returns the server UUID and whether it was newly created (false = already existed).
func ImportMCPServer(ctx context.Context, db *sql.DB, srv MCPServerExport, createdBy string) (uuid.UUID, bool, error) {
tid := tenantIDForInsert(ctx)
// Check if server with same name already exists
var existing uuid.UUID
err := db.QueryRowContext(ctx,
"SELECT id FROM mcp_servers WHERE name = $1 AND tenant_id = $2",
srv.Name, tid,
).Scan(&existing)
if err == nil {
return existing, false, nil // already exists — return existing ID
}
if !errors.Is(err, sql.ErrNoRows) {
return uuid.Nil, false, err
}
id := uuid.Must(uuid.NewV7())
_, err = db.ExecContext(ctx,
`INSERT INTO mcp_servers
(id, name, display_name, transport, command, args, url,
tool_prefix, timeout_sec, settings, enabled, require_user_credentials,
created_by, created_at, updated_at, tenant_id)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,NOW(),NOW(),$14)`,
id, srv.Name, srv.DisplayName, srv.Transport,
srv.Command, jsonOrNull(srv.Args), srv.URL,
srv.ToolPrefix, srv.TimeoutSec, jsonOrNull(srv.Settings), srv.Enabled, srv.RequireUserCredentials,
createdBy, tid,
)
if err != nil {
return uuid.Nil, false, err
}
return id, true, nil
}
// ImportMCPGrant upserts an MCP agent grant for an imported server.
func ImportMCPGrant(ctx context.Context, db *sql.DB, serverID, agentID uuid.UUID, g MCPGrantWithKey, grantedBy string) error {
tid := tenantIDForInsert(ctx)
_, err := db.ExecContext(ctx,
`INSERT INTO mcp_agent_grants
(id, server_id, agent_id, enabled, tool_allow, tool_deny, config_overrides, granted_by, created_at, tenant_id)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,NOW(),$9)
ON CONFLICT (server_id, agent_id) DO UPDATE
SET enabled = EXCLUDED.enabled,
tool_allow = EXCLUDED.tool_allow,
tool_deny = EXCLUDED.tool_deny,
config_overrides = EXCLUDED.config_overrides`,
uuid.Must(uuid.NewV7()), serverID, agentID, g.Enabled,
jsonOrNull(g.ToolAllow), jsonOrNull(g.ToolDeny), jsonOrNull(g.ConfigOverrides),
grantedBy, tid,
)
return err
}