feat(bitrix24): replace two MCP text inputs with a filtered dropdown (#1392)

* feat(bitrix24): replace two MCP text inputs with a filtered dropdown [B24:2794]

Bitrix24 channel creation used to demand two hand-typed strings —
mcp_server_name and mcp_base_url — plus zero indication of which MCP
servers can actually auto-onboard. Typos silently disabled provisioning
and admin had to know which servers implement /api/auto-onboard.

Ship a single dropdown backed by mcp_servers.require_user_credentials,
plus the machinery to make it work end-to-end.

Phase 1 — DB & store
* Promote require_user_credentials from settings JSONB to a top-level
  column on mcp_servers (PG migration 000089, SQLite migration v54).
* Backfill from existing settings blobs so no admin needs to re-tick.
* Add MCPServerData.RequireUserCredentials to the Go store layer, plumb
  through Create / Get / GetByName / List / Update on both stores,
  extend the export DTO, and add require_user_credentials to the HTTP
  allowlist.
* Bump RequiredSchemaVersion 87 -> 89 (jumping 88, which was on disk
  but not wired) and SchemaVersion 53 -> 54 with an
  idempotentColumnMigration guard.

Phase 2 — Bitrix24 channel factory
* Add MCPServerID (UUID string) to bitrix24 InstanceConfig, keep
  MCPServerName + MCPBaseURL as legacy fallback with a "deprecated"
  doc comment.
* Factory validation accepts either mcp_server_id alone or the legacy
  pair; half-config still fails fast.
* initMCPProvisioner prefers GetServer(id) and sources the base URL
  from mcp_servers.url when the id path is used. Legacy name path
  unchanged so pre-migration configs keep working.
* Log line now carries mcp_server_id + require_user_credentials so
  operators can eyeball the wiring.

Phase 3 — Frontend types & MCP form
* Add optional top-level require_user_credentials to MCPServerData /
  MCPServerInput in both ui/web and ui/desktop/frontend types.
* mcp-form-dialog reads the top-level flag first and falls back to
  settings.require_user_credentials so cached responses from
  pre-upgrade backends still render correctly.
* On submit send both the top-level flag AND the legacy settings
  entry so mid-rollout backends stay consistent.

Phase 4 — Bitrix24 channel form dropdown
* New mcp-select field type + MCPServerSelect component. Uses the
  shared useMCP() react-query cache and filters client-side to
  servers whose require_user_credentials is true (OR settings
  JSONB during the migration window).
* Explicit "None (disable MCP provisioning)" option so admins can
  clear the binding without editing config JSON.
* Legacy mcp_server_name / mcp_base_url text inputs kept in the
  Advanced panel, relabelled "(legacy)" with pointer help text.

Phase 5 — channel_instances.config backfill
* PG migration 000090 and SQLite migration v55 rewrite existing
  bitrix24 channel_instances.config to add mcp_server_id by
  resolving mcp_server_name against mcp_servers, tenant-scoped
  via agents.tenant_id (channel_instances doesn't carry tenant_id
  directly).
* Idempotent — only touches rows already carrying
  mcp_server_name that lack mcp_server_id. Legacy keys are left
  in place so provisioner.go can still fall back for unmigrated
  or future-created legacy configs.
* down.sql drops the mcp_server_id key. Provisioner immediately
  reverts to the legacy fallback path.

Tests
* provisioner_test.go: three new cases exercise the mcp_server_id
  path (invalid UUID string, valid UUID with missing row, valid
  UUID with a per-user row). fakeMCPStore gains a serversByID
  map and a real GetServer implementation.
* Existing legacy-config tests unchanged and still green.

Verification
* go build ./... && go build -tags sqliteonly ./...
* go vet ./internal/mcp/... ./internal/channels/bitrix24/...
    ./internal/store/... ./internal/http/...
* go test ./internal/mcp/... ./internal/channels/bitrix24/...  -> ok
* Live-tested against a local docker image on the goclaw-deploy
  postgres. Migrations 89 + 90 applied cleanly. Three existing
  bitrix24 channels (bitrix-sales / nguyen-dao-openline / tieu-vi)
  had their configs backfilled with the b24-syn-mcp UUID and the
  provisioner boots with require_user_credentials=true. UI dropdown
  correctly shows only b24-syn-mcp (the only server with the flag
  ticked) alongside a "None" clearer option.

Surface parity
* Gateway server: store + factory + provisioner + HTTP allowlist.
* API contract: adds require_user_credentials + mcp_server_id
  as optional fields on existing routes. No new endpoints.
* Web UI: MCP form + Bitrix24 channel form + shared types.
* CLI/runtime package: N/A because no CLI subcommand reads the
  mcp_server_id field.

* fix(bitrix24): derive auto-onboard base URL from mcp_servers.url origin [B24:2794]

The Phase 2 refactor swapped provisioner base-URL sourcing from the
legacy per-channel MCPBaseURL config field (which historically stored the
MCP server's ORIGIN, e.g. https://mcp.example.com) to mcp_servers.url,
which stores the JSON-RPC ENDPOINT the agent loop dials (e.g.
https://mcp.example.com/mcp). The two are semantically different but
share a single column.

mcp_client.newMCPClient then appends "/api/auto-onboard" to whatever
baseURL it receives, so the id-path started POSTing to
".../mcp/api/auto-onboard" — 404 for every per-user credential mint and
refresh. Existing users kept working only until their cached access
tokens expired.

Fix: derive the origin (scheme://host[:port]) from server.URL before
handing it to the auto-onboard client. The legacy path is untouched
because MCPBaseURL from channel config is already the origin.

  https://b24-mcp-dev.synity.so/mcp   ->  https://b24-mcp-dev.synity.so
  https://mcp.example.com/mcp/        ->  https://mcp.example.com
  https://mcp.example.com             ->  https://mcp.example.com

Table-driven test covers six shapes plus four error cases (empty,
whitespace-only, no scheme, no host). Updated the existing
TestInitMCPProvisioner_MCPServerID fixture to seed a URL with the /mcp
subpath so it regression-guards the same code path.

Verified live: user 614 sent a message that triggered the expired-cred
refresh branch; goclaw logged "self-refreshed user credentials
created=false" and the agent immediately reported
"mcp.user_tools_loaded user=614 tools=2". Before this fix the same event
logged 'auto-onboard failed: mcp auto-onboard: 404 Not Found'.

Surface parity:
  - Gateway server: provisioner + one new helper (deriveAutoOnboardBaseURL).
  - API contract: N/A because the wire shape hasn't changed.
  - Web UI: N/A because the UI still writes mcp_server_id verbatim.
  - CLI/runtime: N/A because no CLI reads the derived base URL.

---------

Co-authored-by: DangTinh311 <dangtinh31193@gmail.com>

internal/store/sqlitestore/mcp_servers.go

@@ -17,7 +17,7 @@ import (
 )
 
 const mcpServerSelectCols = `id, name, display_name, transport, command, args, url, headers, env,
-		 api_key, tool_prefix, timeout_sec, settings, enabled, created_by, created_at, updated_at`
+		 api_key, tool_prefix, timeout_sec, settings, enabled, require_user_credentials, created_by, created_at, updated_at`
 
 // SQLiteMCPServerStore implements store.MCPServerStore backed by SQLite.
 type SQLiteMCPServerStore struct {
@@ -59,12 +59,12 @@ func (s *SQLiteMCPServerStore) CreateServer(ctx context.Context, srv *store.MCPS
 
 	_, err := s.db.ExecContext(ctx,
 		`INSERT INTO mcp_servers (id, name, display_name, transport, command, args, url, headers, env,
-		 api_key, tool_prefix, timeout_sec, settings, enabled, created_by, created_at, updated_at, tenant_id)
-		 VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
+		 api_key, tool_prefix, timeout_sec, settings, enabled, require_user_credentials, created_by, created_at, updated_at, tenant_id)
+		 VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
 		srv.ID, srv.Name, nilStr(srv.DisplayName), srv.Transport, nilStr(srv.Command),
 		jsonOrEmpty(srv.Args), nilStr(srv.URL), encHeaders, encEnv,
 		nilStr(apiKey), nilStr(srv.ToolPrefix), srv.TimeoutSec,
-		jsonOrEmpty(srv.Settings), srv.Enabled, srv.CreatedBy, now, now, tenantID,
+		jsonOrEmpty(srv.Settings), srv.Enabled, srv.RequireUserCredentials, srv.CreatedBy, now, now, tenantID,
 	)
 	return err
 }

internal/store/sqlitestore/schema.go

@@ -16,7 +16,7 @@ var schemaSQL string
 
 // SchemaVersion is the current SQLite schema version.
 // Bump this when adding new migration steps below.
-const SchemaVersion = 55
+const SchemaVersion = 57
 
 // migrations maps version → SQL to apply when upgrading FROM that version.
 // schema.sql always represents the LATEST full schema (for fresh DBs).
@@ -898,6 +898,45 @@ ALTER TABLE usage_event_rollups ADD COLUMN thinking_tokens BIGINT NOT NULL DEFAU
 CREATE INDEX IF NOT EXISTS idx_channel_pending_messages_parent
   ON channel_pending_messages(channel_name, parent_history_key)
   WHERE parent_history_key <> '';`,
+	// Version 55 → 56: promote require_user_credentials from settings JSONB
+	// to a top-level column so channel factories can filter directly.
+	// Backfill reads the legacy JSONB via json_extract so no admin needs to
+	// re-tick after upgrading. Mirrors PG migration 000092. Idempotent-guarded
+	// via idempotentColumnMigration(55).
+	55: `ALTER TABLE mcp_servers ADD COLUMN require_user_credentials BOOLEAN NOT NULL DEFAULT 0;
+UPDATE mcp_servers
+   SET require_user_credentials = COALESCE(CAST(json_extract(settings, '$.require_user_credentials') AS INTEGER), 0)
+ WHERE settings IS NOT NULL
+   AND json_extract(settings, '$.require_user_credentials') IS NOT NULL;`,
+	// Version 56 → 57: backfill Bitrix24 channel_instances.config with
+	// mcp_server_id by resolving the legacy mcp_server_name against
+	// mcp_servers (matched on the channel's agent tenant_id since
+	// channel_instances doesn't carry tenant_id directly). Mirrors PG
+	// migration 000093. Idempotent — only touches rows without an
+	// existing mcp_server_id key.
+	56: `UPDATE channel_instances
+        SET config = json_set(
+                COALESCE(config, '{}'),
+                '$.mcp_server_id',
+                (SELECT srv.id
+                   FROM mcp_servers srv
+                  WHERE srv.name = json_extract(channel_instances.config, '$.mcp_server_name')
+                    AND srv.tenant_id = (
+                        SELECT a.tenant_id FROM agents a WHERE a.id = channel_instances.agent_id
+                    )
+                  LIMIT 1)
+        ),
+            updated_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now')
+      WHERE channel_type = 'bitrix24'
+        AND json_extract(config, '$.mcp_server_name') IS NOT NULL
+        AND json_extract(config, '$.mcp_server_id') IS NULL
+        AND EXISTS (
+            SELECT 1 FROM mcp_servers srv
+             WHERE srv.name = json_extract(channel_instances.config, '$.mcp_server_name')
+               AND srv.tenant_id = (
+                   SELECT a.tenant_id FROM agents a WHERE a.id = channel_instances.agent_id
+               )
+        );`,
 }
 
 const addUsageEventAnalyticsTables = `
@@ -1549,6 +1588,8 @@ func idempotentColumnMigration(version int) (string, string, bool) {
 		return "secure_cli_binaries", "adapter_name", true
 	case 51:
 		return "webhook_calls", "last_heartbeat_at", true
+	case 55:
+		return "mcp_servers", "require_user_credentials", true
 	default:
 		return "", "", false
 	}

internal/store/sqlitestore/schema.sql

@@ -692,6 +692,13 @@ CREATE TABLE IF NOT EXISTS mcp_servers (
     timeout_sec  INT DEFAULT 60,
     settings     TEXT NOT NULL DEFAULT '{}',
     enabled      BOOLEAN NOT NULL DEFAULT 1,
+    -- require_user_credentials mirrors settings.require_user_credentials but
+    -- promoted to a top-level column so the Bitrix24 channel factory can
+    -- filter mcp_servers directly (indexable) rather than parse the full
+    -- JSONB per row. false = shared admin api_key applies to every caller.
+    -- true = the server mints credentials per-user at message time (Bitrix24
+    -- auto-onboard etc.).
+    require_user_credentials BOOLEAN NOT NULL DEFAULT 0,
     created_by   VARCHAR(255) NOT NULL,
     tenant_id    TEXT NOT NULL REFERENCES tenants(id),
     created_at   TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),

internal/store/sqlitestore/sqlx_scan_structs.go

@@ -104,28 +104,30 @@ type mcpServerRow struct {
 	ToolPrefix  *string         `json:"tool_prefix" db:"tool_prefix"`
 	TimeoutSec  int             `json:"timeout_sec" db:"timeout_sec"`
 	Settings    json.RawMessage `json:"settings" db:"settings"`
-	Enabled     bool            `json:"enabled" db:"enabled"`
-	CreatedBy   string          `json:"created_by" db:"created_by"`
-	CreatedAt   sqliteTime      `json:"created_at" db:"created_at"`
-	UpdatedAt   sqliteTime      `json:"updated_at" db:"updated_at"`
+	Enabled        bool       `json:"enabled" db:"enabled"`
+	RequireUserCredentials bool `json:"require_user_credentials" db:"require_user_credentials"`
+	CreatedBy      string     `json:"created_by" db:"created_by"`
+	CreatedAt      sqliteTime `json:"created_at" db:"created_at"`
+	UpdatedAt      sqliteTime `json:"updated_at" db:"updated_at"`
 }
 
 func (r *mcpServerRow) toMCPServerData() store.MCPServerData {
 	return store.MCPServerData{
-		BaseModel:   store.BaseModel{ID: r.ID, CreatedAt: r.CreatedAt.Time, UpdatedAt: r.UpdatedAt.Time},
-		Name:        r.Name,
-		DisplayName: derefStr(r.DisplayName),
-		Transport:   r.Transport,
-		Command:     derefStr(r.Command),
-		Args:        r.Args,
-		URL:         derefStr(r.URL),
-		Headers:     r.Headers,
-		Env:         r.Env,
-		APIKey:      derefStr(r.APIKey),
-		ToolPrefix:  derefStr(r.ToolPrefix),
-		TimeoutSec:  r.TimeoutSec,
-		Settings:    r.Settings,
-		Enabled:     r.Enabled,
-		CreatedBy:   r.CreatedBy,
+		BaseModel:      store.BaseModel{ID: r.ID, CreatedAt: r.CreatedAt.Time, UpdatedAt: r.UpdatedAt.Time},
+		Name:           r.Name,
+		DisplayName:    derefStr(r.DisplayName),
+		Transport:      r.Transport,
+		Command:        derefStr(r.Command),
+		Args:           r.Args,
+		URL:            derefStr(r.URL),
+		Headers:        r.Headers,
+		Env:            r.Env,
+		APIKey:         derefStr(r.APIKey),
+		ToolPrefix:     derefStr(r.ToolPrefix),
+		TimeoutSec:     r.TimeoutSec,
+		Settings:       r.Settings,
+		Enabled:        r.Enabled,
+		RequireUserCredentials: r.RequireUserCredentials,
+		CreatedBy:      r.CreatedBy,
 	}
 }