mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-13 22:15:47 +00:00
* feat(bitrix24): replace two MCP text inputs with a filtered dropdown [B24:2794]
Bitrix24 channel creation used to demand two hand-typed strings —
mcp_server_name and mcp_base_url — plus zero indication of which MCP
servers can actually auto-onboard. Typos silently disabled provisioning
and admin had to know which servers implement /api/auto-onboard.
Ship a single dropdown backed by mcp_servers.require_user_credentials,
plus the machinery to make it work end-to-end.
Phase 1 — DB & store
* Promote require_user_credentials from settings JSONB to a top-level
column on mcp_servers (PG migration 000089, SQLite migration v54).
* Backfill from existing settings blobs so no admin needs to re-tick.
* Add MCPServerData.RequireUserCredentials to the Go store layer, plumb
through Create / Get / GetByName / List / Update on both stores,
extend the export DTO, and add require_user_credentials to the HTTP
allowlist.
* Bump RequiredSchemaVersion 87 -> 89 (jumping 88, which was on disk
but not wired) and SchemaVersion 53 -> 54 with an
idempotentColumnMigration guard.
Phase 2 — Bitrix24 channel factory
* Add MCPServerID (UUID string) to bitrix24 InstanceConfig, keep
MCPServerName + MCPBaseURL as legacy fallback with a "deprecated"
doc comment.
* Factory validation accepts either mcp_server_id alone or the legacy
pair; half-config still fails fast.
* initMCPProvisioner prefers GetServer(id) and sources the base URL
from mcp_servers.url when the id path is used. Legacy name path
unchanged so pre-migration configs keep working.
* Log line now carries mcp_server_id + require_user_credentials so
operators can eyeball the wiring.
Phase 3 — Frontend types & MCP form
* Add optional top-level require_user_credentials to MCPServerData /
MCPServerInput in both ui/web and ui/desktop/frontend types.
* mcp-form-dialog reads the top-level flag first and falls back to
settings.require_user_credentials so cached responses from
pre-upgrade backends still render correctly.
* On submit send both the top-level flag AND the legacy settings
entry so mid-rollout backends stay consistent.
Phase 4 — Bitrix24 channel form dropdown
* New mcp-select field type + MCPServerSelect component. Uses the
shared useMCP() react-query cache and filters client-side to
servers whose require_user_credentials is true (OR settings
JSONB during the migration window).
* Explicit "None (disable MCP provisioning)" option so admins can
clear the binding without editing config JSON.
* Legacy mcp_server_name / mcp_base_url text inputs kept in the
Advanced panel, relabelled "(legacy)" with pointer help text.
Phase 5 — channel_instances.config backfill
* PG migration 000090 and SQLite migration v55 rewrite existing
bitrix24 channel_instances.config to add mcp_server_id by
resolving mcp_server_name against mcp_servers, tenant-scoped
via agents.tenant_id (channel_instances doesn't carry tenant_id
directly).
* Idempotent — only touches rows already carrying
mcp_server_name that lack mcp_server_id. Legacy keys are left
in place so provisioner.go can still fall back for unmigrated
or future-created legacy configs.
* down.sql drops the mcp_server_id key. Provisioner immediately
reverts to the legacy fallback path.
Tests
* provisioner_test.go: three new cases exercise the mcp_server_id
path (invalid UUID string, valid UUID with missing row, valid
UUID with a per-user row). fakeMCPStore gains a serversByID
map and a real GetServer implementation.
* Existing legacy-config tests unchanged and still green.
Verification
* go build ./... && go build -tags sqliteonly ./...
* go vet ./internal/mcp/... ./internal/channels/bitrix24/...
./internal/store/... ./internal/http/...
* go test ./internal/mcp/... ./internal/channels/bitrix24/... -> ok
* Live-tested against a local docker image on the goclaw-deploy
postgres. Migrations 89 + 90 applied cleanly. Three existing
bitrix24 channels (bitrix-sales / nguyen-dao-openline / tieu-vi)
had their configs backfilled with the b24-syn-mcp UUID and the
provisioner boots with require_user_credentials=true. UI dropdown
correctly shows only b24-syn-mcp (the only server with the flag
ticked) alongside a "None" clearer option.
Surface parity
* Gateway server: store + factory + provisioner + HTTP allowlist.
* API contract: adds require_user_credentials + mcp_server_id
as optional fields on existing routes. No new endpoints.
* Web UI: MCP form + Bitrix24 channel form + shared types.
* CLI/runtime package: N/A because no CLI subcommand reads the
mcp_server_id field.
* fix(bitrix24): derive auto-onboard base URL from mcp_servers.url origin [B24:2794]
The Phase 2 refactor swapped provisioner base-URL sourcing from the
legacy per-channel MCPBaseURL config field (which historically stored the
MCP server's ORIGIN, e.g. https://mcp.example.com) to mcp_servers.url,
which stores the JSON-RPC ENDPOINT the agent loop dials (e.g.
https://mcp.example.com/mcp). The two are semantically different but
share a single column.
mcp_client.newMCPClient then appends "/api/auto-onboard" to whatever
baseURL it receives, so the id-path started POSTing to
".../mcp/api/auto-onboard" — 404 for every per-user credential mint and
refresh. Existing users kept working only until their cached access
tokens expired.
Fix: derive the origin (scheme://host[:port]) from server.URL before
handing it to the auto-onboard client. The legacy path is untouched
because MCPBaseURL from channel config is already the origin.
https://b24-mcp-dev.synity.so/mcp -> https://b24-mcp-dev.synity.so
https://mcp.example.com/mcp/ -> https://mcp.example.com
https://mcp.example.com -> https://mcp.example.com
Table-driven test covers six shapes plus four error cases (empty,
whitespace-only, no scheme, no host). Updated the existing
TestInitMCPProvisioner_MCPServerID fixture to seed a URL with the /mcp
subpath so it regression-guards the same code path.
Verified live: user 614 sent a message that triggered the expired-cred
refresh branch; goclaw logged "self-refreshed user credentials
created=false" and the agent immediately reported
"mcp.user_tools_loaded user=614 tools=2". Before this fix the same event
logged 'auto-onboard failed: mcp auto-onboard: 404 Not Found'.
Surface parity:
- Gateway server: provisioner + one new helper (deriveAutoOnboardBaseURL).
- API contract: N/A because the wire shape hasn't changed.
- Web UI: N/A because the UI still writes mcp_server_id verbatim.
- CLI/runtime: N/A because no CLI reads the derived base URL.
---------
Co-authored-by: DangTinh311 <dangtinh31193@gmail.com>
186 lines
9.5 KiB
Go
186 lines
9.5 KiB
Go
package store
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"time"
|
|
|
|
"github.com/google/uuid"
|
|
)
|
|
|
|
// MCPServerData represents an MCP server in the database.
|
|
type MCPServerData struct {
|
|
BaseModel
|
|
Name string `json:"name" db:"name"`
|
|
DisplayName string `json:"display_name,omitempty" db:"display_name"`
|
|
Transport string `json:"transport" db:"transport"`
|
|
Command string `json:"command,omitempty" db:"command"`
|
|
Args json.RawMessage `json:"args,omitempty" db:"args"`
|
|
URL string `json:"url,omitempty" db:"url"`
|
|
Headers json.RawMessage `json:"headers,omitempty" db:"headers"`
|
|
Env json.RawMessage `json:"env,omitempty" db:"env"`
|
|
APIKey string `json:"api_key,omitempty" db:"api_key"`
|
|
ToolPrefix string `json:"tool_prefix,omitempty" db:"tool_prefix"`
|
|
TimeoutSec int `json:"timeout_sec" db:"timeout_sec"`
|
|
Settings json.RawMessage `json:"settings,omitempty" db:"settings"`
|
|
Enabled bool `json:"enabled" db:"enabled"`
|
|
// RequireUserCredentials marks servers that mint credentials per-user at
|
|
// message time (e.g. Bitrix24 channel auto-onboard) instead of sharing a
|
|
// single admin api_key across every caller. Promoted from
|
|
// settings.require_user_credentials (JSONB) to a top-level column so
|
|
// channel factories can filter mcp_servers directly.
|
|
RequireUserCredentials bool `json:"require_user_credentials" db:"require_user_credentials"`
|
|
CreatedBy string `json:"created_by" db:"created_by"`
|
|
}
|
|
|
|
// MCPAgentGrant represents an MCP server grant to an agent.
|
|
type MCPAgentGrant struct {
|
|
ID uuid.UUID `json:"id" db:"id"`
|
|
ServerID uuid.UUID `json:"server_id" db:"server_id"`
|
|
AgentID uuid.UUID `json:"agent_id" db:"agent_id"`
|
|
Enabled bool `json:"enabled" db:"enabled"`
|
|
ToolAllow json.RawMessage `json:"tool_allow,omitempty" db:"tool_allow"` // JSONB
|
|
ToolDeny json.RawMessage `json:"tool_deny,omitempty" db:"tool_deny"` // JSONB
|
|
ConfigOverrides json.RawMessage `json:"config_overrides,omitempty" db:"config_overrides"` // JSONB
|
|
GrantedBy string `json:"granted_by" db:"granted_by"`
|
|
CreatedAt time.Time `json:"created_at" db:"created_at"`
|
|
}
|
|
|
|
// MCPUserGrant represents an MCP server grant to a user.
|
|
type MCPUserGrant struct {
|
|
ID uuid.UUID `json:"id" db:"id"`
|
|
ServerID uuid.UUID `json:"server_id" db:"server_id"`
|
|
UserID string `json:"user_id" db:"user_id"`
|
|
Enabled bool `json:"enabled" db:"enabled"`
|
|
ToolAllow json.RawMessage `json:"tool_allow,omitempty" db:"tool_allow"` // JSONB
|
|
ToolDeny json.RawMessage `json:"tool_deny,omitempty" db:"tool_deny"` // JSONB
|
|
GrantedBy string `json:"granted_by" db:"granted_by"`
|
|
CreatedAt time.Time `json:"created_at" db:"created_at"`
|
|
}
|
|
|
|
// MCPAccessRequest represents a request for MCP server access.
|
|
type MCPAccessRequest struct {
|
|
ID uuid.UUID `json:"id" db:"id"`
|
|
ServerID uuid.UUID `json:"server_id" db:"server_id"`
|
|
AgentID *uuid.UUID `json:"agent_id,omitempty" db:"agent_id"`
|
|
UserID string `json:"user_id,omitempty" db:"user_id"`
|
|
Scope string `json:"scope" db:"scope"` // "agent" or "user"
|
|
Status string `json:"status" db:"status"` // "pending", "approved", "rejected"
|
|
Reason string `json:"reason,omitempty" db:"reason"`
|
|
ToolAllow json.RawMessage `json:"tool_allow,omitempty" db:"tool_allow"` // JSONB
|
|
RequestedBy string `json:"requested_by" db:"requested_by"`
|
|
ReviewedBy string `json:"reviewed_by,omitempty" db:"reviewed_by"`
|
|
ReviewedAt *time.Time `json:"reviewed_at,omitempty" db:"reviewed_at"`
|
|
ReviewNote string `json:"review_note,omitempty" db:"review_note"`
|
|
CreatedAt time.Time `json:"created_at" db:"created_at"`
|
|
}
|
|
|
|
// MCPAccessInfo combines server data with grant-level tool filters for runtime resolution.
|
|
type MCPAccessInfo struct {
|
|
Server MCPServerData `json:"server" db:"-"`
|
|
ToolAllow []string `json:"tool_allow,omitempty" db:"-"` // effective allow list (nil = all)
|
|
ToolDeny []string `json:"tool_deny,omitempty" db:"-"` // effective deny list
|
|
}
|
|
|
|
// MCPUserCredentials holds per-user credential overrides for an MCP server.
|
|
type MCPUserCredentials struct {
|
|
APIKey string `json:"api_key,omitempty" db:"-"` // decrypted
|
|
Headers map[string]string `json:"headers,omitempty" db:"-"` // decrypted
|
|
Env map[string]string `json:"env,omitempty" db:"-"` // decrypted
|
|
}
|
|
|
|
// MCPOAuthToken holds OAuth tokens for an MCP server, either global (UserID="")
|
|
// or per-user (UserID set). Sensitive fields are decrypted before delivery.
|
|
type MCPOAuthToken struct {
|
|
ID uuid.UUID `db:"id"`
|
|
ServerID uuid.UUID `db:"server_id"`
|
|
TenantID uuid.UUID `db:"tenant_id"`
|
|
UserID string `db:"user_id"` // empty = global
|
|
AccessToken string `db:"access_token"`
|
|
RefreshToken string `db:"refresh_token"`
|
|
TokenType string `db:"token_type"`
|
|
Scopes string `db:"scopes"`
|
|
ExpiresAt *time.Time `db:"expires_at"`
|
|
IssuedAt *time.Time `db:"issued_at"`
|
|
DCRClientID string `db:"dcr_client_id"`
|
|
DCRClientSecret string `db:"dcr_client_secret"` // decrypted
|
|
DCRIssuer string `db:"dcr_issuer"`
|
|
TokenEndpoint string `db:"token_endpoint"`
|
|
ResourceURI string `db:"resource_uri"`
|
|
CreatedAt time.Time `db:"created_at"`
|
|
UpdatedAt time.Time `db:"updated_at"`
|
|
}
|
|
|
|
// MCPOAuthTokenStore manages OAuth tokens for MCP servers.
|
|
type MCPOAuthTokenStore interface {
|
|
// GetOAuthToken returns the global (tenant-level) OAuth token for a server.
|
|
GetOAuthToken(ctx context.Context, serverID, tenantID uuid.UUID) (*MCPOAuthToken, error)
|
|
// GetUserOAuthToken returns the per-user OAuth token for a server.
|
|
GetUserOAuthToken(ctx context.Context, serverID, tenantID uuid.UUID, userID string) (*MCPOAuthToken, error)
|
|
// UpsertOAuthToken inserts or replaces an OAuth token record.
|
|
UpsertOAuthToken(ctx context.Context, token *MCPOAuthToken) error
|
|
// DeleteOAuthToken deletes the global OAuth token for a server.
|
|
DeleteOAuthToken(ctx context.Context, serverID, tenantID uuid.UUID) error
|
|
// DeleteUserOAuthToken deletes the per-user OAuth token for a server.
|
|
DeleteUserOAuthToken(ctx context.Context, serverID, tenantID uuid.UUID, userID string) error
|
|
// DeleteServerOAuthTokens deletes ALL OAuth tokens for a server (global +
|
|
// every per-user row). Used when the server URL or OAuth config changes and
|
|
// the previously minted tokens are no longer valid for the new resource/AS.
|
|
DeleteServerOAuthTokens(ctx context.Context, serverID, tenantID uuid.UUID) error
|
|
}
|
|
|
|
// MCPServerStore manages MCP server configs and access grants.
|
|
type MCPServerStore interface {
|
|
// Server CRUD
|
|
CreateServer(ctx context.Context, s *MCPServerData) error
|
|
GetServer(ctx context.Context, id uuid.UUID) (*MCPServerData, error)
|
|
GetServerByName(ctx context.Context, name string) (*MCPServerData, error)
|
|
ListServers(ctx context.Context) ([]MCPServerData, error)
|
|
UpdateServer(ctx context.Context, id uuid.UUID, updates map[string]any) error
|
|
DeleteServer(ctx context.Context, id uuid.UUID) error
|
|
|
|
// Agent grants
|
|
GrantToAgent(ctx context.Context, g *MCPAgentGrant) error
|
|
RevokeFromAgent(ctx context.Context, serverID, agentID uuid.UUID) error
|
|
ListAgentGrants(ctx context.Context, agentID uuid.UUID) ([]MCPAgentGrant, error)
|
|
ListServerGrants(ctx context.Context, serverID uuid.UUID) ([]MCPAgentGrant, error)
|
|
|
|
// User grants
|
|
GrantToUser(ctx context.Context, g *MCPUserGrant) error
|
|
RevokeFromUser(ctx context.Context, serverID uuid.UUID, userID string) error
|
|
|
|
// Counts: agent grant counts per server (for listing UI)
|
|
CountAgentGrantsByServer(ctx context.Context) (map[uuid.UUID]int, error)
|
|
|
|
// Resolution: all accessible MCP servers + tool filters for agent+user
|
|
ListAccessible(ctx context.Context, agentID uuid.UUID, userID string) ([]MCPAccessInfo, error)
|
|
|
|
// Access requests
|
|
CreateRequest(ctx context.Context, req *MCPAccessRequest) error
|
|
ListPendingRequests(ctx context.Context) ([]MCPAccessRequest, error)
|
|
ReviewRequest(ctx context.Context, requestID uuid.UUID, approved bool, reviewedBy, note string) error
|
|
|
|
// Per-user credentials
|
|
GetUserCredentials(ctx context.Context, serverID uuid.UUID, userID string) (*MCPUserCredentials, error)
|
|
SetUserCredentials(ctx context.Context, serverID uuid.UUID, userID string, creds MCPUserCredentials) error
|
|
DeleteUserCredentials(ctx context.Context, serverID uuid.UUID, userID string) error
|
|
|
|
// CacheToolDescriptions stores a map of tool name → cached tool info
|
|
// (description + real parameter schema, when available) into the
|
|
// server's settings JSONB under the "tool_cache" key.
|
|
CacheToolDescriptions(ctx context.Context, serverID uuid.UUID, toolInfo map[string]CachedToolInfo) error
|
|
}
|
|
|
|
// CachedToolInfo is the cached, per-tool information stored under an MCP
|
|
// server's settings "tool_cache" key. It is used to render prompt previews
|
|
// without a live connection to the MCP server.
|
|
type CachedToolInfo struct {
|
|
// Description is the tool's human-readable description.
|
|
Description string `json:"description"`
|
|
// Parameters is the tool's full JSON Schema for its input parameters,
|
|
// captured from the live MCP connection at cache-write time. It may be
|
|
// nil for cache entries written before this field existed, or for
|
|
// servers whose tools genuinely expose no schema.
|
|
Parameters json.RawMessage `json:"parameters,omitempty"`
|
|
}
|