mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-11 12:18:59 +00:00
* feat(bitrix24): replace two MCP text inputs with a filtered dropdown [B24:2794]
Bitrix24 channel creation used to demand two hand-typed strings —
mcp_server_name and mcp_base_url — plus zero indication of which MCP
servers can actually auto-onboard. Typos silently disabled provisioning
and admin had to know which servers implement /api/auto-onboard.
Ship a single dropdown backed by mcp_servers.require_user_credentials,
plus the machinery to make it work end-to-end.
Phase 1 — DB & store
* Promote require_user_credentials from settings JSONB to a top-level
column on mcp_servers (PG migration 000089, SQLite migration v54).
* Backfill from existing settings blobs so no admin needs to re-tick.
* Add MCPServerData.RequireUserCredentials to the Go store layer, plumb
through Create / Get / GetByName / List / Update on both stores,
extend the export DTO, and add require_user_credentials to the HTTP
allowlist.
* Bump RequiredSchemaVersion 87 -> 89 (jumping 88, which was on disk
but not wired) and SchemaVersion 53 -> 54 with an
idempotentColumnMigration guard.
Phase 2 — Bitrix24 channel factory
* Add MCPServerID (UUID string) to bitrix24 InstanceConfig, keep
MCPServerName + MCPBaseURL as legacy fallback with a "deprecated"
doc comment.
* Factory validation accepts either mcp_server_id alone or the legacy
pair; half-config still fails fast.
* initMCPProvisioner prefers GetServer(id) and sources the base URL
from mcp_servers.url when the id path is used. Legacy name path
unchanged so pre-migration configs keep working.
* Log line now carries mcp_server_id + require_user_credentials so
operators can eyeball the wiring.
Phase 3 — Frontend types & MCP form
* Add optional top-level require_user_credentials to MCPServerData /
MCPServerInput in both ui/web and ui/desktop/frontend types.
* mcp-form-dialog reads the top-level flag first and falls back to
settings.require_user_credentials so cached responses from
pre-upgrade backends still render correctly.
* On submit send both the top-level flag AND the legacy settings
entry so mid-rollout backends stay consistent.
Phase 4 — Bitrix24 channel form dropdown
* New mcp-select field type + MCPServerSelect component. Uses the
shared useMCP() react-query cache and filters client-side to
servers whose require_user_credentials is true (OR settings
JSONB during the migration window).
* Explicit "None (disable MCP provisioning)" option so admins can
clear the binding without editing config JSON.
* Legacy mcp_server_name / mcp_base_url text inputs kept in the
Advanced panel, relabelled "(legacy)" with pointer help text.
Phase 5 — channel_instances.config backfill
* PG migration 000090 and SQLite migration v55 rewrite existing
bitrix24 channel_instances.config to add mcp_server_id by
resolving mcp_server_name against mcp_servers, tenant-scoped
via agents.tenant_id (channel_instances doesn't carry tenant_id
directly).
* Idempotent — only touches rows already carrying
mcp_server_name that lack mcp_server_id. Legacy keys are left
in place so provisioner.go can still fall back for unmigrated
or future-created legacy configs.
* down.sql drops the mcp_server_id key. Provisioner immediately
reverts to the legacy fallback path.
Tests
* provisioner_test.go: three new cases exercise the mcp_server_id
path (invalid UUID string, valid UUID with missing row, valid
UUID with a per-user row). fakeMCPStore gains a serversByID
map and a real GetServer implementation.
* Existing legacy-config tests unchanged and still green.
Verification
* go build ./... && go build -tags sqliteonly ./...
* go vet ./internal/mcp/... ./internal/channels/bitrix24/...
./internal/store/... ./internal/http/...
* go test ./internal/mcp/... ./internal/channels/bitrix24/... -> ok
* Live-tested against a local docker image on the goclaw-deploy
postgres. Migrations 89 + 90 applied cleanly. Three existing
bitrix24 channels (bitrix-sales / nguyen-dao-openline / tieu-vi)
had their configs backfilled with the b24-syn-mcp UUID and the
provisioner boots with require_user_credentials=true. UI dropdown
correctly shows only b24-syn-mcp (the only server with the flag
ticked) alongside a "None" clearer option.
Surface parity
* Gateway server: store + factory + provisioner + HTTP allowlist.
* API contract: adds require_user_credentials + mcp_server_id
as optional fields on existing routes. No new endpoints.
* Web UI: MCP form + Bitrix24 channel form + shared types.
* CLI/runtime package: N/A because no CLI subcommand reads the
mcp_server_id field.
* fix(bitrix24): derive auto-onboard base URL from mcp_servers.url origin [B24:2794]
The Phase 2 refactor swapped provisioner base-URL sourcing from the
legacy per-channel MCPBaseURL config field (which historically stored the
MCP server's ORIGIN, e.g. https://mcp.example.com) to mcp_servers.url,
which stores the JSON-RPC ENDPOINT the agent loop dials (e.g.
https://mcp.example.com/mcp). The two are semantically different but
share a single column.
mcp_client.newMCPClient then appends "/api/auto-onboard" to whatever
baseURL it receives, so the id-path started POSTing to
".../mcp/api/auto-onboard" — 404 for every per-user credential mint and
refresh. Existing users kept working only until their cached access
tokens expired.
Fix: derive the origin (scheme://host[:port]) from server.URL before
handing it to the auto-onboard client. The legacy path is untouched
because MCPBaseURL from channel config is already the origin.
https://b24-mcp-dev.synity.so/mcp -> https://b24-mcp-dev.synity.so
https://mcp.example.com/mcp/ -> https://mcp.example.com
https://mcp.example.com -> https://mcp.example.com
Table-driven test covers six shapes plus four error cases (empty,
whitespace-only, no scheme, no host). Updated the existing
TestInitMCPProvisioner_MCPServerID fixture to seed a URL with the /mcp
subpath so it regression-guards the same code path.
Verified live: user 614 sent a message that triggered the expired-cred
refresh branch; goclaw logged "self-refreshed user credentials
created=false" and the agent immediately reported
"mcp.user_tools_loaded user=614 tools=2". Before this fix the same event
logged 'auto-onboard failed: mcp auto-onboard: 404 Not Found'.
Surface parity:
- Gateway server: provisioner + one new helper (deriveAutoOnboardBaseURL).
- API contract: N/A because the wire shape hasn't changed.
- Web UI: N/A because the UI still writes mcp_server_id verbatim.
- CLI/runtime: N/A because no CLI reads the derived base URL.
---------
Co-authored-by: DangTinh311 <dangtinh31193@gmail.com>
265 lines
8.1 KiB
Go
265 lines
8.1 KiB
Go
//go:build sqlite || sqliteonly
|
|
|
|
package sqlitestore
|
|
|
|
import (
|
|
"context"
|
|
"database/sql"
|
|
"encoding/json"
|
|
"fmt"
|
|
"log/slog"
|
|
"time"
|
|
|
|
"github.com/google/uuid"
|
|
|
|
"github.com/nextlevelbuilder/goclaw/internal/crypto"
|
|
"github.com/nextlevelbuilder/goclaw/internal/store"
|
|
)
|
|
|
|
const mcpServerSelectCols = `id, name, display_name, transport, command, args, url, headers, env,
|
|
api_key, tool_prefix, timeout_sec, settings, enabled, require_user_credentials, created_by, created_at, updated_at`
|
|
|
|
// SQLiteMCPServerStore implements store.MCPServerStore backed by SQLite.
|
|
type SQLiteMCPServerStore struct {
|
|
db *sql.DB
|
|
encKey string
|
|
}
|
|
|
|
func NewSQLiteMCPServerStore(db *sql.DB, encryptionKey string) *SQLiteMCPServerStore {
|
|
return &SQLiteMCPServerStore{db: db, encKey: encryptionKey}
|
|
}
|
|
|
|
func (s *SQLiteMCPServerStore) CreateServer(ctx context.Context, srv *store.MCPServerData) error {
|
|
if err := store.ValidateUserID(srv.CreatedBy); err != nil {
|
|
return err
|
|
}
|
|
if srv.ID == uuid.Nil {
|
|
srv.ID = store.GenNewID()
|
|
}
|
|
|
|
apiKey := srv.APIKey
|
|
if s.encKey != "" && apiKey != "" {
|
|
encrypted, err := crypto.Encrypt(apiKey, s.encKey)
|
|
if err != nil {
|
|
return fmt.Errorf("encrypt api key: %w", err)
|
|
}
|
|
apiKey = encrypted
|
|
}
|
|
|
|
now := time.Now().UTC()
|
|
srv.CreatedAt = now
|
|
srv.UpdatedAt = now
|
|
encHeaders := s.encryptJSON(jsonOrEmpty(srv.Headers))
|
|
encEnv := s.encryptJSON(jsonOrEmpty(srv.Env))
|
|
|
|
tenantID := store.TenantIDFromContext(ctx)
|
|
if tenantID == uuid.Nil {
|
|
tenantID = store.MasterTenantID
|
|
}
|
|
|
|
_, err := s.db.ExecContext(ctx,
|
|
`INSERT INTO mcp_servers (id, name, display_name, transport, command, args, url, headers, env,
|
|
api_key, tool_prefix, timeout_sec, settings, enabled, require_user_credentials, created_by, created_at, updated_at, tenant_id)
|
|
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
|
|
srv.ID, srv.Name, nilStr(srv.DisplayName), srv.Transport, nilStr(srv.Command),
|
|
jsonOrEmpty(srv.Args), nilStr(srv.URL), encHeaders, encEnv,
|
|
nilStr(apiKey), nilStr(srv.ToolPrefix), srv.TimeoutSec,
|
|
jsonOrEmpty(srv.Settings), srv.Enabled, srv.RequireUserCredentials, srv.CreatedBy, now, now, tenantID,
|
|
)
|
|
return err
|
|
}
|
|
|
|
func (s *SQLiteMCPServerStore) GetServer(ctx context.Context, id uuid.UUID) (*store.MCPServerData, error) {
|
|
q := `SELECT ` + mcpServerSelectCols + ` FROM mcp_servers WHERE id = ?`
|
|
qArgs := []any{id}
|
|
if !store.IsCrossTenant(ctx) {
|
|
tenantID := store.TenantIDFromContext(ctx)
|
|
if tenantID == uuid.Nil {
|
|
return nil, sql.ErrNoRows
|
|
}
|
|
q += ` AND tenant_id = ?`
|
|
qArgs = append(qArgs, tenantID)
|
|
}
|
|
var row mcpServerRow
|
|
if err := pkgSqlxDB.GetContext(ctx, &row, q, qArgs...); err != nil {
|
|
return nil, err
|
|
}
|
|
srv := row.toMCPServerData()
|
|
s.decryptServerFields(&srv)
|
|
return &srv, nil
|
|
}
|
|
|
|
func (s *SQLiteMCPServerStore) GetServerByName(ctx context.Context, name string) (*store.MCPServerData, error) {
|
|
q := `SELECT ` + mcpServerSelectCols + ` FROM mcp_servers WHERE name = ?`
|
|
qArgs := []any{name}
|
|
if !store.IsCrossTenant(ctx) {
|
|
tenantID := store.TenantIDFromContext(ctx)
|
|
if tenantID == uuid.Nil {
|
|
return nil, sql.ErrNoRows
|
|
}
|
|
q += ` AND tenant_id = ?`
|
|
qArgs = append(qArgs, tenantID)
|
|
}
|
|
var row mcpServerRow
|
|
if err := pkgSqlxDB.GetContext(ctx, &row, q, qArgs...); err != nil {
|
|
return nil, err
|
|
}
|
|
srv := row.toMCPServerData()
|
|
s.decryptServerFields(&srv)
|
|
return &srv, nil
|
|
}
|
|
|
|
// decryptServerFields decrypts api_key, headers, and env after scan.
|
|
func (s *SQLiteMCPServerStore) decryptServerFields(srv *store.MCPServerData) {
|
|
srv.Headers = s.decryptJSON(srv.Headers)
|
|
srv.Env = s.decryptJSON(srv.Env)
|
|
if srv.APIKey != "" && s.encKey != "" {
|
|
if decrypted, err := crypto.Decrypt(srv.APIKey, s.encKey); err == nil {
|
|
srv.APIKey = decrypted
|
|
} else {
|
|
slog.Warn("mcp: failed to decrypt api key", "server", srv.Name, "error", err)
|
|
}
|
|
}
|
|
}
|
|
|
|
func (s *SQLiteMCPServerStore) ListServers(ctx context.Context) ([]store.MCPServerData, error) {
|
|
q := `SELECT ` + mcpServerSelectCols + ` FROM mcp_servers`
|
|
var qArgs []any
|
|
if !store.IsCrossTenant(ctx) {
|
|
tenantID := store.TenantIDFromContext(ctx)
|
|
if tenantID == uuid.Nil {
|
|
return []store.MCPServerData{}, nil
|
|
}
|
|
q += ` WHERE tenant_id = ?`
|
|
qArgs = append(qArgs, tenantID)
|
|
}
|
|
q += ` ORDER BY name`
|
|
|
|
var rows []mcpServerRow
|
|
if err := pkgSqlxDB.SelectContext(ctx, &rows, q, qArgs...); err != nil {
|
|
return nil, err
|
|
}
|
|
result := make([]store.MCPServerData, 0, len(rows))
|
|
for _, r := range rows {
|
|
srv := r.toMCPServerData()
|
|
s.decryptServerFields(&srv)
|
|
result = append(result, srv)
|
|
}
|
|
return result, nil
|
|
}
|
|
|
|
func (s *SQLiteMCPServerStore) UpdateServer(ctx context.Context, id uuid.UUID, updates map[string]any) error {
|
|
if key, ok := updates["api_key"]; ok {
|
|
if keyStr, isStr := key.(string); isStr && keyStr != "" && s.encKey != "" {
|
|
encrypted, err := crypto.Encrypt(keyStr, s.encKey)
|
|
if err != nil {
|
|
return fmt.Errorf("encrypt api key: %w", err)
|
|
}
|
|
updates["api_key"] = encrypted
|
|
}
|
|
}
|
|
for _, field := range []string{"env", "headers"} {
|
|
if v, ok := updates[field]; ok {
|
|
var raw []byte
|
|
switch val := v.(type) {
|
|
case json.RawMessage:
|
|
raw = []byte(val)
|
|
default:
|
|
raw, _ = json.Marshal(val)
|
|
}
|
|
if len(raw) > 0 {
|
|
updates[field] = json.RawMessage(s.encryptJSON(raw))
|
|
}
|
|
}
|
|
}
|
|
updates["updated_at"] = time.Now().UTC()
|
|
if store.IsCrossTenant(ctx) {
|
|
return execMapUpdate(ctx, s.db, "mcp_servers", id, updates)
|
|
}
|
|
tid := store.TenantIDFromContext(ctx)
|
|
if tid == uuid.Nil {
|
|
return fmt.Errorf("tenant_id required for update")
|
|
}
|
|
return execMapUpdateWhereTenant(ctx, s.db, "mcp_servers", updates, id, tid)
|
|
}
|
|
|
|
func (s *SQLiteMCPServerStore) DeleteServer(ctx context.Context, id uuid.UUID) error {
|
|
if store.IsCrossTenant(ctx) {
|
|
_, err := s.db.ExecContext(ctx, "DELETE FROM mcp_servers WHERE id = ?", id)
|
|
return err
|
|
}
|
|
tid := store.TenantIDFromContext(ctx)
|
|
if tid == uuid.Nil {
|
|
return fmt.Errorf("tenant_id required")
|
|
}
|
|
_, err := s.db.ExecContext(ctx, "DELETE FROM mcp_servers WHERE id = ? AND tenant_id = ?", id, tid)
|
|
return err
|
|
}
|
|
|
|
// CacheToolDescriptions stores a map of tool name → cached tool info
|
|
// (description + parameter schema) into the server's settings JSON under
|
|
// the "tool_cache" key.
|
|
// SQLite has no jsonb_set(); we read-modify-write the settings column instead.
|
|
func (s *SQLiteMCPServerStore) CacheToolDescriptions(ctx context.Context, serverID uuid.UUID, toolInfo map[string]store.CachedToolInfo) error {
|
|
row := s.db.QueryRowContext(ctx, `SELECT COALESCE(settings, '{}') FROM mcp_servers WHERE id = ?`, serverID)
|
|
var rawSettings []byte
|
|
if err := row.Scan(&rawSettings); err != nil {
|
|
return fmt.Errorf("mcp_servers.cache_tool_descriptions read: %w", err)
|
|
}
|
|
|
|
var settings map[string]json.RawMessage
|
|
if err := json.Unmarshal(rawSettings, &settings); err != nil {
|
|
settings = make(map[string]json.RawMessage)
|
|
}
|
|
|
|
cacheJSON, err := json.Marshal(toolInfo)
|
|
if err != nil {
|
|
return fmt.Errorf("marshal tool descriptions: %w", err)
|
|
}
|
|
settings["tool_cache"] = json.RawMessage(cacheJSON)
|
|
|
|
merged, err := json.Marshal(settings)
|
|
if err != nil {
|
|
return fmt.Errorf("marshal settings: %w", err)
|
|
}
|
|
|
|
_, err = s.db.ExecContext(ctx, `UPDATE mcp_servers SET settings = ?, updated_at = ? WHERE id = ?`,
|
|
string(merged), time.Now().UTC(), serverID)
|
|
if err != nil {
|
|
return fmt.Errorf("mcp_servers.cache_tool_descriptions write: %w", err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// encryptJSON encrypts a JSON blob by wrapping ciphertext as a JSON string.
|
|
// Unencrypted: {"key":"val"} (JSON object). Encrypted: "aes-gcm:..." (JSON string).
|
|
func (s *SQLiteMCPServerStore) encryptJSON(data []byte) []byte {
|
|
if s.encKey == "" || len(data) == 0 || string(data) == "{}" || string(data) == "null" {
|
|
return data
|
|
}
|
|
enc, err := crypto.Encrypt(string(data), s.encKey)
|
|
if err != nil {
|
|
slog.Warn("mcp: failed to encrypt json", "error", err)
|
|
return data
|
|
}
|
|
wrapped, _ := json.Marshal(enc)
|
|
return wrapped
|
|
}
|
|
|
|
// decryptJSON decrypts a JSON blob if it is an encrypted JSON string.
|
|
func (s *SQLiteMCPServerStore) decryptJSON(data []byte) []byte {
|
|
if s.encKey == "" || len(data) == 0 || data[0] != '"' {
|
|
return data
|
|
}
|
|
var encStr string
|
|
if json.Unmarshal(data, &encStr) != nil {
|
|
return data
|
|
}
|
|
dec, err := crypto.Decrypt(encStr, s.encKey)
|
|
if err != nil {
|
|
slog.Warn("mcp: failed to decrypt json", "error", err)
|
|
return data
|
|
}
|
|
return []byte(dec)
|
|
}
|