mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-14 14:17:40 +00:00
* refactor(bitrix24): rename "Path B" framing to maintainer-specified naming [B24:2794] Per maintainer hard rule #10 (no generic "Path A/B" framing) from PR #1061 review. The Bitrix24 MCP auto-onboard flow is Bitrix-specific glue ("Bitrix24 OAuth -> existing mcp_user_credentials bridge"), NOT a generic MCP architecture pattern. Naming convention applied consistently: - First mention per file: full "Bitrix24 OAuth -> existing mcp_user_credentials bridge" (matches maintainer comment verbatim). - Subsequent mentions in same file: shortened "mcp_user_credentials bridge". - Test/log context referencing literal endpoint /api/auto-onboard: keep "auto-onboard" reference (it's the actual API endpoint name). Changes are documentation-only: - Rename in code comments + test descriptions + plan docs. - Clarify framing in mcp_client.go + provisioner.go doc comments to emphasize Bitrix-specific glue (not generic MCP infra). - Reuse existing mcp_user_credentials table + MCPServerStore methods (no schema / store / abstraction change). Files: - cmd/gateway.go (factory registration doc) - internal/channels/bitrix24/{channel,factory,mcp_client,provisioner}.go - internal/channels/bitrix24/{mcp_client,provisioner}_test.go - plan/goclaw-mcp-integration.md (21 occurrences) Verified: go build + MCP-related tests pass (TestProvision*, TestInitMCPProvisioner*, TestMCPClient*). Phase 1 of Path C execution per plans/reports/decision-log-260519-1555-bitrix24-pr-fork-decision.md. * fix: confine outbound media paths to agent workspace [B24:2794] Tool MEDIA:<path> output reached channel file-upload sinks (Bitrix imbot.v2.File.upload, Telegram sendDocument, etc.) verbatim via parseMediaResult, with no workspace-boundary check. A malicious or buggy tool emitting MEDIA:/etc/passwd could exfiltrate arbitrary files to chat. Extract the EvalSymlinks+Rel containment from extractMediaFromContent into a shared confineToWorkspace helper and apply it at the parseMediaResult sink in processToolResult. Fixing at the source/egress boundary protects every channel at once rather than per-channel. Paths that escape the workspace are dropped and logged (security.media_path_rejected). Add TestConfineToWorkspace (boundary unit) and TestParseMediaResultConfinedToWorkspace (sink regression for H2). * feat(bitrix24): support inbound + outbound media via imbot.v2 File API [B24:2794] Bitrix24 channel was text-only; attachments were parsed but dropped. - Inbound: download chat files via imbot.v2.File.download (one-time URL), forward to the agent with MIME preserved (internal/channels/bitrix24/download.go). - Outbound: upload agent media to the chat via imbot.v2.File.upload (internal/channels/bitrix24/send_media.go). - Add BaseChannel.HandleMessageMedia to preserve MIME/filename through the bus. - Per-channel media_max_mb cap (default 20) applies to both directions. Tests: 92 pass (internal/channels/bitrix24 + internal/channels), go vet clean (PG + sqliteonly). * refactor(bitrix24): migrate messaging/bot-list/unregister to imbot v2 API [B24:2794] Move outbound REST calls to the imbot v2 family (keeps register on v1): - imbot.message.add -> imbot.v2.Chat.Message.send (fields.message shape, live-verified) - imbot.bot.list (+ legacy imbot.list fallback) -> imbot.v2.Bot.list; add botListRows to normalize the v2 {bots:[...]} envelope, legacy array, and id-keyed map forms - imbot.unregister -> imbot.v2.Bot.unregister Bot registration stays on v1 imbot.register: v2 imbot.v2.Bot.register changes the event-delivery model (per-event handler URLs -> eventMode), which would require rewriting the inbound event parser. No user-facing behavior change. Tests: bitrix24 package green; go vet ./... clean. * feat(bitrix24): route whisper via v1 SKIP_CONNECTOR + add v2 replyId [B24:2794] Bot was leaking HiddenMessage (whisper) replies to the external Zalo connector because every outbound call went through imbot.v2.Chat.Message.send, which has no equivalent of the v1 SKIP_CONNECTOR flag. Branch the outbound path on inbound visibility: whisper → imbot.message.add + SKIP_CONNECTOR=Y (v1, send_v1.go) public → imbot.v2.Chat.Message.send + fields.replyId (v2, send_v2.go) Pipeline: events.go parse data[PARAMS][PARAMS][COMPONENT_ID]=HiddenMessage into EventParams.IsHiddenMessage (form + JSON variants) handle.go set bitrix_visibility on InboundMessage.Metadata consumer forward visibility + message_id into OutboundMessage send.go resolveSendOptions + sendChunk dispatcher + shared callWithRateLimitRetry helper metadata_keys.go single source of truth for the keys + values Defaults preserve pre-refactor behaviour: callers that don't populate bitrix_visibility still go through v2 public, and replyId is omitted unless a numeric bitrix_message_id arrives in metadata. Tests: TestParseEvent_FormURLEncoded_IsHiddenMessage (3 cases) TestParseEvent_JSON_IsHiddenMessage (3 cases) TestResolveSendOptions (8 cases) TestSend_BranchesOnVisibility (4 cases) * feat(bitrix24): openline sender-tag echo on replies [B24:2794] Openline sender-tag echo (this change): - Capture the connector sender tag ("[name #id]:" or "[name] #id:") from inbound openline group messages, strip it from the body the agent sees, and re-prepend the canonical "[name] #id:" form to the reply so the Open Channel connector routes the answer back to the right external user. - New sender_prefix.go helper (+ test) accepts both inbound layouts and emits one canonical form; scoped to messages carrying the tag, so plain chats are unaffected. - metadata_keys.go: MetaKeySenderPrefix; handle.go capture/strip/stash; gateway_consumer_normal.go forwards the key; send.go prepends it on the first chunk before chunking. Bundled bitrix24 channel-core work already on this branch: - handle.go: @mention is the sole trigger for both staff and connector customers; unmentioned traffic is dropped (was: drop all connector msgs). - isGroupMessageType: treat SONET_GROUP "B" as a group. - handle_test.go, mcp_client_test.go: cover the above. * feat(bitrix24): accept colon-less openline sender tag, echo [name] #id [B24:2794] The Open Channel connector dropped the trailing colon from its sender tag: inbound now arrives as "[Name] #id <msg>" (was "[Name] #id: <msg>"). The id-bearing patterns required the colon, so the tag fell through to the name-only branch and the reply echoed "[Name]" — dropping the #id the connector needs to route the answer back. - sender_prefix.go: make the trailing ":" optional on both id layouts ([name #id] / [name] #id, with or without colon) and echo the canonical "[name] #id" (no colon) to match the connector's current format. Bare "[name]" (no id) still echoes "[name]" for Open Channel only. - handle.go: gate the bare name-only layout to Open Channel (isOpenChannel) so ordinary group chats starting with "[x] ..." are left untouched. - sender_prefix_test.go: cover colon/no-colon x id-inside/id-outside, the name-only openline case, and the non-openline no-op. * fix: security and robustness fixes from the bitrix24 channel review [B24:2794] - download.go: block redirect-based SSRF on inbound media. CheckRedirect re-validates each hop (http(s) only, reject private/loopback/link-local hosts, cap hops); the initial portal-domain pin is no longer bypassable via a 3xx to an internal service. Public-host redirects still allowed. - handle.go: extract/echo the openline sender tag only for Open Channel sessions (was: any group chat), removing bogus prefixes in CRM group chats and narrowing the forged-tag misroute surface. - loop_tools.go + loop_media.go: confine result.Media to the agent / team / tenant-allowed roots (new confineToAnyRoot) before a channel uploads it, so a prompt-injected out-of-workspace path (e.g. /etc/passwd) cannot exfiltrate, while legitimate cross-workspace media (team files, delegatee output) still flows. - send_media.go: bounded outbound read via io.LimitReader replaces the os.Stat + os.ReadFile pair, closing the TOCTOU size-cap bypass; cap a single message's outbound attachments at 10 (mirrors inbound). - register.go: paginate imbot.v2.Bot.list (limit/offset + hasNextPage, capped at 40 pages) so verify/lookup see bots past the first 50. - mcp_client.go: redact access_token / refresh_token / client_secret from an echoed MCP error body before it is logged or returned (+ test). * fix(security): validate resolved dial IP on Bitrix media redirects [B24:2794] The inbound media download redirect guard only string-checked the redirect hostname (isPrivateOrLoopback on req.URL.Hostname()), so a redirect to a public hostname that resolves to 127.0.0.1 / 169.254.169.254 / an RFC1918 address — or a DNS-rebinding swap between check and dial — still passed the guard and the client would connect. Reported in PR review. Add security.NewRedirectFollowingSafeClient: it follows redirects but validates the RESOLVED destination IP of every hop at dial time via net.Dialer.Control, reusing the existing blocked-CIDR list. The IP it checks is the IP actually dialed, so both redirect-to-internal and DNS rebinding are refused, while legitimate public CDN redirects still succeed. download.go now uses it instead of the hostname-string guard. Tests: deterministic dial-control table (loopback / link-local / private / multicast / unspecified / public, v4 + v6), malformed/non-IP addr, test bypass, loopback-dial-blocked client wiring, and redirect cap + scheme checks. * feat(bitrix24): per-participant Zalo openline identity from 3-token sender tag [B24:2794] Parse the connector's "[Name] #uid #msgId" sender tag so each external customer in a shared Open Channel group gets its own contact + USER.md instead of collapsing onto the connector proxy id. Identity minting is gated on IS_CONNECTOR=Y to reject operator forged tags. Echo back the msgId only ("#msgId") on replies; keep the legacy single-number and name-only layouts unchanged. Zero DB migration. - sender_prefix.go: parseOpenlineSenderTag() classifies 3-token / legacy / name-only - handle.go: synthetic senderID "openlines:{instance}:{chat}:{uid}" + participant_user_id metadata, gated on FromIsConnector - gateway_consumer_normal.go: deriveGroupUserID() routes participant -> per-person scope, group fallback otherwise - send.go: buildAddressMention numeric-id guard so synthetic ids don't emit invalid [USER=...] BBCode - MetaKeyMessageID kept as Bitrix MESSAGE_ID (drives v2 fields.replyId); connector msgId surfaced only via echo prefix --------- Co-authored-by: DangTinh311 <dangtinh31193@gmail.com> Co-authored-by: Chinh Dang <chinhdang@192.168.68.104>
223 lines
9.1 KiB
Go
223 lines
9.1 KiB
Go
package bitrix24
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"net/http"
|
|
"regexp"
|
|
"strings"
|
|
"time"
|
|
)
|
|
|
|
// mcpClient talks to an MCP server's /api/auto-onboard endpoint.
|
|
//
|
|
// This is Bitrix-specific glue ("Bitrix24 OAuth → existing
|
|
// mcp_user_credentials bridge"), NOT a generic MCP architecture pattern —
|
|
// other channels (Telegram, Discord, …) currently require admins to set
|
|
// per-user MCP credentials manually via the HTTP admin API. Bitrix
|
|
// automates the same flow because Bitrix events naturally carry user
|
|
// OAuth tokens.
|
|
//
|
|
// When a Bitrix24 user sends their first message, the bitrix24 channel
|
|
// doesn't yet know which per-user MCP API key that user should use. It POSTs
|
|
// to the MCP server — which is the authoritative identity provider for this
|
|
// integration — with the triggering user's OAuth tokens (access_token +
|
|
// refresh_token + expires_in) harvested from the Bitrix event auth block.
|
|
// The MCP server verifies the access_token against Bitrix `profile` to
|
|
// confirm the caller actually owns bitrix_user_id — no shared admin secret
|
|
// required — then upserts its own tenants + bitrix_users tables keyed by
|
|
// (domain, bitrix_user_id) and returns the per-user api_key we persist via
|
|
// mcp_user_credentials.
|
|
//
|
|
// The client is deliberately thin:
|
|
// - No retries on 4xx (auth config wrong → operator must fix).
|
|
// - One auto-retry on 5xx / network errors (honours a tight timeout so a
|
|
// slow MCP server can't stall the Bitrix webhook handler).
|
|
// - 404 with body {"error":"tenant_not_installed"} surfaces as
|
|
// ErrTenantNotInstalled so the handler can reply with a specific
|
|
// "reinstall the portal" message instead of a generic error.
|
|
// - Other errors surface verbatim so ensureMCPCredentials can debounce
|
|
// them via the pairing-style replyError gate.
|
|
type mcpClient struct {
|
|
httpClient *http.Client
|
|
baseURL string
|
|
}
|
|
|
|
// ErrTenantNotInstalled is returned when the MCP server reports 404
|
|
// tenant_not_installed for the supplied domain. The channel handler treats
|
|
// this as a distinct failure mode (operator must reinstall the Bitrix app
|
|
// against the MCP server) and surfaces a user-visible reinstall message
|
|
// instead of the generic "try again later" debounce.
|
|
var ErrTenantNotInstalled = errors.New("mcp auto-onboard: tenant_not_installed")
|
|
|
|
// newMCPClient builds a client pointed at baseURL. The MCP server
|
|
// authenticates each auto-onboard call via the caller-supplied Bitrix
|
|
// access_token — no shared admin secret is required.
|
|
// baseURL MUST be the MCP server root (e.g. https://mcp.example.com) — we
|
|
// append /api/auto-onboard internally so channel config stays minimal.
|
|
func newMCPClient(baseURL string, timeout time.Duration) *mcpClient {
|
|
if timeout <= 0 {
|
|
timeout = 10 * time.Second
|
|
}
|
|
return &mcpClient{
|
|
httpClient: &http.Client{Timeout: timeout},
|
|
baseURL: strings.TrimRight(baseURL, "/"),
|
|
}
|
|
}
|
|
|
|
// autoOnboardRequest is the rev4 payload we POST to the MCP server.
|
|
//
|
|
// Required: Domain (tenant key on the MCP side), BitrixUserID (senderID),
|
|
// AccessToken + RefreshToken (forwarded from the Bitrix event so MCP can
|
|
// call Bitrix REST as that user). ExpiresIn is forwarded so MCP can
|
|
// compute a token_expires_at without its own clock drift surprising it.
|
|
// DisplayName is optional and lets MCP seed a friendly profile on insert.
|
|
type autoOnboardRequest struct {
|
|
Domain string `json:"domain"`
|
|
BitrixUserID string `json:"bitrix_user_id"`
|
|
AccessToken string `json:"access_token"`
|
|
RefreshToken string `json:"refresh_token"`
|
|
ExpiresIn int `json:"expires_in,omitempty"`
|
|
DisplayName string `json:"display_name,omitempty"`
|
|
}
|
|
|
|
// autoOnboardResponse is what we expect back from the MCP server (rev4).
|
|
//
|
|
// APIKey is the per-user MCP credential we cache in mcp_user_credentials
|
|
// (the goclaw_user_id key on our side is the Bitrix senderID — we do NOT
|
|
// use UserID from the response, that's the MCP internal user row id).
|
|
// UserID + TenantID are echoed for observability / debugging. Created
|
|
// distinguishes the fresh-insert path from the token-refresh path.
|
|
type autoOnboardResponse struct {
|
|
APIKey string `json:"api_key"`
|
|
UserID string `json:"user_id"`
|
|
TenantID string `json:"tenant_id"`
|
|
Created bool `json:"created"`
|
|
}
|
|
|
|
// autoOnboard POSTs to {baseURL}/api/auto-onboard and returns the resolved
|
|
// per-user api_key. Fails closed on: missing base URL, missing domain /
|
|
// bitrix_user_id / tokens, 4xx (config error), 5xx after one retry,
|
|
// malformed JSON, or empty api_key.
|
|
//
|
|
// On 404 with body {"error":"tenant_not_installed"} returns
|
|
// ErrTenantNotInstalled so the caller can render a specific "admin must
|
|
// reinstall the portal" reply instead of the generic failure debounce.
|
|
func (c *mcpClient) autoOnboard(ctx context.Context, req autoOnboardRequest) (*autoOnboardResponse, error) {
|
|
if c.baseURL == "" {
|
|
return nil, errors.New("mcp auto-onboard: base URL not configured")
|
|
}
|
|
if req.Domain == "" || req.BitrixUserID == "" || req.AccessToken == "" || req.RefreshToken == "" {
|
|
return nil, errors.New("mcp auto-onboard: domain, bitrix_user_id, access_token, refresh_token all required")
|
|
}
|
|
|
|
body, err := json.Marshal(req)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("mcp auto-onboard: marshal request: %w", err)
|
|
}
|
|
|
|
url := c.baseURL + "/api/auto-onboard"
|
|
|
|
// One retry on 5xx / transport error. The webhook handler is on a tight
|
|
// path (Bitrix24 expects a response in <30s and will retry on 5xx itself)
|
|
// so we cap total attempts at 2 with a short backoff.
|
|
var lastErr error
|
|
for attempt := 0; attempt < 2; attempt++ {
|
|
if attempt > 0 {
|
|
select {
|
|
case <-ctx.Done():
|
|
return nil, ctx.Err()
|
|
case <-time.After(250 * time.Millisecond):
|
|
}
|
|
}
|
|
|
|
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
|
|
if err != nil {
|
|
return nil, fmt.Errorf("mcp auto-onboard: new request: %w", err)
|
|
}
|
|
httpReq.Header.Set("Content-Type", "application/json")
|
|
|
|
resp, err := c.httpClient.Do(httpReq)
|
|
if err != nil {
|
|
lastErr = fmt.Errorf("mcp auto-onboard: http: %w", err)
|
|
continue
|
|
}
|
|
out, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<16)) // 64KiB cap
|
|
resp.Body.Close()
|
|
if readErr != nil {
|
|
lastErr = fmt.Errorf("mcp auto-onboard: read body: %w", readErr)
|
|
continue
|
|
}
|
|
|
|
switch {
|
|
case resp.StatusCode == http.StatusOK, resp.StatusCode == http.StatusCreated:
|
|
var ob autoOnboardResponse
|
|
if err := json.Unmarshal(out, &ob); err != nil {
|
|
return nil, fmt.Errorf("mcp auto-onboard: decode response (status %d): %w", resp.StatusCode, err)
|
|
}
|
|
if ob.APIKey == "" {
|
|
return nil, fmt.Errorf("mcp auto-onboard: incomplete response (status %d): missing api_key", resp.StatusCode)
|
|
}
|
|
return &ob, nil
|
|
case resp.StatusCode == http.StatusNotFound && isTenantNotInstalledBody(out):
|
|
// MCP server doesn't know this portal → operator must run the
|
|
// install flow on the MCP side before users can onboard.
|
|
return nil, ErrTenantNotInstalled
|
|
case resp.StatusCode >= 400 && resp.StatusCode < 500:
|
|
// Auth / config errors are non-retryable — surface the body so
|
|
// operators can see the domain / access_token mismatch.
|
|
return nil, fmt.Errorf("mcp auto-onboard: %d %s: %s", resp.StatusCode, http.StatusText(resp.StatusCode), truncateMCPBody(redactMCPBody(string(out)), 500))
|
|
default:
|
|
lastErr = fmt.Errorf("mcp auto-onboard: %d %s: %s", resp.StatusCode, http.StatusText(resp.StatusCode), truncateMCPBody(redactMCPBody(string(out)), 500))
|
|
// fall through to retry
|
|
}
|
|
}
|
|
|
|
if lastErr == nil {
|
|
lastErr = errors.New("mcp auto-onboard: unknown error")
|
|
}
|
|
return nil, lastErr
|
|
}
|
|
|
|
// isTenantNotInstalledBody reports whether a 404 body matches the MCP
|
|
// contract {"error":"tenant_not_installed", ...}. Parse failures return
|
|
// false so a generic 404 with a different body falls through to the
|
|
// normal 4xx error path (operator still sees the body in the log).
|
|
func isTenantNotInstalledBody(body []byte) bool {
|
|
var env struct {
|
|
Error string `json:"error"`
|
|
}
|
|
if err := json.Unmarshal(body, &env); err != nil {
|
|
return false
|
|
}
|
|
return env.Error == "tenant_not_installed"
|
|
}
|
|
|
|
// truncateMCPBody keeps error messages a sensible length so we don't log a
|
|
// multi-MB MCP 5xx body on every failed onboard. Named distinctly from the
|
|
// package-local `truncate` in client.go (imbot payload truncation) so the
|
|
// two don't collide on a rename/refactor.
|
|
func truncateMCPBody(s string, n int) string {
|
|
if len(s) <= n {
|
|
return s
|
|
}
|
|
return s[:n] + "…"
|
|
}
|
|
|
|
// mcpTokenRedactRe scrubs OAuth secrets from an MCP response body. The POST we
|
|
// send carries access_token / refresh_token (the Bitrix→MCP onboarding bridge);
|
|
// a naive MCP server that echoes the request back in its 4xx/5xx error body
|
|
// would otherwise leak those tokens into goclaw logs, breaking the package's
|
|
// "log token lengths, never values" discipline.
|
|
var mcpTokenRedactRe = regexp.MustCompile(`(?i)("(?:access_token|refresh_token|client_secret)"\s*:\s*")[^"]*(")`)
|
|
|
|
// redactMCPBody replaces OAuth secret values in an MCP body with a placeholder
|
|
// before the body is interpolated into an error string / log line.
|
|
func redactMCPBody(s string) string {
|
|
return mcpTokenRedactRe.ReplaceAllString(s, `${1}[redacted]${2}`)
|
|
}
|