feat(bitrix24): imbot.v2 migration, 2-way media, openline sender-tag echo, and hardening (#1236)

* refactor(bitrix24): rename "Path B" framing to maintainer-specified naming [B24:2794]

Per maintainer hard rule #10 (no generic "Path A/B" framing) from PR #1061
review. The Bitrix24 MCP auto-onboard flow is Bitrix-specific glue
("Bitrix24 OAuth -> existing mcp_user_credentials bridge"), NOT a generic
MCP architecture pattern.

Naming convention applied consistently:
- First mention per file: full "Bitrix24 OAuth -> existing
  mcp_user_credentials bridge" (matches maintainer comment verbatim).
- Subsequent mentions in same file: shortened "mcp_user_credentials bridge".
- Test/log context referencing literal endpoint /api/auto-onboard: keep
  "auto-onboard" reference (it's the actual API endpoint name).

Changes are documentation-only:
- Rename in code comments + test descriptions + plan docs.
- Clarify framing in mcp_client.go + provisioner.go doc comments to
  emphasize Bitrix-specific glue (not generic MCP infra).
- Reuse existing mcp_user_credentials table + MCPServerStore methods
  (no schema / store / abstraction change).

Files:
- cmd/gateway.go (factory registration doc)
- internal/channels/bitrix24/{channel,factory,mcp_client,provisioner}.go
- internal/channels/bitrix24/{mcp_client,provisioner}_test.go
- plan/goclaw-mcp-integration.md (21 occurrences)

Verified: go build + MCP-related tests pass (TestProvision*,
TestInitMCPProvisioner*, TestMCPClient*).

Phase 1 of Path C execution per
plans/reports/decision-log-260519-1555-bitrix24-pr-fork-decision.md.

* fix: confine outbound media paths to agent workspace [B24:2794]

Tool MEDIA:<path> output reached channel file-upload sinks (Bitrix
imbot.v2.File.upload, Telegram sendDocument, etc.) verbatim via
parseMediaResult, with no workspace-boundary check. A malicious or buggy
tool emitting MEDIA:/etc/passwd could exfiltrate arbitrary files to chat.

Extract the EvalSymlinks+Rel containment from extractMediaFromContent into
a shared confineToWorkspace helper and apply it at the parseMediaResult
sink in processToolResult. Fixing at the source/egress boundary protects
every channel at once rather than per-channel. Paths that escape the
workspace are dropped and logged (security.media_path_rejected).

Add TestConfineToWorkspace (boundary unit) and
TestParseMediaResultConfinedToWorkspace (sink regression for H2).

* feat(bitrix24): support inbound + outbound media via imbot.v2 File API [B24:2794]

Bitrix24 channel was text-only; attachments were parsed but dropped.
- Inbound: download chat files via imbot.v2.File.download (one-time URL),
  forward to the agent with MIME preserved (internal/channels/bitrix24/download.go).
- Outbound: upload agent media to the chat via imbot.v2.File.upload
  (internal/channels/bitrix24/send_media.go).
- Add BaseChannel.HandleMessageMedia to preserve MIME/filename through the bus.
- Per-channel media_max_mb cap (default 20) applies to both directions.

Tests: 92 pass (internal/channels/bitrix24 + internal/channels), go vet clean (PG + sqliteonly).

* refactor(bitrix24): migrate messaging/bot-list/unregister to imbot v2 API [B24:2794]

Move outbound REST calls to the imbot v2 family (keeps register on v1):
- imbot.message.add -> imbot.v2.Chat.Message.send (fields.message shape, live-verified)
- imbot.bot.list (+ legacy imbot.list fallback) -> imbot.v2.Bot.list; add botListRows
  to normalize the v2 {bots:[...]} envelope, legacy array, and id-keyed map forms
- imbot.unregister -> imbot.v2.Bot.unregister

Bot registration stays on v1 imbot.register: v2 imbot.v2.Bot.register changes the
event-delivery model (per-event handler URLs -> eventMode), which would require
rewriting the inbound event parser. No user-facing behavior change.

Tests: bitrix24 package green; go vet ./... clean.

* feat(bitrix24): route whisper via v1 SKIP_CONNECTOR + add v2 replyId [B24:2794]

Bot was leaking HiddenMessage (whisper) replies to the external Zalo
connector because every outbound call went through imbot.v2.Chat.Message.send,
which has no equivalent of the v1 SKIP_CONNECTOR flag. Branch the outbound
path on inbound visibility:

  whisper → imbot.message.add + SKIP_CONNECTOR=Y  (v1, send_v1.go)
  public  → imbot.v2.Chat.Message.send + fields.replyId  (v2, send_v2.go)

Pipeline:
  events.go        parse data[PARAMS][PARAMS][COMPONENT_ID]=HiddenMessage
                   into EventParams.IsHiddenMessage (form + JSON variants)
  handle.go        set bitrix_visibility on InboundMessage.Metadata
  consumer         forward visibility + message_id into OutboundMessage
  send.go          resolveSendOptions + sendChunk dispatcher +
                   shared callWithRateLimitRetry helper
  metadata_keys.go single source of truth for the keys + values

Defaults preserve pre-refactor behaviour: callers that don't populate
bitrix_visibility still go through v2 public, and replyId is omitted
unless a numeric bitrix_message_id arrives in metadata.

Tests:
  TestParseEvent_FormURLEncoded_IsHiddenMessage  (3 cases)
  TestParseEvent_JSON_IsHiddenMessage             (3 cases)
  TestResolveSendOptions                          (8 cases)
  TestSend_BranchesOnVisibility                   (4 cases)

* feat(bitrix24): openline sender-tag echo on replies [B24:2794]

Openline sender-tag echo (this change):
- Capture the connector sender tag ("[name #id]:" or "[name] #id:") from
  inbound openline group messages, strip it from the body the agent sees,
  and re-prepend the canonical "[name] #id:" form to the reply so the Open
  Channel connector routes the answer back to the right external user.
- New sender_prefix.go helper (+ test) accepts both inbound layouts and
  emits one canonical form; scoped to messages carrying the tag, so plain
  chats are unaffected.
- metadata_keys.go: MetaKeySenderPrefix; handle.go capture/strip/stash;
  gateway_consumer_normal.go forwards the key; send.go prepends it on the
  first chunk before chunking.

Bundled bitrix24 channel-core work already on this branch:
- handle.go: @mention is the sole trigger for both staff and connector
  customers; unmentioned traffic is dropped (was: drop all connector msgs).
- isGroupMessageType: treat SONET_GROUP "B" as a group.
- handle_test.go, mcp_client_test.go: cover the above.

* feat(bitrix24): accept colon-less openline sender tag, echo [name] #id [B24:2794]

The Open Channel connector dropped the trailing colon from its sender tag:
inbound now arrives as "[Name] #id <msg>" (was "[Name] #id: <msg>"). The
id-bearing patterns required the colon, so the tag fell through to the
name-only branch and the reply echoed "[Name]" — dropping the #id the
connector needs to route the answer back.

- sender_prefix.go: make the trailing ":" optional on both id layouts
  ([name #id] / [name] #id, with or without colon) and echo the canonical
  "[name] #id" (no colon) to match the connector's current format. Bare
  "[name]" (no id) still echoes "[name]" for Open Channel only.
- handle.go: gate the bare name-only layout to Open Channel (isOpenChannel)
  so ordinary group chats starting with "[x] ..." are left untouched.
- sender_prefix_test.go: cover colon/no-colon x id-inside/id-outside, the
  name-only openline case, and the non-openline no-op.

* fix: security and robustness fixes from the bitrix24 channel review [B24:2794]

- download.go: block redirect-based SSRF on inbound media. CheckRedirect
  re-validates each hop (http(s) only, reject private/loopback/link-local
  hosts, cap hops); the initial portal-domain pin is no longer bypassable
  via a 3xx to an internal service. Public-host redirects still allowed.
- handle.go: extract/echo the openline sender tag only for Open Channel
  sessions (was: any group chat), removing bogus prefixes in CRM group
  chats and narrowing the forged-tag misroute surface.
- loop_tools.go + loop_media.go: confine result.Media to the agent / team /
  tenant-allowed roots (new confineToAnyRoot) before a channel uploads it,
  so a prompt-injected out-of-workspace path (e.g. /etc/passwd) cannot
  exfiltrate, while legitimate cross-workspace media (team files, delegatee
  output) still flows.
- send_media.go: bounded outbound read via io.LimitReader replaces the
  os.Stat + os.ReadFile pair, closing the TOCTOU size-cap bypass; cap a
  single message's outbound attachments at 10 (mirrors inbound).
- register.go: paginate imbot.v2.Bot.list (limit/offset + hasNextPage,
  capped at 40 pages) so verify/lookup see bots past the first 50.
- mcp_client.go: redact access_token / refresh_token / client_secret from an
  echoed MCP error body before it is logged or returned (+ test).

* fix(security): validate resolved dial IP on Bitrix media redirects [B24:2794]

The inbound media download redirect guard only string-checked the redirect
hostname (isPrivateOrLoopback on req.URL.Hostname()), so a redirect to a public
hostname that resolves to 127.0.0.1 / 169.254.169.254 / an RFC1918 address — or a
DNS-rebinding swap between check and dial — still passed the guard and the client
would connect. Reported in PR review.

Add security.NewRedirectFollowingSafeClient: it follows redirects but validates
the RESOLVED destination IP of every hop at dial time via net.Dialer.Control,
reusing the existing blocked-CIDR list. The IP it checks is the IP actually
dialed, so both redirect-to-internal and DNS rebinding are refused, while
legitimate public CDN redirects still succeed. download.go now uses it instead of
the hostname-string guard.

Tests: deterministic dial-control table (loopback / link-local / private /
multicast / unspecified / public, v4 + v6), malformed/non-IP addr, test bypass,
loopback-dial-blocked client wiring, and redirect cap + scheme checks.

* feat(bitrix24): per-participant Zalo openline identity from 3-token sender tag [B24:2794]

Parse the connector's "[Name] #uid #msgId" sender tag so each external
customer in a shared Open Channel group gets its own contact + USER.md
instead of collapsing onto the connector proxy id. Identity minting is
gated on IS_CONNECTOR=Y to reject operator forged tags. Echo back the
msgId only ("#msgId") on replies; keep the legacy single-number and
name-only layouts unchanged. Zero DB migration.

- sender_prefix.go: parseOpenlineSenderTag() classifies 3-token / legacy / name-only
- handle.go: synthetic senderID "openlines:{instance}:{chat}:{uid}" + participant_user_id metadata, gated on FromIsConnector
- gateway_consumer_normal.go: deriveGroupUserID() routes participant -> per-person scope, group fallback otherwise
- send.go: buildAddressMention numeric-id guard so synthetic ids don't emit invalid [USER=...] BBCode
- MetaKeyMessageID kept as Bitrix MESSAGE_ID (drives v2 fields.replyId); connector msgId surfaced only via echo prefix

---------

Co-authored-by: DangTinh311 <dangtinh31193@gmail.com>
Co-authored-by: Chinh Dang <chinhdang@192.168.68.104>

cmd/gateway.go

@@ -604,8 +604,10 @@ func runGateway() {
 		// lazy per-user credential provisioning (via mcp_server_name +
 		// mcp_base_url in their instance config) can reach the partner's
 		// MCPServerStore. The MCP server authenticates each onboard call
-		// via the caller-supplied Bitrix access_token (Path B) — no shared
-		// admin secret is required. Channels with none of those set operate
+		// via the caller-supplied Bitrix access_token (the "Bitrix24
+		// OAuth → existing mcp_user_credentials bridge" — Bitrix-specific
+		// glue, not a generic MCP architecture pattern) — no shared admin
+		// secret is required. Channels with none of those set operate
 		// identically to before — the MCPStore arg is nil-safe inside the
 		// factory.
 		instanceLoader.RegisterFactory(channels.TypeBitrix24, bitrix24.FactoryWithPortalStoreAndMCP(pgStores.BitrixPortals, pgStores.MCP, bitrixEncKey))

cmd/gateway_consumer_normal.go

@@ -13,6 +13,7 @@ import (
 	"github.com/nextlevelbuilder/goclaw/internal/agent"
 	"github.com/nextlevelbuilder/goclaw/internal/bus"
 	"github.com/nextlevelbuilder/goclaw/internal/channels"
+	"github.com/nextlevelbuilder/goclaw/internal/channels/bitrix24"
 	"github.com/nextlevelbuilder/goclaw/internal/channels/telegram/voiceguard"
 	"github.com/nextlevelbuilder/goclaw/internal/i18n"
 	"github.com/nextlevelbuilder/goclaw/internal/scheduler"
@@ -83,21 +84,10 @@ func processNormalMessage(
 	}
 
 	// Group-scoped UserID: context files, memory, traces, and seeding scope.
-	// - Discord guilds: "guild:{guildID}:user:{senderID}" — per-user per-server,
-	//   shared across all channels within the same server. Session key stays per-channel.
-	// - Other platforms: "group:{channel}:{chatID}" — shared by all users in the chat.
-	// Individual senderID is preserved in InboundMessage for pairing/dedup/mention gate.
-	userID := msg.UserID
-	if peerKind == string(sessions.PeerGroup) && msg.ChatID != "" {
-		if guildID := msg.Metadata["guild_id"]; guildID != "" && msg.SenderID != "" {
-			// Discord guild: per-user scope so each member has own profile
-			// across all channels in the same server.
-			userID = fmt.Sprintf("guild:%s:user:%s", guildID, msg.SenderID)
-		} else {
-			groupID := msg.ChatID
-			userID = fmt.Sprintf("group:%s:%s", msg.Channel, groupID)
-		}
-	}
+	// See deriveGroupUserID for the precedence (Discord guild → openline
+	// participant → group fallback). Individual senderID is preserved in
+	// InboundMessage for pairing/dedup/mention gating regardless of scope.
+	userID := deriveGroupUserID(msg, peerKind)
 
 	// Persist friendly names from channel metadata into session + user profile.
 	sessionMeta := extractSessionMetadata(msg, peerKind)
@@ -234,6 +224,22 @@ func processNormalMessage(
 		}
 	}
 
+	// Forward Bitrix24-specific routing keys so Send() can:
+	//   1. Branch v2 public vs v1 whisper (bitrix_visibility)
+	//   2. Set fields.replyId on v2 public reply (bitrix_message_id)
+	// CopyFinalRoutingMeta is channel-agnostic and doesn't include these.
+	if v := msg.Metadata[bitrix24.MetaKeyVisibility]; v != "" {
+		outMeta[bitrix24.MetaKeyVisibility] = v
+	}
+	if v := msg.Metadata[bitrix24.MetaKeyMessageID]; v != "" {
+		outMeta[bitrix24.MetaKeyMessageID] = v
+	}
+	// Openline sender tag captured on inbound → Send() prepends it to the reply
+	// so the connector routes the answer back to the right external user.
+	if v := msg.Metadata[bitrix24.MetaKeySenderPrefix]; v != "" {
+		outMeta[bitrix24.MetaKeySenderPrefix] = v
+	}
+
 	// Register run with channel manager for streaming/reaction event forwarding.
 	// Use localKey (composite key with topic suffix) so streaming/reaction events
 	// route to the correct per-topic state in the channel.
@@ -698,3 +704,32 @@ func isSafeBitrixEntityToken(s string, maxLen int) bool {
 	}
 	return true
 }
+
+// deriveGroupUserID computes the per-message scope userID used for context
+// files, memory, traces, and seeding. Direct messages keep msg.UserID. Group
+// messages pick a synthetic scope, in precedence order:
+//
+//  1. Discord guild member: "guild:{guildID}:user:{senderID}" — per-user across
+//     every channel in the same server.
+//  2. Openline participant: the per-participant id minted by bitrix24/handle.go
+//     ("openlines:{instance}:{chat}:{uid}") when a connector relayed a customer
+//     message carrying a stable uid — so each external person gets their own
+//     USER.md / memory instead of collapsing into the shared connector proxy.
+//     Absent for legacy/name-only/operator messages → falls through.
+//  3. Group fallback: "group:{channel}:{chatID}" — shared by everyone in the chat.
+//
+// The individual senderID stays on InboundMessage for pairing / dedup / mention
+// gating regardless of which scope is chosen.
+func deriveGroupUserID(msg bus.InboundMessage, peerKind string) string {
+	if peerKind != string(sessions.PeerGroup) || msg.ChatID == "" {
+		return msg.UserID
+	}
+	switch {
+	case msg.Metadata["guild_id"] != "" && msg.SenderID != "":
+		return fmt.Sprintf("guild:%s:user:%s", msg.Metadata["guild_id"], msg.SenderID)
+	case msg.Metadata[bitrix24.MetaKeyParticipantUserID] != "":
+		return msg.Metadata[bitrix24.MetaKeyParticipantUserID]
+	default:
+		return fmt.Sprintf("group:%s:%s", msg.Channel, msg.ChatID)
+	}
+}

cmd/gateway_consumer_normal_test.go

@@ -5,6 +5,8 @@ import (
 	"testing"
 
 	"github.com/nextlevelbuilder/goclaw/internal/bus"
+	"github.com/nextlevelbuilder/goclaw/internal/channels/bitrix24"
+	"github.com/nextlevelbuilder/goclaw/internal/sessions"
 )
 
 // TestIsSafeBitrixEntityToken pins the validation contract for webhook-sourced
@@ -44,6 +46,90 @@ func TestIsSafeBitrixEntityToken(t *testing.T) {
 	}
 }
 
+// TestDeriveGroupUserID pins the group-scope userID precedence: Discord guild
+// member → openline participant → group fallback, with direct messages passing
+// msg.UserID through untouched. The openline participant branch is what gives
+// each Zalo customer its own per-person scope instead of the shared connector
+// proxy.
+func TestDeriveGroupUserID(t *testing.T) {
+	const group = string(sessions.PeerGroup)
+	const direct = string(sessions.PeerDirect)
+	cases := []struct {
+		name     string
+		msg      bus.InboundMessage
+		peerKind string
+		want     string
+	}{
+		{
+			name: "openline participant overrides group fallback",
+			msg: bus.InboundMessage{
+				Channel:  "zalo_ol",
+				ChatID:   "chat4878",
+				SenderID: "openlines:tamgiac:chat4878:111222",
+				UserID:   "openlines:tamgiac:chat4878:111222",
+				Metadata: map[string]string{
+					bitrix24.MetaKeyParticipantUserID: "openlines:tamgiac:chat4878:111222",
+				},
+			},
+			peerKind: group,
+			want:     "openlines:tamgiac:chat4878:111222",
+		},
+		{
+			name: "no participant id falls back to group-level",
+			msg: bus.InboundMessage{
+				Channel: "zalo_ol",
+				ChatID:  "chat4878",
+				UserID:  "960",
+			},
+			peerKind: group,
+			want:     "group:zalo_ol:chat4878",
+		},
+		{
+			name: "empty participant id (parse-fail degrade) falls back to group-level",
+			msg: bus.InboundMessage{
+				Channel:  "zalo_ol",
+				ChatID:   "chat4878",
+				UserID:   "960",
+				Metadata: map[string]string{bitrix24.MetaKeyParticipantUserID: ""},
+			},
+			peerKind: group,
+			want:     "group:zalo_ol:chat4878",
+		},
+		{
+			name: "discord guild takes precedence over participant id",
+			msg: bus.InboundMessage{
+				Channel:  "discord",
+				ChatID:   "chan-1",
+				SenderID: "u-9",
+				Metadata: map[string]string{
+					"guild_id":                        "g-1",
+					bitrix24.MetaKeyParticipantUserID: "openlines:x:y:z",
+				},
+			},
+			peerKind: group,
+			want:     "guild:g-1:user:u-9",
+		},
+		{
+			name: "direct message passes UserID through untouched",
+			msg: bus.InboundMessage{
+				Channel:  "zalo_ol",
+				ChatID:   "chat4878",
+				UserID:   "42",
+				Metadata: map[string]string{bitrix24.MetaKeyParticipantUserID: "openlines:x:y:z"},
+			},
+			peerKind: direct,
+			want:     "42",
+		},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := deriveGroupUserID(tc.msg, tc.peerKind); got != tc.want {
+				t.Errorf("deriveGroupUserID() = %q; want %q", got, tc.want)
+			}
+		})
+	}
+}
+
 func TestResolveSenderNameReadsWhatsAppUserName(t *testing.T) {
 	got := resolveSenderName(bus.InboundMessage{
 		Metadata: map[string]string{