WhatsApp LID-format chat IDs contain @ (e.g. 551152861098:5@s.whatsapp.net)
which is invalid in Docker container names, causing sandbox creation to fail
for any WhatsApp-triggered agent session.
Add @ to the sanitizeKey replacer alongside the existing : / . and space
characters. Adds a test case with a realistic WhatsApp LID key.
Fixes#1029
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- 백엔드: LocaleKO 상수, IsSupported에 ko 추가, catalog_ko.go 185개 문자열
- 웹 UI: 38개 네임스페이스 JSON 파일 (locales/ko/)
- constants.ts SUPPORTED_LANGUAGES에 ko 추가
- index.ts에 ko 리소스 등록 및 getInitialLanguage에 ko 감지
- en/topbar.json languages에 한국어 옵션 추가
- webui handler: index.html에 no-cache 헤더 추가하여 배포 후 캐시 문제 방지
Both WS pairing (router.go Path 3a) and HTTP X-GoClaw-Sender-Id auth
were pinned to RoleOperator. Now they map tenant_users.role ->
permissions.Role (owner/admin/operator|member/viewer) so a tenant admin
who logs in via pairing actually gets admin in the dashboard. Falls
back to RoleOperator when the user has no membership row (backward
compatible). Tenant is inferred from a single membership when no hint
is provided.
Signed-off-by: imcvampire <anhnq0@gmail.com>
Mirror the existing /addwriter /removewriter /writers UX for the cron
permission type so group admins can grant cron-management access without
elevating to full file_writer. Bootstrap policy: first call in a group
is free (no cron managers yet); subsequent calls require an existing
cron manager OR a file_writer.
- new commands_cron_perm.go: handleCronPermCommand (add/remove),
handleListCronPerm; uses store.ConfigTypeCron with scope
"group:<channel>:<chatID>".
- commands.go: route /addcron, /removecron, /croners.
- commands_pairing.go: register the 3 commands in the bot menu.
* fix(skills): normalize backslash in ZIP entry names for Windows compatibility
ZIP archives created on Windows may use backslash (\) as path separator
instead of forward slash (/). When extracted on Linux, this causes files
like 'scripts\search.py' to be created as a single flat file instead of
the proper 'scripts/search.py' directory structure.
The ZIP specification (PKZIP APPNOTE) requires forward slashes, but many
Windows tools produce non-conforming archives.
This adds a strings.ReplaceAll backslash normalization step during skill
ZIP extraction to handle this edge case.
* test(skills): cover backslash zip extraction
---------
Co-authored-by: buidackim <221586880+buidackim@users.noreply.github.com>
Co-authored-by: Mỹ Ngân <goclaw-maintain@users.noreply.github.com>
* docs(openapi): add vault endpoints + schemas
The committed OpenAPI spec only documented 41 of ~286 actual routes. This
adds the 19 Vault endpoints — 16 under /v1/vault/* and 3 under
/v1/agents/{agentID}/vault/* — plus their request/response schemas.
Routes covered (matches internal/http/vault_handler_*.go and
vault_graph_handler.go registrations):
- documents: list, create, get, update, delete (per-agent and cross-agent)
- documents/{docID}/links: outlinks + backlinks + name lookup
- links: create, delete, batch get outlinks
- upload: multipart, allowed exts whitelist
- rescan: per-tenant workspace rescan w/ concurrency guard (409)
- tree: lazy sidebar listing
- search: hybrid FTS + vector
- enrichment: status + stop
- graph: lightweight nodes/edges for visualization
Schemas: VaultDocument, VaultDocumentInput, VaultDocumentUpdateInput,
VaultLink, VaultBacklink, VaultSearchResult, VaultTreeEntry,
VaultGraphNode, VaultGraphEdge. Tags: Vault, Vault Graph.
Pure spec addition — no handler changes. The insertion is surgical
(no reformatting of existing entries), so the diff is additive only.
* docs(openapi): cover the remaining HTTP endpoints
Continues the gap-fill started for Vault. Adds 150 more path operations
spanning every other HTTP-route file under internal/http/, with 19 new
component schemas and 8 new tags. After this commit the spec covers 210
paths vs 60 previously committed (the actual handler surface is ~258).
Coverage by area (file → routes added):
knowledge_graph_handlers.go 9 (Knowledge Graph: extract, graph, stats,
dedup scan/dismiss, merge, single entity)
vault_graph_handler.go 1 (kg/graph/compact)
memory_handlers.go 6 (documents CRUD with {path...}, chunks,
index, index-all)
oauth.go 8 (chatgpt + openai: status, start,
callback, logout × 2)
storage.go 4 (files {path...} read/delete, size, move;
POST upload merged into the existing
/v1/storage/files block so the path key
stays unique and YAML-strict parsers like
Swagger UI's loader accept the spec)
agents.go 22 (sync-workspace, shares CRUD, regenerate /
resummon / cancel-summon, system-prompt-
preview, export {preview,download/token},
import {preview,merge}, teams export/import,
instances + metadata, orchestration,
v3-flags)
evolution_handlers.go 3 (metrics, suggestions, suggestion review)
episodic_handlers.go 2 (list, search)
skills.go 18 (versions, files, files/{path...}, upload,
grants agent/user × revoke, install-deps
trio, runtimes, toggle, tenant-config,
export {preview}, import)
mcp.go + mcp_user_credentials.go 16 (test, reconnect, grants + per-user creds,
requests CRUD + review, export {preview},
import, agent-grants list)
providers.go 4 (verify-embedding, codex-pool-activity,
claude-cli auth-status, embedding/status)
tenants.go + users 7 (CRUD + tenant users + tenant-users list,
users/search)
secure_cli.go + agent_grants 17 (cli-credentials full CRUD + presets +
check-binary + test + user-credentials +
agent-grants)
system_configs.go 4 (list, get, put, delete)
backup_handler.go + s3 + tenant 12 (system backup preflight/download/s3 set,
tenant backup + restore)
packages.go 6 (list, install/uninstall, runtimes,
github-releases, shell-deny-groups)
builtin_tools.go 3 (tenant-config get/set/delete)
tts.go + voices.go + caps 6 (config, synthesize, test-connection,
capabilities, voices, voices/refresh)
channel_instances.go +contacts +pending 10 (channel writers, writer groups,
contacts resolve / merge / unmerge / merged,
pending-messages)
files.go + workspace_upload 3 (files {path...} read, files/sign,
team workspace upload + move, team events,
team task attachment download)
edition.go + openapi.go 2 (edition info, openapi.json)
api-keys + traces + usage 3 (revoke, traces/{traceID}/export,
usage/breakdown)
New schemas: KGEntity, KGRelation, KGDedupCandidate, KGStats, TraversalResult,
MemoryDocument, MemoryChunk, MemorySearchHit, OAuthStatus, OAuthStartResponse,
Skill, McpServer, Tenant, TenantUser, CLICredential, Package, EvolutionMetric,
EvolutionSuggestion, EpisodicSummary.
New tags: Tenants, CLI Credentials, Packages, Contacts, Teams, TTS, Evolution,
Episodic Memory.
Purely additive — no existing operation, schema, or tag is modified. The one
overlap (`/v1/storage/files`, previously documented as GET-only) is handled
by inserting the new POST method inside the existing path block in place
rather than re-declaring the key, so each path key appears exactly once and
the spec parses cleanly under both JSON and strict YAML loaders.
Verified by parsing HEAD and the new file with python json.load + yaml.safe_load
and diffing entry-by-entry. Under `git diff --histogram`/`--patience` the diff
shows zero deletions; the default myers algorithm reports cosmetic deletion
lines but they are diff-algorithm noise, not real changes.
Where a body field or response shape was clear from the handler, the spec
captures it; where the handler accepts free-form / additive JSON payloads
the request body is documented as a generic object (additionalProperties:
true) rather than a fabricated rigid schema. Better to be honest about
flexibility than to lock callers into a wrong type.
* docs(openapi): drop nonexistent DELETE /v1/api-keys/{id}
The spec carried a DELETE /v1/api-keys/{id} operation, but
internal/http/api_keys.go registers no such route — only
GET /v1/api-keys, POST /v1/api-keys, and
POST /v1/api-keys/{id}/revoke. The DELETE entry predates this PR
(it was already on dev), so the semantic-preservation check left
it untouched; this removes the stale op so the spec matches the
registered routes. The correct POST /v1/api-keys/{id}/revoke
operation is unchanged.
Verified: json.load parses, go build ./internal/http/ and
go vet ./internal/http/ both clean.
---------
Co-authored-by: Aaron Vu <aaron-tsar@users.noreply.github.com>
* fix(pipeline): defer non-tool messages until all tool results are emitted
Multi-tool responses could interleave synthetic user messages (loop
warnings, nudges) between tool results, breaking OpenAI-compatible
providers that require contiguous tool_result grouping.
Fix both sequential and parallel tool execution paths in tool_stage.go
to defer non-tool messages until after all tool results are appended.
Fix sanitizeHistory lookahead in loop_history_sanitize.go to reorder
interleaved non-tool messages in persisted session history.
Closes#1177
* perf(agent): cache BuildFilteredTools result per run
All filtering inputs (policy, disabled tools, bootstrap, channel,
orchestration mode, user MCP tools) are fixed for the lifetime of a
run. Only the final iteration differs (strips all tools). Cache the
tool definitions after the first call and reuse for iterations
0..maxIter-1, eliminating N-1 redundant 7-step policy evaluations per
agent run.
Currently the web_search tool resolves a tenant-configured provider
chain (e.g. tavily → exa → brave) and stops at the first success.
That's the right default for cost/latency, but it makes cross-engine
corroboration impossible at the agent level: the caller has no way to
ask "search the same query on Exa specifically" once Tavily has
already returned a hit.
Add an optional `provider` argument. When set, the chain is narrowed
to the named provider (case-insensitive); other providers are not
queried. The first-success-wins fallback is preserved when the arg
is omitted.
Use case (concrete): a research agent verifying source freshness
calls web_search twice for the same query — once with provider="tavily"
and once with provider="exa" — and compares the URLs returned. URLs
that appear in both engines are high-confidence original primary
sources; URLs only in one signal a republish or low-circulation outlet
worth verifying further.
Other changes:
- Cache key now includes the requested provider so per-engine
results don't collide. Without this, the second call would just
replay the first engine's cached result, defeating the purpose.
- Unknown provider returns a clear error listing what IS configured
for the tenant — better DX than a silent no-op.
Tests: 5 new cases covering narrowing, case-insensitivity, unknown
provider, default first-success behaviour preservation, and cache
isolation across providers.
* feat(cron): deterministic command payloads (run a shell command, no LLM)
Cron jobs always run an agent turn today, so deterministic work (health
probes, backups, syncs) pays model tokens on every fire. This adds a
"command" payload kind that runs a shell command directly in the gateway
process with zero model tokens, mirroring openclaw's command cron.
- store: CronPayload.Command (*CronCommandSpec — argv/cwd/env/input/
timeouts/output cap). Persists in the existing payload JSON blob, so
there is NO migration and no schema version bump.
- internal/cronexec: in-process runner with wall-clock + no-output
timeouts, per-stream output capping, and process-group termination so a
timed-out command's forked children are also killed.
- gateway_cron handler: command jobs run in-process and deliver stdout on
success (honoring the NO_REPLY sentinel). A non-zero exit / timeout
returns an error so the run is recorded as error and retried per
cron.max_retries; failures are NOT delivered, mirroring the agent path
(only successful output is announced — no channel spam).
- surfaces: cron.create RPC, the agent `cron` tool, and a new
`goclaw cron create` CLI all accept command payloads.
- security: gated by cron.command_enabled (default false). Commands run
with the gateway process's privileges, so the feature is opt-in per
gateway; when disabled the RPC and tool reject command payloads and the
handler refuses to run them.
- i18n (en/vi/zh), docs (08-scheduling-cron.md), and tests for the runner
and the handler command path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cron): gate command payloads on the update surfaces too
handleUpdate (RPC + agent tool) passed CronJobPatch.Command straight to
UpdateJob, which switches the payload to command kind for any non-nil
Command — without the command_enabled gate or ValidateCronCommandSpec that
create enforces. A normal job could therefore be mutated into a command job
(or persisted with an invalid spec, e.g. empty argv) on a gateway where
command cron is disabled, breaking the disabled-gateway contract.
Both update surfaces now require cron.command_enabled and validate the spec
before UpdateJob, matching create. The agent tool parses the command via the
same path as add and drops the raw keys so a shell-string command can't break
the generic patch unmarshal. Regression tests added for RPC and tool update
(command disabled + invalid argv), plus a positive enabled-valid case.
Addresses review feedback from @mrgoonie on #1279.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pairing requests and message handlers were silently using MasterTenantID
instead of the correct tenant because context.Context had no tenant set.
Root cause: instance_loader.go passed bare ctx (no tenant) to
startChannelWithTimeout; channel handlers also missing store.WithTenantID
at entry points.
Channels fixed:
- instance_loader.go: pass instCtx (with tenant) to startChannelWithTimeout
- Telegram, Bitrix24, Feishu: add store.WithTenantID at handler entry
- Facebook: add ctx param to handleMessagingEvent/handleCommentEvent + webhook_handler
- Pancake: add ctx param to handleMessagingEvent/handleCommentEvent + webhook_handler
Also fix consolidation pruner: PruneExpired SQL had tenant_id filter that
caused zero rows to ever be deleted (context had no tenant). Remove tenant
filter — pruning is a global maintenance operation.
Add regression tests:
- internal/channels/pancake/tenant_context_test.go (6 tests)
- internal/channels/facebook/tenant_context_test.go (4 tests)
- internal/consolidation/prune_expired_test.go (5 tests)
Co-authored-by: Bruno Clermont <bruno.clermont@gmail.com>
The async webhook worker double-processed long-running agent calls: the
90s stale-running sweep reclaimed rows whose agent ran longer than 90s
(no heartbeat), re-running the agent and duplicating MCP side-effects.
Add a last_heartbeat_at column + a Heartbeat lease-renewal method (CAS on
lease_token). While an agent runs, the worker renews the lease every 30s;
ReclaimStale now keys off last_heartbeat_at instead of started_at, so a
live run is never reclaimed while a dead worker is still recovered within
the stale window. If a run loses its lease (reclaimed), the heartbeat
cancels the run context so the agent stops immediately and writes no
further side-effects.
- migrations/000085 (PG) + SQLite schema v52 + RequiredSchemaVersion 85
- WebhookCallStore.Heartbeat (PG + SQLite impls); ClaimNext/ReclaimStale
switched to last_heartbeat_at
- worker heartbeat goroutine + cancel-on-lease-loss; invokeAgent honors ctx
- store + worker tests
setupToolRegistry wires the filesystem tools' AllowPaths from the config/JSON5
default before ApplySystemConfigs overlays system_configs['allowed_paths'], so
DB-configured allowed paths never reached read_file / list_files / write_file /
edit / send_file. Only the rate limiter was re-applied after the overlay (#1111);
the AllowPaths analogue was missing, so agents were denied access to configured
shared directories outside their workspace even though the DB value was present
(visible as a read_file "access denied" log whose allowedPrefixes omit the
configured path).
Extract the user-allowed-path application into applyUserAllowedPaths and re-run it
from runGateway after the overlay, mirroring the rate-limiter re-apply. The helper
is idempotent (AllowPaths is additive and the prefix check is membership-based),
so the initial wiring call plus the re-apply is safe.
Test: cmd/gateway_tools_wiring_test.go asserts a path outside the workspace is
denied before the grant and readable after, that an unrelated path stays denied,
that repeated application is safe, and that an empty list is a no-op.
Add RFC 2544 benchmarking range (198.18.0.0/15) and reserved range
(240.0.0.0/4) to both SSRF blocklists (security package + web_fetch
tool) to prevent internal network access via these special-use IPs.
Closes#1218
A run records its terminal run.status (completed/failed/cancelled) only
when the agent loop returns. If the gateway process is killed mid-run
(restart, crash, or SIGKILL after a slow graceful shutdown), the terminal
event is never emitted, so the run's last run.status item stays "started"
forever — it shows as perpetually running in the timeline and is never
counted as failed.
Mirror the cron scheduler's startup reset (recomputeStaleJobs flips stale
'running' jobs to 'interrupted'): add RunTimelineStore.RecoverInterruptedRuns,
invoked once on gateway startup. It finds runs that have a started run.status
but no terminal sibling and appends a terminal failed run.status item, marked
interrupted in metadata so it stays distinguishable from a genuine agent
failure. Running only at startup means nothing from before is still
executing, so there is no false-positive risk — unlike a periodic sweep
(see the deliberately-disabled tracing stale-trace loop).
- store: RecoverInterruptedRuns on the interface + PG and SQLite impls
- cmd/gateway_managed: invoke once after the timeline recorder is built
- tests: SQLite store test — orphan recovered, completed run untouched,
idempotent on re-run
Server-side webhook agent runs (sync, async, and admin test) used Stream:false,
so OpenAI-compatible routers that only cache streaming requests never populated
or served their prompt cache. Webhook runs paid full input-token price on every
turn even with a stable session and an identical multi-turn prefix, while WS chat
(Stream:true) got cache hits from the 3rd turn.
Add gateway.webhook_stream (default true, env GOCLAW_WEBHOOK_STREAM) and apply it
to all three run sites. ChatStream returns the fully assembled response, so the
payload returned to the caller is unchanged. Set to false to restore non-streaming.
Multi-tool responses could interleave synthetic user messages (loop
warnings, nudges) between tool results, breaking OpenAI-compatible
providers that require contiguous tool_result grouping.
Fix both sequential and parallel tool execution paths in tool_stage.go
to defer non-tool messages until after all tool results are appended.
Fix sanitizeHistory lookahead in loop_history_sanitize.go to reorder
interleaved non-tool messages in persisted session history.
Closes#1177
Add total-count pagination to the webhook admin endpoints and the web UI.
Store:
- WebhookStore/WebhookCallStore gain Count; WebhookListFilter gains
IncludeRevoked + Query (PG + SQLite, parameterized, tenant-scoped)
API:
- GET /v1/webhooks and GET /v1/webhooks/{id}/calls return
{items, total, limit, offset} with server-side search + revoked filtering
Web UI:
- server-driven list pager + search/revoked filter; call-history dialog uses
the real total (fixes the full-page "has more" boundary bug)
- i18n pager labels (en/vi/zh)
Tests: store pagination integration test; mock stores implement Count.
No schema migration (read-only COUNT).
Replace the hardcoded 30s webhook agent-run deadline with a configurable
timeout (default 600s, cap 3600s) for both the async worker and the
sync/test HTTP handler. Legitimate multi-step runs were dying at 30s
mid-tool-call; the same request over WebSocket completed fine.
- internal/webhooks/timeout.go: ResolveTimeoutSec helper (<=0 → 600s, cap 3600s)
- WorkerConfig.AsyncAgentTimeout (async worker) + WebhookLLMHandler.syncTimeout
(sync + admin test), wired from config
- config keys gateway.webhook_{async,sync}_timeout_sec, also settable via
GOCLAW_WEBHOOK_{ASYNC,SYNC}_TIMEOUT_SEC env (env overrides config)
- tests: timeout bounds + config file/env override
The agent config UI (tools-profile-section) and i18n already expose a
per-agent "Rate Limit (per hour)" field saved into tools_config, but the
backend ignored it — the tool rate limiter only ever used the global
tools.rate_limit_per_hour. Wire the field through: ToolPolicySpec gains
RateLimitPerHour; the agent loop threads it into the tool-execution
context; the registry passes it to the limiter, which uses it in place
of the global max when > 0 (0 inherits the global).
The per-run session reset was gated on `!job.Stateless`, inverting the
flag: stateless jobs (the token-saving default) skipped the reset and
accumulated unbounded history, while stateful jobs were wiped every run.
Reset for `job.Stateless` instead, and clear BOTH session layers — the
goclaw session store AND the Claude CLI on-disk .jsonl. claude-cli
resumes its own session by a deterministic per-key UUID, so without
clearing the .jsonl a "stateless" run still replayed the entire
accumulated history (and silently grew it run after run).
Follow-up to #1260. A model (notably the claude-cli proxy under a
degraded session) sometimes emits a tool call as a bare
<invoke name="...">...</invoke> block with no <function_calls> wrapper.
fullToolCallBlockPattern only matched the wrapped form, so the bare
block slipped through to the tag-only strip, leaking the inner
<parameter> command text into the reply while the tool never ran.
Match and remove complete bare <invoke> blocks too (after the wrapped
form), add "<invoke name=" as a detection indicator, and log the
dropped tool name(s). Partial/unterminated artifacts still fall through
to the existing tag strip.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stripGarbledToolXML removed only the tags of a model-emitted
<function_calls> block, leaking the inner <parameter> argument values
into the user-facing reply, and the dropped call was completely silent.
Detect a complete tool-call block, remove it whole, and log the
attempted tool name(s) at WARN so the no-op is diagnosable. Partial
artifacts still fall through to the existing tag-level strip.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Allow overriding the cron job execution timeout via env var, matching
the existing GOCLAW_* env var pattern used throughout the codebase.
Falls back to the existing default (10m) when unset or invalid, since
CronConfig.JobTimeoutDuration() already handles parse errors with a
slog.Warn and the DefaultJobTimeout constant.
Co-authored-by: will.nguyen <will.nguyen@gemcommerce.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local pkg-helper installs an arm64 mcp-server binary on macOS hosts; the
existing allowlist (deno, bun, uvx, ...) didn't cover it so MCP server
config validation rejected legitimate entries.
* refactor(bitrix24): rename "Path B" framing to maintainer-specified naming [B24:2794]
Per maintainer hard rule #10 (no generic "Path A/B" framing) from PR #1061
review. The Bitrix24 MCP auto-onboard flow is Bitrix-specific glue
("Bitrix24 OAuth -> existing mcp_user_credentials bridge"), NOT a generic
MCP architecture pattern.
Naming convention applied consistently:
- First mention per file: full "Bitrix24 OAuth -> existing
mcp_user_credentials bridge" (matches maintainer comment verbatim).
- Subsequent mentions in same file: shortened "mcp_user_credentials bridge".
- Test/log context referencing literal endpoint /api/auto-onboard: keep
"auto-onboard" reference (it's the actual API endpoint name).
Changes are documentation-only:
- Rename in code comments + test descriptions + plan docs.
- Clarify framing in mcp_client.go + provisioner.go doc comments to
emphasize Bitrix-specific glue (not generic MCP infra).
- Reuse existing mcp_user_credentials table + MCPServerStore methods
(no schema / store / abstraction change).
Files:
- cmd/gateway.go (factory registration doc)
- internal/channels/bitrix24/{channel,factory,mcp_client,provisioner}.go
- internal/channels/bitrix24/{mcp_client,provisioner}_test.go
- plan/goclaw-mcp-integration.md (21 occurrences)
Verified: go build + MCP-related tests pass (TestProvision*,
TestInitMCPProvisioner*, TestMCPClient*).
Phase 1 of Path C execution per
plans/reports/decision-log-260519-1555-bitrix24-pr-fork-decision.md.
* fix: confine outbound media paths to agent workspace [B24:2794]
Tool MEDIA:<path> output reached channel file-upload sinks (Bitrix
imbot.v2.File.upload, Telegram sendDocument, etc.) verbatim via
parseMediaResult, with no workspace-boundary check. A malicious or buggy
tool emitting MEDIA:/etc/passwd could exfiltrate arbitrary files to chat.
Extract the EvalSymlinks+Rel containment from extractMediaFromContent into
a shared confineToWorkspace helper and apply it at the parseMediaResult
sink in processToolResult. Fixing at the source/egress boundary protects
every channel at once rather than per-channel. Paths that escape the
workspace are dropped and logged (security.media_path_rejected).
Add TestConfineToWorkspace (boundary unit) and
TestParseMediaResultConfinedToWorkspace (sink regression for H2).
* feat(bitrix24): support inbound + outbound media via imbot.v2 File API [B24:2794]
Bitrix24 channel was text-only; attachments were parsed but dropped.
- Inbound: download chat files via imbot.v2.File.download (one-time URL),
forward to the agent with MIME preserved (internal/channels/bitrix24/download.go).
- Outbound: upload agent media to the chat via imbot.v2.File.upload
(internal/channels/bitrix24/send_media.go).
- Add BaseChannel.HandleMessageMedia to preserve MIME/filename through the bus.
- Per-channel media_max_mb cap (default 20) applies to both directions.
Tests: 92 pass (internal/channels/bitrix24 + internal/channels), go vet clean (PG + sqliteonly).
* refactor(bitrix24): migrate messaging/bot-list/unregister to imbot v2 API [B24:2794]
Move outbound REST calls to the imbot v2 family (keeps register on v1):
- imbot.message.add -> imbot.v2.Chat.Message.send (fields.message shape, live-verified)
- imbot.bot.list (+ legacy imbot.list fallback) -> imbot.v2.Bot.list; add botListRows
to normalize the v2 {bots:[...]} envelope, legacy array, and id-keyed map forms
- imbot.unregister -> imbot.v2.Bot.unregister
Bot registration stays on v1 imbot.register: v2 imbot.v2.Bot.register changes the
event-delivery model (per-event handler URLs -> eventMode), which would require
rewriting the inbound event parser. No user-facing behavior change.
Tests: bitrix24 package green; go vet ./... clean.
* feat(bitrix24): route whisper via v1 SKIP_CONNECTOR + add v2 replyId [B24:2794]
Bot was leaking HiddenMessage (whisper) replies to the external Zalo
connector because every outbound call went through imbot.v2.Chat.Message.send,
which has no equivalent of the v1 SKIP_CONNECTOR flag. Branch the outbound
path on inbound visibility:
whisper → imbot.message.add + SKIP_CONNECTOR=Y (v1, send_v1.go)
public → imbot.v2.Chat.Message.send + fields.replyId (v2, send_v2.go)
Pipeline:
events.go parse data[PARAMS][PARAMS][COMPONENT_ID]=HiddenMessage
into EventParams.IsHiddenMessage (form + JSON variants)
handle.go set bitrix_visibility on InboundMessage.Metadata
consumer forward visibility + message_id into OutboundMessage
send.go resolveSendOptions + sendChunk dispatcher +
shared callWithRateLimitRetry helper
metadata_keys.go single source of truth for the keys + values
Defaults preserve pre-refactor behaviour: callers that don't populate
bitrix_visibility still go through v2 public, and replyId is omitted
unless a numeric bitrix_message_id arrives in metadata.
Tests:
TestParseEvent_FormURLEncoded_IsHiddenMessage (3 cases)
TestParseEvent_JSON_IsHiddenMessage (3 cases)
TestResolveSendOptions (8 cases)
TestSend_BranchesOnVisibility (4 cases)
* feat(bitrix24): openline sender-tag echo on replies [B24:2794]
Openline sender-tag echo (this change):
- Capture the connector sender tag ("[name #id]:" or "[name] #id:") from
inbound openline group messages, strip it from the body the agent sees,
and re-prepend the canonical "[name] #id:" form to the reply so the Open
Channel connector routes the answer back to the right external user.
- New sender_prefix.go helper (+ test) accepts both inbound layouts and
emits one canonical form; scoped to messages carrying the tag, so plain
chats are unaffected.
- metadata_keys.go: MetaKeySenderPrefix; handle.go capture/strip/stash;
gateway_consumer_normal.go forwards the key; send.go prepends it on the
first chunk before chunking.
Bundled bitrix24 channel-core work already on this branch:
- handle.go: @mention is the sole trigger for both staff and connector
customers; unmentioned traffic is dropped (was: drop all connector msgs).
- isGroupMessageType: treat SONET_GROUP "B" as a group.
- handle_test.go, mcp_client_test.go: cover the above.
* feat(bitrix24): accept colon-less openline sender tag, echo [name] #id [B24:2794]
The Open Channel connector dropped the trailing colon from its sender tag:
inbound now arrives as "[Name] #id <msg>" (was "[Name] #id: <msg>"). The
id-bearing patterns required the colon, so the tag fell through to the
name-only branch and the reply echoed "[Name]" — dropping the #id the
connector needs to route the answer back.
- sender_prefix.go: make the trailing ":" optional on both id layouts
([name #id] / [name] #id, with or without colon) and echo the canonical
"[name] #id" (no colon) to match the connector's current format. Bare
"[name]" (no id) still echoes "[name]" for Open Channel only.
- handle.go: gate the bare name-only layout to Open Channel (isOpenChannel)
so ordinary group chats starting with "[x] ..." are left untouched.
- sender_prefix_test.go: cover colon/no-colon x id-inside/id-outside, the
name-only openline case, and the non-openline no-op.
* fix: security and robustness fixes from the bitrix24 channel review [B24:2794]
- download.go: block redirect-based SSRF on inbound media. CheckRedirect
re-validates each hop (http(s) only, reject private/loopback/link-local
hosts, cap hops); the initial portal-domain pin is no longer bypassable
via a 3xx to an internal service. Public-host redirects still allowed.
- handle.go: extract/echo the openline sender tag only for Open Channel
sessions (was: any group chat), removing bogus prefixes in CRM group
chats and narrowing the forged-tag misroute surface.
- loop_tools.go + loop_media.go: confine result.Media to the agent / team /
tenant-allowed roots (new confineToAnyRoot) before a channel uploads it,
so a prompt-injected out-of-workspace path (e.g. /etc/passwd) cannot
exfiltrate, while legitimate cross-workspace media (team files, delegatee
output) still flows.
- send_media.go: bounded outbound read via io.LimitReader replaces the
os.Stat + os.ReadFile pair, closing the TOCTOU size-cap bypass; cap a
single message's outbound attachments at 10 (mirrors inbound).
- register.go: paginate imbot.v2.Bot.list (limit/offset + hasNextPage,
capped at 40 pages) so verify/lookup see bots past the first 50.
- mcp_client.go: redact access_token / refresh_token / client_secret from an
echoed MCP error body before it is logged or returned (+ test).
* fix(security): validate resolved dial IP on Bitrix media redirects [B24:2794]
The inbound media download redirect guard only string-checked the redirect
hostname (isPrivateOrLoopback on req.URL.Hostname()), so a redirect to a public
hostname that resolves to 127.0.0.1 / 169.254.169.254 / an RFC1918 address — or a
DNS-rebinding swap between check and dial — still passed the guard and the client
would connect. Reported in PR review.
Add security.NewRedirectFollowingSafeClient: it follows redirects but validates
the RESOLVED destination IP of every hop at dial time via net.Dialer.Control,
reusing the existing blocked-CIDR list. The IP it checks is the IP actually
dialed, so both redirect-to-internal and DNS rebinding are refused, while
legitimate public CDN redirects still succeed. download.go now uses it instead of
the hostname-string guard.
Tests: deterministic dial-control table (loopback / link-local / private /
multicast / unspecified / public, v4 + v6), malformed/non-IP addr, test bypass,
loopback-dial-blocked client wiring, and redirect cap + scheme checks.
* feat(bitrix24): per-participant Zalo openline identity from 3-token sender tag [B24:2794]
Parse the connector's "[Name] #uid #msgId" sender tag so each external
customer in a shared Open Channel group gets its own contact + USER.md
instead of collapsing onto the connector proxy id. Identity minting is
gated on IS_CONNECTOR=Y to reject operator forged tags. Echo back the
msgId only ("#msgId") on replies; keep the legacy single-number and
name-only layouts unchanged. Zero DB migration.
- sender_prefix.go: parseOpenlineSenderTag() classifies 3-token / legacy / name-only
- handle.go: synthetic senderID "openlines:{instance}:{chat}:{uid}" + participant_user_id metadata, gated on FromIsConnector
- gateway_consumer_normal.go: deriveGroupUserID() routes participant -> per-person scope, group fallback otherwise
- send.go: buildAddressMention numeric-id guard so synthetic ids don't emit invalid [USER=...] BBCode
- MetaKeyMessageID kept as Bitrix MESSAGE_ID (drives v2 fields.replyId); connector msgId surfaced only via echo prefix
---------
Co-authored-by: DangTinh311 <dangtinh31193@gmail.com>
Co-authored-by: Chinh Dang <chinhdang@192.168.68.104>
* feat(mcp): MCP OAuth 2.1 client — full implementation with tests
Implements a complete MCP OAuth 2.1 authorization flow for tool servers that
require user-delegated access, covering all layers from DB to UI.
- discovery.go: RFC 9728 protected-resource → RFC 8414 AS metadata → OIDC
fallback chain with 5-min in-memory cache and InvalidateCache()
- dcr.go: RFC 7591 Dynamic Client Registration with response size guard
- flow.go: PKCE (S256) authorization code flow — StartFlow(), ExchangeCode(),
ClientCredentials(), auto-cleanup of expired flows; carries AS issuer through
PendingFlow for status display
- refresher.go: OAuthTokenProvider with in-memory token cache, automatic refresh
on expiry, per-user vs global slot isolation, InvalidateCache/InvalidateServer
- migrations/000074 + SQLite schema: mcp_oauth_tokens with AES-256-GCM encrypted
access/refresh tokens, partial unique index for global vs per-user rows,
ON DELETE CASCADE from mcp_servers
- store.MCPOAuthTokenStore: Upsert, Get/GetUser, Delete/DeleteUser, and
DeleteServerOAuthTokens (purge all rows for a server)
- PostgreSQL + SQLite implementations
- POST /v1/mcp/oauth/start — discovery + optional DCR + PKCE redirect URL;
client_credentials completes server-side (no redirect) and returns completed=true
- GET /v1/mcp/oauth/callback — exchange code, persist token, publish WS event;
payload built via json.Marshal (no reflected XSS via error_description)
- GET /v1/mcp/oauth/status/{id}, DELETE /v1/mcp/oauth/token/{id} — admin-gated
- POST /v1/mcp/oauth/discover/{id} — on-demand discovery probe
- All outbound calls go through the SSRF-safe client with pinned IPs
- pkg/protocol/mcp_events.go: EventMCPOAuthComplete routed only to the initiating
user (admins in-tenant included); fail-closed across tenants
- getUserMCPTools() injects Authorization: Bearer from OAuthTokenProvider; on a
401 for OAuth servers it purges the cached token so the next turn re-resolves
- handleUpdateServer purges all OAuth tokens (global + per-user), drops the
refresher cache, and evicts the pool when a server's URL or OAuth config
(client_id / endpoints / grant_type / scope / auth_type) changes — so the
status UI and agent never use a token minted for the old resource/AS
- MCPOAuthDialog (WS-driven), unified user-credentials dialog, OAuth settings
fields; handles the no-redirect client_credentials completion
- internal/mcp/oauth/*_test.go: discovery cache, PKCE, DCR, refresher
- internal/http/mcp_oauth_test.go + mcp_update_oauth_purge_test.go: routes, auth
gating, WS event, purge-on-URL/OAuth-config-change
- tests/integration: store + encryption + tenant isolation, E2E start→callback,
DeleteServerOAuthTokens
- internal/gateway/event_filter_test.go, internal/agent/loop_mcp_user_test.go
* fix(mcp): return 400 on OAuth callback with code but missing state
The callback handler rendered a 200 HTML page whenever code or state was
absent. An auth code WITH a missing state is a malformed / CSRF-risk
callback (state is the CSRF token), so reject that case with HTTP 400.
A bare hit with neither code nor state (user opening the URL directly),
provider errors, and exchange failures keep their 200 HTML popup page.
Adds a status code parameter to writeCallbackHTML. Fixes the
TestOAuthCallbackMissingState integration regression while keeping
TestHandleCallbackMissingCodeAndState (no params -> 200) green.
* fix(mcp): scope-based OAuth auth + honor manual OAuth endpoints
Addresses the two MCP/OAuth security-review findings.
Finding 1 — authorization. mcp_oauth_tokens is tenant-scoped, but
start/status/revoke were gated only by requireAuth(RoleAdmin), an RBAC
role check, not tenant membership, so a RoleAdmin caller could act on a
tenant they don't administer. A blanket requireTenantAdmin would have
broken per-user self-service, which the UI exposes (the per-user
MCPUserCredentialsDialog shows an "Authorize" button to regular users for
their own credentials). Instead mirror the existing per-user MCP
credentials model (resolveTargetUserID in mcp_user_credentials.go):
- start/status/revoke accept any authenticated user; each handler calls
authorizeOAuthScope.
- a caller may manage their OWN per-user token (self-service); the
global/server token (user_id="") and other users' tokens require
tenant-admin (owner bypass), so a RoleAdmin that is not a tenant admin
is rejected.
- discover stays admin-only (it only previews AS metadata for a server).
Add a TenantStore dependency. Tests cover self-service, on-behalf-of-
another (403), and global-by-non-tenant-admin (403).
Finding 2 — honor manual OAuth config end-to-end. The UI sent use_dcr /
auth_endpoint / token_endpoint and the update path fingerprinted them for
purge, but handleStart always discovered + DCR'd and ignored them. Now:
- use_dcr=false (a *bool, so legacy/absent stays discover+DCR) skips
discovery/registration and uses the operator endpoints, SSRF-validated.
- token_endpoint is always required; auth_endpoint only for auth-code
grants — client_credentials needs no authorization URL, matching the UI
which hides that field for that grant.
- the refresher already refreshes against the stored token_endpoint and
the callback persists it, so manual-mode tokens refresh correctly.
- oauthFingerprint includes use_dcr (nil normalized to true) so toggling
DCR mode purges stale tokens.
- the web form only serializes manual endpoints when use_dcr is off.
Audited all MCP dialogs (form, global OAuth, per-user credentials, grants,
tools): OAuth dialogs handle completed/auth_url identically and read
config from stored server settings; runtime connect uses the stored token
via the refresher (no re-discovery).
Tests: manual auth-code + client_credentials endpoints, missing/SSRF
endpoints, and the full self/global/on-behalf authorization matrix.
* feat(webhooks): add webhook management UI with delivery history & test
Webhooks admin page on the web dashboard for managing inbound HTTP webhooks
(llm + message kinds), backed by new admin endpoints. No schema changes
(reuses migrations 000059-000061).
Backend (internal/http):
- GET /v1/webhooks/{id}/calls - paginated delivery history (trimmed
DTO; status/limit/offset filters; ownership + tenant scoped)
- GET /v1/webhooks/{id}/calls/{callId} - full single-call detail (request
payload, full response, callback URL, idempotency key, timestamps)
- POST /v1/webhooks/{id}/test - server-side test invocation using the
admin session (no secret); dispatched by kind via RunTest() on the llm/message
handlers; message tester nil-guarded (403 on Lite)
Injects WebhookCallStore + SetTesters() in cmd/gateway_http_wiring.go, adds i18n
key webhook.message_test_requires_standard (en/vi/zh), and unit tests for
list/detail (filter, tenant isolation) and test (success/error/edition gate).
Frontend (ui/web):
- /webhooks admin-only page: list with revoked filter + badge, create/edit form
(edition-gated message kind, Lite localhost_only lock), show-once secret dialog
(create + rotate), test dialog, and delivery-history dialog with server-side
pagination plus a click-through full call-detail dialog.
- use-webhooks hooks, query keys, types, routing, sidebar entry (Cable icon to
distinguish from the existing event "Hooks" page), webhooks i18n namespace
(en/vi/zh).
* fix(webhooks): validate admin message tests
---------
Co-authored-by: Goon <duy@wearetopgroup.com>
* feat(tools): implement image reference processing and native provider support
- Support OpenAI image edits via both Multipart form-data and JSON payloads
- Automatically append reference image descriptions to prompt under [Reference Image Roles]
- Support downloading image URLs for Gemini native image generation
- Deduplicate reference images to optimize API request size
- Add unit tests for Codex, DashScope, MiniMax, BytePlus and local/remote path resolution
* fix(tools): SSRF-guard reference-image URL downloads in create_image
downloadImageBytes fetched caller-supplied ref_images[].url with a plain
http.Client and unbounded io.ReadAll — no SSRF validation, redirect
policy, or size cap, letting the gateway dial loopback/private/metadata
hosts or read arbitrarily large responses.
- Validate the URL via security.Validate and pin the resolved IP, then
download through security.NewSafeClient (pinned dial, no redirects).
- Cap the response with a bounded read (refImageMaxBytes, 20 MB).
- Reject non-HTTP(S) reference URLs up front (file://, data:, gopher://)
so provider-forwarded URLs stay HTTP(S)-only; document the trust
boundary between gateway-side fetch and provider-forwarded URLs.
- Add regression tests: blocked loopback/private/metadata, unfollowed
redirect, oversized response, and non-http(s) scheme rejection.
setupToolRegistry creates the rate limiter from cfg.Tools.RateLimitPerHour
during early bootstrap (gateway.go line 137). System_configs DB overlay
runs ~50 lines later via cfg.ApplySystemConfigs (line 191), so any DB
override of tools.rate_limit_per_hour was silently lost - the limiter
object was already initialised from the JSON5 default.
Symptom in production: editing tools.rate_limit_per_hour via system_configs
table or the config HTTP API had no effect on running gateways. Operators
had to inject a config.json file to change the value, defeating the
DB-as-source-of-truth pattern that other tunables rely on.
Re-apply the limiter after ApplySystemConfigs runs. Safe ordering: server
has not started yet, no in-flight tool calls, and SetRateLimiter is a
plain field assignment with no shutdown cost on the discarded limiter.
nil case (rate_limit_per_hour <= 0) also handled so DB writes can disable
the limiter without restart.
Tests: new TestRegistry_SetRateLimiter_ReplacesPriorLimiter covers both
the replace-with-higher-limit path and the nil-disables path. All
existing rate limiter tests still pass.
Note: the same ordering pattern likely affects other config consumed
inside setupToolRegistry (tools.scrub_credentials, MCP server wiring).
Out of scope for this hotfix - they need a deeper restructure.
The CLI setup wizard and 'providers add' command were using
'openai-compat' (hyphen), but the API and database expect
'openai_compat' (underscore), causing provider creation to fail.
Fixes#1046
When a member calls team_tasks(action="complete"), goclaw fans the result
back to the Lead's session via the team-task announce queue. The Lead
resumes a turn whose initial user message is the synthesized
"[System Message] Team member ... completed task. Result: ..." string.
Inside that resumed turn, the Lead is a regular agent — it can call any
tool, including write_file and cron mutations. Those tools route through
CheckFileWriterPermission / CheckCronPermission, which in group-scope
sessions deny when the resumed RunRequest carries no SenderID:
permission denied: system context cannot write files in group chats.
If this is a legitimate user action, ensure the acting sender is
preserved through the tool chain.
team_tool_dispatch.go already stamps MetaOriginSenderID and MetaOriginRole
into the dispatch metadata. consumer_handlers.go already reads inMeta on
the completion-side teammate message. The gap was in between:
- announceRouting (cmd/gateway_announce_queue.go) had no field for
OriginSenderID / OriginRole
- The RunRequest it built had no SenderID / Role set
- loop_context.injectContext skips WithSenderID when req.SenderID is
empty — so the Lead's resumed ctx had no sender attribution, and
every group-scope permission check then tripped the deny path.
Subagent path (subagentAnnounceRouting in gateway_subagent_announce_queue.go)
already had these fields wired since #915. This brings the team-task
path to parity. No security relaxation: empty upstream → still empty
downstream (still denies, as intended for genuine system-initiated
turns); only legitimately-attributed dispatches now flow through.
- cmd/gateway_announce_queue.go: add OriginSenderID + OriginRole to
announceRouting; pass them into the RunRequest.
- cmd/gateway_consumer_handlers.go: read MetaOriginSenderID +
MetaOriginRole from inMeta when building the routing struct.
- cmd/gateway_announce_routing_test.go: 2 unit tests guarding both
"real human propagates through" and "empty stays empty" cases so
this regression can't re-land silently.
Tests: existing cmd/ + internal/tools/ suites pass; new tests pass.
Registering a remote MCP server (sse/streamable-http) whose hostname
resolves to a private IP is rejected at config-validation time by the
SSRF guard, with no production escape hatch -- the only bypass is a
test-only loopback flag. Self-hosted MCP servers on a private network
are therefore unregisterable, even though the runtime MCP client
connects to them fine (Test Connection succeeds and lists tools; only
create/update input validation blocks).
Add an opt-in, operator-configured allowlist (GOCLAW_MCP_ALLOWED_HOSTS,
empty by default) of trusted hostnames exempt from the private/loopback
IP block during MCP server URL validation only:
- security.ValidateAllowingHosts(url, allowedHosts): like Validate but
skips the private/loopback block for allowlisted hostnames. The
cloud-metadata/link-local (169.254/fe80), multicast and unspecified
ranges are never exempted, even for allowlisted hosts.
- mcp.SetAllowedHosts wires the operator allowlist into ValidateURL /
ValidateServerConfig; default empty => no behavior change.
- web_fetch / webhook / redirect SSRF paths are unchanged (those stay
agent-influenced and fully guarded).
Matching is case-insensitive on the pre-resolution hostname.
require_user_credentials MCP servers loaded and connected, but their tool
defs were never sent to the model, so the agent could never call the
per-user mcp_<prefix>__* tools (it only ever saw the shared MCP server).
The per-user tool objects are intentionally kept out of the shared registry
to prevent cross-user credential leaks, so FilterTools could not emit them.
Surface them through a request-scoped overlay registry passed to the policy
engine: per-user tools are now evaluated AND emitted under the same
allow/deny rules as registry tools, then resolved per-actor at execution
time via executeToolForActor. The overlay is discarded after filtering, so
the shared registry and other users stay unaffected.
- add tools.NewUserToolOverlay (request-scoped, discarded post-filter)
- buildFilteredTools: wrap registry in overlay when per-user tools present;
no-policy path appends their defs directly
- fix stale getUserMCPTools docstring (claimed shared-registry registration)
- tests: overlay policy allow/deny + per-user surfacing / strip / dedup
Co-authored-by: DangTinh311 <dangtinh31193@gmail.com>
The BridgeToolNames allowlist omitted "delegate", so claude-cli /
Claude-Code-backed agents never received the tool even when their
orchestration mode resolved to "delegate" with active agent_links.
The native agent loop injects delegate per orchestration mode, but
bridge sessions only see this curated allowlist, making inter-agent
delegation impossible from bridge-backed agents.
delegate is safe to expose unconditionally: it self-gates via
CanDelegate/agent_links and resolves its source agent from the
X-Agent-ID header context, exactly like team_tasks (already bridged).
Both PG and SQLite skill stores returned SKILL.md content without the
{baseDir} substitution that skills.Loader.LoadSkill performs for
file-based access. Agents reading SKILL.md via this RPC saw the literal
placeholder, breaking script-invocation lines that expect absolute paths.
Mirror the loader's substitution in both store implementations so
behavior is consistent regardless of access path.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The legacy /open-apis/bot/v3/info endpoint returns the bot object at the
top level of the response, not inside "data" like newer Lark APIs. The
probe always failed to extract open_id, leaving botOpenID empty, which
made every group mention count as a bot mention — the bot replied to
messages mentioning other users even with requireMention enabled.
- Read top-level "bot" first, fall back to data.bot defensively
- Retry the startup probe with backoff to survive transient failures
- Fix test fixtures to match the real API response shape
Fixes#895
Use the existing relative-time guard when deciding whether to show channel diagnostics, so Go zero-value timestamps do not make healthy channels look broken.
Add regression coverage for zero-value first_failed_at and active failure states.
Co-authored-by: evgyur <evgyur@gmail.com>
Clear targetAgentID in handleMessage so the gateway consumer's
resolveAgentRoute matches via cfg.Bindings instead of always
using the channel instance's agent_id.
Without this patch, all Discord channels route to the channel
instance's default agent, ignoring bindings in config.json.
Add slog.Info call for routing diagnostics.
Co-authored-by: Duy /zuey/ <duy@wearetopgroup.com>
Co-authored-by: Said <msaidf@users.noreply.github.com>
The sessions.reset RPC only reset the native session store, leaving the
Claude CLI-backed history (.jsonl + CLAUDE.md) in place. The /reset chat
command already clears it via providers.ResetCLISession, so RPC callers
got an incomplete reset for claude-cli agents — stale history was resumed
on the next turn.
Mirror the chat path by calling ResetCLISession from handleReset. It is a
no-op when the CLI provider is unused. The call is wired through an
overridable package var so the behavior can be regression-tested.
Zezae Oh