diff --git a/docs/03-tools-system.md b/docs/03-tools-system.md index 49fd1b6d..e188980b 100644 --- a/docs/03-tools-system.md +++ b/docs/03-tools-system.md @@ -1,10 +1,22 @@ # 03 - Tools System -The tools system is the bridge between the agent loop and the external environment. When the LLM emits a tool call, the agent loop delegates execution to the tool registry, which handles rate limiting, credential scrubbing, policy enforcement, and virtual filesystem routing before returning results for the next LLM iteration. +The tools system bridges the agent loop and the external environment. When the LLM emits a tool call, the registry handles rate limiting, credential scrubbing, policy enforcement, and virtual filesystem routing before returning results for the next LLM iteration. --- -## 1. Tool Execution Flow +## 1. Overview + +Tools are the agent's hands. The registry owns all tool instances and mediates every invocation — injecting request context, enforcing rate limits, running deny-pattern checks, and scrubbing credentials from output. Agents never call tools directly; everything routes through `Registry.ExecuteWithContext`. + +Key concerns the tool system manages: +- **Discovery** — which tools exist and which a given agent may use (policy engine) +- **Invocation** — per-call context injection, rate limiting, execution, result scrubbing +- **Isolation** — workspace path containment, path traversal prevention, Docker sandbox routing +- **Customization** — per-tenant config overlay, custom shell-based tools at runtime + +--- + +## 2. Registry & Lifecycle ```mermaid sequenceDiagram @@ -15,7 +27,7 @@ sequenceDiagram participant SC as Scrubber AL->>R: ExecuteWithContext(name, args, channel, chatID, ...) - R->>R: Inject context values into ctx + R->>R: Inject per-call context values R->>RL: Allow(sessionKey)? alt Rate limited RL-->>R: Error: rate limit exceeded @@ -30,656 +42,179 @@ sequenceDiagram R-->>AL: Result ``` -ExecuteWithContext performs 8 steps: +Per-call context values injected before execution: channel identity, chat ID, peer kind, sandbox key. Tool instances are shared safely across goroutines because mutable state lives in context, not structs. -1. Lock registry, find tool by name, unlock -2. Inject `WithToolChannel(ctx, channel)` -3. Inject `WithToolChatID(ctx, chatID)` -4. Inject `WithToolPeerKind(ctx, peerKind)` -5. Inject `WithToolSandboxKey(ctx, sessionKey)` -6. Rate limit check via `rateLimiter.Allow(sessionKey)` -7. Execute `tool.Execute(ctx, args)` -8. Scrub credentials from both `ForLLM` and `ForUser` output, log duration - -Context keys ensure each tool call receives the correct per-call values without mutable fields, allowing tool instances to be shared safely across concurrent goroutines. +**Custom tools** (shell-based, stored in DB) use a two-phase registry: +- Global tools (no agent scope) are loaded at startup into a shared registry. +- Per-agent tools are merged on first access; the registry is cloned, never mutated globally. +- Cache invalidation via pub/sub causes a reload + router invalidation on next agent turn. --- -## 2. Complete Tool Inventory +## 3. Tool Capabilities -### Filesystem (group: `fs`) +Each tool carries structured metadata describing side-effect class, group membership, and workspace requirements. + +| Capability | Meaning | Examples | +|---|---|---| +| `read-only` | No side effects; safe to retry | `read_file`, `web_search`, `memory_search` | +| `mutating` | Modifies state or external systems | `write_file`, `exec`, `cron`, `team_tasks` | +| `async` | Returns immediately; result delivered later | `spawn` | +| `mcp-bridged` | Proxied to an external MCP server | MCP tools registered dynamically | + +Capabilities are inferred from tool name when no explicit metadata is registered. The policy engine can use capability class to gate entire sets (e.g. restrict an agent to read-only tools). + +--- + +## 4. Built-in Tool Inventory + +### Filesystem (`group:fs`) | Tool | Description | -|------|-------------| +|---|---| | `read_file` | Read file contents with optional line range | | `write_file` | Write or create a file | -| `edit` | Apply targeted edits to a file | +| `edit` | Apply targeted edits to a file (old/new string replace) | | `list_files` | List directory contents | -| `search` | Search file contents with regex | -| `glob` | Find files matching a glob pattern | -### Runtime (group: `runtime`) +### Runtime (`group:runtime`) | Tool | Description | -|------|-------------| -| `exec` | Execute a shell command | -| `credentialed_exec` | Execute CLI with injected credentials (direct exec mode, no shell) | +|---|---| +| `exec` | Execute a shell command; supports credentialed CLI mode for secure credential injection | -### Web (group: `web`) +**Credentialed CLI mode** — when the invoked binary is registered in `secure_cli_binaries`, the exec tool injects encrypted env vars directly into the child process (no shell involved) and verifies the agent has an explicit grant. Shell-wrapper unwrapping (up to depth 3) prevents bypass via `sh -c`. Fail-closed on DB error. + +### Web (`group:web`) | Tool | Description | -|------|-------------| -| `web_search` | Search the web (Brave, DuckDuckGo) | -| `web_fetch` | Fetch and parse a URL | +|---|---| +| `web_search` | Search the web (Exa, Tavily, Brave, DuckDuckGo provider chain) | +| `web_fetch` | Fetch and parse a URL (HTML → Markdown); domain allow/block policy | -### Memory (group: `memory`) +### Memory (`group:memory`) | Tool | Description | -|------|-------------| -| `memory_search` | Search memory documents (BM25 + vector hybrid search) | -| `memory_get` | Retrieve a specific memory document (L1 summary) | -| `memory_expand` | Load full episodic memory content by ID (L2 deep retrieval) | +|---|---| +| `memory_search` | Search memory documents (BM25 + vector hybrid) — returns L1 abstracts | +| `memory_get` | Retrieve a specific memory document by ID | +| `memory_expand` | Load full episodic memory content (L2 deep retrieval) | -**Memory Layers:** -- **L1 (Search)**: `memory_search` returns abstracts + vector scores for ranking -- **L2 (Expand)**: `memory_expand` retrieves full summary for a given episodic ID -- **Vault**: `vault_search` unified discovery across memory + vault docs + knowledge graph +Memory layers: L1 (`memory_search`) returns ranked abstracts; L2 (`memory_expand`) loads the full summary for a given episodic ID. -### Sessions (group: `sessions`) +### Sessions (`group:sessions`) | Tool | Description | -|------|-------------| +|---|---| | `sessions_list` | List active sessions | | `sessions_history` | View session message history | | `sessions_send` | Send a message to a session | -| `spawn` | Spawn subagent or delegate to another agent | | `session_status` | Get current session status | +| `spawn` | Spawn a subagent or delegate to another session | -### Knowledge & Vault (group: `knowledge`) +### Knowledge & Vault (`group:knowledge` / `group:goclaw`) | Tool | Description | -|------|-------------| -| `vault_search` | Primary discovery: unified search across vault docs, memory, knowledge graph (hybrid FTS + vector) | -| `knowledge_graph_search` | Search knowledge graph for entities and relationships | +|---|---| +| `vault_search` | Unified hybrid search across vault docs, memory, and knowledge graph | +| `vault_read` | Read full content of a vault document by doc_id | +| `knowledge_graph_search` | Search knowledge graph entities and relationships | | `skill_search` | Search available skills (BM25) | -### Automation (group: `automation`) +### Automation (`group:automation`) | Tool | Description | -|------|-------------| -| `cron` | Manage scheduled tasks | +|---|---| +| `cron` | Manage scheduled tasks (create, list, delete) | | `datetime` | Get current date/time with timezone support | +| `heartbeat` | Configure agent periodic proactive check-ins | -### Messaging (group: `messaging`) +### Messaging (`group:messaging`) | Tool | Description | -|------|-------------| +|---|---| | `message` | Send a message to a channel | | `create_forum_topic` | Create a Telegram forum topic | +| `list_group_members` | List members in a group chat (Feishu/Lark) | -### Delegation (group: `delegation`) +### Delegation | Tool | Description | -|------|-------------| +|---|---| | `delegate` | Inter-agent task delegation via agent_links (async/sync modes with timeout) | -### Teams (group: `teams`) +### Teams (`group:team`) | Tool | Description | -|------|-------------| -| `team_tasks` | Task board: create, list, get, claim, complete, cancel, assign, review, approve, reject, comment (with type: note/blocker), progress, attach, ask_user, search, update | -| `team_message` | Mailbox: send direct message to teammate, broadcast to all, read unread messages | +|---|---| +| `team_tasks` | Task board: create, list, get, claim, complete, cancel, assign, review, approve, reject, comment, progress, attach, ask_user, search, update | ### Media Generation -#### Images | Tool | Description | -|------|-------------| -| `create_image` | Generate images from text description (OpenAI, Gemini, MiniMax, DashScope) | - -#### Audio & Music -| Tool | Description | -|------|-------------| -| `create_audio` | Generate audio/music/sound effects (MiniMax music, ElevenLabs effects) | +|---|---| +| `create_image` | Generate images from text (OpenAI, Gemini, MiniMax, DashScope, BytePlus) | +| `create_audio` | Generate audio/music/sound effects (MiniMax, ElevenLabs) | +| `create_video` | Generate video from text/image (MiniMax, Gemini, BytePlus) | | `tts` | Text-to-speech synthesis (OpenAI, ElevenLabs, Edge, MiniMax) | -#### Video -| Tool | Description | -|------|-------------| -| `create_video` | Generate video from text/image (MiniMax) | - ### Media Reading -#### Images | Tool | Description | -|------|-------------| +|---|---| | `read_image` | Analyze/describe images using vision AI (Gemini, Anthropic, OpenRouter, DashScope) | - -#### Audio & Speech -| Tool | Description | -|------|-------------| -| `read_audio` | Transcribe audio to text (Resolve transcription service) | - -#### Documents -| Tool | Description | -|------|-------------| -| `read_document` | Extract and analyze documents (PDF, images, etc) via Gemini or Resolve service | - -#### Video -| Tool | Description | -|------|-------------| -| `read_video` | Analyze/transcribe video content via Resolve service | +| `read_audio` | Transcribe audio to text | +| `read_document` | Extract and analyze documents (PDF, images) via Gemini or Resolve service | +| `read_video` | Analyze/transcribe video content | ### Skills & Content | Tool | Description | -|------|-------------| +|---|---| | `use_skill` | Activate a skill (marker tool for observability) | | `publish_skill` | Register a skill directory in the database | - -### AI & LLM - -| Tool | Description | -|------|-------------| -| `openai_compat_call` | Call OpenAI-compatible endpoints with custom request formats | - -### Workspace & Administration - -| Tool | Description | -|------|-------------| -| `workspace_dir` | Resolve workspace directory for team/user context | +| `skill_manage` | Manage skill lifecycle (admin operations) | --- -## 3. Filesystem Tools and Virtual FS Routing +## 5. Tool Contracts (JSON Schemas) -Filesystem operations are intercepted before hitting the host disk. Two interceptor layers route specific paths to the database instead. +User-facing parameter schemas for the most commonly configured tools. -```mermaid -flowchart TD - CALL["read_file / write_file"] --> INT1{"ContextFile
Interceptor?"} - INT1 -->|Handled| DB1[("DB: agent_context_files
/ user_context_files")] - INT1 -->|Not handled| INT2{"Memory
Interceptor?"} - INT2 -->|Handled| DB2[("DB: memory_documents")] - INT2 -->|Not handled| SBX{"Sandbox enabled?"} - SBX -->|Yes| DOCKER["Docker container"] - SBX -->|No| HOST["Host filesystem
resolvePath -> os.ReadFile / WriteFile"] -``` - -### ContextFileInterceptor -- 7 Routed Files - -| File | Description | -|------|-------------| -| `SOUL.md` | Agent personality and behavior | -| `IDENTITY.md` | Agent identity information | -| `AGENTS.md` | Sub-agent definitions | -| `TOOLS.md` | Tool usage guidance | -| `USER.md` | Per-user preferences and context | -| `BOOTSTRAP.md` | First-run instructions (write empty = delete row) | - -### Routing by Agent Type - -```mermaid -flowchart TD - FILE{"Path is one of
7 context files?"} -->|No| PASS["Pass through to disk"] - FILE -->|Yes| TYPE{"Agent type?"} - TYPE -->|open| USER_CF["user_context_files
fallback: agent_context_files"] - TYPE -->|predefined| PRED{"File = USER.md?"} - PRED -->|Yes| USER_CF2["user_context_files"] - PRED -->|No| AGENT_CF["agent_context_files"] -``` - -- **Open agents**: All 7 files are per-user. If a user file does not exist, the agent-level template is returned as fallback. -- **Predefined agents**: Only `USER.md` is per-user. All other files come from the agent-level store. - -### MemoryInterceptor - -Routes `MEMORY.md`, `memory.md`, and `memory/*` paths. Per-user results take priority with a fallback to global scope. Writing a `.md` file automatically triggers `IndexDocument()` (chunking + embedding). - -### PathDenyable Interface - -Tools that access the filesystem implement the `PathDenyable` interface, allowing specific path prefixes to be denied at runtime: - -```go -type PathDenyable interface { - DenyPaths(...string) -} -``` - -All four filesystem tools (`read_file`, `write_file`, `list_files`, `edit_file`) implement it. `list_files` additionally filters denied directories from its output entirely -- the agent doesn't even know the directory exists. Used to prevent agents from accessing `.goclaw` directories within workspaces. - -### Workspace Context Injection - -Filesystem and shell tools read their workspace from `ToolWorkspaceFromCtx(ctx)`, which is injected by the agent loop based on the current user and agent. This enables per-user workspace isolation without changing any tool code. Falls back to the struct field for backward compatibility. - -### Path Security - -`resolvePath()` joins relative paths with the workspace root, applies `filepath.Clean()`, and verifies the result with `HasPrefix()`. This prevents path traversal attacks (e.g., `../../../etc/passwd`). The extended `resolvePathWithAllowed()` permits additional prefixes for skills directories. - ---- - -## 4. Shell Execution - -The `exec` tool allows the LLM to run shell commands, with multiple defense layers. - -### Credentialed CLI Tools - -**Direct Exec Mode** allows secure credential injection for CLI tools without exposing credentials via shell. Credentials are auto-injected as environment variables directly into the child process (no shell involved). - -**How it works:** -1. **Credential lookup** — Administrator configures binary → encrypted env vars in `secure_cli_binaries` table -2. **Shell operator detection** — Blocks unsafe command chaining (`;`, `|`, `&&`, `||`, `>`, `<`, `$()`, backticks) -3. **Path verification** — Binary is resolved to absolute path and matched against configured path -4. **Per-binary deny check** — Optional regex patterns block specific arguments (e.g., `auth\s+`, `ssh-key`) -5. **Direct exec** — Command runs as `exec.CommandContext(binary, args...)` with credentials in env - -**Available presets:** `gh`, `gcloud`, `aws`, `kubectl`, `terraform` - -**Security layers:** -- **No shell** — Direct exec prevents shell command injection -- **Path verification** — Binary spoofing (e.g., `./gh` in workspace) is blocked -- **Per-binary deny** — Admins can block sensitive operations per CLI -- **Output scrubbing** — Credential values registered for automatic redaction - -**Configuration (JSON in `secure_cli_binaries` table):** +### `web_search` tenant config shape ```json { - "binary_name": "gh", - "encrypted_env": {"GH_TOKEN": "ghp_..."}, - "deny_args": ["auth\\s+", "ssh-key"], - "deny_verbose": ["--verbose", "-v"], - "timeout_seconds": 30, - "tips": "GitHub CLI. Available: gh api, gh repo, gh issue, etc." + "provider_order": ["brave", "exa"], + "brave": { "enabled": true, "max_results": 5 }, + "exa": { "enabled": false }, + "duckduckgo": { "enabled": false } } ``` +- `provider_order`: provider preference list; unknown names silently ignored. +- Per-provider: `enabled` (bool) + `max_results` (int). DuckDuckGo `enabled: false` is ignored — it is always the final fallback. +- API keys go in `config_secrets`, never in settings JSON. -#### Grant enforcement (hard deny) - -Registering a binary is not the same as letting every agent exec it. A registered -binary with `is_global=false` **requires a row in `secure_cli_agent_grants`** -for the calling agent; otherwise shell exec is denied before any process runs. -Closes the class of bypass where an ungranted agent could still invoke the -binary via host fallthrough and pick up inherited env (`$GH_TOKEN`, etc.) or -on-disk OAuth state (`~/.config/gh`). - -- **Enforcement site:** `internal/tools/shell.go` — gate runs after normalization - and before exec approval, keyed on `store.SecureCLIStore.IsRegisteredBinary`. -- **Name matching:** `filepath.Base(strings.TrimSpace(strings.ToLower(name)))` — - case-insensitive (macOS APFS), directory-prefix stripped, trims whitespace. -- **Shell wrapper unwrap:** detects and peels `sh -c / bash -c / zsh -c / dash -c`, - leading `env K=V ...`, `nohup`, `stdbuf`, `timeout ...` up to depth 3. Commands - nested deeper than 3 wrappers are rejected as adversarial. -- **Global binaries:** `is_global=true` skips the gate (no grant needed). Shared - utilities without per-agent scoping. -- **Fail-CLOSED:** if the grant lookup errors (DB down, timeout) the exec is - denied with a retry message. 2s per-lookup context timeout. -- **Env scrubbing on fall-through:** when a command bypasses the credentialed - path and runs on the host, the child process env is scrubbed of credential - keys (static deny list + dynamic keys from every registered binary of the - same tenant) before spawn. Prevents `$GH_TOKEN` leaking into a non-gh command. -- **Log events:** - - `security.credentialed_binary_denied` — agent attempted an ungranted - registered binary. Fields: `binary, wrapper, agent_id, tenant_id, command_prefix`. - - `security.credentialed_binary_gate_error` — lookup failed; exec denied. - - `security.credentialed_binary_wrapper_too_deep` — >3 shell wrappers. -- **Subagents:** subagent `ExecTool`s are wired with the same `SecureCLIStore` - (`cmd/gateway_agents.go` → `buildSubagentToolsRegistry`) so a parent cannot - bypass the gate by delegating the exec to a spawned child. - ---- - -### Deny Patterns - -| Category | Blocked Patterns | -|----------|------------------| -| Destructive file ops | `rm -rf`, `del /f`, `rmdir /s` | -| Disk destruction | `mkfs`, `dd if=`, `> /dev/sd*` | -| System control | `shutdown`, `reboot`, `poweroff` | -| Fork bombs | `:(){ ... };:` | -| Remote code exec | `curl \| sh`, `wget -O - \| sh` | -| Reverse shells | `/dev/tcp/`, `nc -e` | -| Eval injection | `eval $()`, `base64 -d \| sh` | - -### Approval Workflow - -```mermaid -flowchart TD - CMD["Shell Command"] --> DENY{"Matches deny
pattern?"} - DENY -->|Yes| BLOCK["Blocked by safety policy"] - DENY -->|No| APPROVAL{"Approval manager
configured?"} - APPROVAL -->|No| EXEC["Execute on host"] - APPROVAL -->|Yes| CHECK{"CheckCommand()"} - CHECK -->|deny| BLOCK2["Command denied"] - CHECK -->|allow| EXEC - CHECK -->|ask| REQUEST["Request approval
(2-minute timeout)"] - REQUEST -->|allow-once| EXEC - REQUEST -->|allow-always| ADD["Add to dynamic allowlist"] --> EXEC - REQUEST -->|deny / timeout| BLOCK3["Command denied"] -``` - -### Sandbox Routing - -When a sandbox manager is configured and a `sandboxKey` exists in context, commands execute inside a Docker container. The host working directory maps to `/workspace` in the container. Host timeout is 60 seconds; sandbox timeout is 300 seconds. If sandbox returns `ErrSandboxDisabled`, execution falls back to the host. - ---- - -## 4a. Tool Capabilities & Metadata (v3) - -Each tool is annotated with structured metadata describing capabilities, group membership, and requirements: - -```go -type ToolCapability string - -const ( - CapReadOnly ToolCapability = "read-only" // no side effects - CapMutating ToolCapability = "mutating" // modifies state - CapAsync ToolCapability = "async" // returns immediately - CapMCPBridged ToolCapability = "mcp-bridged" // proxied to external MCP -) - -type ToolMetadata struct { - Name string - Capabilities []ToolCapability - Group string // "fs", "web", "runtime", "memory", "team", etc. - RequiresWorkspace bool - ProviderHints map[string]any -} -``` - -**Default capability inference** (based on tool name): -- **Read-only**: `read_file`, `list_files`, `memory_search`, `memory_expand`, `web_fetch`, `skill_search`, `knowledge_graph_search`, `sessions_history`, `datetime`, `web_search`, `read_image`, `read_audio`, `read_video`, `read_document` -- **Async**: `spawn` (subagent spawning) -- **Mutating**: All other tools (write, exec, message, team tasks, etc.) - -Metadata enables capability-aware tool filtering (e.g., restrict agents to read-only tools, gate async operations). - ---- - -## 5. Policy Engine - -The policy engine determines which tools the LLM can use through a 7-step allow pipeline followed by deny subtraction and additive alsoAllow. - -```mermaid -flowchart TD - ALL["All registered tools"] --> S1 - - S1["Step 1: Global Profile
full / minimal / coding / messaging"] --> S2 - S2["Step 2: Provider Profile Override
byProvider.{name}.profile"] --> S3 - S3["Step 3: Global Allow List
Intersection with allow list"] --> S4 - S4["Step 4: Provider Allow Override
byProvider.{name}.allow"] --> S5 - S5["Step 5: Agent Allow
Per-agent allow list"] --> S6 - S6["Step 6: Agent + Provider Allow
Per-agent per-provider allow"] --> S7 - S7["Step 7: Group Allow
Group-level allow list"] - - S7 --> DENY["Apply Deny Lists
Global deny, then Agent deny"] - DENY --> ALSO["Apply AlsoAllow
Global alsoAllow, Agent alsoAllow
(additive union)"] - ALSO --> SUB{"Subagent?"} - SUB -->|Yes| SUBDENY["Apply subagent deny list
+ leaf deny list if at max depth"] - SUB -->|No| FINAL["Final tool list sent to LLM"] - SUBDENY --> FINAL -``` - -### Profiles - -| Profile | Tools Included | -|---------|---------------| -| `full` | All registered tools (no restriction) | -| `coding` | `group:fs`, `group:runtime`, `group:sessions`, `group:memory`, `group:web`, `group:knowledge`, `group:media_gen`, `group:media_read`, `group:skills` | -| `messaging` | `group:messaging`, `group:web`, `group:sessions`, `group:media_read`, `skill_search` | -| `minimal` | `session_status` only | - -### Tool Groups - -| Group | Members | -|-------|---------| -| `fs` | `read_file`, `write_file`, `list_files`, `edit`, `search`, `glob` | -| `runtime` | `exec`, `credentialed_exec` | -| `web` | `web_search`, `web_fetch` | -| `memory` | `memory_search`, `memory_get` | -| `sessions` | `sessions_list`, `sessions_history`, `sessions_send`, `spawn`, `session_status` | -| `knowledge` | `knowledge_graph_search`, `skill_search` | -| `automation` | `cron`, `datetime` | -| `messaging` | `message`, `create_forum_topic` | -| `delegation` | ~~`delegate`~~ (removed) | -| `teams` | `team_tasks`, `team_message` | -| `media_gen` | `create_image`, `create_audio`, `create_video`, `tts` | -| `media_read` | `read_image`, `read_audio`, `read_document`, `read_video` | -| `skills` | `use_skill`, `publish_skill` | -| `goclaw` | All native tools (composite group) | - -Groups can be referenced in allow/deny lists with the `group:` prefix (e.g., `group:fs`). The MCP manager dynamically registers `mcp` and `mcp:{serverName}` groups at runtime. - -### Per-Request Tool Allow List - -In addition to the static policy pipeline, channels can inject a per-request tool allow list via message metadata. For example, Telegram forum topics can restrict tools per-topic (see [05-channels-messaging.md](./05-channels-messaging.md) Section 5). The allow list is applied as a final intersection step after the policy pipeline completes. - ---- - -## 6. Subagent System - -Subagents are child agent instances spawned to handle parallel or complex tasks. They run in background goroutines with restricted tool access. - -### Lifecycle - -```mermaid -stateDiagram-v2 - [*] --> Spawning: spawn(task, label) - Spawning --> Running: Limits pass
(depth, concurrent, children) - Spawning --> Rejected: Limit exceeded - - Running --> Completed: Task finished - Running --> Failed: LLM error - Running --> Cancelled: cancel / steer / parent abort - - Completed --> Archived: After 60 min - Failed --> Archived: After 60 min - Cancelled --> Archived: After 60 min -``` - -### Limits - -| Constraint | Default | Description | -|------------|---------|-------------| -| MaxConcurrent | 8 | Total running subagents across all parents | -| MaxSpawnDepth | 1 | Maximum nesting depth | -| MaxChildrenPerAgent | 5 | Maximum children per parent agent | -| ArchiveAfterMinutes | 60 | Auto-archive completed tasks | -| Max iterations | 20 | LLM loop iterations per subagent | - -### Subagent Actions - -| Action | Behavior | -|--------|----------| -| `spawn` (async) | Launch in goroutine, return immediately with acceptance message | -| `run` (sync) | Block until subagent completes, return result directly | -| `list` | List all subagent tasks with status | -| `cancel` | Cancel by specific ID, `"all"`, or `"last"` | -| `steer` | Cancel + settle 500ms + respawn with new message | - -### Tool Deny Lists - -| List | Denied Tools | -|------|-------------| -| Always denied (all depths) | `gateway`, `agents_list`, `whatsapp_login`, `session_status`, `cron`, `memory_search`, `memory_get`, `sessions_send` | -| Leaf denied (max depth) | `sessions_list`, `sessions_history`, `sessions_spawn`, `spawn`, `subagent` | - -Results are announced back to the parent agent via the message bus, optionally batched through an AnnounceQueue with debouncing. - ---- - -## 7. Delegation System (v3) - -The `delegate` tool enables inter-agent task delegation using agent_links for permission management. Unlike subagents (anonymous clones), delegation crosses agent boundaries to fully independent agents with distinct identities, tools, providers, and context files. - -### Delegate Tool - -Invoked with agent_key, task, mode (async/sync), and optional timeout: - +### `web_fetch` tenant config shape ```json { - "agent_key": "data-analyst", - "task": "Analyze Q3 sales trends", - "mode": "sync", - "timeout": 300 + "policy": "allow_all", + "allowed_domains": ["github.com", "*.example.com"], + "blocked_domains": ["malicious.com"] } ``` -**Modes:** -- **async** (default) — Fire-and-forget; result announced via message bus as `delegate:{delegationID}`. No blocking. -- **sync** — Block up to timeout seconds; return result directly. Max timeout: 600s. - -### Permission Model - -Delegation requires an `agent_link` from caller → target. Link status checked at runtime: - -```go -allowed, err := links.CanDelegate(ctx, fromAgentID, toAgentID) +### `tts` tenant config shape +```json +{ + "primary": "elevenlabs", + "default_voice_id": "pMsXgVXv3BLzUgSXRplE", + "default_model": "eleven_flash_v2_5" +} ``` +- `primary`: provider (`elevenlabs`, `openai`, `edge`, `minimax`). +- Per-agent overrides: `agent.other_config.tts_voice_id`, `agent.other_config.tts_model_id`. -If link missing or disabled, returns *"no delegation link from current agent to '{targetKey}'"*. - -### Event Emission - -Emits `delegate.sent` event with delegation ID, from/to agents, task description, and mode. Enables audit trails and async result routing. - -### Coordination with Teams - -Delegation is independent of teams. However, team leads often delegate to team members rather than spawning. For structured workflows with shared task board, use agent teams instead (see [11-agent-teams.md](11-agent-teams.md)). - ---- - -## 8. Agent Teams - -Teams add a shared coordination layer on top of delegation: a task board for parallel work and a mailbox for peer-to-peer communication. - -### Architecture - -An admin creates a team via the dashboard, assigns a **lead** and **members**. When a user messages the lead: -1. The lead sees `TEAM.md` in its system prompt (teammate list + role) -2. The lead posts tasks to the board -3. Teammates are activated, claim tasks, and work in parallel -4. Teammates message each other for coordination -5. The lead synthesizes results and replies to the user - -### Task Board (`team_tasks` tool) - -| Action | Description | -|--------|-------------| -| `list` | List tasks (filter: active/completed/all, order: priority/newest) | -| `create` | Create task with subject, description, priority, blocked_by | -| `claim` | Atomically claim a pending task (race-safe via row-level lock) | -| `complete` | Mark task done with result; auto-unblocks dependent tasks | -| `search` | FTS search over task subject + description | - -### Mailbox (`team_message` tool) - -| Action | Description | -|--------|-------------| -| `send` | Send direct message to a specific teammate | -| `broadcast` | Send message to all teammates | -| `read` | Read unread messages | - -### Lead-Centric Design - -Only the lead gets `TEAM.md` in its system prompt. Teammates discover context on demand through tools -- no wasted tokens on idle agents. When a teammate message arrives, the message itself carries context (e.g., *"[Team message from lead]: please claim a task from the board."*). - -### Message Routing - -Teammate results route through the message bus with a `"teammate:"` prefix. The consumer publishes the outbound response so the lead (and ultimately the user) sees the result. - ---- - -## 10. MCP Bridge Tools - -GoClaw integrates with Model Context Protocol (MCP) servers via `internal/mcp/`. The MCP Manager connects to external tool servers and registers their tools in the tool registry with a configurable prefix. - -### Transports - -| Transport | Description | -|-----------|-------------| -| `stdio` | Launch process with command + args, communicate via stdin/stdout | -| `sse` | Connect to SSE endpoint via URL | -| `streamable-http` | Connect to HTTP streaming endpoint | - -### Behavior - -- Health checks run every 30 seconds per server -- Reconnection uses exponential backoff (2s initial, 60s max, 10 attempts) -- Tools are registered with a prefix (e.g., `mcp_servername_toolname`) -- Dynamic tool group registration: `mcp` and `mcp:{serverName}` groups - -### Access Control - -MCP server access is controlled through per-agent and per-user grants stored in PostgreSQL. - -```mermaid -flowchart TD - REQ["LoadForAgent(agentID, userID)"] --> QUERY["ListAccessible()
JOIN mcp_servers + agent_grants + user_grants"] - QUERY --> SERVERS["Accessible servers list
(with ToolAllow/ToolDeny per grant)"] - SERVERS --> CONNECT["Connect each server
(stdio/sse/streamable-http)"] - CONNECT --> DISCOVER["ListTools() from server"] - DISCOVER --> FILTER["filterTools()
1. Remove tools in deny list
2. Keep only tools in allow list (if set)
3. Deny takes priority over allow"] - FILTER --> REGISTER["Register filtered tools
in tool registry"] -``` - -**Grant types**: - -| Grant | Table | Scope | Fields | -|-------|-------|-------|--------| -| Agent grant | `mcp_agent_grants` | Per server + agent | `tool_allow`, `tool_deny` (JSONB arrays), `config_overrides`, `enabled` | -| User grant | `mcp_user_grants` | Per server + user | `tool_allow`, `tool_deny` (JSONB arrays), `enabled` | - -**Access request workflow**: Users can request access to MCP servers. Admins review and approve or reject. On approval, a corresponding grant is created transactionally. - -```mermaid -flowchart LR - USER["CreateRequest()
scope: agent/user
status: pending"] --> ADMIN["ReviewRequest()
approve or reject"] - ADMIN -->|approved| GRANT["Create agent/user grant
with requested tool_allow"] - ADMIN -->|rejected| DONE["Request closed"] -``` - ---- - -## 11. Custom Tools - -Define shell-based tools at runtime via the HTTP API -- no recompile or restart needed. Custom tools are stored in the `custom_tools` PostgreSQL table and loaded dynamically into the agent's tool registry. - -### Lifecycle - -```mermaid -flowchart TD - subgraph Startup - GLOBAL["LoadGlobal()
Fetch all tools with agent_id IS NULL
Register into global registry"] - end - - subgraph "Per-Agent Resolution" - RESOLVE["LoadForAgent(globalReg, agentID)"] --> CHECK{"Agent has
custom tools?"} - CHECK -->|No| USE_GLOBAL["Use global registry as-is"] - CHECK -->|Yes| CLONE["Clone global registry
Register per-agent tools
Return cloned registry"] - end - - subgraph "Cache Invalidation" - EVENT["cache:custom_tools event"] --> RELOAD["ReloadGlobal()
Unregister old, register new"] - RELOAD --> INVALIDATE["AgentRouter.InvalidateAll()
Force re-resolve on next request"] - end -``` - -### Scope - -| Scope | `agent_id` | Behavior | -|-------|-----------|----------| -| Global | `NULL` | Available to all agents | -| Per-agent | UUID | Available only to the specified agent | - -### Command Execution - -1. **Template rendering**: `{{.key}}` placeholders replaced with shell-escaped argument values (single-quote wrapping with embedded quote escaping) -2. **Deny pattern check**: Same deny patterns as the `exec` tool (blocks `curl|sh`, reverse shells, etc.) -3. **Execution**: `sh -c ` with configurable timeout (default 60s) and optional working directory -4. **Environment variables**: Stored encrypted (AES-256-GCM) in the database, decrypted at runtime and injected into the command environment - -### JSON Config Example - +### Custom tool definition ```json { "name": "dns_lookup", @@ -698,290 +233,287 @@ flowchart TD } ``` ---- - -## 12. Credential Scrubbing - -Tool output is automatically scrubbed before being returned to the LLM. Enabled by default in the registry. - -### Static Patterns - -| Type | Pattern | -|------|---------| -| OpenAI | `sk-[a-zA-Z0-9]{20,}` | -| Anthropic | `sk-ant-[a-zA-Z0-9-]{20,}` | -| GitHub PAT | `ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_` + 36 alphanumeric characters | -| AWS | `AKIA[A-Z0-9]{16}` | -| Generic key-value | `(api_key\|token\|secret\|password\|bearer\|authorization)[:=]value` (case-insensitive) | -| Connection strings | `postgres://`, `mysql://`, `mongodb://`, `redis://` patterns | -| Env var patterns | `KEY=`, `SECRET=`, `CREDENTIAL=`, `DSN=`, `VIRTUAL_*=` patterns | -| Long hex strings | 64+ character hex strings (potential encryption keys) | - -All matches are replaced with `[REDACTED]`. - -### Dynamic Scrubbing - -In addition to static patterns, values can be registered at runtime for scrubbing. This is used for server IPs and other deployment-specific secrets that are not known at compile time. The `AddDynamicScrubValues()` function thread-safely adds values to a scrub list that is checked alongside static patterns. - ---- - -## 13. Rate Limiter - -The tool registry supports per-session rate limiting via `ToolRateLimiter`. When configured, each `ExecuteWithContext` call checks `rateLimiter.Allow(sessionKey)` before tool execution. Rate-limited calls receive an error result without executing the tool. - ---- - -## 14. Per-Tenant Tool Configuration - -Tools expose configuration via a **4-tier overlay** resolved at Execute time. Callers read the merged view through `tools.BuiltinToolSettingsFromCtx(ctx)` — no tool-interface changes, no per-tenant tool instances. - -### Resolution order (high → low) - -``` -Execute(ctx, params) → BuiltinToolSettingsFromCtx(ctx) - 1. Per-agent override (agents.builtin_tool_settings) - 2. Tenant override (builtin_tool_tenant_configs.settings) - 3. Global default (builtin_tools.settings) - 4. Hardcoded default (embedded in Go) -``` - -Merge is tool-name keyed: each tool's config is an independent JSON object; the highest-priority tier with a non-nil value wins (no deep merge). Resolver lives in `internal/tools/context_keys.go`. - -### Opt-in for tool authors - -Reading `BuiltinToolSettingsFromCtx(ctx)` auto-participates in the overlay: - -```go -func (t *MyTool) Execute(ctx context.Context, params MyParams) (*Result, error) { - settings := tools.BuiltinToolSettingsFromCtx(ctx) - cfg := t.defaults - if raw, ok := settings["my_tool"]; ok { - _ = json.Unmarshal(raw, &cfg) // fall back to defaults on error - } - return t.doWork(ctx, cfg, params) +### Credentialed binary config (`secure_cli_binaries` table) +```json +{ + "binary_name": "gh", + "encrypted_env": {"GH_TOKEN": "ghp_..."}, + "deny_args": ["auth\\s+", "ssh-key"], + "timeout_seconds": 30, + "tips": "GitHub CLI. Available: gh api, gh repo, gh issue, etc." } ``` - -### Secret vs non-secret split - -- **Non-secret** → `builtin_tool_tenant_configs.settings` (tool authors own the schema): provider priorities, `max_results`, `allowed_domains`, UI-tunable params -- **Secret** → `config_secrets` (already tenant-scoped): API keys, tokens - -The overlay does **not** validate the split. Tool authors must never put credentials in `settings`. Current adopters: `web_search` (Exa/Tavily/Brave/DuckDuckGo provider chain), `create_image`/`read_image`/`create_audio`/`read_audio` (Gemini/OpenAI/Qwen chain), `web_fetch` (allowlist policy), `knowledge_graph_search` (traversal config). - -### Cache invalidation - -Changes propagate via `bus.TopicCacheBuiltinTools` with tenant scoping (Phase 1 refactor, commit `d77a3664`). Handler emits `emitCacheInvalidate(toolName, tenantID)` → subscribers wipe cached loop contexts → next agent turn re-resolves. - -### Security: master-scope guard pattern - -Writes to global tables (e.g. `PUT /v1/tools/builtin/{name}` touching `builtin_tools`) require **master tenant scope**, enforced by `http.requireMasterScope(w, r)` symmetric to `requireTenantAdmin`. Non-master tenant admins must use the `/tenant-config` endpoint. Same predicate (`store.IsMasterScope(ctx)`) guards WS `config.*`. See commit `6d7473b5` (Phase 0b hotfix) + `plans/260412-0726-tenant-tool-config-refactor/phase-0b-tenant-scope-hotfix.md`. - -```go -func (h *MyHandler) handleWrite(w http.ResponseWriter, r *http.Request) { - if !requireMasterScope(w, r) { // reject non-master; 403 written - return - } - // ... global mutation -} -``` - -Tenant-scoped writes use the symmetric `requireTenantAdmin(w, r, tenantStore)` instead. +Available presets: `gh`, `gcloud`, `aws`, `kubectl`, `terraform`. --- -## 14. Per-Tenant Tool Configuration (4-Tier Overlay) +## 6. Interception Layer -Tenant admins can override tool config without affecting other tenants or master. Overlay priority (most specific wins): +### Virtual Filesystem Routing + +Filesystem calls are intercepted before hitting host disk. Two interceptors route specific paths to the database. + +```mermaid +flowchart TD + CALL["read_file / write_file"] --> INT1{"ContextFile
Interceptor?"} + INT1 -->|Handled| DB1[("DB: agent_context_files
/ user_context_files")] + INT1 -->|Not handled| INT2{"Memory
Interceptor?"} + INT2 -->|Handled| DB2[("DB: memory_documents")] + INT2 -->|Not handled| SBX{"Sandbox enabled?"} + SBX -->|Yes| DOCKER["Docker container"] + SBX -->|No| HOST["Host filesystem (workspace-scoped)"] +``` + +**ContextFileInterceptor** — routes 6 known bootstrap files (`SOUL.md`, `IDENTITY.md`, `AGENTS.md`, `TOOLS.md`, `USER.md`, `BOOTSTRAP.md`) to the DB instead of disk. Routing depends on agent type: +- **Open agents**: all files are per-user; agent-level file is the fallback template. +- **Predefined agents**: only `USER.md` is per-user; all others come from agent-level store. + +**MemoryInterceptor** — routes `MEMORY.md`, `memory.md`, and `memory/*` paths to `memory_documents`. Writing a `.md` file automatically triggers chunking + embedding. + +### Path Security + +`resolvePath()` joins relative paths with the workspace root, applies `filepath.Clean()`, and verifies the result starts with the workspace prefix — preventing path traversal attacks. + +### Workspace Context + +Filesystem and shell tools read their workspace from context (injected by the agent loop per user and agent). This enables per-user workspace isolation without tool code changes. + +### Policy Engine + +The policy engine determines which tools reach the LLM through a layered allow pipeline: + +```mermaid +flowchart TD + ALL["All registered tools"] --> S1 + S1["1. Global Profile (full/minimal/coding/messaging)"] --> S2 + S2["2. Provider Profile Override"] --> S3 + S3["3. Global Allow List (intersection)"] --> S4 + S4["4. Provider Allow Override"] --> S5 + S5["5. Agent Allow"] --> S6 + S6["6. Agent + Provider Allow"] --> S7 + S7["7. Group Allow"] + S7 --> DENY["Apply Deny Lists (global → agent)"] + DENY --> ALSO["Apply AlsoAllow (additive union)"] + ALSO --> SUB{"Subagent?"} + SUB -->|Yes| SUBDENY["Apply subagent deny list
+ leaf deny at max depth"] + SUB -->|No| FINAL["Final tool list → LLM"] + SUBDENY --> FINAL +``` + +**Profiles:** + +| Profile | Tool Set | +|---|---| +| `full` | All registered tools | +| `coding` | `group:fs`, `group:runtime`, `group:sessions`, `group:memory`, `group:web`, `read_image`, `create_image`, `skill_search` | +| `messaging` | `group:messaging`, `group:web`, sessions read, `read_image`, `skill_search` | +| `minimal` | `session_status` only | + +**Tool Groups** (reference `group:` in allow/deny lists): + +| Group | Members | +|---|---| +| `fs` | `read_file`, `write_file`, `list_files`, `edit` | +| `runtime` | `exec` | +| `web` | `web_search`, `web_fetch` | +| `memory` | `memory_search`, `memory_get` | +| `sessions` | `sessions_list`, `sessions_history`, `sessions_send`, `spawn`, `session_status` | +| `automation` | `cron` | +| `messaging` | `message`, `create_forum_topic`, `list_group_members` | +| `team` | `team_tasks` | +| `goclaw` | All native built-in tools (composite) | + +MCP groups (`mcp`, `mcp:{serverName}`) are registered dynamically at connection time. + +**Per-request allow list** — channels can inject a final intersection step via message metadata (e.g. Telegram forum topics restrict tools per topic). + +--- + +## 7. Custom Tools + +Custom tools are shell-based tools defined at runtime via the HTTP API — no recompile or restart needed. Stored in the `custom_tools` table. + +**Authoring fields:** + +| Field | Required | Purpose | +|---|---|---| +| `name` | yes | Tool identifier (lowercase, underscores) | +| `description` | yes | Shown to the LLM as tool description | +| `parameters` | yes | JSON Schema object defining tool params | +| `command` | yes | Shell command with `{{.param}}` Go template placeholders | +| `timeout_seconds` | no | Execution timeout (default 60s) | +| `env` | no | Encrypted environment variables injected at runtime | +| `enabled` | no | Toggle without deleting (default true) | + +**Execution:** Template placeholders are rendered with shell-escaped argument values, then run via `sh -c`. The same deny-pattern check as the `exec` tool applies — no reverse shells, no `curl | sh`, etc. + +**Scope:** +- `agent_id = NULL` → global; available to all agents. +- `agent_id = ` → available only to that agent. + +--- + +## 8. Credential Scrubbing + +Tool output is automatically scrubbed before being returned to the LLM or the user. The scrubber intercepts both the `ForLLM` and `ForUser` result fields. + +**Static patterns** cover well-known credential shapes including provider API keys, cloud access keys, generic key-value assignments, connection string URIs, and long hex strings. All matches are replaced with `[REDACTED]`. + +**Dynamic scrubbing** — values can be registered at runtime (e.g. server IPs, deployment-specific tokens). Thread-safe; checked alongside static patterns on every tool result. + +The exact patterns are intentionally not published here (defense-in-depth). The scrubber is always enabled in the registry by default. + +--- + +## 9. Per-Tenant Config (4-Tier Overlay) + +Tools that support tenant configuration resolve a merged settings map at execution time. Priority order (most specific wins): ``` -Tool call Execute(ctx, params) +Tool Execute(ctx, params) ↓ -BuiltinToolSettingsFromCtx(ctx) returns merged map: - 1. Per-agent override (agents.builtin_tool_settings; reserved for future) - 2. Tenant override (builtin_tool_tenant_configs.settings column) - 3. Global default (builtin_tools.settings column, resolver-loaded) - 4. Hardcoded fallback (tool internal default when no entry exists) +Merged settings map: + 1. Per-agent override (agents.builtin_tool_settings) + 2. Tenant override (builtin_tool_tenant_configs.settings) + 3. Global default (builtin_tools.settings) + 4. Hardcoded fallback (tool internal defaults) ``` -Each tool that needs tenant-aware config calls `BuiltinToolSettingsFromCtx(ctx)` and unmarshals its settings key. Merge happens at tool-name level: tenant entry for `web_search` wins wholesale over global, no deep field-level merge. - -**Opt-in pattern** — Any tool can participate by: -```go -func (t *MyTool) Execute(ctx context.Context, params MyParams) (*Result, error) { - settings := tools.BuiltinToolSettingsFromCtx(ctx) - var cfg MyToolConfig = t.defaults - if raw, ok := settings["my_tool"]; ok { - if err := json.Unmarshal(raw, &cfg); err != nil { - slog.Warn("my_tool: invalid tenant settings, using defaults", "error", err) - } - } - return t.doWork(ctx, cfg, params) -} -``` - -### web_search Configuration - -**Overview:** web_search is tenant-scoped only (as of v3.2, issue #952). Config is edited exclusively via `/tools/builtin/web_search` UI (or `/v1/tools/builtin/web_search/tenant-config` HTTP API for automation). - -**Key behaviors:** -- Supports provider chain: Exa, Tavily, Brave, DuckDuckGo -- Tenant admins specify `provider_order` to prioritize which providers to try (in order) -- Per-provider `enabled` flag gates usage; unknown providers are silently skipped -- **DuckDuckGo is always appended as the last-resort fallback regardless of tenant `enabled` setting — tenant admins cannot fully disable web_search.** -- API keys stored in `config_secrets` table (encrypted, tenant-scoped); never in `settings` JSON -- Defaults to DuckDuckGo only if no other provider keys are configured - -**Configuration path (old, removed):** `config.Tools.Web.*` in `config.json5` — no longer supported. Use UI only. - -**Migration:** Inline API keys from `config.json5` and legacy `builtin_tool_tenant_configs.settings` blobs are auto-migrated to `config_secrets` on first server startup (data hook 055). - -### Schema Contracts - -#### web_search (tenant override shape) -```json -{ - "provider_order": ["brave", "exa"], - "brave": { "enabled": true, "max_results": 5 }, - "exa": { "enabled": false }, - "duckduckgo": { "enabled": false } -} -``` - -**Fields:** -- `provider_order`: List of provider names in preference order (unknown names silently ignored). Optional. -- Per-provider object: `enabled` (bool, optional) + `max_results` (int, optional). If `enabled: false`, provider is skipped even if API key exists. DuckDuckGo `enabled: false` is ignored (always-on). -- Secrets (API keys): Configure via `/v1/tools/builtin/web_search/tenant-config` PUT endpoint, not in this settings blob - -#### web_fetch (tenant override shape) -```json -{ - "policy": "allow_all", - "allowed_domains": ["github.com", "*.example.com"], - "blocked_domains": ["malicious.com"] -} -``` - -#### tts (tenant override shape) -```json -{ - "primary": "elevenlabs", - "default_voice_id": "pMsXgVXv3BLzUgSXRplE", - "default_model": "eleven_flash_v2_5" -} -``` - -**Fields:** -- `primary`: Provider selection (elevenlabs, openai, edge, minimax) -- `default_voice_id`: Tenant-level voice ID fallback (ElevenLabs); overridable per-agent via `agent.other_config.tts_voice_id` -- `default_model`: Tenant-level model choice (eleven_v3, eleven_flash_v2_5, eleven_multilingual_v2, eleven_turbo_v2_5); overridable per-agent via `agent.other_config.tts_model_id` +Merge is per-tool-name: tenant entry for `web_search` wins wholesale over global default — no deep field merge. Tools that do not read the settings map are unaffected. **Secret vs non-secret split:** -- Non-secret (provider priorities, max_results, allowed_domains): `builtin_tool_tenant_configs.settings` (editable via UI) -- Secret (API keys): `config_secrets` table (encrypted, tenant-scoped) +- Non-secret config (provider priorities, limits, domain policies) → `builtin_tool_tenant_configs.settings` +- Secrets (API keys, tokens) → `config_secrets` table (AES-256-GCM encrypted, tenant-scoped) -Never put credentials in tool settings JSON — backend does not validate. +Never put credentials in the settings JSON blob — backend does not validate the split. -### Tenant Admin Workflow +**Cache invalidation:** settings changes propagate via pub/sub (tenant-scoped). Next agent turn re-resolves automatically. -1. Log in as tenant admin → Settings → Builtin Tools -2. Click gear icon next to a tool → JSON editor opens -3. Edit settings per schema contracts above → Save -4. Changes take effect immediately on next agent turn (cache invalidated via pub/sub) -5. Click "Reset to default" to revert to platform defaults +**Tenant admin workflow:** Settings → Builtin Tools → gear icon → JSON editor → Save. Changes take effect on the next agent turn. "Reset to default" reverts to platform defaults. -### Feature Flag: Tenant-Scoped Singleton Pools +**Master-scope guard:** Writes to global `builtin_tools` table require master tenant scope. Tenant admins use the `/tenant-config` endpoint. Same guard applies on the WS config methods. -`config.Tools.TenantScopedSingletons` (default: false) gates Phase 8 refactor — per-tenant connection pool instances for stateful tools (TTS manager, web_fetch policy validator). Disable for legacy singleton behavior. Enable after 1-week staging smoke test. +Current adopters: `web_search`, `web_fetch`, `tts`, `create_image`, `read_image`, `create_audio`, `read_audio`, `knowledge_graph_search`. --- -## File Reference +## 10. MCP Integration -### Core Infrastructure -| File | Purpose | -|------|---------| -| `internal/tools/{registry,types,policy,result}.go` | Registry, interfaces, PolicyEngine (7-step pipeline), result types | -| `internal/tools/capability.go` | Tool metadata: capabilities (read-only, mutating, async, mcp-bridged), groups, hints | -| `internal/tools/{context_keys,rate_limiter}.go` | Context key definitions, per-session rate limiting | -| `internal/tools/{scrub,scrub_server}.go` | Credential scrubbing and dynamic value registration | +GoClaw integrates with Model Context Protocol (MCP) servers. The MCP Manager connects to external tool servers and registers their tools in the tool registry with a configurable prefix (e.g. `mcp_servername_toolname`). -### Filesystem Tools -| File | Purpose | -|------|---------| -| `internal/tools/filesystem{,_list,_write}.go` | read_file, write_file, list_files, edit tools | -| `internal/tools/edit.go` | edit tool: targeted file modifications | -| `internal/tools/{context_file,memory,workspace}_interceptor.go` | File routing: context files, memory, team workspace | -| `internal/tools/workspace_dir.go` | Workspace directory resolution for team/user context | +**Transports:** -### Runtime & Shell Tools -| File | Purpose | -|------|---------| -| `internal/tools/shell.go` | exec tool: deny patterns, approval workflow, sandbox routing | -| `internal/tools/exec_approval.go` | Approval workflow for restricted shell commands | -| `internal/tools/credentialed_exec.go` | credentialed_exec: direct exec mode with credential injection | -| `internal/tools/credential_{context,presets}.go` | TOOLS.md supplement + preset definitions (gh, gcloud, aws, etc.) | +| Transport | Description | +|---|---| +| `stdio` | Launch subprocess; communicate via stdin/stdout | +| `sse` | Connect to SSE endpoint via URL | +| `streamable-http` | Connect to HTTP streaming endpoint | -### Web Tools -| File | Purpose | -|------|---------| -| `internal/tools/web_search{,_brave,_ddg}.go` | web_search tool (Brave, DuckDuckGo) | -| `internal/tools/web_fetch{,_convert,_convert_handlers,_convert_utils,_hidden}.go` | web_fetch tool: fetch, HTML→Markdown, element handlers | -| `internal/tools/web_shared.go` | Shared web utilities | +**Reliability:** Health checks every 30 seconds. Reconnection via exponential backoff (2s initial, 60s max, 10 attempts). -### Memory, Vault & Knowledge -| File | Purpose | -|------|---------| -| `internal/tools/memory{,_expand}.go` | Memory search (L1) + expand (L2 deep retrieval) | -| `internal/tools/vault_search.go` | Vault search: unified hybrid FTS + vector search across vault/memory/KG | -| `internal/tools/{knowledge_graph,skill_search}.go` | Knowledge graph + skill BM25 search | +**Access control:** -### Delegation & Sessions -| File | Purpose | -|------|---------| -| `internal/tools/delegate_tool.go` | Delegate tool: inter-agent task delegation via agent_links (async/sync) | -| `internal/tools/sessions{,_history,_send}.go` | Session list, history, send tools | -| `internal/tools/subagent{,_spawn_tool,_config,_exec,_control,_tracing}.go` | SubagentManager: spawn, cancel, steer, tracing | +```mermaid +flowchart TD + REQ["LoadForAgent(agentID, userID)"] --> QUERY["Join mcp_servers + agent_grants + user_grants"] + QUERY --> SERVERS["Accessible servers list"] + SERVERS --> CONNECT["Connect (stdio/sse/http)"] + CONNECT --> DISCOVER["ListTools() from server"] + DISCOVER --> FILTER["filterTools()\n1. Remove deny-listed tools\n2. Intersect with allow list (if set)\n3. Deny takes priority"] + FILTER --> REGISTER["Register in tool registry"] +``` -### Media Tools -| File | Purpose | -|------|---------| -| `internal/tools/create_image.go` | create_image tool (OpenAI, Gemini, MiniMax, DashScope) | -| `internal/tools/create_image_{dashscope,minimax}.go` | Provider-specific image generation | -| `internal/tools/create_audio.go` | create_audio tool — delegates to `audio.Manager` for music/SFX (ElevenLabs, MiniMax) | -| `internal/tools/create_video.go` | create_video tool (MiniMax) | -| `internal/tools/tts.go` | tts tool: text-to-speech (OpenAI, ElevenLabs, Edge, MiniMax) | -| `internal/tools/read_{image,audio,video,document}.go` | Media reading tools (vision, transcription, analysis) | -| `internal/tools/read_{audio,video,document}_resolve.go` | Resolve service integrations | -| `internal/tools/read_document_gemini.go` | Gemini file API for documents | -| `internal/tools/gemini_file_api.go` | Google Gemini file API wrapper | -| `internal/tools/media_provider_chain.go` | Media provider routing and fallback chain | +**Grant types:** -### Skills & Content Tools -| File | Purpose | -|------|---------| -| `internal/tools/use_skill.go` | use_skill tool: marker for observability | -| `internal/tools/publish_skill.go` | publish_skill tool: skill registration in database | +| Grant | Table | Scope | +|---|---|---| +| Agent grant | `mcp_agent_grants` | Per server + agent; `tool_allow`, `tool_deny` JSONB arrays | +| User grant | `mcp_user_grants` | Per server + user; `tool_allow`, `tool_deny` JSONB arrays | -### Team Collaboration Tools -| File | Purpose | -|------|---------| -| `internal/tools/team_tool_manager.go` | Shared backend: team cache (5-min TTL), resolution | -| `internal/tools/team_tasks_tool.go` | team_tasks tool: task board operations | -| `internal/tools/team_tasks_{read,mutations,lifecycle,followup}.go` | Task CRUD, state transitions, dependency handling | -| `internal/tools/team_message_tool.go` | team_message tool: mailbox (send, broadcast, read) | -| `internal/tools/team_tool_{cache,dispatch,helpers,validation}.go` | Team tool infrastructure | -| `internal/tools/team_access_policy.go` | Access control policies for team tools | +**Access request workflow:** Users request server access → admins approve/reject → on approval a grant is created transactionally. -### Messaging, Automation & Custom Tools -| File | Purpose | -|------|---------| -| `internal/tools/{message,telegram_forum,cron,datetime}.go` | Messaging, forum topics, cron scheduling, datetime | -| `internal/tools/announce_queue.go` | Message queueing with debouncing | -| `internal/tools/{dynamic_loader,dynamic_tool}.go` | Dynamic/custom tool loading and execution | -| `internal/tools/openai_compat_call.go` | OpenAI-compatible endpoint calling utilities | +--- -### MCP & Infrastructure -| File | Purpose | -|------|---------| -| `internal/mcp/{manager,bridge_tool}.go` | MCP server connections, bridge tool | +## 11. Team Tools + +Teams add a shared coordination layer on top of delegation: a task board and peer communication. + +### Architecture + +When a user messages the team lead: +1. Lead sees `TEAM.md` in its system prompt (teammate list + roles). +2. Lead posts tasks to the shared board. +3. Teammates claim tasks and work in parallel (each has their own session). +4. Lead synthesizes results and replies to the user. + +Only the lead receives `TEAM.md` — teammates discover context through tools, saving tokens on idle agents. + +### Task Board (`team_tasks`) + +Actions: `create`, `list`, `get`, `claim` (race-safe row lock), `complete`, `cancel`, `assign`, `review`, `approve`, `reject`, `comment` (note/blocker types), `progress`, `attach`, `ask_user`, `search`, `update`. + +Completing a task auto-unblocks dependent tasks listed in `blocked_by`. + +### Message Routing + +Teammate results route through the message bus with a `"teammate:"` prefix, surfacing to the lead and ultimately the user. + +--- + +## 12. Media Tools + +Media tools follow a provider-chain pattern: multiple backends are tried in priority order; the first successful response wins. + +**Generation chain** (configurable via per-tenant config): +- Images: OpenAI DALL-E → Gemini → MiniMax → DashScope → BytePlus +- Audio/music: MiniMax → ElevenLabs +- Video: MiniMax → Gemini → BytePlus +- TTS: ElevenLabs → OpenAI → Edge → MiniMax + +**Reading chain:** +- Images: Gemini → Anthropic → OpenRouter → DashScope +- Audio/documents/video: Resolve service or Gemini File API + +Media outputs are written to the agent workspace and referenced by path or URL in the result. File naming is deterministic (content-hashed) to enable deduplication. + +--- + +## 13. Subagent System + +Subagents are child agent instances spawned to handle parallel or complex tasks. + +```mermaid +stateDiagram-v2 + [*] --> Spawning: spawn(task, label) + Spawning --> Running: Limits pass + Spawning --> Rejected: Limit exceeded + Running --> Completed: Task finished + Running --> Failed: LLM error + Running --> Cancelled: cancel / steer / parent abort + Completed --> Archived: After 60 min +``` + +**Limits:** + +| Constraint | Default | +|---|---| +| Max concurrent | 8 (across all parents) | +| Max spawn depth | 1 | +| Max children per agent | 5 | +| Archive after | 60 min | +| Max iterations | 20 per subagent | + +**Actions:** `spawn` (async, returns immediately), `run` (sync, blocks until done), `list`, `cancel` (by ID / `"all"` / `"last"`), `steer` (cancel + respawn with new message). + +Subagents share the same `SecureCLIStore` as their parent — the credentialed binary gate cannot be bypassed by delegating exec to a child. + +--- + +## 14. File Reference + +| Module | Path | Purpose | +|---|---|---| +| Registry & policy | `internal/tools/` | Tool registry, policy engine, rate limiter, scrubber, capability metadata | +| MCP bridge | `internal/mcp/` | MCP server connections, tool bridge, access grants | +| Custom tools | `internal/tools/` (`dynamic_loader.go`, `dynamic_tool.go`) | Runtime shell-based custom tool loading and execution | +| Team tools | `internal/tools/` (`team_tasks_tool.go`, `team_tool_*.go`) | Task board backend, team tool dispatch and cache |