fix(collaboration): harden delegated task isolation (#1486)

* feat(collaboration): isolate delegated artifacts and child runs

Isolate delegated inputs and outputs behind secure artifact exchange lifecycles. Scope Agent Link tasks by tenant and root agent, and enforce delegation spawn-tree boundaries. Add process-wide child-run admission and preserve logical media paths across native, MCP, and sandbox execution.

* fix(collaboration): harden delegated task isolation

Enforce tenant and root-agent task scope across migrations and stores. Add exactly-once async completion delivery, delegated sandbox boundaries, and confined artifact and media recovery across runtime surfaces.

* fix(collaboration): recover interrupted async tasks

* fix(collaboration): normalize persisted child-run status

---------

Co-authored-by: ntduc <ntduc@cpp.ai.vn>
This commit is contained in:
Duc Nguyenandntduc authored and GitHub committed 2026-07-30 14:17:40 +07:00
1 parent 496b7ffce6
commit bb7712a9ff
186 files changed
+18858 -1233

No files matched your search

+2 -2
View File
@@ -115,7 +115,7 @@ flowchart TD
| `internal/skills/` | SKILL.md loader (5-tier hierarchy) + BM25 search + hot-reload via fsnotify |
| `internal/channels/` | Channel manager + adapters: Telegram (forum topics, STT, bot commands), Feishu/Lark (streaming cards, media), Zalo OA, Zalo Personal, Discord, WhatsApp, Slack |
| `internal/mcp/` | MCP server bridge (stdio, SSE, streamable-HTTP transports) |
| `internal/scheduler/` | Lane-based concurrency control (main, subagent, cron, team lanes) with per-session serialization. Per-edition rate limits (`MaxSubagentConcurrent`, `MaxSubagentDepth`) with tenant-scoped concurrency |
| `internal/scheduler/` | Lane-based concurrency control (main, subagent, cron, team lanes) with per-session serialization. Self-spawn and Agent Link callbacks additionally use process-wide child-run admission with per-root limits |
| `internal/memory/` | Memory system (pgvector hybrid search) |
| `internal/subagent/` | Subagent lifecycle: spawn, roster, task persistence (subagent_tasks table), announce queue (producer-consumer), auto-retry, per-edition rate limiting |
| `internal/permissions/` | RBAC policy engine (admin, operator, viewer roles) |
@@ -192,7 +192,7 @@ flowchart TD
- **SSRF hardening (HTTPHandler)**: Caller supplies net.Dialer pinning resolved IP, blocking loopback/link-local/private ranges; no HTTP redirects (CheckRedirect returns ErrUseLastResponse)
- **Auth header encryption**: `Authorization` + other sensitive fields in cfg.Config["headers"] encrypted at rest via AES-256-GCM; decrypted only at HTTP send-time
- **Audit logging**: All hook invocations logged to `hook_executions` table (encrypted, PII-redacted) with dedup_key for idempotency
- **Loop-depth guard (M5)**: SubagentStart checks recursion depth; max 3 levels prevents infinite delegation chains
- **Hook recursion guard (M5)**: Hook dispatch tracks recursive hook re-entry and rejects it after 3 levels. Agent Link delegation uses separate admission, concurrency, and timeout safeguards.
- **Circuit breaker**: Auto-disables hook after 3 consecutive failures in recent window (C4 mitigation)
### Pipeline Integration
+4 -2
View File
@@ -111,7 +111,7 @@ Agents support three orchestration modes that determine which inter-agent tools
- **Use case**: Agent with linked delegate targets
- **Tools available**: `spawn`, `delegate` (dispatch to linked agents)
- **Tools hidden**: `team_tasks`
- **Resolution**: When `agent_links` table has rows with source = this agent
- **Resolution**: When `agent_links` grants this agent an effective outbound delegation edge
### ModeTeam
- **Use case**: Agent in a team (multiple agents collaborating)
@@ -121,7 +121,9 @@ Agents support three orchestration modes that determine which inter-agent tools
**Mode Resolution Priority**: Team > Delegate > Spawn
The system prompt includes relevant details for each mode (delegate targets, team context, shared workspace paths).
The system prompt includes relevant details for each mode. Agent Team runs use
their existing shared workspace. Agent Link runs receive only delegation-scoped
`inputs/` and `outputs/` aliases.
---
+54 -3
View File
@@ -144,7 +144,7 @@ Memory layers: L1 (`memory_search`) returns ranked abstracts; L2 (`memory_expand
| Tool | Description |
|---|---|
| `delegate` | Inter-agent task delegation via agent_links (async/sync modes with timeout) |
| `delegate` | Inter-agent task delegation via `agent_links` (async/sync, optional structured input files, isolated output publication) |
### Teams (`group:team`)
@@ -170,6 +170,19 @@ Memory layers: L1 (`memory_search`) returns ranked abstracts; L2 (`memory_expand
| `read_document` | Extract and analyze documents (PDF, DOCX, images). Supports local-first extraction via pdftotext/pandoc before falling back to cloud vision (opt-in via config) |
| `read_video` | Analyze/transcribe video content |
Image attachments keep absolute paths only in internal `MediaRef` storage.
Model-visible `<media:image>` tags expose an exact media ID plus a logical path
such as `.uploads/photo.jpg` or delegated `inputs/photo.jpg`. Image tools resolve
those values against the active workspace; they never infer another agent's
absolute workspace path. Unknown image IDs fail instead of falling back to an
unrelated image. Exact `media_id` lookup is an in-process convenience; Claude
CLI/MCP uses the workspace-confined logical `path` under signed bridge context.
For Agent Link artifact runs, Claude CLI MCP requests carry a signed delegation
ID and staged-input root. Bridge tools therefore apply the same read-only
`inputs/` policy as in-process tools, and generated media remains inside
`outputs/` until the delegation manifest is validated and atomically published.
### Skills & Content
| Tool | Description |
@@ -613,13 +626,51 @@ stateDiagram-v2
| Constraint | Default |
|---|---|
| Max concurrent | 8 (across all parents) |
| Max concurrent | 20 executing descendants per root agent |
| Max spawn depth | 1 |
| Max children per agent | 5 |
| Archive after | 60 min |
| Max iterations | 20 per subagent |
**Actions:** `spawn` (async, returns immediately), `run` (sync, blocks until done), `list`, `cancel` (by ID / `"all"` / `"last"`), `steer` (cancel + respawn with new message).
`maxSpawnDepth` applies only to an agent's own subagent tree. Agent Link
delegation starts a new tree whose spawn depth, concurrency, fanout, retry, and
model settings are resolved from the target agent.
**Actions:** `spawn(action="spawn", mode="async"|"sync")`, `get`, `list`,
`cancel` (by ID / `"all"` / `"last"`), `steer` (cancel + respawn with new
message), and `wait`.
An accepted async spawn returns its short runtime `task_id` plus a durable
`completion_id`. Terminal status, the full text result, and workspace-safe
logical media descriptors are written to `subagent_tasks` before the parent
announcement is attempted. Announcement delivery retries a bounded number of
times; if the inbound queue is still full or the gateway restarts after
terminal persistence, the owning root agent can recover the result and logical
file paths with:
```text
spawn(action="get", completion_id="<uuid>")
```
The lookup is scoped by tenant and immutable root-agent UUID. A same-key agent
created later, another agent in the tenant, and delegation completion rows
cannot satisfy the lookup.
Terminal persistence uses per-attempt database deadlines and a longer bounded
retry window than announcement delivery. If the database remains unavailable
for the entire window, GoClaw does not falsely mark the announcement as
delivered; a live announcement is still attempted. On the next startup that
can reach the database, every non-terminal completion row left by the previous
process is marked `failed` with an interruption reason, so it cannot remain
`queued` or `running` forever. The single-process gateway retries this
reconciliation before accepting traffic when the database is temporarily
unavailable. Graceful gateway shutdown drains this completion lifecycle before
provider and database teardown.
Self-spawn and Agent Link callbacks also share a process safety cap (Standard:
32; Lite: 2) and a bounded pending queue of 128. Inside an Agent Link artifact
run, async `spawn` and async nested `delegate` are rejected; configured
synchronous descendants complete before the outer artifact is published.
Subagents share the same `SecureCLIStore` as their parent — the credentialed binary gate cannot be bypassed by delegating exec to a child.
+9 -3
View File
@@ -461,13 +461,19 @@ Browser pairing allows web UI clients to authenticate without full admin credent
## 12. Delegation Security
Agent delegation is protected through delegation history tracking and concurrency controls.
Agent delegation is protected through isolated artifact exchange, directional
link permission checks, and bounded child-run admission.
| Control | Scope | Description |
|---------|-------|-------------|
| Per-agent load cap | B (all sources) | `other_config.max_delegation_load` limits total concurrent delegations targeting B |
| Artifact boundary | One Agent Link run | Staged inputs are read-only; the delegatee writes only to ephemeral outputs that are validated before atomic publish |
| Per-root limit | One root agent's self-spawn tree | Agent `subagents.maxConcurrent`, default 20 |
| Process safety cap | All self-spawn and Agent Link callbacks | Standard 32 / Lite 2, with at most 128 independent pending chains |
| Nested lifetime | Agent Link artifact run | Async descendants are rejected; sync descendants complete before publication |
| Durable async result | Source/root agent UUID within one tenant | Terminal text and logical media paths use deadline-bounded persistence retries before announcement; single-process startup retries pre-traffic recovery that marks previous-process non-terminal rows failed, and successful terminal writes remain explicitly retrievable by completion UUID |
When concurrency limits are hit, the error message is written for LLM reasoning: *"Agent at capacity (5/5). Try a different agent or handle it yourself."*
`agent_links.max_concurrent` is retained as compatibility metadata and is not
currently enforced by runtime admission.
---
+52 -6
View File
@@ -579,10 +579,17 @@ Each delegation link (lead→member) has its own settings:
| Layer | Scope | Default |
|-------|-------|---------|
| Per-link | Simultaneous delegations from lead to a specific member | 3 |
| Per-agent | Total concurrent delegations targeting any single member | 5 |
| Per-root agent | Executing self-spawn descendants across the full tree | 20 |
| Process safety cap | Executing self-spawn and Agent Link callbacks | Standard 32 / Lite 2 |
| Pending admission | Independent child-run chains waiting for capacity | 128 |
When limits are hit, the error message is written for LLM reasoning: "Agent at capacity (5/5). Try a different agent or handle it yourself."
Each Agent Link delegation begins a new target-owned spawn tree. The source
agent's `maxSpawnDepth` does not limit the target agent; any subagents created by
the target use the target's own subagent settings.
`agent_links.max_concurrent` remains serialized compatibility metadata and is
not currently an admission limit. Agent Team task scheduling keeps its existing
lane/workspace contract.
---
@@ -593,7 +600,7 @@ Agent links define directed delegation relationships between agents, separate fr
- **Target agent** (can receive delegations from)
- **Direction** (outbound from source, inbound to target)
- **Team context** (optional `team_id` if created by team setup)
- **Concurrency limit** (max simultaneous delegations for this link)
- **Reserved concurrency metadata** (`max_concurrent`; retained for compatibility, not currently enforced)
**Team-created links** are automatically created when a team is set up (lead → each member). Links remain even if the team is deleted or members are removed, allowing manual cleanup.
@@ -601,9 +608,48 @@ Agent links define directed delegation relationships between agents, separate fr
## 12. Delegation Context
### SenderID Clearing
### Authorization and artifact scope
In sync delegations, the delegate agent's context has the `senderID` cleared. This is critical because delegations are system-initiated — the delegate should not inherit the caller's group writer permissions, which would incorrectly deny file writes. Each delegate agent has its own writer list.
Agent Link dispatch preserves the caller's authorization scope for policy
checks but attenuates filesystem authority. The delegatee receives staged
read-only copies through `inputs/` and writes only to `outputs/`. After the run,
regular-file, path, symlink, count, and size validation completes before an
atomic publication to the caller's `.delegations/<delegation-id>/` directory.
Agent Team shared-workspace behavior is unchanged.
Successful publications persist with `manifest.json` and logical `outputs/...`
entries. Failed/cancelled exchanges are retained for 60 minutes for diagnostics
and removed by a restart-safe janitor whose per-sweep item and filesystem-entry
work are both bounded. Traces correlate by
`delegation_id` and record only logical paths, sizes, hashes, status, and
timestamps—never physical exchange or caller host paths.
Async Agent Link dispatch also creates a durable completion row before the
child-run ticket is activated. The full terminal result and logical published
artifact paths are persisted before the automatic parent announcement. If
bounded announcement retries cannot enter the inbound queue, the source agent
can retrieve the result after the run—or after a gateway restart—with:
```text
delegate(action="get", delegation_id="<uuid>")
```
Retrieval is scoped by tenant and the immutable UUID of the source agent. It
does not grant the target agent, another linked source, or a later same-key
agent access to the result. Delegation completion rows are excluded from
`/subagents` and self-clone roster queries.
Terminal database writes use per-attempt deadlines plus an extended bounded
retry window. If the database remains unavailable for that whole window, the
live announcement is still attempted but is not recorded as durably delivered;
a crash during the same outage may prevent the terminal result from being
recorded. On the next startup that can reach the database, every non-terminal
completion row from the previous process is marked `failed` with an interruption
reason instead of remaining `queued` or `running` forever. The gateway retries
that reconciliation before accepting traffic if the database is temporarily
unavailable. The current runtime assumes one active gateway process. Graceful
shutdown waits for accepted completion persistence before tearing down runtime
dependencies.
### Trace Linking
+10 -3
View File
@@ -166,13 +166,20 @@ Uploaded files (from web chat, Telegram, Discord, etc.) are persisted to:
Uploads without a usable original filename fall back to `{uuid}.{ext}`. Legacy media refs may still resolve from `.media/{sessionHash}/{uuid}.{ext}`.
The `enrichDocumentPaths()` function injects the full path into `<media:document>` tags:
The `enrichDocumentPaths()` function injects the exact media ID and a logical
path relative to the active agent workspace into `<media:document>` tags:
```
<media:document name="report.pdf" path="/app/workspace/.uploads/report-a1b2c3d4.pdf">
<media:document name="report.pdf" id="..." path=".uploads/report-a1b2c3d4.pdf">
```
Agents can read these files directly via exec — no copy to `/tmp` needed. For archive uploads such as `.zip`, inspect or extract with commands like `unzip -l "<path>"` or `unzip -q "<path>" -d <output-dir>`.
Normal agent runs can read these workspace files directly via `exec` — no copy
to `/tmp` is needed. Agent Link delegations instead receive selected files as
read-only `inputs/...` paths in an isolated delegation exchange. Their `exec`
calls fail closed unless an active sandbox is available, and generated files
must be written under `outputs/` for validation and publication back to the
caller. For archive uploads such as `.zip`, inspect or extract them within the
authorized workspace or delegation paths.
---