mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-15 00:17:27 +00:00
feat: add prepare-env.sh script, update documentation, and refine session scheduling with new lanes and stop commands.
This commit is contained in:
1 parent
1088ffceae
commit
31f7e1e22e
21 files changed
+5391
-161
No files matched your search
+26
-9
@@ -1,9 +1,26 @@
|
||||
export GOCLAW_PROVIDER=
|
||||
export GOCLAW_MODEL=
|
||||
export GOCLAW_MINIMAX_API_KEY=
|
||||
export GOCLAW_OPENROUTER_API_KEY=
|
||||
export GOCLAW_GATEWAY_TOKEN=
|
||||
export GOCLAW_TELEGRAM_TOKEN=
|
||||
export GOCLAW_POSTGRES_DSN=
|
||||
export GOCLAW_ENCRYPTION_KEY=
|
||||
export GOCLAW_TRACE_VERBOSE=1
|
||||
# GoClaw environment variables.
|
||||
# Copy to .env and fill in values. For Docker Compose, do NOT use 'export' prefix.
|
||||
#
|
||||
# Required: at least one LLM provider API key.
|
||||
# Auto-generated by prepare-env.sh: GOCLAW_GATEWAY_TOKEN, GOCLAW_ENCRYPTION_KEY.
|
||||
|
||||
# --- LLM Provider (at least one required) ---
|
||||
GOCLAW_PROVIDER=
|
||||
GOCLAW_MODEL=
|
||||
GOCLAW_OPENROUTER_API_KEY=
|
||||
GOCLAW_ANTHROPIC_API_KEY=
|
||||
GOCLAW_MINIMAX_API_KEY=
|
||||
|
||||
# --- Gateway ---
|
||||
GOCLAW_GATEWAY_TOKEN=
|
||||
GOCLAW_ENCRYPTION_KEY=
|
||||
|
||||
# --- Channels (optional) ---
|
||||
GOCLAW_TELEGRAM_TOKEN=
|
||||
|
||||
# --- Database (only for non-Docker deployments) ---
|
||||
# Docker Compose auto-builds this from POSTGRES_USER/PASSWORD/DB.
|
||||
# GOCLAW_POSTGRES_DSN=postgres://user:pass@host:5432/dbname?sslmode=disable
|
||||
|
||||
# --- Debug ---
|
||||
# GOCLAW_TRACE_VERBOSE=1
|
||||
@@ -27,7 +27,6 @@ prompt.md
|
||||
ui/web/node_modules/
|
||||
ui/web/dist/
|
||||
ui/web/.vite/
|
||||
ui/web/pnpm-lock.yaml
|
||||
|
||||
.mcp.json
|
||||
tmp
|
||||
|
||||
@@ -364,49 +364,50 @@ source .env.local && ./goclaw
|
||||
|
||||
### With Docker
|
||||
|
||||
When `GOCLAW_*_API_KEY` environment variables are set, the gateway **auto-onboards** without interactive prompts — it detects the provider, generates a gateway token, and (in managed mode) connects to Postgres, runs migrations, and seeds default data.
|
||||
|
||||
**Standalone mode** — requires only a provider API key:
|
||||
**1. Prepare environment:**
|
||||
|
||||
```bash
|
||||
# .env
|
||||
GOCLAW_OPENROUTER_API_KEY=sk-or-your-key # Required: at least one provider key
|
||||
GOCLAW_GATEWAY_TOKEN=your-token # Optional (auto-generated if not set)
|
||||
# Generate .env with auto-generated secrets (GOCLAW_ENCRYPTION_KEY, GOCLAW_GATEWAY_TOKEN)
|
||||
chmod +x prepare-env.sh
|
||||
./prepare-env.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.standalone.yml up
|
||||
```
|
||||
The script creates `.env` from `.env.example`, auto-generates `GOCLAW_ENCRYPTION_KEY` and `GOCLAW_GATEWAY_TOKEN`, and prompts for a provider API key. Edit `.env` afterward to fill in any remaining values.
|
||||
|
||||
**Managed mode** — additionally needs Postgres credentials:
|
||||
**2. Start services:**
|
||||
|
||||
```bash
|
||||
# .env
|
||||
GOCLAW_OPENROUTER_API_KEY=sk-or-your-key # Required: at least one provider key
|
||||
GOCLAW_GATEWAY_TOKEN=your-token # Optional (auto-generated)
|
||||
GOCLAW_ENCRYPTION_KEY=hex-64-chars # Optional (auto-generated)
|
||||
GOCLAW_OWNER_IDS=admin # Optional: admin user IDs (can manage all agents)
|
||||
POSTGRES_USER=goclaw # Postgres container credentials
|
||||
POSTGRES_PASSWORD=your-secure-password
|
||||
POSTGRES_DB=goclaw
|
||||
```
|
||||
# Recommended: Managed mode + Web Dashboard (http://localhost:3000)
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.selfservice.yml up -d --build
|
||||
|
||||
```bash
|
||||
# Managed mode (PostgreSQL)
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml up
|
||||
# Standalone mode (file-based, no database)
|
||||
docker compose -f docker-compose.yml -f docker-compose.standalone.yml up -d --build
|
||||
|
||||
# + Web Dashboard (http://localhost:3000)
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.selfservice.yml up
|
||||
# Managed mode without dashboard
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml up -d --build
|
||||
|
||||
# + OpenTelemetry tracing (Jaeger at http://localhost:16686)
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.otel.yml up
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.otel.yml up -d --build
|
||||
|
||||
# + Tailscale (secure remote access)
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.tailscale.yml up
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.tailscale.yml up -d --build
|
||||
```
|
||||
|
||||
When `GOCLAW_*_API_KEY` environment variables are set, the gateway **auto-onboards** without interactive prompts — it detects the provider, generates a gateway token, and (in managed mode) connects to Postgres, runs migrations, and seeds default data.
|
||||
|
||||
**Auto-onboard detects** the first available API key in priority order: OpenRouter → Anthropic → OpenAI → Groq → DeepSeek → Gemini → Mistral → xAI → MiniMax → Cohere → Perplexity. Override with `GOCLAW_PROVIDER` and `GOCLAW_MODEL`. Memory is auto-enabled with embedding support if an OpenAI, OpenRouter, or Gemini key is detected.
|
||||
|
||||
**Minimum `.env` for managed mode:**
|
||||
|
||||
```bash
|
||||
GOCLAW_OPENROUTER_API_KEY=sk-or-your-key # Required: at least one provider key
|
||||
GOCLAW_GATEWAY_TOKEN=... # Auto-generated by prepare-env.sh
|
||||
GOCLAW_ENCRYPTION_KEY=... # Auto-generated by prepare-env.sh
|
||||
# GOCLAW_PROVIDER=openrouter # Optional: override default provider
|
||||
# GOCLAW_MODEL=anthropic/claude-sonnet-4 # Optional: override default model
|
||||
# POSTGRES_PASSWORD=your-secure-password # Optional: defaults to "goclaw"
|
||||
```
|
||||
|
||||
## Deployment Modes
|
||||
|
||||
> **Recommendation:** Use **managed mode** for the best experience. Most of GoClaw's advanced features — agent teams, delegation, handoff, evaluate loops, quality gates, tracing, skills with embedding search, MCP integration, and the web dashboard — require managed mode. Standalone mode is suitable for quick evaluation or single-user setups only.
|
||||
@@ -726,27 +727,30 @@ Seven composable files for different deployment scenarios:
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Prepare .env (auto-generates secrets, prompts for API key)
|
||||
chmod +x prepare-env.sh && ./prepare-env.sh
|
||||
|
||||
# Standalone
|
||||
docker compose -f docker-compose.yml -f docker-compose.standalone.yml up -d
|
||||
docker compose -f docker-compose.yml -f docker-compose.standalone.yml up -d --build
|
||||
|
||||
# Managed (PostgreSQL)
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml up -d
|
||||
docker compose -f docker-compose.yml -f docker-compose.managed.yml up -d --build
|
||||
|
||||
# Managed + Web Dashboard (http://localhost:3000)
|
||||
docker compose -f docker-compose.yml \
|
||||
-f docker-compose.managed.yml \
|
||||
-f docker-compose.selfservice.yml up -d
|
||||
-f docker-compose.selfservice.yml up -d --build
|
||||
|
||||
# Managed + Web Dashboard + OpenTelemetry (Jaeger UI at http://localhost:16686)
|
||||
docker compose -f docker-compose.yml \
|
||||
-f docker-compose.managed.yml \
|
||||
-f docker-compose.selfservice.yml \
|
||||
-f docker-compose.otel.yml up -d
|
||||
-f docker-compose.otel.yml up -d --build
|
||||
|
||||
# Managed + Tailscale (secure remote access via VPN mesh)
|
||||
docker compose -f docker-compose.yml \
|
||||
-f docker-compose.managed.yml \
|
||||
-f docker-compose.tailscale.yml up -d
|
||||
-f docker-compose.tailscale.yml up -d --build
|
||||
|
||||
# Check health
|
||||
curl http://localhost:18790/health
|
||||
@@ -754,21 +758,14 @@ curl http://localhost:18790/health
|
||||
|
||||
### Environment File (.env)
|
||||
|
||||
Create a `.env` file in the project root for Docker Compose:
|
||||
Use the `prepare-env.sh` script to generate `.env` with auto-generated secrets:
|
||||
|
||||
```bash
|
||||
# Required: at least one provider API key
|
||||
GOCLAW_OPENROUTER_API_KEY=sk-or-your-key
|
||||
|
||||
# Optional: gateway token (auto-generated if not set)
|
||||
GOCLAW_GATEWAY_TOKEN=your-token
|
||||
|
||||
# Optional: Postgres credentials (managed mode)
|
||||
POSTGRES_USER=goclaw
|
||||
POSTGRES_PASSWORD=your-secure-password
|
||||
POSTGRES_DB=goclaw
|
||||
./prepare-env.sh
|
||||
```
|
||||
|
||||
This creates `.env` with `GOCLAW_ENCRYPTION_KEY` and `GOCLAW_GATEWAY_TOKEN` pre-filled. You only need to add your provider API key. See `.env.example` for all available variables.
|
||||
|
||||
## Built-in Tools
|
||||
|
||||
| Tool | Group | Description |
|
||||
|
||||
+5
-1
@@ -59,7 +59,11 @@ func runGateway() {
|
||||
}
|
||||
|
||||
// Auto-detect: if no provider API key is configured, help the user.
|
||||
if !cfg.HasAnyProvider() {
|
||||
// Also trigger auto-onboard when config file doesn't exist (first run),
|
||||
// even if env vars provide API keys — managed mode needs DB seeding.
|
||||
_, cfgStatErr := os.Stat(cfgPath)
|
||||
configMissing := os.IsNotExist(cfgStatErr)
|
||||
if !cfg.HasAnyProvider() || configMissing {
|
||||
// Docker / CI: env vars provide API keys → non-interactive auto-onboard.
|
||||
if canAutoOnboard() {
|
||||
if runAutoOnboard(cfgPath) {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Managed overlay — PostgreSQL (pgvector) for multi-tenant storage.
|
||||
#
|
||||
# Usage:
|
||||
# docker compose -f docker-compose.yml -f docker-compose.managed.yml up
|
||||
# Recommended usage (with Web Dashboard):
|
||||
# docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.selfservice.yml up -d --build
|
||||
#
|
||||
# Required env vars (set in .env or shell):
|
||||
# GOCLAW_OPENROUTER_API_KEY (or another provider key)
|
||||
@@ -10,12 +10,14 @@
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg18
|
||||
ports:
|
||||
- "${POSTGRES_PORT:-5432}:5432"
|
||||
environment:
|
||||
POSTGRES_USER: ${POSTGRES_USER:-goclaw}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-goclaw}
|
||||
POSTGRES_DB: ${POSTGRES_DB:-goclaw}
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
- postgres-data:/var/lib/postgresql
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-goclaw}"]
|
||||
interval: 5s
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
# Self-service overlay — adds web dashboard UI (nginx + React SPA).
|
||||
#
|
||||
# Usage:
|
||||
# docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.selfservice.yml up
|
||||
# docker compose -f docker-compose.yml -f docker-compose.standalone.yml -f docker-compose.selfservice.yml up
|
||||
# Recommended usage:
|
||||
# docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.selfservice.yml up -d --build
|
||||
#
|
||||
# Dashboard: http://localhost:3000
|
||||
|
||||
|
||||
+27
-4
@@ -1,9 +1,12 @@
|
||||
# Base docker-compose — shared service definition.
|
||||
# Combine with an overlay for your deployment mode:
|
||||
#
|
||||
# Standalone: docker compose -f docker-compose.yml -f docker-compose.standalone.yml up
|
||||
# Managed: docker compose -f docker-compose.yml -f docker-compose.managed.yml up
|
||||
# Managed+OTel: docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.otel.yml up
|
||||
# Recommended (Managed + Web Dashboard):
|
||||
# docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.selfservice.yml up -d --build
|
||||
#
|
||||
# Standalone: docker compose -f docker-compose.yml -f docker-compose.standalone.yml up -d --build
|
||||
# Managed only: docker compose -f docker-compose.yml -f docker-compose.managed.yml up -d --build
|
||||
# Managed+OTel: docker compose -f docker-compose.yml -f docker-compose.managed.yml -f docker-compose.otel.yml up -d --build
|
||||
|
||||
services:
|
||||
goclaw:
|
||||
@@ -18,9 +21,29 @@ services:
|
||||
- path: .env
|
||||
required: false
|
||||
environment:
|
||||
# Core
|
||||
- GOCLAW_HOST=0.0.0.0
|
||||
- GOCLAW_PORT=18790
|
||||
- GOCLAW_CONFIG=/app/data/config.json
|
||||
- GOCLAW_GATEWAY_TOKEN=${GOCLAW_GATEWAY_TOKEN:-}
|
||||
- GOCLAW_ENCRYPTION_KEY=${GOCLAW_ENCRYPTION_KEY:-}
|
||||
- GOCLAW_OWNER_IDS=${GOCLAW_OWNER_IDS:-}
|
||||
# Provider (at least one required)
|
||||
- GOCLAW_PROVIDER=${GOCLAW_PROVIDER:-}
|
||||
- GOCLAW_MODEL=${GOCLAW_MODEL:-}
|
||||
- GOCLAW_OPENROUTER_API_KEY=${GOCLAW_OPENROUTER_API_KEY:-}
|
||||
- GOCLAW_ANTHROPIC_API_KEY=${GOCLAW_ANTHROPIC_API_KEY:-}
|
||||
- GOCLAW_OPENAI_API_KEY=${GOCLAW_OPENAI_API_KEY:-}
|
||||
- GOCLAW_MINIMAX_API_KEY=${GOCLAW_MINIMAX_API_KEY:-}
|
||||
- GOCLAW_GROQ_API_KEY=${GOCLAW_GROQ_API_KEY:-}
|
||||
- GOCLAW_DEEPSEEK_API_KEY=${GOCLAW_DEEPSEEK_API_KEY:-}
|
||||
- GOCLAW_GEMINI_API_KEY=${GOCLAW_GEMINI_API_KEY:-}
|
||||
- GOCLAW_MISTRAL_API_KEY=${GOCLAW_MISTRAL_API_KEY:-}
|
||||
- GOCLAW_XAI_API_KEY=${GOCLAW_XAI_API_KEY:-}
|
||||
# Channels
|
||||
- GOCLAW_TELEGRAM_TOKEN=${GOCLAW_TELEGRAM_TOKEN:-}
|
||||
# Debug
|
||||
- GOCLAW_TRACE_VERBOSE=${GOCLAW_TRACE_VERBOSE:-0}
|
||||
volumes:
|
||||
- goclaw-data:/app/data
|
||||
- goclaw-workspace:/app/workspace # standalone mode workspace
|
||||
@@ -31,12 +54,12 @@ services:
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp:rw,noexec,nosuid,size=256m
|
||||
pids_limit: 200
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 1G
|
||||
cpus: '2.0'
|
||||
pids: 200
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
|
||||
@@ -4,10 +4,10 @@
|
||||
|
||||
GoClaw is an AI agent gateway written in Go. It exposes a WebSocket RPC (v3) interface and an OpenAI-compatible HTTP API for orchestrating LLM-powered agents. The system supports two operating modes:
|
||||
|
||||
- **Standalone** -- file-based storage, single-user, zero external dependencies beyond an LLM API key.
|
||||
- **Managed** -- PostgreSQL-backed multi-tenant mode with HTTP CRUD APIs, per-user context files, encrypted credentials, and LLM call tracing.
|
||||
- **Standalone** -- file-based storage with SQLite for per-user data, zero external dependencies beyond an LLM API key.
|
||||
- **Managed** -- PostgreSQL-backed multi-tenant mode with HTTP CRUD APIs, per-user context files, encrypted credentials, agent delegation, teams, and LLM call tracing.
|
||||
|
||||
> **Documentation scope**: This documentation is written for **managed mode** (PostgreSQL multi-tenant), which is the recommended production deployment. Standalone mode is functionally equivalent to the original OpenClaw -- all users share the same workspace, sessions, and context files with no per-user isolation, no encrypted secrets, no tracing, and no HTTP CRUD APIs. The remainder of this documentation assumes managed mode unless stated otherwise.
|
||||
> **Documentation scope**: This documentation covers both modes. Standalone mode now has near-parity with managed mode for core features (per-user context files, workspace isolation, agent types, bootstrap onboarding). Managed mode adds agent delegation, teams, quality gates, tracing, HTTP CRUD APIs, and encrypted secrets.
|
||||
|
||||
## 2. Component Diagram
|
||||
|
||||
@@ -38,7 +38,7 @@ flowchart TD
|
||||
|
||||
subgraph Core["Core Engine"]
|
||||
BUS[Message Bus]
|
||||
SCHED[Scheduler -- 3 Lanes]
|
||||
SCHED[Scheduler -- 4 Lanes]
|
||||
AR[Agent Router]
|
||||
LOOP[Agent Loop -- Think / Act / Observe]
|
||||
end
|
||||
@@ -54,6 +54,10 @@ flowchart TD
|
||||
WEB[Web Search / Fetch]
|
||||
MEM[Memory]
|
||||
SUB[Subagent]
|
||||
DEL[Delegation]
|
||||
TEAM_T[Teams]
|
||||
EVAL[Evaluate Loop]
|
||||
HO[Handoff]
|
||||
TTS_T[TTS]
|
||||
BROW[Browser]
|
||||
SK[Skills]
|
||||
@@ -61,6 +65,12 @@ flowchart TD
|
||||
CT[Custom Tools]
|
||||
end
|
||||
|
||||
subgraph Hooks["Hook Engine"]
|
||||
HE[Engine]
|
||||
CMD_E[Command Evaluator]
|
||||
AGT_E[Agent Evaluator]
|
||||
end
|
||||
|
||||
subgraph Store["Store Layer"]
|
||||
SESS[SessionStore]
|
||||
AGENT_S[AgentStore]
|
||||
@@ -71,6 +81,8 @@ flowchart TD
|
||||
TRACE_S[TracingStore]
|
||||
MCP_S[MCPServerStore]
|
||||
CT_S[CustomToolStore]
|
||||
AL_S[AgentLinkStore]
|
||||
TM_S[TeamStore]
|
||||
end
|
||||
|
||||
WS --> WSS
|
||||
@@ -89,6 +101,8 @@ flowchart TD
|
||||
LOOP --> Providers
|
||||
LOOP --> Tools
|
||||
Tools --> Store
|
||||
Tools --> Hooks
|
||||
Hooks --> Tools
|
||||
LOOP --> Store
|
||||
```
|
||||
|
||||
@@ -97,21 +111,22 @@ flowchart TD
|
||||
| Module | Description |
|
||||
|--------|-------------|
|
||||
| `internal/gateway/` | WebSocket + HTTP server, client handling, method router |
|
||||
| `internal/gateway/methods/` | RPC method handlers: chat, agents, sessions, config, skills, cron, pairing, exec approval, usage, send |
|
||||
| `internal/agent/` | Agent loop (think, act, observe), router, resolver, system prompt builder, sanitization, pruning, tracing, memory flush |
|
||||
| `internal/gateway/methods/` | RPC method handlers: chat, agents, agent_links, teams, delegations, sessions, config, skills, cron, pairing, exec approval, usage, send |
|
||||
| `internal/agent/` | Agent loop (think, act, observe), router, resolver, system prompt builder, sanitization, pruning, tracing, memory flush, DELEGATION.md + TEAM.md injection |
|
||||
| `internal/providers/` | LLM providers: Anthropic (native HTTP + SSE streaming), OpenAI-compatible (HTTP + SSE), retry logic |
|
||||
| `internal/tools/` | Tool registry, filesystem ops, exec/shell, policy engine, subagent, context file + memory interceptors, credential scrubbing, rate limiting |
|
||||
| `internal/tools/` | Tool registry, filesystem ops, exec/shell, policy engine, subagent, delegation manager, team tools, evaluate loop, handoff, context file + memory interceptors, credential scrubbing, rate limiting, PathDenyable |
|
||||
| `internal/tools/dynamic_loader.go` | Custom tool loader: LoadGlobal (startup), LoadForAgent (per-agent clone), ReloadGlobal (cache invalidation) |
|
||||
| `internal/tools/dynamic_tool.go` | Custom tool executor: command template rendering, shell escaping, encrypted env vars |
|
||||
| `internal/store/` | Store interfaces: SessionStore, AgentStore, ProviderStore, SkillStore, MemoryStore, CronStore, PairingStore, TracingStore, MCPServerStore |
|
||||
| `internal/hooks/` | Hook engine: quality gates, command evaluator, agent evaluator, recursion prevention (`WithSkipHooks`) |
|
||||
| `internal/store/` | Store interfaces: SessionStore, AgentStore, ProviderStore, SkillStore, MemoryStore, CronStore, PairingStore, TracingStore, MCPServerStore, AgentLinkStore, TeamStore, ChannelInstanceStore, ConfigSecretsStore |
|
||||
| `internal/store/pg/` | PostgreSQL implementations (`database/sql` + `pgx/v5`) |
|
||||
| `internal/store/file/` | File-based implementations (standalone mode) |
|
||||
| `internal/store/file/` | File-based implementations: sessions, memory (SQLite), cron, pairing, skills, agents (filesystem + SQLite) |
|
||||
| `internal/bootstrap/` | System prompt files (AGENTS.md, SOUL.md, TOOLS.md, IDENTITY.md, USER.md, HEARTBEAT.md, BOOTSTRAP.md) + seeding + truncation |
|
||||
| `internal/config/` | Config loading (JSON5) + env var overlay |
|
||||
| `internal/skills/` | SKILL.md loader (5-tier hierarchy) + BM25 search + hot-reload via fsnotify |
|
||||
| `internal/channels/` | Channel manager + adapters: Telegram, Feishu/Lark, Zalo, Discord, WhatsApp |
|
||||
| `internal/mcp/` | MCP server bridge (stdio, SSE, streamable-HTTP transports) |
|
||||
| `internal/scheduler/` | Lane-based concurrency control (main, subagent, cron lanes) with per-session serialization |
|
||||
| `internal/scheduler/` | Lane-based concurrency control (main, subagent, cron, delegate lanes) with per-session serialization |
|
||||
| `internal/memory/` | Memory system (SQLite FTS5 + embeddings for standalone mode) |
|
||||
| `internal/permissions/` | RBAC policy engine (admin, operator, viewer roles) |
|
||||
| `internal/pairing/` | DM/device pairing service (8-character codes) |
|
||||
@@ -119,7 +134,7 @@ flowchart TD
|
||||
| `internal/bus/` | Event pub/sub (Message Bus) |
|
||||
| `internal/sandbox/` | Docker-based code execution sandbox |
|
||||
| `internal/tts/` | Text-to-Speech providers: OpenAI, ElevenLabs, Edge, MiniMax |
|
||||
| `internal/http/` | HTTP API handlers: /v1/chat/completions, /v1/agents, /v1/skills, /v1/traces, /v1/mcp |
|
||||
| `internal/http/` | HTTP API handlers: /v1/chat/completions, /v1/agents, /v1/skills, /v1/traces, /v1/mcp, /v1/delegations, summoner |
|
||||
| `internal/crypto/` | AES-256-GCM encryption for API keys |
|
||||
| `internal/tracing/` | LLM call tracing (traces + spans), in-memory buffer with periodic store flush |
|
||||
| `internal/tracing/otelexport/` | Optional OpenTelemetry OTLP exporter (opt-in via build tags; adds gRPC + protobuf) |
|
||||
@@ -132,19 +147,24 @@ flowchart TD
|
||||
| Aspect | Standalone | Managed |
|
||||
|--------|-----------|---------|
|
||||
| Config source | `config.json` + env vars | `config.json` + `GOCLAW_POSTGRES_DSN` |
|
||||
| Storage | JSON files on disk | PostgreSQL |
|
||||
| Storage | JSON files + SQLite (`~/.goclaw/data/agents.db`) | PostgreSQL |
|
||||
| Agents | Defined in `config.json` `agents.list`, created eagerly at startup | `agents` table, lazy-resolved via `ManagedResolver` |
|
||||
| Context files | Workspace filesystem (SOUL.md, IDENTITY.md, etc.) | `agent_context_files` + `user_context_files` tables |
|
||||
| Agent types | N/A | `open` (7 per-user files) / `predefined` (agent-level + USER.md per-user) |
|
||||
| Agent store | `FileAgentStore` (filesystem + SQLite) | `PGAgentStore` |
|
||||
| Context files | Agent-level on filesystem, per-user in SQLite | `agent_context_files` + `user_context_files` tables |
|
||||
| Agent types | `open` / `predefined` (via config) | `open` (7 per-user files) / `predefined` (agent-level + USER.md per-user) |
|
||||
| Per-user isolation | Workspace subdirectories (`user_alice/`, `user_bob/`) | Same + DB-scoped context files |
|
||||
| Bootstrap onboarding | Per-user BOOTSTRAP.md seeding (SQLite) | Same (PostgreSQL) |
|
||||
| Agent delegation | N/A | Sync/async delegation, agent links, quality gates |
|
||||
| Agent teams | N/A | Shared task board, mailbox, handoff |
|
||||
| Skills | Filesystem only (workspace + global dirs) | PostgreSQL + filesystem + embedding search |
|
||||
| Memory | SQLite FTS5 + embeddings | pgvector hybrid (full-text search + vector similarity) |
|
||||
| Tracing | N/A | `traces` + `spans` tables + optional OTel OTLP export |
|
||||
| MCP servers | `config.json` `tools.mcp_servers` | `mcp_servers` table + grants |
|
||||
| API key storage | `.env.local` / env vars only | PostgreSQL (AES-256-GCM encrypted) |
|
||||
| HTTP CRUD API | N/A | `/v1/agents`, `/v1/skills`, `/v1/traces`, `/v1/mcp` |
|
||||
| Virtual FS | Direct disk I/O | `ContextFileInterceptor` routes read_file/write_file to database |
|
||||
| HTTP CRUD API | N/A | `/v1/agents`, `/v1/skills`, `/v1/traces`, `/v1/mcp`, `/v1/delegations` |
|
||||
| Virtual FS | `ContextFileInterceptor` routes to SQLite | `ContextFileInterceptor` routes to PostgreSQL |
|
||||
| Custom tools | N/A | `custom_tools` table + `DynamicToolLoader` |
|
||||
| Managed-only stores (nil in standalone) | -- | AgentStore, ProviderStore, TracingStore, MCPServerStore, CustomToolStore |
|
||||
| Managed-only stores (nil in standalone) | -- | ProviderStore, TracingStore, MCPServerStore, CustomToolStore, AgentLinkStore, TeamStore |
|
||||
|
||||
---
|
||||
|
||||
@@ -241,12 +261,13 @@ sequenceDiagram
|
||||
GW->>GW: 16. Wire managed HTTP handlers (agents, skills, traces, MCP)
|
||||
else Standalone mode
|
||||
GW->>GW: 14. Create agents eagerly from config
|
||||
GW->>GW: 15. wireStandaloneExtras (FileAgentStore, interceptors, callbacks)
|
||||
end
|
||||
|
||||
GW->>Engine: 17. Create gateway server (WS + HTTP)
|
||||
GW->>Engine: 18. Register RPC methods
|
||||
GW->>Engine: 19. Register + start channels (Telegram, Discord, Feishu, Zalo, WhatsApp)
|
||||
GW->>Engine: 20. Start cron, heartbeat, scheduler (3 lanes)
|
||||
GW->>Engine: 20. Start cron, heartbeat, scheduler (4 lanes)
|
||||
GW->>Engine: 21. Start skills watcher + inbound consumer
|
||||
GW->>Engine: 22. Listen on host:port
|
||||
```
|
||||
@@ -255,7 +276,7 @@ sequenceDiagram
|
||||
|
||||
## 7. Managed Mode Wiring
|
||||
|
||||
The `wireManagedExtras()` function in `cmd/gateway_managed.go` performs 7 steps to wire multi-tenant components:
|
||||
The `wireManagedExtras()` function in `cmd/gateway_managed.go` wires multi-tenant components:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
@@ -265,9 +286,15 @@ flowchart TD
|
||||
W4["4. ManagedResolver<br/>Lazy-creates agent Loops from DB on cache miss"] --> W5
|
||||
W5["5. Virtual FS Interceptors<br/>Wire interceptors on read_file + write_file + memory tools"] --> W6
|
||||
W6["6. Memory Store Wiring<br/>Wire PGMemoryStore on memory_search + memory_get tools"] --> W7
|
||||
W7["7. Cache Invalidation Subscribers<br/>Subscribe to MessageBus events"]
|
||||
W7["7. Cache Invalidation Subscribers<br/>Subscribe to MessageBus events"] --> W8
|
||||
W8["8. Delegation Tools<br/>DelegateManager + delegate_search + agent links"] --> W9
|
||||
W9["9. Team Tools<br/>team_tasks + team_message + team auto-linking"] --> W10
|
||||
W10["10. Hook Engine<br/>Quality gates with command + agent evaluators"] --> W11
|
||||
W11["11. Evaluate Loop + Handoff<br/>evaluate_loop tool + handoff tool"]
|
||||
```
|
||||
|
||||
A separate `wireStandaloneExtras()` in `cmd/gateway_standalone.go` wires the same core callbacks (user seeding, context file loading) using `FileAgentStore` instead of PostgreSQL.
|
||||
|
||||
### Cache Invalidation Events
|
||||
|
||||
| Event | Subscriber | Action |
|
||||
@@ -282,7 +309,7 @@ flowchart TD
|
||||
|
||||
## 8. Scheduler Lanes
|
||||
|
||||
The scheduler uses a lane-based concurrency model. Each lane is a named worker pool with a bounded semaphore. Per-session serialization ensures only one agent run executes at a time per session key.
|
||||
The scheduler uses a lane-based concurrency model. Each lane is a named worker pool with a bounded semaphore. Per-session queues control concurrency within each session.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
@@ -295,21 +322,43 @@ flowchart TD
|
||||
S1[Subagent executions]
|
||||
end
|
||||
|
||||
subgraph Del["Lane: delegate (concurrency 100)"]
|
||||
D1[Delegation executions]
|
||||
end
|
||||
|
||||
subgraph Cron["Lane: cron (concurrency 1)"]
|
||||
C1[Cron job executions]
|
||||
end
|
||||
|
||||
Main --> SEM1[Semaphore]
|
||||
Sub --> SEM2[Semaphore]
|
||||
Cron --> SEM3[Semaphore]
|
||||
Del --> SEM3[Semaphore]
|
||||
Cron --> SEM4[Semaphore]
|
||||
|
||||
SEM1 --> Q[Per-Session Queue<br/>1 run at a time per session]
|
||||
SEM1 --> Q[Per-Session Queue]
|
||||
SEM2 --> Q
|
||||
SEM3 --> Q
|
||||
SEM4 --> Q
|
||||
|
||||
Q --> AGENT[Agent Loop]
|
||||
```
|
||||
|
||||
### Lane Defaults
|
||||
|
||||
| Lane | Concurrency | Env Override | Purpose |
|
||||
|------|:-----------:|-------------|---------|
|
||||
| `main` | 2 | `GOCLAW_LANE_MAIN` | Primary user chat sessions |
|
||||
| `subagent` | 4 | `GOCLAW_LANE_SUBAGENT` | Spawned subagents |
|
||||
| `delegate` | 100 | `GOCLAW_LANE_DELEGATE` | Agent delegation executions |
|
||||
| `cron` | 1 | `GOCLAW_LANE_CRON` | Scheduled cron jobs |
|
||||
|
||||
### Session Queue Concurrency
|
||||
|
||||
Per-session queues now support configurable `maxConcurrent`:
|
||||
- **DMs**: `maxConcurrent = 1` (single-threaded per user)
|
||||
- **Groups**: `maxConcurrent = 3` (multiple concurrent responses)
|
||||
- **Adaptive throttle**: When session history exceeds 60% of context window, concurrency drops to 1
|
||||
|
||||
### Queue Modes
|
||||
|
||||
| Mode | Behavior |
|
||||
@@ -320,6 +369,13 @@ flowchart TD
|
||||
|
||||
Default queue config: capacity 10, drop policy `old` (drops oldest on overflow), debounce 800ms.
|
||||
|
||||
### /stop and /stopall
|
||||
|
||||
- `/stop` -- Cancel the oldest running task (others keep going)
|
||||
- `/stopall` -- Cancel all running tasks + drain the queue
|
||||
|
||||
Both are intercepted before the debouncer to avoid being merged with normal messages.
|
||||
|
||||
---
|
||||
|
||||
## 9. Graceful Shutdown
|
||||
@@ -382,6 +438,9 @@ flowchart TD
|
||||
| `cmd/root.go` | Cobra CLI entry point, flag parsing |
|
||||
| `cmd/gateway.go` | Gateway startup orchestrator (`runGateway()`) |
|
||||
| `cmd/gateway_managed.go` | Managed mode wiring (`wireManagedExtras()`, `wireManagedHTTP()`) |
|
||||
| `cmd/gateway_standalone.go` | Standalone mode wiring (`wireStandaloneExtras()`) |
|
||||
| `cmd/gateway_callbacks.go` | Shared callbacks for managed + standalone (user seeding, context file loading) |
|
||||
| `cmd/gateway_consumer.go` | Inbound message consumer (subagent, delegate, teammate, handoff routing) |
|
||||
| `cmd/gateway_providers.go` | Provider registration (config-based + DB-based) |
|
||||
| `cmd/gateway_methods.go` | RPC method registration |
|
||||
| `internal/config/config.go` | Config struct definitions |
|
||||
@@ -392,7 +451,11 @@ flowchart TD
|
||||
| `internal/gateway/router.go` | RPC method routing |
|
||||
| `internal/scheduler/lanes.go` | Lane definitions, semaphore-based concurrency |
|
||||
| `internal/scheduler/queue.go` | Per-session queue, queue modes, debounce |
|
||||
| `internal/store/stores.go` | `Stores` container struct (all store interfaces) |
|
||||
| `internal/hooks/engine.go` | Hook engine: evaluator registry, `EvaluateHooks` |
|
||||
| `internal/hooks/command_evaluator.go` | Shell command evaluator (exit 0 = pass) |
|
||||
| `internal/hooks/agent_evaluator.go` | Agent delegation evaluator (APPROVED/REJECTED) |
|
||||
| `internal/hooks/context.go` | `WithSkipHooks` / `SkipHooksFromContext` (recursion prevention) |
|
||||
| `internal/store/stores.go` | `Stores` container struct (all 14 store interfaces) |
|
||||
| `internal/store/types.go` | `StoreConfig`, `BaseModel` |
|
||||
|
||||
---
|
||||
|
||||
+21
-8
@@ -15,10 +15,11 @@ flowchart TD
|
||||
START([RunRequest]) --> PH1
|
||||
|
||||
subgraph PH1["Phase 1: Setup"]
|
||||
P1A[Lock mutex - one run at a time] --> P1B[Emit run.started event]
|
||||
P1A[Increment activeRuns atomic counter] --> P1B[Emit run.started event]
|
||||
P1B --> P1C[Create trace record]
|
||||
P1C --> P1D[Inject agentType / userID / agentID into context]
|
||||
P1D --> P1E[Ensure per-user files via sync.Map cache]
|
||||
P1D --> P1E0[Compute per-user workspace + WithToolWorkspace]
|
||||
P1E0 --> P1E[Ensure per-user files via sync.Map cache]
|
||||
P1E --> P1F[Persist agent + user IDs on session]
|
||||
end
|
||||
|
||||
@@ -34,7 +35,7 @@ flowchart TD
|
||||
P3A[Build system prompt - 15+ sections] --> P3B[Inject conversation summary if present]
|
||||
P3B --> P3C["History pipeline: limitHistoryTurns --> pruneContextMessages --> sanitizeHistory"]
|
||||
P3C --> P3D[Append current user message]
|
||||
P3D --> P3E[Save user message to session]
|
||||
P3D --> P3E[Buffer user message locally - deferred write]
|
||||
end
|
||||
|
||||
PH3 --> PH4
|
||||
@@ -60,7 +61,7 @@ flowchart TD
|
||||
|
||||
subgraph PH6["Phase 6: Response Finalization"]
|
||||
P6A["SanitizeAssistantContent (7-step pipeline)"] --> P6B["Detect NO_REPLY - suppress delivery if silent"]
|
||||
P6B --> P6C[Save assistant message to session]
|
||||
P6B --> P6C[Flush all buffered messages atomically to session]
|
||||
P6C --> P6D[Update metadata: model, provider, token counts]
|
||||
end
|
||||
|
||||
@@ -85,10 +86,11 @@ flowchart TD
|
||||
|
||||
### Phase 1: Setup
|
||||
|
||||
- Lock a mutex so only one run executes per Loop at a time.
|
||||
- Increment the `activeRuns` atomic counter (no mutex -- true concurrency, especially in group chats with `maxConcurrent = 3`).
|
||||
- Emit a `run.started` event to notify connected clients.
|
||||
- Create a trace record (managed mode) with a generated trace UUID.
|
||||
- Propagate context values: `WithAgentID()`, `WithUserID()`, `WithAgentType()`. Downstream tools and interceptors rely on these.
|
||||
- Compute per-user workspace: `base + "/" + sanitize(userID)`. Inject via `WithToolWorkspace(ctx)` so all filesystem and shell tools use the correct directory.
|
||||
- Ensure per-user files exist. A `sync.Map` cache guarantees the seeding function runs at most once per user.
|
||||
- Persist the agent ID and user ID on the session for later reference.
|
||||
|
||||
@@ -102,7 +104,7 @@ flowchart TD
|
||||
- Build the system prompt (15+ sections). Context files are resolved dynamically based on agent type.
|
||||
- Inject the conversation summary (if one exists from a previous compaction) as the first two messages.
|
||||
- Run the history pipeline (3 stages, see Section 5).
|
||||
- Append the current user message and save it to the session store.
|
||||
- Append the current user message. Messages are buffered locally (deferred write) to avoid race conditions with concurrent runs on the same session.
|
||||
|
||||
### Phase 4: LLM Iteration Loop
|
||||
|
||||
@@ -126,15 +128,23 @@ flowchart TD
|
||||
|
||||
- Run `SanitizeAssistantContent` -- a 7-step cleanup pipeline (see Section 3).
|
||||
- Detect `NO_REPLY` in the final content. If present, suppress message delivery (silent reply).
|
||||
- Save the final assistant message to the session.
|
||||
- Flush all buffered messages atomically to the session (user message, tool messages, assistant message). This prevents concurrent runs from interleaving partial history.
|
||||
- Update session metadata: model name, provider name, cumulative token counts.
|
||||
|
||||
### Phase 7: Auto-Summarization
|
||||
|
||||
- **Trigger condition**: the history has more than 50 messages OR the estimated token count exceeds 75% of the context window.
|
||||
- **Per-session TryLock**: before summarizing, acquire a non-blocking per-session lock. If another concurrent run is already summarizing, skip. This prevents concurrent summarization from corrupting session history.
|
||||
- **Memory flush first**: run synchronously so the agent can persist durable memories before history is truncated. Max 5 LLM iterations, 90-second timeout.
|
||||
- **Summarize**: launch a background goroutine with a 120-second timeout. The LLM produces a summary of all messages except the last 4. The summary is saved and the history is truncated to those 4 messages. The compaction counter is incremented.
|
||||
|
||||
### Cancel Handling
|
||||
|
||||
When the context is cancelled (via `/stop` or `/stopall`), the loop exits immediately:
|
||||
- Trace finalization uses `context.Background()` fallback when `ctx.Err() != nil` to ensure the final DB write succeeds.
|
||||
- Trace status is set to `"cancelled"` instead of `"error"`.
|
||||
- An empty outbound message triggers cleanup (stop typing indicator, clear reactions).
|
||||
|
||||
---
|
||||
|
||||
## 2. System Prompt
|
||||
@@ -163,7 +173,9 @@ The system prompt is assembled dynamically from 15+ sections. Two modes control
|
||||
15. **Silent Replies** -- instructions for the NO_REPLY convention.
|
||||
16. **Heartbeats** -- instructions for periodic wake-up behavior.
|
||||
17. **Sub-Agent Spawning** -- rules for launching child agents.
|
||||
18. **Runtime** -- runtime metadata (agent ID, session key, provider info).
|
||||
18. **Delegation** -- auto-generated `DELEGATION.md` listing available delegation targets (inline if ≤15, search instruction if >15).
|
||||
19. **Team** -- `TEAM.md` injected for team leads only (team name, role, teammate list).
|
||||
20. **Runtime** -- runtime metadata (agent ID, session key, provider info).
|
||||
|
||||
---
|
||||
|
||||
@@ -410,6 +422,7 @@ The Loop publishes events via an `onEvent` callback. The WebSocket gateway forwa
|
||||
| `tool.result` | Tool execution completes | `{"name": "...", "id": "...", "is_error": bool}` |
|
||||
| `run.completed` | Run finishes successfully | -- |
|
||||
| `run.failed` | Run finishes with an error | `{"error": "..."}` |
|
||||
| `handoff` | Conversation transferred to another agent | `{"from": "...", "to": "...", "reason": "..."}` |
|
||||
|
||||
### Event Flow
|
||||
|
||||
|
||||
@@ -236,6 +236,12 @@ flowchart LR
|
||||
|
||||
---
|
||||
|
||||
## 8. Agent Evaluators (Hook System)
|
||||
|
||||
Agent evaluators in the quality gate / hook system (see [03-tools-system.md](./03-tools-system.md)) use the same provider resolution as normal agent runs. When a quality gate is configured with `"type": "agent"`, the hook engine delegates to the specified reviewer agent, which resolves its own provider through the standard provider registry. No separate provider configuration is needed for evaluator agents.
|
||||
|
||||
---
|
||||
|
||||
## File Reference
|
||||
|
||||
| File | Purpose |
|
||||
|
||||
+247
-4
@@ -110,6 +110,22 @@ Context keys ensure each tool call receives the correct per-call values without
|
||||
|------|-------------|
|
||||
| `message` | Send a message to a channel |
|
||||
|
||||
### Delegation (group: `delegation`)
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `delegate` | Delegate task to another agent (actions: delegate, cancel, list, history) |
|
||||
| `delegate_search` | Hybrid FTS + semantic agent discovery for delegation targets |
|
||||
| `evaluate_loop` | Generate-evaluate-revise cycle with two agents (max 5 rounds) |
|
||||
| `handoff` | Transfer conversation to another agent (routing override) |
|
||||
|
||||
### Teams (group: `teams`)
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `team_tasks` | Task board: list, create, claim, complete, search |
|
||||
| `team_message` | Mailbox: send, broadcast, read unread messages |
|
||||
|
||||
### Other Tools
|
||||
|
||||
| Tool | Description |
|
||||
@@ -168,6 +184,22 @@ flowchart TD
|
||||
|
||||
Routes `MEMORY.md`, `memory.md`, and `memory/*` paths. Per-user results take priority with a fallback to global scope. Writing a `.md` file automatically triggers `IndexDocument()` (chunking + embedding).
|
||||
|
||||
### PathDenyable Interface
|
||||
|
||||
Tools that access the filesystem implement the `PathDenyable` interface, allowing specific path prefixes to be denied at runtime:
|
||||
|
||||
```go
|
||||
type PathDenyable interface {
|
||||
DenyPaths(...string)
|
||||
}
|
||||
```
|
||||
|
||||
All four filesystem tools (`read_file`, `write_file`, `list_files`, `edit_file`) implement it. `list_files` additionally filters denied directories from its output entirely -- the agent doesn't even know the directory exists. Used to prevent agents from accessing `.goclaw` directories within workspaces.
|
||||
|
||||
### Workspace Context Injection
|
||||
|
||||
Filesystem and shell tools read their workspace from `ToolWorkspaceFromCtx(ctx)`, which is injected by the agent loop based on the current user and agent. This enables per-user workspace isolation without changing any tool code. Falls back to the struct field for backward compatibility.
|
||||
|
||||
### Path Security
|
||||
|
||||
`resolvePath()` joins relative paths with the workspace root, applies `filepath.Clean()`, and verifies the result with `HasPrefix()`. This prevents path traversal attacks (e.g., `../../../etc/passwd`). The extended `resolvePathWithAllowed()` permits additional prefixes for skills directories.
|
||||
@@ -258,6 +290,8 @@ flowchart TD
|
||||
| `ui` | `browser`, `canvas` |
|
||||
| `automation` | `cron`, `gateway` |
|
||||
| `messaging` | `message` |
|
||||
| `delegation` | `delegate`, `delegate_search`, `evaluate_loop`, `handoff` |
|
||||
| `teams` | `team_tasks`, `team_message` |
|
||||
| `goclaw` | All native tools (composite group) |
|
||||
|
||||
Groups can be referenced in allow/deny lists with the `group:` prefix (e.g., `group:fs`). The MCP manager dynamically registers `mcp` and `mcp:{serverName}` groups at runtime.
|
||||
@@ -316,7 +350,204 @@ Results are announced back to the parent agent via the message bus, optionally b
|
||||
|
||||
---
|
||||
|
||||
## 7. MCP Bridge Tools
|
||||
## 7. Delegation System
|
||||
|
||||
Delegation allows named agents to delegate tasks to other fully independent agents (each with its own identity, tools, provider, model, and context files). Unlike subagents (anonymous clones), delegation crosses agent boundaries via explicit permission links.
|
||||
|
||||
### DelegateManager
|
||||
|
||||
The `DelegateManager` in `internal/tools/delegate.go` orchestrates all delegation operations:
|
||||
|
||||
| Action | Mode | Behavior |
|
||||
|--------|------|----------|
|
||||
| `delegate` | `sync` | Caller waits for result (quick lookups, fact checks) |
|
||||
| `delegate` | `async` | Caller moves on; result announced later via message bus (`delegate:{id}`) |
|
||||
| `cancel` | -- | Cancel a running async delegation by ID |
|
||||
| `list` | -- | List active delegations |
|
||||
| `history` | -- | Query past delegations from `delegation_history` table |
|
||||
|
||||
### Callback Pattern
|
||||
|
||||
The `tools` package cannot import `agent` (import cycle). A callback function bridges the gap:
|
||||
|
||||
```go
|
||||
type AgentRunFunc func(ctx context.Context, agentKey string, req DelegateRunRequest) (*DelegateRunResult, error)
|
||||
```
|
||||
|
||||
The `cmd` layer provides the implementation at wiring time. The `tools` package never knows `agent` exists.
|
||||
|
||||
### Agent Links (Permission Control)
|
||||
|
||||
Delegation requires an explicit link in the `agent_links` table. Links are directed edges:
|
||||
|
||||
- **outbound** (A→B): Only A can delegate to B
|
||||
- **bidirectional** (A↔B): Both can delegate to each other
|
||||
|
||||
Each link has `max_concurrent` and per-user `settings` (JSONB) for deny/allow lists.
|
||||
|
||||
### Concurrency Control
|
||||
|
||||
Two layers prevent overload:
|
||||
|
||||
| Layer | Config | Scope |
|
||||
|-------|--------|-------|
|
||||
| Per-link | `agent_links.max_concurrent` | A→B specifically |
|
||||
| Per-agent | `other_config.max_delegation_load` | B from all sources |
|
||||
|
||||
When limits hit, the error message is written for LLM reasoning: *"Agent at capacity (5/5). Try a different agent or handle it yourself."*
|
||||
|
||||
### DELEGATION.md Auto-Injection
|
||||
|
||||
During agent resolution, `DELEGATION.md` is auto-generated and injected into the system prompt:
|
||||
|
||||
- **≤15 targets**: Full inline list with agent keys, names, and frontmatter
|
||||
- **>15 targets**: Search instruction pointing to the `delegate_search` tool (hybrid FTS + pgvector cosine)
|
||||
|
||||
### Context File Merging (Open Agents)
|
||||
|
||||
For open agents, per-user context files merge with resolver-injected base files. Per-user files override same-name base files, but base-only files like `DELEGATION.md` are preserved:
|
||||
|
||||
```
|
||||
Base files (resolver): DELEGATION.md
|
||||
Per-user files (DB): AGENTS.md, SOUL.md, TOOLS.md, USER.md, ...
|
||||
Merged result: AGENTS.md, SOUL.md, TOOLS.md, USER.md, ..., DELEGATION.md ✓
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Agent Teams
|
||||
|
||||
Teams add a shared coordination layer on top of delegation: a task board for parallel work and a mailbox for peer-to-peer communication.
|
||||
|
||||
### Architecture
|
||||
|
||||
An admin creates a team via the dashboard, assigns a **lead** and **members**. When a user messages the lead:
|
||||
1. The lead sees `TEAM.md` in its system prompt (teammate list + role)
|
||||
2. The lead posts tasks to the board
|
||||
3. Teammates are activated, claim tasks, and work in parallel
|
||||
4. Teammates message each other for coordination
|
||||
5. The lead synthesizes results and replies to the user
|
||||
|
||||
### Task Board (`team_tasks` tool)
|
||||
|
||||
| Action | Description |
|
||||
|--------|-------------|
|
||||
| `list` | List tasks (filter: active/completed/all, order: priority/newest) |
|
||||
| `create` | Create task with subject, description, priority, blocked_by |
|
||||
| `claim` | Atomically claim a pending task (race-safe via row-level lock) |
|
||||
| `complete` | Mark task done with result; auto-unblocks dependent tasks |
|
||||
| `search` | FTS search over task subject + description |
|
||||
|
||||
### Mailbox (`team_message` tool)
|
||||
|
||||
| Action | Description |
|
||||
|--------|-------------|
|
||||
| `send` | Send direct message to a specific teammate |
|
||||
| `broadcast` | Send message to all teammates |
|
||||
| `read` | Read unread messages |
|
||||
|
||||
### Lead-Centric Design
|
||||
|
||||
Only the lead gets `TEAM.md` in its system prompt. Teammates discover context on demand through tools -- no wasted tokens on idle agents. When a teammate message arrives, the message itself carries context (e.g., *"[Team message from lead]: please claim a task from the board."*).
|
||||
|
||||
### Message Routing
|
||||
|
||||
Teammate results route through the message bus with a `"teammate:"` prefix. The consumer publishes the outbound response so the lead (and ultimately the user) sees the result.
|
||||
|
||||
---
|
||||
|
||||
## 9. Evaluate-Optimize Loop
|
||||
|
||||
A structured revision cycle between two agents: a generator and an evaluator.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant L as Calling Agent
|
||||
participant G as Generator
|
||||
participant V as Evaluator
|
||||
|
||||
L->>G: "Write product announcement"
|
||||
G->>L: Draft v1
|
||||
L->>V: "Evaluate against criteria"
|
||||
V->>L: "REJECTED: Too long, missing pricing"
|
||||
L->>G: "Revise. Feedback: too long, missing pricing"
|
||||
G->>L: Draft v2
|
||||
L->>V: "Evaluate revised version"
|
||||
V->>L: "APPROVED"
|
||||
L->>L: Return v2 as final output
|
||||
```
|
||||
|
||||
The `evaluate_loop` tool orchestrates this. Parameters: generator agent, evaluator agent, pass criteria, and max rounds (default 3, cap 5). Each round is a pair of sync delegations. If the evaluator responds with "APPROVED" (case-insensitive prefix match), the loop exits. If "REJECTED: feedback", the generator gets another shot.
|
||||
|
||||
Internal delegations use `WithSkipHooks(ctx)` to prevent quality gates from triggering recursion.
|
||||
|
||||
---
|
||||
|
||||
## 10. Agent Handoff
|
||||
|
||||
Handoff transfers a conversation from one agent to another. Unlike delegation (which keeps the source agent in the loop), handoff removes it entirely.
|
||||
|
||||
| | Delegation | Handoff |
|
||||
|---|---|---|
|
||||
| Who talks to the user? | Source agent (always) | Target agent (after transfer) |
|
||||
| Source agent involvement | Waits for result, reformulates | Steps away completely |
|
||||
| Session | Target runs in source's context | Target gets a new session |
|
||||
| Duration | One task | Until cleared or handed back |
|
||||
|
||||
### Mechanism
|
||||
|
||||
When agent A calls `handoff(agent="billing", reason="billing question")`:
|
||||
1. A row is written to `handoff_routes`: this channel + chat ID now routes to billing
|
||||
2. A `handoff` event is broadcast (WS clients can react)
|
||||
3. An initial message is published to billing via the message bus with conversation context
|
||||
|
||||
Subsequent messages from the user on that channel are routed to billing (consumer checks `handoff_routes` before normal routing). Billing can hand back via `handoff(action="clear")`.
|
||||
|
||||
---
|
||||
|
||||
## 11. Quality Gates (Hook System)
|
||||
|
||||
A general-purpose hook system for validating agent output before it reaches the user. Located in `internal/hooks/`.
|
||||
|
||||
### Evaluator Types
|
||||
|
||||
| Type | How it works | Example |
|
||||
|------|-------------|---------|
|
||||
| **command** | Run a shell command. Exit 0 = pass. Stderr = feedback. | `npm test`, `eslint --stdin` |
|
||||
| **agent** | Delegate to a reviewer agent. Parse "APPROVED" or "REJECTED: feedback". | QA reviewer checks tone/accuracy |
|
||||
|
||||
### Configuration
|
||||
|
||||
Quality gates live in the source agent's `other_config` JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"quality_gates": [
|
||||
{
|
||||
"event": "delegation.completed",
|
||||
"type": "agent",
|
||||
"agent": "qa-reviewer",
|
||||
"block_on_failure": true,
|
||||
"max_retries": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
When `block_on_failure` is true and retries remain, the system re-runs the target agent with the evaluator's feedback injected as a revision prompt.
|
||||
|
||||
### Recursion Prevention
|
||||
|
||||
Quality gates with agent evaluators can cause infinite recursion (gate delegates to reviewer → reviewer completes → gate fires again). The fix is a context flag: `hooks.WithSkipHooks(ctx, true)`. Three places set it:
|
||||
1. **Agent evaluator** -- when delegating to the reviewer
|
||||
2. **Evaluate loop** -- for all internal generator/evaluator delegations
|
||||
3. **Agent eval callback in cmd layer** -- when the hook engine itself triggers delegation
|
||||
|
||||
`DelegateManager.Delegate()` checks `hooks.SkipHooksFromContext(ctx)` before applying gates. If set, gates are skipped.
|
||||
|
||||
---
|
||||
|
||||
## 12. MCP Bridge Tools
|
||||
|
||||
GoClaw integrates with Model Context Protocol (MCP) servers via `internal/mcp/`. The MCP Manager connects to external tool servers and registers their tools in the tool registry with a configurable prefix.
|
||||
|
||||
@@ -367,7 +598,7 @@ flowchart LR
|
||||
|
||||
---
|
||||
|
||||
## 8. Custom Tools (Managed Mode)
|
||||
## 13. Custom Tools (Managed Mode)
|
||||
|
||||
Define shell-based tools at runtime via the HTTP API -- no recompile or restart needed. Custom tools are stored in the `custom_tools` PostgreSQL table and loaded dynamically into the agent's tool registry.
|
||||
|
||||
@@ -427,7 +658,7 @@ flowchart TD
|
||||
|
||||
---
|
||||
|
||||
## 8. Credential Scrubbing
|
||||
## 14. Credential Scrubbing
|
||||
|
||||
Tool output is automatically scrubbed before being returned to the LLM. Enabled by default in the registry.
|
||||
|
||||
@@ -445,7 +676,7 @@ All matches are replaced with `[REDACTED]`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Rate Limiter
|
||||
## 15. Rate Limiter
|
||||
|
||||
The tool registry supports per-session rate limiting via `ToolRateLimiter`. When configured, each `ExecuteWithContext` call checks `rateLimiter.Allow(sessionKey)` before tool execution. Rate-limited calls receive an error result without executing the tool.
|
||||
|
||||
@@ -464,6 +695,18 @@ The tool registry supports per-session rate limiting via `ToolRateLimiter`. When
|
||||
| `internal/tools/shell.go` | ExecTool: deny patterns, approval workflow, sandbox routing |
|
||||
| `internal/tools/scrub.go` | ScrubCredentials: credential pattern matching and redaction |
|
||||
| `internal/tools/subagent.go` | SubagentManager: spawn, cancel, steer, run sync, deny lists |
|
||||
| `internal/tools/delegate.go` | DelegateManager: sync, async, cancel, concurrency, per-user checks |
|
||||
| `internal/tools/delegate_tool.go` | Delegate tool wrapper (action: delegate/cancel/list/history) |
|
||||
| `internal/tools/delegate_search_tool.go` | Hybrid FTS + semantic agent discovery |
|
||||
| `internal/tools/evaluate_loop_tool.go` | Generate-evaluate-revise loop (max 5 rounds) |
|
||||
| `internal/tools/handoff_tool.go` | Conversation transfer (routing override + context carry) |
|
||||
| `internal/tools/team_tool_manager.go` | Shared backend for team tools |
|
||||
| `internal/tools/team_tasks_tool.go` | Task board: list, create, claim, complete, search |
|
||||
| `internal/tools/team_message_tool.go` | Mailbox: send, broadcast, read |
|
||||
| `internal/hooks/engine.go` | Hook engine: evaluator registry, EvaluateHooks |
|
||||
| `internal/hooks/command_evaluator.go` | Shell command evaluator |
|
||||
| `internal/hooks/agent_evaluator.go` | Agent delegation evaluator |
|
||||
| `internal/hooks/context.go` | WithSkipHooks / SkipHooksFromContext |
|
||||
| `internal/tools/context_file_interceptor.go` | ContextFileInterceptor: 7-file routing by agent type |
|
||||
| `internal/tools/memory_interceptor.go` | MemoryInterceptor: MEMORY.md and memory/* routing |
|
||||
| `internal/tools/skill_search.go` | Skill search tool (BM25) |
|
||||
|
||||
@@ -74,7 +74,7 @@ The first request from a client must be `connect`. Any other method sent before
|
||||
### Event Frame Structure
|
||||
|
||||
- `type`: always `"event"`
|
||||
- `event`: event name (e.g., `chat`, `agent`, `status`)
|
||||
- `event`: event name (e.g., `chat`, `agent`, `status`, `handoff`)
|
||||
- `payload`: event data
|
||||
- `seq`: ordering sequence number
|
||||
- `stateVersion`: version counters for optimistic state sync
|
||||
@@ -112,7 +112,7 @@ flowchart LR
|
||||
|------|--------------------|
|
||||
| viewer | `agents.list`, `config.get`, `sessions.list`, `sessions.preview`, `health`, `status`, `models.list`, `skills.list`, `skills.get`, `channels.list`, `channels.status`, `cron.list`, `cron.status`, `cron.runs`, `usage.get`, `usage.summary` |
|
||||
| operator | All viewer methods plus: `chat.send`, `chat.abort`, `chat.history`, `chat.inject`, `sessions.delete`, `sessions.reset`, `sessions.patch`, `cron.create`, `cron.update`, `cron.delete`, `cron.toggle`, `cron.run`, `skills.update`, `send`, `exec.approval.list`, `exec.approval.approve`, `exec.approval.deny`, `device.pair.request`, `device.pair.list` |
|
||||
| admin | All operator methods plus: `config.apply`, `config.patch`, `agents.create`, `agents.update`, `agents.delete`, `agents.files.*`, `channels.toggle`, `device.pair.approve`, `device.pair.revoke` |
|
||||
| admin | All operator methods plus: `config.apply`, `config.patch`, `agents.create`, `agents.update`, `agents.delete`, `agents.files.*`, `agents.links.*`, `teams.*`, `channels.toggle`, `device.pair.approve`, `device.pair.revoke` |
|
||||
|
||||
---
|
||||
|
||||
@@ -261,6 +261,32 @@ flowchart TD
|
||||
| `browser.snapshot` | Get DOM snapshot |
|
||||
| `browser.screenshot` | Take screenshot |
|
||||
|
||||
### Agent Links
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `agents.links.list` | List agent links (by source agent) |
|
||||
| `agents.links.create` | Create an agent link (outbound or bidirectional) |
|
||||
| `agents.links.update` | Update a link (max_concurrent, settings, status) |
|
||||
| `agents.links.delete` | Delete an agent link |
|
||||
|
||||
### Teams
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `teams.list` | List agent teams |
|
||||
| `teams.create` | Create a team (lead + members) |
|
||||
| `teams.get` | Get team details with members |
|
||||
| `teams.delete` | Delete a team |
|
||||
| `teams.tasks.list` | List team tasks |
|
||||
|
||||
### Delegations
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `delegations.list` | List delegation history (result truncated to 500 runes) |
|
||||
| `delegations.get` | Get delegation detail (result truncated to 8000 runes) |
|
||||
|
||||
### Other
|
||||
|
||||
| Method | Description |
|
||||
@@ -351,6 +377,30 @@ All managed endpoints require `Authorization: Bearer <token>` and `X-GoClaw-User
|
||||
| GET | `/v1/mcp/requests` | List pending access requests |
|
||||
| POST | `/v1/mcp/requests/{id}/review` | Approve or reject a request |
|
||||
|
||||
**Agent Sharing** (`/v1/agents/{id}/sharing`):
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/v1/agents/{id}/sharing` | List shares for an agent |
|
||||
| POST | `/v1/agents/{id}/sharing` | Share agent with a user |
|
||||
| DELETE | `/v1/agents/{id}/sharing/{userID}` | Revoke user access |
|
||||
|
||||
**Agent Links** (`/v1/agents/{id}/links`):
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/v1/agents/{id}/links` | List links for an agent |
|
||||
| POST | `/v1/agents/{id}/links` | Create a new link |
|
||||
| PUT | `/v1/agents/{id}/links/{linkID}` | Update a link |
|
||||
| DELETE | `/v1/agents/{id}/links/{linkID}` | Delete a link |
|
||||
|
||||
**Delegations** (`/v1/delegations`):
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/v1/delegations` | List delegation history (full records, paginated) |
|
||||
| GET | `/v1/delegations/{id}` | Get delegation detail |
|
||||
|
||||
**Skills** (`/v1/skills`):
|
||||
|
||||
| Method | Path | Description |
|
||||
@@ -422,6 +472,9 @@ Error responses include `retryable` (boolean) and `retryAfterMs` (integer) field
|
||||
| `internal/gateway/methods/config.go` | config.get/apply/patch/schema handlers |
|
||||
| `internal/gateway/methods/skills.go` | skills.list/get/update handlers |
|
||||
| `internal/gateway/methods/cron.go` | cron.list/create/update/delete/toggle/run/runs handlers |
|
||||
| `internal/gateway/methods/agent_links.go` | agents.links.* handlers + agent router cache invalidation |
|
||||
| `internal/gateway/methods/teams.go` | teams.* handlers + auto-linking teammates |
|
||||
| `internal/gateway/methods/delegations.go` | delegations.list/get handlers |
|
||||
| `internal/gateway/methods/channels.go` | channels.list/status handlers |
|
||||
| `internal/gateway/methods/pairing.go` | device.pair.* handlers |
|
||||
| `internal/gateway/methods/exec_approval.go` | exec.approval.* handlers |
|
||||
@@ -433,6 +486,8 @@ Error responses include `retryable` (boolean) and `retryAfterMs` (integer) field
|
||||
| `internal/http/agents.go` | Agent CRUD HTTP handlers (managed mode) |
|
||||
| `internal/http/skills.go` | Skills HTTP handlers (managed mode) |
|
||||
| `internal/http/traces.go` | Traces HTTP handlers (managed mode) |
|
||||
| `internal/http/delegations.go` | Delegation history HTTP handlers |
|
||||
| `internal/http/summoner.go` | LLM-powered agent setup (XML parsing, context file generation) |
|
||||
| `internal/http/auth.go` | Bearer token authentication, timing-safe comparison |
|
||||
| `internal/permissions/policy.go` | PolicyEngine: role hierarchy, method-to-role mapping |
|
||||
| `pkg/protocol/frames.go` | Frame types: RequestFrame, ResponseFrame, EventFrame, ErrorShape |
|
||||
@@ -51,6 +51,32 @@ flowchart LR
|
||||
|
||||
Internal channels (`cli`, `system`, `subagent`) are silently skipped by the outbound dispatcher and never forwarded to external platforms.
|
||||
|
||||
### Handoff Routing (Managed Mode)
|
||||
|
||||
Before normal agent routing, the consumer checks the `handoff_routes` table for an active routing override. If a handoff route exists for the incoming channel + chat ID, the message is redirected to the target agent instead of the original agent.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
MSG["Inbound message"] --> CHECK{"handoff_routes<br/>has override?"}
|
||||
CHECK -->|Yes| TARGET["Route to target agent<br/>(billing, support, etc.)"]
|
||||
CHECK -->|No| NORMAL["Route to default agent"]
|
||||
TARGET --> SESSION["New session for target agent"]
|
||||
NORMAL --> SESSION2["Existing session"]
|
||||
```
|
||||
|
||||
Handoff routes are created by the `handoff` tool (see [03-tools-system.md](./03-tools-system.md)) and can be cleared by the target agent calling `handoff(action="clear")` or by handing back to the original agent.
|
||||
|
||||
### Message Routing Prefixes
|
||||
|
||||
The consumer routes system messages based on sender ID prefixes:
|
||||
|
||||
| Prefix | Route | Outbound Delivery |
|
||||
|--------|-------|:-:|
|
||||
| `subagent:` | Parent session queue | Yes |
|
||||
| `delegate:` | Delegate scheduler lane | Yes |
|
||||
| `teammate:` | Lead agent session queue | Yes |
|
||||
| `handoff:` | Target agent via delegate lane | Yes |
|
||||
|
||||
### Managed Mode Behavior
|
||||
|
||||
In managed mode, channels provide per-user isolation through compound sender IDs and context propagation:
|
||||
@@ -155,6 +181,8 @@ The Telegram channel uses long polling via the `telego` library (Telegram Bot AP
|
||||
- **Group mention gating**: By default, bot must be @mentioned in groups (`requireMention: true`). Pending group messages without a mention are stored in a history buffer (default 50 messages) and included as context when the bot is eventually mentioned.
|
||||
- **Typing indicator**: A "typing" action is sent while the agent is processing.
|
||||
- **Proxy support**: Optional HTTP proxy configured via the channel config.
|
||||
- **Cancel commands**: `/stop` (cancel oldest running task) and `/stopall` (cancel all + drain queue). Both are intercepted before the 800ms debouncer to avoid being merged with subsequent messages. See [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md) for details.
|
||||
- **Concurrent group support**: Group sessions support up to 3 concurrent agent runs, allowing multiple users to get responses in parallel.
|
||||
|
||||
### Formatting Pipeline
|
||||
|
||||
@@ -290,6 +318,7 @@ sequenceDiagram
|
||||
| `internal/channels/channel.go` | Channel interface, BaseChannel, DMPolicy/GroupPolicy types, HandleMessage |
|
||||
| `internal/channels/manager.go` | Manager: channel registration, StartAll, StopAll, outbound dispatch |
|
||||
| `internal/channels/telegram/telegram.go` | Telegram channel: long polling, mention gating, typing indicators |
|
||||
| `internal/channels/telegram/commands.go` | /stop, /stopall command handlers, menu registration |
|
||||
| `internal/channels/telegram/format.go` | Markdown-to-Telegram-HTML pipeline, table rendering, CJK width |
|
||||
| `internal/channels/telegram/format_test.go` | Tests for Telegram formatting pipeline |
|
||||
| `internal/channels/feishu/feishu.go` | Feishu/Lark channel: WS/Webhook modes, card messages |
|
||||
@@ -302,3 +331,4 @@ sequenceDiagram
|
||||
| `internal/channels/whatsapp/whatsapp.go` | WhatsApp channel: external WS bridge |
|
||||
| `internal/channels/zalo/zalo.go` | Zalo channel: OA Bot API, long polling, DM only |
|
||||
| `internal/pairing/service.go` | Pairing service: code generation, approval, persistence |
|
||||
| `cmd/gateway_consumer.go` | Message consumer: routing prefixes, handoff check, cancel interception |
|
||||
+180
-10
@@ -12,9 +12,9 @@ flowchart TD
|
||||
CHECK -->|Yes| PG["PostgreSQL Backend"]
|
||||
CHECK -->|No| FILE["File Backend"]
|
||||
|
||||
PG --> PG_STORES["PGSessionStore<br/>PGAgentStore<br/>PGProviderStore<br/>PGCronStore<br/>PGPairingStore<br/>PGSkillStore<br/>PGMemoryStore<br/>PGTracingStore<br/>PGMCPServerStore<br/>PGCustomToolStore"]
|
||||
PG --> PG_STORES["PGSessionStore<br/>PGAgentStore<br/>PGProviderStore<br/>PGCronStore<br/>PGPairingStore<br/>PGSkillStore<br/>PGMemoryStore<br/>PGTracingStore<br/>PGMCPServerStore<br/>PGCustomToolStore<br/>PGChannelInstanceStore<br/>PGConfigSecretsStore<br/>PGAgentLinkStore<br/>PGTeamStore"]
|
||||
|
||||
FILE --> FILE_STORES["FileSessionStore<br/>FileMemoryStore (SQLite + FTS5)<br/>FileCronStore<br/>FilePairingStore<br/>FileSkillStore<br/>AgentStore = nil<br/>ProviderStore = nil<br/>TracingStore = nil<br/>MCPServerStore = nil<br/>CustomToolStore = nil"]
|
||||
FILE --> FILE_STORES["FileSessionStore<br/>FileMemoryStore (SQLite + FTS5)<br/>FileCronStore<br/>FilePairingStore<br/>FileSkillStore<br/>FileAgentStore (filesystem + SQLite)<br/>ProviderStore = nil<br/>TracingStore = nil<br/>MCPServerStore = nil<br/>CustomToolStore = nil<br/>AgentLinks = nil<br/>Teams = nil"]
|
||||
```
|
||||
|
||||
---
|
||||
@@ -30,11 +30,29 @@ The `Stores` struct is the top-level container holding all storage backends. In
|
||||
| CronStore | `FileCronStore` | `PGCronStore` | Both |
|
||||
| PairingStore | `FilePairingStore` via `pairing.Service` | `PGPairingStore` | Both |
|
||||
| SkillStore | `FileSkillStore` via `skills.Loader` | `PGSkillStore` | Both |
|
||||
| AgentStore | `nil` | `PGAgentStore` | Managed only |
|
||||
| AgentStore | `FileAgentStore` (filesystem + SQLite) | `PGAgentStore` | Both |
|
||||
| ProviderStore | `nil` | `PGProviderStore` | Managed only |
|
||||
| TracingStore | `nil` | `PGTracingStore` | Managed only |
|
||||
| MCPServerStore | `nil` | `PGMCPServerStore` | Managed only |
|
||||
| CustomToolStore | `nil` | `PGCustomToolStore` | Managed only |
|
||||
| ChannelInstanceStore | `nil` | `PGChannelInstanceStore` | Managed only |
|
||||
| ConfigSecretsStore | `nil` | `PGConfigSecretsStore` | Managed only |
|
||||
| AgentLinkStore | `nil` | `PGAgentLinkStore` | Managed only |
|
||||
| TeamStore | `nil` | `PGTeamStore` | Managed only |
|
||||
|
||||
### Standalone AgentStore (FileAgentStore)
|
||||
|
||||
In standalone mode, `FileAgentStore` provides per-user context files and profiles without PostgreSQL. It combines filesystem storage (agent-level files like SOUL.md) with SQLite (`~/.goclaw/data/agents.db`) for per-user data:
|
||||
|
||||
| Data | Storage |
|
||||
|------|---------|
|
||||
| Agent metadata | In-memory from `config.json` |
|
||||
| Agent-level files (SOUL.md, IDENTITY.md, ...) | Filesystem at workspace root |
|
||||
| Per-user files (USER.md, BOOTSTRAP.md) | SQLite `user_context_files` |
|
||||
| User profiles | SQLite `user_profiles` |
|
||||
| Group file writers | SQLite `group_file_writers` |
|
||||
|
||||
Agent UUIDs use UUID v5 (deterministic): `uuid.NewSHA1(namespace, "goclaw-standalone:{agentKey}")` -- stable across restarts without database sequences.
|
||||
|
||||
---
|
||||
|
||||
@@ -246,7 +264,115 @@ Dynamic tool definitions stored in PostgreSQL. Each tool defines a shell command
|
||||
|
||||
---
|
||||
|
||||
## 10. Database Schema
|
||||
## 10. Agent Link Store
|
||||
|
||||
The agent link store manages inter-agent delegation permissions -- directed edges that control which agents can delegate to which others.
|
||||
|
||||
### Table: `agent_links`
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `id` | UUID v7 | Primary key |
|
||||
| `source_agent_id` | UUID | Agent that can delegate (FK → agents) |
|
||||
| `target_agent_id` | UUID | Agent being delegated to (FK → agents) |
|
||||
| `direction` | VARCHAR(20) | `outbound` (A→B only), `bidirectional` (A↔B) |
|
||||
| `team_id` | UUID | Non-nil = auto-created by team setup (FK → agent_teams, SET NULL on delete) |
|
||||
| `description` | TEXT | Link description |
|
||||
| `max_concurrent` | INT | Per-link concurrency cap (default 3) |
|
||||
| `settings` | JSONB | Per-user deny/allow lists for fine-grained access control |
|
||||
| `status` | VARCHAR(20) | `active` or `disabled` |
|
||||
| `created_by` | VARCHAR | Audit trail |
|
||||
|
||||
**Constraints**: `UNIQUE(source_agent_id, target_agent_id)`, `CHECK (source_agent_id != target_agent_id)`
|
||||
|
||||
### Agent Search Columns (migration 000002)
|
||||
|
||||
The `agents` table gains three columns for agent discovery during delegation:
|
||||
|
||||
| Column | Type | Purpose |
|
||||
|--------|------|---------|
|
||||
| `frontmatter` | TEXT | Short expertise summary (distinct from `other_config.description` which is the summoning prompt) |
|
||||
| `tsv` | TSVECTOR | Auto-generated from `display_name + frontmatter`, GIN-indexed |
|
||||
| `embedding` | VECTOR(1536) | For cosine similarity search, HNSW-indexed |
|
||||
|
||||
### AgentLinkStore Interface (12 methods)
|
||||
|
||||
- **CRUD**: `CreateLink`, `DeleteLink`, `UpdateLink`, `GetLink`
|
||||
- **Queries**: `ListLinksFrom(agentID)`, `ListLinksTo(agentID)`
|
||||
- **Permission**: `CanDelegate(from, to)`, `GetLinkBetween(from, to)` (returns full link with Settings for per-user checks)
|
||||
- **Discovery**: `DelegateTargets(agentID)` (all targets with joined agent_key + display_name for DELEGATION.md), `SearchDelegateTargets` (FTS), `SearchDelegateTargetsByEmbedding` (vector cosine)
|
||||
|
||||
### Table: `delegation_history`
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `id` | UUID v7 | Primary key |
|
||||
| `source_agent_id` | UUID | Delegating agent |
|
||||
| `target_agent_id` | UUID | Target agent |
|
||||
| `team_id` | UUID | Team context (nullable) |
|
||||
| `team_task_id` | UUID | Related team task (nullable) |
|
||||
| `user_id` | VARCHAR | User who triggered the delegation |
|
||||
| `task` | TEXT | Task description sent to target |
|
||||
| `mode` | VARCHAR(10) | `sync` or `async` |
|
||||
| `status` | VARCHAR(20) | `completed`, `failed`, `cancelled` |
|
||||
| `result` | TEXT | Target agent's response |
|
||||
| `error` | TEXT | Error message on failure |
|
||||
| `iterations` | INT | Number of LLM iterations |
|
||||
| `trace_id` | UUID | Linked trace for observability |
|
||||
| `duration_ms` | INT | Wall-clock duration |
|
||||
| `completed_at` | TIMESTAMPTZ | Completion timestamp |
|
||||
|
||||
Every sync and async delegation is persisted here automatically via `SaveDelegationHistory()`. Results are truncated for WS transport (500 runes for list, 8000 runes for detail).
|
||||
|
||||
---
|
||||
|
||||
## 11. Team Store
|
||||
|
||||
The team store manages collaborative multi-agent teams with a shared task board, peer-to-peer mailbox, and handoff routing.
|
||||
|
||||
### Tables
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `agent_teams` | Team definitions | `name`, `lead_agent_id` (FK → agents), `status`, `settings` (JSONB) |
|
||||
| `agent_team_members` | Team membership | PK `(team_id, agent_id)`, `role` (lead/member) |
|
||||
| `team_tasks` | Shared task board | `subject`, `status` (pending/in_progress/completed/blocked), `owner_agent_id`, `blocked_by` (UUID[]), `priority`, `result`, `tsv` (FTS) |
|
||||
| `team_messages` | Peer-to-peer mailbox | `from_agent_id`, `to_agent_id` (NULL = broadcast), `content`, `message_type` (chat/broadcast), `read` |
|
||||
| `handoff_routes` | Active routing overrides | UNIQUE `(channel, chat_id)`, `from_agent_key`, `to_agent_key`, `reason` |
|
||||
|
||||
### TeamStore Interface (22 methods)
|
||||
|
||||
**Team CRUD**: `CreateTeam`, `GetTeam`, `DeleteTeam`, `ListTeams`
|
||||
|
||||
**Members**: `AddMember`, `RemoveMember`, `ListMembers`, `GetTeamForAgent` (find team by agent)
|
||||
|
||||
**Tasks**: `CreateTask`, `UpdateTask`, `ListTasks` (orderBy: priority/newest, statusFilter: active/completed/all), `GetTask`, `SearchTasks` (FTS on subject+description), `ClaimTask`, `CompleteTask`
|
||||
|
||||
**Delegation History**: `SaveDelegationHistory`, `ListDelegationHistory` (with filter opts), `GetDelegationHistory`
|
||||
|
||||
**Handoff Routes**: `SetHandoffRoute`, `GetHandoffRoute`, `ClearHandoffRoute`
|
||||
|
||||
**Messages**: `SendMessage`, `GetUnread`, `MarkRead`
|
||||
|
||||
### Atomic Task Claiming
|
||||
|
||||
Two agents grabbing the same task is prevented at the database level:
|
||||
|
||||
```sql
|
||||
UPDATE team_tasks
|
||||
SET status = 'in_progress', owner_agent_id = $1
|
||||
WHERE id = $2 AND status = 'pending' AND owner_agent_id IS NULL
|
||||
```
|
||||
|
||||
One row updated = claimed. Zero rows = someone else got it. Row-level locking, no distributed mutex needed.
|
||||
|
||||
### Task Dependencies
|
||||
|
||||
Tasks can declare `blocked_by` (UUID array) pointing to prerequisite tasks. When a task is completed via `CompleteTask`, all dependent tasks whose blockers are now all completed are automatically unblocked (status transitions from `blocked` to `pending`).
|
||||
|
||||
---
|
||||
|
||||
## 12. Database Schema
|
||||
|
||||
All tables use UUID v7 (time-ordered) as primary keys via `GenNewID()`.
|
||||
|
||||
@@ -263,6 +389,21 @@ flowchart TD
|
||||
AG --> UAP["user_agent_profiles"]
|
||||
end
|
||||
|
||||
subgraph "Agent Links"
|
||||
AG --> AL["agent_links"]
|
||||
AL --> DH["delegation_history"]
|
||||
end
|
||||
|
||||
subgraph Teams
|
||||
AT["agent_teams"] --> ATM["agent_team_members"]
|
||||
AT --> TT["team_tasks"]
|
||||
AT --> TM["team_messages"]
|
||||
end
|
||||
|
||||
subgraph Handoff
|
||||
HR["handoff_routes"]
|
||||
end
|
||||
|
||||
subgraph Sessions
|
||||
SE["sessions"]
|
||||
end
|
||||
@@ -304,22 +445,39 @@ flowchart TD
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `agents` | Agent definitions | `agent_key` (UNIQUE), `owner_id`, `agent_type` (open/predefined), `is_default`, soft delete via `deleted_at` |
|
||||
| `agents` | Agent definitions | `agent_key` (UNIQUE), `owner_id`, `agent_type` (open/predefined), `is_default`, `frontmatter`, `tsv`, `embedding`, soft delete via `deleted_at` |
|
||||
| `agent_shares` | Agent RBAC sharing | UNIQUE(agent_id, user_id), `role` (user/admin/operator) |
|
||||
| `agent_context_files` | Agent-level context | UNIQUE(agent_id, file_name) |
|
||||
| `user_context_files` | Per-user context | UNIQUE(agent_id, user_id, file_name) |
|
||||
| `user_agent_profiles` | User tracking | `first_seen_at`, `last_seen_at`, `workspace` |
|
||||
| `agent_links` | Inter-agent delegation permissions | UNIQUE(source, target), `direction`, `max_concurrent`, `settings` (JSONB) |
|
||||
| `agent_teams` | Team definitions | `name`, `lead_agent_id`, `status`, `settings` (JSONB) |
|
||||
| `agent_team_members` | Team membership | PK(team_id, agent_id), `role` (lead/member) |
|
||||
| `team_tasks` | Shared task board | `subject`, `status`, `owner_agent_id`, `blocked_by` (UUID[]), `tsv` (FTS) |
|
||||
| `team_messages` | Peer-to-peer mailbox | `from_agent_id`, `to_agent_id`, `message_type`, `read` |
|
||||
| `delegation_history` | Persisted delegation records | `source_agent_id`, `target_agent_id`, `mode`, `status`, `result`, `trace_id` |
|
||||
| `handoff_routes` | Active routing overrides | UNIQUE(channel, chat_id), `from_agent_key`, `to_agent_key` |
|
||||
| `sessions` | Conversation history | `session_key` (UNIQUE), `messages` (JSONB), `summary`, token counts |
|
||||
| `memory_documents` | Memory docs | UNIQUE(agent_id, COALESCE(user_id, ''), path) |
|
||||
| `memory_chunks` | Chunked + embedded text | `embedding` (VECTOR), `tsv` (TSVECTOR) |
|
||||
| `llm_providers` | Provider configuration | `api_key` (AES-256-GCM encrypted) |
|
||||
| `traces` | LLM call traces | `agent_id`, `user_id`, `status`, aggregated token counts |
|
||||
| `traces` | LLM call traces | `agent_id`, `user_id`, `status`, `parent_trace_id`, aggregated token counts |
|
||||
| `spans` | Individual operations | `span_type` (llm_call, tool_call, agent, embedding), `parent_span_id` |
|
||||
| `skills` | Skill definitions | Content, metadata, grants |
|
||||
| `cron_jobs` | Scheduled tasks | `schedule_kind` (at/every/cron), `payload` (JSONB) |
|
||||
| `mcp_servers` | MCP server configs | `transport`, `api_key` (encrypted), `tool_prefix` |
|
||||
| `custom_tools` | Dynamic tool definitions | `command` (template), `agent_id` (NULL = global), `env` (encrypted) |
|
||||
|
||||
### Migrations
|
||||
|
||||
| Migration | Purpose |
|
||||
|-----------|---------|
|
||||
| `000001_init_schema` | Core tables (agents, sessions, providers, memory, cron, pairing, skills, traces, MCP, custom tools) |
|
||||
| `000002_agent_links` | `agent_links` table + `frontmatter`, `tsv`, `embedding` on agents + `parent_trace_id` on traces |
|
||||
| `000003_agent_teams` | `agent_teams`, `agent_team_members`, `team_tasks`, `team_messages` + `team_id` on agent_links |
|
||||
| `000004_teams_v2` | FTS on `team_tasks` (tsv column) + `delegation_history` table |
|
||||
| `000005_phase4` | `handoff_routes` table |
|
||||
|
||||
### Required PostgreSQL Extensions
|
||||
|
||||
- **pgvector**: Vector similarity search for memory embeddings
|
||||
@@ -327,7 +485,7 @@ flowchart TD
|
||||
|
||||
---
|
||||
|
||||
## 11. Context Propagation
|
||||
## 13. Context Propagation
|
||||
|
||||
Metadata flows through `context.Context` instead of mutable state, ensuring thread safety across concurrent agent runs.
|
||||
|
||||
@@ -345,6 +503,7 @@ flowchart TD
|
||||
| `goclaw_user_id` | string | External user ID (e.g., Telegram user ID) |
|
||||
| `goclaw_agent_id` | uuid.UUID | Agent UUID (managed mode) |
|
||||
| `goclaw_agent_type` | string | Agent type: `"open"` or `"predefined"` |
|
||||
| `goclaw_sender_id` | string | Original individual sender ID (in group chats, `user_id` is group-scoped but `sender_id` preserves the actual person) |
|
||||
|
||||
### Tool Context Keys
|
||||
|
||||
@@ -355,10 +514,11 @@ flowchart TD
|
||||
| `tool_peer_kind` | Peer type: `"direct"` or `"group"` |
|
||||
| `tool_sandbox_key` | Docker sandbox scope key |
|
||||
| `tool_async_cb` | Callback for async tool execution |
|
||||
| `tool_workspace` | Per-user workspace directory (injected by agent loop, read by filesystem/shell tools) |
|
||||
|
||||
---
|
||||
|
||||
## 12. Key PostgreSQL Patterns
|
||||
## 14. Key PostgreSQL Patterns
|
||||
|
||||
### Database Driver
|
||||
|
||||
@@ -400,31 +560,41 @@ All "create or update" operations use `INSERT ... ON CONFLICT DO UPDATE`, ensuri
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `internal/store/stores.go` | `Stores` container struct (all 9 store interfaces) |
|
||||
| `internal/store/stores.go` | `Stores` container struct (all 14 store interfaces) |
|
||||
| `internal/store/types.go` | `BaseModel`, `StoreConfig`, `GenNewID()` |
|
||||
| `internal/store/context.go` | Context propagation: `WithUserID`, `WithAgentID`, `WithAgentType` |
|
||||
| `internal/store/context.go` | Context propagation: `WithUserID`, `WithAgentID`, `WithAgentType`, `WithSenderID` |
|
||||
| `internal/store/session_store.go` | `SessionStore` interface, `SessionData`, `SessionInfo` |
|
||||
| `internal/store/memory_store.go` | `MemoryStore` interface, `MemorySearchResult`, `EmbeddingProvider` |
|
||||
| `internal/store/skill_store.go` | `SkillStore` interface |
|
||||
| `internal/store/agent_store.go` | `AgentStore` interface |
|
||||
| `internal/store/agent_link_store.go` | `AgentLinkStore` interface, `AgentLinkData`, link constants |
|
||||
| `internal/store/team_store.go` | `TeamStore` interface, `TeamData`, `TeamTaskData`, `DelegationHistoryData`, `HandoffRouteData`, `TeamMessageData` |
|
||||
| `internal/store/provider_store.go` | `ProviderStore` interface |
|
||||
| `internal/store/tracing_store.go` | `TracingStore` interface, `TraceData`, `SpanData` |
|
||||
| `internal/store/mcp_store.go` | `MCPServerStore` interface, grant types, access request types |
|
||||
| `internal/store/channel_instance_store.go` | `ChannelInstanceStore` interface |
|
||||
| `internal/store/config_secrets_store.go` | `ConfigSecretsStore` interface |
|
||||
| `internal/store/pairing_store.go` | `PairingStore` interface |
|
||||
| `internal/store/cron_store.go` | `CronStore` interface |
|
||||
| `internal/store/custom_tool_store.go` | `CustomToolStore` interface |
|
||||
| `internal/store/file/agents.go` | `FileAgentStore`: filesystem + SQLite backend for standalone mode |
|
||||
| `internal/store/pg/factory.go` | PG store factory: creates all PG store instances from a connection pool |
|
||||
| `internal/store/pg/sessions.go` | `PGSessionStore`: session cache, Save, GetOrCreate |
|
||||
| `internal/store/pg/agents.go` | `PGAgentStore`: CRUD, soft delete, access control |
|
||||
| `internal/store/pg/agents_context.go` | Agent and user context file operations |
|
||||
| `internal/store/pg/agent_links.go` | `PGAgentLinkStore`: link CRUD, permissions, FTS + vector search |
|
||||
| `internal/store/pg/teams.go` | `PGTeamStore`: teams, tasks (atomic claim), messages, delegation history, handoff routes |
|
||||
| `internal/store/pg/memory_docs.go` | `PGMemoryStore`: document CRUD, indexing, chunking |
|
||||
| `internal/store/pg/memory_search.go` | Hybrid search: FTS, vector, ILIKE fallback, merge |
|
||||
| `internal/store/pg/skills.go` | `PGSkillStore`: skill CRUD and grants |
|
||||
| `internal/store/pg/skills_grants.go` | Skill agent and user grants |
|
||||
| `internal/store/pg/mcp_servers.go` | `PGMCPServerStore`: server CRUD, grants, access requests |
|
||||
| `internal/store/pg/channel_instances.go` | `PGChannelInstanceStore`: channel instance CRUD |
|
||||
| `internal/store/pg/config_secrets.go` | `PGConfigSecretsStore`: encrypted config secrets |
|
||||
| `internal/store/pg/custom_tools.go` | `PGCustomToolStore`: custom tool CRUD with encrypted env |
|
||||
| `internal/store/pg/providers.go` | `PGProviderStore`: provider CRUD with encrypted keys |
|
||||
| `internal/store/pg/tracing.go` | `PGTracingStore`: traces and spans with batch insert |
|
||||
| `internal/store/pg/pool.go` | Connection pool management |
|
||||
| `internal/store/pg/helpers.go` | Nullable helpers, JSON helpers, `execMapUpdate()` |
|
||||
| `internal/store/validate.go` | Input validation utilities |
|
||||
| `internal/tools/context_keys.go` | Tool context keys including `WithToolWorkspace` |
|
||||
@@ -71,6 +71,16 @@ flowchart TD
|
||||
SA2 -->|No| CREATE1["Create template file on disk"]
|
||||
end
|
||||
|
||||
subgraph "Standalone Mode -- Per-User (FileAgentStore)"
|
||||
SU["SeedUserFiles()"] --> SU1{"Agent type?"}
|
||||
SU1 -->|open| SU_OPEN["Seed all 7 files to SQLite"]
|
||||
SU1 -->|predefined| SU_PRED["Seed USER.md + BOOTSTRAP.md to SQLite"]
|
||||
SU_OPEN --> SU_CHK{"Row already exists?"}
|
||||
SU_PRED --> SU_CHK
|
||||
SU_CHK -->|Yes| SKIP_SU["Skip"]
|
||||
SU_CHK -->|No| SU_WRITE["INSERT into user_context_files"]
|
||||
end
|
||||
|
||||
subgraph "Managed Mode -- Agent Level"
|
||||
SB["SeedToStore()"] --> SB1{"Agent type = open?"}
|
||||
SB1 -->|Yes| SKIP_AGENT["Skip (open agents use per-user only)"]
|
||||
@@ -83,7 +93,7 @@ flowchart TD
|
||||
subgraph "Managed Mode -- Per-User"
|
||||
MC["SeedUserFiles()"] --> MC1{"Agent type?"}
|
||||
MC1 -->|open| OPEN["Seed all 7 files to user_context_files"]
|
||||
MC1 -->|predefined| PRED["Seed only USER.md to user_context_files"]
|
||||
MC1 -->|predefined| PRED["Seed USER.md + BOOTSTRAP.md to user_context_files"]
|
||||
OPEN --> CHECK{"File already has content?"}
|
||||
PRED --> CHECK
|
||||
CHECK -->|Yes| SKIP3["Skip -- never overwrite"]
|
||||
@@ -93,22 +103,35 @@ flowchart TD
|
||||
|
||||
`SeedUserFiles()` is idempotent -- safe to call multiple times without overwriting personalized content.
|
||||
|
||||
### Standalone UUID Generation
|
||||
|
||||
Standalone agents are defined in `config.json` without database-generated UUIDs. `FileAgentStore` uses UUID v5 (`uuid.NewSHA1(namespace, "goclaw-standalone:{agentKey}")`) to produce deterministic IDs from agent keys. This ensures SQLite rows for per-user files survive process restarts without coordination.
|
||||
|
||||
### Predefined Agent Bootstrap
|
||||
|
||||
Both standalone and managed mode now seed `BOOTSTRAP.md` for predefined agents (per-user). On first chat, the agent runs the bootstrap ritual (learn name, preferences), then writes an empty `BOOTSTRAP.md` which triggers deletion. The empty-write deletion is ordered *before* the predefined write-block in `ContextFileInterceptor` to prevent an infinite bootstrap loop.
|
||||
|
||||
---
|
||||
|
||||
## 4. Agent Type Routing
|
||||
|
||||
Two agent types determine which context files live at the agent level versus the per-user level.
|
||||
Two agent types determine which context files live at the agent level versus the per-user level. Agent types are now available in both managed and standalone modes.
|
||||
|
||||
| Agent Type | Agent-Level Files | Per-User Files |
|
||||
|------------|-------------------|----------------|
|
||||
| `open` | None | All 7 files (AGENTS, SOUL, TOOLS, IDENTITY, USER, HEARTBEAT, BOOTSTRAP) |
|
||||
| `predefined` | 6 files (shared across all users) | Only USER.md |
|
||||
| `predefined` | 6 files (shared across all users) | USER.md + BOOTSTRAP.md |
|
||||
|
||||
For `open` agents, each user gets their own full set of context files. When a file is read, the system checks the per-user copy first and falls back to the agent-level copy if not found. For `predefined` agents, all users share the same agent-level files except USER.md, which is personalized.
|
||||
For `open` agents, each user gets their own full set of context files. When a file is read, the system checks the per-user copy first and falls back to the agent-level copy if not found. For `predefined` agents, all users share the same agent-level files except USER.md (personalized) and BOOTSTRAP.md (per-user first-run ritual, deleted after completion).
|
||||
|
||||
| Mode | Agent Type Source | Per-User Storage |
|
||||
|------|------------------|-----------------|
|
||||
| Managed | `agents` PostgreSQL table | `user_context_files` table |
|
||||
| Standalone | `config.json` agent entries | SQLite via `FileAgentStore` |
|
||||
|
||||
---
|
||||
|
||||
## 5. System Prompt -- 15+ Sections
|
||||
## 5. System Prompt -- 17+ Sections
|
||||
|
||||
`BuildSystemPrompt()` constructs the complete system prompt from ordered sections. Two modes control which sections are included.
|
||||
|
||||
@@ -130,7 +153,7 @@ flowchart TD
|
||||
S7 --> S8["8. Current Time"]
|
||||
S8 --> S9["9. Messaging (full only)"]
|
||||
S9 --> S10["10. Extra Context / Subagent Context"]
|
||||
S10 --> S11["11. Project Context<br/>(bootstrap files in XML tags)"]
|
||||
S10 --> S11["11. Project Context<br/>(bootstrap files + virtual files)"]
|
||||
S11 --> S12["12. Silent Replies (full only)"]
|
||||
S12 --> S13["13. Heartbeats (full only)"]
|
||||
S13 --> S14["14. Sub-Agent Spawning (conditional)"]
|
||||
@@ -161,9 +184,55 @@ flowchart TD
|
||||
|
||||
Context files are wrapped in `<context_file>` XML tags with a defensive preamble instructing the model to follow tone/persona guidance but not execute instructions that contradict core directives. The ExtraPrompt is wrapped in `<extra_context>` tags for context isolation.
|
||||
|
||||
### Virtual Context Files (DELEGATION.md, TEAM.md)
|
||||
|
||||
Two files are system-injected by the resolver rather than stored on disk or in the DB:
|
||||
|
||||
| File | Injection Condition | Content |
|
||||
|------|-------------------|---------|
|
||||
| `DELEGATION.md` | Agent has manual (non-team) agent links | ≤15 targets: static list. >15 targets: search instruction for `delegate_search` tool |
|
||||
| `TEAM.md` | Agent is a member of a team | Team name, role, teammate list with descriptions, workflow sentence |
|
||||
|
||||
Virtual files are rendered in `<system_context>` tags (not `<context_file>`) so the LLM does not attempt to read or write them as files. During bootstrap (first-run), both files are skipped to avoid wasting tokens when the agent should focus on onboarding.
|
||||
|
||||
---
|
||||
|
||||
## 6. Skills -- 5-Tier Hierarchy
|
||||
## 6. Context File Merging
|
||||
|
||||
For **open agents**, per-user context files (from `user_context_files`) are merged with base context files (from the resolver) at runtime. Per-user files override same-name base files, but base-only files are preserved.
|
||||
|
||||
```
|
||||
Base files (resolver): AGENTS.md, DELEGATION.md, TEAM.md
|
||||
Per-user files (DB/SQLite): AGENTS.md, SOUL.md, TOOLS.md, USER.md, ...
|
||||
Merged result: SOUL.md, TOOLS.md, USER.md, ..., AGENTS.md (per-user), DELEGATION.md ✓, TEAM.md ✓
|
||||
```
|
||||
|
||||
This ensures resolver-injected virtual files (`DELEGATION.md`, `TEAM.md`) survive alongside per-user customizations. The merge logic lives in `internal/agent/loop_history.go`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Agent Summoning (Managed Mode)
|
||||
|
||||
Creating a predefined agent requires 5 context files (SOUL.md, IDENTITY.md, AGENTS.md, TOOLS.md, HEARTBEAT.md) with specific formatting conventions. Agent summoning generates all 5 files from a natural language description in a single LLM call.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
USER["User: 'sarcastic Rust reviewer'"] --> API["Backend (POST /v1/agents/{id}/summon)"]
|
||||
API -->|"status: summoning"| DB["Database"]
|
||||
API --> LLM["LLM call with structured XML prompt"]
|
||||
LLM --> PARSE["Parse XML output into 5 files"]
|
||||
PARSE --> STORE["Write files to agent_context_files"]
|
||||
STORE -->|"status: active"| READY["Agent ready"]
|
||||
LLM -.->|"WS events"| UI["Dashboard modal with progress"]
|
||||
```
|
||||
|
||||
The LLM outputs structured XML with each file in a tagged block. Parsing is done server-side in `internal/http/summoner.go`. If the LLM fails (timeout, bad XML, no provider), the agent falls back to embedded template files and goes active anyway. The user can retry via "Edit with AI" later.
|
||||
|
||||
**Why not `write_file`?** The `ContextFileInterceptor` blocks predefined file writes from chat by design. Bypassing it would create a security hole. Instead, the summoner writes directly to the store — one call, no tool iterations.
|
||||
|
||||
---
|
||||
|
||||
## 8. Skills -- 5-Tier Hierarchy
|
||||
|
||||
Skills are loaded from multiple directories with a priority ordering. Higher-tier skills override lower-tier skills with the same name.
|
||||
|
||||
@@ -183,7 +252,7 @@ Each skill directory contains a `SKILL.md` file with YAML/JSON frontmatter (`nam
|
||||
|
||||
---
|
||||
|
||||
## 7. Skills -- Inline vs Search Mode
|
||||
## 9. Skills -- Inline vs Search Mode
|
||||
|
||||
The system dynamically decides whether to embed skill summaries directly in the prompt (inline mode) or instruct the agent to use the `skill_search` tool (search mode).
|
||||
|
||||
@@ -198,7 +267,7 @@ This decision is re-evaluated each time the system prompt is built, so newly hot
|
||||
|
||||
---
|
||||
|
||||
## 8. Skills -- BM25 Search
|
||||
## 10. Skills -- BM25 Search
|
||||
|
||||
An in-memory BM25 index provides keyword-based skill search. The index is lazily rebuilt whenever the skill version changes.
|
||||
|
||||
@@ -216,7 +285,7 @@ IDF is computed as: `log((N - df + 0.5) / (df + 0.5) + 1)`
|
||||
|
||||
---
|
||||
|
||||
## 9. Skills -- Embedding Search (Managed Mode)
|
||||
## 11. Skills -- Embedding Search (Managed Mode)
|
||||
|
||||
In managed mode, skill search uses a hybrid approach combining BM25 and vector similarity.
|
||||
|
||||
@@ -239,7 +308,7 @@ flowchart TD
|
||||
|
||||
---
|
||||
|
||||
## 10. Skills Grants & Visibility (Managed Mode)
|
||||
## 12. Skills Grants & Visibility (Managed Mode)
|
||||
|
||||
In managed mode, skill access is controlled through a 3-tier visibility model with explicit agent and user grants.
|
||||
|
||||
@@ -275,7 +344,7 @@ flowchart TD
|
||||
|
||||
---
|
||||
|
||||
## 11. Hot-Reload
|
||||
## 13. Hot-Reload
|
||||
|
||||
An fsnotify-based watcher monitors all skill directories for changes to SKILL.md files.
|
||||
|
||||
@@ -290,7 +359,7 @@ New skill directories created inside a watched root are automatically added to t
|
||||
|
||||
---
|
||||
|
||||
## 11. Memory -- Indexing Pipeline
|
||||
## 14. Memory -- Indexing Pipeline
|
||||
|
||||
Memory documents are chunked, embedded, and stored for hybrid search.
|
||||
|
||||
@@ -321,7 +390,7 @@ flowchart TD
|
||||
|
||||
---
|
||||
|
||||
## 12. Hybrid Search
|
||||
## 15. Hybrid Search
|
||||
|
||||
Combines full-text search and vector search with weighted merging.
|
||||
|
||||
@@ -351,7 +420,7 @@ When both FTS and vector search return results, scores are merged using the weig
|
||||
|
||||
---
|
||||
|
||||
## 13. Memory Flush -- Pre-Compaction
|
||||
## 16. Memory Flush -- Pre-Compaction
|
||||
|
||||
Before session history is compacted (summarized + truncated), the agent is given an opportunity to write durable memories to disk.
|
||||
|
||||
@@ -391,8 +460,13 @@ The flush is idempotent per compaction cycle -- it will not run again until the
|
||||
| `internal/bootstrap/seed_store.go` | Managed mode seeding (SeedToStore, SeedUserFiles) |
|
||||
| `internal/bootstrap/load_store.go` | Load context files from DB (LoadFromStore) |
|
||||
| `internal/bootstrap/templates/*.md` | Embedded template files |
|
||||
| `internal/agent/systemprompt.go` | System prompt builder (BuildSystemPrompt, 15+ sections) |
|
||||
| `internal/agent/systemprompt.go` | System prompt builder (BuildSystemPrompt, 17+ sections) |
|
||||
| `internal/agent/systemprompt_sections.go` | Section renderers, virtual file handling (DELEGATION.md, TEAM.md) |
|
||||
| `internal/agent/resolver.go` | Agent resolution, DELEGATION.md + TEAM.md injection |
|
||||
| `internal/agent/loop_history.go` | Context file merging (base + per-user, base-only preserved) |
|
||||
| `internal/agent/memoryflush.go` | Memory flush logic (shouldRunMemoryFlush, runMemoryFlush) |
|
||||
| `internal/store/file/agents.go` | FileAgentStore -- filesystem + SQLite backend for standalone |
|
||||
| `internal/http/summoner.go` | Agent summoning -- LLM-powered context file generation |
|
||||
| `internal/skills/loader.go` | Skill loader (5-tier hierarchy, BuildSummary, filtering) |
|
||||
| `internal/skills/search.go` | BM25 search index (tokenization, IDF scoring) |
|
||||
| `internal/skills/watcher.go` | fsnotify watcher (500ms debounce, version bumping) |
|
||||
|
||||
@@ -30,6 +30,11 @@ flowchart TD
|
||||
S4["Subagent 4"]
|
||||
end
|
||||
|
||||
subgraph "Lane: delegate (concurrency = 100)"
|
||||
D1["Delegation 1"]
|
||||
D2["Delegation 2"]
|
||||
end
|
||||
|
||||
subgraph "Lane: cron (concurrency = 1)"
|
||||
C1["Cron job"]
|
||||
end
|
||||
@@ -42,44 +47,35 @@ flowchart TD
|
||||
|
||||
### Lane Defaults
|
||||
|
||||
| Lane | Concurrency | Purpose |
|
||||
|------|:-----------:|---------|
|
||||
| `main` | 2 | Primary user chat sessions |
|
||||
| `subagent` | 4 | Sub-agents spawned by the main agent |
|
||||
| `cron` | 1 | Scheduled cron jobs (sequential to avoid conflicts) |
|
||||
| Lane | Concurrency | Env Override | Purpose |
|
||||
|------|:-----------:|-------------|---------|
|
||||
| `main` | 2 | `GOCLAW_LANE_MAIN` | Primary user chat sessions |
|
||||
| `subagent` | 4 | `GOCLAW_LANE_SUBAGENT` | Sub-agents spawned by the main agent |
|
||||
| `delegate` | 100 | `GOCLAW_LANE_DELEGATE` | Agent delegation executions |
|
||||
| `cron` | 1 | `GOCLAW_LANE_CRON` | Scheduled cron jobs (sequential to avoid conflicts) |
|
||||
|
||||
`GetOrCreate()` allows creating new lanes on demand with custom concurrency. Session serialization ensures only one agent run executes at a time per session key.
|
||||
`GetOrCreate()` allows creating new lanes on demand with custom concurrency. All lane concurrency values are configurable via environment variables.
|
||||
|
||||
---
|
||||
|
||||
## 2. Session Queue -- 3 Modes
|
||||
## 2. Session Queue
|
||||
|
||||
Each session key gets a dedicated queue that serializes agent runs. Only one run is active at a time; additional messages are queued according to the configured mode.
|
||||
Each session key gets a dedicated queue that manages agent runs. The queue supports configurable concurrent runs per session.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Enqueue: Message arrives
|
||||
Enqueue --> Running: No active run
|
||||
Enqueue --> Queued: Active run exists
|
||||
### Concurrent Runs
|
||||
|
||||
state "Queue Mode" as QM {
|
||||
Queued --> Running: Previous run completed
|
||||
}
|
||||
| Context | `maxConcurrent` | Rationale |
|
||||
|---------|:--------------:|-----------|
|
||||
| DMs | 1 | Single-threaded per user (no interleaving) |
|
||||
| Groups | 3 | Multiple users can get responses in parallel |
|
||||
|
||||
state "Interrupt Mode" as IM {
|
||||
Queued --> CancelActive: Cancel current run
|
||||
CancelActive --> Running: Start immediately
|
||||
}
|
||||
|
||||
Running --> [*]: Completed
|
||||
Running --> Queued: Next message waiting
|
||||
```
|
||||
**Adaptive throttle**: When session history exceeds 60% of the context window, concurrency drops to 1 to prevent context window overflow.
|
||||
|
||||
### Queue Modes
|
||||
|
||||
| Mode | Behavior |
|
||||
|------|----------|
|
||||
| `queue` (default) | FIFO -- messages wait until the current run completes |
|
||||
| `queue` (default) | FIFO -- messages wait until a run slot is available |
|
||||
| `followup` | Same as `queue` -- messages are queued as follow-ups |
|
||||
| `interrupt` | Cancel the active run, drain the queue, start the new message immediately |
|
||||
|
||||
@@ -103,7 +99,26 @@ When the queue reaches capacity, one of two drop policies applies.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cron Lifecycle
|
||||
## 3. /stop and /stopall Commands
|
||||
|
||||
Cancel commands for Telegram and other channels.
|
||||
|
||||
| Command | Behavior |
|
||||
|---------|----------|
|
||||
| `/stop` | Cancel the oldest running task; others keep going |
|
||||
| `/stopall` | Cancel all running tasks + drain the queue |
|
||||
|
||||
### Implementation Details
|
||||
|
||||
- **Debouncer bypass**: `/stop` and `/stopall` are intercepted before the 800ms debouncer to avoid being merged with the next user message
|
||||
- **Cancel mechanism**: `SessionQueue.Cancel()` exposes the `CancelFunc` from the scheduler. Context cancellation propagates to the agent loop
|
||||
- **Empty outbound**: On cancel, an empty outbound message is published to trigger cleanup (stop typing indicator, clear reactions)
|
||||
- **Trace finalization**: When `ctx.Err() != nil`, trace finalization falls back to `context.Background()` for the final DB write. Status is set to `"cancelled"`
|
||||
- **Context survival**: Context values (traceID, collector) survive cancellation -- only the Done channel fires
|
||||
|
||||
---
|
||||
|
||||
## 4. Cron Lifecycle
|
||||
|
||||
Scheduled tasks that run agent turns automatically. The run loop checks every second for due jobs.
|
||||
|
||||
@@ -147,7 +162,7 @@ Jobs can be `active` or `paused`. Paused jobs skip execution during the due chec
|
||||
|
||||
---
|
||||
|
||||
## 4. Heartbeat -- 5 Steps
|
||||
## 5. Heartbeat -- 5 Steps
|
||||
|
||||
Periodically wakes the agent to check on events (calendar, inbox, alerts) and surfaces anything that needs attention.
|
||||
|
||||
|
||||
+62
-4
@@ -14,7 +14,7 @@ flowchart TD
|
||||
L1 --> L2["Layer 2: Input<br/>Injection detection (6 patterns), message truncation"]
|
||||
L2 --> L3["Layer 3: Tool<br/>Shell deny patterns, path traversal, SSRF, exec approval"]
|
||||
L3 --> L4["Layer 4: Output<br/>Credential scrubbing, content wrapping"]
|
||||
L4 --> L5["Layer 5: Isolation<br/>Docker sandbox, read-only root FS, network restrictions"]
|
||||
L4 --> L5["Layer 5: Isolation<br/>Workspace isolation, Docker sandbox, read-only FS"]
|
||||
```
|
||||
|
||||
### Layer 1: Transport Security
|
||||
@@ -77,6 +77,16 @@ flowchart TD
|
||||
|
||||
**Path traversal**: `resolvePath()` applies `filepath.Clean()` then `HasPrefix()` to ensure all paths stay within the workspace. With `restrict = true`, any path outside the workspace is blocked.
|
||||
|
||||
**PathDenyable** -- An interface that lets filesystem tools reject specific path prefixes:
|
||||
|
||||
```go
|
||||
type PathDenyable interface {
|
||||
DenyPaths(...string)
|
||||
}
|
||||
```
|
||||
|
||||
All four filesystem tools (`read_file`, `write_file`, `list_files`, `edit`) implement `PathDenyable`. The agent loop calls `DenyPaths(".goclaw")` at startup to prevent agents from accessing internal data directories. `list_files` additionally filters denied directories from output entirely -- the agent does not see denied paths in directory listings.
|
||||
|
||||
### Layer 4: Output Security
|
||||
|
||||
| Mechanism | Detail |
|
||||
@@ -84,7 +94,18 @@ flowchart TD
|
||||
| Credential scrubbing | Regex detection of: OpenAI (`sk-...`), Anthropic (`sk-ant-...`), GitHub (`ghp_/gho_/ghu_/ghs_/ghr_`), AWS (`AKIA...`), generic key-value patterns. All replaced with `[REDACTED]`. |
|
||||
| Web content wrapping | Fetched content wrapped in `<<<EXTERNAL_UNTRUSTED_CONTENT>>>` tags with security warning |
|
||||
|
||||
### Layer 5: Isolation (Docker Sandbox)
|
||||
### Layer 5: Isolation
|
||||
|
||||
**Per-user workspace isolation** -- Two levels prevent cross-user file access:
|
||||
|
||||
| Level | Scope | Directory Pattern |
|
||||
|-------|-------|------------------|
|
||||
| Per-agent | Each agent gets its own base directory | `~/.goclaw/{agent-key}-workspace/` |
|
||||
| Per-user | Each user gets a subdirectory within the agent workspace | `{agent-workspace}/user_{sanitized_id}/` |
|
||||
|
||||
The workspace is injected into tools via `WithToolWorkspace(ctx)` context injection. Tools read the workspace from context at execution time (fallback to the struct field for backward compatibility). User IDs are sanitized: anything outside `[a-zA-Z0-9_-]` becomes an underscore (`group:telegram:-1001234` → `group_telegram_-1001234`).
|
||||
|
||||
**Docker sandbox** -- Container-based isolation for shell command execution:
|
||||
|
||||
| Hardening | Configuration |
|
||||
|-----------|---------------|
|
||||
@@ -255,6 +276,37 @@ Filter all security events by grepping for the `security.` prefix in log output.
|
||||
|
||||
---
|
||||
|
||||
## 7. Hook Recursion Prevention
|
||||
|
||||
The hook system (quality gates) can trigger infinite recursion: an agent evaluator delegates to a reviewer → delegation completes → fires quality gate → delegates to reviewer again → infinite loop.
|
||||
|
||||
A context flag `hooks.WithSkipHooks(ctx, true)` prevents this. Three injection points set the flag:
|
||||
|
||||
| Injection Point | Why |
|
||||
|----------------|-----|
|
||||
| Agent evaluator | Delegating to the reviewer for quality checks must not re-trigger gates |
|
||||
| Evaluate-optimize loop | All internal generator/evaluator delegations skip gates |
|
||||
| Agent eval callback (cmd layer) | When the hook engine itself triggers delegation |
|
||||
|
||||
`DelegateManager.Delegate()` checks `hooks.SkipHooksFromContext(ctx)` before applying quality gates. If the flag is set, gates are skipped entirely.
|
||||
|
||||
---
|
||||
|
||||
## 8. Delegation Security
|
||||
|
||||
Agent delegation uses directed permissions via the `agent_links` table.
|
||||
|
||||
| Control | Scope | Description |
|
||||
|---------|-------|-------------|
|
||||
| Directed links | A → B | A single row `(A→B, outbound)` means A can delegate to B, not the reverse |
|
||||
| Per-user deny/allow | Per-link | `settings` JSONB on each link holds per-user restrictions (premium users only, blocked accounts) |
|
||||
| Per-link concurrency | A → B | `agent_links.max_concurrent` limits simultaneous delegations from A to B |
|
||||
| Per-agent load cap | B (all sources) | `other_config.max_delegation_load` limits total concurrent delegations targeting B |
|
||||
|
||||
When concurrency limits are hit, the error message is written for LLM reasoning: *"Agent at capacity (5/5). Try a different agent or handle it yourself."*
|
||||
|
||||
---
|
||||
|
||||
## File Reference
|
||||
|
||||
| File | Description |
|
||||
@@ -267,6 +319,11 @@ Filter all security events by grepping for the `security.` prefix in log output.
|
||||
| `internal/gateway/ratelimit.go` | Gateway-level token bucket rate limiter |
|
||||
| `internal/sandbox/` | Docker sandbox manager, FsBridge |
|
||||
| `internal/crypto/aes.go` | AES-256-GCM encrypt/decrypt |
|
||||
| `internal/tools/types.go` | PathDenyable interface definition |
|
||||
| `internal/tools/filesystem.go` | Denied path checking (`checkDeniedPath` helper) |
|
||||
| `internal/tools/filesystem_list.go` | Denied path support + directory filtering |
|
||||
| `internal/hooks/context.go` | WithSkipHooks / SkipHooksFromContext (recursion prevention) |
|
||||
| `internal/hooks/engine.go` | Hook engine, evaluator registry |
|
||||
|
||||
---
|
||||
|
||||
@@ -274,8 +331,9 @@ Filter all security events by grepping for the `security.` prefix in log output.
|
||||
|
||||
| Document | Relevant Content |
|
||||
|----------|-----------------|
|
||||
| [03-tools-system.md](./03-tools-system.md) | Shell deny patterns, exec approval, policy engine |
|
||||
| [03-tools-system.md](./03-tools-system.md) | Shell deny patterns, exec approval, PathDenyable, delegation system, quality gates |
|
||||
| [04-gateway-protocol.md](./04-gateway-protocol.md) | WebSocket auth, RBAC, rate limiting |
|
||||
| [06-store-data-model.md](./06-store-data-model.md) | API key encryption, agent access control pipeline |
|
||||
| [06-store-data-model.md](./06-store-data-model.md) | API key encryption, agent access control pipeline, agent_links table |
|
||||
| [07-bootstrap-skills-memory.md](./07-bootstrap-skills-memory.md) | Context file merging, virtual files |
|
||||
| [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md) | Scheduler lanes, cron lifecycle |
|
||||
| [10-tracing-observability.md](./10-tracing-observability.md) | Tracing and OTel export |
|
||||
@@ -28,6 +28,12 @@ flowchart LR
|
||||
ES --> FT["FinishTrace()<br/>(status, error, output preview)"]
|
||||
```
|
||||
|
||||
### Cancel Handling
|
||||
|
||||
When a run is cancelled via `/stop` or `/stopall`, the run context is cancelled but trace finalization still needs to persist. `FinishTrace()` detects `ctx.Err() != nil` and switches to `context.Background()` for the final database write. The trace status is set to `"cancelled"` instead of `"error"`.
|
||||
|
||||
Context values (traceID, collector) survive cancellation -- only `ctx.Done()` and `ctx.Err()` change. This allows trace finalization to find everything it needs with a fresh context for the DB call.
|
||||
|
||||
---
|
||||
|
||||
## 2. Span Types & Hierarchy
|
||||
@@ -110,13 +116,27 @@ The exporter lives in a separate sub-package (`internal/tracing/otelexport/`) so
|
||||
|-----------|------|-------------|
|
||||
| `agent_id` | UUID | Filter by agent |
|
||||
| `user_id` | string | Filter by user |
|
||||
| `status` | string | Filter by status (running, success, error) |
|
||||
| `status` | string | Filter by status (running, success, error, cancelled) |
|
||||
| `from` / `to` | timestamp | Date range filter |
|
||||
| `limit` | int | Page size (default 50) |
|
||||
| `offset` | int | Pagination offset |
|
||||
|
||||
---
|
||||
|
||||
## 6. Delegation History (Managed Mode)
|
||||
|
||||
Delegation history records are stored in the `delegation_history` table and exposed alongside traces for cross-referencing agent interactions.
|
||||
|
||||
| Channel | Endpoint | Details |
|
||||
|---------|----------|---------|
|
||||
| WebSocket RPC | `delegations.list` / `delegations.get` | Results truncated (500 runes for list, 8000 for detail) |
|
||||
| HTTP API | `GET /v1/delegations` / `GET /v1/delegations/{id}` | Full records |
|
||||
| Agent tool | `delegate(action="history")` | Agent self-checking past delegations |
|
||||
|
||||
Delegation history is automatically recorded by `DelegateManager.saveDelegationHistory()` for every delegation (sync/async). Each record includes source agent, target agent, input, result, duration, and status.
|
||||
|
||||
---
|
||||
|
||||
## File Reference
|
||||
|
||||
| File | Description |
|
||||
@@ -128,6 +148,8 @@ The exporter lives in a separate sub-package (`internal/tracing/otelexport/`) so
|
||||
| `internal/store/pg/tracing.go` | PostgreSQL trace/span persistence + aggregation |
|
||||
| `internal/http/traces.go` | Trace HTTP API handler (GET /v1/traces) |
|
||||
| `internal/agent/loop_tracing.go` | Span emission from agent loop (LLM, tool, agent spans) |
|
||||
| `internal/http/delegations.go` | Delegation history HTTP API handler |
|
||||
| `internal/gateway/methods/delegations.go` | Delegation history RPC handlers |
|
||||
|
||||
---
|
||||
|
||||
@@ -135,6 +157,8 @@ The exporter lives in a separate sub-package (`internal/tracing/otelexport/`) so
|
||||
|
||||
| Document | Relevant Content |
|
||||
|----------|-----------------|
|
||||
| [01-agent-loop.md](./01-agent-loop.md) | Span emission during agent execution |
|
||||
| [06-store-data-model.md](./06-store-data-model.md) | traces/spans tables schema |
|
||||
| [01-agent-loop.md](./01-agent-loop.md) | Span emission during agent execution, cancel handling |
|
||||
| [03-tools-system.md](./03-tools-system.md) | Delegation system, delegation history via agent tool |
|
||||
| [06-store-data-model.md](./06-store-data-model.md) | traces/spans tables schema, delegation_history table |
|
||||
| [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md) | /stop and /stopall commands |
|
||||
| [09-security.md](./09-security.md) | Rate limiting, RBAC access control |
|
||||
@@ -69,6 +69,8 @@ func Load(path string) (*Config, error) {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
cfg.applyEnvOverrides()
|
||||
cfg.applyContextPruningDefaults()
|
||||
return cfg, nil
|
||||
}
|
||||
return nil, fmt.Errorf("read config: %w", err)
|
||||
|
||||
Executable
+133
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env bash
|
||||
# prepare-env.sh — Create or update .env with auto-generated secrets.
|
||||
# Safe to run multiple times: only fills in missing values, never overwrites existing ones.
|
||||
#
|
||||
# Usage:
|
||||
# ./prepare-env.sh # interactive: prompts for provider key
|
||||
# ./prepare-env.sh --quiet # non-interactive: skip prompts, only generate secrets
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ENV_FILE=".env"
|
||||
QUIET=false
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--quiet|-q) QUIET=true ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --- helpers ---
|
||||
|
||||
gen_hex() { openssl rand -hex "$1" 2>/dev/null || head -c "$1" /dev/urandom | xxd -p | tr -d '\n'; }
|
||||
|
||||
# Read current value from .env (KEY=VALUE format, no export prefix).
|
||||
get_env_val() {
|
||||
local key="$1"
|
||||
if [ -f "$ENV_FILE" ]; then
|
||||
grep -E "^${key}=" "$ENV_FILE" 2>/dev/null | tail -1 | cut -d'=' -f2-
|
||||
fi
|
||||
}
|
||||
|
||||
# Set a key in .env. Appends if missing, replaces if empty.
|
||||
set_env_val() {
|
||||
local key="$1" val="$2"
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "${key}=${val}" >> "$ENV_FILE"
|
||||
elif grep -qE "^${key}=" "$ENV_FILE" 2>/dev/null; then
|
||||
# Key exists — only replace if current value is empty
|
||||
local current
|
||||
current="$(get_env_val "$key")"
|
||||
if [ -z "$current" ]; then
|
||||
if [[ "$OSTYPE" == "darwin"* ]]; then
|
||||
sed -i '' "s|^${key}=.*|${key}=${val}|" "$ENV_FILE"
|
||||
else
|
||||
sed -i "s|^${key}=.*|${key}=${val}|" "$ENV_FILE"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
echo "${key}=${val}" >> "$ENV_FILE"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- main ---
|
||||
|
||||
echo "=== GoClaw Environment Preparation ==="
|
||||
echo ""
|
||||
|
||||
# 1. Create .env from .env.example if it doesn't exist
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
if [ -f ".env.example" ]; then
|
||||
# Strip 'export ' prefix for Docker Compose compatibility
|
||||
sed 's/^export //' .env.example > "$ENV_FILE"
|
||||
echo "[created] .env from .env.example (stripped 'export' prefix)"
|
||||
else
|
||||
touch "$ENV_FILE"
|
||||
echo "[created] .env (empty)"
|
||||
fi
|
||||
else
|
||||
echo "[exists] .env"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# 2. Auto-generate GOCLAW_ENCRYPTION_KEY if missing
|
||||
current_enc="$(get_env_val GOCLAW_ENCRYPTION_KEY)"
|
||||
if [ -z "$current_enc" ]; then
|
||||
new_key="$(gen_hex 32)"
|
||||
set_env_val "GOCLAW_ENCRYPTION_KEY" "$new_key"
|
||||
echo "[generated] GOCLAW_ENCRYPTION_KEY"
|
||||
else
|
||||
echo "[exists] GOCLAW_ENCRYPTION_KEY"
|
||||
fi
|
||||
|
||||
# 3. Auto-generate GOCLAW_GATEWAY_TOKEN if missing
|
||||
current_tok="$(get_env_val GOCLAW_GATEWAY_TOKEN)"
|
||||
if [ -z "$current_tok" ]; then
|
||||
new_tok="$(gen_hex 16)"
|
||||
set_env_val "GOCLAW_GATEWAY_TOKEN" "$new_tok"
|
||||
echo "[generated] GOCLAW_GATEWAY_TOKEN"
|
||||
else
|
||||
echo "[exists] GOCLAW_GATEWAY_TOKEN"
|
||||
fi
|
||||
|
||||
# 4. Check provider API key
|
||||
has_provider=false
|
||||
for key in GOCLAW_OPENROUTER_API_KEY GOCLAW_ANTHROPIC_API_KEY GOCLAW_OPENAI_API_KEY \
|
||||
GOCLAW_MINIMAX_API_KEY GOCLAW_GROQ_API_KEY GOCLAW_DEEPSEEK_API_KEY \
|
||||
GOCLAW_GEMINI_API_KEY GOCLAW_MISTRAL_API_KEY GOCLAW_XAI_API_KEY; do
|
||||
val="$(get_env_val "$key")"
|
||||
if [ -n "$val" ]; then
|
||||
has_provider=true
|
||||
echo "[exists] $key"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$has_provider" = false ]; then
|
||||
echo ""
|
||||
echo "[missing] No LLM provider API key found."
|
||||
if [ "$QUIET" = false ]; then
|
||||
echo ""
|
||||
echo "Enter your API key (or press Enter to skip):"
|
||||
echo " Supported: OpenRouter, Anthropic, OpenAI, MiniMax, Groq, DeepSeek, Gemini, Mistral, xAI"
|
||||
echo ""
|
||||
read -rp " GOCLAW_OPENROUTER_API_KEY= " api_key
|
||||
if [ -n "$api_key" ]; then
|
||||
set_env_val "GOCLAW_OPENROUTER_API_KEY" "$api_key"
|
||||
echo "[set] GOCLAW_OPENROUTER_API_KEY"
|
||||
else
|
||||
echo "[skipped] No API key set — you must add one to .env before starting"
|
||||
fi
|
||||
else
|
||||
echo " Add at least one GOCLAW_*_API_KEY to .env before starting."
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== Done ==="
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo " 1. Review .env and fill in any remaining values"
|
||||
echo " 2. Run: docker compose -f docker-compose.yml -f docker-compose.managed.yml up -d --build"
|
||||
echo ""
|
||||
Generated
+4303
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user