refactor: remove standalone mode, consolidate to managed-only (PostgreSQL) (#70)

- Remove standalone mode code: file-based stores, standalone gateway,
  heartbeat service, SQLite memory, standalone docker-compose
- Rename docker-compose.managed.yml → docker-compose.postgres.yml
- Clean up ~130 Go comments referencing "managed mode" qualifier
- Simplify docker-compose.yml env vars (providers/channels via web UI)
- Update .env.example to essential vars only (token + encryption key)
- Add setup wizard UI (provider → agent → channel bootstrap flow)
- Add logs.tail WebSocket handler for live log streaming
- Add cursor-pointer to interactive UI components
- Clean up config page (remove standalone-only sections)
- Update README and docs for managed-only architecture
This commit is contained in:
Viet Tran authored and GitHub committed 2026-03-06 18:51:11 +07:00
1 parent 0f44f18b0c
commit 6895e369f6
183 files changed
+2280 -5340

No files matched your search

+23 -69
View File
@@ -2,12 +2,7 @@
## 1. Overview
GoClaw is an AI agent gateway written in Go. It exposes a WebSocket RPC (v3) interface and an OpenAI-compatible HTTP API for orchestrating LLM-powered agents. The system supports two operating modes:
- **Standalone** -- file-based storage with SQLite for per-user data, zero external dependencies beyond an LLM API key.
- **Managed** -- PostgreSQL-backed multi-tenant mode with HTTP CRUD APIs, per-user context files, encrypted credentials, agent delegation, teams, and LLM call tracing.
> **Documentation scope**: This documentation covers both modes. Standalone mode now has near-parity with managed mode for core features (per-user context files, workspace isolation, agent types, bootstrap onboarding). Managed mode adds agent delegation, teams, quality gates, tracing, HTTP CRUD APIs, and encrypted secrets.
GoClaw is an AI agent gateway written in Go. It exposes a WebSocket RPC (v3) interface and an OpenAI-compatible HTTP API for orchestrating LLM-powered agents. The system uses PostgreSQL as its storage backend with full multi-tenant isolation, per-user context files, encrypted credentials, agent delegation, teams, and LLM call tracing.
## 2. Component Diagram
@@ -121,17 +116,16 @@ flowchart TD
| `internal/hooks/` | Hook engine: quality gates, command evaluator, agent evaluator, recursion prevention (`WithSkipHooks`) |
| `internal/store/` | Store interfaces: SessionStore, AgentStore, ProviderStore, SkillStore, MemoryStore, CronStore, PairingStore, TracingStore, MCPServerStore, AgentLinkStore, TeamStore, ChannelInstanceStore, ConfigSecretsStore |
| `internal/store/pg/` | PostgreSQL implementations (`database/sql` + `pgx/v5`) |
| `internal/store/file/` | File-based implementations: sessions, memory (SQLite), cron, pairing, skills, agents (filesystem + SQLite) |
| `internal/bootstrap/` | System prompt files (AGENTS.md, SOUL.md, TOOLS.md, IDENTITY.md, USER.md, HEARTBEAT.md, BOOTSTRAP.md) + seeding + truncation |
| `internal/bootstrap/` | System prompt files (AGENTS.md, SOUL.md, TOOLS.md, IDENTITY.md, USER.md, BOOTSTRAP.md) + seeding + truncation |
| `internal/config/` | Config loading (JSON5) + env var overlay |
| `internal/skills/` | SKILL.md loader (5-tier hierarchy) + BM25 search + hot-reload via fsnotify |
| `internal/channels/` | Channel manager + adapters: Telegram (forum topics, STT, bot commands), Feishu/Lark (streaming cards, media), Zalo OA, Zalo Personal, Discord, WhatsApp |
| `internal/mcp/` | MCP server bridge (stdio, SSE, streamable-HTTP transports) |
| `internal/scheduler/` | Lane-based concurrency control (main, subagent, cron, delegate lanes) with per-session serialization |
| `internal/memory/` | Memory system (SQLite FTS5 + embeddings for standalone mode) |
| `internal/memory/` | Memory system (pgvector hybrid search) |
| `internal/permissions/` | RBAC policy engine (admin, operator, viewer roles) |
| `internal/pairing/` | DM/device pairing service (8-character codes) |
| `internal/sessions/` | File-based session manager (standalone mode) |
| `internal/sessions/` | Session manager |
| `internal/bus/` | Event pub/sub (Message Bus) |
| `internal/sandbox/` | Docker-based code execution sandbox |
| `internal/tts/` | Text-to-Speech providers: OpenAI, ElevenLabs, Edge, MiniMax |
@@ -139,37 +133,10 @@ flowchart TD
| `internal/crypto/` | AES-256-GCM encryption for API keys |
| `internal/tracing/` | LLM call tracing (traces + spans), in-memory buffer with periodic store flush |
| `internal/tracing/otelexport/` | Optional OpenTelemetry OTLP exporter (opt-in via build tags; adds gRPC + protobuf) |
| `internal/heartbeat/` | Periodic agent wake-up service |
---
## 4. Two Operating Modes
| Aspect | Standalone | Managed |
|--------|-----------|---------|
| Config source | `config.json` + env vars | `config.json` + `GOCLAW_POSTGRES_DSN` |
| Storage | JSON files + SQLite (`~/.goclaw/data/agents.db`) | PostgreSQL |
| Agents | Defined in `config.json` `agents.list`, created eagerly at startup | `agents` table, lazy-resolved via `ManagedResolver` |
| Agent store | `FileAgentStore` (filesystem + SQLite) | `PGAgentStore` |
| Context files | Agent-level on filesystem, per-user in SQLite | `agent_context_files` + `user_context_files` tables |
| Agent types | `open` / `predefined` (via config) | `open` (7 per-user files) / `predefined` (agent-level + USER.md per-user) |
| Per-user isolation | Workspace subdirectories (`user_alice/`, `user_bob/`) | Same + DB-scoped context files |
| Bootstrap onboarding | Per-user BOOTSTRAP.md seeding (SQLite) | Same (PostgreSQL) |
| Agent delegation | N/A | Sync/async delegation, agent links, quality gates |
| Agent teams | N/A | Shared task board, mailbox, handoff |
| Skills | Filesystem only (workspace + global dirs) | PostgreSQL + filesystem + embedding search |
| Memory | SQLite FTS5 + embeddings | pgvector hybrid (full-text search + vector similarity) |
| Tracing | N/A | `traces` + `spans` tables + optional OTel OTLP export |
| MCP servers | `config.json` `tools.mcp_servers` | `mcp_servers` table + grants |
| API key storage | `.env.local` / env vars only | PostgreSQL (AES-256-GCM encrypted) |
| HTTP CRUD API | N/A | `/v1/agents`, `/v1/skills`, `/v1/traces`, `/v1/mcp`, `/v1/delegations` |
| Virtual FS | `ContextFileInterceptor` routes to SQLite | `ContextFileInterceptor` routes to PostgreSQL |
| Custom tools | N/A | `custom_tools` table + `DynamicToolLoader` |
| Managed-only stores (nil in standalone) | -- | ProviderStore, TracingStore, MCPServerStore, CustomToolStore, AgentLinkStore, TeamStore |
---
## 5. Multi-Tenant Identity Model
## 4. Multi-Tenant Identity Model
GoClaw uses the **Identity Propagation** pattern (also known as **Trusted Subsystem**). It does not implement authentication or authorization — instead, it trusts the upstream service that authenticates with the gateway token to provide accurate user identity.
@@ -196,8 +163,8 @@ flowchart LR
| Entry Point | How user_id is provided | Enforcement |
|-------------|------------------------|-------------|
| HTTP API | `X-GoClaw-User-Id` header | Required in managed mode |
| WebSocket | `user_id` field in `connect` handshake | Required in managed mode |
| HTTP API | `X-GoClaw-User-Id` header | Required |
| WebSocket | `user_id` field in `connect` handshake | Required |
| Channels | Derived from platform sender ID (e.g., Telegram user ID) | Automatic |
### Compound User ID Convention
@@ -239,36 +206,27 @@ sequenceDiagram
GW->>GW: 2. Resolve workspace + data dirs
GW->>GW: 3. Create Message Bus
alt Managed mode
GW->>PG: 4. Connect to Postgres (pg.NewPGStores)
PG-->>GW: PG stores created
GW->>GW: 5. Start tracing collector
GW->>PG: 6. Register providers from DB
GW->>PG: 7. Wire embedding provider to PGMemoryStore
GW->>PG: 8. Backfill memory embeddings (background)
else Standalone mode
GW->>GW: 4. Create file-based stores
end
GW->>PG: 4. Connect to Postgres (pg.NewPGStores)
PG-->>GW: PG stores created
GW->>GW: 5. Start tracing collector
GW->>PG: 6. Register providers from DB
GW->>PG: 7. Wire embedding provider to PGMemoryStore
GW->>PG: 8. Backfill memory embeddings (background)
GW->>GW: 9. Register config-based providers
GW->>GW: 10. Create tool registry (filesystem, exec, web, memory, browser, TTS, subagent, MCP)
GW->>GW: 11. Load bootstrap files (DB or filesystem)
GW->>GW: 11. Load bootstrap files (DB)
GW->>GW: 12. Create skills loader + register skill_search tool
GW->>GW: 13. Wire skill embeddings (managed only)
GW->>GW: 13. Wire skill embeddings
alt Managed mode
GW->>GW: 14. Create agents lazily (set ManagedResolver)
GW->>GW: 15. wireManagedExtras (interceptors, cache subscribers)
GW->>GW: 16. Wire managed HTTP handlers (agents, skills, traces, MCP)
else Standalone mode
GW->>GW: 14. Create agents eagerly from config
GW->>GW: 15. wireStandaloneExtras (FileAgentStore, interceptors, callbacks)
end
GW->>GW: 14. Create agents lazily (set ManagedResolver)
GW->>GW: 15. wireManagedExtras (interceptors, cache subscribers)
GW->>GW: 16. Wire managed HTTP handlers (agents, skills, traces, MCP)
GW->>Engine: 17. Create gateway server (WS + HTTP)
GW->>Engine: 18. Register RPC methods
GW->>Engine: 19. Register + start channels (Telegram, Discord, Feishu, Zalo, WhatsApp)
GW->>Engine: 20. Start cron, heartbeat, scheduler (4 lanes)
GW->>Engine: 20. Start cron, scheduler (4 lanes)
GW->>Engine: 21. Start skills watcher + inbound consumer
GW->>Engine: 22. Listen on host:port
```
@@ -294,8 +252,6 @@ flowchart TD
W11["11. Evaluate Loop + Handoff<br/>evaluate_loop tool + handoff tool"]
```
A separate `wireStandaloneExtras()` in `cmd/gateway_standalone.go` wires the same core callbacks (user seeding, context file loading) using `FileAgentStore` instead of PostgreSQL.
### Cache Invalidation Events
| Event | Subscriber | Action |
@@ -386,8 +342,7 @@ When the process receives SIGINT or SIGTERM:
1. Broadcast `shutdown` event to all connected WebSocket clients.
2. `channelMgr.StopAll()` -- stop all channel adapters.
3. `cronStore.Stop()` -- stop cron scheduler.
4. `heartbeatSvc.Stop()` -- stop heartbeat service.
5. `sandboxMgr.Stop()` + `ReleaseAll()` -- release Docker containers.
4. `sandboxMgr.Stop()` + `ReleaseAll()` -- release Docker containers.
6. `cancel()` -- cancel root context, propagating to consumer + scheduler.
7. Deferred cleanup: flush tracing collector, close memory store, close browser manager, stop scheduler lanes.
8. HTTP server shutdown with a **5-second timeout** (`context.WithTimeout`).
@@ -420,7 +375,7 @@ flowchart TD
| `agents` | defaults (provider, model, context_window) + list (per-agent overrides) |
| `tools` | profile, allow/deny lists, exec_approval, web, browser, mcp_servers, rate_limit_per_hour |
| `channels` | Per-channel: enabled, token, dm_policy, group_policy, allow_from |
| `database` | mode (standalone/managed); postgres_dsn read only from env var |
| `database` | postgres_dsn read only from env var |
### Secret Handling
@@ -439,8 +394,7 @@ flowchart TD
| `cmd/root.go` | Cobra CLI entry point, flag parsing |
| `cmd/gateway.go` | Gateway startup orchestrator (`runGateway()`) |
| `cmd/gateway_managed.go` | Managed mode wiring (`wireManagedExtras()`, `wireManagedHTTP()`) |
| `cmd/gateway_standalone.go` | Standalone mode wiring (`wireStandaloneExtras()`) |
| `cmd/gateway_callbacks.go` | Shared callbacks for managed + standalone (user seeding, context file loading) |
| `cmd/gateway_callbacks.go` | Shared callbacks (user seeding, context file loading) |
| `cmd/gateway_consumer.go` | Inbound message consumer (subagent, delegate, teammate, handoff routing) |
| `cmd/gateway_providers.go` | Provider registration (config-based + DB-based) |
| `cmd/gateway_methods.go` | RPC method registration |
@@ -472,7 +426,7 @@ flowchart TD
| [05-channels-messaging.md](./05-channels-messaging.md) | Channel adapters, Telegram formatting, pairing, managed-mode user scoping |
| [06-store-data-model.md](./06-store-data-model.md) | Store interfaces, PostgreSQL schema, session caching, custom tool store |
| [07-bootstrap-skills-memory.md](./07-bootstrap-skills-memory.md) | Bootstrap files, skills system, memory, skills grants |
| [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md) | Scheduler lanes, cron lifecycle, heartbeat |
| [08-scheduling-cron.md](./08-scheduling-cron.md) | Scheduler lanes, cron lifecycle |
| [09-security.md](./09-security.md) | Defense layers, encryption, rate limiting, RBAC, sandbox |
| [10-tracing-observability.md](./10-tracing-observability.md) | Tracing collector, span hierarchy, OTel export, trace API |
| [11-agent-teams.md](./11-agent-teams.md) | Agent teams, task board, mailbox, delegation integration |
+1 -2
View File
@@ -171,8 +171,7 @@ The system prompt is assembled dynamically from 15+ sections. Two modes control
13. **Extra context** -- additional prompt text wrapped in `<extra_context>` XML tags.
14. **Project Context** -- context files loaded from the database or filesystem, wrapped in `<context_file>` XML tags with a defensive preamble.
15. **Silent Replies** -- instructions for the NO_REPLY convention.
16. **Heartbeats** -- instructions for periodic wake-up behavior.
17. **Sub-Agent Spawning** -- rules for launching child agents.
16. **Sub-Agent Spawning** -- rules for launching child agents.
18. **Delegation** -- auto-generated `DELEGATION.md` listing available delegation targets (inline if ≤15, search instruction if >15).
19. **Team** -- `TEAM.md` injected for team leads only (team name, role, teammate list).
20. **Runtime** -- runtime metadata (agent ID, session key, provider info).
-1
View File
@@ -162,7 +162,6 @@ flowchart TD
| `IDENTITY.md` | Agent identity information |
| `AGENTS.md` | Sub-agent definitions |
| `TOOLS.md` | Tool usage guidance |
| `HEARTBEAT.md` | Periodic wake-up instructions |
| `USER.md` | Per-user preferences and context |
| `BOOTSTRAP.md` | First-run instructions (write empty = delete row) |
-1
View File
@@ -292,7 +292,6 @@ flowchart TD
| Method | Description |
|--------|-------------|
| `logs.tail` | Tail gateway logs |
| `heartbeat` | Trigger heartbeat check |
---
+5 -5
View File
@@ -182,7 +182,7 @@ The Telegram channel uses long polling via the `telego` library (Telegram Bot AP
- **Group mention gating**: By default, bot must be @mentioned in groups (`requireMention: true`). Pending messages without a mention are stored in a history buffer (default 50 messages) and included as context when the bot is eventually mentioned.
- **Typing indicator**: A "typing" action is sent while the agent is processing.
- **Proxy support**: Optional HTTP proxy configured via channel config.
- **Cancel commands**: `/stop` and `/stopall` intercepted before the 800ms debouncer. See [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md).
- **Cancel commands**: `/stop` and `/stopall` intercepted before the 800ms debouncer. See [08-scheduling-cron.md](./08-scheduling-cron.md) for details.
- **Concurrent group support**: Group sessions support up to 3 concurrent agent runs.
- **Bot reply as implicit mention**: Replying to a bot message in a group counts as mentioning the bot.
@@ -478,13 +478,13 @@ All channel state — placeholders, streams, reactions, typing controllers, thre
---
## 13. Managed Mode Behavior
## 13. Per-User Isolation
In managed mode, channels provide per-user isolation through compound sender IDs and context propagation:
Channels provide per-user isolation through compound sender IDs and context propagation:
- **User scoping**: Each channel constructs a compound sender ID (e.g., `telegram:123456`) which maps to a `user_id`. The session key format `agent:{agentId}:{channel}:direct:{peerId}` ensures each user has isolated conversation history per agent.
- **Context propagation**: `HandleMessage()` injects `AgentID`, `UserID`, and `AgentType` into the context. These flow to the ContextFileInterceptor, MemoryInterceptor, and per-user file seeding.
- **Pairing storage**: PostgreSQL (`pairing_requests` and `paired_devices` tables) in managed mode; JSON files in standalone mode.
- **Pairing storage**: PostgreSQL (`pairing_requests` and `paired_devices` tables).
- **Session persistence**: PostgreSQL `sessions` table with write-behind caching.
---
@@ -554,6 +554,6 @@ flowchart TD
|----------|-----------------|
| [00-architecture-overview.md](./00-architecture-overview.md) | Channel startup in gateway sequence |
| [03-tools-system.md](./03-tools-system.md) | Tool policy engine, per-request tool allow list |
| [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md) | /stop and /stopall commands, scheduler lanes |
| [08-scheduling-cron.md](./08-scheduling-cron.md) | /stop and /stopall commands, scheduler lanes, cron |
| [09-security.md](./09-security.md) | Group file writer restrictions, security logging |
| [11-agent-teams.md](./11-agent-teams.md) | Team message routing, delegation result delivery |
+30 -55
View File
@@ -1,58 +1,40 @@
# 06 - Store Layer and Data Model
The store layer abstracts all persistence behind Go interfaces, allowing the same core engine to run with file-based storage (standalone mode) or PostgreSQL (managed mode). Each store interface has independent implementations, and the system determines which backend to use based on configuration at startup.
The store layer abstracts all persistence behind Go interfaces backed by PostgreSQL. Each store interface has a PostgreSQL implementation wired at startup.
---
## 1. Store Layer Routing
## 1. Store Layer
```mermaid
flowchart TD
START["Gateway Startup"] --> CHECK{"StoreConfig.IsManaged()?<br/>(DSN + mode = managed)"}
CHECK -->|Yes| PG["PostgreSQL Backend"]
CHECK -->|No| FILE["File Backend"]
START["Gateway Startup"] --> PG["PostgreSQL Backend"]
PG --> PG_STORES["PGSessionStore<br/>PGAgentStore<br/>PGProviderStore<br/>PGCronStore<br/>PGPairingStore<br/>PGSkillStore<br/>PGMemoryStore<br/>PGTracingStore<br/>PGMCPServerStore<br/>PGCustomToolStore<br/>PGChannelInstanceStore<br/>PGConfigSecretsStore<br/>PGAgentLinkStore<br/>PGTeamStore"]
FILE --> FILE_STORES["FileSessionStore<br/>FileMemoryStore (SQLite + FTS5)<br/>FileCronStore<br/>FilePairingStore<br/>FileSkillStore<br/>FileAgentStore (filesystem + SQLite)<br/>ProviderStore = nil<br/>TracingStore = nil<br/>MCPServerStore = nil<br/>CustomToolStore = nil<br/>AgentLinks = nil<br/>Teams = nil"]
```
---
## 2. Store Interface Map
The `Stores` struct is the top-level container holding all storage backends. In standalone mode, managed-only stores are `nil`.
The `Stores` struct is the top-level container holding all PostgreSQL-backed storage implementations.
| Interface | Standalone Implementation | Managed Implementation | Mode |
|-----------|--------------------------|------------------------|------|
| SessionStore | `FileSessionStore` via `sessions.Manager` | `PGSessionStore` | Both |
| MemoryStore | `FileMemoryStore` (SQLite + FTS5 + embeddings) | `PGMemoryStore` (tsvector + pgvector) | Both |
| CronStore | `FileCronStore` | `PGCronStore` | Both |
| PairingStore | `FilePairingStore` via `pairing.Service` | `PGPairingStore` | Both |
| SkillStore | `FileSkillStore` via `skills.Loader` | `PGSkillStore` | Both |
| AgentStore | `FileAgentStore` (filesystem + SQLite) | `PGAgentStore` | Both |
| ProviderStore | `nil` | `PGProviderStore` | Managed only |
| TracingStore | `nil` | `PGTracingStore` | Managed only |
| MCPServerStore | `nil` | `PGMCPServerStore` | Managed only |
| CustomToolStore | `nil` | `PGCustomToolStore` | Managed only |
| ChannelInstanceStore | `nil` | `PGChannelInstanceStore` | Managed only |
| ConfigSecretsStore | `nil` | `PGConfigSecretsStore` | Managed only |
| AgentLinkStore | `nil` | `PGAgentLinkStore` | Managed only |
| TeamStore | `nil` | `PGTeamStore` | Managed only |
### Standalone AgentStore (FileAgentStore)
In standalone mode, `FileAgentStore` provides per-user context files and profiles without PostgreSQL. It combines filesystem storage (agent-level files like SOUL.md) with SQLite (`~/.goclaw/data/agents.db`) for per-user data:
| Data | Storage |
|------|---------|
| Agent metadata | In-memory from `config.json` |
| Agent-level files (SOUL.md, IDENTITY.md, ...) | Filesystem at workspace root |
| Per-user files (USER.md, BOOTSTRAP.md) | SQLite `user_context_files` |
| User profiles | SQLite `user_profiles` |
| Group file writers | SQLite `group_file_writers` |
Agent UUIDs use UUID v5 (deterministic): `uuid.NewSHA1(namespace, "goclaw-standalone:{agentKey}")` -- stable across restarts without database sequences.
| Interface | Implementation |
|-----------|---------------|
| SessionStore | `PGSessionStore` |
| MemoryStore | `PGMemoryStore` (tsvector + pgvector) |
| CronStore | `PGCronStore` |
| PairingStore | `PGPairingStore` |
| SkillStore | `PGSkillStore` |
| AgentStore | `PGAgentStore` |
| ProviderStore | `PGProviderStore` |
| TracingStore | `PGTracingStore` |
| MCPServerStore | `PGMCPServerStore` |
| CustomToolStore | `PGCustomToolStore` |
| ChannelInstanceStore | `PGChannelInstanceStore` |
| ConfigSecretsStore | `PGConfigSecretsStore` |
| AgentLinkStore | `PGAgentLinkStore` |
| TeamStore | `PGTeamStore` |
---
@@ -70,7 +52,7 @@ flowchart TD
CACHE --> GETSM["GetSummary()"]
end
CACHE -->|"Save(key)"| DB[("PostgreSQL / JSON file")]
CACHE -->|"Save(key)"| DB[("PostgreSQL")]
DB -->|"Cache miss via GetOrCreate"| CACHE
```
@@ -91,12 +73,6 @@ flowchart TD
| Cron | `agent:{agentId}:cron:{jobId}:run:{runId}` | `agent:default:cron:reminder:run:abc123` |
| Main | `agent:{agentId}:{mainKey}` | `agent:default:main` |
### File-Based Persistence (Standalone)
- Startup: `loadAll()` reads all `.json` files into memory
- Save: temp file + rename (atomic write, prevents corruption on crash)
- Filename: session key with `:` replaced by `_`, plus `.json` extension
---
## 4. Agent Access Control
@@ -182,14 +158,14 @@ flowchart TD
When FTS returns no results (e.g., cross-language queries), a `likeSearch()` fallback runs ILIKE queries using up to 5 keywords (minimum 3 characters each), scoped to the agent's index.
### Standalone vs Managed
### Search Implementation
| Aspect | Standalone | Managed |
|--------|-----------|---------|
| FTS engine | SQLite FTS5 | PostgreSQL tsvector |
| Vector | Embedding cache | pgvector extension |
| Search function | `plainto_tsquery('simple', ...)` | Same |
| Distance operator | N/A | `<=>` (cosine) |
| Aspect | Detail |
|--------|--------|
| FTS engine | PostgreSQL tsvector |
| Vector | pgvector extension |
| Search function | `plainto_tsquery('simple', ...)` |
| Distance operator | `<=>` (cosine) |
---
@@ -208,8 +184,8 @@ Context files are stored in two tables and routed based on agent type.
| Agent Type | Agent-Level Files | Per-User Files |
|------------|-------------------|----------------|
| `open` | Template fallback only | All 7 files (SOUL, IDENTITY, AGENTS, TOOLS, HEARTBEAT, BOOTSTRAP, USER) |
| `predefined` | 6 files (SOUL, IDENTITY, AGENTS, TOOLS, HEARTBEAT, BOOTSTRAP) | Only USER.md |
| `open` | Template fallback only | All files (SOUL, IDENTITY, AGENTS, TOOLS, BOOTSTRAP, USER) |
| `predefined` | Agent-level files (SOUL, IDENTITY, AGENTS, TOOLS, BOOTSTRAP) | Only USER.md |
The `ContextFileInterceptor` checks agent type from context and routes read/write operations accordingly. For open agents, per-user files take priority with agent-level as fallback.
@@ -577,7 +553,6 @@ All "create or update" operations use `INSERT ... ON CONFLICT DO UPDATE`, ensuri
| `internal/store/pairing_store.go` | `PairingStore` interface |
| `internal/store/cron_store.go` | `CronStore` interface |
| `internal/store/custom_tool_store.go` | `CustomToolStore` interface |
| `internal/store/file/agents.go` | `FileAgentStore`: filesystem + SQLite backend for standalone mode |
| `internal/store/pg/factory.go` | PG store factory: creates all PG store instances from a connection pool |
| `internal/store/pg/sessions.go` | `PGSessionStore`: session cache, Save, GetOrCreate |
| `internal/store/pg/agents.go` | `PGAgentStore`: CRUD, soft delete, access control |
+22 -50
View File
@@ -22,8 +22,7 @@ Markdown files loaded at agent initialization and embedded into the system promp
| 3 | TOOLS.md | Local tool notes (camera, SSH, TTS, etc.) | Yes | Yes |
| 4 | IDENTITY.md | Agent name, creature, vibe, emoji | Yes | No |
| 5 | USER.md | User profile (name, timezone, preferences) | Yes | No |
| 6 | HEARTBEAT.md | Periodic check task list | Yes | No |
| 7 | BOOTSTRAP.md | First-run ritual (deleted after completion) | Yes | No |
| 6 | BOOTSTRAP.md | First-run ritual (deleted after completion) | Yes | No |
Subagent and cron sessions load only AGENTS.md + TOOLS.md (the `minimalAllowlist`).
@@ -60,28 +59,11 @@ When a file is truncated, a marker is inserted between the head and tail section
## 3. Seeding -- Template Creation
Templates are embedded in the binary via Go `embed` (directory: `internal/bootstrap/templates/`). Seeding automatically creates default files for new workspaces or new users.
Templates are embedded in the binary via Go `embed` (directory: `internal/bootstrap/templates/`). Seeding automatically creates default files for new users.
```mermaid
flowchart TD
subgraph "Standalone Mode"
SA["EnsureWorkspaceFiles()"] --> SA1["Iterate over embedded templates"]
SA1 --> SA2{"File already exists?<br/>(O_EXCL atomic check)"}
SA2 -->|Yes| SKIP1["Skip"]
SA2 -->|No| CREATE1["Create template file on disk"]
end
subgraph "Standalone Mode -- Per-User (FileAgentStore)"
SU["SeedUserFiles()"] --> SU1{"Agent type?"}
SU1 -->|open| SU_OPEN["Seed all 7 files to SQLite"]
SU1 -->|predefined| SU_PRED["Seed USER.md + BOOTSTRAP.md to SQLite"]
SU_OPEN --> SU_CHK{"Row already exists?"}
SU_PRED --> SU_CHK
SU_CHK -->|Yes| SKIP_SU["Skip"]
SU_CHK -->|No| SU_WRITE["INSERT into user_context_files"]
end
subgraph "Managed Mode -- Agent Level"
subgraph "Agent Level"
SB["SeedToStore()"] --> SB1{"Agent type = open?"}
SB1 -->|Yes| SKIP_AGENT["Skip (open agents use per-user only)"]
SB1 -->|No| SB2["Seed 6 files to agent_context_files<br/>(all except BOOTSTRAP.md)"]
@@ -90,7 +72,7 @@ flowchart TD
SB3 -->|No| WRITE2["Write embedded template"]
end
subgraph "Managed Mode -- Per-User"
subgraph "Per-User"
MC["SeedUserFiles()"] --> MC1{"Agent type?"}
MC1 -->|open| OPEN["Seed all 7 files to user_context_files"]
MC1 -->|predefined| PRED["Seed USER.md + BOOTSTRAP.md to user_context_files"]
@@ -103,31 +85,26 @@ flowchart TD
`SeedUserFiles()` is idempotent -- safe to call multiple times without overwriting personalized content.
### Standalone UUID Generation
Standalone agents are defined in `config.json` without database-generated UUIDs. `FileAgentStore` uses UUID v5 (`uuid.NewSHA1(namespace, "goclaw-standalone:{agentKey}")`) to produce deterministic IDs from agent keys. This ensures SQLite rows for per-user files survive process restarts without coordination.
### Predefined Agent Bootstrap
Both standalone and managed mode now seed `BOOTSTRAP.md` for predefined agents (per-user). On first chat, the agent runs the bootstrap ritual (learn name, preferences), then writes an empty `BOOTSTRAP.md` which triggers deletion. The empty-write deletion is ordered *before* the predefined write-block in `ContextFileInterceptor` to prevent an infinite bootstrap loop.
`BOOTSTRAP.md` is seeded for predefined agents (per-user). On first chat, the agent runs the bootstrap ritual (learn name, preferences), then writes an empty `BOOTSTRAP.md` which triggers deletion. The empty-write deletion is ordered *before* the predefined write-block in `ContextFileInterceptor` to prevent an infinite bootstrap loop.
---
## 4. Agent Type Routing
Two agent types determine which context files live at the agent level versus the per-user level. Agent types are now available in both managed and standalone modes.
Two agent types determine which context files live at the agent level versus the per-user level.
| Agent Type | Agent-Level Files | Per-User Files |
|------------|-------------------|----------------|
| `open` | None | All 7 files (AGENTS, SOUL, TOOLS, IDENTITY, USER, HEARTBEAT, BOOTSTRAP) |
| `open` | None | All files (AGENTS, SOUL, TOOLS, IDENTITY, USER, BOOTSTRAP) |
| `predefined` | 6 files (shared across all users) | USER.md + BOOTSTRAP.md |
For `open` agents, each user gets their own full set of context files. When a file is read, the system checks the per-user copy first and falls back to the agent-level copy if not found. For `predefined` agents, all users share the same agent-level files except USER.md (personalized) and BOOTSTRAP.md (per-user first-run ritual, deleted after completion).
| Mode | Agent Type Source | Per-User Storage |
|------|------------------|-----------------|
| Managed | `agents` PostgreSQL table | `user_context_files` table |
| Standalone | `config.json` agent entries | SQLite via `FileAgentStore` |
| Source | Per-User Storage |
|--------|-----------------|
| `agents` PostgreSQL table | `user_context_files` table |
---
@@ -155,8 +132,7 @@ flowchart TD
S9 --> S10["10. Extra Context / Subagent Context"]
S10 --> S11["11. Project Context<br/>(bootstrap files + virtual files)"]
S11 --> S12["12. Silent Replies (full only)"]
S12 --> S13["13. Heartbeats (full only)"]
S13 --> S14["14. Sub-Agent Spawning (conditional)"]
S12 --> S14["14. Sub-Agent Spawning (conditional)"]
S14 --> S15["15. Runtime"]
```
@@ -178,7 +154,6 @@ flowchart TD
| 10. Extra Context | Conditional | Conditional |
| 11. Project Context | Yes | Yes |
| 12. Silent Replies | Yes | No |
| 13. Heartbeats | Yes | No |
| 14. Sub-Agent Spawning | Conditional | Conditional |
| 15. Runtime | Yes | Yes |
@@ -213,7 +188,7 @@ This ensures resolver-injected virtual files (`DELEGATION.md`, `TEAM.md`) surviv
## 7. Agent Summoning (Managed Mode)
Creating a predefined agent requires 5 context files (SOUL.md, IDENTITY.md, AGENTS.md, TOOLS.md, HEARTBEAT.md) with specific formatting conventions. Agent summoning generates all 5 files from a natural language description in a single LLM call.
Creating a predefined agent requires 4 context files (SOUL.md, IDENTITY.md, AGENTS.md, TOOLS.md) with specific formatting conventions. Agent summoning generates all 4 files from a natural language description in a single LLM call.
```mermaid
flowchart TD
@@ -432,8 +407,8 @@ Combines full-text search and vector search with weighted merging.
```mermaid
flowchart TD
Q["Search(query)"] --> FTS["FTS Search<br/>Standalone: SQLite FTS5 (BM25)<br/>Managed: tsvector + plainto_tsquery"]
Q --> VEC["Vector Search<br/>Standalone: cosine similarity<br/>Managed: pgvector (cosine distance)"]
Q["Search(query)"] --> FTS["FTS Search<br/>tsvector + plainto_tsquery"]
Q --> VEC["Vector Search<br/>pgvector (cosine distance)"]
FTS --> MERGE["hybridMerge()"]
VEC --> MERGE
MERGE --> NORM["Normalize FTS scores to 0..1<br/>Vector scores already in 0..1"]
@@ -442,15 +417,14 @@ flowchart TD
BOOST --> RESULT["Sorted + filtered results"]
```
### Standalone vs Managed Comparison
### Search Implementation
| Aspect | Standalone | Managed |
|--------|-----------|---------|
| Storage | SQLite + FTS5 | PostgreSQL + tsvector + pgvector |
| FTS | `porter unicode61` tokenizer | `plainto_tsquery('simple')` |
| Vector | JSON array embedding | pgvector type |
| Scope | Global (single agent) | Per-agent + per-user |
| File watcher | fsnotify (1500ms debounce) | Not needed (DB-backed) |
| Aspect | Detail |
|--------|--------|
| Storage | PostgreSQL + tsvector + pgvector |
| FTS | `plainto_tsquery('simple')` |
| Vector | pgvector type |
| Scope | Per-agent + per-user |
When both FTS and vector search return results, scores are merged using the weighted sum. When only one channel returns results, its scores are used directly (weights normalized to 1.0).
@@ -492,8 +466,7 @@ The flush is idempotent per compaction cycle -- it will not run again until the
|------|-------------|
| `internal/bootstrap/files.go` | Bootstrap file constants, loading, session filtering |
| `internal/bootstrap/truncate.go` | Truncation pipeline (head/tail split, budget clamping) |
| `internal/bootstrap/seed.go` | Standalone mode seeding (EnsureWorkspaceFiles) |
| `internal/bootstrap/seed_store.go` | Managed mode seeding (SeedToStore, SeedUserFiles) |
| `internal/bootstrap/seed_store.go` | Seeding: SeedToStore, SeedUserFiles |
| `internal/bootstrap/load_store.go` | Load context files from DB (LoadFromStore) |
| `internal/bootstrap/templates/*.md` | Embedded template files |
| `internal/agent/systemprompt.go` | System prompt builder (BuildSystemPrompt, 17+ sections) |
@@ -501,7 +474,6 @@ The flush is idempotent per compaction cycle -- it will not run again until the
| `internal/agent/resolver.go` | Agent resolution, DELEGATION.md + TEAM.md injection |
| `internal/agent/loop_history.go` | Context file merging (base + per-user, base-only preserved) |
| `internal/agent/memoryflush.go` | Memory flush logic (shouldRunMemoryFlush, runMemoryFlush) |
| `internal/store/file/agents.go` | FileAgentStore -- filesystem + SQLite backend for standalone |
| `internal/http/summoner.go` | Agent summoning -- LLM-powered context file generation |
| `internal/skills/loader.go` | Skill loader (5-tier hierarchy, BuildSummary, filtering) |
| `internal/skills/search.go` | BM25 search index (tokenization, IDF scoring) |
@@ -1,14 +1,13 @@
# 08 - Scheduling, Cron & Heartbeat
# 08 - Scheduling & Cron
Concurrency control and periodic task execution. The scheduler provides lane-based isolation and per-session serialization. Cron and heartbeat extend the agent loop with time-triggered behavior.
Concurrency control and periodic task execution. The scheduler provides lane-based isolation and per-session serialization. Cron extends the agent loop with time-triggered behavior.
> **Managed mode**: Cron jobs and run logs are stored in the `cron_jobs` and `cron_run_logs` PostgreSQL tables. Cache invalidation propagates via the `cache:cron` event on the message bus. In standalone mode, cron state is persisted to JSON files.
> Cron jobs and run logs are stored in the `cron_jobs` and `cron_run_logs` PostgreSQL tables. Cache invalidation propagates via the `cache:cron` event on the message bus.
### Responsibilities
- Scheduler: lane-based concurrency control, per-session message queue serialization
- Cron: three schedule kinds (at/every/cron), run logging, retry with exponential backoff
- Heartbeat: periodic agent wake-up, HEARTBEAT_OK detection, dedup within 24h
---
@@ -165,45 +164,6 @@ Jobs can be `active` or `paused`. Paused jobs skip execution during the due chec
---
## 5. Heartbeat -- 5 Steps
Periodically wakes the agent to check on events (calendar, inbox, alerts) and surfaces anything that needs attention.
> **Standalone mode only in practice**: HEARTBEAT.md is seeded from bootstrap templates only in standalone mode. In managed mode, HEARTBEAT.md is excluded from seeding (replaced by cron jobs). The heartbeat service still runs in both modes, but without a HEARTBEAT.md file with content, Step 2 skips execution.
```mermaid
flowchart TD
TICK["tick() -- every interval (default 30 min)"] --> S1{"Step 1:<br/>Within Active Hours?"}
S1 -->|Outside hours| SKIP1["Skip"]
S1 -->|Within hours| S2{"Step 2:<br/>HEARTBEAT.md exists<br/>and has meaningful content?"}
S2 -->|No| SKIP2["Skip"]
S2 -->|Yes| S3["Step 3: runner()<br/>Run agent with heartbeat prompt"]
S3 --> S4{"Step 4:<br/>Reply contains HEARTBEAT_OK?"}
S4 -->|OK| LOG["Log debug, discard reply"]
S4 -->|Has content| S5{"Step 5:<br/>Dedup -- same content<br/>within 24h?"}
S5 -->|Duplicate| SKIP3["Skip"]
S5 -->|New| DELIVER["deliver() via resolveTarget()<br/>then msgBus.PublishOutbound()"]
```
### Heartbeat Configuration
| Parameter | Default | Description |
|-----------|---------|-------------|
| Interval (`every`) | 30 minutes | Time between heartbeat wakes (`"0m"` to disable) |
| ActiveHours | (none) | Time window restriction, supports wrap-around midnight |
| Target | `"last"` | `"last"` (last-used channel), `"none"`, or explicit channel name |
| To | (empty) | Explicit chat ID (overrides target resolver) |
| Prompt | (default) | Custom system prompt for heartbeat runs |
| Model | (agent default) | Override LLM model for heartbeat runs |
| Session | `agent:{id}:heartbeat:main` | Custom session key |
| AckMaxChars | 300 | Content alongside HEARTBEAT_OK up to this length is still treated as OK |
### HEARTBEAT_OK Detection
Recognizes multiple formatting variants: `HEARTBEAT_OK`, `**HEARTBEAT_OK**`, `` `HEARTBEAT_OK` ``, `<b>HEARTBEAT_OK</b>`. Content accompanying the token is treated as an acknowledgment (OK) if it does not exceed `AckMaxChars`.
---
## File Reference
| File | Description |
@@ -212,13 +172,11 @@ Recognizes multiple formatting variants: `HEARTBEAT_OK`, `**HEARTBEAT_OK**`, ``
| `internal/scheduler/queue.go` | SessionQueue, Scheduler, drop policies, debounce |
| `internal/cron/service.go` | Cron run loop, schedule parsing, job lifecycle |
| `internal/cron/retry.go` | Retry with exponential backoff + jitter |
| `internal/heartbeat/service.go` | Heartbeat loop, HEARTBEAT_OK detection, active hours |
| `internal/store/cron_store.go` | CronStore interface (jobs + run logs) |
| `internal/store/file/cron.go` | File-based cron implementation (standalone mode) |
| `internal/store/pg/cron.go` | PostgreSQL cron implementation |
| `internal/store/pg/cron_scheduler.go` | PG job cache, due-job detection, execution |
| `cmd/gateway_cron.go` | makeCronJobHandler (routes cron execution to scheduler) |
| `cmd/gateway_agents.go` | setupHeartbeat, agent initialization |
| `cmd/gateway_agents.go` | Agent initialization |
| `internal/gateway/methods/cron.go` | RPC method handlers (list, create, update, delete, toggle, run, runs) |
---
+1 -1
View File
@@ -368,5 +368,5 @@ When concurrency limits are hit, the error message is written for LLM reasoning:
| [04-gateway-protocol.md](./04-gateway-protocol.md) | WebSocket auth, RBAC, rate limiting |
| [06-store-data-model.md](./06-store-data-model.md) | API key encryption, agent access control pipeline, agent_links table |
| [07-bootstrap-skills-memory.md](./07-bootstrap-skills-memory.md) | Context file merging, virtual files |
| [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md) | Scheduler lanes, cron lifecycle |
| [08-scheduling-cron.md](./08-scheduling-cron.md) | Scheduler lanes, cron lifecycle, /stop and /stopall |
| [10-tracing-observability.md](./10-tracing-observability.md) | Tracing and OTel export |
+2 -2
View File
@@ -2,7 +2,7 @@
Records agent run activities asynchronously. Spans are buffered in memory and flushed to the TracingStore in batches, with optional export to external OpenTelemetry backends.
> **Managed mode only**: Tracing requires PostgreSQL. In standalone mode, `TracingStore` is nil and no traces are recorded. The `traces` and `spans` tables store all tracing data. Optional OTel export sends spans to external backends (Jaeger, Grafana Tempo, Datadog) in addition to PostgreSQL.
> Tracing uses PostgreSQL. The `traces` and `spans` tables store all tracing data. Optional OTel export sends spans to external backends (Jaeger, Grafana Tempo, Datadog) in addition to PostgreSQL.
---
@@ -163,5 +163,5 @@ Delegation history is automatically recorded by `DelegateManager.saveDelegationH
| [01-agent-loop.md](./01-agent-loop.md) | Span emission during agent execution, cancel handling |
| [03-tools-system.md](./03-tools-system.md) | Delegation system, delegation history via agent tool |
| [06-store-data-model.md](./06-store-data-model.md) | traces/spans tables schema, delegation_history table |
| [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md) | /stop and /stopall commands |
| [08-scheduling-cron.md](./08-scheduling-cron.md) | Scheduler lanes, /stop and /stopall commands |
| [09-security.md](./09-security.md) | Rate limiting, RBAC access control |
+1 -1
View File
@@ -435,5 +435,5 @@ Teams emit events for real-time UI updates and observability.
|----------|-----------------|
| [03-tools-system.md](./03-tools-system.md) | Delegation system, agent links, quality gates, evaluate loop |
| [06-store-data-model.md](./06-store-data-model.md) | Team tables schema, delegation_history, handoff_routes |
| [08-scheduling-cron-heartbeat.md](./08-scheduling-cron-heartbeat.md) | Delegate scheduler lane (concurrency 100) |
| [08-scheduling-cron.md](./08-scheduling-cron.md) | Delegate scheduler lane (concurrency 100), cron |
| [09-security.md](./09-security.md) | Delegation security, hook recursion prevention |