diff --git a/plans/260406-1304-goclaw-v3-core-design/phase-01-foundation-interfaces.md b/plans/260406-1304-goclaw-v3-core-design/phase-01-foundation-interfaces.md deleted file mode 100644 index e3c693a3..00000000 --- a/plans/260406-1304-goclaw-v3-core-design/phase-01-foundation-interfaces.md +++ /dev/null @@ -1,638 +0,0 @@ -# Phase 1: Foundation Interfaces - -## Context Links - -- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (decisions 5, 8, 14, 10) -- Architecture: `plans/reports/Explore-260406-thorough-v3-architecture.md` (sections 1, 2, 6, 7) -- Loop deep dive: `plans/reports/Explore-260406-GoClaw-Agent-Loop-v3-Deep-Dive.md` (section 2, 3) -- Workspace forcing: `plans/reports/Explore-260406-workspace-forcing.md` - -## Overview - -- **Priority:** P1 (all other phases depend on these) -- **Status:** Complete -- **Effort:** 8h design - -4 foundational components with zero cross-dependencies. Can be designed and implemented in parallel. Every subsequent phase (pipeline, memory, integration) depends on at least one of these. - -## Key Insights - -- Token counting currently `chars/4` estimate -- drives ALL compaction/pruning decisions. Replacing with tiktoken-go is highest-ROI change -- Workspace resolution has 5 identified confusion sources (see workspace forcing report). Single immutable WorkspaceContext eliminates dual-key confusion -- EventBus currently `map[string]any` payloads, string topic constants. V3 needs typed events for consolidation pipeline -- Provider interface (`providers.Provider`) is functional but loop handles provider quirks directly. Adapter pattern isolates quirks - ---- - -## A. TokenCounter Interface - -### Interface Contract - -```go -// Package: internal/tokencount - -// TokenCounter provides accurate per-model token counting. -// Replaces chars/4 estimation used in compaction + pruning. -type TokenCounter interface { - // Count returns token count for raw text using model's tokenizer. - Count(model string, text string) int - - // CountMessages returns token count for a message list, - // including per-message overhead (role tokens, separators). - CountMessages(model string, msgs []providers.Message) int - - // ModelContextWindow returns max context tokens for a model. - // Falls back to provider default if model unknown. - ModelContextWindow(model string) int -} - -// TokenizerID identifies which tokenizer a model uses. -type TokenizerID string - -const ( - TokenizerCL100K TokenizerID = "cl100k_base" // Claude, GPT-3.5/4 - TokenizerO200K TokenizerID = "o200k_base" // GPT-4o, GPT-5 - TokenizerFallback TokenizerID = "fallback" // chars/4 -) - -// ModelRegistry maps model name prefixes to tokenizer + context window. -type ModelInfo struct { - TokenizerID TokenizerID - ContextWindow int // default context window for this model -} - -// DefaultRegistry provides built-in model mappings. -// Extend at runtime via RegisterModel(). -var DefaultRegistry = map[string]ModelInfo{ - "claude-": {TokenizerCL100K, 200_000}, - "gpt-4o": {TokenizerO200K, 128_000}, - "gpt-4": {TokenizerCL100K, 128_000}, - "gpt-5": {TokenizerO200K, 1_000_000}, - "qwen-": {TokenizerCL100K, 128_000}, // approx - "deepseek-": {TokenizerCL100K, 128_000}, // approx -} -``` - -### Caching Strategy - -```go -// cachedCounter wraps a tokenizer with per-message hash cache. -// Key: SHA-256(model + role + content + tool_call_ids) -// Value: token count (int) -// Eviction: LRU, 10K entries (typical session < 500 messages) -type cachedCounter struct { - cache *lru.Cache[string, int] // hashicorp/golang-lru/v2 - mu sync.RWMutex -} -``` - -Cache avoids re-counting unchanged messages during mid-loop re-estimation. Only new/modified messages hit tokenizer. - -### Per-Message Overhead - -``` -Per message: +4 tokens (role marker + separators) -System message: +4 tokens additional -Tool calls: name + JSON args counted as text -Tool results: content text counted -Images: model-specific (Claude: tile-based, GPT: similar) -``` - -### Data Flow - -``` -Pipeline stage needs token count - -> TokenCounter.CountMessages(model, messages) - -> For each message: - -> Hash(model, msg) -> check cache - -> Cache miss: tokenizer.Encode(text) -> count -> store - -> Sum + per-message overhead - -> Return total -``` - -### Affected Files (current codebase) - -| File | What changes | -|------|-------------| -| `internal/agent/pruning.go` | Replace `EstimateHistoryTokens()` (chars/4) with `TokenCounter.CountMessages()` | -| `internal/agent/loop_compact.go` | Replace token estimation in compaction trigger | -| `internal/agent/loop.go:352-411` | Mid-loop budget calculation uses token counter | -| `internal/agent/loop_history.go:624-699` | `maybeSummarize()` token estimation | -| `internal/agent/loop_types.go` | `runState.overheadTokens` recalibration logic | - -### Breaking Changes - -- `EstimateHistoryTokens()`, `EstimateTokensWithCalibration()` replaced -- `overheadCalibrated` field in `runState` removed (overhead now computed accurately upfront) -- `pruning.go` constants (`softTrimThreshold`, `hardClearThreshold`) stay but use accurate counts - ---- - -## B. WorkspaceContext - -### Interface Contract - -```go -// Package: internal/workspace - -// Scope defines workspace access boundary. -type Scope string - -const ( - ScopePersonal Scope = "personal" // single user, isolated - ScopeTeam Scope = "team" // team context, shared or isolated - ScopeDelegate Scope = "delegate" // delegated task, scoped access -) - -// WorkspaceContext is resolved ONCE at run start, immutable for the entire run. -// Eliminates dual ctxWorkspace + ctxTeamWorkspace confusion. -type WorkspaceContext struct { - // ActivePath is THE path for all file operations (read/write/list/exec). - // No other workspace path is accessible unless in ReadOnlyPaths. - ActivePath string - - // Scope describes the access boundary type. - Scope Scope - - // ReadOnlyPaths are additional paths the agent can read but NOT write. - // Used for: delegator exports, shared reference dirs. - ReadOnlyPaths []string - - // SharedPath is the shared delegate area (read/write by both delegator + delegatee). - // nil when not in delegation context. - SharedPath *string - - // TeamPath is the team workspace root (nil if not in team context). - TeamPath *string - - // MemoryScope determines memory isolation. - // Defaults to workspace scope. "shared" = all users in agent see same memory. - MemoryScope string - - // KGScope determines knowledge graph isolation. - // Defaults to workspace scope. "shared" = all users in agent see same KG. - KGScope string - - // OwnerID identifies who owns this workspace context (user ID or chat ID). - OwnerID string - - // EnforcementLabel is injected into system prompt verbatim. - // e.g. "ENFORCED -- all file paths resolve here, absolute paths rejected" - EnforcementLabel string -} - -// Resolver produces a WorkspaceContext from request parameters. -// Called once at ContextStage. Result is immutable. -type Resolver interface { - Resolve(ctx context.Context, params ResolveParams) (*WorkspaceContext, error) -} - -// ResolveParams captures all inputs needed to determine workspace. -type ResolveParams struct { - AgentID string - AgentType string // "open" | "predefined" - UserID string - ChatID string - TenantID string - PeerKind string // "direct" | "group" - TeamID *string - TeamConfig *TeamWorkspaceConfig // from teams table - DelegateCtx *DelegateContext // from delegation task - BaseDir string // global workspace root - Sharing *store.WorkspaceSharingConfig -} - -// TeamWorkspaceConfig maps to team.settings JSON. -type TeamWorkspaceConfig struct { - SharedWorkspace bool `json:"shared_workspace"` - WorkspacePath string `json:"workspace_path,omitempty"` -} - -// DelegateContext carries delegation-specific workspace overrides. -type DelegateContext struct { - LinkID string - SharedPath string - ExportPaths []string // read-only exports from delegator -} -``` - -### Resolution Rules - -``` -1. Solo DM, not shared: - ActivePath = {baseDir}/{tenantID}/{agentID}/{userID} - Scope = personal - -2. Solo DM, shared workspace: - ActivePath = {baseDir}/{tenantID}/{agentID} - Scope = personal (shared files, memory still per-user unless overridden) - -3. Team dispatch (member executing task): - ActivePath = {baseDir}/{tenantID}/teams/{teamID}/{chatID} - Scope = team - TeamPath = &ActivePath - -4. Team inbound (leader receiving message): - ActivePath = {baseDir}/{tenantID}/{agentID}/{userID} - Scope = personal - TeamPath = &teamWorkspacePath (read-only reference) - -5. Delegate: - ActivePath = own workspace - Scope = delegate - SharedPath = &sharedDelegatePath - ReadOnlyPaths = delegator exports - -6. Group chat: - ActivePath = {baseDir}/{tenantID}/{agentID}/{chatID} - Scope = personal -``` - -### System Prompt Template Section - -``` -## Workspace - -Active workspace: {{.ActivePath}} ({{.Scope}}) -{{if .EnforcementLabel}}[{{.EnforcementLabel}}]{{end}} -{{if .TeamPath}}Team workspace: {{.TeamPath}} (accessible via team file tools){{end}} -{{if .ReadOnlyPaths}}Read-only paths: {{range .ReadOnlyPaths}}- {{.}} -{{end}}{{end}} - -Context files (SOUL.md, USER.md, MEMORY.md): managed by system, read/write intercepted. -{{if eq .Scope "team"}}This is your active workspace. Personal workspace not accessible during this task.{{end}} -``` - -### Affected Files - -| File | What changes | -|------|-------------| -| `internal/agent/loop_context.go:110-176` | Replace workspace resolution with `Resolver.Resolve()` | -| `internal/tools/context_keys.go` | Remove `ctxWorkspace`, `ctxTeamWorkspace`, `effectiveRestrict()` | -| `internal/tools/filesystem.go:308-340` | `resolvePathWithAllowed()` reads from `WorkspaceContext` | -| `internal/tools/filesystem_write.go` | Same | -| `internal/tools/shell.go:243-267` | Working dir from `WorkspaceContext.ActivePath` | -| `internal/agent/systemprompt.go:498-518` | Replace workspace section with template | -| `internal/agent/systemprompt_sections.go:503-520` | Merge team workspace section into workspace template | - -### Breaking Changes - -- `ctxWorkspace` / `ctxTeamWorkspace` context keys removed -- `effectiveRestrict()` removed (always true, now implicit via WorkspaceContext) -- `WithToolWorkspace()` / `WithToolTeamWorkspace()` replaced by single `WithWorkspaceContext()` -- `tools.WithRestrictToWorkspace()` removed (enforcement = default) - ---- - -## C. EventBus Redesign - -### Interface Contract - -```go -// Package: internal/eventbus (new package, replaces internal/bus for domain events) -// Note: internal/bus MessageBus retained for channel message routing (InboundMessage/OutboundMessage). - -// DomainEvent is a typed event with metadata for consolidation pipeline. -type DomainEvent struct { - ID string // UUID v7 for ordering - Type EventType - SourceID string // dedup key (e.g. session key, run ID) - TenantID string - AgentID string - UserID string - Timestamp time.Time - Payload any // typed per EventType (see below) -} - -// EventType identifies the event category. -type EventType string - -const ( - EventSessionCompleted EventType = "session.completed" - EventEpisodicCreated EventType = "episodic.created" - EventEntityUpserted EventType = "entity.upserted" - EventMemoryLint EventType = "memory.lint" - EventRunCompleted EventType = "run.completed" - EventToolExecuted EventType = "tool.executed" -) - -// Typed payloads -- one per EventType. - -type SessionCompletedPayload struct { - SessionKey string - MessageCount int - TokensUsed int - Summary string // compaction summary if available -} - -type EpisodicCreatedPayload struct { - EpisodicID string - SessionKey string - Summary string - KeyEntities []string // entity names for downstream extraction -} - -type EntityUpsertedPayload struct { - EntityIDs []string // newly upserted entity UUIDs -} - -type MemoryLintPayload struct { - Scope string // "agent" or "user" -} - -type RunCompletedPayload struct { - RunID string - Iterations int - TokensUsed int - ToolCalls int - LoopKilled bool -} - -type ToolExecutedPayload struct { - ToolName string - Duration time.Duration - Success bool - ReadOnly bool -} - -// DomainEventBus manages typed event publishing + worker subscriptions. -type DomainEventBus interface { - // Publish enqueues an event for async processing. - // Non-blocking: returns immediately. - Publish(event DomainEvent) - - // Subscribe registers a worker for a specific event type. - // handler called in worker pool goroutine. - // Returns unsubscribe function. - Subscribe(eventType EventType, handler DomainEventHandler) func() - - // Start launches worker pool. Must be called before Publish. - Start(ctx context.Context) - - // Drain waits for all queued events to be processed. For graceful shutdown. - Drain(timeout time.Duration) error -} - -type DomainEventHandler func(ctx context.Context, event DomainEvent) error -``` - -### Worker Pool Design - -```go -// Config for the domain event bus. -type Config struct { - QueueSize int // buffered channel capacity (default 1000) - WorkerCount int // goroutines per event type (default 2) - RetryAttempts int // retry on handler error (default 3) - RetryDelay time.Duration // backoff base (default 1s, exponential) -} -``` - -Lane-based concurrency: each EventType gets its own worker pool. Events of different types processed in parallel. Events of same type processed sequentially per-worker (but multiple workers can process different events of same type concurrently). - -### Idempotency - -Dedup by `SourceID`: events with same SourceID + EventType processed at most once. Implementation: in-memory set with TTL (1h). If event bus restarts, re-processing is safe because all handlers are idempotent. - -```go -// dedup check before handler dispatch -func (b *bus) dispatch(event DomainEvent) { - key := event.Type + ":" + event.SourceID - if b.seen.Has(key) { - return // already processed - } - b.seen.Set(key, time.Now().Add(b.dedupTTL)) - // dispatch to handler... -} -``` - -### Integration Points - -| Producer | Event | Consumer | -|----------|-------|----------| -| Pipeline ObserveStage | `run.completed` | Metrics, tracing | -| Pipeline ObserveStage | `tool.executed` | Self-evolution metrics | -| Session close / compaction | `session.completed` | Episodic summary worker | -| Episodic worker | `episodic.created` | Semantic extraction worker | -| KG ingest | `entity.upserted` | Dedup check worker | -| Cron scheduler | `memory.lint` | Memory lint worker | - -### Affected Files - -| File | What changes | -|------|-------------| -| `internal/bus/types.go` | Retained for channel routing. DomainEvent types in new package | -| `internal/bus/bus.go` | Retained. New `internal/eventbus/` package alongside | -| `internal/agent/loop_run.go` | Publish `run.completed` after finalize | -| `internal/agent/loop_tools.go` | Publish `tool.executed` per tool call | -| `internal/agent/loop_history.go:624-699` | Publish `session.completed` in `maybeSummarize()` | - -### Breaking Changes - -- None for existing `bus.MessageBus` (retained) -- New `internal/eventbus` package added -- Agent loop gains `DomainEventBus` dependency (injected via `Loop` struct) - ---- - -## D. ProviderAdapter Interface - -### Interface Contract - -```go -// Package: internal/providers (extends existing package) - -// ProviderCapabilities declares what a provider supports. -// Queried by pipeline to choose code paths (streaming vs non-streaming, etc.) -type ProviderCapabilities struct { - Streaming bool // supports ChatStream() - ToolCalling bool // supports tools in request - StreamWithTools bool // can stream while tool calls are in-flight (false for DashScope) - Thinking bool // supports extended thinking / reasoning - Vision bool // supports image inputs - CacheControl bool // supports cache_control blocks (Anthropic) - MaxContextWindow int // default context window for default model - TokenizerID string // for tokencount package mapping -} - -// ProviderAdapter transforms between internal wire format and provider-specific format. -// Each provider implements this to isolate quirks from the pipeline loop. -type ProviderAdapter interface { - // ToRequest converts internal ChatRequest to provider-specific wire bytes. - // Handles: schema transforms, thinking passback, cache control injection. - ToRequest(req ChatRequest) ([]byte, http.Header, error) - - // FromResponse converts provider-specific response bytes to internal ChatResponse. - // Handles: thinking extraction, usage normalization, finish reason mapping. - FromResponse(data []byte) (*ChatResponse, error) - - // FromStreamChunk converts a single SSE chunk to internal StreamChunk. - // Returns nil if chunk should be skipped (keep-alive, metadata). - FromStreamChunk(data []byte) (*StreamChunk, error) - - // Capabilities returns static capability declaration. - Capabilities() ProviderCapabilities - - // Name returns provider identifier. - Name() string -} - -// CapabilitiesAware is optionally implemented by Provider. -// Pipeline checks this to choose code path. -type CapabilitiesAware interface { - Capabilities() ProviderCapabilities -} -``` - -### Adapter Registry - -```go -// AdapterRegistry maps provider names to adapter factories. -// Enables plugin-style provider registration. -type AdapterRegistry struct { - mu sync.RWMutex - adapters map[string]AdapterFactory -} - -type AdapterFactory func(cfg ProviderConfig) (ProviderAdapter, error) - -// Register adds a provider adapter. Called at init() or runtime. -func (r *AdapterRegistry) Register(name string, factory AdapterFactory) { - r.mu.Lock() - defer r.mu.Unlock() - r.adapters[name] = factory -} - -// Get returns adapter for provider name. Error if not registered. -func (r *AdapterRegistry) Get(name string, cfg ProviderConfig) (ProviderAdapter, error) { - r.mu.RLock() - factory, ok := r.adapters[name] - r.mu.RUnlock() - if !ok { - return nil, fmt.Errorf("unknown provider: %s", name) - } - return factory(cfg) -} -``` - -### What Moves to Adapter - -| Currently in loop/provider | Moves to adapter | -|---------------------------|-----------------| -| `RawAssistantContent` passback (Anthropic thinking blocks) | `AnthropicAdapter.ToRequest()` | -| `ThinkingSignature` accumulation | `AnthropicAdapter.FromStreamChunk()` | -| DashScope non-streaming fallback for tool calls | `DashScopeAdapter.Capabilities().StreamWithTools = false` | -| Codex `Phase` field injection | `CodexAdapter.ToRequest()` / `FromResponse()` | -| Schema normalization (tool definitions → provider format) | Each adapter's `ToRequest()` | -| Usage normalization (different field names) | Each adapter's `FromResponse()` | - -### Pipeline Usage - -``` -ThinkStage: - 1. caps := provider.Capabilities() // if CapabilitiesAware - 2. If caps.StreamWithTools == false && len(tools) > 0: - -> use Chat() not ChatStream() - 3. If caps.Thinking: - -> inject thinking options - 4. Call Chat/ChatStream as before (Provider interface unchanged) - 5. Provider internally uses its adapter for wire transforms -``` - -Important: `providers.Provider` interface UNCHANGED. `ProviderAdapter` is internal to each provider implementation. Pipeline queries capabilities via optional `CapabilitiesAware` interface on the provider. - -### Affected Files - -| File | What changes | -|------|-------------| -| `internal/providers/types.go` | Add `ProviderCapabilities`, `CapabilitiesAware` | -| `internal/providers/anthropic.go` | Extract wire transforms to `AnthropicAdapter` | -| `internal/providers/openai.go` | Extract wire transforms to `OpenAIAdapter` | -| `internal/providers/dashscope.go` | `StreamWithTools = false` in capabilities | -| `internal/providers/codex.go` | Phase handling in adapter | -| `internal/agent/loop.go:217-323` | ThinkStage queries capabilities instead of provider-type checks | - -### Breaking Changes - -- `ThinkingCapable` interface superseded by `CapabilitiesAware` (backward compatible -- both can coexist during transition) -- Provider-specific branching in loop.go removed (replaced by capability checks) -- No external API changes - ---- - -## Requirements - -### Functional -- F1: Token counting accurate within 5% of provider-reported usage -- F2: WorkspaceContext resolves all 6 scenarios correctly (solo DM, shared, team dispatch, team inbound, delegate, group) -- F3: DomainEventBus processes events within 100ms p99 (in-process) -- F4: Provider capabilities queryable without LLM call - -### Non-Functional -- NF1: TokenCounter cache hit rate >90% in typical session (messages rarely change) -- NF2: WorkspaceContext immutable after creation (no data races) -- NF3: EventBus handles 1000 events/sec sustained (consolidation pipeline load) -- NF4: All 4 components have zero cross-dependencies (can implement in parallel) - -## Design Steps - -1. Define `TokenCounter` interface + `ModelRegistry` mapping table -2. Implement `cachedCounter` with tiktoken-go + LRU cache -3. Define `WorkspaceContext` struct + `Resolver` interface -4. Implement resolver with resolution rules table -5. Define `DomainEvent` types + `DomainEventBus` interface -6. Implement worker pool with lane-based concurrency + dedup -7. Define `ProviderCapabilities` + `CapabilitiesAware` interface -8. Define `AdapterRegistry` for plugin registration - -## Todo List - -- [x] A1: TokenCounter interface definition — `internal/tokencount/token_counter.go` -- [x] A2: ModelRegistry with prefix-based mapping — `internal/tokencount/token_counter.go` -- [x] A3: Per-message hash cache design — documented in phase, cache impl deferred to implementation -- [x] A4: Fallback strategy for unknown models — `internal/tokencount/fallback_counter.go` -- [x] B1: WorkspaceContext struct definition — `internal/workspace/workspace_context.go` -- [x] B2: Resolver interface + ResolveParams — `internal/workspace/workspace_context.go` -- [x] B3: Resolution rules for all 6 scenarios — documented in phase design -- [x] B4: System prompt template section for workspace — documented in phase design -- [x] B5: Enforcement label design — EnforcementLabel field in WorkspaceContext -- [x] C1: DomainEvent type definitions — `internal/eventbus/event_types.go` -- [x] C2: Typed payload structs for each event — `internal/eventbus/event_types.go` -- [x] C3: DomainEventBus interface — `internal/eventbus/domain_event_bus.go` -- [x] C4: Worker pool config + lane-based concurrency design — `internal/eventbus/domain_event_bus.go` -- [x] C5: Idempotency / dedup strategy — documented in phase design (SourceID-based) -- [x] D1: ProviderCapabilities struct — `internal/providers/capabilities.go` -- [x] D2: CapabilitiesAware optional interface — `internal/providers/capabilities.go` -- [x] D3: AdapterRegistry for plugin registration — `internal/providers/adapter_registry.go` -- [x] D4: Document what moves from loop to each adapter — documented in phase design - -## Success Criteria - -- All 4 interfaces defined with complete method signatures -- Each interface has zero imports from the other 3 -- WorkspaceContext resolution rules cover all 6 scenarios -- Token counting accuracy validated against Anthropic/OpenAI reported usage -- EventBus worker pool supports lane-based concurrency with configurable workers -- Provider capabilities capture all known provider quirks (DashScope, Codex, Anthropic) - -## Risk Assessment - -| Risk | Severity | Mitigation | -|------|----------|------------| -| tiktoken-go tokenizer not exact match for Claude models | MEDIUM | Claude uses custom tokenizer, cl100k_base is approximation. Validate against API usage. Accept 5% error. | -| WorkspaceContext missing edge case for new workspace scenario | LOW | Resolution rules table is explicit. Add new row when new scenario appears. | -| EventBus in-process limitation (single instance, no persistence) | LOW | Sufficient for v3.0. External bus (Redis streams) can implement same interface later. | -| ProviderAdapter abstraction leaks (provider needs loop context) | MEDIUM | Keep adapter stateless. Pass all needed data via `ChatRequest` fields. | - -## Security Considerations - -- WorkspaceContext enforcement is security-critical: `ActivePath` must be validated (no path traversal) -- Existing `resolvePath()` security (symlink, hardlink checks) preserved -- WorkspaceContext provides the boundary, not the validation -- EventBus handlers must not leak tenant data across boundaries (TenantID in every event) -- TokenCounter is read-only, no security concerns - -## Next Steps - -- Phase 2 (Pipeline Loop) consumes all 4 interfaces -- Phase 3 (Memory & KG) consumes TokenCounter + DomainEventBus -- Phase 4 (System Integration) consumes WorkspaceContext + ProviderCapabilities diff --git a/plans/260406-1304-goclaw-v3-core-design/phase-02-pipeline-loop-design.md b/plans/260406-1304-goclaw-v3-core-design/phase-02-pipeline-loop-design.md deleted file mode 100644 index 1c18e283..00000000 --- a/plans/260406-1304-goclaw-v3-core-design/phase-02-pipeline-loop-design.md +++ /dev/null @@ -1,653 +0,0 @@ -# Phase 2: Pipeline Loop Design - -## Context Links - -- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (decision 1) -- Loop deep dive: `plans/reports/Explore-260406-GoClaw-Agent-Loop-v3-Deep-Dive.md` (all sections) -- Phase 1 dependencies: `phase-01-foundation-interfaces.md` (TokenCounter, WorkspaceContext, EventBus, ProviderCapabilities) -- Current loop: `internal/agent/loop.go` (804 lines), `loop_types.go`, `loop_context.go`, `loop_history.go`, `loop_compact.go`, `pruning.go`, `loop_tools.go`, `loop_run.go` - -## Overview - -- **Priority:** P1 (core execution engine, blocks phases 3-6) -- **Status:** Complete -- **Effort:** 8h design -- **Depends on:** Phase 1 (all 4 foundation interfaces) - -Replace monolithic `runLoop()` (804 lines, 20+ field `runState`) with a pipeline of pluggable stages. Each stage owns a substage of state, is independently testable, and communicates via a shared `RunState` struct. - -## Key Insights - -- Current loop has 7 exit conditions scattered across 800 lines. Pipeline model makes each exit condition a stage responsibility. -- `runState` has 20+ fields mixing concerns (loop control, output accumulation, crash safety, bootstrap detection, team task tracking, skill evolution). Split into typed substates. -- Mid-loop compaction (lines 352-411) interleaves with think phase. Pipeline separates this into `PruneStage` between think and tool. -- Parallel tool execution already well-isolated (goroutine per tool, results collected via channel). Wrapping in `ToolStage` is clean. -- Memory flush currently triggers inside `maybeSummarize()` which runs post-loop. Moving to `MemoryFlushStage` pre-compaction aligns with 3-tier memory design. - ---- - -## A. Stage Interface - -```go -// Package: internal/pipeline - -// Stage is a single step in the agent pipeline. -// Stages are stateless -- all mutable state lives in RunState. -type Stage interface { - // Name returns a human-readable identifier for logging/tracing. - Name() string - - // Execute performs the stage's work. Returns error to abort pipeline. - // Stages modify RunState in place. - Execute(ctx context.Context, state *RunState) error -} - -// StageResult signals how the pipeline should proceed after a stage. -type StageResult int - -const ( - Continue StageResult = iota // proceed to next stage - BreakLoop // exit iteration loop (normal completion) - AbortRun // abort entire run (error/kill) -) - -// StageWithResult extends Stage to control pipeline flow. -// If a stage does not implement this, pipeline assumes Continue. -type StageWithResult interface { - Stage - Result() StageResult -} -``` - -## B. RunState Redesign - -Split current 20+ field `runState` into typed substates. Each substate grouped by owning stage. - -```go -// RunState is the shared mutable state for a single pipeline run. -// Passed by pointer through all stages. -type RunState struct { - // Identity (set once at pipeline start, immutable during run) - Request *agent.RunRequest - Workspace *workspace.WorkspaceContext - Model string - Provider providers.Provider - - // Message buffer (read/write by multiple stages) - Messages *MessageBuffer - - // Per-stage substates - Context ContextState - Think ThinkState - Prune PruneState - Tool ToolState - Observe ObserveState - Compact CompactState - Evolution EvolutionState - - // Cross-cutting concerns - Iteration int - RunID string - ExitCode StageResult -} - -// MessageBuffer wraps the message list with append/replace semantics. -// Thread-safe for read; write only from owning stage (sequential pipeline). -type MessageBuffer struct { - system providers.Message // system prompt (rebuilt by ContextStage) - history []providers.Message // conversation history - pending []providers.Message // new messages this run (flushed at checkpoint) -} - -func (mb *MessageBuffer) All() []providers.Message { /* system + history + pending */ } -func (mb *MessageBuffer) AppendPending(msg providers.Message) { /* ... */ } -func (mb *MessageBuffer) FlushPending() []providers.Message { /* move pending to history */ } -func (mb *MessageBuffer) ReplaceHistory(msgs []providers.Message) { /* post-compaction */ } -``` - -### Substates - -```go -// ContextState: owned by ContextStage, read by ThinkStage -type ContextState struct { - ContextFiles []bootstrap.ContextFile - SkillsSummary string - TeamContext *TeamContextInfo // nil if not in team - HadBootstrap bool - OverheadTokens int // system prompt + context files token count (accurate via TokenCounter) -} - -// ThinkState: owned by ThinkStage -type ThinkState struct { - LastResponse *providers.ChatResponse - TotalUsage providers.Usage - TruncRetries int // consecutive truncation retries (max 3) - StreamingActive bool // true during active stream -} - -// PruneState: owned by PruneStage -type PruneState struct { - MidLoopCompacted bool // true after first in-loop compaction - HistoryTokens int // last computed history token count - HistoryBudget int // contextWindow * maxHistoryShare -} - -// ToolState: owned by ToolStage -type ToolState struct { - LoopDetector *ToolLoopDetector - TotalToolCalls int - AsyncToolCalls []string // tool names that executed async (spawn) - MediaResults []MediaResult // files produced by tools - Deliverables []string // content for team task results - LoopKilled bool // set when loop detector triggers critical -} - -// ObserveState: owned by ObserveStage -type ObserveState struct { - FinalContent string // accumulated response text - FinalThinking string // reasoning output - BlockReplies int - LastBlockReply string -} - -// CompactState: owned by CheckpointStage + MemoryFlushStage -type CompactState struct { - CheckpointFlushedMsgs int - MemoryFlushedThisCycle bool - CompactionCount int // tracks compaction cycles -} - -// EvolutionState: owned by skill evolution nudge logic -type EvolutionState struct { - Nudge70Sent bool - Nudge90Sent bool - PostscriptSent bool - BootstrapWrite bool // BOOTSTRAP.md write detected - TeamTaskCreates int - TeamTaskSpawns int -} -``` - -### Migration from current runState - -| Current field | Moves to | -|--------------|----------| -| `loopDetector` | `ToolState.LoopDetector` | -| `totalUsage` | `ThinkState.TotalUsage` | -| `iteration` | `RunState.Iteration` | -| `totalToolCalls` | `ToolState.TotalToolCalls` | -| `finalContent` | `ObserveState.FinalContent` | -| `finalThinking` | `ObserveState.FinalThinking` | -| `asyncToolCalls` | `ToolState.AsyncToolCalls` | -| `mediaResults` | `ToolState.MediaResults` | -| `deliverables` | `ToolState.Deliverables` | -| `pendingMsgs` | `MessageBuffer.pending` | -| `blockReplies` | `ObserveState.BlockReplies` | -| `lastBlockReply` | `ObserveState.LastBlockReply` | -| `checkpointFlushedMsgs` | `CompactState.CheckpointFlushedMsgs` | -| `midLoopCompacted` | `PruneState.MidLoopCompacted` | -| `overheadTokens` | `ContextState.OverheadTokens` | -| `overheadCalibrated` | REMOVED (overhead now accurate upfront via TokenCounter) | -| `bootstrapWriteDetected` | `EvolutionState.BootstrapWrite` | -| `teamTaskCreates/Spawns` | `EvolutionState.TeamTaskCreates/Spawns` | -| `skillNudge*` | `EvolutionState.Nudge*` | -| `loopKilled` | `ToolState.LoopKilled` | -| `truncationRetries` | `ThinkState.TruncRetries` | - ---- - -## C. Stage Catalog - -### Setup Pipeline (runs once before iteration loop) - -``` -1. ContextStage - - Resolve WorkspaceContext via Resolver - - Load + filter context files (bootstrap, per-user, team) - - Build skills summary (BM25) - - Build system prompt via template - - Compute overhead tokens (TokenCounter) - - Enrich input media - - Inject team task reminders - - Reads: RunState.Request, providers - Writes: RunState.Workspace, ContextState, MessageBuffer.system - - Replaces: injectContext() + buildMessages() + enrichInputMedia() + injectTeamTaskReminders() -``` - -### Iteration Pipeline (runs per iteration, max N times) - -``` -2. ThinkStage - - Inject iteration budget nudges (70%, 90%) - - Build per-iteration tool definitions (policy + bootstrap filters) - - Build ChatRequest from MessageBuffer - - Call LLM (Chat or ChatStream based on capabilities) - - Emit events: chunk, thinking - - Accumulate token usage - - Handle truncation retries - - Reads: MessageBuffer, ContextState, ProviderCapabilities - Writes: ThinkState, MessageBuffer (append assistant message) - Result: Continue (has tool calls) | BreakLoop (no tool calls, final content) - - Replaces: loop.go lines 217-446 - -3. PruneStage - - Compute history tokens via TokenCounter - - Phase 1 @ 70% budget: pruneContextMessages (soft-trim + hard-clear) - - Phase 2 @ 100% budget: compactMessagesInPlace (LLM summarization) - - If still over budget after Phase 2: AbortRun - - Reads: MessageBuffer, TokenCounter, context window - Writes: PruneState, MessageBuffer (may replace history) - Result: Continue | AbortRun (budget exceeded after compaction) - - Replaces: loop.go lines 352-411, loop_compact.go, pruning.go - -4. ToolStage - - Extract tool calls from ThinkState.LastResponse - - Generate unique tool call IDs - - Single tool: sequential execution - - Multiple tools: parallel goroutines, collect via channel, sort by index - - Per-tool: policy check, lazy MCP activate, execute, process result - - Loop detection: same-tool, read-only streak, same-result - - Tool budget check (totalToolCalls > maxToolCalls) - - Emit events: tool.call, tool.result - - Reads: ThinkState.LastResponse, ToolExecutor, WorkspaceContext - Writes: ToolState, MessageBuffer (append tool results) - Result: Continue | BreakLoop (loop killed) | AbortRun (policy denied) - - Replaces: loop.go lines 450-738, loop_tools.go - -5. ObserveStage - - Process tool results: extract deliverables, media - - Drain InjectCh (mid-run message injection, Point A + Point B) - - Check if final content ready (no tool calls + no injected messages) - - Sanitize final content - - Skill evolution postscript - - Publish domain events: tool.executed, run metrics - - Reads: ToolState, ThinkState, InjectCh - Writes: ObserveState, MessageBuffer (may append injected messages) - Result: Continue (more iterations needed) | BreakLoop (final content ready) - - Replaces: loop.go lines 450-466 (no-tool path), finalize portions - -6. CheckpointStage - - Every N iterations (default 5): flush pending messages to session store - - Crash safety: intermediate state persisted - - Conditional: skip if iteration % N != 0 - - Reads: MessageBuffer.pending, SessionStore - Writes: CompactState.CheckpointFlushedMsgs, MessageBuffer (clear pending) - - Replaces: loop.go lines 748-761 - -7. MemoryFlushStage - - Runs before compaction (when PruneStage detects budget pressure) - - Triggered by PruneStage setting a flag, NOT by iteration count - - Two-stage fallback: LLM flush -> extractive regex - - Dedup guard: once per compaction cycle - - Publishes: session.completed event (for episodic pipeline) - - Reads: MessageBuffer, MemoryStore, compaction config - Writes: CompactState.MemoryFlushedThisCycle - - Replaces: memoryflush.go, extractive_memory.go -``` - -### Post-Loop Pipeline (runs once after iteration loop exits) - -``` -8. FinalizeStage - - Sanitize final content (HTML entities, markdown cleanup) - - Media dedup + size population - - Flush remaining pending messages to session - - Token accumulation to session metadata - - Bootstrap auto-cleanup (delete BOOTSTRAP.md if write detected) - - Post-run summarization (maybeSummarize) - - Publish run.completed event - - Construct RunResult - - Reads: All substates - Writes: RunResult (returned to caller) - - Replaces: finalizeRun(), maybeSummarize() -``` - ---- - -## D. Pipeline Orchestrator - -```go -// Pipeline orchestrates stage execution for a single agent run. -type Pipeline struct { - setup []Stage // runs once before loop - iteration []Stage // runs per iteration - finalize []Stage // runs once after loop - - maxIterations int - tokenCounter tokencount.TokenCounter - eventBus eventbus.DomainEventBus -} - -// NewDefaultPipeline creates the standard agent pipeline. -func NewDefaultPipeline(deps PipelineDeps) *Pipeline { - return &Pipeline{ - setup: []Stage{ - NewContextStage(deps), - }, - iteration: []Stage{ - NewThinkStage(deps), - NewPruneStage(deps), - NewToolStage(deps), - NewObserveStage(deps), - NewCheckpointStage(deps), - }, - finalize: []Stage{ - NewFinalizeStage(deps), - }, - maxIterations: deps.Config.MaxIterations, - tokenCounter: deps.TokenCounter, - eventBus: deps.EventBus, - } -} - -// PipelineDeps bundles all dependencies injected into stages. -type PipelineDeps struct { - Provider providers.Provider - TokenCounter tokencount.TokenCounter - EventBus eventbus.DomainEventBus - SessionStore store.SessionStore - ToolExecutor tools.ToolExecutor - ToolPolicy *tools.PolicyEngine - SkillsLoader *skills.Loader - MemoryStore store.MemoryStore - WorkspaceResolver workspace.Resolver - Config PipelineConfig -} - -type PipelineConfig struct { - MaxIterations int - MaxToolCalls int - CheckpointInterval int // flush every N iterations (default 5) - ContextWindow int - MaxTokens int - Compaction *config.CompactionConfig - ContextPruning *config.ContextPruningConfig -} -``` - -### Execution Flow - -```go -// Run executes the full pipeline for a single agent run. -func (p *Pipeline) Run(ctx context.Context, req *agent.RunRequest) (*agent.RunResult, error) { - state := NewRunState(req) - - // 1. Setup pipeline (once) - for _, stage := range p.setup { - if err := stage.Execute(ctx, state); err != nil { - return nil, fmt.Errorf("setup %s: %w", stage.Name(), err) - } - } - - // 2. Iteration loop - for state.Iteration = 0; state.Iteration < p.maxIterations; state.Iteration++ { - for _, stage := range p.iteration { - if err := stage.Execute(ctx, state); err != nil { - return nil, fmt.Errorf("iter %d %s: %w", state.Iteration, stage.Name(), err) - } - - // Check if stage wants to control flow - if swr, ok := stage.(StageWithResult); ok { - switch swr.Result() { - case BreakLoop: - goto finalize - case AbortRun: - goto finalize - } - } - } - - // Check context cancellation (/stop) - if ctx.Err() != nil { - break - } - } - -finalize: - // 3. Finalize pipeline (once) - for _, stage := range p.finalize { - if err := stage.Execute(ctx, state); err != nil { - slog.Warn("finalize stage error", "stage", stage.Name(), "err", err) - // finalize errors are logged, not fatal - } - } - - return state.BuildResult(), nil -} -``` - -### Error Propagation - -``` -Stage returns error: - - Setup stage error -> abort run, return error - - Iteration stage error -> abort run, still run finalize - - Finalize stage error -> log warning, continue other finalize stages - -Stage returns StageResult: - - Continue -> next stage in iteration - - BreakLoop -> skip remaining iteration stages, jump to finalize - - AbortRun -> skip remaining stages, jump to finalize (with error flag) -``` - ---- - -## Data Flow Between Stages - -``` -ContextStage - writes: Workspace, ContextState, MessageBuffer.system - | - v -ThinkStage - reads: MessageBuffer.All(), ContextState.OverheadTokens - writes: ThinkState.LastResponse, ThinkState.TotalUsage - writes: MessageBuffer.AppendPending(assistant msg) - | - v -PruneStage - reads: MessageBuffer, TokenCounter, PruneState - writes: PruneState.HistoryTokens - may trigger: MemoryFlushStage (inline, before compaction) - may write: MessageBuffer.ReplaceHistory(compacted) - | - v -ToolStage - reads: ThinkState.LastResponse.ToolCalls - writes: ToolState.*, MessageBuffer.AppendPending(tool results) - | - v -ObserveStage - reads: ToolState, ThinkState, InjectCh - writes: ObserveState.FinalContent (if done) - may write: MessageBuffer.AppendPending(injected msgs) - | - v -CheckpointStage - reads: MessageBuffer.pending - writes: SessionStore, CompactState.CheckpointFlushedMsgs - | - v (loop back to ThinkStage or exit to FinalizeStage) - -FinalizeStage - reads: All substates - writes: RunResult -``` - ---- - -## MemoryFlushStage Integration with PruneStage - -MemoryFlushStage is NOT a regular iteration stage. It is invoked inline by PruneStage when compaction is needed. - -```go -// PruneStage.Execute -func (s *PruneStage) Execute(ctx context.Context, state *RunState) error { - tokens := s.tokenCounter.CountMessages(state.Model, state.Messages.All()) - state.Prune.HistoryTokens = tokens - - budget := s.contextWindow * 80 / 100 // 80% for history - - // Phase 1: soft pruning at 70% - if tokens > budget*70/100 { - pruneContextMessages(state.Messages, s.config) - tokens = s.tokenCounter.CountMessages(state.Model, state.Messages.All()) - } - - // Phase 2: full compaction at 100% - if tokens > budget { - // Memory flush BEFORE compaction (saves memories before they're summarized away) - if !state.Compact.MemoryFlushedThisCycle && s.memoryFlush != nil { - s.memoryFlush.Execute(ctx, state) // inline call, not pipeline stage - state.Compact.MemoryFlushedThisCycle = true - } - - state.Prune.MidLoopCompacted = true - compactMessagesInPlace(state.Messages, s.compactionConfig) - - // Re-check after compaction - tokens = s.tokenCounter.CountMessages(state.Model, state.Messages.All()) - if tokens > budget { - state.ExitCode = AbortRun - } - } - return nil -} -``` - ---- - -## Requirements - -### Functional -- F1: Pipeline produces identical `RunResult` as current monolithic loop for same inputs -- F2: All 7 exit conditions preserved: no-tool-calls, max iterations, truncation limit, loop kill, read-only streak, tool budget, ctx.Done -- F3: Parallel tool execution behavior unchanged (goroutine per tool, sorted results) -- F4: Mid-run injection (InjectCh) works at both Point A (after tools) and Point B (no tools) -- F5: Checkpoint flush every N iterations, crash safety preserved - -### Non-Functional -- NF1: No measurable latency increase from stage dispatch overhead (stages are coarse-grained) -- NF2: Each stage independently testable with mock RunState -- NF3: Adding new stage requires only: implement Stage interface + insert in pipeline constructor -- NF4: RunState substates compile-time typed (no `map[string]any`) - -## Related Code Files - -### Files to decompose - -| Current file | Lines | Maps to stages | -|-------------|-------|---------------| -| `internal/agent/loop.go` | 804 | ThinkStage, ToolStage, ObserveStage, orchestrator | -| `internal/agent/loop_run.go` | 245 | Pipeline.Run() entry point | -| `internal/agent/loop_context.go` | 275 | ContextStage | -| `internal/agent/loop_history.go` | 859 | ContextStage (buildMessages), FinalizeStage (maybeSummarize) | -| `internal/agent/loop_compact.go` | 119 | PruneStage | -| `internal/agent/pruning.go` | 327 | PruneStage | -| `internal/agent/loop_tools.go` | ~180 | ToolStage | -| `internal/agent/loop_tool_filter.go` | ~100 | ThinkStage (buildFilteredTools) | -| `internal/agent/memoryflush.go` | ~150 | MemoryFlushStage | -| `internal/agent/extractive_memory.go` | ~150 | MemoryFlushStage (fallback) | -| `internal/agent/loop_types.go` | ~150 | RunState, substates | -| `internal/agent/loop_input_media.go` | ~100 | ContextStage | -| `internal/agent/loop_team_reminders.go` | ~60 | ContextStage | - -### New files to create - -| File | Purpose | -|------|---------| -| `internal/pipeline/stage.go` | Stage interface, StageResult, StageWithResult | -| `internal/pipeline/run_state.go` | RunState + MessageBuffer + all substates | -| `internal/pipeline/pipeline.go` | Pipeline orchestrator | -| `internal/pipeline/context_stage.go` | ContextStage implementation | -| `internal/pipeline/think_stage.go` | ThinkStage implementation | -| `internal/pipeline/prune_stage.go` | PruneStage + inline MemoryFlush | -| `internal/pipeline/tool_stage.go` | ToolStage implementation | -| `internal/pipeline/observe_stage.go` | ObserveStage implementation | -| `internal/pipeline/checkpoint_stage.go` | CheckpointStage implementation | -| `internal/pipeline/finalize_stage.go` | FinalizeStage implementation | - -## Design Steps - -1. Define Stage interface + StageResult enum -2. Design RunState with typed substates (map current fields) -3. Define MessageBuffer with append/replace/flush semantics -4. Design each stage's Execute signature and data flow -5. Design Pipeline orchestrator with setup/iteration/finalize phases -6. Map error propagation rules (setup fatal, iteration -> finalize, finalize logged) -7. Design MemoryFlushStage inline integration with PruneStage -8. Verify all 7 exit conditions map to StageResult values - -## Todo List - -- [x] A1: Stage interface + StageResult + StageWithResult — `internal/pipeline/stage.go` -- [x] B1: RunState top-level struct — `internal/pipeline/run_state.go` -- [x] B2: MessageBuffer with thread-safe read, owning-stage write — `internal/pipeline/run_state.go` -- [x] B3: ContextState substate — `internal/pipeline/substates.go` -- [x] B4: ThinkState substate — `internal/pipeline/substates.go` -- [x] B5: PruneState substate — `internal/pipeline/substates.go` -- [x] B6: ToolState substate — `internal/pipeline/substates.go` -- [x] B7: ObserveState substate — `internal/pipeline/substates.go` -- [x] B8: CompactState substate — `internal/pipeline/substates.go` -- [x] B9: EvolutionState substate — `internal/pipeline/substates.go` -- [x] C1: ContextStage design — documented in phase (impl deferred) -- [x] C2: ThinkStage design — documented in phase (impl deferred) -- [x] C3: PruneStage design — documented in phase (impl deferred) -- [x] C4: ToolStage design — documented in phase (impl deferred) -- [x] C5: ObserveStage design — documented in phase (impl deferred) -- [x] C6: CheckpointStage design — documented in phase (impl deferred) -- [x] C7: MemoryFlushStage design — documented in phase (impl deferred) -- [x] C8: FinalizeStage design — documented in phase (impl deferred) -- [x] D1: Pipeline orchestrator with 3-phase execution — `internal/pipeline/pipeline.go` -- [x] D2: PipelineDeps dependency bundle — via NewPipeline() params -- [x] D3: Error propagation rules — setup fatal, iteration fatal+finalize, finalize logged -- [x] E1: Verify all 7 exit conditions preserved — documented in phase design -- [x] E2: Map current runState fields to substates — documented in phase design - -## Success Criteria - -- Every line of current `loop.go` maps to exactly one stage -- `runState` fields fully accounted for in substates (no orphans) -- Pipeline.Run() produces same RunResult semantics as current Loop.Run() -- Each stage can be tested with a constructed RunState (no Loop struct dependency) -- MemoryFlushStage integrated with PruneStage without circular dependency -- InjectCh drain preserved at both injection points - -## Risk Assessment - -| Risk | Severity | Mitigation | -|------|----------|------------| -| Subtle behavior change from stage ordering | HIGH | Integration tests comparing v2 vs v3 output for same inputs. Run existing test suite. | -| InjectCh timing sensitivity (Point A vs Point B) | MEDIUM | ObserveStage handles both points explicitly. Document injection semantics. | -| MemoryFlushStage inline call breaks stage isolation | LOW | Acceptable trade-off: flush MUST happen before compaction. Document as intentional coupling. | -| RunState too large (sum of substates) | LOW | Substates are small structs. Total < 500 bytes. No heap pressure. | -| Stage interface too simple for complex stages | LOW | StageWithResult extension handles flow control. Further extension possible via StageWithCleanup etc. | - -## Security Considerations - -- ToolStage must preserve all existing security checks: policy engine, shell deny patterns, workspace restriction -- ContextStage must preserve input guard scanning (injection pattern detection) -- CheckpointStage flushes to session store with existing tenant scoping -- No new attack surface introduced by pipeline refactor - -## Next Steps - -- Phase 3 (Memory & KG) adds MemoryFlushStage consumer for `session.completed` events -- Phase 4 (System Integration) adds template-based system prompt in ContextStage -- Phase 5 (Orchestration) may add DelegateStage or modify ToolStage for delegation diff --git a/plans/260406-1304-goclaw-v3-core-design/phase-03-memory-kg-design.md b/plans/260406-1304-goclaw-v3-core-design/phase-03-memory-kg-design.md deleted file mode 100644 index 19def25a..00000000 --- a/plans/260406-1304-goclaw-v3-core-design/phase-03-memory-kg-design.md +++ /dev/null @@ -1,797 +0,0 @@ -# Phase 3: Memory & KG Design - -## Context Links - -- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (decisions 2, 3, 4, 6, 7) -- Research: `plans/reports/researcher-260406-1059-memory-context-kg-research.md` -- Architecture: `plans/reports/Explore-260406-thorough-v3-architecture.md` (sections 1, 2, 8, 10) -- Phase 1: `phase-01-foundation-interfaces.md` (TokenCounter, DomainEventBus) -- Phase 2: `phase-02-pipeline-loop-design.md` (PruneStage, MemoryFlushStage, ObserveStage) -- Current memory store: `internal/store/memory_store.go` -- Current KG store: `internal/store/knowledge_graph_store.go` -- Current KG migration: `migrations/000013_knowledge_graph.up.sql` -- Current memory migration: `migrations/000001_init_schema.up.sql` (lines 163-192) - -## Overview - -- **Priority:** P1 (core cognitive capability) -- **Status:** Complete -- **Effort:** 8h design -- **Depends on:** Phase 1 (TokenCounter for L0 budget, DomainEventBus for consolidation), Phase 2 (pipeline integration points) - -Design 3-tier memory system, temporal KG, consolidation pipeline, L0/L1/L2 context tiering, and smart auto-inject retrieval. This phase transforms GoClaw from append-only memory to a curated, searchable, self-maintaining knowledge system. - -## Key Insights - -- Current memory is append-only daily .md files, never revisited. Contradictions accumulate, no consolidation, no dedup -- KG extraction is synchronous (blocks loop). Dedup thresholds hardcoded (0.98/0.90). No temporal awareness -- stale facts never expire -- Memory search is hybrid FTS+vector but flat -- no tier awareness, no relevance-based injection -- Karpathy insight: memory should be curated wiki, not append-only log -- OpenViking insight: L0/L1/L2 tiering saves 10x tokens vs full-context loading -- Graphiti insight: temporal validity windows on KG facts enable contradiction resolution - ---- - -## A. 3-Tier Memory Schema - -### Tier 1: Working Memory - -In context window. No new storage needed. Managed by MessageBuffer in pipeline. - -- Content: recent 20-50 messages (conversation turns) -- Lifecycle: per-turn, free (no persistence cost) -- Size: up to 80% of context window budget -- Pruned by: PruneStage (token-accurate via TokenCounter) - -### Tier 2: Episodic Memory - -New table. Session summaries + key interactions with pgvector embeddings. - -```sql --- Migration: 000XXX_episodic_summaries.up.sql - -CREATE TABLE episodic_summaries ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE, - agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE, - user_id VARCHAR(255) NOT NULL DEFAULT '', - session_key VARCHAR(500) NOT NULL, - - -- Content - summary TEXT NOT NULL, -- LLM-generated session summary - key_entities JSONB DEFAULT '[]', -- entity names extracted (for downstream KG) - key_topics JSONB DEFAULT '[]', -- topic tags for BM25 - message_count INT NOT NULL DEFAULT 0, -- messages in summarized session - token_count INT NOT NULL DEFAULT 0, -- total tokens in session - - -- Embeddings for semantic search - embedding vector(1536), - tsv tsvector GENERATED ALWAYS AS (to_tsvector('simple', summary)) STORED, - - -- L0 abstract (pre-computed, ~50 tokens) - l0_abstract TEXT NOT NULL DEFAULT '', - - -- Lifecycle - source_id VARCHAR(255) NOT NULL DEFAULT '', -- dedup key (session_key + compaction_count) - created_at TIMESTAMPTZ DEFAULT NOW(), - expires_at TIMESTAMPTZ, -- NULL = no expiry, default 90d from creation - - UNIQUE(agent_id, user_id, source_id) -); - -CREATE INDEX idx_episodic_scope ON episodic_summaries(agent_id, user_id); -CREATE INDEX idx_episodic_tenant ON episodic_summaries(tenant_id); -CREATE INDEX idx_episodic_created ON episodic_summaries(agent_id, user_id, created_at DESC); -CREATE INDEX idx_episodic_expires ON episodic_summaries(expires_at) WHERE expires_at IS NOT NULL; -CREATE INDEX idx_episodic_tsv ON episodic_summaries USING gin(tsv); -CREATE INDEX idx_episodic_embedding ON episodic_summaries USING ivfflat (embedding vector_cosine_ops) - WITH (lists = 50); -``` - -### Tier 3: Semantic Memory - -Existing KG tables (`kg_entities`, `kg_relations`) + temporal fields (section B). Facts extracted from episodic summaries via consolidation pipeline. Permanent, deduplicated. - -No new table needed. Temporal fields added to existing tables. - -### Tier Flow - -``` -Working (Tier 1) Episodic (Tier 2) Semantic (Tier 3) -[in context window] [episodic_summaries] [kg_entities + kg_relations] - | | | - |-- session end / compaction ------>| | - | (session.completed event) | | - | |-- extraction worker -------->| - | | (episodic.created event) | - | | | - |<-- auto-inject L0 summaries ------| | - |<-- memory_search L1/L2 ---------->| | - |<-- kg_search ------------------------------------------->| | -``` - ---- - -## B. Temporal KG Schema - -### Schema Changes - -```sql --- Migration: 000XXX_kg_temporal.up.sql - --- Add temporal validity to entities -ALTER TABLE kg_entities - ADD COLUMN valid_from TIMESTAMPTZ DEFAULT NOW(), - ADD COLUMN valid_until TIMESTAMPTZ; -- NULL = currently valid - --- Add temporal validity to relations -ALTER TABLE kg_relations - ADD COLUMN valid_from TIMESTAMPTZ DEFAULT NOW(), - ADD COLUMN valid_until TIMESTAMPTZ; - --- Backfill existing data: valid_from = created_at, valid_until = NULL (current) -UPDATE kg_entities SET valid_from = created_at WHERE valid_from IS NULL; -UPDATE kg_relations SET valid_from = created_at WHERE valid_from IS NULL; - --- Partial index for current facts (most common query) -CREATE INDEX idx_kg_entities_current ON kg_entities(agent_id, user_id) - WHERE valid_until IS NULL; -CREATE INDEX idx_kg_relations_current ON kg_relations(agent_id, user_id) - WHERE valid_until IS NULL; - --- Add configurable dedup thresholds (per-agent, stored in agents.other_config JSONB) --- No schema change needed -- uses existing agents.other_config field: --- { "kg_dedup_auto_threshold": 0.98, "kg_dedup_flag_threshold": 0.90 } -``` - -### Query Patterns - -```sql --- Current facts (default, most queries) -SELECT * FROM kg_entities -WHERE agent_id = $1 AND user_id = $2 AND valid_until IS NULL; - --- Historical: what was true at a specific time? -SELECT * FROM kg_entities -WHERE agent_id = $1 AND user_id = $2 - AND valid_from <= $3 - AND (valid_until IS NULL OR valid_until >= $3); - --- Supersede a fact: new contradicting fact arrives -UPDATE kg_entities SET valid_until = NOW(), updated_at = NOW() -WHERE agent_id = $1 AND user_id = $2 AND external_id = $3 AND valid_until IS NULL; --- Then INSERT new entity with valid_from = NOW() -``` - -### Contradiction Resolution - -When extraction produces a fact that contradicts an existing current fact: - -``` -1. Extractor produces: "User works at CompanyB" (confidence 0.85) -2. Existing current fact: "User works at CompanyA" (confidence 0.90) -3. If same external_id + same entity_type: - a. Supersede old: SET valid_until = NOW() on CompanyA fact - b. Insert new: CompanyB with valid_from = NOW() -4. If different external_id but related via relation: - a. Flag as dedup candidate (human review) -``` - -### Configurable Thresholds - -```go -// Per-agent KG config (stored in agents.other_config JSONB) -type KGConfig struct { - DedupAutoThreshold float64 `json:"kg_dedup_auto_threshold"` // default 0.98 - DedupFlagThreshold float64 `json:"kg_dedup_flag_threshold"` // default 0.90 - ExtractionMinConf float64 `json:"kg_extraction_min_conf"` // default 0.75 - EnableTemporal bool `json:"kg_enable_temporal"` // default true -} -``` - ---- - -## C. Consolidation Pipeline - -Event-driven, async, non-blocking. User never waits for consolidation. - -### Event Flow - -``` -session.completed - | - v -EpisodicWorker - |-- Summarize session messages via LLM - |-- Extract key entities (names only, for downstream) - |-- Generate L0 abstract (~1 sentence, ~50 tokens) - |-- Store in episodic_summaries - |-- Generate embedding (async) - |-- Publish: episodic.created - | - v -SemanticExtractionWorker - |-- Read episodic summary - |-- Run KG extractor on summary (not full session -- cheaper) - |-- Upsert entities + relations with temporal fields - |-- Publish: entity.upserted - | - v -DedupWorker - |-- Read newly upserted entity IDs - |-- Compare embeddings against existing entities - |-- Auto-merge (> auto_threshold + name match) - |-- Flag candidates (> flag_threshold) - |-- Handle temporal supersession for contradictions -``` - -### Worker Implementations - -```go -// Package: internal/consolidation - -// EpisodicWorker processes session.completed events. -type EpisodicWorker struct { - store store.EpisodicStore - provider providers.Provider // for LLM summarization - tokenCounter tokencount.TokenCounter - eventBus eventbus.DomainEventBus -} - -func (w *EpisodicWorker) Handle(ctx context.Context, event eventbus.DomainEvent) error { - payload := event.Payload.(eventbus.SessionCompletedPayload) - - // Skip if already processed (idempotent by source_id) - sourceID := fmt.Sprintf("%s:%d", payload.SessionKey, payload.CompactionCount) - if w.store.ExistsBySourceID(ctx, event.AgentID, event.UserID, sourceID) { - return nil // already processed - } - - // 1. Summarize session - summary, err := w.summarizeSession(ctx, payload) - if err != nil { - return fmt.Errorf("summarize: %w", err) - } - - // 2. Extract key entities (lightweight -- name extraction, not full KG) - entities := extractEntityNames(summary) - - // 3. Generate L0 abstract - l0 := generateL0Abstract(summary) // extractive: first sentence or topic sentence - - // 4. Store - ep := &store.EpisodicSummary{ - AgentID: event.AgentID, - UserID: event.UserID, - TenantID: event.TenantID, - SessionKey: payload.SessionKey, - Summary: summary, - KeyEntities: entities, - L0Abstract: l0, - MessageCount: payload.MessageCount, - TokenCount: payload.TokensUsed, - SourceID: sourceID, - ExpiresAt: timePtr(time.Now().Add(90 * 24 * time.Hour)), // 90d TTL - } - if err := w.store.Create(ctx, ep); err != nil { - return fmt.Errorf("store episodic: %w", err) - } - - // 5. Publish for downstream - w.eventBus.Publish(eventbus.DomainEvent{ - Type: eventbus.EventEpisodicCreated, - AgentID: event.AgentID, - UserID: event.UserID, - Payload: eventbus.EpisodicCreatedPayload{ - EpisodicID: ep.ID, - SessionKey: payload.SessionKey, - Summary: summary, - KeyEntities: entities, - }, - }) - - return nil -} -``` - -### Summarization Prompt (Episodic) - -``` -Summarize this conversation session concisely. Focus on: -- Key decisions made -- Facts learned about the user or project -- Tasks completed or in-progress -- Important technical details -- User preferences expressed - -Output: 2-4 paragraph summary. Include entity names (people, projects, tools) explicitly. -Do NOT include greetings, filler, or metadata. -``` - -### L0 Abstract Generation - -V3.0: extractive (first meaningful sentence of summary). V3.1+: LLM-generated. - -```go -// generateL0Abstract produces a ~50 token abstract from episodic summary. -// V3.0: extractive (first sentence). Future: LLM-generated. -func generateL0Abstract(summary string) string { - // Split into sentences, take first non-trivial one - sentences := splitSentences(summary) - for _, s := range sentences { - s = strings.TrimSpace(s) - if len(s) > 20 { // skip very short fragments - if len(s) > 200 { // truncate if too long - s = s[:200] + "..." - } - return s - } - } - // Fallback: first 200 chars - if len(summary) > 200 { - return summary[:200] + "..." - } - return summary -} -``` - ---- - -## D. L0/L1/L2 Context Tiering - -### Layer Definitions - -| Layer | Content | Token cost | When loaded | Storage | -|-------|---------|-----------|-------------|---------| -| **L0** | 1-sentence abstract | ~50 tokens | Always (auto-inject if relevant) | `episodic_summaries.l0_abstract` | -| **L1** | Structured overview | ~200 tokens | When topic relevant | Generated on-demand, cached | -| **L2** | Full content | Full | On-demand via tool call | `episodic_summaries.summary` (episodic) or original document (memory docs) | - -### L1 Generation - -On-demand when agent requests deeper context or auto-inject detects high relevance. - -```go -// L1Cache caches generated L1 overviews. LRU, 500 entries, 1h TTL. -type L1Cache struct { - cache *lru.Cache[string, string] // key: episodic_id or doc_path -} - -// GenerateL1 creates structured overview from episodic summary or memory doc. -func (c *L1Cache) GenerateL1(id string, fullContent string) string { - if cached, ok := c.cache.Get(id); ok { - return cached - } - // Extractive: key bullet points from full content - l1 := extractKeyPoints(fullContent, 200) // ~200 tokens budget - c.cache.Add(id, l1) - return l1 -} -``` - -### L2 Access - -Full content loaded via tool call. Agent calls `memory_search(query, depth="full")` or `memory_expand(id)`. - -```go -// memory_expand tool: loads L2 full content for a specific memory entry -// Parameters: id (episodic ID or document path) -// Returns: full content (summary or document) -``` - ---- - -## E. Smart Auto-Inject Retrieval - -Per-turn relevance check + L0 injection. Non-blocking, lightweight. - -### Flow - -``` -User message arrives (in ContextStage) - | - v -1. Extract intent keywords from user message - - Tokenize, remove stopwords, keep nouns/verbs/proper nouns - - If message too short ("hello", "ok"): skip injection (no relevant context) - | - v -2. BM25 + embedding relevance check against episodic index - - BM25 on episodic_summaries.tsv (fast, FTS) - - Embedding similarity on episodic_summaries.embedding (if available) - - Merge scores (configurable weights) - | - v -3. Filter by relevance threshold - - Default: 0.3 (tunable per agent via agent config) - - If no results above threshold: skip injection - | - v -4. Inject "## Memory Context" section into system prompt - - Max 5 entries, max ~200 tokens total (L0 summaries only) - - Format per entry: "- {topic}: {L0 abstract} [use memory_search for details]" - | - v -5. Agent sees hint, can call memory_search for L1/L2 -``` - -### Interface - -```go -// Package: internal/memory - -// AutoInjector checks relevance and produces L0 injection for system prompt. -type AutoInjector interface { - // Inject checks user message against memory index. - // Returns formatted section for system prompt, or "" if nothing relevant. - // Budget: max ~200 tokens of L0 summaries. - Inject(ctx context.Context, params InjectParams) (string, error) -} - -type InjectParams struct { - AgentID string - UserID string - TenantID string - UserMessage string - MaxEntries int // default 5 - MaxTokens int // default 200 - Threshold float64 // relevance threshold (default 0.3) -} - -// InjectResult for observability -type InjectResult struct { - Section string // formatted prompt section - MatchCount int // total matches found - Injected int // entries injected (after budget trim) - TopScore float64 // highest relevance score -} -``` - -### System Prompt Section (generated by AutoInjector) - -``` -## Memory Context - -Relevant memories from past sessions (summaries only -- use memory_search for details): -- Project Alpha: Discussed migration from PostgreSQL to CockroachDB, decided to stay with PG for now. -- User preferences: Prefers Go over Python for backend, uses Neovim as primary editor. -- Bug fix: Resolved auth token expiration issue in session middleware (2026-03-15). - -Use `memory_search` tool to retrieve full details on any topic above. -``` - -### Agent Config - -```go -// Per-agent memory config (extends existing config.MemoryConfig) -type MemoryConfig struct { - // Existing fields... - Enabled bool `json:"enabled"` - - // V3 additions - AutoInjectEnabled bool `json:"auto_inject_enabled"` // default true - AutoInjectThreshold float64 `json:"auto_inject_threshold"` // default 0.3 - AutoInjectMaxTokens int `json:"auto_inject_max_tokens"` // default 200 - EpisodicTTLDays int `json:"episodic_ttl_days"` // default 90 (0 = no expiry) - ConsolidationEnabled bool `json:"consolidation_enabled"` // default true -} -``` - ---- - -## F. MemoryStore v3 Interface - -Extends existing `MemoryStore` with tier-aware operations. - -```go -// Package: internal/store - -// EpisodicSummary represents a Tier 2 episodic memory entry. -type EpisodicSummary struct { - ID string `json:"id"` - TenantID string `json:"tenant_id"` - AgentID string `json:"agent_id"` - UserID string `json:"user_id"` - SessionKey string `json:"session_key"` - Summary string `json:"summary"` - KeyEntities []string `json:"key_entities"` - KeyTopics []string `json:"key_topics"` - L0Abstract string `json:"l0_abstract"` - MessageCount int `json:"message_count"` - TokenCount int `json:"token_count"` - SourceID string `json:"source_id"` - CreatedAt time.Time `json:"created_at"` - ExpiresAt *time.Time `json:"expires_at,omitempty"` -} - -// EpisodicSearchResult is a search hit with tier indicator. -type EpisodicSearchResult struct { - EpisodicID string `json:"episodic_id"` - L0Abstract string `json:"l0_abstract"` // always populated - Score float64 `json:"score"` - CreatedAt time.Time `json:"created_at"` - SessionKey string `json:"session_key"` -} - -// EpisodicStore manages Tier 2 episodic memory. -type EpisodicStore interface { - // CRUD - Create(ctx context.Context, ep *EpisodicSummary) error - Get(ctx context.Context, id string) (*EpisodicSummary, error) - Delete(ctx context.Context, id string) error - List(ctx context.Context, agentID, userID string, limit, offset int) ([]EpisodicSummary, error) - - // Search (hybrid FTS + vector, returns L0 by default) - Search(ctx context.Context, query string, agentID, userID string, opts EpisodicSearchOptions) ([]EpisodicSearchResult, error) - - // Lifecycle - ExistsBySourceID(ctx context.Context, agentID, userID, sourceID string) (bool, error) - PruneExpired(ctx context.Context) (int, error) // delete where expires_at < NOW() - - // Embedding - SetEmbeddingProvider(provider EmbeddingProvider) - Close() error -} - -type EpisodicSearchOptions struct { - MaxResults int - MinScore float64 - VectorWeight float64 - TextWeight float64 -} - -// MemorySearchResultV3 extends search result with tier indicator. -type MemorySearchResultV3 struct { - MemorySearchResult // embed existing fields - Tier string `json:"tier"` // "working", "episodic", "semantic" - EpisodicID string `json:"episodic_id,omitempty"` - L0Abstract string `json:"l0_abstract,omitempty"` -} - -// UnifiedMemorySearch combines Tier 2 (episodic) and existing memory doc search. -// Used by memory_search tool. -type UnifiedMemorySearch interface { - // Search across all tiers. Returns results with tier indicators. - Search(ctx context.Context, query string, agentID, userID string, opts UnifiedSearchOptions) ([]MemorySearchResultV3, error) - - // Expand loads full content for a specific memory entry (L2). - Expand(ctx context.Context, id string, tier string) (string, error) -} - -type UnifiedSearchOptions struct { - MaxResults int - MinScore float64 - Tiers []string // filter: ["episodic", "document"] (empty = all) - Depth string // "l0", "l1", "l2" (default "l1") -} -``` - -### KnowledgeGraphStore v3 Extensions - -```go -// Temporal query additions to existing KnowledgeGraphStore interface. - -// TemporalQueryOptions extends entity queries with time awareness. -type TemporalQueryOptions struct { - AsOf *time.Time // point-in-time query (nil = current only) - IncludeExpired bool // include superseded facts -} - -// New methods added to KnowledgeGraphStore: - -// ListEntitiesTemporal queries entities with temporal awareness. -// When opts.AsOf is nil, returns current facts only (valid_until IS NULL). -// When opts.AsOf is set, returns facts valid at that timestamp. -ListEntitiesTemporal(ctx context.Context, agentID, userID string, - listOpts EntityListOptions, temporal TemporalQueryOptions) ([]Entity, error) - -// SupersedeEntity marks entity as no longer valid and inserts replacement. -// Atomic: UPDATE old SET valid_until=NOW() + INSERT new in single tx. -SupersedeEntity(ctx context.Context, old *Entity, replacement *Entity) error - -// GetDedupConfig returns per-agent dedup thresholds from agents.other_config. -// Returns defaults if not configured. -GetDedupConfig(ctx context.Context, agentID string) (*KGConfig, error) -``` - -### Entity Struct v3 - -```go -// Extended Entity with temporal fields. -type Entity struct { - // ... existing fields unchanged ... - ID string `json:"id"` - AgentID string `json:"agent_id"` - UserID string `json:"user_id,omitempty"` - ExternalID string `json:"external_id"` - Name string `json:"name"` - EntityType string `json:"entity_type"` - Description string `json:"description,omitempty"` - Properties map[string]string `json:"properties,omitempty"` - SourceID string `json:"source_id,omitempty"` - Confidence float64 `json:"confidence"` - CreatedAt int64 `json:"created_at"` - UpdatedAt int64 `json:"updated_at"` - - // V3 temporal fields - ValidFrom *time.Time `json:"valid_from,omitempty"` - ValidUntil *time.Time `json:"valid_until,omitempty"` // nil = currently valid -} - -// Relation with temporal fields (same pattern). -type Relation struct { - // ... existing fields unchanged ... - - // V3 temporal fields - ValidFrom *time.Time `json:"valid_from,omitempty"` - ValidUntil *time.Time `json:"valid_until,omitempty"` -} -``` - ---- - -## Pipeline Integration Points - -### Where Memory/KG Connects to Pipeline Stages - -| Pipeline stage | Memory/KG interaction | -|---------------|----------------------| -| **ContextStage** | AutoInjector.Inject() -> produces ## Memory Context section | -| **PruneStage** | Triggers MemoryFlushStage before compaction | -| **MemoryFlushStage** | Writes to memory docs (existing). Publishes `session.completed` | -| **FinalizeStage** | Publishes `session.completed` for episodic consolidation | -| **ToolStage** | `memory_search` tool calls UnifiedMemorySearch | -| **ToolStage** | `memory_expand` tool calls UnifiedMemorySearch.Expand() | -| **ToolStage** | `kg_search` tool uses temporal queries (default: current facts) | - -### Consolidation Pipeline Registration - -```go -// At gateway startup, register consolidation workers with DomainEventBus. - -func RegisterConsolidationPipeline(eb eventbus.DomainEventBus, deps ConsolidationDeps) { - episodicWorker := NewEpisodicWorker(deps) - semanticWorker := NewSemanticExtractionWorker(deps) - dedupWorker := NewDedupWorker(deps) - - eb.Subscribe(eventbus.EventSessionCompleted, episodicWorker.Handle) - eb.Subscribe(eventbus.EventEpisodicCreated, semanticWorker.Handle) - eb.Subscribe(eventbus.EventEntityUpserted, dedupWorker.Handle) -} -``` - ---- - -## Requirements - -### Functional -- F1: Episodic summaries created automatically after session completion -- F2: L0 abstracts pre-computed at episodic creation (~50 tokens each) -- F3: Auto-inject produces relevant L0 summaries for user messages (skip for trivial messages) -- F4: Temporal KG queries default to current facts; historical queries via `AsOf` parameter -- F5: Contradiction resolution: supersede old fact, insert new with valid_from = NOW() -- F6: Dedup thresholds configurable per agent (not hardcoded) -- F7: Consolidation pipeline non-blocking (user never waits) -- F8: `memory_search` tool returns results with tier indicators -- F9: `memory_expand` tool loads L2 full content -- F10: Expired episodic entries pruned automatically (cron or lazy) - -### Non-Functional -- NF1: Auto-inject latency < 50ms p99 (BM25 + embedding lookup) -- NF2: Episodic summarization completes within 30s (LLM call) -- NF3: Consolidation pipeline handles 100 session.completed events/minute -- NF4: pgvector index efficient up to 100K embeddings per agent -- NF5: L0 injection budget: max 200 tokens per turn - -## Related Code Files - -### Files to modify - -| File | Change | -|------|--------| -| `internal/store/knowledge_graph_store.go` | Add temporal fields to Entity/Relation, new temporal query methods | -| `internal/store/pg/knowledge_graph*.go` | Implement temporal queries, supersede logic | -| `internal/store/memory_store.go` | No change (existing store retained for Tier 1/document memory) | -| `internal/knowledgegraph/extractor.go` | Source extraction from episodic summaries (lighter than raw messages) | -| `internal/agent/systemprompt.go` | Add ## Memory Context section from AutoInjector | -| `internal/tools/memory_tools.go` | Update memory_search to use UnifiedMemorySearch, add memory_expand | - -### New files to create - -| File | Purpose | -|------|---------| -| `internal/store/episodic_store.go` | EpisodicStore interface + types | -| `internal/store/pg/episodic_summaries.go` | PG implementation | -| `internal/memory/auto_injector.go` | AutoInjector implementation | -| `internal/memory/l1_cache.go` | L1 on-demand generation + LRU cache | -| `internal/consolidation/episodic_worker.go` | session.completed -> episodic summary | -| `internal/consolidation/semantic_worker.go` | episodic.created -> KG extraction | -| `internal/consolidation/dedup_worker.go` | entity.upserted -> dedup check | -| `internal/consolidation/registration.go` | Worker registration with EventBus | -| `migrations/000XXX_episodic_summaries.up.sql` | Episodic table + indexes | -| `migrations/000XXX_kg_temporal.up.sql` | Temporal fields on KG tables | - -## Design Steps - -1. Define `episodic_summaries` DDL with indexes -2. Define `EpisodicStore` interface -3. Design temporal KG schema changes (ALTER TABLE + backfill) -4. Define temporal query methods for KnowledgeGraphStore -5. Design consolidation pipeline event flow -6. Design EpisodicWorker with summarization prompt + L0 generation -7. Design AutoInjector with BM25 + embedding relevance check -8. Define UnifiedMemorySearch interface for cross-tier search -9. Design L0/L1/L2 depth parameter for search results -10. Define per-agent memory config extensions - -## Todo List - -- [x] A1: episodic_summaries DDL — documented in phase (SQL in Phase 6) -- [x] A2: EpisodicStore interface — `internal/store/episodic_store.go` -- [x] A3: EpisodicSummary struct + EpisodicSearchResult — `internal/store/episodic_store.go` -- [x] B1: kg_entities temporal fields DDL — documented in phase (SQL in Phase 6) -- [x] B2: kg_relations temporal fields DDL — documented in phase (SQL in Phase 6) -- [x] B3: Backfill migration — documented in phase design -- [x] B4: Temporal query methods — `internal/store/knowledge_graph_temporal.go` -- [x] B5: SupersedeEntity atomic method — `internal/store/knowledge_graph_temporal.go` -- [x] B6: Configurable dedup thresholds — `internal/store/knowledge_graph_temporal.go` KGConfig -- [x] C1: Consolidation event flow design — documented in phase -- [x] C2: EpisodicWorker handler — `internal/consolidation/workers.go` (stub) -- [x] C3: SemanticExtractionWorker handler — `internal/consolidation/workers.go` (stub) -- [x] C4: DedupWorker handler — `internal/consolidation/workers.go` (stub) -- [x] C5: Worker registration at startup — `internal/consolidation/workers.go` Register() -- [x] D1: L0 abstract generation — documented in phase design (extractive v3.0) -- [x] D2: L1 on-demand generation + LRU cache — documented in phase design -- [x] D3: L2 access via memory_expand tool — documented in phase design -- [x] E1: AutoInjector interface — `internal/memory/auto_injector.go` -- [x] E2: Relevance check (BM25 + embedding) — documented in phase design -- [x] E3: System prompt ## Memory Context section — documented in phase design -- [x] E4: Per-agent threshold config — `internal/memory/auto_injector.go` MemoryConfig -- [x] F1: UnifiedMemorySearch interface — documented in phase design -- [x] F2: MemorySearchResultV3 with tier indicator — documented in phase design -- [x] F3: memory_search tool update — documented in phase design -- [x] F4: memory_expand tool definition — documented in phase design - -## Success Criteria - -- episodic_summaries DDL passes `psql` validation -- Temporal KG migration backward-compatible (existing data has valid_from=created_at, valid_until=NULL) -- All consolidation workers idempotent (safe to replay events) -- AutoInjector returns "" for trivial messages ("hello", "ok") -- AutoInjector returns relevant L0s for topic-specific messages -- UnifiedMemorySearch merges results across tiers with correct tier labels -- Per-agent dedup thresholds configurable, defaults match current hardcoded values - -## Risk Assessment - -| Risk | Severity | Mitigation | -|------|----------|------------| -| Episodic summarization LLM cost | MEDIUM | Use smaller/cheaper model for summarization (e.g. GPT-4o-mini). Configurable per agent. | -| Auto-inject false positives (irrelevant memories injected) | MEDIUM | Conservative default threshold (0.3). Agent can override. Monitor via evolution metrics. | -| pgvector index performance at scale (100K+ embeddings) | MEDIUM | ivfflat index with lists=50. Benchmark needed. Increase lists for larger datasets. | -| Consolidation pipeline backlog during high load | LOW | Lane-based concurrency in EventBus. Workers are idempotent. Backlog clears naturally. | -| L0 extractive quality insufficient | LOW | Extractive is "good enough" for v3.0. LLM-generated L0 in v3.1. | -| Temporal backfill migration slow on large KG | LOW | Backfill is simple UPDATE SET DEFAULT. Add timeout guard. | -| Cross-agent memory leakage | HIGH | All queries scoped by tenant_id + agent_id + user_id. Verify in code review. | - -## Security Considerations - -- Tenant isolation: ALL episodic + KG queries must include tenant_id scope. Missing tenant_id = data leak. -- User isolation: episodic summaries scoped by user_id unless shared memory enabled -- L0/L1 content is derived from user conversations -- same access controls as original messages -- Consolidation workers run in system context -- must not expose cross-tenant data -- AutoInjector results filtered by same agent_id + user_id scope as memory_search - -## Unresolved Questions - -1. **Episodic TTL**: 90 days default. Should this be configurable per agent? (Likely yes, via MemoryConfig) -2. **L0 for memory documents**: Should existing memory_documents also get L0 abstracts? (Deferred to v3.1 -- episodic first) -3. **Cross-agent memory**: Should agents in same tenant share semantic memory? (Brainstorm Q2, deferred) -4. **Embedding model**: Continue OpenAI text-embedding-3-small or switch to open model? (Brainstorm Q3, deferred) -5. **Concurrent consolidation**: Multiple sessions ending simultaneously -> same episodic worker. EventBus dedup by source_id handles this, but verify ordering. - -## Next Steps - -- Phase 4 (System Integration) integrates AutoInjector into ContextStage template -- Phase 5 (Orchestration) uses self-evolution metrics from consolidation pipeline -- Phase 6 (Migration) handles v2 daily memory files -> episodic import diff --git a/plans/260406-1304-goclaw-v3-core-design/phase-04-system-integration-design.md b/plans/260406-1304-goclaw-v3-core-design/phase-04-system-integration-design.md deleted file mode 100644 index a44a7009..00000000 --- a/plans/260406-1304-goclaw-v3-core-design/phase-04-system-integration-design.md +++ /dev/null @@ -1,707 +0,0 @@ -# Phase 4: System Integration Design - -## Context Links -- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (sections 9, 12, 14, 7) -- Architecture report: `plans/reports/Explore-260406-thorough-v3-architecture.md` (sections 5, 6) -- Workspace forcing: `plans/reports/Explore-260406-workspace-forcing.md` -- Current system prompt: `internal/agent/systemprompt.go`, `internal/agent/systemprompt_sections.go` -- Current tool registry: `internal/tools/registry.go`, `internal/tools/policy.go`, `internal/tools/types.go` -- Current skills: `internal/skills/loader.go`, `internal/skills/search.go`, `internal/store/skill_store.go` - -## Overview -- **Priority:** P1 (core dependency for pipeline loop + memory system) -- **Status:** Complete -- **Effort:** 6h design - -Redesigns 4 tightly-coupled systems that sit between the agent loop and the LLM: -1. Template-based system prompt (replace string concatenation) -2. Tool registry with typed capabilities + declarative policy -3. Skills system with embedding search + ACL + dependency graph -4. Retrieval integration connecting auto-inject L0 with template + tools - ---- - -## Key Insights - -1. Current `BuildSystemPrompt()` is ~200 lines of conditional string building. 15+ sections toggled by booleans. Adding workspace enforcement language or memory context requires touching fragile concatenation logic. -2. Tool types use 12+ optional interfaces (`InterceptorAware`, `PathAllowable`, etc.) with type assertions at registration. No capability metadata on tools themselves. -3. PolicyEngine's 7-step pipeline works but all checks are string-based group matching. No typed capabilities (read-only vs mutating), no workspace-awareness. -4. Skills BM25-only search misses intent-based queries ("help me debug" won't find "debugger" skill). `EmbeddingSkillSearcher` interface exists in store but not wired to loader's `BuildSummary()`. -5. Workspace confusion root cause: system prompt text says "use relative paths" but doesn't state the enforcement rule. Template must carry WorkspaceContext enforcement language verbatim. - ---- - -## Requirements - -### Functional -- F1: System prompt built from Go `text/template` with embedded template file -- F2: Each template section independently toggleable via PromptConfig -- F3: WorkspaceContext generates explicit enforcement language (not suggestive) -- F4: Tools declare capabilities (read-only, mutating, async, mcp-bridged) at registration -- F5: Policy engine evaluates capabilities (e.g., "deny all mutating tools for viewer role") -- F6: Tools receive WorkspaceContext, not raw path strings -- F7: Skills searched by both BM25 (lexical) and embedding (semantic), results merged -- F8: Skills filtered by agent/tenant ACL before injection -- F9: Skill dependency graph: loader resolves + validates before activation -- F10: Auto-inject L0 memory summaries into template's Memory section -- F11: Provider-specific template variants (e.g., Codex needs different tool format) - -### Non-Functional -- NF1: Template rendering <1ms (no LLM calls during prompt build) -- NF2: Skill embedding search <50ms at 1000 skills -- NF3: Backward compatible: v2 PromptConfig can produce identical v2 output -- NF4: Template testable in isolation (no DB dependency for rendering) - ---- - -## Architecture - -### A. Template-Based System Prompt - -#### PromptConfig Interface Contract - -```go -// PromptConfig controls which template sections to include. -// Each field maps to a named template block. -type PromptConfig struct { - // Section toggles - Identity bool - Persona bool // SOUL.md content - Instructions bool // AGENTS.md content - Tools bool - Skills bool - Team bool - Workspace bool - Memory bool // auto-inject L0 section - Sandbox bool - Recency bool // recency reminder - - // Data payloads (populated when section enabled) - IdentityData IdentityData - PersonaContent string // raw SOUL.md - InstructionContent string // raw AGENTS.md - ToolsData ToolsSectionData - SkillsData SkillsSectionData - TeamData TeamSectionData - WorkspaceData WorkspaceSectionData - MemoryData MemorySectionData - SandboxData SandboxSectionData - ExtraPrompt string - - // Provider variant (selects template file) - ProviderVariant string // "" = default, "codex", "dashscope" -} -``` - -#### Section Data Types - -```go -type IdentityData struct { - AgentName string - Emoji string - Model string - Channel string - ChatTitle string - PeerKind string // "direct" or "group" -} - -type ToolsSectionData struct { - ToolDefs []ToolSummary // name + one-line description - MCPTools []ToolSummary // MCP bridge tools (separate listing) - CoreSummary map[string]string // builtin tool summaries -} - -type ToolSummary struct { - Name string - Description string - Capability ToolCapability // read-only, mutating, async, mcp-bridged -} - -type SkillsSectionData struct { - Mode string // "inline" (inject summaries) or "search" (search tool only) - Summaries []SkillSummaryEntry -} - -type SkillSummaryEntry struct { - Slug string - Name string - Description string - Score float64 // relevance score from search -} - -type TeamSectionData struct { - TeamWorkspace string - Members []TeamMemberEntry - Guidance string // edition-specific (Full vs Lite) - TeamMDContent string // TEAM.md content -} - -type TeamMemberEntry struct { - AgentKey string - DisplayName string - Skills string - Role string // "lead" or "member" -} - -type WorkspaceSectionData struct { - ActivePath string - Scope string // "personal" | "team-shared" | "team-isolated" | "delegate" - Enforced bool // always true in v3 - ReadOnlyPaths []string - SharedPath *string - ContextFiles []string // list of managed context files - EnforcementMsg string // pre-rendered enforcement text -} - -type MemorySectionData struct { - L0Summaries []L0Summary // auto-injected memory hints - HasSearch bool // memory_search tool available - HasKG bool // kg_search tool available -} - -type L0Summary struct { - Topic string - Summary string // 1-sentence abstract - ID string // for memory_expand(id) deep retrieval -} - -type SandboxSectionData struct { - ContainerDir string - AccessLevel string // "full" or "read-only" -} -``` - -#### Template Structure - -Embedded template file at `internal/agent/prompt_template.go.tmpl`: - -``` -{{- /* System Prompt Template — v3 */ -}} - -{{- if .Identity -}} -{{- template "identity" .IdentityData -}} -{{- end -}} - -{{- if .Persona -}} -# Persona -{{ .PersonaContent }} -{{- end -}} - -{{- if .Instructions -}} -# Instructions -{{ .InstructionContent }} -{{- end -}} - -{{- if .Workspace -}} -{{- template "workspace" .WorkspaceData -}} -{{- end -}} - -{{- if .Tools -}} -{{- template "tools" .ToolsData -}} -{{- end -}} - -{{- if .Skills -}} -{{- template "skills" .SkillsData -}} -{{- end -}} - -{{- if .Memory -}} -{{- template "memory" .MemoryData -}} -{{- end -}} - -{{- if .Sandbox -}} -{{- template "sandbox" .SandboxData -}} -{{- end -}} - -{{- if .Team -}} -{{- template "team" .TeamData -}} -{{- end -}} - -{{- if .ExtraPrompt -}} -# Additional Context -{{ .ExtraPrompt }} -{{- end -}} -``` - -#### Workspace Sub-Template (Critical) - -``` -{{- define "workspace" -}} -# Workspace -- **Path:** {{ .ActivePath }} -- **Scope:** {{ .Scope }} -- **ENFORCED:** All file tool paths resolve relative to this directory. Absolute paths outside workspace are REJECTED. -{{- if .ReadOnlyPaths }} -- **Read-only access:** {{ range .ReadOnlyPaths }}{{ . }}, {{ end }} -{{- end }} -{{- if .SharedPath }} -- **Shared area:** {{ .SharedPath }} (read/write by both you and delegator) -{{- end }} -{{- if .ContextFiles }} -- **Managed files:** {{ range .ContextFiles }}{{ . }}, {{ end }}(system-managed, read/write intercepted to DB) -{{- end }} -{{- end -}} -``` - -#### Memory Sub-Template - -``` -{{- define "memory" -}} -# Memory Context -{{- if .L0Summaries }} -{{- range .L0Summaries }} -- **{{ .Topic }}:** {{ .Summary }} [use memory_search("{{ .ID }}") for details] -{{- end }} -{{- end }} -{{- if .HasSearch }} -Use `memory_search` for detailed retrieval. Use `kg_search` for relationship queries. -{{- end }} -{{- end -}} -``` - -#### PromptBuilder Interface - -```go -// PromptBuilder renders system prompts from PromptConfig. -type PromptBuilder interface { - Build(cfg PromptConfig) (string, error) -} - -// TemplatePromptBuilder implements PromptBuilder using text/template. -type TemplatePromptBuilder struct { - templates map[string]*template.Template // variant -> parsed template -} - -// NewTemplatePromptBuilder loads embedded templates. -func NewTemplatePromptBuilder() (*TemplatePromptBuilder, error) - -// Build renders prompt. Selects template variant from cfg.ProviderVariant. -// Returns rendered string. Error only on template parse failure (should not happen with embedded). -func (b *TemplatePromptBuilder) Build(cfg PromptConfig) (string, error) -``` - -#### Provider Variants - -- **Default template:** Standard Anthropic/OpenAI format -- **Codex variant:** Tool descriptions formatted as docstrings (Codex prefers this) -- **DashScope variant:** No streaming+tools section (DashScope limitation) -- Variant selected by `cfg.ProviderVariant` which maps to provider capabilities - ---- - -### B. Tool Registry Redesign - -#### Tool Capability Types - -```go -// ToolCapability describes what a tool can do. -type ToolCapability string - -const ( - CapReadOnly ToolCapability = "read-only" // no side effects (read_file, list_files, memory_search) - CapMutating ToolCapability = "mutating" // modifies state (write_file, exec, edit) - CapAsync ToolCapability = "async" // returns immediately, result via callback - CapMCPBridged ToolCapability = "mcp-bridged" // proxied to external MCP server -) - -// ToolMetadata describes a tool's capabilities and requirements. -type ToolMetadata struct { - Name string - Capabilities []ToolCapability - Group string // "fs", "web", "runtime", "memory", "team", etc. - RequiresWorkspace bool // tool needs WorkspaceContext - ProviderHints map[string]any // provider-specific hints (e.g., cache_control) -} -``` - -#### Enhanced Tool Interface - -```go -// Tool is the base interface all tools must implement. -// V3 adds Metadata() for capability declaration. -type Tool interface { - Name() string - Description() string - Parameters() map[string]any - Execute(ctx context.Context, args map[string]any) *Result - Metadata() ToolMetadata // NEW: declarative capability info -} -``` - -> **Migration note:** Existing tools that don't implement `Metadata()` get a wrapper -> that returns default metadata (capabilities inferred from tool group membership). - -#### WorkspaceContext in Tool Context - -```go -// Tools access WorkspaceContext via context, not struct fields. -// Replaces: ToolWorkspaceFromCtx(), ToolTeamWorkspaceFromCtx() - -func WorkspaceContextFromCtx(ctx context.Context) *WorkspaceContext -func WithWorkspaceContext(ctx context.Context, wc *WorkspaceContext) context.Context -``` - -All file tools use `WorkspaceContextFromCtx(ctx)` instead of separate workspace/team-workspace context keys. WorkspaceContext carries ActivePath, ReadOnlyPaths, SharedPath — one struct, one lookup. - -#### Enhanced PolicyEngine - -```go -// PolicyRule is a declarative access rule. -type PolicyRule struct { - // Match conditions (all must match) - Capabilities []ToolCapability // match tools with these capabilities - Groups []string // match tools in these groups - Names []string // match specific tool names - - // Action - Action PolicyAction // allow or deny - Reason string // audit log reason -} - -type PolicyAction string -const ( - PolicyAllow PolicyAction = "allow" - PolicyDeny PolicyAction = "deny" -) - -// PolicyEngine evaluates tool access via ordered rule list. -// V3 adds capability-based rules alongside existing group/name rules. -type PolicyEngine struct { - globalPolicy *config.ToolsConfig - rules []PolicyRule // declarative rules (evaluated in order, first match wins) -} - -// FilterTools returns allowed tools. Now evaluates both legacy 7-step pipeline -// AND declarative rules. Declarative rules take precedence. -func (pe *PolicyEngine) FilterTools( - registry ToolExecutor, - agentID string, - providerName string, - agentToolPolicy *config.ToolPolicySpec, - groupToolAllow []string, - isSubagent bool, - isLeafAgent bool, - role string, // NEW: RBAC role ("admin", "operator", "viewer") -) []providers.ToolDefinition -``` - -#### Tool Definition Generation for Providers - -```go -// ToolDefAdapter generates provider-specific tool definitions. -// Replaces direct ToProviderDef() calls. -type ToolDefAdapter interface { - // ToProviderDefs converts internal tool list to provider wire format. - // Handles provider-specific quirks (DashScope tool format, Codex docstrings, etc.) - ToProviderDefs(tools []Tool, caps ProviderCapabilities) []providers.ToolDefinition -} -``` - -#### Pipeline Integration - -ToolStage in pipeline receives registry + policy engine: -```go -type ToolStage struct { - registry *Registry - policyEngine *PolicyEngine - adapter ToolDefAdapter -} - -// Execute runs tool, records metrics, returns result for ObserveStage. -func (s *ToolStage) Execute(ctx context.Context, state *RunState) error -``` - -ObserveStage processes tool results: -```go -type ObserveStage struct { - loopDetector *LoopDetector - metricsStore EvolutionMetricsStore // record tool effectiveness -} - -// Execute checks for tool loops, records metrics, decides continue/stop. -func (s *ObserveStage) Execute(ctx context.Context, state *RunState) error -``` - ---- - -### C. Skills System Upgrade - -#### Hybrid Search Interface - -```go -// SkillSearcher provides both lexical and semantic skill search. -type SkillSearcher interface { - // Search returns skills matching query, merging BM25 + embedding results. - // Returns at most limit results, deduplicated by slug. - Search(ctx context.Context, query string, limit int) ([]SkillSearchResult, error) - - // SearchForAgent filters by ACL before search. - SearchForAgent(ctx context.Context, agentID uuid.UUID, userID string, query string, limit int) ([]SkillSearchResult, error) -} - -// HybridSkillSearcher implements SkillSearcher with BM25 + embedding fusion. -type HybridSkillSearcher struct { - bm25Index *skills.Index // existing BM25 index - embSearcher store.EmbeddingSkillSearcher // existing PG embedding search - accessStore store.SkillAccessStore // ACL filtering - bm25Weight float64 // default 0.4 - embedWeight float64 // default 0.6 -} -``` - -#### Score Fusion Strategy - -``` -1. BM25 search → top 2*limit results, normalize scores to [0,1] -2. Embedding search → top 2*limit results, normalize scores to [0,1] -3. Merge by slug: combined_score = bm25Weight * bm25_score + embedWeight * embed_score -4. Sort by combined_score descending -5. Return top limit -``` - -#### ACL Model - -```go -// SkillVisibility controls who can see/use a skill. -type SkillVisibility string - -const ( - SkillVisibilityPublic SkillVisibility = "public" // all agents in tenant - SkillVisibilityPrivate SkillVisibility = "private" // only granted agents - SkillVisibilitySystem SkillVisibility = "system" // system-managed, immutable -) - -// ACL check flow: -// 1. If skill.Visibility == "public" → allow -// 2. If skill.Visibility == "private" → check skill_agent_grants table -// 3. If skill.Visibility == "system" → allow (system skills always accessible) -// Existing SkillAccessStore.ListAccessible() already implements this. -``` - -No new DB tables needed. Existing `managed_skills.visibility` + `skill_agent_grants` + `SkillAccessStore` interface covers ACL. The change is wiring ACL into the search path (currently BM25 search ignores ACL). - -#### Dependency Graph - -```go -// SkillDependency represents skill A requires skill B. -// Stored in skill frontmatter: `depends: [slug-b, slug-c]` -type SkillDependency struct { - Slug string // this skill - Requires []string // slugs of required skills -} - -// DepChecker validates skill dependencies are satisfied. -// Existing internal/skills/dep_checker.go already handles this. -// V3 enhancement: block activation of skill with unresolved deps. -type DepChecker interface { - // Check returns missing dependencies for a skill. - Check(slug string, available []string) []string - - // ResolveOrder returns activation order respecting dependency graph. - // Error if circular dependency detected. - ResolveOrder(slugs []string) ([]string, error) -} -``` - -No new DB schema. Dependencies stored in SKILL.md frontmatter `depends:` field. `dep_checker.go` already parses this. V3 adds `ResolveOrder()` for topological sort + circular detection. - -#### Hot-Reload Enhancement - -```go -// SkillLoader manages skill discovery, loading, and hot-reload. -// V3 adds ACL-aware loading + hybrid search integration. -type SkillLoader interface { - // LoadAll discovers and loads all skills from all tiers. - LoadAll(ctx context.Context) error - - // BuildSummary returns skill summary for system prompt injection. - // V3: uses HybridSkillSearcher instead of BM25-only. - BuildSummary(ctx context.Context, agentID uuid.UUID, userID string, query string) string - - // Version returns current reload version (atomic counter). - Version() int64 - - // OnReload registers callback for hot-reload events. - OnReload(fn func()) -} -``` - ---- - -### D. Retrieval Integration - -#### Auto-Inject L0 Flow - -``` -Per turn (in ContextStage): -1. Extract keywords from latest user message (simple tokenization, same as BM25) -2. Check relevance against memory index: - a. BM25 score against episodic summaries - b. Embedding similarity against semantic memory (KG entities) -3. If max_score > agent.retrieval_threshold (default 0.3): - → Build L0Summaries for PromptConfig.MemoryData - → Cap at max_l0_tokens (default 200 tokens, counted via TokenCounter) -4. PromptBuilder renders Memory section with L0 hints -5. Agent sees hints, can call memory_search(id) for L1/L2 detail -``` - -#### Retrieval Config - -```go -// RetrievalConfig controls auto-inject behavior. Per-agent configurable. -type RetrievalConfig struct { - Enabled bool `json:"enabled"` // default true - RelevanceThreshold float64 `json:"relevance_threshold"` // default 0.3 - MaxL0Tokens int `json:"max_l0_tokens"` // default 200 - MaxL0Items int `json:"max_l0_items"` // default 5 - BM25Weight float64 `json:"bm25_weight"` // default 0.4 - EmbeddingWeight float64 `json:"embedding_weight"` // default 0.6 -} -``` - -Stored in agent settings JSON (`agents.settings` column, existing JSONB field). No new DB columns. - -#### Retrieval Interface - -```go -// Retriever performs auto-inject L0 retrieval for ContextStage. -type Retriever interface { - // RetrieveL0 returns L0 summaries relevant to the query. - // Called once per turn in ContextStage. - RetrieveL0(ctx context.Context, agentID uuid.UUID, userID string, query string, cfg RetrievalConfig) ([]L0Summary, error) -} - -// HybridRetriever implements Retriever using memory + KG stores. -type HybridRetriever struct { - memoryStore store.MemoryStore - kgStore store.KnowledgeGraphStore - tokenCounter TokenCounter - embProvider store.EmbeddingProvider -} -``` - -#### memory_search / kg_search Tool Updates - -```go -// memory_search tool gains L1/L2 mode parameter. -// Args: { "query": "...", "depth": "l1"|"l2" } -// L1: returns structured overview (episodic summaries, ~200 tokens each) -// L2: returns full detail (original content, KG traversal) -// Default: "l1" (backward compatible — current behavior is L1-equivalent) - -// kg_search tool gains temporal parameter. -// Args: { "query": "...", "as_of": "2026-01-01" } -// as_of: returns facts valid at that date. Omit = current facts only. -``` - ---- - -## Related Code Files - -### Files to Modify -| File | Change | -|------|--------| -| `internal/agent/systemprompt.go` | Replace BuildSystemPrompt() with PromptBuilder.Build() | -| `internal/agent/systemprompt_sections.go` | Extract section data builders, remove string concat | -| `internal/agent/loop_history.go` | Use PromptBuilder in buildMessages() | -| `internal/tools/types.go` | Add Metadata() to Tool interface | -| `internal/tools/registry.go` | Store ToolMetadata, add capability queries | -| `internal/tools/policy.go` | Add capability-based rules, role parameter | -| `internal/tools/context_keys.go` | Replace dual workspace keys with WorkspaceContext | -| `internal/tools/filesystem.go` | Use WorkspaceContextFromCtx() | -| `internal/tools/filesystem_write.go` | Use WorkspaceContextFromCtx() | -| `internal/tools/filesystem_list.go` | Use WorkspaceContextFromCtx() | -| `internal/tools/shell.go` | Use WorkspaceContextFromCtx() | -| `internal/skills/loader.go` | Wire HybridSkillSearcher into BuildSummary() | -| `internal/skills/search.go` | Keep BM25, add fusion logic | -| `internal/tools/memory_search.go` | Add depth parameter (l1/l2) | -| `internal/tools/knowledge_graph_search.go` | Add as_of temporal parameter | - -### Files to Create -| File | Purpose | -|------|---------| -| `internal/agent/prompt_builder.go` | TemplatePromptBuilder implementation | -| `internal/agent/prompt_template.go.tmpl` | Embedded template file (default variant) | -| `internal/agent/prompt_template_codex.go.tmpl` | Codex variant | -| `internal/agent/prompt_config_types.go` | PromptConfig, section data types | -| `internal/tools/capability.go` | ToolCapability, ToolMetadata types | -| `internal/tools/workspace_context.go` | WorkspaceContext struct + context helpers | -| `internal/tools/tool_def_adapter.go` | ToolDefAdapter interface + default impl | -| `internal/skills/hybrid_searcher.go` | HybridSkillSearcher | -| `internal/agent/retriever.go` | Retriever interface + HybridRetriever | - ---- - -## Design Steps - -1. Define PromptConfig + all section data types in `prompt_config_types.go` -2. Write default template file with all section sub-templates -3. Implement TemplatePromptBuilder with variant loading -4. Define ToolCapability + ToolMetadata types -5. Add Metadata() to Tool interface with backward-compat wrapper -6. Define WorkspaceContext struct with context helpers -7. Update PolicyEngine: add capability rules + role parameter -8. Define ToolDefAdapter for provider-specific tool def generation -9. Implement HybridSkillSearcher with score fusion -10. Wire ACL into skill search path -11. Define Retriever interface + RetrievalConfig -12. Implement HybridRetriever for L0 auto-inject -13. Update memory_search tool with depth parameter design -14. Update kg_search tool with as_of temporal parameter design -15. Write provider variant templates (Codex, DashScope) - ---- - -## Todo List - -- [x] A1: PromptConfig + section data types — `internal/agent/prompt_config_types.go` -- [x] A2: Default template file — documented in phase design -- [x] A3: TemplatePromptBuilder interface — `internal/agent/prompt_config_types.go` -- [x] A4: Provider variant templates — documented in phase design -- [x] A5: Workspace sub-template — documented in phase design -- [x] B1: ToolCapability + ToolMetadata — `internal/tools/capability.go` -- [x] B2: Backward-compat wrapper — documented in phase design -- [x] B3: WorkspaceContext + context key — done in Phase 1 (`internal/workspace/`) -- [x] B4: PolicyEngine capability rules — documented in phase design -- [x] B5: ToolDefAdapter interface — documented in phase design -- [x] C1: HybridSkillSearcher — documented in phase design -- [x] C2: ACL wiring into search path — documented in phase design -- [x] C3: DepChecker.ResolveOrder() — documented in phase design -- [x] D1: Retriever + RetrievalConfig — `internal/agent/retriever.go` -- [x] D2: L0 auto-inject flow — documented in phase design -- [x] D3: memory_search depth param — documented in phase design -- [x] D4: kg_search as_of temporal param — documented in phase design - ---- - -## Success Criteria - -1. PromptConfig can reproduce exact v2 system prompt output (regression test) -2. All template sections independently toggleable + testable -3. WorkspaceContext carries all info needed by file tools (no separate context keys) -4. Every tool declares capabilities via Metadata() -5. PolicyEngine can express "deny mutating tools for viewer role" in one rule -6. HybridSkillSearcher returns better results than BM25-only on intent queries -7. L0 auto-inject adds <200 tokens per turn, only when relevant -8. Template renders in <1ms (no LLM calls) - ---- - -## Risk Assessment - -| Risk | Severity | Mitigation | -|------|----------|------------| -| Template engine too rigid for edge cases | MEDIUM | Keep ExtraPrompt escape hatch. Provider variants handle outliers. | -| Metadata() breaks all existing tool implementations | HIGH | Backward-compat wrapper: `defaultMetadata(tool)` infers from group/name. Phase implementation. | -| Hybrid skill search slower than BM25-only | LOW | BM25 <1ms, embedding <50ms. Parallel execution. Cache embeddings. | -| WorkspaceContext migration touches every file tool | HIGH | Single context key replaces 2. Grep for `ToolWorkspaceFromCtx` — 15 call sites. Mechanical refactor. | -| L0 auto-inject adds latency to every turn | MEDIUM | Memory index query <10ms. Skip if no memory (hasMemory flag). Threshold prevents injection for low-relevance messages. | -| Provider template variants diverge over time | LOW | Variants inherit from default, override only specific blocks. Test all variants in CI. | - ---- - -## Security Considerations - -- **Template injection:** PromptConfig data (PersonaContent, InstructionContent) comes from DB. Template uses `{{ }}` not `{{ | html }}`. Since output goes to LLM (not browser), HTML escaping not needed. But ensure no Go template directives in user content — use `{{ printf "%s" .Content }}` for raw pass-through. -- **Workspace enforcement:** WorkspaceSectionData.EnforcementMsg must be generated server-side, not from user input. Template renders it verbatim. -- **ACL bypass:** HybridSkillSearcher MUST filter AFTER search, not before (embedding search doesn't support WHERE clauses). Verify: `ListAccessible()` called on search results. -- **Tool capability lying:** Tools self-declare capabilities. PolicyEngine should cross-check: tools in "fs" group claiming "read-only" but actually writing. Add assertion in tests. diff --git a/plans/260406-1304-goclaw-v3-core-design/phase-05-orchestration-intelligence-design.md b/plans/260406-1304-goclaw-v3-core-design/phase-05-orchestration-intelligence-design.md deleted file mode 100644 index 90633482..00000000 --- a/plans/260406-1304-goclaw-v3-core-design/phase-05-orchestration-intelligence-design.md +++ /dev/null @@ -1,923 +0,0 @@ -# Phase 5: Orchestration & Intelligence Design - -## Context Links -- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (sections 11, 15) -- Architecture report: `plans/reports/Explore-260406-thorough-v3-architecture.md` (section 8) -- Agent link store: `internal/store/agent_link_store.go` -- Current subagent: `internal/tools/subagent.go`, `internal/tools/subagent_exec.go` -- Current team tools: `internal/tools/team_tool.go`, `internal/tools/team_tool_backend.go` -- Event bus: `internal/bus/bus.go`, `internal/bus/types.go` -- Phase 4: `phase-04-system-integration-design.md` (WorkspaceContext, ToolCapability, PolicyEngine) - -## Overview -- **Priority:** P1 (builds on Phase 4 integration layer) -- **Status:** Complete -- **Effort:** 6h design - -Designs 4 interconnected systems: -1. **Delegate tool** — lightweight inter-agent delegation via agent_links -2. **Orchestration mode config** — layered: spawn always, delegate base, team optional -3. **Self-evolution** — progressive 3-stage rollout (metrics -> suggestions -> auto-adapt) -4. **EventBus integration** — orchestration + evolution events - ---- - -## Key Insights - -1. Agent Links infrastructure exists (`agent_link_store.go`) with `CanDelegate()`, `GetLinkBetween()`, `DelegateTargets()`, even `SearchDelegateTargetsByEmbedding()` — but NO tool uses it. Full plumbing ready, needs ~100 LOC tool wrapper. -2. Current orchestration: spawn (self-clone, ~300 LOC) + team_tasks (9,400 LOC, 18 actions). No middle ground. Simple delegation requires team overhead or spawn limitations. -3. Self-evolution is greenfield. No metrics infrastructure exists. Design must be additive — no existing behavior changes in Stage 1. -4. EventBus (`bus.MessageBus`) is channel-oriented (Inbound/Outbound messages). Broadcast events are typed but payloads are `any`. V3 needs structured event types for orchestration lifecycle. - ---- - -## Requirements - -### Functional -- F1: `delegate` tool — agent delegates task to linked agent, async or sync -- F2: Delegate permission checked via existing `agentLinkStore.CanDelegate()` -- F3: Delegatee uses own workspace + optional shared area + read-only exports -- F4: Orchestration modes configurable per tenant (default) + per agent (override) -- F5: System prompt only shows tools for active orchestration mode -- F6: Self-evolution Stage 1: record metrics (retrieval quality, tool effectiveness, response feedback) -- F7: Self-evolution Stage 2: analyze metrics, generate suggestions, admin review UI -- F8: Self-evolution Stage 3: auto-adapt with guardrails + rollback -- F9: All orchestration lifecycle events published via EventBus - -### Non-Functional -- NF1: Delegate tool adds <5ms overhead over direct agent invocation -- NF2: Metrics recording non-blocking (goroutine, never delays response) -- NF3: Self-evolution stages independently toggleable (feature flags) -- NF4: Backward compatible: existing spawn + team_tasks unaffected - ---- - -## Architecture - -### A. Delegate Tool - -#### Tool Interface - -```go -// DelegateTool enables inter-agent task delegation via agent_links. -// Parameters: -// agent_key (required): target agent key (from known delegate targets) -// task (required): task description for the delegatee -// mode (optional): "async" (default) or "sync" -// context (optional): additional context string passed to delegatee -// files (optional): list of file paths to export (read-only for delegatee) - -type DelegateTool struct { - agentLinkStore store.AgentLinkStore - agentStore store.AgentStore - runner AgentRunner // interface to invoke agent loop - bus bus.EventPublisher -} - -func (t *DelegateTool) Name() string { return "delegate" } - -func (t *DelegateTool) Metadata() ToolMetadata { - return ToolMetadata{ - Name: "delegate", - Capabilities: []ToolCapability{CapAsync}, - Group: "orchestration", - } -} - -func (t *DelegateTool) Parameters() map[string]any { - return map[string]any{ - "type": "object", - "properties": map[string]any{ - "agent_key": map[string]any{ - "type": "string", - "description": "Key of the agent to delegate to (from your known delegate targets)", - }, - "task": map[string]any{ - "type": "string", - "description": "Task description for the delegatee", - }, - "mode": map[string]any{ - "type": "string", - "enum": []string{"async", "sync"}, - "default": "async", - "description": "async: fire-and-forget with result notification. sync: wait for completion.", - }, - "context": map[string]any{ - "type": "string", - "description": "Additional context passed to delegatee's system prompt", - }, - "files": map[string]any{ - "type": "array", - "items": map[string]any{"type": "string"}, - "description": "File paths (relative to your workspace) to share read-only with delegatee", - }, - }, - "required": []string{"agent_key", "task"}, - } -} -``` - -#### Execution Flow - -``` -delegate(agent_key="analyst", task="analyze Q1 data", mode="async") - | - v -1. Resolve target: agentStore.GetByKey(agent_key) - -> fail if agent not found or inactive - | - v -2. Permission check: agentLinkStore.CanDelegate(fromAgentID, toAgentID) - -> fail if no active link - | - v -3. Get link config: agentLinkStore.GetLinkBetween(from, to) - -> read Settings JSON for workspace config - | - v -4. Build WorkspaceContext for delegatee: - - ActivePath: delegatee's own workspace (resolved normally) - - SharedPath: /data/delegates/{linkID}/shared (created if not exists) - - ReadOnlyPaths: exported files from delegator (validated against workspace) - - Scope: "delegate" - | - v -5. Build delegation request: - - ExtraPrompt: "Delegated task from {source_agent_key}: {task}\n{context}" - - WorkspaceContext: as built above - - Mode: PromptMinimal (subagent-style, reduced sections) - | - v -6. Execute: - async mode: - - Spawn goroutine via scheduler lane - - Return immediately: "Task delegated to {agent_key}. You'll be notified on completion." - - On completion: bus.Broadcast(delegate.completed event) - - Result delivered via message bus (same as spawn announce) - sync mode: - - Block until delegatee completes (with timeout from link.Settings) - - Return delegatee's final response directly - | - v -7. Emit event: bus.Broadcast("delegate.sent", DelegateSentPayload{...}) -``` - -#### Workspace Model for Delegation - -```go -// DelegateWorkspaceConfig is stored in agent_link.settings JSON. -type DelegateWorkspaceConfig struct { - // SharedPath: shared directory both agents can read/write. - // Relative to data dir. Default: "delegates/{linkID}/shared" - SharedPath string `json:"shared_path,omitempty"` - - // SourceExports: glob patterns of delegator files shared read-only. - // Relative to delegator's workspace. E.g., ["docs/", "data/results/"] - SourceExports []string `json:"source_exports,omitempty"` - - // SyncTimeout: max wait time for sync mode (seconds). Default: 300 - SyncTimeout int `json:"sync_timeout,omitempty"` - - // MaxConcurrent: max concurrent delegations on this link. 0 = unlimited. - // Already exists as AgentLinkData.MaxConcurrent field. -} -``` - -No new DB tables. Config stored in existing `agent_links.settings` JSONB column. - -#### AgentRunner Interface - -```go -// AgentRunner abstracts invoking an agent's loop for delegation. -// Implemented by the gateway's run dispatcher. -type AgentRunner interface { - // RunDelegation executes a delegation task and returns the result. - // ctx carries workspace context, tenant scope, timeout. - RunDelegation(ctx context.Context, req DelegationRequest) (*DelegationResult, error) -} - -type DelegationRequest struct { - TargetAgentID uuid.UUID - SourceAgentID uuid.UUID - LinkID uuid.UUID - Task string - Context string - WorkspaceCtx *WorkspaceContext - UserID string // inherit from delegator's user context - TenantID uuid.UUID - SyncTimeout time.Duration // 0 = async -} - -type DelegationResult struct { - Response string // delegatee's final text response - ToolCalls int // number of tool calls made - Duration time.Duration - Error error -} -``` - ---- - -### B. Orchestration Mode Configuration - -#### Mode Definitions - -```go -// OrchestrationMode controls which inter-agent tools are available. -type OrchestrationMode string - -const ( - // ModeSpawn: self-clone only. spawn tool available. - // Simplest mode. Agent can only create copies of itself for background work. - ModeSpawn OrchestrationMode = "spawn" - - // ModeDelegate: agent links + spawn. delegate tool available. - // Agent can send tasks to linked agents. Base inter-agent mode. - ModeDelegate OrchestrationMode = "delegate" - - // ModeTeam: full team tasks + delegate + spawn. team_tasks tool available. - // Full lifecycle: review, approval, progress tracking. - // Only available when agent is member of an active team. - ModeTeam OrchestrationMode = "team" -) -``` - -#### Configuration Layering - -```go -// OrchestrationConfig controls orchestration behavior. -type OrchestrationConfig struct { - // Mode: active orchestration mode. - // Tenant sets default, agent overrides. - Mode OrchestrationMode `json:"orchestration_mode"` -} -``` - -**Storage:** No new DB columns. Uses existing config mechanisms: -- Tenant default: `tenant_settings.orchestration_mode` (existing JSONB settings column on tenants table) -- Agent override: `agents.settings -> 'orchestration_mode'` (existing JSONB settings column) - -**Resolution order:** -``` -1. Agent settings.orchestration_mode (if set) -2. Tenant settings.orchestration_mode (if set) -3. Default: "spawn" -``` - -#### Tool Visibility by Mode - -| Mode | spawn | delegate | team_tasks | -|------|-------|----------|------------| -| spawn | yes | no | no | -| delegate | yes | yes | no | -| team | yes | yes | yes | - -**Implementation:** PolicyEngine checks orchestration mode during FilterTools: - -```go -// In PolicyEngine.FilterTools(), after existing 7-step pipeline: -func (pe *PolicyEngine) applyOrchestrationMode( - allowed []string, - mode OrchestrationMode, - isTeamMember bool, -) []string { - switch mode { - case ModeSpawn: - allowed = subtractSet(allowed, []string{"delegate", "team_tasks"}) - case ModeDelegate: - allowed = subtractSet(allowed, []string{"team_tasks"}) - case ModeTeam: - if !isTeamMember { - // Downgrade to delegate if not in team - allowed = subtractSet(allowed, []string{"team_tasks"}) - } - } - return allowed -} -``` - -#### System Prompt Integration - -PromptConfig (from Phase 4) carries orchestration info: - -```go -// Added to PromptConfig: -type OrchestrationSectionData struct { - Mode OrchestrationMode - DelegateTargets []DelegateTargetEntry // from agentLinkStore.DelegateTargets() - TeamContext *TeamSectionData // only if ModeTeam + team member -} - -type DelegateTargetEntry struct { - AgentKey string - DisplayName string - Description string -} -``` - -Template renders delegate targets when mode is delegate/team: - -``` -{{- define "orchestration" -}} -{{- if eq .Mode "delegate" "team" }} -# Delegation -You can delegate tasks to these agents: -{{- range .DelegateTargets }} -- **{{ .AgentKey }}**: {{ .Description }} -{{- end }} -Use `delegate(agent_key, task)` to send work. -{{- end }} -{{- end -}} -``` - ---- - -### C. Self-Evolution Design (Progressive) - -#### Stage 1: Metrics Infrastructure (v3.0) - -##### DB Schema - -```sql --- New table: agent_evolution_metrics -CREATE TABLE agent_evolution_metrics ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - tenant_id UUID NOT NULL REFERENCES tenants(id), - agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE, - session_key TEXT NOT NULL, - - metric_type TEXT NOT NULL, -- 'retrieval', 'tool', 'feedback' - metric_key TEXT NOT NULL, -- specific metric (e.g., tool name, retrieval query) - value JSONB NOT NULL, -- metric-specific payload - - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - --- Indexes for efficient querying -CREATE INDEX idx_evo_metrics_agent_type ON agent_evolution_metrics(agent_id, metric_type); -CREATE INDEX idx_evo_metrics_created ON agent_evolution_metrics(created_at); -CREATE INDEX idx_evo_metrics_tenant ON agent_evolution_metrics(tenant_id); - --- Partition-ready: created_at is in schema for future range partitioning --- For now, TTL cleanup via cron: DELETE WHERE created_at < NOW() - INTERVAL '90 days' -``` - -##### Metric Types - -```go -// MetricType constants -const ( - MetricRetrieval MetricType = "retrieval" - MetricTool MetricType = "tool" - MetricFeedback MetricType = "feedback" -) - -// RetrievalMetricValue records per-retrieval quality. -type RetrievalMetricValue struct { - Query string `json:"query"` - ResultCount int `json:"result_count"` - TopScore float64 `json:"top_score"` - UsedInReply bool `json:"used_in_reply"` // heuristic: result text appears in LLM output - Source string `json:"source"` // "memory_search", "kg_search", "auto_inject" -} - -// ToolMetricValue records per-tool-call effectiveness. -type ToolMetricValue struct { - ToolName string `json:"tool_name"` - Success bool `json:"success"` // !result.IsError - Duration time.Duration `json:"duration_ms"` - ResultSize int `json:"result_size"` // bytes - Capability string `json:"capability"` // from ToolMetadata -} - -// FeedbackMetricValue records implicit satisfaction signal. -type FeedbackMetricValue struct { - Signal string `json:"signal"` // "correction", "followup", "thanks", "new_topic" - TurnGap int `json:"turn_gap"` // turns between response and signal - PreviousTool string `json:"previous_tool"` // last tool used before signal -} -``` - -##### Heuristic: "Was retrieval result used?" - -``` -After LLM response: -1. Collect all retrieval results from this turn (memory_search, auto-inject L0) -2. For each result, check if any 3+ word substring appears in LLM output -3. Mark used_in_reply = true if found -4. Record metric with used_in_reply flag -``` - -Simple substring check. Not perfect but sufficient for Stage 1 metrics. No LLM call needed. - -##### Implicit Satisfaction Signals - -``` -After user's next message: -1. If message is a correction ("no", "that's wrong", "I meant..."): signal = "correction" -2. If message continues topic (similar embedding): signal = "followup" -3. If message changes topic entirely: signal = "new_topic" -4. If message is appreciation ("thanks", "perfect"): signal = "thanks" -5. Record feedback metric with previous turn's tool context -``` - -Classification via keyword matching + embedding similarity. No LLM call. - -##### Metrics Store Interface - -```go -// EvolutionMetricsStore manages self-evolution metrics. -type EvolutionMetricsStore interface { - // RecordMetric stores a single metric (non-blocking, goroutine-safe). - RecordMetric(ctx context.Context, metric EvolutionMetric) error - - // QueryMetrics returns metrics for analysis. - QueryMetrics(ctx context.Context, agentID uuid.UUID, metricType MetricType, since time.Time, limit int) ([]EvolutionMetric, error) - - // AggregateToolMetrics returns per-tool success rate, avg duration, call count. - AggregateToolMetrics(ctx context.Context, agentID uuid.UUID, since time.Time) ([]ToolAggregate, error) - - // AggregateRetrievalMetrics returns per-source usage rate. - AggregateRetrievalMetrics(ctx context.Context, agentID uuid.UUID, since time.Time) ([]RetrievalAggregate, error) - - // Cleanup removes metrics older than TTL. - Cleanup(ctx context.Context, olderThan time.Time) (int64, error) -} - -type EvolutionMetric struct { - ID uuid.UUID - TenantID uuid.UUID - AgentID uuid.UUID - SessionKey string - MetricType MetricType - MetricKey string - Value json.RawMessage // serialized *MetricValue - CreatedAt time.Time -} - -type ToolAggregate struct { - ToolName string - CallCount int - SuccessRate float64 - AvgDuration time.Duration -} - -type RetrievalAggregate struct { - Source string // "memory_search", "kg_search", "auto_inject" - QueryCount int - UsageRate float64 // fraction of results used in reply - AvgScore float64 -} -``` - -##### Pipeline Integration - -Metrics recorded in ObserveStage (Phase 2 pipeline): - -```go -// In ObserveStage.Execute(): -// After tool result processing: -if toolResult != nil { - go metricsStore.RecordMetric(ctx, EvolutionMetric{ - MetricType: MetricTool, - MetricKey: toolResult.ToolName, - Value: marshalToolMetric(toolResult), - }) -} - -// After retrieval in ContextStage: -if len(l0Results) > 0 { - go metricsStore.RecordMetric(ctx, EvolutionMetric{ - MetricType: MetricRetrieval, - MetricKey: "auto_inject", - Value: marshalRetrievalMetric(query, l0Results), - }) -} -``` - -Always non-blocking (`go` goroutine). Metrics are best-effort — dropped on error, never block agent loop. - -#### Stage 2: Suggestion Engine (v3.1) - -##### DB Schema - -```sql --- New table: agent_evolution_suggestions -CREATE TABLE agent_evolution_suggestions ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - tenant_id UUID NOT NULL REFERENCES tenants(id), - agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE, - - suggestion_type TEXT NOT NULL, -- 'threshold', 'tool_order', 'skill_add', 'memory_prune' - suggestion TEXT NOT NULL, -- human-readable suggestion - rationale TEXT NOT NULL, -- data-driven reasoning - parameters JSONB, -- machine-readable params for auto-apply - - status TEXT NOT NULL DEFAULT 'pending', -- 'pending', 'approved', 'rejected', 'applied', 'rolled_back' - reviewed_by TEXT, -- admin user who reviewed - reviewed_at TIMESTAMPTZ, - - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - -CREATE INDEX idx_evo_suggestions_agent ON agent_evolution_suggestions(agent_id, status); -CREATE INDEX idx_evo_suggestions_tenant ON agent_evolution_suggestions(tenant_id); -``` - -##### Suggestion Types - -```go -type SuggestionType string - -const ( - SuggestThreshold SuggestionType = "threshold" // adjust retrieval threshold - SuggestToolOrder SuggestionType = "tool_order" // reorder tool list for this agent - SuggestSkillAdd SuggestionType = "skill_add" // recommend adding a skill - SuggestMemoryPrune SuggestionType = "memory_prune" // prune low-value memory -) - -// SuggestionParams is the machine-readable payload for auto-apply. -// Each SuggestionType has its own params structure. -type ThresholdParams struct { - Param string `json:"param"` // "retrieval_threshold", "l0_max_tokens" - OldValue float64 `json:"old_value"` - NewValue float64 `json:"new_value"` -} - -type ToolOrderParams struct { - ToolName string `json:"tool_name"` - Action string `json:"action"` // "promote", "demote", "suggest_removal" - Reasoning string `json:"reasoning"` -} -``` - -##### Analysis Rules (Examples) - -``` -Rule 1: Low retrieval usage - IF retrieval.usage_rate < 0.2 over 7 days AND query_count > 50 - THEN suggest: "Raise relevance_threshold from {current} to {current+0.1}" - REASON: "Only 20% of auto-injected memory results appear in responses" - -Rule 2: Tool never succeeds - IF tool.success_rate < 0.1 over 7 days AND call_count > 20 - THEN suggest: "Consider removing {tool_name} or fixing configuration" - REASON: "{tool_name} failed 90% of {call_count} calls" - -Rule 3: Repeated tool usage - IF tool.call_count > 100/week for specific tool AND no skill exists - THEN suggest: "Add skill for {common_task_pattern}" - REASON: "{tool_name} called {count} times this week, mostly for {pattern}" - -Rule 4: Memory bloat - IF episodic_count > 500 AND avg_usage_rate < 0.05 - THEN suggest: "Prune episodic memories older than 60 days" - REASON: "95% of older episodic summaries never retrieved" -``` - -##### Suggestion Store Interface - -```go -type EvolutionSuggestionStore interface { - CreateSuggestion(ctx context.Context, s EvolutionSuggestion) error - ListSuggestions(ctx context.Context, agentID uuid.UUID, status string, limit int) ([]EvolutionSuggestion, error) - UpdateSuggestionStatus(ctx context.Context, id uuid.UUID, status string, reviewedBy string) error - GetSuggestion(ctx context.Context, id uuid.UUID) (*EvolutionSuggestion, error) -} - -type EvolutionSuggestion struct { - ID uuid.UUID - TenantID uuid.UUID - AgentID uuid.UUID - SuggestionType SuggestionType - Suggestion string - Rationale string - Parameters json.RawMessage - Status string - ReviewedBy string - ReviewedAt *time.Time - CreatedAt time.Time -} -``` - -##### Suggestion Engine Interface - -```go -// SuggestionEngine analyzes metrics and generates suggestions. -// Runs as periodic cron job (configurable interval, default: daily). -type SuggestionEngine interface { - // Analyze runs all rules for an agent and creates pending suggestions. - Analyze(ctx context.Context, agentID uuid.UUID) ([]EvolutionSuggestion, error) - - // AnalyzeAll runs analysis for all agents in tenant. - AnalyzeAll(ctx context.Context, tenantID uuid.UUID) error -} - -// RuleSuggestionEngine implements SuggestionEngine with configurable rules. -type RuleSuggestionEngine struct { - metricsStore EvolutionMetricsStore - suggestionStore EvolutionSuggestionStore - rules []AnalysisRule -} - -// AnalysisRule is a single suggestion rule. -type AnalysisRule interface { - // Evaluate checks if the rule fires for the given agent metrics. - // Returns nil if rule doesn't apply. - Evaluate(ctx context.Context, agentID uuid.UUID, metrics AnalysisInput) (*EvolutionSuggestion, error) -} - -type AnalysisInput struct { - ToolAggregates []ToolAggregate - RetrievalAggregates []RetrievalAggregate - FeedbackSummary FeedbackSummary - AnalysisPeriod time.Duration // lookback window (default 7 days) -} -``` - -#### Stage 3: Auto-Adaptation (v3.2+) - -##### Guardrails - -```go -// AdaptationGuardrails control auto-adaptation limits. -type AdaptationGuardrails struct { - // MaxDeltaPerCycle: max parameter change per adaptation cycle. - // E.g., threshold can only change by +/- 0.1 per cycle. - MaxDeltaPerCycle float64 `json:"max_delta_per_cycle"` - - // MinDataPoints: minimum metrics before adaptation allowed. - MinDataPoints int `json:"min_data_points"` - - // RollbackOnDrop: if quality metric drops by this %, rollback. - RollbackOnDrop float64 `json:"rollback_on_drop_pct"` // e.g., 20.0 = 20% - - // AdminOverride: admin can lock parameters (prevent auto-change). - LockedParams []string `json:"locked_params,omitempty"` -} - -// Default guardrails -var DefaultGuardrails = AdaptationGuardrails{ - MaxDeltaPerCycle: 0.1, - MinDataPoints: 100, - RollbackOnDrop: 20.0, -} -``` - -##### Adaptation Flow - -``` -1. SuggestionEngine.Analyze() produces suggestion with status="pending" -2. If auto_adapt enabled for this agent: - a. Check guardrails (min data points, max delta, locked params) - b. If passes: apply parameters, set status="applied" - c. Record baseline quality metric -3. After 7 days: compare quality metric to baseline - a. If quality dropped > RollbackOnDrop%: rollback, set status="rolled_back" - b. If quality stable/improved: keep, emit event -4. Admin can always: approve pending, reject, rollback applied, lock params -``` - -##### Auto-Adapt Store (Extension of Suggestion Store) - -```go -// Added to EvolutionSuggestionStore for Stage 3: -type EvolutionAdaptStore interface { - EvolutionSuggestionStore - - // ApplySuggestion applies a suggestion's parameters to agent config. - ApplySuggestion(ctx context.Context, suggestionID uuid.UUID) error - - // RollbackSuggestion reverts an applied suggestion. - RollbackSuggestion(ctx context.Context, suggestionID uuid.UUID) error - - // GetAdaptationHistory returns applied/rolled-back suggestions for audit. - GetAdaptationHistory(ctx context.Context, agentID uuid.UUID, limit int) ([]EvolutionSuggestion, error) -} -``` - -##### Feature Flags - -```go -// Feature flags in agent settings JSON (agents.settings column) -type SelfEvolutionFlags struct { - MetricsEnabled bool `json:"self_evolution_metrics"` // Stage 1 (default: true for v3) - SuggestionsEnabled bool `json:"self_evolution_suggestions"` // Stage 2 (default: false) - AutoAdaptEnabled bool `json:"self_evolution_auto_adapt"` // Stage 3 (default: false) -} -``` - -No new DB columns. Flags stored in existing `agents.settings` JSONB. - ---- - -### D. EventBus Integration - -#### New Event Topics - -```go -// Orchestration events -const ( - TopicDelegateSent = "delegate:sent" - TopicDelegateCompleted = "delegate:completed" - TopicDelegateFailed = "delegate:failed" -) - -// Self-evolution events -const ( - TopicMetricRecorded = "evolution:metric" - TopicSuggestionCreated = "evolution:suggestion" - TopicAdaptationApplied = "evolution:adapted" - TopicAdaptationRolledBack = "evolution:rollback" -) -``` - -#### Typed Event Payloads - -```go -// DelegateSentPayload emitted when delegate tool fires. -type DelegateSentPayload struct { - SourceAgentID uuid.UUID `json:"source_agent_id"` - TargetAgentID uuid.UUID `json:"target_agent_id"` - LinkID uuid.UUID `json:"link_id"` - Task string `json:"task"` - Mode string `json:"mode"` // "async" or "sync" - SessionKey string `json:"session_key"` -} - -// DelegateCompletedPayload emitted when delegatee finishes. -type DelegateCompletedPayload struct { - SourceAgentID uuid.UUID `json:"source_agent_id"` - TargetAgentID uuid.UUID `json:"target_agent_id"` - LinkID uuid.UUID `json:"link_id"` - Duration time.Duration `json:"duration_ms"` - ToolCalls int `json:"tool_calls"` - Success bool `json:"success"` - Error string `json:"error,omitempty"` -} - -// DelegateFailedPayload emitted on delegation failure. -type DelegateFailedPayload struct { - SourceAgentID uuid.UUID `json:"source_agent_id"` - TargetAgentID uuid.UUID `json:"target_agent_id"` - LinkID uuid.UUID `json:"link_id"` - Reason string `json:"reason"` // "permission_denied", "agent_inactive", "timeout" -} - -// MetricRecordedPayload emitted on each metric write. -type MetricRecordedPayload struct { - AgentID uuid.UUID `json:"agent_id"` - MetricType MetricType `json:"metric_type"` - MetricKey string `json:"metric_key"` -} - -// SuggestionCreatedPayload emitted when suggestion engine generates one. -type SuggestionCreatedPayload struct { - AgentID uuid.UUID `json:"agent_id"` - SuggestionID uuid.UUID `json:"suggestion_id"` - SuggestionType SuggestionType `json:"suggestion_type"` - Summary string `json:"summary"` -} -``` - -#### Event Subscribers - -```go -// Registered at gateway startup: - -// 1. Delegation result delivery — notifies source agent when async delegation completes -bus.Subscribe("delegate-result-delivery", func(e bus.Event) { - if e.Name == TopicDelegateCompleted { - // Deliver result to source agent's session via sessions_send mechanism - } -}) - -// 2. Metrics recording — non-blocking metric persistence -bus.Subscribe("evolution-metrics-recorder", func(e bus.Event) { - if e.Name == TopicMetricRecorded { - // Already persisted by goroutine in ObserveStage. - // Subscriber only for UI real-time updates (WebSocket broadcast). - } -}) - -// 3. Suggestion notification — notify admin of new suggestions -bus.Subscribe("evolution-suggestion-notify", func(e bus.Event) { - if e.Name == TopicSuggestionCreated { - // Broadcast to WebSocket for admin UI notification - } -}) -``` - ---- - -## Related Code Files - -### Files to Modify -| File | Change | -|------|--------| -| `internal/bus/types.go` | Add orchestration + evolution topic constants + payload types | -| `internal/tools/policy.go` | Add applyOrchestrationMode() to FilterTools | -| `internal/agent/loop.go` | Wire metrics recording in observe phase | -| `internal/agent/systemprompt.go` | Add orchestration section to prompt config | -| `internal/store/agent_link_store.go` | (no change — already has full interface) | -| `cmd/gateway.go` | Register delegate tool, event subscribers, metrics cron | - -### Files to Create -| File | Purpose | -|------|---------| -| `internal/tools/delegate_tool.go` | DelegateTool implementation | -| `internal/tools/delegate_workspace.go` | DelegateWorkspaceConfig, workspace setup | -| `internal/agent/orchestration_mode.go` | OrchestrationMode types, resolution logic | -| `internal/agent/agent_runner.go` | AgentRunner interface for delegation dispatch | -| `internal/store/evolution_store.go` | EvolutionMetricsStore + EvolutionSuggestionStore interfaces | -| `internal/store/pg/evolution_metrics.go` | PG implementation of EvolutionMetricsStore | -| `internal/store/pg/evolution_suggestions.go` | PG implementation of EvolutionSuggestionStore | -| `internal/agent/evolution_metrics.go` | Metric types + recording helpers | -| `internal/agent/evolution_suggestion_engine.go` | SuggestionEngine + AnalysisRule implementations | -| `internal/agent/evolution_guardrails.go` | AdaptationGuardrails + auto-adapt logic | - ---- - -## Design Steps - -1. Define DelegateTool interface + parameters schema -2. Design delegate execution flow (async + sync modes) -3. Define DelegateWorkspaceConfig in agent_link.settings -4. Define AgentRunner interface for delegation dispatch -5. Define OrchestrationMode enum + resolution logic (tenant -> agent) -6. Design tool visibility filtering by orchestration mode -7. Design orchestration section for system prompt template -8. Define agent_evolution_metrics schema + indexes -9. Define metric types (retrieval, tool, feedback) with value structs -10. Define EvolutionMetricsStore interface + aggregate methods -11. Design metrics recording integration in ObserveStage -12. Define agent_evolution_suggestions schema -13. Define SuggestionEngine + AnalysisRule interfaces -14. Design analysis rules (4 initial rules) -15. Define AdaptationGuardrails + auto-adapt flow -16. Define all EventBus topics + typed payloads -17. Design event subscriber wiring at startup - ---- - -## Todo List - -- [ ] A1: DelegateTool interface + parameters -- [ ] A2: Delegate execution flow (async/sync) -- [ ] A3: DelegateWorkspaceConfig schema -- [ ] A4: AgentRunner interface -- [ ] A5: Permission flow (CanDelegate + GetLinkBetween) -- [ ] B1: OrchestrationMode enum + constants -- [ ] B2: Mode resolution (tenant default + agent override) -- [ ] B3: Tool visibility by mode in PolicyEngine -- [ ] B4: Orchestration section in prompt template -- [ ] C1: agent_evolution_metrics DDL + indexes -- [ ] C2: Metric value types (retrieval, tool, feedback) -- [ ] C3: EvolutionMetricsStore interface -- [ ] C4: Retrieval "used in reply" heuristic -- [ ] C5: Implicit satisfaction signal classification -- [ ] C6: agent_evolution_suggestions DDL -- [ ] C7: SuggestionEngine + AnalysisRule interfaces -- [ ] C8: Initial 4 analysis rules -- [ ] C9: AdaptationGuardrails + auto-adapt flow -- [ ] C10: Feature flags in agent settings -- [ ] D1: EventBus topic constants -- [ ] D2: Typed event payloads (delegate, evolution) -- [ ] D3: Event subscriber design (result delivery, notifications) - ---- - -## Success Criteria - -1. DelegateTool ~100 LOC (excluding workspace setup), uses existing agentLinkStore.CanDelegate() -2. Orchestration mode resolvable from tenant + agent settings with no new DB columns -3. PolicyEngine correctly filters tools by mode (spawn/delegate/team) -4. agent_evolution_metrics table handles 10K+ rows/day without index degradation -5. Metrics recording never blocks agent loop (goroutine + best-effort) -6. SuggestionEngine rules are additive (new rule = implement AnalysisRule interface) -7. All orchestration events typed (no `map[string]any` payloads) -8. Feature flags independently toggleable per agent - ---- - -## Risk Assessment - -| Risk | Severity | Mitigation | -|------|----------|------------| -| Delegate tool workspace isolation leaks | HIGH | ReadOnlyPaths enforced by same resolvePath() as all file tools. SharedPath created with restricted permissions. | -| Sync delegation timeout blocks caller | MEDIUM | Configurable timeout per link (default 300s). Timeout returns error, does not kill delegatee. | -| Metrics table grows unbounded | MEDIUM | TTL cleanup cron (default 90 days). Partitioning-ready schema (created_at). Monitor table size. | -| Suggestion engine generates noise | LOW | Minimum data points threshold (100). Suggestions require admin review (Stage 2). Auto-adapt has guardrails (Stage 3). | -| Orchestration mode config not discoverable | LOW | Admin UI shows current mode per agent. CLI: goclaw agent info shows mode. | -| Self-evolution "learns wrong" | HIGH | Progressive rollout. Stage 1 = metrics only (no behavior change). Stage 2 = human review. Stage 3 = guardrails + rollback. | -| EventBus topic explosion | LOW | Strict naming convention. Only add topics with subscribers. No orphan topics. | - ---- - -## Security Considerations - -- **Delegation permission:** `CanDelegate()` is the sole gate. No bypass via direct tool call — delegate tool always calls CanDelegate(). Link status "disabled" blocks delegation. -- **Cross-tenant delegation:** Agent links are tenant-scoped (both agents must be in same tenant). `agentLinkStore` queries filter by tenant_id from context. -- **Delegate workspace isolation:** Delegatee cannot access delegator's full workspace. Only explicitly exported paths (SourceExports) are shared read-only. SharedPath is a new directory, not inside either workspace. -- **Metrics data sensitivity:** Metrics contain query text and tool names, not user data or PII. Scoped by tenant_id. Cleanup cron prevents accumulation. -- **Auto-adapt scope:** Stage 3 only modifies retrieval parameters (thresholds, weights). Cannot modify tool access, workspace rules, or security settings. Admin can lock any parameter. diff --git a/plans/260406-1304-goclaw-v3-core-design/phase-06-migration-audit.md b/plans/260406-1304-goclaw-v3-core-design/phase-06-migration-audit.md deleted file mode 100644 index 582ffe40..00000000 --- a/plans/260406-1304-goclaw-v3-core-design/phase-06-migration-audit.md +++ /dev/null @@ -1,683 +0,0 @@ -# Phase 6: Migration & Audit - -## Context Links -- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (sections 16, 17) -- Architecture report: `plans/reports/Explore-260406-thorough-v3-architecture.md` (all sections) -- Workspace forcing: `plans/reports/Explore-260406-workspace-forcing.md` -- Phase 1-5 designs: all prior phase files in this plan -- Current migrations: `migrations/000001_init_schema.up.sql` through `migrations/000036_secure_cli_agent_grants.up.sql` -- Schema version: `internal/upgrade/version.go` (RequiredSchemaVersion = 36) - -## Overview -- **Priority:** P1 (blocks implementation start) -- **Status:** Complete -- **Effort:** 4h design - -Final design phase — comprehensive migration plan, affected file inventory, test matrix, and audit checklist. Ensures no implementation begins without full impact analysis. - ---- - -## Key Insights - -1. Current RequiredSchemaVersion = 36 (migration 000036). V3 adds 3 new tables + 2 ALTER TABLE. Next migration = 000037. -2. KG tables (000013) lack temporal columns. Adding valid_from/valid_until via ALTER is non-breaking — NULL defaults work. -3. Memory system is hybrid: daily .md files in filesystem + memory_documents/memory_chunks in PG. V3 must ingest filesystem files into new episodic_summaries table. -4. Agent settings already uses JSONB — all new per-agent configs (retrieval thresholds, orchestration mode, self-evolution flags, dedup thresholds) fit without schema changes. -5. No SQLite changes in v3 — design is PG-first per brainstorm decision #17. SQLite compat deferred. - ---- - -## Requirements - -### Functional -- F1: New tables created via standard golang-migrate files -- F2: ALTER TABLE for KG temporal columns with safe defaults -- F3: Lazy data migration: v2 data auto-converted on first access -- F4: Feature flags per agent for gradual v3 rollout -- F5: Full affected file inventory with dependency order -- F6: Integration test plan covering all 6 workspace scenarios -- F7: Rollback path for every schema change - -### Non-Functional -- NF1: Migration runs <30s on databases with 100K entities -- NF2: No downtime required — all changes additive (no DROP/RENAME) -- NF3: v2 agents continue working unchanged until explicitly migrated -- NF4: Test suite runnable in CI (<5 min) - ---- - -## Architecture - -### A. DB Schema Migration - -#### Migration 000037: v3 memory + evolution tables - -```sql --- 000037_v3_memory_evolution.up.sql - --- Episodic summaries (Tier 2 memory) -CREATE TABLE episodic_summaries ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - tenant_id UUID NOT NULL REFERENCES tenants(id), - agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE, - user_id VARCHAR(255) NOT NULL DEFAULT '', - session_key TEXT NOT NULL, - - summary TEXT NOT NULL, -- LLM-generated session summary - key_topics TEXT[] DEFAULT '{}', -- extracted topic keywords - embedding vector(1536), -- for similarity search - source_type TEXT NOT NULL DEFAULT 'session', -- 'session', 'v2_daily', 'manual' - source_id TEXT, -- session key or v2 file path (dedup guard) - turn_count INT NOT NULL DEFAULT 0, -- number of turns summarized - token_count INT NOT NULL DEFAULT 0, -- estimated tokens in original - - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - expires_at TIMESTAMPTZ -- NULL = no expiry. TTL configurable per agent. -); - -CREATE INDEX idx_episodic_agent_user ON episodic_summaries(agent_id, user_id); -CREATE INDEX idx_episodic_tenant ON episodic_summaries(tenant_id); -CREATE INDEX idx_episodic_source ON episodic_summaries(agent_id, source_id); -CREATE INDEX idx_episodic_tsv ON episodic_summaries USING GIN(to_tsvector('simple', summary)); -CREATE INDEX idx_episodic_vec ON episodic_summaries USING hnsw(embedding vector_cosine_ops); -CREATE INDEX idx_episodic_expires ON episodic_summaries(expires_at) WHERE expires_at IS NOT NULL; - --- Evolution metrics (Stage 1 self-evolution) -CREATE TABLE agent_evolution_metrics ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - tenant_id UUID NOT NULL REFERENCES tenants(id), - agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE, - session_key TEXT NOT NULL, - - metric_type TEXT NOT NULL, -- 'retrieval', 'tool', 'feedback' - metric_key TEXT NOT NULL, -- tool name, retrieval source, signal type - value JSONB NOT NULL, -- metric-specific payload - - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - -CREATE INDEX idx_evo_metrics_agent_type ON agent_evolution_metrics(agent_id, metric_type); -CREATE INDEX idx_evo_metrics_created ON agent_evolution_metrics(created_at); -CREATE INDEX idx_evo_metrics_tenant ON agent_evolution_metrics(tenant_id); - --- Evolution suggestions (Stage 2 self-evolution) -CREATE TABLE agent_evolution_suggestions ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - tenant_id UUID NOT NULL REFERENCES tenants(id), - agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE, - - suggestion_type TEXT NOT NULL, -- 'threshold', 'tool_order', 'skill_add', 'memory_prune' - suggestion TEXT NOT NULL, - rationale TEXT NOT NULL, - parameters JSONB, -- machine-readable for auto-apply - - status TEXT NOT NULL DEFAULT 'pending', -- 'pending','approved','rejected','applied','rolled_back' - reviewed_by TEXT, - reviewed_at TIMESTAMPTZ, - - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - -CREATE INDEX idx_evo_suggestions_agent ON agent_evolution_suggestions(agent_id, status); -CREATE INDEX idx_evo_suggestions_tenant ON agent_evolution_suggestions(tenant_id); -``` - -**Down migration:** -```sql --- 000037_v3_memory_evolution.down.sql -DROP TABLE IF EXISTS agent_evolution_suggestions; -DROP TABLE IF EXISTS agent_evolution_metrics; -DROP TABLE IF EXISTS episodic_summaries; -``` - -#### Migration 000038: KG temporal columns - -```sql --- 000038_kg_temporal_columns.up.sql - --- Add temporal validity windows to KG entities and relations -ALTER TABLE kg_entities ADD COLUMN IF NOT EXISTS valid_from TIMESTAMPTZ DEFAULT NOW(); -ALTER TABLE kg_entities ADD COLUMN IF NOT EXISTS valid_until TIMESTAMPTZ; -- NULL = current/valid - -ALTER TABLE kg_relations ADD COLUMN IF NOT EXISTS valid_from TIMESTAMPTZ DEFAULT NOW(); -ALTER TABLE kg_relations ADD COLUMN IF NOT EXISTS valid_until TIMESTAMPTZ; - --- Index for temporal queries: "current facts" (most common query) -CREATE INDEX idx_kg_entities_current ON kg_entities(agent_id, user_id) - WHERE valid_until IS NULL; - --- Index for historical queries: "facts valid at time T" -CREATE INDEX idx_kg_entities_temporal ON kg_entities(agent_id, user_id, valid_from, valid_until); - -CREATE INDEX idx_kg_relations_current ON kg_relations(agent_id, user_id) - WHERE valid_until IS NULL; - -CREATE INDEX idx_kg_relations_temporal ON kg_relations(agent_id, user_id, valid_from, valid_until); -``` - -**Down migration:** -```sql --- 000038_kg_temporal_columns.down.sql -DROP INDEX IF EXISTS idx_kg_relations_temporal; -DROP INDEX IF EXISTS idx_kg_relations_current; -DROP INDEX IF EXISTS idx_kg_entities_temporal; -DROP INDEX IF EXISTS idx_kg_entities_current; - -ALTER TABLE kg_relations DROP COLUMN IF EXISTS valid_until; -ALTER TABLE kg_relations DROP COLUMN IF EXISTS valid_from; -ALTER TABLE kg_entities DROP COLUMN IF EXISTS valid_until; -ALTER TABLE kg_entities DROP COLUMN IF EXISTS valid_from; -``` - -#### RequiredSchemaVersion Bump - -```go -// internal/upgrade/version.go -const RequiredSchemaVersion uint = 38 -``` - ---- - -### B. Data Migration Strategy (Dual-Mode) - -#### Principle: Lazy Migration on First Access - -V3 reads v2 format on first access, auto-migrates in background. No bulk migration script. No downtime. - -#### V2 Memory Files -> Episodic Summaries - -``` -Trigger: Agent with v3_memory_enabled=true, first session start -Flow: -1. Check if agent has unmigrated v2 daily files: - - List files matching memory/YYYY-MM-DD.md in workspace - - Check episodic_summaries for source_id matching file path - - If no matching episodic record: file is unmigrated -2. For each unmigrated file: - a. Read file content - b. Insert episodic_summaries row: - - summary: file content (already summarized by v2 flush) - - source_type: "v2_daily" - - source_id: file path (dedup guard) - - key_topics: extract from content (simple keyword extraction) - - turn_count: 0 (unknown for v2) - - token_count: len(content) / 4 (estimate) - c. Generate embedding in background (non-blocking) -3. Mark migration complete in agent settings: v2_memory_migrated=true -``` - -#### V2 KG Entities -> Temporal - -``` -Trigger: Migration 000038 runs (ALTER TABLE with defaults) -Flow: -1. ALTER TABLE adds valid_from DEFAULT NOW() — new rows get current timestamp -2. Existing rows: valid_from = NOW() at migration time (not created_at) -3. Optional backfill (non-blocking, cron job): - UPDATE kg_entities SET valid_from = created_at WHERE valid_from > created_at + INTERVAL '1 second'; - UPDATE kg_relations SET valid_from = created_at WHERE valid_from > created_at + INTERVAL '1 second'; -4. valid_until remains NULL for all existing entities (all considered "current") -``` - -> **Decision:** Default valid_from to NOW() at migration time, not created_at. Backfill is optional — historical accuracy is nice-to-have, not required. Existing entities are "currently valid" regardless of when they were created. - -#### V2 Sessions -> V3 Sessions - -No migration needed. V3 pipeline operates on same session data. Sessions use same `sessions` table, same `SessionData` struct. V3 adds pipeline stages that wrap existing behavior. - -#### Feature Flags - -All stored in existing `agents.settings` JSONB column: - -```json -{ - "v3_pipeline_enabled": false, - "v3_memory_enabled": false, - "v3_retrieval_enabled": false, - "self_evolution_metrics": false, - "self_evolution_suggestions": false, - "self_evolution_auto_adapt": false, - "v2_memory_migrated": false, - "retrieval_threshold": 0.3, - "dedup_auto_merge_threshold": 0.98, - "dedup_flag_threshold": 0.90, - "orchestration_mode": "spawn" -} -``` - -**Rollout plan:** -1. Deploy v3 binary with all flags false (default) — no behavior change -2. Enable v3_memory_enabled per agent — triggers lazy v2 file migration -3. Enable v3_pipeline_enabled per agent — switches to Stage-based pipeline -4. Enable v3_retrieval_enabled per agent — activates L0 auto-inject -5. Enable self_evolution_metrics per agent — starts recording -6. Enable at tenant level for bulk rollout - ---- - -### C. Code Migration Plan — Affected Files Inventory - -#### Module 1: Foundation (Phase 1) -| File | Change Type | Dependency | -|------|-------------|------------| -| `internal/agent/loop_types.go` | MODIFY: split RunState into substates | None | -| `internal/agent/loop_compact.go` | MODIFY: replace chars/4 with TokenCounter | TokenCounter | -| `internal/agent/pruning.go` | MODIFY: use TokenCounter for thresholds | TokenCounter | -| `internal/tools/context_keys.go` | MODIFY: replace dual workspace keys with WorkspaceContext | None | -| `internal/tools/workspace_resolver.go` | MODIFY: return WorkspaceContext instead of string | None | -| `internal/tools/workspace_dir.go` | MODIFY: use WorkspaceContext | WorkspaceContext | -| `internal/bus/types.go` | MODIFY: add new event topics + typed payloads | None | -| `internal/bus/bus.go` | No change (existing Broadcast mechanism sufficient) | None | -| `internal/providers/types.go` | MODIFY: add ProviderCapabilities struct | None | -| `internal/providers/anthropic.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter | -| `internal/providers/openai.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter | -| `internal/providers/dashscope.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter | -| `internal/providers/codex.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter | -| `internal/providers/claude_cli.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter | -| `internal/providers/acp_provider.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter | - -**New files:** -| File | Purpose | -|------|---------| -| `internal/agent/token_counter.go` | TokenCounter interface + tiktoken impl | -| `internal/tools/workspace_context.go` | WorkspaceContext struct + helpers | -| `internal/providers/adapter.go` | ProviderAdapter interface | -| `internal/providers/capabilities.go` | ProviderCapabilities struct | - -#### Module 2: Pipeline (Phase 2) -| File | Change Type | Dependency | -|------|-------------|------------| -| `internal/agent/loop.go` | MAJOR: refactor into pipeline stages | Phase 1 foundation | -| `internal/agent/loop_run.go` | MODIFY: pipeline entry point | Pipeline stages | -| `internal/agent/loop_context.go` | MODIFY: becomes ContextStage | WorkspaceContext | -| `internal/agent/loop_history.go` | MODIFY: becomes part of ContextStage | PromptBuilder | -| `internal/agent/loop_tools.go` | MODIFY: becomes ToolStage | ToolRegistry | -| `internal/agent/toolloop.go` | MODIFY: integrated into ToolStage | ToolStage | -| `internal/agent/loop_compact.go` | MODIFY: becomes PruneStage | TokenCounter | -| `internal/agent/loop_finalize.go` | MODIFY: becomes FinalizeStage | EventBus | -| `internal/agent/loop_tracing.go` | MODIFY: per-stage tracing | Pipeline stages | - -**New files:** -| File | Purpose | -|------|---------| -| `internal/agent/pipeline.go` | Pipeline, Stage interface, runner | -| `internal/agent/stage_context.go` | ContextStage | -| `internal/agent/stage_think.go` | ThinkStage (LLM call) | -| `internal/agent/stage_prune.go` | PruneStage (token counting + pruning) | -| `internal/agent/stage_tool.go` | ToolStage (tool execution) | -| `internal/agent/stage_observe.go` | ObserveStage (loop detection, metrics) | -| `internal/agent/stage_checkpoint.go` | CheckpointStage (session save) | - -#### Module 3: Memory & KG (Phase 3) -| File | Change Type | Dependency | -|------|-------------|------------| -| `internal/memory/embeddings.go` | MODIFY: integrate with episodic store | EpisodicStore | -| `internal/knowledgegraph/extractor.go` | MODIFY: async extraction via events | EventBus | -| `internal/knowledgegraph/extractor_prompt.go` | No change | | -| `internal/store/memory_store.go` | MODIFY: add EpisodicSummaryStore interface | None | -| `internal/store/knowledge_graph_store.go` | MODIFY: add temporal query methods | None | -| `internal/store/pg/memory_docs.go` | No change (v2 compat) | | -| `internal/store/pg/memory_search.go` | MODIFY: integrate episodic search | EpisodicStore | -| `internal/store/pg/knowledge_graph.go` | MODIFY: temporal WHERE clauses | KG temporal cols | -| `internal/store/pg/knowledge_graph_dedup.go` | MODIFY: configurable thresholds | Agent settings | -| `internal/store/pg/knowledge_graph_relations.go` | MODIFY: temporal WHERE clauses | KG temporal cols | -| `internal/store/pg/knowledge_graph_traversal.go` | MODIFY: temporal filter in traversal | KG temporal cols | -| `internal/store/pg/knowledge_graph_embedding.go` | No change | | -| `internal/agent/memoryflush.go` | MODIFY: v3 path uses event-driven consolidation | EventBus | -| `internal/agent/extractive_memory.go` | No change (fallback for v2 mode) | | - -**New files:** -| File | Purpose | -|------|---------| -| `internal/store/episodic_store.go` | EpisodicSummaryStore interface | -| `internal/store/pg/episodic_summaries.go` | PG implementation | -| `internal/agent/consolidation_worker.go` | Event-driven session -> episodic -> semantic | -| `internal/agent/memory_migrator.go` | Lazy v2 daily files -> episodic migration | - -#### Module 4: System Integration (Phase 4) -| File | Change Type | Dependency | -|------|-------------|------------| -| `internal/agent/systemprompt.go` | MAJOR: replace with PromptBuilder | PromptConfig types | -| `internal/agent/systemprompt_sections.go` | REMOVE: logic moves to template | PromptBuilder | -| `internal/tools/types.go` | MODIFY: add Metadata() to Tool interface | None | -| `internal/tools/registry.go` | MODIFY: store ToolMetadata | ToolMetadata | -| `internal/tools/policy.go` | MODIFY: capability rules + role param | ToolCapability | -| `internal/tools/filesystem.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext | -| `internal/tools/filesystem_write.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext | -| `internal/tools/filesystem_list.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext | -| `internal/tools/shell.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext | -| `internal/tools/edit.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext | -| `internal/tools/memory.go` | MODIFY: add depth parameter | L0/L1/L2 | -| `internal/tools/knowledge_graph.go` | MODIFY: add as_of temporal param | KG temporal | -| `internal/tools/skill_search.go` | MODIFY: hybrid search integration | HybridSearcher | -| `internal/skills/loader.go` | MODIFY: ACL-aware BuildSummary() | SkillAccessStore | -| `internal/skills/search.go` | MODIFY: expose for fusion with embedding | None | - -**New files:** (listed in Phase 4 design) - -#### Module 5: Orchestration & Intelligence (Phase 5) -| File | Change Type | Dependency | -|------|-------------|------------| -| `internal/tools/policy.go` | MODIFY: orchestration mode filtering | OrchestrationMode | -| `internal/bus/types.go` | MODIFY: delegation + evolution event types | None | -| `internal/agent/loop.go` | MODIFY: metrics recording hooks | EvolutionMetrics | -| `cmd/gateway.go` | MODIFY: register delegate tool, event subscribers | All Phase 5 | - -**New files:** (listed in Phase 5 design) - -#### Breaking Changes Inventory - -| Change | Impact | Backward Compat | -|--------|--------|-----------------| -| Tool.Metadata() added to interface | All tool implementations | Wrapper: `defaultMetadata(tool)` | -| WorkspaceContext replaces dual keys | All file tools + loop_context | Single migration: grep `ToolWorkspaceFromCtx` (15 sites) | -| PromptBuilder replaces BuildSystemPrompt() | loop_history.go, tests | Bridge: old func calls PromptBuilder.Build() | -| Pipeline stages replace runLoop() | loop.go (804 lines) | Feature flag: v3_pipeline_enabled per agent | -| TokenCounter replaces chars/4 | loop_compact.go, pruning.go | Fallback: chars/4 for unknown models | -| KG temporal columns | All KG queries | Default: WHERE valid_until IS NULL (same as pre-temporal) | -| Episodic summaries table | New (no existing code depends on it) | Fully additive | -| Evolution metrics/suggestions | New (no existing code depends on it) | Fully additive | - ---- - -### D. Integration Test Plan - -#### Test Matrix: Workspace Scenarios - -From workspace forcing report — 6 scenarios must pass: - -| # | Scenario | ActivePath | Scope | Key Assertion | -|---|----------|------------|-------|---------------| -| 1 | Solo DM, isolated | `{base}/{userID}` | personal | Files scoped to user dir | -| 2 | Solo DM, shared | `{base}` | personal | Files in shared agent dir | -| 3 | Team dispatch (member) | `{base}/teams/{teamID}/{chatID}` | team-shared | Workspace forced to team dir | -| 4 | Team dispatch (leader) | `{base}/teams/{teamID}/{chatID}` | team-shared | Leader sees team workspace | -| 5 | Team inbound (member) | `{base}/teams/{teamID}/{chatID}` | team-isolated | Per-chat isolation within team | -| 6 | Delegate workspace | delegatee workspace + shared area | delegate | Read-only exports enforced | - -#### Memory Tier Tests - -| Test | Input | Expected | -|------|-------|----------| -| Working memory (Tier 1) | 20 messages in session | All messages in context window | -| Episodic creation | Session end event | episodic_summaries row created | -| Episodic dedup | Same session end event replayed | No duplicate (source_id unique) | -| Semantic extraction | Episodic with entities | KG entities upserted | -| L0 auto-inject | User message with relevant topic | Memory section in system prompt | -| L0 skip | "hello" message (low relevance) | No Memory section | -| V2 migration | Agent with memory/2026-01-01.md | Episodic row with source_type=v2_daily | -| TTL expiry | Episodic with expires_at in past | Cleaned by cron | - -#### KG Temporal Tests - -| Test | Input | Expected | -|------|-------|----------| -| New entity | UpsertEntity() | valid_from=NOW(), valid_until=NULL | -| Supersede entity | Contradicting fact | Old: valid_until=NOW(). New: valid_from=NOW() | -| Current query | SearchEntities() default | Only valid_until IS NULL entities | -| Historical query | SearchEntities(as_of="2026-01-01") | Entities valid at that date | -| Temporal traversal | Traverse() with temporal entities | Only current relations followed | -| Configurable dedup | Agent with threshold 0.95 | Auto-merge at 0.95, not hardcoded 0.98 | - -#### Pipeline Stage Tests - -| Test | Stage | Input | Expected | -|------|-------|-------|----------| -| Context build | ContextStage | RunRequest + agent config | WorkspaceContext resolved, system prompt built | -| LLM call | ThinkStage | Messages array | Provider called via adapter | -| Token pruning | PruneStage | Messages exceeding 85% context | Older messages pruned, token count accurate | -| Tool exec | ToolStage | Tool call from LLM | Tool executed via registry with policy check | -| Loop detect | ObserveStage | 3x same tool call | Loop detected, force stop | -| Session save | CheckpointStage | Completed run | Session persisted, events emitted | -| Full pipeline | All stages | Complete request | End-to-end: request -> response -> session saved | - -#### Provider Adapter Tests - -| Test | Provider | Input | Expected | -|------|----------|-------|----------| -| Anthropic wire | Anthropic | Internal ChatRequest | Valid Anthropic API JSON | -| OpenAI wire | OpenAI | Internal ChatRequest | Valid OpenAI API JSON | -| DashScope no-stream-tools | DashScope | ChatRequest with tools | Streaming disabled, tools present | -| Codex OAuth | Codex | ChatRequest | OAuth token refreshed if needed | -| Claude CLI | ClaudeCLI | ChatRequest | Stdio command built correctly | -| Capabilities | All providers | Capabilities() | Correct flags per provider | -| Round-trip | All providers | ToRequest -> FromResponse | Internal format preserved | - -#### Regression Tests - -| Test | Assertion | -|------|-----------| -| V2 agent with all flags false | Identical behavior to pre-v3 binary | -| V2 memory flush | Daily .md files still written when v3_memory_enabled=false | -| V2 KG without temporal | Queries return same results (WHERE valid_until IS NULL matches all) | -| V2 system prompt | PromptBuilder with v2 config produces identical output | -| V2 tool registry | Tools without Metadata() get default metadata wrapper | - ---- - -### E. Audit Checklist - -#### Interface Contracts -- [ ] All interface contracts have method signatures with ctx context.Context as first param -- [ ] All interfaces have at least one implementation (PG store) -- [ ] No interface has >10 methods (split if so) -- [ ] Error returns are typed (not raw string errors) -- [ ] Context propagation: tenant_id, agent_id, user_id carried through all store calls - -#### DB Schemas -- [ ] All new tables have tenant_id column with FK to tenants(id) -- [ ] All FKs have ON DELETE CASCADE where appropriate -- [ ] Indexes exist for: primary lookup, tenant filter, time-range queries -- [ ] No new columns on existing tables (all config in JSONB settings) -- [ ] Down migrations cleanly reverse up migrations -- [ ] No PG-specific features in interface definitions (SQLite compat future) - -#### Data Flow -- [ ] Memory flow: user message -> auto-inject L0 -> response -> session end -> episodic -> semantic -- [ ] KG flow: text -> extract -> upsert -> dedup -> temporal validity -- [ ] Pipeline flow: ContextStage -> ThinkStage -> PruneStage -> ToolStage -> ObserveStage -> CheckpointStage -- [ ] Delegation flow: delegate() -> permission check -> workspace setup -> AgentRunner -> result delivery -- [ ] Event flow: stage completion -> EventBus.Broadcast() -> subscriber handlers - -#### Breaking Changes -- [ ] Tool.Metadata() — wrapper provided for existing tools -- [ ] WorkspaceContext — single context key replaces 2 (grep all call sites) -- [ ] BuildSystemPrompt() — bridge function wraps PromptBuilder -- [ ] Pipeline stages — feature flag gates new behavior -- [ ] Token counting — fallback to chars/4 for unknown models - -#### Security Review -- [ ] Workspace isolation: WorkspaceContext.ActivePath enforced by resolvePath() -- [ ] Delegate read-only exports: validated against delegator's workspace boundary -- [ ] Delegate shared area: created with restrictive permissions (0750) -- [ ] Tenant boundary: all new tables have tenant_id, all queries filter by it -- [ ] Metrics data: no PII stored (queries + tool names, not user content) -- [ ] Template injection: PromptConfig data passed via {{ printf "%s" }} not {{ }} -- [ ] Auto-adapt scope: only modifies retrieval params, not security settings - -#### Performance Review -- [ ] tiktoken-go: benchmark Count() for Claude models (<1ms per 1000 tokens) -- [ ] pgvector: benchmark episodic_summaries search at 100K rows (<50ms) -- [ ] Event bus: benchmark Broadcast throughput (10K events/sec target) -- [ ] Template rendering: benchmark Build() (<1ms, no allocation-heavy paths) -- [ ] L0 auto-inject: benchmark memory index query (<10ms per turn) -- [ ] KG temporal: benchmark index scan with valid_until IS NULL filter -- [ ] Metrics recording: verify non-blocking (goroutine, no mutex on hot path) - ---- - -## Related Code Files - -### Migration Files to Create -| File | Purpose | -|------|---------| -| `migrations/000037_v3_memory_evolution.up.sql` | episodic_summaries, agent_evolution_metrics, agent_evolution_suggestions | -| `migrations/000037_v3_memory_evolution.down.sql` | Drop above tables | -| `migrations/000038_kg_temporal_columns.up.sql` | ALTER kg_entities/kg_relations add valid_from/valid_until | -| `migrations/000038_kg_temporal_columns.down.sql` | DROP added columns + indexes | - -### Files to Modify -| File | Change | -|------|--------| -| `internal/upgrade/version.go` | RequiredSchemaVersion = 38 | - -### Full Affected File Count by Module - -| Module | Files Modified | Files Created | Total | -|--------|---------------|---------------|-------| -| Foundation (Phase 1) | 15 | 4 | 19 | -| Pipeline (Phase 2) | 9 | 7 | 16 | -| Memory & KG (Phase 3) | 14 | 4 | 18 | -| System Integration (Phase 4) | 15 | 9 | 24 | -| Orchestration (Phase 5) | 4 | 10 | 14 | -| Migration (Phase 6) | 1 | 4 | 5 | -| **Total** | **58** | **38** | **96** | - ---- - -## Design Steps - -1. Write migration 000037 DDL (episodic_summaries, evolution tables) -2. Write migration 000038 DDL (KG temporal columns) -3. Write down migrations for both -4. Document lazy v2 memory migration flow -5. Document KG temporal backfill strategy -6. List all feature flags with defaults -7. Inventory all affected files per module (complete) -8. Document breaking changes with migration path -9. Write workspace scenario test matrix -10. Write memory tier test scenarios -11. Write KG temporal test scenarios -12. Write pipeline stage test scenarios -13. Write provider adapter test matrix -14. Write regression test list -15. Complete audit checklist - ---- - -## Todo List - -- [ ] A1: Migration 000037 DDL (3 new tables) -- [ ] A2: Migration 000038 DDL (KG temporal ALTER) -- [ ] A3: Down migrations for both -- [ ] A4: RequiredSchemaVersion bump to 38 -- [ ] B1: V2 memory -> episodic lazy migration design -- [ ] B2: KG temporal backfill strategy -- [ ] B3: Feature flags catalog with defaults -- [ ] B4: Session v2->v3 compatibility notes -- [ ] C1: Affected files inventory per module (complete) -- [ ] C2: Code change dependency order -- [ ] C3: Breaking changes list with backward-compat path -- [ ] D1: Workspace scenario test matrix (6 scenarios) -- [ ] D2: Memory tier tests (8 scenarios) -- [ ] D3: KG temporal tests (6 scenarios) -- [ ] D4: Pipeline stage tests (7 scenarios) -- [ ] D5: Provider adapter tests (7 scenarios) -- [ ] D6: Regression tests (5 scenarios) -- [ ] E1: Interface contracts audit -- [ ] E2: DB schema audit -- [ ] E3: Data flow audit -- [ ] E4: Breaking changes audit -- [ ] E5: Security review -- [ ] E6: Performance review - ---- - -## Success Criteria - -1. Both migrations run cleanly on empty DB and on DB with existing v2 data -2. Down migrations fully reverse up migrations (no orphaned objects) -3. RequiredSchemaVersion matches highest migration number -4. All 6 workspace scenarios pass integration tests -5. V2 agents with all flags false behave identically to pre-v3 -6. Every breaking change has documented backward-compat path -7. Audit checklist 100% complete before implementation begins - ---- - -## Risk Assessment - -| Risk | Severity | Mitigation | -|------|----------|------------| -| Migration fails on large KG tables | MEDIUM | ALTER TABLE ADD COLUMN with DEFAULT is metadata-only in PG12+ (instant). No table rewrite. | -| V2 memory files not found (path differences) | LOW | Lazy migration lists actual files in workspace. Files missing = nothing to migrate. | -| Feature flag combinatorial explosion | MEDIUM | 6 boolean flags = 64 combos. Test key combos only: all-off (v2), all-on (v3), individual toggles. | -| 96 affected files scope | HIGH | Implementation phased by module. Each module independently deployable behind feature flags. | -| Episodic summaries table grows large | MEDIUM | expires_at column + cleanup cron. Index on expires_at for efficient deletion. | -| Temporal KG backfill slow | LOW | Backfill is optional, runs as cron in background. No downtime. Default valid_from=NOW() sufficient. | -| SQLite compat deferred too long | MEDIUM | Interfaces designed PG-agnostic. SQLite impl can be added later without interface changes. | - ---- - -## Rollback Plan - -### Per-Migration Rollback -```bash -./goclaw migrate down 2 # reverts 000038, then 000037 -``` - -### Per-Feature Rollback -Set feature flag to false in agent settings: -```json -{ "v3_pipeline_enabled": false, "v3_memory_enabled": false } -``` -Agent immediately reverts to v2 behavior. No code rollback needed. - -### Full v3 Rollback -1. Disable all v3 feature flags for all agents -2. Run `migrate down 2` to remove v3 tables -3. Deploy v2 binary -4. v2 memory files, KG entities, sessions — all intact (v3 tables were additive) - ---- - -## Security Considerations - -- **Migration safety:** All changes additive (CREATE TABLE, ALTER ADD COLUMN). No DROP, RENAME, or data modification in up migration. Down migration only drops v3-added objects. -- **Feature flag isolation:** v3 code paths gated by flags. Flag false = v2 code path. No mixed state. -- **Tenant isolation in new tables:** All 3 new tables have tenant_id FK. All queries must include tenant_id filter. Store implementations enforce via context. -- **Evolution metrics cleanup:** Cron job with configurable TTL (default 90 days). Prevents unbounded growth. -- **Temporal KG queries:** Default WHERE valid_until IS NULL produces same results as pre-temporal. No data exposure from temporal columns. - ---- - -## Implementation Dependency Order - -``` -Foundation (can start immediately, no dependencies): - 1. TokenCounter (internal/agent/token_counter.go) - 2. WorkspaceContext (internal/tools/workspace_context.go) - 3. EventBus topics (internal/bus/types.go) - 4. ProviderAdapter interface (internal/providers/adapter.go) - -Core (depends on Foundation): - 5. Pipeline stages (internal/agent/pipeline.go + stage_*.go) - 6. PromptBuilder + templates (internal/agent/prompt_builder.go) - 7. Tool Metadata + enhanced registry (internal/tools/capability.go) - -Memory & KG (depends on Foundation + migrations): - 8. DB migrations 000037 + 000038 - 9. EpisodicSummaryStore (internal/store/episodic_store.go) - 10. KG temporal queries (modify existing pg/ files) - 11. Consolidation worker (internal/agent/consolidation_worker.go) - 12. L0 retriever (internal/agent/retriever.go) - -Integration (depends on Core + Memory): - 13. Hybrid skill searcher (internal/skills/hybrid_searcher.go) - 14. Tool policy capability rules (internal/tools/policy.go) - 15. Provider adapters (all 6 providers) - -Orchestration (depends on Integration): - 16. DelegateTool (internal/tools/delegate_tool.go) - 17. Orchestration mode config (internal/agent/orchestration_mode.go) - 18. Evolution metrics (internal/store/pg/evolution_metrics.go) - 19. Evolution suggestions (internal/agent/evolution_suggestion_engine.go) - -Testing (depends on all above): - 20. Integration tests (tests/integration/v3_*.go) - 21. Regression tests (tests/integration/v2_compat_*.go) -``` - -Items 1-4 are independent and can be built in parallel. -Items 5-7 are independent of each other but depend on 1-4. -Items 8-12 are sequential (migrations first, then stores, then workers). diff --git a/plans/260406-1304-goclaw-v3-core-design/plan.md b/plans/260406-1304-goclaw-v3-core-design/plan.md deleted file mode 100644 index 28617c0d..00000000 --- a/plans/260406-1304-goclaw-v3-core-design/plan.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: "GoClaw v3 Core Architecture Design" -description: "Comprehensive design plan for v3 core redesign — 18 modules covering memory, KG, agent loop, providers, workspace, orchestration" -status: complete -priority: P1 -effort: 40h -branch: dev-v3 -tags: [refactor, backend, architecture, design] -blockedBy: [] -blocks: [] -created: 2026-04-06 ---- - -# GoClaw v3 Core Architecture Design - -## Overview - -Design-first plan for comprehensive v3 core redesign. Produces interface contracts, DB schemas, data flow diagrams, and dependency graphs — NOT implementation code. Implementation plans created per-phase after design audit. - -**Brainstorm report:** `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` - -## Cross-Plan Dependencies - -None. This plan blocks future implementation plans (to be created after design audit). - -## Key Decisions (from brainstorm) - -| # | Module | Decision | -|---|--------|----------| -| 1 | Loop | Pipeline stages (pluggable interfaces) | -| 2 | Memory | 3-tier (working/episodic/semantic) | -| 3 | Context | L0/L1/L2 progressive loading | -| 4 | KG | Temporal validity windows | -| 5 | Token counting | tiktoken-go accurate | -| 6 | Consolidation | Async event-driven | -| 7 | Retrieval | Smart auto-inject L0 + on-demand | -| 8 | Workspace | Unified WorkspaceContext | -| 9 | System prompt | Template-based | -| 10 | Providers | Capability + unified wire + plugin | -| 11 | Self-evolution | Progressive (metrics→suggest→auto) | -| 12 | Orchestration | Layered: delegate base + team optional | -| 13 | Tools | Redesign (registry, policy, workspace-aware) | -| 14 | Events | Redesign (structured, typed) | -| 15 | Skills | Semantic search + ACL + deps | -| 16 | Context files | DB + better abstraction | -| 17 | Migration | Dual-mode v2→v3 | -| 18 | SQLite | Deferred | - -## Phases - -| Phase | Name | Status | Effort | -|-------|------|--------|--------| -| 1 | [Foundation Interfaces](./phase-01-foundation-interfaces.md) | **Complete** | 8h | -| 2 | [Pipeline Loop Design](./phase-02-pipeline-loop-design.md) | **Complete** | 8h | -| 3 | [Memory & KG Design](./phase-03-memory-kg-design.md) | **Complete** | 8h | -| 4 | [System Integration Design](./phase-04-system-integration-design.md) | **Complete** | 6h | -| 5 | [Orchestration & Intelligence Design](./phase-05-orchestration-intelligence-design.md) | **Complete** | 6h | -| 6 | [Migration & Audit](./phase-06-migration-audit.md) | **Complete** | 4h | - -## Dependency Graph - -``` -Phase 1: Foundation ─────────────────────────┐ - (TokenCounter, WorkspaceContext, │ - EventBus, ProviderAdapter) │ - │ │ - ▼ │ -Phase 2: Pipeline Loop ──────────┐ │ - (Stage interfaces, RunState) │ │ - │ │ │ - ▼ ▼ ▼ -Phase 3: Memory & KG Phase 4: System Integration - (3-tier, temporal, (Template prompt, tool - consolidation, L0/L1/L2) registry, skills, retrieval) - │ │ - └───────────┬───────────┘ - ▼ - Phase 5: Orchestration & Intelligence - (Delegate, self-evolution) - │ - ▼ - Phase 6: Migration & Audit - (v2→v3 dual-mode, test plan) -``` - -## Deliverables Per Phase - -Each phase produces: -- **Interface contracts** — Go interface definitions with method signatures -- **DB schemas** — SQL DDL for new/modified tables -- **Data flow diagrams** — Textual flow descriptions -- **Config schemas** — JSON/YAML configuration structures -- **Dependency map** — Which existing files are affected - -## References - -- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` -- Research: `plans/reports/researcher-260406-1059-memory-context-kg-research.md` -- Architecture: `plans/reports/Explore-260406-thorough-v3-architecture.md` -- Loop deep dive: `plans/reports/Explore-260406-GoClaw-Agent-Loop-v3-Deep-Dive.md` -- Workspace forcing: `plans/reports/Explore-260406-workspace-forcing.md`