tiennm99/goclaw
1
0 Fork 0
mirror of https://github.com/tiennm99/goclaw.git synced 2026-10-13 00:14:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Delete plans/260406-1304-goclaw-v3-core-design directory

Browse Source
This commit is contained in:
Viet Tran authored and GitHub committed 2026-04-10 00:21:51 +07:00
1 parent ea3353e397
commit 6a34b7ec65
7 files changed
-4502
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

plans/260406-1304-goclaw-v3-core-design/phase-01-foundation-interfaces.md
-638
View File
@@ -1,638 +0,0 @@
# Phase 1: Foundation Interfaces
## Context Links
- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (decisions 5, 8, 14, 10)
- Architecture: `plans/reports/Explore-260406-thorough-v3-architecture.md` (sections 1, 2, 6, 7)
- Loop deep dive: `plans/reports/Explore-260406-GoClaw-Agent-Loop-v3-Deep-Dive.md` (section 2, 3)
- Workspace forcing: `plans/reports/Explore-260406-workspace-forcing.md`
## Overview
- **Priority:** P1 (all other phases depend on these)
- **Status:** Complete
- **Effort:** 8h design
4 foundational components with zero cross-dependencies. Can be designed and implemented in parallel. Every subsequent phase (pipeline, memory, integration) depends on at least one of these.
## Key Insights
- Token counting currently `chars/4` estimate -- drives ALL compaction/pruning decisions. Replacing with tiktoken-go is highest-ROI change
- Workspace resolution has 5 identified confusion sources (see workspace forcing report). Single immutable WorkspaceContext eliminates dual-key confusion
- EventBus currently `map[string]any` payloads, string topic constants. V3 needs typed events for consolidation pipeline
- Provider interface (`providers.Provider`) is functional but loop handles provider quirks directly. Adapter pattern isolates quirks
---
## A. TokenCounter Interface
### Interface Contract
```go
// Package: internal/tokencount
// TokenCounter provides accurate per-model token counting.
// Replaces chars/4 estimation used in compaction + pruning.
type TokenCounter interface {
// Count returns token count for raw text using model's tokenizer.
Count(model string, text string) int
// CountMessages returns token count for a message list,
// including per-message overhead (role tokens, separators).
CountMessages(model string, msgs []providers.Message) int
// ModelContextWindow returns max context tokens for a model.
// Falls back to provider default if model unknown.
ModelContextWindow(model string) int
}
// TokenizerID identifies which tokenizer a model uses.
type TokenizerID string
const (
TokenizerCL100K TokenizerID = "cl100k_base" // Claude, GPT-3.5/4
TokenizerO200K TokenizerID = "o200k_base" // GPT-4o, GPT-5
TokenizerFallback TokenizerID = "fallback" // chars/4
)
// ModelRegistry maps model name prefixes to tokenizer + context window.
type ModelInfo struct {
TokenizerID TokenizerID
ContextWindow int // default context window for this model
}
// DefaultRegistry provides built-in model mappings.
// Extend at runtime via RegisterModel().
var DefaultRegistry = map[string]ModelInfo{
"claude-": {TokenizerCL100K, 200_000},
"gpt-4o": {TokenizerO200K, 128_000},
"gpt-4": {TokenizerCL100K, 128_000},
"gpt-5": {TokenizerO200K, 1_000_000},
"qwen-": {TokenizerCL100K, 128_000}, // approx
"deepseek-": {TokenizerCL100K, 128_000}, // approx
}
```
### Caching Strategy
```go
// cachedCounter wraps a tokenizer with per-message hash cache.
// Key: SHA-256(model + role + content + tool_call_ids)
// Value: token count (int)
// Eviction: LRU, 10K entries (typical session < 500 messages)
type cachedCounter struct {
cache *lru.Cache[string, int] // hashicorp/golang-lru/v2
mu sync.RWMutex
}
```
Cache avoids re-counting unchanged messages during mid-loop re-estimation. Only new/modified messages hit tokenizer.
### Per-Message Overhead
```
Per message: +4 tokens (role marker + separators)
System message: +4 tokens additional
Tool calls: name + JSON args counted as text
Tool results: content text counted
Images: model-specific (Claude: tile-based, GPT: similar)
```
### Data Flow
```
Pipeline stage needs token count
-> TokenCounter.CountMessages(model, messages)
-> For each message:
-> Hash(model, msg) -> check cache
-> Cache miss: tokenizer.Encode(text) -> count -> store
-> Sum + per-message overhead
-> Return total
```
### Affected Files (current codebase)
| File | What changes |
|------|-------------|
| `internal/agent/pruning.go` | Replace `EstimateHistoryTokens()` (chars/4) with `TokenCounter.CountMessages()` |
| `internal/agent/loop_compact.go` | Replace token estimation in compaction trigger |
| `internal/agent/loop.go:352-411` | Mid-loop budget calculation uses token counter |
| `internal/agent/loop_history.go:624-699` | `maybeSummarize()` token estimation |
| `internal/agent/loop_types.go` | `runState.overheadTokens` recalibration logic |
### Breaking Changes
- `EstimateHistoryTokens()`, `EstimateTokensWithCalibration()` replaced
- `overheadCalibrated` field in `runState` removed (overhead now computed accurately upfront)
- `pruning.go` constants (`softTrimThreshold`, `hardClearThreshold`) stay but use accurate counts
---
## B. WorkspaceContext
### Interface Contract
```go
// Package: internal/workspace
// Scope defines workspace access boundary.
type Scope string
const (
ScopePersonal Scope = "personal" // single user, isolated
ScopeTeam Scope = "team" // team context, shared or isolated
ScopeDelegate Scope = "delegate" // delegated task, scoped access
)
// WorkspaceContext is resolved ONCE at run start, immutable for the entire run.
// Eliminates dual ctxWorkspace + ctxTeamWorkspace confusion.
type WorkspaceContext struct {
// ActivePath is THE path for all file operations (read/write/list/exec).
// No other workspace path is accessible unless in ReadOnlyPaths.
ActivePath string
// Scope describes the access boundary type.
Scope Scope
// ReadOnlyPaths are additional paths the agent can read but NOT write.
// Used for: delegator exports, shared reference dirs.
ReadOnlyPaths []string
// SharedPath is the shared delegate area (read/write by both delegator + delegatee).
// nil when not in delegation context.
SharedPath *string
// TeamPath is the team workspace root (nil if not in team context).
TeamPath *string
// MemoryScope determines memory isolation.
// Defaults to workspace scope. "shared" = all users in agent see same memory.
MemoryScope string
// KGScope determines knowledge graph isolation.
// Defaults to workspace scope. "shared" = all users in agent see same KG.
KGScope string
// OwnerID identifies who owns this workspace context (user ID or chat ID).
OwnerID string
// EnforcementLabel is injected into system prompt verbatim.
// e.g. "ENFORCED -- all file paths resolve here, absolute paths rejected"
EnforcementLabel string
}
// Resolver produces a WorkspaceContext from request parameters.
// Called once at ContextStage. Result is immutable.
type Resolver interface {
Resolve(ctx context.Context, params ResolveParams) (*WorkspaceContext, error)
}
// ResolveParams captures all inputs needed to determine workspace.
type ResolveParams struct {
AgentID string
AgentType string // "open" | "predefined"
UserID string
ChatID string
TenantID string
PeerKind string // "direct" | "group"
TeamID *string
TeamConfig *TeamWorkspaceConfig // from teams table
DelegateCtx *DelegateContext // from delegation task
BaseDir string // global workspace root
Sharing *store.WorkspaceSharingConfig
}
// TeamWorkspaceConfig maps to team.settings JSON.
type TeamWorkspaceConfig struct {
SharedWorkspace bool `json:"shared_workspace"`
WorkspacePath string `json:"workspace_path,omitempty"`
}
// DelegateContext carries delegation-specific workspace overrides.
type DelegateContext struct {
LinkID string
SharedPath string
ExportPaths []string // read-only exports from delegator
}
```
### Resolution Rules
```
1. Solo DM, not shared:
ActivePath = {baseDir}/{tenantID}/{agentID}/{userID}
Scope = personal
2. Solo DM, shared workspace:
ActivePath = {baseDir}/{tenantID}/{agentID}
Scope = personal (shared files, memory still per-user unless overridden)
3. Team dispatch (member executing task):
ActivePath = {baseDir}/{tenantID}/teams/{teamID}/{chatID}
Scope = team
TeamPath = &ActivePath
4. Team inbound (leader receiving message):
ActivePath = {baseDir}/{tenantID}/{agentID}/{userID}
Scope = personal
TeamPath = &teamWorkspacePath (read-only reference)
5. Delegate:
ActivePath = own workspace
Scope = delegate
SharedPath = &sharedDelegatePath
ReadOnlyPaths = delegator exports
6. Group chat:
ActivePath = {baseDir}/{tenantID}/{agentID}/{chatID}
Scope = personal
```
### System Prompt Template Section
```
## Workspace
Active workspace: {{.ActivePath}} ({{.Scope}})
{{if .EnforcementLabel}}[{{.EnforcementLabel}}]{{end}}
{{if .TeamPath}}Team workspace: {{.TeamPath}} (accessible via team file tools){{end}}
{{if .ReadOnlyPaths}}Read-only paths: {{range .ReadOnlyPaths}}- {{.}}
{{end}}{{end}}
Context files (SOUL.md, USER.md, MEMORY.md): managed by system, read/write intercepted.
{{if eq .Scope "team"}}This is your active workspace. Personal workspace not accessible during this task.{{end}}
```
### Affected Files
| File | What changes |
|------|-------------|
| `internal/agent/loop_context.go:110-176` | Replace workspace resolution with `Resolver.Resolve()` |
| `internal/tools/context_keys.go` | Remove `ctxWorkspace`, `ctxTeamWorkspace`, `effectiveRestrict()` |
| `internal/tools/filesystem.go:308-340` | `resolvePathWithAllowed()` reads from `WorkspaceContext` |
| `internal/tools/filesystem_write.go` | Same |
| `internal/tools/shell.go:243-267` | Working dir from `WorkspaceContext.ActivePath` |
| `internal/agent/systemprompt.go:498-518` | Replace workspace section with template |
| `internal/agent/systemprompt_sections.go:503-520` | Merge team workspace section into workspace template |
### Breaking Changes
- `ctxWorkspace` / `ctxTeamWorkspace` context keys removed
- `effectiveRestrict()` removed (always true, now implicit via WorkspaceContext)
- `WithToolWorkspace()` / `WithToolTeamWorkspace()` replaced by single `WithWorkspaceContext()`
- `tools.WithRestrictToWorkspace()` removed (enforcement = default)
---
## C. EventBus Redesign
### Interface Contract
```go
// Package: internal/eventbus (new package, replaces internal/bus for domain events)
// Note: internal/bus MessageBus retained for channel message routing (InboundMessage/OutboundMessage).
// DomainEvent is a typed event with metadata for consolidation pipeline.
type DomainEvent struct {
ID string // UUID v7 for ordering
Type EventType
SourceID string // dedup key (e.g. session key, run ID)
TenantID string
AgentID string
UserID string
Timestamp time.Time
Payload any // typed per EventType (see below)
}
// EventType identifies the event category.
type EventType string
const (
EventSessionCompleted EventType = "session.completed"
EventEpisodicCreated EventType = "episodic.created"
EventEntityUpserted EventType = "entity.upserted"
EventMemoryLint EventType = "memory.lint"
EventRunCompleted EventType = "run.completed"
EventToolExecuted EventType = "tool.executed"
)
// Typed payloads -- one per EventType.
type SessionCompletedPayload struct {
SessionKey string
MessageCount int
TokensUsed int
Summary string // compaction summary if available
}
type EpisodicCreatedPayload struct {
EpisodicID string
SessionKey string
Summary string
KeyEntities []string // entity names for downstream extraction
}
type EntityUpsertedPayload struct {
EntityIDs []string // newly upserted entity UUIDs
}
type MemoryLintPayload struct {
Scope string // "agent" or "user"
}
type RunCompletedPayload struct {
RunID string
Iterations int
TokensUsed int
ToolCalls int
LoopKilled bool
}
type ToolExecutedPayload struct {
ToolName string
Duration time.Duration
Success bool
ReadOnly bool
}
// DomainEventBus manages typed event publishing + worker subscriptions.
type DomainEventBus interface {
// Publish enqueues an event for async processing.
// Non-blocking: returns immediately.
Publish(event DomainEvent)
// Subscribe registers a worker for a specific event type.
// handler called in worker pool goroutine.
// Returns unsubscribe function.
Subscribe(eventType EventType, handler DomainEventHandler) func()
// Start launches worker pool. Must be called before Publish.
Start(ctx context.Context)
// Drain waits for all queued events to be processed. For graceful shutdown.
Drain(timeout time.Duration) error
}
type DomainEventHandler func(ctx context.Context, event DomainEvent) error
```
### Worker Pool Design
```go
// Config for the domain event bus.
type Config struct {
QueueSize int // buffered channel capacity (default 1000)
WorkerCount int // goroutines per event type (default 2)
RetryAttempts int // retry on handler error (default 3)
RetryDelay time.Duration // backoff base (default 1s, exponential)
}
```
Lane-based concurrency: each EventType gets its own worker pool. Events of different types processed in parallel. Events of same type processed sequentially per-worker (but multiple workers can process different events of same type concurrently).
### Idempotency
Dedup by `SourceID`: events with same SourceID + EventType processed at most once. Implementation: in-memory set with TTL (1h). If event bus restarts, re-processing is safe because all handlers are idempotent.
```go
// dedup check before handler dispatch
func (b *bus) dispatch(event DomainEvent) {
key := event.Type + ":" + event.SourceID
if b.seen.Has(key) {
return // already processed
}
b.seen.Set(key, time.Now().Add(b.dedupTTL))
// dispatch to handler...
}
```
### Integration Points
| Producer | Event | Consumer |
|----------|-------|----------|
| Pipeline ObserveStage | `run.completed` | Metrics, tracing |
| Pipeline ObserveStage | `tool.executed` | Self-evolution metrics |
| Session close / compaction | `session.completed` | Episodic summary worker |
| Episodic worker | `episodic.created` | Semantic extraction worker |
| KG ingest | `entity.upserted` | Dedup check worker |
| Cron scheduler | `memory.lint` | Memory lint worker |
### Affected Files
| File | What changes |
|------|-------------|
| `internal/bus/types.go` | Retained for channel routing. DomainEvent types in new package |
| `internal/bus/bus.go` | Retained. New `internal/eventbus/` package alongside |
| `internal/agent/loop_run.go` | Publish `run.completed` after finalize |
| `internal/agent/loop_tools.go` | Publish `tool.executed` per tool call |
| `internal/agent/loop_history.go:624-699` | Publish `session.completed` in `maybeSummarize()` |
### Breaking Changes
- None for existing `bus.MessageBus` (retained)
- New `internal/eventbus` package added
- Agent loop gains `DomainEventBus` dependency (injected via `Loop` struct)
---
## D. ProviderAdapter Interface
### Interface Contract
```go
// Package: internal/providers (extends existing package)
// ProviderCapabilities declares what a provider supports.
// Queried by pipeline to choose code paths (streaming vs non-streaming, etc.)
type ProviderCapabilities struct {
Streaming bool // supports ChatStream()
ToolCalling bool // supports tools in request
StreamWithTools bool // can stream while tool calls are in-flight (false for DashScope)
Thinking bool // supports extended thinking / reasoning
Vision bool // supports image inputs
CacheControl bool // supports cache_control blocks (Anthropic)
MaxContextWindow int // default context window for default model
TokenizerID string // for tokencount package mapping
}
// ProviderAdapter transforms between internal wire format and provider-specific format.
// Each provider implements this to isolate quirks from the pipeline loop.
type ProviderAdapter interface {
// ToRequest converts internal ChatRequest to provider-specific wire bytes.
// Handles: schema transforms, thinking passback, cache control injection.
ToRequest(req ChatRequest) ([]byte, http.Header, error)
// FromResponse converts provider-specific response bytes to internal ChatResponse.
// Handles: thinking extraction, usage normalization, finish reason mapping.
FromResponse(data []byte) (*ChatResponse, error)
// FromStreamChunk converts a single SSE chunk to internal StreamChunk.
// Returns nil if chunk should be skipped (keep-alive, metadata).
FromStreamChunk(data []byte) (*StreamChunk, error)
// Capabilities returns static capability declaration.
Capabilities() ProviderCapabilities
// Name returns provider identifier.
Name() string
}
// CapabilitiesAware is optionally implemented by Provider.
// Pipeline checks this to choose code path.
type CapabilitiesAware interface {
Capabilities() ProviderCapabilities
}
```
### Adapter Registry
```go
// AdapterRegistry maps provider names to adapter factories.
// Enables plugin-style provider registration.
type AdapterRegistry struct {
mu sync.RWMutex
adapters map[string]AdapterFactory
}
type AdapterFactory func(cfg ProviderConfig) (ProviderAdapter, error)
// Register adds a provider adapter. Called at init() or runtime.
func (r *AdapterRegistry) Register(name string, factory AdapterFactory) {
r.mu.Lock()
defer r.mu.Unlock()
r.adapters[name] = factory
}
// Get returns adapter for provider name. Error if not registered.
func (r *AdapterRegistry) Get(name string, cfg ProviderConfig) (ProviderAdapter, error) {
r.mu.RLock()
factory, ok := r.adapters[name]
r.mu.RUnlock()
if !ok {
return nil, fmt.Errorf("unknown provider: %s", name)
}
return factory(cfg)
}
```
### What Moves to Adapter
| Currently in loop/provider | Moves to adapter |
|---------------------------|-----------------|
| `RawAssistantContent` passback (Anthropic thinking blocks) | `AnthropicAdapter.ToRequest()` |
| `ThinkingSignature` accumulation | `AnthropicAdapter.FromStreamChunk()` |
| DashScope non-streaming fallback for tool calls | `DashScopeAdapter.Capabilities().StreamWithTools = false` |
| Codex `Phase` field injection | `CodexAdapter.ToRequest()` / `FromResponse()` |
| Schema normalization (tool definitions → provider format) | Each adapter's `ToRequest()` |
| Usage normalization (different field names) | Each adapter's `FromResponse()` |
### Pipeline Usage
```
ThinkStage:
1. caps := provider.Capabilities() // if CapabilitiesAware
2. If caps.StreamWithTools == false && len(tools) > 0:
-> use Chat() not ChatStream()
3. If caps.Thinking:
-> inject thinking options
4. Call Chat/ChatStream as before (Provider interface unchanged)
5. Provider internally uses its adapter for wire transforms
```
Important: `providers.Provider` interface UNCHANGED. `ProviderAdapter` is internal to each provider implementation. Pipeline queries capabilities via optional `CapabilitiesAware` interface on the provider.
### Affected Files
| File | What changes |
|------|-------------|
| `internal/providers/types.go` | Add `ProviderCapabilities`, `CapabilitiesAware` |
| `internal/providers/anthropic.go` | Extract wire transforms to `AnthropicAdapter` |
| `internal/providers/openai.go` | Extract wire transforms to `OpenAIAdapter` |
| `internal/providers/dashscope.go` | `StreamWithTools = false` in capabilities |
| `internal/providers/codex.go` | Phase handling in adapter |
| `internal/agent/loop.go:217-323` | ThinkStage queries capabilities instead of provider-type checks |
### Breaking Changes
- `ThinkingCapable` interface superseded by `CapabilitiesAware` (backward compatible -- both can coexist during transition)
- Provider-specific branching in loop.go removed (replaced by capability checks)
- No external API changes
---
## Requirements
### Functional
- F1: Token counting accurate within 5% of provider-reported usage
- F2: WorkspaceContext resolves all 6 scenarios correctly (solo DM, shared, team dispatch, team inbound, delegate, group)
- F3: DomainEventBus processes events within 100ms p99 (in-process)
- F4: Provider capabilities queryable without LLM call
### Non-Functional
- NF1: TokenCounter cache hit rate >90% in typical session (messages rarely change)
- NF2: WorkspaceContext immutable after creation (no data races)
- NF3: EventBus handles 1000 events/sec sustained (consolidation pipeline load)
- NF4: All 4 components have zero cross-dependencies (can implement in parallel)
## Design Steps
1. Define `TokenCounter` interface + `ModelRegistry` mapping table
2. Implement `cachedCounter` with tiktoken-go + LRU cache
3. Define `WorkspaceContext` struct + `Resolver` interface
4. Implement resolver with resolution rules table
5. Define `DomainEvent` types + `DomainEventBus` interface
6. Implement worker pool with lane-based concurrency + dedup
7. Define `ProviderCapabilities` + `CapabilitiesAware` interface
8. Define `AdapterRegistry` for plugin registration
## Todo List
- [x] A1: TokenCounter interface definition — `internal/tokencount/token_counter.go`
- [x] A2: ModelRegistry with prefix-based mapping — `internal/tokencount/token_counter.go`
- [x] A3: Per-message hash cache design — documented in phase, cache impl deferred to implementation
- [x] A4: Fallback strategy for unknown models — `internal/tokencount/fallback_counter.go`
- [x] B1: WorkspaceContext struct definition — `internal/workspace/workspace_context.go`
- [x] B2: Resolver interface + ResolveParams — `internal/workspace/workspace_context.go`
- [x] B3: Resolution rules for all 6 scenarios — documented in phase design
- [x] B4: System prompt template section for workspace — documented in phase design
- [x] B5: Enforcement label design — EnforcementLabel field in WorkspaceContext
- [x] C1: DomainEvent type definitions — `internal/eventbus/event_types.go`
- [x] C2: Typed payload structs for each event — `internal/eventbus/event_types.go`
- [x] C3: DomainEventBus interface — `internal/eventbus/domain_event_bus.go`
- [x] C4: Worker pool config + lane-based concurrency design — `internal/eventbus/domain_event_bus.go`
- [x] C5: Idempotency / dedup strategy — documented in phase design (SourceID-based)
- [x] D1: ProviderCapabilities struct — `internal/providers/capabilities.go`
- [x] D2: CapabilitiesAware optional interface — `internal/providers/capabilities.go`
- [x] D3: AdapterRegistry for plugin registration — `internal/providers/adapter_registry.go`
- [x] D4: Document what moves from loop to each adapter — documented in phase design
## Success Criteria
- All 4 interfaces defined with complete method signatures
- Each interface has zero imports from the other 3
- WorkspaceContext resolution rules cover all 6 scenarios
- Token counting accuracy validated against Anthropic/OpenAI reported usage
- EventBus worker pool supports lane-based concurrency with configurable workers
- Provider capabilities capture all known provider quirks (DashScope, Codex, Anthropic)
## Risk Assessment
| Risk | Severity | Mitigation |
|------|----------|------------|
| tiktoken-go tokenizer not exact match for Claude models | MEDIUM | Claude uses custom tokenizer, cl100k_base is approximation. Validate against API usage. Accept 5% error. |
| WorkspaceContext missing edge case for new workspace scenario | LOW | Resolution rules table is explicit. Add new row when new scenario appears. |
| EventBus in-process limitation (single instance, no persistence) | LOW | Sufficient for v3.0. External bus (Redis streams) can implement same interface later. |
| ProviderAdapter abstraction leaks (provider needs loop context) | MEDIUM | Keep adapter stateless. Pass all needed data via `ChatRequest` fields. |
## Security Considerations
- WorkspaceContext enforcement is security-critical: `ActivePath` must be validated (no path traversal)
- Existing `resolvePath()` security (symlink, hardlink checks) preserved -- WorkspaceContext provides the boundary, not the validation
- EventBus handlers must not leak tenant data across boundaries (TenantID in every event)
- TokenCounter is read-only, no security concerns
## Next Steps
- Phase 2 (Pipeline Loop) consumes all 4 interfaces
- Phase 3 (Memory & KG) consumes TokenCounter + DomainEventBus
- Phase 4 (System Integration) consumes WorkspaceContext + ProviderCapabilities
plans/260406-1304-goclaw-v3-core-design/phase-02-pipeline-loop-design.md
-653
View File
@@ -1,653 +0,0 @@
# Phase 2: Pipeline Loop Design
## Context Links
- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (decision 1)
- Loop deep dive: `plans/reports/Explore-260406-GoClaw-Agent-Loop-v3-Deep-Dive.md` (all sections)
- Phase 1 dependencies: `phase-01-foundation-interfaces.md` (TokenCounter, WorkspaceContext, EventBus, ProviderCapabilities)
- Current loop: `internal/agent/loop.go` (804 lines), `loop_types.go`, `loop_context.go`, `loop_history.go`, `loop_compact.go`, `pruning.go`, `loop_tools.go`, `loop_run.go`
## Overview
- **Priority:** P1 (core execution engine, blocks phases 3-6)
- **Status:** Complete
- **Effort:** 8h design
- **Depends on:** Phase 1 (all 4 foundation interfaces)
Replace monolithic `runLoop()` (804 lines, 20+ field `runState`) with a pipeline of pluggable stages. Each stage owns a substage of state, is independently testable, and communicates via a shared `RunState` struct.
## Key Insights
- Current loop has 7 exit conditions scattered across 800 lines. Pipeline model makes each exit condition a stage responsibility.
- `runState` has 20+ fields mixing concerns (loop control, output accumulation, crash safety, bootstrap detection, team task tracking, skill evolution). Split into typed substates.
- Mid-loop compaction (lines 352-411) interleaves with think phase. Pipeline separates this into `PruneStage` between think and tool.
- Parallel tool execution already well-isolated (goroutine per tool, results collected via channel). Wrapping in `ToolStage` is clean.
- Memory flush currently triggers inside `maybeSummarize()` which runs post-loop. Moving to `MemoryFlushStage` pre-compaction aligns with 3-tier memory design.
---
## A. Stage Interface
```go
// Package: internal/pipeline
// Stage is a single step in the agent pipeline.
// Stages are stateless -- all mutable state lives in RunState.
type Stage interface {
// Name returns a human-readable identifier for logging/tracing.
Name() string
// Execute performs the stage's work. Returns error to abort pipeline.
// Stages modify RunState in place.
Execute(ctx context.Context, state *RunState) error
}
// StageResult signals how the pipeline should proceed after a stage.
type StageResult int
const (
Continue StageResult = iota // proceed to next stage
BreakLoop // exit iteration loop (normal completion)
AbortRun // abort entire run (error/kill)
)
// StageWithResult extends Stage to control pipeline flow.
// If a stage does not implement this, pipeline assumes Continue.
type StageWithResult interface {
Stage
Result() StageResult
}
```
## B. RunState Redesign
Split current 20+ field `runState` into typed substates. Each substate grouped by owning stage.
```go
// RunState is the shared mutable state for a single pipeline run.
// Passed by pointer through all stages.
type RunState struct {
// Identity (set once at pipeline start, immutable during run)
Request *agent.RunRequest
Workspace *workspace.WorkspaceContext
Model string
Provider providers.Provider
// Message buffer (read/write by multiple stages)
Messages *MessageBuffer
// Per-stage substates
Context ContextState
Think ThinkState
Prune PruneState
Tool ToolState
Observe ObserveState
Compact CompactState
Evolution EvolutionState
// Cross-cutting concerns
Iteration int
RunID string
ExitCode StageResult
}
// MessageBuffer wraps the message list with append/replace semantics.
// Thread-safe for read; write only from owning stage (sequential pipeline).
type MessageBuffer struct {
system providers.Message // system prompt (rebuilt by ContextStage)
history []providers.Message // conversation history
pending []providers.Message // new messages this run (flushed at checkpoint)
}
func (mb *MessageBuffer) All() []providers.Message { /* system + history + pending */ }
func (mb *MessageBuffer) AppendPending(msg providers.Message) { /* ... */ }
func (mb *MessageBuffer) FlushPending() []providers.Message { /* move pending to history */ }
func (mb *MessageBuffer) ReplaceHistory(msgs []providers.Message) { /* post-compaction */ }
```
### Substates
```go
// ContextState: owned by ContextStage, read by ThinkStage
type ContextState struct {
ContextFiles []bootstrap.ContextFile
SkillsSummary string
TeamContext *TeamContextInfo // nil if not in team
HadBootstrap bool
OverheadTokens int // system prompt + context files token count (accurate via TokenCounter)
}
// ThinkState: owned by ThinkStage
type ThinkState struct {
LastResponse *providers.ChatResponse
TotalUsage providers.Usage
TruncRetries int // consecutive truncation retries (max 3)
StreamingActive bool // true during active stream
}
// PruneState: owned by PruneStage
type PruneState struct {
MidLoopCompacted bool // true after first in-loop compaction
HistoryTokens int // last computed history token count
HistoryBudget int // contextWindow * maxHistoryShare
}
// ToolState: owned by ToolStage
type ToolState struct {
LoopDetector *ToolLoopDetector
TotalToolCalls int
AsyncToolCalls []string // tool names that executed async (spawn)
MediaResults []MediaResult // files produced by tools
Deliverables []string // content for team task results
LoopKilled bool // set when loop detector triggers critical
}
// ObserveState: owned by ObserveStage
type ObserveState struct {
FinalContent string // accumulated response text
FinalThinking string // reasoning output
BlockReplies int
LastBlockReply string
}
// CompactState: owned by CheckpointStage + MemoryFlushStage
type CompactState struct {
CheckpointFlushedMsgs int
MemoryFlushedThisCycle bool
CompactionCount int // tracks compaction cycles
}
// EvolutionState: owned by skill evolution nudge logic
type EvolutionState struct {
Nudge70Sent bool
Nudge90Sent bool
PostscriptSent bool
BootstrapWrite bool // BOOTSTRAP.md write detected
TeamTaskCreates int
TeamTaskSpawns int
}
```
### Migration from current runState
| Current field | Moves to |
|--------------|----------|
| `loopDetector` | `ToolState.LoopDetector` |
| `totalUsage` | `ThinkState.TotalUsage` |
| `iteration` | `RunState.Iteration` |
| `totalToolCalls` | `ToolState.TotalToolCalls` |
| `finalContent` | `ObserveState.FinalContent` |
| `finalThinking` | `ObserveState.FinalThinking` |
| `asyncToolCalls` | `ToolState.AsyncToolCalls` |
| `mediaResults` | `ToolState.MediaResults` |
| `deliverables` | `ToolState.Deliverables` |
| `pendingMsgs` | `MessageBuffer.pending` |
| `blockReplies` | `ObserveState.BlockReplies` |
| `lastBlockReply` | `ObserveState.LastBlockReply` |
| `checkpointFlushedMsgs` | `CompactState.CheckpointFlushedMsgs` |
| `midLoopCompacted` | `PruneState.MidLoopCompacted` |
| `overheadTokens` | `ContextState.OverheadTokens` |
| `overheadCalibrated` | REMOVED (overhead now accurate upfront via TokenCounter) |
| `bootstrapWriteDetected` | `EvolutionState.BootstrapWrite` |
| `teamTaskCreates/Spawns` | `EvolutionState.TeamTaskCreates/Spawns` |
| `skillNudge*` | `EvolutionState.Nudge*` |
| `loopKilled` | `ToolState.LoopKilled` |
| `truncationRetries` | `ThinkState.TruncRetries` |
---
## C. Stage Catalog
### Setup Pipeline (runs once before iteration loop)
```
1. ContextStage
- Resolve WorkspaceContext via Resolver
- Load + filter context files (bootstrap, per-user, team)
- Build skills summary (BM25)
- Build system prompt via template
- Compute overhead tokens (TokenCounter)
- Enrich input media
- Inject team task reminders
Reads: RunState.Request, providers
Writes: RunState.Workspace, ContextState, MessageBuffer.system
Replaces: injectContext() + buildMessages() + enrichInputMedia() + injectTeamTaskReminders()
```
### Iteration Pipeline (runs per iteration, max N times)
```
2. ThinkStage
- Inject iteration budget nudges (70%, 90%)
- Build per-iteration tool definitions (policy + bootstrap filters)
- Build ChatRequest from MessageBuffer
- Call LLM (Chat or ChatStream based on capabilities)
- Emit events: chunk, thinking
- Accumulate token usage
- Handle truncation retries
Reads: MessageBuffer, ContextState, ProviderCapabilities
Writes: ThinkState, MessageBuffer (append assistant message)
Result: Continue (has tool calls) | BreakLoop (no tool calls, final content)
Replaces: loop.go lines 217-446
3. PruneStage
- Compute history tokens via TokenCounter
- Phase 1 @ 70% budget: pruneContextMessages (soft-trim + hard-clear)
- Phase 2 @ 100% budget: compactMessagesInPlace (LLM summarization)
- If still over budget after Phase 2: AbortRun
Reads: MessageBuffer, TokenCounter, context window
Writes: PruneState, MessageBuffer (may replace history)
Result: Continue | AbortRun (budget exceeded after compaction)
Replaces: loop.go lines 352-411, loop_compact.go, pruning.go
4. ToolStage
- Extract tool calls from ThinkState.LastResponse
- Generate unique tool call IDs
- Single tool: sequential execution
- Multiple tools: parallel goroutines, collect via channel, sort by index
- Per-tool: policy check, lazy MCP activate, execute, process result
- Loop detection: same-tool, read-only streak, same-result
- Tool budget check (totalToolCalls > maxToolCalls)
- Emit events: tool.call, tool.result
Reads: ThinkState.LastResponse, ToolExecutor, WorkspaceContext
Writes: ToolState, MessageBuffer (append tool results)
Result: Continue | BreakLoop (loop killed) | AbortRun (policy denied)
Replaces: loop.go lines 450-738, loop_tools.go
5. ObserveStage
- Process tool results: extract deliverables, media
- Drain InjectCh (mid-run message injection, Point A + Point B)
- Check if final content ready (no tool calls + no injected messages)
- Sanitize final content
- Skill evolution postscript
- Publish domain events: tool.executed, run metrics
Reads: ToolState, ThinkState, InjectCh
Writes: ObserveState, MessageBuffer (may append injected messages)
Result: Continue (more iterations needed) | BreakLoop (final content ready)
Replaces: loop.go lines 450-466 (no-tool path), finalize portions
6. CheckpointStage
- Every N iterations (default 5): flush pending messages to session store
- Crash safety: intermediate state persisted
- Conditional: skip if iteration % N != 0
Reads: MessageBuffer.pending, SessionStore
Writes: CompactState.CheckpointFlushedMsgs, MessageBuffer (clear pending)
Replaces: loop.go lines 748-761
7. MemoryFlushStage
- Runs before compaction (when PruneStage detects budget pressure)
- Triggered by PruneStage setting a flag, NOT by iteration count
- Two-stage fallback: LLM flush -> extractive regex
- Dedup guard: once per compaction cycle
- Publishes: session.completed event (for episodic pipeline)
Reads: MessageBuffer, MemoryStore, compaction config
Writes: CompactState.MemoryFlushedThisCycle
Replaces: memoryflush.go, extractive_memory.go
```
### Post-Loop Pipeline (runs once after iteration loop exits)
```
8. FinalizeStage
- Sanitize final content (HTML entities, markdown cleanup)
- Media dedup + size population
- Flush remaining pending messages to session
- Token accumulation to session metadata
- Bootstrap auto-cleanup (delete BOOTSTRAP.md if write detected)
- Post-run summarization (maybeSummarize)
- Publish run.completed event
- Construct RunResult
Reads: All substates
Writes: RunResult (returned to caller)
Replaces: finalizeRun(), maybeSummarize()
```
---
## D. Pipeline Orchestrator
```go
// Pipeline orchestrates stage execution for a single agent run.
type Pipeline struct {
setup []Stage // runs once before loop
iteration []Stage // runs per iteration
finalize []Stage // runs once after loop
maxIterations int
tokenCounter tokencount.TokenCounter
eventBus eventbus.DomainEventBus
}
// NewDefaultPipeline creates the standard agent pipeline.
func NewDefaultPipeline(deps PipelineDeps) *Pipeline {
return &Pipeline{
setup: []Stage{
NewContextStage(deps),
},
iteration: []Stage{
NewThinkStage(deps),
NewPruneStage(deps),
NewToolStage(deps),
NewObserveStage(deps),
NewCheckpointStage(deps),
},
finalize: []Stage{
NewFinalizeStage(deps),
},
maxIterations: deps.Config.MaxIterations,
tokenCounter: deps.TokenCounter,
eventBus: deps.EventBus,
}
}
// PipelineDeps bundles all dependencies injected into stages.
type PipelineDeps struct {
Provider providers.Provider
TokenCounter tokencount.TokenCounter
EventBus eventbus.DomainEventBus
SessionStore store.SessionStore
ToolExecutor tools.ToolExecutor
ToolPolicy *tools.PolicyEngine
SkillsLoader *skills.Loader
MemoryStore store.MemoryStore
WorkspaceResolver workspace.Resolver
Config PipelineConfig
}
type PipelineConfig struct {
MaxIterations int
MaxToolCalls int
CheckpointInterval int // flush every N iterations (default 5)
ContextWindow int
MaxTokens int
Compaction *config.CompactionConfig
ContextPruning *config.ContextPruningConfig
}
```
### Execution Flow
```go
// Run executes the full pipeline for a single agent run.
func (p *Pipeline) Run(ctx context.Context, req *agent.RunRequest) (*agent.RunResult, error) {
state := NewRunState(req)
// 1. Setup pipeline (once)
for _, stage := range p.setup {
if err := stage.Execute(ctx, state); err != nil {
return nil, fmt.Errorf("setup %s: %w", stage.Name(), err)
}
}
// 2. Iteration loop
for state.Iteration = 0; state.Iteration < p.maxIterations; state.Iteration++ {
for _, stage := range p.iteration {
if err := stage.Execute(ctx, state); err != nil {
return nil, fmt.Errorf("iter %d %s: %w", state.Iteration, stage.Name(), err)
}
// Check if stage wants to control flow
if swr, ok := stage.(StageWithResult); ok {
switch swr.Result() {
case BreakLoop:
goto finalize
case AbortRun:
goto finalize
}
}
}
// Check context cancellation (/stop)
if ctx.Err() != nil {
break
}
}
finalize:
// 3. Finalize pipeline (once)
for _, stage := range p.finalize {
if err := stage.Execute(ctx, state); err != nil {
slog.Warn("finalize stage error", "stage", stage.Name(), "err", err)
// finalize errors are logged, not fatal
}
}
return state.BuildResult(), nil
}
```
### Error Propagation
```
Stage returns error:
- Setup stage error -> abort run, return error
- Iteration stage error -> abort run, still run finalize
- Finalize stage error -> log warning, continue other finalize stages
Stage returns StageResult:
- Continue -> next stage in iteration
- BreakLoop -> skip remaining iteration stages, jump to finalize
- AbortRun -> skip remaining stages, jump to finalize (with error flag)
```
---
## Data Flow Between Stages
```
ContextStage
writes: Workspace, ContextState, MessageBuffer.system
|
v
ThinkStage
reads: MessageBuffer.All(), ContextState.OverheadTokens
writes: ThinkState.LastResponse, ThinkState.TotalUsage
writes: MessageBuffer.AppendPending(assistant msg)
|
v
PruneStage
reads: MessageBuffer, TokenCounter, PruneState
writes: PruneState.HistoryTokens
may trigger: MemoryFlushStage (inline, before compaction)
may write: MessageBuffer.ReplaceHistory(compacted)
|
v
ToolStage
reads: ThinkState.LastResponse.ToolCalls
writes: ToolState.*, MessageBuffer.AppendPending(tool results)
|
v
ObserveStage
reads: ToolState, ThinkState, InjectCh
writes: ObserveState.FinalContent (if done)
may write: MessageBuffer.AppendPending(injected msgs)
|
v
CheckpointStage
reads: MessageBuffer.pending
writes: SessionStore, CompactState.CheckpointFlushedMsgs
|
v (loop back to ThinkStage or exit to FinalizeStage)
FinalizeStage
reads: All substates
writes: RunResult
```
---
## MemoryFlushStage Integration with PruneStage
MemoryFlushStage is NOT a regular iteration stage. It is invoked inline by PruneStage when compaction is needed.
```go
// PruneStage.Execute
func (s *PruneStage) Execute(ctx context.Context, state *RunState) error {
tokens := s.tokenCounter.CountMessages(state.Model, state.Messages.All())
state.Prune.HistoryTokens = tokens
budget := s.contextWindow * 80 / 100 // 80% for history
// Phase 1: soft pruning at 70%
if tokens > budget*70/100 {
pruneContextMessages(state.Messages, s.config)
tokens = s.tokenCounter.CountMessages(state.Model, state.Messages.All())
}
// Phase 2: full compaction at 100%
if tokens > budget {
// Memory flush BEFORE compaction (saves memories before they're summarized away)
if !state.Compact.MemoryFlushedThisCycle && s.memoryFlush != nil {
s.memoryFlush.Execute(ctx, state) // inline call, not pipeline stage
state.Compact.MemoryFlushedThisCycle = true
}
state.Prune.MidLoopCompacted = true
compactMessagesInPlace(state.Messages, s.compactionConfig)
// Re-check after compaction
tokens = s.tokenCounter.CountMessages(state.Model, state.Messages.All())
if tokens > budget {
state.ExitCode = AbortRun
}
}
return nil
}
```
---
## Requirements
### Functional
- F1: Pipeline produces identical `RunResult` as current monolithic loop for same inputs
- F2: All 7 exit conditions preserved: no-tool-calls, max iterations, truncation limit, loop kill, read-only streak, tool budget, ctx.Done
- F3: Parallel tool execution behavior unchanged (goroutine per tool, sorted results)
- F4: Mid-run injection (InjectCh) works at both Point A (after tools) and Point B (no tools)
- F5: Checkpoint flush every N iterations, crash safety preserved
### Non-Functional
- NF1: No measurable latency increase from stage dispatch overhead (stages are coarse-grained)
- NF2: Each stage independently testable with mock RunState
- NF3: Adding new stage requires only: implement Stage interface + insert in pipeline constructor
- NF4: RunState substates compile-time typed (no `map[string]any`)
## Related Code Files
### Files to decompose
| Current file | Lines | Maps to stages |
|-------------|-------|---------------|
| `internal/agent/loop.go` | 804 | ThinkStage, ToolStage, ObserveStage, orchestrator |
| `internal/agent/loop_run.go` | 245 | Pipeline.Run() entry point |
| `internal/agent/loop_context.go` | 275 | ContextStage |
| `internal/agent/loop_history.go` | 859 | ContextStage (buildMessages), FinalizeStage (maybeSummarize) |
| `internal/agent/loop_compact.go` | 119 | PruneStage |
| `internal/agent/pruning.go` | 327 | PruneStage |
| `internal/agent/loop_tools.go` | ~180 | ToolStage |
| `internal/agent/loop_tool_filter.go` | ~100 | ThinkStage (buildFilteredTools) |
| `internal/agent/memoryflush.go` | ~150 | MemoryFlushStage |
| `internal/agent/extractive_memory.go` | ~150 | MemoryFlushStage (fallback) |
| `internal/agent/loop_types.go` | ~150 | RunState, substates |
| `internal/agent/loop_input_media.go` | ~100 | ContextStage |
| `internal/agent/loop_team_reminders.go` | ~60 | ContextStage |
### New files to create
| File | Purpose |
|------|---------|
| `internal/pipeline/stage.go` | Stage interface, StageResult, StageWithResult |
| `internal/pipeline/run_state.go` | RunState + MessageBuffer + all substates |
| `internal/pipeline/pipeline.go` | Pipeline orchestrator |
| `internal/pipeline/context_stage.go` | ContextStage implementation |
| `internal/pipeline/think_stage.go` | ThinkStage implementation |
| `internal/pipeline/prune_stage.go` | PruneStage + inline MemoryFlush |
| `internal/pipeline/tool_stage.go` | ToolStage implementation |
| `internal/pipeline/observe_stage.go` | ObserveStage implementation |
| `internal/pipeline/checkpoint_stage.go` | CheckpointStage implementation |
| `internal/pipeline/finalize_stage.go` | FinalizeStage implementation |
## Design Steps
1. Define Stage interface + StageResult enum
2. Design RunState with typed substates (map current fields)
3. Define MessageBuffer with append/replace/flush semantics
4. Design each stage's Execute signature and data flow
5. Design Pipeline orchestrator with setup/iteration/finalize phases
6. Map error propagation rules (setup fatal, iteration -> finalize, finalize logged)
7. Design MemoryFlushStage inline integration with PruneStage
8. Verify all 7 exit conditions map to StageResult values
## Todo List
- [x] A1: Stage interface + StageResult + StageWithResult — `internal/pipeline/stage.go`
- [x] B1: RunState top-level struct — `internal/pipeline/run_state.go`
- [x] B2: MessageBuffer with thread-safe read, owning-stage write — `internal/pipeline/run_state.go`
- [x] B3: ContextState substate — `internal/pipeline/substates.go`
- [x] B4: ThinkState substate — `internal/pipeline/substates.go`
- [x] B5: PruneState substate — `internal/pipeline/substates.go`
- [x] B6: ToolState substate — `internal/pipeline/substates.go`
- [x] B7: ObserveState substate — `internal/pipeline/substates.go`
- [x] B8: CompactState substate — `internal/pipeline/substates.go`
- [x] B9: EvolutionState substate — `internal/pipeline/substates.go`
- [x] C1: ContextStage design — documented in phase (impl deferred)
- [x] C2: ThinkStage design — documented in phase (impl deferred)
- [x] C3: PruneStage design — documented in phase (impl deferred)
- [x] C4: ToolStage design — documented in phase (impl deferred)
- [x] C5: ObserveStage design — documented in phase (impl deferred)
- [x] C6: CheckpointStage design — documented in phase (impl deferred)
- [x] C7: MemoryFlushStage design — documented in phase (impl deferred)
- [x] C8: FinalizeStage design — documented in phase (impl deferred)
- [x] D1: Pipeline orchestrator with 3-phase execution — `internal/pipeline/pipeline.go`
- [x] D2: PipelineDeps dependency bundle — via NewPipeline() params
- [x] D3: Error propagation rules — setup fatal, iteration fatal+finalize, finalize logged
- [x] E1: Verify all 7 exit conditions preserved — documented in phase design
- [x] E2: Map current runState fields to substates — documented in phase design
## Success Criteria
- Every line of current `loop.go` maps to exactly one stage
- `runState` fields fully accounted for in substates (no orphans)
- Pipeline.Run() produces same RunResult semantics as current Loop.Run()
- Each stage can be tested with a constructed RunState (no Loop struct dependency)
- MemoryFlushStage integrated with PruneStage without circular dependency
- InjectCh drain preserved at both injection points
## Risk Assessment
| Risk | Severity | Mitigation |
|------|----------|------------|
| Subtle behavior change from stage ordering | HIGH | Integration tests comparing v2 vs v3 output for same inputs. Run existing test suite. |
| InjectCh timing sensitivity (Point A vs Point B) | MEDIUM | ObserveStage handles both points explicitly. Document injection semantics. |
| MemoryFlushStage inline call breaks stage isolation | LOW | Acceptable trade-off: flush MUST happen before compaction. Document as intentional coupling. |
| RunState too large (sum of substates) | LOW | Substates are small structs. Total < 500 bytes. No heap pressure. |
| Stage interface too simple for complex stages | LOW | StageWithResult extension handles flow control. Further extension possible via StageWithCleanup etc. |
## Security Considerations
- ToolStage must preserve all existing security checks: policy engine, shell deny patterns, workspace restriction
- ContextStage must preserve input guard scanning (injection pattern detection)
- CheckpointStage flushes to session store with existing tenant scoping
- No new attack surface introduced by pipeline refactor
## Next Steps
- Phase 3 (Memory & KG) adds MemoryFlushStage consumer for `session.completed` events
- Phase 4 (System Integration) adds template-based system prompt in ContextStage
- Phase 5 (Orchestration) may add DelegateStage or modify ToolStage for delegation
plans/260406-1304-goclaw-v3-core-design/phase-03-memory-kg-design.md
-797
View File
@@ -1,797 +0,0 @@
# Phase 3: Memory & KG Design
## Context Links
- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (decisions 2, 3, 4, 6, 7)
- Research: `plans/reports/researcher-260406-1059-memory-context-kg-research.md`
- Architecture: `plans/reports/Explore-260406-thorough-v3-architecture.md` (sections 1, 2, 8, 10)
- Phase 1: `phase-01-foundation-interfaces.md` (TokenCounter, DomainEventBus)
- Phase 2: `phase-02-pipeline-loop-design.md` (PruneStage, MemoryFlushStage, ObserveStage)
- Current memory store: `internal/store/memory_store.go`
- Current KG store: `internal/store/knowledge_graph_store.go`
- Current KG migration: `migrations/000013_knowledge_graph.up.sql`
- Current memory migration: `migrations/000001_init_schema.up.sql` (lines 163-192)
## Overview
- **Priority:** P1 (core cognitive capability)
- **Status:** Complete
- **Effort:** 8h design
- **Depends on:** Phase 1 (TokenCounter for L0 budget, DomainEventBus for consolidation), Phase 2 (pipeline integration points)
Design 3-tier memory system, temporal KG, consolidation pipeline, L0/L1/L2 context tiering, and smart auto-inject retrieval. This phase transforms GoClaw from append-only memory to a curated, searchable, self-maintaining knowledge system.
## Key Insights
- Current memory is append-only daily .md files, never revisited. Contradictions accumulate, no consolidation, no dedup
- KG extraction is synchronous (blocks loop). Dedup thresholds hardcoded (0.98/0.90). No temporal awareness -- stale facts never expire
- Memory search is hybrid FTS+vector but flat -- no tier awareness, no relevance-based injection
- Karpathy insight: memory should be curated wiki, not append-only log
- OpenViking insight: L0/L1/L2 tiering saves 10x tokens vs full-context loading
- Graphiti insight: temporal validity windows on KG facts enable contradiction resolution
---
## A. 3-Tier Memory Schema
### Tier 1: Working Memory
In context window. No new storage needed. Managed by MessageBuffer in pipeline.
- Content: recent 20-50 messages (conversation turns)
- Lifecycle: per-turn, free (no persistence cost)
- Size: up to 80% of context window budget
- Pruned by: PruneStage (token-accurate via TokenCounter)
### Tier 2: Episodic Memory
New table. Session summaries + key interactions with pgvector embeddings.
```sql
-- Migration: 000XXX_episodic_summaries.up.sql
CREATE TABLE episodic_summaries (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
user_id VARCHAR(255) NOT NULL DEFAULT '',
session_key VARCHAR(500) NOT NULL,
-- Content
summary TEXT NOT NULL, -- LLM-generated session summary
key_entities JSONB DEFAULT '[]', -- entity names extracted (for downstream KG)
key_topics JSONB DEFAULT '[]', -- topic tags for BM25
message_count INT NOT NULL DEFAULT 0, -- messages in summarized session
token_count INT NOT NULL DEFAULT 0, -- total tokens in session
-- Embeddings for semantic search
embedding vector(1536),
tsv tsvector GENERATED ALWAYS AS (to_tsvector('simple', summary)) STORED,
-- L0 abstract (pre-computed, ~50 tokens)
l0_abstract TEXT NOT NULL DEFAULT '',
-- Lifecycle
source_id VARCHAR(255) NOT NULL DEFAULT '', -- dedup key (session_key + compaction_count)
created_at TIMESTAMPTZ DEFAULT NOW(),
expires_at TIMESTAMPTZ, -- NULL = no expiry, default 90d from creation
UNIQUE(agent_id, user_id, source_id)
);
CREATE INDEX idx_episodic_scope ON episodic_summaries(agent_id, user_id);
CREATE INDEX idx_episodic_tenant ON episodic_summaries(tenant_id);
CREATE INDEX idx_episodic_created ON episodic_summaries(agent_id, user_id, created_at DESC);
CREATE INDEX idx_episodic_expires ON episodic_summaries(expires_at) WHERE expires_at IS NOT NULL;
CREATE INDEX idx_episodic_tsv ON episodic_summaries USING gin(tsv);
CREATE INDEX idx_episodic_embedding ON episodic_summaries USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 50);
```
### Tier 3: Semantic Memory
Existing KG tables (`kg_entities`, `kg_relations`) + temporal fields (section B). Facts extracted from episodic summaries via consolidation pipeline. Permanent, deduplicated.
No new table needed. Temporal fields added to existing tables.
### Tier Flow
```
Working (Tier 1) Episodic (Tier 2) Semantic (Tier 3)
[in context window] [episodic_summaries] [kg_entities + kg_relations]
| | |
|-- session end / compaction ------>| |
| (session.completed event) | |
| |-- extraction worker -------->|
| | (episodic.created event) |
| | |
|<-- auto-inject L0 summaries ------| |
|<-- memory_search L1/L2 ---------->| |
|<-- kg_search ------------------------------------------->| |
```
---
## B. Temporal KG Schema
### Schema Changes
```sql
-- Migration: 000XXX_kg_temporal.up.sql
-- Add temporal validity to entities
ALTER TABLE kg_entities
ADD COLUMN valid_from TIMESTAMPTZ DEFAULT NOW(),
ADD COLUMN valid_until TIMESTAMPTZ; -- NULL = currently valid
-- Add temporal validity to relations
ALTER TABLE kg_relations
ADD COLUMN valid_from TIMESTAMPTZ DEFAULT NOW(),
ADD COLUMN valid_until TIMESTAMPTZ;
-- Backfill existing data: valid_from = created_at, valid_until = NULL (current)
UPDATE kg_entities SET valid_from = created_at WHERE valid_from IS NULL;
UPDATE kg_relations SET valid_from = created_at WHERE valid_from IS NULL;
-- Partial index for current facts (most common query)
CREATE INDEX idx_kg_entities_current ON kg_entities(agent_id, user_id)
WHERE valid_until IS NULL;
CREATE INDEX idx_kg_relations_current ON kg_relations(agent_id, user_id)
WHERE valid_until IS NULL;
-- Add configurable dedup thresholds (per-agent, stored in agents.other_config JSONB)
-- No schema change needed -- uses existing agents.other_config field:
-- { "kg_dedup_auto_threshold": 0.98, "kg_dedup_flag_threshold": 0.90 }
```
### Query Patterns
```sql
-- Current facts (default, most queries)
SELECT * FROM kg_entities
WHERE agent_id = $1 AND user_id = $2 AND valid_until IS NULL;
-- Historical: what was true at a specific time?
SELECT * FROM kg_entities
WHERE agent_id = $1 AND user_id = $2
AND valid_from <= $3
AND (valid_until IS NULL OR valid_until >= $3);
-- Supersede a fact: new contradicting fact arrives
UPDATE kg_entities SET valid_until = NOW(), updated_at = NOW()
WHERE agent_id = $1 AND user_id = $2 AND external_id = $3 AND valid_until IS NULL;
-- Then INSERT new entity with valid_from = NOW()
```
### Contradiction Resolution
When extraction produces a fact that contradicts an existing current fact:
```
1. Extractor produces: "User works at CompanyB" (confidence 0.85)
2. Existing current fact: "User works at CompanyA" (confidence 0.90)
3. If same external_id + same entity_type:
a. Supersede old: SET valid_until = NOW() on CompanyA fact
b. Insert new: CompanyB with valid_from = NOW()
4. If different external_id but related via relation:
a. Flag as dedup candidate (human review)
```
### Configurable Thresholds
```go
// Per-agent KG config (stored in agents.other_config JSONB)
type KGConfig struct {
DedupAutoThreshold float64 `json:"kg_dedup_auto_threshold"` // default 0.98
DedupFlagThreshold float64 `json:"kg_dedup_flag_threshold"` // default 0.90
ExtractionMinConf float64 `json:"kg_extraction_min_conf"` // default 0.75
EnableTemporal bool `json:"kg_enable_temporal"` // default true
}
```
---
## C. Consolidation Pipeline
Event-driven, async, non-blocking. User never waits for consolidation.
### Event Flow
```
session.completed
|
v
EpisodicWorker
|-- Summarize session messages via LLM
|-- Extract key entities (names only, for downstream)
|-- Generate L0 abstract (~1 sentence, ~50 tokens)
|-- Store in episodic_summaries
|-- Generate embedding (async)
|-- Publish: episodic.created
|
v
SemanticExtractionWorker
|-- Read episodic summary
|-- Run KG extractor on summary (not full session -- cheaper)
|-- Upsert entities + relations with temporal fields
|-- Publish: entity.upserted
|
v
DedupWorker
|-- Read newly upserted entity IDs
|-- Compare embeddings against existing entities
|-- Auto-merge (> auto_threshold + name match)
|-- Flag candidates (> flag_threshold)
|-- Handle temporal supersession for contradictions
```
### Worker Implementations
```go
// Package: internal/consolidation
// EpisodicWorker processes session.completed events.
type EpisodicWorker struct {
store store.EpisodicStore
provider providers.Provider // for LLM summarization
tokenCounter tokencount.TokenCounter
eventBus eventbus.DomainEventBus
}
func (w *EpisodicWorker) Handle(ctx context.Context, event eventbus.DomainEvent) error {
payload := event.Payload.(eventbus.SessionCompletedPayload)
// Skip if already processed (idempotent by source_id)
sourceID := fmt.Sprintf("%s:%d", payload.SessionKey, payload.CompactionCount)
if w.store.ExistsBySourceID(ctx, event.AgentID, event.UserID, sourceID) {
return nil // already processed
}
// 1. Summarize session
summary, err := w.summarizeSession(ctx, payload)
if err != nil {
return fmt.Errorf("summarize: %w", err)
}
// 2. Extract key entities (lightweight -- name extraction, not full KG)
entities := extractEntityNames(summary)
// 3. Generate L0 abstract
l0 := generateL0Abstract(summary) // extractive: first sentence or topic sentence
// 4. Store
ep := &store.EpisodicSummary{
AgentID: event.AgentID,
UserID: event.UserID,
TenantID: event.TenantID,
SessionKey: payload.SessionKey,
Summary: summary,
KeyEntities: entities,
L0Abstract: l0,
MessageCount: payload.MessageCount,
TokenCount: payload.TokensUsed,
SourceID: sourceID,
ExpiresAt: timePtr(time.Now().Add(90 * 24 * time.Hour)), // 90d TTL
}
if err := w.store.Create(ctx, ep); err != nil {
return fmt.Errorf("store episodic: %w", err)
}
// 5. Publish for downstream
w.eventBus.Publish(eventbus.DomainEvent{
Type: eventbus.EventEpisodicCreated,
AgentID: event.AgentID,
UserID: event.UserID,
Payload: eventbus.EpisodicCreatedPayload{
EpisodicID: ep.ID,
SessionKey: payload.SessionKey,
Summary: summary,
KeyEntities: entities,
},
})
return nil
}
```
### Summarization Prompt (Episodic)
```
Summarize this conversation session concisely. Focus on:
- Key decisions made
- Facts learned about the user or project
- Tasks completed or in-progress
- Important technical details
- User preferences expressed
Output: 2-4 paragraph summary. Include entity names (people, projects, tools) explicitly.
Do NOT include greetings, filler, or metadata.
```
### L0 Abstract Generation
V3.0: extractive (first meaningful sentence of summary). V3.1+: LLM-generated.
```go
// generateL0Abstract produces a ~50 token abstract from episodic summary.
// V3.0: extractive (first sentence). Future: LLM-generated.
func generateL0Abstract(summary string) string {
// Split into sentences, take first non-trivial one
sentences := splitSentences(summary)
for _, s := range sentences {
s = strings.TrimSpace(s)
if len(s) > 20 { // skip very short fragments
if len(s) > 200 { // truncate if too long
s = s[:200] + "..."
}
return s
}
}
// Fallback: first 200 chars
if len(summary) > 200 {
return summary[:200] + "..."
}
return summary
}
```
---
## D. L0/L1/L2 Context Tiering
### Layer Definitions
| Layer | Content | Token cost | When loaded | Storage |
|-------|---------|-----------|-------------|---------|
| **L0** | 1-sentence abstract | ~50 tokens | Always (auto-inject if relevant) | `episodic_summaries.l0_abstract` |
| **L1** | Structured overview | ~200 tokens | When topic relevant | Generated on-demand, cached |
| **L2** | Full content | Full | On-demand via tool call | `episodic_summaries.summary` (episodic) or original document (memory docs) |
### L1 Generation
On-demand when agent requests deeper context or auto-inject detects high relevance.
```go
// L1Cache caches generated L1 overviews. LRU, 500 entries, 1h TTL.
type L1Cache struct {
cache *lru.Cache[string, string] // key: episodic_id or doc_path
}
// GenerateL1 creates structured overview from episodic summary or memory doc.
func (c *L1Cache) GenerateL1(id string, fullContent string) string {
if cached, ok := c.cache.Get(id); ok {
return cached
}
// Extractive: key bullet points from full content
l1 := extractKeyPoints(fullContent, 200) // ~200 tokens budget
c.cache.Add(id, l1)
return l1
}
```
### L2 Access
Full content loaded via tool call. Agent calls `memory_search(query, depth="full")` or `memory_expand(id)`.
```go
// memory_expand tool: loads L2 full content for a specific memory entry
// Parameters: id (episodic ID or document path)
// Returns: full content (summary or document)
```
---
## E. Smart Auto-Inject Retrieval
Per-turn relevance check + L0 injection. Non-blocking, lightweight.
### Flow
```
User message arrives (in ContextStage)
|
v
1. Extract intent keywords from user message
- Tokenize, remove stopwords, keep nouns/verbs/proper nouns
- If message too short ("hello", "ok"): skip injection (no relevant context)
|
v
2. BM25 + embedding relevance check against episodic index
- BM25 on episodic_summaries.tsv (fast, FTS)
- Embedding similarity on episodic_summaries.embedding (if available)
- Merge scores (configurable weights)
|
v
3. Filter by relevance threshold
- Default: 0.3 (tunable per agent via agent config)
- If no results above threshold: skip injection
|
v
4. Inject "## Memory Context" section into system prompt
- Max 5 entries, max ~200 tokens total (L0 summaries only)
- Format per entry: "- {topic}: {L0 abstract} [use memory_search for details]"
|
v
5. Agent sees hint, can call memory_search for L1/L2
```
### Interface
```go
// Package: internal/memory
// AutoInjector checks relevance and produces L0 injection for system prompt.
type AutoInjector interface {
// Inject checks user message against memory index.
// Returns formatted section for system prompt, or "" if nothing relevant.
// Budget: max ~200 tokens of L0 summaries.
Inject(ctx context.Context, params InjectParams) (string, error)
}
type InjectParams struct {
AgentID string
UserID string
TenantID string
UserMessage string
MaxEntries int // default 5
MaxTokens int // default 200
Threshold float64 // relevance threshold (default 0.3)
}
// InjectResult for observability
type InjectResult struct {
Section string // formatted prompt section
MatchCount int // total matches found
Injected int // entries injected (after budget trim)
TopScore float64 // highest relevance score
}
```
### System Prompt Section (generated by AutoInjector)
```
## Memory Context
Relevant memories from past sessions (summaries only -- use memory_search for details):
- Project Alpha: Discussed migration from PostgreSQL to CockroachDB, decided to stay with PG for now.
- User preferences: Prefers Go over Python for backend, uses Neovim as primary editor.
- Bug fix: Resolved auth token expiration issue in session middleware (2026-03-15).
Use `memory_search` tool to retrieve full details on any topic above.
```
### Agent Config
```go
// Per-agent memory config (extends existing config.MemoryConfig)
type MemoryConfig struct {
// Existing fields...
Enabled bool `json:"enabled"`
// V3 additions
AutoInjectEnabled bool `json:"auto_inject_enabled"` // default true
AutoInjectThreshold float64 `json:"auto_inject_threshold"` // default 0.3
AutoInjectMaxTokens int `json:"auto_inject_max_tokens"` // default 200
EpisodicTTLDays int `json:"episodic_ttl_days"` // default 90 (0 = no expiry)
ConsolidationEnabled bool `json:"consolidation_enabled"` // default true
}
```
---
## F. MemoryStore v3 Interface
Extends existing `MemoryStore` with tier-aware operations.
```go
// Package: internal/store
// EpisodicSummary represents a Tier 2 episodic memory entry.
type EpisodicSummary struct {
ID string `json:"id"`
TenantID string `json:"tenant_id"`
AgentID string `json:"agent_id"`
UserID string `json:"user_id"`
SessionKey string `json:"session_key"`
Summary string `json:"summary"`
KeyEntities []string `json:"key_entities"`
KeyTopics []string `json:"key_topics"`
L0Abstract string `json:"l0_abstract"`
MessageCount int `json:"message_count"`
TokenCount int `json:"token_count"`
SourceID string `json:"source_id"`
CreatedAt time.Time `json:"created_at"`
ExpiresAt *time.Time `json:"expires_at,omitempty"`
}
// EpisodicSearchResult is a search hit with tier indicator.
type EpisodicSearchResult struct {
EpisodicID string `json:"episodic_id"`
L0Abstract string `json:"l0_abstract"` // always populated
Score float64 `json:"score"`
CreatedAt time.Time `json:"created_at"`
SessionKey string `json:"session_key"`
}
// EpisodicStore manages Tier 2 episodic memory.
type EpisodicStore interface {
// CRUD
Create(ctx context.Context, ep *EpisodicSummary) error
Get(ctx context.Context, id string) (*EpisodicSummary, error)
Delete(ctx context.Context, id string) error
List(ctx context.Context, agentID, userID string, limit, offset int) ([]EpisodicSummary, error)
// Search (hybrid FTS + vector, returns L0 by default)
Search(ctx context.Context, query string, agentID, userID string, opts EpisodicSearchOptions) ([]EpisodicSearchResult, error)
// Lifecycle
ExistsBySourceID(ctx context.Context, agentID, userID, sourceID string) (bool, error)
PruneExpired(ctx context.Context) (int, error) // delete where expires_at < NOW()
// Embedding
SetEmbeddingProvider(provider EmbeddingProvider)
Close() error
}
type EpisodicSearchOptions struct {
MaxResults int
MinScore float64
VectorWeight float64
TextWeight float64
}
// MemorySearchResultV3 extends search result with tier indicator.
type MemorySearchResultV3 struct {
MemorySearchResult // embed existing fields
Tier string `json:"tier"` // "working", "episodic", "semantic"
EpisodicID string `json:"episodic_id,omitempty"`
L0Abstract string `json:"l0_abstract,omitempty"`
}
// UnifiedMemorySearch combines Tier 2 (episodic) and existing memory doc search.
// Used by memory_search tool.
type UnifiedMemorySearch interface {
// Search across all tiers. Returns results with tier indicators.
Search(ctx context.Context, query string, agentID, userID string, opts UnifiedSearchOptions) ([]MemorySearchResultV3, error)
// Expand loads full content for a specific memory entry (L2).
Expand(ctx context.Context, id string, tier string) (string, error)
}
type UnifiedSearchOptions struct {
MaxResults int
MinScore float64
Tiers []string // filter: ["episodic", "document"] (empty = all)
Depth string // "l0", "l1", "l2" (default "l1")
}
```
### KnowledgeGraphStore v3 Extensions
```go
// Temporal query additions to existing KnowledgeGraphStore interface.
// TemporalQueryOptions extends entity queries with time awareness.
type TemporalQueryOptions struct {
AsOf *time.Time // point-in-time query (nil = current only)
IncludeExpired bool // include superseded facts
}
// New methods added to KnowledgeGraphStore:
// ListEntitiesTemporal queries entities with temporal awareness.
// When opts.AsOf is nil, returns current facts only (valid_until IS NULL).
// When opts.AsOf is set, returns facts valid at that timestamp.
ListEntitiesTemporal(ctx context.Context, agentID, userID string,
listOpts EntityListOptions, temporal TemporalQueryOptions) ([]Entity, error)
// SupersedeEntity marks entity as no longer valid and inserts replacement.
// Atomic: UPDATE old SET valid_until=NOW() + INSERT new in single tx.
SupersedeEntity(ctx context.Context, old *Entity, replacement *Entity) error
// GetDedupConfig returns per-agent dedup thresholds from agents.other_config.
// Returns defaults if not configured.
GetDedupConfig(ctx context.Context, agentID string) (*KGConfig, error)
```
### Entity Struct v3
```go
// Extended Entity with temporal fields.
type Entity struct {
// ... existing fields unchanged ...
ID string `json:"id"`
AgentID string `json:"agent_id"`
UserID string `json:"user_id,omitempty"`
ExternalID string `json:"external_id"`
Name string `json:"name"`
EntityType string `json:"entity_type"`
Description string `json:"description,omitempty"`
Properties map[string]string `json:"properties,omitempty"`
SourceID string `json:"source_id,omitempty"`
Confidence float64 `json:"confidence"`
CreatedAt int64 `json:"created_at"`
UpdatedAt int64 `json:"updated_at"`
// V3 temporal fields
ValidFrom *time.Time `json:"valid_from,omitempty"`
ValidUntil *time.Time `json:"valid_until,omitempty"` // nil = currently valid
}
// Relation with temporal fields (same pattern).
type Relation struct {
// ... existing fields unchanged ...
// V3 temporal fields
ValidFrom *time.Time `json:"valid_from,omitempty"`
ValidUntil *time.Time `json:"valid_until,omitempty"`
}
```
---
## Pipeline Integration Points
### Where Memory/KG Connects to Pipeline Stages
| Pipeline stage | Memory/KG interaction |
|---------------|----------------------|
| **ContextStage** | AutoInjector.Inject() -> produces ## Memory Context section |
| **PruneStage** | Triggers MemoryFlushStage before compaction |
| **MemoryFlushStage** | Writes to memory docs (existing). Publishes `session.completed` |
| **FinalizeStage** | Publishes `session.completed` for episodic consolidation |
| **ToolStage** | `memory_search` tool calls UnifiedMemorySearch |
| **ToolStage** | `memory_expand` tool calls UnifiedMemorySearch.Expand() |
| **ToolStage** | `kg_search` tool uses temporal queries (default: current facts) |
### Consolidation Pipeline Registration
```go
// At gateway startup, register consolidation workers with DomainEventBus.
func RegisterConsolidationPipeline(eb eventbus.DomainEventBus, deps ConsolidationDeps) {
episodicWorker := NewEpisodicWorker(deps)
semanticWorker := NewSemanticExtractionWorker(deps)
dedupWorker := NewDedupWorker(deps)
eb.Subscribe(eventbus.EventSessionCompleted, episodicWorker.Handle)
eb.Subscribe(eventbus.EventEpisodicCreated, semanticWorker.Handle)
eb.Subscribe(eventbus.EventEntityUpserted, dedupWorker.Handle)
}
```
---
## Requirements
### Functional
- F1: Episodic summaries created automatically after session completion
- F2: L0 abstracts pre-computed at episodic creation (~50 tokens each)
- F3: Auto-inject produces relevant L0 summaries for user messages (skip for trivial messages)
- F4: Temporal KG queries default to current facts; historical queries via `AsOf` parameter
- F5: Contradiction resolution: supersede old fact, insert new with valid_from = NOW()
- F6: Dedup thresholds configurable per agent (not hardcoded)
- F7: Consolidation pipeline non-blocking (user never waits)
- F8: `memory_search` tool returns results with tier indicators
- F9: `memory_expand` tool loads L2 full content
- F10: Expired episodic entries pruned automatically (cron or lazy)
### Non-Functional
- NF1: Auto-inject latency < 50ms p99 (BM25 + embedding lookup)
- NF2: Episodic summarization completes within 30s (LLM call)
- NF3: Consolidation pipeline handles 100 session.completed events/minute
- NF4: pgvector index efficient up to 100K embeddings per agent
- NF5: L0 injection budget: max 200 tokens per turn
## Related Code Files
### Files to modify
| File | Change |
|------|--------|
| `internal/store/knowledge_graph_store.go` | Add temporal fields to Entity/Relation, new temporal query methods |
| `internal/store/pg/knowledge_graph*.go` | Implement temporal queries, supersede logic |
| `internal/store/memory_store.go` | No change (existing store retained for Tier 1/document memory) |
| `internal/knowledgegraph/extractor.go` | Source extraction from episodic summaries (lighter than raw messages) |
| `internal/agent/systemprompt.go` | Add ## Memory Context section from AutoInjector |
| `internal/tools/memory_tools.go` | Update memory_search to use UnifiedMemorySearch, add memory_expand |
### New files to create
| File | Purpose |
|------|---------|
| `internal/store/episodic_store.go` | EpisodicStore interface + types |
| `internal/store/pg/episodic_summaries.go` | PG implementation |
| `internal/memory/auto_injector.go` | AutoInjector implementation |
| `internal/memory/l1_cache.go` | L1 on-demand generation + LRU cache |
| `internal/consolidation/episodic_worker.go` | session.completed -> episodic summary |
| `internal/consolidation/semantic_worker.go` | episodic.created -> KG extraction |
| `internal/consolidation/dedup_worker.go` | entity.upserted -> dedup check |
| `internal/consolidation/registration.go` | Worker registration with EventBus |
| `migrations/000XXX_episodic_summaries.up.sql` | Episodic table + indexes |
| `migrations/000XXX_kg_temporal.up.sql` | Temporal fields on KG tables |
## Design Steps
1. Define `episodic_summaries` DDL with indexes
2. Define `EpisodicStore` interface
3. Design temporal KG schema changes (ALTER TABLE + backfill)
4. Define temporal query methods for KnowledgeGraphStore
5. Design consolidation pipeline event flow
6. Design EpisodicWorker with summarization prompt + L0 generation
7. Design AutoInjector with BM25 + embedding relevance check
8. Define UnifiedMemorySearch interface for cross-tier search
9. Design L0/L1/L2 depth parameter for search results
10. Define per-agent memory config extensions
## Todo List
- [x] A1: episodic_summaries DDL — documented in phase (SQL in Phase 6)
- [x] A2: EpisodicStore interface — `internal/store/episodic_store.go`
- [x] A3: EpisodicSummary struct + EpisodicSearchResult — `internal/store/episodic_store.go`
- [x] B1: kg_entities temporal fields DDL — documented in phase (SQL in Phase 6)
- [x] B2: kg_relations temporal fields DDL — documented in phase (SQL in Phase 6)
- [x] B3: Backfill migration — documented in phase design
- [x] B4: Temporal query methods — `internal/store/knowledge_graph_temporal.go`
- [x] B5: SupersedeEntity atomic method — `internal/store/knowledge_graph_temporal.go`
- [x] B6: Configurable dedup thresholds — `internal/store/knowledge_graph_temporal.go` KGConfig
- [x] C1: Consolidation event flow design — documented in phase
- [x] C2: EpisodicWorker handler — `internal/consolidation/workers.go` (stub)
- [x] C3: SemanticExtractionWorker handler — `internal/consolidation/workers.go` (stub)
- [x] C4: DedupWorker handler — `internal/consolidation/workers.go` (stub)
- [x] C5: Worker registration at startup — `internal/consolidation/workers.go` Register()
- [x] D1: L0 abstract generation — documented in phase design (extractive v3.0)
- [x] D2: L1 on-demand generation + LRU cache — documented in phase design
- [x] D3: L2 access via memory_expand tool — documented in phase design
- [x] E1: AutoInjector interface — `internal/memory/auto_injector.go`
- [x] E2: Relevance check (BM25 + embedding) — documented in phase design
- [x] E3: System prompt ## Memory Context section — documented in phase design
- [x] E4: Per-agent threshold config — `internal/memory/auto_injector.go` MemoryConfig
- [x] F1: UnifiedMemorySearch interface — documented in phase design
- [x] F2: MemorySearchResultV3 with tier indicator — documented in phase design
- [x] F3: memory_search tool update — documented in phase design
- [x] F4: memory_expand tool definition — documented in phase design
## Success Criteria
- episodic_summaries DDL passes `psql` validation
- Temporal KG migration backward-compatible (existing data has valid_from=created_at, valid_until=NULL)
- All consolidation workers idempotent (safe to replay events)
- AutoInjector returns "" for trivial messages ("hello", "ok")
- AutoInjector returns relevant L0s for topic-specific messages
- UnifiedMemorySearch merges results across tiers with correct tier labels
- Per-agent dedup thresholds configurable, defaults match current hardcoded values
## Risk Assessment
| Risk | Severity | Mitigation |
|------|----------|------------|
| Episodic summarization LLM cost | MEDIUM | Use smaller/cheaper model for summarization (e.g. GPT-4o-mini). Configurable per agent. |
| Auto-inject false positives (irrelevant memories injected) | MEDIUM | Conservative default threshold (0.3). Agent can override. Monitor via evolution metrics. |
| pgvector index performance at scale (100K+ embeddings) | MEDIUM | ivfflat index with lists=50. Benchmark needed. Increase lists for larger datasets. |
| Consolidation pipeline backlog during high load | LOW | Lane-based concurrency in EventBus. Workers are idempotent. Backlog clears naturally. |
| L0 extractive quality insufficient | LOW | Extractive is "good enough" for v3.0. LLM-generated L0 in v3.1. |
| Temporal backfill migration slow on large KG | LOW | Backfill is simple UPDATE SET DEFAULT. Add timeout guard. |
| Cross-agent memory leakage | HIGH | All queries scoped by tenant_id + agent_id + user_id. Verify in code review. |
## Security Considerations
- Tenant isolation: ALL episodic + KG queries must include tenant_id scope. Missing tenant_id = data leak.
- User isolation: episodic summaries scoped by user_id unless shared memory enabled
- L0/L1 content is derived from user conversations -- same access controls as original messages
- Consolidation workers run in system context -- must not expose cross-tenant data
- AutoInjector results filtered by same agent_id + user_id scope as memory_search
## Unresolved Questions
1. **Episodic TTL**: 90 days default. Should this be configurable per agent? (Likely yes, via MemoryConfig)
2. **L0 for memory documents**: Should existing memory_documents also get L0 abstracts? (Deferred to v3.1 -- episodic first)
3. **Cross-agent memory**: Should agents in same tenant share semantic memory? (Brainstorm Q2, deferred)
4. **Embedding model**: Continue OpenAI text-embedding-3-small or switch to open model? (Brainstorm Q3, deferred)
5. **Concurrent consolidation**: Multiple sessions ending simultaneously -> same episodic worker. EventBus dedup by source_id handles this, but verify ordering.
## Next Steps
- Phase 4 (System Integration) integrates AutoInjector into ContextStage template
- Phase 5 (Orchestration) uses self-evolution metrics from consolidation pipeline
- Phase 6 (Migration) handles v2 daily memory files -> episodic import
plans/260406-1304-goclaw-v3-core-design/phase-04-system-integration-design.md
-707
View File
@@ -1,707 +0,0 @@
# Phase 4: System Integration Design
## Context Links
- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (sections 9, 12, 14, 7)
- Architecture report: `plans/reports/Explore-260406-thorough-v3-architecture.md` (sections 5, 6)
- Workspace forcing: `plans/reports/Explore-260406-workspace-forcing.md`
- Current system prompt: `internal/agent/systemprompt.go`, `internal/agent/systemprompt_sections.go`
- Current tool registry: `internal/tools/registry.go`, `internal/tools/policy.go`, `internal/tools/types.go`
- Current skills: `internal/skills/loader.go`, `internal/skills/search.go`, `internal/store/skill_store.go`
## Overview
- **Priority:** P1 (core dependency for pipeline loop + memory system)
- **Status:** Complete
- **Effort:** 6h design
Redesigns 4 tightly-coupled systems that sit between the agent loop and the LLM:
1. Template-based system prompt (replace string concatenation)
2. Tool registry with typed capabilities + declarative policy
3. Skills system with embedding search + ACL + dependency graph
4. Retrieval integration connecting auto-inject L0 with template + tools
---
## Key Insights
1. Current `BuildSystemPrompt()` is ~200 lines of conditional string building. 15+ sections toggled by booleans. Adding workspace enforcement language or memory context requires touching fragile concatenation logic.
2. Tool types use 12+ optional interfaces (`InterceptorAware`, `PathAllowable`, etc.) with type assertions at registration. No capability metadata on tools themselves.
3. PolicyEngine's 7-step pipeline works but all checks are string-based group matching. No typed capabilities (read-only vs mutating), no workspace-awareness.
4. Skills BM25-only search misses intent-based queries ("help me debug" won't find "debugger" skill). `EmbeddingSkillSearcher` interface exists in store but not wired to loader's `BuildSummary()`.
5. Workspace confusion root cause: system prompt text says "use relative paths" but doesn't state the enforcement rule. Template must carry WorkspaceContext enforcement language verbatim.
---
## Requirements
### Functional
- F1: System prompt built from Go `text/template` with embedded template file
- F2: Each template section independently toggleable via PromptConfig
- F3: WorkspaceContext generates explicit enforcement language (not suggestive)
- F4: Tools declare capabilities (read-only, mutating, async, mcp-bridged) at registration
- F5: Policy engine evaluates capabilities (e.g., "deny all mutating tools for viewer role")
- F6: Tools receive WorkspaceContext, not raw path strings
- F7: Skills searched by both BM25 (lexical) and embedding (semantic), results merged
- F8: Skills filtered by agent/tenant ACL before injection
- F9: Skill dependency graph: loader resolves + validates before activation
- F10: Auto-inject L0 memory summaries into template's Memory section
- F11: Provider-specific template variants (e.g., Codex needs different tool format)
### Non-Functional
- NF1: Template rendering <1ms (no LLM calls during prompt build)
- NF2: Skill embedding search <50ms at 1000 skills
- NF3: Backward compatible: v2 PromptConfig can produce identical v2 output
- NF4: Template testable in isolation (no DB dependency for rendering)
---
## Architecture
### A. Template-Based System Prompt
#### PromptConfig Interface Contract
```go
// PromptConfig controls which template sections to include.
// Each field maps to a named template block.
type PromptConfig struct {
// Section toggles
Identity bool
Persona bool // SOUL.md content
Instructions bool // AGENTS.md content
Tools bool
Skills bool
Team bool
Workspace bool
Memory bool // auto-inject L0 section
Sandbox bool
Recency bool // recency reminder
// Data payloads (populated when section enabled)
IdentityData IdentityData
PersonaContent string // raw SOUL.md
InstructionContent string // raw AGENTS.md
ToolsData ToolsSectionData
SkillsData SkillsSectionData
TeamData TeamSectionData
WorkspaceData WorkspaceSectionData
MemoryData MemorySectionData
SandboxData SandboxSectionData
ExtraPrompt string
// Provider variant (selects template file)
ProviderVariant string // "" = default, "codex", "dashscope"
}
```
#### Section Data Types
```go
type IdentityData struct {
AgentName string
Emoji string
Model string
Channel string
ChatTitle string
PeerKind string // "direct" or "group"
}
type ToolsSectionData struct {
ToolDefs []ToolSummary // name + one-line description
MCPTools []ToolSummary // MCP bridge tools (separate listing)
CoreSummary map[string]string // builtin tool summaries
}
type ToolSummary struct {
Name string
Description string
Capability ToolCapability // read-only, mutating, async, mcp-bridged
}
type SkillsSectionData struct {
Mode string // "inline" (inject summaries) or "search" (search tool only)
Summaries []SkillSummaryEntry
}
type SkillSummaryEntry struct {
Slug string
Name string
Description string
Score float64 // relevance score from search
}
type TeamSectionData struct {
TeamWorkspace string
Members []TeamMemberEntry
Guidance string // edition-specific (Full vs Lite)
TeamMDContent string // TEAM.md content
}
type TeamMemberEntry struct {
AgentKey string
DisplayName string
Skills string
Role string // "lead" or "member"
}
type WorkspaceSectionData struct {
ActivePath string
Scope string // "personal" | "team-shared" | "team-isolated" | "delegate"
Enforced bool // always true in v3
ReadOnlyPaths []string
SharedPath *string
ContextFiles []string // list of managed context files
EnforcementMsg string // pre-rendered enforcement text
}
type MemorySectionData struct {
L0Summaries []L0Summary // auto-injected memory hints
HasSearch bool // memory_search tool available
HasKG bool // kg_search tool available
}
type L0Summary struct {
Topic string
Summary string // 1-sentence abstract
ID string // for memory_expand(id) deep retrieval
}
type SandboxSectionData struct {
ContainerDir string
AccessLevel string // "full" or "read-only"
}
```
#### Template Structure
Embedded template file at `internal/agent/prompt_template.go.tmpl`:
```
{{- /* System Prompt Template — v3 */ -}}
{{- if .Identity -}}
{{- template "identity" .IdentityData -}}
{{- end -}}
{{- if .Persona -}}
# Persona
{{ .PersonaContent }}
{{- end -}}
{{- if .Instructions -}}
# Instructions
{{ .InstructionContent }}
{{- end -}}
{{- if .Workspace -}}
{{- template "workspace" .WorkspaceData -}}
{{- end -}}
{{- if .Tools -}}
{{- template "tools" .ToolsData -}}
{{- end -}}
{{- if .Skills -}}
{{- template "skills" .SkillsData -}}
{{- end -}}
{{- if .Memory -}}
{{- template "memory" .MemoryData -}}
{{- end -}}
{{- if .Sandbox -}}
{{- template "sandbox" .SandboxData -}}
{{- end -}}
{{- if .Team -}}
{{- template "team" .TeamData -}}
{{- end -}}
{{- if .ExtraPrompt -}}
# Additional Context
{{ .ExtraPrompt }}
{{- end -}}
```
#### Workspace Sub-Template (Critical)
```
{{- define "workspace" -}}
# Workspace
- **Path:** {{ .ActivePath }}
- **Scope:** {{ .Scope }}
- **ENFORCED:** All file tool paths resolve relative to this directory. Absolute paths outside workspace are REJECTED.
{{- if .ReadOnlyPaths }}
- **Read-only access:** {{ range .ReadOnlyPaths }}{{ . }}, {{ end }}
{{- end }}
{{- if .SharedPath }}
- **Shared area:** {{ .SharedPath }} (read/write by both you and delegator)
{{- end }}
{{- if .ContextFiles }}
- **Managed files:** {{ range .ContextFiles }}{{ . }}, {{ end }}(system-managed, read/write intercepted to DB)
{{- end }}
{{- end -}}
```
#### Memory Sub-Template
```
{{- define "memory" -}}
# Memory Context
{{- if .L0Summaries }}
{{- range .L0Summaries }}
- **{{ .Topic }}:** {{ .Summary }} [use memory_search("{{ .ID }}") for details]
{{- end }}
{{- end }}
{{- if .HasSearch }}
Use `memory_search` for detailed retrieval. Use `kg_search` for relationship queries.
{{- end }}
{{- end -}}
```
#### PromptBuilder Interface
```go
// PromptBuilder renders system prompts from PromptConfig.
type PromptBuilder interface {
Build(cfg PromptConfig) (string, error)
}
// TemplatePromptBuilder implements PromptBuilder using text/template.
type TemplatePromptBuilder struct {
templates map[string]*template.Template // variant -> parsed template
}
// NewTemplatePromptBuilder loads embedded templates.
func NewTemplatePromptBuilder() (*TemplatePromptBuilder, error)
// Build renders prompt. Selects template variant from cfg.ProviderVariant.
// Returns rendered string. Error only on template parse failure (should not happen with embedded).
func (b *TemplatePromptBuilder) Build(cfg PromptConfig) (string, error)
```
#### Provider Variants
- **Default template:** Standard Anthropic/OpenAI format
- **Codex variant:** Tool descriptions formatted as docstrings (Codex prefers this)
- **DashScope variant:** No streaming+tools section (DashScope limitation)
- Variant selected by `cfg.ProviderVariant` which maps to provider capabilities
---
### B. Tool Registry Redesign
#### Tool Capability Types
```go
// ToolCapability describes what a tool can do.
type ToolCapability string
const (
CapReadOnly ToolCapability = "read-only" // no side effects (read_file, list_files, memory_search)
CapMutating ToolCapability = "mutating" // modifies state (write_file, exec, edit)
CapAsync ToolCapability = "async" // returns immediately, result via callback
CapMCPBridged ToolCapability = "mcp-bridged" // proxied to external MCP server
)
// ToolMetadata describes a tool's capabilities and requirements.
type ToolMetadata struct {
Name string
Capabilities []ToolCapability
Group string // "fs", "web", "runtime", "memory", "team", etc.
RequiresWorkspace bool // tool needs WorkspaceContext
ProviderHints map[string]any // provider-specific hints (e.g., cache_control)
}
```
#### Enhanced Tool Interface
```go
// Tool is the base interface all tools must implement.
// V3 adds Metadata() for capability declaration.
type Tool interface {
Name() string
Description() string
Parameters() map[string]any
Execute(ctx context.Context, args map[string]any) *Result
Metadata() ToolMetadata // NEW: declarative capability info
}
```
> **Migration note:** Existing tools that don't implement `Metadata()` get a wrapper
> that returns default metadata (capabilities inferred from tool group membership).
#### WorkspaceContext in Tool Context
```go
// Tools access WorkspaceContext via context, not struct fields.
// Replaces: ToolWorkspaceFromCtx(), ToolTeamWorkspaceFromCtx()
func WorkspaceContextFromCtx(ctx context.Context) *WorkspaceContext
func WithWorkspaceContext(ctx context.Context, wc *WorkspaceContext) context.Context
```
All file tools use `WorkspaceContextFromCtx(ctx)` instead of separate workspace/team-workspace context keys. WorkspaceContext carries ActivePath, ReadOnlyPaths, SharedPath — one struct, one lookup.
#### Enhanced PolicyEngine
```go
// PolicyRule is a declarative access rule.
type PolicyRule struct {
// Match conditions (all must match)
Capabilities []ToolCapability // match tools with these capabilities
Groups []string // match tools in these groups
Names []string // match specific tool names
// Action
Action PolicyAction // allow or deny
Reason string // audit log reason
}
type PolicyAction string
const (
PolicyAllow PolicyAction = "allow"
PolicyDeny PolicyAction = "deny"
)
// PolicyEngine evaluates tool access via ordered rule list.
// V3 adds capability-based rules alongside existing group/name rules.
type PolicyEngine struct {
globalPolicy *config.ToolsConfig
rules []PolicyRule // declarative rules (evaluated in order, first match wins)
}
// FilterTools returns allowed tools. Now evaluates both legacy 7-step pipeline
// AND declarative rules. Declarative rules take precedence.
func (pe *PolicyEngine) FilterTools(
registry ToolExecutor,
agentID string,
providerName string,
agentToolPolicy *config.ToolPolicySpec,
groupToolAllow []string,
isSubagent bool,
isLeafAgent bool,
role string, // NEW: RBAC role ("admin", "operator", "viewer")
) []providers.ToolDefinition
```
#### Tool Definition Generation for Providers
```go
// ToolDefAdapter generates provider-specific tool definitions.
// Replaces direct ToProviderDef() calls.
type ToolDefAdapter interface {
// ToProviderDefs converts internal tool list to provider wire format.
// Handles provider-specific quirks (DashScope tool format, Codex docstrings, etc.)
ToProviderDefs(tools []Tool, caps ProviderCapabilities) []providers.ToolDefinition
}
```
#### Pipeline Integration
ToolStage in pipeline receives registry + policy engine:
```go
type ToolStage struct {
registry *Registry
policyEngine *PolicyEngine
adapter ToolDefAdapter
}
// Execute runs tool, records metrics, returns result for ObserveStage.
func (s *ToolStage) Execute(ctx context.Context, state *RunState) error
```
ObserveStage processes tool results:
```go
type ObserveStage struct {
loopDetector *LoopDetector
metricsStore EvolutionMetricsStore // record tool effectiveness
}
// Execute checks for tool loops, records metrics, decides continue/stop.
func (s *ObserveStage) Execute(ctx context.Context, state *RunState) error
```
---
### C. Skills System Upgrade
#### Hybrid Search Interface
```go
// SkillSearcher provides both lexical and semantic skill search.
type SkillSearcher interface {
// Search returns skills matching query, merging BM25 + embedding results.
// Returns at most limit results, deduplicated by slug.
Search(ctx context.Context, query string, limit int) ([]SkillSearchResult, error)
// SearchForAgent filters by ACL before search.
SearchForAgent(ctx context.Context, agentID uuid.UUID, userID string, query string, limit int) ([]SkillSearchResult, error)
}
// HybridSkillSearcher implements SkillSearcher with BM25 + embedding fusion.
type HybridSkillSearcher struct {
bm25Index *skills.Index // existing BM25 index
embSearcher store.EmbeddingSkillSearcher // existing PG embedding search
accessStore store.SkillAccessStore // ACL filtering
bm25Weight float64 // default 0.4
embedWeight float64 // default 0.6
}
```
#### Score Fusion Strategy
```
1. BM25 search → top 2*limit results, normalize scores to [0,1]
2. Embedding search → top 2*limit results, normalize scores to [0,1]
3. Merge by slug: combined_score = bm25Weight * bm25_score + embedWeight * embed_score
4. Sort by combined_score descending
5. Return top limit
```
#### ACL Model
```go
// SkillVisibility controls who can see/use a skill.
type SkillVisibility string
const (
SkillVisibilityPublic SkillVisibility = "public" // all agents in tenant
SkillVisibilityPrivate SkillVisibility = "private" // only granted agents
SkillVisibilitySystem SkillVisibility = "system" // system-managed, immutable
)
// ACL check flow:
// 1. If skill.Visibility == "public" → allow
// 2. If skill.Visibility == "private" → check skill_agent_grants table
// 3. If skill.Visibility == "system" → allow (system skills always accessible)
// Existing SkillAccessStore.ListAccessible() already implements this.
```
No new DB tables needed. Existing `managed_skills.visibility` + `skill_agent_grants` + `SkillAccessStore` interface covers ACL. The change is wiring ACL into the search path (currently BM25 search ignores ACL).
#### Dependency Graph
```go
// SkillDependency represents skill A requires skill B.
// Stored in skill frontmatter: `depends: [slug-b, slug-c]`
type SkillDependency struct {
Slug string // this skill
Requires []string // slugs of required skills
}
// DepChecker validates skill dependencies are satisfied.
// Existing internal/skills/dep_checker.go already handles this.
// V3 enhancement: block activation of skill with unresolved deps.
type DepChecker interface {
// Check returns missing dependencies for a skill.
Check(slug string, available []string) []string
// ResolveOrder returns activation order respecting dependency graph.
// Error if circular dependency detected.
ResolveOrder(slugs []string) ([]string, error)
}
```
No new DB schema. Dependencies stored in SKILL.md frontmatter `depends:` field. `dep_checker.go` already parses this. V3 adds `ResolveOrder()` for topological sort + circular detection.
#### Hot-Reload Enhancement
```go
// SkillLoader manages skill discovery, loading, and hot-reload.
// V3 adds ACL-aware loading + hybrid search integration.
type SkillLoader interface {
// LoadAll discovers and loads all skills from all tiers.
LoadAll(ctx context.Context) error
// BuildSummary returns skill summary for system prompt injection.
// V3: uses HybridSkillSearcher instead of BM25-only.
BuildSummary(ctx context.Context, agentID uuid.UUID, userID string, query string) string
// Version returns current reload version (atomic counter).
Version() int64
// OnReload registers callback for hot-reload events.
OnReload(fn func())
}
```
---
### D. Retrieval Integration
#### Auto-Inject L0 Flow
```
Per turn (in ContextStage):
1. Extract keywords from latest user message (simple tokenization, same as BM25)
2. Check relevance against memory index:
a. BM25 score against episodic summaries
b. Embedding similarity against semantic memory (KG entities)
3. If max_score > agent.retrieval_threshold (default 0.3):
→ Build L0Summaries for PromptConfig.MemoryData
→ Cap at max_l0_tokens (default 200 tokens, counted via TokenCounter)
4. PromptBuilder renders Memory section with L0 hints
5. Agent sees hints, can call memory_search(id) for L1/L2 detail
```
#### Retrieval Config
```go
// RetrievalConfig controls auto-inject behavior. Per-agent configurable.
type RetrievalConfig struct {
Enabled bool `json:"enabled"` // default true
RelevanceThreshold float64 `json:"relevance_threshold"` // default 0.3
MaxL0Tokens int `json:"max_l0_tokens"` // default 200
MaxL0Items int `json:"max_l0_items"` // default 5
BM25Weight float64 `json:"bm25_weight"` // default 0.4
EmbeddingWeight float64 `json:"embedding_weight"` // default 0.6
}
```
Stored in agent settings JSON (`agents.settings` column, existing JSONB field). No new DB columns.
#### Retrieval Interface
```go
// Retriever performs auto-inject L0 retrieval for ContextStage.
type Retriever interface {
// RetrieveL0 returns L0 summaries relevant to the query.
// Called once per turn in ContextStage.
RetrieveL0(ctx context.Context, agentID uuid.UUID, userID string, query string, cfg RetrievalConfig) ([]L0Summary, error)
}
// HybridRetriever implements Retriever using memory + KG stores.
type HybridRetriever struct {
memoryStore store.MemoryStore
kgStore store.KnowledgeGraphStore
tokenCounter TokenCounter
embProvider store.EmbeddingProvider
}
```
#### memory_search / kg_search Tool Updates
```go
// memory_search tool gains L1/L2 mode parameter.
// Args: { "query": "...", "depth": "l1"|"l2" }
// L1: returns structured overview (episodic summaries, ~200 tokens each)
// L2: returns full detail (original content, KG traversal)
// Default: "l1" (backward compatible — current behavior is L1-equivalent)
// kg_search tool gains temporal parameter.
// Args: { "query": "...", "as_of": "2026-01-01" }
// as_of: returns facts valid at that date. Omit = current facts only.
```
---
## Related Code Files
### Files to Modify
| File | Change |
|------|--------|
| `internal/agent/systemprompt.go` | Replace BuildSystemPrompt() with PromptBuilder.Build() |
| `internal/agent/systemprompt_sections.go` | Extract section data builders, remove string concat |
| `internal/agent/loop_history.go` | Use PromptBuilder in buildMessages() |
| `internal/tools/types.go` | Add Metadata() to Tool interface |
| `internal/tools/registry.go` | Store ToolMetadata, add capability queries |
| `internal/tools/policy.go` | Add capability-based rules, role parameter |
| `internal/tools/context_keys.go` | Replace dual workspace keys with WorkspaceContext |
| `internal/tools/filesystem.go` | Use WorkspaceContextFromCtx() |
| `internal/tools/filesystem_write.go` | Use WorkspaceContextFromCtx() |
| `internal/tools/filesystem_list.go` | Use WorkspaceContextFromCtx() |
| `internal/tools/shell.go` | Use WorkspaceContextFromCtx() |
| `internal/skills/loader.go` | Wire HybridSkillSearcher into BuildSummary() |
| `internal/skills/search.go` | Keep BM25, add fusion logic |
| `internal/tools/memory_search.go` | Add depth parameter (l1/l2) |
| `internal/tools/knowledge_graph_search.go` | Add as_of temporal parameter |
### Files to Create
| File | Purpose |
|------|---------|
| `internal/agent/prompt_builder.go` | TemplatePromptBuilder implementation |
| `internal/agent/prompt_template.go.tmpl` | Embedded template file (default variant) |
| `internal/agent/prompt_template_codex.go.tmpl` | Codex variant |
| `internal/agent/prompt_config_types.go` | PromptConfig, section data types |
| `internal/tools/capability.go` | ToolCapability, ToolMetadata types |
| `internal/tools/workspace_context.go` | WorkspaceContext struct + context helpers |
| `internal/tools/tool_def_adapter.go` | ToolDefAdapter interface + default impl |
| `internal/skills/hybrid_searcher.go` | HybridSkillSearcher |
| `internal/agent/retriever.go` | Retriever interface + HybridRetriever |
---
## Design Steps
1. Define PromptConfig + all section data types in `prompt_config_types.go`
2. Write default template file with all section sub-templates
3. Implement TemplatePromptBuilder with variant loading
4. Define ToolCapability + ToolMetadata types
5. Add Metadata() to Tool interface with backward-compat wrapper
6. Define WorkspaceContext struct with context helpers
7. Update PolicyEngine: add capability rules + role parameter
8. Define ToolDefAdapter for provider-specific tool def generation
9. Implement HybridSkillSearcher with score fusion
10. Wire ACL into skill search path
11. Define Retriever interface + RetrievalConfig
12. Implement HybridRetriever for L0 auto-inject
13. Update memory_search tool with depth parameter design
14. Update kg_search tool with as_of temporal parameter design
15. Write provider variant templates (Codex, DashScope)
---
## Todo List
- [x] A1: PromptConfig + section data types — `internal/agent/prompt_config_types.go`
- [x] A2: Default template file — documented in phase design
- [x] A3: TemplatePromptBuilder interface — `internal/agent/prompt_config_types.go`
- [x] A4: Provider variant templates — documented in phase design
- [x] A5: Workspace sub-template — documented in phase design
- [x] B1: ToolCapability + ToolMetadata — `internal/tools/capability.go`
- [x] B2: Backward-compat wrapper — documented in phase design
- [x] B3: WorkspaceContext + context key — done in Phase 1 (`internal/workspace/`)
- [x] B4: PolicyEngine capability rules — documented in phase design
- [x] B5: ToolDefAdapter interface — documented in phase design
- [x] C1: HybridSkillSearcher — documented in phase design
- [x] C2: ACL wiring into search path — documented in phase design
- [x] C3: DepChecker.ResolveOrder() — documented in phase design
- [x] D1: Retriever + RetrievalConfig — `internal/agent/retriever.go`
- [x] D2: L0 auto-inject flow — documented in phase design
- [x] D3: memory_search depth param — documented in phase design
- [x] D4: kg_search as_of temporal param — documented in phase design
---
## Success Criteria
1. PromptConfig can reproduce exact v2 system prompt output (regression test)
2. All template sections independently toggleable + testable
3. WorkspaceContext carries all info needed by file tools (no separate context keys)
4. Every tool declares capabilities via Metadata()
5. PolicyEngine can express "deny mutating tools for viewer role" in one rule
6. HybridSkillSearcher returns better results than BM25-only on intent queries
7. L0 auto-inject adds <200 tokens per turn, only when relevant
8. Template renders in <1ms (no LLM calls)
---
## Risk Assessment
| Risk | Severity | Mitigation |
|------|----------|------------|
| Template engine too rigid for edge cases | MEDIUM | Keep ExtraPrompt escape hatch. Provider variants handle outliers. |
| Metadata() breaks all existing tool implementations | HIGH | Backward-compat wrapper: `defaultMetadata(tool)` infers from group/name. Phase implementation. |
| Hybrid skill search slower than BM25-only | LOW | BM25 <1ms, embedding <50ms. Parallel execution. Cache embeddings. |
| WorkspaceContext migration touches every file tool | HIGH | Single context key replaces 2. Grep for `ToolWorkspaceFromCtx` — 15 call sites. Mechanical refactor. |
| L0 auto-inject adds latency to every turn | MEDIUM | Memory index query <10ms. Skip if no memory (hasMemory flag). Threshold prevents injection for low-relevance messages. |
| Provider template variants diverge over time | LOW | Variants inherit from default, override only specific blocks. Test all variants in CI. |
---
## Security Considerations
- **Template injection:** PromptConfig data (PersonaContent, InstructionContent) comes from DB. Template uses `{{ }}` not `{{ | html }}`. Since output goes to LLM (not browser), HTML escaping not needed. But ensure no Go template directives in user content — use `{{ printf "%s" .Content }}` for raw pass-through.
- **Workspace enforcement:** WorkspaceSectionData.EnforcementMsg must be generated server-side, not from user input. Template renders it verbatim.
- **ACL bypass:** HybridSkillSearcher MUST filter AFTER search, not before (embedding search doesn't support WHERE clauses). Verify: `ListAccessible()` called on search results.
- **Tool capability lying:** Tools self-declare capabilities. PolicyEngine should cross-check: tools in "fs" group claiming "read-only" but actually writing. Add assertion in tests.
plans/260406-1304-goclaw-v3-core-design/phase-05-orchestration-intelligence-design.md
-923
View File
@@ -1,923 +0,0 @@
# Phase 5: Orchestration & Intelligence Design
## Context Links
- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (sections 11, 15)
- Architecture report: `plans/reports/Explore-260406-thorough-v3-architecture.md` (section 8)
- Agent link store: `internal/store/agent_link_store.go`
- Current subagent: `internal/tools/subagent.go`, `internal/tools/subagent_exec.go`
- Current team tools: `internal/tools/team_tool.go`, `internal/tools/team_tool_backend.go`
- Event bus: `internal/bus/bus.go`, `internal/bus/types.go`
- Phase 4: `phase-04-system-integration-design.md` (WorkspaceContext, ToolCapability, PolicyEngine)
## Overview
- **Priority:** P1 (builds on Phase 4 integration layer)
- **Status:** Complete
- **Effort:** 6h design
Designs 4 interconnected systems:
1. **Delegate tool** — lightweight inter-agent delegation via agent_links
2. **Orchestration mode config** — layered: spawn always, delegate base, team optional
3. **Self-evolution** — progressive 3-stage rollout (metrics -> suggestions -> auto-adapt)
4. **EventBus integration** — orchestration + evolution events
---
## Key Insights
1. Agent Links infrastructure exists (`agent_link_store.go`) with `CanDelegate()`, `GetLinkBetween()`, `DelegateTargets()`, even `SearchDelegateTargetsByEmbedding()` — but NO tool uses it. Full plumbing ready, needs ~100 LOC tool wrapper.
2. Current orchestration: spawn (self-clone, ~300 LOC) + team_tasks (9,400 LOC, 18 actions). No middle ground. Simple delegation requires team overhead or spawn limitations.
3. Self-evolution is greenfield. No metrics infrastructure exists. Design must be additive — no existing behavior changes in Stage 1.
4. EventBus (`bus.MessageBus`) is channel-oriented (Inbound/Outbound messages). Broadcast events are typed but payloads are `any`. V3 needs structured event types for orchestration lifecycle.
---
## Requirements
### Functional
- F1: `delegate` tool — agent delegates task to linked agent, async or sync
- F2: Delegate permission checked via existing `agentLinkStore.CanDelegate()`
- F3: Delegatee uses own workspace + optional shared area + read-only exports
- F4: Orchestration modes configurable per tenant (default) + per agent (override)
- F5: System prompt only shows tools for active orchestration mode
- F6: Self-evolution Stage 1: record metrics (retrieval quality, tool effectiveness, response feedback)
- F7: Self-evolution Stage 2: analyze metrics, generate suggestions, admin review UI
- F8: Self-evolution Stage 3: auto-adapt with guardrails + rollback
- F9: All orchestration lifecycle events published via EventBus
### Non-Functional
- NF1: Delegate tool adds <5ms overhead over direct agent invocation
- NF2: Metrics recording non-blocking (goroutine, never delays response)
- NF3: Self-evolution stages independently toggleable (feature flags)
- NF4: Backward compatible: existing spawn + team_tasks unaffected
---
## Architecture
### A. Delegate Tool
#### Tool Interface
```go
// DelegateTool enables inter-agent task delegation via agent_links.
// Parameters:
// agent_key (required): target agent key (from known delegate targets)
// task (required): task description for the delegatee
// mode (optional): "async" (default) or "sync"
// context (optional): additional context string passed to delegatee
// files (optional): list of file paths to export (read-only for delegatee)
type DelegateTool struct {
agentLinkStore store.AgentLinkStore
agentStore store.AgentStore
runner AgentRunner // interface to invoke agent loop
bus bus.EventPublisher
}
func (t *DelegateTool) Name() string { return "delegate" }
func (t *DelegateTool) Metadata() ToolMetadata {
return ToolMetadata{
Name: "delegate",
Capabilities: []ToolCapability{CapAsync},
Group: "orchestration",
}
}
func (t *DelegateTool) Parameters() map[string]any {
return map[string]any{
"type": "object",
"properties": map[string]any{
"agent_key": map[string]any{
"type": "string",
"description": "Key of the agent to delegate to (from your known delegate targets)",
},
"task": map[string]any{
"type": "string",
"description": "Task description for the delegatee",
},
"mode": map[string]any{
"type": "string",
"enum": []string{"async", "sync"},
"default": "async",
"description": "async: fire-and-forget with result notification. sync: wait for completion.",
},
"context": map[string]any{
"type": "string",
"description": "Additional context passed to delegatee's system prompt",
},
"files": map[string]any{
"type": "array",
"items": map[string]any{"type": "string"},
"description": "File paths (relative to your workspace) to share read-only with delegatee",
},
},
"required": []string{"agent_key", "task"},
}
}
```
#### Execution Flow
```
delegate(agent_key="analyst", task="analyze Q1 data", mode="async")
|
v
1. Resolve target: agentStore.GetByKey(agent_key)
-> fail if agent not found or inactive
|
v
2. Permission check: agentLinkStore.CanDelegate(fromAgentID, toAgentID)
-> fail if no active link
|
v
3. Get link config: agentLinkStore.GetLinkBetween(from, to)
-> read Settings JSON for workspace config
|
v
4. Build WorkspaceContext for delegatee:
- ActivePath: delegatee's own workspace (resolved normally)
- SharedPath: /data/delegates/{linkID}/shared (created if not exists)
- ReadOnlyPaths: exported files from delegator (validated against workspace)
- Scope: "delegate"
|
v
5. Build delegation request:
- ExtraPrompt: "Delegated task from {source_agent_key}: {task}\n{context}"
- WorkspaceContext: as built above
- Mode: PromptMinimal (subagent-style, reduced sections)
|
v
6. Execute:
async mode:
- Spawn goroutine via scheduler lane
- Return immediately: "Task delegated to {agent_key}. You'll be notified on completion."
- On completion: bus.Broadcast(delegate.completed event)
- Result delivered via message bus (same as spawn announce)
sync mode:
- Block until delegatee completes (with timeout from link.Settings)
- Return delegatee's final response directly
|
v
7. Emit event: bus.Broadcast("delegate.sent", DelegateSentPayload{...})
```
#### Workspace Model for Delegation
```go
// DelegateWorkspaceConfig is stored in agent_link.settings JSON.
type DelegateWorkspaceConfig struct {
// SharedPath: shared directory both agents can read/write.
// Relative to data dir. Default: "delegates/{linkID}/shared"
SharedPath string `json:"shared_path,omitempty"`
// SourceExports: glob patterns of delegator files shared read-only.
// Relative to delegator's workspace. E.g., ["docs/", "data/results/"]
SourceExports []string `json:"source_exports,omitempty"`
// SyncTimeout: max wait time for sync mode (seconds). Default: 300
SyncTimeout int `json:"sync_timeout,omitempty"`
// MaxConcurrent: max concurrent delegations on this link. 0 = unlimited.
// Already exists as AgentLinkData.MaxConcurrent field.
}
```
No new DB tables. Config stored in existing `agent_links.settings` JSONB column.
#### AgentRunner Interface
```go
// AgentRunner abstracts invoking an agent's loop for delegation.
// Implemented by the gateway's run dispatcher.
type AgentRunner interface {
// RunDelegation executes a delegation task and returns the result.
// ctx carries workspace context, tenant scope, timeout.
RunDelegation(ctx context.Context, req DelegationRequest) (*DelegationResult, error)
}
type DelegationRequest struct {
TargetAgentID uuid.UUID
SourceAgentID uuid.UUID
LinkID uuid.UUID
Task string
Context string
WorkspaceCtx *WorkspaceContext
UserID string // inherit from delegator's user context
TenantID uuid.UUID
SyncTimeout time.Duration // 0 = async
}
type DelegationResult struct {
Response string // delegatee's final text response
ToolCalls int // number of tool calls made
Duration time.Duration
Error error
}
```
---
### B. Orchestration Mode Configuration
#### Mode Definitions
```go
// OrchestrationMode controls which inter-agent tools are available.
type OrchestrationMode string
const (
// ModeSpawn: self-clone only. spawn tool available.
// Simplest mode. Agent can only create copies of itself for background work.
ModeSpawn OrchestrationMode = "spawn"
// ModeDelegate: agent links + spawn. delegate tool available.
// Agent can send tasks to linked agents. Base inter-agent mode.
ModeDelegate OrchestrationMode = "delegate"
// ModeTeam: full team tasks + delegate + spawn. team_tasks tool available.
// Full lifecycle: review, approval, progress tracking.
// Only available when agent is member of an active team.
ModeTeam OrchestrationMode = "team"
)
```
#### Configuration Layering
```go
// OrchestrationConfig controls orchestration behavior.
type OrchestrationConfig struct {
// Mode: active orchestration mode.
// Tenant sets default, agent overrides.
Mode OrchestrationMode `json:"orchestration_mode"`
}
```
**Storage:** No new DB columns. Uses existing config mechanisms:
- Tenant default: `tenant_settings.orchestration_mode` (existing JSONB settings column on tenants table)
- Agent override: `agents.settings -> 'orchestration_mode'` (existing JSONB settings column)
**Resolution order:**
```
1. Agent settings.orchestration_mode (if set)
2. Tenant settings.orchestration_mode (if set)
3. Default: "spawn"
```
#### Tool Visibility by Mode
| Mode | spawn | delegate | team_tasks |
|------|-------|----------|------------|
| spawn | yes | no | no |
| delegate | yes | yes | no |
| team | yes | yes | yes |
**Implementation:** PolicyEngine checks orchestration mode during FilterTools:
```go
// In PolicyEngine.FilterTools(), after existing 7-step pipeline:
func (pe *PolicyEngine) applyOrchestrationMode(
allowed []string,
mode OrchestrationMode,
isTeamMember bool,
) []string {
switch mode {
case ModeSpawn:
allowed = subtractSet(allowed, []string{"delegate", "team_tasks"})
case ModeDelegate:
allowed = subtractSet(allowed, []string{"team_tasks"})
case ModeTeam:
if !isTeamMember {
// Downgrade to delegate if not in team
allowed = subtractSet(allowed, []string{"team_tasks"})
}
}
return allowed
}
```
#### System Prompt Integration
PromptConfig (from Phase 4) carries orchestration info:
```go
// Added to PromptConfig:
type OrchestrationSectionData struct {
Mode OrchestrationMode
DelegateTargets []DelegateTargetEntry // from agentLinkStore.DelegateTargets()
TeamContext *TeamSectionData // only if ModeTeam + team member
}
type DelegateTargetEntry struct {
AgentKey string
DisplayName string
Description string
}
```
Template renders delegate targets when mode is delegate/team:
```
{{- define "orchestration" -}}
{{- if eq .Mode "delegate" "team" }}
# Delegation
You can delegate tasks to these agents:
{{- range .DelegateTargets }}
- **{{ .AgentKey }}**: {{ .Description }}
{{- end }}
Use `delegate(agent_key, task)` to send work.
{{- end }}
{{- end -}}
```
---
### C. Self-Evolution Design (Progressive)
#### Stage 1: Metrics Infrastructure (v3.0)
##### DB Schema
```sql
-- New table: agent_evolution_metrics
CREATE TABLE agent_evolution_metrics (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL REFERENCES tenants(id),
agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
session_key TEXT NOT NULL,
metric_type TEXT NOT NULL, -- 'retrieval', 'tool', 'feedback'
metric_key TEXT NOT NULL, -- specific metric (e.g., tool name, retrieval query)
value JSONB NOT NULL, -- metric-specific payload
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Indexes for efficient querying
CREATE INDEX idx_evo_metrics_agent_type ON agent_evolution_metrics(agent_id, metric_type);
CREATE INDEX idx_evo_metrics_created ON agent_evolution_metrics(created_at);
CREATE INDEX idx_evo_metrics_tenant ON agent_evolution_metrics(tenant_id);
-- Partition-ready: created_at is in schema for future range partitioning
-- For now, TTL cleanup via cron: DELETE WHERE created_at < NOW() - INTERVAL '90 days'
```
##### Metric Types
```go
// MetricType constants
const (
MetricRetrieval MetricType = "retrieval"
MetricTool MetricType = "tool"
MetricFeedback MetricType = "feedback"
)
// RetrievalMetricValue records per-retrieval quality.
type RetrievalMetricValue struct {
Query string `json:"query"`
ResultCount int `json:"result_count"`
TopScore float64 `json:"top_score"`
UsedInReply bool `json:"used_in_reply"` // heuristic: result text appears in LLM output
Source string `json:"source"` // "memory_search", "kg_search", "auto_inject"
}
// ToolMetricValue records per-tool-call effectiveness.
type ToolMetricValue struct {
ToolName string `json:"tool_name"`
Success bool `json:"success"` // !result.IsError
Duration time.Duration `json:"duration_ms"`
ResultSize int `json:"result_size"` // bytes
Capability string `json:"capability"` // from ToolMetadata
}
// FeedbackMetricValue records implicit satisfaction signal.
type FeedbackMetricValue struct {
Signal string `json:"signal"` // "correction", "followup", "thanks", "new_topic"
TurnGap int `json:"turn_gap"` // turns between response and signal
PreviousTool string `json:"previous_tool"` // last tool used before signal
}
```
##### Heuristic: "Was retrieval result used?"
```
After LLM response:
1. Collect all retrieval results from this turn (memory_search, auto-inject L0)
2. For each result, check if any 3+ word substring appears in LLM output
3. Mark used_in_reply = true if found
4. Record metric with used_in_reply flag
```
Simple substring check. Not perfect but sufficient for Stage 1 metrics. No LLM call needed.
##### Implicit Satisfaction Signals
```
After user's next message:
1. If message is a correction ("no", "that's wrong", "I meant..."): signal = "correction"
2. If message continues topic (similar embedding): signal = "followup"
3. If message changes topic entirely: signal = "new_topic"
4. If message is appreciation ("thanks", "perfect"): signal = "thanks"
5. Record feedback metric with previous turn's tool context
```
Classification via keyword matching + embedding similarity. No LLM call.
##### Metrics Store Interface
```go
// EvolutionMetricsStore manages self-evolution metrics.
type EvolutionMetricsStore interface {
// RecordMetric stores a single metric (non-blocking, goroutine-safe).
RecordMetric(ctx context.Context, metric EvolutionMetric) error
// QueryMetrics returns metrics for analysis.
QueryMetrics(ctx context.Context, agentID uuid.UUID, metricType MetricType, since time.Time, limit int) ([]EvolutionMetric, error)
// AggregateToolMetrics returns per-tool success rate, avg duration, call count.
AggregateToolMetrics(ctx context.Context, agentID uuid.UUID, since time.Time) ([]ToolAggregate, error)
// AggregateRetrievalMetrics returns per-source usage rate.
AggregateRetrievalMetrics(ctx context.Context, agentID uuid.UUID, since time.Time) ([]RetrievalAggregate, error)
// Cleanup removes metrics older than TTL.
Cleanup(ctx context.Context, olderThan time.Time) (int64, error)
}
type EvolutionMetric struct {
ID uuid.UUID
TenantID uuid.UUID
AgentID uuid.UUID
SessionKey string
MetricType MetricType
MetricKey string
Value json.RawMessage // serialized *MetricValue
CreatedAt time.Time
}
type ToolAggregate struct {
ToolName string
CallCount int
SuccessRate float64
AvgDuration time.Duration
}
type RetrievalAggregate struct {
Source string // "memory_search", "kg_search", "auto_inject"
QueryCount int
UsageRate float64 // fraction of results used in reply
AvgScore float64
}
```
##### Pipeline Integration
Metrics recorded in ObserveStage (Phase 2 pipeline):
```go
// In ObserveStage.Execute():
// After tool result processing:
if toolResult != nil {
go metricsStore.RecordMetric(ctx, EvolutionMetric{
MetricType: MetricTool,
MetricKey: toolResult.ToolName,
Value: marshalToolMetric(toolResult),
})
}
// After retrieval in ContextStage:
if len(l0Results) > 0 {
go metricsStore.RecordMetric(ctx, EvolutionMetric{
MetricType: MetricRetrieval,
MetricKey: "auto_inject",
Value: marshalRetrievalMetric(query, l0Results),
})
}
```
Always non-blocking (`go` goroutine). Metrics are best-effort — dropped on error, never block agent loop.
#### Stage 2: Suggestion Engine (v3.1)
##### DB Schema
```sql
-- New table: agent_evolution_suggestions
CREATE TABLE agent_evolution_suggestions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL REFERENCES tenants(id),
agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
suggestion_type TEXT NOT NULL, -- 'threshold', 'tool_order', 'skill_add', 'memory_prune'
suggestion TEXT NOT NULL, -- human-readable suggestion
rationale TEXT NOT NULL, -- data-driven reasoning
parameters JSONB, -- machine-readable params for auto-apply
status TEXT NOT NULL DEFAULT 'pending', -- 'pending', 'approved', 'rejected', 'applied', 'rolled_back'
reviewed_by TEXT, -- admin user who reviewed
reviewed_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_evo_suggestions_agent ON agent_evolution_suggestions(agent_id, status);
CREATE INDEX idx_evo_suggestions_tenant ON agent_evolution_suggestions(tenant_id);
```
##### Suggestion Types
```go
type SuggestionType string
const (
SuggestThreshold SuggestionType = "threshold" // adjust retrieval threshold
SuggestToolOrder SuggestionType = "tool_order" // reorder tool list for this agent
SuggestSkillAdd SuggestionType = "skill_add" // recommend adding a skill
SuggestMemoryPrune SuggestionType = "memory_prune" // prune low-value memory
)
// SuggestionParams is the machine-readable payload for auto-apply.
// Each SuggestionType has its own params structure.
type ThresholdParams struct {
Param string `json:"param"` // "retrieval_threshold", "l0_max_tokens"
OldValue float64 `json:"old_value"`
NewValue float64 `json:"new_value"`
}
type ToolOrderParams struct {
ToolName string `json:"tool_name"`
Action string `json:"action"` // "promote", "demote", "suggest_removal"
Reasoning string `json:"reasoning"`
}
```
##### Analysis Rules (Examples)
```
Rule 1: Low retrieval usage
IF retrieval.usage_rate < 0.2 over 7 days AND query_count > 50
THEN suggest: "Raise relevance_threshold from {current} to {current+0.1}"
REASON: "Only 20% of auto-injected memory results appear in responses"
Rule 2: Tool never succeeds
IF tool.success_rate < 0.1 over 7 days AND call_count > 20
THEN suggest: "Consider removing {tool_name} or fixing configuration"
REASON: "{tool_name} failed 90% of {call_count} calls"
Rule 3: Repeated tool usage
IF tool.call_count > 100/week for specific tool AND no skill exists
THEN suggest: "Add skill for {common_task_pattern}"
REASON: "{tool_name} called {count} times this week, mostly for {pattern}"
Rule 4: Memory bloat
IF episodic_count > 500 AND avg_usage_rate < 0.05
THEN suggest: "Prune episodic memories older than 60 days"
REASON: "95% of older episodic summaries never retrieved"
```
##### Suggestion Store Interface
```go
type EvolutionSuggestionStore interface {
CreateSuggestion(ctx context.Context, s EvolutionSuggestion) error
ListSuggestions(ctx context.Context, agentID uuid.UUID, status string, limit int) ([]EvolutionSuggestion, error)
UpdateSuggestionStatus(ctx context.Context, id uuid.UUID, status string, reviewedBy string) error
GetSuggestion(ctx context.Context, id uuid.UUID) (*EvolutionSuggestion, error)
}
type EvolutionSuggestion struct {
ID uuid.UUID
TenantID uuid.UUID
AgentID uuid.UUID
SuggestionType SuggestionType
Suggestion string
Rationale string
Parameters json.RawMessage
Status string
ReviewedBy string
ReviewedAt *time.Time
CreatedAt time.Time
}
```
##### Suggestion Engine Interface
```go
// SuggestionEngine analyzes metrics and generates suggestions.
// Runs as periodic cron job (configurable interval, default: daily).
type SuggestionEngine interface {
// Analyze runs all rules for an agent and creates pending suggestions.
Analyze(ctx context.Context, agentID uuid.UUID) ([]EvolutionSuggestion, error)
// AnalyzeAll runs analysis for all agents in tenant.
AnalyzeAll(ctx context.Context, tenantID uuid.UUID) error
}
// RuleSuggestionEngine implements SuggestionEngine with configurable rules.
type RuleSuggestionEngine struct {
metricsStore EvolutionMetricsStore
suggestionStore EvolutionSuggestionStore
rules []AnalysisRule
}
// AnalysisRule is a single suggestion rule.
type AnalysisRule interface {
// Evaluate checks if the rule fires for the given agent metrics.
// Returns nil if rule doesn't apply.
Evaluate(ctx context.Context, agentID uuid.UUID, metrics AnalysisInput) (*EvolutionSuggestion, error)
}
type AnalysisInput struct {
ToolAggregates []ToolAggregate
RetrievalAggregates []RetrievalAggregate
FeedbackSummary FeedbackSummary
AnalysisPeriod time.Duration // lookback window (default 7 days)
}
```
#### Stage 3: Auto-Adaptation (v3.2+)
##### Guardrails
```go
// AdaptationGuardrails control auto-adaptation limits.
type AdaptationGuardrails struct {
// MaxDeltaPerCycle: max parameter change per adaptation cycle.
// E.g., threshold can only change by +/- 0.1 per cycle.
MaxDeltaPerCycle float64 `json:"max_delta_per_cycle"`
// MinDataPoints: minimum metrics before adaptation allowed.
MinDataPoints int `json:"min_data_points"`
// RollbackOnDrop: if quality metric drops by this %, rollback.
RollbackOnDrop float64 `json:"rollback_on_drop_pct"` // e.g., 20.0 = 20%
// AdminOverride: admin can lock parameters (prevent auto-change).
LockedParams []string `json:"locked_params,omitempty"`
}
// Default guardrails
var DefaultGuardrails = AdaptationGuardrails{
MaxDeltaPerCycle: 0.1,
MinDataPoints: 100,
RollbackOnDrop: 20.0,
}
```
##### Adaptation Flow
```
1. SuggestionEngine.Analyze() produces suggestion with status="pending"
2. If auto_adapt enabled for this agent:
a. Check guardrails (min data points, max delta, locked params)
b. If passes: apply parameters, set status="applied"
c. Record baseline quality metric
3. After 7 days: compare quality metric to baseline
a. If quality dropped > RollbackOnDrop%: rollback, set status="rolled_back"
b. If quality stable/improved: keep, emit event
4. Admin can always: approve pending, reject, rollback applied, lock params
```
##### Auto-Adapt Store (Extension of Suggestion Store)
```go
// Added to EvolutionSuggestionStore for Stage 3:
type EvolutionAdaptStore interface {
EvolutionSuggestionStore
// ApplySuggestion applies a suggestion's parameters to agent config.
ApplySuggestion(ctx context.Context, suggestionID uuid.UUID) error
// RollbackSuggestion reverts an applied suggestion.
RollbackSuggestion(ctx context.Context, suggestionID uuid.UUID) error
// GetAdaptationHistory returns applied/rolled-back suggestions for audit.
GetAdaptationHistory(ctx context.Context, agentID uuid.UUID, limit int) ([]EvolutionSuggestion, error)
}
```
##### Feature Flags
```go
// Feature flags in agent settings JSON (agents.settings column)
type SelfEvolutionFlags struct {
MetricsEnabled bool `json:"self_evolution_metrics"` // Stage 1 (default: true for v3)
SuggestionsEnabled bool `json:"self_evolution_suggestions"` // Stage 2 (default: false)
AutoAdaptEnabled bool `json:"self_evolution_auto_adapt"` // Stage 3 (default: false)
}
```
No new DB columns. Flags stored in existing `agents.settings` JSONB.
---
### D. EventBus Integration
#### New Event Topics
```go
// Orchestration events
const (
TopicDelegateSent = "delegate:sent"
TopicDelegateCompleted = "delegate:completed"
TopicDelegateFailed = "delegate:failed"
)
// Self-evolution events
const (
TopicMetricRecorded = "evolution:metric"
TopicSuggestionCreated = "evolution:suggestion"
TopicAdaptationApplied = "evolution:adapted"
TopicAdaptationRolledBack = "evolution:rollback"
)
```
#### Typed Event Payloads
```go
// DelegateSentPayload emitted when delegate tool fires.
type DelegateSentPayload struct {
SourceAgentID uuid.UUID `json:"source_agent_id"`
TargetAgentID uuid.UUID `json:"target_agent_id"`
LinkID uuid.UUID `json:"link_id"`
Task string `json:"task"`
Mode string `json:"mode"` // "async" or "sync"
SessionKey string `json:"session_key"`
}
// DelegateCompletedPayload emitted when delegatee finishes.
type DelegateCompletedPayload struct {
SourceAgentID uuid.UUID `json:"source_agent_id"`
TargetAgentID uuid.UUID `json:"target_agent_id"`
LinkID uuid.UUID `json:"link_id"`
Duration time.Duration `json:"duration_ms"`
ToolCalls int `json:"tool_calls"`
Success bool `json:"success"`
Error string `json:"error,omitempty"`
}
// DelegateFailedPayload emitted on delegation failure.
type DelegateFailedPayload struct {
SourceAgentID uuid.UUID `json:"source_agent_id"`
TargetAgentID uuid.UUID `json:"target_agent_id"`
LinkID uuid.UUID `json:"link_id"`
Reason string `json:"reason"` // "permission_denied", "agent_inactive", "timeout"
}
// MetricRecordedPayload emitted on each metric write.
type MetricRecordedPayload struct {
AgentID uuid.UUID `json:"agent_id"`
MetricType MetricType `json:"metric_type"`
MetricKey string `json:"metric_key"`
}
// SuggestionCreatedPayload emitted when suggestion engine generates one.
type SuggestionCreatedPayload struct {
AgentID uuid.UUID `json:"agent_id"`
SuggestionID uuid.UUID `json:"suggestion_id"`
SuggestionType SuggestionType `json:"suggestion_type"`
Summary string `json:"summary"`
}
```
#### Event Subscribers
```go
// Registered at gateway startup:
// 1. Delegation result delivery — notifies source agent when async delegation completes
bus.Subscribe("delegate-result-delivery", func(e bus.Event) {
if e.Name == TopicDelegateCompleted {
// Deliver result to source agent's session via sessions_send mechanism
}
})
// 2. Metrics recording — non-blocking metric persistence
bus.Subscribe("evolution-metrics-recorder", func(e bus.Event) {
if e.Name == TopicMetricRecorded {
// Already persisted by goroutine in ObserveStage.
// Subscriber only for UI real-time updates (WebSocket broadcast).
}
})
// 3. Suggestion notification — notify admin of new suggestions
bus.Subscribe("evolution-suggestion-notify", func(e bus.Event) {
if e.Name == TopicSuggestionCreated {
// Broadcast to WebSocket for admin UI notification
}
})
```
---
## Related Code Files
### Files to Modify
| File | Change |
|------|--------|
| `internal/bus/types.go` | Add orchestration + evolution topic constants + payload types |
| `internal/tools/policy.go` | Add applyOrchestrationMode() to FilterTools |
| `internal/agent/loop.go` | Wire metrics recording in observe phase |
| `internal/agent/systemprompt.go` | Add orchestration section to prompt config |
| `internal/store/agent_link_store.go` | (no change — already has full interface) |
| `cmd/gateway.go` | Register delegate tool, event subscribers, metrics cron |
### Files to Create
| File | Purpose |
|------|---------|
| `internal/tools/delegate_tool.go` | DelegateTool implementation |
| `internal/tools/delegate_workspace.go` | DelegateWorkspaceConfig, workspace setup |
| `internal/agent/orchestration_mode.go` | OrchestrationMode types, resolution logic |
| `internal/agent/agent_runner.go` | AgentRunner interface for delegation dispatch |
| `internal/store/evolution_store.go` | EvolutionMetricsStore + EvolutionSuggestionStore interfaces |
| `internal/store/pg/evolution_metrics.go` | PG implementation of EvolutionMetricsStore |
| `internal/store/pg/evolution_suggestions.go` | PG implementation of EvolutionSuggestionStore |
| `internal/agent/evolution_metrics.go` | Metric types + recording helpers |
| `internal/agent/evolution_suggestion_engine.go` | SuggestionEngine + AnalysisRule implementations |
| `internal/agent/evolution_guardrails.go` | AdaptationGuardrails + auto-adapt logic |
---
## Design Steps
1. Define DelegateTool interface + parameters schema
2. Design delegate execution flow (async + sync modes)
3. Define DelegateWorkspaceConfig in agent_link.settings
4. Define AgentRunner interface for delegation dispatch
5. Define OrchestrationMode enum + resolution logic (tenant -> agent)
6. Design tool visibility filtering by orchestration mode
7. Design orchestration section for system prompt template
8. Define agent_evolution_metrics schema + indexes
9. Define metric types (retrieval, tool, feedback) with value structs
10. Define EvolutionMetricsStore interface + aggregate methods
11. Design metrics recording integration in ObserveStage
12. Define agent_evolution_suggestions schema
13. Define SuggestionEngine + AnalysisRule interfaces
14. Design analysis rules (4 initial rules)
15. Define AdaptationGuardrails + auto-adapt flow
16. Define all EventBus topics + typed payloads
17. Design event subscriber wiring at startup
---
## Todo List
- [ ] A1: DelegateTool interface + parameters
- [ ] A2: Delegate execution flow (async/sync)
- [ ] A3: DelegateWorkspaceConfig schema
- [ ] A4: AgentRunner interface
- [ ] A5: Permission flow (CanDelegate + GetLinkBetween)
- [ ] B1: OrchestrationMode enum + constants
- [ ] B2: Mode resolution (tenant default + agent override)
- [ ] B3: Tool visibility by mode in PolicyEngine
- [ ] B4: Orchestration section in prompt template
- [ ] C1: agent_evolution_metrics DDL + indexes
- [ ] C2: Metric value types (retrieval, tool, feedback)
- [ ] C3: EvolutionMetricsStore interface
- [ ] C4: Retrieval "used in reply" heuristic
- [ ] C5: Implicit satisfaction signal classification
- [ ] C6: agent_evolution_suggestions DDL
- [ ] C7: SuggestionEngine + AnalysisRule interfaces
- [ ] C8: Initial 4 analysis rules
- [ ] C9: AdaptationGuardrails + auto-adapt flow
- [ ] C10: Feature flags in agent settings
- [ ] D1: EventBus topic constants
- [ ] D2: Typed event payloads (delegate, evolution)
- [ ] D3: Event subscriber design (result delivery, notifications)
---
## Success Criteria
1. DelegateTool ~100 LOC (excluding workspace setup), uses existing agentLinkStore.CanDelegate()
2. Orchestration mode resolvable from tenant + agent settings with no new DB columns
3. PolicyEngine correctly filters tools by mode (spawn/delegate/team)
4. agent_evolution_metrics table handles 10K+ rows/day without index degradation
5. Metrics recording never blocks agent loop (goroutine + best-effort)
6. SuggestionEngine rules are additive (new rule = implement AnalysisRule interface)
7. All orchestration events typed (no `map[string]any` payloads)
8. Feature flags independently toggleable per agent
---
## Risk Assessment
| Risk | Severity | Mitigation |
|------|----------|------------|
| Delegate tool workspace isolation leaks | HIGH | ReadOnlyPaths enforced by same resolvePath() as all file tools. SharedPath created with restricted permissions. |
| Sync delegation timeout blocks caller | MEDIUM | Configurable timeout per link (default 300s). Timeout returns error, does not kill delegatee. |
| Metrics table grows unbounded | MEDIUM | TTL cleanup cron (default 90 days). Partitioning-ready schema (created_at). Monitor table size. |
| Suggestion engine generates noise | LOW | Minimum data points threshold (100). Suggestions require admin review (Stage 2). Auto-adapt has guardrails (Stage 3). |
| Orchestration mode config not discoverable | LOW | Admin UI shows current mode per agent. CLI: goclaw agent info shows mode. |
| Self-evolution "learns wrong" | HIGH | Progressive rollout. Stage 1 = metrics only (no behavior change). Stage 2 = human review. Stage 3 = guardrails + rollback. |
| EventBus topic explosion | LOW | Strict naming convention. Only add topics with subscribers. No orphan topics. |
---
## Security Considerations
- **Delegation permission:** `CanDelegate()` is the sole gate. No bypass via direct tool call — delegate tool always calls CanDelegate(). Link status "disabled" blocks delegation.
- **Cross-tenant delegation:** Agent links are tenant-scoped (both agents must be in same tenant). `agentLinkStore` queries filter by tenant_id from context.
- **Delegate workspace isolation:** Delegatee cannot access delegator's full workspace. Only explicitly exported paths (SourceExports) are shared read-only. SharedPath is a new directory, not inside either workspace.
- **Metrics data sensitivity:** Metrics contain query text and tool names, not user data or PII. Scoped by tenant_id. Cleanup cron prevents accumulation.
- **Auto-adapt scope:** Stage 3 only modifies retrieval parameters (thresholds, weights). Cannot modify tool access, workspace rules, or security settings. Admin can lock any parameter.
plans/260406-1304-goclaw-v3-core-design/phase-06-migration-audit.md
-683
View File
@@ -1,683 +0,0 @@
# Phase 6: Migration & Audit
## Context Links
- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md` (sections 16, 17)
- Architecture report: `plans/reports/Explore-260406-thorough-v3-architecture.md` (all sections)
- Workspace forcing: `plans/reports/Explore-260406-workspace-forcing.md`
- Phase 1-5 designs: all prior phase files in this plan
- Current migrations: `migrations/000001_init_schema.up.sql` through `migrations/000036_secure_cli_agent_grants.up.sql`
- Schema version: `internal/upgrade/version.go` (RequiredSchemaVersion = 36)
## Overview
- **Priority:** P1 (blocks implementation start)
- **Status:** Complete
- **Effort:** 4h design
Final design phase — comprehensive migration plan, affected file inventory, test matrix, and audit checklist. Ensures no implementation begins without full impact analysis.
---
## Key Insights
1. Current RequiredSchemaVersion = 36 (migration 000036). V3 adds 3 new tables + 2 ALTER TABLE. Next migration = 000037.
2. KG tables (000013) lack temporal columns. Adding valid_from/valid_until via ALTER is non-breaking — NULL defaults work.
3. Memory system is hybrid: daily .md files in filesystem + memory_documents/memory_chunks in PG. V3 must ingest filesystem files into new episodic_summaries table.
4. Agent settings already uses JSONB — all new per-agent configs (retrieval thresholds, orchestration mode, self-evolution flags, dedup thresholds) fit without schema changes.
5. No SQLite changes in v3 — design is PG-first per brainstorm decision #17. SQLite compat deferred.
---
## Requirements
### Functional
- F1: New tables created via standard golang-migrate files
- F2: ALTER TABLE for KG temporal columns with safe defaults
- F3: Lazy data migration: v2 data auto-converted on first access
- F4: Feature flags per agent for gradual v3 rollout
- F5: Full affected file inventory with dependency order
- F6: Integration test plan covering all 6 workspace scenarios
- F7: Rollback path for every schema change
### Non-Functional
- NF1: Migration runs <30s on databases with 100K entities
- NF2: No downtime required — all changes additive (no DROP/RENAME)
- NF3: v2 agents continue working unchanged until explicitly migrated
- NF4: Test suite runnable in CI (<5 min)
---
## Architecture
### A. DB Schema Migration
#### Migration 000037: v3 memory + evolution tables
```sql
-- 000037_v3_memory_evolution.up.sql
-- Episodic summaries (Tier 2 memory)
CREATE TABLE episodic_summaries (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL REFERENCES tenants(id),
agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
user_id VARCHAR(255) NOT NULL DEFAULT '',
session_key TEXT NOT NULL,
summary TEXT NOT NULL, -- LLM-generated session summary
key_topics TEXT[] DEFAULT '{}', -- extracted topic keywords
embedding vector(1536), -- for similarity search
source_type TEXT NOT NULL DEFAULT 'session', -- 'session', 'v2_daily', 'manual'
source_id TEXT, -- session key or v2 file path (dedup guard)
turn_count INT NOT NULL DEFAULT 0, -- number of turns summarized
token_count INT NOT NULL DEFAULT 0, -- estimated tokens in original
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expires_at TIMESTAMPTZ -- NULL = no expiry. TTL configurable per agent.
);
CREATE INDEX idx_episodic_agent_user ON episodic_summaries(agent_id, user_id);
CREATE INDEX idx_episodic_tenant ON episodic_summaries(tenant_id);
CREATE INDEX idx_episodic_source ON episodic_summaries(agent_id, source_id);
CREATE INDEX idx_episodic_tsv ON episodic_summaries USING GIN(to_tsvector('simple', summary));
CREATE INDEX idx_episodic_vec ON episodic_summaries USING hnsw(embedding vector_cosine_ops);
CREATE INDEX idx_episodic_expires ON episodic_summaries(expires_at) WHERE expires_at IS NOT NULL;
-- Evolution metrics (Stage 1 self-evolution)
CREATE TABLE agent_evolution_metrics (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL REFERENCES tenants(id),
agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
session_key TEXT NOT NULL,
metric_type TEXT NOT NULL, -- 'retrieval', 'tool', 'feedback'
metric_key TEXT NOT NULL, -- tool name, retrieval source, signal type
value JSONB NOT NULL, -- metric-specific payload
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_evo_metrics_agent_type ON agent_evolution_metrics(agent_id, metric_type);
CREATE INDEX idx_evo_metrics_created ON agent_evolution_metrics(created_at);
CREATE INDEX idx_evo_metrics_tenant ON agent_evolution_metrics(tenant_id);
-- Evolution suggestions (Stage 2 self-evolution)
CREATE TABLE agent_evolution_suggestions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL REFERENCES tenants(id),
agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
suggestion_type TEXT NOT NULL, -- 'threshold', 'tool_order', 'skill_add', 'memory_prune'
suggestion TEXT NOT NULL,
rationale TEXT NOT NULL,
parameters JSONB, -- machine-readable for auto-apply
status TEXT NOT NULL DEFAULT 'pending', -- 'pending','approved','rejected','applied','rolled_back'
reviewed_by TEXT,
reviewed_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_evo_suggestions_agent ON agent_evolution_suggestions(agent_id, status);
CREATE INDEX idx_evo_suggestions_tenant ON agent_evolution_suggestions(tenant_id);
```
**Down migration:**
```sql
-- 000037_v3_memory_evolution.down.sql
DROP TABLE IF EXISTS agent_evolution_suggestions;
DROP TABLE IF EXISTS agent_evolution_metrics;
DROP TABLE IF EXISTS episodic_summaries;
```
#### Migration 000038: KG temporal columns
```sql
-- 000038_kg_temporal_columns.up.sql
-- Add temporal validity windows to KG entities and relations
ALTER TABLE kg_entities ADD COLUMN IF NOT EXISTS valid_from TIMESTAMPTZ DEFAULT NOW();
ALTER TABLE kg_entities ADD COLUMN IF NOT EXISTS valid_until TIMESTAMPTZ; -- NULL = current/valid
ALTER TABLE kg_relations ADD COLUMN IF NOT EXISTS valid_from TIMESTAMPTZ DEFAULT NOW();
ALTER TABLE kg_relations ADD COLUMN IF NOT EXISTS valid_until TIMESTAMPTZ;
-- Index for temporal queries: "current facts" (most common query)
CREATE INDEX idx_kg_entities_current ON kg_entities(agent_id, user_id)
WHERE valid_until IS NULL;
-- Index for historical queries: "facts valid at time T"
CREATE INDEX idx_kg_entities_temporal ON kg_entities(agent_id, user_id, valid_from, valid_until);
CREATE INDEX idx_kg_relations_current ON kg_relations(agent_id, user_id)
WHERE valid_until IS NULL;
CREATE INDEX idx_kg_relations_temporal ON kg_relations(agent_id, user_id, valid_from, valid_until);
```
**Down migration:**
```sql
-- 000038_kg_temporal_columns.down.sql
DROP INDEX IF EXISTS idx_kg_relations_temporal;
DROP INDEX IF EXISTS idx_kg_relations_current;
DROP INDEX IF EXISTS idx_kg_entities_temporal;
DROP INDEX IF EXISTS idx_kg_entities_current;
ALTER TABLE kg_relations DROP COLUMN IF EXISTS valid_until;
ALTER TABLE kg_relations DROP COLUMN IF EXISTS valid_from;
ALTER TABLE kg_entities DROP COLUMN IF EXISTS valid_until;
ALTER TABLE kg_entities DROP COLUMN IF EXISTS valid_from;
```
#### RequiredSchemaVersion Bump
```go
// internal/upgrade/version.go
const RequiredSchemaVersion uint = 38
```
---
### B. Data Migration Strategy (Dual-Mode)
#### Principle: Lazy Migration on First Access
V3 reads v2 format on first access, auto-migrates in background. No bulk migration script. No downtime.
#### V2 Memory Files -> Episodic Summaries
```
Trigger: Agent with v3_memory_enabled=true, first session start
Flow:
1. Check if agent has unmigrated v2 daily files:
- List files matching memory/YYYY-MM-DD.md in workspace
- Check episodic_summaries for source_id matching file path
- If no matching episodic record: file is unmigrated
2. For each unmigrated file:
a. Read file content
b. Insert episodic_summaries row:
- summary: file content (already summarized by v2 flush)
- source_type: "v2_daily"
- source_id: file path (dedup guard)
- key_topics: extract from content (simple keyword extraction)
- turn_count: 0 (unknown for v2)
- token_count: len(content) / 4 (estimate)
c. Generate embedding in background (non-blocking)
3. Mark migration complete in agent settings: v2_memory_migrated=true
```
#### V2 KG Entities -> Temporal
```
Trigger: Migration 000038 runs (ALTER TABLE with defaults)
Flow:
1. ALTER TABLE adds valid_from DEFAULT NOW() — new rows get current timestamp
2. Existing rows: valid_from = NOW() at migration time (not created_at)
3. Optional backfill (non-blocking, cron job):
UPDATE kg_entities SET valid_from = created_at WHERE valid_from > created_at + INTERVAL '1 second';
UPDATE kg_relations SET valid_from = created_at WHERE valid_from > created_at + INTERVAL '1 second';
4. valid_until remains NULL for all existing entities (all considered "current")
```
> **Decision:** Default valid_from to NOW() at migration time, not created_at. Backfill is optional — historical accuracy is nice-to-have, not required. Existing entities are "currently valid" regardless of when they were created.
#### V2 Sessions -> V3 Sessions
No migration needed. V3 pipeline operates on same session data. Sessions use same `sessions` table, same `SessionData` struct. V3 adds pipeline stages that wrap existing behavior.
#### Feature Flags
All stored in existing `agents.settings` JSONB column:
```json
{
"v3_pipeline_enabled": false,
"v3_memory_enabled": false,
"v3_retrieval_enabled": false,
"self_evolution_metrics": false,
"self_evolution_suggestions": false,
"self_evolution_auto_adapt": false,
"v2_memory_migrated": false,
"retrieval_threshold": 0.3,
"dedup_auto_merge_threshold": 0.98,
"dedup_flag_threshold": 0.90,
"orchestration_mode": "spawn"
}
```
**Rollout plan:**
1. Deploy v3 binary with all flags false (default) — no behavior change
2. Enable v3_memory_enabled per agent — triggers lazy v2 file migration
3. Enable v3_pipeline_enabled per agent — switches to Stage-based pipeline
4. Enable v3_retrieval_enabled per agent — activates L0 auto-inject
5. Enable self_evolution_metrics per agent — starts recording
6. Enable at tenant level for bulk rollout
---
### C. Code Migration Plan — Affected Files Inventory
#### Module 1: Foundation (Phase 1)
| File | Change Type | Dependency |
|------|-------------|------------|
| `internal/agent/loop_types.go` | MODIFY: split RunState into substates | None |
| `internal/agent/loop_compact.go` | MODIFY: replace chars/4 with TokenCounter | TokenCounter |
| `internal/agent/pruning.go` | MODIFY: use TokenCounter for thresholds | TokenCounter |
| `internal/tools/context_keys.go` | MODIFY: replace dual workspace keys with WorkspaceContext | None |
| `internal/tools/workspace_resolver.go` | MODIFY: return WorkspaceContext instead of string | None |
| `internal/tools/workspace_dir.go` | MODIFY: use WorkspaceContext | WorkspaceContext |
| `internal/bus/types.go` | MODIFY: add new event topics + typed payloads | None |
| `internal/bus/bus.go` | No change (existing Broadcast mechanism sufficient) | None |
| `internal/providers/types.go` | MODIFY: add ProviderCapabilities struct | None |
| `internal/providers/anthropic.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter |
| `internal/providers/openai.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter |
| `internal/providers/dashscope.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter |
| `internal/providers/codex.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter |
| `internal/providers/claude_cli.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter |
| `internal/providers/acp_provider.go` | MODIFY: implement ProviderAdapter interface | ProviderAdapter |
**New files:**
| File | Purpose |
|------|---------|
| `internal/agent/token_counter.go` | TokenCounter interface + tiktoken impl |
| `internal/tools/workspace_context.go` | WorkspaceContext struct + helpers |
| `internal/providers/adapter.go` | ProviderAdapter interface |
| `internal/providers/capabilities.go` | ProviderCapabilities struct |
#### Module 2: Pipeline (Phase 2)
| File | Change Type | Dependency |
|------|-------------|------------|
| `internal/agent/loop.go` | MAJOR: refactor into pipeline stages | Phase 1 foundation |
| `internal/agent/loop_run.go` | MODIFY: pipeline entry point | Pipeline stages |
| `internal/agent/loop_context.go` | MODIFY: becomes ContextStage | WorkspaceContext |
| `internal/agent/loop_history.go` | MODIFY: becomes part of ContextStage | PromptBuilder |
| `internal/agent/loop_tools.go` | MODIFY: becomes ToolStage | ToolRegistry |
| `internal/agent/toolloop.go` | MODIFY: integrated into ToolStage | ToolStage |
| `internal/agent/loop_compact.go` | MODIFY: becomes PruneStage | TokenCounter |
| `internal/agent/loop_finalize.go` | MODIFY: becomes FinalizeStage | EventBus |
| `internal/agent/loop_tracing.go` | MODIFY: per-stage tracing | Pipeline stages |
**New files:**
| File | Purpose |
|------|---------|
| `internal/agent/pipeline.go` | Pipeline, Stage interface, runner |
| `internal/agent/stage_context.go` | ContextStage |
| `internal/agent/stage_think.go` | ThinkStage (LLM call) |
| `internal/agent/stage_prune.go` | PruneStage (token counting + pruning) |
| `internal/agent/stage_tool.go` | ToolStage (tool execution) |
| `internal/agent/stage_observe.go` | ObserveStage (loop detection, metrics) |
| `internal/agent/stage_checkpoint.go` | CheckpointStage (session save) |
#### Module 3: Memory & KG (Phase 3)
| File | Change Type | Dependency |
|------|-------------|------------|
| `internal/memory/embeddings.go` | MODIFY: integrate with episodic store | EpisodicStore |
| `internal/knowledgegraph/extractor.go` | MODIFY: async extraction via events | EventBus |
| `internal/knowledgegraph/extractor_prompt.go` | No change | |
| `internal/store/memory_store.go` | MODIFY: add EpisodicSummaryStore interface | None |
| `internal/store/knowledge_graph_store.go` | MODIFY: add temporal query methods | None |
| `internal/store/pg/memory_docs.go` | No change (v2 compat) | |
| `internal/store/pg/memory_search.go` | MODIFY: integrate episodic search | EpisodicStore |
| `internal/store/pg/knowledge_graph.go` | MODIFY: temporal WHERE clauses | KG temporal cols |
| `internal/store/pg/knowledge_graph_dedup.go` | MODIFY: configurable thresholds | Agent settings |
| `internal/store/pg/knowledge_graph_relations.go` | MODIFY: temporal WHERE clauses | KG temporal cols |
| `internal/store/pg/knowledge_graph_traversal.go` | MODIFY: temporal filter in traversal | KG temporal cols |
| `internal/store/pg/knowledge_graph_embedding.go` | No change | |
| `internal/agent/memoryflush.go` | MODIFY: v3 path uses event-driven consolidation | EventBus |
| `internal/agent/extractive_memory.go` | No change (fallback for v2 mode) | |
**New files:**
| File | Purpose |
|------|---------|
| `internal/store/episodic_store.go` | EpisodicSummaryStore interface |
| `internal/store/pg/episodic_summaries.go` | PG implementation |
| `internal/agent/consolidation_worker.go` | Event-driven session -> episodic -> semantic |
| `internal/agent/memory_migrator.go` | Lazy v2 daily files -> episodic migration |
#### Module 4: System Integration (Phase 4)
| File | Change Type | Dependency |
|------|-------------|------------|
| `internal/agent/systemprompt.go` | MAJOR: replace with PromptBuilder | PromptConfig types |
| `internal/agent/systemprompt_sections.go` | REMOVE: logic moves to template | PromptBuilder |
| `internal/tools/types.go` | MODIFY: add Metadata() to Tool interface | None |
| `internal/tools/registry.go` | MODIFY: store ToolMetadata | ToolMetadata |
| `internal/tools/policy.go` | MODIFY: capability rules + role param | ToolCapability |
| `internal/tools/filesystem.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext |
| `internal/tools/filesystem_write.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext |
| `internal/tools/filesystem_list.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext |
| `internal/tools/shell.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext |
| `internal/tools/edit.go` | MODIFY: WorkspaceContextFromCtx() | WorkspaceContext |
| `internal/tools/memory.go` | MODIFY: add depth parameter | L0/L1/L2 |
| `internal/tools/knowledge_graph.go` | MODIFY: add as_of temporal param | KG temporal |
| `internal/tools/skill_search.go` | MODIFY: hybrid search integration | HybridSearcher |
| `internal/skills/loader.go` | MODIFY: ACL-aware BuildSummary() | SkillAccessStore |
| `internal/skills/search.go` | MODIFY: expose for fusion with embedding | None |
**New files:** (listed in Phase 4 design)
#### Module 5: Orchestration & Intelligence (Phase 5)
| File | Change Type | Dependency |
|------|-------------|------------|
| `internal/tools/policy.go` | MODIFY: orchestration mode filtering | OrchestrationMode |
| `internal/bus/types.go` | MODIFY: delegation + evolution event types | None |
| `internal/agent/loop.go` | MODIFY: metrics recording hooks | EvolutionMetrics |
| `cmd/gateway.go` | MODIFY: register delegate tool, event subscribers | All Phase 5 |
**New files:** (listed in Phase 5 design)
#### Breaking Changes Inventory
| Change | Impact | Backward Compat |
|--------|--------|-----------------|
| Tool.Metadata() added to interface | All tool implementations | Wrapper: `defaultMetadata(tool)` |
| WorkspaceContext replaces dual keys | All file tools + loop_context | Single migration: grep `ToolWorkspaceFromCtx` (15 sites) |
| PromptBuilder replaces BuildSystemPrompt() | loop_history.go, tests | Bridge: old func calls PromptBuilder.Build() |
| Pipeline stages replace runLoop() | loop.go (804 lines) | Feature flag: v3_pipeline_enabled per agent |
| TokenCounter replaces chars/4 | loop_compact.go, pruning.go | Fallback: chars/4 for unknown models |
| KG temporal columns | All KG queries | Default: WHERE valid_until IS NULL (same as pre-temporal) |
| Episodic summaries table | New (no existing code depends on it) | Fully additive |
| Evolution metrics/suggestions | New (no existing code depends on it) | Fully additive |
---
### D. Integration Test Plan
#### Test Matrix: Workspace Scenarios
From workspace forcing report — 6 scenarios must pass:
| # | Scenario | ActivePath | Scope | Key Assertion |
|---|----------|------------|-------|---------------|
| 1 | Solo DM, isolated | `{base}/{userID}` | personal | Files scoped to user dir |
| 2 | Solo DM, shared | `{base}` | personal | Files in shared agent dir |
| 3 | Team dispatch (member) | `{base}/teams/{teamID}/{chatID}` | team-shared | Workspace forced to team dir |
| 4 | Team dispatch (leader) | `{base}/teams/{teamID}/{chatID}` | team-shared | Leader sees team workspace |
| 5 | Team inbound (member) | `{base}/teams/{teamID}/{chatID}` | team-isolated | Per-chat isolation within team |
| 6 | Delegate workspace | delegatee workspace + shared area | delegate | Read-only exports enforced |
#### Memory Tier Tests
| Test | Input | Expected |
|------|-------|----------|
| Working memory (Tier 1) | 20 messages in session | All messages in context window |
| Episodic creation | Session end event | episodic_summaries row created |
| Episodic dedup | Same session end event replayed | No duplicate (source_id unique) |
| Semantic extraction | Episodic with entities | KG entities upserted |
| L0 auto-inject | User message with relevant topic | Memory section in system prompt |
| L0 skip | "hello" message (low relevance) | No Memory section |
| V2 migration | Agent with memory/2026-01-01.md | Episodic row with source_type=v2_daily |
| TTL expiry | Episodic with expires_at in past | Cleaned by cron |
#### KG Temporal Tests
| Test | Input | Expected |
|------|-------|----------|
| New entity | UpsertEntity() | valid_from=NOW(), valid_until=NULL |
| Supersede entity | Contradicting fact | Old: valid_until=NOW(). New: valid_from=NOW() |
| Current query | SearchEntities() default | Only valid_until IS NULL entities |
| Historical query | SearchEntities(as_of="2026-01-01") | Entities valid at that date |
| Temporal traversal | Traverse() with temporal entities | Only current relations followed |
| Configurable dedup | Agent with threshold 0.95 | Auto-merge at 0.95, not hardcoded 0.98 |
#### Pipeline Stage Tests
| Test | Stage | Input | Expected |
|------|-------|-------|----------|
| Context build | ContextStage | RunRequest + agent config | WorkspaceContext resolved, system prompt built |
| LLM call | ThinkStage | Messages array | Provider called via adapter |
| Token pruning | PruneStage | Messages exceeding 85% context | Older messages pruned, token count accurate |
| Tool exec | ToolStage | Tool call from LLM | Tool executed via registry with policy check |
| Loop detect | ObserveStage | 3x same tool call | Loop detected, force stop |
| Session save | CheckpointStage | Completed run | Session persisted, events emitted |
| Full pipeline | All stages | Complete request | End-to-end: request -> response -> session saved |
#### Provider Adapter Tests
| Test | Provider | Input | Expected |
|------|----------|-------|----------|
| Anthropic wire | Anthropic | Internal ChatRequest | Valid Anthropic API JSON |
| OpenAI wire | OpenAI | Internal ChatRequest | Valid OpenAI API JSON |
| DashScope no-stream-tools | DashScope | ChatRequest with tools | Streaming disabled, tools present |
| Codex OAuth | Codex | ChatRequest | OAuth token refreshed if needed |
| Claude CLI | ClaudeCLI | ChatRequest | Stdio command built correctly |
| Capabilities | All providers | Capabilities() | Correct flags per provider |
| Round-trip | All providers | ToRequest -> FromResponse | Internal format preserved |
#### Regression Tests
| Test | Assertion |
|------|-----------|
| V2 agent with all flags false | Identical behavior to pre-v3 binary |
| V2 memory flush | Daily .md files still written when v3_memory_enabled=false |
| V2 KG without temporal | Queries return same results (WHERE valid_until IS NULL matches all) |
| V2 system prompt | PromptBuilder with v2 config produces identical output |
| V2 tool registry | Tools without Metadata() get default metadata wrapper |
---
### E. Audit Checklist
#### Interface Contracts
- [ ] All interface contracts have method signatures with ctx context.Context as first param
- [ ] All interfaces have at least one implementation (PG store)
- [ ] No interface has >10 methods (split if so)
- [ ] Error returns are typed (not raw string errors)
- [ ] Context propagation: tenant_id, agent_id, user_id carried through all store calls
#### DB Schemas
- [ ] All new tables have tenant_id column with FK to tenants(id)
- [ ] All FKs have ON DELETE CASCADE where appropriate
- [ ] Indexes exist for: primary lookup, tenant filter, time-range queries
- [ ] No new columns on existing tables (all config in JSONB settings)
- [ ] Down migrations cleanly reverse up migrations
- [ ] No PG-specific features in interface definitions (SQLite compat future)
#### Data Flow
- [ ] Memory flow: user message -> auto-inject L0 -> response -> session end -> episodic -> semantic
- [ ] KG flow: text -> extract -> upsert -> dedup -> temporal validity
- [ ] Pipeline flow: ContextStage -> ThinkStage -> PruneStage -> ToolStage -> ObserveStage -> CheckpointStage
- [ ] Delegation flow: delegate() -> permission check -> workspace setup -> AgentRunner -> result delivery
- [ ] Event flow: stage completion -> EventBus.Broadcast() -> subscriber handlers
#### Breaking Changes
- [ ] Tool.Metadata() — wrapper provided for existing tools
- [ ] WorkspaceContext — single context key replaces 2 (grep all call sites)
- [ ] BuildSystemPrompt() — bridge function wraps PromptBuilder
- [ ] Pipeline stages — feature flag gates new behavior
- [ ] Token counting — fallback to chars/4 for unknown models
#### Security Review
- [ ] Workspace isolation: WorkspaceContext.ActivePath enforced by resolvePath()
- [ ] Delegate read-only exports: validated against delegator's workspace boundary
- [ ] Delegate shared area: created with restrictive permissions (0750)
- [ ] Tenant boundary: all new tables have tenant_id, all queries filter by it
- [ ] Metrics data: no PII stored (queries + tool names, not user content)
- [ ] Template injection: PromptConfig data passed via {{ printf "%s" }} not {{ }}
- [ ] Auto-adapt scope: only modifies retrieval params, not security settings
#### Performance Review
- [ ] tiktoken-go: benchmark Count() for Claude models (<1ms per 1000 tokens)
- [ ] pgvector: benchmark episodic_summaries search at 100K rows (<50ms)
- [ ] Event bus: benchmark Broadcast throughput (10K events/sec target)
- [ ] Template rendering: benchmark Build() (<1ms, no allocation-heavy paths)
- [ ] L0 auto-inject: benchmark memory index query (<10ms per turn)
- [ ] KG temporal: benchmark index scan with valid_until IS NULL filter
- [ ] Metrics recording: verify non-blocking (goroutine, no mutex on hot path)
---
## Related Code Files
### Migration Files to Create
| File | Purpose |
|------|---------|
| `migrations/000037_v3_memory_evolution.up.sql` | episodic_summaries, agent_evolution_metrics, agent_evolution_suggestions |
| `migrations/000037_v3_memory_evolution.down.sql` | Drop above tables |
| `migrations/000038_kg_temporal_columns.up.sql` | ALTER kg_entities/kg_relations add valid_from/valid_until |
| `migrations/000038_kg_temporal_columns.down.sql` | DROP added columns + indexes |
### Files to Modify
| File | Change |
|------|--------|
| `internal/upgrade/version.go` | RequiredSchemaVersion = 38 |
### Full Affected File Count by Module
| Module | Files Modified | Files Created | Total |
|--------|---------------|---------------|-------|
| Foundation (Phase 1) | 15 | 4 | 19 |
| Pipeline (Phase 2) | 9 | 7 | 16 |
| Memory & KG (Phase 3) | 14 | 4 | 18 |
| System Integration (Phase 4) | 15 | 9 | 24 |
| Orchestration (Phase 5) | 4 | 10 | 14 |
| Migration (Phase 6) | 1 | 4 | 5 |
| **Total** | **58** | **38** | **96** |
---
## Design Steps
1. Write migration 000037 DDL (episodic_summaries, evolution tables)
2. Write migration 000038 DDL (KG temporal columns)
3. Write down migrations for both
4. Document lazy v2 memory migration flow
5. Document KG temporal backfill strategy
6. List all feature flags with defaults
7. Inventory all affected files per module (complete)
8. Document breaking changes with migration path
9. Write workspace scenario test matrix
10. Write memory tier test scenarios
11. Write KG temporal test scenarios
12. Write pipeline stage test scenarios
13. Write provider adapter test matrix
14. Write regression test list
15. Complete audit checklist
---
## Todo List
- [ ] A1: Migration 000037 DDL (3 new tables)
- [ ] A2: Migration 000038 DDL (KG temporal ALTER)
- [ ] A3: Down migrations for both
- [ ] A4: RequiredSchemaVersion bump to 38
- [ ] B1: V2 memory -> episodic lazy migration design
- [ ] B2: KG temporal backfill strategy
- [ ] B3: Feature flags catalog with defaults
- [ ] B4: Session v2->v3 compatibility notes
- [ ] C1: Affected files inventory per module (complete)
- [ ] C2: Code change dependency order
- [ ] C3: Breaking changes list with backward-compat path
- [ ] D1: Workspace scenario test matrix (6 scenarios)
- [ ] D2: Memory tier tests (8 scenarios)
- [ ] D3: KG temporal tests (6 scenarios)
- [ ] D4: Pipeline stage tests (7 scenarios)
- [ ] D5: Provider adapter tests (7 scenarios)
- [ ] D6: Regression tests (5 scenarios)
- [ ] E1: Interface contracts audit
- [ ] E2: DB schema audit
- [ ] E3: Data flow audit
- [ ] E4: Breaking changes audit
- [ ] E5: Security review
- [ ] E6: Performance review
---
## Success Criteria
1. Both migrations run cleanly on empty DB and on DB with existing v2 data
2. Down migrations fully reverse up migrations (no orphaned objects)
3. RequiredSchemaVersion matches highest migration number
4. All 6 workspace scenarios pass integration tests
5. V2 agents with all flags false behave identically to pre-v3
6. Every breaking change has documented backward-compat path
7. Audit checklist 100% complete before implementation begins
---
## Risk Assessment
| Risk | Severity | Mitigation |
|------|----------|------------|
| Migration fails on large KG tables | MEDIUM | ALTER TABLE ADD COLUMN with DEFAULT is metadata-only in PG12+ (instant). No table rewrite. |
| V2 memory files not found (path differences) | LOW | Lazy migration lists actual files in workspace. Files missing = nothing to migrate. |
| Feature flag combinatorial explosion | MEDIUM | 6 boolean flags = 64 combos. Test key combos only: all-off (v2), all-on (v3), individual toggles. |
| 96 affected files scope | HIGH | Implementation phased by module. Each module independently deployable behind feature flags. |
| Episodic summaries table grows large | MEDIUM | expires_at column + cleanup cron. Index on expires_at for efficient deletion. |
| Temporal KG backfill slow | LOW | Backfill is optional, runs as cron in background. No downtime. Default valid_from=NOW() sufficient. |
| SQLite compat deferred too long | MEDIUM | Interfaces designed PG-agnostic. SQLite impl can be added later without interface changes. |
---
## Rollback Plan
### Per-Migration Rollback
```bash
./goclaw migrate down 2 # reverts 000038, then 000037
```
### Per-Feature Rollback
Set feature flag to false in agent settings:
```json
{ "v3_pipeline_enabled": false, "v3_memory_enabled": false }
```
Agent immediately reverts to v2 behavior. No code rollback needed.
### Full v3 Rollback
1. Disable all v3 feature flags for all agents
2. Run `migrate down 2` to remove v3 tables
3. Deploy v2 binary
4. v2 memory files, KG entities, sessions — all intact (v3 tables were additive)
---
## Security Considerations
- **Migration safety:** All changes additive (CREATE TABLE, ALTER ADD COLUMN). No DROP, RENAME, or data modification in up migration. Down migration only drops v3-added objects.
- **Feature flag isolation:** v3 code paths gated by flags. Flag false = v2 code path. No mixed state.
- **Tenant isolation in new tables:** All 3 new tables have tenant_id FK. All queries must include tenant_id filter. Store implementations enforce via context.
- **Evolution metrics cleanup:** Cron job with configurable TTL (default 90 days). Prevents unbounded growth.
- **Temporal KG queries:** Default WHERE valid_until IS NULL produces same results as pre-temporal. No data exposure from temporal columns.
---
## Implementation Dependency Order
```
Foundation (can start immediately, no dependencies):
1. TokenCounter (internal/agent/token_counter.go)
2. WorkspaceContext (internal/tools/workspace_context.go)
3. EventBus topics (internal/bus/types.go)
4. ProviderAdapter interface (internal/providers/adapter.go)
Core (depends on Foundation):
5. Pipeline stages (internal/agent/pipeline.go + stage_*.go)
6. PromptBuilder + templates (internal/agent/prompt_builder.go)
7. Tool Metadata + enhanced registry (internal/tools/capability.go)
Memory & KG (depends on Foundation + migrations):
8. DB migrations 000037 + 000038
9. EpisodicSummaryStore (internal/store/episodic_store.go)
10. KG temporal queries (modify existing pg/ files)
11. Consolidation worker (internal/agent/consolidation_worker.go)
12. L0 retriever (internal/agent/retriever.go)
Integration (depends on Core + Memory):
13. Hybrid skill searcher (internal/skills/hybrid_searcher.go)
14. Tool policy capability rules (internal/tools/policy.go)
15. Provider adapters (all 6 providers)
Orchestration (depends on Integration):
16. DelegateTool (internal/tools/delegate_tool.go)
17. Orchestration mode config (internal/agent/orchestration_mode.go)
18. Evolution metrics (internal/store/pg/evolution_metrics.go)
19. Evolution suggestions (internal/agent/evolution_suggestion_engine.go)
Testing (depends on all above):
20. Integration tests (tests/integration/v3_*.go)
21. Regression tests (tests/integration/v2_compat_*.go)
```
Items 1-4 are independent and can be built in parallel.
Items 5-7 are independent of each other but depend on 1-4.
Items 8-12 are sequential (migrations first, then stores, then workers).
plans/260406-1304-goclaw-v3-core-design/plan.md
-101
View File
@@ -1,101 +0,0 @@
---
title: "GoClaw v3 Core Architecture Design"
description: "Comprehensive design plan for v3 core redesign — 18 modules covering memory, KG, agent loop, providers, workspace, orchestration"
status: complete
priority: P1
effort: 40h
branch: dev-v3
tags: [refactor, backend, architecture, design]
blockedBy: []
blocks: []
created: 2026-04-06
---
# GoClaw v3 Core Architecture Design
## Overview
Design-first plan for comprehensive v3 core redesign. Produces interface contracts, DB schemas, data flow diagrams, and dependency graphs — NOT implementation code. Implementation plans created per-phase after design audit.
**Brainstorm report:** `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md`
## Cross-Plan Dependencies
None. This plan blocks future implementation plans (to be created after design audit).
## Key Decisions (from brainstorm)
| # | Module | Decision |
|---|--------|----------|
| 1 | Loop | Pipeline stages (pluggable interfaces) |
| 2 | Memory | 3-tier (working/episodic/semantic) |
| 3 | Context | L0/L1/L2 progressive loading |
| 4 | KG | Temporal validity windows |
| 5 | Token counting | tiktoken-go accurate |
| 6 | Consolidation | Async event-driven |
| 7 | Retrieval | Smart auto-inject L0 + on-demand |
| 8 | Workspace | Unified WorkspaceContext |
| 9 | System prompt | Template-based |
| 10 | Providers | Capability + unified wire + plugin |
| 11 | Self-evolution | Progressive (metrics→suggest→auto) |
| 12 | Orchestration | Layered: delegate base + team optional |
| 13 | Tools | Redesign (registry, policy, workspace-aware) |
| 14 | Events | Redesign (structured, typed) |
| 15 | Skills | Semantic search + ACL + deps |
| 16 | Context files | DB + better abstraction |
| 17 | Migration | Dual-mode v2→v3 |
| 18 | SQLite | Deferred |
## Phases
| Phase | Name | Status | Effort |
|-------|------|--------|--------|
| 1 | [Foundation Interfaces](./phase-01-foundation-interfaces.md) | **Complete** | 8h |
| 2 | [Pipeline Loop Design](./phase-02-pipeline-loop-design.md) | **Complete** | 8h |
| 3 | [Memory & KG Design](./phase-03-memory-kg-design.md) | **Complete** | 8h |
| 4 | [System Integration Design](./phase-04-system-integration-design.md) | **Complete** | 6h |
| 5 | [Orchestration & Intelligence Design](./phase-05-orchestration-intelligence-design.md) | **Complete** | 6h |
| 6 | [Migration & Audit](./phase-06-migration-audit.md) | **Complete** | 4h |
## Dependency Graph
```
Phase 1: Foundation ─────────────────────────┐
(TokenCounter, WorkspaceContext, │
EventBus, ProviderAdapter) │
│ │
▼ │
Phase 2: Pipeline Loop ──────────┐ │
(Stage interfaces, RunState) │ │
│ │ │
▼ ▼ ▼
Phase 3: Memory & KG Phase 4: System Integration
(3-tier, temporal, (Template prompt, tool
consolidation, L0/L1/L2) registry, skills, retrieval)
│ │
└───────────┬───────────┘
▼
Phase 5: Orchestration & Intelligence
(Delegate, self-evolution)
│
▼
Phase 6: Migration & Audit
(v2→v3 dual-mode, test plan)
```
## Deliverables Per Phase
Each phase produces:
- **Interface contracts** — Go interface definitions with method signatures
- **DB schemas** — SQL DDL for new/modified tables
- **Data flow diagrams** — Textual flow descriptions
- **Config schemas** — JSON/YAML configuration structures
- **Dependency map** — Which existing files are affected
## References
- Brainstorm: `plans/reports/brainstorm-260406-1059-goclaw-v3-core-redesign.md`
- Research: `plans/reports/researcher-260406-1059-memory-context-kg-research.md`
- Architecture: `plans/reports/Explore-260406-thorough-v3-architecture.md`
- Loop deep dive: `plans/reports/Explore-260406-GoClaw-Agent-Loop-v3-Deep-Dive.md`
- Workspace forcing: `plans/reports/Explore-260406-workspace-forcing.md`