diff --git a/.gitignore b/.gitignore index e12ef95f..b2a5ff69 100644 --- a/.gitignore +++ b/.gitignore @@ -88,4 +88,5 @@ compose.d/* /mise.*.toml /mise /.mise +*.test goclaw-patched-linux-amd64 diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b03c2a0..f640b6a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,35 @@ All notable changes to GoClaw are documented here. For full documentation, see [docs.goclaw.sh](https://docs.goclaw.sh). +## Unreleased + +### Breaking Changes + +- **Context pruning now opt-in.** Previously tool-result trimming ran by default + for all providers; now requires explicit `contextPruning.mode: "cache-ttl"` in + `config.agents.defaults` to enable. Matches upstream TS design and prevents + silent prompt-cache invalidation on Anthropic. + + Migration — add to `config.json5`: + ```json5 + agents: { + defaults: { + contextPruning: { mode: "cache-ttl" } + } + } + ``` + +### Improvements + +- **Context pruning cleanup.** Removed redundant Pass 0 (per-result 30% guard), + deduplicated double prune call per iteration, added SanitizeHistory to + PruneStage for broken tool_use/tool_result pair cleanup. +- **Context pruning config backfill (migration).** Agents with existing custom + `context_pruning` config (e.g., `softTrimRatio`, `keepLastAssistants`) but + missing a `mode` field get auto-backfilled with `mode: "cache-ttl"` to + preserve their intent after the opt-in flip. Rows with NULL config stay + NULL (new opt-in default applies). PG migration 51; SQLite schema v19. + ## Project Status ### Implemented & Tested in Production diff --git a/CLAUDE.md b/CLAUDE.md index 096b67d6..14a77b76 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -184,6 +184,44 @@ Published to GHCR (`ghcr.io/nextlevelbuilder/goclaw`) and Docker Hub (`digitop/g - **Tool gating:** `TeamActionPolicy` in `internal/tools/team_action_policy.go` — lite blocks comment/review/approve/reject/attach/ask_user. `skill_manage`/`publish_skill` not registered in lite - **File serving:** 2-layer path isolation in `internal/http/files.go` — workspace boundary (all editions) + tenant scope (standard only with RBAC) +## Plan Verification Rules + +Apply before finalizing any multi-phase plan. Trust-but-verify between scout → planner → final plan. + +### Verification discipline (what to verify) + +1. **Verify factual claims against code** — re-grep/re-count every number, path, endpoint. Don't copy from scout summaries. +2. **Trace semantics, not just cite lines** — when plan references existing/upstream code, identify WHEN each field mutates and under WHAT conditions. Line-range citation without control-flow trace = how ports silently invert behavior. Check: every call, or specific branches only? +3. **No fabricated identifiers / API families** — every symbol in plan must cite `file:line`. RED FLAGS: plausible-sounding wrappers (`Keyring`, `Validator`, `Manager`), centralized packages (`internal/security`, `internal/auth`) that may be scattered, OTel-style (`StartSpan/EndSpan`) when codebase is emit-based. When unsure, `go doc ` lists actual exported surface. Apply especially when plan says "reuse existing X". +4. **Struct scope audit before adding state** — verify lifetime (per-request/session/agent/process) before adding a field to an existing struct. "Plausibly per-X" is a red flag — grep construction + ownership. Shared-instance state leaks across isolation boundaries. +5. **Gate-premise test math** — before asserting "feature X triggers independently of Y", list all early-returns from function entry to X. Math-verify any fixture claiming "X without Y". +6. **Port = config-shape match** — "faithful port" divergences in config field name/type are silent breaking changes for users copying upstream config. Match upstream shape, or explicitly flag each divergence with rationale in the phase file. +7. **Verify external API endpoints via `docs-seeker`** — before writing endpoint into plan. Sibling APIs often use different roots. + +### Scope & coverage (where to look) + +8. **Grep delete scope deep** — `grep -rn '' .` whole repo. Stubs often have refs in catalogs/routing/switch cases. Enumerate ALL sites in todo. +9. **Signature-change callers enumeration** — grep + list all callers explicitly. "Update all callers" insufficient. +10. **Alias/shim coverage** — enumerate ALL exported symbols via `go doc `. Add compile-time signature guards. +11. **Scout desktop and web separately** — `ui/desktop/frontend/` ≠ `ui/web/`. Different structure, i18n namespaces, test framework presence. + +### Phasing & ordering (when) + +12. **Re-scout on scope change** — if phase promotes from deferred → active, re-scout. Don't reuse brainstorm summary. +13. **Cross-phase gates explicit** — "Phase N-1 merged + tests green" in phase Context. Execution order alone ≠ enforcement. +14. **Zero-coverage characterization test = blocker step** — write byte/request-body fixture test BEFORE migration. Not "recommended". +15. **i18n keys ordering** — add key + 3 catalogs as explicit todo step BEFORE handler code. Missing key = runtime crash. + +### Conventions & finalization + +16. **Context key style convention** — check existing `context.go` pattern before introducing new key types. Mixed = code smell. +17. **Verify pass MANDATORY after rewrite** — spawn fresh Explore/grep to audit planner output. Don't trust self-validation. + +**Pattern to avoid:** user asks → planner writes → report "done". +**Safer pattern:** user asks → scout → planner writes → audit-verify → report. + +**Red-team practice:** After planner completes, run `code-reviewer`/`brainstormer` in audit mode: "spot-check 15+ claims vs live codebase". Past catches: fabricated `crypto.Keyring`/`tracing.StartSpan` (agent-hooks plan); inverted TS-port semantics + wrong struct scope + misread early-return gate (context-pruning plan). See `plans/*/reports/audit-*.md` for concrete examples. + ## Post-Implementation Checklist After implementing or modifying Go code, run these checks: @@ -207,6 +245,7 @@ Go conventions to follow: - **DB query reuse:** Before adding a new DB query for key entities (teams, agents, sessions, users), check if the same data is already fetched earlier in the current flow/pipeline. Prefer passing resolved data through context, event payloads, or function params rather than re-querying. Duplicate queries waste DB resources and add latency - **Solution design:** When designing a fix or feature, identify the root cause first — don't just patch symptoms. Think through production scenarios (high concurrency, multi-tenant isolation, failure cascades, long-running sessions) to ensure the solution holds up. Prefer explicit configuration over runtime heuristics. Prefer the simplest solution that addresses the root cause directly - **Tenant-scope guards on admin writes:** `RoleAdmin` is not a tenant check. Writes to **global** tables (no `tenant_id` column — e.g. `builtin_tools`, disk config, package mgmt) must gate with `http.requireMasterScope` / WS `requireMasterScope(requireOwner(...))`. Writes to **tenant-scoped** tables must gate with `http.requireTenantAdmin` + SQL `WHERE tenant_id = $N`. Shared predicate: `store.IsMasterScope(ctx)`. See `CONTRIBUTING.md` → "Tenant-scope guards" for the full decision table and anti-patterns. +- **Skip load / stress / benchmark tests.** Do NOT write throughput benchmarks, p95/p99 latency assertions, or `runtime.ReadMemStats`-based memory-leak tests for regular feature work. They flake on shared CI runners, waste runner time, and rarely catch real bugs. Only add load tests when explicitly requested for a specific investigation. For normal "prove it works" coverage, use unit + integration + chaos tests. ## Mobile UI/UX Rules diff --git a/Dockerfile b/Dockerfile index b7ab822c..c768a836 100644 --- a/Dockerfile +++ b/Dockerfile @@ -144,7 +144,8 @@ RUN chmod +x /app/docker-entrypoint.sh && \ # while pip/npm subdirs are goclaw-owned (runtime installs by the app process). # Symlink .claude → data volume so Claude CLI credentials persist across container recreates. RUN mkdir -p /app/workspace /app/data/.runtime/pip /app/data/.runtime/npm-global/lib \ - /app/data/.runtime/pip-cache /app/data/.claude /app/skills /app/tsnet-state /app/.goclaw \ + /app/data/.runtime/pip-cache /app/data/.runtime/bin /app/data/.claude /app/skills \ + /app/tsnet-state /app/.goclaw \ && ln -s /app/data/.claude /app/.claude \ && touch /app/data/.runtime/apk-packages \ && chown -R goclaw:goclaw /app/workspace /app/skills /app/tsnet-state /app/.goclaw \ @@ -152,7 +153,8 @@ RUN mkdir -p /app/workspace /app/data/.runtime/pip /app/data/.runtime/npm-global && chown root:goclaw /app/data/.runtime /app/data/.runtime/apk-packages \ && chmod 0750 /app/data/.runtime \ && chmod 0640 /app/data/.runtime/apk-packages \ - && chown -R goclaw:goclaw /app/data/.runtime/pip /app/data/.runtime/npm-global /app/data/.runtime/pip-cache /app/data/.claude + && chown -R goclaw:goclaw /app/data/.runtime/pip /app/data/.runtime/npm-global /app/data/.runtime/pip-cache /app/data/.runtime/bin /app/data/.claude \ + && chmod 0755 /app/data/.runtime/bin # Default environment ENV GOCLAW_CONFIG=/app/config.json \ diff --git a/Makefile b/Makefile index a497b804..679c1dca 100644 --- a/Makefile +++ b/Makefile @@ -2,7 +2,7 @@ VERSION ?= $(shell git describe --tags --abbrev=0 --match "v[0-9]*" 2>/dev/null LDFLAGS = -s -w -X github.com/nextlevelbuilder/goclaw/cmd.Version=$(VERSION) BINARY = goclaw -.PHONY: build build-full build-tui run clean version up down logs reset test vet check-web dev migrate setup ci desktop-dev desktop-build desktop-dmg +.PHONY: build build-full build-tui run clean version up down logs reset test vet check-web dev migrate setup ci desktop-dev desktop-build desktop-dmg test-hooks test-hooks-unit test-hooks-e2e test-hooks-chaos test-hooks-rbac test-hooks-tracing # Build backend only (API-only, no embedded web UI) build: @@ -94,6 +94,26 @@ test-scenarios: # Critical tests (P0 + P1) - run before merge test-critical: test-invariants test-contracts +# ── Agent Hooks targets (phase 4) ── +# Requires TEST_DATABASE_URL pointing at a pgvector:pg18 container on :5433 +test-hooks-unit: + go test -race ./internal/hooks/... ./internal/gateway/methods/ + +test-hooks-e2e: + go test -race -timeout=180s -tags integration -run "TestHooksE2E" ./tests/integration/ + +test-hooks-chaos: + go test -race -timeout=180s -tags integration -run "TestHooksChaos" ./tests/integration/ + +test-hooks-rbac: + go test -race -timeout=90s -tags integration -run "TestHooksRBAC" ./tests/integration/ + +test-hooks-tracing: + go test -race -timeout=90s -tags integration -run "TestHooksTracing" ./tests/integration/ + +# Full hook test suite (unit + integration) +test-hooks: test-hooks-unit test-hooks-e2e test-hooks-chaos test-hooks-rbac test-hooks-tracing + vet: go vet ./... diff --git a/cmd/gateway.go b/cmd/gateway.go index e18efc85..4ca3e32c 100644 --- a/cmd/gateway.go +++ b/cmd/gateway.go @@ -33,6 +33,7 @@ import ( "github.com/nextlevelbuilder/goclaw/internal/edition" "github.com/nextlevelbuilder/goclaw/internal/gateway" "github.com/nextlevelbuilder/goclaw/internal/gateway/methods" + "github.com/nextlevelbuilder/goclaw/internal/hooks" httpapi "github.com/nextlevelbuilder/goclaw/internal/http" mcpbridge "github.com/nextlevelbuilder/goclaw/internal/mcp" "github.com/nextlevelbuilder/goclaw/internal/media" @@ -133,7 +134,7 @@ func runGateway() { tools.DetectServerIPs(context.Background()) } - toolsReg, execApprovalMgr, mcpMgr, sandboxMgr, browserMgr, webFetchTool, ttsTool, permPE, toolPE, dataDir, agentCfg := setupToolRegistry(cfg, workspace, providerRegistry) + toolsReg, execApprovalMgr, mcpMgr, sandboxMgr, browserMgr, webFetchTool, ttsTool, audioMgr, permPE, toolPE, dataDir, agentCfg := setupToolRegistry(cfg, workspace, providerRegistry) if browserMgr != nil { defer browserMgr.Close() } @@ -263,6 +264,9 @@ func runGateway() { // Create all agents — resolved lazily from database by the managed resolver. agentRouter := agent.NewRouter() + if traceCollector != nil { + agentRouter.SetTraceCollector(traceCollector) + } slog.Info("agents will be resolved lazily from database") // Create gateway server and wire enforcement @@ -307,6 +311,7 @@ func runGateway() { workspace: workspace, dataDir: dataDir, domainBus: domainBus, + audioMgr: audioMgr, } gatewayAddr := loopbackAddr(cfg.Gateway.Host, cfg.Gateway.Port) @@ -375,6 +380,18 @@ func runGateway() { server.SetLogTee(logTee) pairingMethods, heartbeatMethods, chatMethods := registerAllMethods(server, agentRouter, pgStores.Sessions, pgStores.Cron, pgStores.Pairing, cfg, cfgPath, workspace, dataDir, msgBus, execApprovalMgr, pgStores.Agents, pgStores.Skills, pgStores.ConfigSecrets, pgStores.Teams, contextFileInterceptor, logTee, pgStores.Heartbeats, pgStores.ConfigPermissions, pgStores.SystemConfigs, pgStores.Tenants, pgStores.SkillTenantCfgs) + // Phase 3: Agent hooks RPC methods (hooks.list/create/update/delete/toggle/test/history). + if hs, ok := pgStores.Hooks.(hooks.HookStore); ok && hs != nil { + hm := methods.NewHookMethods(hs, edition.Current()) + // Reuse dispatcher handlers for dry-run test runner so UI test panel + // exercises the exact code that will run in production. + if sharedHookHandlers != nil { + hm.SetTestRunner(methods.NewDispatcherTestRunner(sharedHookHandlers)) + } + hm.Register(server.Router()) + slog.Info("registered hooks RPC methods") + } + // Wire post-turn processor for team task dispatch (WS chat.send + HTTP API paths). if postTurn != nil { chatMethods.SetPostTurnProcessor(postTurn) @@ -422,12 +439,12 @@ func runGateway() { instanceLoader = channels.NewInstanceLoader(pgStores.ChannelInstances, pgStores.Agents, channelMgr, msgBus, pgStores.Pairing) instanceLoader.SetProviderRegistry(providerRegistry) instanceLoader.SetPendingCompactionConfig(cfg.Channels.PendingCompaction) - instanceLoader.RegisterFactory(channels.TypeTelegram, telegram.FactoryWithStores(pgStores.Agents, pgStores.ConfigPermissions, pgStores.Teams, pgStores.SubagentTasks, pgStores.PendingMessages)) - instanceLoader.RegisterFactory(channels.TypeDiscord, discord.FactoryWithStores(pgStores.Agents, pgStores.ConfigPermissions, pgStores.PendingMessages)) - instanceLoader.RegisterFactory(channels.TypeFeishu, feishu.FactoryWithPendingStore(pgStores.PendingMessages)) + instanceLoader.RegisterFactory(channels.TypeTelegram, telegram.FactoryWithStoresAndAudio(pgStores.Agents, pgStores.ConfigPermissions, pgStores.Teams, pgStores.SubagentTasks, pgStores.PendingMessages, audioMgr)) + instanceLoader.RegisterFactory(channels.TypeDiscord, discord.FactoryWithStoresAndAudio(pgStores.Agents, pgStores.ConfigPermissions, pgStores.PendingMessages, audioMgr)) + instanceLoader.RegisterFactory(channels.TypeFeishu, feishu.FactoryWithPendingStoreAndAudio(pgStores.PendingMessages, audioMgr)) instanceLoader.RegisterFactory(channels.TypeZaloOA, zalo.Factory) instanceLoader.RegisterFactory(channels.TypeZaloPersonal, zalopersonal.FactoryWithPendingStore(pgStores.PendingMessages)) - instanceLoader.RegisterFactory(channels.TypeWhatsApp, whatsapp.FactoryWithDB(pgStores.DB, pgStores.PendingMessages, "pgx")) + instanceLoader.RegisterFactory(channels.TypeWhatsApp, whatsapp.FactoryWithDBAudio(pgStores.DB, pgStores.PendingMessages, "pgx", audioMgr, pgStores.BuiltinTools)) instanceLoader.RegisterFactory(channels.TypeSlack, slackchannel.FactoryWithPendingStore(pgStores.PendingMessages)) instanceLoader.RegisterFactory(channels.TypeFacebook, facebook.Factory) instanceLoader.RegisterFactory(channels.TypePancake, pancake.Factory) @@ -437,7 +454,7 @@ func runGateway() { } // Register config-based channels as fallback when no DB instances loaded. - registerConfigChannels(cfg, channelMgr, msgBus, pgStores, instanceLoader) + registerConfigChannels(cfg, channelMgr, msgBus, pgStores, instanceLoader, audioMgr) // Register channels/instances/links/teams RPC methods wireChannelRPCMethods(server, pgStores, channelMgr, agentRouter, msgBus, workspace) diff --git a/cmd/gateway_agents.go b/cmd/gateway_agents.go index 7ec00a86..e2cb91bf 100644 --- a/cmd/gateway_agents.go +++ b/cmd/gateway_agents.go @@ -4,6 +4,8 @@ import ( "context" "log/slog" + "github.com/nextlevelbuilder/goclaw/internal/audio/elevenlabs" + minimaxaudio "github.com/nextlevelbuilder/goclaw/internal/audio/minimax" "github.com/nextlevelbuilder/goclaw/internal/bus" "github.com/nextlevelbuilder/goclaw/internal/config" "github.com/nextlevelbuilder/goclaw/internal/memory" @@ -187,6 +189,9 @@ func setupSubagents(providerReg *providers.Registry, cfg *config.Config, msgBus if sc.ArchiveAfterMinutes > 0 { subCfg.ArchiveAfterMinutes = sc.ArchiveAfterMinutes } + if sc.MaxRetries > 0 { + subCfg.MaxRetries = sc.MaxRetries + } if sc.Model != "" { subCfg.Model = sc.Model } @@ -274,3 +279,55 @@ func setupTTS(cfg *config.Config) *tts.Manager { return mgr } + +// setupAudioExtras wires Music and SFX providers into the audio Manager. +// ElevenLabs is registered for both SFX and Music when an API key is present. +// MiniMax music is registered when cfg.Audio.Music is configured with a key. +// Phase 4 will add STT providers here. +func setupAudioExtras(cfg *config.Config, mgr *tts.Manager) { + ellKey := cfg.Tts.ElevenLabs.APIKey + ellBase := cfg.Tts.ElevenLabs.BaseURL + + // ElevenLabs SFX — reuse TTS credentials. + if ellKey != "" { + mgr.RegisterSFX(elevenlabs.NewSFXProvider(elevenlabs.Config{ + APIKey: ellKey, + BaseURL: ellBase, + })) + slog.Info("audio.sfx: elevenlabs registered") + } + + // ElevenLabs Music — same credentials, uses /v1/music endpoint. + if ellKey != "" { + mgr.RegisterMusic(elevenlabs.NewMusicProvider(elevenlabs.Config{ + APIKey: ellKey, + BaseURL: ellBase, + })) + slog.Info("audio.music: elevenlabs registered") + } + + // MiniMax Music — optional, from cfg.Audio.Music block. + if cfg.Audio != nil && cfg.Audio.Music != nil { + mc := cfg.Audio.Music + if mc.APIKey != "" { + mgr.RegisterMusic(minimaxaudio.NewMusicProvider(minimaxaudio.MusicConfig{ + APIKey: mc.APIKey, + APIBase: mc.BaseURL, + Model: mc.Model, + })) + slog.Info("audio.music: minimax registered") + } + } + + // ElevenLabs STT (Scribe v2) — reuse TTS credentials. Registered as tenant-scope + // default; per-request tenant override lands via builtin_tools[stt] in Phase 5 + // channel migration. Legacy per-channel STTProxyURL is bridged separately. + if ellKey != "" { + mgr.RegisterSTT(elevenlabs.NewSTTProvider(elevenlabs.Config{ + APIKey: ellKey, + BaseURL: ellBase, + })) + mgr.SetSTTChain([]string{"elevenlabs", "proxy"}) + slog.Info("audio.stt: elevenlabs registered") + } +} diff --git a/cmd/gateway_announce_queue.go b/cmd/gateway_announce_queue.go index bc79a379..561f067d 100644 --- a/cmd/gateway_announce_queue.go +++ b/cmd/gateway_announce_queue.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "log/slog" + "path/filepath" "strings" "github.com/google/uuid" @@ -103,6 +104,7 @@ func processAnnounceLoop( req.ForwardMedia = append(req.ForwardMedia, bus.MediaFile{ Path: mr.Path, MimeType: mr.ContentType, + Filename: filepath.Base(mr.Path), // preserve sanitized stem from producer }) } } diff --git a/cmd/gateway_builtin_tools.go b/cmd/gateway_builtin_tools.go index 2dca1635..548fe1bc 100644 --- a/cmd/gateway_builtin_tools.go +++ b/cmd/gateway_builtin_tools.go @@ -71,6 +71,10 @@ func builtinToolSeedData() []store.BuiltinToolDef { Requires: []string{"tts_provider"}, Metadata: json.RawMessage(`{"config_hint":"Config → TTS"}`), }, + {Name: "stt", DisplayName: "Speech-to-Text", Description: "Transcribe voice/audio messages to text using ElevenLabs Scribe or a proxy service", Category: "media", Enabled: true, + Requires: []string{"stt_provider"}, + Metadata: json.RawMessage(`{"config_hint":"Config → Audio → STT"}`), + }, // browser {Name: "browser", DisplayName: "Browser", Description: "Automate browser interactions: navigate pages, click elements, fill forms, take screenshots", Category: "browser", Enabled: true, diff --git a/cmd/gateway_channels_setup.go b/cmd/gateway_channels_setup.go index d9d97197..df2840a3 100644 --- a/cmd/gateway_channels_setup.go +++ b/cmd/gateway_channels_setup.go @@ -9,6 +9,7 @@ import ( "github.com/google/uuid" "github.com/nextlevelbuilder/goclaw/internal/agent" + "github.com/nextlevelbuilder/goclaw/internal/audio" "github.com/nextlevelbuilder/goclaw/internal/bus" "github.com/nextlevelbuilder/goclaw/internal/channels" "github.com/nextlevelbuilder/goclaw/internal/channels/discord" @@ -27,7 +28,8 @@ import ( ) // registerConfigChannels registers config-based channels as fallback when no DB instances are loaded. -func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, msgBus *bus.MessageBus, pgStores *store.Stores, instanceLoader *channels.InstanceLoader) { +// audioMgr is optional (nil = STT disabled for channels). +func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, msgBus *bus.MessageBus, pgStores *store.Stores, instanceLoader *channels.InstanceLoader, audioMgr *audio.Manager) { if instanceLoader != nil { return } @@ -46,7 +48,7 @@ func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, ms if cfg.Channels.Telegram.Enabled { if cfg.Channels.Telegram.Token == "" { recordMissingConfig(channels.TypeTelegram, "Set channels.telegram.token in config.") - } else if tg, err := telegram.New(cfg.Channels.Telegram, msgBus, pgStores.Pairing); err != nil { + } else if tg, err := telegram.New(cfg.Channels.Telegram, msgBus, pgStores.Pairing, audioMgr); err != nil { channelMgr.RecordFailure(channels.TypeTelegram, "", err) slog.Error("failed to initialize telegram channel", "error", err) } else { @@ -58,7 +60,7 @@ func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, ms if cfg.Channels.Discord.Enabled { if cfg.Channels.Discord.Token == "" { recordMissingConfig(channels.TypeDiscord, "Set channels.discord.token in config.") - } else if dc, err := discord.New(cfg.Channels.Discord, msgBus, nil, nil, nil, nil); err != nil { + } else if dc, err := discord.New(cfg.Channels.Discord, msgBus, nil, nil, nil, nil, audioMgr); err != nil { channelMgr.RecordFailure(channels.TypeDiscord, "", err) slog.Error("failed to initialize discord channel", "error", err) } else { @@ -72,7 +74,7 @@ func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, ms if strings.Contains(fmt.Sprintf("%T", pgStores.DB.Driver()), "sqlite") { waDialect = "sqlite3" } - wa, err := whatsapp.New(cfg.Channels.WhatsApp, msgBus, pgStores.Pairing, pgStores.DB, pgStores.PendingMessages, waDialect) + wa, err := whatsapp.New(cfg.Channels.WhatsApp, msgBus, pgStores.Pairing, pgStores.DB, pgStores.PendingMessages, waDialect, audioMgr, pgStores.BuiltinTools) if err != nil { channelMgr.RecordFailure(channels.TypeWhatsApp, "", err) slog.Error("failed to initialize whatsapp channel", "error", err) @@ -131,7 +133,7 @@ func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, ms feishu.WithAgentStore(pgStores.Agents), feishu.WithConfigPermStore(pgStores.ConfigPermissions), } - if f, err := feishu.New(cfg.Channels.Feishu, msgBus, pgStores.Pairing, nil, feishuOpts...); err != nil { + if f, err := feishu.New(cfg.Channels.Feishu, msgBus, pgStores.Pairing, nil, audioMgr, feishuOpts...); err != nil { channelMgr.RecordFailure(channels.TypeFeishu, "", err) slog.Error("failed to initialize feishu channel", "error", err) } else { diff --git a/cmd/gateway_consumer_handlers.go b/cmd/gateway_consumer_handlers.go index 3be0236b..66216cfe 100644 --- a/cmd/gateway_consumer_handlers.go +++ b/cmd/gateway_consumer_handlers.go @@ -109,6 +109,13 @@ func handleSubagentAnnounce( Iterations: iterations, } + // Preserve real acting sender + RBAC role from original turn so permission + // checks (e.g. write_file in group chat) attribute to the user and can + // bypass per-user grants for authenticated admins, not the synthetic + // "subagent:" sender of the announce message itself (#915). + originSenderID := msg.Metadata[tools.MetaOriginSenderID] + originRole := msg.Metadata[tools.MetaOriginRole] + queueKey := fmt.Sprintf("%s:%s", msg.TenantID, sessionKey) routing := subagentAnnounceRouting{ QueueKey: queueKey, @@ -120,6 +127,8 @@ func handleSubagentAnnounce( OrigPeerKind: origPeerKind, OrigLocalKey: origLocalKey, UserID: announceUserID, + SenderID: originSenderID, + Role: originRole, ParentAgent: parentAgent, ParentTraceID: parentTraceID, ParentRootSpanID: parentRootSpanID, @@ -129,16 +138,14 @@ func handleSubagentAnnounce( // Enqueue into producer-consumer queue using tenant-scoped key from routing. isProcessor := enqueueSubagentAnnounce(queueKey, entry) if isProcessor { - deps.BgWg.Add(1) - go func() { - defer deps.BgWg.Done() + deps.BgWg.Go(func() { defer safego.Recover(nil, "component", "subagent_announce_loop", "session", sessionKey) // Fetch live roster for merged announce context. roster := deps.SubagentMgr.RosterForParent(parentAgent) processSubagentAnnounceLoop(ctx, routing, roster, deps.SubagentMgr, deps.Sched, deps.MsgBus, deps.Cfg) - }() + }) } return true @@ -207,6 +214,13 @@ func handleTeammateMessage( announceUserID = fmt.Sprintf("group:%s:%s", origChannel, origChatID) } + // Preserve real acting sender + RBAC role through teammate dispatch so + // permission checks during the teammate's turn (e.g. write_file in group + // chat) attribute to the original user and can bypass per-user grants + // for authenticated admins (#915). + teammateSenderID := msg.Metadata[tools.MetaOriginSenderID] + teammateRole := msg.Metadata[tools.MetaOriginRole] + outMeta := buildAnnounceOutMeta(origLocalKey) // Link member agent trace back to lead's trace for unified tracing. @@ -235,6 +249,8 @@ func handleTeammateMessage( PeerKind: origPeerKind, LocalKey: origLocalKey, UserID: announceUserID, + SenderID: teammateSenderID, // real user who triggered the teammate dispatch (#915) + Role: teammateRole, // RBAC role for admin bypass during teammate turn (#915) RunID: fmt.Sprintf("teammate-%s-%s", msg.Metadata[tools.MetaFromAgent], msg.Metadata[tools.MetaToAgent]), Stream: false, TeamTaskID: msg.Metadata[tools.MetaTeamTaskID], diff --git a/cmd/gateway_consumer_normal.go b/cmd/gateway_consumer_normal.go index 85313c5f..422d2904 100644 --- a/cmd/gateway_consumer_normal.go +++ b/cmd/gateway_consumer_normal.go @@ -11,6 +11,7 @@ import ( "github.com/nextlevelbuilder/goclaw/internal/agent" "github.com/nextlevelbuilder/goclaw/internal/bus" + "github.com/nextlevelbuilder/goclaw/internal/channels" "github.com/nextlevelbuilder/goclaw/internal/channels/telegram/voiceguard" "github.com/nextlevelbuilder/goclaw/internal/i18n" "github.com/nextlevelbuilder/goclaw/internal/scheduler" @@ -214,22 +215,12 @@ func processNormalMessage( // Build outbound metadata for reply-to + thread routing BEFORE RegisterRun // so block.reply handler can use it for routing intermediate messages. - outMeta := make(map[string]string) + outMeta := channels.CopyFinalRoutingMeta(msg.Metadata) if isGroup { if mid := msg.Metadata["message_id"]; mid != "" { outMeta["reply_to_message_id"] = mid } } - // Channel routing keys — keep in sync with routingMetaKeys in channels/events.go. - for _, k := range []string{ - tools.MetaMessageThreadID, "local_key", "placeholder_key", "group_id", - "feishu_reply_target_id", - "fb_mode", "sender_id", "page_id", "reply_to_comment_id", - } { - if v := msg.Metadata[k]; v != "" { - outMeta[k] = v - } - } // Register run with channel manager for streaming/reaction event forwarding. // Use localKey (composite key with topic suffix) so streaming/reaction events @@ -355,6 +346,29 @@ func processNormalMessage( schedCtx = tools.WithRunKind(schedCtx, rk) } + // Resolve effective sender: prefer MetaOriginSenderID when the on-wire + // SenderID is an internal/synthetic one (e.g. "notification:progress", + // "ticker:system", "system:escalation", "session_send_tool"). This lets + // system-initiated turns that DO have a real user behind them (because + // they propagated the origin via metadata) attribute actions to that user + // — e.g. for CheckFileWriterPermission in group chats (#915). Synthetic + // senders without propagation keep their on-wire value and hit F1's + // deny-in-group rule (safe default). + effectiveSenderID := msg.SenderID + if bus.IsInternalSender(effectiveSenderID) { + // Defense-in-depth: if a propagation bug ever writes a synthetic + // value into MetaOriginSenderID, do NOT honour it. We want only real + // user senders to override the on-wire synthetic. + if realSender := msg.Metadata[tools.MetaOriginSenderID]; realSender != "" && !bus.IsInternalSender(realSender) { + effectiveSenderID = realSender + } + } + // Role propagation: carry the RBAC role of the originating actor so + // permission checks during the re-ingress turn can bypass per-user + // grants for authenticated admins (#915). Only present when the + // upstream dispatch set MetaOriginRole. + effectiveRole := msg.Metadata[tools.MetaOriginRole] + // Schedule through main lane (per-session concurrency controlled by maxConcurrent) outCh := deps.Sched.ScheduleWithOpts(schedCtx, "main", agent.RunRequest{ SessionKey: sessionKey, @@ -368,7 +382,8 @@ func processNormalMessage( PeerKind: peerKind, LocalKey: msg.Metadata["local_key"], UserID: userID, - SenderID: msg.SenderID, + SenderID: effectiveSenderID, + Role: effectiveRole, SenderName: resolveSenderName(msg), RunID: runID, Stream: enableStream, @@ -415,10 +430,19 @@ func processNormalMessage( return } slog.Error("inbound: agent run failed", "error", outcome.Err, "channel", channel) + // Suppress technical error text on public-facing channels (FB, Telegram, etc.) + // Empty Content still triggers placeholder/typing cleanup downstream. + errContent := formatAgentError(outcome.Err) + if deps.ChannelMgr != nil { + if ct := deps.ChannelMgr.ChannelTypeForName(channel); isExternalChannel(ct) { + slog.Info("inbound: suppressed error for external channel", "channel", channel, "type", ct) + errContent = "" + } + } deps.MsgBus.PublishOutbound(bus.OutboundMessage{ Channel: channel, ChatID: chatID, - Content: formatAgentError(outcome.Err), + Content: errContent, Metadata: meta, }) return diff --git a/cmd/gateway_consumer_process.go b/cmd/gateway_consumer_process.go index 08be6a06..4b92a99f 100644 --- a/cmd/gateway_consumer_process.go +++ b/cmd/gateway_consumer_process.go @@ -40,7 +40,7 @@ func makeSchedulerRunFunc(agents *agent.Router, cfg *config.Config) scheduler.Ru // The ctx from the scheduler is already cancellable; we create a child so the router's // cancel func is independent from the scheduler's cancel func. Calling cancel twice is safe. runCtx, cancel := context.WithCancel(ctx) - injectCh := agents.RegisterRun(req.RunID, req.SessionKey, agentID, cancel) + injectCh := agents.RegisterRun(runCtx, req.RunID, req.SessionKey, agentID, cancel) defer agents.UnregisterRun(req.RunID) defer cancel() diff --git a/cmd/gateway_deps.go b/cmd/gateway_deps.go index a57a3852..0487f358 100644 --- a/cmd/gateway_deps.go +++ b/cmd/gateway_deps.go @@ -2,12 +2,14 @@ package cmd import ( "github.com/nextlevelbuilder/goclaw/internal/agent" + "github.com/nextlevelbuilder/goclaw/internal/audio" "github.com/nextlevelbuilder/goclaw/internal/bus" "github.com/nextlevelbuilder/goclaw/internal/cache" "github.com/nextlevelbuilder/goclaw/internal/channels" "github.com/nextlevelbuilder/goclaw/internal/config" "github.com/nextlevelbuilder/goclaw/internal/eventbus" "github.com/nextlevelbuilder/goclaw/internal/gateway" + httpapi "github.com/nextlevelbuilder/goclaw/internal/http" "github.com/nextlevelbuilder/goclaw/internal/providers" "github.com/nextlevelbuilder/goclaw/internal/skills" "github.com/nextlevelbuilder/goclaw/internal/store" @@ -33,4 +35,6 @@ type gatewayDeps struct { workspace string dataDir string domainBus eventbus.DomainEventBus + audioMgr *audio.Manager // nil if TTS not configured; used by TTSHandler + ttsHandler *httpapi.TTSHandler // nil if TTS not configured; for hot-reload } diff --git a/cmd/gateway_errors.go b/cmd/gateway_errors.go index 95ee5db1..61dde32b 100644 --- a/cmd/gateway_errors.go +++ b/cmd/gateway_errors.go @@ -74,6 +74,27 @@ func isContextOverflowError(lower string) bool { containsAny(lower, "overflow", "too large", "too long", "limit", "exceeded")) } +// isExternalChannel reports whether a channel type serves end users on a +// public-facing platform (Facebook, Telegram, etc.). Internal error details +// must not be forwarded to these channels — the caller publishes an empty +// outbound instead so placeholders get cleaned up without leaking technical +// error text to end users. Internal types ("ws", "") return false. +func isExternalChannel(channelType string) bool { + switch channelType { + case channels.TypeFacebook, + channels.TypeTelegram, + channels.TypeDiscord, + channels.TypeFeishu, + channels.TypeWhatsApp, + channels.TypeZaloOA, + channels.TypeZaloPersonal, + channels.TypePancake, + channels.TypeSlack: + return true + } + return false +} + // isMessageFormatError checks for tool_use/tool_result mismatch, role ordering, // and other message format errors that indicate corrupted session history. func isMessageFormatError(lower string) bool { diff --git a/cmd/gateway_errors_test.go b/cmd/gateway_errors_test.go new file mode 100644 index 00000000..6efcf484 --- /dev/null +++ b/cmd/gateway_errors_test.go @@ -0,0 +1,49 @@ +package cmd + +import ( + "testing" + + "github.com/nextlevelbuilder/goclaw/internal/channels" +) + +// TestIsExternalChannel ensures the whitelist matches actual channel type +// constants — a mismatch here silently leaks internal error strings to end +// users (e.g. "zalo" vs "zalo_oa"/"zalo_personal"). Always compare against +// channels.Type* constants, never string literals. +func TestIsExternalChannel(t *testing.T) { + t.Parallel() + + cases := []struct { + name string + channelType string + want bool + }{ + // Public-facing platforms — errors must be suppressed. + {"facebook", channels.TypeFacebook, true}, + {"telegram", channels.TypeTelegram, true}, + {"discord", channels.TypeDiscord, true}, + {"feishu", channels.TypeFeishu, true}, + {"whatsapp", channels.TypeWhatsApp, true}, + {"zalo_oa", channels.TypeZaloOA, true}, + {"zalo_personal", channels.TypeZaloPersonal, true}, + {"pancake", channels.TypePancake, true}, + {"slack", channels.TypeSlack, true}, + + // Internal / unknown channel types — errors must still surface. + {"ws", "ws", false}, + {"empty", "", false}, + {"unknown", "myplatform", false}, + // "line" is a Pancake sub-platform, not a top-level channel type. + {"line_not_top_level", "line", false}, + // Legacy short "zalo" string must NOT match — real constants are zalo_oa / zalo_personal. + {"zalo_short_form", "zalo", false}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := isExternalChannel(tc.channelType); got != tc.want { + t.Fatalf("isExternalChannel(%q) = %v, want %v", tc.channelType, got, tc.want) + } + }) + } +} diff --git a/cmd/gateway_github_installer.go b/cmd/gateway_github_installer.go new file mode 100644 index 00000000..3f91e4fb --- /dev/null +++ b/cmd/gateway_github_installer.go @@ -0,0 +1,63 @@ +package cmd + +import ( + "log/slog" + "os" + "path/filepath" + "strconv" + "strings" + + "github.com/nextlevelbuilder/goclaw/internal/skills" +) + +// initGitHubInstaller constructs the process-wide GitHub Releases installer +// from environment variables and registers it via skills.SetDefaultGitHubInstaller. +// +// Config comes ONLY from env vars (token is a secret — never from config.json): +// +// GOCLAW_PACKAGES_GITHUB_TOKEN optional PAT (boosts rate limit, enables private repos) +// GOCLAW_PACKAGES_MAX_ASSET_SIZE_MB default 200 +// GOCLAW_PACKAGES_GITHUB_ALLOWED_ORGS comma-separated allowlist (empty = all allowed) +// GOCLAW_PACKAGES_GITHUB_BIN_DIR default /app/data/.runtime/bin +// GOCLAW_PACKAGES_GITHUB_MANIFEST default {BIN_DIR}/../github-packages.json +func initGitHubInstaller() { + cfg := &skills.GitHubPackagesConfig{ + Token: os.Getenv("GOCLAW_PACKAGES_GITHUB_TOKEN"), + BinDir: os.Getenv("GOCLAW_PACKAGES_GITHUB_BIN_DIR"), + ManifestPath: os.Getenv("GOCLAW_PACKAGES_GITHUB_MANIFEST"), + } + if v := os.Getenv("GOCLAW_PACKAGES_MAX_ASSET_SIZE_MB"); v != "" { + if n, err := strconv.Atoi(v); err == nil && n > 0 { + cfg.MaxAssetSizeMB = n + } + } + if v := os.Getenv("GOCLAW_PACKAGES_GITHUB_ALLOWED_ORGS"); v != "" { + for _, o := range strings.Split(v, ",") { + if o = strings.TrimSpace(o); o != "" { + cfg.AllowedOrgs = append(cfg.AllowedOrgs, o) + } + } + } + + // NewGitHubInstaller calls cfg.Defaults() — no need to invoke it here. + client := skills.NewGitHubClient(cfg.Token) + installer := skills.NewGitHubInstaller(client, cfg) + skills.SetDefaultGitHubInstaller(installer) + + // Best-effort ensure bin dir + manifest dir exist (entrypoint may run as root + // while Go process runs as goclaw — respect pre-existing permissions). + if err := os.MkdirAll(cfg.BinDir, 0o755); err != nil { + slog.Warn("github.installer: mkdir bin dir failed", "path", cfg.BinDir, "error", err) + } + if err := os.MkdirAll(filepath.Dir(cfg.ManifestPath), 0o755); err != nil { + slog.Warn("github.installer: mkdir manifest dir failed", "path", cfg.ManifestPath, "error", err) + } + + slog.Info("packages: github installer enabled", + "bin_dir", cfg.BinDir, + "manifest", cfg.ManifestPath, + "allowed_orgs", cfg.AllowedOrgs, + "max_asset_mb", cfg.MaxAssetSizeMB, + "token_set", cfg.Token != "", + ) +} diff --git a/cmd/gateway_hooks.go b/cmd/gateway_hooks.go new file mode 100644 index 00000000..26b35887 --- /dev/null +++ b/cmd/gateway_hooks.go @@ -0,0 +1,62 @@ +package cmd + +import ( + "os" + "time" + + "github.com/nextlevelbuilder/goclaw/internal/config" + "github.com/nextlevelbuilder/goclaw/internal/edition" + "github.com/nextlevelbuilder/goclaw/internal/hooks" + hookhandlers "github.com/nextlevelbuilder/goclaw/internal/hooks/handlers" + "github.com/nextlevelbuilder/goclaw/internal/hooks/budget" + "github.com/nextlevelbuilder/goclaw/internal/providers" + "github.com/nextlevelbuilder/goclaw/internal/security" + "github.com/nextlevelbuilder/goclaw/internal/store" + "github.com/nextlevelbuilder/goclaw/internal/store/pg" +) + +// sharedHookHandlers is populated by wireExtras so the gateway.go router +// wiring can reuse the same handler instances for the `hooks.test` runner. +// nil when hook store is absent (hooks disabled). +var sharedHookHandlers map[hooks.HandlerType]hooks.Handler + +// buildHookHandlers constructs the production handler map used by both the +// dispatcher (sync + async chain) and the `hooks.test` test runner. Keeping +// this factory single-source ensures test-panel behavior mirrors production. +// +// Budget wiring (C1 fix): the PromptHandler receives a budget.Store bound +// to pg.NewPGHookBudget so token spend is atomically deducted per tenant. +// When the DB handle is unavailable, budget falls back to nil (Lite desktop). +func buildHookHandlers(stores *store.Stores, providerReg *providers.Registry, hooksCfg config.HooksConfig) map[hooks.HandlerType]hooks.Handler { + encryptKey := os.Getenv("GOCLAW_ENCRYPTION_KEY") + + var budgetStore *budget.Store + if stores != nil && stores.DB != nil { + budgetStore = budget.New(pg.NewPGHookBudget(stores.DB), nil) + } + + promptHandler := &hookhandlers.PromptHandler{ + Resolver: hookhandlers.NewRegistryResolver(providerReg, stores.SystemConfigs), + Budget: budgetStore, + DefaultModel: "haiku", + } + + // ScriptHandler: bounded by cfg.Hooks caps; zero values fall back to + // handler defaults (10 / 3 / 500). Safe for concurrent reuse across + // dispatcher + hooks.test runner (each Execute allocates its own runtime). + scriptHandler := hookhandlers.NewScriptHandler( + hooksCfg.ScriptConcurrency, + hooksCfg.ScriptPerTenantConcurrency, + hooksCfg.ScriptCacheSize, + ) + + return map[hooks.HandlerType]hooks.Handler{ + hooks.HandlerCommand: &hookhandlers.CommandHandler{Edition: edition.Current()}, + hooks.HandlerHTTP: &hookhandlers.HTTPHandler{ + EncryptKey: encryptKey, + Client: security.NewSafeClient(10 * time.Second), + }, + hooks.HandlerPrompt: promptHandler, + hooks.HandlerScript: scriptHandler, + } +} diff --git a/cmd/gateway_http_wiring.go b/cmd/gateway_http_wiring.go index 4672946d..bab60b1d 100644 --- a/cmd/gateway_http_wiring.go +++ b/cmd/gateway_http_wiring.go @@ -3,8 +3,11 @@ package cmd import ( "context" "log/slog" + "time" + "github.com/nextlevelbuilder/goclaw/internal/audio" "github.com/nextlevelbuilder/goclaw/internal/bus" + "github.com/nextlevelbuilder/goclaw/internal/gateway/methods" httpapi "github.com/nextlevelbuilder/goclaw/internal/http" mcpbridge "github.com/nextlevelbuilder/goclaw/internal/mcp" "github.com/nextlevelbuilder/goclaw/internal/media" @@ -130,7 +133,8 @@ func (d *gatewayDeps) wireHTTPHandlersOnServer( d.server.SetUsageHandler(httpapi.NewUsageHandler(d.pgStores.Snapshots, d.pgStores.DB)) } - // Runtime package management (install/uninstall system/pip/npm packages) + // Runtime package management (install/uninstall system/pip/npm/github packages) + initGitHubInstaller() d.server.SetPackagesHandler(httpapi.NewPackagesHandler()) // API documentation (OpenAPI spec + Swagger UI at /docs) @@ -222,6 +226,38 @@ func (d *gatewayDeps) wireHTTPHandlersOnServer( d.server.SetMediaServeHandler(httpapi.NewMediaServeHandler(mediaStore)) } + // ElevenLabs voice list + refresh endpoints (GET /v1/voices, POST /v1/voices/refresh). + // VoiceCache is shared between the HTTP handler and the WS voices.list method. + // TTL 1h + LRU cap 1000 tenants. + { + voiceCache := audio.NewVoiceCache(1*time.Hour, 1000) + var secretStore store.ConfigSecretsStore + if d.pgStores != nil && d.pgStores.ConfigSecrets != nil { + secretStore = d.pgStores.ConfigSecrets + } + var tenantStore store.TenantStore + if d.pgStores != nil && d.pgStores.Tenants != nil { + tenantStore = d.pgStores.Tenants + } + voicesH := httpapi.NewVoicesHandler(voiceCache, secretStore, tenantStore) + d.server.SetVoicesHandler(voicesH) + // Wire WS method — provider nil means each request resolves key via secretStore at HTTP layer. + // For WS, use same cache. Provider is resolved via secretStore at WS level in a future phase. + methods.NewVoicesMethods(voiceCache, nil).Register(d.server.Router()) + } + + // TTS synthesize endpoint — shares audio.Manager with setupTTS. + if d.audioMgr != nil { + ttsH := httpapi.NewTTSHandler(d.audioMgr) + // Reuse the server's rate limiter (per-IP/token; NOT per-user). + // Server.RateLimiter() is non-nil by construction (server.go:104). + if rl := d.server.RateLimiter(); rl != nil && rl.Enabled() { + ttsH.SetRateLimiter(rl.Allow) + } + d.server.SetTTSHandler(ttsH) + d.ttsHandler = ttsH // store for hot-reload + } + // Seed + apply builtin tool disables if d.pgStores.BuiltinTools != nil { seedBuiltinTools(context.Background(), d.pgStores.BuiltinTools) diff --git a/cmd/gateway_lifecycle.go b/cmd/gateway_lifecycle.go index 46b41ba4..75e2ec81 100644 --- a/cmd/gateway_lifecycle.go +++ b/cmd/gateway_lifecycle.go @@ -103,6 +103,9 @@ func (d *gatewayDeps) runLifecycle( return } deps.ttsTool.UpdateManager(newMgr) + if d.ttsHandler != nil { + d.ttsHandler.UpdateManager(newMgr) + } slog.Info("tts config reloaded", "provider", newMgr.PrimaryProvider(), "auto", string(newMgr.AutoMode())) }) diff --git a/cmd/gateway_managed.go b/cmd/gateway_managed.go index 5165a090..af837ef5 100644 --- a/cmd/gateway_managed.go +++ b/cmd/gateway_managed.go @@ -14,9 +14,11 @@ import ( "github.com/nextlevelbuilder/goclaw/internal/agent" "github.com/nextlevelbuilder/goclaw/internal/bus" "github.com/nextlevelbuilder/goclaw/internal/config" - "github.com/nextlevelbuilder/goclaw/internal/orchestration" "github.com/nextlevelbuilder/goclaw/internal/edition" "github.com/nextlevelbuilder/goclaw/internal/eventbus" + "github.com/nextlevelbuilder/goclaw/internal/hooks" + hookbuiltin "github.com/nextlevelbuilder/goclaw/internal/hooks/builtin" + "github.com/nextlevelbuilder/goclaw/internal/orchestration" httpapi "github.com/nextlevelbuilder/goclaw/internal/http" kg "github.com/nextlevelbuilder/goclaw/internal/knowledgegraph" mcpbridge "github.com/nextlevelbuilder/goclaw/internal/mcp" @@ -125,8 +127,10 @@ func wireExtras( // 5. Shared MCP connection pool (eliminates duplicate connections across agents) var mcpPool *mcpbridge.Pool + var mcpGrantChecker mcpbridge.GrantChecker if stores.MCP != nil { mcpPool = mcpbridge.NewPool(mcpbridge.DefaultPoolConfig()) + mcpGrantChecker = mcpbridge.NewStoreGrantChecker(stores.MCP, msgBus) } // 6. Set up agent resolver: lazy-creates Loops from DB @@ -144,6 +148,48 @@ func wireExtras( // vaultIntc is set later by wireVault but captured by closure in OnTextUploaded. var vaultIntc *tools.VaultInterceptor + // Agent Hooks (Issue #875) — lifecycle dispatcher + handlers. + var hookDispatcher hooks.Dispatcher = hooks.NewNoopDispatcher() + if hs, ok := stores.Hooks.(hooks.HookStore); ok && hs != nil { + // Phase 04: wire builtin registry. Install a strip-all lookup FIRST so a + // Load() failure leaves the dispatcher failing closed (no wide fallback + // via the Phase 03 permissive default). On successful Load we swap in the + // real per-id allowlist, then UPSERT canonical rows with stable UUIDv5s. + // Seed failures log but never block startup. + hooks.SetBuiltinAllowlistLookup(func(uuid.UUID) []string { return nil }) + if err := hookbuiltin.Load(); err != nil { + slog.Warn("hooks.builtin_load_failed", "err", err) + } else { + hooks.SetBuiltinAllowlistLookup(hookbuiltin.AllowlistFor) + if err := hookbuiltin.Seed(context.Background(), hs, appCfg.Hooks); err != nil { + slog.Warn("hooks.builtin_seed_failed", "err", err) + } + } + + // Phase 07: runtime migration — auto-disable legacy command-type hooks + // on Standard edition. No-op on Lite. Idempotent. Runs synchronously + // before listeners so traffic never sees a command hook fire on a + // post-Wave-1 Standard instance. + if n, err := hooks.DisableLegacyCommandHooks(context.Background(), hs, edition.Current()); err != nil { + slog.Warn("hooks.command_migration_failed", "err", err) + } else if n > 0 { + slog.Info("hooks.command_migration_ran", + "disabled_count", n, "edition", edition.Current().Name) + } + + handlers := buildHookHandlers(stores, providerReg, appCfg.Hooks) + stdOpts := hooks.StdDispatcherOpts{ + Store: hs, + Audit: hooks.NewAuditWriter(hs, ""), + Handlers: handlers, + } + hookDispatcher = hooks.NewStdDispatcher(stdOpts) + hooks.SubscribeDelegateEvents(domainBus, hookDispatcher) + // Stash handlers for later gateway.go wiring (test runner). + sharedHookHandlers = handlers + slog.Info("agent hooks dispatcher wired", "handlers", "command,http,prompt") + } + resolver := agent.NewManagedResolver(agent.ResolverDeps{ AgentStore: stores.Agents, ProviderStore: stores.Providers, @@ -177,6 +223,7 @@ func wireExtras( BuiltinToolStore: stores.BuiltinTools, MCPStore: stores.MCP, MCPPool: mcpPool, + MCPGrantChecker: mcpGrantChecker, ConfigPermStore: stores.ConfigPermissions, MediaStore: mediaStore, ModelPricing: appCfg.Telemetry.ModelPricing, @@ -191,6 +238,7 @@ func wireExtras( AutoInjector: autoInjector, EvolutionMetricsStore: stores.EvolutionMetrics, DomainBus: domainBus, + HookDispatcher: hookDispatcher, OnTextUploaded: func(ctx context.Context, path, content string) { if vaultIntc != nil { vaultIntc.AfterWrite(ctx, path, content) @@ -385,6 +433,7 @@ func wireExtras( } delegateTool := tools.NewDelegateTool(stores.AgentLinks, stores.Agents, domainBus, delegateRunFn) delegateTool.SetMsgBus(msgBus) + delegateTool.SetHookDispatcher(hookDispatcher) toolsReg.Register(delegateTool) slog.Info("delegate tool wired") } diff --git a/cmd/gateway_providers.go b/cmd/gateway_providers.go index b8e9edd9..b174ef44 100644 --- a/cmd/gateway_providers.go +++ b/cmd/gateway_providers.go @@ -373,16 +373,6 @@ func registerProvidersFromDB(registry *providers.Registry, provStore store.Provi base = "https://ollama.com/v1" } registry.RegisterForTenant(p.TenantID, providers.NewOpenAIProvider(p.Name, p.APIKey, base, "llama3.3")) - case store.ProviderSuno: - // Suno is a media-only provider (music gen). Register as OpenAI-compat - // so credentialProvider interface works for API key/base extraction. - base := p.APIBase - if base == "" { - base = "https://api.sunoapi.org" - } - prov := providers.NewOpenAIProvider(p.Name, p.APIKey, base, "") - prov.WithProviderType(p.ProviderType) - registry.RegisterForTenant(p.TenantID, prov) case store.ProviderNovita: base := p.APIBase if base == "" { diff --git a/cmd/gateway_setup.go b/cmd/gateway_setup.go index a919024a..730f7a8f 100644 --- a/cmd/gateway_setup.go +++ b/cmd/gateway_setup.go @@ -10,6 +10,7 @@ import ( "github.com/google/uuid" + "github.com/nextlevelbuilder/goclaw/internal/audio" "github.com/nextlevelbuilder/goclaw/internal/bootstrap" "github.com/nextlevelbuilder/goclaw/internal/bus" "github.com/nextlevelbuilder/goclaw/internal/config" @@ -44,6 +45,7 @@ func setupToolRegistry( browserMgr *browser.Manager, webFetchTool *tools.WebFetchTool, ttsTool *tools.TtsTool, + audioMgr *audio.Manager, permPE *permissions.PolicyEngine, toolPE *tools.PolicyEngine, dataDir string, @@ -133,15 +135,19 @@ func setupToolRegistry( toolsReg.Register(tools.NewReadImageTool(providerRegistry)) toolsReg.Register(tools.NewCreateImageTool(providerRegistry)) - // Audio generation tool (MiniMax music + ElevenLabs sound effects) - toolsReg.Register(tools.NewCreateAudioTool(providerRegistry, - cfg.Tts.ElevenLabs.APIKey, cfg.Tts.ElevenLabs.BaseURL)) - - // TTS (text-to-speech) system — always create TtsTool so config reload can populate it later + // Audio system: build Manager first so Music/SFX providers are registered + // before the create_audio tool is constructed. ttsMgr := setupTTS(cfg) if ttsMgr == nil { ttsMgr = tts.NewManager(tts.ManagerConfig{}) } + setupAudioExtras(cfg, ttsMgr) // Phase 3: registers Music + SFX providers. + audio.BridgeLegacySTT(ttsMgr, cfg) // Phase 4: bridge per-channel STTProxyURL → channel-scoped providers. + audioMgr = ttsMgr // expose to caller for channel STT wiring (Phase 5) + + // Audio generation tool — backed by audio.Manager (Music + SFX). + toolsReg.Register(tools.NewCreateAudioTool(ttsMgr)) + ttsTool = tools.NewTtsTool(ttsMgr) toolsReg.Register(ttsTool) if ttsMgr.HasProviders() { @@ -297,6 +303,14 @@ func wireTracingAndCron( Payload: map[string]any{"trace_ids": ids}, }) } + // Immediate status broadcast on every successful status write (bypasses 5s flush). + traceCollector.SetStatusBroadcaster(func(p tracing.TraceStatusPayload, tid uuid.UUID) { + msgBus.Broadcast(bus.Event{ + Name: protocol.EventTraceStatusChanged, + Payload: p, + TenantID: tid, + }) + }) traceCollector.Start() slog.Info("LLM tracing enabled") } diff --git a/cmd/gateway_setup_exec_workspace_test.go b/cmd/gateway_setup_exec_workspace_test.go index 6fc79ba8..cd210c92 100644 --- a/cmd/gateway_setup_exec_workspace_test.go +++ b/cmd/gateway_setup_exec_workspace_test.go @@ -22,7 +22,7 @@ func testExecToolFromGatewaySetup(t *testing.T, workspace, dataDir string) *tool cfg.Tools.Browser.Enabled = false providerRegistry := providers.NewRegistry(nil) - toolsReg, _, _, _, _, _, _, _, _, _, _ := setupToolRegistry(cfg, workspace, providerRegistry) + toolsReg, _, _, _, _, _, _, _, _, _, _, _ := setupToolRegistry(cfg, workspace, providerRegistry) execToolAny, ok := toolsReg.Get("exec") if !ok { diff --git a/cmd/gateway_subagent_announce_queue.go b/cmd/gateway_subagent_announce_queue.go index 87264d54..510947d2 100644 --- a/cmd/gateway_subagent_announce_queue.go +++ b/cmd/gateway_subagent_announce_queue.go @@ -47,6 +47,15 @@ func makeDelegateAnnounceCallback( if meta.OriginSessionKey != "" { batchMeta[tools.MetaOriginSessionKey] = meta.OriginSessionKey } + if meta.OriginSenderID != "" { + batchMeta[tools.MetaOriginSenderID] = meta.OriginSenderID + } + if meta.OriginRole != "" { + batchMeta[tools.MetaOriginRole] = meta.OriginRole + } + if meta.OriginUserID != "" { + batchMeta[tools.MetaOriginUserID] = meta.OriginUserID + } // Collect media from all items in the batch. var batchMedia []bus.MediaFile for _, item := range items { @@ -95,6 +104,8 @@ type subagentAnnounceRouting struct { OrigPeerKind string OrigLocalKey string UserID string + SenderID string // real acting sender (preserves permission attribution through re-ingress, #915) + Role string // caller's RBAC role; bypasses per-user grants for admin/operator/owner (#915) ParentAgent string ParentTraceID uuid.UUID ParentRootSpanID uuid.UUID @@ -168,6 +179,8 @@ func processSubagentAnnounceLoop( PeerKind: r.OrigPeerKind, LocalKey: r.OrigLocalKey, UserID: r.UserID, + SenderID: r.SenderID, // preserves real acting sender for permission checks (#915) + Role: r.Role, // preserves RBAC role for admin bypass in group writes (#915) RunID: fmt.Sprintf("subagent-announce-%s-%d", r.ParentAgent, len(entries)), RunKind: "announce", HideInput: true, @@ -182,10 +195,16 @@ func processSubagentAnnounceLoop( if outcome.Err != nil { if !errors.Is(outcome.Err, context.Canceled) { slog.Error("subagent announce: lead run failed", "error", outcome.Err, "batch_size", len(entries)) + errContent := formatAgentError(outcome.Err) + if isExternalChannel(r.OrigChannelType) { + slog.Info("subagent announce: suppressed error for external channel", + "channel", r.OrigChannel, "type", r.OrigChannelType) + errContent = "" + } msgBus.PublishOutbound(bus.OutboundMessage{ Channel: r.OrigChannel, ChatID: r.OrigChatID, - Content: formatAgentError(outcome.Err), + Content: errContent, Metadata: r.OutMeta, }) } diff --git a/cmd/tenant_backup_cli_helpers.go b/cmd/tenant_backup_cli_helpers.go index b11c1019..ddeafe67 100644 --- a/cmd/tenant_backup_cli_helpers.go +++ b/cmd/tenant_backup_cli_helpers.go @@ -5,14 +5,27 @@ import ( "fmt" "os" - _ "github.com/jackc/pgx/v5/stdlib" "github.com/google/uuid" + _ "github.com/jackc/pgx/v5/stdlib" "github.com/spf13/cobra" "github.com/nextlevelbuilder/goclaw/internal/config" "github.com/nextlevelbuilder/goclaw/internal/store/pg" ) +func openTenantBackupDB(cfg *config.Config) (*sql.DB, error) { + dsn := cfg.Database.PostgresDSN + if dsn == "" { + return nil, fmt.Errorf("GOCLAW_POSTGRES_DSN not configured") + } + + db, err := sql.Open("pgx", dsn) + if err != nil { + return nil, fmt.Errorf("open db: %w", err) + } + return db, nil +} + // resolveTenantForCLI opens the DB, looks up the tenant by slug or UUID, // and returns the resolved tenant ID, slug, and an open *sql.DB (caller must close). func resolveTenantForCLI(cmd *cobra.Command, cfg *config.Config, rawID, slug string) (uuid.UUID, string, *sql.DB, error) { @@ -20,14 +33,9 @@ func resolveTenantForCLI(cmd *cobra.Command, cfg *config.Config, rawID, slug str return uuid.Nil, "", nil, fmt.Errorf("--tenant or --tenant-id is required") } - dsn := cfg.Database.PostgresDSN - if dsn == "" { - return uuid.Nil, "", nil, fmt.Errorf("GOCLAW_POSTGRES_DSN not configured") - } - - db, err := sql.Open("pgx", dsn) + db, err := openTenantBackupDB(cfg) if err != nil { - return uuid.Nil, "", nil, fmt.Errorf("open db: %w", err) + return uuid.Nil, "", nil, err } ts := pg.NewPGTenantStore(db) diff --git a/cmd/tenant_restore.go b/cmd/tenant_restore.go index 2ca81f72..57744722 100644 --- a/cmd/tenant_restore.go +++ b/cmd/tenant_restore.go @@ -1,9 +1,12 @@ package cmd import ( + "database/sql" "fmt" "os" + "strings" + "github.com/google/uuid" "github.com/spf13/cobra" "github.com/nextlevelbuilder/goclaw/internal/backup" @@ -12,11 +15,12 @@ import ( func tenantRestoreCmd() *cobra.Command { var ( - tenantSlug string - tenantID string - mode string - force bool - dryRun bool + tenantSlug string + tenantID string + newTenantSlug string + mode string + force bool + dryRun bool ) cmd := &cobra.Command{ @@ -25,9 +29,14 @@ func tenantRestoreCmd() *cobra.Command { Long: `Restores a tenant from a .tar.gz archive produced by 'goclaw tenant-backup'. Modes: - upsert (default) — INSERT … ON CONFLICT DO NOTHING. Non-destructive. - replace — Delete existing tenant data first, then INSERT. Requires --force. - new — Create a new tenant and import data under the new tenant ID.`, + upsert (default) — INSERT ... ON CONFLICT DO NOTHING. Non-destructive. + Requires --tenant or --tenant-id. + replace — Wipes tenant-scoped data (keeps tenant metadata), then INSERT. + Requires --tenant or --tenant-id AND --force. + new — Creates a new tenant from archive metadata. + Requires --new-tenant-slug. Archive's tenant_id is remapped + to the new tenant. Users, API keys, LLM providers, and all + other tenant-scoped data are cloned from the archive.`, Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { archivePath := args[0] @@ -41,6 +50,14 @@ Modes: if _, err := os.Stat(archivePath); err != nil { return fmt.Errorf("archive not found: %s", archivePath) } + + // Validate flag wiring before touching the filesystem state (--force check). + // Catches invalid combinations like mode=new with --tenant before the user + // sees destructive-operation prompts. + if err := validateTenantRestoreFlags(mode, tenantSlug, tenantID, newTenantSlug); err != nil { + return err + } + if mode == "replace" && !dryRun && !force { fmt.Fprintln(os.Stderr, "ERROR: --force is required for replace mode (destructive operation).") os.Exit(1) @@ -51,14 +68,27 @@ Modes: return fmt.Errorf("load config: %w", err) } - // For "new" mode the tenant may not exist yet — allow lookup failure. - tid, slug, db, lookupErr := resolveTenantForCLI(cmd, cfg, tenantID, tenantSlug) - if lookupErr != nil && mode != "new" { - return lookupErr - } - if db != nil { - defer db.Close() + var ( + tid = uuid.Nil + slug string + db *sql.DB + ) + var restoreErr error + + switch mode { + case "new": + db, restoreErr = openTenantBackupDB(cfg) + if restoreErr != nil { + return restoreErr + } + slug = strings.TrimSpace(newTenantSlug) + default: + tid, slug, db, restoreErr = resolveTenantForCLI(cmd, cfg, tenantID, tenantSlug) + if restoreErr != nil { + return restoreErr + } } + defer db.Close() dataDir := config.TenantDataDir(cfg.ResolvedDataDir(), tid, slug) wsDir := config.TenantWorkspace(cfg.WorkspacePath(), tid, slug) @@ -106,10 +136,35 @@ Modes: }, } - cmd.Flags().StringVar(&tenantSlug, "tenant", "", "target tenant slug") - cmd.Flags().StringVar(&tenantID, "tenant-id", "", "target tenant UUID") + cmd.Flags().StringVar(&tenantSlug, "tenant", "", "target tenant slug (for mode=upsert|replace)") + cmd.Flags().StringVar(&tenantID, "tenant-id", "", "target tenant UUID (for mode=upsert|replace, alternative to --tenant)") + cmd.Flags().StringVar(&newTenantSlug, "new-tenant-slug", "", "slug for the new tenant (required for mode=new)") cmd.Flags().StringVar(&mode, "mode", "upsert", "restore mode: upsert, replace, new") cmd.Flags().BoolVar(&force, "force", false, "required for replace mode") cmd.Flags().BoolVar(&dryRun, "dry-run", false, "inspect archive without making changes") return cmd } + +// validateTenantRestoreFlags enforces flag semantics per restore mode. +// Called before opening the database so invalid combinations fail fast. +func validateTenantRestoreFlags(mode, tenant, tenantID, newTenantSlug string) error { + switch mode { + case "new": + if strings.TrimSpace(newTenantSlug) == "" { + return fmt.Errorf("--new-tenant-slug is required for mode=new (choose a unique slug for the new tenant)") + } + if tenant != "" || tenantID != "" { + return fmt.Errorf("mode=new does not accept --tenant or --tenant-id; use --new-tenant-slug to name the new tenant") + } + case "upsert", "replace": + if tenant == "" && tenantID == "" { + return fmt.Errorf("--tenant or --tenant-id is required for mode=%s", mode) + } + if strings.TrimSpace(newTenantSlug) != "" { + fmt.Fprintf(os.Stderr, "warning: --new-tenant-slug is ignored for mode=%s\n", mode) + } + default: + return fmt.Errorf("invalid --mode=%q (allowed: upsert, replace, new)", mode) + } + return nil +} diff --git a/cmd/tenant_restore_test.go b/cmd/tenant_restore_test.go new file mode 100644 index 00000000..29b95ffb --- /dev/null +++ b/cmd/tenant_restore_test.go @@ -0,0 +1,42 @@ +package cmd + +import ( + "strings" + "testing" +) + +func TestValidateTenantRestoreFlags(t *testing.T) { + tests := []struct { + name, mode, tenant, tenantID, newSlug, wantErr string + }{ + {"new ok", "new", "", "", "fresh-slug", ""}, + {"new missing slug", "new", "", "", "", "required for mode=new"}, + {"new with tenant slug", "new", "acme", "", "fresh-slug", "does not accept"}, + {"new with tenant-id", "new", "", "uuid-x", "fresh-slug", "does not accept"}, + {"upsert ok tenant", "upsert", "acme", "", "", ""}, + {"upsert ok tenant-id", "upsert", "", "uuid-x", "", ""}, + {"upsert ok both", "upsert", "acme", "uuid-x", "", ""}, + {"upsert missing", "upsert", "", "", "", "required for mode=upsert"}, + {"upsert ignores new slug (no error)", "upsert", "acme", "", "ignored", ""}, + {"replace ok", "replace", "acme", "", "", ""}, + {"replace missing", "replace", "", "", "", "required for mode=replace"}, + {"invalid mode", "bogus", "acme", "", "", "invalid --mode"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + err := validateTenantRestoreFlags(tc.mode, tc.tenant, tc.tenantID, tc.newSlug) + if tc.wantErr == "" { + if err != nil { + t.Fatalf("want nil, got %v", err) + } + return + } + if err == nil { + t.Fatalf("want err containing %q, got nil", tc.wantErr) + } + if !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("want err containing %q, got %q", tc.wantErr, err.Error()) + } + }) + } +} diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index 1eaa8896..3b7e36c1 100644 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -7,7 +7,7 @@ RUNTIME_DIR="/app/data/.runtime" # Non-fatal: on first start with a fresh volume the directory may not be # writable yet (volume initialisation race on some Docker runtimes). # The app starts fine without .runtime; package installs will fail gracefully. -mkdir -p "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global/lib" "$RUNTIME_DIR/pip-cache" || true +mkdir -p "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global/lib" "$RUNTIME_DIR/pip-cache" "$RUNTIME_DIR/bin" || true # Fix .runtime ownership for split root/goclaw access. # .runtime itself must be root-owned so pkg-helper (root) can write apk-packages. @@ -16,7 +16,7 @@ mkdir -p "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global/lib" "$RUNTIME_DIR/pip-cach if [ "$(id -u)" = "0" ] && [ -d "$RUNTIME_DIR" ]; then chown root:goclaw "$RUNTIME_DIR" 2>/dev/null || true chmod 0750 "$RUNTIME_DIR" 2>/dev/null || true - chown -R goclaw:goclaw "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global" "$RUNTIME_DIR/pip-cache" 2>/dev/null || true + chown -R goclaw:goclaw "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global" "$RUNTIME_DIR/pip-cache" "$RUNTIME_DIR/bin" 2>/dev/null || true fi # Fix workspace directory ownership: handle dirs created by root in previous @@ -38,7 +38,7 @@ export PIP_CACHE_DIR="$RUNTIME_DIR/pip-cache" # NODE_PATH includes both pre-installed system globals and runtime-installed globals. export NPM_CONFIG_PREFIX="$RUNTIME_DIR/npm-global" export NODE_PATH="/usr/local/lib/node_modules:$RUNTIME_DIR/npm-global/lib/node_modules:${NODE_PATH:-}" -export PATH="$RUNTIME_DIR/npm-global/bin:$RUNTIME_DIR/pip/bin:$PATH" +export PATH="$RUNTIME_DIR/bin:$RUNTIME_DIR/npm-global/bin:$RUNTIME_DIR/pip/bin:$PATH" # System packages: re-install on-demand packages persisted across recreates. # After chown above, root owns .runtime and can create this file. diff --git a/docs/00-architecture-overview.md b/docs/00-architecture-overview.md index 82533c5d..d4fba9b6 100644 --- a/docs/00-architecture-overview.md +++ b/docs/00-architecture-overview.md @@ -38,6 +38,7 @@ flowchart TD SCHED[Scheduler -- 4 Lanes] AR[Agent Router] LOOP[Agent Loop -- Think / Act / Observe] + HOOKS[Hook Dispatcher -- Lifecycle Events] end subgraph Providers["LLM Providers"] @@ -120,7 +121,8 @@ flowchart TD | `internal/permissions/` | RBAC policy engine (admin, operator, viewer roles) | | `internal/store/pg/pairing.go` | DM/device pairing service (8-character codes, database-backed) | | `internal/sandbox/` | Docker-based code execution sandbox | -| `internal/tts/` | Text-to-Speech providers: OpenAI, ElevenLabs, Edge, MiniMax | +| `internal/audio/` | Unified audio manager: 4 provider interfaces (TTS active; STT/Music/SFX stubbed/partial). Orchestrates ElevenLabs, OpenAI, Edge, MiniMax TTS providers. `internal/tts/` retained as backward-compat alias | +| `internal/tts/` | Backward-compat alias layer (24 symbols) — all pre-refactor callers compile unchanged | | `internal/http/` | HTTP API handlers: /v1/chat/completions, /v1/agents, /v1/skills, /v1/traces, /v1/mcp, /v1/delegations, summoner | | `internal/crypto/` | AES-256-GCM encryption for API keys | | `internal/tracing/` | LLM call tracing (traces + spans), in-memory buffer with periodic store flush | @@ -142,6 +144,67 @@ flowchart TD | `internal/workspace/` | Workspace context resolver: 6 scenarios (agent default, team lead, team member, dispatch, subagent, cron) | | `internal/vault/` | Knowledge Vault: wikilinks (semantic mesh), hybrid search (BM25+vector), filesystem sync, L0 auto-injection | | `internal/channels/whatsapp/` | Native WhatsApp channel via whatsmeow (replaces WhatsApp API), QR auth, media handling | +| `internal/hooks/` | Agent lifecycle hooks: event dispatcher (sync/async), concrete handlers (command/http), matchers (regex + CEL), audit logging, edition gating, cost safeguards. Events: SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, Stop, SubagentStart/Stop. Handlers: CommandHandler (shell, Lite-only), HTTPHandler (SSRF-hardened, auth decrypt) | +| `internal/hooks/handlers/` | Concrete hook handler implementations: `CommandHandler` (exec wrapper, JSON I/O, env allowlist, edition recheck), `HTTPHandler` (SSRF-hardened HTTP client, retry-once on 5xx, no redirects, encrypted auth headers) | + +--- + +## 3.5 Agent Hooks System + +**Lifecycle hooks** allow agents to perform custom logic at key execution points. The system is event-driven (sync/async), integrated into the 8-stage pipeline, and includes audit logging, edition gating, and security safeguards. + +### Event Types + +| Event | Stage | Sync/Async | Purpose | +|-------|-------|-----------|---------| +| **SessionStart** | ContextStage | Async | Fires once per session (first turn); before history loading | +| **UserPromptSubmit** | ContextStage | Sync | Fires on user message arrival; blocks with hook reason or mutates input | +| **PreToolUse** | ToolStage | Sync | Fires before each tool execution; blocks tool or mutates arguments | +| **PostToolUse** | ToolStage | Async | Fires after tool result is processed; non-blocking | +| **Stop** | FinalizeStage | Async | Fires when session terminates (stop/complete/error) | +| **SubagentStart** | (Delegate tool) | Sync | Fires before delegated task spawns; blocks delegation | +| **SubagentStop** | (Domain events) | Async | Fires on delegation completion/failure; non-blocking | + +### Handler Types + +| Handler | Edition | Config | Semantics | +|---------|---------|--------|-----------| +| **Command** | Lite only | `cmd`, `allowedEnvVars` | Exec shell command; stdin=event JSON; stdout=decision JSON; exit 0→allow, exit 2→block; timeout→block (fail-closed) | +| **HTTP** | All | `url`, `headers` | POST event JSON to webhook URL; parse response for decision, additionalContext, updatedInput; 5xx retry once; 4xx error no-retry; SSRF-hardened with pinned IP | +| **Prompt** | Phase 3+ | TBD | Integrates custom prompting logic (deferred) | + +### Sync vs Async + +**Sync hooks** (UserPromptSubmit, PreToolUse, SubagentStart): +- Block pipeline until decision received +- Support Copy-on-Write (COW) staged mutations: `updatedInput` buffered, committed only if ALL sync hooks succeed +- Timeout ≤5s per hook; chain total ≤10s wall-time +- Rejection blocks the step (e.g., tool not executed, user message not processed) + +**Async hooks** (SessionStart, PostToolUse, Stop, SubagentStop): +- Fire-and-forget via worker pool (default 16 workers, bounded queue 512) +- Non-blocking (pipeline continues immediately) +- Timeout per hook; chain budget enforced but no blocking + +### Security Model + +- **Edition gating**: `CommandHandler` only on Lite edition; attempts on Standard/other editions rejected (defense-in-depth) +- **SSRF hardening (HTTPHandler)**: Caller supplies net.Dialer pinning resolved IP, blocking loopback/link-local/private ranges; no HTTP redirects (CheckRedirect returns ErrUseLastResponse) +- **Auth header encryption**: `Authorization` + other sensitive fields in cfg.Config["headers"] encrypted at rest via AES-256-GCM; decrypted only at HTTP send-time +- **Audit logging**: All hook invocations logged to `hook_executions` table (encrypted, PII-redacted) with dedup_key for idempotency +- **Loop-depth guard (M5)**: SubagentStart checks recursion depth; max 3 levels prevents infinite delegation chains +- **Circuit breaker**: Auto-disables hook after 3 consecutive failures in recent window (C4 mitigation) + +### Pipeline Integration + +Dispatcher wired into `PipelineDeps.HookDispatcher` (nil-safe noop fallback). All 8 stages support hook firing with zero overhead when dispatcher not configured. Example sync hook flow: + +``` +1. ContextStage: UserPromptSubmit → Hooks.Fire(sync) +2. Sync hooks buffer mutations (updatedInput) +3. All hooks succeed? → Commit mutations, proceed +4. Any hook rejects? → Discard buffer, abort pipeline, user sees reason +``` --- @@ -474,6 +537,29 @@ Six distinct workspace scenarios: - **L0 Auto-Injection**: Top-K vault entries injected into system prompt as "relevant context from vault" - **Filesystem Sync**: Vault entries exported as `.md` files for manual editing, re-imported with change tracking +### Audio & Voice System (ElevenLabs + Streaming TTS) + +**Provider architecture** (`internal/audio/`): +- **`Manager`**: Central orchestrator dispatching TTS/STT/Music/SFX requests to pluggable providers +- **`TTSProvider` interface**: Core text-to-speech contract (blocking, buffered response) +- **`StreamingTTSProvider` interface**: Optional interface for ElevenLabs `/stream` endpoint (chunked audio via `io.ReadCloser`) +- **Implementations**: ElevenLabs, OpenAI, Edge, MiniMax (phase-gated; STT/Music/SFX partial) + +**Voice discovery & caching**: +- **Voice cache** (`internal/audio/voice_cache.go`): In-memory LRU (cap 1000 tenants, TTL 1h) shared by HTTP `/v1/voices` + WS `voices.list` handlers. Thread-safe with `sync.Mutex` (LRU updates require write lock) +- **Cache miss recovery**: HTTP handler auto-fetches from ElevenLabs; WS handler requires prior cache warm (provider resolution deferred to Phase 3) +- **Agent audio context** (`store.WithAgentAudio` / `AgentAudioFromCtx`): Immutable snapshot bundle (AgentID + OtherConfig JSONB) injected by dispatcher before tool dispatch; consumed by `TtsTool.Execute` for voice/model resolution + +**Agent-level configuration** (`agents.other_config` JSONB): +- `tts_voice_id`: ElevenLabs voice ID (e.g., "pMsXgVXv3BLzUgSXRplE") +- `tts_model_id`: Model choice (eleven_v3, eleven_flash_v2_5, eleven_multilingual_v2, eleven_turbo_v2_5) +- Resolution precedence: CLI args → agent config → tenant override → provider default + +**Web UI voice picker** (`ui/web/src/components/voice-picker.tsx`): +- Combobox with BM25 search, preview playback button (HTML `