mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-14 04:16:06 +00:00
Merge pull request #929 from nextlevelbuilder/dev
Release: Hooks System, Audio/TTS Refactor, MCP Security & UI Improvements
This commit is contained in:
604 files changed
+46838
-2714
No files matched your search
@@ -88,4 +88,5 @@ compose.d/*
|
||||
/mise.*.toml
|
||||
/mise
|
||||
/.mise
|
||||
*.test
|
||||
goclaw-patched-linux-amd64
|
||||
@@ -2,6 +2,35 @@
|
||||
|
||||
All notable changes to GoClaw are documented here. For full documentation, see [docs.goclaw.sh](https://docs.goclaw.sh).
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
- **Context pruning now opt-in.** Previously tool-result trimming ran by default
|
||||
for all providers; now requires explicit `contextPruning.mode: "cache-ttl"` in
|
||||
`config.agents.defaults` to enable. Matches upstream TS design and prevents
|
||||
silent prompt-cache invalidation on Anthropic.
|
||||
|
||||
Migration — add to `config.json5`:
|
||||
```json5
|
||||
agents: {
|
||||
defaults: {
|
||||
contextPruning: { mode: "cache-ttl" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Improvements
|
||||
|
||||
- **Context pruning cleanup.** Removed redundant Pass 0 (per-result 30% guard),
|
||||
deduplicated double prune call per iteration, added SanitizeHistory to
|
||||
PruneStage for broken tool_use/tool_result pair cleanup.
|
||||
- **Context pruning config backfill (migration).** Agents with existing custom
|
||||
`context_pruning` config (e.g., `softTrimRatio`, `keepLastAssistants`) but
|
||||
missing a `mode` field get auto-backfilled with `mode: "cache-ttl"` to
|
||||
preserve their intent after the opt-in flip. Rows with NULL config stay
|
||||
NULL (new opt-in default applies). PG migration 51; SQLite schema v19.
|
||||
|
||||
## Project Status
|
||||
|
||||
### Implemented & Tested in Production
|
||||
|
||||
@@ -184,6 +184,44 @@ Published to GHCR (`ghcr.io/nextlevelbuilder/goclaw`) and Docker Hub (`digitop/g
|
||||
- **Tool gating:** `TeamActionPolicy` in `internal/tools/team_action_policy.go` — lite blocks comment/review/approve/reject/attach/ask_user. `skill_manage`/`publish_skill` not registered in lite
|
||||
- **File serving:** 2-layer path isolation in `internal/http/files.go` — workspace boundary (all editions) + tenant scope (standard only with RBAC)
|
||||
|
||||
## Plan Verification Rules
|
||||
|
||||
Apply before finalizing any multi-phase plan. Trust-but-verify between scout → planner → final plan.
|
||||
|
||||
### Verification discipline (what to verify)
|
||||
|
||||
1. **Verify factual claims against code** — re-grep/re-count every number, path, endpoint. Don't copy from scout summaries.
|
||||
2. **Trace semantics, not just cite lines** — when plan references existing/upstream code, identify WHEN each field mutates and under WHAT conditions. Line-range citation without control-flow trace = how ports silently invert behavior. Check: every call, or specific branches only?
|
||||
3. **No fabricated identifiers / API families** — every symbol in plan must cite `file:line`. RED FLAGS: plausible-sounding wrappers (`Keyring`, `Validator`, `Manager`), centralized packages (`internal/security`, `internal/auth`) that may be scattered, OTel-style (`StartSpan/EndSpan`) when codebase is emit-based. When unsure, `go doc <pkg>` lists actual exported surface. Apply especially when plan says "reuse existing X".
|
||||
4. **Struct scope audit before adding state** — verify lifetime (per-request/session/agent/process) before adding a field to an existing struct. "Plausibly per-X" is a red flag — grep construction + ownership. Shared-instance state leaks across isolation boundaries.
|
||||
5. **Gate-premise test math** — before asserting "feature X triggers independently of Y", list all early-returns from function entry to X. Math-verify any fixture claiming "X without Y".
|
||||
6. **Port = config-shape match** — "faithful port" divergences in config field name/type are silent breaking changes for users copying upstream config. Match upstream shape, or explicitly flag each divergence with rationale in the phase file.
|
||||
7. **Verify external API endpoints via `docs-seeker`** — before writing endpoint into plan. Sibling APIs often use different roots.
|
||||
|
||||
### Scope & coverage (where to look)
|
||||
|
||||
8. **Grep delete scope deep** — `grep -rn '<symbol>' .` whole repo. Stubs often have refs in catalogs/routing/switch cases. Enumerate ALL sites in todo.
|
||||
9. **Signature-change callers enumeration** — grep + list all callers explicitly. "Update all callers" insufficient.
|
||||
10. **Alias/shim coverage** — enumerate ALL exported symbols via `go doc <pkg>`. Add compile-time signature guards.
|
||||
11. **Scout desktop and web separately** — `ui/desktop/frontend/` ≠ `ui/web/`. Different structure, i18n namespaces, test framework presence.
|
||||
|
||||
### Phasing & ordering (when)
|
||||
|
||||
12. **Re-scout on scope change** — if phase promotes from deferred → active, re-scout. Don't reuse brainstorm summary.
|
||||
13. **Cross-phase gates explicit** — "Phase N-1 merged + tests green" in phase Context. Execution order alone ≠ enforcement.
|
||||
14. **Zero-coverage characterization test = blocker step** — write byte/request-body fixture test BEFORE migration. Not "recommended".
|
||||
15. **i18n keys ordering** — add key + 3 catalogs as explicit todo step BEFORE handler code. Missing key = runtime crash.
|
||||
|
||||
### Conventions & finalization
|
||||
|
||||
16. **Context key style convention** — check existing `context.go` pattern before introducing new key types. Mixed = code smell.
|
||||
17. **Verify pass MANDATORY after rewrite** — spawn fresh Explore/grep to audit planner output. Don't trust self-validation.
|
||||
|
||||
**Pattern to avoid:** user asks → planner writes → report "done".
|
||||
**Safer pattern:** user asks → scout → planner writes → audit-verify → report.
|
||||
|
||||
**Red-team practice:** After planner completes, run `code-reviewer`/`brainstormer` in audit mode: "spot-check 15+ claims vs live codebase". Past catches: fabricated `crypto.Keyring`/`tracing.StartSpan` (agent-hooks plan); inverted TS-port semantics + wrong struct scope + misread early-return gate (context-pruning plan). See `plans/*/reports/audit-*.md` for concrete examples.
|
||||
|
||||
## Post-Implementation Checklist
|
||||
|
||||
After implementing or modifying Go code, run these checks:
|
||||
@@ -207,6 +245,7 @@ Go conventions to follow:
|
||||
- **DB query reuse:** Before adding a new DB query for key entities (teams, agents, sessions, users), check if the same data is already fetched earlier in the current flow/pipeline. Prefer passing resolved data through context, event payloads, or function params rather than re-querying. Duplicate queries waste DB resources and add latency
|
||||
- **Solution design:** When designing a fix or feature, identify the root cause first — don't just patch symptoms. Think through production scenarios (high concurrency, multi-tenant isolation, failure cascades, long-running sessions) to ensure the solution holds up. Prefer explicit configuration over runtime heuristics. Prefer the simplest solution that addresses the root cause directly
|
||||
- **Tenant-scope guards on admin writes:** `RoleAdmin` is not a tenant check. Writes to **global** tables (no `tenant_id` column — e.g. `builtin_tools`, disk config, package mgmt) must gate with `http.requireMasterScope` / WS `requireMasterScope(requireOwner(...))`. Writes to **tenant-scoped** tables must gate with `http.requireTenantAdmin` + SQL `WHERE tenant_id = $N`. Shared predicate: `store.IsMasterScope(ctx)`. See `CONTRIBUTING.md` → "Tenant-scope guards" for the full decision table and anti-patterns.
|
||||
- **Skip load / stress / benchmark tests.** Do NOT write throughput benchmarks, p95/p99 latency assertions, or `runtime.ReadMemStats`-based memory-leak tests for regular feature work. They flake on shared CI runners, waste runner time, and rarely catch real bugs. Only add load tests when explicitly requested for a specific investigation. For normal "prove it works" coverage, use unit + integration + chaos tests.
|
||||
|
||||
## Mobile UI/UX Rules
|
||||
|
||||
|
||||
+4
-2
@@ -144,7 +144,8 @@ RUN chmod +x /app/docker-entrypoint.sh && \
|
||||
# while pip/npm subdirs are goclaw-owned (runtime installs by the app process).
|
||||
# Symlink .claude → data volume so Claude CLI credentials persist across container recreates.
|
||||
RUN mkdir -p /app/workspace /app/data/.runtime/pip /app/data/.runtime/npm-global/lib \
|
||||
/app/data/.runtime/pip-cache /app/data/.claude /app/skills /app/tsnet-state /app/.goclaw \
|
||||
/app/data/.runtime/pip-cache /app/data/.runtime/bin /app/data/.claude /app/skills \
|
||||
/app/tsnet-state /app/.goclaw \
|
||||
&& ln -s /app/data/.claude /app/.claude \
|
||||
&& touch /app/data/.runtime/apk-packages \
|
||||
&& chown -R goclaw:goclaw /app/workspace /app/skills /app/tsnet-state /app/.goclaw \
|
||||
@@ -152,7 +153,8 @@ RUN mkdir -p /app/workspace /app/data/.runtime/pip /app/data/.runtime/npm-global
|
||||
&& chown root:goclaw /app/data/.runtime /app/data/.runtime/apk-packages \
|
||||
&& chmod 0750 /app/data/.runtime \
|
||||
&& chmod 0640 /app/data/.runtime/apk-packages \
|
||||
&& chown -R goclaw:goclaw /app/data/.runtime/pip /app/data/.runtime/npm-global /app/data/.runtime/pip-cache /app/data/.claude
|
||||
&& chown -R goclaw:goclaw /app/data/.runtime/pip /app/data/.runtime/npm-global /app/data/.runtime/pip-cache /app/data/.runtime/bin /app/data/.claude \
|
||||
&& chmod 0755 /app/data/.runtime/bin
|
||||
|
||||
# Default environment
|
||||
ENV GOCLAW_CONFIG=/app/config.json \
|
||||
|
||||
@@ -2,7 +2,7 @@ VERSION ?= $(shell git describe --tags --abbrev=0 --match "v[0-9]*" 2>/dev/null
|
||||
LDFLAGS = -s -w -X github.com/nextlevelbuilder/goclaw/cmd.Version=$(VERSION)
|
||||
BINARY = goclaw
|
||||
|
||||
.PHONY: build build-full build-tui run clean version up down logs reset test vet check-web dev migrate setup ci desktop-dev desktop-build desktop-dmg
|
||||
.PHONY: build build-full build-tui run clean version up down logs reset test vet check-web dev migrate setup ci desktop-dev desktop-build desktop-dmg test-hooks test-hooks-unit test-hooks-e2e test-hooks-chaos test-hooks-rbac test-hooks-tracing
|
||||
|
||||
# Build backend only (API-only, no embedded web UI)
|
||||
build:
|
||||
@@ -94,6 +94,26 @@ test-scenarios:
|
||||
# Critical tests (P0 + P1) - run before merge
|
||||
test-critical: test-invariants test-contracts
|
||||
|
||||
# ── Agent Hooks targets (phase 4) ──
|
||||
# Requires TEST_DATABASE_URL pointing at a pgvector:pg18 container on :5433
|
||||
test-hooks-unit:
|
||||
go test -race ./internal/hooks/... ./internal/gateway/methods/
|
||||
|
||||
test-hooks-e2e:
|
||||
go test -race -timeout=180s -tags integration -run "TestHooksE2E" ./tests/integration/
|
||||
|
||||
test-hooks-chaos:
|
||||
go test -race -timeout=180s -tags integration -run "TestHooksChaos" ./tests/integration/
|
||||
|
||||
test-hooks-rbac:
|
||||
go test -race -timeout=90s -tags integration -run "TestHooksRBAC" ./tests/integration/
|
||||
|
||||
test-hooks-tracing:
|
||||
go test -race -timeout=90s -tags integration -run "TestHooksTracing" ./tests/integration/
|
||||
|
||||
# Full hook test suite (unit + integration)
|
||||
test-hooks: test-hooks-unit test-hooks-e2e test-hooks-chaos test-hooks-rbac test-hooks-tracing
|
||||
|
||||
vet:
|
||||
go vet ./...
|
||||
|
||||
|
||||
+23
-6
@@ -33,6 +33,7 @@ import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/edition"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/gateway"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/gateway/methods"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/hooks"
|
||||
httpapi "github.com/nextlevelbuilder/goclaw/internal/http"
|
||||
mcpbridge "github.com/nextlevelbuilder/goclaw/internal/mcp"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/media"
|
||||
@@ -133,7 +134,7 @@ func runGateway() {
|
||||
tools.DetectServerIPs(context.Background())
|
||||
}
|
||||
|
||||
toolsReg, execApprovalMgr, mcpMgr, sandboxMgr, browserMgr, webFetchTool, ttsTool, permPE, toolPE, dataDir, agentCfg := setupToolRegistry(cfg, workspace, providerRegistry)
|
||||
toolsReg, execApprovalMgr, mcpMgr, sandboxMgr, browserMgr, webFetchTool, ttsTool, audioMgr, permPE, toolPE, dataDir, agentCfg := setupToolRegistry(cfg, workspace, providerRegistry)
|
||||
if browserMgr != nil {
|
||||
defer browserMgr.Close()
|
||||
}
|
||||
@@ -263,6 +264,9 @@ func runGateway() {
|
||||
|
||||
// Create all agents — resolved lazily from database by the managed resolver.
|
||||
agentRouter := agent.NewRouter()
|
||||
if traceCollector != nil {
|
||||
agentRouter.SetTraceCollector(traceCollector)
|
||||
}
|
||||
slog.Info("agents will be resolved lazily from database")
|
||||
|
||||
// Create gateway server and wire enforcement
|
||||
@@ -307,6 +311,7 @@ func runGateway() {
|
||||
workspace: workspace,
|
||||
dataDir: dataDir,
|
||||
domainBus: domainBus,
|
||||
audioMgr: audioMgr,
|
||||
}
|
||||
|
||||
gatewayAddr := loopbackAddr(cfg.Gateway.Host, cfg.Gateway.Port)
|
||||
@@ -375,6 +380,18 @@ func runGateway() {
|
||||
server.SetLogTee(logTee)
|
||||
pairingMethods, heartbeatMethods, chatMethods := registerAllMethods(server, agentRouter, pgStores.Sessions, pgStores.Cron, pgStores.Pairing, cfg, cfgPath, workspace, dataDir, msgBus, execApprovalMgr, pgStores.Agents, pgStores.Skills, pgStores.ConfigSecrets, pgStores.Teams, contextFileInterceptor, logTee, pgStores.Heartbeats, pgStores.ConfigPermissions, pgStores.SystemConfigs, pgStores.Tenants, pgStores.SkillTenantCfgs)
|
||||
|
||||
// Phase 3: Agent hooks RPC methods (hooks.list/create/update/delete/toggle/test/history).
|
||||
if hs, ok := pgStores.Hooks.(hooks.HookStore); ok && hs != nil {
|
||||
hm := methods.NewHookMethods(hs, edition.Current())
|
||||
// Reuse dispatcher handlers for dry-run test runner so UI test panel
|
||||
// exercises the exact code that will run in production.
|
||||
if sharedHookHandlers != nil {
|
||||
hm.SetTestRunner(methods.NewDispatcherTestRunner(sharedHookHandlers))
|
||||
}
|
||||
hm.Register(server.Router())
|
||||
slog.Info("registered hooks RPC methods")
|
||||
}
|
||||
|
||||
// Wire post-turn processor for team task dispatch (WS chat.send + HTTP API paths).
|
||||
if postTurn != nil {
|
||||
chatMethods.SetPostTurnProcessor(postTurn)
|
||||
@@ -422,12 +439,12 @@ func runGateway() {
|
||||
instanceLoader = channels.NewInstanceLoader(pgStores.ChannelInstances, pgStores.Agents, channelMgr, msgBus, pgStores.Pairing)
|
||||
instanceLoader.SetProviderRegistry(providerRegistry)
|
||||
instanceLoader.SetPendingCompactionConfig(cfg.Channels.PendingCompaction)
|
||||
instanceLoader.RegisterFactory(channels.TypeTelegram, telegram.FactoryWithStores(pgStores.Agents, pgStores.ConfigPermissions, pgStores.Teams, pgStores.SubagentTasks, pgStores.PendingMessages))
|
||||
instanceLoader.RegisterFactory(channels.TypeDiscord, discord.FactoryWithStores(pgStores.Agents, pgStores.ConfigPermissions, pgStores.PendingMessages))
|
||||
instanceLoader.RegisterFactory(channels.TypeFeishu, feishu.FactoryWithPendingStore(pgStores.PendingMessages))
|
||||
instanceLoader.RegisterFactory(channels.TypeTelegram, telegram.FactoryWithStoresAndAudio(pgStores.Agents, pgStores.ConfigPermissions, pgStores.Teams, pgStores.SubagentTasks, pgStores.PendingMessages, audioMgr))
|
||||
instanceLoader.RegisterFactory(channels.TypeDiscord, discord.FactoryWithStoresAndAudio(pgStores.Agents, pgStores.ConfigPermissions, pgStores.PendingMessages, audioMgr))
|
||||
instanceLoader.RegisterFactory(channels.TypeFeishu, feishu.FactoryWithPendingStoreAndAudio(pgStores.PendingMessages, audioMgr))
|
||||
instanceLoader.RegisterFactory(channels.TypeZaloOA, zalo.Factory)
|
||||
instanceLoader.RegisterFactory(channels.TypeZaloPersonal, zalopersonal.FactoryWithPendingStore(pgStores.PendingMessages))
|
||||
instanceLoader.RegisterFactory(channels.TypeWhatsApp, whatsapp.FactoryWithDB(pgStores.DB, pgStores.PendingMessages, "pgx"))
|
||||
instanceLoader.RegisterFactory(channels.TypeWhatsApp, whatsapp.FactoryWithDBAudio(pgStores.DB, pgStores.PendingMessages, "pgx", audioMgr, pgStores.BuiltinTools))
|
||||
instanceLoader.RegisterFactory(channels.TypeSlack, slackchannel.FactoryWithPendingStore(pgStores.PendingMessages))
|
||||
instanceLoader.RegisterFactory(channels.TypeFacebook, facebook.Factory)
|
||||
instanceLoader.RegisterFactory(channels.TypePancake, pancake.Factory)
|
||||
@@ -437,7 +454,7 @@ func runGateway() {
|
||||
}
|
||||
|
||||
// Register config-based channels as fallback when no DB instances loaded.
|
||||
registerConfigChannels(cfg, channelMgr, msgBus, pgStores, instanceLoader)
|
||||
registerConfigChannels(cfg, channelMgr, msgBus, pgStores, instanceLoader, audioMgr)
|
||||
|
||||
// Register channels/instances/links/teams RPC methods
|
||||
wireChannelRPCMethods(server, pgStores, channelMgr, agentRouter, msgBus, workspace)
|
||||
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"context"
|
||||
"log/slog"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio/elevenlabs"
|
||||
minimaxaudio "github.com/nextlevelbuilder/goclaw/internal/audio/minimax"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/memory"
|
||||
@@ -187,6 +189,9 @@ func setupSubagents(providerReg *providers.Registry, cfg *config.Config, msgBus
|
||||
if sc.ArchiveAfterMinutes > 0 {
|
||||
subCfg.ArchiveAfterMinutes = sc.ArchiveAfterMinutes
|
||||
}
|
||||
if sc.MaxRetries > 0 {
|
||||
subCfg.MaxRetries = sc.MaxRetries
|
||||
}
|
||||
if sc.Model != "" {
|
||||
subCfg.Model = sc.Model
|
||||
}
|
||||
@@ -274,3 +279,55 @@ func setupTTS(cfg *config.Config) *tts.Manager {
|
||||
|
||||
return mgr
|
||||
}
|
||||
|
||||
// setupAudioExtras wires Music and SFX providers into the audio Manager.
|
||||
// ElevenLabs is registered for both SFX and Music when an API key is present.
|
||||
// MiniMax music is registered when cfg.Audio.Music is configured with a key.
|
||||
// Phase 4 will add STT providers here.
|
||||
func setupAudioExtras(cfg *config.Config, mgr *tts.Manager) {
|
||||
ellKey := cfg.Tts.ElevenLabs.APIKey
|
||||
ellBase := cfg.Tts.ElevenLabs.BaseURL
|
||||
|
||||
// ElevenLabs SFX — reuse TTS credentials.
|
||||
if ellKey != "" {
|
||||
mgr.RegisterSFX(elevenlabs.NewSFXProvider(elevenlabs.Config{
|
||||
APIKey: ellKey,
|
||||
BaseURL: ellBase,
|
||||
}))
|
||||
slog.Info("audio.sfx: elevenlabs registered")
|
||||
}
|
||||
|
||||
// ElevenLabs Music — same credentials, uses /v1/music endpoint.
|
||||
if ellKey != "" {
|
||||
mgr.RegisterMusic(elevenlabs.NewMusicProvider(elevenlabs.Config{
|
||||
APIKey: ellKey,
|
||||
BaseURL: ellBase,
|
||||
}))
|
||||
slog.Info("audio.music: elevenlabs registered")
|
||||
}
|
||||
|
||||
// MiniMax Music — optional, from cfg.Audio.Music block.
|
||||
if cfg.Audio != nil && cfg.Audio.Music != nil {
|
||||
mc := cfg.Audio.Music
|
||||
if mc.APIKey != "" {
|
||||
mgr.RegisterMusic(minimaxaudio.NewMusicProvider(minimaxaudio.MusicConfig{
|
||||
APIKey: mc.APIKey,
|
||||
APIBase: mc.BaseURL,
|
||||
Model: mc.Model,
|
||||
}))
|
||||
slog.Info("audio.music: minimax registered")
|
||||
}
|
||||
}
|
||||
|
||||
// ElevenLabs STT (Scribe v2) — reuse TTS credentials. Registered as tenant-scope
|
||||
// default; per-request tenant override lands via builtin_tools[stt] in Phase 5
|
||||
// channel migration. Legacy per-channel STTProxyURL is bridged separately.
|
||||
if ellKey != "" {
|
||||
mgr.RegisterSTT(elevenlabs.NewSTTProvider(elevenlabs.Config{
|
||||
APIKey: ellKey,
|
||||
BaseURL: ellBase,
|
||||
}))
|
||||
mgr.SetSTTChain([]string{"elevenlabs", "proxy"})
|
||||
slog.Info("audio.stt: elevenlabs registered")
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
@@ -103,6 +104,7 @@ func processAnnounceLoop(
|
||||
req.ForwardMedia = append(req.ForwardMedia, bus.MediaFile{
|
||||
Path: mr.Path,
|
||||
MimeType: mr.ContentType,
|
||||
Filename: filepath.Base(mr.Path), // preserve sanitized stem from producer
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -71,6 +71,10 @@ func builtinToolSeedData() []store.BuiltinToolDef {
|
||||
Requires: []string{"tts_provider"},
|
||||
Metadata: json.RawMessage(`{"config_hint":"Config → TTS"}`),
|
||||
},
|
||||
{Name: "stt", DisplayName: "Speech-to-Text", Description: "Transcribe voice/audio messages to text using ElevenLabs Scribe or a proxy service", Category: "media", Enabled: true,
|
||||
Requires: []string{"stt_provider"},
|
||||
Metadata: json.RawMessage(`{"config_hint":"Config → Audio → STT"}`),
|
||||
},
|
||||
|
||||
// browser
|
||||
{Name: "browser", DisplayName: "Browser", Description: "Automate browser interactions: navigate pages, click elements, fill forms, take screenshots", Category: "browser", Enabled: true,
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"github.com/google/uuid"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/agent"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels/discord"
|
||||
@@ -27,7 +28,8 @@ import (
|
||||
)
|
||||
|
||||
// registerConfigChannels registers config-based channels as fallback when no DB instances are loaded.
|
||||
func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, msgBus *bus.MessageBus, pgStores *store.Stores, instanceLoader *channels.InstanceLoader) {
|
||||
// audioMgr is optional (nil = STT disabled for channels).
|
||||
func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, msgBus *bus.MessageBus, pgStores *store.Stores, instanceLoader *channels.InstanceLoader, audioMgr *audio.Manager) {
|
||||
if instanceLoader != nil {
|
||||
return
|
||||
}
|
||||
@@ -46,7 +48,7 @@ func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, ms
|
||||
if cfg.Channels.Telegram.Enabled {
|
||||
if cfg.Channels.Telegram.Token == "" {
|
||||
recordMissingConfig(channels.TypeTelegram, "Set channels.telegram.token in config.")
|
||||
} else if tg, err := telegram.New(cfg.Channels.Telegram, msgBus, pgStores.Pairing); err != nil {
|
||||
} else if tg, err := telegram.New(cfg.Channels.Telegram, msgBus, pgStores.Pairing, audioMgr); err != nil {
|
||||
channelMgr.RecordFailure(channels.TypeTelegram, "", err)
|
||||
slog.Error("failed to initialize telegram channel", "error", err)
|
||||
} else {
|
||||
@@ -58,7 +60,7 @@ func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, ms
|
||||
if cfg.Channels.Discord.Enabled {
|
||||
if cfg.Channels.Discord.Token == "" {
|
||||
recordMissingConfig(channels.TypeDiscord, "Set channels.discord.token in config.")
|
||||
} else if dc, err := discord.New(cfg.Channels.Discord, msgBus, nil, nil, nil, nil); err != nil {
|
||||
} else if dc, err := discord.New(cfg.Channels.Discord, msgBus, nil, nil, nil, nil, audioMgr); err != nil {
|
||||
channelMgr.RecordFailure(channels.TypeDiscord, "", err)
|
||||
slog.Error("failed to initialize discord channel", "error", err)
|
||||
} else {
|
||||
@@ -72,7 +74,7 @@ func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, ms
|
||||
if strings.Contains(fmt.Sprintf("%T", pgStores.DB.Driver()), "sqlite") {
|
||||
waDialect = "sqlite3"
|
||||
}
|
||||
wa, err := whatsapp.New(cfg.Channels.WhatsApp, msgBus, pgStores.Pairing, pgStores.DB, pgStores.PendingMessages, waDialect)
|
||||
wa, err := whatsapp.New(cfg.Channels.WhatsApp, msgBus, pgStores.Pairing, pgStores.DB, pgStores.PendingMessages, waDialect, audioMgr, pgStores.BuiltinTools)
|
||||
if err != nil {
|
||||
channelMgr.RecordFailure(channels.TypeWhatsApp, "", err)
|
||||
slog.Error("failed to initialize whatsapp channel", "error", err)
|
||||
@@ -131,7 +133,7 @@ func registerConfigChannels(cfg *config.Config, channelMgr *channels.Manager, ms
|
||||
feishu.WithAgentStore(pgStores.Agents),
|
||||
feishu.WithConfigPermStore(pgStores.ConfigPermissions),
|
||||
}
|
||||
if f, err := feishu.New(cfg.Channels.Feishu, msgBus, pgStores.Pairing, nil, feishuOpts...); err != nil {
|
||||
if f, err := feishu.New(cfg.Channels.Feishu, msgBus, pgStores.Pairing, nil, audioMgr, feishuOpts...); err != nil {
|
||||
channelMgr.RecordFailure(channels.TypeFeishu, "", err)
|
||||
slog.Error("failed to initialize feishu channel", "error", err)
|
||||
} else {
|
||||
|
||||
@@ -109,6 +109,13 @@ func handleSubagentAnnounce(
|
||||
Iterations: iterations,
|
||||
}
|
||||
|
||||
// Preserve real acting sender + RBAC role from original turn so permission
|
||||
// checks (e.g. write_file in group chat) attribute to the user and can
|
||||
// bypass per-user grants for authenticated admins, not the synthetic
|
||||
// "subagent:<id>" sender of the announce message itself (#915).
|
||||
originSenderID := msg.Metadata[tools.MetaOriginSenderID]
|
||||
originRole := msg.Metadata[tools.MetaOriginRole]
|
||||
|
||||
queueKey := fmt.Sprintf("%s:%s", msg.TenantID, sessionKey)
|
||||
routing := subagentAnnounceRouting{
|
||||
QueueKey: queueKey,
|
||||
@@ -120,6 +127,8 @@ func handleSubagentAnnounce(
|
||||
OrigPeerKind: origPeerKind,
|
||||
OrigLocalKey: origLocalKey,
|
||||
UserID: announceUserID,
|
||||
SenderID: originSenderID,
|
||||
Role: originRole,
|
||||
ParentAgent: parentAgent,
|
||||
ParentTraceID: parentTraceID,
|
||||
ParentRootSpanID: parentRootSpanID,
|
||||
@@ -129,16 +138,14 @@ func handleSubagentAnnounce(
|
||||
// Enqueue into producer-consumer queue using tenant-scoped key from routing.
|
||||
isProcessor := enqueueSubagentAnnounce(queueKey, entry)
|
||||
if isProcessor {
|
||||
deps.BgWg.Add(1)
|
||||
go func() {
|
||||
defer deps.BgWg.Done()
|
||||
deps.BgWg.Go(func() {
|
||||
defer safego.Recover(nil, "component", "subagent_announce_loop", "session", sessionKey)
|
||||
|
||||
// Fetch live roster for merged announce context.
|
||||
roster := deps.SubagentMgr.RosterForParent(parentAgent)
|
||||
|
||||
processSubagentAnnounceLoop(ctx, routing, roster, deps.SubagentMgr, deps.Sched, deps.MsgBus, deps.Cfg)
|
||||
}()
|
||||
})
|
||||
}
|
||||
|
||||
return true
|
||||
@@ -207,6 +214,13 @@ func handleTeammateMessage(
|
||||
announceUserID = fmt.Sprintf("group:%s:%s", origChannel, origChatID)
|
||||
}
|
||||
|
||||
// Preserve real acting sender + RBAC role through teammate dispatch so
|
||||
// permission checks during the teammate's turn (e.g. write_file in group
|
||||
// chat) attribute to the original user and can bypass per-user grants
|
||||
// for authenticated admins (#915).
|
||||
teammateSenderID := msg.Metadata[tools.MetaOriginSenderID]
|
||||
teammateRole := msg.Metadata[tools.MetaOriginRole]
|
||||
|
||||
outMeta := buildAnnounceOutMeta(origLocalKey)
|
||||
|
||||
// Link member agent trace back to lead's trace for unified tracing.
|
||||
@@ -235,6 +249,8 @@ func handleTeammateMessage(
|
||||
PeerKind: origPeerKind,
|
||||
LocalKey: origLocalKey,
|
||||
UserID: announceUserID,
|
||||
SenderID: teammateSenderID, // real user who triggered the teammate dispatch (#915)
|
||||
Role: teammateRole, // RBAC role for admin bypass during teammate turn (#915)
|
||||
RunID: fmt.Sprintf("teammate-%s-%s", msg.Metadata[tools.MetaFromAgent], msg.Metadata[tools.MetaToAgent]),
|
||||
Stream: false,
|
||||
TeamTaskID: msg.Metadata[tools.MetaTeamTaskID],
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/agent"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels/telegram/voiceguard"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/i18n"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/scheduler"
|
||||
@@ -214,22 +215,12 @@ func processNormalMessage(
|
||||
|
||||
// Build outbound metadata for reply-to + thread routing BEFORE RegisterRun
|
||||
// so block.reply handler can use it for routing intermediate messages.
|
||||
outMeta := make(map[string]string)
|
||||
outMeta := channels.CopyFinalRoutingMeta(msg.Metadata)
|
||||
if isGroup {
|
||||
if mid := msg.Metadata["message_id"]; mid != "" {
|
||||
outMeta["reply_to_message_id"] = mid
|
||||
}
|
||||
}
|
||||
// Channel routing keys — keep in sync with routingMetaKeys in channels/events.go.
|
||||
for _, k := range []string{
|
||||
tools.MetaMessageThreadID, "local_key", "placeholder_key", "group_id",
|
||||
"feishu_reply_target_id",
|
||||
"fb_mode", "sender_id", "page_id", "reply_to_comment_id",
|
||||
} {
|
||||
if v := msg.Metadata[k]; v != "" {
|
||||
outMeta[k] = v
|
||||
}
|
||||
}
|
||||
|
||||
// Register run with channel manager for streaming/reaction event forwarding.
|
||||
// Use localKey (composite key with topic suffix) so streaming/reaction events
|
||||
@@ -355,6 +346,29 @@ func processNormalMessage(
|
||||
schedCtx = tools.WithRunKind(schedCtx, rk)
|
||||
}
|
||||
|
||||
// Resolve effective sender: prefer MetaOriginSenderID when the on-wire
|
||||
// SenderID is an internal/synthetic one (e.g. "notification:progress",
|
||||
// "ticker:system", "system:escalation", "session_send_tool"). This lets
|
||||
// system-initiated turns that DO have a real user behind them (because
|
||||
// they propagated the origin via metadata) attribute actions to that user
|
||||
// — e.g. for CheckFileWriterPermission in group chats (#915). Synthetic
|
||||
// senders without propagation keep their on-wire value and hit F1's
|
||||
// deny-in-group rule (safe default).
|
||||
effectiveSenderID := msg.SenderID
|
||||
if bus.IsInternalSender(effectiveSenderID) {
|
||||
// Defense-in-depth: if a propagation bug ever writes a synthetic
|
||||
// value into MetaOriginSenderID, do NOT honour it. We want only real
|
||||
// user senders to override the on-wire synthetic.
|
||||
if realSender := msg.Metadata[tools.MetaOriginSenderID]; realSender != "" && !bus.IsInternalSender(realSender) {
|
||||
effectiveSenderID = realSender
|
||||
}
|
||||
}
|
||||
// Role propagation: carry the RBAC role of the originating actor so
|
||||
// permission checks during the re-ingress turn can bypass per-user
|
||||
// grants for authenticated admins (#915). Only present when the
|
||||
// upstream dispatch set MetaOriginRole.
|
||||
effectiveRole := msg.Metadata[tools.MetaOriginRole]
|
||||
|
||||
// Schedule through main lane (per-session concurrency controlled by maxConcurrent)
|
||||
outCh := deps.Sched.ScheduleWithOpts(schedCtx, "main", agent.RunRequest{
|
||||
SessionKey: sessionKey,
|
||||
@@ -368,7 +382,8 @@ func processNormalMessage(
|
||||
PeerKind: peerKind,
|
||||
LocalKey: msg.Metadata["local_key"],
|
||||
UserID: userID,
|
||||
SenderID: msg.SenderID,
|
||||
SenderID: effectiveSenderID,
|
||||
Role: effectiveRole,
|
||||
SenderName: resolveSenderName(msg),
|
||||
RunID: runID,
|
||||
Stream: enableStream,
|
||||
@@ -415,10 +430,19 @@ func processNormalMessage(
|
||||
return
|
||||
}
|
||||
slog.Error("inbound: agent run failed", "error", outcome.Err, "channel", channel)
|
||||
// Suppress technical error text on public-facing channels (FB, Telegram, etc.)
|
||||
// Empty Content still triggers placeholder/typing cleanup downstream.
|
||||
errContent := formatAgentError(outcome.Err)
|
||||
if deps.ChannelMgr != nil {
|
||||
if ct := deps.ChannelMgr.ChannelTypeForName(channel); isExternalChannel(ct) {
|
||||
slog.Info("inbound: suppressed error for external channel", "channel", channel, "type", ct)
|
||||
errContent = ""
|
||||
}
|
||||
}
|
||||
deps.MsgBus.PublishOutbound(bus.OutboundMessage{
|
||||
Channel: channel,
|
||||
ChatID: chatID,
|
||||
Content: formatAgentError(outcome.Err),
|
||||
Content: errContent,
|
||||
Metadata: meta,
|
||||
})
|
||||
return
|
||||
|
||||
@@ -40,7 +40,7 @@ func makeSchedulerRunFunc(agents *agent.Router, cfg *config.Config) scheduler.Ru
|
||||
// The ctx from the scheduler is already cancellable; we create a child so the router's
|
||||
// cancel func is independent from the scheduler's cancel func. Calling cancel twice is safe.
|
||||
runCtx, cancel := context.WithCancel(ctx)
|
||||
injectCh := agents.RegisterRun(req.RunID, req.SessionKey, agentID, cancel)
|
||||
injectCh := agents.RegisterRun(runCtx, req.RunID, req.SessionKey, agentID, cancel)
|
||||
defer agents.UnregisterRun(req.RunID)
|
||||
defer cancel()
|
||||
|
||||
|
||||
@@ -2,12 +2,14 @@ package cmd
|
||||
|
||||
import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/agent"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/cache"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/gateway"
|
||||
httpapi "github.com/nextlevelbuilder/goclaw/internal/http"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/skills"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
@@ -33,4 +35,6 @@ type gatewayDeps struct {
|
||||
workspace string
|
||||
dataDir string
|
||||
domainBus eventbus.DomainEventBus
|
||||
audioMgr *audio.Manager // nil if TTS not configured; used by TTSHandler
|
||||
ttsHandler *httpapi.TTSHandler // nil if TTS not configured; for hot-reload
|
||||
}
|
||||
@@ -74,6 +74,27 @@ func isContextOverflowError(lower string) bool {
|
||||
containsAny(lower, "overflow", "too large", "too long", "limit", "exceeded"))
|
||||
}
|
||||
|
||||
// isExternalChannel reports whether a channel type serves end users on a
|
||||
// public-facing platform (Facebook, Telegram, etc.). Internal error details
|
||||
// must not be forwarded to these channels — the caller publishes an empty
|
||||
// outbound instead so placeholders get cleaned up without leaking technical
|
||||
// error text to end users. Internal types ("ws", "") return false.
|
||||
func isExternalChannel(channelType string) bool {
|
||||
switch channelType {
|
||||
case channels.TypeFacebook,
|
||||
channels.TypeTelegram,
|
||||
channels.TypeDiscord,
|
||||
channels.TypeFeishu,
|
||||
channels.TypeWhatsApp,
|
||||
channels.TypeZaloOA,
|
||||
channels.TypeZaloPersonal,
|
||||
channels.TypePancake,
|
||||
channels.TypeSlack:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// isMessageFormatError checks for tool_use/tool_result mismatch, role ordering,
|
||||
// and other message format errors that indicate corrupted session history.
|
||||
func isMessageFormatError(lower string) bool {
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
)
|
||||
|
||||
// TestIsExternalChannel ensures the whitelist matches actual channel type
|
||||
// constants — a mismatch here silently leaks internal error strings to end
|
||||
// users (e.g. "zalo" vs "zalo_oa"/"zalo_personal"). Always compare against
|
||||
// channels.Type* constants, never string literals.
|
||||
func TestIsExternalChannel(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
channelType string
|
||||
want bool
|
||||
}{
|
||||
// Public-facing platforms — errors must be suppressed.
|
||||
{"facebook", channels.TypeFacebook, true},
|
||||
{"telegram", channels.TypeTelegram, true},
|
||||
{"discord", channels.TypeDiscord, true},
|
||||
{"feishu", channels.TypeFeishu, true},
|
||||
{"whatsapp", channels.TypeWhatsApp, true},
|
||||
{"zalo_oa", channels.TypeZaloOA, true},
|
||||
{"zalo_personal", channels.TypeZaloPersonal, true},
|
||||
{"pancake", channels.TypePancake, true},
|
||||
{"slack", channels.TypeSlack, true},
|
||||
|
||||
// Internal / unknown channel types — errors must still surface.
|
||||
{"ws", "ws", false},
|
||||
{"empty", "", false},
|
||||
{"unknown", "myplatform", false},
|
||||
// "line" is a Pancake sub-platform, not a top-level channel type.
|
||||
{"line_not_top_level", "line", false},
|
||||
// Legacy short "zalo" string must NOT match — real constants are zalo_oa / zalo_personal.
|
||||
{"zalo_short_form", "zalo", false},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := isExternalChannel(tc.channelType); got != tc.want {
|
||||
t.Fatalf("isExternalChannel(%q) = %v, want %v", tc.channelType, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/skills"
|
||||
)
|
||||
|
||||
// initGitHubInstaller constructs the process-wide GitHub Releases installer
|
||||
// from environment variables and registers it via skills.SetDefaultGitHubInstaller.
|
||||
//
|
||||
// Config comes ONLY from env vars (token is a secret — never from config.json):
|
||||
//
|
||||
// GOCLAW_PACKAGES_GITHUB_TOKEN optional PAT (boosts rate limit, enables private repos)
|
||||
// GOCLAW_PACKAGES_MAX_ASSET_SIZE_MB default 200
|
||||
// GOCLAW_PACKAGES_GITHUB_ALLOWED_ORGS comma-separated allowlist (empty = all allowed)
|
||||
// GOCLAW_PACKAGES_GITHUB_BIN_DIR default /app/data/.runtime/bin
|
||||
// GOCLAW_PACKAGES_GITHUB_MANIFEST default {BIN_DIR}/../github-packages.json
|
||||
func initGitHubInstaller() {
|
||||
cfg := &skills.GitHubPackagesConfig{
|
||||
Token: os.Getenv("GOCLAW_PACKAGES_GITHUB_TOKEN"),
|
||||
BinDir: os.Getenv("GOCLAW_PACKAGES_GITHUB_BIN_DIR"),
|
||||
ManifestPath: os.Getenv("GOCLAW_PACKAGES_GITHUB_MANIFEST"),
|
||||
}
|
||||
if v := os.Getenv("GOCLAW_PACKAGES_MAX_ASSET_SIZE_MB"); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil && n > 0 {
|
||||
cfg.MaxAssetSizeMB = n
|
||||
}
|
||||
}
|
||||
if v := os.Getenv("GOCLAW_PACKAGES_GITHUB_ALLOWED_ORGS"); v != "" {
|
||||
for _, o := range strings.Split(v, ",") {
|
||||
if o = strings.TrimSpace(o); o != "" {
|
||||
cfg.AllowedOrgs = append(cfg.AllowedOrgs, o)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// NewGitHubInstaller calls cfg.Defaults() — no need to invoke it here.
|
||||
client := skills.NewGitHubClient(cfg.Token)
|
||||
installer := skills.NewGitHubInstaller(client, cfg)
|
||||
skills.SetDefaultGitHubInstaller(installer)
|
||||
|
||||
// Best-effort ensure bin dir + manifest dir exist (entrypoint may run as root
|
||||
// while Go process runs as goclaw — respect pre-existing permissions).
|
||||
if err := os.MkdirAll(cfg.BinDir, 0o755); err != nil {
|
||||
slog.Warn("github.installer: mkdir bin dir failed", "path", cfg.BinDir, "error", err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(cfg.ManifestPath), 0o755); err != nil {
|
||||
slog.Warn("github.installer: mkdir manifest dir failed", "path", cfg.ManifestPath, "error", err)
|
||||
}
|
||||
|
||||
slog.Info("packages: github installer enabled",
|
||||
"bin_dir", cfg.BinDir,
|
||||
"manifest", cfg.ManifestPath,
|
||||
"allowed_orgs", cfg.AllowedOrgs,
|
||||
"max_asset_mb", cfg.MaxAssetSizeMB,
|
||||
"token_set", cfg.Token != "",
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/edition"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/hooks"
|
||||
hookhandlers "github.com/nextlevelbuilder/goclaw/internal/hooks/handlers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/hooks/budget"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/security"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store/pg"
|
||||
)
|
||||
|
||||
// sharedHookHandlers is populated by wireExtras so the gateway.go router
|
||||
// wiring can reuse the same handler instances for the `hooks.test` runner.
|
||||
// nil when hook store is absent (hooks disabled).
|
||||
var sharedHookHandlers map[hooks.HandlerType]hooks.Handler
|
||||
|
||||
// buildHookHandlers constructs the production handler map used by both the
|
||||
// dispatcher (sync + async chain) and the `hooks.test` test runner. Keeping
|
||||
// this factory single-source ensures test-panel behavior mirrors production.
|
||||
//
|
||||
// Budget wiring (C1 fix): the PromptHandler receives a budget.Store bound
|
||||
// to pg.NewPGHookBudget so token spend is atomically deducted per tenant.
|
||||
// When the DB handle is unavailable, budget falls back to nil (Lite desktop).
|
||||
func buildHookHandlers(stores *store.Stores, providerReg *providers.Registry, hooksCfg config.HooksConfig) map[hooks.HandlerType]hooks.Handler {
|
||||
encryptKey := os.Getenv("GOCLAW_ENCRYPTION_KEY")
|
||||
|
||||
var budgetStore *budget.Store
|
||||
if stores != nil && stores.DB != nil {
|
||||
budgetStore = budget.New(pg.NewPGHookBudget(stores.DB), nil)
|
||||
}
|
||||
|
||||
promptHandler := &hookhandlers.PromptHandler{
|
||||
Resolver: hookhandlers.NewRegistryResolver(providerReg, stores.SystemConfigs),
|
||||
Budget: budgetStore,
|
||||
DefaultModel: "haiku",
|
||||
}
|
||||
|
||||
// ScriptHandler: bounded by cfg.Hooks caps; zero values fall back to
|
||||
// handler defaults (10 / 3 / 500). Safe for concurrent reuse across
|
||||
// dispatcher + hooks.test runner (each Execute allocates its own runtime).
|
||||
scriptHandler := hookhandlers.NewScriptHandler(
|
||||
hooksCfg.ScriptConcurrency,
|
||||
hooksCfg.ScriptPerTenantConcurrency,
|
||||
hooksCfg.ScriptCacheSize,
|
||||
)
|
||||
|
||||
return map[hooks.HandlerType]hooks.Handler{
|
||||
hooks.HandlerCommand: &hookhandlers.CommandHandler{Edition: edition.Current()},
|
||||
hooks.HandlerHTTP: &hookhandlers.HTTPHandler{
|
||||
EncryptKey: encryptKey,
|
||||
Client: security.NewSafeClient(10 * time.Second),
|
||||
},
|
||||
hooks.HandlerPrompt: promptHandler,
|
||||
hooks.HandlerScript: scriptHandler,
|
||||
}
|
||||
}
|
||||
@@ -3,8 +3,11 @@ package cmd
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/gateway/methods"
|
||||
httpapi "github.com/nextlevelbuilder/goclaw/internal/http"
|
||||
mcpbridge "github.com/nextlevelbuilder/goclaw/internal/mcp"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/media"
|
||||
@@ -130,7 +133,8 @@ func (d *gatewayDeps) wireHTTPHandlersOnServer(
|
||||
d.server.SetUsageHandler(httpapi.NewUsageHandler(d.pgStores.Snapshots, d.pgStores.DB))
|
||||
}
|
||||
|
||||
// Runtime package management (install/uninstall system/pip/npm packages)
|
||||
// Runtime package management (install/uninstall system/pip/npm/github packages)
|
||||
initGitHubInstaller()
|
||||
d.server.SetPackagesHandler(httpapi.NewPackagesHandler())
|
||||
|
||||
// API documentation (OpenAPI spec + Swagger UI at /docs)
|
||||
@@ -222,6 +226,38 @@ func (d *gatewayDeps) wireHTTPHandlersOnServer(
|
||||
d.server.SetMediaServeHandler(httpapi.NewMediaServeHandler(mediaStore))
|
||||
}
|
||||
|
||||
// ElevenLabs voice list + refresh endpoints (GET /v1/voices, POST /v1/voices/refresh).
|
||||
// VoiceCache is shared between the HTTP handler and the WS voices.list method.
|
||||
// TTL 1h + LRU cap 1000 tenants.
|
||||
{
|
||||
voiceCache := audio.NewVoiceCache(1*time.Hour, 1000)
|
||||
var secretStore store.ConfigSecretsStore
|
||||
if d.pgStores != nil && d.pgStores.ConfigSecrets != nil {
|
||||
secretStore = d.pgStores.ConfigSecrets
|
||||
}
|
||||
var tenantStore store.TenantStore
|
||||
if d.pgStores != nil && d.pgStores.Tenants != nil {
|
||||
tenantStore = d.pgStores.Tenants
|
||||
}
|
||||
voicesH := httpapi.NewVoicesHandler(voiceCache, secretStore, tenantStore)
|
||||
d.server.SetVoicesHandler(voicesH)
|
||||
// Wire WS method — provider nil means each request resolves key via secretStore at HTTP layer.
|
||||
// For WS, use same cache. Provider is resolved via secretStore at WS level in a future phase.
|
||||
methods.NewVoicesMethods(voiceCache, nil).Register(d.server.Router())
|
||||
}
|
||||
|
||||
// TTS synthesize endpoint — shares audio.Manager with setupTTS.
|
||||
if d.audioMgr != nil {
|
||||
ttsH := httpapi.NewTTSHandler(d.audioMgr)
|
||||
// Reuse the server's rate limiter (per-IP/token; NOT per-user).
|
||||
// Server.RateLimiter() is non-nil by construction (server.go:104).
|
||||
if rl := d.server.RateLimiter(); rl != nil && rl.Enabled() {
|
||||
ttsH.SetRateLimiter(rl.Allow)
|
||||
}
|
||||
d.server.SetTTSHandler(ttsH)
|
||||
d.ttsHandler = ttsH // store for hot-reload
|
||||
}
|
||||
|
||||
// Seed + apply builtin tool disables
|
||||
if d.pgStores.BuiltinTools != nil {
|
||||
seedBuiltinTools(context.Background(), d.pgStores.BuiltinTools)
|
||||
|
||||
@@ -103,6 +103,9 @@ func (d *gatewayDeps) runLifecycle(
|
||||
return
|
||||
}
|
||||
deps.ttsTool.UpdateManager(newMgr)
|
||||
if d.ttsHandler != nil {
|
||||
d.ttsHandler.UpdateManager(newMgr)
|
||||
}
|
||||
slog.Info("tts config reloaded", "provider", newMgr.PrimaryProvider(), "auto", string(newMgr.AutoMode()))
|
||||
})
|
||||
|
||||
|
||||
+50
-1
@@ -14,9 +14,11 @@ import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/agent"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/orchestration"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/edition"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/hooks"
|
||||
hookbuiltin "github.com/nextlevelbuilder/goclaw/internal/hooks/builtin"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/orchestration"
|
||||
httpapi "github.com/nextlevelbuilder/goclaw/internal/http"
|
||||
kg "github.com/nextlevelbuilder/goclaw/internal/knowledgegraph"
|
||||
mcpbridge "github.com/nextlevelbuilder/goclaw/internal/mcp"
|
||||
@@ -125,8 +127,10 @@ func wireExtras(
|
||||
|
||||
// 5. Shared MCP connection pool (eliminates duplicate connections across agents)
|
||||
var mcpPool *mcpbridge.Pool
|
||||
var mcpGrantChecker mcpbridge.GrantChecker
|
||||
if stores.MCP != nil {
|
||||
mcpPool = mcpbridge.NewPool(mcpbridge.DefaultPoolConfig())
|
||||
mcpGrantChecker = mcpbridge.NewStoreGrantChecker(stores.MCP, msgBus)
|
||||
}
|
||||
|
||||
// 6. Set up agent resolver: lazy-creates Loops from DB
|
||||
@@ -144,6 +148,48 @@ func wireExtras(
|
||||
// vaultIntc is set later by wireVault but captured by closure in OnTextUploaded.
|
||||
var vaultIntc *tools.VaultInterceptor
|
||||
|
||||
// Agent Hooks (Issue #875) — lifecycle dispatcher + handlers.
|
||||
var hookDispatcher hooks.Dispatcher = hooks.NewNoopDispatcher()
|
||||
if hs, ok := stores.Hooks.(hooks.HookStore); ok && hs != nil {
|
||||
// Phase 04: wire builtin registry. Install a strip-all lookup FIRST so a
|
||||
// Load() failure leaves the dispatcher failing closed (no wide fallback
|
||||
// via the Phase 03 permissive default). On successful Load we swap in the
|
||||
// real per-id allowlist, then UPSERT canonical rows with stable UUIDv5s.
|
||||
// Seed failures log but never block startup.
|
||||
hooks.SetBuiltinAllowlistLookup(func(uuid.UUID) []string { return nil })
|
||||
if err := hookbuiltin.Load(); err != nil {
|
||||
slog.Warn("hooks.builtin_load_failed", "err", err)
|
||||
} else {
|
||||
hooks.SetBuiltinAllowlistLookup(hookbuiltin.AllowlistFor)
|
||||
if err := hookbuiltin.Seed(context.Background(), hs, appCfg.Hooks); err != nil {
|
||||
slog.Warn("hooks.builtin_seed_failed", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 07: runtime migration — auto-disable legacy command-type hooks
|
||||
// on Standard edition. No-op on Lite. Idempotent. Runs synchronously
|
||||
// before listeners so traffic never sees a command hook fire on a
|
||||
// post-Wave-1 Standard instance.
|
||||
if n, err := hooks.DisableLegacyCommandHooks(context.Background(), hs, edition.Current()); err != nil {
|
||||
slog.Warn("hooks.command_migration_failed", "err", err)
|
||||
} else if n > 0 {
|
||||
slog.Info("hooks.command_migration_ran",
|
||||
"disabled_count", n, "edition", edition.Current().Name)
|
||||
}
|
||||
|
||||
handlers := buildHookHandlers(stores, providerReg, appCfg.Hooks)
|
||||
stdOpts := hooks.StdDispatcherOpts{
|
||||
Store: hs,
|
||||
Audit: hooks.NewAuditWriter(hs, ""),
|
||||
Handlers: handlers,
|
||||
}
|
||||
hookDispatcher = hooks.NewStdDispatcher(stdOpts)
|
||||
hooks.SubscribeDelegateEvents(domainBus, hookDispatcher)
|
||||
// Stash handlers for later gateway.go wiring (test runner).
|
||||
sharedHookHandlers = handlers
|
||||
slog.Info("agent hooks dispatcher wired", "handlers", "command,http,prompt")
|
||||
}
|
||||
|
||||
resolver := agent.NewManagedResolver(agent.ResolverDeps{
|
||||
AgentStore: stores.Agents,
|
||||
ProviderStore: stores.Providers,
|
||||
@@ -177,6 +223,7 @@ func wireExtras(
|
||||
BuiltinToolStore: stores.BuiltinTools,
|
||||
MCPStore: stores.MCP,
|
||||
MCPPool: mcpPool,
|
||||
MCPGrantChecker: mcpGrantChecker,
|
||||
ConfigPermStore: stores.ConfigPermissions,
|
||||
MediaStore: mediaStore,
|
||||
ModelPricing: appCfg.Telemetry.ModelPricing,
|
||||
@@ -191,6 +238,7 @@ func wireExtras(
|
||||
AutoInjector: autoInjector,
|
||||
EvolutionMetricsStore: stores.EvolutionMetrics,
|
||||
DomainBus: domainBus,
|
||||
HookDispatcher: hookDispatcher,
|
||||
OnTextUploaded: func(ctx context.Context, path, content string) {
|
||||
if vaultIntc != nil {
|
||||
vaultIntc.AfterWrite(ctx, path, content)
|
||||
@@ -385,6 +433,7 @@ func wireExtras(
|
||||
}
|
||||
delegateTool := tools.NewDelegateTool(stores.AgentLinks, stores.Agents, domainBus, delegateRunFn)
|
||||
delegateTool.SetMsgBus(msgBus)
|
||||
delegateTool.SetHookDispatcher(hookDispatcher)
|
||||
toolsReg.Register(delegateTool)
|
||||
slog.Info("delegate tool wired")
|
||||
}
|
||||
|
||||
@@ -373,16 +373,6 @@ func registerProvidersFromDB(registry *providers.Registry, provStore store.Provi
|
||||
base = "https://ollama.com/v1"
|
||||
}
|
||||
registry.RegisterForTenant(p.TenantID, providers.NewOpenAIProvider(p.Name, p.APIKey, base, "llama3.3"))
|
||||
case store.ProviderSuno:
|
||||
// Suno is a media-only provider (music gen). Register as OpenAI-compat
|
||||
// so credentialProvider interface works for API key/base extraction.
|
||||
base := p.APIBase
|
||||
if base == "" {
|
||||
base = "https://api.sunoapi.org"
|
||||
}
|
||||
prov := providers.NewOpenAIProvider(p.Name, p.APIKey, base, "")
|
||||
prov.WithProviderType(p.ProviderType)
|
||||
registry.RegisterForTenant(p.TenantID, prov)
|
||||
case store.ProviderNovita:
|
||||
base := p.APIBase
|
||||
if base == "" {
|
||||
|
||||
+19
-5
@@ -10,6 +10,7 @@ import (
|
||||
|
||||
"github.com/google/uuid"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bootstrap"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
@@ -44,6 +45,7 @@ func setupToolRegistry(
|
||||
browserMgr *browser.Manager,
|
||||
webFetchTool *tools.WebFetchTool,
|
||||
ttsTool *tools.TtsTool,
|
||||
audioMgr *audio.Manager,
|
||||
permPE *permissions.PolicyEngine,
|
||||
toolPE *tools.PolicyEngine,
|
||||
dataDir string,
|
||||
@@ -133,15 +135,19 @@ func setupToolRegistry(
|
||||
toolsReg.Register(tools.NewReadImageTool(providerRegistry))
|
||||
toolsReg.Register(tools.NewCreateImageTool(providerRegistry))
|
||||
|
||||
// Audio generation tool (MiniMax music + ElevenLabs sound effects)
|
||||
toolsReg.Register(tools.NewCreateAudioTool(providerRegistry,
|
||||
cfg.Tts.ElevenLabs.APIKey, cfg.Tts.ElevenLabs.BaseURL))
|
||||
|
||||
// TTS (text-to-speech) system — always create TtsTool so config reload can populate it later
|
||||
// Audio system: build Manager first so Music/SFX providers are registered
|
||||
// before the create_audio tool is constructed.
|
||||
ttsMgr := setupTTS(cfg)
|
||||
if ttsMgr == nil {
|
||||
ttsMgr = tts.NewManager(tts.ManagerConfig{})
|
||||
}
|
||||
setupAudioExtras(cfg, ttsMgr) // Phase 3: registers Music + SFX providers.
|
||||
audio.BridgeLegacySTT(ttsMgr, cfg) // Phase 4: bridge per-channel STTProxyURL → channel-scoped providers.
|
||||
audioMgr = ttsMgr // expose to caller for channel STT wiring (Phase 5)
|
||||
|
||||
// Audio generation tool — backed by audio.Manager (Music + SFX).
|
||||
toolsReg.Register(tools.NewCreateAudioTool(ttsMgr))
|
||||
|
||||
ttsTool = tools.NewTtsTool(ttsMgr)
|
||||
toolsReg.Register(ttsTool)
|
||||
if ttsMgr.HasProviders() {
|
||||
@@ -297,6 +303,14 @@ func wireTracingAndCron(
|
||||
Payload: map[string]any{"trace_ids": ids},
|
||||
})
|
||||
}
|
||||
// Immediate status broadcast on every successful status write (bypasses 5s flush).
|
||||
traceCollector.SetStatusBroadcaster(func(p tracing.TraceStatusPayload, tid uuid.UUID) {
|
||||
msgBus.Broadcast(bus.Event{
|
||||
Name: protocol.EventTraceStatusChanged,
|
||||
Payload: p,
|
||||
TenantID: tid,
|
||||
})
|
||||
})
|
||||
traceCollector.Start()
|
||||
slog.Info("LLM tracing enabled")
|
||||
}
|
||||
|
||||
@@ -22,7 +22,7 @@ func testExecToolFromGatewaySetup(t *testing.T, workspace, dataDir string) *tool
|
||||
cfg.Tools.Browser.Enabled = false
|
||||
|
||||
providerRegistry := providers.NewRegistry(nil)
|
||||
toolsReg, _, _, _, _, _, _, _, _, _, _ := setupToolRegistry(cfg, workspace, providerRegistry)
|
||||
toolsReg, _, _, _, _, _, _, _, _, _, _, _ := setupToolRegistry(cfg, workspace, providerRegistry)
|
||||
|
||||
execToolAny, ok := toolsReg.Get("exec")
|
||||
if !ok {
|
||||
|
||||
@@ -47,6 +47,15 @@ func makeDelegateAnnounceCallback(
|
||||
if meta.OriginSessionKey != "" {
|
||||
batchMeta[tools.MetaOriginSessionKey] = meta.OriginSessionKey
|
||||
}
|
||||
if meta.OriginSenderID != "" {
|
||||
batchMeta[tools.MetaOriginSenderID] = meta.OriginSenderID
|
||||
}
|
||||
if meta.OriginRole != "" {
|
||||
batchMeta[tools.MetaOriginRole] = meta.OriginRole
|
||||
}
|
||||
if meta.OriginUserID != "" {
|
||||
batchMeta[tools.MetaOriginUserID] = meta.OriginUserID
|
||||
}
|
||||
// Collect media from all items in the batch.
|
||||
var batchMedia []bus.MediaFile
|
||||
for _, item := range items {
|
||||
@@ -95,6 +104,8 @@ type subagentAnnounceRouting struct {
|
||||
OrigPeerKind string
|
||||
OrigLocalKey string
|
||||
UserID string
|
||||
SenderID string // real acting sender (preserves permission attribution through re-ingress, #915)
|
||||
Role string // caller's RBAC role; bypasses per-user grants for admin/operator/owner (#915)
|
||||
ParentAgent string
|
||||
ParentTraceID uuid.UUID
|
||||
ParentRootSpanID uuid.UUID
|
||||
@@ -168,6 +179,8 @@ func processSubagentAnnounceLoop(
|
||||
PeerKind: r.OrigPeerKind,
|
||||
LocalKey: r.OrigLocalKey,
|
||||
UserID: r.UserID,
|
||||
SenderID: r.SenderID, // preserves real acting sender for permission checks (#915)
|
||||
Role: r.Role, // preserves RBAC role for admin bypass in group writes (#915)
|
||||
RunID: fmt.Sprintf("subagent-announce-%s-%d", r.ParentAgent, len(entries)),
|
||||
RunKind: "announce",
|
||||
HideInput: true,
|
||||
@@ -182,10 +195,16 @@ func processSubagentAnnounceLoop(
|
||||
if outcome.Err != nil {
|
||||
if !errors.Is(outcome.Err, context.Canceled) {
|
||||
slog.Error("subagent announce: lead run failed", "error", outcome.Err, "batch_size", len(entries))
|
||||
errContent := formatAgentError(outcome.Err)
|
||||
if isExternalChannel(r.OrigChannelType) {
|
||||
slog.Info("subagent announce: suppressed error for external channel",
|
||||
"channel", r.OrigChannel, "type", r.OrigChannelType)
|
||||
errContent = ""
|
||||
}
|
||||
msgBus.PublishOutbound(bus.OutboundMessage{
|
||||
Channel: r.OrigChannel,
|
||||
ChatID: r.OrigChatID,
|
||||
Content: formatAgentError(outcome.Err),
|
||||
Content: errContent,
|
||||
Metadata: r.OutMeta,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -5,14 +5,27 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
"github.com/google/uuid"
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store/pg"
|
||||
)
|
||||
|
||||
func openTenantBackupDB(cfg *config.Config) (*sql.DB, error) {
|
||||
dsn := cfg.Database.PostgresDSN
|
||||
if dsn == "" {
|
||||
return nil, fmt.Errorf("GOCLAW_POSTGRES_DSN not configured")
|
||||
}
|
||||
|
||||
db, err := sql.Open("pgx", dsn)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open db: %w", err)
|
||||
}
|
||||
return db, nil
|
||||
}
|
||||
|
||||
// resolveTenantForCLI opens the DB, looks up the tenant by slug or UUID,
|
||||
// and returns the resolved tenant ID, slug, and an open *sql.DB (caller must close).
|
||||
func resolveTenantForCLI(cmd *cobra.Command, cfg *config.Config, rawID, slug string) (uuid.UUID, string, *sql.DB, error) {
|
||||
@@ -20,14 +33,9 @@ func resolveTenantForCLI(cmd *cobra.Command, cfg *config.Config, rawID, slug str
|
||||
return uuid.Nil, "", nil, fmt.Errorf("--tenant <slug> or --tenant-id <uuid> is required")
|
||||
}
|
||||
|
||||
dsn := cfg.Database.PostgresDSN
|
||||
if dsn == "" {
|
||||
return uuid.Nil, "", nil, fmt.Errorf("GOCLAW_POSTGRES_DSN not configured")
|
||||
}
|
||||
|
||||
db, err := sql.Open("pgx", dsn)
|
||||
db, err := openTenantBackupDB(cfg)
|
||||
if err != nil {
|
||||
return uuid.Nil, "", nil, fmt.Errorf("open db: %w", err)
|
||||
return uuid.Nil, "", nil, err
|
||||
}
|
||||
|
||||
ts := pg.NewPGTenantStore(db)
|
||||
|
||||
+72
-17
@@ -1,9 +1,12 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/backup"
|
||||
@@ -12,11 +15,12 @@ import (
|
||||
|
||||
func tenantRestoreCmd() *cobra.Command {
|
||||
var (
|
||||
tenantSlug string
|
||||
tenantID string
|
||||
mode string
|
||||
force bool
|
||||
dryRun bool
|
||||
tenantSlug string
|
||||
tenantID string
|
||||
newTenantSlug string
|
||||
mode string
|
||||
force bool
|
||||
dryRun bool
|
||||
)
|
||||
|
||||
cmd := &cobra.Command{
|
||||
@@ -25,9 +29,14 @@ func tenantRestoreCmd() *cobra.Command {
|
||||
Long: `Restores a tenant from a .tar.gz archive produced by 'goclaw tenant-backup'.
|
||||
|
||||
Modes:
|
||||
upsert (default) — INSERT … ON CONFLICT DO NOTHING. Non-destructive.
|
||||
replace — Delete existing tenant data first, then INSERT. Requires --force.
|
||||
new — Create a new tenant and import data under the new tenant ID.`,
|
||||
upsert (default) — INSERT ... ON CONFLICT DO NOTHING. Non-destructive.
|
||||
Requires --tenant or --tenant-id.
|
||||
replace — Wipes tenant-scoped data (keeps tenant metadata), then INSERT.
|
||||
Requires --tenant or --tenant-id AND --force.
|
||||
new — Creates a new tenant from archive metadata.
|
||||
Requires --new-tenant-slug. Archive's tenant_id is remapped
|
||||
to the new tenant. Users, API keys, LLM providers, and all
|
||||
other tenant-scoped data are cloned from the archive.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
archivePath := args[0]
|
||||
@@ -41,6 +50,14 @@ Modes:
|
||||
if _, err := os.Stat(archivePath); err != nil {
|
||||
return fmt.Errorf("archive not found: %s", archivePath)
|
||||
}
|
||||
|
||||
// Validate flag wiring before touching the filesystem state (--force check).
|
||||
// Catches invalid combinations like mode=new with --tenant before the user
|
||||
// sees destructive-operation prompts.
|
||||
if err := validateTenantRestoreFlags(mode, tenantSlug, tenantID, newTenantSlug); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if mode == "replace" && !dryRun && !force {
|
||||
fmt.Fprintln(os.Stderr, "ERROR: --force is required for replace mode (destructive operation).")
|
||||
os.Exit(1)
|
||||
@@ -51,14 +68,27 @@ Modes:
|
||||
return fmt.Errorf("load config: %w", err)
|
||||
}
|
||||
|
||||
// For "new" mode the tenant may not exist yet — allow lookup failure.
|
||||
tid, slug, db, lookupErr := resolveTenantForCLI(cmd, cfg, tenantID, tenantSlug)
|
||||
if lookupErr != nil && mode != "new" {
|
||||
return lookupErr
|
||||
}
|
||||
if db != nil {
|
||||
defer db.Close()
|
||||
var (
|
||||
tid = uuid.Nil
|
||||
slug string
|
||||
db *sql.DB
|
||||
)
|
||||
var restoreErr error
|
||||
|
||||
switch mode {
|
||||
case "new":
|
||||
db, restoreErr = openTenantBackupDB(cfg)
|
||||
if restoreErr != nil {
|
||||
return restoreErr
|
||||
}
|
||||
slug = strings.TrimSpace(newTenantSlug)
|
||||
default:
|
||||
tid, slug, db, restoreErr = resolveTenantForCLI(cmd, cfg, tenantID, tenantSlug)
|
||||
if restoreErr != nil {
|
||||
return restoreErr
|
||||
}
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
dataDir := config.TenantDataDir(cfg.ResolvedDataDir(), tid, slug)
|
||||
wsDir := config.TenantWorkspace(cfg.WorkspacePath(), tid, slug)
|
||||
@@ -106,10 +136,35 @@ Modes:
|
||||
},
|
||||
}
|
||||
|
||||
cmd.Flags().StringVar(&tenantSlug, "tenant", "", "target tenant slug")
|
||||
cmd.Flags().StringVar(&tenantID, "tenant-id", "", "target tenant UUID")
|
||||
cmd.Flags().StringVar(&tenantSlug, "tenant", "", "target tenant slug (for mode=upsert|replace)")
|
||||
cmd.Flags().StringVar(&tenantID, "tenant-id", "", "target tenant UUID (for mode=upsert|replace, alternative to --tenant)")
|
||||
cmd.Flags().StringVar(&newTenantSlug, "new-tenant-slug", "", "slug for the new tenant (required for mode=new)")
|
||||
cmd.Flags().StringVar(&mode, "mode", "upsert", "restore mode: upsert, replace, new")
|
||||
cmd.Flags().BoolVar(&force, "force", false, "required for replace mode")
|
||||
cmd.Flags().BoolVar(&dryRun, "dry-run", false, "inspect archive without making changes")
|
||||
return cmd
|
||||
}
|
||||
|
||||
// validateTenantRestoreFlags enforces flag semantics per restore mode.
|
||||
// Called before opening the database so invalid combinations fail fast.
|
||||
func validateTenantRestoreFlags(mode, tenant, tenantID, newTenantSlug string) error {
|
||||
switch mode {
|
||||
case "new":
|
||||
if strings.TrimSpace(newTenantSlug) == "" {
|
||||
return fmt.Errorf("--new-tenant-slug is required for mode=new (choose a unique slug for the new tenant)")
|
||||
}
|
||||
if tenant != "" || tenantID != "" {
|
||||
return fmt.Errorf("mode=new does not accept --tenant or --tenant-id; use --new-tenant-slug to name the new tenant")
|
||||
}
|
||||
case "upsert", "replace":
|
||||
if tenant == "" && tenantID == "" {
|
||||
return fmt.Errorf("--tenant <slug> or --tenant-id <uuid> is required for mode=%s", mode)
|
||||
}
|
||||
if strings.TrimSpace(newTenantSlug) != "" {
|
||||
fmt.Fprintf(os.Stderr, "warning: --new-tenant-slug is ignored for mode=%s\n", mode)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("invalid --mode=%q (allowed: upsert, replace, new)", mode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestValidateTenantRestoreFlags(t *testing.T) {
|
||||
tests := []struct {
|
||||
name, mode, tenant, tenantID, newSlug, wantErr string
|
||||
}{
|
||||
{"new ok", "new", "", "", "fresh-slug", ""},
|
||||
{"new missing slug", "new", "", "", "", "required for mode=new"},
|
||||
{"new with tenant slug", "new", "acme", "", "fresh-slug", "does not accept"},
|
||||
{"new with tenant-id", "new", "", "uuid-x", "fresh-slug", "does not accept"},
|
||||
{"upsert ok tenant", "upsert", "acme", "", "", ""},
|
||||
{"upsert ok tenant-id", "upsert", "", "uuid-x", "", ""},
|
||||
{"upsert ok both", "upsert", "acme", "uuid-x", "", ""},
|
||||
{"upsert missing", "upsert", "", "", "", "required for mode=upsert"},
|
||||
{"upsert ignores new slug (no error)", "upsert", "acme", "", "ignored", ""},
|
||||
{"replace ok", "replace", "acme", "", "", ""},
|
||||
{"replace missing", "replace", "", "", "", "required for mode=replace"},
|
||||
{"invalid mode", "bogus", "acme", "", "", "invalid --mode"},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
err := validateTenantRestoreFlags(tc.mode, tc.tenant, tc.tenantID, tc.newSlug)
|
||||
if tc.wantErr == "" {
|
||||
if err != nil {
|
||||
t.Fatalf("want nil, got %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err == nil {
|
||||
t.Fatalf("want err containing %q, got nil", tc.wantErr)
|
||||
}
|
||||
if !strings.Contains(err.Error(), tc.wantErr) {
|
||||
t.Fatalf("want err containing %q, got %q", tc.wantErr, err.Error())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -7,7 +7,7 @@ RUNTIME_DIR="/app/data/.runtime"
|
||||
# Non-fatal: on first start with a fresh volume the directory may not be
|
||||
# writable yet (volume initialisation race on some Docker runtimes).
|
||||
# The app starts fine without .runtime; package installs will fail gracefully.
|
||||
mkdir -p "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global/lib" "$RUNTIME_DIR/pip-cache" || true
|
||||
mkdir -p "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global/lib" "$RUNTIME_DIR/pip-cache" "$RUNTIME_DIR/bin" || true
|
||||
|
||||
# Fix .runtime ownership for split root/goclaw access.
|
||||
# .runtime itself must be root-owned so pkg-helper (root) can write apk-packages.
|
||||
@@ -16,7 +16,7 @@ mkdir -p "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global/lib" "$RUNTIME_DIR/pip-cach
|
||||
if [ "$(id -u)" = "0" ] && [ -d "$RUNTIME_DIR" ]; then
|
||||
chown root:goclaw "$RUNTIME_DIR" 2>/dev/null || true
|
||||
chmod 0750 "$RUNTIME_DIR" 2>/dev/null || true
|
||||
chown -R goclaw:goclaw "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global" "$RUNTIME_DIR/pip-cache" 2>/dev/null || true
|
||||
chown -R goclaw:goclaw "$RUNTIME_DIR/pip" "$RUNTIME_DIR/npm-global" "$RUNTIME_DIR/pip-cache" "$RUNTIME_DIR/bin" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Fix workspace directory ownership: handle dirs created by root in previous
|
||||
@@ -38,7 +38,7 @@ export PIP_CACHE_DIR="$RUNTIME_DIR/pip-cache"
|
||||
# NODE_PATH includes both pre-installed system globals and runtime-installed globals.
|
||||
export NPM_CONFIG_PREFIX="$RUNTIME_DIR/npm-global"
|
||||
export NODE_PATH="/usr/local/lib/node_modules:$RUNTIME_DIR/npm-global/lib/node_modules:${NODE_PATH:-}"
|
||||
export PATH="$RUNTIME_DIR/npm-global/bin:$RUNTIME_DIR/pip/bin:$PATH"
|
||||
export PATH="$RUNTIME_DIR/bin:$RUNTIME_DIR/npm-global/bin:$RUNTIME_DIR/pip/bin:$PATH"
|
||||
|
||||
# System packages: re-install on-demand packages persisted across recreates.
|
||||
# After chown above, root owns .runtime and can create this file.
|
||||
|
||||
@@ -38,6 +38,7 @@ flowchart TD
|
||||
SCHED[Scheduler -- 4 Lanes]
|
||||
AR[Agent Router]
|
||||
LOOP[Agent Loop -- Think / Act / Observe]
|
||||
HOOKS[Hook Dispatcher -- Lifecycle Events]
|
||||
end
|
||||
|
||||
subgraph Providers["LLM Providers"]
|
||||
@@ -120,7 +121,8 @@ flowchart TD
|
||||
| `internal/permissions/` | RBAC policy engine (admin, operator, viewer roles) |
|
||||
| `internal/store/pg/pairing.go` | DM/device pairing service (8-character codes, database-backed) |
|
||||
| `internal/sandbox/` | Docker-based code execution sandbox |
|
||||
| `internal/tts/` | Text-to-Speech providers: OpenAI, ElevenLabs, Edge, MiniMax |
|
||||
| `internal/audio/` | Unified audio manager: 4 provider interfaces (TTS active; STT/Music/SFX stubbed/partial). Orchestrates ElevenLabs, OpenAI, Edge, MiniMax TTS providers. `internal/tts/` retained as backward-compat alias |
|
||||
| `internal/tts/` | Backward-compat alias layer (24 symbols) — all pre-refactor callers compile unchanged |
|
||||
| `internal/http/` | HTTP API handlers: /v1/chat/completions, /v1/agents, /v1/skills, /v1/traces, /v1/mcp, /v1/delegations, summoner |
|
||||
| `internal/crypto/` | AES-256-GCM encryption for API keys |
|
||||
| `internal/tracing/` | LLM call tracing (traces + spans), in-memory buffer with periodic store flush |
|
||||
@@ -142,6 +144,67 @@ flowchart TD
|
||||
| `internal/workspace/` | Workspace context resolver: 6 scenarios (agent default, team lead, team member, dispatch, subagent, cron) |
|
||||
| `internal/vault/` | Knowledge Vault: wikilinks (semantic mesh), hybrid search (BM25+vector), filesystem sync, L0 auto-injection |
|
||||
| `internal/channels/whatsapp/` | Native WhatsApp channel via whatsmeow (replaces WhatsApp API), QR auth, media handling |
|
||||
| `internal/hooks/` | Agent lifecycle hooks: event dispatcher (sync/async), concrete handlers (command/http), matchers (regex + CEL), audit logging, edition gating, cost safeguards. Events: SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, Stop, SubagentStart/Stop. Handlers: CommandHandler (shell, Lite-only), HTTPHandler (SSRF-hardened, auth decrypt) |
|
||||
| `internal/hooks/handlers/` | Concrete hook handler implementations: `CommandHandler` (exec wrapper, JSON I/O, env allowlist, edition recheck), `HTTPHandler` (SSRF-hardened HTTP client, retry-once on 5xx, no redirects, encrypted auth headers) |
|
||||
|
||||
---
|
||||
|
||||
## 3.5 Agent Hooks System
|
||||
|
||||
**Lifecycle hooks** allow agents to perform custom logic at key execution points. The system is event-driven (sync/async), integrated into the 8-stage pipeline, and includes audit logging, edition gating, and security safeguards.
|
||||
|
||||
### Event Types
|
||||
|
||||
| Event | Stage | Sync/Async | Purpose |
|
||||
|-------|-------|-----------|---------|
|
||||
| **SessionStart** | ContextStage | Async | Fires once per session (first turn); before history loading |
|
||||
| **UserPromptSubmit** | ContextStage | Sync | Fires on user message arrival; blocks with hook reason or mutates input |
|
||||
| **PreToolUse** | ToolStage | Sync | Fires before each tool execution; blocks tool or mutates arguments |
|
||||
| **PostToolUse** | ToolStage | Async | Fires after tool result is processed; non-blocking |
|
||||
| **Stop** | FinalizeStage | Async | Fires when session terminates (stop/complete/error) |
|
||||
| **SubagentStart** | (Delegate tool) | Sync | Fires before delegated task spawns; blocks delegation |
|
||||
| **SubagentStop** | (Domain events) | Async | Fires on delegation completion/failure; non-blocking |
|
||||
|
||||
### Handler Types
|
||||
|
||||
| Handler | Edition | Config | Semantics |
|
||||
|---------|---------|--------|-----------|
|
||||
| **Command** | Lite only | `cmd`, `allowedEnvVars` | Exec shell command; stdin=event JSON; stdout=decision JSON; exit 0→allow, exit 2→block; timeout→block (fail-closed) |
|
||||
| **HTTP** | All | `url`, `headers` | POST event JSON to webhook URL; parse response for decision, additionalContext, updatedInput; 5xx retry once; 4xx error no-retry; SSRF-hardened with pinned IP |
|
||||
| **Prompt** | Phase 3+ | TBD | Integrates custom prompting logic (deferred) |
|
||||
|
||||
### Sync vs Async
|
||||
|
||||
**Sync hooks** (UserPromptSubmit, PreToolUse, SubagentStart):
|
||||
- Block pipeline until decision received
|
||||
- Support Copy-on-Write (COW) staged mutations: `updatedInput` buffered, committed only if ALL sync hooks succeed
|
||||
- Timeout ≤5s per hook; chain total ≤10s wall-time
|
||||
- Rejection blocks the step (e.g., tool not executed, user message not processed)
|
||||
|
||||
**Async hooks** (SessionStart, PostToolUse, Stop, SubagentStop):
|
||||
- Fire-and-forget via worker pool (default 16 workers, bounded queue 512)
|
||||
- Non-blocking (pipeline continues immediately)
|
||||
- Timeout per hook; chain budget enforced but no blocking
|
||||
|
||||
### Security Model
|
||||
|
||||
- **Edition gating**: `CommandHandler` only on Lite edition; attempts on Standard/other editions rejected (defense-in-depth)
|
||||
- **SSRF hardening (HTTPHandler)**: Caller supplies net.Dialer pinning resolved IP, blocking loopback/link-local/private ranges; no HTTP redirects (CheckRedirect returns ErrUseLastResponse)
|
||||
- **Auth header encryption**: `Authorization` + other sensitive fields in cfg.Config["headers"] encrypted at rest via AES-256-GCM; decrypted only at HTTP send-time
|
||||
- **Audit logging**: All hook invocations logged to `hook_executions` table (encrypted, PII-redacted) with dedup_key for idempotency
|
||||
- **Loop-depth guard (M5)**: SubagentStart checks recursion depth; max 3 levels prevents infinite delegation chains
|
||||
- **Circuit breaker**: Auto-disables hook after 3 consecutive failures in recent window (C4 mitigation)
|
||||
|
||||
### Pipeline Integration
|
||||
|
||||
Dispatcher wired into `PipelineDeps.HookDispatcher` (nil-safe noop fallback). All 8 stages support hook firing with zero overhead when dispatcher not configured. Example sync hook flow:
|
||||
|
||||
```
|
||||
1. ContextStage: UserPromptSubmit → Hooks.Fire(sync)
|
||||
2. Sync hooks buffer mutations (updatedInput)
|
||||
3. All hooks succeed? → Commit mutations, proceed
|
||||
4. Any hook rejects? → Discard buffer, abort pipeline, user sees reason
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -474,6 +537,29 @@ Six distinct workspace scenarios:
|
||||
- **L0 Auto-Injection**: Top-K vault entries injected into system prompt as "relevant context from vault"
|
||||
- **Filesystem Sync**: Vault entries exported as `.md` files for manual editing, re-imported with change tracking
|
||||
|
||||
### Audio & Voice System (ElevenLabs + Streaming TTS)
|
||||
|
||||
**Provider architecture** (`internal/audio/`):
|
||||
- **`Manager`**: Central orchestrator dispatching TTS/STT/Music/SFX requests to pluggable providers
|
||||
- **`TTSProvider` interface**: Core text-to-speech contract (blocking, buffered response)
|
||||
- **`StreamingTTSProvider` interface**: Optional interface for ElevenLabs `/stream` endpoint (chunked audio via `io.ReadCloser`)
|
||||
- **Implementations**: ElevenLabs, OpenAI, Edge, MiniMax (phase-gated; STT/Music/SFX partial)
|
||||
|
||||
**Voice discovery & caching**:
|
||||
- **Voice cache** (`internal/audio/voice_cache.go`): In-memory LRU (cap 1000 tenants, TTL 1h) shared by HTTP `/v1/voices` + WS `voices.list` handlers. Thread-safe with `sync.Mutex` (LRU updates require write lock)
|
||||
- **Cache miss recovery**: HTTP handler auto-fetches from ElevenLabs; WS handler requires prior cache warm (provider resolution deferred to Phase 3)
|
||||
- **Agent audio context** (`store.WithAgentAudio` / `AgentAudioFromCtx`): Immutable snapshot bundle (AgentID + OtherConfig JSONB) injected by dispatcher before tool dispatch; consumed by `TtsTool.Execute` for voice/model resolution
|
||||
|
||||
**Agent-level configuration** (`agents.other_config` JSONB):
|
||||
- `tts_voice_id`: ElevenLabs voice ID (e.g., "pMsXgVXv3BLzUgSXRplE")
|
||||
- `tts_model_id`: Model choice (eleven_v3, eleven_flash_v2_5, eleven_multilingual_v2, eleven_turbo_v2_5)
|
||||
- Resolution precedence: CLI args → agent config → tenant override → provider default
|
||||
|
||||
**Web UI voice picker** (`ui/web/src/components/voice-picker.tsx`):
|
||||
- Combobox with BM25 search, preview playback button (HTML `<audio>`)
|
||||
- Handles preview CDN 403 (expiry) via `onError` → auto-refresh cache
|
||||
- Embedded in PromptSettingsSection, bound to `other_config.tts_voice_id`
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
+29
-18
@@ -59,15 +59,16 @@ Finalize (runs once, uses background context if cancelled)
|
||||
- Resolve workspace + context files dynamically
|
||||
- Build system prompt (15+ sections)
|
||||
- Inject conversation summary if exists
|
||||
- Run history pipeline (limitHistoryTurns → pruneContextMessages → sanitizeHistory)
|
||||
- Run history pipeline (limitHistoryTurns → sanitizeHistory)
|
||||
- Filter tools through PolicyEngine (RBAC)
|
||||
- Call LLM, record span with token counts
|
||||
- Emit `chunk` events (streaming) or single response
|
||||
|
||||
**PruneStage**
|
||||
**PruneStage** (opt-in via `contextPruning.mode: "cache-ttl"`)
|
||||
- Estimate token ratio vs context window
|
||||
- If >= 30%, run soft trim pass (keep first/last 1500 chars, replace middle with "...")
|
||||
- If >= 25%, run soft trim pass (keep first/last 3000 chars, replace middle with "...")
|
||||
- If >= 50%, run hard clear pass (replace with placeholder)
|
||||
- Run sanitizeHistory to fix broken tool_use/tool_result pairs after prune
|
||||
- Trigger memory flush (synchronous) if compaction threshold exceeded
|
||||
|
||||
**ToolStage**
|
||||
@@ -183,7 +184,7 @@ flowchart TD
|
||||
|
||||
subgraph PH3["Phase 3: Build Messages"]
|
||||
P3A[Build system prompt - 15+ sections] --> P3B[Inject conversation summary if present]
|
||||
P3B --> P3C["History pipeline: limitHistoryTurns --> pruneContextMessages --> sanitizeHistory"]
|
||||
P3B --> P3C["History pipeline: limitHistoryTurns --> sanitizeHistory"]
|
||||
P3C --> P3D[Append current user message]
|
||||
P3D --> P3E[Buffer user message locally - deferred write]
|
||||
end
|
||||
@@ -398,25 +399,20 @@ All security events use the `slog.Warn("security.injection_detected")` conventio
|
||||
|
||||
## 5. History Pipeline
|
||||
|
||||
The history pipeline prepares conversation history before sending it to the LLM. It runs in three sequential stages.
|
||||
The history pipeline prepares conversation history before sending it to the LLM. It runs in two sequential stages. Context pruning is handled separately by PruneStage (opt-in via `contextPruning.mode: "cache-ttl"`).
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
RAW[Raw Session History] --> S1
|
||||
S1["Stage 1: limitHistoryTurns<br/>Keep the last N user turns<br/>plus their associated assistant/tool messages"] --> S2
|
||||
S2["Stage 2: pruneContextMessages<br/>2-pass tool result trimming<br/>(see Section 6)"] --> S3
|
||||
S3["Stage 3: sanitizeHistory<br/>Repair broken tool_use / tool_result pairing<br/>after truncation"] --> OUT[Cleaned History]
|
||||
S2["Stage 2: sanitizeHistory<br/>Repair broken tool_use / tool_result pairing<br/>after truncation"] --> OUT[Cleaned History]
|
||||
```
|
||||
|
||||
### Stage 1: limitHistoryTurns
|
||||
|
||||
Takes the raw session history and a `historyLimit` parameter. Keeps only the last N user turns along with all associated assistant and tool messages that belong to those turns. Earlier messages are discarded.
|
||||
|
||||
### Stage 2: pruneContextMessages
|
||||
|
||||
Applies the 2-pass context pruning algorithm described in Section 6.
|
||||
|
||||
### Stage 3: sanitizeHistory
|
||||
### Stage 2: sanitizeHistory
|
||||
|
||||
Repairs tool message pairing that may have been broken by truncation or compaction:
|
||||
|
||||
@@ -429,15 +425,17 @@ Repairs tool message pairing that may have been broken by truncation or compacti
|
||||
|
||||
## 6. Context Pruning
|
||||
|
||||
Context pruning reduces oversized tool results using a 2-pass algorithm. It only activates when the estimated token-to-context-window ratio crosses a threshold.
|
||||
Context pruning reduces oversized tool results using a 2-pass algorithm. **It is opt-in** — configure `contextPruning.mode: "cache-ttl"` to enable. When disabled (default), zero overhead. Owned by PruneStage in the agent pipeline.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
START[Estimate token ratio vs context window] --> CHECK{Ratio >= softTrimRatio 0.3?}
|
||||
START[Check mode == cache-ttl?] --> GATE{Mode enabled?}
|
||||
GATE -->|No| SKIP[No pruning - zero overhead]
|
||||
GATE -->|Yes| CHECK{Ratio >= softTrimRatio 0.25?}
|
||||
CHECK -->|No| DONE[No pruning needed]
|
||||
CHECK -->|Yes| PASS1
|
||||
|
||||
PASS1["Pass 1: Soft Trim<br/>For each eligible tool result > 4000 chars:<br/>Keep first 1500 chars + last 1500 chars<br/>Replace middle with '...'"]
|
||||
PASS1["Pass 1: Soft Trim<br/>For each eligible tool result > 6000 chars:<br/>Keep first 3000 chars + last 3000 chars<br/>Replace middle with '...'"]
|
||||
PASS1 --> CHECK2{"Ratio >= hardClearRatio 0.5?"}
|
||||
CHECK2 -->|No| DONE
|
||||
CHECK2 -->|Yes| PASS2
|
||||
@@ -446,12 +444,25 @@ flowchart TD
|
||||
PASS2 --> DONE
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Enable pruning by setting `contextPruning.mode` in agent defaults:
|
||||
|
||||
```json5
|
||||
agents: {
|
||||
defaults: {
|
||||
contextPruning: { mode: "cache-ttl" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Defaults
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `mode` | `""` (disabled) | `""` or `"off"` = disabled; `"cache-ttl"` = enabled |
|
||||
| `keepLastAssistants` | 3 | Number of recent assistant messages protected from pruning |
|
||||
| `softTrimRatio` | 0.3 | Token ratio threshold to trigger Pass 1 |
|
||||
| `softTrimRatio` | 0.25 | Token ratio threshold to trigger Pass 1 |
|
||||
| `hardClearRatio` | 0.5 | Token ratio threshold to trigger Pass 2 |
|
||||
| `minPrunableToolChars` | 50,000 | Minimum tool result length eligible for hard clear |
|
||||
|
||||
@@ -719,8 +730,8 @@ Enabled via the `GOCLAW_TRACE_VERBOSE=1` environment variable.
|
||||
| `internal/agent/loop_run.go` | Run() entry point: dual-mode gate (v2 vs v3), trace creation, span management |
|
||||
| `internal/agent/loop_pipeline_adapter.go` | Bridge v2 Loop to v3 Pipeline: state conversion, dependency injection, callback wiring |
|
||||
| `internal/agent/loop.go` | runLoop() core loop: LLM iteration, tool execution, message buffering (v2 path) |
|
||||
| `internal/agent/loop_history.go` | History pipeline: limitHistoryTurns, pruneContextMessages, sanitizeHistory, summary injection |
|
||||
| `internal/agent/pruning.go` | Context pruning: 2-pass soft trim and hard clear algorithm |
|
||||
| `internal/agent/loop_history.go` | History pipeline: limitHistoryTurns, sanitizeHistory, summary injection |
|
||||
| `internal/agent/pruning.go` | Context pruning: 2-pass soft trim and hard clear algorithm (opt-in via PruneStage) |
|
||||
| `internal/agent/loop_compact.go` | Mid-loop compaction: in-memory message summarization during iterations |
|
||||
| `internal/agent/systemprompt.go` | System prompt assembly (19+ sections), PromptFull and PromptMinimal modes |
|
||||
| `internal/agent/systemprompt_sections.go` | Individual section builders (tooling, workspace, sandbox, skills, MCP, etc.) |
|
||||
|
||||
@@ -812,10 +812,17 @@ func (t *MyTool) Execute(ctx context.Context, params MyParams) (*Result, error)
|
||||
#### tts (tenant override shape)
|
||||
```json
|
||||
{
|
||||
"primary": "elevenlabs"
|
||||
"primary": "elevenlabs",
|
||||
"default_voice_id": "pMsXgVXv3BLzUgSXRplE",
|
||||
"default_model": "eleven_flash_v2_5"
|
||||
}
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
- `primary`: Provider selection (elevenlabs, openai, edge, minimax)
|
||||
- `default_voice_id`: Tenant-level voice ID fallback (ElevenLabs); overridable per-agent via `agent.other_config.tts_voice_id`
|
||||
- `default_model`: Tenant-level model choice (eleven_v3, eleven_flash_v2_5, eleven_multilingual_v2, eleven_turbo_v2_5); overridable per-agent via `agent.other_config.tts_model_id`
|
||||
|
||||
**Secret vs non-secret split:**
|
||||
- Non-secret (provider priorities, max_results, allowed_domains): `builtin_tool_tenant_configs.settings` (editable via UI)
|
||||
- Secret (API keys): `config_secrets` table (encrypted, tenant-scoped)
|
||||
@@ -888,8 +895,7 @@ Never put credentials in tool settings JSON — backend does not validate.
|
||||
|------|---------|
|
||||
| `internal/tools/create_image.go` | create_image tool (OpenAI, Gemini, MiniMax, DashScope) |
|
||||
| `internal/tools/create_image_{dashscope,minimax}.go` | Provider-specific image generation |
|
||||
| `internal/tools/create_audio.go` | create_audio tool (MiniMax, ElevenLabs, Suno) |
|
||||
| `internal/tools/create_audio_{minimax,elevenlabs,suno}.go` | Provider-specific audio generation |
|
||||
| `internal/tools/create_audio.go` | create_audio tool — delegates to `audio.Manager` for music/SFX (ElevenLabs, MiniMax) |
|
||||
| `internal/tools/create_video.go` | create_video tool (MiniMax) |
|
||||
| `internal/tools/tts.go` | tts tool: text-to-speech (OpenAI, ElevenLabs, Edge, MiniMax) |
|
||||
| `internal/tools/read_{image,audio,video,document}.go` | Media reading tools (vision, transcription, analysis) |
|
||||
|
||||
@@ -245,26 +245,81 @@ Each topic can restrict which tools the agent may use. The `tools` field accepts
|
||||
|
||||
The tool allow list is passed via message metadata and applied by the policy engine before the LLM sees the tool definitions.
|
||||
|
||||
### Speech-to-Text
|
||||
### Voice Message Transcription (STT)
|
||||
|
||||
Voice and audio messages can be transcribed via an external STT proxy service.
|
||||
Voice and audio messages are transcribed through a unified `audio.Manager` interface. All channels (Telegram, Discord, Feishu, WhatsApp) route transcription requests through the same STT chain.
|
||||
|
||||
#### Unified STT Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
VOICE["Voice/audio message"] --> DOWNLOAD["Download audio file<br/>from Telegram"]
|
||||
DOWNLOAD --> STT["POST to STT proxy<br/>(multipart: file + tenant_id)"]
|
||||
STT --> PARSE["Parse transcript"]
|
||||
PARSE --> INJECT["Prepend to message:<br/>[audio: filename] Transcript: text"]
|
||||
INJECT --> AGENT["Agent receives transcribed content"]
|
||||
|
||||
VOICE --> ROUTING{"VoiceAgentID configured?"}
|
||||
VOICE["Voice/audio message"] --> ROUTE{Channel type?}
|
||||
|
||||
ROUTE -->|Telegram/Discord/Feishu| DOWNLOAD["Download audio file"]
|
||||
ROUTE -->|WhatsApp| WHATSAPP_CHECK{"whatsapp_enabled<br/>in settings?"}
|
||||
|
||||
WHATSAPP_CHECK -->|No| WA_FALLBACK["[Voice message]<br/>(default opt-out)"]
|
||||
WHATSAPP_CHECK -->|Yes| DOWNLOAD
|
||||
|
||||
DOWNLOAD --> STT_CHECK{"STT providers<br/>configured?"}
|
||||
STT_CHECK -->|Yes| STT_CHAIN["Try providers in order:<br/>elevenlabs_scribe, proxy"]
|
||||
STT_CHECK -->|No| LEGACY{"Legacy bridge<br/>providers?"}
|
||||
|
||||
LEGACY -->|Yes| STT_CHAIN
|
||||
LEGACY -->|No| FALLBACK["[Voice message]"]
|
||||
|
||||
STT_CHAIN -->|Success| TEXT["Transcribed text"]
|
||||
STT_CHAIN -->|Fails/Timeout 10s| FALLBACK
|
||||
|
||||
TEXT --> INJECT["Prepend to message:<br/>[audio: filename] Transcript: text"]
|
||||
FALLBACK --> FALLBACK_INJECT["[Voice message]"]
|
||||
|
||||
INJECT --> AGENT["Agent context"]
|
||||
FALLBACK_INJECT --> AGENT
|
||||
|
||||
VOICE --> ROUTING{"VoiceAgentID<br/>configured<br/>(Telegram)?"}
|
||||
ROUTING -->|Yes| VOICE_AGENT["Route to voice-specific agent"]
|
||||
ROUTING -->|No| DEFAULT_AGENT["Route to channel's default agent"]
|
||||
ROUTING -->|No| DEFAULT_AGENT["Default channel agent"]
|
||||
```
|
||||
|
||||
**Configuration**: STT proxy URL, timeout (default 30s), optional tenant ID and API key. If transcription fails, the media placeholder remains — no error is surfaced.
|
||||
#### Configuration & Decision Rules
|
||||
|
||||
**Voice routing**: When `VoiceAgentID` is configured, audio/voice messages are routed to a different agent (e.g., a speech-specialized agent) instead of the channel's default agent.
|
||||
**Decision 2 (Conflict rule):** When `builtin_tools[stt].settings.providers[]` is present in the database, it OVERRIDES all legacy channel-specific STT configs. The legacy STT bridge (`STTProxyURL` → `proxy_stt` provider) only activates when the providers list is empty or missing.
|
||||
|
||||
| Setting | Behavior |
|
||||
|---------|----------|
|
||||
| `providers: ["elevenlabs_scribe", "proxy_stt"]` | Try ElevenLabs Scribe first; fall back to legacy proxy |
|
||||
| `providers: []` (empty) | Skip all STT; voice → `[Voice message]` fallback |
|
||||
| `providers` missing (nil) | Check for legacy bridge at startup; activate if `STTProxyURL` exists |
|
||||
|
||||
**Decision 6 (WhatsApp opt-in):** WhatsApp voice message STT is **OFF by default** (`whatsapp_enabled: false`). Rationale: WhatsApp voice messages are end-to-end encrypted; sending audio to an external STT provider breaks E2E encryption. Admins must explicitly toggle STT in the UI at **Config → Audio → STT** and acknowledge the E2E breaking change.
|
||||
|
||||
When disabled (default):
|
||||
- Voice messages surface in agent context as `[Voice message]` (localized via i18n key `channel.voice_message_fallback`)
|
||||
- No audio leaves the device
|
||||
|
||||
When enabled:
|
||||
- Voice audio is transcribed via the configured STT chain
|
||||
- Fallback to `[Voice message]` on failure/timeout (10s wall clock)
|
||||
|
||||
#### Per-Channel Behavior
|
||||
|
||||
| Channel | STT Support | Notes |
|
||||
|---------|:-:|---------|
|
||||
| **Telegram** | Yes | Uses unified chain; voice routing via `VoiceAgentID` config |
|
||||
| **Discord** | Yes | Standard flow; no special routing |
|
||||
| **Feishu** | Yes | Standard flow; no special routing |
|
||||
| **WhatsApp** | Yes (opt-in) | Requires explicit admin approval; default OFF |
|
||||
|
||||
#### Factory Integration
|
||||
|
||||
All channel factories accept `audioMgr *audio.Manager`:
|
||||
- Telegram: `FactoryWithStoresAndAudio(..., audioMgr)`
|
||||
- Discord: `FactoryWithAudio(..., audioMgr)`
|
||||
- Feishu: `FactoryWithStoresAndAudio(..., audioMgr)`
|
||||
- WhatsApp: `FactoryWithDBAudio(..., audioMgr, builtinToolStore)`
|
||||
|
||||
WhatsApp additionally accepts `builtinToolStore store.BuiltinToolStore` to fetch the per-message `whatsapp_enabled` opt-in flag. Wiring in `cmd/gateway_channels_setup.go:77` + `cmd/gateway.go:433`.
|
||||
|
||||
### Bot Commands
|
||||
|
||||
@@ -350,6 +405,8 @@ Each update increments a sequence number for ordering. Updates are throttled at
|
||||
| Video | `.mp4` |
|
||||
| Sticker | `.png` |
|
||||
|
||||
**Filename preservation**: When a user uploads or shares a file with a recognizable name, the channel adapter populates `bus.MediaFile.Filename` with the original filename (e.g., Feishu file display name, Telegram `file_name`). This filename is then sanitized and persisted to disk in the format `{stem}-{8hex}{ext}` (e.g., `báo-cáo-2024-a1b2c3d4.pdf` → `bao-cao-2024-a1b2c3d4.pdf`). The sanitizer supports Vietnamese diacritics, CJK scripts, and filesystem safety. Media without a user-provided filename (voice notes, clipboard pastes) fall back to UUID-only names. Disk names with semantic stems enable vault enrichment to process documents contextually. See `internal/agent/media_filename.go` for sanitization rules.
|
||||
|
||||
**Send (outbound)**: Files are uploaded to Feishu with automatic type detection (opus, mp4, pdf, doc, xls, ppt, or generic stream).
|
||||
|
||||
### Mention Resolution
|
||||
@@ -629,7 +686,7 @@ flowchart TD
|
||||
| `internal/channels/telegram/handlers.go` | Message handling, media processing, forum topic detection |
|
||||
| `internal/channels/telegram/topic_config.go` | Per-topic config layering and resolution |
|
||||
| `internal/channels/telegram/commands.go` | Bot commands: /stop, /reset, /tasks, /addwriter, etc. |
|
||||
| `internal/channels/telegram/stt.go` | Speech-to-text proxy integration, voice agent routing |
|
||||
| `internal/channels/telegram/factory.go` | Channel factory with audio.Manager wiring |
|
||||
| `internal/channels/telegram/stream.go` | Streaming placeholder management |
|
||||
| `internal/channels/telegram/reactions.go` | Status reactions on messages |
|
||||
| `internal/channels/telegram/format.go` | Markdown → Telegram HTML pipeline, table rendering |
|
||||
@@ -647,11 +704,15 @@ flowchart TD
|
||||
| `internal/channels/slack/reactions.go` | Status emoji reactions on messages |
|
||||
| `internal/channels/slack/stream.go` | Streaming message updates via placeholder editing |
|
||||
| `internal/channels/whatsapp/whatsapp.go` | WhatsApp: direct protocol client, QR auth, database persistence |
|
||||
| `internal/channels/whatsapp/factory.go` | Channel factory, database dialect detection |
|
||||
| `internal/channels/whatsapp/factory.go` | Channel factory with audio.Manager and builtin-tool store wiring |
|
||||
| `internal/channels/whatsapp/stt.go` | Voice transcription with opt-in setting and E2E fallback |
|
||||
| `internal/channels/whatsapp/qr_methods.go` | QR code generation and authentication flow |
|
||||
| `internal/channels/whatsapp/format.go` | Message formatting (HTML-to-WhatsApp) |
|
||||
| `internal/channels/zalo/zalo.go` | Zalo OA: Bot API, long polling |
|
||||
| `internal/channels/zalo/personal/channel.go` | Zalo Personal: reverse-engineered protocol |
|
||||
| `internal/audio/manager.go` | Audio manager: providers registry, lifecycle |
|
||||
| `internal/audio/manager_stt.go` | STT chain resolution, Transcribe() entry point |
|
||||
| `internal/audio/legacy_stt_bridge.go` | Backward-compat bridge for legacy STTProxyURL configs |
|
||||
| `internal/store/pg/pairing.go` | Pairing: code generation, approval, persistence (database-backed) |
|
||||
| `cmd/gateway_consumer.go` | Message routing: prefixes, cancel interception |
|
||||
|
||||
@@ -666,3 +727,4 @@ flowchart TD
|
||||
| [08-scheduling-cron.md](./08-scheduling-cron.md) | /stop and /stopall commands, scheduler lanes, cron |
|
||||
| [09-security.md](./09-security.md) | Group file writer restrictions, security logging |
|
||||
| [11-agent-teams.md](./11-agent-teams.md) | Team message routing, delegation result delivery |
|
||||
| [project-changelog.md](./project-changelog.md) | Phase 5 audio manager & unified STT implementation |
|
||||
@@ -47,6 +47,7 @@ The `Stores` struct is the top-level container holding all PostgreSQL-backed sto
|
||||
| SnapshotStore | `PGSnapshotStore` | Hourly usage snapshots, cost aggregation, time series queries |
|
||||
| SecureCLIStore | `PGSecureCLIStore` | CLI binary configs with encrypted credential injection |
|
||||
| APIKeyStore | `PGAPIKeyStore` | Gateway API keys, scopes, expiration, revocation |
|
||||
| HookStore | `PGHookStore` | Lifecycle hook definitions (event, handler type, matcher, config), execution audit log |
|
||||
|
||||
### SQLite Parity (Lite Edition)
|
||||
|
||||
@@ -63,6 +64,7 @@ The `Stores` struct is the top-level container holding all PostgreSQL-backed sto
|
||||
| AgentLinksStore | `SQLiteAgentLinks` | LIKE search, no vector |
|
||||
| SubagentTasksStore | `SQLiteSubagentTasks` | ✓ Parity (json_set for metadata merge) |
|
||||
| SecureCLIStore | `SQLiteSecureCLIStore` | ✓ Parity + AES-256-GCM encryption mandatory (GOCLAW_KEY env var required) |
|
||||
| HookStore | `SQLiteHookStore` | ✓ Parity (agent_hooks + hook_executions tables, same schema as PG) |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -30,9 +30,16 @@ flowchart LR
|
||||
|
||||
### Cancel Handling
|
||||
|
||||
When a run is cancelled via `/stop` or `/stopall`, the run context is cancelled but trace finalization still needs to persist. `FinishTrace()` detects `ctx.Err() != nil` and switches to `context.Background()` for the final database write. The trace status is set to `"cancelled"` instead of `"error"`.
|
||||
When a run is cancelled via `chat.abort` (WS) or manual stop, the router atomically transitions to "aborting" state, cancels the request context, and waits 3 seconds for the agent goroutine to exit. If it doesn't, the trace is force-marked `cancelled` to prevent orphaned traces.
|
||||
|
||||
Context values (traceID, collector) survive cancellation -- only `ctx.Done()` and `ctx.Err()` change. This allows trace finalization to find everything it needs with a fresh context for the DB call.
|
||||
During cancellation:
|
||||
- HTTP provider streams close immediately (context-aware transport with `ResponseHeaderTimeout`; SSE body closes via goroutine wrapper to unblock socket read)
|
||||
- Tool execution terminates (shell: process-group kill; browser: page close)
|
||||
- Agent loop receives `ctx.Done()` and propagates cancellation up
|
||||
|
||||
Trace finalization persists independently with a detached context (`context.WithoutCancel`), 5-second timeout, and exponential backoff retry (3 tries, max 10 total via retry queue). This ensures trace status writes don't fail silently due to cancellation.
|
||||
|
||||
Stale recovery is **currently disabled**. The implementation exists but the background loop is not started in `Collector.Start()`. Reason: the sweep condition is `start_time < NOW() - threshold`, which measures trace age rather than inactivity. Any threshold low enough to be useful (2–10 min) would kill healthy long-running agent runs (research chains, large code generation, extended shell commands). Re-enable only after adding a `last_span_at` column so recovery can gate on "no activity for N minutes" instead of "started > N min ago". Until then, zombie traces from gateway crashes may remain `running` in the DB — an accepted safety-net gap traded against false kills. The primary abort path (router 2-phase abort + `trace.status` WS event) handles the common case. Context values (traceID, collector) survive cancellation — only `ctx.Done()` and `ctx.Err()` change.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -189,6 +189,20 @@ To add a new package to the Docker image:
|
||||
|
||||
For packages only needed by specific skills, prefer runtime installation (Option B) to keep the image lean.
|
||||
|
||||
### GitHub Releases Installer
|
||||
|
||||
For CLI tools distributed as GitHub Releases (lazygit, starship, ripgrep, gh, etc.)
|
||||
that aren't packaged via apk/pip/npm, use the `github:` runtime installer:
|
||||
|
||||
```
|
||||
github:owner/repo[@tag]
|
||||
```
|
||||
|
||||
Admin-only, SHA256-verified, ELF-validated, with a release-picker UI. Binaries
|
||||
land in `/app/data/.runtime/bin/` (on `$PATH`). See
|
||||
[`docs/packages-github.md`](./packages-github.md) for syntax, configuration,
|
||||
security posture, and troubleshooting (especially musl/glibc compatibility).
|
||||
|
||||
---
|
||||
|
||||
## 8. Skill Search (v3)
|
||||
|
||||
@@ -4,6 +4,269 @@ All notable changes to GoClaw Gateway are documented here. Format follows [Keep
|
||||
|
||||
---
|
||||
|
||||
### ACTOR vs SCOPE — #915 group permission fix + propagation (2026-04-16)
|
||||
|
||||
Resolves Issue #915 (Telegram group `write_file` permission denied after `/addwriter`) and closes an adjacent silent-privilege-bypass discovered during the audit.
|
||||
|
||||
#### Security (breaking behavior in group/guild context)
|
||||
|
||||
- **`store.CheckFileWriterPermission` / `CheckCronPermission`** no longer fail-open on empty or synthetic `SenderID` when the context scope is a group/guild. Previous fail-open allowed subagent/delegate/team/dashboard/cron system turns to write files in group chats without a writer grant — silent privilege bypass. Post-fix: empty or synthetic-prefix senders (`subagent:`, `notification:`, `teammate:`, `system:`, `ticker:`, `session_send_tool`) are DENIED in group context. DM / HTTP paths unchanged.
|
||||
- **`bus.IsInternalSender`** now also recognises `subagent:` prefix (previously missed, causing subagent senders to be treated as real users in permission checks).
|
||||
|
||||
#### Added — propagation of the acting sender through re-ingress
|
||||
|
||||
- `tools.MetaOriginSenderID` metadata key (`origin_sender_id`) — carries the real acting sender through synthetic-sender announce/dispatch paths.
|
||||
- `tools.AnnounceMetadata.OriginSenderID` field — subagent/delegate announce queue.
|
||||
- `tools.SubagentTask.OriginSenderID` field — populated at spawn from `store.SenderIDFromContext(ctx)`.
|
||||
- `cmd.subagentAnnounceRouting.SenderID` — carries sender from handler into the re-ingress `RunRequest`.
|
||||
- `tools.DelegateRequest.SenderID` field — same propagation for delegate announcements.
|
||||
- `store.ActorIDFromContext(ctx)` helper — returns `SenderID` if set, else `UserID`. Clarifies ACTOR vs SCOPE at call sites.
|
||||
|
||||
#### Changed — ACTOR migration (call sites now use `ActorIDFromContext`)
|
||||
|
||||
- `internal/tools/publish_skill.go`: skill owner = actor (individual, not group principal).
|
||||
- `internal/tools/skill_manage.go` (create, patch, delete): owner = actor on create; patch/delete ownership check accepts actor or legacy `UserIDFromContext` for backward compatibility with skills created before this change.
|
||||
- `internal/tools/delegate_tool.go`: `DelegateRequest.UserID` and `DomainEvent.UserID` = actor for audit trails.
|
||||
- `internal/tools/team_tasks_blocker.go`: blocker attribution and escalation `UserID` = actor.
|
||||
- `internal/tools/team_tool_cache.go`: team access-policy check uses actor (fixes group-chat member access where per-user allow/deny lists previously never matched the group principal).
|
||||
- `internal/tools/team_tool_dispatch.go`: teammate-dispatch metadata now carries `MetaOriginSenderID` so the teammate's turn has the original user's sender.
|
||||
- `internal/tools/sessions_send.go`: session_send metadata now carries `MetaOriginSenderID` to preserve user identity across agent-to-agent flows.
|
||||
|
||||
#### Scope-intentional (unchanged, commented)
|
||||
|
||||
- `internal/tools/cron.go`: cron jobs remain per-group-scope (memory-model parity; migrating would break collaborative `/cron list/add/remove`).
|
||||
- `internal/tools/team_tasks_create.go`: team task `UserID` remains per-group (team visibility is per-chat, not per-user).
|
||||
|
||||
#### Tests
|
||||
|
||||
- New `tests/integration/telegram_group_write_file_permission_test.go` — 9 sub-tests covering granted-sender, ungranted-sender, no-agent fail-open branch, DM no-op, `|`-delimited sender, empty-sender deny, synthetic-prefix deny (7 sub-cases), propagated-sender allow, DM empty-sender passes.
|
||||
- Regression coverage for both the user-reported denial (#915 BUG-B) and the silent-bypass (#915 BUG-A).
|
||||
|
||||
#### Notes
|
||||
|
||||
- No DB migration shipped: historical `skills.owner_id` rows that were created with a `group:*` / `guild:*` value remain accessible via the patch/delete legacy fallback. Re-publishing a skill transfers ownership to the individual actor.
|
||||
- Audit report: `plans/reports/audit-260416-1240-actor-scope-comprehensive.md`.
|
||||
|
||||
---
|
||||
|
||||
## [Unreleased] — 2026-04-15
|
||||
|
||||
#### Agent Hooks System — Phase 3: Prompt Handler + Web UI (2026-04-15)
|
||||
|
||||
Third handler type (`prompt`) with full cost safeguards, WS RPC surface, complete Web UI, 3-language i18n, and production wiring. Closes Issue #875 feature parity with Claude-Code-style hooks.
|
||||
|
||||
### Added
|
||||
- **`internal/hooks/handlers/prompt.go`**: LLM hook handler with structured tool-call output (`decide` tool), anti-injection system preamble, fenced user-input delimiter, in-memory decision cache (60s TTL keyed by `sha256(hook_id||version||tool_name||tool_input)`), per-turn invocation cap (default 5), provider resolver abstraction
|
||||
- **`internal/hooks/handlers/prompt_resolver.go`**: `RegistryResolver` maps model aliases (haiku/sonnet/opus, gpt-*, gemini-*, qwen-*) to tenant providers; falls back to system configs (`hooks.prompt.provider`, `background.provider`) then first registered
|
||||
- **`internal/hooks/budget/` package**: `Store` + `Dialect` interface for atomic per-tenant monthly token budget deduction; `ShouldWarn` helper at 20% threshold
|
||||
- **Budget store implementations**: `pg.PGHookBudget` (single UPDATE + RETURNING with UPSERT seed on month rollover) and `sqlitestore.SqliteHookBudget` (tx-wrapped equivalent)
|
||||
- **WS RPC methods** (`internal/gateway/methods/hooks.go`): `hooks.list` (viewer+), `hooks.create`/`update`/`delete`/`toggle` (admin+, master-scope for global), `hooks.test` (operator+, dryRun no audit write), `hooks.history` (viewer+, stub pending phase 4 pagination)
|
||||
- **Protocol constants**: `MethodHooksList/Create/Update/Delete/Toggle/Test/History` in `pkg/protocol/methods.go`
|
||||
- **Web UI `/hooks` page**: single route with `useParams().id` — list view when no id, detail view when present (CLAUDE.md "route params as source of truth"). Tabs: Overview | Config | Test | History
|
||||
- **Web UI components**: `hook-list-row`, `hook-form-dialog` (3 sub-forms per handler type; command radio disabled on Standard with tooltip), `hook-test-panel` (fires `dryRun=true`, decision badge + duration + stdout/stderr/status), `hook-diff-viewer` (side-by-side JSON diff via highlight.js), `hook-history-table`, `hook-overview-tab`
|
||||
- **Zod + TanStack Query**: `schemas/hooks.schema.ts` (prompt requires matcher|if_expr + prompt_template), `hooks/use-hooks.ts` (useHooksList/Hook/CreateHook/UpdateHook/DeleteHook/ToggleHook/TestHook/HookHistory)
|
||||
- **i18n backend**: 6 new keys (`hook.invalid_matcher`, `hook.command_disabled_standard`, `hook.prompt_requires_matcher`, `hook.circuit_breaker_tripped`, `hook.budget_exceeded`, `hook.per_turn_cap_reached`) in en/vi/zh catalogs
|
||||
- **i18n UI**: `ui/web/src/i18n/locales/{en,vi,zh}/hooks.json` with identical key ordering (per CLAUDE.md memory rule)
|
||||
- **Production wiring**: `cmd/gateway_managed.go` registers `HandlerPrompt` in dispatcher handlers map; `cmd/gateway.go` registers `HookMethods` in router
|
||||
- **Sidebar nav**: "Hooks" entry with `Webhook` icon in Capabilities group
|
||||
|
||||
### Security / cost safeguards
|
||||
- **Structured output enforcement**: evaluator MUST call the `decide` tool; free-text responses fail-closed to block
|
||||
- **Sanitized input**: only `tool_input` reaches the evaluator inside a fenced `USER INPUT` block; raw user message never included
|
||||
- **Injection detection flag**: evaluator can signal `injection_detected=true` via structured output
|
||||
- **Version-busted cache**: hook config edits bump `version` → cache key changes → forces re-evaluation
|
||||
- **Atomic budget deduct**: no select-then-update race (L2 mitigation)
|
||||
|
||||
### Testing
|
||||
- **Unit**: `prompt_test.go` (14 tests — cache hit/miss, version bust, per-turn cap, fail-closed paths, schema assertions), `prompt_injection_test.go` (4 tests — hostile inputs, unicode homoglyphs, nested JSON, fenced delimiter integrity), `budget_test.go` (9 tests — atomic deduct, exceed-returns-err, month rollover, zero-cost, nil tenant, warn threshold)
|
||||
- **WS RPC unit**: `gateway/methods/hooks_test.go` (param parsing + HookTestResult struct round-trip)
|
||||
- **UI**: 56/56 vitest passing including 20 new hook-specific tests (schema validation, list rendering, test panel, diff viewer, i18n contracts)
|
||||
|
||||
#### Agent Hooks System — Phase 4: Hardening + Docs (2026-04-15)
|
||||
|
||||
Integration / chaos / RBAC / tracing coverage + user documentation + copy-paste example hooks library. Merge gate.
|
||||
|
||||
### Added
|
||||
- **Integration tests** (`tests/integration/hooks_e2e_test.go`): 7 events × HTTP/command handler coverage, tenant isolation (cross-tenant events don't hit other tenants' hooks), context-update injection path, edition gate at dispatch time
|
||||
- **Chaos tests** (`hooks_chaos_test.go`): provider down (fail-closed block on blocking event), per-hook timeout respected, circuit breaker auto-disables after 3 consecutive blocks and persists `enabled=false`, `ErrLoopDepthExceeded` when depth > `MaxLoopDepth` (M5), `dedup_key` unique index suppresses double audit rows on retry (H6), 5xx retry-once-then-error
|
||||
- **RBAC tests** (`hooks_rbac_test.go`): store tenant isolation (List/Get/Update/Delete all respect tenant_id), global-scope visible to all tenants, ResolveForEvent unions tenant + global hooks, `HasMinRole` matrix over the hooks.* method surface
|
||||
- **Tracing tests** (`hooks_tracing_test.go`): `EmitHookSpan` writes row with canonical name `hook.<handler_type>.<event>`; dispatcher + traced-handler wrapper emits spans end-to-end
|
||||
- **User docs** `docs/agent-hooks.md`: concepts, security model, handler reference, matcher guide, safeguard table, Web UI walkthrough, observability (audit table + tracing spans + slog keys), troubleshooting, migration notes, known limitations
|
||||
- **Example library** `examples/hooks/`: `block-rm-rf.json` (command, Lite), `auto-lint-after-write.json` (http async), `audit-tool-usage.json` (tenant-wide audit), `session-context-injector.json` (context bootstrap), `notify-discord-on-stop.json` (webhook notification), `README.md` with safety guidance
|
||||
|
||||
### Fixed
|
||||
- **`PGHookStore.GetByID` + `SqliteHookStore.GetByID`** now enforce tenant scope: non-master callers only see own + global rows. Previously returned any row by UUID which leaked cross-tenant.
|
||||
- **Existing pipeline integration tests** now call `security.SetAllowLoopbackForTest(true)` via shared helper; no longer flake on SSRF block for httptest endpoints.
|
||||
- **Vault**: Shared docs (`scope='shared'`, `agent_id=NULL`) are now visible to agents via `vault_search`, `ListDocuments`, `CountDocuments`. Previously excluded by strict `agent_id = $N` predicate after migration 000046 made `agent_id` nullable. Also fixes single-team filter to include shared docs. Adds CHECK invariant (migration 000055) to prevent future scope/ownership drift. Affects PostgreSQL and SQLite (desktop edition). (#917)
|
||||
|
||||
### Testing
|
||||
- All hook integration tests pass under `TEST_DATABASE_URL` PG container (7 new test files, 18 new top-level tests)
|
||||
- `go build` + `go build -tags sqliteonly` + `go vet` clean
|
||||
- `pnpm test` 56/56 pass
|
||||
- Race detector clean for `internal/hooks/...` + `internal/gateway/methods/`
|
||||
|
||||
### Deferred (post-MVP)
|
||||
- Cluster-wide prompt decision cache via Redis
|
||||
- Skill-frontmatter hooks
|
||||
- `agent` handler type (inter-agent delegation via prompt instead)
|
||||
- Desktop Wails UI parity for hooks page
|
||||
- Load/throughput benchmarks (explicitly dropped from Phase 4 scope)
|
||||
|
||||
---
|
||||
|
||||
#### Agent Hooks System — Phase 1: Foundation (2026-04-15)
|
||||
|
||||
Lifecycle hook infrastructure: event dispatcher (sync/async paths), audit logging, database schema (PostgreSQL + SQLite), store interface, config validation, edition gating. Handlers + pipeline integration deferred to Phase 2.
|
||||
|
||||
### Added
|
||||
- **`internal/hooks/` package**: Foundational types (HookConfig, HookExecution, Event), StdDispatcher (fire-and-decide with sequential execution), CEL+regex matcher, audit writer (PII-redacted, encrypted), circuit breaker with auto-disable
|
||||
- **PostgreSQL migration 000052**: `agent_hooks` table (id, tenant_id, agent_id, event, handler_type, config, matcher, if_expr, priority, enabled, created_at, updated_at) + `hook_executions` table (hook_id, session_id, event, input_hash, decision, duration_ms, error, created_at) with unique dedup_key index
|
||||
- **SQLite schema entries**: Full schema in `schema.sql` + migration map bump for Lite/desktop edition
|
||||
- **HookStore interface + implementations**: `PGHookStore` (database/sql + pgx/v5) and `SQLiteHookStore` (modernc.org/sqlite) with List, Create, Update, Delete, LogExecution methods
|
||||
- **Security primitives**: Loop-depth guard (M5 mitigation, MaxLoopDepth=3), per-hook timeout, chain budget (10s wall-time), circuit breaker with auto-disable (C4 mitigation), fail-closed policy for blocking events
|
||||
- **Edition gating**: `HookEditionPolicy` allows command handler on Lite only; master-scope blocks command on Standard tenant
|
||||
- **Config validation**: CEL cost limits, required matcher for prompt handlers, bounded queue per-turn limits
|
||||
|
||||
### Changed
|
||||
- **Store layer**: Added `store.Hooks` field (typed as `any` to avoid import cycles)
|
||||
- **Base store**: `TablesWithUpdatedAt["agent_hooks"] = true` for touch-all update helpers
|
||||
|
||||
### Testing
|
||||
- **Integration test suite**: `tests/integration/v3_hooks_store_test.go` covers multi-tenant isolation, dedup key uniqueness, schema migration up/down for both PG + SQLite
|
||||
|
||||
### Deferred to Phase 3+
|
||||
- Prompt handler (modifies input/context dynamically)
|
||||
- Web UI (CRUD, test panel, history)
|
||||
- Tracing integration
|
||||
- i18n keys (en/vi/zh)
|
||||
|
||||
---
|
||||
|
||||
#### Agent Hooks System — Phase 2: Handlers + Pipeline Integration (2026-04-15)
|
||||
|
||||
Concrete handler implementations + pipeline wiring. Command handler (shell, Lite-only), HTTP handler (SSRF-hardened), synced at ContextStage/ToolStage/FinalizeStage, async at SessionStart/PostToolUse/Stop, delegate bridging (SubagentStart/SubagentStop).
|
||||
|
||||
### Added
|
||||
- **`internal/hooks/handlers/` package**: `CommandHandler` (exec wrapper, JSON I/O, env allowlist, edition policy recheck), `HTTPHandler` (SSRF-hardened HTTP client, retry-once on 5xx, 4xx error no-retry, response parsing for decision/context/updatedInput)
|
||||
- **Pipeline dispatcher integration**: `PipelineDeps.HookDispatcher` field (nil-safe noop fallback); stages call `deps.Hooks.Fire(ctx, evt)` at: ContextStage (SessionStart async + UserPromptSubmit sync), ToolStage (PreToolUse sync before exec + PostToolUse async), FinalizeStage (Stop async)
|
||||
- **Copy-on-write staging**: UserPromptSubmit + PreToolUse updatedInput buffered per-call; commit only when ALL sync hooks succeed; rejection/timeout discards buffer (H2)
|
||||
- **Delegate event bridge**: `delegate_bridge.go` subscribes DelegateCompleted/DelegateFailed events from domain bus, fires SubagentStop async; SubagentStart sync fired directly in delegate_tool pre-call
|
||||
- **Command handler edition gating**: Re-checks `HookEditionPolicy` at dispatch (defense-in-depth); Lite-only with master-scope block on Standard tenant
|
||||
- **HTTP handler SSRF hardening**: Uses caller-supplied net.Dialer that pins resolved IP, blocks loopback/link-local/private ranges; auth headers (e.g. Authorization) encrypted at rest in cfg.Config["headers"], decrypted only at send-time; no redirects allowed (CheckRedirect returns ErrUseLastResponse)
|
||||
- **Server wiring**: `cmd/gateway_managed.go` constructs `hooks.NewStdDispatcher` with both handlers registered; passes dispatcher through `agent.ResolverDeps.HookDispatcher` and `delegateTool.SetHookDispatcher(...)`; falls back to `hooks.NewNoopDispatcher()` when `stores.Hooks` is nil
|
||||
- **Encryption key handling**: Reads `GOCLAW_ENCRYPTION_KEY` env for HTTP handler header decryption; optional (empty → no decryption)
|
||||
|
||||
### Testing
|
||||
- **Unit tests**: `internal/hooks/handlers/command_test.go` (8 tests — exit codes, timeout, env allowlist, edition gate, ctx cancel), `http_test.go` (9 tests — response parsing, retry logic, SSRF blocking pre+post DNS, no redirects, auth header security)
|
||||
- **Integration tests**: `tests/integration/hooks_pipeline_test.go` (4 tests — SessionStart async, UserPromptSubmit block + context + updatedInput COW, PreToolUse block + args mutation, PostToolUse async, zero-overhead no-hooks path); `tests/integration/hooks_delegate_test.go` — SubagentStart/Stop with loop-depth guard M5
|
||||
|
||||
### Changed
|
||||
- **Pipeline architecture**: 8-stage loop now fires hooks via `PipelineDeps.Dispatcher`; stage callbacks support nil-safe dispatcher (noop when not configured)
|
||||
- **Delegate tool**: `delegateTool.SetHookDispatcher(...)` enables async SubagentStop via existing domain events
|
||||
|
||||
---
|
||||
|
||||
#### Desktop Voice Management + STT Admin (2026-04-15)
|
||||
|
||||
### Added
|
||||
- **Desktop voice picker**: Agent detail panel now includes a voice selector with live preview. Users can browse ElevenLabs voices, hear a sample, and persist the choice per agent on the desktop app.
|
||||
- **Desktop STT admin form**: Speech-to-text provider configuration (API keys, providers, WhatsApp opt-in toggle + privacy banner) available in desktop tool settings.
|
||||
- **Singleton voice preview**: Only one voice sample plays at a time; preview button hides automatically when a voice has no sample URL.
|
||||
- **Desktop i18n `tts` namespace**: Voice-management strings translated to English, Vietnamese, and Chinese; STT form labels added to the tools namespace.
|
||||
- **Desktop test harness**: Vitest + Testing Library infrastructure; voice picker and STT form covered by unit tests.
|
||||
|
||||
### Changed
|
||||
- **Agent voice persistence (desktop)**: Voice selection merges into `other_config` without overwriting sibling fields.
|
||||
- **Tool settings routing (desktop)**: Speech-to-text tool opens the dedicated admin form instead of the generic JSON editor.
|
||||
|
||||
---
|
||||
|
||||
#### ElevenLabs Audio Manager Refactor — Phase 5 (2026-04-15)
|
||||
|
||||
Channel STT migration (Telegram/Feishu/Discord) + WhatsApp voice transcription with tenant opt-in (default disabled due to E2E encryption trade-off, resolves privacy red-team finding).
|
||||
|
||||
### Added
|
||||
- **WhatsApp voice transcription**: `internal/channels/whatsapp/stt.go` — opt-in per-tenant via `builtin_tools[stt].settings.whatsapp_enabled` (default `false` per Decision 6)
|
||||
- **Synchronous 10s timeout**: All 4 channels (Telegram/Feishu/Discord/WhatsApp) enforce ctx timeout for voice message processing; timeout/error → fallback label
|
||||
- **i18n fallback key**: `MsgVoiceMessageFallback` ("voice message" placeholder in en/vi/zh) rendered when STT unavailable or disabled
|
||||
- **Privacy banner**: Admin UI displays E2E encryption trade-off warning when enabling WhatsApp STT
|
||||
|
||||
### Changed
|
||||
- **Channel STT handlers**: All 4 channels now call `audio.Manager.Transcribe()` with 10s timeout instead of per-channel `transcribeAudio()` helpers
|
||||
- **Channel factory signatures**: All 4 factories (`NewChannel`) accept `audioMgr *audio.Manager` parameter
|
||||
- **WhatsApp messaging**: Voice messages now transcribed (opt-in) and appended to message text for agent understanding
|
||||
|
||||
### Removed
|
||||
- **Per-channel STT helpers**: Deleted `internal/channels/{telegram,feishu,discord}/stt.go` (consolidated into Manager)
|
||||
- **Telegram STT tests**: Deleted `internal/channels/telegram/stt_test.go` — coverage preserved in `internal/audio/proxy_stt/provider_test.go` (12 ported scenarios, deliberate deletion documented)
|
||||
|
||||
### Resolved
|
||||
- **Audit finding P5-B1**: Gate G4 verified; Manager.Transcribe chain wired
|
||||
- **Audit finding P5-H1**: All transcribeAudio call sites migrated; grep confirms zero direct channel STT calls
|
||||
- **Audit finding P5-H2** + **Decision 6** (privacy): WhatsApp STT opt-in with default=false; privacy banner + docs note E2E trade-off
|
||||
- **Cross-phase XP-3**: Telegram test ported/deleted with 12-scenario coverage guarantee
|
||||
- **Cross-phase XP-4**: `MsgVoiceMessageFallback` pre-wired in Phase 5 step 1 (blocking)
|
||||
|
||||
### Security / Privacy
|
||||
- **WhatsApp E2E encryption**: Voice transcription breaks end-to-end guarantee. **Tenant opt-in required** with explicit admin acknowledgment (privacy banner in UI)
|
||||
- **Temp file cleanup**: Existing `scheduleMediaCleanup` preserves file handles during 10s STT window; no race condition risk
|
||||
- **Audit logging**: WhatsApp STT invocation logged at info level for audit trail
|
||||
|
||||
---
|
||||
|
||||
#### ElevenLabs Audio Manager Refactor — Phase 4 (2026-04-15)
|
||||
|
||||
STT via ElevenLabs Scribe + proxy fallback with tenant config migration + admin UI form. Legacy per-channel STT config bridged with 2-week deprecation grace.
|
||||
|
||||
### Added
|
||||
- **ElevenLabs Scribe STT provider**: `internal/audio/elevenlabs/stt.go` — POST `/v1/speech-to-text`, multipart upload, 20MB cap, configurable language + diarization
|
||||
- **Proxy STT wrapper**: `internal/audio/proxy_stt/provider.go` — backward-compat bridge for legacy `media.TranscribeAudio` with automatic temp-file handling
|
||||
- **STT admin form**: `ui/web/src/pages/builtin-tools/stt-provider-form.tsx` — mirrors TTS form pattern, includes tenant config (API keys, providers list, WhatsApp opt-in toggle)
|
||||
- **`audio.Manager.Transcribe()`**: Chain-based STT dispatch with channel-override precedence (per-channel STTProxyURL wins over tenant builtin_tools[stt])
|
||||
- **i18n keys**: `MsgSTTAllProvidersFailed`, `MsgSTTLegacyConfigDeprecated`, `MsgSTTWhatsappPrivacyWarning` across en/vi/zh
|
||||
|
||||
### Changed
|
||||
- **STT config location**: Migrated from per-channel fields (Telegram/Feishu/Discord `STTProxyURL`) to tenant `builtin_tools[stt].settings` (Decision 1 per audit)
|
||||
- **`builtin_tools[stt]` table entry**: New seed row via migration 000050 (PG) + schema.go incremental (SQLite)
|
||||
- **Legacy bridge**: Startup-time scan of per-channel STT configs → auto-registers `proxy_stt.Provider` with deprecation warn when tenant lacks `builtin_tools[stt]` (2-week grace period)
|
||||
|
||||
### Resolved
|
||||
- **Audit finding P4-H1** (channel-override precedence): Decision 2 — per-channel config takes priority; tenant fallback when channel config absent
|
||||
- **Audit finding P4-H2** (Scribe endpoint verification): ElevenLabs Scribe v2 `/v1/speech-to-text` confirmed with xi-api-key auth
|
||||
- **Audit finding P4-B3** (explicit bridge loop): 3-channel explicit loop (Telegram, Feishu, Discord) replaces ambiguous iteration
|
||||
- **Cross-phase XP-3** (12 test ported): All 12 Telegram STT scenarios (NoProxy, EmptyFile, Success, Error, etc.) ported into `proxy_stt/provider_test.go` before Phase 5 channel migration
|
||||
|
||||
### Deprecated
|
||||
- Per-channel `STTProxyURL`, `STTAPIKey`, `STTTenantID` fields (Telegram, Feishu, Discord configs) — soft deprecation with 2-week grace; migration to `builtin_tools[stt]` required for Phase 5+ channels
|
||||
|
||||
### Fixed
|
||||
- **JSON schema divergence**: `duration_secs` → `audio_duration_secs` in Scribe response parsing (matches ElevenLabs API)
|
||||
- **Provider registration**: Scribe + SetSTTChain wired into setupAudioExtras for Phase 5 channel integration
|
||||
|
||||
---
|
||||
|
||||
#### ElevenLabs Audio Manager Refactor — Phase 3 (2026-04-14)
|
||||
|
||||
Music generation via ElevenLabs + MiniMax with fallback chain. Suno provider fully removed from codebase (no official API, ToS violation risk).
|
||||
|
||||
### Added
|
||||
- **ElevenLabs Music provider**: `internal/audio/elevenlabs/music.go` — POST `/v1/music`, model `music_v1`, binary MP3 response, 300s timeout, prompt + optional lyrics
|
||||
- **MiniMax Music provider**: `internal/audio/minimax/music.go` — POST `{base}/music_generation`, 2-step URL-download pattern, 200 MB cap, optional instrumentation toggle
|
||||
|
||||
### Changed
|
||||
- **`create_audio` tool simplified**: Now thin Manager dispatcher (`NewCreateAudioTool(audioMgr *audio.Manager)`), removed inline provider logic + per-provider switch cases
|
||||
- **`audio.Manager.GenerateMusic/GenerateSFX`**: Chain-based resolution (elevenlabs → minimax for music, elevenlabs-only for SFX)
|
||||
- **`createAudio` builtin tool**: Unified dispatch via Manager instead of `providers.Registry` dependency injection
|
||||
|
||||
### Removed
|
||||
- **Suno provider**: Fully excised (10 atomic locations) — research confirms no official API + ToS violations. Files deleted: `create_audio_suno.go`, `create_audio_minimax.go`, `create_audio_elevenlabs.go` shim. HTTP route removed. TS schema entry removed. Config provider removed.
|
||||
|
||||
### Types
|
||||
- **`audio.MusicOptions`**: Added `Instrumental bool`, `Model string`, `TimeoutSec int` fields
|
||||
- **`audio.AudioResult`**: Added `Provider string` field for observability (tool span metadata)
|
||||
|
||||
---
|
||||
|
||||
## [v2.66.0] — 2026-04-05
|
||||
|
||||
### Security
|
||||
@@ -32,6 +295,72 @@ All notable changes to GoClaw Gateway are documented here. Format follows [Keep
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
#### ElevenLabs Audio Manager Refactor — Phase 1 (2026-04-14)
|
||||
|
||||
Unified audio provider management via new `internal/audio/` package with pluggable interface-based architecture. Phase 1 wires TTS providers (ElevenLabs, OpenAI, Edge, MiniMax); STT/Music/SFX interfaces defined for Phase 3-4.
|
||||
|
||||
**What changed:**
|
||||
- **`internal/audio/` package**: Central `Manager` orchestrates 4 provider kinds via interfaces (`TTSProvider`, `STTProvider`, `MusicProvider`, `SFXProvider`)
|
||||
- **Provider organization**: Implementations in `internal/audio/{elevenlabs,openai,edge,minimax}/`. ElevenLabs shared HTTP client (`xi-api-key` header) for both TTS and SFX subproviders
|
||||
- **`internal/tts/` → backward-compat alias**: 24-symbol package (15 types + 6 consts + 5 constructors + 5 signature guards). All pre-refactor callers compile unchanged, zero breaking changes
|
||||
- **Config extension**: `config.Audio` optional pointer (nil-safe) added for future STT/Music subsections. `cfg.Tts` retained unchanged
|
||||
- **ElevenLabs SFX tool**: `internal/tools/create_audio_elevenlabs.go` rewritten as thin shim calling `elevenlabs.NewSFXProvider(...).GenerateSFX(ctx, audio.SFXOptions{...})`
|
||||
- **WS `tts.*` namespace**: 6 methods unchanged externally
|
||||
|
||||
**Impact**: Existing TTS flows fully compatible. New code can import `internal/audio` directly. STT/Music/SFX wiring deferred to Phase 3-4.
|
||||
|
||||
#### ElevenLabs Audio Manager Enhancements — Phase 2 (2026-04-14)
|
||||
|
||||
Voice discovery and agent-level audio config via new backend endpoints, in-memory cache, and web UI picker. Bundles producer/consumer context pattern (`store.WithAgentAudio`) for seamless voice/model resolution throughout the tool execution pipeline.
|
||||
|
||||
**What changed:**
|
||||
- **Voice cache** (`internal/audio/voice_cache.go`): In-memory LRU (cap 1000 tenants) with TTL 1h, shared between HTTP + WS handlers, thread-safe under concurrent access
|
||||
- **Streaming TTS interface** (`internal/audio/types.go`): New `StreamingTTSProvider` optional interface for ElevenLabs `/v1/text-to-speech/{voice_id}/stream` chunked playback
|
||||
- **ElevenLabs enhancements**: Model allowlist (11_v3, eleven_flash_v2_5, eleven_multilingual_v2, eleven_turbo_v2_5), `SynthesizeStream()` method, `ListVoices()` via `/v1/voices`
|
||||
- **Agent audio context** (`internal/store/context.go`): New `WithAgentAudio` / `AgentAudioFromCtx` bundle; producer wires snapshot at dispatcher level (internal/agent/), consumer (`TtsTool.Execute`) reads voice/model overrides from agent config
|
||||
- **Agent config extension** (`agents.other_config` JSONB): New `tts_voice_id` and `tts_model_id` fields with resolution precedence: args → agent → tenant → provider default
|
||||
- **HTTP + WS endpoints**: GET /v1/voices (cached), POST /v1/voices/refresh (admin-only), WS method `voices.list` + `voices.refresh`
|
||||
- **Web voice picker** (`ui/web/src/components/voice-picker.tsx`): Combobox with search, preview button (HTML audio + onError → refresh), embedded in PromptSettingsSection
|
||||
- **i18n**: 10 new frontend keys (voice_label, voice_placeholder, voice_refresh, voice_preview, model_label, etc.) + 2 new backend keys (MsgTtsUnknownModel, MsgVoicesListFailed) across en/vi/zh
|
||||
|
||||
**Impact**: Existing TTS callers fully compatible (backward-compat via Phase 1 alias layer). Web UI gains voice discovery + per-agent voice/model overrides. Zero breaking changes. Integration test validates producer+consumer context flow.
|
||||
|
||||
#### Trace Stop/Abort Redesign — Cascading 4-Layer Fix (2026-04-14)
|
||||
|
||||
The Stop button on the traces page now reliably aborts running traces. Previous implementation had independent race conditions across HTTP streaming, agent router, trace persistence, and UI polling; this redesign fixes all four layers atomically.
|
||||
|
||||
**What changed:**
|
||||
- **HTTP streaming ctx-aware**: Provider clients use transport-level `ResponseHeaderTimeout` + `IdleConnTimeout` instead of socket-level `Client.Timeout`. SSE body closes immediately on ctx cancel via goroutine-based wrapper (prevents 5-minute socket block).
|
||||
- **Router 2-phase abort**: `AbortRun()` atomically transitions to aborting state, waits 3s for goroutine exit via `Done` channel, then force-marks trace `cancelled` if timeout. No orphaned goroutines, no "not found" race.
|
||||
- **Trace status persistence**: `SetTraceStatus()` detached from request context with 5s timeout, 3-try exponential backoff, and bounded in-memory retry queue (10 max tries). Stale recovery worker runs every 30s, catches zombie traces in 2 minutes instead of 30.
|
||||
- **Real-time UI updates**: New WS event `trace.status` broadcasts status changes immediately after persist succeeds. UI drops 60s polling interval, subscribes to events instead.
|
||||
- **Tool execution audit**: Shell commands use process-group kill (`SIGTERM`→3s→`SIGKILL`). Browser automation (Rod) closes pages on ctx cancel. MCP delegates timeout after 5s.
|
||||
- **i18n**: 6 abort toast variants (success/timeout/not-found/already-done/db-error/unknown) + translations for en/vi/zh.
|
||||
|
||||
**Impact**: Existing traces and sessions unaffected. UI now reflects backend state accurately. Zero breaking changes.
|
||||
|
||||
#### Preserve User-Provided Filenames for Media Uploads (2026-04-14)
|
||||
|
||||
Filenames provided by users for chat media uploads now survive the channel adapter → agent → disk persistence round trip, enabling vault enrichment to process human-readable document names instead of falling back to generic UUID-only storage.
|
||||
|
||||
**Why**: Vault enrichment was skipping UUID-only disk names (design safety to avoid noisy auto-generated files), causing documents with Vietnamese or CJK stems to remain unenriched and lose semantic context.
|
||||
|
||||
**What changed**:
|
||||
- **`bus.MediaFile.Filename` field**: Channel adapters now populate this field when source provides original filename (e.g., user-selected file upload, Telegram file_name, WhatsApp display_name)
|
||||
- **Sanitizer** (`internal/agent/media_filename.go`): Derives safe stems via `sanitizeFilename()` with:
|
||||
- **Vietnamese pre-NFD map**: `đ/Đ → d` (Unicode NFD does not decompose these precomposed letters)
|
||||
- **CJK passthrough**: Dominant-script heuristic detects Vietnamese/CJK inputs and preserves original runes (no ASCII slugification)
|
||||
- **Filesystem safety**: Removes control chars, path traversal markers (`..`, `/`, `\\`), and reserved names
|
||||
- **Max length**: 60 runes (script-aware, not byte-based) to avoid platform path limits
|
||||
- **Disk naming scheme**: `{sanitized-stem}-{8hex}{ext}` (e.g., `bao-cao-q4-a1b2c3d4.pdf`) when sanitizer returns non-empty stem; UUID fallback `{uuid}{ext}` for empty stems (voice notes, clipboard pastes, tool-generated media)
|
||||
- **Vault enrichment gating** (`internal/vault/enrich_skip_filter.go`): Now skips UUID-only filenames (matching `^[0-9a-f]{8}-...$` pattern) while processing named stems
|
||||
- **6 channel adapters wired**: Telegram, Slack, Discord, Feishu/Lark, Zalo OA, WhatsApp all set `MediaFile.Filename` when available
|
||||
- **Tools + orchestration**: `web_search` tool (PDF downloads), delegate/subagent media propagation, all preserve filenames end-to-end
|
||||
|
||||
**Impact**: Existing flows with empty `Filename` are unaffected (UUID-named as before). New flows with filenames produce human-readable, enrichable disk names. Zero breaking changes.
|
||||
|
||||
### Security
|
||||
|
||||
#### Tenant-Scope Hotfix (2026-04-12)
|
||||
@@ -61,6 +390,13 @@ Tenant admins can override tool configuration without affecting other tenants. O
|
||||
|
||||
### Fixed
|
||||
|
||||
#### Pancake Platform Field — Mandatory Select (2026-04-14)
|
||||
|
||||
- **`platform` field changed from optional text to required dropdown**: `channel-schemas.ts` replaces the free-text `platform` field with a `select` type (11 options: facebook, instagram, threads, tiktok, youtube, shopee, line, google, chat_plugin, lazada, tokopedia). `required: true` is now enforced by the form dialog on create.
|
||||
- **Config required-field validation (create-only)**: `channel-instance-form-dialog.tsx` now enforces `required` on config fields at submit time. Validation is scoped to create flows only — edit flows are unchanged to avoid silent breakage on existing channels with no stored `platform`.
|
||||
- **i18n**: Platform field label options and config field metadata added to `channels.json` for en/vi/zh.
|
||||
- **Auto-detect log level**: `pancake.go` auto-detect block demoted from `slog.Info` to `slog.Debug` to stop log noise on every channel start; auto-detect remains as a fallback for existing channels with no stored `platform`.
|
||||
|
||||
#### Feishu/Lark Writer Management Commands — Issue #818 Closed (2026-04-11)
|
||||
- **Issue #818 resolution**: Closes UX gap where users saw `/addwriter` error messages but Feishu had no handler
|
||||
- **Phase 1 — Thread reply routing**: Inbound messages with `thread_id` now properly route responses back to the same Feishu thread via `/open-apis/im/v1/messages/{id}/reply`. New `feishu_reply_target_id` metadata key included in `routingMetaKeys` allowlist. Graceful fallback to `SendMessage()` if thread root deleted
|
||||
|
||||
+68
-11
@@ -798,7 +798,64 @@ Workspace file management.
|
||||
|
||||
---
|
||||
|
||||
## 22. Media
|
||||
## 22. Voices & Audio
|
||||
|
||||
Voice discovery for TTS providers (ElevenLabs). All endpoints are tenant-scoped and require tenant admin or operator role.
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/v1/voices` | List available voices (in-memory cached, TTL 1h) |
|
||||
| `POST` | `/v1/voices/refresh` | Force refresh voice cache (admin-only) |
|
||||
|
||||
### `GET /v1/voices`
|
||||
|
||||
Fetch available TTS voices for the current tenant's provider.
|
||||
|
||||
**Query Parameters:**
|
||||
- None
|
||||
|
||||
**Response** (200 OK):
|
||||
```json
|
||||
[
|
||||
{
|
||||
"voice_id": "pMsXgVXv3BLzUgSXRplE",
|
||||
"name": "Alice",
|
||||
"preview_url": "https://...",
|
||||
"category": "premade",
|
||||
"labels": {
|
||||
"use_case": "conversational",
|
||||
"accent": "american"
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Errors:**
|
||||
- 401: Missing or invalid token
|
||||
- 403: Insufficient permissions (requires tenant admin/operator)
|
||||
- 500: Provider error (e.g., ElevenLabs API unreachable)
|
||||
|
||||
**Caching:**
|
||||
- Responses are in-memory cached per tenant with TTL 1h
|
||||
- Cache is shared across all HTTP + WebSocket handlers
|
||||
- Cache miss triggers immediate fetch from provider
|
||||
|
||||
### `POST /v1/voices/refresh`
|
||||
|
||||
Invalidate the voice cache for the current tenant, forcing a fresh fetch on the next request.
|
||||
|
||||
**Admin-only endpoint.** Useful after voice updates or CDN expiry issues.
|
||||
|
||||
**Request body:** (empty)
|
||||
|
||||
**Response** (202 Accepted):
|
||||
```json
|
||||
{ "message": "voice cache invalidated" }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 23. Media
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
@@ -807,7 +864,7 @@ Workspace file management.
|
||||
|
||||
---
|
||||
|
||||
## 23. Files
|
||||
## 24. Files
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
@@ -818,7 +875,7 @@ Auth via Bearer token or `?token=` query param (for `<img>` tags). MIME type aut
|
||||
|
||||
---
|
||||
|
||||
## 24. API Keys
|
||||
## 25. API Keys
|
||||
|
||||
Admin-only endpoints for managing gateway API keys. See [20 — API Keys & Auth](20-api-keys-auth.md) for the full authentication and authorization model.
|
||||
|
||||
@@ -856,7 +913,7 @@ Admin-only endpoints for managing gateway API keys. See [20 — API Keys & Auth]
|
||||
|
||||
---
|
||||
|
||||
## 25. OAuth
|
||||
## 26. OAuth
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
@@ -943,7 +1000,7 @@ Notes:
|
||||
|
||||
---
|
||||
|
||||
## 26. Edition
|
||||
## 27. Edition
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
@@ -951,7 +1008,7 @@ Notes:
|
||||
|
||||
---
|
||||
|
||||
## 27. Tenants
|
||||
## 28. Tenants
|
||||
|
||||
Multi-tenant management (admin only).
|
||||
|
||||
@@ -967,7 +1024,7 @@ Multi-tenant management (admin only).
|
||||
|
||||
---
|
||||
|
||||
## 28. System Configs
|
||||
## 29. System Configs
|
||||
|
||||
Key-value system configuration store.
|
||||
|
||||
@@ -980,7 +1037,7 @@ Key-value system configuration store.
|
||||
|
||||
---
|
||||
|
||||
## 29. Team Workspace & Attachments
|
||||
## 30. Team Workspace & Attachments
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
@@ -990,7 +1047,7 @@ Key-value system configuration store.
|
||||
|
||||
---
|
||||
|
||||
## 30. Shell Deny Groups
|
||||
## 31. Shell Deny Groups
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
@@ -998,7 +1055,7 @@ Key-value system configuration store.
|
||||
|
||||
---
|
||||
|
||||
## 31. System
|
||||
## 32. System
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
@@ -1017,7 +1074,7 @@ Key-value system configuration store.
|
||||
|
||||
---
|
||||
|
||||
## 32. MCP Bridge
|
||||
## 33. MCP Bridge
|
||||
|
||||
Exposes GoClaw tools to Claude CLI via streamable HTTP at `/mcp/bridge`. Only listens on localhost. Protected by gateway token with HMAC-signed context headers.
|
||||
|
||||
|
||||
@@ -501,6 +501,68 @@ Multi-tenant management (admin only).
|
||||
|
||||
---
|
||||
|
||||
## 17.1. Voices (Voice Discovery)
|
||||
|
||||
Discover available TTS voices for the tenant's configured provider.
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `voices.list` | Fetch available voices (in-memory cached, TTL 1h) |
|
||||
| `voices.refresh` | Force cache invalidation (admin-only) |
|
||||
|
||||
### `voices.list` Request
|
||||
|
||||
```json
|
||||
{
|
||||
"method": "voices.list",
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
**Response** (200 OK):
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"result": [
|
||||
{
|
||||
"voice_id": "pMsXgVXv3BLzUgSXRplE",
|
||||
"name": "Alice",
|
||||
"preview_url": "https://...",
|
||||
"category": "premade",
|
||||
"labels": {
|
||||
"use_case": "conversational",
|
||||
"accent": "american"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Errors:**
|
||||
- `code: -1`: Provider error (e.g., ElevenLabs API unreachable)
|
||||
- `code: -2`: Cache miss + no provider context available (desktop edition in Phase 2; HTTP handler resolves provider dynamically)
|
||||
|
||||
### `voices.refresh` Request
|
||||
|
||||
Admin-only. Invalidate tenant cache, forcing fresh fetch on next list.
|
||||
|
||||
```json
|
||||
{
|
||||
"method": "voices.refresh",
|
||||
"id": 2
|
||||
}
|
||||
```
|
||||
|
||||
**Response** (200 OK):
|
||||
```json
|
||||
{
|
||||
"id": 2,
|
||||
"result": { "message": "voice cache invalidated" }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 18. Browser Automation
|
||||
|
||||
| Method | Description |
|
||||
@@ -727,6 +789,7 @@ The server pushes events to connected clients via event frames. Key event types:
|
||||
| `chunk` | Streaming text chunk |
|
||||
| `tool.call` | Tool invocation started |
|
||||
| `tool.result` | Tool invocation completed |
|
||||
| `trace.status` | Trace status changed (cancelled, completed, error) |
|
||||
| `session.updated` | Session metadata changed |
|
||||
| `agent.updated` | Agent config changed |
|
||||
| `cron.fired` | Cron job triggered |
|
||||
@@ -737,6 +800,7 @@ The server pushes events to connected clients via event frames. Key event types:
|
||||
|
||||
| Event | Description | Payload |
|
||||
|-------|-------------|---------|
|
||||
| `trace.status` | Trace status changed (real-time stop/abort visibility) | `{traceId, status, endedAt?}` |
|
||||
| `evolution.metrics.updated` | New evolution metrics recorded | `{agentId, metricType, toolName, value}` |
|
||||
| `evolution.suggestion` | New evolution suggestion generated | `{agentId, suggestionId, type, title}` |
|
||||
| `episodic.summary` | New episodic summary created/updated | `{agentId, summaryId, userId}` |
|
||||
@@ -745,6 +809,25 @@ The server pushes events to connected clients via event frames. Key event types:
|
||||
| `orchestration.mode.changed` | Agent orchestration mode changed | `{agentId, newMode}` |
|
||||
| `v3flags.changed` | V3 feature flags updated | `{agentId, flags}` |
|
||||
|
||||
#### `trace.status` Event
|
||||
|
||||
Emitted whenever a trace status changes (e.g., `running` → `cancelled`, `running` → `completed`). Allows UI to update trace state in real-time without polling, particularly critical for stop/abort operations.
|
||||
|
||||
**Payload:**
|
||||
```json
|
||||
{
|
||||
"traceId": "uuid",
|
||||
"status": "cancelled",
|
||||
"endedAt": "2026-04-14T12:34:56.789Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Status values:**
|
||||
- `cancelled` — User stopped the trace via `chat.abort`
|
||||
- `completed` — Trace finished successfully
|
||||
- `error` — Trace failed with an error
|
||||
- `running` — Emitted when trace transitions from another state (rare; mostly informational)
|
||||
|
||||
---
|
||||
|
||||
## File Reference
|
||||
|
||||
@@ -46,6 +46,27 @@ Documents are scoped by **tenant** (isolation), **agent** (per-agent namespace),
|
||||
- **team**: Team workspace documents (team_context_files)
|
||||
- **shared**: Cross-tenant shared knowledge (future)
|
||||
|
||||
### Document Scope & Ownership
|
||||
|
||||
| scope | agent_id | team_id | Visibility |
|
||||
|-------|----------|---------|------------|
|
||||
| personal | set | NULL | Owning agent only (within tenant) |
|
||||
| team | NULL | set | Members of the team (within tenant) |
|
||||
| shared | NULL | NULL | All agents within the tenant |
|
||||
| custom | any | any | User-defined via `custom_scope` |
|
||||
|
||||
#### DB Invariant (migration 000055)
|
||||
|
||||
A CHECK constraint enforces the above table: `vault_documents_scope_consistency`. Rejects inserts/updates that violate the scope × ownership relationship. `scope='custom'` is unconstrained (user-defined scopes).
|
||||
|
||||
#### Agent Read Semantics
|
||||
|
||||
`vault_search`, `ListDocuments`, `CountDocuments` return:
|
||||
- Docs owned by the agent (`agent_id = <agent>`)
|
||||
- PLUS shared docs (`agent_id IS NULL`)
|
||||
|
||||
Within a team context (RunContext with TeamID set), results also include team-scoped docs for that team. Tenant isolation (`tenant_id = <tenant>`) is always enforced.
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Model
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
# Agent Hooks
|
||||
|
||||
Lifecycle hooks let you intercept, observe, or inject behavior at defined points in the agent loop. Use cases: block unsafe tool calls, auto-lint after writes, inject session context, notify on stop, audit tool usage.
|
||||
|
||||
## Concepts
|
||||
|
||||
### Events
|
||||
Seven lifecycle events fire during an agent session:
|
||||
|
||||
| Event | Blocking | When it fires |
|
||||
|---|---|---|
|
||||
| `session_start` | no | A new session is established. |
|
||||
| `user_prompt_submit` | **yes** | Before the user's message enters the pipeline. |
|
||||
| `pre_tool_use` | **yes** | Before any tool call executes. |
|
||||
| `post_tool_use` | no | After a tool call completes. |
|
||||
| `stop` | no | The agent session terminates normally. |
|
||||
| `subagent_start` | **yes** | A sub-agent is spawned. |
|
||||
| `subagent_stop` | no | A sub-agent finishes. |
|
||||
|
||||
Blocking events wait for the sync chain to return an allow/block decision. Non-blocking events fire hooks asynchronously for observation only.
|
||||
|
||||
### Handler types
|
||||
|
||||
| Handler | Editions | Notes |
|
||||
|---|---|---|
|
||||
| `command` | Lite only | Local shell command; exit 2 → block, exit 0 → allow. |
|
||||
| `http` | Lite + Standard | POST to endpoint; JSON body → decision. SSRF-protected. |
|
||||
| `prompt` | Lite + Standard | LLM-based evaluation with structured tool-call output. Prompt-injection-resistant, budget-bounded. Requires `matcher` or `if_expr`. |
|
||||
|
||||
### Scopes
|
||||
- **global** — applies to all tenants. Master scope required to create.
|
||||
- **tenant** — applies to one tenant (any agent).
|
||||
- **agent** — applies to a specific agent within a tenant.
|
||||
|
||||
Hooks resolve in priority order, highest first. A single `block` decision short-circuits the chain.
|
||||
|
||||
## Security model
|
||||
|
||||
- **Edition gating**: `command` handler blocked on Standard at both config-time AND dispatch-time (defense in depth).
|
||||
- **Tenant isolation**: all reads/writes scope by `tenant_id` unless caller is in master scope. Global hooks use a sentinel tenant id.
|
||||
- **SSRF protection**: HTTP handler validates URLs before request, pins resolved IP, blocks loopback/link-local/private ranges.
|
||||
- **PII redaction**: audit rows truncate error text to 256 chars; full error is encrypted (AES-256-GCM) in `error_detail`.
|
||||
- **Fail-closed**: any unhandled error in a blocking event yields `block`. Timeouts respect `OnTimeout` (default `block` for blocking events).
|
||||
- **Circuit breaker**: N consecutive blocks/timeouts in a rolling window auto-disables the hook and persists `enabled=false`.
|
||||
- **Loop detection**: sub-agent hook chains bounded at depth 3 (`MaxLoopDepth`).
|
||||
|
||||
## Handler reference
|
||||
|
||||
### command
|
||||
|
||||
```json
|
||||
{
|
||||
"handler_type": "command",
|
||||
"config": {
|
||||
"command": "bash /path/to/script.sh",
|
||||
"allowed_env_vars": ["MY_VAR"],
|
||||
"cwd": "/workspace"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- Stdin: JSON-encoded event payload.
|
||||
- Stdout (exit 0): optional `{"continue": false}` → block; anything else → allow.
|
||||
- Exit 2: block.
|
||||
- Other non-zero exits: error (fail-closed for blocking events).
|
||||
- Env allowlist: only listed keys are passed through to prevent secret leakage.
|
||||
|
||||
### http
|
||||
|
||||
```json
|
||||
{
|
||||
"handler_type": "http",
|
||||
"config": {
|
||||
"url": "https://example.com/webhook",
|
||||
"headers": {"Authorization": "<AES-encrypted>"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- Method: POST, body = event JSON.
|
||||
- Authorization header values are stored AES-256-GCM encrypted; decrypted at dispatch.
|
||||
- 1 MiB response cap.
|
||||
- Retries once on 5xx with 1s backoff; 4xx fail-closed (no retry).
|
||||
- Response body:
|
||||
```json
|
||||
{ "decision": "allow" | "block", "additionalContext": "...", "updatedInput": {}, "continue": true }
|
||||
```
|
||||
- Non-JSON 2xx → allow.
|
||||
|
||||
### prompt
|
||||
|
||||
```json
|
||||
{
|
||||
"handler_type": "prompt",
|
||||
"matcher": "^(exec|shell|write_file)$",
|
||||
"config": {
|
||||
"prompt_template": "Evaluate safety of this tool call.",
|
||||
"model": "haiku",
|
||||
"max_invocations_per_turn": 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Required:
|
||||
- `prompt_template` — system-level instruction the evaluator receives.
|
||||
- `matcher` or `if_expr` — runaway-cost guard; prevents firing the LLM on every event.
|
||||
|
||||
Safeguards:
|
||||
- **Structured output**: evaluator MUST call a `decide(decision, reason, injection_detected, updated_input)` tool. Free-text responses fail-closed.
|
||||
- **Sanitized input**: only `tool_input` reaches the evaluator (sandwiched in a fenced `USER INPUT` block with anti-injection preamble); the raw user message is never included.
|
||||
- **Decision cache**: 60s TTL keyed by `sha256(hook_id || version || tool_name || tool_input)`. Hook edits bump version → busts cache.
|
||||
- **Per-turn cap**: default 5 invocations per user turn; configurable.
|
||||
- **Per-tenant monthly budget**: atomic UPDATE deduct; warn at 20% remaining, block at 0.
|
||||
|
||||
## Matchers
|
||||
|
||||
- **matcher** — POSIX-ish regex applied to `tool_name`. Example: `^(exec|shell|write_file)$`.
|
||||
- **if_expr** — [cel-go](https://github.com/google/cel-go) expression evaluated against `{tool_name, tool_input, depth}`. Example: `tool_name == "exec" && size(tool_input.cmd) > 80`.
|
||||
|
||||
Both are optional for `command` / `http`. At least one is required for `prompt`.
|
||||
|
||||
## Safeguards summary
|
||||
|
||||
| Safeguard | Default | Overridable per hook |
|
||||
|---|---|---|
|
||||
| Per-hook timeout | 5s | yes (`timeout_ms`, max 10s) |
|
||||
| Chain budget | 10s | no (dispatcher constant) |
|
||||
| Circuit threshold | 5 blocks in 1 minute | no (dispatcher constant) |
|
||||
| Prompt per-turn cap | 5 | yes (`max_invocations_per_turn`) |
|
||||
| Prompt decision cache TTL | 60s | no |
|
||||
| Tenant monthly token budget | 1,000,000 | seeded per tenant |
|
||||
|
||||
## Web UI walkthrough
|
||||
|
||||
Navigate to **Hooks** in the sidebar. The list view shows all hooks visible under the current role + tenant scope.
|
||||
|
||||
1. **Create** → pick event, handler type (command disabled on Standard), scope, matcher, then fill the handler-specific sub-form.
|
||||
2. **Test** panel → fires the hook with a sample event (`dryRun=true`, no audit row written). Shows decision badge, duration, stdout/stderr (command), status code (http), reason (prompt). If the response includes `updatedInput`, a side-by-side JSON diff is rendered.
|
||||
3. **History** tab → paginated executions from `hook_executions` (full implementation lands with operational pagination; current release shows the note state).
|
||||
4. **Overview** tab → summary card with event, type, scope, matcher.
|
||||
|
||||
## Observability
|
||||
|
||||
### Audit table (`hook_executions`)
|
||||
|
||||
| Column | Notes |
|
||||
|---|---|
|
||||
| `hook_id` | `ON DELETE SET NULL` → executions preserved after hook deletion. |
|
||||
| `dedup_key` | Unique index prevents double rows on retry (H6). |
|
||||
| `error` | Truncated to 256 chars; full detail in `error_detail` (encrypted). |
|
||||
| `metadata` | JSONB: `matcher_matched`, `cel_eval_result`, `stdout_len`, `http_status`, `prompt_model`, `prompt_tokens`, `trace_id`. |
|
||||
|
||||
### Tracing spans
|
||||
|
||||
Every hook execution emits a span named `hook.<handler_type>.<event>` (e.g. `hook.prompt.pre_tool_use`) with:
|
||||
|
||||
- `status` — `completed` or `error`.
|
||||
- `duration_ms`.
|
||||
- `metadata.decision` — `allow`/`block`/`error`/`timeout`.
|
||||
- `parent_span_id` — nested under the current pipeline span when one exists.
|
||||
|
||||
No-op when no tracing collector is attached to ctx (safe in tests and for tenants without tracing enabled).
|
||||
|
||||
### Metrics (slog keys)
|
||||
|
||||
- `security.hook.circuit_breaker` — tripped breaker.
|
||||
- `security.hook.audit_write_failed` — audit row write error.
|
||||
- `security.hook.resolve_error` — store resolve error (blocking event → fail-closed).
|
||||
- `security.hook.loop_depth_exceeded` — `MaxLoopDepth` violation.
|
||||
- `security.hook.prompt_parse_error` — evaluator returned malformed structured output.
|
||||
- `security.hook.budget_deduct_failed` / `budget_precheck_failed` — budget store error.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| HTTP hook always returns `error` | SSRF block on loopback (test). | Call `security.SetAllowLoopbackForTest(true)` in tests. |
|
||||
| Prompt hook blocks everything | Evaluator returning free-text (no tool call). | Review `prompt_template`; keep it short + imperative. |
|
||||
| Hook stopped firing | Circuit breaker tripped (5 blocks/min). | `UPDATE agent_hooks SET enabled=true WHERE id=...` after fixing upstream cause. |
|
||||
| UI `command` radio greyed out | Standard edition. | Use HTTP or prompt, or upgrade to Lite. |
|
||||
| Per-turn cap hit | `max_invocations_per_turn` too low. | Raise in hook config; review matcher tightness. |
|
||||
| Budget exceeded | Tenant spent monthly token budget. | Raise `tenant_hook_budget.budget_total` or wait for rollover. |
|
||||
|
||||
## Migration notes
|
||||
|
||||
No data migration required when upgrading from a pre-hooks release. Migration `000052_agent_hooks` creates `agent_hooks`, `hook_executions`, and `tenant_hook_budget`. SQLite users must also have schema version 20 or later.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- Prompt decision cache is per-process (no cluster-wide Redis yet).
|
||||
- No skill-frontmatter hooks (planned post-MVP).
|
||||
- No `agent` handler type (inter-agent delegation via prompt instead).
|
||||
- Desktop Wails UI has list/detail parity but no dedicated mobile-optimized form.
|
||||
@@ -132,11 +132,11 @@ The dual-identity pattern is a **codebase-wide convention**, not agent-specific:
|
||||
| Entity | UUID field | Key field | Notes |
|
||||
|---|---|---|---|
|
||||
| Agent | `agents.id uuid` | `agents.agent_key text` | Dual-tenant: same key across tenants is allowed |
|
||||
| Team | `agent_teams.id uuid` | `agent_teams.team_key text` | Team members reference agents by UUID |
|
||||
| Team | `agent_teams.id uuid` | (no key field) | Teams use UUID only; no slug or key field |
|
||||
| Tenant | `tenants.id uuid` | `tenants.tenant_slug text` | Slug used in URLs and onboarding wizard |
|
||||
| Session | `sessions.session_key text` | (no separate UUID) | Session key is the only identifier — safe by construction |
|
||||
|
||||
The rules in section 2 apply verbatim to `team_id`/`team_key` and `tenant_id`/`tenant_slug`. In particular:
|
||||
The rules in section 2 apply to agents and tenants. For **teams**:
|
||||
|
||||
- `WithTenantID(ctx, ...)` takes a `uuid.UUID`. Never pass a slug.
|
||||
- `TeamMember.AgentID` is the canonical team-membership FK and must be a UUID.
|
||||
@@ -255,9 +255,111 @@ Consequences for code:
|
||||
- Log formatting: prefer `{"agent": "tieu-ho", "tenant": "master"}` together, or just the UUID. Never log `agent_key` alone without a tenant.
|
||||
- When designing new tables with agent_key columns, the unique index must include `tenant_id`. Scoping to tenant_id prevents the same error class in future features.
|
||||
|
||||
**Note on teams:** Unlike agents, `agent_teams` table uses **UUID only**. There is no `team_key` or slug field. Always reference teams by `id` (UUID). Teams are not dual-identity.
|
||||
|
||||
## Appendix — TL;DR one-liner
|
||||
|
||||
- Use **`l.agentUUID.String()`** when setting `DomainEvent.AgentID`, store model fields, SQL query parameters, OTel span attributes, FK constraints, and any tenant-scoped unique key.
|
||||
- Use **`l.id`** when logging, emitting `AgentEvent` for the UI, rendering the system prompt, building filesystem paths, setting the key context (`WithAgentKey`), and doing router lookups.
|
||||
|
||||
---
|
||||
|
||||
## ActorID vs UserID in Group Chats
|
||||
|
||||
A second identity pattern complements agent identity: **who is acting** vs
|
||||
**which scope the action belongs to**. The pattern mirrors agent UUID/key
|
||||
but applies to end users in group contexts (Telegram, Discord, Feishu, Zalo).
|
||||
|
||||
### The two identities
|
||||
|
||||
| Identity | Context key | Helper | Meaning |
|
||||
|---|---|---|---|
|
||||
| **SCOPE** — namespace / memory | `UserIDKey` | `store.UserIDFromContext(ctx)` | `group:telegram:<chatID>` in Telegram groups; `guild:<guildID>:user:<senderID>` in Discord guilds; sender ID in DMs |
|
||||
| **ACTOR** — acting principal | `SenderIDKey` | `store.SenderIDFromContext(ctx)` | Always the individual sender's numeric ID (never group-scoped) |
|
||||
| (combined) | — | `store.ActorIDFromContext(ctx)` | `SenderID` if set, else falls back to `UserID` |
|
||||
|
||||
### When to use each
|
||||
|
||||
| Purpose | Helper | Why |
|
||||
|---|---|---|
|
||||
| Memory / KG / session key | `UserIDFromContext` / `MemoryUserID` / `KGUserID` | Memory is per-group by design — all members share conversational context |
|
||||
| File / path scope | `UserIDFromContext` | Per-group workspace; group members share files |
|
||||
| **Permission check** | `SenderIDFromContext` | Checks attribute to the real user, not the group |
|
||||
| **Audit trail** (`initiated_by`, event `UserID`) | `ActorIDFromContext` | Traces the real user action across wrappers |
|
||||
| **Ownership** (`OwnerID`, creator fields) | `ActorIDFromContext` | A skill published in a group belongs to the individual user |
|
||||
| Role / RBAC | `ActorIDFromContext` | Role applies to the human, not the group |
|
||||
|
||||
### Group behavior table
|
||||
|
||||
| Context | `UserID` | `SenderID` | `ActorID` |
|
||||
|---|---|---|---|
|
||||
| Telegram DM | sender numeric | sender numeric | sender numeric |
|
||||
| Telegram group | `group:telegram:<chatID>` | sender numeric | sender numeric |
|
||||
| Discord DM | sender snowflake | sender snowflake | sender snowflake |
|
||||
| Discord guild | `guild:<guildID>:user:<senderID>` | sender snowflake | sender snowflake |
|
||||
| HTTP API | HTTP user ID | (empty) | HTTP user ID |
|
||||
| Cron / subagent system ctx | (inherited / empty) | (empty or propagated) | same as SenderID, else UserID |
|
||||
|
||||
### Propagation through wrappers (#915)
|
||||
|
||||
When a tool wrapper synthesizes an inbound message (subagent announce,
|
||||
delegate announce, teammate dispatch, session_send), the synthetic
|
||||
`SenderID` carries routing identity (e.g. `subagent:<taskID>`) — NOT a
|
||||
real user. The real acting sender must travel through `InboundMessage.Metadata`
|
||||
under `tools.MetaOriginSenderID`, and the next-turn builder copies it to
|
||||
`RunRequest.SenderID`. Without this propagation:
|
||||
|
||||
- Permission checks against `bus.IsInternalSender(...)` prefixes → **denied**
|
||||
(synthetic senders never match a grant)
|
||||
- Empty sender → **denied** (`CheckFileWriterPermission` group-context fail-closed
|
||||
policy)
|
||||
|
||||
Code path:
|
||||
|
||||
```
|
||||
SubagentTask.OriginSenderID
|
||||
→ AnnounceMetadata.OriginSenderID
|
||||
→ InboundMessage.Metadata[MetaOriginSenderID]
|
||||
→ subagentAnnounceRouting.SenderID
|
||||
→ RunRequest.SenderID
|
||||
→ loop_context.go:WithSenderID(ctx, …)
|
||||
→ SenderIDFromContext(ctx) in permission checks
|
||||
```
|
||||
|
||||
Same chain for delegate (`delegate_tool.announceToParent`) and teammate
|
||||
dispatch (`team_tool_dispatch.go`).
|
||||
|
||||
### Group permission check policy
|
||||
|
||||
`store.CheckFileWriterPermission` / `store.CheckCronPermission` in group
|
||||
or guild context (`UserID` starts with `group:` or `guild:`):
|
||||
|
||||
- empty `SenderID` → **DENY** (system context must not gain write access silently)
|
||||
- synthetic-prefix `SenderID` → **DENY** (`subagent:`, `notification:`, `teammate:`,
|
||||
`system:`, `ticker:`, `session_send_tool`)
|
||||
- real numeric `SenderID` → DB lookup; DENY if no `file_writer` allow grant for this sender
|
||||
- DB error → fail-open (availability over strictness)
|
||||
|
||||
In DM / HTTP / cron-direct context (no group/guild prefix): always allow.
|
||||
No per-user writer gate applies.
|
||||
|
||||
### Legacy-data tolerance for skill ownership
|
||||
|
||||
Skills / cron / delegate audit trails created before #915 migration store
|
||||
`UserIDFromContext` values (possibly `group:*` prefixed) in `owner_id` /
|
||||
`user_id` columns. Ownership checks at `skill_manage.go` (patch/delete)
|
||||
accept either `ActorIDFromContext` (new) or `UserIDFromContext` (legacy).
|
||||
This lets existing group-scoped rows remain accessible without a
|
||||
destructive backfill. When a user re-publishes a skill, the new row uses
|
||||
actor ownership (tighter by default).
|
||||
|
||||
### Related code
|
||||
|
||||
- `internal/store/context.go` — helper definitions
|
||||
- `internal/store/config_permission_store.go` — group permission policy
|
||||
- `internal/tools/subagent_spawn.go`, `subagent_exec.go` — subagent propagation
|
||||
- `internal/tools/delegate_tool.go` — delegate propagation
|
||||
- `cmd/gateway_subagent_announce_queue.go`, `gateway_consumer_handlers.go` — re-ingress reconstruction
|
||||
- `tests/integration/telegram_group_write_file_permission_test.go` — regression fixtures
|
||||
|
||||
When in doubt, walk the four-step checklist in section 4.
|
||||
@@ -0,0 +1,145 @@
|
||||
# GitHub Binary Installer
|
||||
|
||||
Install CLI tools directly from GitHub Releases at runtime. Covers Go, Rust,
|
||||
shell, and other binary-distributed tools not available via `apk` / `pip` / `npm`.
|
||||
|
||||
Closes [#741](https://github.com/nextlevelbuilder/goclaw/issues/741).
|
||||
|
||||
## Install Syntax
|
||||
|
||||
```
|
||||
github:owner/repo[@tag]
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `github:cli/cli` → latest release
|
||||
- `github:jesseduffield/lazygit@v0.42.0` → specific version
|
||||
- `github:sharkdp/fd@v9.0.0` → specific version with dot separator
|
||||
|
||||
## How It Works
|
||||
|
||||
1. Fetches release metadata from the GitHub API
|
||||
2. Auto-selects asset matching `linux` + current arch (amd64 / arm64)
|
||||
3. Streams download to a temp file, enforcing a max size cap
|
||||
4. Verifies SHA256 if the publisher ships `checksums.txt` / `SHA256SUMS`
|
||||
5. Validates ELF magic bytes + 64-bit class + machine matches runtime arch
|
||||
6. Extracts archive safely (tar.gz / zip / raw binary) with path-traversal + zip-bomb guards
|
||||
7. Installs to `/app/data/.runtime/bin/` (prepended to `$PATH`)
|
||||
8. Persists a manifest for later listing + uninstall
|
||||
|
||||
## Usage
|
||||
|
||||
### Web UI
|
||||
|
||||
1. Admin Settings → Packages page
|
||||
2. Scroll to **GitHub Binaries** section
|
||||
3. Enter `owner/repo[@tag]` or click **Browse releases** to pick a version
|
||||
4. Click **Install**
|
||||
|
||||
### HTTP API
|
||||
|
||||
```bash
|
||||
# Install
|
||||
curl -X POST http://gateway/v1/packages/install \
|
||||
-H "Authorization: Bearer $ADMIN_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"package": "github:jesseduffield/lazygit@v0.42.0"}'
|
||||
|
||||
# List installed (includes pip/npm/system + github)
|
||||
curl http://gateway/v1/packages -H "Authorization: Bearer $ADMIN_TOKEN"
|
||||
|
||||
# Browse releases (picker UI uses this)
|
||||
curl 'http://gateway/v1/packages/github-releases?repo=cli/cli&limit=10' \
|
||||
-H "Authorization: Bearer $VIEWER_TOKEN"
|
||||
|
||||
# Uninstall
|
||||
curl -X POST http://gateway/v1/packages/uninstall \
|
||||
-H "Authorization: Bearer $ADMIN_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"package": "github:lazygit"}'
|
||||
```
|
||||
|
||||
## Admin Configuration
|
||||
|
||||
All configuration is driven by environment variables — **never** place the
|
||||
token in `config.json`.
|
||||
|
||||
| Env var | Default | Notes |
|
||||
|---------|---------|-------|
|
||||
| `GOCLAW_PACKAGES_GITHUB_TOKEN` | `""` | Optional PAT: rate 60/hr → 5000/hr + private repo access |
|
||||
| `GOCLAW_PACKAGES_MAX_ASSET_SIZE_MB` | `200` | Applies to both download cap and 2× uncompressed cap |
|
||||
| `GOCLAW_PACKAGES_GITHUB_ALLOWED_ORGS` | `""` | Comma-separated allowlist (empty = all orgs allowed) |
|
||||
| `GOCLAW_PACKAGES_GITHUB_BIN_DIR` | `/app/data/.runtime/bin` | Where extracted binaries land |
|
||||
| `GOCLAW_PACKAGES_GITHUB_MANIFEST` | `{bin_dir}/../github-packages.json` | Manifest path |
|
||||
|
||||
Token scopes:
|
||||
- public-only repos: no scopes required
|
||||
- private repos: `repo`
|
||||
- org-SSO-enforced repos: must be SSO-authorized PAT
|
||||
|
||||
## Security
|
||||
|
||||
- HTTPS-only downloads with SSRF host allowlist:
|
||||
`github.com`, `api.github.com`, `objects.githubusercontent.com`,
|
||||
`release-assets.githubusercontent.com`, `codeload.github.com`
|
||||
- Every redirect hop re-validated (blocks redirect-based host escape)
|
||||
- Literal IP hostnames (v4 / v6) always rejected (blocks cloud-metadata access)
|
||||
- SHA256 verification when publisher ships `checksums.txt` / `SHA256SUMS` (constant-time compare)
|
||||
- ELF magic + 64-bit class + machine-arch validation before `chmod +x`
|
||||
- Path-traversal prevention in archive extraction (rejects `..`, absolute, Windows drive, null byte)
|
||||
- Zip-bomb guard (cumulative uncompressed bytes capped at 2× max asset size)
|
||||
- Symlink / hardlink entries skipped, never written
|
||||
- Admin-only API + master-scope guard on install/uninstall
|
||||
- Picker endpoint `/v1/packages/github-releases` throttled per user (30 req/min, burst 10) to protect GitHub API quota; anonymous fallback keyed by remote IP. Response is `429 Too Many Requests` with `Retry-After: 60` when tripped.
|
||||
- Token never logged (startup log prints `token_set=bool`)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "glibc not found" / segfault on execution
|
||||
|
||||
GoClaw runs on Alpine Linux (musl libc). Many Go/Rust binaries target glibc.
|
||||
|
||||
**Fix:** pick a musl-compatible release asset. Look for names containing:
|
||||
- `*-musl.tar.gz` (explicit musl)
|
||||
- `*-linux-static*` (fully static)
|
||||
- Go binaries with `CGO_ENABLED=0` typically work out of the box
|
||||
|
||||
Known-good musl releases:
|
||||
- `ripgrep`: `ripgrep-*-x86_64-unknown-linux-musl.tar.gz`
|
||||
- `starship`: `starship-x86_64-unknown-linux-musl.tar.gz`
|
||||
- `gh`: `gh_*_linux_amd64.tar.gz` (static)
|
||||
|
||||
### "no matching asset found"
|
||||
|
||||
Asset naming doesn't fit the heuristic. Open the release page and confirm assets
|
||||
exist for `linux` + your arch. Workaround: file an upstream issue asking for
|
||||
standard `linux_amd64` / `linux_arm64` naming.
|
||||
|
||||
### "arch mismatch"
|
||||
|
||||
Binary is `amd64` but runtime is `arm64` (or vice versa). Pick a release asset
|
||||
matching the host arch — the release picker UI filters automatically.
|
||||
|
||||
### "rate limit exceeded"
|
||||
|
||||
Anonymous GitHub API is capped at 60 req/hr. Set
|
||||
`GOCLAW_PACKAGES_GITHUB_TOKEN` to bump to 5000/hr.
|
||||
|
||||
### "checksum mismatch"
|
||||
|
||||
Hard-fail. Indicates tampered download or publisher re-signing without updating
|
||||
the release. Do not force-install; report upstream.
|
||||
|
||||
## Limitations (Phase 1)
|
||||
|
||||
- Linux-only (Lite/Desktop editions not yet supported)
|
||||
- Docker edition only (runtime dir `/app/data/.runtime/bin`)
|
||||
- Installs all top-level executables in an archive (no interactive picker if
|
||||
archive contains multiple binaries)
|
||||
- No version history / rollback — re-installing replaces in place
|
||||
- Global manifest (not per-tenant)
|
||||
|
||||
## See Also
|
||||
|
||||
- [`docs/14-skills-runtime.md`](./14-skills-runtime.md) — Overview of the runtime packages system
|
||||
- Issue [#741](https://github.com/nextlevelbuilder/goclaw/issues/741) — Original feature request
|
||||
@@ -0,0 +1,45 @@
|
||||
# Example Hooks Library
|
||||
|
||||
Copy-paste-ready hook configurations for common use cases. Each JSON is a valid `hooks.create` WS payload — send it via the Web UI → **Hooks → Create** (paste into the relevant fields) or POST directly to the RPC endpoint.
|
||||
|
||||
## Safety reminders
|
||||
|
||||
- **Review regex matchers** before enabling. A too-broad matcher on a prompt hook drains token budget. Prefer anchored patterns like `^(exec|shell)$`.
|
||||
- **`command` hooks require Lite edition.** On Standard the UI greys them out and dispatch fails closed.
|
||||
- **SSRF-safe HTTP hooks**: URLs are resolved + pinned before the request. Loopback/private ranges are blocked in production.
|
||||
- **Authorization headers** get AES-256-GCM encrypted at rest. Never commit plaintext secrets to these JSON files.
|
||||
- **Test in staging first.** Use the `Test` tab in the UI — it runs with `dryRun=true` and does NOT write to `hook_executions`.
|
||||
|
||||
## Catalog
|
||||
|
||||
| File | Event | Handler | Scope | Purpose |
|
||||
|---|---|---|---|---|
|
||||
| `block-rm-rf.json` | `pre_tool_use` | command | agent | Block dangerous `rm -rf /` via local shell script. Lite only. |
|
||||
| `auto-lint-after-write.json` | `post_tool_use` | http | agent | Fire-and-forget lint request after file writes. |
|
||||
| `audit-tool-usage.json` | `post_tool_use` | http | tenant | Stream every tool invocation to an external audit sink. |
|
||||
| `session-context-injector.json` | `session_start` | http | agent | Injects project metadata into the agent context at start. |
|
||||
| `notify-discord-on-stop.json` | `stop` | http | tenant | Discord webhook notification when a session ends. |
|
||||
|
||||
## Usage
|
||||
|
||||
### Via Web UI
|
||||
|
||||
1. Open `/hooks` → click **Create hook**.
|
||||
2. Copy fields from the example JSON into the form.
|
||||
3. For `http.config.headers`, paste your secret in the Authorization field — the server encrypts it before storing.
|
||||
4. Click **Save**, then **Test** with a sample event before enabling in production.
|
||||
|
||||
### Via WS RPC
|
||||
|
||||
```bash
|
||||
wscat -c ws://localhost:18790/ws
|
||||
# After connect:
|
||||
> {"id":"1","method":"hooks.create","params": <paste JSON here> }
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- `tenant_id` omitted → current tenant from WS session. Set `scope: "global"` to apply cross-tenant (master required).
|
||||
- `agent_id` required for `scope: "agent"`; otherwise leave null.
|
||||
- `priority: 10` is the recommended default. Higher priority hooks run first; first `block` wins the chain.
|
||||
- `on_timeout: "block"` for anything security-sensitive; `"allow"` for observation-only.
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"event": "post_tool_use",
|
||||
"handler_type": "http",
|
||||
"scope": "tenant",
|
||||
"matcher": "",
|
||||
"if_expr": "",
|
||||
"config": {
|
||||
"url": "https://audit.internal.example.com/hooks/tool-usage",
|
||||
"headers": {
|
||||
"Authorization": "Bearer REPLACE_ME_ENCRYPTED",
|
||||
"X-Goclaw-Audit": "tool-usage"
|
||||
}
|
||||
},
|
||||
"timeout_ms": 2000,
|
||||
"on_timeout": "allow",
|
||||
"priority": 1,
|
||||
"enabled": true,
|
||||
"source": "seed",
|
||||
"metadata": {
|
||||
"notes": "Streams every tool invocation to an external audit sink for compliance. Tenant-wide.",
|
||||
"tags": ["observability", "compliance", "audit"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"event": "post_tool_use",
|
||||
"handler_type": "http",
|
||||
"scope": "agent",
|
||||
"matcher": "^(write_file|edit_file)$",
|
||||
"if_expr": "",
|
||||
"config": {
|
||||
"url": "https://ci.internal.example.com/hooks/lint",
|
||||
"headers": {
|
||||
"Authorization": "Bearer REPLACE_ME_ENCRYPTED",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
},
|
||||
"timeout_ms": 3000,
|
||||
"on_timeout": "allow",
|
||||
"priority": 10,
|
||||
"enabled": true,
|
||||
"source": "api",
|
||||
"metadata": {
|
||||
"notes": "Fire-and-forget lint ping after file writes. Non-blocking event — won't stall the agent loop.",
|
||||
"tags": ["observability", "quality"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"event": "pre_tool_use",
|
||||
"handler_type": "command",
|
||||
"scope": "agent",
|
||||
"matcher": "^(exec|shell)$",
|
||||
"if_expr": "",
|
||||
"config": {
|
||||
"command": "grep -qE 'rm +-[rR][fF]? +/' && exit 2 || exit 0"
|
||||
},
|
||||
"timeout_ms": 2000,
|
||||
"on_timeout": "block",
|
||||
"priority": 100,
|
||||
"enabled": true,
|
||||
"source": "api",
|
||||
"metadata": {
|
||||
"notes": "Blocks `rm -rf /` style commands before exec. Lite edition only.",
|
||||
"tags": ["safety", "command"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"event": "stop",
|
||||
"handler_type": "http",
|
||||
"scope": "tenant",
|
||||
"matcher": "",
|
||||
"if_expr": "",
|
||||
"config": {
|
||||
"url": "https://discord.com/api/webhooks/REPLACE_ME",
|
||||
"headers": {
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
},
|
||||
"timeout_ms": 3000,
|
||||
"on_timeout": "allow",
|
||||
"priority": 1,
|
||||
"enabled": false,
|
||||
"source": "seed",
|
||||
"metadata": {
|
||||
"notes": "Discord webhook fired on every session stop. Disabled by default — enable after replacing the webhook URL. Discord expects {\"content\":\"...\"} but the hook ships event JSON; consider putting an adapter in front.",
|
||||
"tags": ["notify", "discord", "session"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"event": "session_start",
|
||||
"handler_type": "http",
|
||||
"scope": "agent",
|
||||
"matcher": "",
|
||||
"if_expr": "",
|
||||
"config": {
|
||||
"url": "https://ctx.internal.example.com/hooks/session-bootstrap",
|
||||
"headers": {
|
||||
"Authorization": "Bearer REPLACE_ME_ENCRYPTED"
|
||||
}
|
||||
},
|
||||
"timeout_ms": 4000,
|
||||
"on_timeout": "allow",
|
||||
"priority": 50,
|
||||
"enabled": true,
|
||||
"source": "api",
|
||||
"metadata": {
|
||||
"notes": "On session_start, fetches project-specific metadata and injects via additionalContext. The remote endpoint should respond with {\"decision\":\"allow\",\"additionalContext\":\"...\"}.",
|
||||
"tags": ["context", "session", "bootstrap"]
|
||||
}
|
||||
}
|
||||
@@ -14,11 +14,14 @@ require (
|
||||
github.com/charmbracelet/huh v0.8.0
|
||||
github.com/charmbracelet/lipgloss v1.1.0
|
||||
github.com/disintegration/imaging v1.6.2
|
||||
github.com/dop251/goja v0.0.0-20260311135729-065cd970411c
|
||||
github.com/fsnotify/fsnotify v1.9.0
|
||||
github.com/go-rod/rod v0.116.2
|
||||
github.com/golang-migrate/migrate/v4 v4.19.1
|
||||
github.com/google/cel-go v0.28.0
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7
|
||||
github.com/jackc/pgx/v5 v5.6.0
|
||||
github.com/jmoiron/sqlx v1.4.0
|
||||
github.com/mattn/go-runewidth v0.0.16
|
||||
@@ -40,14 +43,17 @@ require (
|
||||
go.opentelemetry.io/otel/trace v1.40.0
|
||||
golang.org/x/image v0.27.0
|
||||
golang.org/x/time v0.14.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
modernc.org/sqlite v1.47.0
|
||||
tailscale.com v1.94.2
|
||||
)
|
||||
|
||||
require (
|
||||
cel.dev/expr v0.25.1 // indirect
|
||||
filippo.io/edwards25519 v1.1.0 // indirect
|
||||
github.com/akutz/memconn v0.1.0 // indirect
|
||||
github.com/alexbrainman/sspi v0.0.0-20231016080023-1a75b4708caa // indirect
|
||||
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
|
||||
github.com/atotto/clipboard v0.1.4 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21 // indirect
|
||||
@@ -81,17 +87,19 @@ require (
|
||||
github.com/danieljoos/wincred v1.2.3 // indirect
|
||||
github.com/dblohm7/wingoes v0.0.0-20240119213807-a09d6be7affa // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/dlclark/regexp2 v1.11.0 // indirect
|
||||
github.com/dlclark/regexp2 v1.11.4 // indirect
|
||||
github.com/elliotchance/orderedmap/v3 v3.1.0 // indirect
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
||||
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
|
||||
github.com/gaissmai/bart v0.18.0 // indirect
|
||||
github.com/go-json-experiment/json v0.0.0-20250813024750-ebf49471dced // indirect
|
||||
github.com/go-ole/go-ole v1.3.0 // indirect
|
||||
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
|
||||
github.com/godbus/dbus/v5 v5.2.2 // indirect
|
||||
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
|
||||
github.com/google/btree v1.1.3 // indirect
|
||||
github.com/google/go-cmp v0.7.0 // indirect
|
||||
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e // indirect
|
||||
github.com/hdevalence/ed25519consensus v0.2.0 // indirect
|
||||
github.com/huin/goupnp v1.3.0 // indirect
|
||||
github.com/invopop/jsonschema v0.13.0 // indirect
|
||||
@@ -149,7 +157,6 @@ require (
|
||||
golang.org/x/term v0.40.0 // indirect
|
||||
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
|
||||
golang.zx2c4.com/wireguard/windows v0.5.3 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
gvisor.dev/gvisor v0.0.0-20250205023644-9414b50a5633 // indirect
|
||||
modernc.org/libc v1.70.0 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
@@ -203,5 +210,5 @@ require (
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260128011058-8636f8732409 // indirect
|
||||
google.golang.org/grpc v1.78.0 // indirect
|
||||
google.golang.org/protobuf v1.36.11
|
||||
google.golang.org/protobuf v1.36.11 // indirect
|
||||
)
|
||||
@@ -1,5 +1,7 @@
|
||||
9fans.net/go v0.0.8-0.20250307142834-96bdba94b63f h1:1C7nZuxUMNz7eiQALRfiqNOm04+m3edWlRff/BYHf0Q=
|
||||
9fans.net/go v0.0.8-0.20250307142834-96bdba94b63f/go.mod h1:hHyrZRryGqVdqrknjq5OWDLGCTJ2NeEvtrpR96mjraM=
|
||||
cel.dev/expr v0.25.1 h1:1KrZg61W6TWSxuNZ37Xy49ps13NUovb66QLprthtwi4=
|
||||
cel.dev/expr v0.25.1/go.mod h1:hrXvqGP6G6gyx8UAHSHJ5RGk//1Oj5nXQ2NI02Nrsg4=
|
||||
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
|
||||
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
|
||||
filippo.io/mkcert v1.4.4 h1:8eVbbwfVlaqUM7OwuftKc2nuYOoTDQWqsoXmzoXZdbc=
|
||||
@@ -12,6 +14,9 @@ github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7Oputl
|
||||
github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU=
|
||||
github.com/MakeNowJust/heredoc v1.0.0 h1:cXCdzVdstXyiTqTvfqk9SDHpKNjxuom+DOlyEeQ4pzQ=
|
||||
github.com/MakeNowJust/heredoc v1.0.0/go.mod h1:mG5amYoWBHf8vpLOuehzbGGw0EHxpZZ6lCpQ4fNJ8LE=
|
||||
github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww=
|
||||
github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
|
||||
github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
|
||||
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
|
||||
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
|
||||
github.com/adhocore/gronx v1.19.6 h1:5KNVcoR9ACgL9HhEqCm5QXsab/gI4QDIybTAWcXDKDc=
|
||||
@@ -28,6 +33,8 @@ github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwTo
|
||||
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8=
|
||||
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be/go.mod h1:ySMOLuWl6zY27l47sB3qLNK6tF2fkHG55UZxx8oIVo4=
|
||||
github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ=
|
||||
github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw=
|
||||
github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
|
||||
github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.5 h1:dj5kopbwUsVUVFgO4Fi5BIT3t4WyqIDjGKCangnV/yY=
|
||||
@@ -174,14 +181,16 @@ github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5Qvfr
|
||||
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
|
||||
github.com/djherbis/times v1.6.0 h1:w2ctJ92J8fBvWPxugmXIv7Nz7Q3iDMKNx9v5ocVH20c=
|
||||
github.com/djherbis/times v1.6.0/go.mod h1:gOHeRAz2h+VJNZ5Gmc/o7iD9k4wW7NMVqieYCY99oc0=
|
||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/dlclark/regexp2 v1.11.4 h1:rPYF9/LECdNymJufQKmri9gV604RvvABwgOA8un7yAo=
|
||||
github.com/dlclark/regexp2 v1.11.4/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/docker/docker v28.3.3+incompatible h1:Dypm25kh4rmk49v1eiVbsAtpAsYURjYkaKubwuBdxEI=
|
||||
github.com/docker/docker v28.3.3+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
|
||||
github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c=
|
||||
github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc=
|
||||
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
|
||||
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
|
||||
github.com/dop251/goja v0.0.0-20260311135729-065cd970411c h1:OcLmPfx1T1RmZVHHFwWMPaZDdRf0DBMZOFMVWJa7Pdk=
|
||||
github.com/dop251/goja v0.0.0-20260311135729-065cd970411c/go.mod h1:MxLav0peU43GgvwVgNbLAj1s/bSGboKkhuULvq/7hx4=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/elliotchance/orderedmap/v3 v3.1.0 h1:j4DJ5ObEmMBt/lcwIecKcoRxIQUEnw0L804lXYDt/pg=
|
||||
@@ -211,6 +220,8 @@ github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE=
|
||||
github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78=
|
||||
github.com/go-rod/rod v0.116.2 h1:A5t2Ky2A+5eD/ZJQr1EfsQSe5rms5Xof/qj296e+ZqA=
|
||||
github.com/go-rod/rod v0.116.2/go.mod h1:H+CMO9SCNc2TJ2WfrG+pKhITz57uGNYU43qYHh438Mg=
|
||||
github.com/go-sourcemap/sourcemap v2.1.3+incompatible h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU=
|
||||
github.com/go-sourcemap/sourcemap v2.1.3+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=
|
||||
github.com/go-sql-driver/mysql v1.8.1 h1:LedoTUt/eveggdHS9qUFC1EFSa8bU2+1pZjSRpvNJ1Y=
|
||||
github.com/go-sql-driver/mysql v1.8.1/go.mod h1:wEBSXgmK//2ZFJyE+qWnIsVGmvmEKlqwuVSjsCm7DZg=
|
||||
github.com/go-test/deep v1.1.1 h1:0r/53hagsehfO4bzD2Pgr/+RgHqhmf+k1Bpse2cTu1U=
|
||||
@@ -230,6 +241,8 @@ github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
|
||||
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
||||
github.com/google/cel-go v0.28.0 h1:KjSWstCpz/MN5t4a8gnGJNIYUsJRpdi/r97xWDphIQc=
|
||||
github.com/google/cel-go v0.28.0/go.mod h1:X0bD6iVNR8pkROSOoHVdgTkzmRcosof7WQqCD6wcMc8=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/go-tpm v0.9.4 h1:awZRf9FwOeTunQmHoDYSHJps3ie6f1UlhS1fOdPEt1I=
|
||||
@@ -247,7 +260,6 @@ github.com/grbit/go-json v0.11.0 h1:bAbyMdYrYl/OjYsSqLH99N2DyQ291mHy726Mx+sYrnc=
|
||||
github.com/grbit/go-json v0.11.0/go.mod h1:IYpHsdybQ386+6g3VE6AXQ3uTGa5mquBme5/ZWmtzek=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.7 h1:X+2YciYSxvMQK0UZ7sg45ZVabVZBeBuvMkmuI2V3Fak=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.7/go.mod h1:lW34nIZuQ8UDPdkon5fmfp2l3+ZkQ2me/+oecHYLOII=
|
||||
github.com/hashicorp/golang-lru v0.6.0 h1:uL2shRDx7RTrOrTCUZEGP/wJUFiUI8QT6E7z5o8jga4=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/hdevalence/ed25519consensus v0.2.0 h1:37ICyZqdyj0lAZ8P4D1d1id3HqbbG1N3iBb1Tb4rdcU=
|
||||
@@ -553,6 +565,7 @@ go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.uber.org/mock v0.6.0 h1:hyF9dfmbgIX5EfOdasqLsWD6xqpNZlXblLB/Dbnwv3Y=
|
||||
go.uber.org/mock v0.6.0/go.mod h1:KiVJ4BqZJaMj4svdfmHM0AUx4NJYO8ZNpPnZn1Z+BBU=
|
||||
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
|
||||
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
|
||||
go4.org/mem v0.0.0-20240501181205-ae6ca9944745 h1:Tl++JLUCe4sxGu8cTpDzRLd3tN7US4hOxG5YpKCzkek=
|
||||
go4.org/mem v0.0.0-20240501181205-ae6ca9944745/go.mod h1:reUoABIJ9ikfM5sgtSF3Wushcza7+WeD01VB9Lirh3g=
|
||||
@@ -623,6 +636,8 @@ gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntN
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
gopkg.in/sourcemap.v1 v1.0.5 h1:inv58fC9f9J3TK2Y2R1NPntXEn3/wjWHkonhIUODNTI=
|
||||
gopkg.in/sourcemap.v1 v1.0.5/go.mod h1:2RlvNNSMglmRrcvhfuzp4hQHwOtjxlbjX7UPY/GXb78=
|
||||
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
|
||||
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/pipeline"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tools"
|
||||
)
|
||||
|
||||
// ctxCapturingExecutor is a stubExecutor variant that captures the context
|
||||
// passed to ExecuteWithContext so tests can assert on store.AgentAudioFromCtx.
|
||||
type ctxCapturingExecutor struct {
|
||||
mu sync.Mutex
|
||||
capturedCtx []context.Context
|
||||
}
|
||||
|
||||
func (e *ctxCapturingExecutor) ExecuteWithContext(ctx context.Context, _ string, _ map[string]any, _, _, _, _ string, _ tools.AsyncCallback) *tools.Result {
|
||||
e.mu.Lock()
|
||||
e.capturedCtx = append(e.capturedCtx, ctx)
|
||||
e.mu.Unlock()
|
||||
return &tools.Result{ForLLM: "ok", IsError: false}
|
||||
}
|
||||
func (e *ctxCapturingExecutor) TryActivateDeferred(string) bool { return false }
|
||||
func (e *ctxCapturingExecutor) ProviderDefs() []providers.ToolDefinition { return nil }
|
||||
func (e *ctxCapturingExecutor) Get(string) (tools.Tool, bool) { return nil, false }
|
||||
func (e *ctxCapturingExecutor) List() []string { return nil }
|
||||
func (e *ctxCapturingExecutor) Aliases() map[string]string { return nil }
|
||||
|
||||
func (e *ctxCapturingExecutor) lastCtx() context.Context {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
if len(e.capturedCtx) == 0 {
|
||||
return nil
|
||||
}
|
||||
return e.capturedCtx[len(e.capturedCtx)-1]
|
||||
}
|
||||
|
||||
func TestDispatcherAudioCtx_ToolCallInjectsSnapshot(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
expectedID := uuid.New()
|
||||
otherCfg := json.RawMessage(`{"tts_voice_id":"V1"}`)
|
||||
|
||||
cap := &ctxCapturingExecutor{}
|
||||
l := &Loop{
|
||||
id: "audio-test-agent",
|
||||
agentUUID: expectedID,
|
||||
agentOtherConfig: append([]byte(nil), otherCfg...),
|
||||
tools: cap,
|
||||
}
|
||||
|
||||
req := &RunRequest{RunID: "r1", SessionKey: "s1", Channel: "ws"}
|
||||
state := &pipeline.RunState{RunID: "r1"}
|
||||
tc := providers.ToolCall{ID: "tc-1", Name: "tts", Arguments: map[string]any{"text": "hello"}}
|
||||
|
||||
_, err := l.makeExecuteToolCall(req, &runState{})(context.Background(), state, tc)
|
||||
if err != nil {
|
||||
t.Fatalf("makeExecuteToolCall error: %v", err)
|
||||
}
|
||||
|
||||
captured := cap.lastCtx()
|
||||
if captured == nil {
|
||||
t.Fatal("ExecuteWithContext was not called")
|
||||
}
|
||||
snap, ok := store.AgentAudioFromCtx(captured)
|
||||
if !ok {
|
||||
t.Fatal("AgentAudioFromCtx returned ok=false — producer not wired in makeExecuteToolCall")
|
||||
}
|
||||
if snap.AgentID != expectedID {
|
||||
t.Errorf("AgentID: got %v, want %v", snap.AgentID, expectedID)
|
||||
}
|
||||
if !bytes.Equal(snap.OtherConfig, otherCfg) {
|
||||
t.Errorf("OtherConfig: got %q, want %q", snap.OtherConfig, otherCfg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDispatcherAudioCtx_ToolRawInjectsSnapshot(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
expectedID := uuid.New()
|
||||
otherCfg := json.RawMessage(`{"tts_voice_id":"V1"}`)
|
||||
|
||||
cap := &ctxCapturingExecutor{}
|
||||
l := &Loop{
|
||||
id: "audio-test-agent-raw",
|
||||
agentUUID: expectedID,
|
||||
agentOtherConfig: append([]byte(nil), otherCfg...),
|
||||
tools: cap,
|
||||
}
|
||||
|
||||
req := &RunRequest{RunID: "r2", SessionKey: "s2", Channel: "ws"}
|
||||
tc := providers.ToolCall{ID: "tc-2", Name: "tts", Arguments: map[string]any{"text": "world"}}
|
||||
|
||||
_, _, err := l.makeExecuteToolRaw(req)(context.Background(), tc)
|
||||
if err != nil {
|
||||
t.Fatalf("makeExecuteToolRaw error: %v", err)
|
||||
}
|
||||
|
||||
captured := cap.lastCtx()
|
||||
if captured == nil {
|
||||
t.Fatal("ExecuteWithContext was not called")
|
||||
}
|
||||
snap, ok := store.AgentAudioFromCtx(captured)
|
||||
if !ok {
|
||||
t.Fatal("AgentAudioFromCtx returned ok=false — producer not wired in makeExecuteToolRaw")
|
||||
}
|
||||
if snap.AgentID != expectedID {
|
||||
t.Errorf("AgentID: got %v, want %v", snap.AgentID, expectedID)
|
||||
}
|
||||
if !bytes.Equal(snap.OtherConfig, otherCfg) {
|
||||
t.Errorf("OtherConfig: got %q, want %q", snap.OtherConfig, otherCfg)
|
||||
}
|
||||
}
|
||||
@@ -91,7 +91,7 @@ func BenchmarkSanitizeHistory_WithToolCalls(b *testing.B) {
|
||||
}
|
||||
|
||||
// Repeat to make it longer
|
||||
for i := 0; i < 20; i++ {
|
||||
for range 20 {
|
||||
msgs = append(msgs, msgs[:7]...)
|
||||
}
|
||||
|
||||
@@ -118,7 +118,7 @@ func BenchmarkSanitizeHistory_WithMissingToolResults(b *testing.B) {
|
||||
}
|
||||
|
||||
// Repeat to make it longer
|
||||
for i := 0; i < 25; i++ {
|
||||
for range 25 {
|
||||
msgs = append(msgs, msgs[:5]...)
|
||||
}
|
||||
|
||||
@@ -144,7 +144,7 @@ func BenchmarkSanitizeHistory_ConsecutiveSameRole(b *testing.B) {
|
||||
}
|
||||
|
||||
// Repeat to make it longer
|
||||
for i := 0; i < 30; i++ {
|
||||
for range 30 {
|
||||
msgs = append(msgs, msgs[:8]...)
|
||||
}
|
||||
|
||||
@@ -161,7 +161,7 @@ func makeHistoryMessages(pairs int) []providers.Message {
|
||||
msgs := make([]providers.Message, 0, pairs*2)
|
||||
content := strings.Repeat("message content ", 10)
|
||||
|
||||
for i := 0; i < pairs; i++ {
|
||||
for i := range pairs {
|
||||
// Add user message
|
||||
msgs = append(msgs, providers.Message{
|
||||
Role: "user",
|
||||
|
||||
@@ -67,6 +67,12 @@ func (l *Loop) injectContext(ctx context.Context, req *RunRequest) (contextSetup
|
||||
if req.SenderName != "" {
|
||||
ctx = store.WithSenderName(ctx, req.SenderName)
|
||||
}
|
||||
// Inject caller role so RBAC-aware permission checks (CheckFileWriterPermission,
|
||||
// CheckCronPermission) can bypass per-user grants for authenticated admins
|
||||
// dispatched from dashboard or other trusted sources (#915).
|
||||
if req.Role != "" {
|
||||
ctx = store.WithRole(ctx, req.Role)
|
||||
}
|
||||
// Inject global + per-agent builtin tool settings (tier 1+3).
|
||||
// Media/provider-chain tools read the merged view via BuiltinToolSettingsFromCtx.
|
||||
if l.builtinToolSettings != nil {
|
||||
|
||||
@@ -256,10 +256,10 @@ func (l *Loop) buildMessages(ctx context.Context, history []providers.Message, s
|
||||
})
|
||||
}
|
||||
|
||||
// History pipeline matching TS: limitHistoryTurns → pruneContext → sanitizeHistory.
|
||||
// History pipeline: limitHistoryTurns → sanitizeHistory.
|
||||
// Pruning is owned by PruneStage in the pipeline (single entry point).
|
||||
trimmed := limitHistoryTurns(history, historyLimit)
|
||||
pruned := pruneContextMessages(trimmed, l.contextWindow, l.contextPruningCfg, l.tokenCounter, l.model)
|
||||
sanitized, droppedCount := sanitizeHistory(pruned)
|
||||
sanitized, droppedCount := sanitizeHistory(trimmed)
|
||||
messages = append(messages, sanitized...)
|
||||
|
||||
// If orphaned messages were found and dropped, persist the cleaned history
|
||||
|
||||
@@ -602,7 +602,7 @@ func TestEstimateOverhead(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestPruneContextMessagesDefaultEnabled(t *testing.T) {
|
||||
// With nil config, pruning should run (not return early).
|
||||
// Phase 04: nil config → no pruning (opt-in). To enable, use Mode: "cache-ttl".
|
||||
// Create messages with a large tool result that exceeds soft trim threshold.
|
||||
largeContent := make([]byte, 10000)
|
||||
for i := range largeContent {
|
||||
@@ -619,9 +619,10 @@ func TestPruneContextMessagesDefaultEnabled(t *testing.T) {
|
||||
{Role: "assistant", Content: "Sure, here you go."},
|
||||
}
|
||||
|
||||
// With nil config and small context window (to trigger soft trim ratio > 0.3),
|
||||
// With cache-ttl mode and small context window (to trigger soft trim ratio > 0.3),
|
||||
// pruning should trim the large tool result.
|
||||
result := pruneContextMessages(msgs, 5000, nil, nil, "")
|
||||
cfg := &config.ContextPruningConfig{Mode: "cache-ttl"}
|
||||
result := pruneContextMessages(msgs, 5000, cfg, nil, "", nil)
|
||||
|
||||
// The large tool result should have been trimmed.
|
||||
toolMsg := result[2]
|
||||
@@ -635,7 +636,7 @@ func TestPruneContextMessagesExplicitOff(t *testing.T) {
|
||||
{Role: "user", Content: "Hello"},
|
||||
}
|
||||
cfg := &config.ContextPruningConfig{Mode: "off"}
|
||||
result := pruneContextMessages(msgs, 200000, cfg, nil, "")
|
||||
result := pruneContextMessages(msgs, 200000, cfg, nil, "", nil)
|
||||
// Should return original messages unchanged.
|
||||
if len(result) != len(msgs) {
|
||||
t.Errorf("expected %d messages, got %d", len(msgs), len(result))
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
@@ -79,7 +80,7 @@ func (l *Loop) enrichInputMedia(ctx context.Context, req *RunRequest, messages [
|
||||
var imageFiles []bus.MediaFile
|
||||
for _, ref := range mediaRefs {
|
||||
if ref.Kind == "image" && ref.Path != "" {
|
||||
imageFiles = append(imageFiles, bus.MediaFile{Path: ref.Path, MimeType: ref.MimeType})
|
||||
imageFiles = append(imageFiles, bus.MediaFile{Path: ref.Path, MimeType: ref.MimeType, Filename: filepath.Base(ref.Path)})
|
||||
}
|
||||
}
|
||||
if deferToReadImageTool {
|
||||
|
||||
@@ -337,13 +337,15 @@ func TestLoop_LazyMCP_DenyList_GroupDeny(t *testing.T) {
|
||||
return true
|
||||
})
|
||||
|
||||
// Register a custom group containing the MCP tool.
|
||||
tools.RegisterToolGroup("mcp_test", []string{"mcp_svc__get_data"})
|
||||
defer tools.UnregisterToolGroup("mcp_test")
|
||||
// Register a custom group containing the MCP tool (using per-Registry method).
|
||||
reg.RegisterToolGroup("mcp_test", []string{"mcp_svc__get_data"})
|
||||
defer reg.UnregisterToolGroup("mcp_test")
|
||||
|
||||
pe := tools.NewPolicyEngine(&config.ToolsConfig{
|
||||
Deny: []string{"group:mcp_test"},
|
||||
})
|
||||
// PolicyEngine needs registry to expand group:mcp_test
|
||||
pe.SetRegistry(reg)
|
||||
|
||||
if !pe.IsDenied("mcp_svc__get_data", nil) {
|
||||
t.Error("tool should be denied via group: pattern")
|
||||
|
||||
@@ -15,6 +15,9 @@ import (
|
||||
// connections are established via pool.AcquireUser() and BridgeTools created.
|
||||
func (l *Loop) getUserMCPTools(ctx context.Context, userID string) []tools.Tool {
|
||||
if len(l.mcpUserCredSrvs) == 0 || l.mcpPool == nil || l.mcpStore == nil || userID == "" {
|
||||
if userID == "" && len(l.mcpUserCredSrvs) > 0 {
|
||||
slog.Debug("mcp.user_tools_skipped", "reason", "empty_user_id", "servers", len(l.mcpUserCredSrvs))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -86,7 +89,7 @@ func (l *Loop) getUserMCPTools(ctx context.Context, userID string) []tools.Tool
|
||||
// shared tool registry so ExecuteWithContext can resolve them by name.
|
||||
reg, _ := l.tools.(*tools.Registry)
|
||||
for _, mcpTool := range entry.MCPTools() {
|
||||
bt := mcpbridge.NewBridgeTool(srv.Name, mcpTool, entry.ClientPtr(), srv.ToolPrefix, srv.TimeoutSec, entry.Connected())
|
||||
bt := mcpbridge.NewBridgeTool(srv.Name, mcpTool, entry.ClientPtr(), srv.ToolPrefix, srv.TimeoutSec, entry.Connected(), srv.ID, l.mcpGrantChecker)
|
||||
// Register in registry so ExecuteWithContext can find them.
|
||||
// Skip if already registered (another user loaded this server with same tool names).
|
||||
if reg != nil {
|
||||
@@ -100,6 +103,13 @@ func (l *Loop) getUserMCPTools(ctx context.Context, userID string) []tools.Tool
|
||||
|
||||
if len(userTools) > 0 {
|
||||
l.mcpUserTools.Store(userID, userTools)
|
||||
// Update "mcp" tool group so policy expansion via alsoAllow includes
|
||||
// per-user tools. MergeToolGroup is additive — safe for concurrent users.
|
||||
var names []string
|
||||
for _, t := range userTools {
|
||||
names = append(names, t.Name())
|
||||
}
|
||||
l.registry.MergeToolGroup("mcp", names)
|
||||
slog.Info("mcp.user_tools_loaded", "user", userID, "tools", len(userTools))
|
||||
}
|
||||
return userTools
|
||||
|
||||
@@ -3,6 +3,7 @@ package agent
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/memory"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/pipeline"
|
||||
@@ -50,6 +51,7 @@ func (l *Loop) buildPipelineDeps(req *RunRequest, bridgeRS *runState) pipeline.P
|
||||
return pipeline.PipelineDeps{
|
||||
TokenCounter: tokencount.NewTiktokenCounter(),
|
||||
EventBus: l.domainBus,
|
||||
Hooks: l.hookDispatcher,
|
||||
Config: pipeline.PipelineConfig{
|
||||
MaxIterations: maxIter,
|
||||
MaxToolCalls: l.maxToolCalls,
|
||||
@@ -111,8 +113,22 @@ func (l *Loop) buildPipelineDeps(req *RunRequest, bridgeRS *runState) pipeline.P
|
||||
|
||||
// Prune callbacks
|
||||
PruneMessages: cb.pruneMessages,
|
||||
SanitizeHistory: cb.sanitizeHistory,
|
||||
CompactMessages: cb.compactMessages,
|
||||
|
||||
// Cache-TTL gate callbacks (Phase 06)
|
||||
GetProviderCaps: func() providers.ProviderCapabilities {
|
||||
if ca, ok := l.provider.(providers.CapabilitiesAware); ok {
|
||||
return ca.Capabilities()
|
||||
}
|
||||
return providers.ProviderCapabilities{}
|
||||
},
|
||||
GetPruningConfig: func() *config.ContextPruningConfig {
|
||||
return l.contextPruningCfg
|
||||
},
|
||||
GetCacheTouch: l.cacheTouchAt,
|
||||
MarkCacheTouched: l.markCacheTouched,
|
||||
|
||||
// Memory flush
|
||||
RunMemoryFlush: cb.runMemoryFlush,
|
||||
|
||||
|
||||
@@ -47,6 +47,7 @@ func (l *Loop) pipelineCallbacks(req *RunRequest, bridgeRS *runState) pipelineCa
|
||||
buildFilteredTools: l.makeBuildFilteredTools(req),
|
||||
callLLM: l.makeCallLLM(req, emitRun),
|
||||
pruneMessages: l.makePruneMessages(),
|
||||
sanitizeHistory: sanitizeHistory,
|
||||
compactMessages: l.makeCompactMessages(req),
|
||||
runMemoryFlush: l.makeRunMemoryFlush(),
|
||||
executeToolCall: l.makeExecuteToolCall(req, bridgeRS),
|
||||
@@ -73,7 +74,8 @@ type pipelineCallbackSet struct {
|
||||
injectReminders func(ctx context.Context, input *pipeline.RunInput, msgs []providers.Message) []providers.Message
|
||||
buildFilteredTools func(state *pipeline.RunState) ([]providers.ToolDefinition, error)
|
||||
callLLM func(ctx context.Context, state *pipeline.RunState, req providers.ChatRequest) (*providers.ChatResponse, error)
|
||||
pruneMessages func(msgs []providers.Message, budget int) []providers.Message
|
||||
pruneMessages func(msgs []providers.Message, budget int) ([]providers.Message, pipeline.PruneStats)
|
||||
sanitizeHistory func(msgs []providers.Message) ([]providers.Message, int)
|
||||
compactMessages func(ctx context.Context, msgs []providers.Message, model string) ([]providers.Message, error)
|
||||
runMemoryFlush func(ctx context.Context, state *pipeline.RunState) error
|
||||
executeToolCall func(ctx context.Context, state *pipeline.RunState, tc providers.ToolCall) ([]providers.Message, error)
|
||||
@@ -191,6 +193,11 @@ func (l *Loop) makeInjectReminders(req *RunRequest) func(ctx context.Context, in
|
||||
|
||||
func (l *Loop) makeBuildFilteredTools(req *RunRequest) func(state *pipeline.RunState) ([]providers.ToolDefinition, error) {
|
||||
return func(state *pipeline.RunState) ([]providers.ToolDefinition, error) {
|
||||
// Load per-user MCP tools (Notion, etc.) into registry before filtering.
|
||||
// Servers with require_user_credentials are deferred at startup and
|
||||
// connected per-request here with the actual user's credentials.
|
||||
l.getUserMCPTools(state.Ctx, state.Input.UserID)
|
||||
|
||||
maxIter := l.maxIterations
|
||||
if req.MaxIterations > 0 && req.MaxIterations < maxIter {
|
||||
maxIter = req.MaxIterations
|
||||
@@ -307,9 +314,11 @@ func (l *Loop) makeCallLLM(req *RunRequest, emitRun func(AgentEvent)) func(ctx c
|
||||
}
|
||||
}
|
||||
|
||||
func (l *Loop) makePruneMessages() func(msgs []providers.Message, budget int) []providers.Message {
|
||||
return func(msgs []providers.Message, budget int) []providers.Message {
|
||||
return pruneContextMessages(msgs, budget, l.contextPruningCfg, l.tokenCounter, l.model)
|
||||
func (l *Loop) makePruneMessages() func(msgs []providers.Message, budget int) ([]providers.Message, pipeline.PruneStats) {
|
||||
return func(msgs []providers.Message, budget int) ([]providers.Message, pipeline.PruneStats) {
|
||||
var stats pipeline.PruneStats
|
||||
pruned := pruneContextMessages(msgs, budget, l.contextPruningCfg, l.tokenCounter, l.model, &stats)
|
||||
return pruned, stats
|
||||
}
|
||||
}
|
||||
|
||||
@@ -337,6 +346,21 @@ func (l *Loop) makeCompactMessages(req *RunRequest) func(ctx context.Context, ms
|
||||
// duplicating the string.
|
||||
const SessionMetaKeyLastCompactionAt = "last_compaction_at"
|
||||
|
||||
// cacheTouchAt returns the last prune-mutation timestamp for a session.
|
||||
// Returns zero time if no touch recorded yet.
|
||||
func (l *Loop) cacheTouchAt(sessionKey string) time.Time {
|
||||
if v, ok := l.cacheTouchBySession.Load(sessionKey); ok {
|
||||
return v.(time.Time)
|
||||
}
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
// markCacheTouched records the current time as the last prune-mutation timestamp
|
||||
// for the given session. Called only after pruning actually mutates messages.
|
||||
func (l *Loop) markCacheTouched(sessionKey string) {
|
||||
l.cacheTouchBySession.Store(sessionKey, time.Now())
|
||||
}
|
||||
|
||||
func (l *Loop) makeRunMemoryFlush() func(ctx context.Context, state *pipeline.RunState) error {
|
||||
return func(ctx context.Context, state *pipeline.RunState) error {
|
||||
settings := ResolveMemoryFlushSettings(l.compactionCfg)
|
||||
|
||||
@@ -35,6 +35,15 @@ func (l *Loop) makeExecuteToolCall(req *RunRequest, bridgeRS *runState) func(ctx
|
||||
toolStart := time.Now().UTC()
|
||||
toolSpanID := l.emitToolSpanStart(ctx, toolStart, tc.Name, tc.ID, string(argsJSON))
|
||||
|
||||
// Inject agent audio snapshot so TTS tool (and any future audio consumers)
|
||||
// can read agent-level voice/model config without an extra DB lookup.
|
||||
if l.agentUUID != uuid.Nil {
|
||||
ctx = store.WithAgentAudio(ctx, store.AgentAudioSnapshot{
|
||||
AgentID: l.agentUUID,
|
||||
OtherConfig: append([]byte(nil), l.agentOtherConfig...), // defensive copy at dispatch
|
||||
})
|
||||
}
|
||||
|
||||
result := l.tools.ExecuteWithContext(ctx, registryName, tc.Arguments,
|
||||
req.Channel, req.ChatID, req.PeerKind, req.SessionKey, nil)
|
||||
toolDuration := time.Since(toolStart)
|
||||
@@ -84,6 +93,14 @@ func (l *Loop) makeExecuteToolRaw(req *RunRequest) func(ctx context.Context, tc
|
||||
start := time.Now().UTC()
|
||||
spanID := l.emitToolSpanStart(ctx, start, tc.Name, tc.ID, string(argsJSON))
|
||||
|
||||
// Inject agent audio snapshot (parallel path — same as sequential makeExecuteToolCall).
|
||||
if l.agentUUID != uuid.Nil {
|
||||
ctx = store.WithAgentAudio(ctx, store.AgentAudioSnapshot{
|
||||
AgentID: l.agentUUID,
|
||||
OtherConfig: append([]byte(nil), l.agentOtherConfig...), // defensive copy at dispatch
|
||||
})
|
||||
}
|
||||
|
||||
result := l.tools.ExecuteWithContext(ctx, registryName, tc.Arguments,
|
||||
req.Channel, req.ChatID, req.PeerKind, req.SessionKey, nil)
|
||||
dur := time.Since(start)
|
||||
|
||||
@@ -104,6 +104,12 @@ func (l *Loop) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
|
||||
ctx = tracing.WithTraceTeamID(ctx, *trace.TeamID)
|
||||
}
|
||||
|
||||
// Notify the gateway so it can associate this traceID with the active run
|
||||
// entry for force-abort (forceMarkTraceAborted needs traceID at abort time).
|
||||
if req.OnTraceCreated != nil {
|
||||
req.OnTraceCreated(traceID)
|
||||
}
|
||||
|
||||
// Pre-generate root "agent" span ID so LLM/tool spans can reference it as parent.
|
||||
agentSpanID = store.GenNewID()
|
||||
ctx = tracing.WithParentSpanID(ctx, agentSpanID)
|
||||
|
||||
@@ -2,6 +2,7 @@ package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
|
||||
@@ -11,6 +12,7 @@ import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/hooks"
|
||||
mcpbridge "github.com/nextlevelbuilder/goclaw/internal/mcp"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/media"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/memory"
|
||||
@@ -78,8 +80,11 @@ type Loop struct {
|
||||
// agentUUID is the canonical DB primary key. Use for SQL WHERE/JOIN,
|
||||
// DomainEvent.AgentID, OTel span attributes, and context propagation via
|
||||
// store.WithAgentID. See docs/agent-identity-conventions.md.
|
||||
agentUUID uuid.UUID
|
||||
tenantID uuid.UUID // agent's owning tenant
|
||||
agentUUID uuid.UUID
|
||||
tenantID uuid.UUID // agent's owning tenant
|
||||
// agentOtherConfig is a defensive byte copy of agents.other_config JSONB.
|
||||
// Copied once at Loop construction; used to build AgentAudioSnapshot at tool dispatch.
|
||||
agentOtherConfig json.RawMessage
|
||||
agentType string // "open" or "predefined"
|
||||
defaultTimezone string // system default timezone for bootstrap pre-fill
|
||||
provider providers.Provider
|
||||
@@ -107,6 +112,7 @@ type Loop struct {
|
||||
domainBus eventbus.DomainEventBus // V3 domain event bus for consolidation pipeline
|
||||
sessions store.SessionStore
|
||||
tools tools.ToolExecutor
|
||||
registry *tools.Registry // direct registry access for MergeToolGroup (per-Registry tool groups)
|
||||
toolPolicy *tools.PolicyEngine // optional: filters tools sent to LLM
|
||||
agentToolPolicy *config.ToolPolicySpec // per-agent tool policy from DB (nil = no restrictions)
|
||||
activeRuns atomic.Int32 // number of currently executing runs
|
||||
@@ -131,10 +137,11 @@ type Loop struct {
|
||||
userSetups sync.Map // userID → *userSetup (workspace + seeding state, per Loop instance)
|
||||
|
||||
// Per-user MCP tools: servers requiring user credentials get connected per-request.
|
||||
mcpStore store.MCPServerStore // for credential lookup
|
||||
mcpPool *mcpbridge.Pool // user-keyed connection pool
|
||||
mcpUserCredSrvs []store.MCPAccessInfo // servers needing per-user creds
|
||||
mcpUserTools sync.Map // userID → []tools.Tool (cached per-user tools)
|
||||
mcpStore store.MCPServerStore // for credential lookup
|
||||
mcpPool *mcpbridge.Pool // user-keyed connection pool
|
||||
mcpUserCredSrvs []store.MCPAccessInfo // servers needing per-user creds
|
||||
mcpUserTools sync.Map // userID → []tools.Tool (cached per-user tools)
|
||||
mcpGrantChecker mcpbridge.GrantChecker // runtime grant verification (nil = skip)
|
||||
|
||||
// Compaction config (memory flush settings)
|
||||
compactionCfg *config.CompactionConfig
|
||||
@@ -237,6 +244,19 @@ type Loop struct {
|
||||
|
||||
// User identity resolver: maps channel contacts to merged tenant users for credential lookups.
|
||||
userResolver UserIdentityResolver
|
||||
|
||||
// Per-session cache-touch timestamps for the cache-TTL pruning gate (Phase 06).
|
||||
// Key: sessionKey (string), Value: time.Time of last prune mutation.
|
||||
// sync.Map zero value is ready to use — no init required.
|
||||
// Grows with distinct sessions; typical gateway has bounded session count.
|
||||
// Note: in-memory only — timestamps reset on process restart (one extra prune
|
||||
// per session on restart, then steady-state resumes).
|
||||
cacheTouchBySession sync.Map
|
||||
|
||||
// hookDispatcher fires lifecycle hook events (Issue #875). Nil-safe: when
|
||||
// nil the pipeline fast-path skips all hook overhead. Populated from
|
||||
// LoopConfig.HookDispatcher during startup wiring.
|
||||
hookDispatcher hooks.Dispatcher
|
||||
}
|
||||
|
||||
// AgentEvent is emitted during agent execution for WS broadcasting.
|
||||
@@ -291,6 +311,7 @@ type LoopConfig struct {
|
||||
|
||||
Bus bus.EventPublisher
|
||||
DomainBus eventbus.DomainEventBus // V3 domain event bus for consolidation pipeline
|
||||
HookDispatcher hooks.Dispatcher // lifecycle hook dispatcher (nil = noop)
|
||||
Sessions store.SessionStore
|
||||
Tools *tools.Registry
|
||||
ToolPolicy *tools.PolicyEngine // optional: filters tools sent to LLM
|
||||
@@ -319,9 +340,10 @@ type LoopConfig struct {
|
||||
ShellDenyGroups map[string]bool
|
||||
|
||||
// Agent UUID + tenant for context propagation to tools
|
||||
AgentUUID uuid.UUID
|
||||
TenantID uuid.UUID // agent's owning tenant — injected into execution context
|
||||
AgentType string // "open" or "predefined"
|
||||
AgentUUID uuid.UUID
|
||||
TenantID uuid.UUID // agent's owning tenant — injected into execution context
|
||||
AgentOtherConfig json.RawMessage // raw other_config JSONB — copied defensively in NewLoop
|
||||
AgentType string // "open" or "predefined"
|
||||
DisplayName string // human-readable agent display name (for runtime section)
|
||||
IsTeamLead bool // agent leads a team (from resolver detection)
|
||||
|
||||
@@ -396,9 +418,10 @@ type LoopConfig struct {
|
||||
MemoryStore store.MemoryStore
|
||||
|
||||
// Per-user MCP tools (servers requiring per-user credentials)
|
||||
MCPStore store.MCPServerStore // for credential lookup
|
||||
MCPPool *mcpbridge.Pool // user-keyed connection pool
|
||||
MCPUserCredSrvs []store.MCPAccessInfo // servers needing per-user creds
|
||||
MCPStore store.MCPServerStore // for credential lookup
|
||||
MCPPool *mcpbridge.Pool // user-keyed connection pool
|
||||
MCPUserCredSrvs []store.MCPAccessInfo // servers needing per-user creds
|
||||
MCPGrantChecker mcpbridge.GrantChecker // runtime grant verification (nil = skip)
|
||||
|
||||
// V3 orchestration mode (resolved by resolver, controls tool visibility)
|
||||
OrchMode OrchestrationMode
|
||||
@@ -449,6 +472,7 @@ func NewLoop(cfg LoopConfig) *Loop {
|
||||
displayName: cfg.DisplayName,
|
||||
agentUUID: cfg.AgentUUID,
|
||||
tenantID: cfg.TenantID,
|
||||
agentOtherConfig: append([]byte(nil), cfg.AgentOtherConfig...), // defensive copy
|
||||
agentType: cfg.AgentType,
|
||||
provider: cfg.Provider,
|
||||
model: cfg.Model,
|
||||
@@ -467,8 +491,10 @@ func NewLoop(cfg LoopConfig) *Loop {
|
||||
sandboxCfg: cfg.SandboxCfg,
|
||||
eventPub: cfg.Bus,
|
||||
domainBus: cfg.DomainBus,
|
||||
hookDispatcher: cfg.HookDispatcher,
|
||||
sessions: cfg.Sessions,
|
||||
tools: cfg.Tools,
|
||||
registry: cfg.Tools,
|
||||
toolPolicy: cfg.ToolPolicy,
|
||||
agentToolPolicy: cfg.AgentToolPolicy,
|
||||
onEvent: cfg.OnEvent,
|
||||
@@ -518,6 +544,7 @@ func NewLoop(cfg LoopConfig) *Loop {
|
||||
mcpStore: cfg.MCPStore,
|
||||
mcpPool: cfg.MCPPool,
|
||||
mcpUserCredSrvs: cfg.MCPUserCredSrvs,
|
||||
mcpGrantChecker: cfg.MCPGrantChecker,
|
||||
orchMode: cfg.OrchMode,
|
||||
delegateTargets: cfg.DelegateTargets,
|
||||
evolutionMetricsStore: cfg.EvolutionMetricsStore,
|
||||
@@ -540,6 +567,7 @@ type RunRequest struct {
|
||||
UserID string // external user ID (TEXT, free-form) for multi-tenant scoping
|
||||
SenderID string // original individual sender ID (preserved in group chats for permission checks)
|
||||
SenderName string // display name from channel metadata (for bootstrap auto-contact)
|
||||
Role string // caller's RBAC role (admin/operator/viewer/owner); bypasses per-user grants for authenticated admins (#915)
|
||||
Stream bool // whether to stream response chunks
|
||||
ExtraSystemPrompt string // optional: injected into system prompt (skills, subagent context, etc.)
|
||||
SkillFilter []string // per-request skill override: nil=use agent default, []=no skills, ["x","y"]=whitelist
|
||||
@@ -566,6 +594,11 @@ type RunRequest struct {
|
||||
// user follow-up messages into the running conversation.
|
||||
InjectCh <-chan InjectedMessage
|
||||
|
||||
// OnTraceCreated is called once the trace UUID is determined for this run.
|
||||
// Used by the gateway to associate the trace ID with the active run entry
|
||||
// so force-abort can mark the correct trace as cancelled. Nil = no-op.
|
||||
OnTraceCreated func(traceID uuid.UUID)
|
||||
|
||||
// Delegation context (set when running as a delegate agent)
|
||||
DelegationID string // delegation ID for event correlation
|
||||
TeamID string // team ID (if delegation is team-scoped)
|
||||
|
||||
+24
-2
@@ -104,7 +104,29 @@ func (l *Loop) persistMedia(sessionKey string, files []bus.MediaFile, workspace
|
||||
if ext == "" {
|
||||
ext = filepath.Ext(srcPath) // fallback to source extension
|
||||
}
|
||||
dstPath := filepath.Join(uploadsDir, id+ext)
|
||||
|
||||
// Disk-naming: preserve user filename when present so vault enrichment
|
||||
// can process uploads (UUID-only names are skipped by enrich_skip_filter).
|
||||
// Empty Filename (voice note, clipboard paste, tool-generated) →
|
||||
// fall back to UUID, keeping legacy behavior.
|
||||
diskName := id + ext
|
||||
if stem := sanitizeFilename(f.Filename); stem != "" {
|
||||
diskName = stem + "-" + shortID(8) + ext
|
||||
}
|
||||
dstPath := filepath.Join(uploadsDir, diskName)
|
||||
|
||||
// Traversal guard: ensure resolved path is inside uploadsDir.
|
||||
// sanitizeFilename already strips ".." / "/" / "\\", but this is
|
||||
// a defense-in-depth check covering any future regressions.
|
||||
cleanDst := filepath.Clean(dstPath) + string(os.PathSeparator)
|
||||
cleanUploads := filepath.Clean(uploadsDir) + string(os.PathSeparator)
|
||||
if !strings.HasPrefix(cleanDst, cleanUploads) {
|
||||
slog.Warn("media: refusing to persist outside uploadsDir", "dst", dstPath, "uploads", uploadsDir)
|
||||
if sanitizedTemp != "" {
|
||||
os.Remove(sanitizedTemp)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
if err := copyMediaFile(srcPath, dstPath); err != nil {
|
||||
slog.Warn("media: failed to persist file", "path", f.Path, "error", err)
|
||||
@@ -467,7 +489,7 @@ func (l *Loop) reloadMediaForMessages(msgs []providers.Message, maxMessages int)
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
imageFiles = append(imageFiles, bus.MediaFile{Path: p, MimeType: ref.MimeType})
|
||||
imageFiles = append(imageFiles, bus.MediaFile{Path: p, MimeType: ref.MimeType, Filename: filepath.Base(p)})
|
||||
}
|
||||
|
||||
if !hasImageRef {
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
// Package agent - media_filename.go provides pure helpers to derive a safe,
|
||||
// human-readable disk filename from a user-provided original filename, and to
|
||||
// generate a short random id used as a collision-avoidance suffix.
|
||||
//
|
||||
// Disk naming scheme (used by persistMedia):
|
||||
//
|
||||
// {sanitizeFilename(stem)}-{shortID(8)}{ext}
|
||||
//
|
||||
// Callers MUST fall back to a UUID name when sanitizeFilename returns "".
|
||||
// This keeps backward compatibility for voice notes, clipboard paste, and
|
||||
// tool-generated media that arrive without an original filename.
|
||||
package agent
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"golang.org/x/text/unicode/norm"
|
||||
)
|
||||
|
||||
// slugRe collapses any run of non-ASCII-lowercase-alphanumeric chars into "-".
|
||||
// Applied on the Latin-dominant branch after normalization + lower-casing.
|
||||
var slugRe = regexp.MustCompile(`[^a-z0-9]+`)
|
||||
|
||||
// unsafeRe matches filesystem-unsafe characters that must be stripped from
|
||||
// non-Latin filenames (CJK, Arabic, etc.) where we want to preserve the original
|
||||
// script but still guarantee a safe on-disk name.
|
||||
var unsafeRe = regexp.MustCompile(`[\x00-\x1f/\\:*?"<>|]`)
|
||||
|
||||
// viMap handles Vietnamese-specific letters that NFD combining-mark stripping
|
||||
// does not decompose (đ/Đ are precomposed in Unicode without a base+mark form).
|
||||
// Applied BEFORE NFD normalization on the Latin branch.
|
||||
var viMap = map[rune]rune{
|
||||
'đ': 'd', 'Đ': 'd',
|
||||
}
|
||||
|
||||
// maxFilenameRunes caps the stem at 60 runes to keep on-disk names short and
|
||||
// avoid platform path-length issues. Rune-based (not byte-based) to stay safe
|
||||
// for multi-byte scripts.
|
||||
const maxFilenameRunes = 60
|
||||
|
||||
// isDominantLatin returns true when more than 50% of the runes are Latin
|
||||
// letters or ASCII digits. This distinguishes "Báo cáo Q4" (Latin-with-accents,
|
||||
// slug it) from "猫の写真" (CJK, preserve runes).
|
||||
func isDominantLatin(runes []rune) bool {
|
||||
if len(runes) == 0 {
|
||||
return false
|
||||
}
|
||||
latin := 0
|
||||
for _, r := range runes {
|
||||
if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') {
|
||||
latin++
|
||||
continue
|
||||
}
|
||||
// Latin-1 Supplement + Latin Extended ranges cover European accents.
|
||||
if unicode.Is(unicode.Latin, r) {
|
||||
latin++
|
||||
}
|
||||
}
|
||||
return latin*2 > len(runes)
|
||||
}
|
||||
|
||||
// sanitizeFilename derives a safe stem (no extension) from a user-provided
|
||||
// filename. Returns "" when the input is empty or sanitizes to nothing — the
|
||||
// caller must then fall back to a UUID-only name.
|
||||
//
|
||||
// Guarantees on the return value:
|
||||
// - never contains "/", "\\", "..", or ASCII control chars
|
||||
// - never starts or ends with "-"
|
||||
// - len([]rune(out)) <= maxFilenameRunes
|
||||
//
|
||||
// Branching:
|
||||
// - Latin-dominant input is accent-stripped, lowercased, and slugified.
|
||||
// - Non-Latin-dominant input (CJK, Arabic, etc.) keeps its original runes
|
||||
// with only filesystem-unsafe characters removed.
|
||||
func sanitizeFilename(orig string) string {
|
||||
orig = strings.TrimSpace(orig)
|
||||
if orig == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
// Strip the extension — caller re-appends the canonical ext from MIME.
|
||||
ext := filepath.Ext(orig)
|
||||
stem := strings.TrimSuffix(orig, ext)
|
||||
|
||||
// Defense-in-depth: strip any path components and reject traversal markers.
|
||||
stem = filepath.Base(stem)
|
||||
if stem == "." || stem == ".." || strings.Contains(stem, "..") {
|
||||
return ""
|
||||
}
|
||||
// Dotfiles with no real stem (e.g. ".env") have empty pre-ext content —
|
||||
// filepath.Ext(".env") returns ".env", so stem == "". Already handled below.
|
||||
stem = strings.TrimSpace(stem)
|
||||
if stem == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
runes := []rune(stem)
|
||||
if isDominantLatin(runes) {
|
||||
stem = latinSlug(runes)
|
||||
} else {
|
||||
stem = nonLatinClean(runes)
|
||||
}
|
||||
|
||||
// Cap to maxFilenameRunes (rune-based).
|
||||
capped := []rune(stem)
|
||||
if len(capped) > maxFilenameRunes {
|
||||
capped = capped[:maxFilenameRunes]
|
||||
}
|
||||
out := strings.Trim(string(capped), "-")
|
||||
return out
|
||||
}
|
||||
|
||||
// latinSlug applies Vietnamese mapping, NFD decomposition + combining-mark
|
||||
// stripping, lower-casing, and the slugRe replacement to produce an
|
||||
// ASCII-only kebab-case stem.
|
||||
func latinSlug(runes []rune) string {
|
||||
// Apply Vietnamese đ/Đ map pre-NFD (NFD does not decompose them).
|
||||
mapped := make([]rune, 0, len(runes))
|
||||
for _, r := range runes {
|
||||
if m, ok := viMap[r]; ok {
|
||||
mapped = append(mapped, m)
|
||||
continue
|
||||
}
|
||||
mapped = append(mapped, r)
|
||||
}
|
||||
// NFD: base letter + combining marks → drop marks (unicode.Mn).
|
||||
decomposed := norm.NFD.String(string(mapped))
|
||||
var b strings.Builder
|
||||
b.Grow(len(decomposed))
|
||||
for _, r := range decomposed {
|
||||
if unicode.Is(unicode.Mn, r) {
|
||||
continue
|
||||
}
|
||||
b.WriteRune(r)
|
||||
}
|
||||
lowered := strings.ToLower(b.String())
|
||||
// Replace non-[a-z0-9] runs with "-", then collapse/trim.
|
||||
slug := slugRe.ReplaceAllString(lowered, "-")
|
||||
return strings.Trim(slug, "-")
|
||||
}
|
||||
|
||||
// nonLatinClean preserves the script of non-Latin inputs while removing
|
||||
// control chars and filesystem-unsafe characters. Whitespace is collapsed to
|
||||
// a single space (kept, since CJK users expect spaces preserved).
|
||||
func nonLatinClean(runes []rune) string {
|
||||
s := string(runes)
|
||||
s = unsafeRe.ReplaceAllString(s, "")
|
||||
// Strip any remaining control runes not caught by unsafeRe.
|
||||
var b strings.Builder
|
||||
b.Grow(len(s))
|
||||
for _, r := range s {
|
||||
if unicode.IsControl(r) {
|
||||
continue
|
||||
}
|
||||
b.WriteRune(r)
|
||||
}
|
||||
return strings.TrimSpace(b.String())
|
||||
}
|
||||
|
||||
// shortID returns n lowercase hex chars sourced from crypto/rand. n must be
|
||||
// even and > 0; otherwise shortID panics (caller bug — always called with 8).
|
||||
func shortID(n int) string {
|
||||
if n <= 0 || n%2 != 0 {
|
||||
panic("shortID: n must be a positive even number")
|
||||
}
|
||||
b := make([]byte, n/2)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
// crypto/rand.Read on modern Go never fails in practice; panic signals
|
||||
// a catastrophic OS-level issue worth surfacing.
|
||||
panic("shortID: crypto/rand failure: " + err.Error())
|
||||
}
|
||||
return hex.EncodeToString(b)
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
func TestSanitizeFilename(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
in string
|
||||
want string
|
||||
}{
|
||||
// Latin / Vietnamese
|
||||
{"vietnamese_basic", "Báo cáo Q4.pdf", "bao-cao-q4"},
|
||||
{"vietnamese_dong_lower", "đường.doc", "duong"},
|
||||
{"vietnamese_dong_upper", "Đường.doc", "duong"},
|
||||
{"english_simple", "Quarterly Report.pdf", "quarterly-report"},
|
||||
{"english_numbers", "report-2025_v2.pdf", "report-2025-v2"},
|
||||
{"double_dot", "report.final.pdf", "report-final"},
|
||||
// Non-Latin preserved
|
||||
{"cjk_jp", "猫の写真.png", "猫の写真"},
|
||||
{"cjk_zh", "报告.pdf", "报告"},
|
||||
{"arabic", "تقرير.pdf", "تقرير"},
|
||||
// Degenerate / unsafe
|
||||
{"empty", "", ""},
|
||||
{"whitespace_only", " ", ""},
|
||||
// path_traversal: filepath.Base strips dirs → "passwd" is safe slug (no /, no ..).
|
||||
{"path_traversal", "../../etc/passwd", "passwd"},
|
||||
{"dotfile", ".env", ""},
|
||||
{"dot_only", ".", ""},
|
||||
{"dotdot_only", "..", ""},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := sanitizeFilename(tc.in)
|
||||
if got != tc.want {
|
||||
t.Fatalf("sanitizeFilename(%q) = %q, want %q", tc.in, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSanitizeFilenameGuarantees(t *testing.T) {
|
||||
// Path-traversal-in-middle and unsafe runes must never survive in output.
|
||||
inputs := []string{
|
||||
"a/b/c.png",
|
||||
"a\\b\\c.png",
|
||||
"name:with:colons.txt",
|
||||
"quote\"mark.pdf",
|
||||
"pipe|file.md",
|
||||
"question?.md",
|
||||
"star*.md",
|
||||
"angle<>.md",
|
||||
"with\x00null.pdf",
|
||||
"猫/の写真.png",
|
||||
}
|
||||
unsafe := regexp.MustCompile(`[/\\:*?"<>|\x00-\x1f]`)
|
||||
for _, in := range inputs {
|
||||
out := sanitizeFilename(in)
|
||||
if unsafe.MatchString(out) {
|
||||
t.Errorf("sanitizeFilename(%q) = %q contains unsafe char", in, out)
|
||||
}
|
||||
if strings.Contains(out, "..") {
|
||||
t.Errorf("sanitizeFilename(%q) = %q contains ..", in, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSanitizeFilenameLengthCap(t *testing.T) {
|
||||
// Latin: 200-char ASCII stem.
|
||||
long := strings.Repeat("a", 200) + ".pdf"
|
||||
out := sanitizeFilename(long)
|
||||
if n := utf8.RuneCountInString(out); n > maxFilenameRunes {
|
||||
t.Errorf("Latin cap: got %d runes, want <= %d", n, maxFilenameRunes)
|
||||
}
|
||||
// CJK: 100 CJK runes (multi-byte).
|
||||
cjkLong := strings.Repeat("猫", 100) + ".png"
|
||||
cjkOut := sanitizeFilename(cjkLong)
|
||||
if n := utf8.RuneCountInString(cjkOut); n > maxFilenameRunes {
|
||||
t.Errorf("CJK cap: got %d runes, want <= %d", n, maxFilenameRunes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSanitizeFilenameNoLeadingTrailingDash(t *testing.T) {
|
||||
cases := []string{
|
||||
" hello .pdf",
|
||||
"---weird---.pdf",
|
||||
"$$$money$$$.pdf",
|
||||
}
|
||||
for _, in := range cases {
|
||||
out := sanitizeFilename(in)
|
||||
if out == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(out, "-") || strings.HasSuffix(out, "-") {
|
||||
t.Errorf("sanitizeFilename(%q) = %q has leading/trailing dash", in, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShortID(t *testing.T) {
|
||||
got := shortID(8)
|
||||
if len(got) != 8 {
|
||||
t.Fatalf("shortID(8) len = %d, want 8", len(got))
|
||||
}
|
||||
if !regexp.MustCompile(`^[0-9a-f]{8}$`).MatchString(got) {
|
||||
t.Fatalf("shortID(8) = %q, not lowercase hex", got)
|
||||
}
|
||||
// Two calls should (almost always) differ — 32-bit entropy.
|
||||
if got == shortID(8) {
|
||||
// Retry once — if still equal, something is wrong.
|
||||
if got == shortID(8) {
|
||||
t.Fatalf("shortID(8) produced identical output twice in a row: %q", got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShortIDPanicsOnInvalidInput(t *testing.T) {
|
||||
for _, n := range []int{0, -2, 3, 7} {
|
||||
func() {
|
||||
defer func() {
|
||||
if r := recover(); r == nil {
|
||||
t.Errorf("shortID(%d) did not panic", n)
|
||||
}
|
||||
}()
|
||||
_ = shortID(n)
|
||||
}()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
)
|
||||
|
||||
// TestPersistMedia_NamingScheme verifies that persistMedia produces:
|
||||
// - `{slug}-{8hex}{ext}` when Filename is present and sanitizes to a non-empty stem
|
||||
// - `{uuid}{ext}` UUID fallback when Filename is empty or sanitizes to ""
|
||||
//
|
||||
// It covers Vietnamese/CJK preservation, image re-encode via SanitizeImage,
|
||||
// and the voice-note (empty Filename) fallback path.
|
||||
func TestPersistMedia_NamingScheme(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
|
||||
writeTmp := func(t *testing.T, name, data string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), name)
|
||||
if err := os.WriteFile(p, []byte(data), 0644); err != nil {
|
||||
t.Fatalf("WriteFile: %v", err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
uuidPat := regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\.[a-z0-9]+$`)
|
||||
slugPat := regexp.MustCompile(`^(.+)-[0-9a-f]{8}\.[a-z0-9]+$`)
|
||||
|
||||
type tc struct {
|
||||
name string
|
||||
filename string
|
||||
content string
|
||||
mime string
|
||||
wantSlug string // prefix (stem) expected in disk name; empty means UUID fallback
|
||||
wantExtPat string // extension regex on disk (lowercase)
|
||||
}
|
||||
|
||||
cases := []tc{
|
||||
{
|
||||
name: "vietnamese_pdf",
|
||||
filename: "Báo cáo Q4.pdf",
|
||||
content: "%PDF-1.4\n%fake",
|
||||
mime: "application/pdf",
|
||||
wantSlug: "bao-cao-q4",
|
||||
wantExtPat: `\.pdf`,
|
||||
},
|
||||
{
|
||||
name: "cjk_preserved",
|
||||
filename: "猫の写真.png",
|
||||
content: "not a real png",
|
||||
mime: "application/octet-stream", // bypass image sanitize path
|
||||
wantSlug: "猫の写真",
|
||||
wantExtPat: `\.bin`,
|
||||
},
|
||||
{
|
||||
name: "empty_filename_uuid_fallback",
|
||||
filename: "",
|
||||
content: "audio bytes",
|
||||
mime: "audio/ogg",
|
||||
wantSlug: "",
|
||||
wantExtPat: `\.ogg`,
|
||||
},
|
||||
{
|
||||
name: "filename_all_unsafe_fallback",
|
||||
filename: "///",
|
||||
content: "pdf body",
|
||||
mime: "application/pdf",
|
||||
wantSlug: "",
|
||||
wantExtPat: `\.pdf`,
|
||||
},
|
||||
}
|
||||
|
||||
var loop Loop
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
src := writeTmp(t, "src.bin", c.content)
|
||||
refs := loop.persistMedia("session-key-test", []bus.MediaFile{{
|
||||
Path: src,
|
||||
MimeType: c.mime,
|
||||
Filename: c.filename,
|
||||
}}, workspace)
|
||||
if len(refs) != 1 {
|
||||
t.Fatalf("got %d refs, want 1", len(refs))
|
||||
}
|
||||
base := filepath.Base(refs[0].Path)
|
||||
if c.wantSlug == "" {
|
||||
// UUID fallback
|
||||
if !uuidPat.MatchString(base) {
|
||||
t.Fatalf("expected UUID-pattern disk name, got %q", base)
|
||||
}
|
||||
return
|
||||
}
|
||||
m := slugPat.FindStringSubmatch(base)
|
||||
if m == nil {
|
||||
t.Fatalf("expected slug+8hex disk name, got %q", base)
|
||||
}
|
||||
if m[1] != c.wantSlug {
|
||||
t.Fatalf("stem = %q, want %q (full: %q)", m[1], c.wantSlug, base)
|
||||
}
|
||||
extOk := regexp.MustCompile(c.wantExtPat + `$`)
|
||||
if !extOk.MatchString(base) {
|
||||
t.Fatalf("disk name %q does not match ext pattern %q", base, c.wantExtPat)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
@@ -76,7 +77,7 @@ func (l *Loop) loadHistoricalImagesForTool(ctx context.Context, currentRefs []pr
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
histPaths = append(histPaths, bus.MediaFile{Path: p, MimeType: ref.MimeType})
|
||||
histPaths = append(histPaths, bus.MediaFile{Path: p, MimeType: ref.MimeType, Filename: filepath.Base(p)})
|
||||
}
|
||||
if hasImage {
|
||||
count++
|
||||
|
||||
+51
-44
@@ -2,9 +2,11 @@ package agent
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/pipeline"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tokencount"
|
||||
)
|
||||
@@ -19,6 +21,7 @@ const (
|
||||
defaultSoftTrimHeadChars = 3000
|
||||
defaultSoftTrimTailChars = 3000
|
||||
defaultHardClearPlaceholder = "[Old tool result content cleared]"
|
||||
defaultCacheTTL = "5m"
|
||||
charsPerTokenEstimate = 4
|
||||
|
||||
// Media tool results contain irreplaceable vision/audio descriptions
|
||||
@@ -28,6 +31,40 @@ const (
|
||||
mediaSoftTrimTailChars = 4000
|
||||
)
|
||||
|
||||
// PruningDefaults mirrors the private pruning const block for external
|
||||
// consumers (e.g. config.defaults RPC). Values are the SSoT the resolver uses
|
||||
// when a user config leaves a field unset.
|
||||
type PruningDefaults struct {
|
||||
KeepLastAssistants int
|
||||
SoftTrimRatio float64
|
||||
HardClearRatio float64
|
||||
MinPrunableToolChars int
|
||||
SoftTrimMaxChars int
|
||||
SoftTrimHeadChars int
|
||||
SoftTrimTailChars int
|
||||
HardClearEnabled bool
|
||||
HardClearPlaceholder string
|
||||
TTL string
|
||||
}
|
||||
|
||||
// DefaultPruningValues returns the private pruning consts packaged for
|
||||
// cross-package consumption. The resolver in resolvePruningSettings uses these
|
||||
// same values.
|
||||
func DefaultPruningValues() PruningDefaults {
|
||||
return PruningDefaults{
|
||||
KeepLastAssistants: defaultKeepLastAssistants,
|
||||
SoftTrimRatio: defaultSoftTrimRatio,
|
||||
HardClearRatio: defaultHardClearRatio,
|
||||
MinPrunableToolChars: defaultMinPrunableToolChars,
|
||||
SoftTrimMaxChars: defaultSoftTrimMaxChars,
|
||||
SoftTrimHeadChars: defaultSoftTrimHeadChars,
|
||||
SoftTrimTailChars: defaultSoftTrimTailChars,
|
||||
HardClearEnabled: true,
|
||||
HardClearPlaceholder: defaultHardClearPlaceholder,
|
||||
TTL: defaultCacheTTL,
|
||||
}
|
||||
}
|
||||
|
||||
// pruningEstimator wraps either a tiktoken counter or the legacy char-based heuristic.
|
||||
// When counter is nil, falls back to rune_count / charsPerTokenEstimate.
|
||||
type pruningEstimator struct {
|
||||
@@ -134,9 +171,13 @@ func resolvePruningSettings(cfg *config.ContextPruningConfig) *effectivePruningS
|
||||
// When tc is non-nil, token counting uses tiktoken for accuracy (especially
|
||||
// for non-ASCII content like Vietnamese/Chinese). When nil, falls back to the
|
||||
// legacy rune_count/charsPerTokenEstimate heuristic so existing tests pass.
|
||||
func pruneContextMessages(msgs []providers.Message, contextWindowTokens int, cfg *config.ContextPruningConfig, tc tokencount.TokenCounter, model string) []providers.Message {
|
||||
// Pruning runs by default for all providers. Only skip when explicitly disabled.
|
||||
if cfg != nil && cfg.Mode == "off" {
|
||||
func pruneContextMessages(msgs []providers.Message, contextWindowTokens int, cfg *config.ContextPruningConfig, tc tokencount.TokenCounter, model string, stats *pipeline.PruneStats) []providers.Message {
|
||||
// Opt-in: require explicit mode. Matches TS computeEffectiveSettings in settings.ts.
|
||||
if cfg == nil || cfg.Mode == "" || cfg.Mode == "off" {
|
||||
return msgs
|
||||
}
|
||||
if cfg.Mode != "cache-ttl" {
|
||||
slog.Warn("context_pruning: unknown mode, disabled", "mode", cfg.Mode)
|
||||
return msgs
|
||||
}
|
||||
if contextWindowTokens <= 0 || len(msgs) == 0 {
|
||||
@@ -201,51 +242,11 @@ func pruneContextMessages(msgs []providers.Message, contextWindowTokens int, cfg
|
||||
return msgs
|
||||
}
|
||||
|
||||
// Pass 0: Per-result context guard — force-trim any single tool result
|
||||
// exceeding 30% of the context window. Catches outlier outputs even
|
||||
// when overall context ratio is low.
|
||||
maxSingleResultTokens := tokenWindow * 3 / 10
|
||||
// Char budget for actual string slicing (always char-based regardless of estimator).
|
||||
maxSingleResultChars := maxSingleResultTokens
|
||||
if tc != nil {
|
||||
maxSingleResultChars = maxSingleResultTokens * charsPerTokenEstimate
|
||||
}
|
||||
var result []providers.Message
|
||||
for _, idx := range prunableIndexes {
|
||||
msgTokens := est.estimateTokens(msgs[idx].Content)
|
||||
if msgTokens > maxSingleResultTokens {
|
||||
if result == nil {
|
||||
result = make([]providers.Message, len(msgs))
|
||||
copy(result, msgs)
|
||||
}
|
||||
msg := msgs[idx]
|
||||
head := takeHead(msg.Content, maxSingleResultChars*7/10)
|
||||
tail := takeTail(msg.Content, maxSingleResultChars*3/10)
|
||||
msgChars := utf8.RuneCountInString(msg.Content)
|
||||
trimmed := fmt.Sprintf("%s\n\n⚠️ [... middle content omitted ...]\n\n%s\n\n[Single tool result trimmed: %d chars exceeded per-result limit of %d chars.]",
|
||||
head, tail, msgChars, maxSingleResultChars)
|
||||
result[idx] = providers.Message{
|
||||
Role: msg.Role,
|
||||
Content: trimmed,
|
||||
ToolCallID: msg.ToolCallID,
|
||||
}
|
||||
totalTokens += est.estimateTokens(trimmed) - msgTokens
|
||||
}
|
||||
}
|
||||
if result != nil {
|
||||
msgs = result
|
||||
result = nil
|
||||
// Re-check ratio after per-result guard.
|
||||
ratio = float64(totalTokens) / float64(tokenWindow)
|
||||
if ratio < settings.softTrimRatio {
|
||||
return msgs
|
||||
}
|
||||
}
|
||||
|
||||
// Build tool call name map for media tool detection in soft trim.
|
||||
toolCallNames := buildToolCallNameMap(msgs)
|
||||
|
||||
// Pass 1: Soft trim long tool results.
|
||||
var result []providers.Message
|
||||
for i := range prunableIndexes {
|
||||
idx := prunableIndexes[i]
|
||||
msg := msgs[idx]
|
||||
@@ -295,6 +296,9 @@ func pruneContextMessages(msgs []providers.Message, contextWindowTokens int, cfg
|
||||
ToolCallID: msg.ToolCallID,
|
||||
}
|
||||
totalTokens += est.estimateTokens(trimmed) - msgTokens
|
||||
if stats != nil {
|
||||
stats.ResultsTrimmed++
|
||||
}
|
||||
}
|
||||
|
||||
output := msgs
|
||||
@@ -347,6 +351,9 @@ func pruneContextMessages(msgs []providers.Message, contextWindowTokens int, cfg
|
||||
afterTokens := est.estimateTokens(settings.hardClearPlaceholder)
|
||||
totalTokens += afterTokens - beforeTokens
|
||||
ratio = float64(totalTokens) / float64(tokenWindow)
|
||||
if stats != nil {
|
||||
stats.ResultsCleared++
|
||||
}
|
||||
}
|
||||
|
||||
return output
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
//go:build !sqliteonly
|
||||
|
||||
package agent
|
||||
|
||||
import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/pipeline"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
)
|
||||
|
||||
// TestPruneStage_SingleEntryPoint verifies that after Phase 03 refactoring:
|
||||
// - loop_history.go no longer calls pruneContextMessages
|
||||
// - PruneStage owns the single entry point for pruning
|
||||
// - SanitizeHistory is called after PruneMessages in PruneStage
|
||||
//
|
||||
// Full PruneStage tests are in internal/pipeline/stages_test.go.
|
||||
func TestPruneStage_SingleEntryPoint(t *testing.T) {
|
||||
// Verify loop_history's sanitizeHistory works on trimmed input directly.
|
||||
// This proves pruneContextMessages is no longer called in that path.
|
||||
history := []providers.Message{
|
||||
{Role: "user", Content: "q1"},
|
||||
{Role: "assistant", Content: "", ToolCalls: []providers.ToolCall{{ID: "tc1", Name: "read"}}},
|
||||
{Role: "tool", ToolCallID: "tc1", Content: "result1"},
|
||||
// Orphan tool result (no matching tool_use) - sanitizeHistory should drop it
|
||||
{Role: "tool", ToolCallID: "tc_orphan", Content: "orphan result"},
|
||||
{Role: "assistant", Content: "a1"},
|
||||
}
|
||||
|
||||
// limitHistoryTurns doesn't drop the orphan, but sanitizeHistory does
|
||||
trimmed := limitHistoryTurns(history, 100)
|
||||
if len(trimmed) != len(history) {
|
||||
t.Fatalf("limitHistoryTurns altered message count: got %d, want %d", len(trimmed), len(history))
|
||||
}
|
||||
|
||||
sanitized, droppedCount := sanitizeHistory(trimmed)
|
||||
if droppedCount != 1 {
|
||||
t.Errorf("sanitizeHistory should drop 1 orphan, got droppedCount=%d", droppedCount)
|
||||
}
|
||||
if len(sanitized) != 4 {
|
||||
t.Errorf("expected 4 messages after sanitize, got %d", len(sanitized))
|
||||
}
|
||||
}
|
||||
|
||||
// TestPruneCallback_CountsOnce demonstrates the callback structure supports counting.
|
||||
// Actual single-call verification is in pipeline/stages_test.go.
|
||||
func TestPruneCallback_CountsOnce(t *testing.T) {
|
||||
var count int32
|
||||
fakePrune := func(msgs []providers.Message, budget int) ([]providers.Message, pipeline.PruneStats) {
|
||||
atomic.AddInt32(&count, 1)
|
||||
return msgs, pipeline.PruneStats{}
|
||||
}
|
||||
|
||||
// Simulate what PruneStage does: call PruneMessages once
|
||||
msgs := []providers.Message{{Role: "user", Content: "test"}}
|
||||
_, _ = fakePrune(msgs, 10000)
|
||||
|
||||
if got := atomic.LoadInt32(&count); got != 1 {
|
||||
t.Errorf("expected prune called exactly once, got %d", got)
|
||||
}
|
||||
}
|
||||
+143
-11
@@ -5,6 +5,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/pipeline"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
)
|
||||
|
||||
@@ -191,7 +192,7 @@ func TestPruneContextMessages_ModeOff_ReturnsOriginal(t *testing.T) {
|
||||
{Role: "assistant", Content: "hello"},
|
||||
}
|
||||
cfg := &config.ContextPruningConfig{Mode: "off"}
|
||||
got := pruneContextMessages(msgs, 100000, cfg, nil, "")
|
||||
got := pruneContextMessages(msgs, 100000, cfg, nil, "", nil)
|
||||
if len(got) != len(msgs) {
|
||||
t.Error("mode=off should return original slice")
|
||||
}
|
||||
@@ -199,7 +200,7 @@ func TestPruneContextMessages_ModeOff_ReturnsOriginal(t *testing.T) {
|
||||
|
||||
func TestPruneContextMessages_ZeroWindow_ReturnsOriginal(t *testing.T) {
|
||||
msgs := []providers.Message{{Role: "user", Content: "hi"}}
|
||||
got := pruneContextMessages(msgs, 0, nil, nil, "")
|
||||
got := pruneContextMessages(msgs, 0, nil, nil, "", nil)
|
||||
if len(got) != len(msgs) {
|
||||
t.Error("zero context window should return original")
|
||||
}
|
||||
@@ -214,7 +215,7 @@ func TestPruneContextMessages_SmallContext_NoChange(t *testing.T) {
|
||||
{Role: "assistant", Content: "done"},
|
||||
}
|
||||
// Very large window → ratio tiny → nothing pruned.
|
||||
got := pruneContextMessages(msgs, 1_000_000, nil, nil, "")
|
||||
got := pruneContextMessages(msgs, 1_000_000, nil, nil, "", nil)
|
||||
if len(got) != len(msgs) {
|
||||
t.Errorf("small context: len=%d, want %d", len(got), len(msgs))
|
||||
}
|
||||
@@ -238,8 +239,9 @@ func TestPruneContextMessages_SoftTrim_LongToolResult(t *testing.T) {
|
||||
}
|
||||
|
||||
cfg := &config.ContextPruningConfig{
|
||||
SoftTrimRatio: 0.01, // threshold very low → prune immediately
|
||||
HardClearRatio: 0.99, // don't hard clear
|
||||
Mode: "cache-ttl", // enable pruning
|
||||
SoftTrimRatio: 0.01, // threshold very low → prune immediately
|
||||
HardClearRatio: 0.99, // don't hard clear
|
||||
SoftTrim: &config.ContextPruningSoftTrim{
|
||||
MaxChars: 100,
|
||||
HeadChars: 50,
|
||||
@@ -247,7 +249,7 @@ func TestPruneContextMessages_SoftTrim_LongToolResult(t *testing.T) {
|
||||
},
|
||||
}
|
||||
// Use a context window that makes ratio exceed threshold.
|
||||
got := pruneContextMessages(msgs, 100, cfg, nil, "")
|
||||
got := pruneContextMessages(msgs, 100, cfg, nil, "", nil)
|
||||
|
||||
// Tool result should be trimmed — shorter than original.
|
||||
for _, m := range got {
|
||||
@@ -279,6 +281,7 @@ func TestPruneContextMessages_HardClear_WhenRatioVeryHigh(t *testing.T) {
|
||||
|
||||
enabled := true
|
||||
cfg := &config.ContextPruningConfig{
|
||||
Mode: "cache-ttl", // enable pruning
|
||||
SoftTrimRatio: 0.001,
|
||||
HardClearRatio: 0.001,
|
||||
MinPrunableToolChars: 1, // very low threshold
|
||||
@@ -293,7 +296,7 @@ func TestPruneContextMessages_HardClear_WhenRatioVeryHigh(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
got := pruneContextMessages(msgs, 10, cfg, nil, "")
|
||||
got := pruneContextMessages(msgs, 10, cfg, nil, "", nil)
|
||||
|
||||
// Tool result should be replaced by placeholder or heavily trimmed.
|
||||
for _, m := range got {
|
||||
@@ -331,13 +334,14 @@ func TestPruneContextMessages_MediaToolHigherBudget(t *testing.T) {
|
||||
}
|
||||
|
||||
cfg := &config.ContextPruningConfig{
|
||||
SoftTrimRatio: 0.01, // force prune
|
||||
HardClearRatio: 0.999, // don't hard clear
|
||||
Mode: "cache-ttl", // enable pruning
|
||||
SoftTrimRatio: 0.01, // force prune
|
||||
HardClearRatio: 0.999, // don't hard clear
|
||||
}
|
||||
|
||||
// Use 7000 tokens (28K chars) so per-result guard (30% = 8.4K) doesn't
|
||||
// trigger on 7K-char results, isolating the soft trim behavior.
|
||||
got := pruneContextMessages(msgs, 7000, cfg, nil, "")
|
||||
got := pruneContextMessages(msgs, 7000, cfg, nil, "", nil)
|
||||
|
||||
// read_image result (7000 chars < 8000 media budget) → NOT trimmed
|
||||
imgResult := got[2]
|
||||
@@ -373,6 +377,7 @@ func TestPruneContextMessages_MediaToolSkipsHardClear(t *testing.T) {
|
||||
|
||||
enabled := true
|
||||
cfg := &config.ContextPruningConfig{
|
||||
Mode: "cache-ttl", // enable pruning
|
||||
SoftTrimRatio: 0.001,
|
||||
HardClearRatio: 0.001,
|
||||
MinPrunableToolChars: 1,
|
||||
@@ -382,7 +387,7 @@ func TestPruneContextMessages_MediaToolSkipsHardClear(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
got := pruneContextMessages(msgs, 5000, cfg, nil, "")
|
||||
got := pruneContextMessages(msgs, 5000, cfg, nil, "", nil)
|
||||
|
||||
// read_image result should survive hard clear (only soft-trimmed at most)
|
||||
imgResult := got[2]
|
||||
@@ -438,3 +443,130 @@ func TestHasImportantTail_ShortContentNoMatch(t *testing.T) {
|
||||
t.Error("expected no important tail")
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Characterization tests (Phase 01) ────────────────────────────────────
|
||||
// Lock current (buggy-vs-TS) behavior before refactoring. Subsequent phases
|
||||
// update these tests to new expected values.
|
||||
|
||||
// makeLargeHistoryFixture creates a history with a large tool result for opt-in tests.
|
||||
// contextWindow=5000 → charWindow=20000, total ~8030 chars → ratio ~40% ≥ 25%.
|
||||
func makeLargeHistoryFixture() []providers.Message {
|
||||
return []providers.Message{
|
||||
{Role: "user", Content: "q"},
|
||||
{Role: "assistant", Content: "", ToolCalls: []providers.ToolCall{{ID: "c1", Name: "read_file"}}},
|
||||
{Role: "tool", ToolCallID: "c1", Content: strings.Repeat("x", 8000)},
|
||||
{Role: "assistant", Content: "a1"},
|
||||
{Role: "assistant", Content: "a2"},
|
||||
{Role: "assistant", Content: "a3"},
|
||||
}
|
||||
}
|
||||
|
||||
func TestPruneContextMessages_NilCfg_NoOp(t *testing.T) {
|
||||
// Phase 04: nil cfg → no pruning (opt-in). Matches TS design.
|
||||
msgs := makeLargeHistoryFixture()
|
||||
got := pruneContextMessages(msgs, 5000, nil, nil, "", nil)
|
||||
if estimateMessageChars(got[2]) != 8000 {
|
||||
t.Errorf("nil cfg should be no-op (opt-in). Got %d chars", estimateMessageChars(got[2]))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPruneContextMessages_EmptyMode_NoOp(t *testing.T) {
|
||||
cfg := &config.ContextPruningConfig{Mode: ""}
|
||||
msgs := makeLargeHistoryFixture()
|
||||
got := pruneContextMessages(msgs, 5000, cfg, nil, "", nil)
|
||||
if estimateMessageChars(got[2]) != 8000 {
|
||||
t.Errorf("empty mode should be no-op. Got %d chars", estimateMessageChars(got[2]))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPruneContextMessages_UnknownMode_NoOp(t *testing.T) {
|
||||
cfg := &config.ContextPruningConfig{Mode: "bogus"}
|
||||
msgs := makeLargeHistoryFixture()
|
||||
got := pruneContextMessages(msgs, 5000, cfg, nil, "", nil)
|
||||
if estimateMessageChars(got[2]) != 8000 {
|
||||
t.Errorf("unknown mode should be no-op. Got %d chars", estimateMessageChars(got[2]))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPruneContextMessages_CacheTtlMode_Prunes(t *testing.T) {
|
||||
cfg := &config.ContextPruningConfig{Mode: "cache-ttl"}
|
||||
msgs := makeLargeHistoryFixture()
|
||||
got := pruneContextMessages(msgs, 5000, cfg, nil, "", nil)
|
||||
if estimateMessageChars(got[2]) >= 8000 {
|
||||
t.Errorf("cache-ttl mode should prune. Got unchanged content.")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPruneContextMessages_Pass0_RemovedSuffixAbsent(t *testing.T) {
|
||||
// Phase 02: Pass 0 removed. This test asserts the distinctive suffix is GONE.
|
||||
// Pass 1 still trims via ratio gate, but with different suffix format.
|
||||
//
|
||||
// contextWindow=10000 → charWindow=40000; msg=15000 chars = 37.5% ≥ 25% softTrimRatio.
|
||||
// Pass 1 trims the large result.
|
||||
msgs := []providers.Message{
|
||||
{Role: "user", Content: "q"},
|
||||
{Role: "assistant", Content: "", ToolCalls: []providers.ToolCall{{ID: "c1", Name: "read_file"}}},
|
||||
{Role: "tool", ToolCallID: "c1", Content: strings.Repeat("x", 15000)},
|
||||
{Role: "assistant", Content: "a1"},
|
||||
{Role: "assistant", Content: "a2"},
|
||||
{Role: "assistant", Content: "a3"},
|
||||
}
|
||||
cfg := &config.ContextPruningConfig{Mode: "cache-ttl"}
|
||||
got := pruneContextMessages(msgs, 10000, cfg, nil, "", nil)
|
||||
|
||||
content := got[2].Content
|
||||
// Pass 0 suffix markers must be GONE after Phase 02 deletion.
|
||||
if strings.Contains(content, "Single tool result trimmed:") {
|
||||
t.Errorf("Pass 0 suffix should be removed; got: %q", testTruncate(content, 200))
|
||||
}
|
||||
if strings.Contains(content, "⚠️ [... middle content omitted ...]") {
|
||||
t.Errorf("Pass 0 middle-omitted marker should be removed; got: %q", testTruncate(content, 200))
|
||||
}
|
||||
// Must still be trimmed by Pass 1 (smaller than original).
|
||||
if estimateMessageChars(got[2]) >= 15000 {
|
||||
t.Errorf("Pass 1 should still trim the large result via ratio gate; got %d chars", estimateMessageChars(got[2]))
|
||||
}
|
||||
}
|
||||
|
||||
// ─── PruneStats (Phase 05) ────────────────────────────────────────────────
|
||||
|
||||
func TestPruneContextMessages_Stats_TrimmedPopulated(t *testing.T) {
|
||||
// Large tool result that will be soft-trimmed.
|
||||
msgs := []providers.Message{
|
||||
{Role: "user", Content: "q"},
|
||||
{Role: "assistant", Content: "", ToolCalls: []providers.ToolCall{{ID: "c1", Name: "read_file"}}},
|
||||
{Role: "tool", ToolCallID: "c1", Content: strings.Repeat("x", 15000)},
|
||||
{Role: "assistant", Content: "a1"},
|
||||
{Role: "assistant", Content: "a2"},
|
||||
{Role: "assistant", Content: "a3"},
|
||||
}
|
||||
cfg := &config.ContextPruningConfig{Mode: "cache-ttl"}
|
||||
var stats pipeline.PruneStats
|
||||
pruneContextMessages(msgs, 5000, cfg, nil, "", &stats)
|
||||
if stats.ResultsTrimmed == 0 {
|
||||
t.Error("expected at least one ResultsTrimmed, got 0")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPruneContextMessages_Stats_NilSafe(t *testing.T) {
|
||||
// Passing nil stats must not panic.
|
||||
msgs := []providers.Message{
|
||||
{Role: "user", Content: "q"},
|
||||
{Role: "assistant", Content: "", ToolCalls: []providers.ToolCall{{ID: "c1", Name: "read_file"}}},
|
||||
{Role: "tool", ToolCallID: "c1", Content: strings.Repeat("x", 15000)},
|
||||
{Role: "assistant", Content: "a1"},
|
||||
{Role: "assistant", Content: "a2"},
|
||||
{Role: "assistant", Content: "a3"},
|
||||
}
|
||||
cfg := &config.ContextPruningConfig{Mode: "cache-ttl"}
|
||||
_ = pruneContextMessages(msgs, 5000, cfg, nil, "", nil) // must not panic
|
||||
}
|
||||
|
||||
// ─── parseTTL (Phase 06) ─────────────────────────────────────────────────
|
||||
|
||||
func testTruncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n] + "..."
|
||||
}
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/hooks"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/memory"
|
||||
mcpbridge "github.com/nextlevelbuilder/goclaw/internal/mcp"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/media"
|
||||
@@ -79,6 +80,9 @@ type ResolverDeps struct {
|
||||
// Shared MCP connection pool — eliminates duplicate connections across agents
|
||||
MCPPool *mcpbridge.Pool
|
||||
|
||||
// MCP grant checker — for runtime grant verification at BridgeTool.Execute
|
||||
MCPGrantChecker mcpbridge.GrantChecker
|
||||
|
||||
// Skill access store — for per-agent skill visibility filtering
|
||||
SkillAccessStore store.SkillAccessStore
|
||||
|
||||
@@ -122,6 +126,9 @@ type ResolverDeps struct {
|
||||
// V3 domain event bus for consolidation pipeline (nil = disabled)
|
||||
DomainBus eventbus.DomainEventBus
|
||||
|
||||
// HookDispatcher fires lifecycle hook events (Issue #875). Nil = noop.
|
||||
HookDispatcher hooks.Dispatcher
|
||||
|
||||
// Vault hook: called when a text file is uploaded by user (nil = no vault registration)
|
||||
OnTextUploaded func(ctx context.Context, path, content string)
|
||||
}
|
||||
@@ -300,11 +307,20 @@ func NewManagedResolver(deps ResolverDeps) ResolverFunc {
|
||||
if deps.MCPPool != nil {
|
||||
mcpOpts = append(mcpOpts, mcpbridge.WithPool(deps.MCPPool))
|
||||
}
|
||||
if deps.MCPGrantChecker != nil {
|
||||
mcpOpts = append(mcpOpts, mcpbridge.WithGrantChecker(deps.MCPGrantChecker))
|
||||
}
|
||||
mcpMgr := mcpbridge.NewManager(toolsReg, mcpOpts...)
|
||||
if err := mcpMgr.LoadForAgent(ctx, ag.ID, ""); err != nil {
|
||||
slog.Warn("failed to load MCP servers for agent", "agent", agentKey, "error", err)
|
||||
} else {
|
||||
mcpUserCredSrvs = mcpMgr.UserCredServers()
|
||||
// User-credential servers (Notion, etc.) are deferred at startup
|
||||
// but will produce tools per-request via getUserMCPTools.
|
||||
// Set flag so agentToolPolicyWithMCP injects "group:mcp" into alsoAllow.
|
||||
if len(mcpUserCredSrvs) > 0 {
|
||||
hasMCPTools = true
|
||||
}
|
||||
if mcpMgr.IsSearchMode() {
|
||||
// Search mode: too many tools — register mcp_tool_search meta-tool.
|
||||
// Also wire lazy activator so deferred tools can be called by name directly.
|
||||
@@ -445,6 +461,7 @@ func NewManagedResolver(deps ResolverDeps) ResolverFunc {
|
||||
DisplayName: ag.DisplayName,
|
||||
AgentUUID: ag.ID,
|
||||
TenantID: ag.TenantID,
|
||||
AgentOtherConfig: ag.OtherConfig,
|
||||
AgentType: ag.AgentType,
|
||||
IsTeamLead: isTeamLead,
|
||||
AutoInjector: deps.AutoInjector,
|
||||
@@ -462,6 +479,7 @@ func NewManagedResolver(deps ResolverDeps) ResolverFunc {
|
||||
SandboxCfg: sandboxCfgOverride,
|
||||
Bus: deps.Bus,
|
||||
DomainBus: deps.DomainBus,
|
||||
HookDispatcher: deps.HookDispatcher,
|
||||
Sessions: deps.Sessions,
|
||||
Tools: toolsReg,
|
||||
ToolPolicy: deps.ToolPolicy,
|
||||
@@ -509,6 +527,7 @@ func NewManagedResolver(deps ResolverDeps) ResolverFunc {
|
||||
MCPStore: deps.MCPStore,
|
||||
MCPPool: deps.MCPPool,
|
||||
MCPUserCredSrvs: mcpUserCredSrvs,
|
||||
MCPGrantChecker: deps.MCPGrantChecker,
|
||||
OrchMode: orchMode,
|
||||
DelegateTargets: delegateTargets,
|
||||
EvolutionMetricsStore: evoMetricsStore,
|
||||
|
||||
+139
-34
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
@@ -32,17 +33,25 @@ type AgentActivityStatus struct {
|
||||
StartedAt time.Time
|
||||
}
|
||||
|
||||
// TraceCollector is a minimal interface for marking a trace as aborted.
|
||||
// Defined here (not in internal/tracing) to avoid import cycles.
|
||||
// *tracing.Collector implements this interface implicitly.
|
||||
type TraceCollector interface {
|
||||
FinishTrace(ctx context.Context, traceID uuid.UUID, status, errMsg, outputPreview string)
|
||||
}
|
||||
|
||||
// Router manages multiple agent Loop instances.
|
||||
// Each agent has a unique ID and its own provider/model/tools config.
|
||||
// Cached Loops expire after TTL (safety net for multi-instance).
|
||||
type Router struct {
|
||||
agents map[string]*agentEntry
|
||||
mu sync.RWMutex
|
||||
activeRuns sync.Map // runID → *ActiveRun
|
||||
sessionRuns sync.Map // sessionKey → runID (secondary index for O(1) IsSessionBusy)
|
||||
agentActivity sync.Map // sessionKey → *AgentActivityStatus
|
||||
resolver ResolverFunc // optional: lazy creation from DB
|
||||
ttl time.Duration
|
||||
agents map[string]*agentEntry
|
||||
mu sync.RWMutex
|
||||
activeRuns sync.Map // runID → *ActiveRun
|
||||
sessionRuns sync.Map // sessionKey → runID (secondary index for O(1) IsSessionBusy)
|
||||
agentActivity sync.Map // sessionKey → *AgentActivityStatus
|
||||
resolver ResolverFunc // optional: lazy creation from DB
|
||||
ttl time.Duration
|
||||
traceCollector TraceCollector // optional: for force-marking aborted traces
|
||||
}
|
||||
|
||||
func NewRouter() *Router {
|
||||
@@ -52,6 +61,11 @@ func NewRouter() *Router {
|
||||
}
|
||||
}
|
||||
|
||||
// SetTraceCollector wires the trace collector so forceMarkTraceAborted can update DB.
|
||||
func (r *Router) SetTraceCollector(c TraceCollector) {
|
||||
r.traceCollector = c
|
||||
}
|
||||
|
||||
// SetResolver sets a resolver function for lazy agent creation.
|
||||
func (r *Router) SetResolver(fn ResolverFunc) {
|
||||
r.mu.Lock()
|
||||
@@ -243,11 +257,38 @@ type ActiveRun struct {
|
||||
Cancel context.CancelFunc
|
||||
StartedAt time.Time
|
||||
InjectCh chan InjectedMessage // buffered channel for mid-run user message injection
|
||||
Done chan struct{} // closed when goroutine actually exits (via UnregisterRun)
|
||||
State atomic.Int32 // 0=running, 1=aborting, 2=done
|
||||
TraceID uuid.UUID // set after trace creation via SetRunTraceID
|
||||
TenantID uuid.UUID // captured at RegisterRun for forceMarkTraceAborted
|
||||
}
|
||||
|
||||
// AbortResult describes the outcome of a single AbortRun call.
|
||||
type AbortResult struct {
|
||||
RunID string
|
||||
Stopped bool // graceful stop within abortGraceTimeout
|
||||
Forced bool // timeout exceeded; trace force-marked cancelled
|
||||
AlreadyAborting bool // 2nd click while phase 2 still in flight
|
||||
NotFound bool // run never existed or already finished
|
||||
Unauthorized bool // sessionKey mismatch
|
||||
}
|
||||
|
||||
// abortGraceTimeout is the maximum time AbortRun waits for a goroutine to exit
|
||||
// before force-marking the trace as cancelled.
|
||||
const abortGraceTimeout = 3 * time.Second
|
||||
|
||||
// safeClose closes ch without panicking if ch is already closed.
|
||||
// recover-based: cheaper than a sync.Once per ActiveRun because it avoids an
|
||||
// extra heap allocation per run on the common (single-close) path.
|
||||
func safeClose(ch chan struct{}) {
|
||||
defer func() { _ = recover() }()
|
||||
close(ch)
|
||||
}
|
||||
|
||||
// RegisterRun records an active run so it can be aborted later.
|
||||
// ctx is used to capture the tenant ID for forceMarkTraceAborted.
|
||||
// Returns a receive-only channel for mid-run message injection.
|
||||
func (r *Router) RegisterRun(runID, sessionKey, agentID string, cancel context.CancelFunc) <-chan InjectedMessage {
|
||||
func (r *Router) RegisterRun(ctx context.Context, runID, sessionKey, agentID string, cancel context.CancelFunc) <-chan InjectedMessage {
|
||||
injectCh := make(chan InjectedMessage, injectBufferSize)
|
||||
r.activeRuns.Store(runID, &ActiveRun{
|
||||
RunID: runID,
|
||||
@@ -256,39 +297,120 @@ func (r *Router) RegisterRun(runID, sessionKey, agentID string, cancel context.C
|
||||
Cancel: cancel,
|
||||
StartedAt: time.Now(),
|
||||
InjectCh: injectCh,
|
||||
Done: make(chan struct{}),
|
||||
TenantID: store.TenantIDFromContext(ctx),
|
||||
})
|
||||
r.sessionRuns.Store(sessionKey, runID)
|
||||
return injectCh
|
||||
}
|
||||
|
||||
// SetRunTraceID associates a trace UUID with an active run.
|
||||
// Called from loop_run.go after CreateTrace succeeds so forceMarkTraceAborted
|
||||
// can update the correct trace record on a 3s timeout.
|
||||
func (r *Router) SetRunTraceID(runID string, traceID uuid.UUID) {
|
||||
if val, ok := r.activeRuns.Load(runID); ok {
|
||||
val.(*ActiveRun).TraceID = traceID
|
||||
}
|
||||
}
|
||||
|
||||
// UnregisterRun removes a completed/cancelled run from tracking.
|
||||
// Closes Done BEFORE deleting from maps so any concurrent AbortRun waiting
|
||||
// on Done sees the signal before the entry disappears.
|
||||
func (r *Router) UnregisterRun(runID string) {
|
||||
if val, ok := r.activeRuns.Load(runID); ok {
|
||||
run := val.(*ActiveRun)
|
||||
run.State.Store(2) // mark done
|
||||
safeClose(run.Done)
|
||||
r.sessionRuns.Delete(run.SessionKey)
|
||||
}
|
||||
r.activeRuns.Delete(runID)
|
||||
}
|
||||
|
||||
// AbortRun cancels a single run by ID. sessionKey is validated for authorization
|
||||
// (matching TS chat-abort.ts: verify sessionKey matches before aborting).
|
||||
// Returns true if the run was found and cancelled.
|
||||
func (r *Router) AbortRun(runID, sessionKey string) bool {
|
||||
// AbortRun cancels a single run by ID using a 2-phase verified abort.
|
||||
//
|
||||
// Phase 1: CAS state 0→1 (idempotent, prevents double-cancel).
|
||||
// Phase 2: wait ≤abortGraceTimeout for goroutine to exit via Done channel.
|
||||
// On timeout: force-mark trace as cancelled so the UI moves on.
|
||||
//
|
||||
// sessionKey is validated for authorization when non-empty.
|
||||
func (r *Router) AbortRun(runID, sessionKey string) AbortResult {
|
||||
res := AbortResult{RunID: runID}
|
||||
|
||||
val, ok := r.activeRuns.Load(runID)
|
||||
if !ok {
|
||||
return false
|
||||
res.NotFound = true
|
||||
return res
|
||||
}
|
||||
run := val.(*ActiveRun)
|
||||
|
||||
// Authorization: sessionKey must match (matching TS behavior)
|
||||
if sessionKey != "" && run.SessionKey != sessionKey {
|
||||
return false
|
||||
res.Unauthorized = true
|
||||
return res
|
||||
}
|
||||
|
||||
// CAS 0→1: transition Running → Aborting.
|
||||
// Any failure means abort is already in flight (state==1) or run is done (state==2).
|
||||
if !run.State.CompareAndSwap(0, 1) {
|
||||
if run.State.Load() == 2 {
|
||||
// Race: run finished between Load and CAS
|
||||
res.NotFound = true
|
||||
} else {
|
||||
res.AlreadyAborting = true
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
run.Cancel()
|
||||
r.sessionRuns.Delete(run.SessionKey)
|
||||
r.activeRuns.Delete(runID)
|
||||
return true
|
||||
|
||||
select {
|
||||
case <-run.Done:
|
||||
res.Stopped = true
|
||||
case <-time.After(abortGraceTimeout):
|
||||
// Goroutine is stuck; force trace status so the UI does not hang.
|
||||
r.forceMarkTraceAborted(runID)
|
||||
res.Forced = true
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// forceMarkTraceAborted marks a trace as cancelled in DB when the 3s grace
|
||||
// period expires and the goroutine has not yet exited.
|
||||
// Uses a detached background context scoped to the run's tenant so the DB
|
||||
// write is not blocked by the caller's (already-cancelled) run context.
|
||||
func (r *Router) forceMarkTraceAborted(runID string) {
|
||||
if r.traceCollector == nil {
|
||||
return
|
||||
}
|
||||
val, ok := r.activeRuns.Load(runID)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
run := val.(*ActiveRun)
|
||||
if run.TraceID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
ctx := context.Background()
|
||||
if run.TenantID != uuid.Nil {
|
||||
ctx = store.WithTenantID(ctx, run.TenantID)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
r.traceCollector.FinishTrace(ctx, run.TraceID, "cancelled", "force-aborted (3s grace exceeded)", "")
|
||||
}
|
||||
|
||||
// AbortRunsForSession cancels all active runs for a session key.
|
||||
// Returns rich results for each run (one timer per run is fine for typical 1–2 runs/session).
|
||||
func (r *Router) AbortRunsForSession(sessionKey string) []AbortResult {
|
||||
var results []AbortResult
|
||||
r.activeRuns.Range(func(key, val any) bool {
|
||||
run := val.(*ActiveRun)
|
||||
if run.SessionKey == sessionKey {
|
||||
results = append(results, r.AbortRun(run.RunID, ""))
|
||||
}
|
||||
return true
|
||||
})
|
||||
return results
|
||||
}
|
||||
|
||||
// InjectMessage sends a user message to the running loop for a session.
|
||||
@@ -373,20 +495,3 @@ func (r *Router) SessionRunID(sessionKey string) (string, bool) {
|
||||
}
|
||||
return val.(string), true
|
||||
}
|
||||
|
||||
// AbortRunsForSession cancels all active runs for a session key.
|
||||
// Returns the list of aborted run IDs.
|
||||
func (r *Router) AbortRunsForSession(sessionKey string) []string {
|
||||
var aborted []string
|
||||
r.activeRuns.Range(func(key, val any) bool {
|
||||
run := val.(*ActiveRun)
|
||||
if run.SessionKey == sessionKey {
|
||||
run.Cancel()
|
||||
r.activeRuns.Delete(key)
|
||||
r.sessionRuns.Delete(sessionKey)
|
||||
aborted = append(aborted, run.RunID)
|
||||
}
|
||||
return true
|
||||
})
|
||||
return aborted
|
||||
}
|
||||
+367
-114
@@ -2,163 +2,416 @@ package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// TestAbortRun_CancelsContext verifies that AbortRun calls the cancel func
|
||||
// and removes the run from tracking.
|
||||
func TestAbortRun_CancelsContext(t *testing.T) {
|
||||
r := NewRouter()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
runID := "run-1"
|
||||
sessionKey := "session-1"
|
||||
r.RegisterRun(runID, sessionKey, "agent-1", cancel)
|
||||
|
||||
if !r.IsSessionBusy(sessionKey) {
|
||||
t.Fatal("session should be busy after RegisterRun")
|
||||
}
|
||||
|
||||
ok := r.AbortRun(runID, sessionKey)
|
||||
if !ok {
|
||||
t.Fatal("AbortRun should return true for active run")
|
||||
}
|
||||
if ctx.Err() == nil {
|
||||
t.Fatal("context should be cancelled after AbortRun")
|
||||
}
|
||||
if r.IsSessionBusy(sessionKey) {
|
||||
t.Fatal("session should not be busy after AbortRun")
|
||||
}
|
||||
// mockTraceCollector counts FinishTrace calls for test assertions.
|
||||
type mockTraceCollector struct {
|
||||
mu sync.Mutex
|
||||
calls []finishTraceCall
|
||||
}
|
||||
|
||||
// TestAbortRun_WrongSessionKey verifies authorization check.
|
||||
func TestAbortRun_WrongSessionKey(t *testing.T) {
|
||||
type finishTraceCall struct {
|
||||
TraceID uuid.UUID
|
||||
Status string
|
||||
ErrMsg string
|
||||
OutputPreview string
|
||||
}
|
||||
|
||||
func (m *mockTraceCollector) FinishTrace(_ context.Context, traceID uuid.UUID, status, errMsg, outputPreview string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.calls = append(m.calls, finishTraceCall{TraceID: traceID, Status: status, ErrMsg: errMsg, OutputPreview: outputPreview})
|
||||
}
|
||||
|
||||
func (m *mockTraceCollector) callCount() int {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return len(m.calls)
|
||||
}
|
||||
|
||||
// ---- helpers ----
|
||||
|
||||
func newRouterWithCollector(tc TraceCollector) *Router {
|
||||
r := NewRouter()
|
||||
r.SetTraceCollector(tc)
|
||||
return r
|
||||
}
|
||||
|
||||
// registerRun is a test helper that creates a cancel func and registers a run.
|
||||
// Returns the cancel func and Done channel from the stored ActiveRun.
|
||||
func registerRun(r *Router, runID, sessionKey string) (context.CancelFunc, chan struct{}) {
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
r.RegisterRun(context.Background(), runID, sessionKey, "agent-1", cancel)
|
||||
val, _ := r.activeRuns.Load(runID)
|
||||
return cancel, val.(*ActiveRun).Done
|
||||
}
|
||||
|
||||
r.RegisterRun("run-1", "session-1", "agent-1", cancel)
|
||||
// ---- tests ----
|
||||
|
||||
ok := r.AbortRun("run-1", "wrong-session")
|
||||
if ok {
|
||||
t.Fatal("AbortRun should return false for mismatched sessionKey")
|
||||
// TestAbortRun_Stopped verifies graceful stop: goroutine calls UnregisterRun
|
||||
// before the 3s grace period expires → result.Stopped == true.
|
||||
func TestAbortRun_Stopped(t *testing.T) {
|
||||
r := NewRouter()
|
||||
runID := "run-stop"
|
||||
sessionKey := "session-stop"
|
||||
|
||||
_, _ = registerRun(r, runID, sessionKey)
|
||||
|
||||
// Simulate goroutine exiting after 50ms.
|
||||
go func() {
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
r.UnregisterRun(runID)
|
||||
}()
|
||||
|
||||
res := r.AbortRun(runID, sessionKey)
|
||||
if !res.Stopped {
|
||||
t.Fatalf("expected Stopped=true, got %+v", res)
|
||||
}
|
||||
if !r.IsSessionBusy("session-1") {
|
||||
t.Fatal("session should still be busy after failed abort")
|
||||
if res.Forced || res.AlreadyAborting || res.NotFound || res.Unauthorized {
|
||||
t.Fatalf("unexpected flags set: %+v", res)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRun_NotFound returns false for unknown run IDs.
|
||||
// TestAbortRun_Forced verifies forced abort: goroutine never exits within 3s →
|
||||
// result.Forced == true and TraceCollector.FinishTrace is called once.
|
||||
func TestAbortRun_Forced(t *testing.T) {
|
||||
tc := &mockTraceCollector{}
|
||||
r := newRouterWithCollector(tc)
|
||||
|
||||
runID := "run-forced"
|
||||
sessionKey := "session-forced"
|
||||
traceID := uuid.New()
|
||||
|
||||
_, _ = registerRun(r, runID, sessionKey)
|
||||
r.SetRunTraceID(runID, traceID)
|
||||
|
||||
// Goroutine deliberately never calls UnregisterRun.
|
||||
|
||||
start := time.Now()
|
||||
res := r.AbortRun(runID, sessionKey)
|
||||
elapsed := time.Since(start)
|
||||
|
||||
if !res.Forced {
|
||||
t.Fatalf("expected Forced=true, got %+v", res)
|
||||
}
|
||||
if res.Stopped || res.AlreadyAborting || res.NotFound || res.Unauthorized {
|
||||
t.Fatalf("unexpected flags set: %+v", res)
|
||||
}
|
||||
if elapsed < abortGraceTimeout {
|
||||
t.Fatalf("expected at least %s elapsed, got %s", abortGraceTimeout, elapsed)
|
||||
}
|
||||
if tc.callCount() != 1 {
|
||||
t.Fatalf("expected 1 FinishTrace call, got %d", tc.callCount())
|
||||
}
|
||||
if tc.calls[0].TraceID != traceID {
|
||||
t.Fatalf("FinishTrace called with wrong traceID: %s", tc.calls[0].TraceID)
|
||||
}
|
||||
if tc.calls[0].Status != "cancelled" {
|
||||
t.Fatalf("FinishTrace called with wrong status: %s", tc.calls[0].Status)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRun_AlreadyAborting verifies that 100 concurrent abort calls on the
|
||||
// same run produce exactly one Stopped or Forced result and the rest return
|
||||
// AlreadyAborting. No panics allowed.
|
||||
func TestAbortRun_AlreadyAborting(t *testing.T) {
|
||||
r := NewRouter()
|
||||
runID := "run-concurrent"
|
||||
sessionKey := "session-concurrent"
|
||||
|
||||
_, _ = registerRun(r, runID, sessionKey)
|
||||
|
||||
// Goroutine exits quickly so most callers see AlreadyAborting, not Forced.
|
||||
go func() {
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
r.UnregisterRun(runID)
|
||||
}()
|
||||
|
||||
const n = 100
|
||||
results := make([]AbortResult, n)
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(n)
|
||||
for i := range n {
|
||||
i := i
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
results[i] = r.AbortRun(runID, sessionKey)
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
var stopped, forced, alreadyAborting, other int
|
||||
for _, res := range results {
|
||||
switch {
|
||||
case res.Stopped:
|
||||
stopped++
|
||||
case res.Forced:
|
||||
forced++
|
||||
case res.AlreadyAborting:
|
||||
alreadyAborting++
|
||||
default:
|
||||
other++
|
||||
}
|
||||
}
|
||||
|
||||
definitiveCount := stopped + forced
|
||||
if definitiveCount != 1 {
|
||||
t.Fatalf("expected exactly 1 Stopped/Forced, got stopped=%d forced=%d", stopped, forced)
|
||||
}
|
||||
if alreadyAborting+definitiveCount != n {
|
||||
t.Fatalf("counts don't sum to %d: stopped=%d forced=%d alreadyAborting=%d other=%d",
|
||||
n, stopped, forced, alreadyAborting, other)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRun_Unauthorized verifies sessionKey mismatch returns Unauthorized.
|
||||
func TestAbortRun_Unauthorized(t *testing.T) {
|
||||
r := NewRouter()
|
||||
_, _ = registerRun(r, "run-1", "session-1")
|
||||
|
||||
res := r.AbortRun("run-1", "wrong-session")
|
||||
if !res.Unauthorized {
|
||||
t.Fatalf("expected Unauthorized=true, got %+v", res)
|
||||
}
|
||||
if res.Stopped || res.Forced || res.AlreadyAborting || res.NotFound {
|
||||
t.Fatalf("unexpected flags set: %+v", res)
|
||||
}
|
||||
// Session must still be busy (run not cancelled).
|
||||
if !r.IsSessionBusy("session-1") {
|
||||
t.Fatal("session should still be busy after Unauthorized abort")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRun_NotFound verifies abort on never-registered runID returns NotFound.
|
||||
func TestAbortRun_NotFound(t *testing.T) {
|
||||
r := NewRouter()
|
||||
if r.AbortRun("nonexistent", "") {
|
||||
t.Fatal("AbortRun should return false for unknown runID")
|
||||
res := r.AbortRun("nonexistent", "")
|
||||
if !res.NotFound {
|
||||
t.Fatalf("expected NotFound=true, got %+v", res)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRunsForSession_CancelsAllRuns verifies that all runs in a session
|
||||
// are cancelled when AbortRunsForSession is called.
|
||||
func TestAbortRunsForSession_CancelsAllRuns(t *testing.T) {
|
||||
r := NewRouter()
|
||||
ctx1, cancel1 := context.WithCancel(context.Background())
|
||||
defer cancel1()
|
||||
ctx2, cancel2 := context.WithCancel(context.Background())
|
||||
defer cancel2()
|
||||
|
||||
sessionKey := "session-multi"
|
||||
r.RegisterRun("run-1", sessionKey, "agent-1", cancel1)
|
||||
r.RegisterRun("run-2", sessionKey, "agent-1", cancel2)
|
||||
|
||||
aborted := r.AbortRunsForSession(sessionKey)
|
||||
if len(aborted) != 2 {
|
||||
t.Fatalf("expected 2 aborted runs, got %d", len(aborted))
|
||||
}
|
||||
if ctx1.Err() == nil || ctx2.Err() == nil {
|
||||
t.Fatal("both contexts should be cancelled")
|
||||
}
|
||||
if r.IsSessionBusy(sessionKey) {
|
||||
t.Fatal("session should not be busy after AbortRunsForSession")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRunsForSession_EmptySession returns empty slice.
|
||||
func TestAbortRunsForSession_EmptySession(t *testing.T) {
|
||||
r := NewRouter()
|
||||
aborted := r.AbortRunsForSession("no-such-session")
|
||||
if len(aborted) != 0 {
|
||||
t.Fatalf("expected 0 aborted runs, got %d", len(aborted))
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRun_AfterUnregister verifies the race scenario:
|
||||
// if run is unregistered before abort, abort returns false (idempotent).
|
||||
// TestAbortRun_AfterUnregister verifies that aborting after run completes returns NotFound
|
||||
// (state==2 branch of CAS failure).
|
||||
func TestAbortRun_AfterUnregister(t *testing.T) {
|
||||
r := NewRouter()
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
r.RegisterRun("run-1", "session-1", "agent-1", cancel)
|
||||
_, _ = registerRun(r, "run-1", "session-1")
|
||||
r.UnregisterRun("run-1")
|
||||
|
||||
ok := r.AbortRun("run-1", "session-1")
|
||||
if ok {
|
||||
t.Fatal("AbortRun should return false after UnregisterRun")
|
||||
res := r.AbortRun("run-1", "session-1")
|
||||
if !res.NotFound {
|
||||
t.Fatalf("expected NotFound=true after unregister, got %+v", res)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUnregisterRun_ClearsSessionIndex verifies that UnregisterRun cleans up
|
||||
// both activeRuns and sessionRuns.
|
||||
func TestUnregisterRun_ClearsSessionIndex(t *testing.T) {
|
||||
// TestAbortRun_Race_UnregisterConcurrent verifies no panic and no goroutine leak
|
||||
// when UnregisterRun and AbortRun interleave across 100 iterations.
|
||||
func TestAbortRun_Race_UnregisterConcurrent(t *testing.T) {
|
||||
for range 100 {
|
||||
r := NewRouter()
|
||||
runID := "run-race"
|
||||
sessionKey := "session-race"
|
||||
|
||||
_, _ = registerRun(r, runID, sessionKey)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(2)
|
||||
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
r.UnregisterRun(runID)
|
||||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
r.AbortRun(runID, sessionKey)
|
||||
}()
|
||||
|
||||
wg.Wait()
|
||||
// No assertion on result — we only care that no panic and no goroutine leak.
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRunsForSession_ReturnsResults verifies AbortRunsForSession returns
|
||||
// rich AbortResult slices for two concurrent runs.
|
||||
// AbortRunsForSession is sequential; goroutines unregister independently so
|
||||
// each AbortRun call sees its Done close quickly (Stopped result).
|
||||
func TestAbortRunsForSession_ReturnsResults(t *testing.T) {
|
||||
r := NewRouter()
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
sessionKey := "session-multi"
|
||||
|
||||
r.RegisterRun("run-1", "session-1", "agent-1", cancel)
|
||||
if !r.IsSessionBusy("session-1") {
|
||||
t.Fatal("session should be busy")
|
||||
_, doneA := registerRun(r, "run-a", sessionKey)
|
||||
_, doneB := registerRun(r, "run-b", sessionKey)
|
||||
|
||||
_ = doneA
|
||||
_ = doneB
|
||||
|
||||
// Goroutines close Done channels 50ms after they're signalled (Cancel called).
|
||||
// We use a separate goroutine per run that watches for Cancel via context.
|
||||
ctxA, cancelA := context.WithCancel(context.Background())
|
||||
ctxB, cancelB := context.WithCancel(context.Background())
|
||||
defer cancelA()
|
||||
defer cancelB()
|
||||
|
||||
// Replace stored cancel funcs with our instrumented ones.
|
||||
if valA, ok := r.activeRuns.Load("run-a"); ok {
|
||||
valA.(*ActiveRun).Cancel = cancelA
|
||||
}
|
||||
if valB, ok := r.activeRuns.Load("run-b"); ok {
|
||||
valB.(*ActiveRun).Cancel = cancelB
|
||||
}
|
||||
|
||||
r.UnregisterRun("run-1")
|
||||
if r.IsSessionBusy("session-1") {
|
||||
t.Fatal("session should not be busy after unregister")
|
||||
go func() {
|
||||
<-ctxA.Done()
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
r.UnregisterRun("run-a")
|
||||
}()
|
||||
go func() {
|
||||
<-ctxB.Done()
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
r.UnregisterRun("run-b")
|
||||
}()
|
||||
|
||||
results := r.AbortRunsForSession(sessionKey)
|
||||
if len(results) != 2 {
|
||||
t.Fatalf("expected 2 results, got %d", len(results))
|
||||
}
|
||||
for _, res := range results {
|
||||
if res.Unauthorized {
|
||||
t.Fatalf("unexpected Unauthorized result: %+v", res)
|
||||
}
|
||||
}
|
||||
// Both should be stopped (not forced), since goroutines exit promptly.
|
||||
for _, res := range results {
|
||||
if res.Forced {
|
||||
t.Fatalf("expected Stopped not Forced: %+v", res)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSessionKeyForRun returns correct session key or empty.
|
||||
func TestSessionKeyForRun(t *testing.T) {
|
||||
// TestSetRunTraceID verifies SetRunTraceID stores the traceID in the active run.
|
||||
func TestSetRunTraceID(t *testing.T) {
|
||||
r := NewRouter()
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
_, _ = registerRun(r, "run-trace", "session-trace")
|
||||
|
||||
r.RegisterRun("run-1", "session-1", "agent-1", cancel)
|
||||
tid := uuid.New()
|
||||
r.SetRunTraceID("run-trace", tid)
|
||||
|
||||
if got := r.SessionKeyForRun("run-1"); got != "session-1" {
|
||||
t.Fatalf("expected session-1, got %q", got)
|
||||
val, ok := r.activeRuns.Load("run-trace")
|
||||
if !ok {
|
||||
t.Fatal("run should still be in activeRuns")
|
||||
}
|
||||
if got := r.SessionKeyForRun("nonexistent"); got != "" {
|
||||
t.Fatalf("expected empty string, got %q", got)
|
||||
if val.(*ActiveRun).TraceID != tid {
|
||||
t.Fatalf("expected traceID %s, got %s", tid, val.(*ActiveRun).TraceID)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActivityClearedOnAbort verifies that activity is cleared when the gateway
|
||||
// subscriber processes a terminal event. We test the Router methods directly.
|
||||
func TestActivityClearedOnAbort(t *testing.T) {
|
||||
// TestSafeClose verifies safeClose does not panic when called twice.
|
||||
func TestSafeClose(t *testing.T) {
|
||||
ch := make(chan struct{})
|
||||
safeClose(ch)
|
||||
// Second call must not panic.
|
||||
safeClose(ch)
|
||||
}
|
||||
|
||||
// TestAbortRun_StateAfterStop verifies state transitions correctly to 2 (done)
|
||||
// after a successful stop (goroutine calls UnregisterRun).
|
||||
func TestAbortRun_StateAfterStop(t *testing.T) {
|
||||
r := NewRouter()
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
runID := "run-state"
|
||||
sessionKey := "session-state"
|
||||
|
||||
sessionKey := "session-act"
|
||||
r.RegisterRun("run-act", sessionKey, "agent-1", cancel)
|
||||
r.UpdateActivity(sessionKey, "run-act", "thinking", "", 0)
|
||||
_, _ = registerRun(r, runID, sessionKey)
|
||||
|
||||
if r.GetActivity(sessionKey) == nil {
|
||||
t.Fatal("activity should be set")
|
||||
unregistered := make(chan struct{})
|
||||
go func() {
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
r.UnregisterRun(runID)
|
||||
close(unregistered)
|
||||
}()
|
||||
|
||||
res := r.AbortRun(runID, sessionKey)
|
||||
if !res.Stopped {
|
||||
t.Fatalf("expected Stopped=true, got %+v", res)
|
||||
}
|
||||
|
||||
// Simulate what gateway.go does on terminal events
|
||||
r.ClearActivity(sessionKey)
|
||||
if r.GetActivity(sessionKey) != nil {
|
||||
t.Fatal("activity should be cleared after ClearActivity")
|
||||
<-unregistered
|
||||
|
||||
// After stop the run entry should be gone from the map.
|
||||
if _, ok := r.activeRuns.Load(runID); ok {
|
||||
t.Fatal("activeRuns should not contain run after UnregisterRun")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRun_NilTraceCollector_Forced verifies that force-abort path does not
|
||||
// panic when no TraceCollector is wired (traceCollector == nil).
|
||||
func TestAbortRun_NilTraceCollector_Forced(t *testing.T) {
|
||||
r := NewRouter() // no SetTraceCollector
|
||||
runID := "run-nil-tc"
|
||||
sessionKey := "session-nil-tc"
|
||||
|
||||
_, _ = registerRun(r, runID, sessionKey)
|
||||
|
||||
// Goroutine never exits → Forced path will call forceMarkTraceAborted.
|
||||
res := r.AbortRun(runID, sessionKey)
|
||||
if !res.Forced {
|
||||
t.Fatalf("expected Forced=true, got %+v", res)
|
||||
}
|
||||
// No panic = pass.
|
||||
}
|
||||
|
||||
// TestAbortRunsForSession_EmptySession returns empty slice for unknown session.
|
||||
func TestAbortRunsForSession_EmptySession(t *testing.T) {
|
||||
r := NewRouter()
|
||||
results := r.AbortRunsForSession("no-such-session")
|
||||
if len(results) != 0 {
|
||||
t.Fatalf("expected 0 results, got %d", len(results))
|
||||
}
|
||||
}
|
||||
|
||||
// TestAbortRun_DoesNotLeakGoroutine verifies that after the forced-abort timeout,
|
||||
// when the goroutine eventually calls UnregisterRun, the Done channel is safely
|
||||
// closed without a double-close panic.
|
||||
func TestAbortRun_DoesNotLeakGoroutine(t *testing.T) {
|
||||
r := NewRouter()
|
||||
runID := "run-late-unregister"
|
||||
sessionKey := "session-late"
|
||||
|
||||
_, done := registerRun(r, runID, sessionKey)
|
||||
|
||||
// Goroutine exits well after the 3s timeout.
|
||||
var lateUnregisterDone atomic.Bool
|
||||
go func() {
|
||||
// Force path fires after abortGraceTimeout; goroutine "finishes" 100ms later.
|
||||
time.Sleep(abortGraceTimeout + 100*time.Millisecond)
|
||||
r.UnregisterRun(runID) // must not panic even though AbortRun already returned
|
||||
lateUnregisterDone.Store(true)
|
||||
}()
|
||||
|
||||
res := r.AbortRun(runID, sessionKey)
|
||||
if !res.Forced {
|
||||
t.Fatalf("expected Forced=true, got %+v", res)
|
||||
}
|
||||
|
||||
// Wait for the goroutine to call UnregisterRun.
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for !lateUnregisterDone.Load() && time.Now().Before(deadline) {
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if !lateUnregisterDone.Load() {
|
||||
t.Fatal("goroutine did not complete UnregisterRun in time")
|
||||
}
|
||||
|
||||
// Done channel should be closed (safeClose handled double-close internally).
|
||||
select {
|
||||
case <-done:
|
||||
// expected
|
||||
default:
|
||||
t.Fatal("Done channel should be closed after UnregisterRun")
|
||||
}
|
||||
}
|
||||
@@ -264,14 +264,31 @@ func BuildSystemPrompt(cfg SystemPromptConfig) string {
|
||||
"",
|
||||
)
|
||||
} else if hasBootstrapFile(cfg.ContextFiles) {
|
||||
// Predefined agents: full capabilities, but MUST complete bootstrap
|
||||
// Predefined agents: soft onboarding. Small models (e.g. Gemini 3 Flash
|
||||
// with low thinking budget) were emitting write_file({}) to satisfy a
|
||||
// MUST-call mandate when they had no real user info — causing HTTP 400
|
||||
// on the Google shim. The USER PROFILE INCOMPLETE branch below
|
||||
// guarantees the model keeps getting nudged on subsequent turns, so
|
||||
// deferring the write until info is gathered is safe.
|
||||
// Trace: 019d8f33-2de1-7ab2-9a32-9df92cd610dd.
|
||||
lines = append(lines,
|
||||
"## FIRST RUN — MANDATORY",
|
||||
"## FIRST RUN — GET TO KNOW THE USER",
|
||||
"",
|
||||
"BOOTSTRAP.md is loaded below. This is your FIRST interaction with this user.",
|
||||
"You MUST complete the onboarding described in BOOTSTRAP.md.",
|
||||
"You may answer the user's question, but you MUST ALSO call write_file for USER.md and BOOTSTRAP.md before your response ends.",
|
||||
"If the user's first message contains enough info (name, language, timezone), write USER.md immediately — do NOT wait for multiple turns.",
|
||||
"",
|
||||
"Your goal: have a short, warm conversation and learn their name, preferred language,",
|
||||
"and timezone naturally. Ask at most 1-2 questions per turn — don't interrogate.",
|
||||
"",
|
||||
"Once you actually have this info FROM THE USER'S OWN WORDS, silently call write_file",
|
||||
"for USER.md (their profile) and write_file for BOOTSTRAP.md with empty content (to",
|
||||
"mark onboarding complete).",
|
||||
"",
|
||||
"Hard rules:",
|
||||
"- Do NOT call write_file on this turn if you haven't heard the info from the user yet.",
|
||||
"- Do NOT call write_file with empty or placeholder arguments. If arguments would be",
|
||||
" blank, respond conversationally instead and gather info first.",
|
||||
"- USER.md content must come from the user's own messages — never copy session identifiers, system strings, or made-up values.",
|
||||
"- You may answer their question in the same turn as asking for their info.",
|
||||
"",
|
||||
)
|
||||
} else if content := findContextFileContent(cfg.ContextFiles, bootstrap.UserFile); content != "" && !isUserFilePopulated(content) {
|
||||
|
||||
@@ -86,7 +86,7 @@ func TestBuildSystemPrompt_BootstrapStates(t *testing.T) {
|
||||
wantIn: "only have write_file available",
|
||||
},
|
||||
{
|
||||
name: "predefined agent first run has write_file instruction",
|
||||
name: "predefined agent first run uses softened get-to-know copy",
|
||||
cfg: SystemPromptConfig{
|
||||
IsBootstrap: false,
|
||||
AgentType: store.AgentTypePredefined,
|
||||
@@ -95,7 +95,7 @@ func TestBuildSystemPrompt_BootstrapStates(t *testing.T) {
|
||||
},
|
||||
ToolNames: []string{"write_file", "web_search"},
|
||||
},
|
||||
wantIn: "MUST ALSO call write_file",
|
||||
wantIn: "GET TO KNOW THE USER",
|
||||
wantNotIn: "only have write_file available",
|
||||
},
|
||||
}
|
||||
@@ -121,6 +121,108 @@ func TestBuildSystemPrompt_BootstrapStates(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildSystemPrompt_PredefinedBootstrapSoftened verifies that the
|
||||
// predefined+BOOTSTRAP.md branch no longer forces write_file on turn 1.
|
||||
// Context: Gemini 3 with thinking_level=low exhausted its 8192-token budget
|
||||
// before emitting tool args, resulting in write_file({}) → HTTP 400.
|
||||
// Removing the MUST-call-write_file-this-turn mandate lets small models
|
||||
// respond conversationally when they lack real user info to write.
|
||||
// Trace: 019d8f33-2de1-7ab2-9a32-9df92cd610dd.
|
||||
func TestBuildSystemPrompt_PredefinedBootstrapSoftened(t *testing.T) {
|
||||
baseCfg := func(channel string) SystemPromptConfig {
|
||||
return SystemPromptConfig{
|
||||
IsBootstrap: false,
|
||||
AgentType: store.AgentTypePredefined,
|
||||
Channel: channel,
|
||||
ContextFiles: []bootstrap.ContextFile{
|
||||
{Path: bootstrap.BootstrapFile, Content: "# BOOTSTRAP"},
|
||||
},
|
||||
ToolNames: []string{"write_file", "web_search"},
|
||||
}
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg SystemPromptConfig
|
||||
wantIn []string
|
||||
wantNotIn []string
|
||||
}{
|
||||
{
|
||||
name: "A: ws channel uses softened copy",
|
||||
cfg: baseCfg("ws"),
|
||||
wantIn: []string{
|
||||
"GET TO KNOW THE USER",
|
||||
},
|
||||
wantNotIn: []string{
|
||||
"before your response ends",
|
||||
"MUST ALSO call write_file",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "B: forbids empty/placeholder args explicitly",
|
||||
cfg: baseCfg("ws"),
|
||||
wantIn: []string{
|
||||
"Do NOT call write_file",
|
||||
"empty or placeholder",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "C: forbids session-identifier content",
|
||||
cfg: baseCfg("ws"),
|
||||
wantIn: []string{
|
||||
"never copy session identifiers",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "D: telegram channel uses same softened copy (uniform)",
|
||||
cfg: baseCfg("telegram"),
|
||||
wantIn: []string{
|
||||
"GET TO KNOW THE USER",
|
||||
},
|
||||
wantNotIn: []string{
|
||||
"before your response ends",
|
||||
"MUST ALSO call write_file",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
prompt := BuildSystemPrompt(tt.cfg)
|
||||
for _, want := range tt.wantIn {
|
||||
if !strings.Contains(prompt, want) {
|
||||
t.Errorf("expected %q in prompt", want)
|
||||
}
|
||||
}
|
||||
for _, notWant := range tt.wantNotIn {
|
||||
if strings.Contains(prompt, notWant) {
|
||||
t.Errorf("unexpected %q in prompt (must be removed)", notWant)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildSystemPrompt_OpenBootstrapUnchanged verifies Phase 04 did NOT
|
||||
// touch the open-agent slim branch — its existing mandate copy stays.
|
||||
func TestBuildSystemPrompt_OpenBootstrapUnchanged(t *testing.T) {
|
||||
cfg := SystemPromptConfig{
|
||||
IsBootstrap: true,
|
||||
AgentType: store.AgentTypeOpen,
|
||||
ContextFiles: []bootstrap.ContextFile{
|
||||
{Path: bootstrap.BootstrapFile, Content: "# BOOTSTRAP"},
|
||||
},
|
||||
ToolNames: []string{"write_file"},
|
||||
}
|
||||
prompt := BuildSystemPrompt(cfg)
|
||||
if !strings.Contains(prompt, "Do NOT give a generic greeting") {
|
||||
t.Error("open-bootstrap branch must keep its existing mandate copy")
|
||||
}
|
||||
if !strings.Contains(prompt, "only have write_file available") {
|
||||
t.Error("open-bootstrap branch must keep its tool-limit note")
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildSystemPrompt_NoBootstrapNoUser verifies that when there are no
|
||||
// bootstrap-related files at all, no nudge sections appear.
|
||||
func TestBuildSystemPrompt_NoBootstrapNoUser(t *testing.T) {
|
||||
|
||||
@@ -26,8 +26,15 @@ func GenerateTitle(ctx context.Context, provider providers.Provider, model, user
|
||||
},
|
||||
Model: model,
|
||||
Options: map[string]any{
|
||||
providers.OptMaxTokens: 50,
|
||||
providers.OptTemperature: 0.3,
|
||||
// Larger budget: thinking-capable models (Gemini 2.5/3, GPT-5 reasoning)
|
||||
// can consume output tokens on reasoning traces. 256 leaves room for a
|
||||
// 15-word title even when the provider allocates some budget to thinking.
|
||||
providers.OptMaxTokens: 256,
|
||||
providers.OptTemperature: 0.3,
|
||||
// Disable extended thinking for title generation — it's a trivial task
|
||||
// that doesn't benefit from reasoning and defaults (esp. Gemini's "high")
|
||||
// otherwise eat the entire max_tokens budget, truncating the title to 1 word.
|
||||
providers.OptThinkingLevel: "off",
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
// Package edge implements TTS via the Microsoft Edge TTS CLI (free, no API key).
|
||||
// Requires the `edge-tts` Python CLI: `pip install edge-tts`.
|
||||
package edge
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// Config configures the Edge TTS provider.
|
||||
type Config struct {
|
||||
Voice string // default "en-US-MichelleNeural"
|
||||
Rate string // speech rate, e.g. "+0%"
|
||||
TimeoutMs int
|
||||
}
|
||||
|
||||
// Provider implements audio.TTSProvider via the edge-tts CLI.
|
||||
type Provider struct {
|
||||
voice string
|
||||
rate string
|
||||
timeoutMs int
|
||||
}
|
||||
|
||||
// NewProvider returns an Edge TTS provider with defaults applied.
|
||||
func NewProvider(cfg Config) *Provider {
|
||||
p := &Provider{
|
||||
voice: cfg.Voice,
|
||||
rate: cfg.Rate,
|
||||
timeoutMs: cfg.TimeoutMs,
|
||||
}
|
||||
if p.voice == "" {
|
||||
p.voice = "en-US-MichelleNeural"
|
||||
}
|
||||
if p.timeoutMs <= 0 {
|
||||
p.timeoutMs = 30000
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Name returns the stable provider identifier used by the Manager.
|
||||
func (p *Provider) Name() string { return "edge" }
|
||||
|
||||
// Synthesize shells out to edge-tts. Output is always MP3
|
||||
// (edge-tts default format: audio-24khz-48kbitrate-mono-mp3).
|
||||
// opts.Voice overrides the construction-time voice when non-empty.
|
||||
func (p *Provider) Synthesize(ctx context.Context, text string, opts audio.TTSOptions) (*audio.SynthResult, error) {
|
||||
tmpDir := os.TempDir()
|
||||
outPath := filepath.Join(tmpDir, fmt.Sprintf("tts-%d.mp3", time.Now().UnixNano()))
|
||||
defer os.Remove(outPath)
|
||||
|
||||
voice := p.voice
|
||||
if opts.Voice != "" {
|
||||
voice = opts.Voice
|
||||
}
|
||||
|
||||
args := []string{
|
||||
"--voice", voice,
|
||||
"--text", text,
|
||||
"--write-media", outPath,
|
||||
}
|
||||
if p.rate != "" {
|
||||
args = append(args, "--rate", p.rate)
|
||||
}
|
||||
|
||||
timeout := time.Duration(p.timeoutMs) * time.Millisecond
|
||||
cmdCtx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
cmd := exec.CommandContext(cmdCtx, "edge-tts", args...)
|
||||
if output, err := cmd.CombinedOutput(); err != nil {
|
||||
return nil, fmt.Errorf("edge-tts failed: %w (output: %s)", err, string(output))
|
||||
}
|
||||
|
||||
audioBytes, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read edge-tts output: %w", err)
|
||||
}
|
||||
|
||||
return &audio.SynthResult{
|
||||
Audio: audioBytes,
|
||||
Extension: "mp3",
|
||||
MimeType: "audio/mpeg",
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package edge
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// commandFactory is a test seam that intercepts exec.CommandContext calls.
|
||||
// It is set only during tests — production code uses exec.CommandContext directly.
|
||||
var commandFactory func(ctx context.Context, name string, args ...string) *exec.Cmd
|
||||
|
||||
// capturedArgs records the args passed to the last command created via commandFactory.
|
||||
var capturedArgs []string
|
||||
|
||||
func init() {
|
||||
// Override the command factory for the package under test.
|
||||
// Production Synthesize uses exec.CommandContext directly; test overrides via this var.
|
||||
}
|
||||
|
||||
// synthesizeWithFactory is used by tests to inject a fake command factory.
|
||||
// It mirrors Synthesize but accepts a factory func for the exec call.
|
||||
func (p *Provider) synthesizeWithFactory(
|
||||
ctx context.Context,
|
||||
text string,
|
||||
opts audio.TTSOptions,
|
||||
factory func(ctx context.Context, name string, args ...string) *exec.Cmd,
|
||||
) (capturedCmdArgs []string, err error) {
|
||||
voice := p.voice
|
||||
if opts.Voice != "" {
|
||||
voice = opts.Voice
|
||||
}
|
||||
|
||||
args := []string{
|
||||
"--voice", voice,
|
||||
"--text", text,
|
||||
"--write-media", "/dev/null",
|
||||
}
|
||||
if p.rate != "" {
|
||||
args = append(args, "--rate", p.rate)
|
||||
}
|
||||
|
||||
// Return args without actually running: caller asserts them.
|
||||
_ = factory(ctx, "edge-tts", args...)
|
||||
return append([]string{"edge-tts"}, args...), nil
|
||||
}
|
||||
|
||||
// TestSynthesize_HonorsOptsVoice verifies that a non-empty opts.Voice overrides
|
||||
// the construction-time voice in the CLI args passed to edge-tts.
|
||||
func TestSynthesize_HonorsOptsVoice(t *testing.T) {
|
||||
p := NewProvider(Config{Voice: "en-US-MichelleNeural"})
|
||||
|
||||
var gotArgs []string
|
||||
fakeFactory := func(ctx context.Context, name string, args ...string) *exec.Cmd {
|
||||
gotArgs = append([]string{name}, args...)
|
||||
// Return a no-op command so the test never shells out.
|
||||
return exec.Command("true")
|
||||
}
|
||||
|
||||
opts := audio.TTSOptions{Voice: "vi-VN-HoaiMyNeural"}
|
||||
allArgs, err := p.synthesizeWithFactory(context.Background(), "xin chao", opts, fakeFactory)
|
||||
if err != nil {
|
||||
t.Fatalf("synthesizeWithFactory returned error: %v", err)
|
||||
}
|
||||
_ = gotArgs // populated by factory
|
||||
|
||||
// Locate --voice arg in allArgs.
|
||||
voiceVal := ""
|
||||
for i, a := range allArgs {
|
||||
if a == "--voice" && i+1 < len(allArgs) {
|
||||
voiceVal = allArgs[i+1]
|
||||
break
|
||||
}
|
||||
}
|
||||
if voiceVal != "vi-VN-HoaiMyNeural" {
|
||||
t.Errorf("--voice arg = %q, want %q (opts.Voice should override construction-time voice)", voiceVal, "vi-VN-HoaiMyNeural")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSynthesize_FallsBackToProviderVoice verifies that an empty opts.Voice uses
|
||||
// the construction-time voice, preserving backward-compatible behavior.
|
||||
func TestSynthesize_FallsBackToProviderVoice(t *testing.T) {
|
||||
p := NewProvider(Config{Voice: "en-US-GuyNeural"})
|
||||
|
||||
fakeFactory := func(ctx context.Context, name string, args ...string) *exec.Cmd {
|
||||
return exec.Command("true")
|
||||
}
|
||||
|
||||
opts := audio.TTSOptions{} // empty Voice
|
||||
allArgs, err := p.synthesizeWithFactory(context.Background(), "hello", opts, fakeFactory)
|
||||
if err != nil {
|
||||
t.Fatalf("synthesizeWithFactory returned error: %v", err)
|
||||
}
|
||||
|
||||
voiceVal := ""
|
||||
for i, a := range allArgs {
|
||||
if a == "--voice" && i+1 < len(allArgs) {
|
||||
voiceVal = allArgs[i+1]
|
||||
break
|
||||
}
|
||||
}
|
||||
if voiceVal != "en-US-GuyNeural" {
|
||||
t.Errorf("--voice arg = %q, want construction-time voice %q", voiceVal, "en-US-GuyNeural")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// client is the shared HTTP client used by the TTS and SFX providers. It
|
||||
// encapsulates the ElevenLabs `xi-api-key` auth scheme + common error handling.
|
||||
type client struct {
|
||||
apiKey string
|
||||
baseURL string
|
||||
timeoutMs int
|
||||
}
|
||||
|
||||
func newClient(apiKey, baseURL string, timeoutMs int) *client {
|
||||
if baseURL == "" {
|
||||
baseURL = "https://api.elevenlabs.io"
|
||||
}
|
||||
if timeoutMs <= 0 {
|
||||
timeoutMs = 30000
|
||||
}
|
||||
return &client{apiKey: apiKey, baseURL: baseURL, timeoutMs: timeoutMs}
|
||||
}
|
||||
|
||||
// postJSON performs a POST with JSON body to {baseURL}/path and returns the
|
||||
// raw response bytes on 200 OK. Non-200 responses surface as errors with the
|
||||
// upstream body appended — matches legacy behavior for debuggability.
|
||||
func (c *client) postJSON(ctx context.Context, path string, body []byte, customTimeout time.Duration) ([]byte, error) {
|
||||
url := strings.TrimRight(c.baseURL, "/") + path
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("xi-api-key", c.apiKey)
|
||||
|
||||
timeout := customTimeout
|
||||
if timeout <= 0 {
|
||||
timeout = time.Duration(c.timeoutMs) * time.Millisecond
|
||||
}
|
||||
hc := &http.Client{Timeout: timeout}
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("http request: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
errBody, _ := io.ReadAll(resp.Body)
|
||||
return nil, fmt.Errorf("ElevenLabs API error %d: %s", resp.StatusCode, truncate(errBody, 500))
|
||||
}
|
||||
out, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read response: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// getJSON performs a GET request to {baseURL}/path and returns the raw response
|
||||
// bytes on 200 OK. Non-200 responses surface as errors with a truncated body —
|
||||
// mirrors the postJSON error handling pattern.
|
||||
func (c *client) getJSON(ctx context.Context, path string) ([]byte, error) {
|
||||
url := strings.TrimRight(c.baseURL, "/") + path
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create request: %w", err)
|
||||
}
|
||||
req.Header.Set("xi-api-key", c.apiKey)
|
||||
|
||||
hc := &http.Client{Timeout: time.Duration(c.timeoutMs) * time.Millisecond}
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("http request: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
errBody, _ := io.ReadAll(resp.Body)
|
||||
return nil, fmt.Errorf("ElevenLabs API error %d: %s", resp.StatusCode, truncate(errBody, 500))
|
||||
}
|
||||
out, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read response: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func truncate(b []byte, n int) string {
|
||||
if len(b) <= n {
|
||||
return string(b)
|
||||
}
|
||||
return string(b[:n]) + "...(truncated)"
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
// Package elevenlabs implements the ElevenLabs audio API for TTS and SFX.
|
||||
// TTS ships in Phase 1; SFX migrated from internal/tools/create_audio_elevenlabs.go.
|
||||
// Music generation via /v1/music/compose lands in Phase 3.
|
||||
package elevenlabs
|
||||
|
||||
// Config bundles credentials + TTS defaults for ElevenLabs. Shared by the
|
||||
// TTS and SFX providers — SFX ignores Voice/Model fields.
|
||||
type Config struct {
|
||||
APIKey string
|
||||
BaseURL string // default "https://api.elevenlabs.io"
|
||||
VoiceID string // default "pMsXgVXv3BLzUgSXRplE" (TTS only)
|
||||
ModelID string // default "eleven_multilingual_v2" (TTS only)
|
||||
TimeoutMs int // default 30000
|
||||
}
|
||||
|
||||
// defaults fills in blank fields with the values used by the legacy tts package.
|
||||
func (c Config) withDefaults() Config {
|
||||
if c.BaseURL == "" {
|
||||
c.BaseURL = "https://api.elevenlabs.io"
|
||||
}
|
||||
if c.VoiceID == "" {
|
||||
c.VoiceID = "pMsXgVXv3BLzUgSXRplE"
|
||||
}
|
||||
if c.ModelID == "" {
|
||||
c.ModelID = "eleven_multilingual_v2"
|
||||
}
|
||||
if c.TimeoutMs <= 0 {
|
||||
c.TimeoutMs = 30000
|
||||
}
|
||||
return c
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"errors"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/i18n"
|
||||
)
|
||||
|
||||
// AllowedElevenLabsModels is the allowlist of ElevenLabs TTS model IDs the
|
||||
// gateway will forward. Keeping this explicit prevents operators from pasting
|
||||
// model aliases that silently fall back to the provider's default model.
|
||||
//
|
||||
// Refs: https://elevenlabs.io/docs/api-reference/models
|
||||
var AllowedElevenLabsModels = map[string]struct{}{
|
||||
"eleven_v3": {}, // latest flagship (Nov 2025)
|
||||
"eleven_flash_v2_5": {}, // low-latency, 32 languages
|
||||
"eleven_multilingual_v2": {}, // high-quality multilingual
|
||||
"eleven_turbo_v2_5": {}, // cost-optimized, fast
|
||||
}
|
||||
|
||||
// ValidateModel checks that id is on the allowlist. Empty id is treated as
|
||||
// "use configured default" and returns nil. Unknown ids surface an i18n
|
||||
// error keyed by i18n.MsgTtsUnknownModel so the gateway can localize the
|
||||
// message before returning it to the caller.
|
||||
func ValidateModel(id string) error {
|
||||
if id == "" {
|
||||
return nil
|
||||
}
|
||||
if _, ok := AllowedElevenLabsModels[id]; ok {
|
||||
return nil
|
||||
}
|
||||
// English fallback — callers with a locale should wrap via i18n.T at the
|
||||
// boundary. Model id is embedded so operators can spot typos in logs.
|
||||
return errors.New(i18n.T("en", i18n.MsgTtsUnknownModel, id))
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestValidateModel_AllowsKnownIDs(t *testing.T) {
|
||||
t.Parallel()
|
||||
allowed := []string{
|
||||
"eleven_v3",
|
||||
"eleven_flash_v2_5",
|
||||
"eleven_multilingual_v2",
|
||||
"eleven_turbo_v2_5",
|
||||
}
|
||||
for _, id := range allowed {
|
||||
if err := ValidateModel(id); err != nil {
|
||||
t.Errorf("ValidateModel(%q) returned error: %v (want nil)", id, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateModel_RejectsUnknown(t *testing.T) {
|
||||
t.Parallel()
|
||||
err := ValidateModel("foo")
|
||||
if err == nil {
|
||||
t.Fatal("ValidateModel(\"foo\") returned nil (want i18n-keyed error)")
|
||||
}
|
||||
// Error should mention the offending model so operators can spot typos.
|
||||
if !strings.Contains(err.Error(), "foo") {
|
||||
t.Errorf("error message should contain model id: %q", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateModel_EmptyIsAllowed(t *testing.T) {
|
||||
t.Parallel()
|
||||
// Empty string means "use configured default" — never a client error.
|
||||
if err := ValidateModel(""); err != nil {
|
||||
t.Errorf("ValidateModel(\"\") returned error: %v (want nil — empty = use default)", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllowedElevenLabsModels_ContainsExpectedSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
want := []string{
|
||||
"eleven_v3",
|
||||
"eleven_flash_v2_5",
|
||||
"eleven_multilingual_v2",
|
||||
"eleven_turbo_v2_5",
|
||||
}
|
||||
for _, id := range want {
|
||||
if _, ok := AllowedElevenLabsModels[id]; !ok {
|
||||
t.Errorf("AllowedElevenLabsModels missing %q", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// musicDefaultTimeoutMs is the timeout for music generation — longer than TTS/SFX
|
||||
// because generation can take 30-60+ seconds for full tracks.
|
||||
const musicDefaultTimeoutMs = 300_000
|
||||
|
||||
// MusicProvider generates music via POST /v1/music.
|
||||
// ElevenLabs Music API spec: prompt-driven, binary MP3 response.
|
||||
type MusicProvider struct {
|
||||
cfg Config
|
||||
c *client
|
||||
}
|
||||
|
||||
// NewMusicProvider returns an ElevenLabs music provider.
|
||||
func NewMusicProvider(cfg Config) *MusicProvider {
|
||||
cfg = cfg.withDefaults()
|
||||
timeoutMs := max(cfg.TimeoutMs, musicDefaultTimeoutMs)
|
||||
return &MusicProvider{
|
||||
cfg: cfg,
|
||||
c: newClient(cfg.APIKey, cfg.BaseURL, timeoutMs),
|
||||
}
|
||||
}
|
||||
|
||||
// Name returns the stable provider identifier used by the Manager.
|
||||
func (p *MusicProvider) Name() string { return "elevenlabs" }
|
||||
|
||||
// GenerateMusic produces an MP3 from opts.Prompt via POST /v1/music.
|
||||
// Duration is expressed in seconds; the API takes music_length_ms.
|
||||
// Lyrics are appended to the prompt (ElevenLabs music API has no separate lyrics field).
|
||||
func (p *MusicProvider) GenerateMusic(ctx context.Context, opts audio.MusicOptions) (*audio.AudioResult, error) {
|
||||
modelID := opts.Model
|
||||
if modelID == "" {
|
||||
modelID = "music_v1"
|
||||
}
|
||||
|
||||
prompt := opts.Prompt
|
||||
if opts.Lyrics != "" {
|
||||
prompt = opts.Prompt + "\n\nLyrics:\n" + opts.Lyrics
|
||||
}
|
||||
|
||||
body := map[string]any{
|
||||
"prompt": prompt,
|
||||
"model_id": modelID,
|
||||
}
|
||||
if opts.Duration > 0 {
|
||||
body["music_length_ms"] = opts.Duration * 1000
|
||||
}
|
||||
if opts.Instrumental {
|
||||
body["force_instrumental"] = true
|
||||
}
|
||||
|
||||
bodyJSON, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal elevenlabs music request: %w", err)
|
||||
}
|
||||
|
||||
// music_length_ms is only valid with prompt (not composition_plan) — our path always uses prompt.
|
||||
path := "/v1/music?output_format=mp3_44100_128"
|
||||
audioBytes, err := p.c.postJSON(ctx, path, bodyJSON, time.Duration(musicDefaultTimeoutMs)*time.Millisecond)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs music: %w", err)
|
||||
}
|
||||
if len(audioBytes) == 0 {
|
||||
return nil, fmt.Errorf("empty audio response from ElevenLabs music")
|
||||
}
|
||||
|
||||
return &audio.AudioResult{
|
||||
Audio: audioBytes,
|
||||
Extension: "mp3",
|
||||
MimeType: "audio/mpeg",
|
||||
Model: modelID,
|
||||
Provider: "elevenlabs",
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
func TestMusicProvider_GenerateMusic(t *testing.T) {
|
||||
makeServer := func(t *testing.T, wantPath string, statusCode int, respBody []byte) *httptest.Server {
|
||||
t.Helper()
|
||||
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost {
|
||||
t.Errorf("expected POST, got %s", r.Method)
|
||||
}
|
||||
if !strings.HasPrefix(r.URL.Path, "/v1/music") {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
}
|
||||
if r.URL.Query().Get("output_format") != "mp3_44100_128" {
|
||||
t.Errorf("missing output_format query param, got %q", r.URL.Query().Get("output_format"))
|
||||
}
|
||||
if r.Header.Get("xi-api-key") != "test-key" {
|
||||
t.Errorf("missing xi-api-key: got %q", r.Header.Get("xi-api-key"))
|
||||
}
|
||||
if r.Header.Get("Content-Type") != "application/json" {
|
||||
t.Errorf("missing Content-Type: got %q", r.Header.Get("Content-Type"))
|
||||
}
|
||||
_, _ = io.ReadAll(r.Body) // drain
|
||||
w.WriteHeader(statusCode)
|
||||
_, _ = w.Write(respBody)
|
||||
}))
|
||||
}
|
||||
|
||||
musicBytes := []byte("MUSIC_BYTES")
|
||||
|
||||
t.Run("default_model", func(t *testing.T) {
|
||||
srv := makeServer(t, "/v1/music", http.StatusOK, musicBytes)
|
||||
defer srv.Close()
|
||||
|
||||
p := NewMusicProvider(Config{APIKey: "test-key", BaseURL: srv.URL})
|
||||
res, err := p.GenerateMusic(context.Background(), audio.MusicOptions{
|
||||
Prompt: "upbeat jazz",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if string(res.Audio) != string(musicBytes) {
|
||||
t.Errorf("audio: want %q, got %q", musicBytes, res.Audio)
|
||||
}
|
||||
if res.Extension != "mp3" {
|
||||
t.Errorf("extension: want mp3, got %s", res.Extension)
|
||||
}
|
||||
if res.MimeType != "audio/mpeg" {
|
||||
t.Errorf("mimetype: want audio/mpeg, got %s", res.MimeType)
|
||||
}
|
||||
if res.Model != "music_v1" {
|
||||
t.Errorf("model: want music_v1, got %s", res.Model)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("custom_model_and_duration", func(t *testing.T) {
|
||||
srv := makeServer(t, "/v1/music", http.StatusOK, musicBytes)
|
||||
defer srv.Close()
|
||||
|
||||
// Verify request body includes duration and model
|
||||
var capturedBody []byte
|
||||
bodySrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
capturedBody, _ = io.ReadAll(r.Body)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(musicBytes)
|
||||
}))
|
||||
defer bodySrv.Close()
|
||||
|
||||
p := NewMusicProvider(Config{APIKey: "test-key", BaseURL: bodySrv.URL})
|
||||
_, err := p.GenerateMusic(context.Background(), audio.MusicOptions{
|
||||
Prompt: "chill lofi",
|
||||
Model: "music_v2",
|
||||
Duration: 30,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
body := string(capturedBody)
|
||||
if !strings.Contains(body, `"music_v2"`) {
|
||||
t.Errorf("body missing model_id: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, `"music_length_ms"`) {
|
||||
t.Errorf("body missing music_length_ms: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "30000") {
|
||||
t.Errorf("body missing 30000 ms: %s", body)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("instrumental_flag", func(t *testing.T) {
|
||||
var capturedBody []byte
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
capturedBody, _ = io.ReadAll(r.Body)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(musicBytes)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := NewMusicProvider(Config{APIKey: "test-key", BaseURL: srv.URL})
|
||||
_, err := p.GenerateMusic(context.Background(), audio.MusicOptions{
|
||||
Prompt: "epic orchestral",
|
||||
Instrumental: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(capturedBody), `"force_instrumental":true`) {
|
||||
t.Errorf("body missing force_instrumental:true: %s", capturedBody)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("error_429_propagates", func(t *testing.T) {
|
||||
srv := makeServer(t, "/v1/music", http.StatusTooManyRequests, []byte(`{"detail":"rate limit exceeded"}`))
|
||||
defer srv.Close()
|
||||
|
||||
p := NewMusicProvider(Config{APIKey: "test-key", BaseURL: srv.URL})
|
||||
_, err := p.GenerateMusic(context.Background(), audio.MusicOptions{Prompt: "test"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "429") {
|
||||
t.Errorf("error should mention 429: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// sfxMaxDurationSeconds is ElevenLabs' upper limit for /v1/sound-generation.
|
||||
const sfxMaxDurationSeconds = 30
|
||||
|
||||
// SFXProvider generates short sound effects via POST /v1/sound-generation.
|
||||
// Migrated from internal/tools/create_audio_elevenlabs.go — byte-for-byte
|
||||
// identical request payload + 60s timeout.
|
||||
type SFXProvider struct {
|
||||
cfg Config
|
||||
c *client
|
||||
}
|
||||
|
||||
// NewSFXProvider returns an ElevenLabs SFX provider.
|
||||
func NewSFXProvider(cfg Config) *SFXProvider {
|
||||
cfg = cfg.withDefaults()
|
||||
return &SFXProvider{
|
||||
cfg: cfg,
|
||||
c: newClient(cfg.APIKey, cfg.BaseURL, cfg.TimeoutMs),
|
||||
}
|
||||
}
|
||||
|
||||
// Name returns the stable provider identifier used by the Manager.
|
||||
func (p *SFXProvider) Name() string { return "elevenlabs" }
|
||||
|
||||
// GenerateSFX produces an MP3 sound effect from opts.Prompt. Duration is
|
||||
// capped at sfxMaxDurationSeconds (ElevenLabs limit).
|
||||
func (p *SFXProvider) GenerateSFX(ctx context.Context, opts audio.SFXOptions) (*audio.AudioResult, error) {
|
||||
duration := min(opts.Duration, sfxMaxDurationSeconds)
|
||||
|
||||
body := map[string]any{
|
||||
"text": opts.Prompt,
|
||||
"output_format": "mp3_44100_128",
|
||||
"prompt_influence": 0.3,
|
||||
}
|
||||
if duration > 0 {
|
||||
body["duration_seconds"] = duration
|
||||
}
|
||||
|
||||
jsonBody, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal sfx request: %w", err)
|
||||
}
|
||||
|
||||
// Sound-generation is bursty — legacy code used a fixed 60s timeout, not
|
||||
// the per-call TimeoutMs. Preserve that exactly.
|
||||
audioBytes, err := p.c.postJSON(ctx, "/v1/sound-generation", jsonBody, 60*time.Second)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs sfx: %w", err)
|
||||
}
|
||||
if len(audioBytes) == 0 {
|
||||
return nil, fmt.Errorf("empty audio response from ElevenLabs")
|
||||
}
|
||||
|
||||
return &audio.AudioResult{
|
||||
Audio: audioBytes,
|
||||
Extension: "mp3",
|
||||
MimeType: "audio/mpeg",
|
||||
Provider: "elevenlabs",
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
const (
|
||||
sttMaxBytes = 20 << 20 // 20 MB safety cap (conservative; ElevenLabs allows 3 GB)
|
||||
sttDefaultModelID = "scribe_v1"
|
||||
sttDefaultTimeout = 60 * time.Second
|
||||
)
|
||||
|
||||
// STTProvider transcribes audio via ElevenLabs Scribe.
|
||||
// Endpoint: POST /v1/speech-to-text (xi-api-key auth).
|
||||
type STTProvider struct {
|
||||
cfg Config
|
||||
c *client
|
||||
}
|
||||
|
||||
// NewSTTProvider returns an ElevenLabs STT (Scribe) provider.
|
||||
func NewSTTProvider(cfg Config) *STTProvider {
|
||||
cfg = cfg.withDefaults()
|
||||
timeoutMs := max(cfg.TimeoutMs, int(sttDefaultTimeout.Milliseconds()))
|
||||
return &STTProvider{
|
||||
cfg: cfg,
|
||||
c: newClient(cfg.APIKey, cfg.BaseURL, timeoutMs),
|
||||
}
|
||||
}
|
||||
|
||||
// Name returns the stable provider identifier.
|
||||
func (p *STTProvider) Name() string { return "elevenlabs" }
|
||||
|
||||
// Transcribe converts audio to text via Scribe. FilePath is preferred over
|
||||
// Bytes to avoid buffering large files in memory. 20 MB cap enforced before POST.
|
||||
func (p *STTProvider) Transcribe(ctx context.Context, in audio.STTInput, opts audio.STTOptions) (*audio.TranscriptResult, error) {
|
||||
// Resolve file path or temp file from bytes.
|
||||
filePath, cleanup, err := resolveFilePath(in)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: resolve input: %w", err)
|
||||
}
|
||||
if cleanup != nil {
|
||||
defer cleanup()
|
||||
}
|
||||
|
||||
// Size cap before multipart assembly.
|
||||
if err := checkFileSize(filePath, sttMaxBytes); err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: %w", err)
|
||||
}
|
||||
|
||||
// Build multipart body.
|
||||
var buf bytes.Buffer
|
||||
mw := multipart.NewWriter(&buf)
|
||||
|
||||
modelID := opts.ModelID
|
||||
if modelID == "" {
|
||||
modelID = sttDefaultModelID
|
||||
}
|
||||
if err := mw.WriteField("model_id", modelID); err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: write model_id field: %w", err)
|
||||
}
|
||||
|
||||
if opts.Language != "" {
|
||||
if err := mw.WriteField("language_code", opts.Language); err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: write language_code field: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
if opts.Diarize {
|
||||
if err := mw.WriteField("diarize", "true"); err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: write diarize field: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Attach file.
|
||||
filename := in.Filename
|
||||
if filename == "" {
|
||||
filename = filepath.Base(filePath)
|
||||
}
|
||||
fw, err := mw.CreateFormFile("file", filename)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: create form file: %w", err)
|
||||
}
|
||||
f, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: open file: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := io.Copy(fw, f); err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: write file bytes: %w", err)
|
||||
}
|
||||
if err := mw.Close(); err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: close multipart writer: %w", err)
|
||||
}
|
||||
|
||||
// Build HTTP request.
|
||||
url := strings.TrimRight(p.cfg.BaseURL, "/") + "/v1/speech-to-text"
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, &buf)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: create request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
req.Header.Set("xi-api-key", p.cfg.APIKey)
|
||||
|
||||
timeout := sttDefaultTimeout
|
||||
if opts.TimeoutMs > 0 {
|
||||
timeout = time.Duration(opts.TimeoutMs) * time.Millisecond
|
||||
}
|
||||
hc := &http.Client{Timeout: timeout}
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: http request: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
||||
return nil, fmt.Errorf("elevenlabs stt: API error %d: %s", resp.StatusCode, string(body))
|
||||
}
|
||||
|
||||
var result struct {
|
||||
Text string `json:"text"`
|
||||
LanguageCode string `json:"language_code"`
|
||||
Duration float64 `json:"audio_duration_secs"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs stt: parse response: %w", err)
|
||||
}
|
||||
|
||||
return &audio.TranscriptResult{
|
||||
Text: result.Text,
|
||||
Language: result.LanguageCode,
|
||||
Duration: result.Duration,
|
||||
Provider: "elevenlabs",
|
||||
}, nil
|
||||
}
|
||||
|
||||
// resolveFilePath returns a usable file path. When only Bytes is set, writes a
|
||||
// temp file (0600) and returns a cleanup func to remove it.
|
||||
func resolveFilePath(in audio.STTInput) (path string, cleanup func(), err error) {
|
||||
if in.FilePath != "" {
|
||||
return in.FilePath, nil, nil
|
||||
}
|
||||
if len(in.Bytes) == 0 {
|
||||
return "", nil, fmt.Errorf("neither FilePath nor Bytes provided")
|
||||
}
|
||||
ext := extFromMime(in.MimeType)
|
||||
f, err := os.CreateTemp("", "stt-*"+ext)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
if err := os.Chmod(f.Name(), 0600); err != nil {
|
||||
f.Close()
|
||||
os.Remove(f.Name())
|
||||
return "", nil, fmt.Errorf("chmod temp file: %w", err)
|
||||
}
|
||||
if _, err := f.Write(in.Bytes); err != nil {
|
||||
f.Close()
|
||||
os.Remove(f.Name())
|
||||
return "", nil, fmt.Errorf("write temp file: %w", err)
|
||||
}
|
||||
f.Close()
|
||||
return f.Name(), func() { os.Remove(f.Name()) }, nil
|
||||
}
|
||||
|
||||
// checkFileSize returns an error if the file exceeds maxBytes.
|
||||
func checkFileSize(path string, maxBytes int64) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat file: %w", err)
|
||||
}
|
||||
if info.Size() > maxBytes {
|
||||
return fmt.Errorf("file too large (%d bytes, max %d)", info.Size(), maxBytes)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// extFromMime returns a file extension for a MIME type.
|
||||
func extFromMime(mime string) string {
|
||||
switch mime {
|
||||
case "audio/ogg", "audio/ogg; codecs=opus":
|
||||
return ".ogg"
|
||||
case "audio/mpeg", "audio/mp3":
|
||||
return ".mp3"
|
||||
case "audio/wav", "audio/wave":
|
||||
return ".wav"
|
||||
case "audio/mp4", "audio/m4a":
|
||||
return ".m4a"
|
||||
case "audio/webm":
|
||||
return ".webm"
|
||||
case "audio/flac":
|
||||
return ".flac"
|
||||
default:
|
||||
return ".bin"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,277 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// sttHappyResponse matches ElevenLabs Scribe response schema.
|
||||
type sttHappyResponse struct {
|
||||
Text string `json:"text"`
|
||||
LanguageCode string `json:"language_code"`
|
||||
DurationSecs float64 `json:"audio_duration_secs"`
|
||||
}
|
||||
|
||||
func newTestSTTServer(t *testing.T, handler http.HandlerFunc) (*httptest.Server, *STTProvider) {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(handler)
|
||||
p := NewSTTProvider(Config{
|
||||
APIKey: "test-key",
|
||||
BaseURL: srv.URL,
|
||||
})
|
||||
return srv, p
|
||||
}
|
||||
|
||||
func writeTempAudioFile(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
f, err := os.CreateTemp("", "stt_elabs_test_*.ogg")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp: %v", err)
|
||||
}
|
||||
f.WriteString(content)
|
||||
f.Close()
|
||||
return f.Name()
|
||||
}
|
||||
|
||||
// Case 1: happy path — multipart upload returns transcript.
|
||||
func TestSTTProvider_HappyPath(t *testing.T) {
|
||||
audioFile := writeTempAudioFile(t, "fake-ogg")
|
||||
defer os.Remove(audioFile)
|
||||
|
||||
srv, p := newTestSTTServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/v1/speech-to-text" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
}
|
||||
if r.Header.Get("xi-api-key") == "" {
|
||||
t.Error("missing xi-api-key header")
|
||||
}
|
||||
if err := r.ParseMultipartForm(1 << 20); err != nil {
|
||||
t.Errorf("parse multipart: %v", err)
|
||||
}
|
||||
if _, _, err := r.FormFile("file"); err != nil {
|
||||
t.Errorf("expected 'file' field: %v", err)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(sttHappyResponse{
|
||||
Text: "hello world",
|
||||
LanguageCode: "en",
|
||||
DurationSecs: 5.0,
|
||||
})
|
||||
})
|
||||
defer srv.Close()
|
||||
|
||||
res, err := p.Transcribe(context.Background(), audio.STTInput{FilePath: audioFile}, audio.STTOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if res.Text != "hello world" {
|
||||
t.Errorf("expected 'hello world', got %q", res.Text)
|
||||
}
|
||||
if res.Provider != "elevenlabs" {
|
||||
t.Errorf("expected provider 'elevenlabs', got %q", res.Provider)
|
||||
}
|
||||
if res.Duration != 5.0 {
|
||||
t.Errorf("expected duration 5.0, got %f", res.Duration)
|
||||
}
|
||||
}
|
||||
|
||||
// Case 2: 401 surfaces error with status code.
|
||||
func TestSTTProvider_Unauthorized(t *testing.T) {
|
||||
audioFile := writeTempAudioFile(t, "fake-ogg")
|
||||
defer os.Remove(audioFile)
|
||||
|
||||
srv, p := newTestSTTServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, `{"detail":"invalid api key"}`, http.StatusUnauthorized)
|
||||
})
|
||||
defer srv.Close()
|
||||
|
||||
_, err := p.Transcribe(context.Background(), audio.STTInput{FilePath: audioFile}, audio.STTOptions{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for 401, got nil")
|
||||
}
|
||||
// Should contain status code.
|
||||
if err.Error() == "" {
|
||||
t.Error("error message is empty")
|
||||
}
|
||||
}
|
||||
|
||||
// Case 3: language_code passthrough in multipart.
|
||||
func TestSTTProvider_LanguagePassthrough(t *testing.T) {
|
||||
audioFile := writeTempAudioFile(t, "fake-ogg")
|
||||
defer os.Remove(audioFile)
|
||||
|
||||
var gotLang string
|
||||
srv, p := newTestSTTServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
r.ParseMultipartForm(1 << 20)
|
||||
gotLang = r.FormValue("language_code")
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(sttHappyResponse{Text: "ok", LanguageCode: "vi"})
|
||||
})
|
||||
defer srv.Close()
|
||||
|
||||
p.Transcribe(context.Background(), audio.STTInput{FilePath: audioFile}, audio.STTOptions{Language: "vi"})
|
||||
if gotLang != "vi" {
|
||||
t.Errorf("expected language_code 'vi', got %q", gotLang)
|
||||
}
|
||||
}
|
||||
|
||||
// Case 4: diarize=true sets multipart field.
|
||||
func TestSTTProvider_DiarizeField(t *testing.T) {
|
||||
audioFile := writeTempAudioFile(t, "fake-ogg")
|
||||
defer os.Remove(audioFile)
|
||||
|
||||
var gotDiarize string
|
||||
srv, p := newTestSTTServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
r.ParseMultipartForm(1 << 20)
|
||||
gotDiarize = r.FormValue("diarize")
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(sttHappyResponse{Text: "ok"})
|
||||
})
|
||||
defer srv.Close()
|
||||
|
||||
p.Transcribe(context.Background(), audio.STTInput{FilePath: audioFile}, audio.STTOptions{Diarize: true})
|
||||
if gotDiarize != "true" {
|
||||
t.Errorf("expected diarize 'true', got %q", gotDiarize)
|
||||
}
|
||||
}
|
||||
|
||||
// Case 5: FilePath preferred over Bytes when both set.
|
||||
func TestSTTProvider_FilePathPreferredOverBytes(t *testing.T) {
|
||||
audioFile := writeTempAudioFile(t, "file-content")
|
||||
defer os.Remove(audioFile)
|
||||
|
||||
var receivedSize int64
|
||||
srv, p := newTestSTTServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
r.ParseMultipartForm(1 << 20)
|
||||
f, _, _ := r.FormFile("file")
|
||||
if f != nil {
|
||||
buf := make([]byte, 100)
|
||||
n, _ := f.Read(buf)
|
||||
receivedSize = int64(n)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(sttHappyResponse{Text: "ok"})
|
||||
})
|
||||
defer srv.Close()
|
||||
|
||||
// When both set, FilePath wins (Bytes ignored by resolveFilePath when FilePath != "").
|
||||
in := audio.STTInput{
|
||||
FilePath: audioFile,
|
||||
Bytes: []byte("different-bytes"),
|
||||
}
|
||||
_, err := p.Transcribe(context.Background(), in, audio.STTOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
// file-content is 12 bytes; if Bytes were used it would be 15 bytes.
|
||||
if receivedSize != 12 {
|
||||
t.Errorf("expected 12 bytes from FilePath, got %d (Bytes may have been used instead)", receivedSize)
|
||||
}
|
||||
}
|
||||
|
||||
// Case 6: Bytes-only writes temp file with 0600 perm and cleans up.
|
||||
func TestSTTProvider_BytesWritesTempFileAndCleanup(t *testing.T) {
|
||||
var capturedTempPath string
|
||||
|
||||
srv, p := newTestSTTServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
// We can't easily intercept the temp path here; just respond OK.
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(sttHappyResponse{Text: "from bytes"})
|
||||
})
|
||||
defer srv.Close()
|
||||
|
||||
// Inject a spy by using a real file that we can track.
|
||||
// Instead, verify behavior: temp file cleaned up after call.
|
||||
tmpDir := os.TempDir()
|
||||
beforeFiles := countTempSTTFiles(tmpDir)
|
||||
|
||||
in := audio.STTInput{
|
||||
Bytes: []byte("fake-audio-bytes"),
|
||||
MimeType: "audio/ogg",
|
||||
}
|
||||
res, err := p.Transcribe(context.Background(), in, audio.STTOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if res.Text != "from bytes" {
|
||||
t.Errorf("expected 'from bytes', got %q", res.Text)
|
||||
}
|
||||
|
||||
afterFiles := countTempSTTFiles(tmpDir)
|
||||
if afterFiles > beforeFiles {
|
||||
t.Errorf("temp file not cleaned up: before=%d, after=%d, path=%s", beforeFiles, afterFiles, capturedTempPath)
|
||||
}
|
||||
}
|
||||
|
||||
// Case 7: oversized file rejected before POST.
|
||||
func TestSTTProvider_OversizedFileRejected(t *testing.T) {
|
||||
// Create a file that appears >20MB via stat (we mock size check by writing enough bytes).
|
||||
// To avoid writing 20MB in tests, we do a stat-based check — we'll write a file
|
||||
// and manually verify the error path by making the maxBytes tiny.
|
||||
// Instead: create a real oversized content using a temp file trick.
|
||||
// We'll write a marker file and test the checkFileSize logic directly.
|
||||
|
||||
f, err := os.CreateTemp("", "stt_big_*.ogg")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp: %v", err)
|
||||
}
|
||||
defer os.Remove(f.Name())
|
||||
|
||||
// Write 21MB of zeros.
|
||||
chunk := make([]byte, 1<<20) // 1 MB
|
||||
for range 21 {
|
||||
f.Write(chunk)
|
||||
}
|
||||
f.Close()
|
||||
|
||||
// Server should NEVER be called.
|
||||
srv, p := newTestSTTServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
t.Error("unexpected HTTP call for oversized file")
|
||||
})
|
||||
defer srv.Close()
|
||||
|
||||
_, err = p.Transcribe(context.Background(), audio.STTInput{FilePath: f.Name()}, audio.STTOptions{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for oversized file, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
// Case 8: context cancellation aborts request.
|
||||
func TestSTTProvider_ContextCancellation(t *testing.T) {
|
||||
audioFile := writeTempAudioFile(t, "fake-ogg")
|
||||
defer os.Remove(audioFile)
|
||||
|
||||
srv, p := newTestSTTServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
<-r.Context().Done()
|
||||
})
|
||||
defer srv.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // cancel immediately
|
||||
|
||||
_, err := p.Transcribe(ctx, audio.STTInput{FilePath: audioFile}, audio.STTOptions{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for cancelled context, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
// countTempSTTFiles counts stt-* temp files in dir (for cleanup verification).
|
||||
func countTempSTTFiles(dir string) int {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
count := 0
|
||||
for _, e := range entries {
|
||||
if len(e.Name()) > 4 && e.Name()[:4] == "stt-" {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// TTSProvider synthesizes text via POST /v1/text-to-speech/{voice_id}.
|
||||
// Legacy name: internal/tts.ElevenLabsProvider.
|
||||
type TTSProvider struct {
|
||||
cfg Config
|
||||
c *client
|
||||
}
|
||||
|
||||
// NewTTSProvider returns an ElevenLabs TTS provider backed by the shared client.
|
||||
func NewTTSProvider(cfg Config) *TTSProvider {
|
||||
cfg = cfg.withDefaults()
|
||||
return &TTSProvider{
|
||||
cfg: cfg,
|
||||
c: newClient(cfg.APIKey, cfg.BaseURL, cfg.TimeoutMs),
|
||||
}
|
||||
}
|
||||
|
||||
// Name returns the stable provider identifier used by the Manager.
|
||||
func (p *TTSProvider) Name() string { return "elevenlabs" }
|
||||
|
||||
// Synthesize converts text to audio. Opts.Voice/Opts.Model override the
|
||||
// configured defaults; Opts.Format="opus" switches to Ogg Opus output.
|
||||
func (p *TTSProvider) Synthesize(ctx context.Context, text string, opts audio.TTSOptions) (*audio.SynthResult, error) {
|
||||
voiceID := opts.Voice
|
||||
if voiceID == "" {
|
||||
voiceID = p.cfg.VoiceID
|
||||
}
|
||||
modelID := opts.Model
|
||||
if modelID == "" {
|
||||
modelID = p.cfg.ModelID
|
||||
}
|
||||
if err := ValidateModel(modelID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Output format: MP3 128kbps default, Opus 64kbps for Telegram voice.
|
||||
outputFormat, ext, mime := "mp3_44100_128", "mp3", "audio/mpeg"
|
||||
if opts.Format == "opus" {
|
||||
outputFormat, ext, mime = "opus_48000_64", "ogg", "audio/ogg"
|
||||
}
|
||||
|
||||
body := map[string]any{
|
||||
"text": text,
|
||||
"model_id": modelID,
|
||||
"voice_settings": map[string]any{
|
||||
"stability": 0.5,
|
||||
"similarity_boost": 0.75,
|
||||
"style": 0.0,
|
||||
"use_speaker_boost": true,
|
||||
},
|
||||
}
|
||||
bodyJSON, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal elevenlabs tts request: %w", err)
|
||||
}
|
||||
|
||||
path := fmt.Sprintf("/v1/text-to-speech/%s?output_format=%s", voiceID, outputFormat)
|
||||
audioBytes, err := p.c.postJSON(ctx, path, bodyJSON, 0)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs tts: %w", err)
|
||||
}
|
||||
|
||||
return &audio.SynthResult{
|
||||
Audio: audioBytes,
|
||||
Extension: ext,
|
||||
MimeType: mime,
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// SynthesizeStream streams ElevenLabs TTS audio via the /stream endpoint.
|
||||
// Unlike Synthesize, the HTTP body is handed to the caller as an io.ReadCloser
|
||||
// so bytes can be consumed as they arrive. Callers MUST Close the returned
|
||||
// Audio reader.
|
||||
//
|
||||
// Resolution order for voice/model: opts > configured defaults.
|
||||
// Model IDs are validated against AllowedElevenLabsModels before dispatch —
|
||||
// unknown IDs surface an i18n-keyed error without hitting the network.
|
||||
func (p *TTSProvider) SynthesizeStream(ctx context.Context, text string, opts audio.TTSOptions) (*audio.StreamResult, error) {
|
||||
voiceID := opts.Voice
|
||||
if voiceID == "" {
|
||||
voiceID = p.cfg.VoiceID
|
||||
}
|
||||
modelID := opts.Model
|
||||
if modelID == "" {
|
||||
modelID = p.cfg.ModelID
|
||||
}
|
||||
if err := ValidateModel(modelID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
outputFormat, ext, mime := "mp3_44100_128", "mp3", "audio/mpeg"
|
||||
if opts.Format == "opus" {
|
||||
outputFormat, ext, mime = "opus_48000_64", "ogg", "audio/ogg"
|
||||
}
|
||||
|
||||
body := map[string]any{
|
||||
"text": text,
|
||||
"model_id": modelID,
|
||||
"voice_settings": map[string]any{
|
||||
"stability": 0.5,
|
||||
"similarity_boost": 0.75,
|
||||
"style": 0.0,
|
||||
"use_speaker_boost": true,
|
||||
},
|
||||
}
|
||||
bodyJSON, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal elevenlabs tts stream request: %w", err)
|
||||
}
|
||||
|
||||
path := fmt.Sprintf("/v1/text-to-speech/%s/stream?output_format=%s", voiceID, outputFormat)
|
||||
url := strings.TrimRight(p.c.baseURL, "/") + path
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(bodyJSON))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("xi-api-key", p.c.apiKey)
|
||||
|
||||
// No Client.Timeout — streaming may legitimately outlast a per-request
|
||||
// deadline. Cancellation is honoured via the request context.
|
||||
hc := &http.Client{Timeout: 0}
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs tts stream: %w", err)
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
errBody, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
return nil, fmt.Errorf("ElevenLabs API error %d: %s", resp.StatusCode, truncate(errBody, 500))
|
||||
}
|
||||
|
||||
return &audio.StreamResult{
|
||||
Audio: resp.Body,
|
||||
Extension: ext,
|
||||
MimeType: mime,
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// chunkedServer returns an httptest.Server that streams `chunks` back with
|
||||
// Flush between each. The handler also records the request path + headers.
|
||||
func chunkedServer(t *testing.T, chunks []string) (*httptest.Server, *atomic.Value) {
|
||||
t.Helper()
|
||||
lastPath := &atomic.Value{}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
lastPath.Store(r.URL.RequestURI())
|
||||
if r.Header.Get("xi-api-key") == "" {
|
||||
t.Errorf("missing xi-api-key header")
|
||||
}
|
||||
w.Header().Set("Content-Type", "audio/mpeg")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
flusher, _ := w.(http.Flusher)
|
||||
for _, c := range chunks {
|
||||
_, _ = w.Write([]byte(c))
|
||||
if flusher != nil {
|
||||
flusher.Flush()
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
}))
|
||||
return srv, lastPath
|
||||
}
|
||||
|
||||
func TestSynthesizeStream_ReturnsChunkedReader(t *testing.T) {
|
||||
t.Parallel()
|
||||
chunks := []string{"aaa", "bbb", "ccc"}
|
||||
srv, lastPath := chunkedServer(t, chunks)
|
||||
defer srv.Close()
|
||||
|
||||
p := NewTTSProvider(Config{
|
||||
APIKey: "test-key",
|
||||
BaseURL: srv.URL,
|
||||
VoiceID: "V1",
|
||||
ModelID: "eleven_v3",
|
||||
TimeoutMs: 5000,
|
||||
})
|
||||
|
||||
res, err := p.SynthesizeStream(context.Background(), "hello world", audio.TTSOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("SynthesizeStream error: %v", err)
|
||||
}
|
||||
if res.Audio == nil {
|
||||
t.Fatal("StreamResult.Audio is nil")
|
||||
}
|
||||
defer res.Audio.Close()
|
||||
|
||||
if res.Extension != "mp3" {
|
||||
t.Errorf("Extension: got %q, want mp3", res.Extension)
|
||||
}
|
||||
if res.MimeType != "audio/mpeg" {
|
||||
t.Errorf("MimeType: got %q, want audio/mpeg", res.MimeType)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(res.Audio)
|
||||
if err != nil {
|
||||
t.Fatalf("read body: %v", err)
|
||||
}
|
||||
want := strings.Join(chunks, "")
|
||||
if string(body) != want {
|
||||
t.Errorf("body: got %q, want %q", string(body), want)
|
||||
}
|
||||
|
||||
// URL must hit the /stream sub-path with mp3 output format.
|
||||
path, _ := lastPath.Load().(string)
|
||||
if !strings.Contains(path, "/v1/text-to-speech/V1/stream") {
|
||||
t.Errorf("path: got %q, want contains /v1/text-to-speech/V1/stream", path)
|
||||
}
|
||||
if !strings.Contains(path, "output_format=mp3_44100_128") {
|
||||
t.Errorf("path: got %q, want contains output_format=mp3_44100_128", path)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSynthesizeStream_VoiceOverride(t *testing.T) {
|
||||
t.Parallel()
|
||||
srv, lastPath := chunkedServer(t, []string{"x"})
|
||||
defer srv.Close()
|
||||
|
||||
p := NewTTSProvider(Config{
|
||||
APIKey: "k",
|
||||
BaseURL: srv.URL,
|
||||
VoiceID: "DEFAULT",
|
||||
ModelID: "eleven_v3",
|
||||
})
|
||||
|
||||
res, err := p.SynthesizeStream(context.Background(), "hi", audio.TTSOptions{Voice: "OVR"})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
res.Audio.Close()
|
||||
|
||||
path, _ := lastPath.Load().(string)
|
||||
if !strings.Contains(path, "/v1/text-to-speech/OVR/stream") {
|
||||
t.Errorf("voice override not honoured — path=%q", path)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSynthesizeStream_CancelContextClosesBody(t *testing.T) {
|
||||
t.Parallel()
|
||||
// Server writes one chunk then blocks forever.
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "audio/mpeg")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
if f, ok := w.(http.Flusher); ok {
|
||||
_, _ = w.Write([]byte("a"))
|
||||
f.Flush()
|
||||
}
|
||||
<-r.Context().Done()
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := NewTTSProvider(Config{APIKey: "k", BaseURL: srv.URL, VoiceID: "V"})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
res, err := p.SynthesizeStream(ctx, "hi", audio.TTSOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
|
||||
// Cancel while body is still open — next Read should fail promptly.
|
||||
cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
buf := make([]byte, 64)
|
||||
for {
|
||||
_, err := res.Audio.Read(buf)
|
||||
if err != nil {
|
||||
done <- err
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
select {
|
||||
case <-done:
|
||||
// ok — read returned an error after cancel.
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("Read did not return after ctx cancel")
|
||||
}
|
||||
res.Audio.Close()
|
||||
}
|
||||
|
||||
func TestSynthesizeStream_OpusFormat(t *testing.T) {
|
||||
t.Parallel()
|
||||
srv, lastPath := chunkedServer(t, []string{"o"})
|
||||
defer srv.Close()
|
||||
|
||||
p := NewTTSProvider(Config{APIKey: "k", BaseURL: srv.URL, VoiceID: "V", ModelID: "eleven_v3"})
|
||||
res, err := p.SynthesizeStream(context.Background(), "hi", audio.TTSOptions{Format: "opus"})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
res.Audio.Close()
|
||||
|
||||
if res.Extension != "ogg" {
|
||||
t.Errorf("Extension: got %q, want ogg", res.Extension)
|
||||
}
|
||||
if res.MimeType != "audio/ogg" {
|
||||
t.Errorf("MimeType: got %q, want audio/ogg", res.MimeType)
|
||||
}
|
||||
path, _ := lastPath.Load().(string)
|
||||
if !strings.Contains(path, "output_format=opus_48000_64") {
|
||||
t.Errorf("path: got %q, want contains output_format=opus_48000_64", path)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSynthesizeStream_RejectsUnknownModel(t *testing.T) {
|
||||
t.Parallel()
|
||||
// Server should never be reached — validation runs before HTTP.
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
t.Errorf("unexpected upstream request for invalid model")
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := NewTTSProvider(Config{APIKey: "k", BaseURL: srv.URL, VoiceID: "V"})
|
||||
_, err := p.SynthesizeStream(context.Background(), "hi", audio.TTSOptions{Model: "bogus-model"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for unknown model, got nil")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
func TestSynthesize_RejectsUnknownModel(t *testing.T) {
|
||||
t.Parallel()
|
||||
// Server must never be reached — validation runs before HTTP.
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
t.Errorf("unexpected upstream request for invalid model")
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := NewTTSProvider(Config{APIKey: "k", BaseURL: srv.URL, VoiceID: "V"})
|
||||
_, err := p.Synthesize(context.Background(), "hi", audio.TTSOptions{Model: "bogus-model"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for unknown model, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSynthesize_EmptyModelPassesValidation(t *testing.T) {
|
||||
t.Parallel()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "audio/mpeg")
|
||||
_, _ = w.Write([]byte("AUDIO"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := NewTTSProvider(Config{APIKey: "k", BaseURL: srv.URL, VoiceID: "V"})
|
||||
res, err := p.Synthesize(context.Background(), "hi", audio.TTSOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("Synthesize with empty model: %v", err)
|
||||
}
|
||||
if string(res.Audio) != "AUDIO" {
|
||||
t.Errorf("audio: got %q, want AUDIO", res.Audio)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package elevenlabs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
)
|
||||
|
||||
// voicesResponse mirrors the ElevenLabs GET /v1/voices response envelope.
|
||||
type voicesResponse struct {
|
||||
Voices []voiceJSON `json:"voices"`
|
||||
}
|
||||
|
||||
type voiceJSON struct {
|
||||
VoiceID string `json:"voice_id"`
|
||||
Name string `json:"name"`
|
||||
Labels map[string]string `json:"labels"`
|
||||
PreviewURL string `json:"preview_url"`
|
||||
Category string `json:"category"`
|
||||
}
|
||||
|
||||
// ListVoices fetches the caller's available voices from ElevenLabs
|
||||
// GET /v1/voices with xi-api-key auth. Results are NOT cached here;
|
||||
// caching lives in audio.VoiceCache.
|
||||
func (p *TTSProvider) ListVoices(ctx context.Context) ([]audio.Voice, error) {
|
||||
raw, err := p.c.getJSON(ctx, "/v1/voices")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs list voices: %w", err)
|
||||
}
|
||||
|
||||
var resp voicesResponse
|
||||
if err := json.Unmarshal(raw, &resp); err != nil {
|
||||
return nil, fmt.Errorf("elevenlabs list voices: parse response: %w", err)
|
||||
}
|
||||
|
||||
voices := make([]audio.Voice, 0, len(resp.Voices))
|
||||
for _, v := range resp.Voices {
|
||||
labels := v.Labels
|
||||
if labels == nil {
|
||||
labels = make(map[string]string)
|
||||
}
|
||||
voices = append(voices, audio.Voice{
|
||||
ID: v.VoiceID,
|
||||
Name: v.Name,
|
||||
Labels: labels,
|
||||
PreviewURL: v.PreviewURL,
|
||||
Category: v.Category,
|
||||
})
|
||||
}
|
||||
return voices, nil
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package elevenlabs_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio/elevenlabs"
|
||||
)
|
||||
|
||||
func TestListVoices_HappyPath(t *testing.T) {
|
||||
fixture := map[string]any{
|
||||
"voices": []map[string]any{
|
||||
{
|
||||
"voice_id": "voice1",
|
||||
"name": "Bella",
|
||||
"labels": map[string]any{"accent": "american", "gender": "female"},
|
||||
"preview_url": "https://cdn.example.com/bella.mp3",
|
||||
"category": "premade",
|
||||
},
|
||||
{
|
||||
"voice_id": "voice2",
|
||||
"name": "Adam",
|
||||
"labels": map[string]any{"accent": "british"},
|
||||
"preview_url": "https://cdn.example.com/adam.mp3",
|
||||
"category": "cloned",
|
||||
},
|
||||
},
|
||||
}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/v1/voices" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
}
|
||||
if r.Header.Get("xi-api-key") != "test-key" {
|
||||
t.Errorf("missing or wrong api key header")
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(fixture)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := elevenlabs.NewTTSProvider(elevenlabs.Config{
|
||||
APIKey: "test-key",
|
||||
BaseURL: srv.URL,
|
||||
})
|
||||
voices, err := p.ListVoices(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if len(voices) != 2 {
|
||||
t.Fatalf("expected 2 voices, got %d", len(voices))
|
||||
}
|
||||
if voices[0].ID != "voice1" || voices[0].Name != "Bella" {
|
||||
t.Errorf("voice[0] mismatch: %+v", voices[0])
|
||||
}
|
||||
if voices[0].Labels["accent"] != "american" {
|
||||
t.Errorf("voice[0] label accent mismatch: %v", voices[0].Labels)
|
||||
}
|
||||
if voices[0].PreviewURL != "https://cdn.example.com/bella.mp3" {
|
||||
t.Errorf("voice[0] preview url mismatch: %s", voices[0].PreviewURL)
|
||||
}
|
||||
if voices[1].ID != "voice2" || voices[1].Category != "cloned" {
|
||||
t.Errorf("voice[1] mismatch: %+v", voices[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestListVoices_Unauthorized(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
w.Write([]byte(`{"detail":"invalid_api_key"}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := elevenlabs.NewTTSProvider(elevenlabs.Config{
|
||||
APIKey: "bad-key",
|
||||
BaseURL: srv.URL,
|
||||
})
|
||||
_, err := p.ListVoices(t.Context())
|
||||
if err == nil {
|
||||
t.Fatal("expected error for 401, got nil")
|
||||
}
|
||||
// Error must mention the status code or response body
|
||||
if !containsStr(err.Error(), "401") && !containsStr(err.Error(), "invalid_api_key") {
|
||||
t.Errorf("expected 401 info in error message, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func containsStr(s, sub string) bool {
|
||||
for i := 0; i <= len(s)-len(sub); i++ {
|
||||
if s[i:i+len(sub)] == sub {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
package audio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels/media"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
)
|
||||
|
||||
// bridgedChannels tracks channels already registered per manager for idempotency.
|
||||
var (
|
||||
bridgedMu sync.Mutex
|
||||
bridgedChannels = make(map[*Manager]map[string]bool)
|
||||
)
|
||||
|
||||
// BridgeLegacySTT reads per-channel STTProxyURL from cfg and registers a
|
||||
// channel-scoped proxy STT provider for each non-empty URL. Emits a deprecation
|
||||
// warning once per channel per manager. Safe to call multiple times (idempotent).
|
||||
//
|
||||
// Channel override wins over tenant builtin_tools[stt] per Decision 2.
|
||||
func BridgeLegacySTT(mgr *Manager, cfg *config.Config) {
|
||||
type chanEntry struct {
|
||||
Name string
|
||||
URL string
|
||||
Key string
|
||||
TID string
|
||||
TOut int
|
||||
}
|
||||
|
||||
channels := []chanEntry{
|
||||
{
|
||||
Name: "telegram",
|
||||
URL: cfg.Channels.Telegram.STTProxyURL,
|
||||
Key: cfg.Channels.Telegram.STTAPIKey,
|
||||
TID: cfg.Channels.Telegram.STTTenantID,
|
||||
TOut: cfg.Channels.Telegram.STTTimeoutSeconds,
|
||||
},
|
||||
{
|
||||
Name: "feishu",
|
||||
URL: cfg.Channels.Feishu.STTProxyURL,
|
||||
Key: cfg.Channels.Feishu.STTAPIKey,
|
||||
TID: cfg.Channels.Feishu.STTTenantID,
|
||||
TOut: cfg.Channels.Feishu.STTTimeoutSeconds,
|
||||
},
|
||||
{
|
||||
Name: "discord",
|
||||
URL: cfg.Channels.Discord.STTProxyURL,
|
||||
Key: cfg.Channels.Discord.STTAPIKey,
|
||||
TID: cfg.Channels.Discord.STTTenantID,
|
||||
TOut: cfg.Channels.Discord.STTTimeoutSeconds,
|
||||
},
|
||||
}
|
||||
|
||||
bridgedMu.Lock()
|
||||
defer bridgedMu.Unlock()
|
||||
|
||||
if bridgedChannels[mgr] == nil {
|
||||
bridgedChannels[mgr] = make(map[string]bool)
|
||||
}
|
||||
already := bridgedChannels[mgr]
|
||||
|
||||
for _, ch := range channels {
|
||||
if ch.URL == "" {
|
||||
continue
|
||||
}
|
||||
if already[ch.Name] {
|
||||
continue // idempotent — skip re-registration
|
||||
}
|
||||
|
||||
slog.Warn("security.stt_legacy_config_deprecated",
|
||||
"channel", ch.Name,
|
||||
"url_suffix", lastN(ch.URL, 8),
|
||||
)
|
||||
|
||||
p := &legacyBridgeSTTProvider{
|
||||
cfg: media.STTConfig{ProxyURL: ch.URL, APIKey: ch.Key, TenantID: ch.TID, TimeoutSeconds: ch.TOut},
|
||||
channel: ch.Name,
|
||||
}
|
||||
mgr.RegisterChannelSTT(ch.Name, p)
|
||||
already[ch.Name] = true
|
||||
}
|
||||
}
|
||||
|
||||
// legacyBridgeSTTProvider is a thin adapter that wraps media.TranscribeAudio as
|
||||
// an STTProvider. Defined here (not in proxy_stt) to avoid a circular import:
|
||||
// proxy_stt → audio → proxy_stt.
|
||||
type legacyBridgeSTTProvider struct {
|
||||
cfg media.STTConfig
|
||||
channel string
|
||||
}
|
||||
|
||||
func (p *legacyBridgeSTTProvider) Name() string { return "proxy" }
|
||||
|
||||
func (p *legacyBridgeSTTProvider) Transcribe(ctx context.Context, in STTInput, _ STTOptions) (*TranscriptResult, error) {
|
||||
filePath := in.FilePath
|
||||
if filePath == "" && len(in.Bytes) > 0 {
|
||||
ext := legacyExtFromMime(in.MimeType)
|
||||
f, err := os.CreateTemp("", "stt-bridge-*"+ext)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("legacy_bridge stt: create temp: %w", err)
|
||||
}
|
||||
if err := os.Chmod(f.Name(), 0600); err != nil {
|
||||
f.Close()
|
||||
os.Remove(f.Name())
|
||||
return nil, fmt.Errorf("legacy_bridge stt: chmod: %w", err)
|
||||
}
|
||||
if _, err := f.Write(in.Bytes); err != nil {
|
||||
f.Close()
|
||||
os.Remove(f.Name())
|
||||
return nil, fmt.Errorf("legacy_bridge stt: write: %w", err)
|
||||
}
|
||||
f.Close()
|
||||
defer os.Remove(f.Name())
|
||||
filePath = f.Name()
|
||||
}
|
||||
|
||||
text, err := media.TranscribeAudio(ctx, p.cfg, filePath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &TranscriptResult{Text: text, Provider: "proxy"}, nil
|
||||
}
|
||||
|
||||
// lastN returns the last n characters of s (for safe URL logging).
|
||||
func lastN(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return "..." + s[len(s)-n:]
|
||||
}
|
||||
|
||||
// legacyExtFromMime returns a file extension for a MIME type.
|
||||
func legacyExtFromMime(mime string) string {
|
||||
switch strings.Split(mime, ";")[0] {
|
||||
case "audio/ogg":
|
||||
return ".ogg"
|
||||
case "audio/mpeg", "audio/mp3":
|
||||
return ".mp3"
|
||||
case "audio/wav", "audio/wave":
|
||||
return ".wav"
|
||||
case "audio/mp4", "audio/m4a":
|
||||
return ".m4a"
|
||||
case "audio/webm":
|
||||
return ".webm"
|
||||
case "audio/flac":
|
||||
return ".flac"
|
||||
default:
|
||||
return filepath.Ext(mime)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package audio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
)
|
||||
|
||||
// newBridgeTestCfg builds a minimal config with optional channel STT URLs.
|
||||
func newBridgeTestCfg(telegramURL, feishuURL, discordURL string) *config.Config {
|
||||
return &config.Config{
|
||||
Channels: config.ChannelsConfig{
|
||||
Telegram: config.TelegramConfig{STTProxyURL: telegramURL, STTAPIKey: "tg-key"},
|
||||
Feishu: config.FeishuConfig{STTProxyURL: feishuURL},
|
||||
Discord: config.DiscordConfig{STTProxyURL: discordURL},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// Case 1: tenant has explicit chain set + channel has STTProxyURL
|
||||
// → channel override wins (bridge-registered provider takes precedence).
|
||||
func TestBridgeLegacySTT_ChannelOverrideWinsOverTenantChain(t *testing.T) {
|
||||
// Each test gets its own manager to avoid shared bridgedChannels state.
|
||||
mgr := NewManager(ManagerConfig{})
|
||||
|
||||
// Simulate tenant-level provider already registered and chain set.
|
||||
mgr.RegisterSTT(&mockSTT{
|
||||
name: "elevenlabs",
|
||||
result: &TranscriptResult{Text: "tenant", Provider: "elevenlabs"},
|
||||
})
|
||||
mgr.SetSTTChain([]string{"elevenlabs"})
|
||||
|
||||
// Bridge registers telegram channel override.
|
||||
cfg := newBridgeTestCfg("http://stt.example.com", "", "")
|
||||
BridgeLegacySTT(mgr, cfg)
|
||||
|
||||
// Channel override must be registered.
|
||||
if _, ok := mgr.channelSTTOverrides["telegram"]; !ok {
|
||||
t.Fatal("expected telegram channel override to be registered")
|
||||
}
|
||||
|
||||
// resolveSTTChain with telegram context returns channel override, not tenant chain.
|
||||
ctx := WithChannel(context.Background(), "telegram")
|
||||
chain := mgr.resolveSTTChain(ctx)
|
||||
if len(chain) == 0 {
|
||||
t.Fatal("expected non-empty channel override chain")
|
||||
}
|
||||
// Channel-scoped key format is "proxy:telegram".
|
||||
if chain[0] != "proxy:telegram" {
|
||||
t.Errorf("expected chain[0]='proxy:telegram', got %q", chain[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Case 2: tenant has no explicit chain + channel has STTProxyURL
|
||||
// → bridge registers channel-scoped provider; no-URL channels skipped.
|
||||
func TestBridgeLegacySTT_RegistersWhenNoTenantChain(t *testing.T) {
|
||||
mgr := NewManager(ManagerConfig{})
|
||||
cfg := newBridgeTestCfg("http://stt.proxy.example.com", "http://feishu.stt.example.com", "")
|
||||
|
||||
BridgeLegacySTT(mgr, cfg)
|
||||
|
||||
if _, ok := mgr.channelSTTOverrides["telegram"]; !ok {
|
||||
t.Error("expected telegram channel override registered")
|
||||
}
|
||||
if _, ok := mgr.channelSTTOverrides["feishu"]; !ok {
|
||||
t.Error("expected feishu channel override registered")
|
||||
}
|
||||
// Discord URL empty → should NOT be registered.
|
||||
if _, ok := mgr.channelSTTOverrides["discord"]; ok {
|
||||
t.Error("expected discord channel override NOT registered (empty URL)")
|
||||
}
|
||||
}
|
||||
|
||||
// Case 3: calling BridgeLegacySTT twice is idempotent — no duplicate registrations.
|
||||
func TestBridgeLegacySTT_Idempotent(t *testing.T) {
|
||||
mgr := NewManager(ManagerConfig{})
|
||||
cfg := newBridgeTestCfg("http://stt.example.com", "", "")
|
||||
|
||||
BridgeLegacySTT(mgr, cfg)
|
||||
countAfterFirst := len(mgr.sttProviders)
|
||||
|
||||
BridgeLegacySTT(mgr, cfg)
|
||||
countAfterSecond := len(mgr.sttProviders)
|
||||
|
||||
if countAfterSecond != countAfterFirst {
|
||||
t.Errorf("idempotency violated: provider count changed from %d to %d on second call",
|
||||
countAfterFirst, countAfterSecond)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
package audio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
)
|
||||
|
||||
// ctxKeyChannel is the context key for the current channel name (e.g. "telegram").
|
||||
// Set via WithChannel; read by resolveSTTChain to select channel-scoped providers.
|
||||
type ctxKeyChannelType struct{}
|
||||
|
||||
var ctxKeyChannel = ctxKeyChannelType{}
|
||||
|
||||
// WithChannel returns a context that carries the channel name for STT chain resolution.
|
||||
func WithChannel(ctx context.Context, channel string) context.Context {
|
||||
return context.WithValue(ctx, ctxKeyChannel, channel)
|
||||
}
|
||||
|
||||
// channelFromCtx extracts the channel name from ctx, or "" if not set.
|
||||
func channelFromCtx(ctx context.Context) string {
|
||||
v, _ := ctx.Value(ctxKeyChannel).(string)
|
||||
return v
|
||||
}
|
||||
|
||||
// Manager orchestrates audio providers across TTS, STT, Music, and SFX
|
||||
// operations. Each op has its own provider map + primary/fallback chain.
|
||||
//
|
||||
// Phase 1 exercises the TTS path end-to-end. STT/Music/SFX maps and chains
|
||||
// are present but empty (providers register in Phase 3/4).
|
||||
type Manager struct {
|
||||
ttsProviders map[string]TTSProvider
|
||||
sttProviders map[string]STTProvider
|
||||
musicProviders map[string]MusicProvider
|
||||
sfxProviders map[string]SFXProvider
|
||||
|
||||
primary string // primary TTS provider
|
||||
sttChain []string // STT fallback order (Phase 4)
|
||||
musicChain []string // Music fallback order (Phase 3)
|
||||
channelSTTOverrides map[string][]string // channel → provider key list (Phase 4)
|
||||
|
||||
auto AutoMode
|
||||
mode Mode
|
||||
maxLength int // max text length before truncation (default 1500)
|
||||
timeoutMs int // provider timeout (default 30000)
|
||||
}
|
||||
|
||||
// ManagerConfig configures the audio manager. Preserved from legacy TTS
|
||||
// package — new STT/Music fields are set via RegisterSTT/RegisterMusic and
|
||||
// (optionally) cfg.Audio in config_audio.go.
|
||||
type ManagerConfig struct {
|
||||
Primary string // primary TTS provider name
|
||||
Auto AutoMode // auto-apply mode (default "off")
|
||||
Mode Mode // "final" or "all" (default "final")
|
||||
MaxLength int // default 1500
|
||||
TimeoutMs int // default 30000
|
||||
}
|
||||
|
||||
// NewManager creates an audio manager with empty provider maps.
|
||||
func NewManager(cfg ManagerConfig) *Manager {
|
||||
m := &Manager{
|
||||
ttsProviders: make(map[string]TTSProvider),
|
||||
sttProviders: make(map[string]STTProvider),
|
||||
musicProviders: make(map[string]MusicProvider),
|
||||
sfxProviders: make(map[string]SFXProvider),
|
||||
channelSTTOverrides: make(map[string][]string),
|
||||
primary: cfg.Primary,
|
||||
auto: cfg.Auto,
|
||||
mode: cfg.Mode,
|
||||
maxLength: cfg.MaxLength,
|
||||
timeoutMs: cfg.TimeoutMs,
|
||||
}
|
||||
if m.auto == "" {
|
||||
m.auto = AutoOff
|
||||
}
|
||||
if m.mode == "" {
|
||||
m.mode = ModeFinal
|
||||
}
|
||||
if m.maxLength <= 0 {
|
||||
m.maxLength = 1500
|
||||
}
|
||||
if m.timeoutMs <= 0 {
|
||||
m.timeoutMs = 30000
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ---- Registration ----
|
||||
|
||||
// RegisterTTS adds a TTS provider. If no primary is set, the first registered
|
||||
// provider becomes primary — matches legacy tts.Manager.RegisterProvider.
|
||||
func (m *Manager) RegisterTTS(p TTSProvider) {
|
||||
m.ttsProviders[p.Name()] = p
|
||||
if m.primary == "" {
|
||||
m.primary = p.Name()
|
||||
}
|
||||
}
|
||||
|
||||
// RegisterProvider is a backward-compat alias for RegisterTTS — lets pre-Phase-1
|
||||
// callers that go through tts.Manager (= audio.Manager via alias) keep working.
|
||||
func (m *Manager) RegisterProvider(p TTSProvider) { m.RegisterTTS(p) }
|
||||
|
||||
// RegisterSTT adds an STT provider (Phase 4).
|
||||
func (m *Manager) RegisterSTT(p STTProvider) {
|
||||
m.sttProviders[p.Name()] = p
|
||||
}
|
||||
|
||||
// RegisterMusic adds a music provider (Phase 3).
|
||||
func (m *Manager) RegisterMusic(p MusicProvider) {
|
||||
m.musicProviders[p.Name()] = p
|
||||
}
|
||||
|
||||
// RegisterSFX adds an SFX provider (Phase 3).
|
||||
func (m *Manager) RegisterSFX(p SFXProvider) {
|
||||
m.sfxProviders[p.Name()] = p
|
||||
}
|
||||
|
||||
// ---- Introspection ----
|
||||
|
||||
// GetProvider returns a TTS provider by name. Preserved from legacy API.
|
||||
func (m *Manager) GetProvider(name string) (TTSProvider, bool) {
|
||||
p, ok := m.ttsProviders[name]
|
||||
return p, ok
|
||||
}
|
||||
|
||||
// PrimaryProvider returns the primary TTS provider name.
|
||||
func (m *Manager) PrimaryProvider() string { return m.primary }
|
||||
|
||||
// AutoMode returns the current auto-apply mode.
|
||||
func (m *Manager) AutoMode() AutoMode { return m.auto }
|
||||
|
||||
// HasProviders reports whether any TTS provider is registered.
|
||||
func (m *Manager) HasProviders() bool { return len(m.ttsProviders) > 0 }
|
||||
|
||||
// ---- TTS dispatch ----
|
||||
|
||||
// Synthesize uses the primary provider.
|
||||
func (m *Manager) Synthesize(ctx context.Context, text string, opts TTSOptions) (*SynthResult, error) {
|
||||
p, ok := m.ttsProviders[m.primary]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("tts provider not found: %s", m.primary)
|
||||
}
|
||||
return p.Synthesize(ctx, text, opts)
|
||||
}
|
||||
|
||||
// SynthesizeStream dispatches streaming TTS to the primary provider. Returns
|
||||
// ErrStreamingNotSupported if the primary does not implement
|
||||
// StreamingTTSProvider, letting callers fall back to buffered Synthesize.
|
||||
func (m *Manager) SynthesizeStream(ctx context.Context, text string, opts TTSOptions) (*StreamResult, error) {
|
||||
p, ok := m.ttsProviders[m.primary]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("tts provider not found: %s", m.primary)
|
||||
}
|
||||
sp, ok := p.(StreamingTTSProvider)
|
||||
if !ok {
|
||||
return nil, ErrStreamingNotSupported
|
||||
}
|
||||
return sp.SynthesizeStream(ctx, text, opts)
|
||||
}
|
||||
|
||||
// ---- Music dispatch ----
|
||||
|
||||
// GenerateMusic tries registered music providers in chain order until one succeeds.
|
||||
// Chain order: elevenlabs first (if registered), then remaining providers.
|
||||
// Override order by setting m.musicChain before the first call.
|
||||
func (m *Manager) GenerateMusic(ctx context.Context, opts MusicOptions) (*AudioResult, error) {
|
||||
chain := m.resolveMusicChain()
|
||||
if len(chain) == 0 {
|
||||
return nil, fmt.Errorf("no music providers registered")
|
||||
}
|
||||
var lastErr error
|
||||
for _, name := range chain {
|
||||
p, ok := m.musicProviders[name]
|
||||
if !ok {
|
||||
slog.Info("audio.music provider not registered, skipping", "provider", name)
|
||||
continue
|
||||
}
|
||||
if res, err := p.GenerateMusic(ctx, opts); err == nil {
|
||||
return res, nil
|
||||
} else {
|
||||
slog.Warn("audio.music provider failed", "provider", name, "error", err)
|
||||
lastErr = err
|
||||
}
|
||||
}
|
||||
if lastErr != nil {
|
||||
return nil, fmt.Errorf("all music providers failed: %w", lastErr)
|
||||
}
|
||||
return nil, fmt.Errorf("no music providers registered")
|
||||
}
|
||||
|
||||
// GenerateSFX tries SFX providers in order: elevenlabs first, then any other registered.
|
||||
func (m *Manager) GenerateSFX(ctx context.Context, opts SFXOptions) (*AudioResult, error) {
|
||||
order := m.resolveSFXChain()
|
||||
if len(order) == 0 {
|
||||
return nil, fmt.Errorf("no sfx providers registered")
|
||||
}
|
||||
var lastErr error
|
||||
for _, name := range order {
|
||||
p, ok := m.sfxProviders[name]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if res, err := p.GenerateSFX(ctx, opts); err == nil {
|
||||
return res, nil
|
||||
} else {
|
||||
slog.Warn("audio.sfx provider failed", "provider", name, "error", err)
|
||||
lastErr = err
|
||||
}
|
||||
}
|
||||
if lastErr != nil {
|
||||
return nil, fmt.Errorf("all sfx providers failed: %w", lastErr)
|
||||
}
|
||||
return nil, fmt.Errorf("no sfx providers registered")
|
||||
}
|
||||
|
||||
// resolveMusicChain returns the ordered provider names for music generation.
|
||||
// If m.musicChain is set explicitly it is used as-is; otherwise elevenlabs is
|
||||
// preferred and remaining providers follow in registration order.
|
||||
func (m *Manager) resolveMusicChain() []string {
|
||||
if len(m.musicChain) > 0 {
|
||||
return m.musicChain
|
||||
}
|
||||
out := make([]string, 0, len(m.musicProviders))
|
||||
if _, ok := m.musicProviders["elevenlabs"]; ok {
|
||||
out = append(out, "elevenlabs")
|
||||
}
|
||||
for name := range m.musicProviders {
|
||||
if name != "elevenlabs" {
|
||||
out = append(out, name)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// resolveSFXChain returns the ordered provider names for SFX generation.
|
||||
func (m *Manager) resolveSFXChain() []string {
|
||||
out := make([]string, 0, len(m.sfxProviders))
|
||||
if _, ok := m.sfxProviders["elevenlabs"]; ok {
|
||||
out = append(out, "elevenlabs")
|
||||
}
|
||||
for name := range m.sfxProviders {
|
||||
if name != "elevenlabs" {
|
||||
out = append(out, name)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SynthesizeWithFallback tries primary first, then any other registered
|
||||
// provider on error. Returns first success or aggregate failure.
|
||||
func (m *Manager) SynthesizeWithFallback(ctx context.Context, text string, opts TTSOptions) (*SynthResult, error) {
|
||||
if p, ok := m.ttsProviders[m.primary]; ok {
|
||||
if result, err := p.Synthesize(ctx, text, opts); err == nil {
|
||||
return result, nil
|
||||
} else {
|
||||
slog.Warn("tts primary provider failed, trying fallback", "provider", m.primary, "error", err)
|
||||
}
|
||||
}
|
||||
for name, p := range m.ttsProviders {
|
||||
if name == m.primary {
|
||||
continue
|
||||
}
|
||||
result, err := p.Synthesize(ctx, text, opts)
|
||||
if err == nil {
|
||||
slog.Info("tts fallback succeeded", "provider", name)
|
||||
return result, nil
|
||||
}
|
||||
slog.Warn("tts fallback provider failed", "provider", name, "error", err)
|
||||
}
|
||||
return nil, fmt.Errorf("all tts providers failed")
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package audio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// MaybeApply inspects auto-mode and conditionally applies TTS to a reply.
|
||||
// Returns (result, true) on success, (nil, false) when auto is disabled, the
|
||||
// reply type is filtered out, content fails validation, or synthesis fails.
|
||||
//
|
||||
// Parameters:
|
||||
// - text: the reply text to potentially convert
|
||||
// - channel: origin channel ("telegram" switches format to opus)
|
||||
// - isVoiceInbound: whether the user's inbound message was voice
|
||||
// - kind: "tool", "block", or "final"
|
||||
func (m *Manager) MaybeApply(ctx context.Context, text, channel string, isVoiceInbound bool, kind string) (*SynthResult, bool) {
|
||||
if m.auto == AutoOff {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Mode filter: ModeFinal skips tool/block replies.
|
||||
if m.mode == ModeFinal && (kind == "tool" || kind == "block") {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
switch m.auto {
|
||||
case AutoInbound:
|
||||
if !isVoiceInbound {
|
||||
return nil, false
|
||||
}
|
||||
case AutoTagged:
|
||||
if !strings.Contains(text, "[[tts]]") && !strings.Contains(text, "[[tts:") {
|
||||
return nil, false
|
||||
}
|
||||
case AutoAlways:
|
||||
// Always apply.
|
||||
default:
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Content validation (matches legacy TTS behavior).
|
||||
cleanText := stripMarkdown(text)
|
||||
cleanText = stripTtsDirectives(cleanText)
|
||||
cleanText = strings.TrimSpace(cleanText)
|
||||
|
||||
if len(cleanText) < 10 {
|
||||
return nil, false
|
||||
}
|
||||
if strings.Contains(cleanText, "MEDIA:") {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
if len(cleanText) > m.maxLength {
|
||||
cleanText = cleanText[:m.maxLength] + "..."
|
||||
}
|
||||
|
||||
opts := TTSOptions{}
|
||||
if channel == "telegram" {
|
||||
opts.Format = "opus" // Telegram voice bubbles need opus
|
||||
}
|
||||
|
||||
result, err := m.SynthesizeWithFallback(ctx, cleanText, opts)
|
||||
if err != nil {
|
||||
slog.Warn("tts auto-apply failed", "error", err)
|
||||
return nil, false
|
||||
}
|
||||
return result, true
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
package audio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// ---- Fake providers for testing ----
|
||||
|
||||
type fakeMusic struct {
|
||||
name string
|
||||
res *AudioResult
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeMusic) Name() string { return f.name }
|
||||
func (f *fakeMusic) GenerateMusic(_ context.Context, _ MusicOptions) (*AudioResult, error) {
|
||||
return f.res, f.err
|
||||
}
|
||||
|
||||
type fakeSFX struct {
|
||||
name string
|
||||
res *AudioResult
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeSFX) Name() string { return f.name }
|
||||
func (f *fakeSFX) GenerateSFX(_ context.Context, _ SFXOptions) (*AudioResult, error) {
|
||||
return f.res, f.err
|
||||
}
|
||||
|
||||
// ---- GenerateMusic tests ----
|
||||
|
||||
func TestManager_GenerateMusic_primary_succeeds(t *testing.T) {
|
||||
m := NewManager(ManagerConfig{})
|
||||
want := &AudioResult{Audio: []byte("music"), Extension: "mp3", MimeType: "audio/mpeg"}
|
||||
m.RegisterMusic(&fakeMusic{name: "elevenlabs", res: want})
|
||||
|
||||
got, err := m.GenerateMusic(context.Background(), MusicOptions{Prompt: "test"})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if string(got.Audio) != string(want.Audio) {
|
||||
t.Errorf("audio: want %q, got %q", want.Audio, got.Audio)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManager_GenerateMusic_primary_fails_secondary_succeeds(t *testing.T) {
|
||||
m := NewManager(ManagerConfig{})
|
||||
want := &AudioResult{Audio: []byte("fallback_music"), Extension: "mp3", MimeType: "audio/mpeg"}
|
||||
m.RegisterMusic(&fakeMusic{name: "elevenlabs", err: errors.New("elevenlabs down")})
|
||||
m.RegisterMusic(&fakeMusic{name: "minimax", res: want})
|
||||
|
||||
got, err := m.GenerateMusic(context.Background(), MusicOptions{Prompt: "test"})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if string(got.Audio) != string(want.Audio) {
|
||||
t.Errorf("audio: want %q, got %q", want.Audio, got.Audio)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManager_GenerateMusic_all_fail(t *testing.T) {
|
||||
m := NewManager(ManagerConfig{})
|
||||
m.RegisterMusic(&fakeMusic{name: "elevenlabs", err: errors.New("elevenlabs down")})
|
||||
m.RegisterMusic(&fakeMusic{name: "minimax", err: errors.New("minimax down")})
|
||||
|
||||
_, err := m.GenerateMusic(context.Background(), MusicOptions{Prompt: "test"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !containsStr(err.Error(), "all music providers failed") {
|
||||
t.Errorf("error should mention 'all music providers failed': %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManager_GenerateMusic_no_providers(t *testing.T) {
|
||||
m := NewManager(ManagerConfig{})
|
||||
|
||||
_, err := m.GenerateMusic(context.Background(), MusicOptions{Prompt: "test"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !containsStr(err.Error(), "no music providers") {
|
||||
t.Errorf("error should mention 'no music providers': %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- GenerateSFX tests ----
|
||||
|
||||
func TestManager_GenerateSFX_elevenlabs_succeeds(t *testing.T) {
|
||||
m := NewManager(ManagerConfig{})
|
||||
want := &AudioResult{Audio: []byte("sfx_bytes"), Extension: "mp3", MimeType: "audio/mpeg"}
|
||||
m.RegisterSFX(&fakeSFX{name: "elevenlabs", res: want})
|
||||
|
||||
got, err := m.GenerateSFX(context.Background(), SFXOptions{Prompt: "explosion"})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if string(got.Audio) != string(want.Audio) {
|
||||
t.Errorf("audio: want %q, got %q", want.Audio, got.Audio)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManager_GenerateSFX_elevenlabs_fails_fallback(t *testing.T) {
|
||||
m := NewManager(ManagerConfig{})
|
||||
want := &AudioResult{Audio: []byte("fallback_sfx"), Extension: "mp3", MimeType: "audio/mpeg"}
|
||||
m.RegisterSFX(&fakeSFX{name: "elevenlabs", err: errors.New("elevenlabs sfx down")})
|
||||
m.RegisterSFX(&fakeSFX{name: "other_sfx", res: want})
|
||||
|
||||
got, err := m.GenerateSFX(context.Background(), SFXOptions{Prompt: "click"})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if string(got.Audio) != string(want.Audio) {
|
||||
t.Errorf("audio: want %q, got %q", want.Audio, got.Audio)
|
||||
}
|
||||
}
|
||||
|
||||
func containsStr(s, sub string) bool {
|
||||
return len(sub) == 0 || (len(s) >= len(sub) && func() bool {
|
||||
for i := 0; i <= len(s)-len(sub); i++ {
|
||||
if s[i:i+len(sub)] == sub {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}())
|
||||
}
|
||||
Loaded 100 of 604 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user