mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-13 00:14:33 +00:00
Release: vault improvements, MCP reconnect, cron fixes, Facebook/Pancake channels (#822)
* fix(ci): skip CI condition in semantic-release for main branch
go-semantic-release auto-detects the default branch from GitHub API
(which is dev), but releases are cut from main. The CI condition
rejects runs on non-default branches. Use --no-ci to bypass this
check since the workflow already gates on push to main.
* docs: document CI/CD pipelines, release flow, and v2.66.0 changelog
- CLAUDE.md: add CI/CD & Releases section with workflow table, tag
patterns, Docker variants, beta/desktop release commands
- CONTRIBUTING.md: expand Releases section with standard (auto),
beta (manual tag), and desktop release workflows
- docs/17-changelog.md: add v2.66.0 entry covering IDOR fix, BytePlus
provider, per-agent grants, beta pipeline, and CI fixes
* fix(telegram): handle group-to-supergroup migration seamlessly
When a Telegram group upgrades to a supergroup, the chat ID changes and
all existing references become stale. This caused send failures (400),
orphaned sessions, and required manual re-pairing.
Add dual-path migration handling:
- Proactive: intercept inbound MigrateToChatID before isServiceMessage
- Reactive: detect 400 + MigrateToChatID on send, migrate DB, retry
DB migration updates in a single transaction (scoped by tenant + channel):
- paired_devices: sender_id, chat_id
- sessions: session_key, user_id
- channel_contacts: sender_id
- channel_pending_messages: history_key
Also invalidates in-memory caches (approvedGroups, pairingReplySent,
groupHistory) and handles media sends via migration retry in Send().
* fix(tools): quote-aware shell operator detection in credentialed exec (#700) (#702)
* fix(tools): quote-aware shell operator detection in credentialed exec (#700)
- Replace detectShellOperators with detectUnquotedShellOperators in
credentialed exec path — respects single/double quoting so that
characters like | inside argument values (e.g. --jq '.[0] | .name')
are not falsely flagged as shell operators
- Pass raw command string (preserving quotes) to executeCredentialed
instead of reconstructing from parsed args
- Downgrade "no credential found" log from Warn to Debug (fires for
every non-credentialed command, too noisy at Warn)
- Add extractUnquotedSegments() helper with comprehensive tests
* fix(tools): handle backslash escape outside quotes in shell operator detection
extractUnquotedSegments did not handle \ as an escape character outside
of quotes, causing \" to incorrectly enter double-quote mode. This hid
subsequent shell operators from detection (e.g. gh \"arg\" | env would
not detect the unquoted pipe).
Add backslash escape handling in the unquoted state to match
go-shellwords parsing behavior. Both \ and the escaped character are
emitted as unquoted content so operator detection still catches them.
---------
Co-authored-by: viettranx <viettranx@gmail.com>
* feat(infra): tracing recovery, browser cleanup, CLI fixes, UI workspace split (#709)
- Tracing: recover stale running traces/spans on startup (PG + SQLite)
- Browser: Chrome orphan cleanup via launcher PID, timeouts, Leakless
- Claude CLI: WaitDelay 5s + context-cancel early exit
- Agent loop: safety-net defer to finalize orphan root traces
- UI: split workspace sharing into separate Memory and KG toggles
- Minor: for-range idiom, min() builtin
* fix(prompt): skip credentialed CLI context when exec tool is denied
Agents with exec in their deny list cannot run CLI commands, so
injecting wrangler/gh credential context is misleading — the LLM
sees instructions for tools it cannot use. Gate the section on
exec being present in the filtered tool list.
* fix(ui): improve traces table layout and readability
Compact columns: status as icon-only, merge time+duration into one column.
Truncate long user IDs, clean <media:*> tags from preview, move badges to second line.
* fix(security): harden exec path exemption matching
- Add absolute path exemption for dataDir/skills-store/ (fixes skill
scripts using absolute paths like /app/data/skills-store/ being denied)
- Strip surrounding quotes before prefix matching (LLMs often quote paths)
- Reject path traversal ("..") in exempt fields to prevent escape
- Switch from "any field exempt → skip" to per-field matching: only exempt
if ALL fields that match the deny pattern are individually exempt
- Closes pipe/comment bypass vectors where an exempt path in one argument
would exempt the entire command including non-exempt paths
Includes 27 test cases covering: legitimate access, quoted paths,
path traversal, unicode bypass, pipe/comment bypass, mixed args.
* feat(whatsapp): add native WhatsApp channel with whatsmeow (#720)
Replace Node.js Baileys bridge with native go.mau.fi/whatsmeow — zero
external dependencies. QR auth, media support, markdown formatting,
typing indicators, dual JID/LID identity, group policies, pairing.
Resolves #703
* fix(chat): load message history on first conversation click (#730)
* fix(chat): load message history when selecting existing conversation from clean state
The skipNextHistoryRef was unconditionally set when sessionKey transitioned
from empty to non-empty. This prevented loadHistory() from running when
clicking an existing conversation from the initial /chat page. The skip
was only intended for the new-chat send flow where the optimistic message
is already displayed.
Guard the skip with expectingRunRef so it only activates when a message
send is in flight.
Closes #729
* docs: add UI diff evidence for PR #730
Before/after screenshots and HTML comparison report showing
first conversation click behavior fix.
* fix(permissions): use cron-specific permission check for cron tool (#725)
* fix(security): harden exec path exemption matching (#721)
- Add absolute path exemption for dataDir/skills-store/ (fixes skill
scripts using absolute paths like /app/data/skills-store/ being denied)
- Strip surrounding quotes before prefix matching (LLMs often quote paths)
- Reject path traversal ("..") in exempt fields to prevent escape
- Switch from "any field exempt → skip" to per-field matching: only exempt
if ALL fields that match the deny pattern are individually exempt
- Closes pipe/comment bypass vectors where an exempt path in one argument
would exempt the entire command including non-exempt paths
Includes 27 test cases covering: legitimate access, quoted paths,
path traversal, unicode bypass, pipe/comment bypass, mixed args.
* fix(permissions): use cron-specific permission check for cron tool
Cron tool was hardcoded to check `file_writer` configType via
CheckFileWriterPermission(), ignoring the `cron` configType that
the UI actually saves when granting cron permissions. This caused
agents in group chats to be denied cron access even with correct
permission configured.
Add ConfigTypeCron constant and CheckCronPermission() that checks
`cron` configType first, falling back to `file_writer`.
---------
Co-authored-by: Viet Tran <viettranx@gmail.com>
* fix(exec): allow uploaded files in active workspaces (#748)
Shell-aware command parsing, dynamic workspace exemptions, and symlink canonicalization for exec path denial. Fixes #739.
* refactor(exec): extract path exemption logic to separate file
Move shell-aware parsing, dynamic workspace exemptions, path alias
variants, and canonicalization functions from shell.go (688 LOC) to
shell_path_exemption.go (284 LOC) for maintainability.
* feat(agent): centralized tenant user identity resolution for credentials
Add CredentialUserID context key that resolves channel contacts to merged
tenant users for credential lookups (SecureCLI, MCP). Keeps UserID
unchanged for session/workspace scoping. Resolves group senders, group
contacts, and unresolved DMs via ContactStore with 60s TTL cache.
* fix(ui): improve traces table column layout
Add width constraints and whitespace-nowrap to prevent column wrapping
on narrow viewports. Cherry-picked from dev-v3.
* fix(ui): enable search filtering on knowledge graph view (#758)
Search box on /knowledge-graph only filtered table view. Add useMemo client-side filtering of graph entities by name/type/description, only show relations where both endpoints match.
Closes #759
* feat: add zoom controls to knowledge graph (#757)
Add +/- buttons and percentage display to the knowledge graph stats bar
so users can zoom without relying solely on mouse wheel. Uses existing
react-force-graph-2D zoom() API with 1.5x step and 300ms animation.
Closes #755
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(providers): use DB name for ClaudeCLI, ACP, and Anthropic registration
Provider Name() methods returned hardcoded strings, so DB-registered
providers with custom names got wrong registry key — causing "provider
not found" fallback. Add WithClaudeCLIName/WithACPName/WithAnthropicName
options, pass p.Name from DB registration paths. Config-based paths
keep hardcoded defaults.
Closes #742
* fix(agent): correct soft-trim head/tail budget allocation when tail is important (#723)
Co-authored-by: quxy5 <quxy5@outlook.com>
* feat(v3): core architecture redesign — pipeline, memory, vault, evolution, providers, orchestration (#790)
* feat(v3): add core interface contracts and migration for v3 redesign
Foundation interfaces: TokenCounter, WorkspaceContext, DomainEventBus,
ProviderAdapter/Capabilities. Pipeline: Stage, RunState, MessageBuffer,
substates, Pipeline orchestrator. Memory: EpisodicStore, AutoInjector,
KG temporal extensions, consolidation workers. System integration:
PromptConfig, ToolCapability, Retriever. Orchestration: OrchestrationMode,
EvolutionMetrics/SuggestionStore. Migration 000037: episodic_summaries,
evolution tables, KG temporal columns. Schema version 36→37.
* refactor(plans): mark all v3 design phases complete with file references
* fix(v3): address code review findings on design contracts
- C1: add missing l0_abstract column to episodic_summaries migration
- C2: align EpisodicSummary ID/TenantID/AgentID to uuid.UUID
- H1: document tenant_id scoping requirement on EpisodicStore
- H2: add UNIQUE constraint on (agent_id, user_id, source_id) for dedup
- H4: clarify ProviderAdapter vs Provider relationship in doc
- M3: set state.ExitCode on BreakLoop/AbortRun in pipeline
- M6: store full PipelineConfig in Pipeline struct
- Edge: add WHERE embedding IS NOT NULL on HNSW index
* fix(v3): second-pass review fixes
- H1: use context.WithoutCancel for finalize + set ExitCode on ctx cancel
- H2: use utf8.RuneCountInString consistently in FallbackCounter
- H3: longest-prefix-match in ModelContextWindow (prevents wrong tokenizer)
- H4: return unsubscribe cleanup func from consolidation.Register
* feat(v3): implement DomainEventBus with worker pool, dedup, and retry
Worker pool processes events from buffered channel. SourceID-based dedup
prevents duplicate processing. Exponential backoff retry on handler error.
Panic recovery per handler. Graceful shutdown via Drain(). 8/8 tests pass
with race detector.
* feat(v3): implement ProviderAdapter for Anthropic, OpenAI, DashScope, Codex
Add CapabilitiesAware to all 6 providers. Create ProviderAdapter
implementations that delegate to existing buildRequestBody/parseResponse
for DRY. ClaudeCLI and ACP get capabilities only (subprocess transport).
DashScope wraps OpenAI adapter with StreamWithTools=false override.
* feat(v3): implement WorkspaceContext Resolver for 6 scenarios
Stateless resolver produces immutable WorkspaceContext at run start.
Handles personal/group/predefined/team-shared/team-isolated/delegation.
Wired into loop_context.go behind v3PipelineEnabled flag (additive,
v2 path unchanged). Includes delegation path boundary check,
master tenant bypass, and tenant slug path composition.
* feat(v3): implement tiktoken TokenCounter with BPE encoding + cache
Adds tiktoken-go for accurate cl100k_base/o200k_base token counting.
Per-message FNV-1a hash cache avoids re-encoding unchanged history.
Falls back to rune/3 heuristic for unknown models. NewTokenCounter
factory selects implementation at build time.
* feat(v3): promote 12 other_config JSONB fields to dedicated agent columns
Extract emoji, agent_description, thinking_level, max_tokens,
self_evolve, skill_evolve, skill_nudge_interval, reasoning_config,
workspace_sharing, chatgpt_oauth_routing, shell_deny_groups, and
kg_dedup_config from the catch-all other_config JSONB into proper
columns with DB-level types and defaults.
- Migration: PG (000037) + SQLite (schema v6→7) with backfill
- Go: AgentData struct + simplified Parse* methods
- Store: SELECT/INSERT/scan updated for both PG and SQLite
- Gateway: create/update handlers accept promoted fields
- HTTP: export/import with legacy backward compat
- Web UI: all 15 frontend files read/write from top level
* feat(v3): implement Knowledge Vault with unified search, wikilinks, and FS sync
Migration 000038 adds vault_documents (FTS+pgvector), vault_links, vault_versions
tables. VaultStore interface with PG implementation for document CRUD, hybrid
FTS+vector search, and bidirectional link management. All queries enforce
tenant_id isolation including JOIN-based scoping on link operations.
FS sync layer: SHA-256 content hashing, VaultInterceptor hooks into write_file/
read_file for auto-registration and lazy sync, fsnotify watcher with 500ms
debounce. Wikilink engine parses [[target]] syntax, resolves targets via
3-step strategy, and maintains vault_links on write.
VaultSearchService fans out queries across vault, episodic, and KG stores in
parallel with per-source score normalization and weighted merge. AutoInjector
and Retriever implementations for pipeline integration.
Three agent tools: vault_search (unified discovery), vault_link (explicit
linking), vault_backlinks (dependency tracing). Feature-flagged via
v3_vault_enabled agent setting.
* feat(v3): wire vault into gateway startup + add unit tests
Wire VaultStore embedding provider, VaultSearchService, VaultInterceptor
on read/write tools, and register vault_search/vault_link/vault_backlinks
tools in gateway_vault_wiring.go. All wiring gated by stores.Vault != nil.
Add 28 unit tests for ContentHash, ContentHashFile, and ExtractWikilinks
covering edge cases, unicode, display text, context windows, and offsets.
* feat(v3): implement stage-based pipeline loop with 8 pluggable stages
Decompose monolithic agent loop into internal/pipeline/ package:
- 6 stages: Context, Think, Prune+MemoryFlush, Tool, Observe+Checkpoint, Finalize
- Foundation types: Stage interface, RunState with 7 typed substates, MessageBuffer
- Pipeline orchestrator with setup/iteration/finalize 3-phase execution
- Callback-based PipelineDeps avoids circular import with agent package
- Feature-flagged via v3PipelineEnabled in Loop.Run()
- All 7 exit conditions preserved (no tools, max iter, truncation, loop kill,
read-only streak, tool budget, ctx cancel)
* feat(v3): wire pipeline callbacks to Loop methods + add 71 unit tests
Wire 15 of 17 PipelineDeps callbacks from Loop methods via closures:
- Context: LoadContextFiles, BuildMessages, EnrichMedia, InjectReminders
- Think: BuildFilteredTools, CallLLM (stream/sync)
- Prune: PruneMessages, CompactMessages
- Memory: RunMemoryFlush
- Finalize: SanitizeContent, FlushMessages, UpdateMetadata, BootstrapCleanup, MaybeSummarize
- Remaining: ExecuteToolCall, CheckReadOnly (deep loop.go integration)
Add comprehensive test suite (71 tests, all passing with -race):
- MessageBuffer: 10 tests (append, flush, replace, counts)
- Pipeline.Run: 14 tests (3-phase flow, exit conditions, ctx cancel)
- Stage tests: 47 tests (ThinkStage nudges/truncation, PruneStage budget,
ToolStage parallel/exit, ObserveStage content, CheckpointStage interval,
FinalizeStage cleanup)
* feat(v3): wire remaining 2 callbacks (ExecuteToolCall, CheckReadOnly)
Complete callback wiring — 17/17 PipelineDeps callbacks now active:
- ExecuteToolCall: resolves tool name, executes via registry, processes
result via existing processToolResult with loop detection bridge
- CheckReadOnly: delegates to checkReadOnlyStreak via bridge runState
- Bridge runState shares loop detection state between pipeline and agent
* fix(v3): eliminate data race in tool execution + capture injected messages
- Remove parallel tool execution path — serialize all tool calls to avoid
data races on shared bridgeRS (loop detector, media results, deliverables)
- Loop kill checked after each tool (mid-batch early exit)
- BuildFilteredTools: capture and append injected tool-awareness messages
- Rename test to reflect sequential execution
* feat(v3): wire ResolveWorkspace, safe parallel tools, ContextStage tests
- Wire ResolveWorkspace callback via workspace.NewResolver() with
ResolveParams from Loop fields (no longer a nil stub)
- Re-add safe parallel tool execution: split into ExecuteToolRaw
(parallel I/O) + ProcessToolResult (sequential state mutation)
with opaque rawData pass-through (no double execution)
- Add 12 unit tests for ContextStage (8) + MemoryFlushStage (3)
- Split tool callbacks to loop_pipeline_tool_callbacks.go (under 200 lines)
- Capture buildFilteredTools injected messages
* feat(v3): add episodic memory store + temporal KG columns
Phase 1 — Episodic Store:
- Migration 000039: episodic_summaries table with pgvector, FTS, L0 abstracts
- EpisodicStore PG impl: CRUD, hybrid FTS+vector search, ExistsBySourceID,
PruneExpired. Idempotent via source_id UNIQUE constraint.
Phase 2 — Temporal KG:
- Migration 000040: valid_from/valid_until on kg_entities + kg_relations,
partial indexes for current-facts queries, epoch→timestamptz backfill
- ListEntitiesTemporal: current-only, point-in-time, or include-expired modes
- SupersedeEntity: atomic expire-old + insert-new in single transaction
Schema version bumped to 40.
* fix(v3): review fixes for episodic store + temporal KG
- C1: Fix column name mismatch turn_count vs message_count in Go SQL
- C2: Remove redundant migration 000040 (000037 already adds temporal KG columns)
- H1: Use time.Time not int64 for TIMESTAMPTZ columns in SupersedeEntity
- H2: Add tenant_id scoping to Get/Delete for tenant isolation
- M2: Fix scanEntityTemporal to convert TIMESTAMPTZ→UnixMilli correctly
- L1: Remove unused uuid import from episodic_search.go
- Schema version corrected to 39 (only 000039 is new)
* feat(v3): implement consolidation pipeline with 3 event-driven workers
Event chain: session.completed → EpisodicWorker → episodic.created →
SemanticWorker → entity.upserted → DedupWorker
- EpisodicWorker: reuses compaction summary or calls LLM, generates L0
abstract (extractive), idempotent via source_id check
- SemanticWorker: extracts KG facts from episodic summary via existing
Extractor, sets temporal valid_from, publishes entity.upserted
- DedupWorker: runs DedupAfterExtraction on new entity IDs (terminal)
- L0 abstract: sentence-based extraction (~50 tokens), no LLM needed
- All workers registered via DomainEventBus.Subscribe()
* feat(v3): implement progressive loading with L0 auto-inject + unified search
- AutoInjector: searches episodic store, builds L0 prompt section (~200 tokens),
skips trivial messages via stopword filter
- L1Cache: in-memory LRU (500 entries, 1h TTL) for structured overviews
- UnifiedSearch: cross-tier search merging episodic + document results by score
- ContextStage integration: AutoInject callback appends memory section to system prompt
- MemorySection field added to ContextState for observability
* feat(v3): add memory_expand tool for L2 episodic retrieval
New tool: memory_expand(id) returns full episodic summary with metadata.
Complements memory_search L0/L1 results with deep L2 access.
Nil-safe: returns error message when episodic store not available.
Gateway wiring + memory_search depth param + kg_search temporal param
deferred to runtime integration phase.
* feat(v3): complete Phase 5 — tool extensions + gateway wiring
- memory_search: add depth param + episodic tier search merged with docs
- kg_search: add as_of temporal param, use ListEntitiesTemporal
- memory_expand: registered in gateway startup
- Gateway: Episodic field in Stores, PGEpisodicStore in factory,
embedding provider wired, tools connected to episodic store
* fix(v3): Phase 3 review fixes — tenant isolation + AutoInject args
- C1: Add tenant_id filter to ftsSearch, vectorSearch, List queries
(prevents cross-tenant episodic memory leaks)
- C2: Fix AutoInject callback signature — agent/tenant captured by
closure, only userMessage + userID passed explicitly
- H1: Add tenant_id to List query
* feat(v3): wire per-agent v3 flags from DB into dual-mode gate
Parse v3_pipeline_enabled, v3_memory_enabled, v3_retrieval_enabled from
agent other_config JSONB via ParseV3Flags(). Resolver now sets all flags
on LoopConfig so the existing gate in loop_run.go reads from DB.
- V3Flags struct + ParseV3Flags() + ValidateV3Flags() in store layer
- v3MemoryEnabled/v3RetrievalEnabled added to Loop, LoopConfig, PipelineConfig
- Auto-inject gated on V3RetrievalEnabled (was unconditional)
- Structured perf logging for v3 pipeline runs
- v3 flag validation on both WS agent.update and HTTP PUT endpoints
* feat(v3): wire AutoInjector into pipeline for L0 memory auto-inject
Create AutoInjector at gateway startup from episodic store, pass through
ResolverDeps → LoopConfig → Loop. Pipeline adapter builds AutoInject
callback capturing agent/tenant context via closure.
ContextStage already gates on V3RetrievalEnabled + AutoInject != nil.
* feat(v3): add tool metadata map + capability-based deny rules
Registry gains per-tool ToolMetadata map with RegisterWithMetadata()
and GetMetadata() (infers defaults from tool name when not explicit).
PolicyEngine gains DenyCapability() for RBAC integration — tools with
denied capabilities filtered at step 8 after existing 7-step pipeline.
* fix(v3): add RWMutex to PolicyEngine capability deny fields
DenyCapability() and SetRegistry() now guarded by sync.RWMutex.
FilterTools reads snapshot under RLock. Prevents data race when
capability rules are modified concurrently with tool filtering.
* feat(v3): implement delegate tool for inter-agent task delegation
New `delegate` tool wraps existing agent_links infrastructure
(CanDelegate, DelegateTargets). Supports async (fire-and-forget)
and sync (block with timeout) modes. Permission checked via
AgentLinkStore. Events emitted: delegate.sent/completed/failed.
DelegateRunFunc injected by gateway to avoid circular dependency.
* feat(v3): complete 3 deferred implementations
1. OrchestrationMode resolution: ResolveOrchestrationMode() checks
team membership → delegate links → spawn (priority order).
2. PG EvolutionMetricsStore: RecordMetric, QueryMetrics, aggregate
tool/retrieval metrics, TTL cleanup. All queries tenant-scoped.
3. BridgePromptBuilder: implements PromptBuilder interface by
delegating to existing BuildSystemPrompt(). Appends v3 memory
L0 section when enabled. Ready for template engine swap later.
* fix(v3): address code review findings on commits 5-6
- C1: CanDelegate now tenant-scoped (fail-closed on missing tenant)
- H1: Sync delegate timeout capped at 600s
- H2: Async goroutine gets 10min deadline (prevents leaks)
- H3: JSONB casts use COALESCE/NULLIF guards (handles missing fields)
- M1/M2: Remove dead code (formatVaultSection, memoryL0ToStrings)
* fix(teams): stop auto-creating agent_links for team members
Teams use agent_team_members table directly — agent_links caused
context confusion between team dispatch and delegation systems.
- Remove autoCreateTeamLinks() calls from team create + member add
- Remove link cleanup from member remove
- Remove dead autoCreateTeamLinks() function
- Append DELETE to migration 000039: clear team-created agent_links
* fix(v3): tenant isolation for all agent_links queries + PromptBuilder Instructions
- DelegateTargets, GetLinkBetween, SearchDelegateTargets,
SearchDelegateTargetsByEmbedding, DeleteTeamLinksForAgent all now
scoped by tenant_id (fail-closed on missing tenant)
- BridgePromptBuilder now maps Instructions/InstructionContent to
AGENTS.md context file (was silently dropped)
* feat(v3): wire orchestration mode + evolution metrics into agent loop
- Orchestration mode: resolver resolves mode from team/links, tool filter
hides delegate/team_tasks based on mode, prompt builder injects delegation
targets section
- Evolution metrics: non-blocking goroutine records tool execution metrics
(name, success, duration) via EvolutionMetricsStore in both v2 loop and
v3 pipeline paths (sequential + parallel)
- Fix review findings: tenant ID propagated via store.WithTenantID in
background goroutine, 5s timeout prevents goroutine leak
* feat(v3): implement suggestion engine with pluggable analysis rules
- PG EvolutionSuggestionStore: CRUD for agent_evolution_suggestions table
- SuggestionEngine: aggregates 7-day metrics, runs rules, deduplicates
pending suggestions per type before creating new ones
- 3 initial rules: LowRetrievalUsage (usage_rate<0.2), ToolFailure
(success_rate<0.1), RepeatedTool (>100 calls/week → suggest skill)
- EventSuggestionCreated event type added to eventbus
- Cron wiring deferred to gateway startup integration pass
* feat(v3): implement auto-adapt guardrails with apply/rollback
- AdaptationGuardrails: max delta per cycle, min data points, locked
params, rollback-on-drop percentage
- ApplySuggestion: applies threshold suggestions to agent other_config
JSONB, stores baseline for rollback
- RollbackSuggestion: restores baseline values from suggestion params
- EvaluateApplied: compares post-apply metrics to baseline, auto-rolls
back when quality drops beyond threshold
- Scope limited to retrieval params only (never security settings)
* feat(v3): wire evolution stores + daily/weekly cron for suggestions
- Add EvolutionMetrics + EvolutionSuggestions to Stores struct + PG factory
- Wire EvolutionMetricsStore into ResolverDeps (cmd/gateway_managed.go)
- Add gateway_evolution_cron.go: daily suggestion analysis + weekly
evaluation/rollback for applied suggestions
- Cron runs as background goroutine with 5-min timeout per cycle
* fix(v3): address code review findings on evolution engine
- C1: persist baseline parameters before marking suggestion as applied
(was building map but never saving — rollback would always fail)
- H1: add tenant_id isolation to UpdateSuggestionStatus, GetSuggestion,
and new UpdateSuggestionParameters method
* test(v3): add unit tests for orchestration, suggestions, guardrails, prompt
- orchestration_mode_test: orchModeDenyTools (4 modes) + ResolveOrchestrationMode
(4 scenarios with mock stores)
- suggestion_rules_test: LowRetrievalUsage, ToolFailure, RepeatedTool with
threshold boundary tests (at/below/above min data points)
- evolution_guardrails_test: DefaultGuardrails values + CheckGuardrails
(insufficient data, locked params, zero-min fallback)
- prompt_builder_orchestration_test: BridgePromptBuilder orchestration section
presence/absence across 4 scenarios + target content verification
* test(v3): add integration tests for evolution metrics + suggestions
- Test helper: shared PG connection with sync.Once migration, per-test
tenant+agent seed with cleanup
- Evolution metrics: RecordMetric, AggregateToolMetrics (success rate),
Cleanup (TTL deletion)
- Evolution suggestions: full CRUD, UpdateSuggestionParameters (baseline
persist), tenant isolation (cross-tenant read blocked)
- Pipeline E2E: seed 25 failed tools + 55 low-usage retrievals, verify
SuggestionEngine creates suggestions, verify dedup on second run
- Fix: migration 039 de-duped (episodic_summaries already in 037)
- Fix: NULL reviewed_by scan via sql.NullString
* feat(v3): add HTTP API handlers for evolution, vault, episodic, orchestration, v3-flags
5 new handler files exposing v3 backend stores as REST endpoints:
- evolution_handlers.go: metrics query/aggregate + suggestions CRUD
- vault_handlers.go: cross-agent document listing + search + links
- episodic_handlers.go: episodic summaries list + hybrid search
- orchestration_handlers.go: computed mode + delegate targets (read-only)
- v3_flags_handlers.go: per-agent v3 feature flag get/toggle
Store fixes from code review:
- episodic FTS: use inline to_tsvector (no stored tsv column)
- episodic: conditional user_id filter in List + Search (admin view)
- episodic: add tenant_id to ExistsBySourceID + PruneExpired
- evolution: require tenant_id in context (no struct fallback)
- evolution: check RowsAffected on suggestion updates
- vault: optional agent_id filter in ListDocuments (cross-agent)
* feat(v3): add web UI for evolution tab, v3 settings, vault page, episodic memory
Agent Detail enhancements:
- V3 Settings section: pipeline/memory/retrieval flag toggles
- Orchestration section: mode badge + delegate targets display
- Evolution section: added metrics + suggestions v3 flag toggles
- Evolution tab: Recharts metrics charts + suggestion review table
with approve/reject/rollback actions + guardrails card
New pages:
- /vault: Knowledge Vault document registry with cross-agent listing,
hybrid search dialog, document detail with wikilinks
- Memory page: added Episodic Memory tab with summary cards,
expandable details, key topic badges, and hybrid search
Infrastructure:
- HttpClient: added patch() method
- Query keys: v3Flags, orchestration, evolution namespaces
- 4 new hooks: use-v3-flags, use-orchestration, use-evolution-metrics,
use-evolution-suggestions, use-vault, use-episodic
- i18n: vault namespace (en/vi/zh), agents + memory keys updated
- Reused formatRelativeTime from lib/format.ts (eliminated 3 duplicates)
* refactor(http): add bindJSON helper and migrate all decode call sites
Replace 36 json.NewDecoder(r.Body).Decode + error blocks with bindJSON
across 20 HTTP handler files. Standardizes decode error responses to
structured writeError format. Fixes unchecked decode in handleIndexAll.
* refactor(store): adopt sqlx for PG scan operations (Phase 1+2)
Add jmoiron/sqlx v1.4.0 with camelToSnake json tag mapper.
Migrate scan-heavy PG store methods to sqlx Get/Select:
- tracing.go: GetTrace, ListTraces, ListChildTraces, GetTraceSpans, GetCostSummary
- heartbeat.go: Get, ListDue, ListLogs
- providers.go: GetProvider, GetProviderByName, ListProviders, ListAllProviders
- mcp_servers.go: GetServer, GetServerByName, ListServers
- pairing.go: ListPending, ListPaired
- agents_export_queries.go: 5 export functions
- agents_export_team_queries.go: exportTeamMembers, ExportAgentLinks
All writes (INSERT/UPDATE/DELETE), execMapUpdate, and dynamic WHERE
builders remain raw SQL. Zero behavior change.
* refactor(store): adopt sqlx for SQLite scan operations (Phase 3)
Migrate SQLite store scan methods to sqlx Get/Select:
- providers.go: GetProvider, GetProviderByName, ListProviders, ListAllProviders
- tenants.go: GetTenant, GetTenantBySlug, ListTenants, GetTenantUser, ListUsers, ListUserTenants
- mcp_servers.go: GetServer, GetServerByName, ListServers
Create sqlx_scan_structs.go with sqliteTime-aware scan structs
(providerRow, tenantRow, tenantUserRow, mcpServerRow) to handle
SQLite TEXT timestamp parsing via StructScan.
* refactor(store): migrate PG bulk scan operations to sqlx (Phase 4)
Migrate scan-heavy methods across 6 PG store files:
- tenant_store.go: GetTenant, GetTenantBySlug, ListTenants, GetTenantUser,
ListUsers, ListUserTenants — removed 3 scan helpers
- teams.go: ListTeams, GetTeam, ListMembers, ListMembersByTenant
- teams_tasks_activity.go: ListComments, ListEvents, ListFollowUps
- pending_message_store.go: ListPending, ListByHistoryKey
- skills_grants.go: ListAgentGrants
- config_permissions.go: CheckPermission
~20 scan ops converted. Files with encryption post-processing,
pq.Array, pgvector, or dynamic SQL kept raw.
* refactor(store): extract shared CamelToSnake mapper, add UUIDArray usage note
- Move camelToSnake to internal/store/column_mapper.go (DRY)
- Both pg and sqlitestore packages now import shared CamelToSnake
- Add planned-use comment on UUIDArray type
* refactor(cli): migrate commands from config.json to HTTP API, add providers/setup/TUI
- Add unified HTTP client (gateway_http_client.go) with auth, error parsing, typed generics
- Rewrite agent list/add/delete to use gateway HTTP API instead of config.json
- Rewrite channels list to HTTP API, add channels add/delete subcommands
- Replace models command with full providers CRUD (list/add/update/delete/verify)
- Add setup wizard command (provider → agent → channel post-onboard flow)
- Add Bubble Tea TUI behind build tag (tui/!tui with noop fallback)
- Update onboard next-steps to mention goclaw setup
- Add build-tui Makefile target
- Fix URL path injection (url.PathEscape on all user-supplied path segments)
- Fix UTF-8 truncation in skills description display
* refactor(store): add explicit db struct tags, fix sqlx mapper for heartbeat scan error
Switch sqlx mapper from NewMapperFunc (which only applies CamelToSnake to
field names, not tag values) to NewMapperFunc("db", CamelToSnake) with
explicit db:"column_name" tags on all store structs.
Root cause: NewMapperFunc("json", fn) sets mapFunc but not tagMapFunc,
so camelCase json tags like "agentId" were used as-is instead of being
converted to "agent_id", causing "missing destination name" scan errors.
Fix: use db struct tags as the source of truth for column mapping.
Every DB entity field gets db:"column_name", nested JSON configs and
runtime-only structs get db:"-".
* test(store): add integration tests for 13 store interfaces (70 tests)
Cover Tier 1 (critical) + Tier 2 (security) stores with integration tests
running against pgvector pg18. Coverage from 2.4% to ~54%.
Stores tested: Session, Agent, Team/Task, Memory, KnowledgeGraph, Vault,
MCP Server, API Key, ConfigPermission, Contact.
Infrastructure: fixture builders (seedTeam, seedMCPServer, etc.),
mock EmbeddingProvider, multi-tenant helpers, expanded cleanup.
* fix(store): resolve NULL scan bugs in MCP server and task metadata
- mcp_servers: COALESCE nullable TEXT columns (display_name, command,
url, api_key, tool_prefix) to prevent sqlx scan failures
- mcp_servers_access: COALESCE nullable JSONB columns in ListAgentGrants
(tool_allow, tool_deny, config_overrides) to prevent silent row drops
- teams_tasks: default task metadata to '{}' instead of nil to satisfy
NOT NULL constraint on CreateTask
- sqlx_helpers: export InitSqlx for integration test setup
* feat(pipeline): fix v3 pipeline context injection, tracing, KG temporal filters
- Pipeline context: add InjectContext + LoadSessionHistory callbacks to
ContextStage, propagate enriched ctx via state.Ctx for iteration stages
- Pipeline tracing: wrap makeCallLLM with emitLLMSpanStart/End, wrap
makeExecuteToolCall/Raw with emitToolSpanStart/End
- Token counter: switch pipeline from FallbackCounter to TiktokenCounter
- KG temporal: add valid_until IS NULL filter to all entity/relation
queries (list, search, vector, FTS, traversal CTE, stats)
- Skills: add SkillEmbedder interface for future hybrid BM25+vector search
- Cache: remove unused tenantResolve dead code from PermissionCache
- Store: fix NULL scan bugs in tracing metadata and agent skill_nudge
- Test: add TestStoreKG_TemporalFilter integration test
- UI: add v3 version badge, evolution section, memory/traces improvements
* refactor(store): migrate KG store from raw sql.Rows to sqlx StructScan
Migrate 6 knowledge graph store files from manual rows.Scan() to
pkgSqlxDB.GetContext/SelectContext with intermediate scan row structs.
- Add entityRow, relationRow, traversalRow, dedupCandidateRow structs
with json.RawMessage for jsonb and time.Time for timestamptz columns
- Add toEntity()/toRelation() converters (UnixMilli + json.Unmarshal)
- Add sqlxTx() helper for wrapping *sql.Tx with sqlx mapper
- Fix ScanDuplicates passing time.Now().Unix() to TIMESTAMPTZ column
- Fix ListEntitiesTemporal missing tenant scope (scopeClause)
- Fix SupersedeEntity missing tenant scope and tenant_id on INSERT
- Fix DedupCandidate.CreatedAt using Unix() instead of UnixMilli()
- Update agents_export_queries.go to reuse new scan row structs
- Net -160 lines of manual scan boilerplate removed
* refactor(store): migrate memory, skills, agents, sessions, mcp, cron, vault stores to sqlx
Batch migration of 19 store files from raw rows.Scan() to
pkgSqlxDB.GetContext/SelectContext with intermediate scan row structs.
Groups migrated:
- Memory: memory_docs, memory_admin, memory_search, memory_embedding_cache
- Episodic: episodic_search, episodic_summaries
- Skills: skills, skills_admin, skills_embedding, skills_export_queries
- Agents: agents (backfill+shares), agents_context, agents_export_team_standalone
- Sessions: sessions_list (List, ListPaged, ListPagedRich)
- MCP: mcp_servers_access, mcp_export_queries
- Cron: cron_exec (GetRunLog)
- Vault: vault_documents (ListDocuments, ftsSearch, vectorSearch)
- Tenant: tenant_configs (ListDisabled, ListAll)
7 new scan row files created. Net -510 lines of manual scan boilerplate.
INSERT/UPDATE/DELETE and scalar COUNT queries kept as raw SQL.
* fix(store): fix 3 sqlx scan struct db tag issues found by audit
- Fix vault FTS alias mismatch: `AS rank` → `AS score` (critical: runtime scan error)
- Fix episodic key_topics type: json.RawMessage → pq.StringArray (TEXT[] column)
- Fix agentShareRow.CreatedAt: string → time.Time, wire to output struct
* feat(providers): implement Wave 2 provider resilience and intelligence
9-phase implementation covering:
- Request middleware chain with composable body transformers
- OpenAI prompt caching, service tier, and fast mode middlewares
- Error classification (9 categories) with two-tier failover
- Model registry with forward-compat resolvers (Anthropic + OpenAI)
- Embedding providers (OpenAI + Voyage) with 1536-dim validation
- Cooldown/probe system with per-provider:model state tracking
- Markdown-aware chunking shared across 5 channels
- Session recall via FTS + pgvector on episodic summaries
- Dreaming/promotion pipeline for long-term memory consolidation
Migrations: 000040 (episodic search index), 000041 (promoted_at column)
Schema version: 39 → 41
* feat(providers): wire model registry into gateway provider construction
Create InMemoryRegistry with Anthropic + OpenAI forward-compat resolvers
at gateway startup. Pass to all Anthropic and OpenAI providers created
from both config and DB sources.
* feat(consolidation): wire DomainEventBus and consolidation pipeline
Create DomainEventBus at gateway startup, thread through resolver →
LoopConfig → Loop → PipelineDeps. Emit session.completed event after
each run finalization. Register consolidation pipeline (episodic →
semantic → KG dedup → dreaming) with event bus subscriptions.
* fix(store): fix episodic key_topics pq.Array, ON CONFLICT, and migration 040 immutability
- episodic_summaries.go Create: json.Marshal(KeyTopics) → pq.Array (text[] column)
- episodic_search.go scanEpisodic/scanEpisodicRow: json.RawMessage → pq.StringArray
- episodic_summaries.go Create: ON CONFLICT add WHERE source_id IS NOT NULL for partial index
- migration 040: add immutable_array_to_string wrapper (array_to_string is STABLE in PG)
* test(store): add 17 integration tests for skills, cron, episodic, tenant configs
- Skills store: 6 tests (CRUD, grants, tenant isolation)
- Cron store: 4 tests (job CRUD, run log sqlx scan, pagination, tenant isolation)
- Episodic store: 4 tests (summary CRUD, list, FTS search, tenant isolation)
- Tenant configs: 3 tests (tool/skill disable, list, tenant isolation)
- Test helper: add cleanup for skills, cron, episodic tables
* fix(permissions): use cron-specific permission check for cron tool (#725)
* fix(security): harden exec path exemption matching (#721)
- Add absolute path exemption for dataDir/skills-store/ (fixes skill
scripts using absolute paths like /app/data/skills-store/ being denied)
- Strip surrounding quotes before prefix matching (LLMs often quote paths)
- Reject path traversal ("..") in exempt fields to prevent escape
- Switch from "any field exempt → skip" to per-field matching: only exempt
if ALL fields that match the deny pattern are individually exempt
- Closes pipe/comment bypass vectors where an exempt path in one argument
would exempt the entire command including non-exempt paths
Includes 27 test cases covering: legitimate access, quoted paths,
path traversal, unicode bypass, pipe/comment bypass, mixed args.
* fix(permissions): use cron-specific permission check for cron tool
Cron tool was hardcoded to check `file_writer` configType via
CheckFileWriterPermission(), ignoring the `cron` configType that
the UI actually saves when granting cron permissions. This caused
agents in group chats to be denied cron access even with correct
permission configured.
Add ConfigTypeCron constant and CheckCronPermission() that checks
`cron` configType first, falling back to `file_writer`.
---------
Co-authored-by: Viet Tran <viettranx@gmail.com>
* fix(chat): load message history on first conversation click (#730)
* fix(chat): load message history when selecting existing conversation from clean state
The skipNextHistoryRef was unconditionally set when sessionKey transitioned
from empty to non-empty. This prevented loadHistory() from running when
clicking an existing conversation from the initial /chat page. The skip
was only intended for the new-chat send flow where the optimistic message
is already displayed.
Guard the skip with expectingRunRef so it only activates when a message
send is in flight.
Closes #729
* docs: add UI diff evidence for PR #730
Before/after screenshots and HTML comparison report showing
first conversation click behavior fix.
* feat(whatsapp): port native WhatsApp channel with whatsmeow from dev
Cherry-pick 0db1e93a with manual conflict resolution:
- cmd/channels_cmd.go: kept dev-v3 HTTP API approach (not config-based)
- go.mod/go.sum: merged deps, ran go mod tidy
* feat(ui): v3 web UI enhancement — branded loading, rich markdown, vault graph, sidebar polish
- Branded loading: HTML pre-loader with logo pulse/shimmer, fade-out on app ready, PageLoader logo swap
- Rich markdown: wikilinks, mermaid (lazy-loaded), math/KaTeX, callouts/admonitions plugins
- Vault graph: force-directed document graph view with table/graph toggle
- Agent filters: type filter (open/predefined) on agents page
- Sidebar: tenant name + role badge in footer
- Query keys: add vault + episodic entries
* fix(ui): address code review — mermaid XSS, Safari compat, cache key
- Change mermaid securityLevel from "loose" to "strict" (XSS prevention)
- Add requestIdleCallback fallback for Safari < 17
- Fix vault all-links query key to use sorted doc IDs (stale cache fix)
- Remove dead abortRef from MermaidBlock
- Add prefers-reduced-motion to loader animation
* docs: update CLAUDE.md with v3 architecture + complete changelog
- Add 10 new v3 internal modules to project structure (pipeline, eventbus, consolidation, tokencount, vault, workspace, etc.)
- Add native WhatsApp channel, edition system, providerresolve, updater to structure
- Update Key Patterns with 8 v3-specific patterns (pipeline, eventbus, 3-tier memory, vault, evolution, orchestration, middleware)
- Add comprehensive V3 Redesign section to changelog covering:
- 8-stage pipeline with dual-mode gate
- DomainEventBus + consolidation workers
- 3-tier memory (working/episodic/semantic)
- Knowledge Vault with wikilinks
- Self-evolution engine
- Orchestration + delegate tool
- WorkspaceContext resolver
- ModelRegistry + provider adapter
- Feature flags
- Request middleware
- sqlx migration + 70+ integration tests
* fix(security): harden file path validation, tenant isolation, and tool access
- handleSign: validate path within workspace/dataDir before signing HMAC token
- handleSign: enforce tenant-scoped restriction for RBAC-enabled editions
- handleServe: add workspace/dataDir boundary check for ft= signed requests
- handleServe: remove cross-tenant findInWorkspace fallback for ft= requests
- TenantDataDir/TenantWorkspace: guard against empty slug resolving to parent
- exec tool: add tenants/ to AllowPathExemptions for tenant skill execution
- list_files: add AllowPaths support and wire skills directory access
* docs(v3): update 16 docs + add 2 new docs for v3 architecture
Update all docs/ to reflect v3 implementation (64 commits, 31K insertions):
- 00: architecture overview with 7 new packages
- 01: agent loop with pipeline, orchestration, evolution
- 02: providers with Wave 2 resilience (middleware, failover, registry)
- 03: tools with delegate, vault_search, vault_link, memory_expand
- 04,08,10,11,14,23: targeted v3 additions
- 06: store with 6 new tables, promoted columns, sqlx
- 07: memory with 3-tier architecture, consolidation pipeline
- 18,19: HTTP/WS API with v3 endpoints and methods
- 21: evolution system (metrics, suggestions, auto-adapt)
- model-steering: model registry relationship
New docs:
- 22-v3-http-endpoints.md: 12 v3 HTTP endpoints
- 24-knowledge-vault.md: vault architecture, wikilinks, search
* feat(v3): wire delegate tool, fix pipeline callbacks, clean dead code
Wire delegate tool end-to-end:
- Register DelegateTool in gateway with DelegateRunFunc
- Implement runFn: resolve target agent, build session key, propagate tracing
- Add async announce via msgBus (reuses subagent announce handler)
- Populate context fields (TenantID, Channel, ChatID) for routing
- Set DelegationID + ParentAgentID on RunRequest for event correlation
Fix pipeline callbacks:
- BreakLoop now completes remaining stages in current iteration
- EnrichMedia signature updated to use RunState for message buffer access
- Add non-streaming event emission for channel compatibility
- Fix user message flush tracking in v3 pipeline
Clean dead code (510 LOC removed):
- Delete memory/l1_cache.go, unified_search.go (superseded by vault search)
- Delete vault/auto_injector_impl.go, retriever_impl.go (never wired)
- Delete vault/sync_worker.go (never started)
- Remove orphaned EventMemoryLint, EventSuggestionCreated constants
* feat(v3): finalize stage — emit session.completed, NO_REPLY, strip directives
- Emit session.completed event for consolidation pipeline (episodic → semantic → dreaming)
- Detect NO_REPLY before flush so silent content is persisted for context
- Strip [[...]] message directives from user-facing content (v2 parity)
- Wire StripMessageDirectives, IsSilentReply, EmitSessionCompleted callbacks
* feat(vault): full CRUD — backend endpoints, UI dialogs, content preview
Backend (5 new HTTP endpoints):
- POST/PUT/DELETE /v1/agents/{id}/vault/documents — create, update, delete
- POST/DELETE /v1/agents/{id}/vault/links — create, delete
- Server-side validation for doc_type and scope enums
- Agent ownership verification on link creation
- FK cascade handles link cleanup on document delete
Frontend (React):
- Create document dialog (title, path, type, scope)
- Edit mode in detail dialog (inline title/type/scope editing)
- Delete document with confirmation
- Create link dialog (from/to doc, link type, context)
- Delete link with inline confirmation on badges
- Content preview (collapsible, lazy-loads via /v1/storage/files/)
- Mutation hooks with query invalidation
- i18n keys for en/vi/zh
* fix(v3): critical pipeline parity fixes — ChatRequest, reasoning, passback, media
- Enrich ChatRequest with all provider options (temperature, sessionKey,
agentID, userID, channel, workspace, tenantID) matching v2
- Add ResolveReasoningDecision for thinking models (o3, DeepSeek-R1, Kimi)
- Wire uniquifyToolCallIDs to prevent OpenAI 400 on duplicate IDs
- Add assistant message passback (Phase, RawAssistantContent) for Anthropic
- Emit block.reply for intermediate content (non-streaming channels)
- Add ContentSuffix append + ForwardMedia merge in FinalizeStage
- Build final assistant message with MediaRefs for session persistence
- Use effectiveMaxTokens() + OptMaxTokens constant
- Guard truncation retry on len(ToolCalls) > 0 + parseError check
- Accumulate ThinkingTokens in usage
- Deduplicate emitRun closure (shared from callbackSet)
- Fix EnrichMedia to receive RunState with actual messages
- Persist user message in makeFlushMessages (matching v2)
* fix(store): coerce NOT NULL JSONB columns to empty object on agent update
When switching provider away from ChatGPT OAuth, the UI sends
chatgpt_oauth_routing: null which violates the NOT NULL constraint.
Coerce null → '{}' for all NOT NULL JSONB promoted columns:
chatgpt_oauth_routing, reasoning_config, workspace_sharing,
shell_deny_groups, kg_dedup_config.
* feat(v3): v3 info modal redesign, agent links CRUD tab, sidebar rename
- Rewrite v3 info modal with 8 feature cards (pipeline, memory,
retrieval, vault, evolution, orchestration, resilience, registry)
with v2→v3 comparisons, stat badges, full i18n (en/vi/zh)
- Add tabbed layout to teams page: Agent Teams | Agent Links tabs
- Agent Links tab with full CRUD via existing WS RPC methods
(list/create/edit/delete) with Radix Select, Combobox, mutual
agent exclusion in create dialog
- Sidebar menu renamed to "Agent Link & Team"
- Backend: add source_display_name, source_emoji, target_emoji
to agent link joined queries for consistent display
* fix(store): include personal chats in cron delivery targets
Add 'user' contact_type to ListDeliveryTargets SQL filter in both
PG and SQLite stores. Previously only group/topic contacts appeared
in cron channel/chat dropdowns.
* fix(v3): duplicate messages, missing thinking, span numbering
- ThinkStage: skip AppendPending for final answer (no tool calls),
let FinalizeStage build the definitive message with sanitization
and MediaRefs — fixes duplicate assistant messages in session history
- RunResult: add Thinking field, propagate through v2 finalizeRun,
v3 convertRunResult, run.completed event, and chat.send response
- UI: capture thinkingRef before clearing on run.completed, include
in final message object so thinking renders without page refresh
- Span numbering: pass Iteration+1 in v3 callback to match v2's
1-based iteration display in trace span names
* fix(v3): wire delegation targets into BuildSystemPrompt
buildOrchestrationSection() was implemented and tested but never
wired into the actual BuildSystemPrompt() flow. Only the unused
BridgePromptBuilder had it. Add DelegateTargets + OrchMode fields
to SystemPromptConfig and inject "## Delegation Targets" section
so agents with agent_links see their delegation targets in prompt.
* fix(v3): sync mediaResults from bridgeRS to pipeline state
syncBridgeToState copied loopKilled, asyncToolCalls, deliverables
from the v2 bridgeRS but missed mediaResults. Tool results with
MEDIA: prefix were extracted by processToolResult into bridgeRS
but never propagated to state.Tool.MediaResults — causing
FinalizeStage to produce empty MediaRefs and RunResult.Media.
* fix(v3): populate SessionCompletedPayload in session.completed events
Both v2 loop and v3 pipeline emitted session.completed events with
nil Payload, causing episodicWorker type assertion to fail silently.
Episodic summaries were never created.
- Expand EmitSessionCompleted callback to pass msgCount, tokensUsed,
compactionCount from pipeline state
- V3 path: build payload from state.Messages.TotalLen(),
state.Think.TotalUsage, state.Compact.CompactionCount
- V2 path: build payload from history + rs.totalUsage +
sessions.GetCompactionCount()
* fix(v3): remaining pipeline parity gaps — skill postscript, team task count
- Add SkillPostscript callback to PipelineDeps + FinalizeStage,
matching v2's skill evolution nudge after complex tool runs
- Wire makeSkillPostscript() in adapter with same logic as v2
(skillEvolve + skillNudgeInterval + totalToolCalls threshold)
- Sync teamTaskCreates from bridgeRS to pipeline EvolutionState
(was already syncing teamTaskSpawns but missed creates)
* fix(evolution): add JSON struct tags to metric aggregates
ToolAggregate and RetrievalAggregate had no JSON tags, causing
Go to marshal field names as PascalCase (ToolName, CallCount)
while the UI expects snake_case (tool_name, call_count). Charts
rendered empty containers with no data bars.
Also change AvgDuration (time.Duration) to AvgDurationMs (float64)
for JSON-friendly serialization — time.Duration marshals as
nanoseconds which is unusable in frontend.
* fix(v3): add debug logging to episodic worker for consolidation pipeline
Add INFO/WARN logs at entry, summary decision, and creation points
to diagnose why episodic_summaries table stays empty in production.
* chore: silence noisy tenant_cache debug logs
* refactor(ui): replace v3 settings section with engine version picker + tabbed info modal
Replace flat toggle list with radio-style version cards (v2/v3) and
feature mini-cards. Redesign v3 info modal from single scroll to 3-tab
layout (Core Engine, Memory & Knowledge, Orchestration) with Lucide
icons and v2→v3 comparison cards.
- Add batchUpdate to use-v3-flags hook for atomic v2 switch
- Create engine-version-section with VersionCard + FeatureMiniCard
- Create v3-info-modal/ with 5 modular components (<45 LOC each)
- Add i18n keys (detail.engine + v3Info.tabs) for en/vi/zh
- Wire into agent-overview-tab, agent-header, agent-card
- Delete v3-settings-section.tsx + agent-v3-info-modal.tsx
* feat(v3): pass media files through delegate tool results
Delegate tool now carries media files (images, audio, etc.) produced
by the delegatee back to the parent agent. DelegateRunFunc returns
DelegateResult{Content, Media} instead of plain string.
- Add DelegateResult struct with Content + Media fields
- Convert agent.MediaResult to bus.MediaFile in gateway wire
- Attach media to sync result and async announce message
- Add MediaCount to DelegateCompletedPayload for observability
- Set MetaParentAgent in async announce metadata
* merge: bring main bug fixes into dev-v3
Cherry-pick 8 commits from origin/main:
- fix(telegram): handle group-to-supergroup migration (#698)
- feat(providers): add OpenRouter identification headers (#705)
- fix: deterministic prompt ordering for LLM cache hit (#719)
- fix(security): harden exec path exemption matching (#721)
- fix: invalidate storage size cache on delete and move (#726)
- fix: use errors.Is() for sentinel comparisons (#727)
- fix(desktop): add defaultValues to form dialogs (#737)
- Release: credential resolver, WhatsApp native, exec hardening (#754)
Conflict resolution:
- store/pg exports: accept main's errors.Is() additions
- shell.go: accept main's extracted matchesAnyPathExemption helper
- gateway_providers: merge both WithAnthropicName + WithAnthropicRegistry
- loop_types + resolver: merge v3 fields (OrchMode, DelegateTargets,
EvolutionMetricsStore) with main's new UserResolver/ContactStore
- gateway_setup: keep dev-v3's tenant-scoped path exemptions
- channels_cmd: keep dev-v3's HTTP API approach
* feat(v3): add foundation packages for architecture refactor (Phase 1)
Purely additive — zero changes to existing files. Creates shared types
and helpers that Phase 2-4 will migrate callers to:
- internal/store/base/: Dialect interface, BuildMapUpdate, nullable/JSON
helpers, scope clause builder, table metadata (44 tests)
- internal/orchestration/: ChildResult capture from v2/v3, media type
conversion with round-trip tests (10 tests)
- internal/providers/sse_reader.go: Shared SSE scanner replacing inline
bufio.Scanner boilerplate in 3 providers (8 tests)
* refactor(store): unify pg/ and sqlitestore/ helpers via base/ package (Phase 2)
- Create PG and SQLite dialect implementations (base.Dialect interface)
- Replace 15 duplicate helpers in pg/helpers.go with aliases to base.*
- Replace 13 duplicate helpers in sqlitestore/helpers.go with aliases
- Rewrite execMapUpdate and execMapUpdateWhereTenant to use
base.BuildMapUpdate with dialect-specific placeholders
- Rewrite scopeClause/scopeClauseAlias as thin wrappers around
base.BuildScopeClause
- Remove duplicate execMapUpdateWhereTenant from pg/agents.go
pg/helpers.go: 267→175 LOC, sqlitestore/helpers.go: 226→130 LOC
* refactor(orch): extract BatchQueue[T] generic for announce queues (Phase 3)
- Create internal/orchestration/batch_queue.go: generic producer-consumer
queue replacing duplicated sync.Map+mutex pattern (10 tests, race-safe)
- Simplify cmd/gateway_announce_queue.go: team queue uses BatchQueue,
removes announceQueueState/getOrCreate/drain/tryFinish (~40 LOC saved)
- Simplify cmd/gateway_subagent_announce_queue.go: subagent queue uses
BatchQueue, same pattern reduction (~40 LOC saved)
- Update cmd/gateway_consumer_handlers.go: callers use new signatures
* fix: remove defer accumulation in announce loop, clean comment tombstone
- Remove `defer ptd.ReleaseTeamLock()` inside for loop that accumulated
deferred calls per iteration (explicit ReleaseTeamLock already called)
- Remove dead comment tombstone in pg/agents.go
* fix: scope defer in announce loop via closure for panic safety
Wrap announce processing in closure so defer ptd.ReleaseTeamLock()
runs once per iteration instead of accumulating. Explicit release
still called for normal path; defer catches panics.
* refactor(providers): wire shared SSEScanner into 3 providers (Phase 4b)
Replace inline bufio.Scanner+SSE parsing boilerplate with shared
SSEScanner from sse_reader.go in:
- openai.go: data-only SSE (OpenAI, DashScope, Kimi)
- codex.go: data-only SSE (OpenAI Codex)
- anthropic_stream.go: event+data SSE (uses EventType() for switch)
Removes ~24 LOC of duplicated scanner setup + manual line parsing.
* refactor(agent): force v3 pipeline, remove v2 runLoop (Phase 4A)
- Delete runLoop() from loop.go (~745 LOC removed), keep shared helpers
(resolveToolCallName, hasParseErrors, truncateToolArgs)
- Remove v2/v3 gate in loop_run.go: always call runViaPipeline()
- Remove v3PipelineEnabled field from Loop struct + LoopConfig + resolver
- Always resolve workspace in loop_context.go (was behind v3 gate)
- Deprecate PipelineEnabled in V3Flags (kept for backward compat parsing)
All agents now always use v3 pipeline. No behavioral change for agents
that already had v3_pipeline_enabled=true (which was all production agents).
* refactor(gateway): decompose gateway.go from 1295 to 476 LOC (Phase 4B)
Extract sections of runGateway() into focused files:
- gateway_deps.go: gatewayDeps struct for shared state
- gateway_http_wiring.go: wireHTTPHandlersOnServer (~207 LOC)
- gateway_events.go: event subscribers + teamTaskEventType (~367 LOC)
- gateway_lifecycle.go: signal handling, shutdown, server start (~222 LOC)
- gateway_tools_wiring.go: cron/heartbeat/session tool wiring (~116 LOC)
Also extracted: startCronAndHeartbeat, makeDelegateAnnounceCallback.
Pure structural refactoring — no behavior change.
* test(agent): add v3 force migration guard tests
Verify v2 runLoop is deleted, V3PipelineEnabled field removed from
LoopConfig, and V3Flags backward compat parsing still works.
Compile-time guards prevent accidental re-introduction of v2 code.
* feat(delegate): wire ChildResult + fix media passthrough (Phase 3 gap)
- Use orchestration.CaptureFromRunResult in delegate run callback
to standardize result capture via ChildResult
- Use MediaResultToBusFiles in CaptureFromRunResult (DRY)
- Fix delegate_tool.go metadata key: delegate_id → delegation_id
* refactor(ui): remove v2/v3 pipeline toggle, always show V3 (Phase 4E)
- engine-version-section.tsx: remove pipeline toggle, show V3 read-only
- use-agent-version.ts: always return "v3" (no flag check)
- use-v3-flags.ts: remove v3_pipeline_enabled from toggleable flags
- Update i18n strings (en/vi/zh): remove v2-specific tooltip text
* test: add Phase 5 test infrastructure for v3 architecture refactor
- pg/pg_dialect_test.go: 4 tests for PG Dialect (placeholder, transform,
returning, interface compliance)
- sse_reader_test.go: 4 edge case tests (empty data, scanner error,
event type persistence, no data after [DONE])
- gateway_announce_format_test.go: 10 tests for team + subagent
announce formatting (single/batch, success/failure, snapshot)
Coverage: base/ 96%, orchestration/ 100%, providers/ 57%
* chore: remove stale runLoop references + apply go fix (Phase 6)
- Update comments referencing deleted runLoop in pipeline callbacks,
loop_types, stage.go
- go fix: reflect.TypeFor, range over int, strings.Builder
* docs: update architecture docs for v3 refactor completion (Phase 6)
- CLAUDE.md: add store/base/, orchestration/ to project structure;
remove v2 runLoop and dual-mode gate references; rename Pipeline (v3)
to Pipeline; add SSEScanner and BatchQueue to key patterns
- docs/00-architecture-overview.md: update module map with new packages,
remove [V3] markers (now standard)
- docs/17-changelog.md: add V3 Architecture Refactor entry (6 phases)
* feat(evolution): skill draft auto-generation + go fix cleanup
- Add skill draft template generation from evolution suggestions
- Wire skill apply endpoint in evolution HTTP handlers
- Apply go fix across codebase (range-over-int, reflect.TypeFor, etc.)
- Minor refactors: simplify switch/case, reduce string builder allocs
- Gateway deps: add skills loader field
* perf(prompt): deterministic tool order + Anthropic cache boundary split
Sort tool names in buildToolingSection for cache-stable output.
Insert GOCLAW_CACHE_BOUNDARY marker before Time section; Anthropic
provider splits system prompt into 2 blocks (stable cached, dynamic not).
Backward compat: no marker → single cached block.
* perf(prompt): optimize cache boundary position + add Execution Bias
Move Memory Recall and stable context files (AGENTS.md, TOOLS.md,
USER_PREDEFINED.md) above cache boundary. Dynamic per-user files
(USER.md, BOOTSTRAP.md) stay below. Add Execution Bias section
(full mode only) forcing action-oriented tool use. Fix duplicate
header when Project Context split across boundary.
* feat(prompt): add PromptMode task/none + 3-layer resolution
Expand PromptMode from full|minimal to full|task|minimal|none.
Task mode = enterprise automation: keeps Tooling, Execution Bias,
Safety-slim, Persona-slim, Skills-search, MCP-search, Memory-slim,
Workspace, Runtime, Delegation. Drops verbose sections (Tool Call
Style, Self-Evolution, Spawning, Recency, etc.).
None mode returns identity line only. 3-layer mode resolution:
runtime override > auto-detect (subagent/cron) > agent config
(other_config.prompt_mode) > default (full).
* feat(prompt): provider prompt contributions (stable/dynamic/overrides)
Add PromptContribution struct + PromptContributor interface for
provider-specific prompt customizations. Providers can inject
StablePrefix (before cache boundary), DynamicSuffix (after boundary),
or override sections by ID (e.g. execution_bias). Nil-safe: providers
that don't implement the interface get default behavior.
* feat(prompt): pinned skills with hybrid inline+search mode
Add per-agent pinned_skills config (max 10, from other_config JSONB).
Pinned skills always inline in prompt via BuildPinnedSummary.
Non-pinned discovered via skill_search. Hybrid section shows both
pinned XML and search instructions. Works in full and task modes.
* fix(prompt): validate prompt_mode from DB before cast
Reject unknown prompt_mode values from OtherConfig JSONB
(e.g. typos like "taks") by checking against validPromptModes set.
Invalid values default to "" (full mode). Prevents broken prompts
where no mode flags match.
* fix(prompt): wire SectionIDToolCallStyle for provider override
Tool Call Style section now uses sectionContent() like Execution Bias,
allowing providers to override it via PromptContribution.SectionOverrides.
* feat(store): implement 9 SQLite store backends for v3 parity
Close feature gap between PostgreSQL and SQLite (desktop/lite) editions.
96 methods across 9 stores: AgentLinks, SubagentTasks, SecureCLI,
SecureCLIGrants, EvolutionMetrics, EvolutionSuggestions, Episodic,
KnowledgeGraph, Vault. Schema v8→v9 adds 4 tables.
Key design decisions:
- LIKE-based search replaces tsvector/FTS5 (unavailable in modernc.org/sqlite)
- Go-side StringSimilarity for KG dedup (replaces pgvector cosine)
- Recursive CTE traversal with comma-delimited path cycle detection
- AES-256-GCM encryption on all SecureCLI read/write paths
- F15: SecureCLI disabled when EncryptionKey empty
- ON CONFLICT DO UPDATE (never INSERT OR REPLACE) to preserve FK cascades
* docs(store): add SQLite parity section to store data model docs
Document 9 new SQLite store implementations, schema v9, and
feature parity gaps (LIKE vs FTS, Jaro-Winkler vs vector dedup).
* fix(docker): skip web-builder stage when ENABLE_EMBEDUI=false
Use BuildKit conditional stage pattern so web-builder is not executed
when embedding is disabled. Also update pnpm-lock.yaml for 6 new deps
that were missing from the lockfile (markdown/math/mermaid packages).
* feat(vault): embed metadata.summary for richer vector search
Include summary from metadata JSONB in embedding text (title + path +
summary) for better semantic search. Update tsvector generated column
to include summary in FTS index. Zero new columns — summary stored in
existing metadata JSONB field. Backward compat: docs without summary
still embed title+path only.
* fix(vault-ui): wider detail dialog, markdown rendering, create tooltip
- Detail dialog: sm:max-w-lg → sm:max-w-2xl, content area 200→300px
- Content preview: render markdown via MarkdownRenderer instead of <pre>
- Create button: add tooltip explaining "select agent first" when disabled
* feat(ui): prompt_mode dropdown + pinned_skills multi-select
Add PromptSettingsSection to agent overview tab:
- prompt_mode: Select dropdown (full/task/minimal/none)
- pinned_skills: Tag input with max 10, click to remove
Both save to agent other_config JSONB via existing onUpdate flow.
Self-contained save button appears only when values change.
* fix(vault-ui): enlarge dialog to max-w-4xl, constrain markdown heading sizes
Dialog sm:max-w-2xl → sm:max-w-4xl. Markdown headings capped at
text-base/text-sm via Tailwind child selectors to prevent oversized
h1/h2 in content preview. Content area raised to 400px.
* refactor(ui): move pinned skills to dedicated section with skill select
Split pinned_skills out of PromptSettingsSection into PinnedSkillsSection.
Uses useAgentSkills hook for proper dropdown with granted skills list.
Placed right below SkillsSection in agent overview tab. Badge chips
with X to remove, Select dropdown to add. Max 10 enforced.
* fix(ui): add border wrap to capabilities section for visual consistency
* feat(ui): redesign prompt mode as compact cards, replace v3 badge
- Rewrite prompt settings from select dropdown to 2×2 compact cards
with lucide icons (Zap/Wrench/Package/CircleOff) and ring selection
- Move prompt settings to top of agent overview tab
- Replace v3 badge with prompt mode badge in header, card, and list row
- Add i18n keys for prompt mode labels/descriptions (en/vi/zh)
- Extract readPromptMode() to shared agent-display-utils
- Remove dead useAgentVersion hook and v3Tooltip/v2Tooltip i18n keys
- Remove v3 badge from EngineVersionSection (keep feature toggles)
- Use cn() for badge class composition with twMerge safety
* feat(ui): add section tags and token estimates to prompt mode cards
Each card now shows which system prompt sections are included
(Persona, Tools, Safety, Skills, MCP, Memory, etc.) as compact
tags, plus estimated base token range (~2-4K for full, ~10 for none).
Helps users understand the impact of each mode at a glance.
* fix(ui): correct token estimates with actual tiktoken measurements
Measured via tiktoken (cl100k) on realistic config:
full=~1.7K+, task=~1.1K+, minimal=~660, none=~6 base tokens.
The "+" suffix indicates context files/skills/MCP add more.
* fix(ui): use tiktoken-measured token ranges for prompt mode cards
Measured across 3 scenarios (bare/typical/heavy) via tiktoken cl100k:
full=~500-2.9K, task=~350-1.2K, minimal=~350-820, none=~6 tokens.
Ranges reflect real configs from minimal (3 tools) to heavy
(10 tools, long persona, MCP, sandbox, pinned skills).
* fix(ui): use production-measured token counts for prompt mode cards
Measured via tiktoken on real production agent (tieu-ho) with
4 context files (AGENTS.md, SOUL.md, IDENTITY.md, USER_PREDEFINED.md),
14 tools, memory, KG, skills, Telegram channel:
full=~3.1K, task=~2.2K, minimal=~1.9K, none=~6 tokens.
* refactor(ui): accurate section tags per prompt mode from systemprompt.go
Section tags now match exact gating logic in BuildSystemPrompt():
- full: persona, tools, exec bias, call style, safety, skills,
MCP, memory, sandbox, evolution, channel hints (11 sections)
- task: style echo, tools, exec bias, safety (slim), skills (search),
MCP (search), memory (slim) (7 sections)
- minimal: tools, safety (2 sections + shared context files)
- none: no tags shown, no token count (trivially ~6 tokens)
Removed workspace/identityOnly tags (shared across all modes).
* feat(ui): replace engine version toggles with V3 capabilities modal
- Remove 3 toggle switches (Pipeline, Memory, Retrieval) from agent
detail — all features are now always-on for v3 agents
- Replace with compact static badges layout
- New V3 Capabilities modal with 4 tabs:
Pipeline (8-stage flow), Memory (L0/L1/L2 tiers),
Knowledge (KG, Vault, Dreaming), Orchestration (Delegate, Evolution)
- Each tab has Lucide icons and technical descriptions of how
features actually work
- Separate i18n namespace v3-capabilities in en/vi/zh locales
* refactor(ui): move V3 badge to agent header, remove engine version section
- Add clickable V3 badge next to agent name in header
- Clicking opens V3 Capabilities modal
- Remove Engine Version section from overview tab entirely
- Remove unused EngineVersionSection import
* feat: v3 prompt engine overhaul — 7-phase restructuring
Phase 1: Fix mode resolution — subagent/cron cap at task (not minimal),
heartbeat stays minimal, pinned skills injected in all modes, USER_PREDEFINED
added to minimal allowlist.
Phase 2: Context file restructuring — new CAPABILITIES.md (domain expertise,
separated from SOUL.md), AGENTS_MINIMAL.md for heartbeat sessions, both added
to stable context files and minimal allowlist.
Phase 3: Summoner update — all 4 prompt builders generate CAPABILITIES.md,
fallback 2-call stores capabilities alongside SOUL.md.
Phase 4: Open agent deprecation — creation silently upgrades open→predefined
in both HTTP and WS endpoints.
Phase 5: Bootstrap auto-contact — sender name from channel metadata injected
into bootstrap context for 1-turn onboarding (DM only).
Phase 6: System prompt preview — GET /v1/agents/{id}/system-prompt-preview
endpoint with mode param, token counting, section parsing.
Phase 7: Agent creation UX — removed open agent type toggle, schema simplified,
description always required, prompt mode cards updated with v3 token estimates.
* fix(vault-ui): redesign detail dialog and link dialog UX
- Vault detail: show content preview by default (not collapsed),
move type/scope to header badges, hash to subtle footer
- Link dialog: replace native <select> with searchable Combobox
for target document and link type (5 presets + custom)
- Fix Vietnamese i18n: add proper diacritics to v3-capabilities
* fix(i18n): keep prompt section badges in English across all locales
Technical terms like Persona, Tools, Safety, Memory, Skills, MCP,
Sandbox, Evolution should not be translated — they are UI labels
matching system prompt section names.
* feat(ui): system prompt preview in Files tab + README restructure
Files tab: add "System Prompt" item in sidebar below context files.
When selected, shows readonly preview with mode selector (full/task/
minimal/none), token count badge, and cache boundary highlighting.
Fetches from GET /v1/agents/{id}/system-prompt-preview.
README: remove Claw Ecosystem comparison tables, remove OpenClaw port
reference, rename "What Makes It Different" to "Core Features" with v3
additions (8-stage pipeline, 4-mode prompt, 3-tier memory, knowledge
vault, self-evolution), slim built-in tools to category summary table.
* feat: replace architecture images with v3 sketchnotes
Add 9 new architecture sketchnote images generated for v3:
- 8-Stage Agent Pipeline
- 4-Mode Prompt System
- 3-Tier Memory Architecture
- Multi-Tenant Architecture
- Agent Orchestration
- Knowledge Vault
- Provider Adapter System
- Self-Evolution System
- DomainEventBus
Remove old architecture images (architecture.jpg, goclaw_multi_tenant.png,
agent-delegation.jpg, agent-teams.jpg).
Update README to reference new images in Architecture and Orchestration
sections. Consolidate orchestration section (remove separate delegation
and teams subsections).
* feat(readme): add remaining 4 architecture sketchnotes with sections
Add Knowledge Vault, Self-Evolution, Provider Adapters, and Event-Driven
Architecture sections with corresponding sketchnote images and concise
descriptions. All 9 v3 architecture diagrams now in README.
* feat: reorder README architecture images, evolution guardrails fix, memory tools enhancement
README: reorder architecture sketchnotes (Multi-Tenant first), remove
pinnedSkills from task mode sections badge.
Backend: evolution guardrails fix, memory auto-injector and tools
enhancement, gateway HTTP wiring update.
* feat(desktop): add 12 v3 feature sections to agent detail panel
Add evolution expansion (skill learning, v3 flags), prompt mode selector,
evolution dashboard tab (CSS bar charts, suggestions, guardrails),
thinking/reasoning config, orchestration display, context pruning,
compaction, subagents (lite-limited), tool policy, sandbox config,
and pinned skills management.
Extract agent detail state into use-agent-detail-state hook.
Add getWithParams to ApiClient. Add i18n keys to en/vi/zh locales.
* fix(vault): team-scope security — prevent cross-team data corruption and leaks
- Add team_id UUID + custom_scope to vault_documents (PG migration 043, SQLite migration 10)
- COALESCE-based UNIQUE prevents silent cross-team data overwrite on ON CONFLICT
- PG trigger auto-corrects scope…
* ci(release): switch from auto-deploy to tag-based release
Replace go-semantic-release (auto on push main) with tag-based trigger:
- Push tag v1.2.3 → auto release (stable only, not beta/rc)
- Manual dispatch with optional tag input as fallback
- gh release create with --generate-notes replaces semantic-release changelog
- Beta/RC tags still handled by release-beta.yaml (unchanged)
* fix(ci): resolve 4 workflow failures — tests, race, Windows build
- slack: update splitAtLimit tests to match ChunkMarkdown behavior
- providerresolve: use ChatGPTOAuthRouting field instead of OtherConfig
- scheduler: buffer started channel to prevent race in DropOldPolicy
- backup: split checkDiskSpace into platform files (Unix/Windows)
* feat(vault): modal document view, compact sidebar, non-owner team access
UI:
- Replace bottom detail panel with modal dialog for all doc views
- Redesign sidebar with type icons, compact layout, inline metadata
- Always show team select, fix missing load() call for teams hook
- Align header heights between sidebar and main panel
Backend:
- Add TeamIDs filter to VaultListOptions and VaultSearchOptions
- Non-owner users now see personal + their teams' docs (was personal-only)
- Refactor PG search to use searchTeamFilter for FTS and vector queries
- Update both PG and SQLite stores with shared team filter helpers
* fix(backup): guard nil context in checkPgDump preflight
* feat(channels): add Facebook Messenger and Pancake channel integrations (#731)
Add two new channel implementations for Facebook Fanpage (comment + Messenger
auto-reply, first inbox DM) and Pancake/pages.fm (multi-platform inbox via
Facebook, Zalo, Instagram, TikTok, WhatsApp, LINE).
Key features:
- Facebook: comment auto-reply, Messenger auto-reply, first inbox DM,
HMAC-SHA256 webhook verification, multi-page webhook routing
- Pancake: multi-platform inbox, outbound echo dedup with HTML normalization,
race-condition-safe echo fingerprinting, platform-aware formatting
- Bootstrap skip: pre-fill USER.md from channel metadata (Pancake)
- SanitizeDisplayName across all channels (defense in depth)
Code audit fixes:
- Fix truncateForTikTok byte→rune slicing (UTF-8 corruption)
- Fix empty message.ID shared dedup slot (silent message loss)
- Fix DisplayName markdown injection in buildPrefilledUser
- Consolidate duplicate ChannelMeta type (agent→bootstrap)
- Compile-time interface assertions, alphabetical type constants
- Per-message logs demoted to slog.Debug, errors.As for wrapped errors
- UI: alphabetical channel ordering, complete config schemas
- Remove deprecated WhatsApp bridge_url, fix nested error parsing
* Delete plans/260406-1304-goclaw-v3-core-design directory
* fix(vault): adaptive label truncation and zoom-aware link labels in graph view
Node labels now truncate based on zoom level (10→18→30→full chars),
showing full text only when zoomed in close enough to read.
Link labels hidden below 1.5x zoom to reduce visual clutter.
Selected nodes always show full label regardless of zoom.
* refactor(vault): remove vault_link/vault_backlinks tools, auto-sync wikilinks in enrichment pipeline
- Remove vault_link and vault_backlinks builtin tools (replaced by auto-linking)
- Add DeleteDocLinksByType to VaultStore interface (PG + SQLite) to selectively
delete links by type without destroying semantic links
- Integrate SyncDocLinks into enrichment worker: [[wikilinks]] are now
auto-extracted from document content and synced as vault links on every write
- Fix SyncDocLinks to use DeleteDocLinksByType("wikilink") instead of
DeleteDocLinks which was deleting all link types including semantic
- Add missing coreToolSummaries for delegate, memory_expand, vault_search
(previously showed as "(custom tool)" in system prompt)
- Increase enrichSimilarityLimit from 5 to 10 for richer auto-linking
* fix(ui): persist chat messages across sessions and fix first-message drop
Migrate chat message state from component-local useState to a Zustand
store so messages survive session switches and back-navigation.
- Infinite re-render loop (React #185): inline `?? []` fallback in
Zustand selector created a new array reference every render; fixed
with a stable EMPTY_MESSAGES module constant.
- First message disappearing in new chat: addLocalMessage used the
component-level sessionKey which is still "" when send fires
(navigate is async); fixed by threading explicit sessionKey through
onMessageAdded in useChatSend.
* fix(ui): add LRU eviction to chat messages store
Keep at most 25 cached sessions in memory. When the limit is exceeded,
evict the oldest idle sessions (never evict sessions with an active run).
* feat(vault): replace auto-linking with LLM-classified relationship types
Replace vector-similarity-only auto-linking (creating generic "semantic"
links) with an LLM classification step that determines actual relationship
types (reference, depends_on, extends, related, supersedes, contradicts)
and generates meaningful context descriptions.
- Add enrich_classify.go: orchestration, retry with escalating timeouts,
candidate gathering with bidirectional dedup, 20-doc cap per batch
- Add enrich_classify_prompt.go: system prompt, JSON parsing with partial
success model, UTF-8 safe truncation
- Restructure processBatch: summarize → embed → classify → dedup+wikilinks
- Move dedup recording after classify (failed classify allows re-enrichment)
- Remove autoLink method (fully replaced by classifyLinks)
- Add DeleteDocLinksByTypes to VaultStore interface + PG/SQLite (IN clause)
- Guard old link deletion behind len(newLinks) > 0 (no data loss on all-SKIP)
* fix(evolution): fix 7 bugs in evolution flow — guardrails, cron, dedup, dead types
- Remove broken delta comparison in CheckGuardrails (compared usage rate vs max delta)
- Replace hardcoded +0.05 threshold bump with configurable MaxDeltaPerCycle, cap at 0.95
- Remove dead MetricFeedback and SuggestMemoryPrune constants + UI references
- Make SuggestToolOrder approval actionable — disables tool via BuiltinToolTenantConfigStore
- Replace time.Ticker(24h) with wall-clock scheduling (3 AM daily, Sundays weekly eval)
- Add PG advisory lock on pinned connection for multi-instance safety
- Add EvolutionSuggest flag check in cron (metrics-only agents skip suggestion analysis)
- Change suggestion dedup from type-only to (type, metric_key) composite key
* fix(tracing): include thinking tokens in cost calculation
CalculateCost previously ignored usage.ThinkingTokens entirely, causing
cost underestimation for every reasoning-capable model (OpenAI o3/o4-mini,
Codex/GPT-5, Anthropic extended thinking).
Thinking tokens are reported by providers as a sub-count of completion
tokens — OpenAI completion_tokens already includes reasoning_tokens,
Codex output_tokens likewise, and Anthropic output_tokens counts thinking
at the output rate. Default behaviour (ReasoningPerMillion=0) now matches
provider billing semantics: full CompletionTokens priced at OutputPerMillion,
no double-counting.
When ReasoningPerMillion is explicitly configured, the completion bucket is
split into visible output + thinking and each priced independently. Defensive
clamp to zero when Anthropic's thinkingChars/4 estimate exceeds reported
OutputTokens.
Adds ReasoningPerMillion field to ModelPricing and 7 regression tests
covering the double-count gate, distinct-rate split, Anthropic estimate
overrun, and backward compatibility for non-reasoning models.
* feat(pipeline): per-provider context window, cache/tenant/pipeline hardening
Ship Group A (bug fixes) + Group B (pipeline enhancements) from
plans/260410-1009-openclaw-ts-feature-port — five changes that share
cmd/ wiring and pipeline plumbing so they commit as a unit.
cache: InMemoryCache now supports periodic sweep + max-size cap via
variadic options. PermissionCache wires 60s sweep + 10k entry cap +
Close() hook in gateway shutdown so long-running gateways don't leak
per-user permission entries. Backward compatible for zero-arg callers.
store: ContactCollector.seen cache key now includes tenant_id + channel_
instance so the same sender in different tenants (or different bot
instances in the same tenant) no longer silently skip upserts against
each other. Zero-tenant (Desktop) behaviour preserved.
pipeline: EffectiveContextWindow is resolved once per run in ContextStage
via a ResolveContextWindow callback (backed by providers.ModelRegistry)
so PruneStage bills history against the actual model window instead of
a stale static config. Nil resolver / unknown model fall back to
Config.ContextWindow for backward compatibility. Locked to the model
observed at context build time to prevent mid-run budget drift.
pipeline: PipelineConfig.ReserveTokens carves out an optional safety
buffer subtracted from the history budget so compaction fires slightly
before the hard limit — protects against provider over-delivery and
token counter drift on streaming responses. Zero (default) preserves
legacy budget math.
agent: ModelRegistry flows gateway → ResolverDeps → LoopConfig → Loop →
pipeline adapter so resolver can look up per-model capabilities at run
time without re-touching gateway internals.
20 regression tests across cache, contact collector, and pipeline cover
the critical paths: cross-tenant isolation, cache sweep + eviction +
Close idempotency, per-model window override + fallback, and reserve
token buffer behaviour. All passing with -race on both go build ./...
and go build -tags sqliteonly ./....
* feat(memory): context-aware recall query for auto-inject
Auto-inject previously searched episodic memory using only the latest
user message. Follow-up questions like "what's my favorite?" returned
poor matches because the embedding lost the conversational frame.
InjectParams now carries an optional RecentContext field that pgAuto
Injector prepends to the search query as "Context: ... \nQuery: ..."
before running the FTS+vector hybrid search. The "Context:"/"Query:"
framing works with both instruction-tuned embedding models (which
respect the labels) and plain models (neutral separators).
ContextStage walks the message history backward, collects up to 2
trailing user turns capped at 300 runes total, and threads the snippet
through the AutoInject callback to the injector. Empty RecentContext
preserves legacy single-message search semantics — zero-risk fallback
for callers that haven't adopted the new field.
Rune-based truncation (not byte) keeps vi/zh locales safe: a byte-wise
tail-clip would slice multi-byte runes and emit invalid UTF-8 to the
embedding model, degrading exactly the cases Phase 9 is meant to fix.
tailClipRunes helper covers Vietnamese, Chinese, Japanese, emoji.
13 regression tests: recall query builder (unicode-safe clip,
whitespace handling, position ordering), buildRecentContext (order
preservation, turn cap, truncation, non-user skip), and
tailClipRunes (CJK, short input, zero cap). All passing with -race.
Refs plans/260410-1009-openclaw-ts-feature-port/phase-09-active-
memory-recall.md — minimal-viable delivery; Tier 2 LLM re-ranking
and per-session recall cache deferred until operational data shows
context-aware search alone is insufficient.
* feat(ui/graph): migrate vault + KG graph to Sigma.js + Graphology
Replace react-force-graph-2d (Canvas 2D, 500-node ceiling) with Sigma.js
(WebGL, 5k+ nodes at 60fps) for vault and knowledge graph visualization.
Layout pipeline uses Louvain community detection + grid seeding + FA2
refinement + noverlap post-processing. Communities are placed in a grid
(sorted by size) so related nodes cluster visually instead of collapsing
into a ring shape.
Shared graph infrastructure in ui/web/src/components/graph/:
- sigma-graph-container: lifecycle, synchronous pre-computed layout,
2-hop BFS highlight on hover, opacity edge pulse, prefers-reduced-motion
- sigma-graph-controls: reusable zoom/stats bar with camera tracking
- sigma-graph-search: combobox with ARIA wiring and keyboard navigation
- sigma-graph-filters: type filter dropdown (outside-click + Escape)
- sigma-graph-minimap: canvas overlay with viewport rect + click-to-jump
- sigma-graph-keyboard-help: help popover triggered by ? key
- use-sigma-keyboard: +/- R F Esc / shortcuts
- graph-utils: sizing, middle-truncation helpers
Adapters rewritten to output Graphology Graph instances. Both views use
curved arrow edges, focus ring for keyboard users, role="application"
ARIA, responsive top bar, and node limits up to 5k.
Removes react-force-graph-2d, d3-force, @types/d3-force.
Adds sigma, @sigma/edge-curve, graphology, graphology-layout,
graphology-layout-forceatlas2, graphology-layout-noverlap,
graphology-communities-louvain, graphology-types.
* feat(vault): workspace rescan endpoint with symlink-safe walker
Add POST /v1/agents/{agentID}/vault/rescan to backfill vault_documents
from filesystem. Walks agent workspace, registers missing/changed files,
publishes enrichment events for async summarize + embed + link classify.
Backend:
- SafeWalkWorkspace: symlink-safe walker with exclusion patterns,
resource limits (5K files, 500MB, 50MB/file), context deadline
- RescanWorkspace: hash-based dedup, path-based scope detection
(teams/{id}/ → team scope), idempotent via UpsertDocument ON CONFLICT
- Per-agent mutex (409 Conflict for concurrent rescans)
- Tenant-scoped workspace resolution (config.TenantWorkspace)
- Enrichment worker: semaphore-based parallel summarize (max 3 concurrent)
- Media files without summary skip LLM summarize, embed title+path only
- DRY: export InferTitle/InferDocType from vault pkg, remove from interceptor
- Auto-register text uploads in vault via onTextUploaded callback
Frontend:
- Rescan button in vault page header (FolderSync icon)
- Toast with result counts, 409 warning, error handling
- i18n strings for en/vi/zh
Security: skip all symlinks, boundary checks, per-file size limit,
tenant isolation via server-side workspace resolution.
* feat(ts-port): reasoning strip, dreaming config + weighted scoring
Phase 6 — Reasoning token stripping:
- ReasoningDecision.StripThinking auto-flags Kimi + DeepSeek-Reasoner
- Guard clauses in Anthropic/OpenAI/Codex stream handlers
- Usage.ThinkingTokens + RawAssistantContent preserved (billing + tool passback safe)
Phase 8 — Per-agent dreaming config:
- MemoryConfig.Dreaming JSONB (no migration), resolver callback pattern
- Enabled/DebounceMs/Threshold/VerboseLog fields with partial-override merge
- ConsolidationDeps gains optional AgentStore
Phase 10 — Dreaming weighted scoring:
- Migration 000045 adds recall_count/recall_score/last_recalled_at on episodic_summaries
- ComputeRecallScore 4-component formula (freq/rel/recency/freshness, 14d half-life)
- memory_search fire-and-forget RecordRecall; ListUnpromotedScored in DreamingWorker
- Bootstrap-friendly filter: unrecalled entries bypass thresholds
- Debounce stamped on filter-empty skip to prevent starvation loop
Phase 5 follow-up — last_compaction_at in sessions.metadata JSONB:
- v3 PruneStage.CompactMessages and v2 maybeSummarize both stamp timestamp
- Zero migration; exported const SessionMetaKeyLastCompactionAt
RequiredSchemaVersion: 44 → 45 (PG), SchemaVersion: 12 → 13 (SQLite).
27 new tests; builds pass under PG and sqliteonly tags.
* feat(ui): dreaming config section and context usage badge
Phase 11 — web UI components:
- DreamingConfig TS interface + nested block in MemorySection (4 controls)
- Context usage badge in ChatTopBar: used/max/percent with color ramp
- Badge tooltip surfaces last_compaction_at from session.metadata
- i18n keys in en/vi/zh: configSections.dreaming.*, chat.contextUsage.*
- Hidden on mobile via hidden sm:flex (follow-up: compact variant)
* feat(vault): tenant-wide rescan with nullable agent_id + media preview
Vault rescan redesigned from per-agent to tenant-wide:
- POST /v1/vault/rescan replaces POST /v1/agents/{id}/vault/rescan
- agent_id nullable in vault_documents (PG migration 046 + SQLite v14)
- Path-based scope inference: agents/{key}/ → personal, teams/{uuid}/ → team, root → shared
- Interceptor sets agent_id=NULL for team-scoped file writes
- Enrichment worker batch key handles empty agent_id
- web-fetch/ directory excluded from vault scan at any depth
- Media preview: images render via authenticated blob URL, binary files show metadata
- Scan button no longer requires agent selection
* fix(vault): rescan path resolution, batch enrichment pipeline, cross-agent create
- Fix rescan storing stripped paths (teams/{uuid}/... → file.md) causing
enrichment worker to fail locating files on disk. Now stores full
workspace-relative path with backward-compat fallback for old records.
- Refactor enrichment into chunked pipeline: batch 5 files per LLM call
(was 1:1), 3 parallel goroutines via semaphore — reduces LLM calls ~80%.
- Extract shared chatWithRetry for summarize + classify (3 retries,
escalating timeouts 5m→7m→10m).
- Add POST /v1/vault/documents route for cross-agent doc creation
(agentID optional, defaults scope to shared when no agent).
- Add path traversal validation (reject .. and leading /) on create.
- Skip media files in syncWikilinks to prevent binary garbage parsing.
- Bound syncWikilinks file read to 4MB.
* feat(vault): cross-agent API, WS enrichment progress, nullable agent_id
Backend:
- Add cross-agent routes for all vault CRUD (/v1/vault/documents/*, /v1/vault/links/*)
alongside existing per-agent routes for backward compat.
- All handlers skip agent ownership check when agentID is empty.
- Path traversal validation (reject .. and leading /) on create.
- Scope defaults to "shared" when creating without agent.
- Enrichment progress broadcast via WS event (vault.enrich.progress)
using existing MessageBus infrastructure.
- GET /v1/vault/enrichment/status for polling fallback.
Frontend:
- Fix VaultDocument.agent_id type to optional/nullable — prevents
runtime crash on shared/team docs without agent.
- All vault hooks use cross-agent URLs (/v1/vault/...) — no more
skipping docs with null agent_id in graph/links/mutations.
- Remove dead agentId params from useUpdateDocument, useDeleteDocument,
useCreateLink, useDeleteLink, useVaultLinks, LinkBadge.
- Pass team_id from selected team filter to create dialog.
- WS-based enrichment progress bar on vault page (useWsEvent hook).
- Disable rescan button during enrichment, auto-clear after 3s.
* fix(ui/graph): use random+strongGravity FA2 for sparse vault graphs
Sparse graphs (edges/node < 0.8) previously used Louvain community
detection which created hundreds of singleton communities → grid layout.
Now uses random disc init + FA2 with strongGravityMode + higher gravity
to pull orphan nodes inward. Dense graphs keep Louvain-seeded layout.
* perf: eliminate N+1 query patterns across import, vault, and usage pipelines
- Convert importCron/importUserProfiles/importUserOverrides to multi-row
UPSERT with chunking (200→2 queries per import)
- Add GetTenantsByIDs batch method to TenantStore (PG+SQLite), refactor
handleMine() to single-query tenant fetch (N→1)
- Refactor buildAgentKeyMap() to use existing GetByKeys with tenant-scoped
context instead of per-key SELECT (N→1)
- Add GetDocumentsByIDs, GetDocumentByBasename, CreateLinks batch methods
to VaultStore (PG+SQLite)
- Batch-fetch docs in enrichment Phase 0, carry title through pipeline to
avoid per-doc refetch in classify phase
- Replace ListDocuments(limit=500) fallback in wikilink resolution with
targeted GetDocumentByBasename DB query
- Batch CreateLink calls in SyncDocLinks and classifyLinks
- Refactor usage handleGet/handleSummary to use ListPagedRich instead of
List+GetOrCreate N+1 loop (100→1 queries)
- Batch INSERT for team import members/comments/events/links (250→5)
- Add migration 47: UNIQUE constraint on cron_jobs(agent_id,tenant_id,name)
with dedup, SQLite schema v15
* fix(sqlite): fix pre-existing test failures in cron and schema migration tests
- Add missing reloadClaimed=false arg to executeOneJob call in cron toggle test
- Fix schema migration tests that applied full schema then rolled back version,
causing duplicate column errors on re-migration. Now recreates table without
post-targetVersion columns before re-running migrations.
* fix(ui/vault): adaptive graph layout, color palette, and link name resolution
- Replace density-branched FA2 layout with unified adaptive algorithm
based on orphan ratio instead of misleading edge density threshold
- Remove Louvain community grid seeding (caused ring/rectangle artifacts)
- Enable strongGravity + linLog + outboundAttractionDistribution for
organic cluster layout on both vault and KG graphs
- Update vault type colors: note→teal, media→pink (softer for dominant types)
- Increase light-mode edge visibility (slate-400 @ 80% vs gray-300 @ 60%)
- Enrich vault links API with doc_names map for outlink target titles
- Show document names instead of truncated UUIDs in link badges
- Use existing backlink title field from backend JOIN
- Horizontal scroll for link badges to preserve content viewport
- Move "Add Link" button to bottom bar in detail dialog
- Remove unused graphology-communities-louvain dependency
* fix(cron): prevent scheduler loop from blocking when a job hangs (#820)
* fix(cron): prevent scheduler loop from blocking when a job hangs
The cron scheduler's runLoop calls checkAndRunDueJobs() every second,
which previously used wg.Wait() to block until ALL claimed jobs complete.
If any single job hung (LLM provider timeout, agent loop stuck, network
issue), wg.Wait() would block indefinitely, preventing the scheduler
from ever checking for new due jobs — effectively killing all cron
scheduling until a container restart.
Changes:
- Remove wg.Wait() from both PG and SQLite cron schedulers — jobs now
run as independent goroutines that don't block the check loop
- Add panic recovery to PG runLoop (safeCheckAndRunDueJobs wrapper)
and per-job goroutines, matching the existing safego.Recover pattern
in the SQLite scheduler
- Add 10-minute context timeout to the cron job handler so a hung
agent run is cancelled instead of blocking forever
- Use select with context.Done() in the handler to respect the timeout
when waiting for the scheduler outcome
- Invalidate PG job cache per-job on completion instead of after the
(now-removed) batch wait
The SQLite scheduler already had safego.Recover on job goroutines but
still used wg.Wait() — this commit removes that blocking wait as well.
* fix(cron): make job timeout configurable + add SQLite runLoop panic recovery
- Add `cron.job_timeout` config field (Go duration string, default "10m")
so operators can tune the per-job timeout for complex agent workloads
without code changes
- Add `safeCheckJobs` panic recovery wrapper to SQLite cron runLoop,
matching the PG scheduler's `safeCheckAndRunDueJobs` for consistency
- Use dynamic timeout string in error message for better diagnostics
* fix: remove unused "time" import from gateway_cron.go
* fix(cron): apply same fixes to SQLite DB scheduler (sqlitestore)
The SQLite DB-backed scheduler (used by desktop edition with SQLite
backend) had the exact same wg.Wait() blocking issue and missing
panic recovery as the PG scheduler. Apply identical fixes:
- Remove wg.Wait() — jobs run as independent goroutines
- Add safeCheckAndRunDueJobs panic recovery wrapper for runLoop
- Add per-job panic recovery and cache invalidation
---------
Co-authored-by: Luvu182 <208665161+Luvu182@users.noreply.github.com>
* fix(cron): improve panic recovery and config validation
- Use safego.Recover for per-job panic recovery in PG and SQLite cron
schedulers (captures stack traces vs inline fmt.Sprint)
- Log warning when cron job_timeout config value is invalid instead of
silently falling back to default
* feat(vault): replace create dialog with multi-file upload modal
- Add POST /v1/vault/upload multipart endpoint with extension whitelist,
50MB/file + 50 file count limits, path traversal prevention
- Replace manual create dialog with upload dialog: radio destination
(Shared/Agent/Team), drag-drop zone, scrollable file list
- Files written to workspace folders, auto-enrich via EventVaultDocUpserted
- Split vault_handlers.go (991 lines) into 4 focused modules (~260 lines each)
- Add i18n keys for upload dialog (en/vi/zh)
* fix(mcp): full reconnect for SSE/HTTP after server-side restart (#812)
* fix(mcp): full reconnect for SSE/HTTP after server-side restart
When an MCP server using SSE or streamable-http transport restarts
(container redeploy, crash, OOM), the old client holds a stale session
ID. tryReconnect only called Ping() on the dead client, which keeps
POSTing to /message?sessionId=<dead> — the server returns 404, and
after maxReconnectAttempts the connection is permanently dead.
For pool connections, poolHealthLoop only set connected=false without
attempting any reconnect at all.
Fix:
- Store connection params (url, headers, command, args, env) in
serverState so tryReconnect can create fresh connections.
- tryReconnect now has two phases: fast path (ping existing client for
transient blips) and slow path (close old client, create fresh one
via createClient + Start + Initialize, swap ss.client).
- Add updateBridgeToolClients to propagate the new client pointer to
all registered BridgeTools after a full reconnect.
- Add poolTryReconnect with the same two-phase pattern for
pool-managed connections.
- Add BridgeTool.swapClient for safe client pointer replacement.
Closes #810
* fix(mcp): address review findings — atomic client pointer, close-after-swap, cooldown
Red-team review found critical issues in the initial reconnect fix:
1. Data race: BridgeTool.client written by healthLoop, read by Execute
concurrently without synchronization. Fix: BridgeTool now holds
*atomic.Pointer[mcpclient.Client] shared with serverState.clientPtr.
Execute uses atomic Load(), reconnect uses atomic Store().
2. Close-before-swap: old client was closed before new one was ready.
If createClient failed, ss.client pointed to closed client permanently.
Fix: create and validate new client first, swap atomically, then close old.
3. Pool BridgeTool orphaning: poolTryReconnect swapped ss.client but
existing BridgeTools held old pointer with no update mechanism.
Fix: atomic.Pointer propagates automatically — no explicit update needed.
4. Permanent death: after maxReconnectAttempts, server was permanently
dead with no recovery. Fix: 5-minute cooldown then reset attempts.
5. healthFailures not reset in fast path: minor inconsistency fixed.
Also fixes external caller in loop_mcp_user.go and adds ClientPtr()
accessor to poolEntry for atomic pointer sharing.
* refactor(mcp): extract fullReconnect helper, add nil guard, clarify dual-pointer design
- Extract shared fullReconnect() used by both tryReconnect and
poolTryReconnect — eliminates 30-line duplication (-17 lines net)
- Add nil guard on clientPtr.Load() in BridgeTool.Execute to prevent
nil deref in edge cases (test fixtures, initialization race)
- Add doc comment on serverState explaining dual-pointer design
(client for healthLoop, clientPtr for BridgeTools)
* refactor(mcp): DRY reconnect logic, fix client pointer thread safety
- Extract reconnectWithBackoff() shared by Manager and Pool reconnect paths
- Use clientPtr.Load() in Stop/Evict/evict paths to avoid race with fullReconnect
- Add tool staleness note to fullReconnect
- Remove dead poolEntry.Client()
---------
Co-authored-by: Duy /zuey/ <duy@wearetopgroup.com>
Co-authored-by: Duc Nguyen <me@vanducng.dev>
Co-authored-by: Kai (Tam Nhu) Tran <61256810+kaitranntt@users.noreply.github.com>
Co-authored-by: Plateau Nguyen <nguyennlt.ncc@gmail.com>
Co-authored-by: Reski Rukmantiyo <reski.rukmantio@lintasarta.co.id>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: QuXiangyu <771744189@qq.com>
Co-authored-by: quxy5 <quxy5@outlook.com>
Co-authored-by: Luan Vu <luanvuseo@gmail.com>
Co-authored-by: Luvu182 <208665161+Luvu182@users.noreply.github.com>
This commit is contained in:
229 files changed
+15809
-7080
No files matched your search
+73
-29
@@ -20,6 +20,8 @@ import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
kg "github.com/nextlevelbuilder/goclaw/internal/knowledgegraph"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels/discord"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels/facebook"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels/pancake"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels/feishu"
|
||||
slackchannel "github.com/nextlevelbuilder/goclaw/internal/channels/slack"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels/telegram"
|
||||
@@ -179,17 +181,16 @@ func runGateway() {
|
||||
}
|
||||
setupMemoryEmbeddings(pgStores, providerRegistry)
|
||||
|
||||
// Resolve background provider for consolidation + vault enrichment.
|
||||
// Fallback: background.provider → agent.default_provider → first registered provider.
|
||||
bgProvider, bgModel := resolveBackgroundProvider(cfg, providerRegistry)
|
||||
|
||||
// V3: Wire consolidation pipeline (episodic → semantic → KG → dreaming)
|
||||
if pgStores.Episodic != nil {
|
||||
var consolidationProvider providers.Provider
|
||||
if names := providerRegistry.ListForTenant(providers.MasterTenantID); len(names) > 0 {
|
||||
consolidationProvider, _ = providerRegistry.GetForTenant(providers.MasterTenantID, names[0])
|
||||
}
|
||||
if consolidationProvider != nil {
|
||||
// Create KG extractor for semantic worker (entity/relation extraction from episodic summaries)
|
||||
if bgProvider != nil {
|
||||
var kgExtractor *kg.Extractor
|
||||
if pgStores.KnowledgeGraph != nil {
|
||||
kgExtractor = kg.NewExtractor(consolidationProvider, consolidationProvider.DefaultModel(), 0)
|
||||
kgExtractor = kg.NewExtractor(bgProvider, bgModel, 0)
|
||||
}
|
||||
cleanupConsolidation := consolidation.Register(consolidation.ConsolidationDeps{
|
||||
EpisodicStore: pgStores.Episodic,
|
||||
@@ -197,34 +198,31 @@ func runGateway() {
|
||||
KGStore: pgStores.KnowledgeGraph,
|
||||
SessionStore: pgStores.Sessions,
|
||||
EventBus: domainBus,
|
||||
Provider: consolidationProvider,
|
||||
Model: consolidationProvider.DefaultModel(),
|
||||
Provider: bgProvider,
|
||||
Model: bgModel,
|
||||
Extractor: kgExtractor,
|
||||
AgentStore: pgStores.Agents,
|
||||
})
|
||||
defer cleanupConsolidation()
|
||||
slog.Info("consolidation pipeline registered")
|
||||
slog.Info("consolidation pipeline registered", "provider", bgProvider.Name(), "model", bgModel)
|
||||
} else {
|
||||
slog.Warn("consolidation pipeline skipped: no provider available")
|
||||
}
|
||||
}
|
||||
|
||||
// V3: Wire vault enrichment worker (async summary + embedding + auto-linking).
|
||||
// Resolves provider independently from consolidation pipeline.
|
||||
if pgStores.Vault != nil {
|
||||
var vaultProvider providers.Provider
|
||||
if names := providerRegistry.ListForTenant(providers.MasterTenantID); len(names) > 0 {
|
||||
vaultProvider, _ = providerRegistry.GetForTenant(providers.MasterTenantID, names[0])
|
||||
}
|
||||
if vaultProvider != nil {
|
||||
cleanupVaultEnrich := vault.RegisterEnrichWorker(vault.EnrichWorkerDeps{
|
||||
VaultStore: pgStores.Vault,
|
||||
Provider: vaultProvider,
|
||||
Model: vaultProvider.DefaultModel(),
|
||||
EventBus: domainBus,
|
||||
})
|
||||
defer cleanupVaultEnrich()
|
||||
slog.Info("vault enrichment worker registered")
|
||||
}
|
||||
var enrichProgress *vault.EnrichProgress
|
||||
if pgStores.Vault != nil && bgProvider != nil {
|
||||
cleanupVaultEnrich, ep := vault.RegisterEnrichWorker(vault.EnrichWorkerDeps{
|
||||
VaultStore: pgStores.Vault,
|
||||
Provider: bgProvider,
|
||||
Model: bgModel,
|
||||
EventBus: domainBus,
|
||||
MsgBus: msgBus,
|
||||
})
|
||||
enrichProgress = ep
|
||||
defer cleanupVaultEnrich()
|
||||
slog.Info("vault enrichment worker registered", "provider", bgProvider.Name(), "model", bgModel)
|
||||
}
|
||||
|
||||
loadBootstrapFiles(pgStores, workspace, agentCfg)
|
||||
@@ -282,7 +280,7 @@ func runGateway() {
|
||||
var mcpPool *mcpbridge.Pool
|
||||
var mediaStore *media.Store
|
||||
var postTurn tools.PostTurnProcessor
|
||||
contextFileInterceptor, mcpPool, mediaStore, postTurn = wireExtras(pgStores, agentRouter, providerRegistry, msgBus, pgStores.Sessions, toolsReg, toolPE, skillsLoader, hasMemory, traceCollector, workspace, cfg.Gateway.InjectionAction, cfg, sandboxMgr, redisClient, domainBus)
|
||||
contextFileInterceptor, mcpPool, mediaStore, postTurn = wireExtras(pgStores, agentRouter, providerRegistry, modelReg, msgBus, pgStores.Sessions, toolsReg, toolPE, skillsLoader, hasMemory, traceCollector, workspace, cfg.Gateway.InjectionAction, cfg, sandboxMgr, redisClient, domainBus)
|
||||
if mcpPool != nil {
|
||||
defer mcpPool.Stop()
|
||||
}
|
||||
@@ -297,8 +295,10 @@ func runGateway() {
|
||||
agentRouter: agentRouter,
|
||||
toolsReg: toolsReg,
|
||||
skillsLoader: skillsLoader,
|
||||
enrichProgress: enrichProgress,
|
||||
workspace: workspace,
|
||||
dataDir: dataDir,
|
||||
domainBus: domainBus,
|
||||
}
|
||||
|
||||
gatewayAddr := loopbackAddr(cfg.Gateway.Host, cfg.Gateway.Port)
|
||||
@@ -314,7 +314,11 @@ func runGateway() {
|
||||
// Wire dependencies for system prompt preview parity.
|
||||
if agentsH != nil {
|
||||
agentsH.SetPreviewDeps(toolsReg, skillsLoader)
|
||||
agentsH.SetPreviewStores(pgStores.Teams, pgStores.AgentLinks)
|
||||
var skillAccess store.SkillAccessStore
|
||||
if pgStores.Skills != nil {
|
||||
skillAccess, _ = pgStores.Skills.(store.SkillAccessStore)
|
||||
}
|
||||
agentsH.SetPreviewStores(pgStores.Teams, pgStores.AgentLinks, skillAccess)
|
||||
}
|
||||
|
||||
// External wake/trigger API
|
||||
@@ -417,6 +421,8 @@ func runGateway() {
|
||||
instanceLoader.RegisterFactory(channels.TypeZaloPersonal, zalopersonal.FactoryWithPendingStore(pgStores.PendingMessages))
|
||||
instanceLoader.RegisterFactory(channels.TypeWhatsApp, whatsapp.FactoryWithDB(pgStores.DB, pgStores.PendingMessages, "pgx"))
|
||||
instanceLoader.RegisterFactory(channels.TypeSlack, slackchannel.FactoryWithPendingStore(pgStores.PendingMessages))
|
||||
instanceLoader.RegisterFactory(channels.TypeFacebook, facebook.Factory)
|
||||
instanceLoader.RegisterFactory(channels.TypePancake, pancake.Factory)
|
||||
if err := instanceLoader.LoadAll(context.Background()); err != nil {
|
||||
slog.Error("failed to load channel instances from DB", "error", err)
|
||||
}
|
||||
@@ -507,8 +513,10 @@ func runGateway() {
|
||||
methods.NewTenantsMethods(pgStores.Tenants, msgBus, workspace).Register(server.Router())
|
||||
server.SetTenantsHandler(httpapi.NewTenantsHandler(pgStores.Tenants, msgBus, workspace))
|
||||
server.Router().SetTenantStore(pgStores.Tenants)
|
||||
// Permission cache for tenant membership checks
|
||||
// Permission cache for tenant membership checks. Store on deps so
|
||||
// lifecycle shutdown can call Close() to stop the sweep goroutines.
|
||||
permCache := cache.NewPermissionCache()
|
||||
deps.permCache = permCache
|
||||
msgBus.Subscribe("permission-cache", func(e bus.Event) {
|
||||
if p, ok := e.Payload.(bus.CacheInvalidatePayload); ok {
|
||||
permCache.HandleInvalidation(p)
|
||||
@@ -534,3 +542,39 @@ func runGateway() {
|
||||
sigCh: sigCh,
|
||||
})
|
||||
}
|
||||
|
||||
// resolveBackgroundProvider picks the LLM provider+model for background workers
|
||||
// (vault enrichment, consolidation). Fallback chain:
|
||||
//
|
||||
// background.provider/model → agent.default_provider/model → first registered provider.
|
||||
func resolveBackgroundProvider(cfg *config.Config, reg *providers.Registry) (providers.Provider, string) {
|
||||
try := func(name, model string) (providers.Provider, string, bool) {
|
||||
if name == "" {
|
||||
return nil, "", false
|
||||
}
|
||||
p, err := reg.GetForTenant(providers.MasterTenantID, name)
|
||||
if err != nil || p == nil {
|
||||
return nil, "", false
|
||||
}
|
||||
if model == "" {
|
||||
model = p.DefaultModel()
|
||||
}
|
||||
return p, model, true
|
||||
}
|
||||
|
||||
// 1. Explicit background config
|
||||
if p, m, ok := try(cfg.Gateway.BackgroundProvider, cfg.Gateway.BackgroundModel); ok {
|
||||
return p, m
|
||||
}
|
||||
// 2. Agent default provider
|
||||
if p, m, ok := try(cfg.Agents.Defaults.Provider, cfg.Agents.Defaults.Model); ok {
|
||||
return p, m
|
||||
}
|
||||
// 3. First registered provider (legacy fallback)
|
||||
if names := reg.ListForTenant(providers.MasterTenantID); len(names) > 0 {
|
||||
if p, m, ok := try(names[0], ""); ok {
|
||||
return p, m
|
||||
}
|
||||
}
|
||||
return nil, ""
|
||||
}
|
||||
@@ -30,8 +30,8 @@ func buildEnsureUserProfile(as store.AgentStore) agent.EnsureUserProfileFunc {
|
||||
// isNew=true seeds all files; isNew=false only seeds if user has zero files
|
||||
// (avoids re-seeding BOOTSTRAP.md after auto-cleanup on server restart).
|
||||
func buildSeedUserFiles(as store.AgentStore) agent.SeedUserFilesFunc {
|
||||
return func(ctx context.Context, agentID uuid.UUID, userID, agentType string, isNew bool) error {
|
||||
_, err := bootstrap.SeedUserFiles(ctx, as, agentID, userID, agentType, !isNew)
|
||||
return func(ctx context.Context, agentID uuid.UUID, userID, agentType string, isNew bool, channelMeta *bootstrap.ChannelMeta) error {
|
||||
_, err := bootstrap.SeedUserFiles(ctx, as, agentID, userID, agentType, !isNew, channelMeta)
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -91,6 +91,18 @@ func extractSessionMetadata(msg bus.InboundMessage, peerKind string) map[string]
|
||||
return meta
|
||||
}
|
||||
|
||||
// buildPancakeSessionLabel returns "Pancake:{senderName}:{pageName}" with non-empty parts only.
|
||||
func buildPancakeSessionLabel(senderName, pageName string) string {
|
||||
label := "Pancake"
|
||||
if senderName != "" {
|
||||
label += ":" + senderName
|
||||
}
|
||||
if pageName != "" {
|
||||
label += ":" + pageName
|
||||
}
|
||||
return label
|
||||
}
|
||||
|
||||
// buildAnnounceOutMeta builds outbound metadata for announce messages so that
|
||||
// Send() can route replies to the correct forum topic or DM thread.
|
||||
func buildAnnounceOutMeta(localKey string) map[string]string {
|
||||
|
||||
@@ -108,6 +108,12 @@ func processNormalMessage(
|
||||
}
|
||||
}
|
||||
|
||||
// Set session label for Pancake channels: "Pancake:{SenderName}:{PageName}"
|
||||
if msg.Metadata["pancake_mode"] != "" {
|
||||
label := buildPancakeSessionLabel(msg.Metadata["display_name"], msg.Metadata["page_name"])
|
||||
deps.SessStore.SetLabel(ctx, sessionKey, label)
|
||||
}
|
||||
|
||||
// Auto-collect channel contacts for the contact selector.
|
||||
// Skip internal senders (system:*, notification:*, teammate:*, ticker:*, session_send_tool).
|
||||
if deps.ContactCollector != nil && msg.SenderID != "" && !bus.IsInternalSender(msg.SenderID) {
|
||||
|
||||
+13
-4
@@ -78,8 +78,12 @@ func makeCronJobHandler(sched *scheduler.Scheduler, msgBus *bus.MessageBus, cfg
|
||||
)
|
||||
}
|
||||
|
||||
// Build context with tenant scope so agent loop events are scoped correctly.
|
||||
cronCtx := store.WithTenantID(context.Background(), job.TenantID)
|
||||
// Build context with tenant scope and timeout so agent loop events are
|
||||
// scoped correctly and a hung agent can't block the cron scheduler forever.
|
||||
jobTimeout := cfg.Cron.JobTimeoutDuration()
|
||||
cronCtx, cancelCron := context.WithTimeout(context.Background(), jobTimeout)
|
||||
defer cancelCron()
|
||||
cronCtx = store.WithTenantID(cronCtx, job.TenantID)
|
||||
|
||||
// Reset session before each cron run to prevent tool errors from previous
|
||||
// runs from polluting the context and blocking future executions (#294).
|
||||
@@ -106,8 +110,13 @@ func makeCronJobHandler(sched *scheduler.Scheduler, msgBus *bus.MessageBus, cfg
|
||||
TraceTags: []string{"cron"},
|
||||
})
|
||||
|
||||
// Block until the scheduled run completes
|
||||
outcome := <-outCh
|
||||
// Block until the scheduled run completes or the timeout fires.
|
||||
var outcome scheduler.RunOutcome
|
||||
select {
|
||||
case outcome = <-outCh:
|
||||
case <-cronCtx.Done():
|
||||
return nil, fmt.Errorf("cron job %s timed out after %s", job.Name, jobTimeout)
|
||||
}
|
||||
if outcome.Err != nil {
|
||||
return nil, outcome.Err
|
||||
}
|
||||
|
||||
@@ -3,13 +3,16 @@ package cmd
|
||||
import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/agent"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/cache"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/gateway"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/skills"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tools"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/vault"
|
||||
)
|
||||
|
||||
// gatewayDeps holds shared dependencies used across the extracted gateway setup functions.
|
||||
@@ -24,6 +27,9 @@ type gatewayDeps struct {
|
||||
agentRouter *agent.Router
|
||||
toolsReg *tools.Registry
|
||||
skillsLoader *skills.Loader // optional: enables skill creation in evolution approval
|
||||
permCache *cache.PermissionCache // nil if no tenant store; closed on shutdown to stop sweep goroutines
|
||||
enrichProgress *vault.EnrichProgress // nil if enrichment worker not registered
|
||||
workspace string
|
||||
dataDir string
|
||||
domainBus eventbus.DomainEventBus
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package cmd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
@@ -9,36 +10,94 @@ import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// runEvolutionCron runs the v3 evolution suggestion engine (daily) and
|
||||
// evaluation/rollback check (weekly) as background goroutines.
|
||||
// evolutionCronLockID is a PG advisory lock ID for preventing duplicate cron runs.
|
||||
// Only one gateway instance should run evolution analysis at a time.
|
||||
const evolutionCronLockID int64 = 0x65766F6C // "evol"
|
||||
|
||||
// runEvolutionCron runs the v3 evolution suggestion engine (daily at 3:00 AM)
|
||||
// and evaluation/rollback check (weekly on Sundays at 3:00 AM) as a background goroutine.
|
||||
// Designed to be called with `go runEvolutionCron(...)`.
|
||||
func runEvolutionCron(stores *store.Stores, engine *agent.SuggestionEngine) {
|
||||
dailyTicker := time.NewTicker(24 * time.Hour)
|
||||
defer dailyTicker.Stop()
|
||||
|
||||
weeklyTicker := time.NewTicker(7 * 24 * time.Hour)
|
||||
defer weeklyTicker.Stop()
|
||||
|
||||
// Run first analysis 1 minute after startup (warm-up).
|
||||
// Wait 1 minute after startup for warm-up, then run first analysis.
|
||||
time.Sleep(1 * time.Minute)
|
||||
runSuggestionAnalysis(stores, engine)
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-dailyTicker.C:
|
||||
runSuggestionAnalysis(stores, engine)
|
||||
case <-weeklyTicker.C:
|
||||
nextDaily := nextDailyRun(3) // 3:00 AM server time
|
||||
|
||||
timer := time.NewTimer(time.Until(nextDaily))
|
||||
<-timer.C
|
||||
timer.Stop()
|
||||
|
||||
runSuggestionAnalysis(stores, engine)
|
||||
|
||||
// Weekly evaluation on Sundays.
|
||||
if time.Now().Weekday() == time.Sunday {
|
||||
runEvolutionEvaluation(stores)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// runSuggestionAnalysis lists agents with evolution metrics enabled and runs analysis.
|
||||
// nextDailyRun returns the next occurrence of the given hour (0-23) today or tomorrow.
|
||||
// Uses AddDate to handle DST transitions correctly (calendar day, not 24h).
|
||||
func nextDailyRun(hour int) time.Time {
|
||||
now := time.Now()
|
||||
next := time.Date(now.Year(), now.Month(), now.Day(), hour, 0, 0, 0, now.Location())
|
||||
if !now.Before(next) {
|
||||
next = next.AddDate(0, 0, 1) // calendar day, not 24h — DST-safe
|
||||
}
|
||||
return next
|
||||
}
|
||||
|
||||
// tryAdvisoryLock acquires a PG session-level advisory lock on a pinned connection.
|
||||
// Returns the pinned connection (caller must close) and whether the lock was acquired.
|
||||
// No-op (returns nil, true) when db is nil (SQLite/desktop edition runs single-instance).
|
||||
func tryAdvisoryLock(ctx context.Context, db *sql.DB) (*sql.Conn, bool) {
|
||||
if db == nil {
|
||||
return nil, true
|
||||
}
|
||||
conn, err := db.Conn(ctx)
|
||||
if err != nil {
|
||||
slog.Debug("evolution.cron.conn_failed", "error", err)
|
||||
return nil, false
|
||||
}
|
||||
var acquired bool
|
||||
if err := conn.QueryRowContext(ctx, "SELECT pg_try_advisory_lock($1)", evolutionCronLockID).Scan(&acquired); err != nil {
|
||||
conn.Close()
|
||||
slog.Debug("evolution.cron.advisory_lock_failed", "error", err)
|
||||
return nil, false
|
||||
}
|
||||
if !acquired {
|
||||
conn.Close()
|
||||
return nil, false
|
||||
}
|
||||
return conn, true
|
||||
}
|
||||
|
||||
// releaseAdvisoryLock releases a PG advisory lock on the pinned connection and closes it.
|
||||
func releaseAdvisoryLock(ctx context.Context, conn *sql.Conn) {
|
||||
if conn == nil {
|
||||
return
|
||||
}
|
||||
_, _ = conn.ExecContext(ctx, "SELECT pg_advisory_unlock($1)", evolutionCronLockID)
|
||||
conn.Close()
|
||||
}
|
||||
|
||||
// runSuggestionAnalysis lists agents with evolution enabled and runs analysis.
|
||||
// Note: List(ctx, "") uses bare context (no tenant) to list ALL agents cross-tenant.
|
||||
// Per-agent calls are scoped via WithTenantID.
|
||||
func runSuggestionAnalysis(stores *store.Stores, engine *agent.SuggestionEngine) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
// List all agents (empty ownerID = all, tenant-scoped via context).
|
||||
// Acquire advisory lock on a pinned connection to prevent duplicate runs across instances.
|
||||
conn, acquired := tryAdvisoryLock(ctx, stores.DB)
|
||||
if !acquired {
|
||||
slog.Debug("evolution.cron.skipped_lock_held")
|
||||
return
|
||||
}
|
||||
defer releaseAdvisoryLock(ctx, conn)
|
||||
|
||||
agents, err := stores.Agents.List(ctx, "")
|
||||
if err != nil {
|
||||
slog.Warn("evolution.cron.list_agents_failed", "error", err)
|
||||
@@ -54,6 +113,9 @@ func runSuggestionAnalysis(stores *store.Stores, engine *agent.SuggestionEngine)
|
||||
if !flags.EvolutionMetrics {
|
||||
continue
|
||||
}
|
||||
if !flags.EvolutionSuggest {
|
||||
continue // metrics enabled but suggestions disabled — skip analysis
|
||||
}
|
||||
agentCtx := store.WithTenantID(ctx, ag.TenantID)
|
||||
if _, err := engine.Analyze(agentCtx, ag.ID); err != nil {
|
||||
slog.Debug("evolution.cron.analyze_failed", "agent", ag.ID, "error", err)
|
||||
@@ -86,6 +148,9 @@ func runEvolutionEvaluation(stores *store.Stores) {
|
||||
if !flags.EvolutionMetrics {
|
||||
continue
|
||||
}
|
||||
if !flags.EvolutionSuggest {
|
||||
continue // skip evaluation for agents with suggestions disabled
|
||||
}
|
||||
agentCtx := store.WithTenantID(ctx, ag.TenantID)
|
||||
if err := agent.EvaluateApplied(agentCtx, ag.ID, guardrails, stores.EvolutionMetrics, stores.EvolutionSuggestions, stores.Agents); err != nil {
|
||||
slog.Debug("evolution.cron.eval_failed", "agent", ag.ID, "error", err)
|
||||
|
||||
@@ -167,12 +167,17 @@ func (d *gatewayDeps) wireHTTPHandlersOnServer(
|
||||
if d.pgStores.Agents != nil {
|
||||
evoOpts = append(evoOpts, httpapi.WithAgentStore(d.pgStores.Agents))
|
||||
}
|
||||
if d.pgStores.BuiltinToolTenantCfgs != nil {
|
||||
evoOpts = append(evoOpts, httpapi.WithToolTenantCfgs(d.pgStores.BuiltinToolTenantCfgs))
|
||||
}
|
||||
d.server.SetEvolutionHandler(httpapi.NewEvolutionHandler(d.pgStores.EvolutionMetrics, d.pgStores.EvolutionSuggestions, evoOpts...))
|
||||
}
|
||||
|
||||
// V3: Knowledge Vault document API
|
||||
if d.pgStores != nil && d.pgStores.Vault != nil {
|
||||
d.server.SetVaultHandler(httpapi.NewVaultHandler(d.pgStores.Vault, d.pgStores.Teams))
|
||||
vh := httpapi.NewVaultHandler(d.pgStores.Vault, d.pgStores.Teams, d.workspace, d.domainBus, d.pgStores.Agents, d.pgStores.Teams)
|
||||
vh.SetEnrichProgress(d.enrichProgress)
|
||||
d.server.SetVaultHandler(vh)
|
||||
}
|
||||
|
||||
// V3: Episodic memory summaries API
|
||||
|
||||
@@ -159,6 +159,11 @@ func (d *gatewayDeps) runLifecycle(
|
||||
// Close provider resources (e.g. Claude CLI temp files)
|
||||
d.providerRegistry.Close()
|
||||
|
||||
// Stop permission cache sweep goroutines so they don't leak past shutdown.
|
||||
if d.permCache != nil {
|
||||
d.permCache.Close()
|
||||
}
|
||||
|
||||
// Stop sandbox pruning + release containers
|
||||
if deps.sandboxMgr != nil {
|
||||
deps.sandboxMgr.Stop()
|
||||
|
||||
+12
-1
@@ -42,6 +42,7 @@ func wireExtras(
|
||||
stores *store.Stores,
|
||||
agentRouter *agent.Router,
|
||||
providerReg *providers.Registry,
|
||||
modelReg providers.ModelRegistry,
|
||||
msgBus *bus.MessageBus,
|
||||
sessStore store.SessionStore,
|
||||
toolsReg *tools.Registry,
|
||||
@@ -140,10 +141,14 @@ func wireExtras(
|
||||
autoInjector = memorypkg.NewAutoInjector(stores.Episodic, stores.EvolutionMetrics)
|
||||
}
|
||||
|
||||
// vaultIntc is set later by wireVault but captured by closure in OnTextUploaded.
|
||||
var vaultIntc *tools.VaultInterceptor
|
||||
|
||||
resolver := agent.NewManagedResolver(agent.ResolverDeps{
|
||||
AgentStore: stores.Agents,
|
||||
ProviderStore: stores.Providers,
|
||||
ProviderReg: providerReg,
|
||||
ModelRegistry: modelReg,
|
||||
Bus: msgBus,
|
||||
Sessions: sessStore,
|
||||
Tools: toolsReg,
|
||||
@@ -157,6 +162,7 @@ func wireExtras(
|
||||
ContextFileLoader: contextFileLoader,
|
||||
BootstrapCleanup: buildBootstrapCleanup(stores.Agents),
|
||||
CacheInvalidate: buildCacheInvalidate(contextFileInterceptor),
|
||||
DefaultTimezone: appCfg.Cron.DefaultTimezone,
|
||||
InjectionAction: injectionAction,
|
||||
MaxMessageChars: appCfg.Gateway.MaxMessageChars,
|
||||
CompactionCfg: appCfg.Agents.Defaults.Compaction,
|
||||
@@ -184,6 +190,11 @@ func wireExtras(
|
||||
AutoInjector: autoInjector,
|
||||
EvolutionMetricsStore: stores.EvolutionMetrics,
|
||||
DomainBus: domainBus,
|
||||
OnTextUploaded: func(ctx context.Context, path, content string) {
|
||||
if vaultIntc != nil {
|
||||
vaultIntc.AfterWrite(ctx, path, content)
|
||||
}
|
||||
},
|
||||
OnEvent: func(event agent.AgentEvent) {
|
||||
// Sign /v1/files/ and /v1/media/ URLs in content before delivery.
|
||||
// Sessions store clean paths; signing happens only at delivery time.
|
||||
@@ -339,7 +350,7 @@ func wireExtras(
|
||||
}
|
||||
|
||||
// Wire vault tools and interceptors (conditional on vault store availability)
|
||||
wireVault(stores, toolsReg, workspace, domainBus)
|
||||
vaultIntc = wireVault(stores, toolsReg, workspace, domainBus)
|
||||
|
||||
// Wire delegate tool for inter-agent delegation via agent_links.
|
||||
if stores.AgentLinks != nil && stores.Agents != nil {
|
||||
|
||||
@@ -106,6 +106,10 @@ func seedConfigForContext(ctx context.Context, sc store.SystemConfigStore, cfg *
|
||||
setBool("gateway.tool_status", cfg.Gateway.ToolStatus)
|
||||
setInt("gateway.task_recovery_interval_sec", cfg.Gateway.TaskRecoveryIntervalSec)
|
||||
|
||||
// Background workers
|
||||
set("background.provider", cfg.Gateway.BackgroundProvider)
|
||||
set("background.model", cfg.Gateway.BackgroundModel)
|
||||
|
||||
// Tools
|
||||
set("tools.profile", cfg.Tools.Profile)
|
||||
setInt("tools.rate_limit_per_hour", cfg.Tools.RateLimitPerHour)
|
||||
|
||||
@@ -12,22 +12,17 @@ import (
|
||||
// wireVault wires Knowledge Vault tools and interceptors into the tool registry.
|
||||
// All wiring is skipped if stores.Vault is nil.
|
||||
// Pattern mirrors wireExtras KG wiring: register tools, set stores, set interceptors.
|
||||
func wireVault(stores *store.Stores, toolsReg *tools.Registry, workspace string, bus eventbus.DomainEventBus) {
|
||||
// wireVault wires Knowledge Vault tools and interceptors into the tool registry.
|
||||
// Returns the shared VaultInterceptor for use by other subsystems (e.g. agent upload hook).
|
||||
// Returns nil if stores.Vault is nil.
|
||||
func wireVault(stores *store.Stores, toolsReg *tools.Registry, workspace string, bus eventbus.DomainEventBus) *tools.VaultInterceptor {
|
||||
if stores.Vault == nil {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
|
||||
// Register vault tools — these are always available when vault store is present.
|
||||
vaultSearchTool := tools.NewVaultSearchTool()
|
||||
vaultLinkTool := tools.NewVaultLinkTool()
|
||||
vaultBacklinksTool := tools.NewVaultBacklinksTool()
|
||||
toolsReg.Register(vaultSearchTool)
|
||||
toolsReg.Register(vaultLinkTool)
|
||||
toolsReg.Register(vaultBacklinksTool)
|
||||
|
||||
// Wire vault store onto link/backlinks tools.
|
||||
vaultLinkTool.SetVaultStore(stores.Vault)
|
||||
vaultBacklinksTool.SetVaultStore(stores.Vault)
|
||||
|
||||
// Build VaultSearchService: fan-out across vault + KG (episodic store pending impl).
|
||||
// EpisodicStore is nil until a PG implementation exists.
|
||||
@@ -78,5 +73,6 @@ func wireVault(stores *store.Stores, toolsReg *tools.Registry, workspace string,
|
||||
}
|
||||
}
|
||||
|
||||
slog.Info("vault tools registered", "tools", "vault_search,vault_link,vault_backlinks,create_image,create_video,create_audio,tts,edit")
|
||||
slog.Info("vault tools registered", "tools", "vault_search,create_image,create_video,create_audio,tts,edit")
|
||||
return vaultIntc
|
||||
}
|
||||
@@ -88,9 +88,9 @@ Every channel must implement the base interface:
|
||||
| Interface | Purpose | Implemented By |
|
||||
|-----------|---------|----------------|
|
||||
| `StreamingChannel` | Real-time streaming updates | Telegram, Slack |
|
||||
| `WebhookChannel` | Webhook HTTP handler mounting | Feishu |
|
||||
| `WebhookChannel` | Webhook HTTP handler mounting | Facebook, Feishu/Lark, Pancake |
|
||||
| `ReactionChannel` | Status reactions on messages | Telegram, Slack, Feishu |
|
||||
| `BlockReplyChannel` | Override gateway block_reply setting | Slack |
|
||||
| `BlockReplyChannel` | Override gateway block_reply setting | Discord, Feishu/Lark, Pancake, Slack, Zalo OA, Zalo Personal |
|
||||
|
||||
`BaseChannel` provides a shared implementation that all channels embed: allowlist matching, `HandleMessage()`, `CheckPolicy()`, and user ID extraction.
|
||||
|
||||
|
||||
@@ -101,6 +101,18 @@ flowchart TD
|
||||
| Cron | `agent:{agentId}:cron:{jobId}:run:{runId}` | `agent:default:cron:reminder:run:abc123` |
|
||||
| Main | `agent:{agentId}:{mainKey}` | `agent:default:main` |
|
||||
|
||||
### Session Metadata - Compaction Tracking
|
||||
|
||||
**New well-known metadata key** (Phase 5 follow-up): `last_compaction_at` (RFC3339 string)
|
||||
|
||||
This timestamp is written to `sessions.metadata` JSONB after successful message compaction (context pruning). Both execution paths update it:
|
||||
- **V3 pipeline**: `PruneStage.CompactMessages()` after successful compaction
|
||||
- **V2 legacy**: `maybeSummarize()` goroutine after successful summarization
|
||||
|
||||
Operators can read this via `GetSessionMetadata()` to understand when a session was last compacted. The web UI optionally displays this timestamp in a context-usage tooltip.
|
||||
|
||||
Go constant export: `agent.SessionMetaKeyLastCompactionAt = "last_compaction_at"`
|
||||
|
||||
---
|
||||
|
||||
## 4. Agent Access Control
|
||||
@@ -661,7 +673,7 @@ L0 (Working Memory) L1 (Episodic Memory) L2 (Semantic Memory)
|
||||
|
||||
| Table | Purpose | Key Columns |
|
||||
|-------|---------|-------------|
|
||||
| `episodic_summaries` | Session conversation summaries | `agent_id`, `user_id`, `session_key`, `summary`, `l0_abstract`, `key_topics` (TEXT[]), `embedding` (vector), `source_id` (dedup), `expires_at` |
|
||||
| `episodic_summaries` | Session conversation summaries | `agent_id`, `user_id`, `session_key`, `summary`, `l0_abstract`, `key_topics` (TEXT[]), `embedding` (vector), `source_id` (dedup), `expires_at`, `recall_count` (INT), `recall_score` (FLOAT), `last_recalled_at` (TIMESTAMPTZ) |
|
||||
| `agent_evolution_metrics` | Self-evolution performance data | `agent_id`, `session_key`, `metric_type` (retrieval/tool/feedback), `metric_key`, `value` (JSONB) |
|
||||
| `agent_evolution_suggestions` | Data-driven improvement suggestions | `agent_id`, `suggestion_type`, `suggestion`, `rationale`, `parameters` (JSONB), `status` (pending/approved/rejected/applied) |
|
||||
| `vault_documents` | Knowledge Vault document registry | `agent_id`, `scope` (personal/team/shared), `path`, `title`, `doc_type`, `content_hash`, `embedding` (vector), `metadata` (JSONB) |
|
||||
@@ -757,7 +769,25 @@ flowchart TD
|
||||
| **EpisodicWorker** | `run.completed` | Extract session summary via LLM or compaction summary. Generate L0 abstract. Store in `episodic_summaries`. Emit `episodic.created` |
|
||||
| **SemanticWorker** | `episodic.created` | Parse summary for entity mentions and relationships. Extract via regex/NER. Insert into KG tables (`kg_entities`, `kg_relations`). Emit `entity.upserted` |
|
||||
| **DedupWorker** | `entity.upserted` | Check for duplicate entities via embedding similarity. Merge duplicate nodes by redirecting relations. Update timestamps to reflect consolidation |
|
||||
| **DreamingWorker** | `episodic.created` (debounced 10m) | Batch collect unpromoted episodic summaries. Call LLM for synthesis/insight pass. Write results to long-term memory (update KG, write to vault, etc.) |
|
||||
| **DreamingWorker** | `episodic.created` (debounced 10m) | Batch collect unpromoted episodic summaries scored by usefulness (recall signal). Call LLM for synthesis/insight pass. Write results to long-term memory (update KG, write to vault, etc.) |
|
||||
|
||||
### Dreaming Weighted Scoring (Phase 10, Migration 000045)
|
||||
|
||||
The DreamingWorker prioritizes unpromoted episodic summaries by usefulness via a 4-component running-average score:
|
||||
|
||||
**ComputeRecallScore formula** (14-day half-life):
|
||||
```
|
||||
score = 0.30 * frequency + 0.35 * relevance + 0.20 * recency + 0.15 * freshness
|
||||
```
|
||||
|
||||
**Tracking columns** (added to `episodic_summaries`):
|
||||
- `recall_count INT DEFAULT 0` — Number of times this summary was returned in memory searches
|
||||
- `recall_score DOUBLE PRECISION DEFAULT 0` — Weighted average score (0 to 1)
|
||||
- `last_recalled_at TIMESTAMPTZ` — Timestamp of most recent search hit
|
||||
|
||||
**Index for DreamingWorker**: `idx_episodic_recall_unpromoted` on `(agent_id, user_id, recall_score DESC) WHERE promoted_at IS NULL`. Enables efficient `ListUnpromotedScored()` queries to fetch highest-scoring summaries first.
|
||||
|
||||
**Integration with memory_search tool**: After search results are returned to agent, a fire-and-forget task increments `recall_count`, updates `recall_score` via running average, and sets `last_recalled_at`. No blocking — search returns immediately.
|
||||
|
||||
### Configuration
|
||||
|
||||
|
||||
@@ -227,7 +227,43 @@ OpenAI-compatible providers handle thinking/reasoning content as metadata. The `
|
||||
|
||||
---
|
||||
|
||||
## 6. Observability
|
||||
## 6. Reasoning Content Stripping (Phase 6 — OpenClaw TS port)
|
||||
|
||||
Some models emit chain-of-thought reasoning tokens even when `effort="off"` is specified. To prevent that raw CoT from reaching end users, GoClaw supports a `StripThinking` flag on `ReasoningDecision`.
|
||||
|
||||
### Models Known to Leak CoT
|
||||
|
||||
Auto-flagged via `modelLeaksReasoning(model)` in `internal/providers/reasoning_resolution.go`:
|
||||
- **Kimi family**: any model name containing `kimi` (case-insensitive, e.g. `kimi-k2`, `moonshot/kimi-k2-thinking`)
|
||||
- **DeepSeek-Reasoner**: any model name containing `deepseek-reasoner`
|
||||
|
||||
The allowlist is a simple substring check — extendable as new leaky models appear.
|
||||
|
||||
### Auto-Enable Flow
|
||||
|
||||
1. `ResolveReasoningDecision` runs its normal flow, then a `defer` checks: if `EffectiveEffort == "off" && modelLeaksReasoning(model)`, it sets `decision.StripThinking = true`.
|
||||
2. The agent loop (`loop_pipeline_callbacks.go`) reads `decision.StripThinking` and propagates it into `chatReq.Options[providers.OptStripThinking] = true`.
|
||||
3. Each provider's streaming handler reads `OptStripThinking` from options at the top of `ChatStream`/`Chat` and applies guard clauses.
|
||||
|
||||
### Implementation Per Provider
|
||||
|
||||
- **Anthropic** (`anthropic_stream.go`, `anthropic.go`): `thinking_delta` events skip `result.Thinking` accumulation AND the `onChunk` emit when stripping, but `thinkingChars` still increments so `Usage.ThinkingTokens` stays billable. `RawAssistantContent` (content blocks for tool-use passback) is never touched. Non-streaming `Chat()` clears `resp.Thinking` post-parse.
|
||||
- **OpenAI** (`openai_chat.go`): streaming guards `reasoning`/`reasoning_content` delta accumulation; non-streaming clears `resp.Thinking` post-parse. `Usage.ThinkingTokens` read from the usage chunk independently.
|
||||
- **Codex** (`codex.go`): `processSSEEvent` takes an extra `stripThinking bool` param; the `reasoning` item case skips summary text appending when set. Usage still extracted from `response.completed`.
|
||||
- **DashScope**: inherits both guards via `OpenAIProvider` embedding — no separate implementation needed.
|
||||
|
||||
### Invariants Preserved
|
||||
|
||||
| Field | Stripping effect | Rationale |
|
||||
|---|---|---|
|
||||
| `ChatResponse.Thinking` | Cleared | User-visible output |
|
||||
| `Usage.ThinkingTokens` | **Unchanged** | Billing accuracy (Phase 1 depends on it) |
|
||||
| `RawAssistantContent` | **Unchanged** | Anthropic tool-use replay requires raw thinking blocks |
|
||||
| `onChunk(StreamChunk{Thinking})` | Not emitted | Streaming UI display |
|
||||
|
||||
---
|
||||
|
||||
## 7. Observability
|
||||
|
||||
Each LLM span can now include a `metadata.reasoning` section with:
|
||||
- `source`
|
||||
|
||||
@@ -32,6 +32,59 @@ All notable changes to GoClaw Gateway are documented here. Format follows [Keep
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
#### Episodic Memory Weighted Scoring — Dreaming Enhancement (2026-04-10, Phase 10)
|
||||
- **Recall signal tracking**: `episodic_summaries` table gains 3 columns: `recall_count INT`, `recall_score DOUBLE PRECISION`, `last_recalled_at TIMESTAMPTZ` to track usefulness of each memory
|
||||
- **ComputeRecallScore formula**: 4-component running average (30% frequency + 35% relevance + 20% recency + 15% freshness, 14-day half-life) quantifies memory value
|
||||
- **DreamingWorker prioritization**: `ListUnpromotedScored()` queries sort by `recall_score DESC` instead of `created_at ASC`, promoting high-signal summaries for synthesis
|
||||
- **fire-and-forget updates**: `memory_search` tool fire-and-forget tasks increment recall counts asynchronously without blocking search results
|
||||
- **Index optimization**: New partial index `idx_episodic_recall_unpromoted ON episodic_summaries(agent_id, user_id, recall_score DESC) WHERE promoted_at IS NULL` for efficient DreamingWorker queries
|
||||
- **Migration 000045**: PG schema v44→45 + SQLite schema v12→13
|
||||
|
||||
#### Compaction Telemetry — Message Context Tracking (2026-04-10, Phase 5 Follow-up)
|
||||
- **Session metadata tracking**: `sessions.metadata` JSONB gains well-known key `last_compaction_at` (RFC3339 timestamp) after successful message compaction
|
||||
- **Dual execution paths**: Both v3 `PruneStage.CompactMessages` and v2 legacy `maybeSummarize` update timestamp on successful compaction
|
||||
- **Operator visibility**: `GetSessionMetadata()` exposes compaction timestamp; web UI shows in context-usage tooltip
|
||||
- **Go constant export**: `agent.SessionMetaKeyLastCompactionAt = "last_compaction_at"`
|
||||
|
||||
#### Provider Reasoning Content Stripping (2026-04-10, Phase 6)
|
||||
- **Auto-strip known leakers**: Models known to leak chain-of-thought at effort="off" (Kimi family, DeepSeek-Reasoner) auto-enable `StripThinking` so user-visible `ChatResponse.Thinking` stays empty
|
||||
- **Multi-provider support**: Guard clauses in Anthropic streaming, Anthropic non-streaming `Chat()`, OpenAI `ChatStream`/`Chat`, Codex `processSSEEvent`; DashScope inherits via OpenAIProvider embedding
|
||||
- **Billing-safe invariants**: `Usage.ThinkingTokens` still counted from raw bytes; `RawAssistantContent` untouched so Anthropic tool-use passback continues to work
|
||||
- **Option plumbing**: New `providers.OptStripThinking` key propagated via `ChatRequest.Options`; `ReasoningDecision.StripThinking` auto-set in `ResolveReasoningDecision` defer
|
||||
- **Helper**: `modelLeaksReasoning(model string) bool` — extensible allowlist
|
||||
|
||||
#### Dreaming Config Per-Agent (2026-04-10, Phase 8)
|
||||
- **Per-agent overrides**: `MemoryConfig.Dreaming` JSONB on `agents.memory_config` (nested, no migration) controls dreaming worker behaviour per-agent
|
||||
- **Fields**: `Enabled *bool`, `DebounceMs int`, `Threshold int`, `VerboseLog *bool` — all pointer/zero-default for partial-override merge semantics
|
||||
- **Resolver pattern**: `DreamingConfigResolver func(ctx, agentID) *DreamingConfig` wired via `newAgentStoreResolver(AgentStore)`. `ConsolidationDeps` gains optional `AgentStore store.AgentCRUDStore`
|
||||
- **Backward compat**: Nil resolver → struct defaults; empty JSONB → defaults via zero-value short-circuit
|
||||
- **Merge helper**: `mergeDreamingConfig` applies override fields only when explicitly set, preserving base values
|
||||
|
||||
#### Per-Provider Context Window & Hardening (2026-04-10, Phase 4, commit 8d37dc45)
|
||||
- **EffectiveContextWindow**: Resolved once per run in ContextStage from `ModelRegistry` (via provider+model lookup); `PruneStage` uses it with fallback to `Config.ContextWindow`
|
||||
- **ReserveTokens safety buffer**: New `PipelineConfig.ReserveTokens` subtracted from history budget so PruneStage compacts slightly before the hard limit
|
||||
- **InMemoryCache hardening**: TTL sweep (60s) + max-size cap (10k) with oldest-first eviction; `Close()` wired to gateway shutdown
|
||||
- **ContactCollector tenant fix**: Cache key now includes `tenantID + channelInstance` (was silently skipping upserts for same sender across tenants)
|
||||
|
||||
#### Context-Aware Auto-Inject Query (2026-04-10, Phase 9, commit 2731f99a)
|
||||
- **RecentContext enrichment**: `InjectParams.RecentContext` field supplies last 1-2 user turns; `pgAutoInjector.Inject` prepends "Context:" frame before "Query:" focus
|
||||
- **Helper**: `pipeline.buildRecentContext()` walks history backward, picks last 2 user turns, caps at 300 chars
|
||||
- **Callback signature**: `AutoInject func(ctx, msg, userID, recentContext string)` — additive, backward compatible
|
||||
- **Why**: Vector search on "what's my favorite?" alone returns nothing useful; context-aware query captures conversational intent
|
||||
|
||||
#### Cost Calculation Thinking Tokens Fix (2026-04-10, Phase 1, commit 77a80680)
|
||||
- **Critical billing bug**: `CalculateCost` now properly handles `ThinkingTokens` as sub-count of `CompletionTokens` (not double-counted for OpenAI o3/o4-mini, Codex/GPT-5; properly split for Anthropic extended thinking)
|
||||
- **Provider-aware**: Splits only when `ReasoningPerMillion > 0`, otherwise leaves as-is (default matches provider billing)
|
||||
|
||||
#### Web UI Enhancements (2026-04-10, Phase 11)
|
||||
- **Context usage badge**: Chat top bar shows `{used}/{max} ({percent}%)` with color ramp (amber ≥75%, destructive ≥90%); hidden on mobile via `hidden sm:flex`
|
||||
- **Compaction indicator**: Context badge tooltip includes compaction count + last compaction timestamp (read from `session.metadata.last_compaction_at`)
|
||||
- **DreamingConfig UI**: Agent detail MemorySection renders nested dreaming block (4 controls: enabled, threshold, debounce_ms, verbose_log)
|
||||
- **i18n**: New keys in en/vi/zh: `agents.configSections.dreaming.*`, `chat.contextUsage.*`
|
||||
- **Not shipped (YAGNI)**: "Memory recall config section" (existing MemorySection already covers all MemoryConfig fields), "Session types extension" (fields already present or out of scope)
|
||||
|
||||
### Refactored
|
||||
|
||||
#### V3 Architecture Refactor — Phase 6 Completion (2026-04-08)
|
||||
@@ -233,9 +286,11 @@ All notable changes to GoClaw Gateway are documented here. Format follows [Keep
|
||||
- **File viewer**: Improved workspace file view/download and storage depth control
|
||||
- **Pairing DB errors**: Handle transient errors gracefully
|
||||
- **Provider thinking**: Corrected DashScope per-model thinking logic
|
||||
- **Pancake Page loop guard**: Narrowed webhook ingress to `messaging` + `INBOX` events and normalized HTML-formatted echoes before short-TTL outbound echo suppression, reducing Facebook Page self-reply loops in Pancake inbox conversations
|
||||
|
||||
### Documentation
|
||||
|
||||
- Updated `05-channels-messaging.md` — Refreshed `WebhookChannel` / `BlockReplyChannel` implementation tables for Facebook, Pancake, Discord, and Zalo-family channels
|
||||
- Updated `18-http-api.md` — Added section 17 for Runtime & Packages Management endpoints
|
||||
- Updated `09-security.md` — Added Docker entrypoint documentation, pkg-helper architecture, privilege separation
|
||||
- Updated `17-changelog.md` — New entries for packages management, Docker security, and auth fix
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"math"
|
||||
"slices"
|
||||
|
||||
"github.com/google/uuid"
|
||||
@@ -55,22 +54,8 @@ func CheckGuardrails(g AdaptationGuardrails, sg store.EvolutionSuggestion, dataP
|
||||
}
|
||||
}
|
||||
|
||||
// Check delta constraint for threshold-type suggestions.
|
||||
if sg.SuggestionType == store.SuggestThreshold {
|
||||
maxDelta := g.MaxDeltaPerCycle
|
||||
if maxDelta <= 0 {
|
||||
maxDelta = 0.1
|
||||
}
|
||||
var params map[string]any
|
||||
if err := json.Unmarshal(sg.Parameters, ¶ms); err == nil {
|
||||
if currentRate, ok := params["current_usage_rate"].(float64); ok {
|
||||
// Proposed change is bounded by max delta.
|
||||
if math.Abs(currentRate) > maxDelta {
|
||||
return fmt.Errorf("delta %.2f exceeds max %.2f per cycle", currentRate, maxDelta)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Threshold suggestions: the applied delta is capped at MaxDeltaPerCycle in ApplySuggestion,
|
||||
// so no additional delta check is needed here. MinDataPoints and LockedParams already guard safety.
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -78,7 +63,7 @@ func CheckGuardrails(g AdaptationGuardrails, sg store.EvolutionSuggestion, dataP
|
||||
// ApplySuggestion applies an approved suggestion's parameters to the agent's other_config JSONB.
|
||||
// Stores the previous values in the suggestion's parameters for rollback.
|
||||
// Scope: only retrieval-related parameters. Never security or core settings.
|
||||
func ApplySuggestion(ctx context.Context, agentStore store.AgentStore, sugStore store.EvolutionSuggestionStore, sg store.EvolutionSuggestion) error {
|
||||
func ApplySuggestion(ctx context.Context, agentStore store.AgentStore, sugStore store.EvolutionSuggestionStore, sg store.EvolutionSuggestion, guardrails AdaptationGuardrails) error {
|
||||
// Load current agent config.
|
||||
agent, err := agentStore.GetByID(ctx, sg.AgentID)
|
||||
if err != nil {
|
||||
@@ -102,12 +87,20 @@ func ApplySuggestion(ctx context.Context, agentStore store.AgentStore, sugStore
|
||||
// Apply suggestion-specific changes based on type.
|
||||
switch sg.SuggestionType {
|
||||
case store.SuggestThreshold:
|
||||
// Raise retrieval threshold slightly (bounded by guardrails).
|
||||
// Raise retrieval threshold by MaxDeltaPerCycle (bounded by guardrails).
|
||||
current, _ := otherConfig["retrieval_threshold"].(float64)
|
||||
if current == 0 {
|
||||
current = 0.3 // default threshold
|
||||
}
|
||||
otherConfig["retrieval_threshold"] = current + 0.05 // conservative bump
|
||||
delta := guardrails.MaxDeltaPerCycle
|
||||
if delta <= 0 {
|
||||
delta = 0.05 // fallback
|
||||
}
|
||||
newThreshold := current + delta
|
||||
if newThreshold > 0.95 {
|
||||
newThreshold = 0.95
|
||||
}
|
||||
otherConfig["retrieval_threshold"] = newThreshold
|
||||
default:
|
||||
// Non-threshold suggestions are informational only, no auto-apply.
|
||||
return fmt.Errorf("suggestion type %q does not support auto-apply", sg.SuggestionType)
|
||||
|
||||
@@ -2,6 +2,7 @@ package agent
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
@@ -51,7 +52,7 @@ func TestCheckGuardrails(t *testing.T) {
|
||||
dataPoints: 200,
|
||||
params: map[string]any{"source": "mem"},
|
||||
sgType: store.SuggestThreshold,
|
||||
wantErr: `parameter "source" is locked`,
|
||||
wantErr: "parameter \"source\" is locked",
|
||||
},
|
||||
{
|
||||
name: "no locked params passes",
|
||||
@@ -67,6 +68,14 @@ func TestCheckGuardrails(t *testing.T) {
|
||||
sgType: store.SuggestThreshold,
|
||||
wantErr: "insufficient data",
|
||||
},
|
||||
{
|
||||
name: "threshold with usage rate passes (no broken delta check)",
|
||||
guardrails: DefaultGuardrails(),
|
||||
dataPoints: 200,
|
||||
params: map[string]any{"current_usage_rate": 0.15, "source": "vault"},
|
||||
sgType: store.SuggestThreshold,
|
||||
// Previously this would fail with "delta 0.15 exceeds max 0.10" — now passes.
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -84,24 +93,10 @@ func TestCheckGuardrails(t *testing.T) {
|
||||
t.Errorf("expected error containing %q, got nil", tt.wantErr)
|
||||
}
|
||||
if tt.wantErr != "" && err != nil {
|
||||
if got := err.Error(); !contains(got, tt.wantErr) {
|
||||
t.Errorf("error %q does not contain %q", got, tt.wantErr)
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Errorf("error %q does not contain %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func contains(s, substr string) bool {
|
||||
return len(s) >= len(substr) && (s == substr || len(substr) == 0 ||
|
||||
(len(s) > 0 && len(substr) > 0 && searchStr(s, substr)))
|
||||
}
|
||||
|
||||
func searchStr(s, sub string) bool {
|
||||
for i := 0; i <= len(s)-len(sub); i++ {
|
||||
if s[i:i+len(sub)] == sub {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -115,7 +115,8 @@ func (l *Loop) injectContext(ctx context.Context, req *RunRequest) (contextSetup
|
||||
// Seeding must run before buildMessages→resolveContextFiles reads context files.
|
||||
// Team sessions skip seeding: members process tasks from leader, not end-user onboarding.
|
||||
isTeamSession := bootstrap.IsTeamSession(req.SessionKey)
|
||||
setup := l.getOrCreateUserSetup(ctx, req.UserID, req.Channel, isTeamSession)
|
||||
channelMeta := l.buildChannelMeta(req)
|
||||
setup := l.getOrCreateUserSetup(ctx, req.UserID, req.Channel, isTeamSession, channelMeta)
|
||||
|
||||
// Workspace resolution (layered pipeline).
|
||||
// Layer order: tenant → team → project (future) → user/chat
|
||||
|
||||
@@ -295,6 +295,11 @@ func (l *Loop) maybeSummarize(ctx context.Context, sessionKey string) {
|
||||
l.sessions.SetSummary(sctx, sessionKey, SanitizeAssistantContent(resp.Content))
|
||||
l.sessions.TruncateHistory(sctx, sessionKey, keepLast)
|
||||
l.sessions.IncrementCompaction(sctx, sessionKey)
|
||||
// Mirror SessionMetaKeyLastCompactionAt from the v3 prune/compact path
|
||||
// so the legacy v2 post-turn summarizer also surfaces compaction cadence.
|
||||
l.sessions.SetSessionMetadata(sctx, sessionKey, map[string]string{
|
||||
SessionMetaKeyLastCompactionAt: time.Now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
l.sessions.Save(sctx, sessionKey)
|
||||
}()
|
||||
}
|
||||
|
||||
@@ -3,12 +3,28 @@ package agent
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tools"
|
||||
)
|
||||
|
||||
// isTextMime returns true for MIME types representing human-readable text content.
|
||||
// Covers text/* family plus common application/* types that are text-based.
|
||||
func isTextMime(mime string) bool {
|
||||
if strings.HasPrefix(mime, "text/") {
|
||||
return true
|
||||
}
|
||||
switch mime {
|
||||
case "application/json", "application/xml", "application/yaml",
|
||||
"application/x-yaml", "application/javascript":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// collectRefsByKind gathers MediaRefs of a given kind from message history
|
||||
// (reverse order) and current-turn refs. Historical first, current last.
|
||||
func collectRefsByKind(messages []providers.Message, currentRefs []providers.MediaRef, kind string) []providers.MediaRef {
|
||||
@@ -47,6 +63,18 @@ func (l *Loop) enrichInputMedia(ctx context.Context, req *RunRequest, messages [
|
||||
var mediaRefs []providers.MediaRef
|
||||
if len(req.Media) > 0 {
|
||||
mediaRefs = l.persistMedia(req.SessionKey, req.Media, tools.ToolWorkspaceFromCtx(ctx))
|
||||
|
||||
// Register persisted text uploads in vault (async, non-blocking).
|
||||
if l.onTextUploaded != nil {
|
||||
for _, ref := range mediaRefs {
|
||||
if ref.Path != "" && isTextMime(ref.MimeType) {
|
||||
if content, err := os.ReadFile(ref.Path); err == nil {
|
||||
go l.onTextUploaded(context.WithoutCancel(ctx), ref.Path, string(content))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Load current-turn images from persisted refs (Path is always set for new uploads).
|
||||
var imageFiles []bus.MediaFile
|
||||
for _, ref := range mediaRefs {
|
||||
|
||||
@@ -86,7 +86,7 @@ func (l *Loop) getUserMCPTools(ctx context.Context, userID string) []tools.Tool
|
||||
// shared tool registry so ExecuteWithContext can resolve them by name.
|
||||
reg, _ := l.tools.(*tools.Registry)
|
||||
for _, mcpTool := range entry.MCPTools() {
|
||||
bt := mcpbridge.NewBridgeTool(srv.Name, mcpTool, entry.Client(), srv.ToolPrefix, srv.TimeoutSec, entry.Connected())
|
||||
bt := mcpbridge.NewBridgeTool(srv.Name, mcpTool, entry.ClientPtr(), srv.ToolPrefix, srv.TimeoutSec, entry.Connected())
|
||||
// Register in registry so ExecuteWithContext can find them.
|
||||
// Skip if already registered (another user loaded this server with same tool names).
|
||||
if reg != nil {
|
||||
|
||||
@@ -59,6 +59,19 @@ func (l *Loop) buildPipelineDeps(req *RunRequest, bridgeRS *runState) pipeline.P
|
||||
Compaction: l.compactionCfg,
|
||||
// V3 memory/retrieval flags removed — always true at runtime.
|
||||
},
|
||||
// Resolve per-model context window once per run. Falls back to
|
||||
// Config.ContextWindow when registry/model is unknown (existing
|
||||
// behaviour unchanged for tests and lite edition).
|
||||
ResolveContextWindow: func(provider, model string) int {
|
||||
if l.modelRegistry == nil || model == "" {
|
||||
return 0
|
||||
}
|
||||
spec := l.modelRegistry.Resolve(provider, model)
|
||||
if spec == nil {
|
||||
return 0
|
||||
}
|
||||
return spec.ContextWindow
|
||||
},
|
||||
EmitEvent: func(event any) {
|
||||
if ae, ok := event.(AgentEvent); ok {
|
||||
l.emit(ae)
|
||||
@@ -233,16 +246,19 @@ func convertRunResult(pr *pipeline.RunResult) *RunResult {
|
||||
|
||||
// makeAutoInjectCallback creates the AutoInject callback that captures agent/tenant context.
|
||||
// Returns nil if autoInjector is not configured (v3 retrieval disabled or no episodic store).
|
||||
func (l *Loop) makeAutoInjectCallback(req *RunRequest) func(ctx context.Context, userMessage, userID string) (string, error) {
|
||||
// Phase 9: plumbs recentContext through to enrich vector search queries for
|
||||
// context-aware recall.
|
||||
func (l *Loop) makeAutoInjectCallback(req *RunRequest) func(ctx context.Context, userMessage, userID, recentContext string) (string, error) {
|
||||
if l.autoInjector == nil {
|
||||
return nil
|
||||
}
|
||||
return func(ctx context.Context, userMessage, userID string) (string, error) {
|
||||
return func(ctx context.Context, userMessage, userID, recentContext string) (string, error) {
|
||||
result, err := l.autoInjector.Inject(ctx, memory.InjectParams{
|
||||
AgentID: l.agentUUID.String(),
|
||||
UserID: userID,
|
||||
TenantID: store.TenantIDFromContext(ctx).String(),
|
||||
UserMessage: userMessage,
|
||||
AgentID: l.agentUUID.String(),
|
||||
UserID: userID,
|
||||
TenantID: store.TenantIDFromContext(ctx).String(),
|
||||
UserMessage: userMessage,
|
||||
RecentContext: recentContext,
|
||||
})
|
||||
if err != nil || result == nil {
|
||||
return "", err
|
||||
|
||||
@@ -47,7 +47,7 @@ func (l *Loop) pipelineCallbacks(req *RunRequest, bridgeRS *runState) pipelineCa
|
||||
buildFilteredTools: l.makeBuildFilteredTools(req),
|
||||
callLLM: l.makeCallLLM(req, emitRun),
|
||||
pruneMessages: l.makePruneMessages(),
|
||||
compactMessages: l.makeCompactMessages(),
|
||||
compactMessages: l.makeCompactMessages(req),
|
||||
runMemoryFlush: l.makeRunMemoryFlush(),
|
||||
executeToolCall: l.makeExecuteToolCall(req, bridgeRS),
|
||||
executeToolRaw: l.makeExecuteToolRaw(req),
|
||||
@@ -241,6 +241,9 @@ func (l *Loop) makeCallLLM(req *RunRequest, emitRun func(AgentEvent)) func(ctx c
|
||||
if effort := reasoningDecision.RequestEffort(); effort != "" {
|
||||
chatReq.Options[providers.OptThinkingLevel] = effort
|
||||
}
|
||||
if reasoningDecision.StripThinking {
|
||||
chatReq.Options[providers.OptStripThinking] = true
|
||||
}
|
||||
|
||||
// Emit LLM span start for tracing.
|
||||
start := time.Now().UTC()
|
||||
@@ -309,16 +312,30 @@ func (l *Loop) makePruneMessages() func(msgs []providers.Message, budget int) []
|
||||
}
|
||||
}
|
||||
|
||||
func (l *Loop) makeCompactMessages() func(ctx context.Context, msgs []providers.Message, model string) ([]providers.Message, error) {
|
||||
func (l *Loop) makeCompactMessages(req *RunRequest) func(ctx context.Context, msgs []providers.Message, model string) ([]providers.Message, error) {
|
||||
return func(ctx context.Context, msgs []providers.Message, model string) ([]providers.Message, error) {
|
||||
compacted := l.compactMessagesInPlace(ctx, msgs)
|
||||
if compacted == nil {
|
||||
return msgs, nil // compaction failed, return original
|
||||
}
|
||||
// Stamp session metadata with the compaction timestamp so operators
|
||||
// can diagnose compaction cadence without a dedicated column. Stored
|
||||
// as RFC3339 string in sessions.metadata JSONB (flushed on next save).
|
||||
if l.sessions != nil && req != nil && req.SessionKey != "" {
|
||||
l.sessions.SetSessionMetadata(ctx, req.SessionKey, map[string]string{
|
||||
SessionMetaKeyLastCompactionAt: time.Now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
return compacted, nil
|
||||
}
|
||||
}
|
||||
|
||||
// SessionMetaKeyLastCompactionAt is the sessions.metadata JSONB key used to
|
||||
// record the RFC3339 timestamp of the most recent compaction. Exported so
|
||||
// the web UI code path can read it back via GetSessionMetadata without
|
||||
// duplicating the string.
|
||||
const SessionMetaKeyLastCompactionAt = "last_compaction_at"
|
||||
|
||||
func (l *Loop) makeRunMemoryFlush() func(ctx context.Context, state *pipeline.RunState) error {
|
||||
return func(ctx context.Context, state *pipeline.RunState) error {
|
||||
settings := ResolveMemoryFlushSettings(l.compactionCfg)
|
||||
|
||||
@@ -45,7 +45,8 @@ type EnsureUserProfileFunc func(ctx context.Context, agentID uuid.UUID, userID,
|
||||
// Called once per user per Loop instance, independent of workspace.
|
||||
// isNew indicates whether the profile was just created (seed all) or already existed
|
||||
// (only seed if user has zero files — avoids re-seeding after BOOTSTRAP.md cleanup).
|
||||
type SeedUserFilesFunc func(ctx context.Context, agentID uuid.UUID, userID, agentType string, isNew bool) error
|
||||
// channelMeta carries optional channel-provided contact info for bootstrap skip decisions.
|
||||
type SeedUserFilesFunc func(ctx context.Context, agentID uuid.UUID, userID, agentType string, isNew bool, channelMeta *bootstrap.ChannelMeta) error
|
||||
|
||||
// EnsureUserFilesFunc is the legacy combined callback (profile + seed + workspace).
|
||||
// Deprecated: use EnsureUserProfileFunc + SeedUserFilesFunc separately.
|
||||
@@ -72,8 +73,10 @@ type Loop struct {
|
||||
agentUUID uuid.UUID // set for context propagation
|
||||
tenantID uuid.UUID // agent's owning tenant
|
||||
agentType string // "open" or "predefined"
|
||||
defaultTimezone string // system default timezone for bootstrap pre-fill
|
||||
provider providers.Provider
|
||||
model string
|
||||
modelRegistry providers.ModelRegistry // resolves per-model context window at run time (nil = use static contextWindow)
|
||||
contextWindow int
|
||||
maxTokens int // max output tokens per LLM call (0 = default 8192)
|
||||
maxIterations int
|
||||
@@ -186,6 +189,10 @@ type Loop struct {
|
||||
// Secure CLI store for credentialed exec context injection
|
||||
secureCLIStore store.SecureCLIStore
|
||||
|
||||
// Vault hook: called when a text file is persisted from user upload.
|
||||
// Enables vault registration without agent package importing vault.
|
||||
onTextUploaded func(ctx context.Context, path, content string)
|
||||
|
||||
// Persistent media storage for cross-turn image/document access
|
||||
mediaStore *media.Store
|
||||
|
||||
@@ -256,6 +263,10 @@ type LoopConfig struct {
|
||||
MemoryCfg *config.MemoryConfig
|
||||
SandboxCfg *sandbox.Config
|
||||
|
||||
// ModelRegistry resolves provider/model → ModelSpec for per-run context
|
||||
// window lookup. Nil = fall back to static LoopConfig.ContextWindow.
|
||||
ModelRegistry providers.ModelRegistry
|
||||
|
||||
Bus bus.EventPublisher
|
||||
DomainBus eventbus.DomainEventBus // V3 domain event bus for consolidation pipeline
|
||||
Sessions store.SessionStore
|
||||
@@ -299,6 +310,7 @@ type LoopConfig struct {
|
||||
ContextFileLoader ContextFileLoaderFunc
|
||||
BootstrapCleanup BootstrapCleanupFunc
|
||||
CacheInvalidate CacheInvalidateFunc // invalidate context file cache after seeding
|
||||
DefaultTimezone string // system default timezone for bootstrap pre-fill
|
||||
|
||||
// Tracing collector (nil = no tracing)
|
||||
TraceCollector *tracing.Collector
|
||||
@@ -339,6 +351,9 @@ type LoopConfig struct {
|
||||
// Secure CLI store for credentialed exec context injection
|
||||
SecureCLIStore store.SecureCLIStore
|
||||
|
||||
// Vault hook: called asynchronously when a text file is persisted from user upload.
|
||||
OnTextUploaded func(ctx context.Context, path, content string)
|
||||
|
||||
// Persistent media storage for cross-turn image/document access
|
||||
MediaStore *media.Store
|
||||
|
||||
@@ -409,6 +424,7 @@ func NewLoop(cfg LoopConfig) *Loop {
|
||||
agentType: cfg.AgentType,
|
||||
provider: cfg.Provider,
|
||||
model: cfg.Model,
|
||||
modelRegistry: cfg.ModelRegistry,
|
||||
contextWindow: cfg.ContextWindow,
|
||||
maxTokens: cfg.MaxTokens,
|
||||
maxIterations: cfg.MaxIterations,
|
||||
@@ -433,6 +449,7 @@ func NewLoop(cfg LoopConfig) *Loop {
|
||||
skillAllowList: cfg.SkillAllowList,
|
||||
hasMemory: cfg.HasMemory,
|
||||
contextFiles: cfg.ContextFiles,
|
||||
defaultTimezone: cfg.DefaultTimezone,
|
||||
ensureUserProfile: cfg.EnsureUserProfile,
|
||||
seedUserFiles: cfg.SeedUserFiles,
|
||||
ensureUserFiles: cfg.EnsureUserFiles,
|
||||
@@ -461,6 +478,7 @@ func NewLoop(cfg LoopConfig) *Loop {
|
||||
configPermStore: cfg.ConfigPermStore,
|
||||
teamStore: cfg.TeamStore,
|
||||
secureCLIStore: cfg.SecureCLIStore,
|
||||
onTextUploaded: cfg.OnTextUploaded,
|
||||
mediaStore: cfg.MediaStore,
|
||||
modelPricing: cfg.ModelPricing,
|
||||
budgetMonthlyCents: cfg.BudgetMonthlyCents,
|
||||
|
||||
@@ -64,10 +64,23 @@ func (l *Loop) shouldShareKnowledgeGraph() bool {
|
||||
return l.workspaceSharing != nil && l.workspaceSharing.ShareKnowledgeGraph
|
||||
}
|
||||
|
||||
// buildChannelMeta extracts channel metadata from RunRequest for bootstrap decisions.
|
||||
// Returns nil when channel type is unknown (preserves normal bootstrap flow).
|
||||
func (l *Loop) buildChannelMeta(req *RunRequest) *bootstrap.ChannelMeta {
|
||||
if req == nil || req.ChannelType == "" {
|
||||
return nil
|
||||
}
|
||||
return &bootstrap.ChannelMeta{
|
||||
ChannelType: req.ChannelType,
|
||||
DisplayName: req.SenderName,
|
||||
DefaultTimezone: l.defaultTimezone,
|
||||
}
|
||||
}
|
||||
|
||||
// getOrCreateUserSetup returns the cached userSetup for a user, creating it on first call.
|
||||
// On first call: seeds context files (non-team) and resolves workspace from user profile.
|
||||
// On subsequent calls: returns cached setup immediately (no DB calls).
|
||||
func (l *Loop) getOrCreateUserSetup(ctx context.Context, userID, channel string, isTeamSession bool) *userSetup {
|
||||
func (l *Loop) getOrCreateUserSetup(ctx context.Context, userID, channel string, isTeamSession bool, channelMeta *bootstrap.ChannelMeta) *userSetup {
|
||||
if userID == "" {
|
||||
return &userSetup{workspace: l.workspace}
|
||||
}
|
||||
@@ -92,7 +105,7 @@ func (l *Loop) getOrCreateUserSetup(ctx context.Context, userID, channel string,
|
||||
}
|
||||
// Step 2: Seed context files (must run before buildMessages reads them).
|
||||
// Passes isNew so SeedUserFiles knows whether to skip existing files.
|
||||
if err := l.seedUserFiles(ctx, l.agentUUID, userID, l.agentType, isNew); err != nil {
|
||||
if err := l.seedUserFiles(ctx, l.agentUUID, userID, l.agentType, isNew, channelMeta); err != nil {
|
||||
slog.Warn("failed to seed user context files", "error", err)
|
||||
// Seeding failed (e.g. SQLITE_BUSY after retries). Inject
|
||||
// embedded bootstrap templates in-memory so the first turn
|
||||
|
||||
@@ -4,34 +4,49 @@ import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bootstrap"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tokencount"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tools"
|
||||
)
|
||||
|
||||
// PreviewDeps holds optional dependencies for building a preview system prompt.
|
||||
// All fields are nil-safe — missing deps simply skip resolution for that section.
|
||||
type PreviewDeps struct {
|
||||
AgentStore store.AgentStore
|
||||
TeamStore store.TeamStore
|
||||
AgentLinks store.AgentLinkStore
|
||||
ProviderReg *providers.Registry
|
||||
ToolLister interface{ List() []string }
|
||||
SkillsLoader interface {
|
||||
AgentStore store.AgentStore
|
||||
TeamStore store.TeamStore
|
||||
AgentLinks store.AgentLinkStore
|
||||
ProviderReg *providers.Registry
|
||||
SkillAccessStore store.SkillAccessStore
|
||||
ToolLister interface {
|
||||
List() []string
|
||||
Get(name string) (tools.Tool, bool)
|
||||
Aliases() map[string]string
|
||||
}
|
||||
SkillsLoader interface {
|
||||
BuildPinnedSummary(ctx context.Context, names []string) string
|
||||
BuildSummary(ctx context.Context, allowList []string) string
|
||||
}
|
||||
DataDir string // for team workspace path construction
|
||||
}
|
||||
|
||||
// PreviewResult holds the output of BuildPreviewPrompt.
|
||||
type PreviewResult struct {
|
||||
Prompt string
|
||||
ToolDefs []providers.ToolDefinition // tool definitions (schemas) as sent to the LLM
|
||||
}
|
||||
|
||||
// BuildPreviewPrompt builds a system prompt for preview purposes.
|
||||
// Reuses the same BuildSystemPrompt() as the LLM pipeline, resolving as many
|
||||
// fields as possible from agent data + DB stores. Runtime-only fields
|
||||
// (channel, peer kind, session context, credentials) are left at zero values —
|
||||
// BuildSystemPrompt already nil-checks every field.
|
||||
func BuildPreviewPrompt(ctx context.Context, ag *store.AgentData, mode PromptMode, userID string, deps PreviewDeps) string {
|
||||
func BuildPreviewPrompt(ctx context.Context, ag *store.AgentData, mode PromptMode, userID string, deps PreviewDeps) PreviewResult {
|
||||
// --- Context files ---
|
||||
var contextFiles []bootstrap.ContextFile
|
||||
if deps.AgentStore != nil {
|
||||
@@ -62,6 +77,63 @@ func BuildPreviewPrompt(ctx context.Context, ag *store.AgentData, mode PromptMod
|
||||
toolNames = fallbackPreviewToolNames
|
||||
}
|
||||
|
||||
// --- skill_manage gating (matches loop_history.go:124-131) ---
|
||||
if !ag.ParseSkillEvolve() {
|
||||
filtered := make([]string, 0, len(toolNames))
|
||||
for _, n := range toolNames {
|
||||
if n != "skill_manage" {
|
||||
filtered = append(filtered, n)
|
||||
}
|
||||
}
|
||||
toolNames = filtered
|
||||
}
|
||||
|
||||
// --- Basic agent tool policy: deny-only (Allow/AlsoAllow/ByProvider not applied).
|
||||
// Full PolicyEngine requires runtime state (provider name, channel) not available in preview. ---
|
||||
if toolPolicy := ag.ParseToolsConfig(); toolPolicy != nil && len(toolPolicy.Deny) > 0 {
|
||||
denySet := make(map[string]bool, len(toolPolicy.Deny))
|
||||
for _, d := range toolPolicy.Deny {
|
||||
denySet[d] = true
|
||||
}
|
||||
filtered := make([]string, 0, len(toolNames))
|
||||
for _, n := range toolNames {
|
||||
if !denySet[n] {
|
||||
filtered = append(filtered, n)
|
||||
}
|
||||
}
|
||||
toolNames = filtered
|
||||
}
|
||||
|
||||
// --- Alias exclusion (matches loop_history.go:136-146) ---
|
||||
if deps.ToolLister != nil {
|
||||
if aliasSet := deps.ToolLister.Aliases(); len(aliasSet) > 0 {
|
||||
filtered := make([]string, 0, len(toolNames))
|
||||
for _, n := range toolNames {
|
||||
if _, isAlias := aliasSet[n]; !isAlias {
|
||||
filtered = append(filtered, n)
|
||||
}
|
||||
}
|
||||
toolNames = filtered
|
||||
}
|
||||
}
|
||||
|
||||
// --- MCP tool descriptions (matches loop_history_supplement.go:44-58) ---
|
||||
var mcpToolDescs map[string]string
|
||||
if deps.ToolLister != nil {
|
||||
descs := make(map[string]string)
|
||||
for _, name := range toolNames {
|
||||
if !strings.HasPrefix(name, "mcp_") || name == "mcp_tool_search" {
|
||||
continue
|
||||
}
|
||||
if tool, ok := deps.ToolLister.Get(name); ok {
|
||||
descs[name] = tool.Description()
|
||||
}
|
||||
}
|
||||
if len(descs) > 0 {
|
||||
mcpToolDescs = descs
|
||||
}
|
||||
}
|
||||
|
||||
// --- Sandbox ---
|
||||
sandboxCfg := ag.ParseSandboxConfig()
|
||||
sandboxEnabled := sandboxCfg != nil && sandboxCfg.Mode != "" && sandboxCfg.Mode != "off"
|
||||
@@ -76,6 +148,32 @@ func BuildPreviewPrompt(ctx context.Context, ag *store.AgentData, mode PromptMod
|
||||
pinnedSummary = deps.SkillsLoader.BuildPinnedSummary(ctx, pinnedSkills)
|
||||
}
|
||||
|
||||
// --- Skills summary (BuildSummary + token count) ---
|
||||
var skillsSummary string
|
||||
if deps.SkillsLoader != nil {
|
||||
var skillAllowList []string
|
||||
if deps.SkillAccessStore != nil {
|
||||
if accessible, err := deps.SkillAccessStore.ListAccessible(ctx, ag.ID, userID); err == nil {
|
||||
skillAllowList = make([]string, 0, len(accessible))
|
||||
for _, sk := range accessible {
|
||||
skillAllowList = append(skillAllowList, sk.Slug)
|
||||
}
|
||||
} else {
|
||||
// On error: empty list (no skills). Preview is diagnostic; safer than showing all.
|
||||
skillAllowList = []string{}
|
||||
}
|
||||
}
|
||||
|
||||
summary := deps.SkillsLoader.BuildSummary(ctx, skillAllowList)
|
||||
if summary != "" {
|
||||
tokens := tokencount.NewFallbackCounter().Count("claude-3", summary)
|
||||
if tokens <= skillInlineMaxTokens {
|
||||
skillsSummary = summary
|
||||
}
|
||||
// Over threshold → search-only mode (skillsSummary stays empty)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Provider contribution ---
|
||||
var providerContrib *providers.PromptContribution
|
||||
if deps.ProviderReg != nil && ag.Provider != "" {
|
||||
@@ -121,8 +219,31 @@ func BuildPreviewPrompt(ctx context.Context, ag *store.AgentData, mode PromptMod
|
||||
}
|
||||
}
|
||||
|
||||
// --- Tool definitions (schemas sent to LLM alongside the system prompt) ---
|
||||
var toolDefs []providers.ToolDefinition
|
||||
if deps.ToolLister != nil {
|
||||
for _, name := range toolNames {
|
||||
if tool, ok := deps.ToolLister.Get(name); ok {
|
||||
toolDefs = append(toolDefs, tools.ToProviderDef(tool))
|
||||
}
|
||||
}
|
||||
// Include alias definitions (LLM receives both canonical + aliases)
|
||||
for alias, canonical := range deps.ToolLister.Aliases() {
|
||||
if tool, ok := deps.ToolLister.Get(canonical); ok {
|
||||
toolDefs = append(toolDefs, providers.ToolDefinition{
|
||||
Type: "function",
|
||||
Function: providers.ToolFunctionSchema{
|
||||
Name: alias,
|
||||
Description: tool.Description(),
|
||||
Parameters: tool.Parameters(),
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- Build system prompt (same function as LLM pipeline) ---
|
||||
return BuildSystemPrompt(SystemPromptConfig{
|
||||
prompt := BuildSystemPrompt(SystemPromptConfig{
|
||||
AgentID: ag.AgentKey,
|
||||
AgentUUID: ag.ID.String(),
|
||||
DisplayName: ag.DisplayName,
|
||||
@@ -146,6 +267,8 @@ func BuildPreviewPrompt(ctx context.Context, ag *store.AgentData, mode PromptMod
|
||||
SandboxEnabled: sandboxEnabled,
|
||||
SandboxContainerDir: sandboxContainerDir,
|
||||
PinnedSkillsSummary: pinnedSummary,
|
||||
SkillsSummary: skillsSummary,
|
||||
MCPToolDescs: mcpToolDescs,
|
||||
IsTeamContext: isTeamCtx,
|
||||
TeamWorkspace: teamWorkspace,
|
||||
TeamMembers: teamMembers,
|
||||
@@ -154,8 +277,9 @@ func BuildPreviewPrompt(ctx context.Context, ag *store.AgentData, mode PromptMod
|
||||
OrchMode: orchMode,
|
||||
// Runtime-only fields left at zero: Channel, ChannelType, ChatTitle,
|
||||
// PeerKind, OwnerIDs, ExtraPrompt, CredentialCLIContext, IsBootstrap,
|
||||
// MCPToolDescs, SkillsSummary, SandboxWorkspaceAccess
|
||||
// SandboxWorkspaceAccess
|
||||
})
|
||||
return PreviewResult{Prompt: prompt, ToolDefs: toolDefs}
|
||||
}
|
||||
|
||||
// mergePreviewUserFiles overlays per-user files onto base agent-level files.
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
func baseAgent() *store.AgentData {
|
||||
return &store.AgentData{
|
||||
BaseModel: store.BaseModel{ID: uuid.New()},
|
||||
AgentKey: "test-agent",
|
||||
AgentType: store.AgentTypePredefined,
|
||||
Workspace: "/workspace",
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_NilDeps(t *testing.T) {
|
||||
r := BuildPreviewPrompt(context.Background(), baseAgent(), PromptFull, "", PreviewDeps{})
|
||||
if r.Prompt == "" {
|
||||
t.Fatal("expected non-empty prompt with nil deps")
|
||||
}
|
||||
if !strings.Contains(r.Prompt, "read_file") {
|
||||
t.Error("expected fallback tool names in prompt")
|
||||
}
|
||||
// No tool lister → no tool defs
|
||||
if len(r.ToolDefs) != 0 {
|
||||
t.Errorf("expected no tool defs with nil ToolLister, got %d", len(r.ToolDefs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_SkillsInline(t *testing.T) {
|
||||
r := BuildPreviewPrompt(context.Background(), baseAgent(), PromptFull, "", PreviewDeps{
|
||||
SkillsLoader: &mockSkillsLoader{
|
||||
summary: "<available_skills>\n<skill name=\"git\">Git operations</skill>\n</available_skills>",
|
||||
},
|
||||
})
|
||||
if !strings.Contains(r.Prompt, "<available_skills>") {
|
||||
t.Error("expected skills XML inlined in prompt")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_SkillsSearchMode(t *testing.T) {
|
||||
bigSummary := strings.Repeat("x", 10000)
|
||||
r := BuildPreviewPrompt(context.Background(), baseAgent(), PromptFull, "", PreviewDeps{
|
||||
SkillsLoader: &mockSkillsLoader{summary: bigSummary},
|
||||
})
|
||||
if strings.Contains(r.Prompt, bigSummary) {
|
||||
t.Error("expected large summary to be excluded (search-only mode)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_PinnedSkillsHybrid(t *testing.T) {
|
||||
ag := baseAgent()
|
||||
ag.OtherConfig = []byte(`{"pinned_skills":["deploy"]}`)
|
||||
r := BuildPreviewPrompt(context.Background(), ag, PromptFull, "", PreviewDeps{
|
||||
SkillsLoader: &mockSkillsLoader{
|
||||
pinned: "<skill name=\"deploy\">Deploy to prod</skill>",
|
||||
summary: "<available_skills>\n<skill name=\"git\">Git ops</skill>\n</available_skills>",
|
||||
},
|
||||
})
|
||||
if !strings.Contains(r.Prompt, "deploy") || !strings.Contains(r.Prompt, "Pinned skills") {
|
||||
t.Error("expected pinned skills section in prompt")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_SkillAllowList(t *testing.T) {
|
||||
ag := baseAgent()
|
||||
loader := &mockSkillsLoader{
|
||||
summary: "<available_skills><skill name=\"allowed\">ok</skill></available_skills>",
|
||||
}
|
||||
r := BuildPreviewPrompt(context.Background(), ag, PromptFull, "user1", PreviewDeps{
|
||||
SkillsLoader: loader,
|
||||
SkillAccessStore: &mockSkillAccessStore{
|
||||
accessible: []store.SkillInfo{{Slug: "allowed-skill"}},
|
||||
},
|
||||
})
|
||||
if !strings.Contains(r.Prompt, "<available_skills>") {
|
||||
t.Error("expected filtered skills in prompt")
|
||||
}
|
||||
if len(loader.capturedAllow) != 1 || loader.capturedAllow[0] != "allowed-skill" {
|
||||
t.Errorf("expected allow list [allowed-skill], got %v", loader.capturedAllow)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_SkillAccessStoreError(t *testing.T) {
|
||||
ag := baseAgent()
|
||||
loader := &mockSkillsLoader{
|
||||
summary: "<available_skills><skill name=\"s\">desc</skill></available_skills>",
|
||||
}
|
||||
r := BuildPreviewPrompt(context.Background(), ag, PromptFull, "user1", PreviewDeps{
|
||||
SkillsLoader: loader,
|
||||
SkillAccessStore: &mockSkillAccessStore{err: errors.New("db error")},
|
||||
})
|
||||
if r.Prompt == "" {
|
||||
t.Fatal("expected non-empty prompt on SkillAccessStore error")
|
||||
}
|
||||
if loader.capturedAllow == nil || len(loader.capturedAllow) != 0 {
|
||||
t.Errorf("expected empty (non-nil) allow list on error, got %v", loader.capturedAllow)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_MCPToolDescs(t *testing.T) {
|
||||
r := BuildPreviewPrompt(context.Background(), baseAgent(), PromptFull, "", PreviewDeps{
|
||||
ToolLister: &mockToolLister{
|
||||
tools: map[string]string{
|
||||
"read_file": "Read a file",
|
||||
"mcp_pg_query": "Run PostgreSQL queries",
|
||||
},
|
||||
},
|
||||
})
|
||||
if !strings.Contains(r.Prompt, "mcp_pg_query") {
|
||||
t.Error("expected MCP tool description in prompt")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_MCPToolSearchExcluded(t *testing.T) {
|
||||
r := BuildPreviewPrompt(context.Background(), baseAgent(), PromptFull, "", PreviewDeps{
|
||||
ToolLister: &mockToolLister{
|
||||
tools: map[string]string{
|
||||
"read_file": "Read a file",
|
||||
"mcp_tool_search": "Search MCP tools",
|
||||
},
|
||||
},
|
||||
})
|
||||
if strings.Contains(r.Prompt, "Search MCP tools") {
|
||||
t.Error("mcp_tool_search should not appear in MCP tool descriptions")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_AliasExclusion(t *testing.T) {
|
||||
r := BuildPreviewPrompt(context.Background(), baseAgent(), PromptFull, "", PreviewDeps{
|
||||
ToolLister: &mockToolLister{
|
||||
tools: map[string]string{
|
||||
"read_file": "Read a file",
|
||||
"Read": "Alias for read_file",
|
||||
"exec": "Execute shell",
|
||||
"Bash": "Alias for exec",
|
||||
},
|
||||
aliases: map[string]string{
|
||||
"Read": "read_file",
|
||||
"Bash": "exec",
|
||||
},
|
||||
},
|
||||
})
|
||||
if strings.Contains(r.Prompt, "- Read\n") || strings.Contains(r.Prompt, "- Bash\n") {
|
||||
t.Error("aliases should be excluded from tool list")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_SkillManageGating(t *testing.T) {
|
||||
ag := baseAgent()
|
||||
r := BuildPreviewPrompt(context.Background(), ag, PromptFull, "", PreviewDeps{
|
||||
ToolLister: &mockToolLister{
|
||||
tools: map[string]string{
|
||||
"read_file": "Read a file",
|
||||
"skill_manage": "Manage skills",
|
||||
},
|
||||
},
|
||||
})
|
||||
if strings.Contains(r.Prompt, "skill_manage") {
|
||||
t.Error("skill_manage should be excluded when skill_evolve is off")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_SkillManageEnabled(t *testing.T) {
|
||||
ag := baseAgent()
|
||||
ag.SkillEvolve = true
|
||||
r := BuildPreviewPrompt(context.Background(), ag, PromptFull, "", PreviewDeps{
|
||||
ToolLister: &mockToolLister{
|
||||
tools: map[string]string{
|
||||
"read_file": "Read a file",
|
||||
"skill_manage": "Manage skills",
|
||||
"skill_search": "Search skills",
|
||||
},
|
||||
},
|
||||
})
|
||||
if !strings.Contains(r.Prompt, "skill_manage") {
|
||||
t.Error("skill_manage should be present when skill_evolve is on")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_ToolPolicyDeny(t *testing.T) {
|
||||
ag := baseAgent()
|
||||
ag.ToolsConfig = []byte(`{"deny":["exec","web_fetch"]}`)
|
||||
r := BuildPreviewPrompt(context.Background(), ag, PromptFull, "", PreviewDeps{
|
||||
ToolLister: &mockToolLister{
|
||||
tools: map[string]string{
|
||||
"read_file": "Read a file",
|
||||
"exec": "Execute shell",
|
||||
"web_fetch": "Fetch web page",
|
||||
},
|
||||
},
|
||||
})
|
||||
if strings.Contains(r.Prompt, "- exec\n") {
|
||||
t.Error("denied tool 'exec' should be excluded")
|
||||
}
|
||||
if strings.Contains(r.Prompt, "- web_fetch\n") {
|
||||
t.Error("denied tool 'web_fetch' should be excluded")
|
||||
}
|
||||
if !strings.Contains(r.Prompt, "read_file") {
|
||||
t.Error("non-denied tool 'read_file' should be present")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPreviewPrompt_ToolDefs(t *testing.T) {
|
||||
r := BuildPreviewPrompt(context.Background(), baseAgent(), PromptFull, "", PreviewDeps{
|
||||
ToolLister: &mockToolLister{
|
||||
tools: map[string]string{
|
||||
"read_file": "Read a file",
|
||||
"exec": "Execute shell",
|
||||
},
|
||||
aliases: map[string]string{
|
||||
"Read": "read_file",
|
||||
},
|
||||
},
|
||||
})
|
||||
// Should have canonical tools + aliases in tool defs
|
||||
if len(r.ToolDefs) != 3 { // read_file + exec + Read alias
|
||||
t.Errorf("expected 3 tool defs (2 canonical + 1 alias), got %d", len(r.ToolDefs))
|
||||
}
|
||||
// Verify alias is included in defs even though excluded from system prompt
|
||||
found := false
|
||||
for _, td := range r.ToolDefs {
|
||||
if td.Function.Name == "Read" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("expected alias 'Read' in tool defs")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
|
||||
"github.com/google/uuid"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tools"
|
||||
)
|
||||
|
||||
// mockToolLister implements the widened ToolLister interface for testing.
|
||||
type mockToolLister struct {
|
||||
tools map[string]string // name → description
|
||||
aliases map[string]string // alias → canonical
|
||||
}
|
||||
|
||||
func (m *mockToolLister) List() []string {
|
||||
names := make([]string, 0, len(m.tools))
|
||||
for n := range m.tools {
|
||||
names = append(names, n)
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
|
||||
func (m *mockToolLister) Get(name string) (tools.Tool, bool) {
|
||||
desc, ok := m.tools[name]
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
return &mockTool{name: name, desc: desc}, true
|
||||
}
|
||||
|
||||
func (m *mockToolLister) Aliases() map[string]string {
|
||||
if m.aliases == nil {
|
||||
return nil
|
||||
}
|
||||
return m.aliases
|
||||
}
|
||||
|
||||
// mockTool is a minimal tools.Tool implementation for testing.
|
||||
type mockTool struct {
|
||||
name string
|
||||
desc string
|
||||
}
|
||||
|
||||
func (t *mockTool) Name() string { return t.name }
|
||||
func (t *mockTool) Description() string { return t.desc }
|
||||
func (t *mockTool) Parameters() map[string]any { return nil }
|
||||
func (t *mockTool) Execute(_ context.Context, _ map[string]any) *tools.Result { return nil }
|
||||
|
||||
// mockSkillsLoader implements the widened SkillsLoader interface.
|
||||
type mockSkillsLoader struct {
|
||||
pinned string // pre-built pinned XML
|
||||
summary string // pre-built full summary
|
||||
capturedAllow []string // set by BuildSummary for test assertions
|
||||
}
|
||||
|
||||
func (m *mockSkillsLoader) BuildPinnedSummary(_ context.Context, _ []string) string {
|
||||
return m.pinned
|
||||
}
|
||||
|
||||
func (m *mockSkillsLoader) BuildSummary(_ context.Context, allowList []string) string {
|
||||
m.capturedAllow = allowList
|
||||
return m.summary
|
||||
}
|
||||
|
||||
// mockSkillAccessStore returns canned skill access lists.
|
||||
type mockSkillAccessStore struct {
|
||||
accessible []store.SkillInfo
|
||||
err error
|
||||
}
|
||||
|
||||
func (m *mockSkillAccessStore) ListAccessible(_ context.Context, _ uuid.UUID, _ string) ([]store.SkillInfo, error) {
|
||||
return m.accessible, m.err
|
||||
}
|
||||
@@ -30,6 +30,7 @@ type ResolverDeps struct {
|
||||
AgentStore store.AgentStore
|
||||
ProviderStore store.ProviderStore
|
||||
ProviderReg *providers.Registry
|
||||
ModelRegistry providers.ModelRegistry // per-model context window + capabilities lookup
|
||||
Bus bus.EventPublisher
|
||||
Sessions store.SessionStore
|
||||
Tools *tools.Registry
|
||||
@@ -45,6 +46,7 @@ type ResolverDeps struct {
|
||||
ContextFileLoader ContextFileLoaderFunc
|
||||
BootstrapCleanup BootstrapCleanupFunc
|
||||
CacheInvalidate CacheInvalidateFunc
|
||||
DefaultTimezone string // system default timezone for bootstrap pre-fill
|
||||
|
||||
// Security
|
||||
InjectionAction string // "log", "warn", "block", "off"
|
||||
@@ -115,6 +117,9 @@ type ResolverDeps struct {
|
||||
|
||||
// V3 domain event bus for consolidation pipeline (nil = disabled)
|
||||
DomainBus eventbus.DomainEventBus
|
||||
|
||||
// Vault hook: called when a text file is uploaded by user (nil = no vault registration)
|
||||
OnTextUploaded func(ctx context.Context, path, content string)
|
||||
}
|
||||
|
||||
// NewManagedResolver creates a ResolverFunc that builds Loops from DB agent data.
|
||||
@@ -411,6 +416,7 @@ func NewManagedResolver(deps ResolverDeps) ResolverFunc {
|
||||
AutoInjector: deps.AutoInjector,
|
||||
Provider: provider,
|
||||
Model: ag.Model,
|
||||
ModelRegistry: deps.ModelRegistry,
|
||||
ContextWindow: contextWindow,
|
||||
MaxTokens: ag.ParseMaxTokens(),
|
||||
MaxIterations: maxIter,
|
||||
@@ -435,6 +441,7 @@ func NewManagedResolver(deps ResolverDeps) ResolverFunc {
|
||||
ContextFileLoader: deps.ContextFileLoader,
|
||||
BootstrapCleanup: deps.BootstrapCleanup,
|
||||
CacheInvalidate: deps.CacheInvalidate,
|
||||
DefaultTimezone: deps.DefaultTimezone,
|
||||
OnEvent: deps.OnEvent,
|
||||
TraceCollector: deps.TraceCollector,
|
||||
InjectionAction: deps.InjectionAction,
|
||||
@@ -457,6 +464,7 @@ func NewManagedResolver(deps ResolverDeps) ResolverFunc {
|
||||
ConfigPermStore: deps.ConfigPermStore,
|
||||
TeamStore: deps.TeamStore,
|
||||
SecureCLIStore: deps.SecureCLIStore,
|
||||
OnTextUploaded: deps.OnTextUploaded,
|
||||
MediaStore: deps.MediaStore,
|
||||
ModelPricing: deps.ModelPricing,
|
||||
BudgetMonthlyCents: derefInt(ag.BudgetMonthlyCents),
|
||||
|
||||
@@ -45,6 +45,30 @@ func NewSuggestionEngine(metrics store.EvolutionMetricsStore, suggestions store.
|
||||
}
|
||||
}
|
||||
|
||||
// dedupKey uniquely identifies a suggestion by type + metric key (e.g., tool name or source).
|
||||
type dedupKey struct {
|
||||
Type store.SuggestionType
|
||||
Key string
|
||||
}
|
||||
|
||||
// extractMetricKey returns the distinguishing key from suggestion parameters.
|
||||
func extractMetricKey(params json.RawMessage, st store.SuggestionType) string {
|
||||
var p map[string]any
|
||||
if json.Unmarshal(params, &p) != nil {
|
||||
return ""
|
||||
}
|
||||
switch st {
|
||||
case store.SuggestThreshold:
|
||||
s, _ := p["source"].(string)
|
||||
return s
|
||||
case store.SuggestToolOrder, store.SuggestSkillAdd:
|
||||
s, _ := p["tool"].(string)
|
||||
return s
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// Analyze runs all rules against a single agent's metrics (7-day window).
|
||||
// Returns newly created suggestions. Skips rules that produce duplicates.
|
||||
func (e *SuggestionEngine) Analyze(ctx context.Context, agentID uuid.UUID) ([]store.EvolutionSuggestion, error) {
|
||||
@@ -65,11 +89,12 @@ func (e *SuggestionEngine) Analyze(ctx context.Context, agentID uuid.UUID) ([]st
|
||||
Since: since,
|
||||
}
|
||||
|
||||
// Load existing pending suggestions to avoid duplicates.
|
||||
// Load existing pending suggestions to avoid duplicates (composite key: type + metric key).
|
||||
existing, _ := e.suggestions.ListSuggestions(ctx, agentID, "pending", 100)
|
||||
existingTypes := make(map[store.SuggestionType]bool, len(existing))
|
||||
existingKeys := make(map[dedupKey]bool, len(existing))
|
||||
for _, sg := range existing {
|
||||
existingTypes[sg.SuggestionType] = true
|
||||
mk := extractMetricKey(sg.Parameters, sg.SuggestionType)
|
||||
existingKeys[dedupKey{sg.SuggestionType, mk}] = true
|
||||
}
|
||||
|
||||
var created []store.EvolutionSuggestion
|
||||
@@ -82,8 +107,9 @@ func (e *SuggestionEngine) Analyze(ctx context.Context, agentID uuid.UUID) ([]st
|
||||
if sg == nil {
|
||||
continue
|
||||
}
|
||||
// Skip if pending suggestion of same type already exists.
|
||||
if existingTypes[sg.SuggestionType] {
|
||||
// Skip if pending suggestion with same type + metric key already exists.
|
||||
mk := extractMetricKey(sg.Parameters, sg.SuggestionType)
|
||||
if existingKeys[dedupKey{sg.SuggestionType, mk}] {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -95,7 +121,7 @@ func (e *SuggestionEngine) Analyze(ctx context.Context, agentID uuid.UUID) ([]st
|
||||
continue
|
||||
}
|
||||
created = append(created, *sg)
|
||||
existingTypes[sg.SuggestionType] = true
|
||||
existingKeys[dedupKey{sg.SuggestionType, mk}] = true
|
||||
}
|
||||
|
||||
return created, nil
|
||||
|
||||
@@ -205,6 +205,9 @@ var coreToolSummaries = map[string]string{
|
||||
"team_tasks": "Team task board — track progress, manage dependencies (spawn auto-creates delegation tasks)",
|
||||
"list_group_members": "List all members of the current group chat (Feishu/Lark only)",
|
||||
"create_forum_topic": "Create a forum topic in a Telegram supergroup",
|
||||
"delegate": "Delegate a task to a linked agent (requires agent_links). See ## Delegation Targets for available agents",
|
||||
"memory_expand": "Retrieve full session details from episodic memory results — use after memory_search returns episodic hits",
|
||||
"vault_search": "Search documents in the knowledge vault (hybrid keyword + semantic)",
|
||||
|
||||
// Tool aliases (edit_file, sessions_spawn, Read, Write, Edit, Bash, etc.)
|
||||
// are registered in the tool registry but excluded from the system prompt
|
||||
|
||||
@@ -2,6 +2,7 @@ package bootstrap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -9,9 +10,47 @@ import (
|
||||
|
||||
"github.com/google/uuid"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// ChannelMeta carries channel-provided contact info for bootstrap decisions.
|
||||
type ChannelMeta struct {
|
||||
ChannelType string
|
||||
DisplayName string
|
||||
DefaultTimezone string
|
||||
}
|
||||
|
||||
// shouldSkipBootstrap returns true when channel provides enough user info
|
||||
// to pre-fill USER.md, making the interactive bootstrap unnecessary.
|
||||
// Currently only Pancake channel qualifies (provides Facebook profile name).
|
||||
func shouldSkipBootstrap(meta *ChannelMeta) bool {
|
||||
return meta != nil &&
|
||||
meta.ChannelType == "pancake" &&
|
||||
meta.DisplayName != ""
|
||||
}
|
||||
|
||||
// buildPrefilledUser generates USER.md content pre-filled with channel-provided contact info.
|
||||
func buildPrefilledUser(meta *ChannelMeta) string {
|
||||
tz := meta.DefaultTimezone
|
||||
if tz == "" {
|
||||
tz = "(unknown)"
|
||||
}
|
||||
name := channels.SanitizeDisplayName(meta.DisplayName)
|
||||
return fmt.Sprintf(`# USER.md - About This User
|
||||
|
||||
- **Name:** %s
|
||||
- **What to call them:** %s
|
||||
- **Timezone:** %s
|
||||
|
||||
## Context
|
||||
|
||||
_(First contact via %s channel. Profile info auto-filled from channel data.)_
|
||||
`, name, name, tz, meta.ChannelType)
|
||||
}
|
||||
|
||||
|
||||
|
||||
// retryOnBusy retries fn up to 3 times on SQLITE_BUSY errors with 500ms delay.
|
||||
func retryOnBusy(fn func() error) error {
|
||||
var lastErr error
|
||||
@@ -140,7 +179,7 @@ var userSeedFilesPredefined = []string{
|
||||
// This ensures wizard-configured owner profiles are preserved on first chat.
|
||||
//
|
||||
// Returns the list of file names that were seeded.
|
||||
func SeedUserFiles(ctx context.Context, agentStore store.AgentStore, agentID uuid.UUID, userID, agentType string, skipIfAnyExist bool) ([]string, error) {
|
||||
func SeedUserFiles(ctx context.Context, agentStore store.AgentStore, agentID uuid.UUID, userID, agentType string, skipIfAnyExist bool, channelMeta *ChannelMeta) ([]string, error) {
|
||||
files := userSeedFilesOpen
|
||||
if agentType == store.AgentTypePredefined {
|
||||
files = userSeedFilesPredefined
|
||||
@@ -160,6 +199,44 @@ func SeedUserFiles(ctx context.Context, agentStore store.AgentStore, agentID uui
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Channel-provided contact info: skip bootstrap, pre-fill USER.md directly.
|
||||
// Currently only Pancake channel (Facebook Messenger) provides enough user info
|
||||
// (display_name from Facebook profile) to skip the interactive onboarding flow.
|
||||
if shouldSkipBootstrap(channelMeta) {
|
||||
userContent := buildPrefilledUser(channelMeta)
|
||||
if err := retryOnBusy(func() error {
|
||||
return agentStore.SetUserContextFile(ctx, agentID, userID, UserFile, userContent)
|
||||
}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
seeded := []string{UserFile}
|
||||
|
||||
// For open agents: still seed SOUL.md, IDENTITY.md, AGENTS.md (default templates)
|
||||
// so the agent has its personality/identity files — just skip BOOTSTRAP.md.
|
||||
if agentType == store.AgentTypeOpen {
|
||||
for _, name := range userSeedFilesOpen {
|
||||
if name == BootstrapFile || name == UserFile {
|
||||
continue // skip bootstrap (the whole point) and user (already filled)
|
||||
}
|
||||
content, err := templateFS.ReadFile(filepath.Join("templates", name))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if err := retryOnBusy(func() error {
|
||||
return agentStore.SetUserContextFile(ctx, agentID, userID, name, string(content))
|
||||
}); err != nil {
|
||||
return seeded, err
|
||||
}
|
||||
seeded = append(seeded, name)
|
||||
}
|
||||
}
|
||||
|
||||
slog.Info("bootstrap skipped (channel contact info)",
|
||||
"agent", agentID, "user", userID, "channel", channelMeta.ChannelType,
|
||||
"display_name", channelMeta.DisplayName, "files", seeded)
|
||||
return seeded, nil
|
||||
}
|
||||
|
||||
hasFile := make(map[string]bool, len(existing))
|
||||
for _, f := range existing {
|
||||
if f.Content != "" {
|
||||
|
||||
@@ -3,6 +3,7 @@ package bootstrap
|
||||
import (
|
||||
"context"
|
||||
"maps"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
@@ -116,6 +117,23 @@ func (s *seedStubStore) PropagateContextFile(_ context.Context, _ uuid.UUID, _ s
|
||||
}
|
||||
// ---- Tests ----
|
||||
|
||||
// TestBuildPrefilledUser_SanitizesMarkdownInjection verifies that DisplayName with
|
||||
// newlines or markdown syntax does not inject into USER.md structure.
|
||||
func TestBuildPrefilledUser_SanitizesMarkdownInjection(t *testing.T) {
|
||||
meta := &ChannelMeta{
|
||||
ChannelType: "pancake",
|
||||
DisplayName: "Evil\n- **Admin:** true\n## Override",
|
||||
DefaultTimezone: "Asia/Ho_Chi_Minh",
|
||||
}
|
||||
content := buildPrefilledUser(meta)
|
||||
if strings.Contains(content, "## Override") {
|
||||
t.Error("DisplayName markdown injection not sanitized")
|
||||
}
|
||||
if strings.Contains(content, "\n- **Admin:**") {
|
||||
t.Error("DisplayName newline injection not sanitized")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSeedUserFiles_PredefinedAgent_UsesAgentLevelUserMD is the primary regression test.
|
||||
// When a predefined agent has wizard-written USER.md in agent_context_files, SeedUserFiles
|
||||
// must seed that content into user_context_files — NOT the blank embedded template.
|
||||
@@ -127,7 +145,7 @@ func TestSeedUserFiles_PredefinedAgent_UsesAgentLevelUserMD(t *testing.T) {
|
||||
// Simulate wizard writing USER.md at agent level via agents.files.set
|
||||
as.agentFiles[UserFile] = wizardContent
|
||||
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-alice", store.AgentTypePredefined, false)
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-alice", store.AgentTypePredefined, false, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("SeedUserFiles returned error: %v", err)
|
||||
}
|
||||
@@ -160,7 +178,7 @@ func TestSeedUserFiles_PredefinedAgent_FallsBackToTemplateWhenNoAgentLevelUserMD
|
||||
agentID := uuid.New()
|
||||
// No agent-level USER.md — wizard did not write one
|
||||
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-bob", store.AgentTypePredefined, false)
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-bob", store.AgentTypePredefined, false, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("SeedUserFiles returned error: %v", err)
|
||||
}
|
||||
@@ -198,7 +216,7 @@ func TestSeedUserFiles_PredefinedAgent_DoesNotOverwriteExistingPerUserContent(t
|
||||
// Also set wizard content at agent level
|
||||
as.agentFiles[UserFile] = "wizard content that should NOT override personal content"
|
||||
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-charlie", store.AgentTypePredefined, false)
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-charlie", store.AgentTypePredefined, false, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("SeedUserFiles returned error: %v", err)
|
||||
}
|
||||
@@ -223,7 +241,7 @@ func TestSeedUserFiles_OpenAgent_UsesEmbeddedTemplate(t *testing.T) {
|
||||
agentID := uuid.New()
|
||||
// Open agents should never check agent_context_files for USER.md
|
||||
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-dave", store.AgentTypeOpen, false)
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-dave", store.AgentTypeOpen, false, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("SeedUserFiles returned error: %v", err)
|
||||
}
|
||||
@@ -256,14 +274,14 @@ func TestSeedUserFiles_IdempotentOnSecondCall(t *testing.T) {
|
||||
agentID := uuid.New()
|
||||
|
||||
// First call — seeds files
|
||||
SeedUserFiles(context.Background(), as, agentID, "user-frank", store.AgentTypePredefined, false)
|
||||
SeedUserFiles(context.Background(), as, agentID, "user-frank", store.AgentTypePredefined, false, nil)
|
||||
|
||||
// Simulate what the first call wrote (move seededUserFiles → userFiles)
|
||||
maps.Copy(as.userFiles, as.seededUserFiles)
|
||||
as.seededUserFiles = make(map[string]string)
|
||||
|
||||
// Second call — must seed nothing (all files already exist)
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-frank", store.AgentTypePredefined, false)
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-frank", store.AgentTypePredefined, false, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("second SeedUserFiles returned error: %v", err)
|
||||
}
|
||||
@@ -284,7 +302,7 @@ func TestSeedUserFiles_SkipIfAnyExist(t *testing.T) {
|
||||
agentID := uuid.New()
|
||||
|
||||
// Step 1: Seed initial files (new user, skipIfAnyExist=false)
|
||||
SeedUserFiles(context.Background(), as, agentID, "user-eve", store.AgentTypePredefined, false)
|
||||
SeedUserFiles(context.Background(), as, agentID, "user-eve", store.AgentTypePredefined, false, nil)
|
||||
maps.Copy(as.userFiles, as.seededUserFiles)
|
||||
as.seededUserFiles = make(map[string]string)
|
||||
|
||||
@@ -292,7 +310,7 @@ func TestSeedUserFiles_SkipIfAnyExist(t *testing.T) {
|
||||
delete(as.userFiles, BootstrapFile)
|
||||
|
||||
// Step 3: skipIfAnyExist=true (existing profile) → must NOT re-seed
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-eve", store.AgentTypePredefined, true)
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-eve", store.AgentTypePredefined, true, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("SeedUserFiles(skipIfAnyExist=true) returned error: %v", err)
|
||||
}
|
||||
@@ -301,7 +319,7 @@ func TestSeedUserFiles_SkipIfAnyExist(t *testing.T) {
|
||||
}
|
||||
|
||||
// Step 4: skipIfAnyExist=false (force) → SHOULD re-seed missing BOOTSTRAP.md
|
||||
seeded, err = SeedUserFiles(context.Background(), as, agentID, "user-eve", store.AgentTypePredefined, false)
|
||||
seeded, err = SeedUserFiles(context.Background(), as, agentID, "user-eve", store.AgentTypePredefined, false, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("SeedUserFiles(skipIfAnyExist=false) returned error: %v", err)
|
||||
}
|
||||
@@ -323,7 +341,7 @@ func TestSeedUserFiles_SkipIfAnyExist_EmptyUser(t *testing.T) {
|
||||
agentID := uuid.New()
|
||||
|
||||
// No existing user files → skipIfAnyExist=true should still seed
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-ghost", store.AgentTypePredefined, true)
|
||||
seeded, err := SeedUserFiles(context.Background(), as, agentID, "user-ghost", store.AgentTypePredefined, true, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("SeedUserFiles returned error: %v", err)
|
||||
}
|
||||
|
||||
Vendored
+127
-7
@@ -2,29 +2,63 @@ package cache
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// entry wraps a cached value with expiration metadata.
|
||||
// entry wraps a cached value with expiration and creation metadata.
|
||||
// createdAt is used for oldest-first eviction when maxSize is exceeded.
|
||||
type entry[V any] struct {
|
||||
value V
|
||||
expiresAt time.Time // zero means no expiry
|
||||
createdAt time.Time // set on Set(), used for eviction ordering
|
||||
}
|
||||
|
||||
func (e entry[V]) expired() bool {
|
||||
return !e.expiresAt.IsZero() && time.Now().After(e.expiresAt)
|
||||
}
|
||||
|
||||
// InMemoryCache is a thread-safe in-memory Cache implementation with TTL support.
|
||||
// InMemoryCache is a thread-safe in-memory Cache implementation with TTL support,
|
||||
// periodic sweep goroutine for expired entries, and optional max-size cap with
|
||||
// oldest-first eviction.
|
||||
type InMemoryCache[V any] struct {
|
||||
data sync.Map
|
||||
data sync.Map
|
||||
maxSize int // 0 = unlimited
|
||||
sweepInterval time.Duration // 0 = no periodic sweep (lazy eviction only)
|
||||
cancel context.CancelFunc
|
||||
closeOnce sync.Once
|
||||
}
|
||||
|
||||
// NewInMemoryCache creates a new in-memory cache.
|
||||
func NewInMemoryCache[V any]() *InMemoryCache[V] {
|
||||
return &InMemoryCache[V]{}
|
||||
// CacheOption configures an InMemoryCache during construction.
|
||||
type CacheOption[V any] func(*InMemoryCache[V])
|
||||
|
||||
// WithMaxSize sets a maximum entry count. When exceeded during sweep, the
|
||||
// oldest 20% of entries are evicted. Zero = unlimited.
|
||||
func WithMaxSize[V any](n int) CacheOption[V] {
|
||||
return func(c *InMemoryCache[V]) { c.maxSize = n }
|
||||
}
|
||||
|
||||
// WithSweepInterval sets the periodic sweep interval for expired entries.
|
||||
// Zero disables the sweep goroutine (lazy eviction on Get only).
|
||||
func WithSweepInterval[V any](d time.Duration) CacheOption[V] {
|
||||
return func(c *InMemoryCache[V]) { c.sweepInterval = d }
|
||||
}
|
||||
|
||||
// NewInMemoryCache creates a new in-memory cache. Without options it behaves
|
||||
// exactly as before (lazy eviction, no size cap, no sweep goroutine).
|
||||
func NewInMemoryCache[V any](opts ...CacheOption[V]) *InMemoryCache[V] {
|
||||
c := &InMemoryCache[V]{}
|
||||
for _, opt := range opts {
|
||||
opt(c)
|
||||
}
|
||||
if c.sweepInterval > 0 {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
c.cancel = cancel
|
||||
go c.sweepLoop(ctx)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func (c *InMemoryCache[V]) Get(_ context.Context, key string) (V, bool) {
|
||||
@@ -43,7 +77,7 @@ func (c *InMemoryCache[V]) Get(_ context.Context, key string) (V, bool) {
|
||||
}
|
||||
|
||||
func (c *InMemoryCache[V]) Set(_ context.Context, key string, value V, ttl time.Duration) {
|
||||
e := entry[V]{value: value}
|
||||
e := entry[V]{value: value, createdAt: time.Now()}
|
||||
if ttl > 0 {
|
||||
e.expiresAt = time.Now().Add(ttl)
|
||||
}
|
||||
@@ -69,3 +103,89 @@ func (c *InMemoryCache[V]) Clear(_ context.Context) {
|
||||
return true
|
||||
})
|
||||
}
|
||||
|
||||
// Close stops the background sweep goroutine. Safe to call multiple times.
|
||||
// After Close, the cache remains readable/writable but without periodic sweep.
|
||||
func (c *InMemoryCache[V]) Close() {
|
||||
c.closeOnce.Do(func() {
|
||||
if c.cancel != nil {
|
||||
c.cancel()
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// sweepLoop runs the periodic expiry + size-cap cleanup in a background goroutine.
|
||||
func (c *InMemoryCache[V]) sweepLoop(ctx context.Context) {
|
||||
ticker := time.NewTicker(c.sweepInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
c.sweepOnce()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sweepOnce deletes expired entries and, if maxSize is set and exceeded,
|
||||
// evicts the oldest 20% of remaining entries (by createdAt).
|
||||
func (c *InMemoryCache[V]) sweepOnce() {
|
||||
// Phase 1: collect expired keys (don't delete during Range iteration).
|
||||
// sync.Map's Range is safe with concurrent Store/Delete, but deleting
|
||||
// during iteration can cause the same key to be visited twice in
|
||||
// pathological cases. Collecting first is cleaner.
|
||||
var expiredKeys []string
|
||||
type keyAge struct {
|
||||
key string
|
||||
createdAt time.Time
|
||||
}
|
||||
var allAlive []keyAge
|
||||
|
||||
c.data.Range(func(k, v any) bool {
|
||||
key, ok := k.(string)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
e, ok := v.(entry[V])
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if e.expired() {
|
||||
expiredKeys = append(expiredKeys, key)
|
||||
} else {
|
||||
allAlive = append(allAlive, keyAge{key: key, createdAt: e.createdAt})
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
for _, k := range expiredKeys {
|
||||
c.data.Delete(k)
|
||||
}
|
||||
|
||||
// Phase 2: size-cap eviction. Sort alive entries by createdAt ascending,
|
||||
// evict oldest 20% if over maxSize.
|
||||
if c.maxSize > 0 && len(allAlive) > c.maxSize {
|
||||
sort.Slice(allAlive, func(i, j int) bool {
|
||||
return allAlive[i].createdAt.Before(allAlive[j].createdAt)
|
||||
})
|
||||
toEvict := len(allAlive) - c.maxSize + (c.maxSize / 5) // bring below cap + 20% headroom
|
||||
if toEvict > len(allAlive) {
|
||||
toEvict = len(allAlive)
|
||||
}
|
||||
for i := 0; i < toEvict; i++ {
|
||||
c.data.Delete(allAlive[i].key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sizeLocked returns the current entry count (for tests and metrics).
|
||||
// Named sizeLocked for historical reasons; sync.Map needs no lock.
|
||||
func (c *InMemoryCache[V]) sizeLocked() int {
|
||||
n := 0
|
||||
c.data.Range(func(_, _ any) bool {
|
||||
n++
|
||||
return true
|
||||
})
|
||||
return n
|
||||
}
|
||||
Vendored
+104
@@ -90,3 +90,107 @@ func TestInMemoryCache_Clear(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestInMemoryCache_PeriodicSweep verifies expired entries are removed by
|
||||
// the background sweep goroutine (not just lazy on Get).
|
||||
func TestInMemoryCache_PeriodicSweep(t *testing.T) {
|
||||
c := NewInMemoryCache[string](
|
||||
WithSweepInterval[string](20*time.Millisecond),
|
||||
)
|
||||
defer c.Close()
|
||||
ctx := context.Background()
|
||||
|
||||
c.Set(ctx, "short", "v1", 10*time.Millisecond)
|
||||
c.Set(ctx, "long", "v2", 0) // no expiry
|
||||
|
||||
// Wait for sweep to run at least once after short entry expires
|
||||
time.Sleep(60 * time.Millisecond)
|
||||
|
||||
// Short entry should be removed by sweep even without Get
|
||||
if count := c.sizeLocked(); count != 1 {
|
||||
t.Errorf("expected 1 entry after sweep, got %d", count)
|
||||
}
|
||||
if _, ok := c.Get(ctx, "long"); !ok {
|
||||
t.Error("long-lived entry should still exist")
|
||||
}
|
||||
}
|
||||
|
||||
// TestInMemoryCache_MaxSizeEviction verifies oldest entries are evicted when
|
||||
// max size cap is reached.
|
||||
func TestInMemoryCache_MaxSizeEviction(t *testing.T) {
|
||||
c := NewInMemoryCache[int](
|
||||
WithMaxSize[int](5),
|
||||
WithSweepInterval[int](10*time.Millisecond),
|
||||
)
|
||||
defer c.Close()
|
||||
ctx := context.Background()
|
||||
|
||||
// Insert 10 entries with distinct creation times to ensure oldest-first ordering
|
||||
for i := 0; i < 10; i++ {
|
||||
c.Set(ctx, string(rune('a'+i)), i, 0)
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
}
|
||||
|
||||
// Trigger sweep by waiting for interval
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
|
||||
if count := c.sizeLocked(); count > 5 {
|
||||
t.Errorf("expected size ≤ 5 after max-size eviction, got %d", count)
|
||||
}
|
||||
}
|
||||
|
||||
// TestInMemoryCache_Close verifies Close stops the sweep goroutine (no leak).
|
||||
func TestInMemoryCache_Close(t *testing.T) {
|
||||
c := NewInMemoryCache[string](
|
||||
WithSweepInterval[string](10*time.Millisecond),
|
||||
)
|
||||
ctx := context.Background()
|
||||
c.Set(ctx, "k", "v", 0)
|
||||
|
||||
c.Close()
|
||||
// Close should be idempotent
|
||||
c.Close()
|
||||
|
||||
// After Close, cache is still readable for lazy access but sweep is stopped
|
||||
if _, ok := c.Get(ctx, "k"); !ok {
|
||||
t.Error("Get should still work after Close")
|
||||
}
|
||||
}
|
||||
|
||||
// TestInMemoryCache_ConcurrentSweepAndSet verifies no race between sweep and Set.
|
||||
func TestInMemoryCache_ConcurrentSweepAndSet(t *testing.T) {
|
||||
c := NewInMemoryCache[int](
|
||||
WithSweepInterval[int](1*time.Millisecond),
|
||||
WithMaxSize[int](100),
|
||||
)
|
||||
defer c.Close()
|
||||
ctx := context.Background()
|
||||
|
||||
done := make(chan bool)
|
||||
go func() {
|
||||
for i := 0; i < 500; i++ {
|
||||
c.Set(ctx, string(rune('a'+(i%26))), i, 5*time.Millisecond)
|
||||
}
|
||||
done <- true
|
||||
}()
|
||||
go func() {
|
||||
for i := 0; i < 500; i++ {
|
||||
_, _ = c.Get(ctx, string(rune('a'+(i%26))))
|
||||
}
|
||||
done <- true
|
||||
}()
|
||||
<-done
|
||||
<-done
|
||||
}
|
||||
|
||||
// TestInMemoryCache_BackwardCompatZeroArg verifies existing call sites with
|
||||
// zero-arg constructor still work (variadic options).
|
||||
func TestInMemoryCache_BackwardCompatZeroArg(t *testing.T) {
|
||||
c := NewInMemoryCache[string]()
|
||||
defer c.Close()
|
||||
ctx := context.Background()
|
||||
c.Set(ctx, "k", "v", 0)
|
||||
if v, ok := c.Get(ctx, "k"); !ok || v != "v" {
|
||||
t.Errorf("backward compat broken: got %v, %v", v, ok)
|
||||
}
|
||||
}
|
||||
Vendored
+32
-4
@@ -23,15 +23,43 @@ type PermissionCache struct {
|
||||
teamAccess *InMemoryCache[bool]
|
||||
}
|
||||
|
||||
// NewPermissionCache creates a new permission cache.
|
||||
// permissionCacheSweepInterval and permissionCacheMaxSize bound background
|
||||
// growth of per-user cache entries. Without these, long-running gateways with
|
||||
// many distinct users would accumulate unbounded entries (tenant_roles, agent
|
||||
// access, team access) even with a 30s TTL — lazy eviction only fires on Get,
|
||||
// so entries for disconnected users never get reclaimed.
|
||||
const (
|
||||
permissionCacheSweepInterval = 60 * time.Second
|
||||
permissionCacheMaxSize = 10_000
|
||||
)
|
||||
|
||||
// NewPermissionCache creates a new permission cache with periodic sweep
|
||||
// goroutines for all three inner caches. Call Close() on gateway shutdown to
|
||||
// stop the sweep goroutines.
|
||||
func NewPermissionCache() *PermissionCache {
|
||||
return &PermissionCache{
|
||||
tenantRole: NewInMemoryCache[string](),
|
||||
agentAccess: NewInMemoryCache[agentAccessEntry](),
|
||||
teamAccess: NewInMemoryCache[bool](),
|
||||
tenantRole: NewInMemoryCache[string](
|
||||
WithSweepInterval[string](permissionCacheSweepInterval),
|
||||
WithMaxSize[string](permissionCacheMaxSize),
|
||||
),
|
||||
agentAccess: NewInMemoryCache[agentAccessEntry](
|
||||
WithSweepInterval[agentAccessEntry](permissionCacheSweepInterval),
|
||||
WithMaxSize[agentAccessEntry](permissionCacheMaxSize),
|
||||
),
|
||||
teamAccess: NewInMemoryCache[bool](
|
||||
WithSweepInterval[bool](permissionCacheSweepInterval),
|
||||
WithMaxSize[bool](permissionCacheMaxSize),
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
// Close stops all background sweep goroutines. Safe to call multiple times.
|
||||
func (pc *PermissionCache) Close() {
|
||||
pc.tenantRole.Close()
|
||||
pc.agentAccess.Close()
|
||||
pc.teamAccess.Close()
|
||||
}
|
||||
|
||||
const (
|
||||
tenantRoleTTL = 30 * time.Second
|
||||
agentAccessTTL = 30 * time.Second
|
||||
|
||||
@@ -70,10 +70,12 @@ const (
|
||||
|
||||
// Channel type constants used across channel packages and gateway wiring.
|
||||
const (
|
||||
TypeTelegram = "telegram"
|
||||
TypeDiscord = "discord"
|
||||
TypeSlack = "slack"
|
||||
TypeFacebook = "facebook"
|
||||
TypeFeishu = "feishu"
|
||||
TypePancake = "pancake"
|
||||
TypeSlack = "slack"
|
||||
TypeTelegram = "telegram"
|
||||
TypeWhatsApp = "whatsapp"
|
||||
TypeZaloOA = "zalo_oa"
|
||||
TypeZaloPersonal = "zalo_personal"
|
||||
|
||||
@@ -264,7 +264,7 @@ func (c *Channel) handleMessage(_ *discordgo.Session, m *discordgo.MessageCreate
|
||||
"message_id": m.ID,
|
||||
"user_id": senderID,
|
||||
"username": m.Author.Username,
|
||||
"display_name": senderName,
|
||||
"display_name": channels.SanitizeDisplayName(senderName),
|
||||
"guild_id": m.GuildID,
|
||||
"channel_id": channelID,
|
||||
"is_dm": fmt.Sprintf("%t", isDM),
|
||||
|
||||
@@ -71,6 +71,9 @@ func (m *Manager) dispatchOutbound(ctx context.Context) {
|
||||
if err := channel.Send(ctx, msg); err != nil {
|
||||
slog.Error("error sending message to channel",
|
||||
"channel", msg.Channel,
|
||||
"chat_id", msg.ChatID,
|
||||
"content_len", len(msg.Content),
|
||||
"content_preview", Truncate(msg.Content, 160),
|
||||
"error", err,
|
||||
)
|
||||
// Try to send a text-only error notification back to the chat.
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// handleCommentEvent processes a feed webhook change where item == "comment".
|
||||
func (ch *Channel) handleCommentEvent(entry WebhookEntry, change ChangeValue) {
|
||||
// Feature gate.
|
||||
if !ch.config.Features.CommentReply {
|
||||
return
|
||||
}
|
||||
|
||||
// Only process new comments (not edits or deletions).
|
||||
if change.Verb != "add" {
|
||||
return
|
||||
}
|
||||
|
||||
// Page routing: ensure this event belongs to our page instance (before dedup write).
|
||||
if entry.ID != ch.pageID {
|
||||
return
|
||||
}
|
||||
|
||||
// Self-reply prevention: skip comments posted by the page itself.
|
||||
if change.From.ID == ch.pageID {
|
||||
return
|
||||
}
|
||||
|
||||
// Dedup: Facebook may deliver the same event more than once.
|
||||
if ch.isDup("comment:" + change.CommentID) {
|
||||
slog.Debug("facebook: duplicate comment event skipped", "comment_id", change.CommentID)
|
||||
return
|
||||
}
|
||||
|
||||
// Build message content — optionally enriched with post + thread context.
|
||||
// Use stopCtx so inflight Graph API calls are cancelled when the channel stops.
|
||||
content := change.Message
|
||||
if ch.config.CommentReplyOptions.IncludePostContext && change.PostID != "" {
|
||||
content = ch.buildEnrichedContent(change)
|
||||
}
|
||||
|
||||
senderID := change.From.ID
|
||||
// Session key groups all comments by the same user on the same post.
|
||||
chatID := fmt.Sprintf("%s:%s", change.PostID, senderID)
|
||||
|
||||
metadata := map[string]string{
|
||||
"comment_id": change.CommentID,
|
||||
"post_id": change.PostID,
|
||||
"parent_id": change.ParentID,
|
||||
"sender_name": change.From.Name,
|
||||
"sender_id": senderID,
|
||||
"fb_mode": "comment",
|
||||
"reply_to_comment_id": change.CommentID,
|
||||
}
|
||||
|
||||
ch.HandleMessage(senderID, chatID, content, nil, metadata, "direct")
|
||||
}
|
||||
|
||||
// buildEnrichedContent fetches post content and comment thread, assembles a rich
|
||||
// context string for the agent. Uses stopCtx so it is cancelled on channel Stop().
|
||||
func (ch *Channel) buildEnrichedContent(change ChangeValue) string {
|
||||
ctx := ch.stopCtx
|
||||
|
||||
var sb strings.Builder
|
||||
|
||||
// Post content.
|
||||
post, err := ch.postFetcher.GetPost(ctx, change.PostID)
|
||||
if err == nil && post != nil && post.Message != "" {
|
||||
sb.WriteString("[Bài đăng] ")
|
||||
sb.WriteString(post.Message)
|
||||
sb.WriteString("\n\n")
|
||||
}
|
||||
|
||||
// Comment thread (when this is a nested reply).
|
||||
if change.ParentID != "" {
|
||||
depth := ch.config.CommentReplyOptions.MaxThreadDepth
|
||||
if depth <= 0 {
|
||||
depth = 10
|
||||
}
|
||||
thread, err := ch.postFetcher.GetCommentThread(ctx, change.ParentID, depth)
|
||||
if err == nil && len(thread) > 0 {
|
||||
sb.WriteString("[Thread]\n")
|
||||
for _, c := range thread {
|
||||
sb.WriteString(fmt.Sprintf("- %s: %s\n", c.From.Name, c.Message))
|
||||
}
|
||||
sb.WriteString("\n")
|
||||
}
|
||||
}
|
||||
|
||||
// Current comment.
|
||||
sb.WriteString("[Comment mới] ")
|
||||
sb.WriteString(change.From.Name)
|
||||
sb.WriteString(": ")
|
||||
sb.WriteString(change.Message)
|
||||
|
||||
return sb.String()
|
||||
}
|
||||
@@ -0,0 +1,260 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// Compile-time interface assertions.
|
||||
var (
|
||||
_ channels.Channel = (*Channel)(nil)
|
||||
_ channels.WebhookChannel = (*Channel)(nil)
|
||||
)
|
||||
|
||||
const (
|
||||
webhookPath = "/channels/facebook/webhook"
|
||||
dedupTTL = 24 * time.Hour // matches Facebook's max retry window
|
||||
dedupCleanEvery = 5 * time.Minute // how often to evict stale dedup entries
|
||||
)
|
||||
|
||||
// Channel implements channels.Channel and channels.WebhookChannel for Facebook Fanpage.
|
||||
// Supports comment auto-reply, Messenger auto-reply, and first inbox DM.
|
||||
type Channel struct {
|
||||
*channels.BaseChannel
|
||||
config facebookInstanceConfig
|
||||
graphClient *GraphClient
|
||||
webhookH *WebhookHandler
|
||||
pageID string
|
||||
|
||||
// dedup prevents processing duplicate webhook deliveries.
|
||||
// Value is the time.Time the event was first seen; entries are evicted after dedupTTL.
|
||||
dedup sync.Map // eventKey(string) → time.Time
|
||||
|
||||
// firstInboxSent tracks which senders have already received a first-inbox DM (in-memory).
|
||||
firstInboxSent sync.Map // senderID(string) → struct{}
|
||||
|
||||
// postFetcher caches post content to enrich comment context.
|
||||
postFetcher *PostFetcher
|
||||
|
||||
// stopCh + stopCtx serve complementary roles:
|
||||
// stopCh for select{} in goroutines (runDedupCleaner), stopCtx for HTTP client cancellation.
|
||||
stopCh chan struct{}
|
||||
stopCtx context.Context
|
||||
stopFn context.CancelFunc
|
||||
}
|
||||
|
||||
// New creates a Facebook channel from parsed credentials and config.
|
||||
func New(cfg facebookInstanceConfig, creds facebookCreds,
|
||||
msgBus *bus.MessageBus, _ store.PairingStore) (*Channel, error) {
|
||||
|
||||
if creds.PageAccessToken == "" {
|
||||
return nil, fmt.Errorf("facebook: page_access_token is required")
|
||||
}
|
||||
if cfg.PageID == "" {
|
||||
return nil, fmt.Errorf("facebook: page_id is required")
|
||||
}
|
||||
if creds.AppSecret == "" {
|
||||
return nil, fmt.Errorf("facebook: app_secret is required")
|
||||
}
|
||||
if creds.VerifyToken == "" {
|
||||
return nil, fmt.Errorf("facebook: verify_token is required")
|
||||
}
|
||||
|
||||
base := channels.NewBaseChannel(channels.TypeFacebook, msgBus, cfg.AllowFrom)
|
||||
|
||||
graphClient := NewGraphClient(creds.PageAccessToken, cfg.PageID)
|
||||
postFetcher := NewPostFetcher(graphClient, cfg.PostContextCacheTTL)
|
||||
|
||||
stopCtx, stopFn := context.WithCancel(context.Background())
|
||||
|
||||
ch := &Channel{
|
||||
BaseChannel: base,
|
||||
config: cfg,
|
||||
graphClient: graphClient,
|
||||
pageID: cfg.PageID,
|
||||
postFetcher: postFetcher,
|
||||
stopCh: make(chan struct{}),
|
||||
stopCtx: stopCtx,
|
||||
stopFn: stopFn,
|
||||
}
|
||||
|
||||
wh := NewWebhookHandler(creds.AppSecret, creds.VerifyToken)
|
||||
wh.onComment = ch.handleCommentEvent
|
||||
wh.onMessage = ch.handleMessagingEvent
|
||||
ch.webhookH = wh
|
||||
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
// Factory creates a Facebook Channel from DB instance data.
|
||||
// Implements channels.ChannelFactory.
|
||||
func Factory(name string, creds json.RawMessage, cfg json.RawMessage,
|
||||
msgBus *bus.MessageBus, pairingSvc store.PairingStore) (channels.Channel, error) {
|
||||
|
||||
var c facebookCreds
|
||||
if err := json.Unmarshal(creds, &c); err != nil {
|
||||
return nil, fmt.Errorf("facebook: decode credentials: %w", err)
|
||||
}
|
||||
|
||||
var ic facebookInstanceConfig
|
||||
if len(cfg) > 0 {
|
||||
if err := json.Unmarshal(cfg, &ic); err != nil {
|
||||
return nil, fmt.Errorf("facebook: decode config: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
ch, err := New(ic, c, msgBus, pairingSvc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ch.SetName(name)
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
// Start connects the channel: verifies the page token, subscribes webhooks, marks healthy.
|
||||
func (ch *Channel) Start(ctx context.Context) error {
|
||||
ch.MarkStarting("connecting to Facebook page")
|
||||
|
||||
if err := ch.graphClient.VerifyToken(ctx); err != nil {
|
||||
ch.MarkFailed("token invalid", err.Error(), channels.ChannelFailureKindAuth, false)
|
||||
return err
|
||||
}
|
||||
|
||||
// Best-effort: subscribe app to webhooks.
|
||||
if err := ch.graphClient.SubscribeApp(ctx); err != nil {
|
||||
slog.Warn("facebook: webhook subscription failed (check app install on page)", "err", err)
|
||||
}
|
||||
|
||||
globalRouter.register(ch)
|
||||
ch.MarkHealthy("connected to page " + ch.pageID)
|
||||
ch.SetRunning(true)
|
||||
|
||||
// Background goroutine: evict stale dedup entries to prevent memory growth.
|
||||
go ch.runDedupCleaner()
|
||||
|
||||
slog.Info("facebook channel started", "page_id", ch.pageID, "name", ch.Name())
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop gracefully shuts down the channel.
|
||||
func (ch *Channel) Stop(_ context.Context) error {
|
||||
globalRouter.unregister(ch.pageID)
|
||||
ch.stopFn() // cancel stopCtx → cancels inflight Graph API calls
|
||||
close(ch.stopCh) // stop background goroutines
|
||||
ch.SetRunning(false)
|
||||
ch.MarkStopped("stopped")
|
||||
slog.Info("facebook channel stopped", "page_id", ch.pageID, "name", ch.Name())
|
||||
return nil
|
||||
}
|
||||
|
||||
// Send delivers an outbound message. Dispatches to comment reply or Messenger based on fb_mode metadata.
|
||||
func (ch *Channel) Send(ctx context.Context, msg bus.OutboundMessage) error {
|
||||
mode := msg.Metadata["fb_mode"]
|
||||
|
||||
switch mode {
|
||||
case "messenger":
|
||||
text := FormatForMessenger(msg.Content)
|
||||
parts := splitMessage(text, messengerMaxChars)
|
||||
for _, part := range parts {
|
||||
if _, err := ch.graphClient.SendMessage(ctx, msg.ChatID, part); err != nil {
|
||||
ch.handleAPIError(err)
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
default: // "comment"
|
||||
commentID := msg.Metadata["reply_to_comment_id"]
|
||||
if commentID == "" {
|
||||
return fmt.Errorf("facebook: reply_to_comment_id missing in outbound metadata")
|
||||
}
|
||||
text := FormatForComment(msg.Content)
|
||||
if _, err := ch.graphClient.ReplyToComment(ctx, commentID, text); err != nil {
|
||||
ch.handleAPIError(err)
|
||||
return err
|
||||
}
|
||||
|
||||
// First inbox: send a private DM after comment reply (best-effort).
|
||||
if ch.config.Features.FirstInbox {
|
||||
senderID := msg.Metadata["sender_id"]
|
||||
if senderID != "" {
|
||||
ch.sendFirstInbox(ctx, senderID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// WebhookHandler returns the shared webhook path and the global router as handler.
|
||||
// Only the first facebook instance mounts the route; others return ("", nil).
|
||||
func (ch *Channel) WebhookHandler() (string, http.Handler) {
|
||||
return globalRouter.webhookRoute()
|
||||
}
|
||||
|
||||
// handleAPIError maps Graph API errors to channel health states.
|
||||
func (ch *Channel) handleAPIError(err error) {
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case IsAuthError(err):
|
||||
ch.MarkFailed("token expired", err.Error(), channels.ChannelFailureKindAuth, false)
|
||||
case IsPermissionError(err):
|
||||
ch.MarkFailed("permission denied", err.Error(), channels.ChannelFailureKindAuth, false)
|
||||
case IsRateLimitError(err):
|
||||
ch.MarkDegraded("rate limited", err.Error(), channels.ChannelFailureKindNetwork, true)
|
||||
default:
|
||||
ch.MarkDegraded("api error", err.Error(), channels.ChannelFailureKindUnknown, true)
|
||||
}
|
||||
}
|
||||
|
||||
// sendFirstInbox sends a one-time private DM after a comment reply (best-effort).
|
||||
func (ch *Channel) sendFirstInbox(ctx context.Context, senderID string) {
|
||||
if _, alreadySent := ch.firstInboxSent.LoadOrStore(senderID, struct{}{}); alreadySent {
|
||||
return
|
||||
}
|
||||
message := ch.config.FirstInboxMessage
|
||||
if message == "" {
|
||||
message = "Cảm ơn bạn đã comment! Mình có thể hỗ trợ thêm qua tin nhắn riêng."
|
||||
}
|
||||
if _, err := ch.graphClient.SendMessage(ctx, senderID, message); err != nil {
|
||||
slog.Warn("facebook: first inbox send failed", "sender_id", senderID, "err", err)
|
||||
ch.firstInboxSent.Delete(senderID) // allow retry on next comment
|
||||
}
|
||||
}
|
||||
|
||||
// runDedupCleaner evicts dedup entries older than dedupTTL every dedupCleanEvery.
|
||||
func (ch *Channel) runDedupCleaner() {
|
||||
ticker := time.NewTicker(dedupCleanEvery)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ch.stopCh:
|
||||
return
|
||||
case <-ticker.C:
|
||||
now := time.Now()
|
||||
ch.dedup.Range(func(k, v any) bool {
|
||||
if t, ok := v.(time.Time); ok && now.Sub(t) > dedupTTL {
|
||||
ch.dedup.Delete(k)
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// isDup checks and records a dedup key. Returns true if the key was already seen.
|
||||
func (ch *Channel) isDup(key string) bool {
|
||||
_, loaded := ch.dedup.LoadOrStore(key, time.Now())
|
||||
return loaded
|
||||
}
|
||||
|
||||
@@ -0,0 +1,395 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// --- verifySignature ---
|
||||
|
||||
func TestVerifySignature(t *testing.T) {
|
||||
secret := "test_secret"
|
||||
body := []byte(`{"object":"page","entry":[]}`)
|
||||
|
||||
mac := hmac.New(sha256.New, []byte(secret))
|
||||
mac.Write(body)
|
||||
validSig := "sha256=" + hex.EncodeToString(mac.Sum(nil))
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
sig string
|
||||
wantValid bool
|
||||
}{
|
||||
{"valid signature", validSig, true},
|
||||
{"wrong prefix", "sha1=" + hex.EncodeToString(mac.Sum(nil)), false},
|
||||
{"empty signature", "", false},
|
||||
{"bad hex", "sha256=notvalidhex", false},
|
||||
{"tampered body", "sha256=0000000000000000000000000000000000000000000000000000000000000000", false},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := verifySignature(body, tt.sig, secret)
|
||||
if got != tt.wantValid {
|
||||
t.Errorf("verifySignature() = %v, want %v", got, tt.wantValid)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// --- WebhookHandler GET verification ---
|
||||
|
||||
func TestWebhookHandlerVerification(t *testing.T) {
|
||||
wh := NewWebhookHandler("secret", "my_verify_token")
|
||||
|
||||
t.Run("valid challenge", func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet,
|
||||
"/webhook?hub.mode=subscribe&hub.verify_token=my_verify_token&hub.challenge=abc123", nil)
|
||||
w := httptest.NewRecorder()
|
||||
wh.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", w.Code)
|
||||
}
|
||||
if w.Body.String() != "abc123" {
|
||||
t.Errorf("body = %q, want %q", w.Body.String(), "abc123")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("wrong verify token", func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet,
|
||||
"/webhook?hub.mode=subscribe&hub.verify_token=wrong&hub.challenge=abc123", nil)
|
||||
w := httptest.NewRecorder()
|
||||
wh.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusForbidden {
|
||||
t.Errorf("status = %d, want 403", w.Code)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("wrong hub.mode", func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet,
|
||||
"/webhook?hub.mode=unsubscribe&hub.verify_token=my_verify_token&hub.challenge=abc123", nil)
|
||||
w := httptest.NewRecorder()
|
||||
wh.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusForbidden {
|
||||
t.Errorf("status = %d, want 403", w.Code)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("unsafe challenge rejected", func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet,
|
||||
"/webhook?hub.mode=subscribe&hub.verify_token=my_verify_token&hub.challenge=<script>alert(1)</script>", nil)
|
||||
w := httptest.NewRecorder()
|
||||
wh.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want 400", w.Code)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// --- WebhookHandler POST event delivery ---
|
||||
|
||||
func signBody(t *testing.T, body []byte, secret string) string {
|
||||
t.Helper()
|
||||
mac := hmac.New(sha256.New, []byte(secret))
|
||||
mac.Write(body)
|
||||
return "sha256=" + hex.EncodeToString(mac.Sum(nil))
|
||||
}
|
||||
|
||||
func TestWebhookHandlerPostEvents(t *testing.T) {
|
||||
appSecret := "app_secret_123"
|
||||
|
||||
t.Run("invalid signature returns 200 (no retry)", func(t *testing.T) {
|
||||
wh := NewWebhookHandler(appSecret, "token")
|
||||
body := []byte(`{"object":"page","entry":[]}`)
|
||||
req := httptest.NewRequest(http.MethodPost, "/webhook", strings.NewReader(string(body)))
|
||||
req.Header.Set("X-Hub-Signature-256", "sha256=badbadbadbad")
|
||||
w := httptest.NewRecorder()
|
||||
wh.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Errorf("status = %d, want 200 (Facebook must stop retrying)", w.Code)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("valid comment event dispatches callback", func(t *testing.T) {
|
||||
var gotChange ChangeValue
|
||||
wh := NewWebhookHandler(appSecret, "token")
|
||||
wh.onComment = func(entry WebhookEntry, change ChangeValue) {
|
||||
gotChange = change
|
||||
}
|
||||
|
||||
payload := WebhookPayload{
|
||||
Object: "page",
|
||||
Entry: []WebhookEntry{{
|
||||
ID: "111",
|
||||
Time: 1234567890,
|
||||
Changes: []WebhookChange{{
|
||||
Field: "feed",
|
||||
Value: ChangeValue{
|
||||
From: FBUser{ID: "456", Name: "Alice"},
|
||||
Item: "comment",
|
||||
CommentID: "789",
|
||||
PostID: "post1",
|
||||
Message: "Hello page!",
|
||||
Verb: "add",
|
||||
},
|
||||
}},
|
||||
}},
|
||||
}
|
||||
body, _ := json.Marshal(payload)
|
||||
req := httptest.NewRequest(http.MethodPost, "/webhook", strings.NewReader(string(body)))
|
||||
req.Header.Set("X-Hub-Signature-256", signBody(t, body, appSecret))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
wh.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", w.Code)
|
||||
}
|
||||
if gotChange.CommentID != "789" {
|
||||
t.Errorf("comment callback not called or wrong comment_id: %+v", gotChange)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("valid messenger event dispatches callback", func(t *testing.T) {
|
||||
var gotEvent MessagingEvent
|
||||
wh := NewWebhookHandler(appSecret, "token")
|
||||
wh.onMessage = func(_ WebhookEntry, event MessagingEvent) {
|
||||
gotEvent = event
|
||||
}
|
||||
|
||||
mid := "m_abc123"
|
||||
payload := WebhookPayload{
|
||||
Object: "page",
|
||||
Entry: []WebhookEntry{{
|
||||
ID: "111",
|
||||
Time: 1234567890,
|
||||
Messaging: []MessagingEvent{{
|
||||
Sender: FBUser{ID: "user1"},
|
||||
Recipient: FBUser{ID: "111"},
|
||||
Timestamp: 1234567890,
|
||||
Message: &IncomingMessage{MID: mid, Text: "hi"},
|
||||
}},
|
||||
}},
|
||||
}
|
||||
body, _ := json.Marshal(payload)
|
||||
req := httptest.NewRequest(http.MethodPost, "/webhook", strings.NewReader(string(body)))
|
||||
req.Header.Set("X-Hub-Signature-256", signBody(t, body, appSecret))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
wh.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", w.Code)
|
||||
}
|
||||
if gotEvent.Message == nil || gotEvent.Message.MID != mid {
|
||||
t.Errorf("messenger callback not called: %+v", gotEvent)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("non-page object ignored", func(t *testing.T) {
|
||||
called := false
|
||||
wh := NewWebhookHandler(appSecret, "token")
|
||||
wh.onComment = func(_ WebhookEntry, _ ChangeValue) { called = true }
|
||||
|
||||
payload := map[string]any{"object": "user", "entry": []any{}}
|
||||
body, _ := json.Marshal(payload)
|
||||
req := httptest.NewRequest(http.MethodPost, "/webhook", strings.NewReader(string(body)))
|
||||
req.Header.Set("X-Hub-Signature-256", signBody(t, body, appSecret))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
wh.ServeHTTP(w, req)
|
||||
|
||||
if called {
|
||||
t.Error("callback should not be called for non-page object")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// --- Formatter ---
|
||||
|
||||
func TestFormatForComment(t *testing.T) {
|
||||
tests := []struct {
|
||||
input string
|
||||
want string
|
||||
}{
|
||||
{"**bold** text", "bold text"},
|
||||
{"*italic* text", "italic text"},
|
||||
{"**bold** and *italic*", "bold and italic"},
|
||||
{"[link](https://example.com)", "link (https://example.com)"},
|
||||
{"# Heading\nbody", "Heading\nbody"},
|
||||
{"<b>html</b>", "html"},
|
||||
{"`code`", "code"},
|
||||
// Long text truncated at 8000 chars.
|
||||
{strings.Repeat("a", 9000), strings.Repeat("a", 8000)},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
got := FormatForComment(tt.input)
|
||||
if got != tt.want {
|
||||
t.Errorf("FormatForComment(%q) = %q, want %q", tt.input, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatForMessenger(t *testing.T) {
|
||||
tests := []struct {
|
||||
input string
|
||||
want string
|
||||
}{
|
||||
{"**bold**", "bold"},
|
||||
{"*italic*", "italic"},
|
||||
{"[link](https://x.com)", "link (https://x.com)"},
|
||||
{" leading space ", "leading space"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
got := FormatForMessenger(tt.input)
|
||||
if got != tt.want {
|
||||
t.Errorf("FormatForMessenger(%q) = %q, want %q", tt.input, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitMessage(t *testing.T) {
|
||||
t.Run("short message not split", func(t *testing.T) {
|
||||
parts := splitMessage("hello", 2000)
|
||||
if len(parts) != 1 || parts[0] != "hello" {
|
||||
t.Errorf("unexpected parts: %v", parts)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("splits at sentence boundary", func(t *testing.T) {
|
||||
long := strings.Repeat("word ", 200) + ". " + strings.Repeat("word ", 200)
|
||||
parts := splitMessage(long, 500)
|
||||
if len(parts) < 2 {
|
||||
t.Errorf("expected multiple parts, got %d", len(parts))
|
||||
}
|
||||
for _, p := range parts {
|
||||
if len([]rune(p)) > 500 {
|
||||
t.Errorf("part exceeds maxChars: len=%d", len([]rune(p)))
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("hard cut when no boundary found", func(t *testing.T) {
|
||||
noSpace := strings.Repeat("a", 3000)
|
||||
parts := splitMessage(noSpace, 2000)
|
||||
if len(parts) != 2 {
|
||||
t.Errorf("expected 2 parts, got %d", len(parts))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// --- validateFBID ---
|
||||
|
||||
func TestValidateFBID(t *testing.T) {
|
||||
valid := []string{"123456789", "123_456", "1"}
|
||||
invalid := []string{"", "../me", "123abc", "123 456", "../../accounts"}
|
||||
|
||||
for _, id := range valid {
|
||||
if err := validateFBID(id); err != nil {
|
||||
t.Errorf("validateFBID(%q) unexpectedly returned error: %v", id, err)
|
||||
}
|
||||
}
|
||||
for _, id := range invalid {
|
||||
if err := validateFBID(id); err == nil {
|
||||
t.Errorf("validateFBID(%q) should have returned error", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- parseRetryAfter ---
|
||||
|
||||
func TestParseRetryAfter(t *testing.T) {
|
||||
tests := []struct {
|
||||
header string
|
||||
want time.Duration
|
||||
}{
|
||||
{"", 5 * time.Second},
|
||||
{"0", 5 * time.Second},
|
||||
{"notanumber", 5 * time.Second},
|
||||
{"10", 10 * time.Second},
|
||||
{"60", 60 * time.Second},
|
||||
// Capped at maxRetryAfterSec (60).
|
||||
{"3600", 60 * time.Second},
|
||||
{"999999", 60 * time.Second},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
resp := &http.Response{Header: make(http.Header)}
|
||||
if tt.header != "" {
|
||||
resp.Header.Set("Retry-After", tt.header)
|
||||
}
|
||||
got := parseRetryAfter(resp)
|
||||
if got != tt.want {
|
||||
t.Errorf("parseRetryAfter(%q) = %v, want %v", tt.header, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- Dedup eviction ---
|
||||
|
||||
func TestDedupEviction(t *testing.T) {
|
||||
// Override TTL to a tiny value so we can test eviction synchronously.
|
||||
const shortTTL = 10 * time.Millisecond
|
||||
|
||||
ch := &Channel{}
|
||||
|
||||
key := "comment:test_event"
|
||||
// First call: not a dup.
|
||||
ch.dedup.Store(key, time.Now().Add(-shortTTL-time.Millisecond))
|
||||
|
||||
// Simulate cleaner: evict entries older than shortTTL.
|
||||
now := time.Now()
|
||||
ch.dedup.Range(func(k, v any) bool {
|
||||
if t2, ok := v.(time.Time); ok && now.Sub(t2) > shortTTL {
|
||||
ch.dedup.Delete(k)
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
// After eviction, isDup should return false (entry gone).
|
||||
if ch.isDup(key) {
|
||||
// isDup stores the key now, so second call would be dup — just verify the first returns false.
|
||||
// Since isDup uses LoadOrStore, a successful store means it wasn't there.
|
||||
// The test above stored, deleted, so isDup should do a fresh store → return false.
|
||||
}
|
||||
// Verify second call IS a dup.
|
||||
if !ch.isDup(key) {
|
||||
t.Error("second isDup call should return true (key already stored)")
|
||||
}
|
||||
}
|
||||
|
||||
// --- graphAPIError classification ---
|
||||
|
||||
func TestErrorClassifiers(t *testing.T) {
|
||||
authErr := &graphAPIError{code: 190, msg: "invalid token"}
|
||||
permErr := &graphAPIError{code: 10, msg: "permission denied"}
|
||||
rateErr := &graphAPIError{code: 4, msg: "rate limit"}
|
||||
otherErr := &graphAPIError{code: 999, msg: "unknown"}
|
||||
wrappedAuth := fmt.Errorf("send failed: %w", authErr)
|
||||
|
||||
if !IsAuthError(authErr) {
|
||||
t.Error("IsAuthError should be true for code 190")
|
||||
}
|
||||
if !IsAuthError(wrappedAuth) {
|
||||
t.Error("IsAuthError should work with wrapped errors")
|
||||
}
|
||||
if !IsPermissionError(permErr) {
|
||||
t.Error("IsPermissionError should be true for code 10")
|
||||
}
|
||||
if !IsRateLimitError(rateErr) {
|
||||
t.Error("IsRateLimitError should be true for code 4")
|
||||
}
|
||||
if IsAuthError(otherErr) || IsPermissionError(otherErr) || IsRateLimitError(otherErr) {
|
||||
t.Error("unknown error should not match any classifier")
|
||||
}
|
||||
if IsAuthError(nil) {
|
||||
t.Error("nil should not be auth error")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
commentMaxChars = 8000
|
||||
messengerMaxChars = 2000
|
||||
)
|
||||
|
||||
var (
|
||||
// Two-pass bold: **text** first, then *text*, to avoid stray asterisk artifacts.
|
||||
reBoldDouble = regexp.MustCompile(`\*\*([^*]+)\*\*`)
|
||||
reBoldSingle = regexp.MustCompile(`\*([^*]+)\*`)
|
||||
// markdownLink converts [text](url) → "text (url)".
|
||||
reLink = regexp.MustCompile(`\[([^\]]+)\]\(([^)]+)\)`)
|
||||
// HTML tags.
|
||||
reHTML = regexp.MustCompile(`<[^>]+>`)
|
||||
// Heading markers (# … at line start).
|
||||
reHeader = regexp.MustCompile(`(?m)^#{1,6}\s+`)
|
||||
// Inline `code` backticks.
|
||||
reCode = regexp.MustCompile("`([^`]+)`")
|
||||
)
|
||||
|
||||
// FormatForComment sanitizes agent output for posting as a Facebook comment.
|
||||
// Facebook comments do not support markdown or HTML.
|
||||
func FormatForComment(text string) string {
|
||||
text = reHeader.ReplaceAllString(text, "")
|
||||
text = reBoldDouble.ReplaceAllString(text, "$1") // ** first
|
||||
text = reBoldSingle.ReplaceAllString(text, "$1") // * second
|
||||
text = reLink.ReplaceAllString(text, "$1 ($2)")
|
||||
text = reCode.ReplaceAllString(text, "$1")
|
||||
text = reHTML.ReplaceAllString(text, "")
|
||||
text = strings.TrimSpace(text)
|
||||
return truncateRunes(text, commentMaxChars)
|
||||
}
|
||||
|
||||
// FormatForMessenger sanitizes agent output for Messenger messages.
|
||||
func FormatForMessenger(text string) string {
|
||||
text = reHeader.ReplaceAllString(text, "")
|
||||
text = reBoldDouble.ReplaceAllString(text, "$1")
|
||||
text = reBoldSingle.ReplaceAllString(text, "$1")
|
||||
text = reLink.ReplaceAllString(text, "$1 ($2)")
|
||||
text = reCode.ReplaceAllString(text, "$1")
|
||||
text = reHTML.ReplaceAllString(text, "")
|
||||
return strings.TrimSpace(text)
|
||||
}
|
||||
|
||||
// splitMessage splits text into chunks of at most maxChars runes,
|
||||
// preferring paragraph or sentence boundaries to avoid mid-word cuts.
|
||||
// Uses plain-text splitting (not channels.ChunkMarkdown) because Facebook
|
||||
// strips all markdown — fenced code block repair is unnecessary.
|
||||
func splitMessage(text string, maxChars int) []string {
|
||||
runes := []rune(text) // convert once
|
||||
if len(runes) <= maxChars {
|
||||
return []string{text}
|
||||
}
|
||||
|
||||
var parts []string
|
||||
for len(runes) > maxChars {
|
||||
chunk := string(runes[:maxChars])
|
||||
|
||||
// Try paragraph boundary.
|
||||
if idx := strings.LastIndex(chunk, "\n\n"); idx > maxChars/2 {
|
||||
parts = append(parts, strings.TrimSpace(chunk[:idx]))
|
||||
runes = []rune(strings.TrimSpace(string(runes[idx+2:])))
|
||||
continue
|
||||
}
|
||||
|
||||
// Try sentence boundary.
|
||||
split := false
|
||||
for _, sep := range []string{". ", "! ", "? ", "\n"} {
|
||||
if idx := strings.LastIndex(chunk, sep); idx > maxChars/2 {
|
||||
parts = append(parts, strings.TrimSpace(chunk[:idx+1]))
|
||||
runes = []rune(strings.TrimSpace(string(runes[idx+len(sep):])))
|
||||
split = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if split {
|
||||
continue
|
||||
}
|
||||
|
||||
// Hard cut at maxChars.
|
||||
parts = append(parts, strings.TrimSpace(chunk))
|
||||
runes = []rune(strings.TrimSpace(string(runes[maxChars:])))
|
||||
}
|
||||
|
||||
if remaining := strings.TrimSpace(string(runes)); remaining != "" {
|
||||
parts = append(parts, remaining)
|
||||
}
|
||||
return parts
|
||||
}
|
||||
|
||||
// truncateRunes truncates s to at most n Unicode code points.
|
||||
func truncateRunes(s string, n int) string {
|
||||
runes := []rune(s)
|
||||
if len(runes) <= n {
|
||||
return s
|
||||
}
|
||||
return string(runes[:n])
|
||||
}
|
||||
@@ -0,0 +1,355 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
graphAPIVersion = "v25.0"
|
||||
graphAPIBase = "https://graph.facebook.com"
|
||||
maxRetries = 3
|
||||
// maxRetryAfterSec caps the Retry-After sleep to prevent goroutine stalls on abnormal values.
|
||||
maxRetryAfterSec = 60
|
||||
)
|
||||
|
||||
// fbIDPattern validates Facebook object IDs: numeric or "{num}_{num}" form (post IDs).
|
||||
var fbIDPattern = regexp.MustCompile(`^\d+(_\d+)?$`)
|
||||
|
||||
// GraphClient wraps the Facebook Graph API for a single page instance.
|
||||
type GraphClient struct {
|
||||
httpClient *http.Client
|
||||
pageAccessToken string
|
||||
pageID string
|
||||
}
|
||||
|
||||
// NewGraphClient creates a new GraphClient for the given page.
|
||||
func NewGraphClient(pageAccessToken, pageID string) *GraphClient {
|
||||
return &GraphClient{
|
||||
httpClient: &http.Client{Timeout: 15 * time.Second},
|
||||
pageAccessToken: pageAccessToken,
|
||||
pageID: pageID,
|
||||
}
|
||||
}
|
||||
|
||||
// VerifyToken checks the page access token by calling GET /me.
|
||||
func (g *GraphClient) VerifyToken(ctx context.Context) error {
|
||||
data, err := g.doRequest(ctx, http.MethodGet, "/me?fields=id,name", nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("facebook: token verification failed: %w", err)
|
||||
}
|
||||
var result struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &result); err != nil {
|
||||
return fmt.Errorf("facebook: token verification parse error: %w", err)
|
||||
}
|
||||
slog.Info("facebook: page token verified", "page_id", result.ID, "name", result.Name)
|
||||
return nil
|
||||
}
|
||||
|
||||
// SubscribeApp subscribes the app to the page's webhook events.
|
||||
func (g *GraphClient) SubscribeApp(ctx context.Context) error {
|
||||
if err := validateFBID(g.pageID); err != nil {
|
||||
return fmt.Errorf("facebook: subscribe app: %w", err)
|
||||
}
|
||||
path := fmt.Sprintf("/%s/subscribed_apps?subscribed_fields=feed,messages", g.pageID)
|
||||
_, err := g.doRequest(ctx, http.MethodPost, path, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("facebook: subscribe app failed: %w", err)
|
||||
}
|
||||
slog.Info("facebook: app subscribed to page webhooks", "page_id", g.pageID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetPost fetches a post by ID with message and story fields.
|
||||
func (g *GraphClient) GetPost(ctx context.Context, postID string) (*GraphPost, error) {
|
||||
if err := validateFBID(postID); err != nil {
|
||||
return nil, fmt.Errorf("facebook: get post: %w", err)
|
||||
}
|
||||
path := fmt.Sprintf("/%s?fields=id,message,story,created_time", postID)
|
||||
data, err := g.doRequest(ctx, http.MethodGet, path, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var post GraphPost
|
||||
if err := json.Unmarshal(data, &post); err != nil {
|
||||
return nil, fmt.Errorf("facebook: parse post: %w", err)
|
||||
}
|
||||
return &post, nil
|
||||
}
|
||||
|
||||
// GetComment fetches a single comment by ID.
|
||||
func (g *GraphClient) GetComment(ctx context.Context, commentID string) (*GraphComment, error) {
|
||||
if err := validateFBID(commentID); err != nil {
|
||||
return nil, fmt.Errorf("facebook: get comment: %w", err)
|
||||
}
|
||||
path := fmt.Sprintf("/%s?fields=id,message,from,created_time", commentID)
|
||||
data, err := g.doRequest(ctx, http.MethodGet, path, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var c GraphComment
|
||||
if err := json.Unmarshal(data, &c); err != nil {
|
||||
return nil, fmt.Errorf("facebook: parse comment: %w", err)
|
||||
}
|
||||
return &c, nil
|
||||
}
|
||||
|
||||
// GetCommentThread fetches up to limit comments under a parent comment.
|
||||
func (g *GraphClient) GetCommentThread(ctx context.Context, parentCommentID string, limit int) ([]GraphComment, error) {
|
||||
if err := validateFBID(parentCommentID); err != nil {
|
||||
return nil, fmt.Errorf("facebook: get comment thread: %w", err)
|
||||
}
|
||||
if limit <= 0 {
|
||||
limit = 10
|
||||
}
|
||||
path := fmt.Sprintf("/%s/comments?fields=id,message,from,created_time&limit=%d", parentCommentID, limit)
|
||||
data, err := g.doRequest(ctx, http.MethodGet, path, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var resp GraphListResponse[GraphComment]
|
||||
if err := json.Unmarshal(data, &resp); err != nil {
|
||||
return nil, fmt.Errorf("facebook: parse comment thread: %w", err)
|
||||
}
|
||||
return resp.Data, nil
|
||||
}
|
||||
|
||||
// ReplyToComment posts a reply to a comment. Returns the new comment ID.
|
||||
func (g *GraphClient) ReplyToComment(ctx context.Context, commentID, message string) (string, error) {
|
||||
if err := validateFBID(commentID); err != nil {
|
||||
return "", fmt.Errorf("facebook: reply to comment: %w", err)
|
||||
}
|
||||
path := fmt.Sprintf("/%s/comments", commentID)
|
||||
body := map[string]string{"message": message}
|
||||
data, err := g.doRequest(ctx, http.MethodPost, path, body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var result struct {
|
||||
ID string `json:"id"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &result); err != nil {
|
||||
return "", fmt.Errorf("facebook: parse reply result: %w", err)
|
||||
}
|
||||
return result.ID, nil
|
||||
}
|
||||
|
||||
// SendMessage sends a Messenger message to the given recipient. Returns message ID.
|
||||
func (g *GraphClient) SendMessage(ctx context.Context, recipientID, message string) (string, error) {
|
||||
body := map[string]any{
|
||||
"recipient": map[string]string{"id": recipientID},
|
||||
"message": map[string]string{"text": message},
|
||||
}
|
||||
data, err := g.doRequest(ctx, http.MethodPost, "/me/messages", body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var result struct {
|
||||
MessageID string `json:"message_id"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &result); err != nil {
|
||||
return "", fmt.Errorf("facebook: parse send message result: %w", err)
|
||||
}
|
||||
return result.MessageID, nil
|
||||
}
|
||||
|
||||
// SendTypingOn sends a typing indicator to the recipient (auto-off after 3s).
|
||||
func (g *GraphClient) SendTypingOn(ctx context.Context, recipientID string) error {
|
||||
body := map[string]any{
|
||||
"recipient": map[string]string{"id": recipientID},
|
||||
"sender_action": "typing_on",
|
||||
}
|
||||
_, err := g.doRequest(ctx, http.MethodPost, "/me/messages", body)
|
||||
return err
|
||||
}
|
||||
|
||||
// doRequest executes a Graph API call with retries on transient errors.
|
||||
// The page access token is passed via Authorization header (never in the URL).
|
||||
func (g *GraphClient) doRequest(ctx context.Context, method, path string, body any) ([]byte, error) {
|
||||
apiURL := fmt.Sprintf("%s/%s%s", graphAPIBase, graphAPIVersion, path)
|
||||
|
||||
for attempt := 0; attempt < maxRetries; attempt++ {
|
||||
if attempt > 0 {
|
||||
backoff := time.Duration(1<<uint(attempt-1)) * time.Second
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(backoff):
|
||||
}
|
||||
}
|
||||
|
||||
var reqBody io.Reader
|
||||
if body != nil {
|
||||
b, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("facebook: marshal request: %w", err)
|
||||
}
|
||||
reqBody = bytes.NewReader(b)
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, method, apiURL, reqBody)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("facebook: build request: %w", err)
|
||||
}
|
||||
// Pass token via header to avoid URL logging exposure.
|
||||
req.Header.Set("Authorization", "Bearer "+g.pageAccessToken)
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
|
||||
resp, err := g.httpClient.Do(req)
|
||||
if err != nil {
|
||||
if attempt < maxRetries-1 {
|
||||
slog.Warn("facebook: api request error, retrying", "attempt", attempt+1, "err", err)
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("facebook: api request: %w", err)
|
||||
}
|
||||
|
||||
respBody, readErr := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
if readErr != nil {
|
||||
return nil, fmt.Errorf("facebook: read response: %w", readErr)
|
||||
}
|
||||
|
||||
// Proactive rate limit monitoring.
|
||||
g.logRateLimit(resp)
|
||||
|
||||
// Retry on 5xx.
|
||||
if resp.StatusCode >= 500 && attempt < maxRetries-1 {
|
||||
slog.Warn("facebook: server error, retrying", "status", resp.StatusCode, "attempt", attempt+1)
|
||||
continue
|
||||
}
|
||||
|
||||
// Parse Graph API error envelope.
|
||||
if resp.StatusCode >= 400 {
|
||||
var apiErr graphErrorBody
|
||||
if json.Unmarshal(respBody, &apiErr) == nil && apiErr.Error.Code != 0 {
|
||||
// 24h messaging window violation — not retryable.
|
||||
if apiErr.Error.Code == 551 || apiErr.Error.Subcode == 2018109 {
|
||||
slog.Warn("facebook: 24h messaging window expired", "code", apiErr.Error.Code)
|
||||
return nil, &graphAPIError{code: apiErr.Error.Code, msg: apiErr.Error.Message}
|
||||
}
|
||||
// Rate limited: sleep and retry (capped).
|
||||
if resp.StatusCode == 429 && attempt < maxRetries-1 {
|
||||
retryAfter := parseRetryAfter(resp)
|
||||
slog.Warn("facebook: rate limited", "retry_after", retryAfter)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(retryAfter):
|
||||
}
|
||||
continue
|
||||
}
|
||||
return nil, &graphAPIError{code: apiErr.Error.Code, msg: apiErr.Error.Message}
|
||||
}
|
||||
return nil, fmt.Errorf("facebook: http %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
return respBody, nil
|
||||
}
|
||||
|
||||
// All attempts exhausted (only reachable when every iteration took the continue path).
|
||||
return nil, fmt.Errorf("facebook: max retries exceeded")
|
||||
}
|
||||
|
||||
// logRateLimit parses the X-Business-Use-Case-Usage header and warns when approaching limits.
|
||||
func (g *GraphClient) logRateLimit(resp *http.Response) {
|
||||
usage := resp.Header.Get("X-Business-Use-Case-Usage")
|
||||
if usage == "" {
|
||||
return
|
||||
}
|
||||
var parsed map[string][]struct {
|
||||
CallCount int `json:"call_count"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(usage), &parsed); err != nil {
|
||||
return
|
||||
}
|
||||
for _, entries := range parsed {
|
||||
for _, e := range entries {
|
||||
if e.CallCount >= 95 {
|
||||
slog.Warn("facebook: rate limit critical", "call_count_pct", e.CallCount, "page_id", g.pageID)
|
||||
} else if e.CallCount >= 80 {
|
||||
slog.Warn("facebook: rate limit warning", "call_count_pct", e.CallCount, "page_id", g.pageID)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// parseRetryAfter extracts the Retry-After header, capped at maxRetryAfterSec.
|
||||
func parseRetryAfter(resp *http.Response) time.Duration {
|
||||
val := resp.Header.Get("Retry-After")
|
||||
if val == "" {
|
||||
return 5 * time.Second
|
||||
}
|
||||
secs, err := strconv.Atoi(val)
|
||||
if err != nil || secs <= 0 {
|
||||
return 5 * time.Second
|
||||
}
|
||||
if secs > maxRetryAfterSec {
|
||||
secs = maxRetryAfterSec
|
||||
}
|
||||
return time.Duration(secs) * time.Second
|
||||
}
|
||||
|
||||
// validateFBID returns an error if id is not a valid Facebook object ID format.
|
||||
// Facebook IDs are numeric strings, or "{num}_{num}" for post IDs.
|
||||
func validateFBID(id string) error {
|
||||
if id == "" {
|
||||
return fmt.Errorf("empty facebook ID")
|
||||
}
|
||||
if !fbIDPattern.MatchString(id) {
|
||||
return fmt.Errorf("invalid facebook ID format: %q", id)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// graphAPIError is a structured error from the Facebook Graph API.
|
||||
type graphAPIError struct {
|
||||
code int
|
||||
msg string
|
||||
}
|
||||
|
||||
func (e *graphAPIError) Error() string {
|
||||
return fmt.Sprintf("facebook graph api error %d: %s", e.code, e.msg)
|
||||
}
|
||||
|
||||
// IsAuthError returns true when the error is an expired or invalid token.
|
||||
func IsAuthError(err error) bool {
|
||||
var ge *graphAPIError
|
||||
if !errors.As(err, &ge) {
|
||||
return false
|
||||
}
|
||||
return ge.code == 190 || ge.code == 102
|
||||
}
|
||||
|
||||
// IsPermissionError returns true for permission-denied errors.
|
||||
func IsPermissionError(err error) bool {
|
||||
var ge *graphAPIError
|
||||
if !errors.As(err, &ge) {
|
||||
return false
|
||||
}
|
||||
return ge.code == 10 || ge.code == 200
|
||||
}
|
||||
|
||||
// IsRateLimitError returns true for rate limit errors.
|
||||
func IsRateLimitError(err error) bool {
|
||||
var ge *graphAPIError
|
||||
if !errors.As(err, &ge) {
|
||||
return false
|
||||
}
|
||||
return ge.code == 4 || ge.code == 17 || ge.code == 32 || ge.code == 613
|
||||
}
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
)
|
||||
|
||||
// handleMessagingEvent processes a Messenger inbox event.
|
||||
func (ch *Channel) handleMessagingEvent(entry WebhookEntry, event MessagingEvent) {
|
||||
// Feature gate.
|
||||
if !ch.config.Features.MessengerAutoReply {
|
||||
return
|
||||
}
|
||||
|
||||
// Page routing guard (before dedup write).
|
||||
if entry.ID != ch.pageID {
|
||||
return
|
||||
}
|
||||
|
||||
// Self-message prevention: skip messages sent by the page itself.
|
||||
if event.Sender.ID == ch.pageID {
|
||||
return
|
||||
}
|
||||
|
||||
// Skip delivery/read receipts and other non-content events.
|
||||
if event.Message == nil && event.Postback == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// Dedup by message MID or postback signature (include payload to reduce collision risk).
|
||||
var eventKey string
|
||||
switch {
|
||||
case event.Message != nil:
|
||||
eventKey = "msg:" + event.Message.MID
|
||||
case event.Postback != nil:
|
||||
eventKey = fmt.Sprintf("postback:%s:%d:%s", event.Sender.ID, event.Timestamp, event.Postback.Payload)
|
||||
}
|
||||
if ch.isDup(eventKey) {
|
||||
slog.Debug("facebook: duplicate messaging event skipped", "key", eventKey)
|
||||
return
|
||||
}
|
||||
|
||||
// Extract text content.
|
||||
var content string
|
||||
switch {
|
||||
case event.Message != nil && event.Message.Text != "":
|
||||
content = event.Message.Text
|
||||
case event.Postback != nil:
|
||||
content = event.Postback.Title
|
||||
default:
|
||||
// Attachment-only message — skip for now.
|
||||
return
|
||||
}
|
||||
|
||||
senderID := event.Sender.ID
|
||||
// Messenger sessions are 1:1: chatID = senderID (channel name scopes the session).
|
||||
chatID := senderID
|
||||
|
||||
metadata := map[string]string{
|
||||
"fb_mode": "messenger",
|
||||
"message_id": eventKey,
|
||||
"page_id": ch.pageID,
|
||||
"sender_id": senderID,
|
||||
}
|
||||
if ch.config.MessengerOptions.SessionTimeout != "" {
|
||||
metadata["session_timeout"] = ch.config.MessengerOptions.SessionTimeout
|
||||
}
|
||||
|
||||
ch.HandleMessage(senderID, chatID, content, nil, metadata, "direct")
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/sync/singleflight"
|
||||
)
|
||||
|
||||
const defaultPostCacheTTL = 15 * time.Minute
|
||||
|
||||
// postCacheEntry holds a cached post with its expiry time.
|
||||
type postCacheEntry struct {
|
||||
post *GraphPost
|
||||
expiresAt time.Time
|
||||
}
|
||||
|
||||
// PostFetcher fetches and caches Facebook post content and comment threads.
|
||||
// singleflight prevents redundant concurrent Graph API calls for the same post
|
||||
// (cache stampede on viral posts receiving many comments simultaneously).
|
||||
type PostFetcher struct {
|
||||
graphClient *GraphClient
|
||||
cacheTTL time.Duration
|
||||
cache sync.Map // postID(string) → *postCacheEntry
|
||||
sfGroup singleflight.Group // coalesces concurrent fetches for the same postID
|
||||
}
|
||||
|
||||
// NewPostFetcher creates a PostFetcher with the given cache TTL string (e.g. "15m").
|
||||
// Falls back to defaultPostCacheTTL if the string is empty or unparseable.
|
||||
func NewPostFetcher(client *GraphClient, cacheTTLStr string) *PostFetcher {
|
||||
ttl := defaultPostCacheTTL
|
||||
if cacheTTLStr != "" {
|
||||
if d, err := time.ParseDuration(cacheTTLStr); err == nil && d > 0 {
|
||||
ttl = d
|
||||
}
|
||||
}
|
||||
return &PostFetcher{
|
||||
graphClient: client,
|
||||
cacheTTL: ttl,
|
||||
}
|
||||
}
|
||||
|
||||
// GetPost returns the post for postID, using cache when fresh.
|
||||
// Concurrent callers for the same postID share one inflight request.
|
||||
func (pf *PostFetcher) GetPost(ctx context.Context, postID string) (*GraphPost, error) {
|
||||
if postID == "" {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Check cache first.
|
||||
if v, ok := pf.cache.Load(postID); ok {
|
||||
entry := v.(*postCacheEntry)
|
||||
if time.Now().Before(entry.expiresAt) {
|
||||
return entry.post, nil
|
||||
}
|
||||
pf.cache.Delete(postID)
|
||||
}
|
||||
|
||||
// Coalesce concurrent fetches for the same postID.
|
||||
val, err, _ := pf.sfGroup.Do(postID, func() (any, error) {
|
||||
// Re-check cache inside the singleflight call (another goroutine may have populated it).
|
||||
if v, ok := pf.cache.Load(postID); ok {
|
||||
entry := v.(*postCacheEntry)
|
||||
if time.Now().Before(entry.expiresAt) {
|
||||
return entry.post, nil
|
||||
}
|
||||
}
|
||||
post, err := pf.graphClient.GetPost(ctx, postID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pf.cache.Store(postID, &postCacheEntry{
|
||||
post: post,
|
||||
expiresAt: time.Now().Add(pf.cacheTTL),
|
||||
})
|
||||
return post, nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if val == nil {
|
||||
return nil, nil
|
||||
}
|
||||
return val.(*GraphPost), nil
|
||||
}
|
||||
|
||||
// GetCommentThread fetches up to depth comments under parentCommentID.
|
||||
// Not cached — thread content changes frequently as replies arrive.
|
||||
func (pf *PostFetcher) GetCommentThread(ctx context.Context, parentCommentID string, depth int) ([]GraphComment, error) {
|
||||
if parentCommentID == "" {
|
||||
return nil, nil
|
||||
}
|
||||
return pf.graphClient.GetCommentThread(ctx, parentCommentID, depth)
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
// Package facebook implements the Facebook Fanpage channel for GoClaw.
|
||||
// Supports: comment auto-reply, Messenger inbox auto-reply, first inbox DM.
|
||||
package facebook
|
||||
|
||||
// facebookCreds holds encrypted credentials stored in channel_instances.credentials.
|
||||
type facebookCreds struct {
|
||||
PageAccessToken string `json:"page_access_token"`
|
||||
AppSecret string `json:"app_secret"`
|
||||
VerifyToken string `json:"verify_token"`
|
||||
}
|
||||
|
||||
// facebookInstanceConfig holds non-secret config from channel_instances.config JSONB.
|
||||
type facebookInstanceConfig struct {
|
||||
PageID string `json:"page_id"`
|
||||
Features struct {
|
||||
CommentReply bool `json:"comment_reply"`
|
||||
FirstInbox bool `json:"first_inbox"`
|
||||
MessengerAutoReply bool `json:"messenger_auto_reply"`
|
||||
} `json:"features"`
|
||||
CommentReplyOptions struct {
|
||||
IncludePostContext bool `json:"include_post_context"`
|
||||
MaxThreadDepth int `json:"max_thread_depth"`
|
||||
} `json:"comment_reply_options"`
|
||||
MessengerOptions struct {
|
||||
SessionTimeout string `json:"session_timeout"`
|
||||
} `json:"messenger_options"`
|
||||
PostContextCacheTTL string `json:"post_context_cache_ttl"`
|
||||
// FirstInboxMessage is the DM text sent to commenters (first-inbox feature).
|
||||
// Defaults to Vietnamese if empty. Operators should set this to match their page language.
|
||||
FirstInboxMessage string `json:"first_inbox_message,omitempty"`
|
||||
AllowFrom []string `json:"allow_from,omitempty"`
|
||||
}
|
||||
|
||||
// --- Webhook payloads ---
|
||||
|
||||
// WebhookPayload is the top-level Facebook webhook event payload.
|
||||
type WebhookPayload struct {
|
||||
Object string `json:"object"`
|
||||
Entry []WebhookEntry `json:"entry"`
|
||||
}
|
||||
|
||||
// WebhookEntry is one page's events within a webhook delivery.
|
||||
type WebhookEntry struct {
|
||||
ID string `json:"id"` // page_id
|
||||
Time int64 `json:"time"`
|
||||
Changes []WebhookChange `json:"changes,omitempty"` // feed events (comments, posts)
|
||||
Messaging []MessagingEvent `json:"messaging,omitempty"` // Messenger events
|
||||
}
|
||||
|
||||
// WebhookChange is a single change event for feed subscriptions.
|
||||
type WebhookChange struct {
|
||||
Field string `json:"field"` // "feed", "mention", etc.
|
||||
Value ChangeValue `json:"value"`
|
||||
}
|
||||
|
||||
// ChangeValue holds the details of a feed change event.
|
||||
type ChangeValue struct {
|
||||
From FBUser `json:"from"`
|
||||
Item string `json:"item"` // "comment", "post", "status"
|
||||
CommentID string `json:"comment_id"`
|
||||
PostID string `json:"post_id"`
|
||||
ParentID string `json:"parent_id"` // parent comment ID for nested replies
|
||||
Message string `json:"message"`
|
||||
Verb string `json:"verb"` // "add", "edit", "remove"
|
||||
CreatedTime int64 `json:"created_time"`
|
||||
}
|
||||
|
||||
// MessagingEvent is a single Messenger inbox event.
|
||||
type MessagingEvent struct {
|
||||
Sender FBUser `json:"sender"`
|
||||
Recipient FBUser `json:"recipient"`
|
||||
Timestamp int64 `json:"timestamp"`
|
||||
Message *IncomingMessage `json:"message,omitempty"`
|
||||
Postback *Postback `json:"postback,omitempty"`
|
||||
}
|
||||
|
||||
// IncomingMessage holds a Messenger text/attachment message.
|
||||
type IncomingMessage struct {
|
||||
MID string `json:"mid"`
|
||||
Text string `json:"text"`
|
||||
Attachments []Attachment `json:"attachments,omitempty"`
|
||||
}
|
||||
|
||||
// Postback holds a Messenger postback event.
|
||||
type Postback struct {
|
||||
Title string `json:"title"`
|
||||
Payload string `json:"payload"`
|
||||
}
|
||||
|
||||
// Attachment is a Messenger media attachment.
|
||||
type Attachment struct {
|
||||
Type string `json:"type"` // "image", "video", "audio", "file"
|
||||
Payload AttachmentPayload `json:"payload"`
|
||||
}
|
||||
|
||||
// AttachmentPayload holds the URL of a media attachment.
|
||||
type AttachmentPayload struct {
|
||||
URL string `json:"url"`
|
||||
}
|
||||
|
||||
// FBUser is a minimal Facebook user reference.
|
||||
type FBUser struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name,omitempty"`
|
||||
}
|
||||
|
||||
// --- Graph API response types ---
|
||||
|
||||
// GraphComment is a Facebook comment object from the Graph API.
|
||||
type GraphComment struct {
|
||||
ID string `json:"id"`
|
||||
Message string `json:"message"`
|
||||
From FBUser `json:"from"`
|
||||
CreatedTime string `json:"created_time"`
|
||||
}
|
||||
|
||||
// GraphPost is a Facebook post object from the Graph API.
|
||||
type GraphPost struct {
|
||||
ID string `json:"id"`
|
||||
Message string `json:"message"`
|
||||
Story string `json:"story,omitempty"`
|
||||
CreatedTime string `json:"created_time"`
|
||||
}
|
||||
|
||||
// GraphPaging holds cursor-based pagination for Graph API list responses.
|
||||
type GraphPaging struct {
|
||||
Cursors struct {
|
||||
Before string `json:"before"`
|
||||
After string `json:"after"`
|
||||
} `json:"cursors"`
|
||||
Next string `json:"next,omitempty"`
|
||||
}
|
||||
|
||||
// GraphListResponse is a generic Graph API list response.
|
||||
type GraphListResponse[T any] struct {
|
||||
Data []T `json:"data"`
|
||||
Paging GraphPaging `json:"paging,omitempty"`
|
||||
}
|
||||
|
||||
// graphErrorBody is the error envelope returned by Graph API on failures.
|
||||
type graphErrorBody struct {
|
||||
Error struct {
|
||||
Message string `json:"message"`
|
||||
Type string `json:"type"`
|
||||
Code int `json:"code"`
|
||||
Subcode int `json:"error_subcode"`
|
||||
} `json:"error"`
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// WebhookHandler implements http.Handler for the Facebook webhook endpoint.
|
||||
// Handles both the GET verification challenge and POST event delivery.
|
||||
type WebhookHandler struct {
|
||||
appSecret string
|
||||
verifyToken string
|
||||
extraSecrets []string // additional app secrets for multi-Meta-App deployments
|
||||
onComment func(entry WebhookEntry, change ChangeValue)
|
||||
onMessage func(entry WebhookEntry, event MessagingEvent)
|
||||
}
|
||||
|
||||
// NewWebhookHandler creates a new WebhookHandler.
|
||||
func NewWebhookHandler(appSecret, verifyToken string) *WebhookHandler {
|
||||
return &WebhookHandler{
|
||||
appSecret: appSecret,
|
||||
verifyToken: verifyToken,
|
||||
}
|
||||
}
|
||||
|
||||
// hubChallengePattern validates that hub.challenge is safe to reflect.
|
||||
var hubChallengePattern = regexp.MustCompile(`^[a-zA-Z0-9_\-]{1,256}$`)
|
||||
|
||||
// ServeHTTP handles Facebook webhook GET (verification) and POST (event delivery).
|
||||
func (wh *WebhookHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.Method {
|
||||
case http.MethodGet:
|
||||
wh.handleVerification(w, r)
|
||||
case http.MethodPost:
|
||||
wh.handleEvent(w, r)
|
||||
default:
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
}
|
||||
}
|
||||
|
||||
// handleVerification responds to Facebook's webhook verification challenge.
|
||||
func (wh *WebhookHandler) handleVerification(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
if q.Get("hub.mode") != "subscribe" {
|
||||
http.Error(w, "invalid hub.mode", http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
if q.Get("hub.verify_token") != wh.verifyToken {
|
||||
slog.Warn("security.facebook_webhook_verify_token_mismatch",
|
||||
"remote_addr", r.RemoteAddr)
|
||||
http.Error(w, "invalid verify token", http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
challenge := q.Get("hub.challenge")
|
||||
// Validate challenge before reflecting to prevent injection if Content-Type changes.
|
||||
if !hubChallengePattern.MatchString(challenge) {
|
||||
http.Error(w, "invalid challenge", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write([]byte(challenge))
|
||||
}
|
||||
|
||||
// handleEvent processes a Facebook webhook event delivery.
|
||||
// Always returns 200 OK — Facebook retries on non-2xx for 24h.
|
||||
func (wh *WebhookHandler) handleEvent(w http.ResponseWriter, r *http.Request) {
|
||||
const maxBodyBytes = 4 << 20 // 4 MB
|
||||
lr := io.LimitReader(r.Body, maxBodyBytes+1)
|
||||
body, err := io.ReadAll(lr)
|
||||
if err != nil {
|
||||
slog.Warn("facebook: webhook read body error", "err", err)
|
||||
w.WriteHeader(http.StatusOK) // 200 so Facebook stops retrying a bad delivery
|
||||
return
|
||||
}
|
||||
if len(body) > maxBodyBytes {
|
||||
slog.Warn("facebook: webhook body exceeded limit, event dropped", "bytes", len(body))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
sig := r.Header.Get("X-Hub-Signature-256")
|
||||
verified := verifySignature(body, sig, wh.appSecret)
|
||||
if !verified {
|
||||
// Try extra secrets (multi-Meta-App deployments share one webhook endpoint).
|
||||
for _, s := range wh.extraSecrets {
|
||||
if verifySignature(body, sig, s) {
|
||||
verified = true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if !verified {
|
||||
slog.Warn("security.facebook_webhook_signature_invalid", "remote_addr", r.RemoteAddr)
|
||||
w.WriteHeader(http.StatusOK) // return 200 so Facebook stops sending
|
||||
return
|
||||
}
|
||||
|
||||
var payload WebhookPayload
|
||||
if err := json.Unmarshal(body, &payload); err != nil {
|
||||
slog.Warn("facebook: webhook parse error", "err", err)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
if payload.Object != "page" {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
for _, entry := range payload.Entry {
|
||||
// Feed events (comments, posts).
|
||||
for _, change := range entry.Changes {
|
||||
if change.Field == "feed" && change.Value.Item == "comment" {
|
||||
if wh.onComment != nil {
|
||||
wh.onComment(entry, change.Value)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Messenger events.
|
||||
for _, event := range entry.Messaging {
|
||||
if wh.onMessage != nil {
|
||||
wh.onMessage(entry, event)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// verifySignature validates the X-Hub-Signature-256 header using HMAC-SHA256.
|
||||
// Facebook sends "sha256=<hex_digest>"; we recompute and compare.
|
||||
func verifySignature(body []byte, signature, appSecret string) bool {
|
||||
const prefix = "sha256="
|
||||
if !strings.HasPrefix(signature, prefix) {
|
||||
return false
|
||||
}
|
||||
expected, err := hex.DecodeString(signature[len(prefix):])
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
mac := hmac.New(sha256.New, []byte(appSecret))
|
||||
mac.Write(body)
|
||||
computed := mac.Sum(nil)
|
||||
return hmac.Equal(computed, expected)
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package facebook
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// webhookRouter routes incoming Facebook webhook events to the correct channel instance by page_id.
|
||||
// A single HTTP handler is shared across all facebook channel instances on the same server.
|
||||
//
|
||||
// Multi-Meta-App note: all page instances registered here are expected to share the same Meta App
|
||||
// (and thus the same app_secret). If instances with different secrets are registered, ServeHTTP
|
||||
// tries all known secrets and accepts the payload if any matches.
|
||||
type webhookRouter struct {
|
||||
mu sync.RWMutex
|
||||
instances map[string]*Channel // pageID → channel
|
||||
routeHandled bool // true after first webhookRoute() call
|
||||
}
|
||||
|
||||
var globalRouter = &webhookRouter{
|
||||
instances: make(map[string]*Channel),
|
||||
}
|
||||
|
||||
func (r *webhookRouter) register(ch *Channel) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.instances[ch.pageID] = ch
|
||||
}
|
||||
|
||||
func (r *webhookRouter) unregister(pageID string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
delete(r.instances, pageID)
|
||||
}
|
||||
|
||||
// webhookRoute returns the path+handler for the first call; ("", nil) for subsequent calls.
|
||||
func (r *webhookRouter) webhookRoute() (string, http.Handler) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if !r.routeHandled {
|
||||
r.routeHandled = true
|
||||
return webhookPath, r
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// ServeHTTP is the shared handler for all Facebook page webhooks.
|
||||
// Routes each entry to the matching channel instance by page_id.
|
||||
//
|
||||
// Multi-Meta-App support: all registered page secrets are collected and tried in order.
|
||||
// A payload is accepted if its signature matches any known app_secret. In the common
|
||||
// case (single Meta App) there is exactly one secret and behavior is unchanged.
|
||||
func (r *webhookRouter) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
||||
r.mu.RLock()
|
||||
var primarySecret, verifyToken string
|
||||
var extraSecrets []string
|
||||
seenSecrets := make(map[string]bool)
|
||||
for _, ch := range r.instances {
|
||||
s := ch.webhookH.appSecret
|
||||
if primarySecret == "" {
|
||||
primarySecret = s
|
||||
verifyToken = ch.webhookH.verifyToken
|
||||
seenSecrets[s] = true
|
||||
} else if !seenSecrets[s] {
|
||||
extraSecrets = append(extraSecrets, s)
|
||||
seenSecrets[s] = true
|
||||
}
|
||||
}
|
||||
r.mu.RUnlock()
|
||||
|
||||
if primarySecret == "" {
|
||||
// No instances registered yet.
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
if len(extraSecrets) > 0 {
|
||||
slog.Warn("security.facebook_multi_meta_app",
|
||||
"extra_app_count", len(extraSecrets),
|
||||
"note", "multiple Meta App secrets registered; payloads verified against all known secrets")
|
||||
}
|
||||
|
||||
routingWH := &WebhookHandler{
|
||||
appSecret: primarySecret,
|
||||
verifyToken: verifyToken,
|
||||
extraSecrets: extraSecrets,
|
||||
}
|
||||
routingWH.onComment = func(entry WebhookEntry, change ChangeValue) {
|
||||
r.mu.RLock()
|
||||
target := r.instances[entry.ID]
|
||||
r.mu.RUnlock()
|
||||
if target != nil {
|
||||
target.handleCommentEvent(entry, change)
|
||||
}
|
||||
}
|
||||
routingWH.onMessage = func(entry WebhookEntry, event MessagingEvent) {
|
||||
r.mu.RLock()
|
||||
target := r.instances[entry.ID]
|
||||
r.mu.RUnlock()
|
||||
if target != nil {
|
||||
target.handleMessagingEvent(entry, event)
|
||||
}
|
||||
}
|
||||
routingWH.ServeHTTP(w, req)
|
||||
}
|
||||
@@ -169,7 +169,7 @@ func (c *Channel) handleMessageEvent(ctx context.Context, event *MessageEvent) {
|
||||
"message_id": messageID,
|
||||
"chat_type": mc.ChatType,
|
||||
"sender_name": senderName,
|
||||
"display_name": senderName,
|
||||
"display_name": channels.SanitizeDisplayName(senderName),
|
||||
"mentioned_bot": fmt.Sprintf("%t", mc.MentionedBot),
|
||||
"platform": channels.TypeFeishu,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
publicAPIBase = "https://pages.fm/api/public_api/v2" // page-level APIs
|
||||
userAPIBase = "https://pages.fm/api/v1" // user-level APIs (list pages, etc.)
|
||||
httpTimeout = 30 * time.Second
|
||||
)
|
||||
|
||||
// APIClient wraps the Pancake REST API for a single page instance.
|
||||
type APIClient struct {
|
||||
pageV1BaseURL string
|
||||
pageV2BaseURL string
|
||||
userBaseURL string
|
||||
pageAccessToken string
|
||||
apiKey string
|
||||
pageID string
|
||||
httpClient *http.Client
|
||||
}
|
||||
|
||||
// NewAPIClient creates a new Pancake APIClient for the given page.
|
||||
func NewAPIClient(apiKey, pageAccessToken, pageID string) *APIClient {
|
||||
return &APIClient{
|
||||
pageV1BaseURL: "https://pages.fm/api/public_api/v1",
|
||||
pageV2BaseURL: publicAPIBase,
|
||||
userBaseURL: userAPIBase,
|
||||
pageAccessToken: pageAccessToken,
|
||||
apiKey: apiKey,
|
||||
pageID: pageID,
|
||||
httpClient: &http.Client{Timeout: httpTimeout},
|
||||
}
|
||||
}
|
||||
|
||||
// VerifyToken validates the page_access_token via a lightweight API call.
|
||||
func (c *APIClient) VerifyToken(ctx context.Context) error {
|
||||
url := fmt.Sprintf("%s/pages/%s/conversations?limit=1", c.pageV2BaseURL, c.pageID)
|
||||
if err := c.doRequest(ctx, http.MethodGet, url, nil); err != nil {
|
||||
return fmt.Errorf("pancake: token verification failed: %w", err)
|
||||
}
|
||||
slog.Info("pancake: page token verified", "page_id", c.pageID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetPage fetches page metadata including platform (facebook/zalo/instagram/tiktok/whatsapp/line).
|
||||
func (c *APIClient) GetPage(ctx context.Context) (*PageInfo, error) {
|
||||
url := fmt.Sprintf("%s/pages", c.userBaseURL)
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pancake: build get-pages request: %w", err)
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+c.apiKey)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
res, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pancake: get pages request failed: %w", err)
|
||||
}
|
||||
defer res.Body.Close()
|
||||
|
||||
body, err := io.ReadAll(res.Body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pancake: read get-pages response: %w", err)
|
||||
}
|
||||
|
||||
var result struct {
|
||||
Data []PageInfo `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &result); err != nil {
|
||||
return nil, fmt.Errorf("pancake: parse get-pages response: %w", err)
|
||||
}
|
||||
|
||||
for i := range result.Data {
|
||||
if result.Data[i].ID == c.pageID {
|
||||
return &result.Data[i], nil
|
||||
}
|
||||
}
|
||||
|
||||
// Page not found in list — return minimal info without platform
|
||||
slog.Warn("pancake: page not found in pages list, platform unknown", "page_id", c.pageID)
|
||||
return &PageInfo{ID: c.pageID}, nil
|
||||
}
|
||||
|
||||
// SendMessage sends a text message to a conversation.
|
||||
func (c *APIClient) SendMessage(ctx context.Context, conversationID, content string) error {
|
||||
body, _ := json.Marshal(SendMessageRequest{
|
||||
Action: "reply_inbox",
|
||||
Message: content,
|
||||
})
|
||||
url := fmt.Sprintf("%s/pages/%s/conversations/%s/messages", c.pageV1BaseURL, c.pageID, conversationID)
|
||||
if err := c.doRequest(ctx, http.MethodPost, url, bytes.NewReader(body)); err != nil {
|
||||
return fmt.Errorf("pancake: send message: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SendAttachmentMessage sends one or more uploaded content IDs to a conversation.
|
||||
func (c *APIClient) SendAttachmentMessage(ctx context.Context, conversationID string, contentIDs []string) error {
|
||||
body, _ := json.Marshal(SendMessageRequest{
|
||||
Action: "reply_inbox",
|
||||
ContentIDs: contentIDs,
|
||||
})
|
||||
url := fmt.Sprintf("%s/pages/%s/conversations/%s/messages", c.pageV1BaseURL, c.pageID, conversationID)
|
||||
if err := c.doRequest(ctx, http.MethodPost, url, bytes.NewReader(body)); err != nil {
|
||||
return fmt.Errorf("pancake: send attachment message: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// UploadMedia uploads a file via multipart/form-data and returns the attachment ID.
|
||||
func (c *APIClient) UploadMedia(ctx context.Context, filename string, data io.Reader, contentType string) (string, error) {
|
||||
var buf bytes.Buffer
|
||||
mw := multipart.NewWriter(&buf)
|
||||
|
||||
fw, err := mw.CreateFormFile("file", filepath.Base(filename))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("pancake: create form file: %w", err)
|
||||
}
|
||||
if _, err := io.Copy(fw, data); err != nil {
|
||||
return "", fmt.Errorf("pancake: copy file data: %w", err)
|
||||
}
|
||||
mw.Close()
|
||||
|
||||
url := fmt.Sprintf("%s/pages/%s/upload_contents", c.pageV1BaseURL, c.pageID)
|
||||
req, err := c.newPageRequest(ctx, http.MethodPost, url, &buf)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("pancake: build upload request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
|
||||
res, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("pancake: upload request failed: %w", err)
|
||||
}
|
||||
defer res.Body.Close()
|
||||
|
||||
respBody, err := io.ReadAll(res.Body)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("pancake: read upload response: %w", err)
|
||||
}
|
||||
|
||||
var uploadResp UploadResponse
|
||||
if err := json.Unmarshal(respBody, &uploadResp); err != nil {
|
||||
return "", fmt.Errorf("pancake: parse upload response: %w", err)
|
||||
}
|
||||
|
||||
if uploadResp.ID == "" {
|
||||
return "", fmt.Errorf("pancake: upload response missing attachment ID")
|
||||
}
|
||||
|
||||
return uploadResp.ID, nil
|
||||
}
|
||||
|
||||
// doRequest executes an authenticated HTTP request using the page_access_token.
|
||||
// Always drains and closes the response body to enable connection reuse.
|
||||
func (c *APIClient) doRequest(ctx context.Context, method, url string, body io.Reader) error {
|
||||
req, err := c.newPageRequest(ctx, method, url, body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
res, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer res.Body.Close()
|
||||
|
||||
// Always read the full body to allow HTTP connection reuse.
|
||||
respBody, _ := io.ReadAll(res.Body)
|
||||
|
||||
if res.StatusCode >= 400 {
|
||||
var apiErr apiError
|
||||
if jsonErr := json.Unmarshal(respBody, &apiErr); jsonErr == nil && apiErr.Message != "" {
|
||||
return &apiErr
|
||||
}
|
||||
return fmt.Errorf("pancake: HTTP %d", res.StatusCode)
|
||||
}
|
||||
|
||||
// Some Pancake endpoints return HTTP 200 with a JSON body carrying success=false.
|
||||
// Treat these as application-level send failures instead of silent success.
|
||||
var appResp struct {
|
||||
Success *bool `json:"success,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
}
|
||||
if err := json.Unmarshal(respBody, &appResp); err == nil && appResp.Success != nil && !*appResp.Success {
|
||||
if appResp.Message != "" {
|
||||
return fmt.Errorf("pancake: %s", appResp.Message)
|
||||
}
|
||||
return fmt.Errorf("pancake: request reported success=false")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *APIClient) newPageRequest(ctx context.Context, method, rawURL string, body io.Reader) (*http.Request, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, method, rawURL, body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
query := req.URL.Query()
|
||||
query.Set("page_access_token", c.pageAccessToken)
|
||||
req.URL.RawQuery = query.Encode()
|
||||
|
||||
// Keep the header for compatibility; official docs require the query token.
|
||||
req.Header.Set("Authorization", "Bearer "+c.pageAccessToken)
|
||||
return req, nil
|
||||
}
|
||||
|
||||
// isAuthError checks if an error is an authentication/authorization failure.
|
||||
// Uses errors.As to handle wrapped errors consistently with Facebook channel pattern.
|
||||
func isAuthError(err error) bool {
|
||||
var ae *apiError
|
||||
if !errors.As(err, &ae) {
|
||||
return false
|
||||
}
|
||||
return ae.Code == 401 || ae.Code == 403 || ae.Code == 4001 || ae.Code == 4003
|
||||
}
|
||||
|
||||
// isRateLimitError checks if an error is a rate limit response.
|
||||
func isRateLimitError(err error) bool {
|
||||
var ae *apiError
|
||||
if !errors.As(err, &ae) {
|
||||
return false
|
||||
}
|
||||
return ae.Code == 429 || ae.Code == 4029
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"html"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
var (
|
||||
htmlBreakTagRe = regexp.MustCompile(`(?i)<br\b[^>]*\/?>`)
|
||||
htmlCloseTagRe = regexp.MustCompile(`(?i)</(?:div|p|li|ul|ol|h[1-6])>`)
|
||||
htmlTagRe = regexp.MustCompile(`(?i)<[^>]+>`)
|
||||
)
|
||||
|
||||
// isDup checks and records a dedup key. Returns true if the key was already seen.
|
||||
func (ch *Channel) isDup(key string) bool {
|
||||
_, loaded := ch.dedup.LoadOrStore(key, time.Now())
|
||||
return loaded
|
||||
}
|
||||
|
||||
// splitMessage splits text into chunks no longer than maxLen Unicode code points.
|
||||
// Operates on runes (not bytes) to avoid splitting multi-byte characters (CJK, emoji, Vietnamese).
|
||||
func splitMessage(text string, maxLen int) []string {
|
||||
runes := []rune(text)
|
||||
if maxLen <= 0 || len(runes) <= maxLen {
|
||||
return []string{text}
|
||||
}
|
||||
var parts []string
|
||||
for len(runes) > maxLen {
|
||||
parts = append(parts, string(runes[:maxLen]))
|
||||
runes = runes[maxLen:]
|
||||
}
|
||||
if len(runes) > 0 {
|
||||
parts = append(parts, string(runes))
|
||||
}
|
||||
return parts
|
||||
}
|
||||
|
||||
func (ch *Channel) forgetOutboundEcho(conversationID, content string) {
|
||||
if conversationID == "" {
|
||||
return
|
||||
}
|
||||
normalized := normalizeEchoContent(content)
|
||||
if normalized == "" {
|
||||
return
|
||||
}
|
||||
ch.recentOutbound.Delete(conversationID + "\x00" + normalized)
|
||||
}
|
||||
|
||||
func (ch *Channel) rememberOutboundEcho(conversationID, content string) {
|
||||
if conversationID == "" {
|
||||
return
|
||||
}
|
||||
normalized := normalizeEchoContent(content)
|
||||
if normalized == "" {
|
||||
return
|
||||
}
|
||||
ch.recentOutbound.Store(conversationID+"\x00"+normalized, time.Now())
|
||||
}
|
||||
|
||||
func (ch *Channel) isRecentOutboundEcho(conversationID, content string) bool {
|
||||
if conversationID == "" {
|
||||
return false
|
||||
}
|
||||
normalized := normalizeEchoContent(content)
|
||||
if normalized == "" {
|
||||
return false
|
||||
}
|
||||
key := conversationID + "\x00" + normalized
|
||||
v, ok := ch.recentOutbound.Load(key)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
ts, ok := v.(time.Time)
|
||||
if !ok {
|
||||
ch.recentOutbound.Delete(key)
|
||||
return false
|
||||
}
|
||||
if time.Since(ts) > outboundEchoTTL {
|
||||
ch.recentOutbound.Delete(key)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func normalizeEchoContent(content string) string {
|
||||
content = strings.TrimSpace(content)
|
||||
if content == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
content = html.UnescapeString(content)
|
||||
content = strings.ReplaceAll(content, "\r\n", "\n")
|
||||
content = strings.ReplaceAll(content, "\r", "\n")
|
||||
content = htmlBreakTagRe.ReplaceAllString(content, "\n")
|
||||
content = htmlCloseTagRe.ReplaceAllString(content, "\n")
|
||||
content = htmlTagRe.ReplaceAllString(content, "")
|
||||
|
||||
lines := strings.Split(content, "\n")
|
||||
normalized := make([]string, 0, len(lines))
|
||||
pendingBlank := false
|
||||
for _, line := range lines {
|
||||
line = strings.Join(strings.Fields(line), " ")
|
||||
if line == "" {
|
||||
if len(normalized) == 0 || pendingBlank {
|
||||
continue
|
||||
}
|
||||
pendingBlank = true
|
||||
continue
|
||||
}
|
||||
if pendingBlank {
|
||||
normalized = append(normalized, "")
|
||||
pendingBlank = false
|
||||
}
|
||||
normalized = append(normalized, line)
|
||||
}
|
||||
|
||||
return strings.TrimSpace(strings.Join(normalized, "\n"))
|
||||
}
|
||||
|
||||
// runDedupCleaner evicts dedup entries older than dedupTTL every dedupCleanEvery.
|
||||
func (ch *Channel) runDedupCleaner() {
|
||||
ticker := time.NewTicker(dedupCleanEvery)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ch.stopCh:
|
||||
return
|
||||
case <-ticker.C:
|
||||
now := time.Now()
|
||||
ch.dedup.Range(func(k, v any) bool {
|
||||
if t, ok := v.(time.Time); ok && now.Sub(t) > dedupTTL {
|
||||
ch.dedup.Delete(k)
|
||||
}
|
||||
return true
|
||||
})
|
||||
ch.recentOutbound.Range(func(k, v any) bool {
|
||||
if t, ok := v.(time.Time); ok && now.Sub(t) > outboundEchoTTL {
|
||||
ch.recentOutbound.Delete(k)
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// FormatOutbound formats agent response text for the target platform.
|
||||
// Each platform has different formatting rules and supported markup.
|
||||
func FormatOutbound(content string, platform string) string {
|
||||
switch platform {
|
||||
case "facebook":
|
||||
return formatForFacebook(content)
|
||||
case "whatsapp":
|
||||
return formatForWhatsApp(content)
|
||||
case "zalo", "instagram", "line":
|
||||
return stripMarkdown(content)
|
||||
case "tiktok":
|
||||
return stripMarkdown(truncateForTikTok(content))
|
||||
default:
|
||||
return stripMarkdown(content)
|
||||
}
|
||||
}
|
||||
|
||||
// formatForFacebook allows basic HTML tags supported by Messenger.
|
||||
// Strips unsupported tags, keeps bold/italic/links.
|
||||
func formatForFacebook(content string) string {
|
||||
// Convert markdown bold (**text** or __text__) to plain (FB Messenger uses plain text)
|
||||
content = reBold.ReplaceAllString(content, "$1")
|
||||
content = reItalic.ReplaceAllString(content, "$1")
|
||||
// Strip markdown code blocks and inline code
|
||||
content = reCodeBlock.ReplaceAllString(content, "$1")
|
||||
content = reInlineCode.ReplaceAllString(content, "$1")
|
||||
// Strip markdown headers (## Heading → Heading)
|
||||
content = reHeader.ReplaceAllString(content, "$1")
|
||||
return strings.TrimSpace(content)
|
||||
}
|
||||
|
||||
// formatForWhatsApp converts markdown to WhatsApp-native formatting.
|
||||
// WhatsApp uses *bold*, _italic_, ~strikethrough~, ```code```.
|
||||
func formatForWhatsApp(content string) string {
|
||||
// Convert **bold** → *bold* (WhatsApp format)
|
||||
content = reDoubleBold.ReplaceAllString(content, "*$1*")
|
||||
// Convert __italic__ → _italic_ (already matches WA format, just clean up __)
|
||||
content = reDoubleUnderline.ReplaceAllString(content, "_$1_")
|
||||
// Strip markdown headers
|
||||
content = reHeader.ReplaceAllString(content, "$1")
|
||||
// Strip inline code backticks (keep content)
|
||||
content = reInlineCode.ReplaceAllString(content, "$1")
|
||||
return strings.TrimSpace(content)
|
||||
}
|
||||
|
||||
// stripMarkdown removes common markdown formatting, returning plain text.
|
||||
func stripMarkdown(content string) string {
|
||||
content = reBold.ReplaceAllString(content, "$1")
|
||||
content = reItalic.ReplaceAllString(content, "$1")
|
||||
content = reCodeBlock.ReplaceAllString(content, "$1")
|
||||
content = reInlineCode.ReplaceAllString(content, "$1")
|
||||
content = reHeader.ReplaceAllString(content, "$1")
|
||||
content = reLink.ReplaceAllString(content, "$1")
|
||||
content = reImage.ReplaceAllString(content, "")
|
||||
return strings.TrimSpace(content)
|
||||
}
|
||||
|
||||
// truncateForTikTok truncates content to TikTok DM limit (500 runes).
|
||||
// Uses rune slicing to avoid corrupting multi-byte UTF-8 (CJK, Vietnamese, emoji).
|
||||
func truncateForTikTok(content string) string {
|
||||
const limit = 500
|
||||
runes := []rune(content)
|
||||
if len(runes) <= limit {
|
||||
return content
|
||||
}
|
||||
return string(runes[:limit-3]) + "..."
|
||||
}
|
||||
|
||||
// Compiled regexes for markdown stripping — package-level for efficiency.
|
||||
var (
|
||||
reBold = regexp.MustCompile(`(?:\*\*|__)(.+?)(?:\*\*|__)`)
|
||||
reDoubleBold = regexp.MustCompile(`\*\*(.+?)\*\*`)
|
||||
reDoubleUnderline = regexp.MustCompile(`__(.+?)__`)
|
||||
reItalic = regexp.MustCompile(`(?:\*|_)(.+?)(?:\*|_)`)
|
||||
reCodeBlock = regexp.MustCompile("(?s)```(?:[a-z]*)?\n?(.+?)```")
|
||||
reInlineCode = regexp.MustCompile("`(.+?)`")
|
||||
reHeader = regexp.MustCompile(`(?m)^#{1,6}\s+(.+)$`)
|
||||
reLink = regexp.MustCompile(`\[([^\]]+)\]\([^)]+\)`)
|
||||
reImage = regexp.MustCompile(`!\[[^\]]*\]\([^)]+\)`)
|
||||
)
|
||||
@@ -0,0 +1,72 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"mime"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
)
|
||||
|
||||
// handleMediaAttachments uploads media files from an OutboundMessage and returns attachment IDs.
|
||||
// Returns an empty slice (not an error) when no media is attached.
|
||||
// On upload failure the error is returned so the caller can decide to send text-only.
|
||||
func (ch *Channel) handleMediaAttachments(ctx context.Context, msg bus.OutboundMessage) ([]string, error) {
|
||||
if len(msg.Media) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
var ids []string
|
||||
for _, att := range msg.Media {
|
||||
if att.URL == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
id, err := ch.uploadMediaFile(ctx, att.URL, att.ContentType)
|
||||
if err != nil {
|
||||
slog.Warn("pancake: skipping media attachment on upload error",
|
||||
"page_id", ch.pageID, "url", att.URL, "err", err)
|
||||
return ids, err
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
// uploadMediaFile opens a local file path and uploads it to the Pancake API.
|
||||
// path must be absolute to prevent directory traversal via relative paths.
|
||||
func (ch *Channel) uploadMediaFile(ctx context.Context, path string, contentType string) (string, error) {
|
||||
cleanPath := filepath.Clean(path)
|
||||
if !filepath.IsAbs(cleanPath) {
|
||||
return "", fmt.Errorf("pancake: media path must be absolute, got: %s", path)
|
||||
}
|
||||
f, err := os.Open(cleanPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
ct := contentType
|
||||
if ct == "" {
|
||||
ct = mimeTypeFromPath(cleanPath)
|
||||
}
|
||||
|
||||
return ch.apiClient.UploadMedia(ctx, filepath.Base(cleanPath), f, ct)
|
||||
}
|
||||
|
||||
// mimeTypeFromPath guesses the MIME type from the file extension.
|
||||
func mimeTypeFromPath(path string) string {
|
||||
ext := filepath.Ext(path)
|
||||
if ext == "" {
|
||||
return "application/octet-stream"
|
||||
}
|
||||
mt := mime.TypeByExtension(ext)
|
||||
if mt == "" {
|
||||
return "application/octet-stream"
|
||||
}
|
||||
return mt
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"html"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
)
|
||||
|
||||
// handleMessagingEvent converts a Pancake "messaging" webhook event to bus.InboundMessage.
|
||||
func (ch *Channel) handleMessagingEvent(data MessagingData) {
|
||||
slog.Debug("pancake: handleMessagingEvent called",
|
||||
"page_id", ch.pageID,
|
||||
"sender_id", data.Message.SenderID,
|
||||
"conversation_id", data.ConversationID,
|
||||
"type", data.Type,
|
||||
"platform", data.Platform,
|
||||
"msg_id", data.Message.ID,
|
||||
"content_len", len(data.Message.Content))
|
||||
|
||||
// Dedup by message ID to handle Pancake's at-least-once delivery.
|
||||
// Skip dedup when message ID is empty — prevents shared slot "msg:" from
|
||||
// silently dropping all subsequent empty-ID messages across conversations.
|
||||
var dedupKey string
|
||||
if data.Message.ID != "" {
|
||||
dedupKey = fmt.Sprintf("msg:%s", data.Message.ID)
|
||||
if ch.isDup(dedupKey) {
|
||||
slog.Info("pancake: duplicate message skipped", "msg_id", data.Message.ID)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Prevent reply loops: skip messages sent by the page itself.
|
||||
if data.Message.SenderID == ch.pageID {
|
||||
slog.Info("pancake: skipping own page message",
|
||||
"page_id", ch.pageID,
|
||||
"sender_id", data.Message.SenderID)
|
||||
return
|
||||
}
|
||||
if isAssignedStaff(data.AssigneeIDs, data.Message.SenderID) {
|
||||
slog.Info("pancake: skipping assigned staff message",
|
||||
"page_id", ch.pageID,
|
||||
"sender_id", data.Message.SenderID,
|
||||
"conversation_id", data.ConversationID)
|
||||
return
|
||||
}
|
||||
|
||||
if data.Message.SenderID == "" {
|
||||
slog.Warn("pancake: message missing sender_id, skipping", "msg_id", data.Message.ID)
|
||||
return
|
||||
}
|
||||
|
||||
// Check echo BEFORE buildMessageContent adds the [From: ...] prefix.
|
||||
// rememberOutboundEcho stores the raw outbound text; the prefix would cause a
|
||||
// key mismatch and silently break loop detection.
|
||||
if ch.isRecentOutboundEcho(data.ConversationID, data.Message.Content) {
|
||||
slog.Info("pancake: skipping recent outbound echo",
|
||||
"page_id", ch.pageID,
|
||||
"conversation_id", data.ConversationID,
|
||||
"msg_id", data.Message.ID)
|
||||
return
|
||||
}
|
||||
|
||||
content := buildMessageContent(data)
|
||||
|
||||
metadata := map[string]string{
|
||||
"pancake_mode": strings.ToLower(data.Type), // "inbox" or "comment"
|
||||
"conversation_type": data.Type,
|
||||
"platform": data.Platform,
|
||||
"conversation_id": data.ConversationID,
|
||||
"message_id": dedupKey,
|
||||
"display_name": channels.SanitizeDisplayName(data.Message.SenderName),
|
||||
"page_name": ch.pageName,
|
||||
}
|
||||
|
||||
ch.HandleMessage(
|
||||
data.Message.SenderID,
|
||||
data.ConversationID, // ChatID = conversation_id for reply routing
|
||||
content,
|
||||
nil, // media handled inline via content URLs
|
||||
metadata,
|
||||
"direct", // Pancake inbox conversations are always treated as direct messages
|
||||
)
|
||||
|
||||
slog.Debug("pancake: inbound message published to bus",
|
||||
"page_id", ch.pageID,
|
||||
"conv_id", data.ConversationID,
|
||||
"sender_id", data.Message.SenderID,
|
||||
"platform", data.Platform,
|
||||
"type", data.Type,
|
||||
"channel_name", ch.Name(),
|
||||
)
|
||||
}
|
||||
|
||||
// buildMessageContent combines text content and attachment URLs into a single string.
|
||||
// Format: [From: {SenderID} ({SenderName})] {content}
|
||||
func buildMessageContent(data MessagingData) string {
|
||||
parts := []string{}
|
||||
|
||||
if data.Message.Content != "" {
|
||||
parts = append(parts, stripHTML(data.Message.Content))
|
||||
}
|
||||
|
||||
for _, att := range data.Message.Attachments {
|
||||
if att.URL != "" {
|
||||
parts = append(parts, att.URL)
|
||||
}
|
||||
}
|
||||
|
||||
body := strings.Join(parts, "\n")
|
||||
|
||||
if data.Message.SenderID != "" {
|
||||
prefix := fmt.Sprintf("[From: %s (%s)]", data.Message.SenderID, data.Message.SenderName)
|
||||
if body != "" {
|
||||
return prefix + " " + body
|
||||
}
|
||||
return prefix
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
// stripHTML removes HTML tags and unescapes HTML entities from s.
|
||||
func stripHTML(s string) string {
|
||||
s = htmlTagRe.ReplaceAllString(s, "")
|
||||
return html.UnescapeString(strings.TrimSpace(s))
|
||||
}
|
||||
|
||||
func isAssignedStaff(assigneeIDs []string, senderID string) bool {
|
||||
if senderID == "" {
|
||||
return false
|
||||
}
|
||||
for _, assigneeID := range assigneeIDs {
|
||||
if assigneeID == senderID {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// Compile-time interface assertions.
|
||||
var (
|
||||
_ channels.Channel = (*Channel)(nil)
|
||||
_ channels.WebhookChannel = (*Channel)(nil)
|
||||
_ channels.BlockReplyChannel = (*Channel)(nil)
|
||||
)
|
||||
|
||||
const (
|
||||
dedupTTL = 24 * time.Hour
|
||||
dedupCleanEvery = 5 * time.Minute
|
||||
outboundEchoTTL = 45 * time.Second
|
||||
)
|
||||
|
||||
// Channel implements channels.Channel and channels.WebhookChannel for Pancake (pages.fm).
|
||||
// One channel instance = one Pancake page, which may serve multiple platforms (FB, Zalo, IG, etc.)
|
||||
type Channel struct {
|
||||
*channels.BaseChannel
|
||||
config pancakeInstanceConfig
|
||||
apiClient *APIClient
|
||||
pageID string
|
||||
pageName string // resolved from Pancake page metadata at Start()
|
||||
platform string // resolved from Pancake page metadata at Start()
|
||||
webhookSecret string // optional HMAC-SHA256 secret for webhook verification
|
||||
|
||||
// dedup prevents processing duplicate webhook deliveries.
|
||||
dedup sync.Map // eventKey(string) → time.Time
|
||||
|
||||
// recentOutbound suppresses short-lived webhook echoes of our own text replies.
|
||||
recentOutbound sync.Map // conversationID + "\x00" + normalized content → time.Time
|
||||
|
||||
stopCh chan struct{}
|
||||
stopCtx context.Context
|
||||
stopFn context.CancelFunc
|
||||
}
|
||||
|
||||
// New creates a Pancake Channel from parsed credentials and config.
|
||||
func New(cfg pancakeInstanceConfig, creds pancakeCreds,
|
||||
msgBus *bus.MessageBus, _ store.PairingStore) (*Channel, error) {
|
||||
|
||||
if creds.APIKey == "" {
|
||||
return nil, fmt.Errorf("pancake: api_key is required")
|
||||
}
|
||||
if creds.PageAccessToken == "" {
|
||||
return nil, fmt.Errorf("pancake: page_access_token is required")
|
||||
}
|
||||
if cfg.PageID == "" {
|
||||
return nil, fmt.Errorf("pancake: page_id is required")
|
||||
}
|
||||
|
||||
base := channels.NewBaseChannel(channels.TypePancake, msgBus, cfg.AllowFrom)
|
||||
stopCtx, stopFn := context.WithCancel(context.Background())
|
||||
|
||||
ch := &Channel{
|
||||
BaseChannel: base,
|
||||
config: cfg,
|
||||
apiClient: NewAPIClient(creds.APIKey, creds.PageAccessToken, cfg.PageID),
|
||||
pageID: cfg.PageID,
|
||||
platform: cfg.Platform,
|
||||
webhookSecret: creds.WebhookSecret,
|
||||
stopCh: make(chan struct{}),
|
||||
stopCtx: stopCtx,
|
||||
stopFn: stopFn,
|
||||
}
|
||||
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
// Factory creates a Pancake Channel from DB instance data.
|
||||
// Implements channels.ChannelFactory.
|
||||
func Factory(name string, creds json.RawMessage, cfg json.RawMessage,
|
||||
msgBus *bus.MessageBus, pairingSvc store.PairingStore) (channels.Channel, error) {
|
||||
|
||||
var c pancakeCreds
|
||||
if err := json.Unmarshal(creds, &c); err != nil {
|
||||
return nil, fmt.Errorf("pancake: decode credentials: %w", err)
|
||||
}
|
||||
|
||||
var ic pancakeInstanceConfig
|
||||
if len(cfg) > 0 {
|
||||
if err := json.Unmarshal(cfg, &ic); err != nil {
|
||||
return nil, fmt.Errorf("pancake: decode config: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
ch, err := New(ic, c, msgBus, pairingSvc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ch.SetName(name)
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
// Start connects the channel: verifies token, resolves platform, registers webhook.
|
||||
func (ch *Channel) Start(ctx context.Context) error {
|
||||
ch.MarkStarting("connecting to Pancake page")
|
||||
|
||||
if err := ch.apiClient.VerifyToken(ctx); err != nil {
|
||||
ch.MarkFailed("token invalid", err.Error(), channels.ChannelFailureKindAuth, false)
|
||||
return err
|
||||
}
|
||||
|
||||
// Resolve platform and page name from page metadata (best-effort — don't fail on this).
|
||||
if ch.platform == "" || ch.pageName == "" {
|
||||
if page, err := ch.apiClient.GetPage(ctx); err != nil {
|
||||
slog.Warn("pancake: could not resolve platform from page metadata", "page_id", ch.pageID, "err", err)
|
||||
} else {
|
||||
if page.Platform != "" {
|
||||
ch.platform = page.Platform
|
||||
}
|
||||
if page.Name != "" {
|
||||
ch.pageName = page.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if ch.webhookSecret == "" {
|
||||
slog.Warn("security.pancake_webhook_no_secret",
|
||||
"page_id", ch.pageID,
|
||||
"note", "webhook_secret not configured; incoming webhook requests will not be authenticated")
|
||||
}
|
||||
|
||||
globalRouter.register(ch)
|
||||
ch.MarkHealthy("connected to page " + ch.pageID)
|
||||
ch.SetRunning(true)
|
||||
|
||||
// Background goroutine: evict stale dedup entries to prevent memory growth.
|
||||
go ch.runDedupCleaner()
|
||||
|
||||
slog.Info("pancake channel started",
|
||||
"page_id", ch.pageID,
|
||||
"platform", ch.platform,
|
||||
"name", ch.Name())
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop gracefully shuts down the channel.
|
||||
func (ch *Channel) Stop(_ context.Context) error {
|
||||
globalRouter.unregister(ch.pageID)
|
||||
ch.stopFn()
|
||||
close(ch.stopCh)
|
||||
ch.SetRunning(false)
|
||||
ch.MarkStopped("stopped")
|
||||
slog.Info("pancake channel stopped", "page_id", ch.pageID, "name", ch.Name())
|
||||
return nil
|
||||
}
|
||||
|
||||
// Send delivers an outbound message via Pancake API.
|
||||
func (ch *Channel) Send(ctx context.Context, msg bus.OutboundMessage) error {
|
||||
slog.Debug("pancake: Send called",
|
||||
"page_id", ch.pageID,
|
||||
"chat_id", msg.ChatID,
|
||||
"content_len", len(msg.Content),
|
||||
"platform", ch.platform,
|
||||
"channel_name", ch.Name())
|
||||
|
||||
conversationID := msg.ChatID
|
||||
if conversationID == "" {
|
||||
return fmt.Errorf("pancake: chat_id (conversation_id) is required for outbound message")
|
||||
}
|
||||
|
||||
text := FormatOutbound(msg.Content, ch.platform)
|
||||
|
||||
// Handle media attachments.
|
||||
attachmentIDs, err := ch.handleMediaAttachments(ctx, msg)
|
||||
if err != nil {
|
||||
slog.Warn("pancake: media upload failed, sending text only",
|
||||
"page_id", ch.pageID, "err", err)
|
||||
}
|
||||
|
||||
// Pancake's official contract makes `message` and `content_ids` mutually exclusive.
|
||||
// Deliver uploaded media first, then follow with text chunks if needed.
|
||||
if len(attachmentIDs) > 0 {
|
||||
if err := ch.apiClient.SendAttachmentMessage(ctx, conversationID, attachmentIDs); err != nil {
|
||||
ch.handleAPIError(err)
|
||||
return err
|
||||
}
|
||||
if text == "" {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// Text-only: split into platform-appropriate chunks.
|
||||
// Store echo fingerprints BEFORE sending so that webhook echoes arriving
|
||||
// while the HTTP round-trip is in flight are already recognized as self-sent.
|
||||
parts := splitMessage(text, ch.maxMessageLength())
|
||||
for _, part := range parts {
|
||||
ch.rememberOutboundEcho(conversationID, part)
|
||||
}
|
||||
for _, part := range parts {
|
||||
if err := ch.apiClient.SendMessage(ctx, conversationID, part); err != nil {
|
||||
ch.handleAPIError(err)
|
||||
// Remove pre-stored echo fingerprints for unsent parts so they
|
||||
// don't suppress genuine customer messages that happen to match.
|
||||
ch.forgetOutboundEcho(conversationID, part)
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// BlockReplyEnabled returns the per-channel block_reply override (nil = inherit gateway default).
|
||||
func (ch *Channel) BlockReplyEnabled() *bool { return ch.config.BlockReply }
|
||||
|
||||
// WebhookHandler returns the shared webhook path and global router as handler.
|
||||
// Only the first pancake instance mounts the route; others return ("", nil).
|
||||
func (ch *Channel) WebhookHandler() (string, http.Handler) {
|
||||
return globalRouter.webhookRoute()
|
||||
}
|
||||
|
||||
// handleAPIError maps Pancake API errors to channel health states.
|
||||
func (ch *Channel) handleAPIError(err error) {
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case isAuthError(err):
|
||||
ch.MarkFailed("token expired or invalid", err.Error(), channels.ChannelFailureKindAuth, false)
|
||||
case isRateLimitError(err):
|
||||
ch.MarkDegraded("rate limited", err.Error(), channels.ChannelFailureKindNetwork, true)
|
||||
default:
|
||||
ch.MarkDegraded("api error", err.Error(), channels.ChannelFailureKindUnknown, true)
|
||||
}
|
||||
}
|
||||
|
||||
// maxMessageLength returns the platform-specific character limit.
|
||||
func (ch *Channel) maxMessageLength() int {
|
||||
switch ch.platform {
|
||||
case "tiktok":
|
||||
return 500
|
||||
case "instagram":
|
||||
return 1000
|
||||
case "facebook", "zalo":
|
||||
return 2000
|
||||
case "whatsapp":
|
||||
return 4096
|
||||
case "line":
|
||||
return 5000
|
||||
default:
|
||||
return 2000
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,396 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
)
|
||||
|
||||
func TestMessageHandlerSkipsRecentOutboundEchoWithHTMLFormatting(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
ch := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
pageID: "page-123",
|
||||
}
|
||||
ch.rememberOutboundEcho("conv-1", "Line 1\nLine 2")
|
||||
|
||||
ch.handleMessagingEvent(MessagingData{
|
||||
PageID: "page-123",
|
||||
ConversationID: "conv-1",
|
||||
Type: "INBOX",
|
||||
Platform: "facebook",
|
||||
Message: MessagingMessage{
|
||||
ID: "msg-echo-html-1",
|
||||
SenderID: "user-1",
|
||||
Content: "<div>Line 1<br key='n_0' />Line 2</div>",
|
||||
},
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
if _, ok := msgBus.ConsumeInbound(ctx); ok {
|
||||
t.Fatal("expected HTML-formatted echo to be dropped")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWebhookRouterSkipsNonInboxConversationEvents(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
target := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
pageID: "page-123",
|
||||
platform: "facebook",
|
||||
}
|
||||
router := &webhookRouter{
|
||||
instances: map[string]*Channel{
|
||||
"page-123": target,
|
||||
},
|
||||
}
|
||||
|
||||
body := `{
|
||||
"page_id": "page-123",
|
||||
"event_type": "messaging",
|
||||
"data": {
|
||||
"conversation": {
|
||||
"id": "page-123_user-1",
|
||||
"type": "COMMENT",
|
||||
"from": {
|
||||
"id": "user-1",
|
||||
"name": "Customer"
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"id": "msg-comment-1",
|
||||
"message": "hi"
|
||||
}
|
||||
}
|
||||
}`
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/channels/pancake/webhook", strings.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
if _, ok := msgBus.ConsumeInbound(ctx); ok {
|
||||
t.Fatal("expected COMMENT conversation webhook to be ignored")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWebhookRouterPrefersMessageSenderOverConversationSender(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
target := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
pageID: "page-123",
|
||||
platform: "facebook",
|
||||
}
|
||||
router := &webhookRouter{
|
||||
instances: map[string]*Channel{
|
||||
"page-123": target,
|
||||
},
|
||||
}
|
||||
|
||||
body := `{
|
||||
"page_id": "page-123",
|
||||
"event_type": "messaging",
|
||||
"data": {
|
||||
"conversation": {
|
||||
"id": "conv-1",
|
||||
"type": "INBOX",
|
||||
"from": {
|
||||
"id": "user-initiator",
|
||||
"name": "Conversation Starter"
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"id": "msg-actual-sender-1",
|
||||
"message": "xin chao",
|
||||
"from": {
|
||||
"id": "user-actual",
|
||||
"name": "Actual Sender",
|
||||
"page_customer_id": "pc-123"
|
||||
}
|
||||
}
|
||||
}
|
||||
}`
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/channels/pancake/webhook", strings.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
msg, ok := msgBus.ConsumeInbound(ctx)
|
||||
if !ok {
|
||||
t.Fatal("expected inbound message to be published")
|
||||
}
|
||||
if got, want := msg.SenderID, "user-actual"; got != want {
|
||||
t.Fatalf("sender_id = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := msg.Metadata["display_name"], "Actual Sender"; got != want {
|
||||
t.Fatalf("metadata.display_name = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWebhookRouterSkipsPageAuthoredInboxReply(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
target := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
pageID: "page-123",
|
||||
platform: "facebook",
|
||||
}
|
||||
router := &webhookRouter{
|
||||
instances: map[string]*Channel{
|
||||
"page-123": target,
|
||||
},
|
||||
}
|
||||
|
||||
body := `{
|
||||
"page_id": "page-123",
|
||||
"event_type": "messaging",
|
||||
"data": {
|
||||
"conversation": {
|
||||
"id": "conv-1",
|
||||
"type": "INBOX",
|
||||
"from": {
|
||||
"id": "user-1",
|
||||
"name": "Customer"
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"id": "msg-page-reply-1",
|
||||
"message": "manual page reply",
|
||||
"from": {
|
||||
"id": "page-123",
|
||||
"name": "Page Bot"
|
||||
}
|
||||
}
|
||||
}
|
||||
}`
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/channels/pancake/webhook", strings.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
if _, ok := msgBus.ConsumeInbound(ctx); ok {
|
||||
t.Fatal("expected page-authored reply not to be published inbound")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWebhookRouterSkipsAssignedStaffReply(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
target := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
pageID: "page-123",
|
||||
platform: "facebook",
|
||||
}
|
||||
router := &webhookRouter{
|
||||
instances: map[string]*Channel{
|
||||
"page-123": target,
|
||||
},
|
||||
}
|
||||
|
||||
body := `{
|
||||
"page_id": "page-123",
|
||||
"event_type": "messaging",
|
||||
"data": {
|
||||
"conversation": {
|
||||
"id": "conv-1",
|
||||
"type": "INBOX",
|
||||
"assignee_ids": ["staff-1"],
|
||||
"from": {
|
||||
"id": "user-1",
|
||||
"name": "Customer"
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"id": "msg-staff-reply-1",
|
||||
"message": "manual staff reply",
|
||||
"from": {
|
||||
"id": "staff-1",
|
||||
"name": "Assigned Staff"
|
||||
}
|
||||
}
|
||||
}
|
||||
}`
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/channels/pancake/webhook", strings.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
if _, ok := msgBus.ConsumeInbound(ctx); ok {
|
||||
t.Fatal("expected assigned staff reply not to be published inbound")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSendThenWebhookEchoDoesNotRepublishInbound(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
api := NewAPIClient("user-token", "page-token", "page-123")
|
||||
sendServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = io.WriteString(w, `{"success":true}`)
|
||||
}))
|
||||
defer sendServer.Close()
|
||||
|
||||
api.pageV1BaseURL = sendServer.URL
|
||||
api.httpClient = sendServer.Client()
|
||||
|
||||
ch := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
apiClient: api,
|
||||
pageID: "page-123",
|
||||
platform: "facebook",
|
||||
}
|
||||
|
||||
if err := ch.Send(context.Background(), bus.OutboundMessage{
|
||||
Channel: "pancake",
|
||||
ChatID: "conv-1",
|
||||
Content: "Line 1\nLine 2",
|
||||
}); err != nil {
|
||||
t.Fatalf("Send returned error: %v", err)
|
||||
}
|
||||
|
||||
router := &webhookRouter{
|
||||
instances: map[string]*Channel{
|
||||
"page-123": ch,
|
||||
},
|
||||
}
|
||||
|
||||
body := `{
|
||||
"page_id": "page-123",
|
||||
"event_type": "messaging",
|
||||
"data": {
|
||||
"conversation": {
|
||||
"id": "conv-1",
|
||||
"type": "INBOX",
|
||||
"from": {
|
||||
"id": "user-1",
|
||||
"name": "Customer"
|
||||
}
|
||||
},
|
||||
"message": {
|
||||
"id": "msg-echo-roundtrip-1",
|
||||
"message": "<div>Line 1<br key='n_0' />Line 2</div>"
|
||||
}
|
||||
}
|
||||
}`
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/channels/pancake/webhook", strings.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
if _, ok := msgBus.ConsumeInbound(ctx); ok {
|
||||
t.Fatal("expected echoed outbound send not to republish inbound work")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSendRaceConditionEchoArrivesBeforeSendReturns simulates the production race:
|
||||
// webhook echo arriving while SendMessage HTTP call is still in flight.
|
||||
// Before the fix, rememberOutboundEcho was called AFTER SendMessage, so an echo
|
||||
// arriving during the HTTP round-trip would not be recognized.
|
||||
func TestSendRaceConditionEchoArrivesBeforeSendReturns(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
api := NewAPIClient("user-token", "page-token", "page-123")
|
||||
|
||||
ch := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
apiClient: api,
|
||||
pageID: "page-123",
|
||||
platform: "facebook",
|
||||
}
|
||||
|
||||
router := &webhookRouter{
|
||||
instances: map[string]*Channel{
|
||||
"page-123": ch,
|
||||
},
|
||||
}
|
||||
|
||||
const outboundContent = "Em đây, alive luôn 😊\nThoát loop thành công rồi nha."
|
||||
|
||||
// Pancake API server that delivers the echo webhook BEFORE returning the
|
||||
// SendMessage response — simulating the real-world race condition.
|
||||
sendServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
// While SendMessage is "in flight", deliver the echo webhook.
|
||||
echoBody := `{
|
||||
"page_id": "page-123",
|
||||
"event_type": "messaging",
|
||||
"data": {
|
||||
"conversation": {
|
||||
"id": "conv-race",
|
||||
"type": "INBOX",
|
||||
"from": {"id": "user-1", "name": "Customer"}
|
||||
},
|
||||
"message": {
|
||||
"id": "msg-race-echo",
|
||||
"message": "<div>Em đây, alive luôn 😊 <br key='n_0' />Thoát loop thành công rồi nha.</div>"
|
||||
}
|
||||
}
|
||||
}`
|
||||
echoReq := httptest.NewRequest(http.MethodPost, "/channels/pancake/webhook",
|
||||
strings.NewReader(echoBody))
|
||||
echoW := httptest.NewRecorder()
|
||||
router.ServeHTTP(echoW, echoReq)
|
||||
|
||||
// Now return SendMessage success.
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = io.WriteString(w, `{"success":true}`)
|
||||
}))
|
||||
defer sendServer.Close()
|
||||
|
||||
api.pageV1BaseURL = sendServer.URL
|
||||
api.httpClient = sendServer.Client()
|
||||
|
||||
if err := ch.Send(context.Background(), bus.OutboundMessage{
|
||||
Channel: "pancake",
|
||||
ChatID: "conv-race",
|
||||
Content: outboundContent,
|
||||
}); err != nil {
|
||||
t.Fatalf("Send returned error: %v", err)
|
||||
}
|
||||
|
||||
// The echo webhook was delivered during Send. It must NOT have been published.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
if _, ok := msgBus.ConsumeInbound(ctx); ok {
|
||||
t.Fatal("echo arrived during SendMessage HTTP call but was not suppressed — race condition not fixed")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,484 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
)
|
||||
|
||||
// TestFactory_Valid verifies Factory creates a Channel from valid JSON creds/config.
|
||||
func TestFactory_Valid(t *testing.T) {
|
||||
creds, _ := json.Marshal(pancakeCreds{
|
||||
APIKey: "test-api-key",
|
||||
PageAccessToken: "test-page-token",
|
||||
})
|
||||
cfg, _ := json.Marshal(pancakeInstanceConfig{PageID: "12345"})
|
||||
|
||||
ch, err := Factory("pancake-test", creds, cfg, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Factory returned unexpected error: %v", err)
|
||||
}
|
||||
if ch == nil {
|
||||
t.Fatal("Factory returned nil channel")
|
||||
}
|
||||
if ch.Name() != "pancake-test" {
|
||||
t.Errorf("Name() = %q, want %q", ch.Name(), "pancake-test")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFactory_MissingAPIKey verifies Factory returns error when api_key is empty.
|
||||
func TestFactory_MissingAPIKey(t *testing.T) {
|
||||
creds, _ := json.Marshal(pancakeCreds{PageAccessToken: "token"})
|
||||
cfg, _ := json.Marshal(pancakeInstanceConfig{PageID: "12345"})
|
||||
|
||||
_, err := Factory("test", creds, cfg, nil, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing api_key, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFactory_MissingPageAccessToken verifies Factory returns error when page_access_token is empty.
|
||||
func TestFactory_MissingPageAccessToken(t *testing.T) {
|
||||
creds, _ := json.Marshal(pancakeCreds{APIKey: "key"})
|
||||
cfg, _ := json.Marshal(pancakeInstanceConfig{PageID: "12345"})
|
||||
|
||||
_, err := Factory("test", creds, cfg, nil, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing page_access_token, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFactory_MissingPageID verifies Factory returns error when page_id is empty.
|
||||
func TestFactory_MissingPageID(t *testing.T) {
|
||||
creds, _ := json.Marshal(pancakeCreds{
|
||||
APIKey: "key",
|
||||
PageAccessToken: "token",
|
||||
})
|
||||
cfg, _ := json.Marshal(pancakeInstanceConfig{}) // no page_id
|
||||
|
||||
_, err := Factory("test", creds, cfg, nil, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing page_id, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFormatOutbound verifies platform-aware formatting for each platform.
|
||||
func TestFormatOutbound(t *testing.T) {
|
||||
input := "**Hello** _world_ `code` ## Header [link](http://example.com)"
|
||||
|
||||
cases := []struct {
|
||||
platform string
|
||||
wantNot string // substring that should NOT appear in output
|
||||
}{
|
||||
{"facebook", "**"},
|
||||
{"zalo", "**"},
|
||||
{"instagram", "_"},
|
||||
{"tiktok", "##"},
|
||||
{"whatsapp", "**"},
|
||||
{"line", "##"},
|
||||
{"unknown", "`"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.platform, func(t *testing.T) {
|
||||
out := FormatOutbound(input, tc.platform)
|
||||
if out == "" {
|
||||
t.Error("FormatOutbound returned empty string")
|
||||
}
|
||||
_ = out // formatting verified visually; we just check no panic + non-empty
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestSplitMessage verifies message splitting at platform character limits.
|
||||
func TestSplitMessage(t *testing.T) {
|
||||
t.Run("short message not split", func(t *testing.T) {
|
||||
parts := splitMessage("hello", 100)
|
||||
if len(parts) != 1 || parts[0] != "hello" {
|
||||
t.Errorf("unexpected parts: %v", parts)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("exact limit not split", func(t *testing.T) {
|
||||
msg := string(make([]byte, 100))
|
||||
parts := splitMessage(msg, 100)
|
||||
if len(parts) != 1 {
|
||||
t.Errorf("expected 1 part, got %d", len(parts))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("over limit is split", func(t *testing.T) {
|
||||
msg := string(make([]byte, 250))
|
||||
parts := splitMessage(msg, 100)
|
||||
if len(parts) != 3 {
|
||||
t.Errorf("expected 3 parts, got %d", len(parts))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("zero limit returns whole string", func(t *testing.T) {
|
||||
parts := splitMessage("hello", 0)
|
||||
if len(parts) != 1 {
|
||||
t.Errorf("expected 1 part with zero limit, got %d", len(parts))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestIsDup verifies dedup returns false first, true on repeat.
|
||||
func TestIsDup(t *testing.T) {
|
||||
ch := &Channel{}
|
||||
|
||||
if ch.isDup("key-1") {
|
||||
t.Error("isDup: first call should return false")
|
||||
}
|
||||
if !ch.isDup("key-1") {
|
||||
t.Error("isDup: second call should return true")
|
||||
}
|
||||
if ch.isDup("key-2") {
|
||||
t.Error("isDup: different key should return false")
|
||||
}
|
||||
}
|
||||
|
||||
// TestWebhookRouterReturns200 verifies the global router always returns HTTP 200.
|
||||
func TestWebhookRouterReturns200(t *testing.T) {
|
||||
// Use a fresh local router to avoid interfering with the package-level globalRouter.
|
||||
router := &webhookRouter{instances: make(map[string]*Channel)}
|
||||
|
||||
t.Run("POST event returns 200", func(t *testing.T) {
|
||||
body := `{"data":{"conversation":{"id":"123_456","type":"INBOX","from":{"id":"456"}},"message":{"id":"m1"}}}`
|
||||
req := httptest.NewRequest(http.MethodPost, "/channels/pancake/webhook",
|
||||
strings.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Errorf("expected 200, got %d", w.Code)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("GET returns 200 (not 405)", func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet, "/channels/pancake/webhook", nil)
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Errorf("expected 200, got %d", w.Code)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("malformed JSON returns 200", func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodPost, "/channels/pancake/webhook",
|
||||
strings.NewReader("not-json"))
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Errorf("expected 200, got %d", w.Code)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestMessageHandlerSkipsSelfReply verifies the page's own messages are not published.
|
||||
// The dedup entry is stored (dedup runs first), but HandleMessage is never called.
|
||||
// If the self-reply guard were absent, ch.bus (nil) would panic — making no-panic the assertion.
|
||||
func TestMessageHandlerSkipsSelfReply(t *testing.T) {
|
||||
const pageID = "page-123"
|
||||
ch := &Channel{pageID: pageID}
|
||||
|
||||
data := MessagingData{
|
||||
PageID: pageID,
|
||||
ConversationID: "conv-1",
|
||||
Type: "INBOX",
|
||||
Platform: "facebook",
|
||||
Message: MessagingMessage{
|
||||
ID: "msg-self-1",
|
||||
SenderID: pageID, // same as page → must be skipped before HandleMessage
|
||||
SenderName: "Page Bot",
|
||||
Content: "Hello",
|
||||
},
|
||||
}
|
||||
|
||||
// Must not panic. If self-reply guard is missing, nil bus dereference panics here.
|
||||
ch.handleMessagingEvent(data)
|
||||
|
||||
// Dedup entry is stored (dedup check runs before self-reply check).
|
||||
_, stored := ch.dedup.Load("msg:msg-self-1")
|
||||
if !stored {
|
||||
t.Error("dedup entry should have been stored (dedup runs before self-reply guard)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageHandlerPublishesMessageIDMetadata(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
ch := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
pageID: "page-123",
|
||||
}
|
||||
|
||||
ch.handleMessagingEvent(MessagingData{
|
||||
PageID: "page-123",
|
||||
ConversationID: "conv-1",
|
||||
Type: "INBOX",
|
||||
Platform: "facebook",
|
||||
Message: MessagingMessage{
|
||||
ID: "msg-123",
|
||||
SenderID: "user-1",
|
||||
Content: "hello",
|
||||
},
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
msg, ok := msgBus.ConsumeInbound(ctx)
|
||||
if !ok {
|
||||
t.Fatal("expected inbound message to be published")
|
||||
}
|
||||
if got, want := msg.Metadata["message_id"], "msg:msg-123"; got != want {
|
||||
t.Fatalf("metadata.message_id = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageHandlerSkipsRecentOutboundEcho(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
ch := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
pageID: "page-123",
|
||||
}
|
||||
ch.rememberOutboundEcho("conv-1", "hello from bot")
|
||||
|
||||
ch.handleMessagingEvent(MessagingData{
|
||||
PageID: "page-123",
|
||||
ConversationID: "conv-1",
|
||||
Type: "INBOX",
|
||||
Platform: "facebook",
|
||||
Message: MessagingMessage{
|
||||
ID: "msg-echo-1",
|
||||
SenderID: "user-1",
|
||||
Content: "hello from bot",
|
||||
},
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
if _, ok := msgBus.ConsumeInbound(ctx); ok {
|
||||
t.Fatal("expected echoed outbound message to be dropped")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlockReplyEnabledUsesChannelOverride(t *testing.T) {
|
||||
enabled := true
|
||||
ch := &Channel{
|
||||
config: pancakeInstanceConfig{
|
||||
BlockReply: &enabled,
|
||||
},
|
||||
}
|
||||
|
||||
got := ch.BlockReplyEnabled()
|
||||
if got == nil || !*got {
|
||||
t.Fatalf("BlockReplyEnabled() = %v, want true", got)
|
||||
}
|
||||
}
|
||||
|
||||
type captureTransport struct {
|
||||
req *http.Request
|
||||
body []byte
|
||||
resp *http.Response
|
||||
}
|
||||
|
||||
func (t *captureTransport) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
t.req = req.Clone(req.Context())
|
||||
if req.Body != nil {
|
||||
body, err := io.ReadAll(req.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
t.body = body
|
||||
req.Body = io.NopCloser(bytes.NewReader(body))
|
||||
}
|
||||
if t.resp != nil {
|
||||
return t.resp, nil
|
||||
}
|
||||
return &http.Response{
|
||||
StatusCode: http.StatusOK,
|
||||
Header: make(http.Header),
|
||||
Body: io.NopCloser(strings.NewReader(`{"success":true}`)),
|
||||
Request: req,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func TestAPIClientSendMessageMatchesOfficialContract(t *testing.T) {
|
||||
transport := &captureTransport{}
|
||||
client := NewAPIClient("user-token", "page-token", "page-123")
|
||||
client.httpClient = &http.Client{Transport: transport}
|
||||
|
||||
if err := client.SendMessage(context.Background(), "conv-456", "xin chao"); err != nil {
|
||||
t.Fatalf("SendMessage returned error: %v", err)
|
||||
}
|
||||
|
||||
if transport.req == nil {
|
||||
t.Fatal("expected outbound request to be captured")
|
||||
}
|
||||
if got, want := transport.req.URL.Path, "/api/public_api/v1/pages/page-123/conversations/conv-456/messages"; got != want {
|
||||
t.Fatalf("request path = %q, want %q", got, want)
|
||||
}
|
||||
if got := transport.req.URL.Query().Get("page_access_token"); got != "page-token" {
|
||||
t.Fatalf("page_access_token query = %q, want %q", got, "page-token")
|
||||
}
|
||||
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(transport.body, &payload); err != nil {
|
||||
t.Fatalf("unmarshal payload: %v", err)
|
||||
}
|
||||
if got, want := payload["action"], "reply_inbox"; got != want {
|
||||
t.Fatalf("payload.action = %#v, want %#v", got, want)
|
||||
}
|
||||
if got, want := payload["message"], "xin chao"; got != want {
|
||||
t.Fatalf("payload.message = %#v, want %#v", got, want)
|
||||
}
|
||||
if _, exists := payload["content"]; exists {
|
||||
t.Fatalf("payload must not contain legacy content field: %s", string(transport.body))
|
||||
}
|
||||
if _, exists := payload["attachment_id"]; exists {
|
||||
t.Fatalf("payload must not contain attachment_id field: %s", string(transport.body))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPIClientSendMessageReturnsBodyLevelError(t *testing.T) {
|
||||
transport := &captureTransport{
|
||||
resp: &http.Response{
|
||||
StatusCode: http.StatusOK,
|
||||
Header: make(http.Header),
|
||||
Body: io.NopCloser(strings.NewReader(`{"success":false,"message":"conversation blocked"}`)),
|
||||
},
|
||||
}
|
||||
client := NewAPIClient("user-token", "page-token", "page-123")
|
||||
client.httpClient = &http.Client{Transport: transport}
|
||||
|
||||
err := client.SendMessage(context.Background(), "conv-456", "xin chao")
|
||||
if err == nil {
|
||||
t.Fatal("expected SendMessage to return body-level error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "conversation blocked") {
|
||||
t.Fatalf("SendMessage error = %v, want body-level message", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTruncateForTikTok_MultiByteCharacters verifies rune-safe truncation for
|
||||
// Vietnamese diacritics and emoji (multi-byte UTF-8 sequences).
|
||||
func TestTruncateForTikTok_MultiByteCharacters(t *testing.T) {
|
||||
// Vietnamese text with diacritics (multi-byte UTF-8)
|
||||
input := strings.Repeat("Xin chào ", 100) // ~900 bytes, <500 runes
|
||||
result := truncateForTikTok(input)
|
||||
if !utf8.ValidString(result) {
|
||||
t.Fatal("truncateForTikTok produced invalid UTF-8")
|
||||
}
|
||||
|
||||
// Emoji string exceeding 500 runes
|
||||
emoji := strings.Repeat("😊", 600)
|
||||
result = truncateForTikTok(emoji)
|
||||
runes := []rune(result)
|
||||
if len(runes) > 500 {
|
||||
t.Errorf("expected <=500 runes, got %d", len(runes))
|
||||
}
|
||||
if !utf8.ValidString(result) {
|
||||
t.Fatal("emoji truncation produced invalid UTF-8")
|
||||
}
|
||||
}
|
||||
|
||||
// TestMessageHandlerEmptyMessageID verifies that two messages with empty IDs
|
||||
// from different conversations are both published (not deduped against each other).
|
||||
func TestMessageHandlerEmptyMessageID(t *testing.T) {
|
||||
msgBus := bus.New()
|
||||
ch := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel(channels.TypePancake, msgBus, nil),
|
||||
pageID: "page-123",
|
||||
}
|
||||
|
||||
// First message with empty ID — should be published
|
||||
ch.handleMessagingEvent(MessagingData{
|
||||
PageID: "page-123", ConversationID: "conv-1",
|
||||
Type: "INBOX", Platform: "facebook",
|
||||
Message: MessagingMessage{ID: "", SenderID: "user-1", Content: "hello"},
|
||||
})
|
||||
|
||||
// Second message with empty ID, different conversation — should ALSO be published
|
||||
ch.handleMessagingEvent(MessagingData{
|
||||
PageID: "page-123", ConversationID: "conv-2",
|
||||
Type: "INBOX", Platform: "facebook",
|
||||
Message: MessagingMessage{ID: "", SenderID: "user-2", Content: "world"},
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, ok1 := msgBus.ConsumeInbound(ctx)
|
||||
if !ok1 {
|
||||
t.Fatal("first empty-ID message should be published")
|
||||
}
|
||||
_, ok2 := msgBus.ConsumeInbound(ctx)
|
||||
if !ok2 {
|
||||
t.Fatal("second empty-ID message should NOT be deduped against first")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPIClientUploadMediaMatchesOfficialContract(t *testing.T) {
|
||||
transport := &captureTransport{
|
||||
resp: &http.Response{
|
||||
StatusCode: http.StatusOK,
|
||||
Header: make(http.Header),
|
||||
Body: io.NopCloser(strings.NewReader(`{"id":"upload-123","success":true}`)),
|
||||
},
|
||||
}
|
||||
client := NewAPIClient("user-token", "page-token", "page-123")
|
||||
client.httpClient = &http.Client{Transport: transport}
|
||||
|
||||
id, err := client.UploadMedia(context.Background(), "photo.jpg", strings.NewReader("file-bytes"), "image/jpeg")
|
||||
if err != nil {
|
||||
t.Fatalf("UploadMedia returned error: %v", err)
|
||||
}
|
||||
if id != "upload-123" {
|
||||
t.Fatalf("UploadMedia id = %q, want %q", id, "upload-123")
|
||||
}
|
||||
if transport.req == nil {
|
||||
t.Fatal("expected upload request to be captured")
|
||||
}
|
||||
if got, want := transport.req.URL.Path, "/api/public_api/v1/pages/page-123/upload_contents"; got != want {
|
||||
t.Fatalf("upload path = %q, want %q", got, want)
|
||||
}
|
||||
if got := transport.req.URL.Query().Get("page_access_token"); got != "page-token" {
|
||||
t.Fatalf("upload page_access_token query = %q, want %q", got, "page-token")
|
||||
}
|
||||
if !strings.HasPrefix(transport.req.Header.Get("Content-Type"), "multipart/form-data; boundary=") {
|
||||
t.Fatalf("upload Content-Type = %q, want multipart/form-data", transport.req.Header.Get("Content-Type"))
|
||||
}
|
||||
}
|
||||
|
||||
// TestIsAuthError_WrappedError verifies errors.As works with wrapped apiError.
|
||||
func TestIsAuthError_WrappedError(t *testing.T) {
|
||||
inner := &apiError{Code: 401, Message: "unauthorized"}
|
||||
wrapped := fmt.Errorf("send failed: %w", inner)
|
||||
if !isAuthError(wrapped) {
|
||||
t.Error("isAuthError should detect wrapped 401 apiError via errors.As")
|
||||
}
|
||||
if isAuthError(fmt.Errorf("random error")) {
|
||||
t.Error("isAuthError should return false for non-apiError")
|
||||
}
|
||||
}
|
||||
|
||||
// TestIsRateLimitError_WrappedError verifies errors.As works with wrapped rate limit.
|
||||
func TestIsRateLimitError_WrappedError(t *testing.T) {
|
||||
inner := &apiError{Code: 429, Message: "too many requests"}
|
||||
wrapped := fmt.Errorf("send failed: %w", inner)
|
||||
if !isRateLimitError(wrapped) {
|
||||
t.Error("isRateLimitError should detect wrapped 429 apiError via errors.As")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
// Package pancake implements the Pancake (pages.fm) channel for GoClaw.
|
||||
// Pancake acts as a unified proxy for Facebook, Zalo OA, Instagram, TikTok, WhatsApp, Line.
|
||||
// A single Pancake API key gives access to all connected platforms — no per-platform OAuth needed.
|
||||
package pancake
|
||||
|
||||
import "encoding/json"
|
||||
|
||||
// pancakeCreds holds encrypted credentials stored in channel_instances.credentials.
|
||||
type pancakeCreds struct {
|
||||
APIKey string `json:"api_key"` // User-level Pancake API key
|
||||
PageAccessToken string `json:"page_access_token"` // Page-level token for all page APIs
|
||||
WebhookSecret string `json:"webhook_secret,omitempty"` // Optional HMAC-SHA256 verification
|
||||
}
|
||||
|
||||
// pancakeInstanceConfig holds non-secret config from channel_instances.config JSONB.
|
||||
type pancakeInstanceConfig struct {
|
||||
PageID string `json:"page_id"`
|
||||
Platform string `json:"platform,omitempty"` // auto-detected at Start(): facebook/zalo/instagram/tiktok/whatsapp/line
|
||||
Features struct {
|
||||
InboxReply bool `json:"inbox_reply"`
|
||||
CommentReply bool `json:"comment_reply"`
|
||||
} `json:"features"`
|
||||
AllowFrom []string `json:"allow_from,omitempty"`
|
||||
BlockReply *bool `json:"block_reply,omitempty"` // override gateway block_reply (nil = inherit)
|
||||
}
|
||||
|
||||
// --- Webhook payload types ---
|
||||
// These types match the actual Pancake (pages.fm) webhook delivery format.
|
||||
// Top-level envelope has optional "event_type" + nested "data" containing
|
||||
// "conversation" and "message" objects.
|
||||
|
||||
// WebhookEvent is the top-level Pancake webhook delivery envelope.
|
||||
type WebhookEvent struct {
|
||||
EventType string `json:"event_type,omitempty"` // "messaging", may be empty
|
||||
PageID string `json:"page_id,omitempty"` // top-level page_id (some formats)
|
||||
Data json.RawMessage `json:"data"`
|
||||
}
|
||||
|
||||
// WebhookData is the "data" envelope inside a Pancake webhook event.
|
||||
type WebhookData struct {
|
||||
Conversation WebhookConversation `json:"conversation"`
|
||||
Message WebhookMessage `json:"message"`
|
||||
PageID string `json:"page_id,omitempty"` // page_id may appear here or top-level
|
||||
}
|
||||
|
||||
// WebhookConversation holds the conversation metadata from a Pancake webhook.
|
||||
type WebhookConversation struct {
|
||||
ID string `json:"id"` // format: "pageID_senderID"
|
||||
Type string `json:"type"` // "INBOX" or "COMMENT"
|
||||
AssigneeIDs []string `json:"assignee_ids,omitempty"` // Pancake staff IDs assigned to this conversation
|
||||
From WebhookSender `json:"from"`
|
||||
Snippet string `json:"snippet,omitempty"`
|
||||
}
|
||||
|
||||
// WebhookSender identifies the message sender.
|
||||
type WebhookSender struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Email string `json:"email,omitempty"`
|
||||
PageCustomerID string `json:"page_customer_id,omitempty"`
|
||||
}
|
||||
|
||||
// WebhookMessage holds the message payload from a Pancake webhook.
|
||||
type WebhookMessage struct {
|
||||
ID string `json:"id"`
|
||||
Message string `json:"message,omitempty"` // primary text content
|
||||
OriginalMessage string `json:"original_message,omitempty"` // unformatted fallback
|
||||
Content string `json:"content,omitempty"` // legacy field
|
||||
From *WebhookSender `json:"from,omitempty"` // actual sender of this message
|
||||
Attachments []MessageAttachment `json:"attachments,omitempty"`
|
||||
CreatedAt json.Number `json:"created_at,omitempty"`
|
||||
}
|
||||
|
||||
// MessageAttachment represents a media attachment in a Pancake webhook message.
|
||||
type MessageAttachment struct {
|
||||
Type string `json:"type"` // "image", "video", "file"
|
||||
URL string `json:"url"`
|
||||
}
|
||||
|
||||
// MessagingData is the normalized internal representation used after parsing.
|
||||
type MessagingData struct {
|
||||
PageID string
|
||||
ConversationID string
|
||||
Type string // "INBOX" or "COMMENT"
|
||||
Platform string // "facebook", "zalo", "instagram", "tiktok", "whatsapp", "line"
|
||||
AssigneeIDs []string
|
||||
Message MessagingMessage
|
||||
}
|
||||
|
||||
// MessagingMessage is the normalized message used by the handler.
|
||||
type MessagingMessage struct {
|
||||
ID string
|
||||
Content string
|
||||
SenderID string
|
||||
SenderName string
|
||||
Attachments []MessageAttachment
|
||||
}
|
||||
|
||||
// --- API response types ---
|
||||
|
||||
// PageInfo holds page metadata from GET /pages response.
|
||||
type PageInfo struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Platform string `json:"platform"` // facebook/zalo/instagram/tiktok/whatsapp/line
|
||||
Avatar string `json:"avatar,omitempty"`
|
||||
}
|
||||
|
||||
// SendMessageRequest is the POST body for sending a message via Pancake API.
|
||||
type SendMessageRequest struct {
|
||||
Action string `json:"action"`
|
||||
Message string `json:"message,omitempty"`
|
||||
ContentIDs []string `json:"content_ids,omitempty"`
|
||||
}
|
||||
|
||||
// UploadResponse is returned by POST /pages/{id}/upload_contents.
|
||||
type UploadResponse struct {
|
||||
ID string `json:"id"`
|
||||
URL string `json:"url,omitempty"`
|
||||
}
|
||||
|
||||
// apiError wraps a Pancake API error response.
|
||||
type apiError struct {
|
||||
Code int `json:"code"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func (e *apiError) Error() string {
|
||||
return e.Message
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
package pancake
|
||||
|
||||
import (
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
const (
|
||||
webhookPath = "/channels/pancake/webhook"
|
||||
maxBodyBytes = 1 << 20 // 1 MB — prevent abuse
|
||||
)
|
||||
|
||||
// verifyHMAC verifies a Pancake HMAC-SHA256 signature.
|
||||
// Expected header format: "sha256=<hex-digest>"
|
||||
func verifyHMAC(body []byte, secret, signature string) bool {
|
||||
const prefix = "sha256="
|
||||
if len(signature) <= len(prefix) {
|
||||
return false
|
||||
}
|
||||
got, err := hex.DecodeString(signature[len(prefix):])
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
mac := hmac.New(sha256.New, []byte(secret))
|
||||
mac.Write(body)
|
||||
expected := mac.Sum(nil)
|
||||
return hmac.Equal(got, expected)
|
||||
}
|
||||
|
||||
// --- Global webhook router for multi-page support ---
|
||||
|
||||
// webhookRouter routes incoming Pancake webhook events to the correct channel instance by page_id.
|
||||
// A single HTTP handler is shared across all pancake channel instances.
|
||||
type webhookRouter struct {
|
||||
mu sync.RWMutex
|
||||
instances map[string]*Channel // pageID → channel
|
||||
routeHandled bool // true after first webhookRoute() call
|
||||
}
|
||||
|
||||
var globalRouter = &webhookRouter{
|
||||
instances: make(map[string]*Channel),
|
||||
}
|
||||
|
||||
func (r *webhookRouter) register(ch *Channel) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.instances[ch.pageID] = ch
|
||||
}
|
||||
|
||||
func (r *webhookRouter) unregister(pageID string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
delete(r.instances, pageID)
|
||||
}
|
||||
|
||||
// webhookRoute returns the path+handler on first call; ("", nil) for subsequent calls.
|
||||
// The HTTP mux retains the route once registered — routeHandled prevents duplicate mounts.
|
||||
func (r *webhookRouter) webhookRoute() (string, http.Handler) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if !r.routeHandled {
|
||||
r.routeHandled = true
|
||||
return webhookPath, r
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// ServeHTTP is the shared handler for all Pancake page webhooks.
|
||||
// Always returns HTTP 200 — Pancake suspends webhooks if >80% errors in a 30-min window.
|
||||
func (r *webhookRouter) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
||||
slog.Info("pancake: webhook request received",
|
||||
"method", req.Method,
|
||||
"remote_addr", req.RemoteAddr,
|
||||
"content_length", req.ContentLength)
|
||||
|
||||
if req.Method != http.MethodPost {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
lr := io.LimitReader(req.Body, maxBodyBytes+1)
|
||||
body, err := io.ReadAll(lr)
|
||||
if err != nil {
|
||||
slog.Warn("pancake: router read body error", "err", err)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
slog.Debug("pancake: webhook body received",
|
||||
"body_len", len(body),
|
||||
"body_preview", truncateBody(body, 1000))
|
||||
|
||||
// Parse top-level envelope.
|
||||
var event WebhookEvent
|
||||
if err := json.Unmarshal(body, &event); err != nil {
|
||||
slog.Warn("pancake: router parse event error", "err", err, "body_preview", truncateBody(body, 300))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
// Parse the nested "data" object containing conversation + message.
|
||||
var data WebhookData
|
||||
if err := json.Unmarshal(event.Data, &data); err != nil {
|
||||
slog.Warn("pancake: router parse data error", "err", err)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
// Resolve page_id: try top-level first, then data-level, then extract from conversation ID.
|
||||
pageID := event.PageID
|
||||
if pageID == "" {
|
||||
pageID = data.PageID
|
||||
}
|
||||
if pageID == "" {
|
||||
// Conversation ID format: "pageID_senderID" — extract page portion.
|
||||
if idx := strings.Index(data.Conversation.ID, "_"); idx > 0 {
|
||||
pageID = data.Conversation.ID[:idx]
|
||||
}
|
||||
}
|
||||
|
||||
// Resolve conversation type — only process INBOX messages.
|
||||
convType := strings.ToUpper(data.Conversation.Type)
|
||||
|
||||
if event.EventType != "" && !strings.EqualFold(event.EventType, "messaging") {
|
||||
slog.Debug("pancake: skipping non-messaging webhook event",
|
||||
"event_type", event.EventType,
|
||||
"page_id", pageID)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
slog.Debug("pancake: webhook event parsed",
|
||||
"event_type", event.EventType,
|
||||
"page_id", pageID,
|
||||
"conv_id", data.Conversation.ID,
|
||||
"conv_type", convType,
|
||||
"sender_id", data.Conversation.From.ID,
|
||||
"sender_name", data.Conversation.From.Name,
|
||||
"msg_id", data.Message.ID)
|
||||
|
||||
if pageID == "" {
|
||||
slog.Warn("pancake: could not determine page_id from webhook payload")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
if convType != "INBOX" {
|
||||
slog.Info("pancake: skipping non-inbox conversation event",
|
||||
"page_id", pageID,
|
||||
"conv_id", data.Conversation.ID,
|
||||
"conv_type", convType,
|
||||
"msg_id", data.Message.ID)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
r.mu.RLock()
|
||||
target := r.instances[pageID]
|
||||
r.mu.RUnlock()
|
||||
|
||||
if target == nil {
|
||||
// Log all registered page IDs for debugging.
|
||||
r.mu.RLock()
|
||||
var registered []string
|
||||
for pid := range r.instances {
|
||||
registered = append(registered, pid)
|
||||
}
|
||||
r.mu.RUnlock()
|
||||
slog.Warn("pancake: no channel instance for page_id",
|
||||
"page_id", pageID,
|
||||
"registered_pages", registered)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
// HMAC signature verification — skip if webhook_secret not configured.
|
||||
if target.webhookSecret != "" {
|
||||
sig := req.Header.Get("X-Pancake-Signature")
|
||||
if !verifyHMAC(body, target.webhookSecret, sig) {
|
||||
slog.Warn("security.pancake_webhook_signature_mismatch",
|
||||
"page_id", pageID,
|
||||
"remote_addr", req.RemoteAddr)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Build normalized MessagingData from actual Pancake payload.
|
||||
msgContent := data.Message.Message
|
||||
if msgContent == "" {
|
||||
msgContent = data.Message.OriginalMessage
|
||||
}
|
||||
if msgContent == "" {
|
||||
msgContent = data.Message.Content
|
||||
}
|
||||
if msgContent == "" {
|
||||
msgContent = data.Conversation.Snippet
|
||||
}
|
||||
|
||||
senderID := data.Conversation.From.ID
|
||||
senderName := data.Conversation.From.Name
|
||||
if data.Message.From != nil {
|
||||
if data.Message.From.ID != "" {
|
||||
senderID = data.Message.From.ID
|
||||
}
|
||||
if data.Message.From.Name != "" {
|
||||
senderName = data.Message.From.Name
|
||||
}
|
||||
}
|
||||
|
||||
normalized := MessagingData{
|
||||
PageID: pageID,
|
||||
ConversationID: data.Conversation.ID,
|
||||
Type: convType,
|
||||
Platform: target.platform,
|
||||
AssigneeIDs: append([]string(nil), data.Conversation.AssigneeIDs...),
|
||||
Message: MessagingMessage{
|
||||
ID: data.Message.ID,
|
||||
Content: msgContent,
|
||||
SenderID: senderID,
|
||||
SenderName: senderName,
|
||||
Attachments: data.Message.Attachments,
|
||||
},
|
||||
}
|
||||
|
||||
target.handleMessagingEvent(normalized)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// truncateBody returns a string preview of body, truncated to maxLen bytes.
|
||||
func truncateBody(body []byte, maxLen int) string {
|
||||
if len(body) <= maxLen {
|
||||
return string(body)
|
||||
}
|
||||
return string(body[:maxLen]) + "..."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
package channels
|
||||
|
||||
import "strings"
|
||||
|
||||
// SanitizeDisplayName strips newlines and markdown heading markers from a
|
||||
// channel-provided display name to prevent injection into markdown templates
|
||||
// (e.g. USER.md). Caps length at 100 runes.
|
||||
func SanitizeDisplayName(name string) string {
|
||||
name = strings.ReplaceAll(name, "\n", " ")
|
||||
name = strings.ReplaceAll(name, "\r", " ")
|
||||
name = strings.ReplaceAll(name, "#", "")
|
||||
name = strings.TrimSpace(name)
|
||||
if len([]rune(name)) > 100 {
|
||||
name = string([]rune(name)[:100])
|
||||
}
|
||||
return name
|
||||
}
|
||||
@@ -251,7 +251,7 @@ func (c *Channel) handleMessage(ev *slackevents.MessageEvent) {
|
||||
"message_id": ev.TimeStamp,
|
||||
"user_id": senderID,
|
||||
"username": displayName,
|
||||
"display_name": displayName,
|
||||
"display_name": channels.SanitizeDisplayName(displayName),
|
||||
"channel_id": channelID,
|
||||
"is_dm": fmt.Sprintf("%t", isDM),
|
||||
"local_key": localKey,
|
||||
|
||||
@@ -70,7 +70,7 @@ func (c *Channel) handleDM(msg protocol.UserMessage) {
|
||||
metadata := map[string]string{
|
||||
"message_id": msg.Data.MsgID,
|
||||
"platform": channels.TypeZaloPersonal,
|
||||
"display_name": senderName,
|
||||
"display_name": channels.SanitizeDisplayName(senderName),
|
||||
}
|
||||
c.HandleMessage(senderID, threadID, content, media, metadata, "direct")
|
||||
}
|
||||
@@ -151,7 +151,7 @@ func (c *Channel) handleGroupMessage(msg protocol.GroupMessage) {
|
||||
"message_id": msg.Data.MsgID,
|
||||
"platform": channels.TypeZaloPersonal,
|
||||
"group_id": threadID,
|
||||
"display_name": senderName,
|
||||
"display_name": channels.SanitizeDisplayName(senderName),
|
||||
}
|
||||
c.HandleMessage(senderID, threadID, finalContent, allMedia, metadata, "group")
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ package config
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -186,6 +187,22 @@ type MemoryConfig struct {
|
||||
VectorWeight float64 `json:"vector_weight,omitempty"` // hybrid search vector weight (default 0.7)
|
||||
TextWeight float64 `json:"text_weight,omitempty"` // hybrid search FTS weight (default 0.3)
|
||||
MinScore float64 `json:"min_score,omitempty"` // minimum relevance score (default 0.35)
|
||||
|
||||
// Dreaming configures the episodic → long-term consolidation worker.
|
||||
// nil = use hardcoded defaults (threshold=5, debounce=10min, enabled).
|
||||
Dreaming *DreamingConfig `json:"dreaming,omitempty"`
|
||||
}
|
||||
|
||||
// DreamingConfig controls per-agent behaviour of the consolidation dreaming
|
||||
// worker (episodic summaries → long-term memory). Pointer fields allow partial
|
||||
// overrides from JSONB to merge cleanly with defaults without clobbering
|
||||
// unset values (e.g. leaving VerboseLog at its default when only Threshold is
|
||||
// overridden).
|
||||
type DreamingConfig struct {
|
||||
Enabled *bool `json:"enabled,omitempty"` // default true (nil = enabled)
|
||||
DebounceMs int `json:"debounce_ms,omitempty"` // min interval between runs per agent/user (default 600000 = 10 min)
|
||||
Threshold int `json:"threshold,omitempty"` // min unpromoted entries before running (default 5)
|
||||
VerboseLog *bool `json:"verbose_log,omitempty"` // log debounce/below-threshold skips at info level (default false)
|
||||
}
|
||||
|
||||
// SandboxConfig configures Docker-based sandbox execution.
|
||||
@@ -301,6 +318,11 @@ type ModelPricing struct {
|
||||
OutputPerMillion float64 `json:"output_per_million"`
|
||||
CacheReadPerMillion float64 `json:"cache_read_per_million,omitempty"`
|
||||
CacheCreatePerMillion float64 `json:"cache_create_per_million,omitempty"`
|
||||
// ReasoningPerMillion is the per-million-token rate for reasoning/thinking tokens
|
||||
// (e.g., Claude extended thinking, GPT-5 reasoning, o3/o4-mini CoT). If zero,
|
||||
// reasoning tokens fall back to OutputPerMillion (providers typically charge
|
||||
// reasoning at the same rate as output tokens).
|
||||
ReasoningPerMillion float64 `json:"reasoning_per_million,omitempty"`
|
||||
}
|
||||
|
||||
// TelemetryConfig configures OpenTelemetry export for traces and spans.
|
||||
@@ -322,6 +344,22 @@ type CronConfig struct {
|
||||
RetryBaseDelay string `json:"retry_base_delay,omitempty"` // initial backoff delay (default "2s", Go duration)
|
||||
RetryMaxDelay string `json:"retry_max_delay,omitempty"` // maximum backoff delay (default "30s", Go duration)
|
||||
DefaultTimezone string `json:"default_timezone,omitempty"` // IANA timezone for cron expressions when not set per-job (e.g. "Asia/Ho_Chi_Minh")
|
||||
JobTimeout string `json:"job_timeout,omitempty"` // max duration per cron job execution (default "10m", Go duration)
|
||||
}
|
||||
|
||||
// DefaultJobTimeout is the fallback timeout for cron job execution.
|
||||
const DefaultJobTimeout = 10 * time.Minute
|
||||
|
||||
// JobTimeoutDuration returns the configured job timeout or the default (10m).
|
||||
func (cc CronConfig) JobTimeoutDuration() time.Duration {
|
||||
if cc.JobTimeout != "" {
|
||||
d, err := time.ParseDuration(cc.JobTimeout)
|
||||
if err == nil && d > 0 {
|
||||
return d
|
||||
}
|
||||
slog.Warn("cron: invalid job_timeout, using default", "value", cc.JobTimeout, "default", DefaultJobTimeout)
|
||||
}
|
||||
return DefaultJobTimeout
|
||||
}
|
||||
|
||||
// ToRetryConfig converts CronConfig to cron.RetryConfig with defaults applied.
|
||||
|
||||
@@ -359,6 +359,8 @@ type GatewayConfig struct {
|
||||
BlockReply *bool `json:"block_reply,omitempty"` // deliver intermediate text during tool iterations (default false)
|
||||
ToolStatus *bool `json:"tool_status,omitempty"` // show tool name in streaming preview during tool execution (default true)
|
||||
TaskRecoveryIntervalSec int `json:"task_recovery_interval_sec,omitempty"` // team task recovery ticker interval in seconds (default 300 = 5min)
|
||||
BackgroundProvider string `json:"background_provider,omitempty"` // LLM provider for background workers (vault enrichment, consolidation)
|
||||
BackgroundModel string `json:"background_model,omitempty"` // LLM model for background workers
|
||||
}
|
||||
|
||||
// ToolsConfig controls tool availability, policy, and web search.
|
||||
|
||||
@@ -52,6 +52,10 @@ func (c *Config) ApplySystemConfigs(configs map[string]string) {
|
||||
boolean("gateway.tool_status", &c.Gateway.ToolStatus)
|
||||
integer("gateway.task_recovery_interval_sec", &c.Gateway.TaskRecoveryIntervalSec)
|
||||
|
||||
// Background workers (vault enrichment, consolidation)
|
||||
str("background.provider", &c.Gateway.BackgroundProvider)
|
||||
str("background.model", &c.Gateway.BackgroundModel)
|
||||
|
||||
// Tools
|
||||
str("tools.profile", &c.Tools.Profile)
|
||||
integer("tools.rate_limit_per_hour", &c.Tools.RateLimitPerHour)
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
package consolidation
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// resolvedDreamingConfig is the fully-populated view of dreaming settings used
|
||||
// inside Handle(). All fields are concrete (no pointers) so the worker can
|
||||
// operate without further nil-checks.
|
||||
type resolvedDreamingConfig struct {
|
||||
Enabled bool
|
||||
Debounce time.Duration
|
||||
Threshold int
|
||||
VerboseLog bool
|
||||
}
|
||||
|
||||
// defaultDreamingConfig returns hardcoded defaults that match the worker's
|
||||
// legacy behaviour before per-agent overrides were added. These values are
|
||||
// mirrored into the struct defaults in workers.go.
|
||||
func defaultDreamingConfig() resolvedDreamingConfig {
|
||||
return resolvedDreamingConfig{
|
||||
Enabled: true,
|
||||
Debounce: dreamingDefaultDebounce,
|
||||
Threshold: dreamingDefaultThreshold,
|
||||
VerboseLog: false,
|
||||
}
|
||||
}
|
||||
|
||||
// mergeDreamingConfig applies an agent-provided override onto the defaults.
|
||||
// Nil pointer fields fall through to defaults; zero integers are treated as
|
||||
// "unset" so a legacy empty JSONB still resolves to defaults. Negative values
|
||||
// are clamped to defaults to defend against malformed config.
|
||||
func mergeDreamingConfig(base resolvedDreamingConfig, override *config.DreamingConfig) resolvedDreamingConfig {
|
||||
if override == nil {
|
||||
return base
|
||||
}
|
||||
if override.Enabled != nil {
|
||||
base.Enabled = *override.Enabled
|
||||
}
|
||||
if override.DebounceMs > 0 {
|
||||
base.Debounce = time.Duration(override.DebounceMs) * time.Millisecond
|
||||
}
|
||||
if override.Threshold > 0 {
|
||||
base.Threshold = override.Threshold
|
||||
}
|
||||
if override.VerboseLog != nil {
|
||||
base.VerboseLog = *override.VerboseLog
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
||||
// DreamingConfigResolver fetches per-agent dreaming config at event handling
|
||||
// time. Implementations should be fast and non-blocking; the consolidation
|
||||
// worker calls this on every event that passes the agent-ID parse.
|
||||
type DreamingConfigResolver func(ctx context.Context, agentID string) *config.DreamingConfig
|
||||
|
||||
// newAgentStoreResolver builds a DreamingConfigResolver that reads from the
|
||||
// given AgentStore and extracts MemoryConfig.Dreaming. Returns nil if the
|
||||
// store is nil so the worker can fall back to defaults.
|
||||
func newAgentStoreResolver(agents store.AgentCRUDStore) DreamingConfigResolver {
|
||||
if agents == nil {
|
||||
return nil
|
||||
}
|
||||
return func(ctx context.Context, agentID string) *config.DreamingConfig {
|
||||
id, err := uuid.Parse(agentID)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
ag, err := agents.GetByIDUnscoped(ctx, id)
|
||||
if err != nil || ag == nil {
|
||||
return nil
|
||||
}
|
||||
mc := ag.ParseMemoryConfig()
|
||||
if mc == nil {
|
||||
return nil
|
||||
}
|
||||
return mc.Dreaming
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
package consolidation
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
func boolPtr(b bool) *bool { return &b }
|
||||
|
||||
func TestMergeDreamingConfigNilOverrideReturnsBase(t *testing.T) {
|
||||
base := defaultDreamingConfig()
|
||||
got := mergeDreamingConfig(base, nil)
|
||||
if got != base {
|
||||
t.Fatalf("nil override mutated base: got %+v, want %+v", got, base)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeDreamingConfigPartialOverride(t *testing.T) {
|
||||
base := defaultDreamingConfig()
|
||||
override := &config.DreamingConfig{
|
||||
Threshold: 3,
|
||||
DebounceMs: 120_000, // 2 min
|
||||
}
|
||||
got := mergeDreamingConfig(base, override)
|
||||
|
||||
if got.Threshold != 3 {
|
||||
t.Errorf("Threshold = %d, want 3", got.Threshold)
|
||||
}
|
||||
if got.Debounce != 2*time.Minute {
|
||||
t.Errorf("Debounce = %v, want 2m", got.Debounce)
|
||||
}
|
||||
// Enabled not set in override → stays at default.
|
||||
if !got.Enabled {
|
||||
t.Errorf("Enabled = false, want default true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeDreamingConfigDisable(t *testing.T) {
|
||||
base := defaultDreamingConfig()
|
||||
override := &config.DreamingConfig{Enabled: boolPtr(false)}
|
||||
got := mergeDreamingConfig(base, override)
|
||||
if got.Enabled {
|
||||
t.Errorf("Enabled = true, want false (override)")
|
||||
}
|
||||
}
|
||||
|
||||
// TestMergeDreamingConfigVerboseLogNilPreservesDefault guards against the
|
||||
// footgun where an override without VerboseLog silently reverts the base
|
||||
// value. Pointer semantics require nil-check, not unconditional assignment.
|
||||
func TestMergeDreamingConfigVerboseLogNilPreservesDefault(t *testing.T) {
|
||||
base := defaultDreamingConfig()
|
||||
base.VerboseLog = true // simulate an operator default of true
|
||||
override := &config.DreamingConfig{Threshold: 2}
|
||||
got := mergeDreamingConfig(base, override)
|
||||
if !got.VerboseLog {
|
||||
t.Errorf("VerboseLog = false, want true (nil override should preserve base)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeDreamingConfigVerboseLogExplicitFalse(t *testing.T) {
|
||||
base := defaultDreamingConfig()
|
||||
base.VerboseLog = true
|
||||
override := &config.DreamingConfig{VerboseLog: boolPtr(false)}
|
||||
got := mergeDreamingConfig(base, override)
|
||||
if got.VerboseLog {
|
||||
t.Errorf("VerboseLog = true, want false (explicit override must apply)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeDreamingConfigZeroFieldsIgnored(t *testing.T) {
|
||||
// Zero values on non-pointer fields must not clobber defaults — this is
|
||||
// critical for agents with an empty memory_config.dreaming JSONB object.
|
||||
base := defaultDreamingConfig()
|
||||
override := &config.DreamingConfig{} // all zero
|
||||
got := mergeDreamingConfig(base, override)
|
||||
if got.Threshold != base.Threshold {
|
||||
t.Errorf("Threshold = %d, want base %d", got.Threshold, base.Threshold)
|
||||
}
|
||||
if got.Debounce != base.Debounce {
|
||||
t.Errorf("Debounce = %v, want base %v", got.Debounce, base.Debounce)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDreamingWorkerHandleHonoursCustomThreshold verifies that a per-agent
|
||||
// resolver lowering the threshold fires even when the worker's global default
|
||||
// would have skipped the event.
|
||||
func TestDreamingWorkerHandleHonoursCustomThreshold(t *testing.T) {
|
||||
summaries := []store.EpisodicSummary{
|
||||
{ID: uuid.New(), Summary: "s1"},
|
||||
{ID: uuid.New(), Summary: "s2"},
|
||||
}
|
||||
mockEpisodic := &mockEpisodicStore{
|
||||
countResult: 2,
|
||||
unpromoted: summaries,
|
||||
promoted: make(map[string]bool),
|
||||
}
|
||||
mockMemory := newMockMemoryStore()
|
||||
mockProvider := &mockProvider{
|
||||
chatResp: &providers.ChatResponse{Content: "ok"},
|
||||
}
|
||||
|
||||
worker := &dreamingWorker{
|
||||
episodicStore: mockEpisodic,
|
||||
memoryStore: mockMemory,
|
||||
provider: mockProvider,
|
||||
model: "test",
|
||||
threshold: 5, // global default says skip at count=2
|
||||
debounce: 1 * time.Second,
|
||||
resolveConfig: func(_ context.Context, _ string) *config.DreamingConfig {
|
||||
return &config.DreamingConfig{Threshold: 2} // per-agent override lowers it
|
||||
},
|
||||
}
|
||||
|
||||
err := worker.Handle(context.Background(), eventbus.DomainEvent{
|
||||
Type: eventbus.EventEpisodicCreated,
|
||||
TenantID: uuid.New().String(),
|
||||
AgentID: "agent-123",
|
||||
UserID: "user-123",
|
||||
Payload: &eventbus.EpisodicCreatedPayload{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Handle failed: %v", err)
|
||||
}
|
||||
if len(mockEpisodic.promoted) != 2 {
|
||||
t.Errorf("Expected 2 promotions with override threshold, got %d", len(mockEpisodic.promoted))
|
||||
}
|
||||
}
|
||||
|
||||
// TestDreamingWorkerHandleDisabledSkips verifies that Enabled=false prevents
|
||||
// any store activity, regardless of threshold/debounce.
|
||||
func TestDreamingWorkerHandleDisabledSkips(t *testing.T) {
|
||||
mockEpisodic := &mockEpisodicStore{
|
||||
countResult: 100, // would normally trigger
|
||||
promoted: make(map[string]bool),
|
||||
}
|
||||
|
||||
worker := &dreamingWorker{
|
||||
episodicStore: mockEpisodic,
|
||||
threshold: 5,
|
||||
debounce: 1 * time.Second,
|
||||
resolveConfig: func(_ context.Context, _ string) *config.DreamingConfig {
|
||||
return &config.DreamingConfig{Enabled: boolPtr(false)}
|
||||
},
|
||||
}
|
||||
|
||||
err := worker.Handle(context.Background(), eventbus.DomainEvent{
|
||||
Type: eventbus.EventEpisodicCreated,
|
||||
TenantID: uuid.New().String(),
|
||||
AgentID: "agent-disabled",
|
||||
UserID: "user-123",
|
||||
Payload: &eventbus.EpisodicCreatedPayload{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Handle failed: %v", err)
|
||||
}
|
||||
if len(mockEpisodic.promoted) > 0 {
|
||||
t.Errorf("Expected 0 promotions when disabled, got %d", len(mockEpisodic.promoted))
|
||||
}
|
||||
// CountUnpromoted must also be skipped — it's a downstream read that
|
||||
// dreaming workers should not perform when an agent opts out.
|
||||
if mockEpisodic.countCalls > 0 {
|
||||
t.Errorf("Expected 0 CountUnpromoted calls when disabled, got %d", mockEpisodic.countCalls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDreamingWorkerHandleNilResolverUsesDefaults verifies backward
|
||||
// compatibility — no resolver means the worker behaves exactly as before.
|
||||
func TestDreamingWorkerHandleNilResolverUsesDefaults(t *testing.T) {
|
||||
mockEpisodic := &mockEpisodicStore{countResult: 2} // below default 5
|
||||
worker := &dreamingWorker{
|
||||
episodicStore: mockEpisodic,
|
||||
threshold: 5,
|
||||
debounce: 1 * time.Second,
|
||||
// resolveConfig: nil
|
||||
}
|
||||
err := worker.Handle(context.Background(), eventbus.DomainEvent{
|
||||
Type: eventbus.EventEpisodicCreated,
|
||||
TenantID: uuid.New().String(),
|
||||
AgentID: "agent-123",
|
||||
UserID: "user-123",
|
||||
Payload: &eventbus.EpisodicCreatedPayload{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Handle failed: %v", err)
|
||||
}
|
||||
// Below threshold — no promotions expected (default behaviour preserved).
|
||||
if len(mockEpisodic.promoted) > 0 {
|
||||
t.Errorf("Expected no promotions below default threshold, got %d", len(mockEpisodic.promoted))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
package consolidation
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// TestDreamingWorkerUsesScoredListing verifies the worker fetches via
|
||||
// ListUnpromotedScored (not the legacy created_at ASC path) so recall-hot
|
||||
// entries synthesise first.
|
||||
func TestDreamingWorkerUsesScoredListing(t *testing.T) {
|
||||
// Two unpromoted entries: one cold-but-old, one hot-recent.
|
||||
now := time.Now().UTC()
|
||||
lastRecall := now.Add(-1 * time.Hour)
|
||||
summaries := []store.EpisodicSummary{
|
||||
{ID: uuid.New(), Summary: "hot", RecallCount: 5, RecallScore: 0.9, LastRecalledAt: &lastRecall, CreatedAt: now.Add(-2 * 24 * time.Hour)},
|
||||
{ID: uuid.New(), Summary: "cold", CreatedAt: now.Add(-5 * 24 * time.Hour)},
|
||||
}
|
||||
// Custom scored-mock flagging whether the correct listing method fired.
|
||||
mockEp := &scoredEpisodicMock{
|
||||
mockEpisodicStore: mockEpisodicStore{
|
||||
countResult: 5,
|
||||
unpromoted: summaries,
|
||||
promoted: make(map[string]bool),
|
||||
},
|
||||
}
|
||||
mockMem := newMockMemoryStore()
|
||||
mockProv := &mockProvider{chatResp: &providers.ChatResponse{Content: "consolidated"}}
|
||||
|
||||
worker := &dreamingWorker{
|
||||
episodicStore: mockEp,
|
||||
memoryStore: mockMem,
|
||||
provider: mockProv,
|
||||
model: "test",
|
||||
threshold: 5,
|
||||
debounce: 1 * time.Second,
|
||||
}
|
||||
|
||||
err := worker.Handle(context.Background(), eventbus.DomainEvent{
|
||||
Type: eventbus.EventEpisodicCreated,
|
||||
TenantID: uuid.New().String(),
|
||||
AgentID: "agent-scored",
|
||||
UserID: "user-scored",
|
||||
Payload: &eventbus.EpisodicCreatedPayload{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Handle failed: %v", err)
|
||||
}
|
||||
if !mockEp.scoredCalled {
|
||||
t.Errorf("worker did not call ListUnpromotedScored (scoring bypass)")
|
||||
}
|
||||
if mockEp.legacyCalled {
|
||||
t.Errorf("worker called legacy ListUnpromoted — should use scored path")
|
||||
}
|
||||
if len(mockEp.promoted) != 2 {
|
||||
t.Errorf("expected 2 entries promoted, got %d", len(mockEp.promoted))
|
||||
}
|
||||
}
|
||||
|
||||
// TestDreamingWorkerFiltersBelowThreshold verifies that even when the agent
|
||||
// has enough unpromoted entries to pass the count check, the recall-score
|
||||
// filter can still skip synthesis if every entry is too weak.
|
||||
func TestDreamingWorkerFiltersBelowThreshold(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
// All entries recalled once (below MinRecallCount=2) with trivial score.
|
||||
summaries := make([]store.EpisodicSummary, 5)
|
||||
for i := range summaries {
|
||||
summaries[i] = store.EpisodicSummary{
|
||||
ID: uuid.New(),
|
||||
Summary: "weak",
|
||||
CreatedAt: now.Add(-30 * 24 * time.Hour),
|
||||
RecallCount: 1,
|
||||
RecallScore: 0.1,
|
||||
}
|
||||
}
|
||||
mockEp := &scoredEpisodicMock{
|
||||
mockEpisodicStore: mockEpisodicStore{
|
||||
countResult: 5,
|
||||
unpromoted: summaries,
|
||||
promoted: make(map[string]bool),
|
||||
},
|
||||
}
|
||||
mockMem := newMockMemoryStore()
|
||||
mockProv := &mockProvider{chatResp: &providers.ChatResponse{Content: "noop"}}
|
||||
|
||||
worker := &dreamingWorker{
|
||||
episodicStore: mockEp,
|
||||
memoryStore: mockMem,
|
||||
provider: mockProv,
|
||||
model: "test",
|
||||
threshold: 5,
|
||||
debounce: 1 * time.Second,
|
||||
}
|
||||
|
||||
err := worker.Handle(context.Background(), eventbus.DomainEvent{
|
||||
Type: eventbus.EventEpisodicCreated,
|
||||
TenantID: uuid.New().String(),
|
||||
AgentID: "agent-filter",
|
||||
UserID: "user-filter",
|
||||
Payload: &eventbus.EpisodicCreatedPayload{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Handle failed: %v", err)
|
||||
}
|
||||
if len(mockEp.promoted) > 0 {
|
||||
t.Errorf("expected all entries filtered, got %d promoted", len(mockEp.promoted))
|
||||
}
|
||||
if len(mockMem.docs) > 0 {
|
||||
t.Errorf("expected no synthesis document, got %d", len(mockMem.docs))
|
||||
}
|
||||
}
|
||||
|
||||
// TestDreamingWorkerFilterEmptyStampsDebounce guards against the starvation
|
||||
// loop: when all entries fail the recall threshold filter we still bump
|
||||
// lastRun, otherwise every subsequent episodic.created event re-runs the
|
||||
// full scoring pipeline in a tight loop. See code review note P10.1.
|
||||
func TestDreamingWorkerFilterEmptyStampsDebounce(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
summaries := make([]store.EpisodicSummary, 5)
|
||||
for i := range summaries {
|
||||
summaries[i] = store.EpisodicSummary{
|
||||
ID: uuid.New(),
|
||||
Summary: "weak",
|
||||
CreatedAt: now.Add(-30 * 24 * time.Hour),
|
||||
RecallCount: 1, // below MinRecallCount=2 → filtered
|
||||
RecallScore: 0.05,
|
||||
}
|
||||
}
|
||||
mockEp := &scoredEpisodicMock{
|
||||
mockEpisodicStore: mockEpisodicStore{
|
||||
countResult: 5,
|
||||
unpromoted: summaries,
|
||||
promoted: make(map[string]bool),
|
||||
},
|
||||
}
|
||||
worker := &dreamingWorker{
|
||||
episodicStore: mockEp,
|
||||
memoryStore: newMockMemoryStore(),
|
||||
provider: &mockProvider{chatResp: &providers.ChatResponse{Content: "noop"}},
|
||||
model: "test",
|
||||
threshold: 5,
|
||||
debounce: 10 * time.Minute, // realistic
|
||||
}
|
||||
|
||||
ev := eventbus.DomainEvent{
|
||||
Type: eventbus.EventEpisodicCreated,
|
||||
TenantID: uuid.New().String(),
|
||||
AgentID: "agent-loop",
|
||||
UserID: "user-loop",
|
||||
Payload: &eventbus.EpisodicCreatedPayload{},
|
||||
}
|
||||
|
||||
// First event: filter drops everything → no promotions.
|
||||
if err := worker.Handle(context.Background(), ev); err != nil {
|
||||
t.Fatalf("first Handle: %v", err)
|
||||
}
|
||||
firstCountCalls := mockEp.countCalls
|
||||
|
||||
// Second event immediately after: debounce should skip BEFORE CountUnpromoted.
|
||||
if err := worker.Handle(context.Background(), ev); err != nil {
|
||||
t.Fatalf("second Handle: %v", err)
|
||||
}
|
||||
if mockEp.countCalls != firstCountCalls {
|
||||
t.Errorf("debounce not stamped on filter-empty skip — CountUnpromoted called %d times, want %d",
|
||||
mockEp.countCalls, firstCountCalls)
|
||||
}
|
||||
}
|
||||
|
||||
// scoredEpisodicMock wraps mockEpisodicStore with tracking flags for the
|
||||
// two ListUnpromoted* methods so tests can assert which path the worker used.
|
||||
type scoredEpisodicMock struct {
|
||||
mockEpisodicStore
|
||||
scoredCalled bool
|
||||
legacyCalled bool
|
||||
}
|
||||
|
||||
func (m *scoredEpisodicMock) ListUnpromoted(ctx context.Context, agentID, userID string, limit int) ([]store.EpisodicSummary, error) {
|
||||
m.legacyCalled = true
|
||||
return m.mockEpisodicStore.ListUnpromoted(ctx, agentID, userID, limit)
|
||||
}
|
||||
|
||||
func (m *scoredEpisodicMock) ListUnpromotedScored(ctx context.Context, agentID, userID string, limit int) ([]store.EpisodicSummary, error) {
|
||||
m.scoredCalled = true
|
||||
return m.mockEpisodicStore.ListUnpromotedScored(ctx, agentID, userID, limit)
|
||||
}
|
||||
@@ -27,10 +27,56 @@ type dreamingWorker struct {
|
||||
episodicStore store.EpisodicStore
|
||||
memoryStore store.MemoryStore
|
||||
provider providers.Provider
|
||||
model string // LLM model for synthesis
|
||||
threshold int // min unpromoted entries before running
|
||||
debounce time.Duration // min interval between runs per agent/user
|
||||
lastRun sync.Map // key: "agentID:userID" → time.Time
|
||||
model string // LLM model for synthesis
|
||||
|
||||
// threshold/debounce are the global defaults. Per-agent overrides come
|
||||
// from resolveConfig which reads the agent's MemoryConfig.Dreaming JSONB.
|
||||
threshold int
|
||||
debounce time.Duration
|
||||
resolveConfig DreamingConfigResolver
|
||||
|
||||
lastRun sync.Map // key: "agentID:userID" → time.Time
|
||||
}
|
||||
|
||||
// formatEntryForSynthesis renders a single episodic entry with recall
|
||||
// metadata for the LLM synthesis prompt. Entries with recall signal are
|
||||
// tagged so the LLM can weight them higher; unrecalled entries pass through
|
||||
// as plain summaries.
|
||||
func formatEntryForSynthesis(e store.EpisodicSummary) string {
|
||||
if e.RecallCount == 0 {
|
||||
return e.Summary
|
||||
}
|
||||
lastRecall := "never"
|
||||
if e.LastRecalledAt != nil {
|
||||
lastRecall = e.LastRecalledAt.Format("Jan 2")
|
||||
}
|
||||
return fmt.Sprintf("[recalled %dx, last: %s] %s", e.RecallCount, lastRecall, e.Summary)
|
||||
}
|
||||
|
||||
// logSkip emits dreaming skip reasons at debug level by default, elevating
|
||||
// to info when verbose logging is enabled via per-agent config.
|
||||
func logSkip(verbose bool, msg string, args ...any) {
|
||||
if verbose {
|
||||
slog.Info(msg, args...)
|
||||
return
|
||||
}
|
||||
slog.Debug(msg, args...)
|
||||
}
|
||||
|
||||
// effectiveConfig merges the worker's defaults with any per-agent override.
|
||||
// Centralised so tests can reason about the precedence in one place.
|
||||
func (w *dreamingWorker) effectiveConfig(ctx context.Context, agentID string) resolvedDreamingConfig {
|
||||
base := defaultDreamingConfig()
|
||||
if w.threshold > 0 {
|
||||
base.Threshold = w.threshold
|
||||
}
|
||||
if w.debounce > 0 {
|
||||
base.Debounce = w.debounce
|
||||
}
|
||||
if w.resolveConfig == nil {
|
||||
return base
|
||||
}
|
||||
return mergeDreamingConfig(base, w.resolveConfig(ctx, agentID))
|
||||
}
|
||||
|
||||
// Handle processes an episodic.created event for the dreaming pipeline.
|
||||
@@ -45,41 +91,55 @@ func (w *dreamingWorker) Handle(ctx context.Context, event eventbus.DomainEvent)
|
||||
agentID := event.AgentID
|
||||
userID := event.UserID
|
||||
|
||||
// Resolve per-agent config (threshold/debounce/enabled). Falls back to
|
||||
// struct defaults when no override is stored on the agent.
|
||||
cfg := w.effectiveConfig(ctx, agentID)
|
||||
if !cfg.Enabled {
|
||||
if cfg.VerboseLog {
|
||||
slog.Info("dreaming: disabled for agent", "agent", agentID, "user", userID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Debounce: skip if ran recently for this pair.
|
||||
key := agentID + ":" + userID
|
||||
debounce := w.debounce
|
||||
if debounce == 0 {
|
||||
debounce = dreamingDefaultDebounce
|
||||
}
|
||||
if v, ok := w.lastRun.Load(key); ok {
|
||||
if last, ok := v.(time.Time); ok && time.Since(last) < debounce {
|
||||
slog.Debug("dreaming: debounce skip", "agent", agentID, "user", userID)
|
||||
if last, ok := v.(time.Time); ok && time.Since(last) < cfg.Debounce {
|
||||
logSkip(cfg.VerboseLog, "dreaming: debounce skip", "agent", agentID, "user", userID)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// Count unpromoted entries; skip if below threshold.
|
||||
threshold := w.threshold
|
||||
if threshold <= 0 {
|
||||
threshold = dreamingDefaultThreshold
|
||||
}
|
||||
count, err := w.episodicStore.CountUnpromoted(ctx, agentID, userID)
|
||||
if err != nil {
|
||||
slog.Warn("dreaming: count unpromoted failed", "err", err, "agent", agentID)
|
||||
return nil
|
||||
}
|
||||
if count < threshold {
|
||||
slog.Debug("dreaming: below threshold", "count", count, "threshold", threshold, "agent", agentID)
|
||||
if count < cfg.Threshold {
|
||||
logSkip(cfg.VerboseLog, "dreaming: below threshold", "count", count, "threshold", cfg.Threshold, "agent", agentID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Fetch unpromoted entries.
|
||||
entries, err := w.episodicStore.ListUnpromoted(ctx, agentID, userID, dreamingFetchLimit)
|
||||
// Fetch unpromoted entries ordered by recall_score DESC (Phase 10). Falls
|
||||
// back to created_at ASC for ties so agents with no recall history still
|
||||
// get oldest-first behaviour.
|
||||
entries, err := w.episodicStore.ListUnpromotedScored(ctx, agentID, userID, dreamingFetchLimit)
|
||||
if err != nil {
|
||||
slog.Warn("dreaming: list unpromoted failed", "err", err, "agent", agentID)
|
||||
return nil
|
||||
}
|
||||
// Filter by recall-score thresholds before synthesis so weak entries
|
||||
// don't burn LLM tokens. Never-recalled entries bypass the count minimum
|
||||
// via the freshness component in ComputeRecallScore.
|
||||
entries = filterByRecallThresholds(entries, defaultRecallThresholds(), time.Now().UTC())
|
||||
if len(entries) == 0 {
|
||||
// Stamp lastRun even on empty-filter skips — otherwise every subsequent
|
||||
// episodic.created event re-runs CountUnpromoted + ListUnpromotedScored
|
||||
// + filter in a tight loop until the agent accumulates fresh content.
|
||||
// See code review note P10.1.
|
||||
w.lastRun.Store(key, time.Now())
|
||||
logSkip(cfg.VerboseLog, "dreaming: all entries below recall thresholds", "agent", agentID, "user", userID)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -119,10 +179,12 @@ func (w *dreamingWorker) Handle(ctx context.Context, event eventbus.DomainEvent)
|
||||
}
|
||||
|
||||
// synthesize calls the LLM to extract long-term facts from session summaries.
|
||||
// Each entry is annotated with its recall metadata so the LLM can weight
|
||||
// frequently-recalled memories higher during synthesis.
|
||||
func (w *dreamingWorker) synthesize(ctx context.Context, entries []store.EpisodicSummary) (string, error) {
|
||||
summaries := make([]string, len(entries))
|
||||
for i, e := range entries {
|
||||
summaries[i] = e.Summary
|
||||
summaries[i] = formatEntryForSynthesis(e)
|
||||
}
|
||||
body := strings.Join(summaries, "\n---\n")
|
||||
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
package consolidation
|
||||
|
||||
import (
|
||||
"math"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// Scoring constants. Chosen to match the plan's 4-component formula (plan has
|
||||
// 6 components but GoClaw lacks per-episode conceptual + consolidation data).
|
||||
//
|
||||
// score = 0.30*frequency + 0.35*relevance + 0.20*recency + 0.15*freshness
|
||||
//
|
||||
// halfLifeDays=14 matches typical short-term memory retention windows; tuning
|
||||
// should be informed by operator feedback once real recall data accumulates.
|
||||
const (
|
||||
recallWeightFrequency = 0.30
|
||||
recallWeightRelevance = 0.35
|
||||
recallWeightRecency = 0.20
|
||||
recallWeightFreshness = 0.15
|
||||
recallHalfLifeDays = 14.0
|
||||
// frequencyCap: recall_count above this yields max frequency component
|
||||
// (avoids runaway-frequency bias for hot entries).
|
||||
recallFrequencyCap = 10
|
||||
)
|
||||
|
||||
// recallThresholds holds the minimum scoring cutoffs applied to
|
||||
// ListUnpromotedScored results before LLM synthesis. Chosen well below the
|
||||
// TS reference (0.75 / 3) because GoClaw boots with zero recall data —
|
||||
// early-life agents must still reach synthesis via the freshness component.
|
||||
type recallThresholds struct {
|
||||
MinScore float64
|
||||
MinRecallCount int
|
||||
}
|
||||
|
||||
func defaultRecallThresholds() recallThresholds {
|
||||
return recallThresholds{MinScore: 0.2, MinRecallCount: 2}
|
||||
}
|
||||
|
||||
// ComputeRecallScore computes the weighted recall score for an episodic entry.
|
||||
// Pure function — no IO, no randomness — so the dreaming worker can compute
|
||||
// scores on demand instead of materialising them into the DB on every event.
|
||||
//
|
||||
// Returned value is in [0, 1+ε]. In practice results cluster in [0, 0.8];
|
||||
// thresholds in defaultRecallThresholds are calibrated to that range.
|
||||
func ComputeRecallScore(entry store.EpisodicSummary, now time.Time) float64 {
|
||||
// 1. Frequency (30%): log-normalised recall count. Log1p smooths the
|
||||
// curve so 0 → 0, 1 → 0.29, 10 → 1.0; anything above cap stays at 1.
|
||||
freq := math.Log1p(float64(entry.RecallCount)) / math.Log1p(recallFrequencyCap)
|
||||
if freq > 1.0 {
|
||||
freq = 1.0
|
||||
}
|
||||
|
||||
// 2. Relevance (35%): running average of memory_search hit scores.
|
||||
// Already lives in recall_score column and is clamped server-side to [0,1]
|
||||
// by RecordRecall callers. Defensive clamp here for robustness.
|
||||
rel := entry.RecallScore
|
||||
if rel < 0 {
|
||||
rel = 0
|
||||
} else if rel > 1 {
|
||||
rel = 1
|
||||
}
|
||||
|
||||
// Exponential decay shared by recency + freshness components.
|
||||
// lambda = ln(2) / halfLife → score halves every `recallHalfLifeDays` days.
|
||||
lambda := math.Ln2 / recallHalfLifeDays
|
||||
|
||||
// 3. Recency (20%): decay from LastRecalledAt (falls back to CreatedAt
|
||||
// for entries that have never been searched). Entries recalled today
|
||||
// score ~1.0; 14 days ago → 0.5; 28 days ago → 0.25.
|
||||
lastRef := entry.CreatedAt
|
||||
if entry.LastRecalledAt != nil {
|
||||
lastRef = *entry.LastRecalledAt
|
||||
}
|
||||
recency := decayFrom(lastRef, now, lambda)
|
||||
|
||||
// 4. Freshness (15%): decay from CreatedAt regardless of recall history.
|
||||
// Keeps brand-new entries eligible even with zero recall — prevents cold
|
||||
// start from starving the synthesis pipeline.
|
||||
freshness := decayFrom(entry.CreatedAt, now, lambda)
|
||||
|
||||
return recallWeightFrequency*freq +
|
||||
recallWeightRelevance*rel +
|
||||
recallWeightRecency*recency +
|
||||
recallWeightFreshness*freshness
|
||||
}
|
||||
|
||||
// decayFrom returns exp(-lambda * ageDays). Negative ages (clock skew,
|
||||
// future-dated rows) clamp to 1.0 so the test fixtures are predictable.
|
||||
func decayFrom(then, now time.Time, lambda float64) float64 {
|
||||
if then.IsZero() || !then.Before(now) {
|
||||
return 1.0
|
||||
}
|
||||
ageDays := now.Sub(then).Hours() / 24
|
||||
return math.Exp(-lambda * ageDays)
|
||||
}
|
||||
|
||||
// filterByRecallThresholds drops entries whose recall signal is too weak
|
||||
// to justify LLM synthesis. Pure function so the dreaming worker can
|
||||
// sort-then-filter in a single pass.
|
||||
//
|
||||
// Bootstrap-friendly rules:
|
||||
// - Never-recalled entries (RecallCount == 0) bypass BOTH thresholds.
|
||||
// Freshness + recency keep them eligible so new agents can synthesise
|
||||
// their earliest sessions without any memory_search activity.
|
||||
// - Entries that HAVE been recalled must clear both MinRecallCount and
|
||||
// MinScore. If an entry has seen 1 weak hit, it's filtered.
|
||||
func filterByRecallThresholds(entries []store.EpisodicSummary, th recallThresholds, now time.Time) []store.EpisodicSummary {
|
||||
if len(entries) == 0 {
|
||||
return entries
|
||||
}
|
||||
kept := entries[:0]
|
||||
for _, e := range entries {
|
||||
if e.RecallCount == 0 {
|
||||
kept = append(kept, e)
|
||||
continue
|
||||
}
|
||||
if e.RecallCount < th.MinRecallCount {
|
||||
continue
|
||||
}
|
||||
if ComputeRecallScore(e, now) < th.MinScore {
|
||||
continue
|
||||
}
|
||||
kept = append(kept, e)
|
||||
}
|
||||
return kept
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
package consolidation
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// floatEq checks two floats are equal within epsilon for the scoring assertions.
|
||||
func floatEq(a, b, eps float64) bool { return math.Abs(a-b) < eps }
|
||||
|
||||
func TestComputeRecallScoreZeroRecallFreshEntry(t *testing.T) {
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
entry := store.EpisodicSummary{
|
||||
ID: uuid.New(),
|
||||
CreatedAt: now, // brand new, zero recalls
|
||||
}
|
||||
// freq=0, rel=0, recency=1.0, freshness=1.0
|
||||
// 0.30*0 + 0.35*0 + 0.20*1 + 0.15*1 = 0.35
|
||||
score := ComputeRecallScore(entry, now)
|
||||
if !floatEq(score, 0.35, 0.001) {
|
||||
t.Errorf("score = %f, want ~0.35", score)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComputeRecallScoreOldUnrecalled(t *testing.T) {
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
entry := store.EpisodicSummary{
|
||||
CreatedAt: now.AddDate(0, 0, -28), // 28 days old, never recalled
|
||||
}
|
||||
// recency = freshness = exp(-ln2/14 * 28) = exp(-ln2*2) = 0.25
|
||||
// score = 0 + 0 + 0.2*0.25 + 0.15*0.25 = 0.0875
|
||||
score := ComputeRecallScore(entry, now)
|
||||
if !floatEq(score, 0.0875, 0.001) {
|
||||
t.Errorf("score = %f, want ~0.0875", score)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComputeRecallScoreExponentialDecayHalfLife(t *testing.T) {
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
// Entry recalled exactly 14 days ago (one half-life). Recency should halve.
|
||||
lastRecall := now.AddDate(0, 0, -14)
|
||||
entry := store.EpisodicSummary{
|
||||
CreatedAt: lastRecall, // created at same moment it was last recalled
|
||||
RecallCount: 1,
|
||||
RecallScore: 1.0,
|
||||
LastRecalledAt: &lastRecall,
|
||||
}
|
||||
// freq = log1p(1) / log1p(10) = 0.693/2.398 ≈ 0.289
|
||||
// rel = 1.0
|
||||
// recency = freshness = exp(-ln2) = 0.5
|
||||
// score = 0.30*0.289 + 0.35*1.0 + 0.20*0.5 + 0.15*0.5 = 0.0867 + 0.35 + 0.175 = 0.6117
|
||||
score := ComputeRecallScore(entry, now)
|
||||
if !floatEq(score, 0.6117, 0.001) {
|
||||
t.Errorf("score = %f, want ~0.6117", score)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComputeRecallScoreHighRecallSignal(t *testing.T) {
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
entry := store.EpisodicSummary{
|
||||
CreatedAt: now.AddDate(0, 0, -3), // 3 days old
|
||||
RecallCount: 10,
|
||||
RecallScore: 0.9,
|
||||
LastRecalledAt: &now, // recalled today
|
||||
}
|
||||
// freq ≈ 1.0 (cap)
|
||||
// rel = 0.9
|
||||
// recency ≈ 1.0 (lastRef=now)
|
||||
// freshness = exp(-ln2/14 * 3) ≈ 0.862
|
||||
// score ≈ 0.30 + 0.315 + 0.20 + 0.1293 ≈ 0.944
|
||||
score := ComputeRecallScore(entry, now)
|
||||
if score < 0.9 {
|
||||
t.Errorf("score = %f, want >0.9 (hot entry)", score)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComputeRecallScoreClampsRelevance(t *testing.T) {
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
// Malformed entry with out-of-range recall_score — scoring must clamp.
|
||||
entry := store.EpisodicSummary{
|
||||
CreatedAt: now,
|
||||
RecallCount: 1,
|
||||
RecallScore: 2.5, // bug: > 1.0
|
||||
}
|
||||
score := ComputeRecallScore(entry, now)
|
||||
if score > 1.1 {
|
||||
t.Errorf("score = %f exceeds reasonable bounds (relevance should clamp)", score)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterByRecallThresholdsFreshBypass(t *testing.T) {
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
entries := []store.EpisodicSummary{
|
||||
// Never recalled, 28 days old → low score but bypasses via RecallCount=0.
|
||||
{ID: uuid.New(), CreatedAt: now.AddDate(0, 0, -28)},
|
||||
}
|
||||
kept := filterByRecallThresholds(entries, recallThresholds{MinScore: 0.5, MinRecallCount: 2}, now)
|
||||
if len(kept) != 1 {
|
||||
t.Errorf("expected 1 kept (fresh bypass), got %d", len(kept))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterByRecallThresholdsPreservesOrder(t *testing.T) {
|
||||
// In-place filter must not scramble the caller's ORDER BY recall_score DESC.
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
entries := []store.EpisodicSummary{
|
||||
{ID: uuid.MustParse("00000000-0000-0000-0000-000000000001"), CreatedAt: now, RecallCount: 10, RecallScore: 0.9, LastRecalledAt: &now},
|
||||
{ID: uuid.MustParse("00000000-0000-0000-0000-000000000002"), CreatedAt: now, RecallCount: 1, RecallScore: 0.1}, // filtered
|
||||
{ID: uuid.MustParse("00000000-0000-0000-0000-000000000003"), CreatedAt: now, RecallCount: 5, RecallScore: 0.7, LastRecalledAt: &now},
|
||||
}
|
||||
kept := filterByRecallThresholds(entries, defaultRecallThresholds(), now)
|
||||
if len(kept) != 2 {
|
||||
t.Fatalf("len=%d, want 2", len(kept))
|
||||
}
|
||||
if kept[0].ID.String() != "00000000-0000-0000-0000-000000000001" ||
|
||||
kept[1].ID.String() != "00000000-0000-0000-0000-000000000003" {
|
||||
t.Errorf("order corrupted: %v", []string{kept[0].ID.String(), kept[1].ID.String()})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterByRecallThresholdsDropsWeakRecalled(t *testing.T) {
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
entries := []store.EpisodicSummary{
|
||||
// Recalled once — below MinRecallCount=2 → filtered.
|
||||
{ID: uuid.New(), CreatedAt: now, RecallCount: 1, RecallScore: 0.9},
|
||||
// Recalled 3 times with good score — kept.
|
||||
{ID: uuid.New(), CreatedAt: now, RecallCount: 3, RecallScore: 0.8, LastRecalledAt: &now},
|
||||
// Never recalled — kept via freshness.
|
||||
{ID: uuid.New(), CreatedAt: now},
|
||||
}
|
||||
kept := filterByRecallThresholds(entries, defaultRecallThresholds(), now)
|
||||
if len(kept) != 2 {
|
||||
t.Errorf("expected 2 kept, got %d", len(kept))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatEntryForSynthesisAnnotatesRecalled(t *testing.T) {
|
||||
now := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
|
||||
entry := store.EpisodicSummary{
|
||||
Summary: "test session",
|
||||
RecallCount: 5,
|
||||
LastRecalledAt: &now,
|
||||
}
|
||||
out := formatEntryForSynthesis(entry)
|
||||
if !strings.Contains(out, "recalled 5x") || !strings.Contains(out, "test session") {
|
||||
t.Errorf("output missing metadata or summary: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatEntryForSynthesisBypassesUnrecalled(t *testing.T) {
|
||||
entry := store.EpisodicSummary{Summary: "fresh content"}
|
||||
if out := formatEntryForSynthesis(entry); out != "fresh content" {
|
||||
t.Errorf("unrecalled entry should pass through unchanged, got %q", out)
|
||||
}
|
||||
}
|
||||
@@ -24,6 +24,10 @@ type ConsolidationDeps struct {
|
||||
Provider providers.Provider // for LLM summarization
|
||||
Model string
|
||||
Extractor EntityExtractor
|
||||
// AgentStore is optional: when present, the dreaming worker reads
|
||||
// per-agent overrides from MemoryConfig.Dreaming. If nil, the worker
|
||||
// uses its built-in defaults for every agent.
|
||||
AgentStore store.AgentCRUDStore
|
||||
}
|
||||
|
||||
// Register wires all consolidation workers to the event bus.
|
||||
@@ -52,6 +56,7 @@ func Register(deps ConsolidationDeps) func() {
|
||||
model: deps.Model,
|
||||
threshold: dreamingDefaultThreshold,
|
||||
debounce: dreamingDefaultDebounce,
|
||||
resolveConfig: newAgentStoreResolver(deps.AgentStore),
|
||||
}
|
||||
|
||||
unsub1 := deps.EventBus.Subscribe(eventbus.EventSessionCompleted, episodic.Handle)
|
||||
|
||||
@@ -15,16 +15,25 @@ import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// recallCall is a single RecordRecall invocation captured by the mock store
|
||||
// so tests can assert the (id, score) stream forwarded by memory_search.
|
||||
type recallCall struct {
|
||||
ID string
|
||||
Score float64
|
||||
}
|
||||
|
||||
// mockEpisodicStore implements store.EpisodicStore for testing.
|
||||
type mockEpisodicStore struct {
|
||||
created []*store.EpisodicSummary
|
||||
existsByID map[string]bool
|
||||
unpromoted []store.EpisodicSummary
|
||||
promoted map[string]bool
|
||||
countResult int
|
||||
pruneErr error
|
||||
pruneCount int
|
||||
mu sync.Mutex
|
||||
created []*store.EpisodicSummary
|
||||
existsByID map[string]bool
|
||||
unpromoted []store.EpisodicSummary
|
||||
promoted map[string]bool
|
||||
countResult int
|
||||
countCalls int // set by CountUnpromoted — used by disabled-skip tests
|
||||
recallCalls []recallCall
|
||||
pruneErr error
|
||||
pruneCount int
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func (m *mockEpisodicStore) Create(_ context.Context, ep *store.EpisodicSummary) error {
|
||||
@@ -68,6 +77,7 @@ func (m *mockEpisodicStore) PruneExpired(_ context.Context) (int, error) {
|
||||
func (m *mockEpisodicStore) CountUnpromoted(_ context.Context, _, _ string) (int, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.countCalls++
|
||||
return m.countResult, nil
|
||||
}
|
||||
|
||||
@@ -77,6 +87,23 @@ func (m *mockEpisodicStore) ListUnpromoted(_ context.Context, _, _ string, _ int
|
||||
return m.unpromoted, nil
|
||||
}
|
||||
|
||||
// ListUnpromotedScored returns the same fixture as ListUnpromoted; tests that
|
||||
// need to verify sort order override the method directly on a dedicated mock.
|
||||
func (m *mockEpisodicStore) ListUnpromotedScored(_ context.Context, _, _ string, _ int) ([]store.EpisodicSummary, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.unpromoted, nil
|
||||
}
|
||||
|
||||
// RecordRecall appends the call into the recallCalls slice so tests can
|
||||
// assert which episodic IDs received which score.
|
||||
func (m *mockEpisodicStore) RecordRecall(_ context.Context, id string, score float64) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.recallCalls = append(m.recallCalls, recallCall{ID: id, Score: score})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *mockEpisodicStore) MarkPromoted(_ context.Context, ids []string) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/adhocore/gronx"
|
||||
@@ -146,11 +145,22 @@ func (cs *Service) runLoop(stopChan chan struct{}) {
|
||||
case <-stopChan:
|
||||
return
|
||||
case <-ticker.C:
|
||||
cs.checkJobs()
|
||||
cs.safeCheckJobs()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// safeCheckJobs wraps checkJobs with panic recovery so a panic in any
|
||||
// check/claim logic doesn't kill the runLoop goroutine.
|
||||
func (cs *Service) safeCheckJobs() {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
slog.Error("cron: checkJobs panicked — runLoop continues", "panic", fmt.Sprint(r))
|
||||
}
|
||||
}()
|
||||
cs.checkJobs()
|
||||
}
|
||||
|
||||
func (cs *Service) checkJobs() {
|
||||
cs.mu.Lock()
|
||||
|
||||
@@ -189,17 +199,16 @@ func (cs *Service) checkJobs() {
|
||||
cs.saveUnsafe()
|
||||
cs.mu.Unlock()
|
||||
|
||||
// Execute jobs in parallel — scheduler enforces per-session serialization
|
||||
var wg sync.WaitGroup
|
||||
// Execute jobs in parallel without blocking the runLoop.
|
||||
// Previously wg.Wait() blocked here — if any job hung (e.g. LLM timeout,
|
||||
// agent loop stuck), the entire cron scheduler would stop checking for new
|
||||
// due jobs. Now each job runs independently with panic recovery.
|
||||
for _, dj := range dueJobs {
|
||||
wg.Add(1)
|
||||
go func(id string, scheduledAtMS int64) {
|
||||
defer wg.Done()
|
||||
defer safego.Recover(nil, "job_id", id)
|
||||
cs.executeJobByID(id, scheduledAtMS)
|
||||
}(dj.id, dj.scheduledAtMS)
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func (cs *Service) executeJobByID(jobID string, scheduledAtMS int64) {
|
||||
|
||||
@@ -381,10 +381,26 @@ func (m *TenantsMethods) handleMine(ctx context.Context, client *gateway.Client,
|
||||
return
|
||||
}
|
||||
|
||||
// Batch-fetch all tenant data in a single query instead of per-membership.
|
||||
ids := make([]uuid.UUID, 0, len(memberships))
|
||||
for _, mem := range memberships {
|
||||
ids = append(ids, mem.TenantID)
|
||||
}
|
||||
tenants, tErr := m.tenantStore.GetTenantsByIDs(ctx, ids)
|
||||
if tErr != nil {
|
||||
slog.Error("tenants.mine: batch fetch failed", "error", tErr, "user_id", userID)
|
||||
client.SendResponse(protocol.NewErrorResponse(req.ID, protocol.ErrInternal, i18n.T(locale, i18n.MsgFailedToList, "tenants")))
|
||||
return
|
||||
}
|
||||
tenantMap := make(map[uuid.UUID]*store.TenantData, len(tenants))
|
||||
for i := range tenants {
|
||||
tenantMap[tenants[i].ID] = &tenants[i]
|
||||
}
|
||||
|
||||
entries := make([]tenantEntry, 0, len(memberships))
|
||||
for _, mem := range memberships {
|
||||
t, err := m.tenantStore.GetTenant(ctx, mem.TenantID)
|
||||
if err != nil || t == nil || t.Status != store.TenantStatusActive {
|
||||
t := tenantMap[mem.TenantID]
|
||||
if t == nil || t.Status != store.TenantStatusActive {
|
||||
continue
|
||||
}
|
||||
entries = append(entries, tenantEntry{ID: t.ID.String(), Name: t.Name, Slug: t.Slug, Role: mem.Role, Status: t.Status})
|
||||
|
||||
@@ -50,28 +50,28 @@ func (m *UsageMethods) handleGet(ctx context.Context, client *gateway.Client, re
|
||||
params.Limit = 20
|
||||
}
|
||||
|
||||
sessions := m.sessions.List(ctx, params.AgentID)
|
||||
// Use ListPagedRich: single query returns model, provider, tokens — no N+1 GetOrCreate loop.
|
||||
// Fetch large batch to filter non-zero tokens, then paginate in-memory.
|
||||
result := m.sessions.ListPagedRich(ctx, store.SessionListOpts{
|
||||
AgentID: params.AgentID,
|
||||
Limit: 10000,
|
||||
})
|
||||
|
||||
records := make([]UsageRecord, 0, len(sessions))
|
||||
for _, s := range sessions {
|
||||
// Get full session data for token info
|
||||
data := m.sessions.GetOrCreate(ctx, s.Key)
|
||||
if data.InputTokens == 0 && data.OutputTokens == 0 {
|
||||
records := make([]UsageRecord, 0, len(result.Sessions))
|
||||
for _, s := range result.Sessions {
|
||||
if s.InputTokens == 0 && s.OutputTokens == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// Extract agentID from session key (format: "agent:<agentID>:<scopeKey>")
|
||||
agentID := extractAgentIDFromKey(s.Key)
|
||||
|
||||
records = append(records, UsageRecord{
|
||||
AgentID: agentID,
|
||||
SessionKey: s.Key,
|
||||
Model: data.Model,
|
||||
Provider: data.Provider,
|
||||
InputTokens: data.InputTokens,
|
||||
OutputTokens: data.OutputTokens,
|
||||
TotalTokens: data.InputTokens + data.OutputTokens,
|
||||
Timestamp: data.Updated.UnixMilli(),
|
||||
Model: s.Model,
|
||||
Provider: s.Provider,
|
||||
InputTokens: s.InputTokens,
|
||||
OutputTokens: s.OutputTokens,
|
||||
TotalTokens: s.InputTokens + s.OutputTokens,
|
||||
Timestamp: s.Updated.UnixMilli(),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -96,7 +96,8 @@ func (m *UsageMethods) handleGet(ctx context.Context, client *gateway.Client, re
|
||||
}
|
||||
|
||||
func (m *UsageMethods) handleSummary(ctx context.Context, client *gateway.Client, req *protocol.RequestFrame) {
|
||||
sessions := m.sessions.List(ctx, "") // all agents
|
||||
// Use ListPagedRich: single query returns all token data — no N+1 GetOrCreate loop.
|
||||
result := m.sessions.ListPagedRich(ctx, store.SessionListOpts{Limit: 10000})
|
||||
|
||||
type agentSummary struct {
|
||||
InputTokens int64 `json:"inputTokens"`
|
||||
@@ -108,9 +109,8 @@ func (m *UsageMethods) handleSummary(ctx context.Context, client *gateway.Client
|
||||
byAgent := make(map[string]*agentSummary)
|
||||
var totalRecords int
|
||||
|
||||
for _, s := range sessions {
|
||||
data := m.sessions.GetOrCreate(ctx, s.Key)
|
||||
if data.InputTokens == 0 && data.OutputTokens == 0 {
|
||||
for _, s := range result.Sessions {
|
||||
if s.InputTokens == 0 && s.OutputTokens == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -119,9 +119,9 @@ func (m *UsageMethods) handleSummary(ctx context.Context, client *gateway.Client
|
||||
byAgent[agentID] = &agentSummary{}
|
||||
}
|
||||
|
||||
byAgent[agentID].InputTokens += data.InputTokens
|
||||
byAgent[agentID].OutputTokens += data.OutputTokens
|
||||
byAgent[agentID].TotalTokens += data.InputTokens + data.OutputTokens
|
||||
byAgent[agentID].InputTokens += s.InputTokens
|
||||
byAgent[agentID].OutputTokens += s.OutputTokens
|
||||
byAgent[agentID].TotalTokens += s.InputTokens + s.OutputTokens
|
||||
byAgent[agentID].Sessions++
|
||||
totalRecords++
|
||||
}
|
||||
|
||||
+16
-8
@@ -18,6 +18,7 @@ import (
|
||||
"github.com/nextlevelbuilder/goclaw/internal/permissions"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tools"
|
||||
"github.com/nextlevelbuilder/goclaw/pkg/protocol"
|
||||
)
|
||||
|
||||
@@ -32,8 +33,9 @@ type AgentsHandler struct {
|
||||
kgStore store.KnowledgeGraphStore // for import (nil = disabled)
|
||||
episodicStore store.EpisodicStore // for import (nil in SQLite/lite builds)
|
||||
vaultStore store.VaultStore // for vault import (nil = disabled)
|
||||
toolsReg ToolLister // for system prompt preview tool resolution (nil = fallback)
|
||||
skillsLoader SkillPinnedBuilder // for system prompt preview pinned skills (nil = skip)
|
||||
toolsReg ToolPreviewLister // for system prompt preview tool resolution (nil = fallback)
|
||||
skillsLoader SkillPreviewBuilder // for system prompt preview pinned skills (nil = skip)
|
||||
skillAccessStore store.SkillAccessStore // for system prompt preview skill filtering (nil = skip)
|
||||
teamStore store.TeamStore // for system prompt preview team context (nil = skip)
|
||||
agentLinkStore store.AgentLinkStore // for system prompt preview delegation targets (nil = skip)
|
||||
defaultWorkspace string // default workspace path template (e.g. "~/.goclaw/workspace")
|
||||
@@ -82,24 +84,30 @@ func (h *AgentsHandler) SetVaultStore(vs store.VaultStore) {
|
||||
h.vaultStore = vs
|
||||
}
|
||||
|
||||
// ToolLister is satisfied by tools.Registry for system prompt preview.
|
||||
type ToolLister interface{ List() []string }
|
||||
// ToolPreviewLister is satisfied by tools.Registry for system prompt preview.
|
||||
type ToolPreviewLister interface {
|
||||
List() []string
|
||||
Get(name string) (tools.Tool, bool)
|
||||
Aliases() map[string]string
|
||||
}
|
||||
|
||||
// SkillPinnedBuilder is satisfied by skills.Loader for pinned skills summary.
|
||||
type SkillPinnedBuilder interface {
|
||||
// SkillPreviewBuilder is satisfied by skills.Loader for system prompt preview.
|
||||
type SkillPreviewBuilder interface {
|
||||
BuildPinnedSummary(ctx context.Context, names []string) string
|
||||
BuildSummary(ctx context.Context, allowList []string) string
|
||||
}
|
||||
|
||||
// SetPreviewDeps attaches optional dependencies for system prompt preview.
|
||||
func (h *AgentsHandler) SetPreviewDeps(tl ToolLister, sl SkillPinnedBuilder) {
|
||||
func (h *AgentsHandler) SetPreviewDeps(tl ToolPreviewLister, sl SkillPreviewBuilder) {
|
||||
h.toolsReg = tl
|
||||
h.skillsLoader = sl
|
||||
}
|
||||
|
||||
// SetPreviewStores attaches team + agent link stores for system prompt preview.
|
||||
func (h *AgentsHandler) SetPreviewStores(ts store.TeamStore, als store.AgentLinkStore) {
|
||||
func (h *AgentsHandler) SetPreviewStores(ts store.TeamStore, als store.AgentLinkStore, sas store.SkillAccessStore) {
|
||||
h.teamStore = ts
|
||||
h.agentLinkStore = als
|
||||
h.skillAccessStore = sas
|
||||
}
|
||||
|
||||
// isOwnerUser checks if the given user ID is a system owner.
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
@@ -151,40 +152,42 @@ func (h *AgentsHandler) importKG(ctx context.Context, ag *store.AgentData, arc *
|
||||
|
||||
func (h *AgentsHandler) importCron(ctx context.Context, ag *store.AgentData, arc *importArchive, summary *ImportSummary, progressFn func(ProgressEvent)) {
|
||||
tid := importTenantID(ctx)
|
||||
for _, j := range arc.cronJobs {
|
||||
// cron_jobs has no UNIQUE constraint on (agent_id, name), so use SELECT+UPDATE/INSERT
|
||||
var exists bool
|
||||
_ = h.db.QueryRowContext(ctx,
|
||||
`SELECT EXISTS(SELECT 1 FROM cron_jobs WHERE agent_id = $1 AND name = $2 AND tenant_id = $3)`,
|
||||
ag.ID, j.Name, tid,
|
||||
).Scan(&exists)
|
||||
if exists {
|
||||
_, err := h.db.ExecContext(ctx,
|
||||
`UPDATE cron_jobs
|
||||
SET schedule_kind=$1, cron_expression=$2, interval_ms=$3,
|
||||
run_at=$4, timezone=$5, payload=$6, delete_after_run=$7
|
||||
WHERE agent_id=$8 AND name=$9 AND tenant_id=$10`,
|
||||
j.ScheduleKind, j.CronExpression, j.IntervalMS,
|
||||
nullStr(j.RunAt), j.Timezone, j.Payload, j.DeleteAfterRun,
|
||||
ag.ID, j.Name, tid,
|
||||
)
|
||||
if err != nil {
|
||||
slog.Warn("agents.import.cron_job.update", "agent_id", ag.ID, "name", j.Name, "error", err)
|
||||
}
|
||||
} else {
|
||||
_, err := h.db.ExecContext(ctx,
|
||||
`INSERT INTO cron_jobs
|
||||
(agent_id, name, enabled, schedule_kind, cron_expression, interval_ms, run_at, timezone, payload, delete_after_run, tenant_id)
|
||||
VALUES ($1, $2, false, $3, $4, $5, $6, $7, $8, $9, $10)`,
|
||||
ag.ID, j.Name, j.ScheduleKind,
|
||||
const paramsPerRow = 10 // agent_id, name, schedule_kind, cron_expression, interval_ms, run_at, timezone, payload, delete_after_run, tenant_id (enabled is literal false)
|
||||
const chunkSize = 5000
|
||||
|
||||
for start := 0; start < len(arc.cronJobs); start += chunkSize {
|
||||
end := min(start+chunkSize, len(arc.cronJobs))
|
||||
chunk := arc.cronJobs[start:end]
|
||||
|
||||
args := make([]any, 0, len(chunk)*paramsPerRow)
|
||||
placeholders := make([]string, 0, len(chunk))
|
||||
for i, j := range chunk {
|
||||
base := i * paramsPerRow
|
||||
placeholders = append(placeholders, fmt.Sprintf(
|
||||
"($%d,$%d,false,$%d,$%d,$%d,$%d,$%d,$%d,$%d,$%d)",
|
||||
base+1, base+2, base+3, base+4, base+5, base+6, base+7, base+8, base+9, base+10,
|
||||
))
|
||||
args = append(args, ag.ID, j.Name, j.ScheduleKind,
|
||||
j.CronExpression, j.IntervalMS, nullStr(j.RunAt), j.Timezone,
|
||||
j.Payload, j.DeleteAfterRun, tid,
|
||||
)
|
||||
if err != nil {
|
||||
slog.Warn("agents.import.cron_job.insert", "agent_id", ag.ID, "name", j.Name, "error", err)
|
||||
}
|
||||
}
|
||||
summary.CronJobs++
|
||||
|
||||
q := `INSERT INTO cron_jobs
|
||||
(agent_id, name, enabled, schedule_kind, cron_expression, interval_ms, run_at, timezone, payload, delete_after_run, tenant_id)
|
||||
VALUES ` + strings.Join(placeholders, ",") + `
|
||||
ON CONFLICT (agent_id, tenant_id, name) DO UPDATE SET
|
||||
schedule_kind = EXCLUDED.schedule_kind,
|
||||
cron_expression = EXCLUDED.cron_expression,
|
||||
interval_ms = EXCLUDED.interval_ms,
|
||||
run_at = EXCLUDED.run_at,
|
||||
timezone = EXCLUDED.timezone,
|
||||
payload = EXCLUDED.payload,
|
||||
delete_after_run = EXCLUDED.delete_after_run`
|
||||
if _, err := h.db.ExecContext(ctx, q, args...); err != nil {
|
||||
slog.Warn("agents.import.cron_jobs.batch", "agent_id", ag.ID, "count", len(chunk), "error", err)
|
||||
}
|
||||
summary.CronJobs += len(chunk)
|
||||
}
|
||||
if progressFn != nil {
|
||||
progressFn(ProgressEvent{Phase: "cron", Status: "done", Current: summary.CronJobs, Total: len(arc.cronJobs)})
|
||||
@@ -194,18 +197,28 @@ func (h *AgentsHandler) importCron(ctx context.Context, ag *store.AgentData, arc
|
||||
func (h *AgentsHandler) importUserProfiles(ctx context.Context, ag *store.AgentData, arc *importArchive, summary *ImportSummary, progressFn func(ProgressEvent)) {
|
||||
// workspace=NULL for portability (auto-created via GetOrCreateUserProfile on first user access)
|
||||
tid := importTenantID(ctx)
|
||||
for _, p := range arc.userProfiles {
|
||||
_, err := h.db.ExecContext(ctx,
|
||||
`INSERT INTO user_agent_profiles (agent_id, user_id, workspace, tenant_id)
|
||||
VALUES ($1, $2, NULL, $3)
|
||||
ON CONFLICT DO NOTHING`,
|
||||
ag.ID, p.UserID, tid,
|
||||
)
|
||||
if err != nil {
|
||||
slog.Warn("agents.import.user_profile", "agent_id", ag.ID, "user_id", p.UserID, "error", err)
|
||||
continue
|
||||
const colsPerRow = 3 // agent_id, user_id, tenant_id
|
||||
const chunkSize = 5000
|
||||
|
||||
for start := 0; start < len(arc.userProfiles); start += chunkSize {
|
||||
end := min(start+chunkSize, len(arc.userProfiles))
|
||||
chunk := arc.userProfiles[start:end]
|
||||
|
||||
args := make([]any, 0, len(chunk)*colsPerRow)
|
||||
placeholders := make([]string, 0, len(chunk))
|
||||
for i, p := range chunk {
|
||||
base := i * colsPerRow
|
||||
placeholders = append(placeholders, fmt.Sprintf("($%d,$%d,NULL,$%d)", base+1, base+2, base+3))
|
||||
args = append(args, ag.ID, p.UserID, tid)
|
||||
}
|
||||
summary.UserProfiles++
|
||||
|
||||
q := `INSERT INTO user_agent_profiles (agent_id, user_id, workspace, tenant_id)
|
||||
VALUES ` + strings.Join(placeholders, ",") + `
|
||||
ON CONFLICT (agent_id, user_id) DO NOTHING`
|
||||
if _, err := h.db.ExecContext(ctx, q, args...); err != nil {
|
||||
slog.Warn("agents.import.user_profiles.batch", "agent_id", ag.ID, "count", len(chunk), "error", err)
|
||||
}
|
||||
summary.UserProfiles += len(chunk)
|
||||
}
|
||||
if progressFn != nil {
|
||||
progressFn(ProgressEvent{Phase: "user_profiles", Status: "done", Current: summary.UserProfiles, Total: len(arc.userProfiles)})
|
||||
@@ -214,22 +227,35 @@ func (h *AgentsHandler) importUserProfiles(ctx context.Context, ag *store.AgentD
|
||||
|
||||
func (h *AgentsHandler) importUserOverrides(ctx context.Context, ag *store.AgentData, arc *importArchive, summary *ImportSummary, progressFn func(ProgressEvent)) {
|
||||
tid := importTenantID(ctx)
|
||||
for _, o := range arc.userOverrides {
|
||||
_, err := h.db.ExecContext(ctx,
|
||||
`INSERT INTO user_agent_overrides (agent_id, user_id, provider, model, settings, tenant_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
ON CONFLICT (agent_id, user_id) DO UPDATE SET
|
||||
provider = EXCLUDED.provider,
|
||||
model = EXCLUDED.model,
|
||||
settings = EXCLUDED.settings,
|
||||
updated_at = NOW()`,
|
||||
ag.ID, o.UserID, o.Provider, o.Model, coalesceJSON(o.Settings), tid,
|
||||
)
|
||||
if err != nil {
|
||||
slog.Warn("agents.import.user_override", "agent_id", ag.ID, "user_id", o.UserID, "error", err)
|
||||
continue
|
||||
const colsPerRow = 6 // agent_id, user_id, provider, model, settings, tenant_id
|
||||
const chunkSize = 5000
|
||||
|
||||
for start := 0; start < len(arc.userOverrides); start += chunkSize {
|
||||
end := min(start+chunkSize, len(arc.userOverrides))
|
||||
chunk := arc.userOverrides[start:end]
|
||||
|
||||
args := make([]any, 0, len(chunk)*colsPerRow)
|
||||
placeholders := make([]string, 0, len(chunk))
|
||||
for i, o := range chunk {
|
||||
base := i * colsPerRow
|
||||
placeholders = append(placeholders, fmt.Sprintf(
|
||||
"($%d,$%d,$%d,$%d,$%d,$%d)",
|
||||
base+1, base+2, base+3, base+4, base+5, base+6,
|
||||
))
|
||||
args = append(args, ag.ID, o.UserID, o.Provider, o.Model, coalesceJSON(o.Settings), tid)
|
||||
}
|
||||
summary.UserOverrides++
|
||||
|
||||
q := `INSERT INTO user_agent_overrides (agent_id, user_id, provider, model, settings, tenant_id)
|
||||
VALUES ` + strings.Join(placeholders, ",") + `
|
||||
ON CONFLICT (agent_id, user_id) DO UPDATE SET
|
||||
provider = EXCLUDED.provider,
|
||||
model = EXCLUDED.model,
|
||||
settings = EXCLUDED.settings,
|
||||
updated_at = NOW()`
|
||||
if _, err := h.db.ExecContext(ctx, q, args...); err != nil {
|
||||
slog.Warn("agents.import.user_overrides.batch", "agent_id", ag.ID, "count", len(chunk), "error", err)
|
||||
}
|
||||
summary.UserOverrides += len(chunk)
|
||||
}
|
||||
if progressFn != nil {
|
||||
progressFn(ProgressEvent{Phase: "user_overrides", Status: "done", Current: summary.UserOverrides, Total: len(arc.userOverrides)})
|
||||
@@ -354,9 +380,10 @@ func (h *AgentsHandler) importVault(ctx context.Context, ag *store.AgentData, ar
|
||||
progressFn(ProgressEvent{Phase: "vault_documents", Status: "running", Total: len(arc.vaultDocuments)})
|
||||
}
|
||||
for _, d := range arc.vaultDocuments {
|
||||
agentIDStr := ag.ID.String()
|
||||
doc := &store.VaultDocument{
|
||||
TenantID: tid.String(),
|
||||
AgentID: ag.ID.String(),
|
||||
AgentID: &agentIDStr,
|
||||
TeamID: nil, // team_id not portable
|
||||
Scope: d.Scope,
|
||||
CustomScope: d.CustomScope,
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
|
||||
@@ -60,20 +61,37 @@ func (h *AgentsHandler) importTeamSection(ctx context.Context, ag *store.AgentDa
|
||||
// Always include the importing agent itself
|
||||
agentKeyToID[ag.AgentKey] = ag.ID
|
||||
|
||||
// Insert non-lead members
|
||||
for _, m := range arc.teamMembers {
|
||||
memberID, ok := agentKeyToID[m.AgentKey]
|
||||
if !ok {
|
||||
slog.Info("import.team: member not found, skipping", "agent_key", m.AgentKey)
|
||||
continue
|
||||
// Insert non-lead members (batch)
|
||||
{
|
||||
type memberRow struct {
|
||||
agentID uuid.UUID
|
||||
role string
|
||||
}
|
||||
if _, err = h.db.ExecContext(ctx,
|
||||
`INSERT INTO agent_team_members (team_id, agent_id, role, tenant_id, joined_at)
|
||||
VALUES ($1, $2, $3, $4, NOW())
|
||||
ON CONFLICT (team_id, agent_id) DO NOTHING`,
|
||||
teamID, memberID, m.Role, tid,
|
||||
); err != nil {
|
||||
slog.Warn("import.team: insert member", "agent_key", m.AgentKey, "error", err)
|
||||
var rows []memberRow
|
||||
for _, m := range arc.teamMembers {
|
||||
memberID, ok := agentKeyToID[m.AgentKey]
|
||||
if !ok {
|
||||
slog.Info("import.team: member not found, skipping", "agent_key", m.AgentKey)
|
||||
continue
|
||||
}
|
||||
rows = append(rows, memberRow{agentID: memberID, role: m.Role})
|
||||
}
|
||||
const cols = 4 // team_id, agent_id, role, tenant_id
|
||||
for start := 0; start < len(rows); start += 1000 {
|
||||
end := min(start+1000, len(rows))
|
||||
chunk := rows[start:end]
|
||||
args := make([]any, 0, len(chunk)*cols)
|
||||
ph := make([]string, 0, len(chunk))
|
||||
for i, r := range chunk {
|
||||
b := i * cols
|
||||
ph = append(ph, fmt.Sprintf("($%d,$%d,$%d,$%d,NOW())", b+1, b+2, b+3, b+4))
|
||||
args = append(args, teamID, r.agentID, r.role, tid)
|
||||
}
|
||||
q := `INSERT INTO agent_team_members (team_id, agent_id, role, tenant_id, joined_at)
|
||||
VALUES ` + strings.Join(ph, ",") + ` ON CONFLICT (team_id, agent_id) DO NOTHING`
|
||||
if _, err = h.db.ExecContext(ctx, q, args...); err != nil {
|
||||
slog.Warn("import.team: batch insert members", "count", len(chunk), "error", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -130,62 +148,132 @@ func (h *AgentsHandler) importTeamSection(ctx context.Context, ag *store.AgentDa
|
||||
}
|
||||
}
|
||||
|
||||
// Insert comments
|
||||
for _, c := range arc.teamComments {
|
||||
if c.TaskIdx < 0 || c.TaskIdx >= len(taskIDByIdx) {
|
||||
continue
|
||||
// Insert comments (batch)
|
||||
{
|
||||
type commentRow struct {
|
||||
id uuid.UUID
|
||||
taskID uuid.UUID
|
||||
agentID *uuid.UUID
|
||||
userID *string
|
||||
content string
|
||||
commentType string
|
||||
metadata any
|
||||
}
|
||||
taskID := taskIDByIdx[c.TaskIdx]
|
||||
|
||||
var agentID *uuid.UUID
|
||||
if c.AgentKey != nil {
|
||||
if id, ok := agentKeyToID[*c.AgentKey]; ok {
|
||||
agentID = &id
|
||||
var cRows []commentRow
|
||||
for _, c := range arc.teamComments {
|
||||
if c.TaskIdx < 0 || c.TaskIdx >= len(taskIDByIdx) {
|
||||
continue
|
||||
}
|
||||
var agentID *uuid.UUID
|
||||
if c.AgentKey != nil {
|
||||
if id, ok := agentKeyToID[*c.AgentKey]; ok {
|
||||
agentID = &id
|
||||
}
|
||||
}
|
||||
cRows = append(cRows, commentRow{
|
||||
id: uuid.Must(uuid.NewV7()), taskID: taskIDByIdx[c.TaskIdx],
|
||||
agentID: agentID, userID: c.UserID, content: c.Content,
|
||||
commentType: c.CommentType, metadata: nullJSON(c.Metadata),
|
||||
})
|
||||
}
|
||||
const cols = 8
|
||||
for start := 0; start < len(cRows); start += 1000 {
|
||||
end := min(start+1000, len(cRows))
|
||||
chunk := cRows[start:end]
|
||||
args := make([]any, 0, len(chunk)*cols)
|
||||
ph := make([]string, 0, len(chunk))
|
||||
for i, r := range chunk {
|
||||
b := i * cols
|
||||
ph = append(ph, fmt.Sprintf("($%d,$%d,$%d,$%d,$%d,$%d,$%d,$%d,NOW())", b+1, b+2, b+3, b+4, b+5, b+6, b+7, b+8))
|
||||
args = append(args, r.id, r.taskID, r.agentID, r.userID, r.content, r.commentType, r.metadata, tid)
|
||||
}
|
||||
q := `INSERT INTO team_task_comments (id, task_id, agent_id, user_id, content, comment_type, metadata, tenant_id, created_at)
|
||||
VALUES ` + strings.Join(ph, ",")
|
||||
if _, err = h.db.ExecContext(ctx, q, args...); err != nil {
|
||||
slog.Warn("import.team: batch insert comments", "count", len(chunk), "error", err)
|
||||
}
|
||||
}
|
||||
if _, err = h.db.ExecContext(ctx,
|
||||
`INSERT INTO team_task_comments (id, task_id, agent_id, user_id, content, comment_type, metadata, tenant_id, created_at)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,NOW())`,
|
||||
uuid.Must(uuid.NewV7()), taskID, agentID, c.UserID,
|
||||
c.Content, c.CommentType, nullJSON(c.Metadata), tid,
|
||||
); err != nil {
|
||||
slog.Warn("import.team: insert comment", "task_id", taskID, "error", err)
|
||||
}
|
||||
|
||||
// Insert events (batch)
|
||||
{
|
||||
type eventRow struct {
|
||||
id uuid.UUID
|
||||
taskID uuid.UUID
|
||||
eventType string
|
||||
actorType string
|
||||
actorID string
|
||||
data any
|
||||
}
|
||||
var eRows []eventRow
|
||||
for _, ev := range arc.teamEvents {
|
||||
if ev.TaskIdx < 0 || ev.TaskIdx >= len(taskIDByIdx) {
|
||||
continue
|
||||
}
|
||||
eRows = append(eRows, eventRow{
|
||||
id: uuid.Must(uuid.NewV7()), taskID: taskIDByIdx[ev.TaskIdx],
|
||||
eventType: ev.EventType, actorType: ev.ActorType,
|
||||
actorID: ev.ActorID, data: nullJSON(ev.Data),
|
||||
})
|
||||
}
|
||||
const cols = 7
|
||||
for start := 0; start < len(eRows); start += 1000 {
|
||||
end := min(start+1000, len(eRows))
|
||||
chunk := eRows[start:end]
|
||||
args := make([]any, 0, len(chunk)*cols)
|
||||
ph := make([]string, 0, len(chunk))
|
||||
for i, r := range chunk {
|
||||
b := i * cols
|
||||
ph = append(ph, fmt.Sprintf("($%d,$%d,$%d,$%d,$%d,$%d,$%d,NOW())", b+1, b+2, b+3, b+4, b+5, b+6, b+7))
|
||||
args = append(args, r.id, r.taskID, r.eventType, r.actorType, r.actorID, r.data, tid)
|
||||
}
|
||||
q := `INSERT INTO team_task_events (id, task_id, event_type, actor_type, actor_id, data, tenant_id, created_at)
|
||||
VALUES ` + strings.Join(ph, ",")
|
||||
if _, err = h.db.ExecContext(ctx, q, args...); err != nil {
|
||||
slog.Warn("import.team: batch insert events", "count", len(chunk), "error", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Insert events
|
||||
for _, ev := range arc.teamEvents {
|
||||
if ev.TaskIdx < 0 || ev.TaskIdx >= len(taskIDByIdx) {
|
||||
continue
|
||||
// Insert agent_links (batch)
|
||||
{
|
||||
type linkRow struct {
|
||||
id uuid.UUID
|
||||
srcID uuid.UUID
|
||||
tgtID uuid.UUID
|
||||
dir string
|
||||
desc string
|
||||
}
|
||||
taskID := taskIDByIdx[ev.TaskIdx]
|
||||
if _, err = h.db.ExecContext(ctx,
|
||||
`INSERT INTO team_task_events (id, task_id, event_type, actor_type, actor_id, data, tenant_id, created_at)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7,NOW())`,
|
||||
uuid.Must(uuid.NewV7()), taskID,
|
||||
ev.EventType, ev.ActorType, ev.ActorID, nullJSON(ev.Data), tid,
|
||||
); err != nil {
|
||||
slog.Warn("import.team: insert event", "task_id", taskID, "error", err)
|
||||
var lRows []linkRow
|
||||
for _, l := range arc.teamLinks {
|
||||
srcID, srcOK := agentKeyToID[l.SourceAgentKey]
|
||||
tgtID, tgtOK := agentKeyToID[l.TargetAgentKey]
|
||||
if !srcOK || !tgtOK {
|
||||
slog.Info("import.team: agent_link skipped — agent not found",
|
||||
"source", l.SourceAgentKey, "target", l.TargetAgentKey)
|
||||
continue
|
||||
}
|
||||
lRows = append(lRows, linkRow{
|
||||
id: uuid.Must(uuid.NewV7()), srcID: srcID, tgtID: tgtID,
|
||||
dir: l.Direction, desc: l.Description,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Insert agent_links
|
||||
for _, l := range arc.teamLinks {
|
||||
srcID, srcOK := agentKeyToID[l.SourceAgentKey]
|
||||
tgtID, tgtOK := agentKeyToID[l.TargetAgentKey]
|
||||
if !srcOK || !tgtOK {
|
||||
slog.Info("import.team: agent_link skipped — agent not found",
|
||||
"source", l.SourceAgentKey, "target", l.TargetAgentKey)
|
||||
continue
|
||||
}
|
||||
if _, err = h.db.ExecContext(ctx,
|
||||
`INSERT INTO agent_links (id, source_agent_id, target_agent_id, direction, description, created_by, tenant_id)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7)
|
||||
ON CONFLICT DO NOTHING`,
|
||||
uuid.Must(uuid.NewV7()), srcID, tgtID,
|
||||
l.Direction, l.Description, userID, tid,
|
||||
); err != nil {
|
||||
slog.Warn("import.team: insert link", "error", err)
|
||||
const cols = 7
|
||||
for start := 0; start < len(lRows); start += 1000 {
|
||||
end := min(start+1000, len(lRows))
|
||||
chunk := lRows[start:end]
|
||||
args := make([]any, 0, len(chunk)*cols)
|
||||
ph := make([]string, 0, len(chunk))
|
||||
for i, r := range chunk {
|
||||
b := i * cols
|
||||
ph = append(ph, fmt.Sprintf("($%d,$%d,$%d,$%d,$%d,$%d,$%d)", b+1, b+2, b+3, b+4, b+5, b+6, b+7))
|
||||
args = append(args, r.id, r.srcID, r.tgtID, r.dir, r.desc, userID, tid)
|
||||
}
|
||||
q := `INSERT INTO agent_links (id, source_agent_id, target_agent_id, direction, description, created_by, tenant_id)
|
||||
VALUES ` + strings.Join(ph, ",") + ` ON CONFLICT DO NOTHING`
|
||||
if _, err = h.db.ExecContext(ctx, q, args...); err != nil {
|
||||
slog.Warn("import.team: batch insert links", "count", len(chunk), "error", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -208,6 +296,7 @@ func (h *AgentsHandler) importTeamSection(ctx context.Context, ag *store.AgentDa
|
||||
}
|
||||
|
||||
// buildAgentKeyMap resolves all agent_keys referenced in the team section to UUIDs.
|
||||
// Uses batch GetByKeys with tenant-scoped context instead of per-key SELECT.
|
||||
func (h *AgentsHandler) buildAgentKeyMap(ctx context.Context, tid uuid.UUID, arc *importArchive) map[string]uuid.UUID {
|
||||
keys := make(map[string]struct{})
|
||||
for _, m := range arc.teamMembers {
|
||||
@@ -231,14 +320,22 @@ func (h *AgentsHandler) buildAgentKeyMap(ctx context.Context, tid uuid.UUID, arc
|
||||
keys[l.TargetAgentKey] = struct{}{}
|
||||
}
|
||||
|
||||
out := make(map[string]uuid.UUID, len(keys))
|
||||
for key := range keys {
|
||||
var id uuid.UUID
|
||||
if err := h.db.QueryRowContext(ctx,
|
||||
`SELECT id FROM agents WHERE agent_key = $1 AND tenant_id = $2`, key, tid,
|
||||
).Scan(&id); err == nil {
|
||||
out[key] = id
|
||||
}
|
||||
keyList := make([]string, 0, len(keys))
|
||||
for k := range keys {
|
||||
keyList = append(keyList, k)
|
||||
}
|
||||
|
||||
// Ensure tenant-scoped context for GetByKeys (Red Team Fix #2).
|
||||
scopedCtx := store.WithTenantID(ctx, tid)
|
||||
agents, err := h.agents.GetByKeys(scopedCtx, keyList)
|
||||
if err != nil {
|
||||
slog.Warn("import.team: batch agent key lookup failed", "error", err)
|
||||
return make(map[string]uuid.UUID)
|
||||
}
|
||||
|
||||
out := make(map[string]uuid.UUID, len(agents))
|
||||
for _, a := range agents {
|
||||
out[a.AgentKey] = a.ID
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/agent"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/tokencount"
|
||||
)
|
||||
|
||||
@@ -18,10 +19,11 @@ type promptPreviewSection struct {
|
||||
|
||||
// promptPreviewResponse is the API response for system prompt preview.
|
||||
type promptPreviewResponse struct {
|
||||
Mode string `json:"mode"`
|
||||
Prompt string `json:"prompt"`
|
||||
TokenCount int `json:"token_count"`
|
||||
Sections []promptPreviewSection `json:"sections"`
|
||||
Mode string `json:"mode"`
|
||||
Prompt string `json:"prompt"`
|
||||
TokenCount int `json:"token_count"`
|
||||
Sections []promptPreviewSection `json:"sections"`
|
||||
Tools []providers.ToolDefinition `json:"tools,omitempty"`
|
||||
}
|
||||
|
||||
// handleSystemPromptPreview renders the actual system prompt for an agent in a given mode.
|
||||
@@ -49,26 +51,28 @@ func (h *AgentsHandler) handleSystemPromptPreview(w http.ResponseWriter, r *http
|
||||
// Build preview prompt — reuses same BuildSystemPrompt() as LLM pipeline.
|
||||
// Runtime-only fields (channel, peer kind, credentials) are zero-valued;
|
||||
// BuildSystemPrompt nil-checks every field so these sections are simply skipped.
|
||||
prompt := agent.BuildPreviewPrompt(ctx, ag, mode, r.URL.Query().Get("user_id"), agent.PreviewDeps{
|
||||
AgentStore: h.agents,
|
||||
TeamStore: h.teamStore,
|
||||
AgentLinks: h.agentLinkStore,
|
||||
ProviderReg: h.providerReg,
|
||||
ToolLister: h.toolsReg,
|
||||
SkillsLoader: h.skillsLoader,
|
||||
DataDir: h.dataDir,
|
||||
result := agent.BuildPreviewPrompt(ctx, ag, mode, r.URL.Query().Get("user_id"), agent.PreviewDeps{
|
||||
AgentStore: h.agents,
|
||||
TeamStore: h.teamStore,
|
||||
AgentLinks: h.agentLinkStore,
|
||||
ProviderReg: h.providerReg,
|
||||
ToolLister: h.toolsReg,
|
||||
SkillsLoader: h.skillsLoader,
|
||||
SkillAccessStore: h.skillAccessStore,
|
||||
DataDir: h.dataDir,
|
||||
})
|
||||
|
||||
counter := tokencount.NewFallbackCounter()
|
||||
tokens := counter.Count("claude-3", prompt)
|
||||
sections := parseSections(prompt)
|
||||
tokens := counter.Count("claude-3", result.Prompt)
|
||||
sections := parseSections(result.Prompt)
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(promptPreviewResponse{
|
||||
Mode: string(mode),
|
||||
Prompt: prompt,
|
||||
Prompt: result.Prompt,
|
||||
TokenCount: tokens,
|
||||
Sections: sections,
|
||||
Tools: result.ToolDefs,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -117,6 +117,9 @@ func (m *mockTenantStore) GetTenantUser(context.Context, uuid.UUID) (*store.Tena
|
||||
func (m *mockTenantStore) CreateTenantUserReturning(context.Context, uuid.UUID, string, string, string) (*store.TenantUserData, error) {
|
||||
return nil, fmt.Errorf("not implemented")
|
||||
}
|
||||
func (m *mockTenantStore) GetTenantsByIDs(context.Context, []uuid.UUID) ([]store.TenantData, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
type mockPairingStore struct {
|
||||
paired map[string]bool
|
||||
|
||||
@@ -540,7 +540,7 @@ func (h *ChannelInstancesHandler) handleResolveContacts(w http.ResponseWriter, r
|
||||
// isValidChannelType checks if the channel type is supported.
|
||||
func isValidChannelType(ct string) bool {
|
||||
switch ct {
|
||||
case "telegram", "discord", "slack", "whatsapp", "zalo_oa", "zalo_personal", "feishu":
|
||||
case "telegram", "discord", "slack", "whatsapp", "zalo_oa", "zalo_personal", "feishu", "facebook", "pancake":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strconv"
|
||||
@@ -26,6 +27,9 @@ type EvolutionHandler struct {
|
||||
|
||||
// Optional: agent store for applying threshold suggestions.
|
||||
agentStore store.AgentStore
|
||||
|
||||
// Optional: tenant tool config for disabling tools on SuggestToolOrder approval.
|
||||
toolTenantCfgs store.BuiltinToolTenantConfigStore
|
||||
}
|
||||
|
||||
// EvolutionHandlerOpt configures optional EvolutionHandler dependencies.
|
||||
@@ -45,6 +49,11 @@ func WithAgentStore(as store.AgentStore) EvolutionHandlerOpt {
|
||||
return func(h *EvolutionHandler) { h.agentStore = as }
|
||||
}
|
||||
|
||||
// WithToolTenantCfgs enables tool disabling on SuggestToolOrder approval.
|
||||
func WithToolTenantCfgs(tc store.BuiltinToolTenantConfigStore) EvolutionHandlerOpt {
|
||||
return func(h *EvolutionHandler) { h.toolTenantCfgs = tc }
|
||||
}
|
||||
|
||||
func NewEvolutionHandler(m store.EvolutionMetricsStore, s store.EvolutionSuggestionStore, opts ...EvolutionHandlerOpt) *EvolutionHandler {
|
||||
h := &EvolutionHandler{metrics: m, suggestions: s}
|
||||
for _, opt := range opts {
|
||||
@@ -222,6 +231,29 @@ func (h *EvolutionHandler) handleUpdateSuggestion(w http.ResponseWriter, r *http
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok", "action": "skill_created"})
|
||||
return
|
||||
|
||||
case store.SuggestToolOrder:
|
||||
action := "tool_order_approved"
|
||||
if h.toolTenantCfgs != nil {
|
||||
// Extract tool name from suggestion parameters.
|
||||
var params map[string]any
|
||||
if err := json.Unmarshal(existing.Parameters, ¶ms); err == nil {
|
||||
if toolName, _ := params["tool"].(string); toolName != "" {
|
||||
// Disable tool at tenant level using existing infrastructure.
|
||||
if err := h.toolTenantCfgs.Set(r.Context(), existing.TenantID, toolName, false); err != nil {
|
||||
slog.Warn("evolution.tool_order.disable_failed", "tool", toolName, "error", err)
|
||||
} else {
|
||||
action = "tool_disabled"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := h.suggestions.UpdateSuggestionStatus(r.Context(), suggestionID, "applied", reviewedBy); err != nil {
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok", "action": action})
|
||||
return
|
||||
|
||||
case store.SuggestThreshold:
|
||||
if h.agentStore == nil {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "threshold auto-apply not available"})
|
||||
@@ -240,7 +272,7 @@ func (h *EvolutionHandler) handleUpdateSuggestion(w http.ResponseWriter, r *http
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if err := agent.ApplySuggestion(r.Context(), h.agentStore, h.suggestions, *existing); err != nil {
|
||||
if err := agent.ApplySuggestion(r.Context(), h.agentStore, h.suggestions, *existing, guardrails); err != nil {
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// handleListAllDocuments lists vault documents across all agents in tenant.
|
||||
// Optional query param agent_id to filter by specific agent.
|
||||
func (h *VaultHandler) handleListAllDocuments(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.URL.Query().Get("agent_id")
|
||||
opts := h.parseListOpts(r)
|
||||
|
||||
// Validate team membership if specific team requested.
|
||||
if opts.TeamID != nil && *opts.TeamID != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, *opts.TeamID) {
|
||||
return
|
||||
}
|
||||
}
|
||||
// Non-owner without team_id filter: show personal + user's teams.
|
||||
if opts.TeamID == nil && !store.IsOwnerRole(r.Context()) {
|
||||
h.applyNonOwnerTeamScope(r.Context(), &opts)
|
||||
}
|
||||
|
||||
docs, err := h.store.ListDocuments(r.Context(), tenantID.String(), agentID, opts)
|
||||
if err != nil {
|
||||
slog.Warn("vault.list_all failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if docs == nil {
|
||||
docs = []store.VaultDocument{}
|
||||
}
|
||||
total, cntErr := h.store.CountDocuments(r.Context(), tenantID.String(), agentID, opts)
|
||||
if cntErr != nil {
|
||||
slog.Warn("vault.count failed", "error", cntErr)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, vaultDocListResponse{Documents: docs, Total: total})
|
||||
}
|
||||
|
||||
// handleListDocuments lists vault documents for a specific agent.
|
||||
func (h *VaultHandler) handleListDocuments(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
opts := h.parseListOpts(r)
|
||||
|
||||
if opts.TeamID != nil && *opts.TeamID != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, *opts.TeamID) {
|
||||
return
|
||||
}
|
||||
}
|
||||
if opts.TeamID == nil && !store.IsOwnerRole(r.Context()) {
|
||||
h.applyNonOwnerTeamScope(r.Context(), &opts)
|
||||
}
|
||||
|
||||
docs, err := h.store.ListDocuments(r.Context(), tenantID.String(), agentID, opts)
|
||||
if err != nil {
|
||||
slog.Warn("vault.list failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if docs == nil {
|
||||
docs = []store.VaultDocument{}
|
||||
}
|
||||
total, cntErr := h.store.CountDocuments(r.Context(), tenantID.String(), agentID, opts)
|
||||
if cntErr != nil {
|
||||
slog.Warn("vault.count failed", "error", cntErr)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, vaultDocListResponse{Documents: docs, Total: total})
|
||||
}
|
||||
|
||||
// handleGetDocument returns a single vault document by ID, scoped to the agent.
|
||||
func (h *VaultHandler) handleGetDocument(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
docID := r.PathValue("docID")
|
||||
|
||||
doc, err := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
if err != nil {
|
||||
slog.Warn("vault.get failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if doc == nil || (agentID != "" && doc.AgentID != nil && *doc.AgentID != agentID) {
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "document not found"})
|
||||
return
|
||||
}
|
||||
// Verify team boundary — non-owner must be team member to view team docs.
|
||||
if doc.TeamID != nil && *doc.TeamID != "" && !store.IsOwnerRole(r.Context()) {
|
||||
if !h.validateTeamMembership(r.Context(), w, *doc.TeamID) {
|
||||
return
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, doc)
|
||||
}
|
||||
|
||||
// handleCreateDocument creates a new vault document.
|
||||
func (h *VaultHandler) handleCreateDocument(w http.ResponseWriter, r *http.Request) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
|
||||
var body struct {
|
||||
Path string `json:"path"`
|
||||
Title string `json:"title"`
|
||||
DocType string `json:"doc_type"`
|
||||
Scope string `json:"scope"`
|
||||
TeamID string `json:"team_id"`
|
||||
Metadata map[string]any `json:"metadata"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
return
|
||||
}
|
||||
if body.Path == "" || body.Title == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "path and title are required"})
|
||||
return
|
||||
}
|
||||
if strings.Contains(body.Path, "..") || strings.HasPrefix(body.Path, "/") {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid path"})
|
||||
return
|
||||
}
|
||||
if body.DocType == "" {
|
||||
body.DocType = "note"
|
||||
}
|
||||
if body.Scope == "" {
|
||||
body.Scope = "personal"
|
||||
}
|
||||
if !validDocType(body.DocType) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid doc_type"})
|
||||
return
|
||||
}
|
||||
if !validScope(body.Scope) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid scope"})
|
||||
return
|
||||
}
|
||||
|
||||
doc := &store.VaultDocument{
|
||||
TenantID: tenantID.String(),
|
||||
Path: body.Path,
|
||||
Title: body.Title,
|
||||
DocType: body.DocType,
|
||||
Scope: body.Scope,
|
||||
Metadata: body.Metadata,
|
||||
}
|
||||
if agentID != "" {
|
||||
doc.AgentID = &agentID
|
||||
} else if doc.Scope == "personal" {
|
||||
doc.Scope = "shared" // no agent → shared scope
|
||||
}
|
||||
if body.TeamID != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, body.TeamID) {
|
||||
return
|
||||
}
|
||||
doc.TeamID = &body.TeamID
|
||||
if body.Scope == "personal" {
|
||||
doc.Scope = "team"
|
||||
}
|
||||
}
|
||||
if err := h.store.UpsertDocument(r.Context(), doc); err != nil {
|
||||
slog.Warn("vault.create failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
// Re-fetch by ID (set via RETURNING) — unambiguous even when same path exists across teams.
|
||||
created, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), doc.ID)
|
||||
if created != nil {
|
||||
writeJSON(w, http.StatusCreated, created)
|
||||
} else {
|
||||
writeJSON(w, http.StatusCreated, doc)
|
||||
}
|
||||
}
|
||||
|
||||
// handleUpdateDocument updates an existing vault document.
|
||||
func (h *VaultHandler) handleUpdateDocument(w http.ResponseWriter, r *http.Request) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
docID := r.PathValue("docID")
|
||||
|
||||
existing, err := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
if err != nil || existing == nil || (agentID != "" && existing.AgentID != nil && *existing.AgentID != agentID) {
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "document not found"})
|
||||
return
|
||||
}
|
||||
|
||||
var body struct {
|
||||
Title *string `json:"title"`
|
||||
DocType *string `json:"doc_type"`
|
||||
Scope *string `json:"scope"`
|
||||
TeamID *string `json:"team_id"` // nil=no change, ""=clear, "uuid"=set
|
||||
Metadata map[string]any `json:"metadata"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
return
|
||||
}
|
||||
|
||||
if body.Title != nil {
|
||||
existing.Title = *body.Title
|
||||
}
|
||||
if body.DocType != nil {
|
||||
existing.DocType = *body.DocType
|
||||
}
|
||||
if body.Scope != nil {
|
||||
existing.Scope = *body.Scope
|
||||
}
|
||||
if body.TeamID != nil {
|
||||
// Only owner/admin can change team assignment.
|
||||
if !store.IsOwnerRole(r.Context()) {
|
||||
writeJSON(w, http.StatusForbidden, map[string]string{"error": "only owner can change document team assignment"})
|
||||
return
|
||||
}
|
||||
if *body.TeamID == "" {
|
||||
existing.TeamID = nil
|
||||
existing.Scope = "personal"
|
||||
} else {
|
||||
existing.TeamID = body.TeamID
|
||||
existing.Scope = "team"
|
||||
}
|
||||
}
|
||||
if body.Metadata != nil {
|
||||
existing.Metadata = body.Metadata
|
||||
}
|
||||
|
||||
if err := h.store.UpsertDocument(r.Context(), existing); err != nil {
|
||||
slog.Warn("vault.update failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
updated, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
if updated != nil {
|
||||
writeJSON(w, http.StatusOK, updated)
|
||||
} else {
|
||||
writeJSON(w, http.StatusOK, existing)
|
||||
}
|
||||
}
|
||||
|
||||
// handleDeleteDocument deletes a vault document by ID.
|
||||
func (h *VaultHandler) handleDeleteDocument(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
docID := r.PathValue("docID")
|
||||
|
||||
existing, err := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
if err != nil || existing == nil || (agentID != "" && existing.AgentID != nil && *existing.AgentID != agentID) {
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "document not found"})
|
||||
return
|
||||
}
|
||||
|
||||
// Verify team boundary before deletion.
|
||||
if existing.TeamID != nil && *existing.TeamID != "" && !store.IsOwnerRole(r.Context()) {
|
||||
if !h.validateTeamMembership(r.Context(), w, *existing.TeamID) {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// DeleteDocument without RunContext applies no team_id filter (broad match on tenant+agent+path).
|
||||
// This is safe because we pre-validated team membership above and use server-derived existing.Path.
|
||||
// Use the doc's actual agent_id (may be empty for team/shared docs).
|
||||
deleteAgentID := ""
|
||||
if existing.AgentID != nil {
|
||||
deleteAgentID = *existing.AgentID
|
||||
}
|
||||
if err := h.store.DeleteDocument(r.Context(), tenantID.String(), deleteAgentID, existing.Path); err != nil {
|
||||
slog.Warn("vault.delete failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
@@ -0,0 +1,255 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
)
|
||||
|
||||
// handleSearchAll runs tenant-wide search (agent_id optional in body).
|
||||
func (h *VaultHandler) handleSearchAll(w http.ResponseWriter, r *http.Request) {
|
||||
h.doSearch(w, r, "")
|
||||
}
|
||||
|
||||
// handleSearch runs hybrid FTS+vector search scoped to a specific agent.
|
||||
func (h *VaultHandler) handleSearch(w http.ResponseWriter, r *http.Request) {
|
||||
h.doSearch(w, r, r.PathValue("agentID"))
|
||||
}
|
||||
|
||||
// doSearch is the shared search implementation for both per-agent and tenant-wide endpoints.
|
||||
func (h *VaultHandler) doSearch(w http.ResponseWriter, r *http.Request, agentID string) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
|
||||
var body struct {
|
||||
Query string `json:"query"`
|
||||
AgentID string `json:"agent_id"`
|
||||
Scope string `json:"scope"`
|
||||
DocTypes []string `json:"doc_types"`
|
||||
MaxResults int `json:"max_results"`
|
||||
TeamID string `json:"team_id"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
return
|
||||
}
|
||||
if body.Query == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "query is required"})
|
||||
return
|
||||
}
|
||||
if body.MaxResults <= 0 {
|
||||
body.MaxResults = 10
|
||||
}
|
||||
// Body agent_id only used when path doesn't provide one (tenant-wide endpoint).
|
||||
if agentID == "" {
|
||||
agentID = body.AgentID
|
||||
}
|
||||
|
||||
searchOpts := store.VaultSearchOptions{
|
||||
Query: body.Query,
|
||||
AgentID: agentID,
|
||||
TenantID: tenantID.String(),
|
||||
Scope: body.Scope,
|
||||
DocTypes: body.DocTypes,
|
||||
MaxResults: body.MaxResults,
|
||||
}
|
||||
if body.TeamID != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, body.TeamID) {
|
||||
return
|
||||
}
|
||||
searchOpts.TeamID = &body.TeamID
|
||||
} else if !store.IsOwnerRole(r.Context()) {
|
||||
if ids := h.userAccessibleTeamIDs(r.Context()); len(ids) > 0 {
|
||||
searchOpts.TeamIDs = ids
|
||||
} else {
|
||||
empty := ""
|
||||
searchOpts.TeamID = &empty
|
||||
}
|
||||
}
|
||||
|
||||
results, err := h.store.Search(r.Context(), searchOpts)
|
||||
if err != nil {
|
||||
slog.Warn("vault.search failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if results == nil {
|
||||
results = []store.VaultSearchResult{}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, results)
|
||||
}
|
||||
|
||||
// handleGetLinks returns outgoing links and backlinks for a vault document.
|
||||
func (h *VaultHandler) handleGetLinks(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
_ = r.PathValue("agentID") // agent scoping done at document level
|
||||
docID := r.PathValue("docID")
|
||||
|
||||
outLinks, err := h.store.GetOutLinks(r.Context(), tenantID.String(), docID)
|
||||
if err != nil {
|
||||
slog.Warn("vault.outlinks failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
backlinks, err := h.store.GetBacklinks(r.Context(), tenantID.String(), docID)
|
||||
if err != nil {
|
||||
slog.Warn("vault.backlinks failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if outLinks == nil {
|
||||
outLinks = []store.VaultLink{}
|
||||
}
|
||||
if backlinks == nil {
|
||||
backlinks = []store.VaultBacklink{}
|
||||
}
|
||||
|
||||
// Filter backlinks by team boundary — derive team context from the target document
|
||||
// itself (not a query param) so clients don't need to supply it correctly.
|
||||
isOwner := store.IsOwnerRole(r.Context())
|
||||
if !isOwner {
|
||||
targetDoc, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
var currentTeamID string
|
||||
if targetDoc != nil && targetDoc.TeamID != nil {
|
||||
currentTeamID = *targetDoc.TeamID
|
||||
}
|
||||
filtered := make([]store.VaultBacklink, 0, len(backlinks))
|
||||
for _, bl := range backlinks {
|
||||
if currentTeamID != "" {
|
||||
if bl.TeamID != nil && *bl.TeamID != currentTeamID {
|
||||
continue
|
||||
}
|
||||
} else {
|
||||
if bl.TeamID != nil && *bl.TeamID != "" {
|
||||
continue
|
||||
}
|
||||
}
|
||||
filtered = append(filtered, bl)
|
||||
}
|
||||
backlinks = filtered
|
||||
}
|
||||
|
||||
// Resolve target doc titles for outlinks so the UI can display names instead of IDs.
|
||||
docNames := make(map[string]string, len(outLinks))
|
||||
for _, l := range outLinks {
|
||||
if _, seen := docNames[l.ToDocID]; seen {
|
||||
continue
|
||||
}
|
||||
if d, err := h.store.GetDocumentByID(r.Context(), tenantID.String(), l.ToDocID); err == nil && d != nil {
|
||||
name := d.Title
|
||||
if name == "" {
|
||||
if idx := strings.LastIndex(d.Path, "/"); idx >= 0 {
|
||||
name = d.Path[idx+1:]
|
||||
} else {
|
||||
name = d.Path
|
||||
}
|
||||
}
|
||||
docNames[l.ToDocID] = name
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"outlinks": outLinks,
|
||||
"backlinks": backlinks,
|
||||
"doc_names": docNames,
|
||||
})
|
||||
}
|
||||
|
||||
// handleCreateLink creates a link between two vault documents.
|
||||
func (h *VaultHandler) handleCreateLink(w http.ResponseWriter, r *http.Request) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
|
||||
var body struct {
|
||||
FromDocID string `json:"from_doc_id"`
|
||||
ToDocID string `json:"to_doc_id"`
|
||||
LinkType string `json:"link_type"`
|
||||
Context string `json:"context"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
return
|
||||
}
|
||||
if body.FromDocID == "" || body.ToDocID == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "from_doc_id and to_doc_id are required"})
|
||||
return
|
||||
}
|
||||
if body.LinkType == "" {
|
||||
body.LinkType = "reference"
|
||||
}
|
||||
|
||||
// Verify both docs exist, same tenant, and at least source belongs to this agent.
|
||||
agentID := r.PathValue("agentID")
|
||||
from, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), body.FromDocID)
|
||||
to, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), body.ToDocID)
|
||||
if from == nil || to == nil {
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "one or both documents not found"})
|
||||
return
|
||||
}
|
||||
if agentID != "" && from.AgentID != nil && *from.AgentID != agentID {
|
||||
writeJSON(w, http.StatusForbidden, map[string]string{"error": "source document does not belong to this agent"})
|
||||
return
|
||||
}
|
||||
// Block cross-team linking (both team docs must be in same team).
|
||||
if from.TeamID != nil && to.TeamID != nil && *from.TeamID != *to.TeamID {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "cannot link documents from different teams"})
|
||||
return
|
||||
}
|
||||
|
||||
link := &store.VaultLink{
|
||||
FromDocID: body.FromDocID,
|
||||
ToDocID: body.ToDocID,
|
||||
LinkType: body.LinkType,
|
||||
Context: body.Context,
|
||||
}
|
||||
if err := h.store.CreateLink(r.Context(), link); err != nil {
|
||||
slog.Warn("vault.create_link failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, link)
|
||||
}
|
||||
|
||||
// handleBatchGetLinks returns all outlinks for a batch of doc IDs in one query.
|
||||
func (h *VaultHandler) handleBatchGetLinks(w http.ResponseWriter, r *http.Request) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
|
||||
var body struct {
|
||||
DocIDs []string `json:"doc_ids"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
return
|
||||
}
|
||||
if len(body.DocIDs) == 0 {
|
||||
writeJSON(w, http.StatusOK, []store.VaultLink{})
|
||||
return
|
||||
}
|
||||
if len(body.DocIDs) > 500 {
|
||||
body.DocIDs = body.DocIDs[:500]
|
||||
}
|
||||
|
||||
links, err := h.store.GetOutLinksBatch(r.Context(), tenantID.String(), body.DocIDs)
|
||||
if err != nil {
|
||||
slog.Warn("vault.batch_links failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if links == nil {
|
||||
links = []store.VaultLink{}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, links)
|
||||
}
|
||||
|
||||
// handleDeleteLink deletes a vault link.
|
||||
func (h *VaultHandler) handleDeleteLink(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
linkID := r.PathValue("linkID")
|
||||
|
||||
if err := h.store.DeleteLink(r.Context(), tenantID.String(), linkID); err != nil {
|
||||
slog.Warn("vault.delete_link failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/vault"
|
||||
)
|
||||
|
||||
// allowedUploadExts is the whitelist of text-based file extensions accepted by vault upload.
|
||||
var allowedUploadExts = map[string]bool{
|
||||
".md": true, ".txt": true, ".json": true, ".yaml": true, ".yml": true,
|
||||
".csv": true, ".toml": true, ".xml": true, ".html": true, ".htm": true,
|
||||
".go": true, ".py": true, ".js": true, ".ts": true, ".tsx": true, ".jsx": true,
|
||||
".rs": true, ".java": true, ".rb": true, ".sh": true, ".sql": true,
|
||||
".swift": true, ".kt": true, ".c": true, ".cpp": true, ".h": true,
|
||||
}
|
||||
|
||||
// handleUpload accepts multipart file uploads and registers them in the vault.
|
||||
// Files are written to the tenant workspace in the appropriate subfolder based on agent/team selection,
|
||||
// then UpsertDocument + EventVaultDocUpserted triggers the existing enrichment pipeline.
|
||||
func (h *VaultHandler) handleUpload(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
tenantIDStr := tenantID.String()
|
||||
|
||||
// 32 MB in-memory; remainder spills to disk.
|
||||
if err := r.ParseMultipartForm(32 << 20); err != nil {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid multipart form"})
|
||||
return
|
||||
}
|
||||
defer r.MultipartForm.RemoveAll()
|
||||
|
||||
agentIDStr := r.FormValue("agent_id")
|
||||
teamIDStr := r.FormValue("team_id")
|
||||
|
||||
// Validate team membership if provided.
|
||||
if teamIDStr != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, teamIDStr) {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Resolve agent UUID → agent_key for folder placement.
|
||||
var agentKey string
|
||||
if agentIDStr != "" {
|
||||
if h.agents != nil {
|
||||
agents, err := h.agents.List(r.Context(), "")
|
||||
if err == nil {
|
||||
for _, a := range agents {
|
||||
if a.ID.String() == agentIDStr {
|
||||
agentKey = a.AgentKey
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if agentKey == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "agent not found"})
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Determine target subfolder and scope.
|
||||
var subDir, scope string
|
||||
switch {
|
||||
case agentIDStr != "":
|
||||
subDir = filepath.Join("agents", agentKey)
|
||||
scope = "personal"
|
||||
case teamIDStr != "":
|
||||
subDir = filepath.Join("teams", teamIDStr)
|
||||
scope = "team"
|
||||
default:
|
||||
scope = "shared"
|
||||
}
|
||||
|
||||
wsPath := h.resolveTenantWorkspace(r.Context())
|
||||
if wsPath == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "workspace not available"})
|
||||
return
|
||||
}
|
||||
targetDir := filepath.Join(wsPath, subDir)
|
||||
if err := os.MkdirAll(targetDir, 0o755); err != nil {
|
||||
slog.Warn("vault.upload: mkdir failed", "dir", targetDir, "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "failed to create target directory"})
|
||||
return
|
||||
}
|
||||
|
||||
files := r.MultipartForm.File["files"]
|
||||
if len(files) == 0 {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "no files provided"})
|
||||
return
|
||||
}
|
||||
if len(files) > 50 {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "too many files (max 50)"})
|
||||
return
|
||||
}
|
||||
|
||||
type uploadResult struct {
|
||||
Doc *store.VaultDocument `json:"document"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
var results []uploadResult
|
||||
var created int
|
||||
|
||||
for _, fh := range files {
|
||||
// Sanitize filename — basename only, no path traversal.
|
||||
name := filepath.Base(fh.Filename)
|
||||
if name == "." || name == ".." || name == "" {
|
||||
results = append(results, uploadResult{Error: "invalid filename: " + fh.Filename})
|
||||
continue
|
||||
}
|
||||
|
||||
// Validate extension.
|
||||
ext := strings.ToLower(filepath.Ext(name))
|
||||
if !allowedUploadExts[ext] {
|
||||
results = append(results, uploadResult{Error: "unsupported file type: " + name})
|
||||
continue
|
||||
}
|
||||
|
||||
// Enforce per-file size limit (50 MB).
|
||||
if fh.Size > 50<<20 {
|
||||
results = append(results, uploadResult{Error: "file too large: " + name})
|
||||
continue
|
||||
}
|
||||
|
||||
// Open uploaded file.
|
||||
src, err := fh.Open()
|
||||
if err != nil {
|
||||
results = append(results, uploadResult{Error: "failed to read: " + name})
|
||||
continue
|
||||
}
|
||||
|
||||
// Write to workspace.
|
||||
dstPath := filepath.Join(targetDir, name)
|
||||
dst, err := os.Create(dstPath)
|
||||
if err != nil {
|
||||
src.Close()
|
||||
results = append(results, uploadResult{Error: "failed to write: " + name})
|
||||
continue
|
||||
}
|
||||
_, copyErr := io.Copy(dst, src)
|
||||
dst.Close()
|
||||
src.Close()
|
||||
if copyErr != nil {
|
||||
results = append(results, uploadResult{Error: "failed to write: " + name})
|
||||
continue
|
||||
}
|
||||
|
||||
// Hash the written file.
|
||||
hash, err := vault.ContentHashFile(dstPath)
|
||||
if err != nil {
|
||||
results = append(results, uploadResult{Error: "failed to hash: " + name})
|
||||
continue
|
||||
}
|
||||
|
||||
// Build workspace-relative path for DB storage.
|
||||
relPath := name
|
||||
if subDir != "" {
|
||||
relPath = subDir + "/" + name
|
||||
}
|
||||
|
||||
doc := &store.VaultDocument{
|
||||
TenantID: tenantIDStr,
|
||||
Path: relPath,
|
||||
Title: vault.InferTitle(relPath),
|
||||
DocType: vault.InferDocType(relPath),
|
||||
ContentHash: hash,
|
||||
Scope: scope,
|
||||
}
|
||||
if agentIDStr != "" {
|
||||
doc.AgentID = &agentIDStr
|
||||
}
|
||||
if teamIDStr != "" {
|
||||
doc.TeamID = &teamIDStr
|
||||
}
|
||||
|
||||
if err := h.store.UpsertDocument(r.Context(), doc); err != nil {
|
||||
slog.Warn("vault.upload: upsert failed", "path", relPath, "error", err)
|
||||
results = append(results, uploadResult{Error: "failed to register: " + name})
|
||||
continue
|
||||
}
|
||||
|
||||
// Publish enrichment event (same pattern as rescan.go).
|
||||
if h.eventBus != nil {
|
||||
agentForEvent := ""
|
||||
if agentIDStr != "" {
|
||||
agentForEvent = agentIDStr
|
||||
}
|
||||
h.eventBus.Publish(eventbus.DomainEvent{
|
||||
ID: uuid.Must(uuid.NewV7()).String(),
|
||||
Type: eventbus.EventVaultDocUpserted,
|
||||
SourceID: doc.ID + ":" + hash,
|
||||
TenantID: tenantIDStr,
|
||||
AgentID: agentForEvent,
|
||||
Timestamp: time.Now(),
|
||||
Payload: eventbus.VaultDocUpsertedPayload{
|
||||
DocID: doc.ID,
|
||||
TenantID: tenantIDStr,
|
||||
AgentID: agentForEvent,
|
||||
Path: relPath,
|
||||
ContentHash: hash,
|
||||
Workspace: wsPath,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
results = append(results, uploadResult{Doc: doc})
|
||||
created++
|
||||
}
|
||||
|
||||
// Signal enrichment progress for WS subscribers.
|
||||
if h.enrichProgress != nil && created > 0 {
|
||||
h.enrichProgress.Start(created, tenantID)
|
||||
}
|
||||
|
||||
slog.Info("vault.upload", "tenant", tenantIDStr, "uploaded", created, "errors", len(files)-created)
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"documents": results,
|
||||
"count": created,
|
||||
})
|
||||
}
|
||||
+108
-417
@@ -6,9 +6,14 @@ import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/eventbus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/vault"
|
||||
)
|
||||
|
||||
// vaultDocListResponse wraps the document list with total count for pagination.
|
||||
@@ -17,14 +22,33 @@ type vaultDocListResponse struct {
|
||||
Total int `json:"total"`
|
||||
}
|
||||
|
||||
// VaultHandler serves Knowledge Vault document and link endpoints.
|
||||
type VaultHandler struct {
|
||||
store store.VaultStore
|
||||
teamAccess store.TeamAccessStore // nil = skip team membership validation (e.g. lite edition)
|
||||
// AgentLister is the subset of AgentStore needed by VaultHandler (rescan agent_key→UUID mapping).
|
||||
type AgentLister interface {
|
||||
List(ctx context.Context, ownerID string) ([]store.AgentData, error)
|
||||
}
|
||||
|
||||
func NewVaultHandler(s store.VaultStore, ta store.TeamAccessStore) *VaultHandler {
|
||||
return &VaultHandler{store: s, teamAccess: ta}
|
||||
// TeamLister is the subset of TeamStore needed by VaultHandler (rescan team validation).
|
||||
type TeamLister interface {
|
||||
ListTeams(ctx context.Context) ([]store.TeamData, error)
|
||||
}
|
||||
|
||||
// VaultHandler serves Knowledge Vault document and link endpoints.
|
||||
type VaultHandler struct {
|
||||
store store.VaultStore
|
||||
teamAccess store.TeamAccessStore // nil = skip team membership validation (e.g. lite edition)
|
||||
agents AgentLister // nil = rescan skips agent resolution
|
||||
teams TeamLister // nil = rescan skips team resolution
|
||||
workspace string
|
||||
eventBus eventbus.DomainEventBus
|
||||
enrichProgress *vault.EnrichProgress // nil = enrichment progress SSE disabled
|
||||
rescanMu sync.Map // key: tenantID → struct{}, per-tenant concurrency guard
|
||||
}
|
||||
|
||||
// SetEnrichProgress injects the enrichment progress tracker for SSE streaming.
|
||||
func (h *VaultHandler) SetEnrichProgress(p *vault.EnrichProgress) { h.enrichProgress = p }
|
||||
|
||||
func NewVaultHandler(s store.VaultStore, ta store.TeamAccessStore, workspace string, bus eventbus.DomainEventBus, agents AgentLister, teams TeamLister) *VaultHandler {
|
||||
return &VaultHandler{store: s, teamAccess: ta, agents: agents, teams: teams, workspace: workspace, eventBus: bus}
|
||||
}
|
||||
|
||||
// validateTeamMembership checks that the requesting user belongs to the given team.
|
||||
@@ -83,16 +107,28 @@ func (h *VaultHandler) applyNonOwnerTeamScope(ctx context.Context, opts *store.V
|
||||
}
|
||||
|
||||
func (h *VaultHandler) RegisterRoutes(mux *http.ServeMux) {
|
||||
// Cross-agent endpoint (agent_id optional query param).
|
||||
// Cross-agent endpoints (agentID = "" from PathValue when no {agentID} in path).
|
||||
mux.HandleFunc("GET /v1/vault/documents", h.auth(h.handleListAllDocuments))
|
||||
// Per-agent endpoints.
|
||||
mux.HandleFunc("POST /v1/vault/documents", h.auth(h.handleCreateDocument))
|
||||
mux.HandleFunc("GET /v1/vault/documents/{docID}", h.auth(h.handleGetDocument))
|
||||
mux.HandleFunc("PUT /v1/vault/documents/{docID}", h.auth(h.handleUpdateDocument))
|
||||
mux.HandleFunc("DELETE /v1/vault/documents/{docID}", h.auth(h.handleDeleteDocument))
|
||||
mux.HandleFunc("GET /v1/vault/documents/{docID}/links", h.auth(h.handleGetLinks))
|
||||
mux.HandleFunc("POST /v1/vault/links", h.auth(h.handleCreateLink))
|
||||
mux.HandleFunc("DELETE /v1/vault/links/{linkID}", h.auth(h.handleDeleteLink))
|
||||
mux.HandleFunc("POST /v1/vault/links/batch", h.auth(h.handleBatchGetLinks))
|
||||
mux.HandleFunc("POST /v1/vault/upload", h.auth(h.handleUpload))
|
||||
mux.HandleFunc("POST /v1/vault/rescan", h.auth(h.handleRescan))
|
||||
mux.HandleFunc("POST /v1/vault/search", h.auth(h.handleSearchAll))
|
||||
mux.HandleFunc("GET /v1/vault/enrichment/status", h.auth(h.handleEnrichmentStatus))
|
||||
// Per-agent endpoints (backward compat — same handlers, agentID from path).
|
||||
mux.HandleFunc("GET /v1/agents/{agentID}/vault/documents", h.auth(h.handleListDocuments))
|
||||
mux.HandleFunc("GET /v1/agents/{agentID}/vault/documents/{docID}", h.auth(h.handleGetDocument))
|
||||
mux.HandleFunc("POST /v1/agents/{agentID}/vault/documents", h.auth(h.handleCreateDocument))
|
||||
mux.HandleFunc("PUT /v1/agents/{agentID}/vault/documents/{docID}", h.auth(h.handleUpdateDocument))
|
||||
mux.HandleFunc("DELETE /v1/agents/{agentID}/vault/documents/{docID}", h.auth(h.handleDeleteDocument))
|
||||
mux.HandleFunc("POST /v1/agents/{agentID}/vault/search", h.auth(h.handleSearch))
|
||||
mux.HandleFunc("GET /v1/agents/{agentID}/vault/documents/{docID}/links", h.auth(h.handleGetLinks))
|
||||
mux.HandleFunc("POST /v1/agents/{agentID}/vault/search", h.auth(h.handleSearch))
|
||||
mux.HandleFunc("POST /v1/agents/{agentID}/vault/links", h.auth(h.handleCreateLink))
|
||||
mux.HandleFunc("DELETE /v1/agents/{agentID}/vault/links/{linkID}", h.auth(h.handleDeleteLink))
|
||||
}
|
||||
@@ -122,436 +158,91 @@ func (h *VaultHandler) parseListOpts(r *http.Request) store.VaultListOptions {
|
||||
return opts
|
||||
}
|
||||
|
||||
// handleListAllDocuments lists vault documents across all agents in tenant.
|
||||
// Optional query param agent_id to filter by specific agent.
|
||||
func (h *VaultHandler) handleListAllDocuments(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.URL.Query().Get("agent_id")
|
||||
opts := h.parseListOpts(r)
|
||||
// handleRescan walks the entire tenant workspace and registers missing/changed files in vault.
|
||||
// Infers agent/team ownership from directory structure: agents/{key}/, teams/{uuid}/, or root shared.
|
||||
func (h *VaultHandler) handleRescan(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context()).String()
|
||||
|
||||
// Validate team membership if specific team requested.
|
||||
if opts.TeamID != nil && *opts.TeamID != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, *opts.TeamID) {
|
||||
return
|
||||
}
|
||||
// Per-tenant concurrency guard.
|
||||
if _, loaded := h.rescanMu.LoadOrStore(tenantID, struct{}{}); loaded {
|
||||
writeJSON(w, http.StatusConflict, map[string]string{"error": "rescan already in progress"})
|
||||
return
|
||||
}
|
||||
// Non-owner without team_id filter: show personal + user's teams.
|
||||
if opts.TeamID == nil && !store.IsOwnerRole(r.Context()) {
|
||||
h.applyNonOwnerTeamScope(r.Context(), &opts)
|
||||
defer h.rescanMu.Delete(tenantID)
|
||||
|
||||
wsPath := h.resolveTenantWorkspace(r.Context())
|
||||
if wsPath == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "workspace not available"})
|
||||
return
|
||||
}
|
||||
|
||||
docs, err := h.store.ListDocuments(r.Context(), tenantID.String(), agentID, opts)
|
||||
// Build agent_key→UUID map and team UUID set for path inference.
|
||||
agentMap, teamSet := h.buildRescanMaps(r.Context())
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
result, err := vault.RescanWorkspace(ctx, vault.RescanParams{
|
||||
TenantID: tenantID,
|
||||
Workspace: wsPath,
|
||||
AgentMap: agentMap,
|
||||
TeamSet: teamSet,
|
||||
}, h.store, h.eventBus)
|
||||
if err != nil {
|
||||
slog.Warn("vault.list_all failed", "error", err)
|
||||
slog.Warn("vault.rescan failed", "tenant", tenantID, "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if docs == nil {
|
||||
docs = []store.VaultDocument{}
|
||||
|
||||
// Signal enrichment progress so WS subscribers see running=true immediately.
|
||||
if h.enrichProgress != nil && (result.New+result.Updated) > 0 {
|
||||
h.enrichProgress.Start(result.New+result.Updated, store.TenantIDFromContext(r.Context()))
|
||||
}
|
||||
total, cntErr := h.store.CountDocuments(r.Context(), tenantID.String(), agentID, opts)
|
||||
if cntErr != nil {
|
||||
slog.Warn("vault.count failed", "error", cntErr)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, vaultDocListResponse{Documents: docs, Total: total})
|
||||
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}
|
||||
|
||||
// handleListDocuments lists vault documents for a specific agent.
|
||||
func (h *VaultHandler) handleListDocuments(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
opts := h.parseListOpts(r)
|
||||
|
||||
if opts.TeamID != nil && *opts.TeamID != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, *opts.TeamID) {
|
||||
return
|
||||
}
|
||||
// resolveTenantWorkspace returns the tenant-scoped workspace root.
|
||||
func (h *VaultHandler) resolveTenantWorkspace(ctx context.Context) string {
|
||||
if h.workspace == "" {
|
||||
return ""
|
||||
}
|
||||
if opts.TeamID == nil && !store.IsOwnerRole(r.Context()) {
|
||||
h.applyNonOwnerTeamScope(r.Context(), &opts)
|
||||
}
|
||||
|
||||
docs, err := h.store.ListDocuments(r.Context(), tenantID.String(), agentID, opts)
|
||||
if err != nil {
|
||||
slog.Warn("vault.list failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if docs == nil {
|
||||
docs = []store.VaultDocument{}
|
||||
}
|
||||
total, cntErr := h.store.CountDocuments(r.Context(), tenantID.String(), agentID, opts)
|
||||
if cntErr != nil {
|
||||
slog.Warn("vault.count failed", "error", cntErr)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, vaultDocListResponse{Documents: docs, Total: total})
|
||||
tenantID := store.TenantIDFromContext(ctx)
|
||||
slug := store.TenantSlugFromContext(ctx)
|
||||
return config.TenantWorkspace(h.workspace, tenantID, slug)
|
||||
}
|
||||
|
||||
// handleGetDocument returns a single vault document by ID, scoped to the agent.
|
||||
func (h *VaultHandler) handleGetDocument(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
docID := r.PathValue("docID")
|
||||
// buildRescanMaps pre-loads agent_key→UUID and team UUID sets for the current tenant.
|
||||
func (h *VaultHandler) buildRescanMaps(ctx context.Context) (map[string]string, map[string]bool) {
|
||||
agentMap := make(map[string]string)
|
||||
teamSet := make(map[string]bool)
|
||||
|
||||
doc, err := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
if err != nil {
|
||||
slog.Warn("vault.get failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if doc == nil || doc.AgentID != agentID {
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "document not found"})
|
||||
return
|
||||
}
|
||||
// Verify team boundary — non-owner must be team member to view team docs.
|
||||
if doc.TeamID != nil && *doc.TeamID != "" && !store.IsOwnerRole(r.Context()) {
|
||||
if !h.validateTeamMembership(r.Context(), w, *doc.TeamID) {
|
||||
return
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, doc)
|
||||
}
|
||||
|
||||
// handleSearch runs hybrid FTS+vector search on vault documents.
|
||||
func (h *VaultHandler) handleSearch(w http.ResponseWriter, r *http.Request) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
|
||||
var body struct {
|
||||
Query string `json:"query"`
|
||||
Scope string `json:"scope"`
|
||||
DocTypes []string `json:"doc_types"`
|
||||
MaxResults int `json:"max_results"`
|
||||
TeamID string `json:"team_id"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
return
|
||||
}
|
||||
if body.Query == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "query is required"})
|
||||
return
|
||||
}
|
||||
if body.MaxResults <= 0 {
|
||||
body.MaxResults = 10
|
||||
}
|
||||
|
||||
searchOpts := store.VaultSearchOptions{
|
||||
Query: body.Query,
|
||||
AgentID: agentID,
|
||||
TenantID: tenantID.String(),
|
||||
Scope: body.Scope,
|
||||
DocTypes: body.DocTypes,
|
||||
MaxResults: body.MaxResults,
|
||||
}
|
||||
if body.TeamID != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, body.TeamID) {
|
||||
return
|
||||
}
|
||||
searchOpts.TeamID = &body.TeamID
|
||||
} else if !store.IsOwnerRole(r.Context()) {
|
||||
if ids := h.userAccessibleTeamIDs(r.Context()); len(ids) > 0 {
|
||||
searchOpts.TeamIDs = ids
|
||||
} else {
|
||||
empty := ""
|
||||
searchOpts.TeamID = &empty
|
||||
}
|
||||
}
|
||||
|
||||
results, err := h.store.Search(r.Context(), searchOpts)
|
||||
if err != nil {
|
||||
slog.Warn("vault.search failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if results == nil {
|
||||
results = []store.VaultSearchResult{}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, results)
|
||||
}
|
||||
|
||||
// handleGetLinks returns outgoing links and backlinks for a vault document.
|
||||
func (h *VaultHandler) handleGetLinks(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
_ = r.PathValue("agentID") // agent scoping done at document level
|
||||
docID := r.PathValue("docID")
|
||||
|
||||
outLinks, err := h.store.GetOutLinks(r.Context(), tenantID.String(), docID)
|
||||
if err != nil {
|
||||
slog.Warn("vault.outlinks failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
backlinks, err := h.store.GetBacklinks(r.Context(), tenantID.String(), docID)
|
||||
if err != nil {
|
||||
slog.Warn("vault.backlinks failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
if outLinks == nil {
|
||||
outLinks = []store.VaultLink{}
|
||||
}
|
||||
if backlinks == nil {
|
||||
backlinks = []store.VaultBacklink{}
|
||||
}
|
||||
|
||||
// Filter backlinks by team boundary — derive team context from the target document
|
||||
// itself (not a query param) so clients don't need to supply it correctly.
|
||||
isOwner := store.IsOwnerRole(r.Context())
|
||||
if !isOwner {
|
||||
targetDoc, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
var currentTeamID string
|
||||
if targetDoc != nil && targetDoc.TeamID != nil {
|
||||
currentTeamID = *targetDoc.TeamID
|
||||
}
|
||||
filtered := make([]store.VaultBacklink, 0, len(backlinks))
|
||||
for _, bl := range backlinks {
|
||||
if currentTeamID != "" {
|
||||
if bl.TeamID != nil && *bl.TeamID != currentTeamID {
|
||||
continue
|
||||
}
|
||||
} else {
|
||||
if bl.TeamID != nil && *bl.TeamID != "" {
|
||||
continue
|
||||
}
|
||||
if h.agents != nil {
|
||||
agents, err := h.agents.List(ctx, "")
|
||||
if err == nil {
|
||||
for _, a := range agents {
|
||||
agentMap[a.AgentKey] = a.ID.String()
|
||||
}
|
||||
filtered = append(filtered, bl)
|
||||
}
|
||||
backlinks = filtered
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"outlinks": outLinks,
|
||||
"backlinks": backlinks,
|
||||
})
|
||||
if h.teams != nil {
|
||||
teams, err := h.teams.ListTeams(ctx)
|
||||
if err == nil {
|
||||
for _, t := range teams {
|
||||
teamSet[t.ID.String()] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return agentMap, teamSet
|
||||
}
|
||||
|
||||
// handleCreateDocument creates a new vault document.
|
||||
func (h *VaultHandler) handleCreateDocument(w http.ResponseWriter, r *http.Request) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
|
||||
var body struct {
|
||||
Path string `json:"path"`
|
||||
Title string `json:"title"`
|
||||
DocType string `json:"doc_type"`
|
||||
Scope string `json:"scope"`
|
||||
TeamID string `json:"team_id"`
|
||||
Metadata map[string]any `json:"metadata"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
// handleEnrichmentStatus returns the current enrichment pipeline progress as JSON.
|
||||
func (h *VaultHandler) handleEnrichmentStatus(w http.ResponseWriter, r *http.Request) {
|
||||
if h.enrichProgress == nil {
|
||||
writeJSON(w, http.StatusOK, vault.EnrichEvent{})
|
||||
return
|
||||
}
|
||||
if body.Path == "" || body.Title == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "path and title are required"})
|
||||
return
|
||||
}
|
||||
if body.DocType == "" {
|
||||
body.DocType = "note"
|
||||
}
|
||||
if body.Scope == "" {
|
||||
body.Scope = "personal"
|
||||
}
|
||||
if !validDocType(body.DocType) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid doc_type"})
|
||||
return
|
||||
}
|
||||
if !validScope(body.Scope) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid scope"})
|
||||
return
|
||||
}
|
||||
|
||||
doc := &store.VaultDocument{
|
||||
TenantID: tenantID.String(),
|
||||
AgentID: agentID,
|
||||
Path: body.Path,
|
||||
Title: body.Title,
|
||||
DocType: body.DocType,
|
||||
Scope: body.Scope,
|
||||
Metadata: body.Metadata,
|
||||
}
|
||||
if body.TeamID != "" {
|
||||
if !h.validateTeamMembership(r.Context(), w, body.TeamID) {
|
||||
return
|
||||
}
|
||||
doc.TeamID = &body.TeamID
|
||||
if body.Scope == "personal" {
|
||||
doc.Scope = "team"
|
||||
}
|
||||
}
|
||||
if err := h.store.UpsertDocument(r.Context(), doc); err != nil {
|
||||
slog.Warn("vault.create failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
// Re-fetch by ID (set via RETURNING) — unambiguous even when same path exists across teams.
|
||||
created, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), doc.ID)
|
||||
if created != nil {
|
||||
writeJSON(w, http.StatusCreated, created)
|
||||
} else {
|
||||
writeJSON(w, http.StatusCreated, doc)
|
||||
}
|
||||
}
|
||||
|
||||
// handleUpdateDocument updates an existing vault document.
|
||||
func (h *VaultHandler) handleUpdateDocument(w http.ResponseWriter, r *http.Request) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
docID := r.PathValue("docID")
|
||||
|
||||
existing, err := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
if err != nil || existing == nil || existing.AgentID != agentID {
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "document not found"})
|
||||
return
|
||||
}
|
||||
|
||||
var body struct {
|
||||
Title *string `json:"title"`
|
||||
DocType *string `json:"doc_type"`
|
||||
Scope *string `json:"scope"`
|
||||
TeamID *string `json:"team_id"` // nil=no change, ""=clear, "uuid"=set
|
||||
Metadata map[string]any `json:"metadata"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
return
|
||||
}
|
||||
|
||||
if body.Title != nil {
|
||||
existing.Title = *body.Title
|
||||
}
|
||||
if body.DocType != nil {
|
||||
existing.DocType = *body.DocType
|
||||
}
|
||||
if body.Scope != nil {
|
||||
existing.Scope = *body.Scope
|
||||
}
|
||||
if body.TeamID != nil {
|
||||
// Only owner/admin can change team assignment.
|
||||
if !store.IsOwnerRole(r.Context()) {
|
||||
writeJSON(w, http.StatusForbidden, map[string]string{"error": "only owner can change document team assignment"})
|
||||
return
|
||||
}
|
||||
if *body.TeamID == "" {
|
||||
existing.TeamID = nil
|
||||
existing.Scope = "personal"
|
||||
} else {
|
||||
existing.TeamID = body.TeamID
|
||||
existing.Scope = "team"
|
||||
}
|
||||
}
|
||||
if body.Metadata != nil {
|
||||
existing.Metadata = body.Metadata
|
||||
}
|
||||
|
||||
if err := h.store.UpsertDocument(r.Context(), existing); err != nil {
|
||||
slog.Warn("vault.update failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
updated, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
if updated != nil {
|
||||
writeJSON(w, http.StatusOK, updated)
|
||||
} else {
|
||||
writeJSON(w, http.StatusOK, existing)
|
||||
}
|
||||
}
|
||||
|
||||
// handleDeleteDocument deletes a vault document and its links.
|
||||
func (h *VaultHandler) handleDeleteDocument(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
agentID := r.PathValue("agentID")
|
||||
docID := r.PathValue("docID")
|
||||
|
||||
existing, err := h.store.GetDocumentByID(r.Context(), tenantID.String(), docID)
|
||||
if err != nil || existing == nil || existing.AgentID != agentID {
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "document not found"})
|
||||
return
|
||||
}
|
||||
|
||||
// Verify team boundary before deletion.
|
||||
if existing.TeamID != nil && *existing.TeamID != "" && !store.IsOwnerRole(r.Context()) {
|
||||
if !h.validateTeamMembership(r.Context(), w, *existing.TeamID) {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// DeleteDocument without RunContext applies no team_id filter (broad match on tenant+agent+path).
|
||||
// This is safe because we pre-validated team membership above and use server-derived existing.Path.
|
||||
if err := h.store.DeleteDocument(r.Context(), tenantID.String(), agentID, existing.Path); err != nil {
|
||||
slog.Warn("vault.delete failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleCreateLink creates a link between two vault documents.
|
||||
func (h *VaultHandler) handleCreateLink(w http.ResponseWriter, r *http.Request) {
|
||||
locale := extractLocale(r)
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
|
||||
var body struct {
|
||||
FromDocID string `json:"from_doc_id"`
|
||||
ToDocID string `json:"to_doc_id"`
|
||||
LinkType string `json:"link_type"`
|
||||
Context string `json:"context"`
|
||||
}
|
||||
if !bindJSON(w, r, locale, &body) {
|
||||
return
|
||||
}
|
||||
if body.FromDocID == "" || body.ToDocID == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "from_doc_id and to_doc_id are required"})
|
||||
return
|
||||
}
|
||||
if body.LinkType == "" {
|
||||
body.LinkType = "reference"
|
||||
}
|
||||
|
||||
// Verify both docs exist, same tenant, and at least source belongs to this agent.
|
||||
agentID := r.PathValue("agentID")
|
||||
from, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), body.FromDocID)
|
||||
to, _ := h.store.GetDocumentByID(r.Context(), tenantID.String(), body.ToDocID)
|
||||
if from == nil || to == nil {
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "one or both documents not found"})
|
||||
return
|
||||
}
|
||||
if from.AgentID != agentID {
|
||||
writeJSON(w, http.StatusForbidden, map[string]string{"error": "source document does not belong to this agent"})
|
||||
return
|
||||
}
|
||||
// Block cross-team linking (both team docs must be in same team).
|
||||
if from.TeamID != nil && to.TeamID != nil && *from.TeamID != *to.TeamID {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "cannot link documents from different teams"})
|
||||
return
|
||||
}
|
||||
|
||||
link := &store.VaultLink{
|
||||
FromDocID: body.FromDocID,
|
||||
ToDocID: body.ToDocID,
|
||||
LinkType: body.LinkType,
|
||||
Context: body.Context,
|
||||
}
|
||||
if err := h.store.CreateLink(r.Context(), link); err != nil {
|
||||
slog.Warn("vault.create_link failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, link)
|
||||
}
|
||||
|
||||
// handleDeleteLink deletes a vault link.
|
||||
func (h *VaultHandler) handleDeleteLink(w http.ResponseWriter, r *http.Request) {
|
||||
tenantID := store.TenantIDFromContext(r.Context())
|
||||
linkID := r.PathValue("linkID")
|
||||
|
||||
if err := h.store.DeleteLink(r.Context(), tenantID.String(), linkID); err != nil {
|
||||
slog.Warn("vault.delete_link failed", "error", err)
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
writeJSON(w, http.StatusOK, h.enrichProgress.Status())
|
||||
}
|
||||
|
||||
var allowedDocTypes = map[string]bool{"context": true, "memory": true, "note": true, "skill": true, "episodic": true, "media": true}
|
||||
|
||||
@@ -15,6 +15,8 @@ import (
|
||||
|
||||
// BridgeTool adapts an MCP tool into the tools.Tool interface.
|
||||
// It delegates Execute calls to the MCP server via the client.
|
||||
// The client pointer is loaded atomically from clientPtr to support
|
||||
// safe reconnection without data races.
|
||||
type BridgeTool struct {
|
||||
serverName string
|
||||
toolName string // original MCP tool name
|
||||
@@ -22,7 +24,7 @@ type BridgeTool struct {
|
||||
description string
|
||||
inputSchema map[string]any // JSON Schema for parameters
|
||||
requiredSet map[string]bool
|
||||
client *mcpclient.Client
|
||||
clientPtr *atomic.Pointer[mcpclient.Client] // shared with serverState for atomic swap on reconnect
|
||||
timeoutSec int
|
||||
connected *atomic.Bool
|
||||
}
|
||||
@@ -30,7 +32,9 @@ type BridgeTool struct {
|
||||
// NewBridgeTool creates a BridgeTool from an MCP Tool definition.
|
||||
// The tool name is always prefixed with "mcp_" to distinguish MCP tools from native tools.
|
||||
// If prefix is empty, it is auto-derived from the server name.
|
||||
func NewBridgeTool(serverName string, mcpTool mcpgo.Tool, client *mcpclient.Client, prefix string, timeoutSec int, connected *atomic.Bool) *BridgeTool {
|
||||
// clientPtr is a shared atomic pointer from serverState — reconnection swaps it
|
||||
// atomically, and all BridgeTools see the new client without explicit notification.
|
||||
func NewBridgeTool(serverName string, mcpTool mcpgo.Tool, clientPtr *atomic.Pointer[mcpclient.Client], prefix string, timeoutSec int, connected *atomic.Bool) *BridgeTool {
|
||||
name := mcpTool.Name
|
||||
effectivePrefix := ensureMCPPrefix(prefix, serverName)
|
||||
registered := effectivePrefix + "__" + name
|
||||
@@ -53,7 +57,7 @@ func NewBridgeTool(serverName string, mcpTool mcpgo.Tool, client *mcpclient.Clie
|
||||
description: mcpTool.Description,
|
||||
inputSchema: schema,
|
||||
requiredSet: reqSet,
|
||||
client: client,
|
||||
clientPtr: clientPtr,
|
||||
timeoutSec: timeoutSec,
|
||||
connected: connected,
|
||||
}
|
||||
@@ -99,6 +103,11 @@ func (t *BridgeTool) Execute(ctx context.Context, args map[string]any) *tools.Re
|
||||
return tools.ErrorResult(fmt.Sprintf("MCP server %q is disconnected", t.serverName))
|
||||
}
|
||||
|
||||
client := t.clientPtr.Load() // atomic load — safe during concurrent reconnect
|
||||
if client == nil {
|
||||
return tools.ErrorResult(fmt.Sprintf("MCP server %q has no active client", t.serverName))
|
||||
}
|
||||
|
||||
callCtx, cancel := context.WithTimeout(ctx, time.Duration(t.timeoutSec)*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -111,7 +120,7 @@ func (t *BridgeTool) Execute(ctx context.Context, args map[string]any) *tools.Re
|
||||
req.Params.Name = t.toolName
|
||||
req.Params.Arguments = cleanedArgs
|
||||
|
||||
result, err := t.client.CallTool(callCtx, req)
|
||||
result, err := client.CallTool(callCtx, req)
|
||||
if err != nil {
|
||||
if errors.Is(callCtx.Err(), context.DeadlineExceeded) {
|
||||
return tools.ErrorResult(fmt.Sprintf("MCP tool %q timeout after %ds", t.registeredName, t.timeoutSec))
|
||||
|
||||
+25
-3
@@ -25,6 +25,7 @@ const (
|
||||
initialBackoff = 2 * time.Second
|
||||
maxBackoff = 60 * time.Second
|
||||
maxReconnectAttempts = 10
|
||||
reconnectCooldown = 5 * time.Minute // wait after exhausting reconnect attempts before retrying
|
||||
|
||||
// mcpToolInlineMaxCount is the threshold above which MCP tools switch
|
||||
// to search mode (deferred loading via mcp_tool_search) instead of
|
||||
@@ -41,15 +42,35 @@ type ServerStatus struct {
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// connParams stores connection parameters needed to re-establish a dead connection.
|
||||
// Populated during initial connectAndDiscover and used by tryReconnect.
|
||||
type connParams struct {
|
||||
command string
|
||||
args []string
|
||||
env map[string]string
|
||||
url string
|
||||
headers map[string]string
|
||||
}
|
||||
|
||||
// serverState tracks a single MCP server connection.
|
||||
//
|
||||
// Dual-pointer design for the MCP client:
|
||||
// - client: direct pointer used by healthLoop (single goroutine, no contention).
|
||||
// - clientPtr: atomic pointer shared with all BridgeTools via NewBridgeTool.
|
||||
// BridgeTools call clientPtr.Load() in Execute for race-safe access.
|
||||
//
|
||||
// On reconnect, fullReconnect() updates BOTH: ss.client for healthLoop and
|
||||
// ss.clientPtr.Store() for BridgeTools. The old client is closed AFTER the swap.
|
||||
type serverState struct {
|
||||
name string
|
||||
transport string
|
||||
client *mcpclient.Client
|
||||
client *mcpclient.Client // direct ref for health checks (single-goroutine access)
|
||||
clientPtr atomic.Pointer[mcpclient.Client] // shared atomic ref for BridgeTools (multi-goroutine safe)
|
||||
connected atomic.Bool
|
||||
toolNames []string // registered tool names in the registry
|
||||
timeoutSec int
|
||||
cancel context.CancelFunc
|
||||
conn connParams // connection params for reconnect
|
||||
|
||||
mu sync.Mutex
|
||||
reconnAttempts int
|
||||
@@ -495,8 +516,9 @@ func (m *Manager) Stop() {
|
||||
if ss.cancel != nil {
|
||||
ss.cancel()
|
||||
}
|
||||
if ss.client != nil {
|
||||
if err := ss.client.Close(); err != nil {
|
||||
// Use atomic pointer — health loop may swap client via fullReconnect concurrently.
|
||||
if client := ss.clientPtr.Load(); client != nil {
|
||||
if err := client.Close(); err != nil {
|
||||
slog.Debug("mcp.server.close_error", "server", name, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,7 +86,15 @@ func connectAndDiscover(ctx context.Context, name, transportType, command string
|
||||
transport: transportType,
|
||||
client: client,
|
||||
timeoutSec: timeoutSec,
|
||||
conn: connParams{
|
||||
command: command,
|
||||
args: args,
|
||||
env: env,
|
||||
url: url,
|
||||
headers: headers,
|
||||
},
|
||||
}
|
||||
ss.clientPtr.Store(client)
|
||||
ss.connected.Store(true)
|
||||
|
||||
return ss, toolsResult.Tools, nil
|
||||
@@ -133,7 +141,7 @@ func (m *Manager) connectServer(ctx context.Context, name, transportType, comman
|
||||
func (m *Manager) registerBridgeTools(ss *serverState, mcpTools []mcpgo.Tool, serverName, toolPrefix string, timeoutSec int) []string {
|
||||
var registeredNames []string
|
||||
for _, mcpTool := range mcpTools {
|
||||
bt := NewBridgeTool(serverName, mcpTool, ss.client, toolPrefix, timeoutSec, &ss.connected)
|
||||
bt := NewBridgeTool(serverName, mcpTool, &ss.clientPtr, toolPrefix, timeoutSec, &ss.connected)
|
||||
|
||||
if _, exists := m.registry.Get(bt.Name()); exists {
|
||||
slog.Warn("mcp.tool.name_collision",
|
||||
@@ -199,7 +207,7 @@ func (m *Manager) connectViaPool(ctx context.Context, tenantID uuid.UUID, name,
|
||||
func (m *Manager) registerPoolBridgeTools(entry *poolEntry, serverName, toolPrefix string, timeoutSec int) []string {
|
||||
var registeredNames []string
|
||||
for _, mcpTool := range entry.tools {
|
||||
bt := NewBridgeTool(serverName, mcpTool, entry.state.client, toolPrefix, timeoutSec, &entry.state.connected)
|
||||
bt := NewBridgeTool(serverName, mcpTool, &entry.state.clientPtr, toolPrefix, timeoutSec, &entry.state.connected)
|
||||
|
||||
if _, exists := m.registry.Get(bt.Name()); exists {
|
||||
slog.Warn("mcp.tool.name_collision",
|
||||
@@ -302,24 +310,36 @@ func (m *Manager) healthLoop(ctx context.Context, ss *serverState) {
|
||||
|
||||
// tryReconnect attempts to reconnect with exponential backoff.
|
||||
func (m *Manager) tryReconnect(ctx context.Context, ss *serverState) {
|
||||
reconnectWithBackoff(ctx, ss, "mcp.server")
|
||||
}
|
||||
|
||||
// reconnectWithBackoff implements the two-phase reconnect strategy shared by
|
||||
// Manager.healthLoop and poolHealthLoop. Handles cooldown after exhausting
|
||||
// max attempts, exponential backoff, fast-path ping (transient blips), and
|
||||
// slow-path full reconnect (dead server-side session).
|
||||
// logPrefix distinguishes log entries (e.g. "mcp.server" vs "mcp.pool").
|
||||
func reconnectWithBackoff(ctx context.Context, ss *serverState, logPrefix string) {
|
||||
ss.mu.Lock()
|
||||
if ss.reconnAttempts >= maxReconnectAttempts {
|
||||
ss.lastErr = fmt.Sprintf("max reconnect attempts (%d) reached", maxReconnectAttempts)
|
||||
ss.lastErr = fmt.Sprintf("max reconnect attempts (%d) reached, entering cooldown", maxReconnectAttempts)
|
||||
ss.mu.Unlock()
|
||||
slog.Error("mcp.server.reconnect_exhausted", "server", ss.name)
|
||||
return
|
||||
slog.Warn(logPrefix+".reconnect_cooldown", "server", ss.name, "cooldown", reconnectCooldown)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(reconnectCooldown):
|
||||
}
|
||||
ss.mu.Lock()
|
||||
ss.reconnAttempts = 0
|
||||
ss.mu.Unlock()
|
||||
return // will retry on next health tick
|
||||
}
|
||||
ss.reconnAttempts++
|
||||
attempt := ss.reconnAttempts
|
||||
ss.mu.Unlock()
|
||||
|
||||
backoff := min(initialBackoff*time.Duration(1<<(attempt-1)), maxBackoff)
|
||||
|
||||
slog.Info("mcp.server.reconnecting",
|
||||
"server", ss.name,
|
||||
"attempt", attempt,
|
||||
"backoff", backoff,
|
||||
)
|
||||
slog.Info(logPrefix+".reconnecting", "server", ss.name, "attempt", attempt, "backoff", backoff)
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
@@ -327,13 +347,74 @@ func (m *Manager) tryReconnect(ctx context.Context, ss *serverState) {
|
||||
case <-time.After(backoff):
|
||||
}
|
||||
|
||||
// Try to ping again — transport may have auto-reconnected
|
||||
// Fast path: ping existing client — works for transient network blips
|
||||
// where the server-side session is still alive.
|
||||
if err := ss.client.Ping(ctx); err == nil {
|
||||
ss.connected.Store(true)
|
||||
ss.mu.Lock()
|
||||
ss.reconnAttempts = 0
|
||||
ss.healthFailures = 0
|
||||
ss.lastErr = ""
|
||||
ss.mu.Unlock()
|
||||
slog.Info("mcp.server.reconnected", "server", ss.name)
|
||||
slog.Info(logPrefix+".reconnected", "server", ss.name)
|
||||
return
|
||||
}
|
||||
|
||||
// Slow path: server-side session is dead (container restart, OOM, etc.).
|
||||
if fullReconnect(ctx, ss) {
|
||||
slog.Info(logPrefix+".reconnected", "server", ss.name, "method", "full_reconnect")
|
||||
}
|
||||
}
|
||||
|
||||
// fullReconnect creates a fresh MCP client, atomically swaps it into serverState,
|
||||
// and closes the old one. Returns true on success. Used by reconnectWithBackoff
|
||||
// as the slow path when pinging the old client fails.
|
||||
//
|
||||
// The new client is created and validated FIRST, then swapped via clientPtr.Store()
|
||||
// so BridgeTools see the new client immediately. The old client is closed AFTER
|
||||
// the swap to avoid a window where ss.client points to a closed client.
|
||||
//
|
||||
// NOTE: Does not re-discover tools (ListTools). If the MCP server restarts with
|
||||
// a different tool set, changes won't be reflected until the Manager reconnects.
|
||||
func fullReconnect(ctx context.Context, ss *serverState) bool {
|
||||
slog.Info("mcp.full_reconnect", "server", ss.name, "transport", ss.transport)
|
||||
|
||||
newClient, err := createClient(ss.transport, ss.conn.command, ss.conn.args, ss.conn.env, ss.conn.url, ss.conn.headers)
|
||||
if err != nil {
|
||||
slog.Warn("mcp.reconnect_create_failed", "server", ss.name, "error", err)
|
||||
return false
|
||||
}
|
||||
|
||||
if ss.transport != "stdio" {
|
||||
if err := newClient.Start(ctx); err != nil {
|
||||
_ = newClient.Close()
|
||||
slog.Warn("mcp.reconnect_start_failed", "server", ss.name, "error", err)
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
initReq := mcpgo.InitializeRequest{}
|
||||
initReq.Params.ProtocolVersion = mcpgo.LATEST_PROTOCOL_VERSION
|
||||
initReq.Params.ClientInfo = mcpgo.Implementation{Name: "goclaw", Version: "1.0.0"}
|
||||
|
||||
if _, err := newClient.Initialize(ctx, initReq); err != nil {
|
||||
_ = newClient.Close()
|
||||
slog.Warn("mcp.reconnect_init_failed", "server", ss.name, "error", err)
|
||||
return false
|
||||
}
|
||||
|
||||
// Swap atomically: store new client, then close old.
|
||||
// BridgeTools use clientPtr.Load() so they see the new client immediately.
|
||||
oldClient := ss.client
|
||||
ss.client = newClient
|
||||
ss.clientPtr.Store(newClient)
|
||||
ss.connected.Store(true)
|
||||
ss.mu.Lock()
|
||||
ss.reconnAttempts = 0
|
||||
ss.healthFailures = 0
|
||||
ss.lastErr = ""
|
||||
ss.mu.Unlock()
|
||||
|
||||
_ = oldClient.Close()
|
||||
return true
|
||||
}
|
||||
+26
-16
@@ -131,8 +131,8 @@ func (p *Pool) Acquire(ctx context.Context, tenantID uuid.UUID, name, transportT
|
||||
if old.state.cancel != nil {
|
||||
old.state.cancel()
|
||||
}
|
||||
if old.state.client != nil {
|
||||
_ = old.state.client.Close()
|
||||
if client := old.state.clientPtr.Load(); client != nil {
|
||||
_ = client.Close()
|
||||
}
|
||||
delete(p.servers, key)
|
||||
// Return slot to semaphore
|
||||
@@ -216,8 +216,8 @@ func (p *Pool) AcquireUser(ctx context.Context, tenantID uuid.UUID, name, userID
|
||||
if old.state.cancel != nil {
|
||||
old.state.cancel()
|
||||
}
|
||||
if old.state.client != nil {
|
||||
_ = old.state.client.Close()
|
||||
if client := old.state.clientPtr.Load(); client != nil {
|
||||
_ = client.Close()
|
||||
}
|
||||
delete(p.userServers, key)
|
||||
// Return slot to per-server semaphore
|
||||
@@ -390,8 +390,8 @@ func (p *Pool) Stop() {
|
||||
if entry.state.cancel != nil {
|
||||
entry.state.cancel()
|
||||
}
|
||||
if entry.state.client != nil {
|
||||
_ = entry.state.client.Close()
|
||||
if client := entry.state.clientPtr.Load(); client != nil {
|
||||
_ = client.Close()
|
||||
}
|
||||
slog.Debug("mcp.pool.stopped", "key", key)
|
||||
}
|
||||
@@ -401,8 +401,8 @@ func (p *Pool) Stop() {
|
||||
if entry.state.cancel != nil {
|
||||
entry.state.cancel()
|
||||
}
|
||||
if entry.state.client != nil {
|
||||
_ = entry.state.client.Close()
|
||||
if client := entry.state.clientPtr.Load(); client != nil {
|
||||
_ = client.Close()
|
||||
}
|
||||
slog.Debug("mcp.pool.user.stopped", "key", key)
|
||||
}
|
||||
@@ -423,8 +423,8 @@ func (p *Pool) Evict(tenantID uuid.UUID, serverName string) {
|
||||
if entry.state.cancel != nil {
|
||||
entry.state.cancel()
|
||||
}
|
||||
if entry.state.client != nil {
|
||||
_ = entry.state.client.Close()
|
||||
if client := entry.state.clientPtr.Load(); client != nil {
|
||||
_ = client.Close()
|
||||
}
|
||||
delete(p.servers, key)
|
||||
select {
|
||||
@@ -481,8 +481,8 @@ func (p *Pool) evictIdle() {
|
||||
if entry.state.cancel != nil {
|
||||
entry.state.cancel()
|
||||
}
|
||||
if entry.state.client != nil {
|
||||
_ = entry.state.client.Close()
|
||||
if client := entry.state.clientPtr.Load(); client != nil {
|
||||
_ = client.Close()
|
||||
}
|
||||
delete(p.servers, key)
|
||||
select {
|
||||
@@ -499,8 +499,8 @@ func (p *Pool) evictIdle() {
|
||||
if entry.state.cancel != nil {
|
||||
entry.state.cancel()
|
||||
}
|
||||
if entry.state.client != nil {
|
||||
_ = entry.state.client.Close()
|
||||
if client := entry.state.clientPtr.Load(); client != nil {
|
||||
_ = client.Close()
|
||||
}
|
||||
delete(p.userServers, key)
|
||||
// Return slot to per-server semaphore
|
||||
@@ -588,8 +588,9 @@ func (p *Pool) evictOldestIdleLocked() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// Client returns the MCP client for this pool entry.
|
||||
func (e *poolEntry) Client() *mcpclient.Client { return e.state.client }
|
||||
// ClientPtr returns the atomic client pointer for this pool entry.
|
||||
// Used by BridgeTools to atomically load the current client during reconnect.
|
||||
func (e *poolEntry) ClientPtr() *atomic.Pointer[mcpclient.Client] { return &e.state.clientPtr }
|
||||
|
||||
// Connected returns a pointer to the connected flag for this pool entry.
|
||||
func (e *poolEntry) Connected() *atomic.Bool { return &e.state.connected }
|
||||
@@ -598,6 +599,8 @@ func (e *poolEntry) Connected() *atomic.Bool { return &e.state.connected }
|
||||
func (e *poolEntry) MCPTools() []mcpgo.Tool { return e.tools }
|
||||
|
||||
// poolHealthLoop is a standalone health loop for pool-managed connections.
|
||||
// After consecutive ping failures, it attempts a full reconnect by creating
|
||||
// a fresh client, mirroring the Manager.tryReconnect slow path.
|
||||
func poolHealthLoop(ctx context.Context, ss *serverState) {
|
||||
ticker := newHealthTicker()
|
||||
defer ticker.Stop()
|
||||
@@ -625,6 +628,7 @@ func poolHealthLoop(ctx context.Context, ss *serverState) {
|
||||
|
||||
if failures >= healthFailThreshold {
|
||||
ss.connected.Store(false)
|
||||
poolTryReconnect(ctx, ss)
|
||||
}
|
||||
} else {
|
||||
ss.connected.Store(true)
|
||||
@@ -637,3 +641,9 @@ func poolHealthLoop(ctx context.Context, ss *serverState) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// poolTryReconnect attempts reconnect for a pool-managed connection.
|
||||
// Delegates to the shared reconnectWithBackoff with pool-specific log prefix.
|
||||
func poolTryReconnect(ctx context.Context, ss *serverState) {
|
||||
reconnectWithBackoff(ctx, ss, "mcp.pool")
|
||||
}
|
||||
@@ -20,6 +20,18 @@ type InjectParams struct {
|
||||
UserID string
|
||||
TenantID string
|
||||
UserMessage string
|
||||
|
||||
// RecentContext carries a short snippet of recent conversation (typically
|
||||
// the last 1-2 user turns concatenated) used to enrich the search query.
|
||||
// Context-aware recall: without this, vector search on "what's my favorite?"
|
||||
// misses memories about the topic under discussion. With it, the query
|
||||
// embedding captures conversational intent and returns materially better
|
||||
// matches for follow-up questions.
|
||||
//
|
||||
// Empty = legacy behaviour (search on UserMessage only).
|
||||
// Target length: ≤ ~400 chars. Longer context dilutes the embedding.
|
||||
RecentContext string
|
||||
|
||||
MaxEntries int // default 5
|
||||
MaxTokens int // default 200
|
||||
Threshold float64 // relevance threshold (default 0.3)
|
||||
|
||||
@@ -41,8 +41,15 @@ func (a *pgAutoInjector) Inject(ctx context.Context, params InjectParams) (*Inje
|
||||
threshold = 0.3
|
||||
}
|
||||
|
||||
// Phase 9: context-aware recall. When the caller supplied RecentContext,
|
||||
// build a richer search query that captures conversational intent. Without
|
||||
// this, vector search on "what's my favorite?" misses memories about the
|
||||
// topic under discussion. With it, the query embedding captures the
|
||||
// follow-up semantics and returns materially better matches.
|
||||
searchQuery := buildRecallQuery(params.UserMessage, params.RecentContext)
|
||||
|
||||
// Search with FTS bias (faster than vector for auto-inject)
|
||||
results, err := a.episodicStore.Search(ctx, params.UserMessage, params.AgentID, params.UserID,
|
||||
results, err := a.episodicStore.Search(ctx, searchQuery, params.AgentID, params.UserID,
|
||||
store.EpisodicSearchOptions{
|
||||
MaxResults: maxEntries * 2, // fetch more, filter by threshold
|
||||
MinScore: threshold,
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
package memory
|
||||
|
||||
import "strings"
|
||||
|
||||
// maxRecallContextRunes bounds the recent-context snippet used to enrich the
|
||||
// recall query. Longer snippets dilute the embedding signal and slow down the
|
||||
// vector search; shorter snippets lose the conversational frame.
|
||||
//
|
||||
// 400 runes ≈ 100 tokens in Latin scripts, fewer in CJK (1-2 short user turns
|
||||
// either way). Tuning knob; change only if recall quality metrics show a
|
||||
// clear trend in either direction.
|
||||
//
|
||||
// Unit is runes, not bytes, because GoClaw supports vi/zh locales: a
|
||||
// byte-wise tail-clip would slice a multi-byte rune in half and emit invalid
|
||||
// UTF-8 to the embedding model.
|
||||
const maxRecallContextRunes = 400
|
||||
|
||||
// buildRecallQuery concatenates the user's latest message with a short recent
|
||||
// context snippet to produce a context-aware search query. Used by auto-inject
|
||||
// to improve recall on follow-up questions where the current message alone is
|
||||
// ambiguous (pronouns, implicit references, one-word replies).
|
||||
//
|
||||
// The recent context is truncated to maxRecallContextRunes (rune-safe for
|
||||
// CJK/vi/zh input) and prepended so embedding models give the latest message
|
||||
// the most weight (position bias). Empty context or empty message return the
|
||||
// unmodified input — zero-risk fallback for legacy callers that don't supply
|
||||
// RecentContext yet.
|
||||
func buildRecallQuery(userMessage, recentContext string) string {
|
||||
userMessage = strings.TrimSpace(userMessage)
|
||||
if recentContext == "" || userMessage == "" {
|
||||
return userMessage
|
||||
}
|
||||
|
||||
ctx := strings.TrimSpace(recentContext)
|
||||
ctx = tailClipRunes(ctx, maxRecallContextRunes)
|
||||
|
||||
// Prepend context with a lightweight separator. "Context:" and "Query:"
|
||||
// tags help instruction-tuned embedding models distinguish frame from
|
||||
// focus; for non-instruction models they act as neutral separators.
|
||||
var sb strings.Builder
|
||||
sb.Grow(len(userMessage) + len(ctx) + 32)
|
||||
sb.WriteString("Context: ")
|
||||
sb.WriteString(ctx)
|
||||
sb.WriteString("\nQuery: ")
|
||||
sb.WriteString(userMessage)
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// tailClipRunes returns the last maxRunes runes of s, rune-safe for multi-byte
|
||||
// scripts (Vietnamese, Chinese, Japanese, emoji). If s has fewer runes than
|
||||
// maxRunes it's returned unchanged. Byte-wise slicing would split multi-byte
|
||||
// runes and emit invalid UTF-8.
|
||||
func tailClipRunes(s string, maxRunes int) string {
|
||||
if maxRunes <= 0 {
|
||||
return ""
|
||||
}
|
||||
runes := []rune(s)
|
||||
if len(runes) <= maxRunes {
|
||||
return s
|
||||
}
|
||||
return string(runes[len(runes)-maxRunes:])
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
package memory
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestBuildRecallQuery_NoContextReturnsMessageUnchanged(t *testing.T) {
|
||||
got := buildRecallQuery("what's my favorite?", "")
|
||||
if got != "what's my favorite?" {
|
||||
t.Errorf("empty context should return message unchanged, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRecallQuery_EmptyMessageReturnsEmpty(t *testing.T) {
|
||||
got := buildRecallQuery("", "some recent context")
|
||||
if got != "" {
|
||||
t.Errorf("empty message should return empty, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRecallQuery_CombinesContextAndMessage(t *testing.T) {
|
||||
ctx := "We were talking about coffee shops in downtown."
|
||||
msg := "what's my favorite?"
|
||||
got := buildRecallQuery(msg, ctx)
|
||||
|
||||
if !strings.Contains(got, msg) {
|
||||
t.Errorf("result must contain user message, got %q", got)
|
||||
}
|
||||
if !strings.Contains(got, ctx) {
|
||||
t.Errorf("result must contain recent context, got %q", got)
|
||||
}
|
||||
// Query label must come AFTER context so position-biased embeddings give
|
||||
// the message the strongest signal.
|
||||
ctxIdx := strings.Index(got, "Context:")
|
||||
queryIdx := strings.Index(got, "Query:")
|
||||
if ctxIdx == -1 || queryIdx == -1 || ctxIdx >= queryIdx {
|
||||
t.Errorf("Context must precede Query label, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRecallQuery_TruncatesOversizedContext(t *testing.T) {
|
||||
// 1000-char context, maxRecallContextChars = 400
|
||||
ctx := strings.Repeat("a", 1000)
|
||||
msg := "test"
|
||||
got := buildRecallQuery(msg, ctx)
|
||||
|
||||
// Stripped context section should be at most maxRecallContextRunes long.
|
||||
// Full output has prefix overhead ("Context: " + "\nQuery: " + msg) ~ 20 chars.
|
||||
// Total should be < 500 chars (bounded by truncation + prefix + msg).
|
||||
if len(got) > maxRecallContextRunes+64 {
|
||||
t.Errorf("oversized context not truncated: got %d chars, want ≤ %d",
|
||||
len(got), maxRecallContextRunes+64)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRecallQuery_TailClipsLongContext(t *testing.T) {
|
||||
// Tail-clip keeps the most recent portion (last N chars) since it's
|
||||
// closer to the current turn in conversation time.
|
||||
ctx := strings.Repeat("OLD ", 200) + strings.Repeat("NEW ", 20) // 820 chars, "NEW" at the end
|
||||
got := buildRecallQuery("follow-up", ctx)
|
||||
if !strings.Contains(got, "NEW") {
|
||||
t.Errorf("tail-clip should preserve recent ('NEW') portion, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRecallQuery_TrimsWhitespace(t *testing.T) {
|
||||
got := buildRecallQuery(" query ", " ctx ")
|
||||
if !strings.Contains(got, "Context: ctx") {
|
||||
t.Errorf("context should be trimmed, got %q", got)
|
||||
}
|
||||
if !strings.Contains(got, "Query: query") {
|
||||
t.Errorf("message should be trimmed, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildRecallQuery_UnicodeSafeTailClip verifies rune-safe truncation for
|
||||
// Vietnamese / Chinese input. Byte-wise slicing would produce invalid UTF-8 at
|
||||
// multi-byte rune boundaries — embedding models on that garbage return lower
|
||||
// quality matches, defeating Phase 9's whole point.
|
||||
func TestBuildRecallQuery_UnicodeSafeTailClip(t *testing.T) {
|
||||
// 500 Vietnamese tiếng Việt runes (each char = 2-3 bytes)
|
||||
viChar := "ế" // 3 bytes in UTF-8
|
||||
ctx := strings.Repeat(viChar, 500)
|
||||
msg := "câu hỏi"
|
||||
got := buildRecallQuery(msg, ctx)
|
||||
// The result must contain valid UTF-8 — no half-runes at the clip boundary.
|
||||
// If the clip cut a rune, the resulting string would contain 0xEF 0xBF 0xBD
|
||||
// replacement chars or fail strings.ContainsRune checks.
|
||||
if !strings.Contains(got, msg) {
|
||||
t.Errorf("message missing from result, got %q", got)
|
||||
}
|
||||
// Must still contain the Vietnamese marker char
|
||||
if !strings.Contains(got, viChar) {
|
||||
t.Errorf("vietnamese context missing, got %q", got)
|
||||
}
|
||||
// Every rune must be valid (conversion round-trip)
|
||||
for _, r := range got {
|
||||
if r == '\uFFFD' {
|
||||
t.Errorf("result contains replacement rune (invalid UTF-8 clip boundary), got %q", got)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTailClipRunes(t *testing.T) {
|
||||
if got := tailClipRunes("hello", 10); got != "hello" {
|
||||
t.Errorf("short input should be unchanged, got %q", got)
|
||||
}
|
||||
if got := tailClipRunes("abcdef", 3); got != "def" {
|
||||
t.Errorf("expected 'def', got %q", got)
|
||||
}
|
||||
// Chinese: 10 Han chars, clip to 3
|
||||
if got := tailClipRunes("一二三四五六七八九十", 3); got != "八九十" {
|
||||
t.Errorf("expected '八九十', got %q", got)
|
||||
}
|
||||
if got := tailClipRunes("anything", 0); got != "" {
|
||||
t.Errorf("zero maxRunes should return empty, got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package pipeline
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bootstrap"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
@@ -34,6 +35,21 @@ func (s *ContextStage) Execute(ctx context.Context, state *RunState) error {
|
||||
state.Ctx = ctx
|
||||
}
|
||||
|
||||
// 0.5. Resolve the effective context window for this run's provider/model.
|
||||
// Done once here so PruneStage reads a stable value on every iteration and
|
||||
// the budget can't drift if the model somehow changes mid-run. A zero
|
||||
// result from the resolver (unknown model, no registry) leaves the field
|
||||
// zero — PruneStage then falls back to Config.ContextWindow.
|
||||
if s.deps.ResolveContextWindow != nil && state.Model != "" {
|
||||
providerID := ""
|
||||
if state.Provider != nil {
|
||||
providerID = state.Provider.Name()
|
||||
}
|
||||
if cw := s.deps.ResolveContextWindow(providerID, state.Model); cw > 0 {
|
||||
state.Context.EffectiveContextWindow = cw
|
||||
}
|
||||
}
|
||||
|
||||
// 1. Resolve workspace
|
||||
if s.deps.ResolveWorkspace != nil {
|
||||
ws, err := s.deps.ResolveWorkspace(ctx, state.Input)
|
||||
@@ -96,8 +112,11 @@ func (s *ContextStage) Execute(ctx context.Context, state *RunState) error {
|
||||
|
||||
// 8. Auto-inject L0 memory context into system prompt.
|
||||
// V3RetrievalEnabled check removed — auto-inject runs whenever AutoInject is available.
|
||||
// Phase 9: pass recent conversation context so vector search can resolve
|
||||
// pronouns and implicit references in follow-up questions.
|
||||
if s.deps.AutoInject != nil && state.Input.Message != "" {
|
||||
section, err := s.deps.AutoInject(ctx, state.Input.Message, state.Input.UserID)
|
||||
recentCtx := buildRecentContext(state.Messages.History())
|
||||
section, err := s.deps.AutoInject(ctx, state.Input.Message, state.Input.UserID, recentCtx)
|
||||
if err == nil && section != "" {
|
||||
state.Context.MemorySection = section
|
||||
sys := state.Messages.System()
|
||||
@@ -109,6 +128,50 @@ func (s *ContextStage) Execute(ctx context.Context, state *RunState) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// buildRecentContext concatenates the trailing user turns from the history
|
||||
// buffer into a short snippet suitable for enriching a recall query. Walks
|
||||
// backward from the end so we keep the most recent turns regardless of
|
||||
// earlier messages being pruned. Returns "" when there's no usable context.
|
||||
//
|
||||
// Budget: up to 2 user turns, max ~300 runes total. Rune (not byte) cap keeps
|
||||
// vi/zh locales safe — a byte-wise clip would slice multi-byte characters
|
||||
// and emit invalid UTF-8 to the embedding model. Tuning knob is intentional
|
||||
// here rather than config-driven — Phase 9 adds it only if operational data
|
||||
// shows variance across agent types.
|
||||
func buildRecentContext(history []providers.Message) string {
|
||||
const maxTurns = 2
|
||||
const maxRunes = 300
|
||||
if len(history) == 0 {
|
||||
return ""
|
||||
}
|
||||
turns := make([]string, 0, maxTurns)
|
||||
for i := len(history) - 1; i >= 0 && len(turns) < maxTurns; i-- {
|
||||
m := history[i]
|
||||
if m.Role != "user" || m.Content == "" {
|
||||
continue
|
||||
}
|
||||
turns = append([]string{m.Content}, turns...) // prepend to preserve order
|
||||
}
|
||||
if len(turns) == 0 {
|
||||
return ""
|
||||
}
|
||||
var sb strings.Builder
|
||||
for i, t := range turns {
|
||||
if i > 0 {
|
||||
sb.WriteString(" | ")
|
||||
}
|
||||
sb.WriteString(t)
|
||||
}
|
||||
joined := sb.String()
|
||||
// Rune-safe tail clip: keep the most recent portion of the conversation
|
||||
// (closest in time to the current turn) rather than the oldest.
|
||||
runes := []rune(joined)
|
||||
if len(runes) > maxRunes {
|
||||
joined = string(runes[len(runes)-maxRunes:])
|
||||
}
|
||||
return joined
|
||||
}
|
||||
|
||||
// toAnySlice converts []bootstrap.ContextFile to []any for ContextState.ContextFiles.
|
||||
// Phase 8 will remove this when ContextState uses typed field.
|
||||
func toAnySlice(files []bootstrap.ContextFile) []any {
|
||||
|
||||
@@ -18,12 +18,20 @@ type PipelineDeps struct {
|
||||
EventBus eventbus.DomainEventBus
|
||||
Config PipelineConfig
|
||||
|
||||
// ResolveContextWindow returns the effective context window (in tokens) for
|
||||
// a given provider/model pair. Nil = always use Config.ContextWindow.
|
||||
// Invoked ONCE per run by ContextStage and stored in RunState.Context.EffectiveContextWindow.
|
||||
ResolveContextWindow func(provider, model string) int
|
||||
|
||||
// Callbacks from agent.Loop — Phase 8 adapter wires these.
|
||||
EmitEvent func(event any)
|
||||
|
||||
// Auto-inject memory context (ContextStage, L0 tier).
|
||||
// Callback captures agent/tenant context via closure.
|
||||
AutoInject func(ctx context.Context, userMessage, userID string) (string, error)
|
||||
// Callback captures agent/tenant context via closure. recentContext carries
|
||||
// a short snippet of recent conversation (last 1-2 user turns) so the
|
||||
// downstream recall query can resolve pronouns and implicit references.
|
||||
// Empty recentContext = legacy single-message search semantics.
|
||||
AutoInject func(ctx context.Context, userMessage, userID, recentContext string) (string, error)
|
||||
|
||||
// InjectContext sets up agent/tenant/user/workspace/tool context values.
|
||||
// Wraps injectContext() for v3 pipeline. Called once at ContextStage start.
|
||||
@@ -92,6 +100,14 @@ type PipelineConfig struct {
|
||||
MaxTokens int
|
||||
Compaction *config.CompactionConfig
|
||||
|
||||
// ReserveTokens is a safety buffer subtracted from the history budget so
|
||||
// PruneStage compacts slightly before the hard limit. Prevents edge cases
|
||||
// where a provider returns more than MaxTokens output or where the token
|
||||
// counter's estimate drifts upward during streaming.
|
||||
// Zero (default) preserves legacy behavior: budget = contextWindow - overhead - MaxTokens.
|
||||
// Recommended: 5-10% of contextWindow for reasoning-heavy models.
|
||||
ReserveTokens int
|
||||
|
||||
// V3 memory/retrieval flags removed — always true at runtime.
|
||||
// Memory flush runs if callback != nil; auto-inject runs if AutoInject != nil.
|
||||
}
|
||||
@@ -31,8 +31,19 @@ func (s *PruneStage) Result() StageResult { return s.result }
|
||||
func (s *PruneStage) Execute(ctx context.Context, state *RunState) error {
|
||||
s.result = Continue
|
||||
|
||||
// Compute budget: context window minus overhead (system prompt + context files) minus output reserve
|
||||
budget := s.deps.Config.ContextWindow - state.Context.OverheadTokens - s.deps.Config.MaxTokens
|
||||
// Compute budget using the effective context window for this run's model.
|
||||
// ContextStage resolves EffectiveContextWindow once per run via ModelRegistry;
|
||||
// if zero (unknown model, registry not wired) fall back to the pipeline-wide
|
||||
// Config.ContextWindow for backward compatibility.
|
||||
//
|
||||
// ReserveTokens (optional, default 0) carves out a safety buffer so compaction
|
||||
// fires slightly before the hard limit — protects against provider over-delivery
|
||||
// and token-counter drift on streaming responses.
|
||||
contextWindow := state.Context.EffectiveContextWindow
|
||||
if contextWindow == 0 {
|
||||
contextWindow = s.deps.Config.ContextWindow
|
||||
}
|
||||
budget := contextWindow - state.Context.OverheadTokens - s.deps.Config.MaxTokens - s.deps.Config.ReserveTokens
|
||||
if budget <= 0 {
|
||||
return nil // no history budget, nothing to prune
|
||||
}
|
||||
|
||||
Loaded 100 of 229 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user