mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-11 03:13:24 +00:00
dev
47
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 thotam
|
3f90057c24 |
fix(pipeline): stop aborting runs on heuristic context budget estimates (#1587)
Runs on models without a registered tokenizer (e.g. 9router brand models)
ended with the generic "Agent couldn't generate a response" fallback even
though the real request used about 55% of the context window.
PruneStage counted history with TokenCounter, which falls back to a
chars/2 heuristic for unregistered models and overcounted about 1.8x.
Once over budget it ran memory flush (~35s, invisible in traces), then
mid-loop compaction, which cannot summarize a history made only of tool
call/result pairs. The callback reported the untouched history as
compacted, PruneStage still saw it over budget and returned AbortRun
before any LLM call, and FinalizeStage replaced the empty reply with the
fallback.
- PruneStage and ContextStage overhead count with the request guard's
BudgetCounter. PruneStage no longer controls loop flow; the final
request guard in ThinkStage decides.
- CompactMessages returns ErrNotCompacted when history is unchanged.
Callers stop counting it as a compaction and do not retry it in the
same run, while post-run summarization still sees the pressure.
- When the guard exhausts every reduction step, ThinkStage stops the run
with a localized chat.context_budget_exceeded notice instead of an
error, so the run's tool results are still persisted. The stop reason
marks the trace and agent span as error; team tasks, cron and
heartbeat treat it as a failure via RunOutcome.Failure().
- Memory flush and mid-loop compaction emit event spans.
- Web and desktop UIs treat an unset context_pruning as enabled (the
backend default since
|
||
|
isaacgao4396andisaacgao4396
|
169e0bafaf |
fix(tools): use_skill inlines content when read_file isn't granted (#1547)
* feat(store): add context helper for per-iteration tool allowlist * feat(pipeline): surface resolved tool allowlist to tool dispatch via context * fix(tools): use_skill inlines content when read_file isn't granted Fixes #1477 --------- Co-authored-by: isaacgao4396 <gaoyuan4396@gmail.com> |
||
|
Minh Dang
|
45680e9ff5 |
fix(pipeline): stop delivering the same generated image twice
An agent that generates an image and then attaches a copy of it under a different name produced two media entries for one picture, and both were sent. In production every image-edit reply arrived as two identical attachments: the file create_image wrote (slug_timestamp_rand.png) plus the path the agent passed to send_file, byte-identical but stored twice. processMedia deduplicated by path only, so the pair survived. It also ran before generated images were folded into MediaResults, leaving that producer past the only dedupe pass entirely. Fold persisted assistant images into MediaResults first, then deduplicate once over the union, matching identical local files by content digest. The assistant message is now built from that same deduplicated set, so session history and outbound delivery cannot disagree. Media that cannot be read (remote or already cleaned up) is left untouched. |
||
|
Justin Truong
|
d13cc0804a |
fix(providers): carry tool name on tool results for Gemini
Gemini drops tool_call_id and pairs functionCall/functionResponse by function name, so its OpenAI-compat shim requires a non-empty FunctionResponse.name. The name was only recoverable through a reverse id->name lookup over assistant tool_calls still present in the request window, which fails after pruning, truncation or tool_call collapse and produces HTTP 400 "Name cannot be empty" on any follow-up iteration. Add Message.ToolName, set it at every tool-result creation site and preserve it through context pruning, then prefer it when serializing. Fall back to the existing index for history persisted before the field, so old sessions keep working without a reset. When neither source resolves a name, drop the unlabelled tool result instead of emitting an empty one: an empty name is a guaranteed 400 and a synthetic name would match no prior functionCall. Gated behind the existing Gemini detection so other OpenAI-compat hosts, which pair by tool_call_id, are unaffected. Field is JSON-optional, so session history needs no migration and older binaries ignore it on rollback. |
||
|
bilogic
|
6e61bd1531 | stop responding with "..." (#1491) | ||
|
nguyenha935andnguyenha935
|
31f973477d |
test(pipeline): lock mid-loop compaction pressure flag end-to-end (#1479)
FinalizeStage forwards state.Prune.MidLoopCompacted into MaybeSummarize so a mid-loop compaction gets persisted (TruncateHistory + IncrementCompaction) instead of being thrown away and redone next turn. Nothing asserted the flag value: TestFinalizeStage_MaybeSummarize_Called binds it as `_ bool`, so hardcoding `false` at the call site keeps the whole suite green while silently restoring the per-turn re-compaction loop and stalling the cumulative compaction count episodic depends on. Adds two cases driving the real NewDefaultPipeline().Run(): - a history large enough to force prune-stage compaction must reach MaybeSummarize with midLoopCompacted=true - a small history must compact nothing and report false, so the flag cannot simply be pinned true Mutation-verified: with the finalize call site hardcoded to false the new test fails while the pre-existing pipeline and agent suites still pass. Test-only change. Co-authored-by: nguyenha935 <nguyenha935@users.noreply.github.com> |
||
|
Clark Cant
|
5ca433b8ac |
fix: persist mid-loop compaction to stop re-compaction loop and revive episodic
The v3 pipeline compacts session history mid-loop (prune_stage + final-request guard) but only mutates the run's message buffer, never the session store. Each turn reloads full history and re-compacts from scratch: message_tokens climb 129k->156k across turns while every turn compacts back down to ~60k. The lossy compaction differs per run, degrading the agent. The same missing persistence stalls episodic memory: the cumulative compaction count never advances, so the episodic worker's idempotency key (sessionKey:count) is pinned and every cycle after the first is skipped. Observed on live traffic: 8 run.completed since deploy, 0 new episodic. Fixes, all reusing existing machinery (no new store methods, no migrations): - Bug A: emitSessionCompleted reads cumulative GetCompactionCount (matching the legacy v2 path) instead of the per-run counter that resets to 0. - Bug B/anti-loop: finalize passes state.Prune.MidLoopCompacted into maybeSummarize; under pressure it lowers the trigger to a unit-aligned threshold (compactionInputCap - overhead, same MaxRequestShare the guard uses) so the compaction is PERSISTED via the existing TruncateHistory + IncrementCompaction path. Defensive floor prevents over-compaction on pathological config; tool-result-only bloat still skips (history-only). - Bug C: SourceID embeds the count (sessionKey:count) so the eventbus dedup key advances per compaction cycle instead of swallowing rapid same-session turns within the 5m TTL. Tests: episodic compaction, maybe_summarize pressure, request budget. go build (PG + sqliteonly), go vet, go test -race all green. |
||
|
thotam
|
503909d3b6 |
fix(sessions): store last-call context tokens instead of run-cumulative total
SetLastPromptTokens received TotalUsage.PromptTokens — the sum of prompt tokens across every think→act→observe iteration of a run — instead of the final iteration's own usage. A tool-heavy run inflated the stored value by the iteration count, so the sessions list showed e.g. 901K "context used" against a 258K window, and maybeSummarize over-triggered compaction once the inflation exceeded the 40% overhead clamp (observed: 83 compactions on a 4-message session). Track the final iteration's usage separately (ThinkState.LastUsage) and persist its actual context size: prompt tokens + cached segments (Anthropic reports input_tokens excluding cache read/creation) + output tokens (the final reply joins history, so it occupies the next request's prompt). The run-cumulative total still feeds AccumulateTokens for billing. |
||
|
ntduc
|
03a196e613 | feat(hooks): add script context injection | ||
|
|
293aa2a5e1 |
(feat) MCP Server (#1365)
* feat(hooks/observe): add structured output validation hook (ObserveHook) - hooks/types.go: Add ObserveHook interface + BuiltInHookType enum - hooks/dispatcher.go: HookDispatcher emits ObserveHook with PhaseResult payload - pipeline/observe_stage.go: ObserveStage emits hook after ObserveResult built - pipeline/substates.go: ObserveStageResult carries hook results + validation errors - hooks/config.go: BuiltInHookTypeObserve added to BuiltInHookType enum PhaseResult carries structured output, token usage, tool calls + validation errors emitted post-ObserveStage. ObserveHook implementations can validate structured output against schemas, detect tool-call loops, enforce token budgets, etc. Hook fires after ObserveStage produces ObserveResult, before results propagate to next stage. ValidationError returned by hook halts pipeline and propagates error to caller without further stage execution. Co-Authored-By: Claude <noreply@anthropic.com> * ui(hooks): add post_model_response event to web UI - Add event to Zod schema, filter dropdown, and form dialog - Implement conditional test panel UI for model response payload - Add translations (en/zh/vi) for new test panel fields - Updated beta description to reference the new event Co-Authored-By: Claude <noreply@anthropic.com> * feat(mcp): add MCP CRUD server exposing goclaw resources at /api/mcp/ with Bearer token auth and X-GoClaw-Tenant-Id header, default to master tenant * feat(mcp): add goclaw_skills_write_file tool to edit skill files on disk The CRUD MCP server's goclaw_skills_update only touched skill DB metadata, with no way to edit a skill's SKILL.md/file content on the filesystem. Extract the versioned write logic from the web UI's skill file editor (SkillsHandler.handleWriteFile) into skills.WriteVersionedFile so both surfaces share identical validation and versioning, and expose it as a new MCP tool. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Bruno Clermont <bruno.clermont@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
thotam
|
0c1ededc92 |
feat(webhooks): return per-call usage breakdown with provider/model/cost (#1421)
feat(webhooks): per-call usage breakdown with provider/model/cost (#1421) |
||
|
thotam
|
e95851e690 |
fix(pipeline): aggregate cache tokens across agent turns (#1419)
ThinkStage folded each turn's usage into state.Think.TotalUsage but summed only prompt/completion/total/thinking tokens, dropping CacheReadTokens, CacheCreationTokens and the PromptTokensIncludeCachedSegments flag. The aggregated RunResult.Usage is the source for webhook usage responses and usage_events analytics, so async multi-turn agent runs reported zero cache tokens even when traces showed heavy prompt-cache use (cache is read per-response for spans, but never summed into the aggregate). Sum cache read/creation across turns and OR the include-cached flag (a provider-level property, consistent across a run). Restores correct cache accounting for webhook usage and usage_events without any envelope change. |
||
|
|
db69ccf882 |
tracing: nest tool calls under LLM spans, emit hook spans, fix usage_events FK (#1415)
- Reparent tool_call spans under their producing llm_call span via RunState.CurrentLLMSpanID so a model turn's trace visibly contains the tool calls it triggered. - Wire internal/hooks EmitHookSpan into the dispatcher writeExec so every hook execution (pre/post tool use, etc.) produces a trace span with input, console output, decision, error, and duration for agent troubleshooting. - Fix usage_events_span_id_fkey violations: usage events were inserted synchronously referencing a span flushed ~5s later, silently dropping token/cost data. Route them through the collector so they flush after spans in the same cycle. No schema migration; FK preserved. Co-authored-by: Bruno Clermont <bruno.clermont@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Duc Nguyen
|
b9a69e355e |
fix(pipeline): deliver generated images to chat channels, not just session history (#1351)
Codex native image_generation output (state.Observe.AssistantImages) was persisted to workspace/media/ and attached to the assistant message's MediaRefs for session history, but never added to state.Tool.MediaResults — the only field that flows into RunResult.Media and from there into the outbound channel message. Generated images showed up in the web UI (which reads session history) but chat channels (Telegram, Zalo, Slack, ...) received text-only replies with no attachment. Append the persisted image refs to state.Tool.MediaResults right after they're attached to the assistant message, so they reach RunResult.Media via the existing pipeline plumbing and cmd/gateway_consumer_normal.go's appendMediaToOutbound — the same path create_image/send_file already use. No new delivery path needed. |
||
|
nguyenha935andGoClaw Operator
|
4d3c6bcb05 |
feat: add Telegram manager channel permissions (#1339)
Co-authored-by: GoClaw Operator <operator@goclaw> |
||
|
nguyenha935andGoClaw Operator
|
22efff8c0d |
fix: filter streamed thinking tags (#1300)
Co-authored-by: GoClaw Operator <operator@goclaw> |
||
|
nguyenha935andGoClaw Operator
|
64596bf0c1 |
fix: preflight compact v3 context before model calls (#1297)
Run pruning before ThinkStage so oversized histories compact before the provider request is built. Persist V3 last_prompt_tokens with the post-flush persisted message count so post-run compaction calibration has current usage data. Tests: go test ./internal/pipeline -run "TestNewDefaultPipeline_PrunesBeforeThink|TestFinalizeStage_UpdateMetadataReceivesPersistedMessageCount" -count=1; go test ./internal/agent -run "^$" -count=1 Co-authored-by: GoClaw Operator <operator@goclaw> |
||
|
Thanh_Dang
|
68ad5953d7 |
fix(pipeline): defer non-tool messages until all tool results are emitted (#1270)
Multi-tool responses could interleave synthetic user messages (loop warnings, nudges) between tool results, breaking OpenAI-compatible providers that require contiguous tool_result grouping. Fix both sequential and parallel tool execution paths in tool_stage.go to defer non-tool messages until after all tool results are appended. Fix sanitizeHistory lookahead in loop_history_sanitize.go to reorder interleaved non-tool messages in persisted session history. Closes #1177 |
||
|
thotam
|
389640ae51 |
fix(agent): check nil td.Function to prevent panic on native tools (#1192)
* fix(agent): check nil td.Function to prevent panic on native tools Fix nil pointer dereference panic when agent processes native tools (such as image_generation) which do not contain function schema wrappers. Key changes: - loop_pipeline_callbacks.go & think_stage.go: add checks for td.Function != nil before accessing td.Function.Name. - loop_tool_filter.go & loop_history_toolnames.go: add safety checks when filtering and extracting tool names. - anthropic_request.go & codex_build.go: verify t.Function != nil when formatting tools for provider API requests. * test: add regression tests for native tools nil function panic Add targeted regression tests to verify that: - ThinkStage builds AllowedTools without panicking when mixed with native tools - Loop's buildFilteredTools filters native tools correctly without panicking across all filters - Anthropic request builder skips native tools safely - Codex request builder handles native tools with nil Function fields gracefully |
||
|
Goon
|
152b2ac0f7 | fix(pipeline): preserve mid-flight follow-up requests | ||
|
Goon
|
59a1773a18 | Merge remote-tracking branch 'origin/dev' into codex/merge-nextlevelbuilder-goclaw | ||
|
Goon
|
a63080714a |
Merge remote-tracking branch 'upstream/dev' into codex/merge-nextlevelbuilder-goclaw
# Conflicts: # internal/pipeline/think_stage.go # internal/pipeline/tool_stage.go # tests/integration/git_adapter_ssh_test.go |
||
|
Duy /zuey/
|
23b18fa6cd |
fix(channels): rely on generated intermediate progress (#143)
* fix(channels): add reasoning delivery modes * fix(telegram): show tool status without placeholder * docs(plans): mark issue 67 behavior plan complete * feat(config): add behavior setting purpose tooltips * fix(channels): rely on generated intermediate progress |
||
|
|
90552531fb |
Fix/pipeline compaction pending loss (#1182)
* fix(pipeline): preserve pending messages during compaction When PruneStage or ThinkStage triggers mid-loop compaction, ReplaceHistory() clears the pending buffer. If the current iteration's assistant(tool_calls) message is in pending, it gets discarded. The next iteration then sends tool-result messages without a preceding tool_calls message, causing DeepSeek/OpenAI to reject the request with HTTP 400: Messages with role 'tool' must be a response to a preceding message with 'tool_calls' Fix: save pending before ReplaceHistory and restore it after. Reproduction: - Agent with 64K context window - 4-5 consecutive tool calls (e.g. read_file) in one session - Token count exceeds soft budget during ToolStage - PruneStage compaction clears pending including assistant(tool_calls) - Next iteration fails with HTTP 400 * test(pipeline): regression guards for pending preservation during compaction Add focused unit tests covering both ReplaceHistory call sites that previously dropped the pending buffer: - PruneStage mid-loop compaction must preserve the in-flight assistant(tool_calls) message so tool_calls -> tool_result pairing survives, preventing the OpenAI/DeepSeek 400 on the next provider call. - ThinkStage emergency overflow compaction must preserve already-staged pending messages (e.g. iteration-budget nudges) across the retry. Both tests fail without the preservation fix and pass with it. Co-authored-by: ollielin <2137816+ollielin@users.noreply.github.com> --------- Co-authored-by: Duy /zuey/ <duy@wearetopgroup.com> Co-authored-by: ollielin <ollielin@tencent.com> Co-authored-by: Thieu Nguyen <thieunguyen.tech@gmail.com> Co-authored-by: ollielin <2137816+ollielin@users.noreply.github.com> |
||
|
|
43837afca3 |
fix(security): consolidate & enhance batched security fixes (#1155, #967, #972, #974, #989, #973) (#1185)
* fix(sandbox): avoid shell in FsBridge writes Replace sh -c with interpolated path by shell-free 'tee -- <path>' argv form, piping content via stdin. Prevents command injection through filenames containing shell metacharacters inside the sandbox container. Co-authored-by: evgyur <evgyur@gmail.com> * fix(security): fail-closed on pairing DB errors across channels On IsPaired lookup error, deny instead of granting access. Covers the shared CheckDMPolicy/CheckGroupPolicy helpers (Slack/Discord/Feishu/WhatsApp/Zalo) and the four inline Telegram pairing checks. Co-authored-by: Srini <srinis.k@gmail.com> * fix(security): harden provider URL validation against SSRF Enforce scheme check for all provider types; restrict local types (ollama, claude_cli, acp) to an explicit localhost allowlist instead of skipping checks; resolve remote hostnames and reject any IP in a private/reserved range via the shared security.IsBlocked CIDR list (covers loopback, link-local, metadata, multicast, and unspecified 0.0.0.0/::). Closes the wildcard-DNS bypass and the local-type escape hatch. Operator opt-in via GOCLAW_ALLOW_PRIVATE_PROVIDER_URLS. Exports security.IsBlocked as the single source of truth for blocked ranges. Co-authored-by: Linh Vo Van <linh.vo@e-cq.net> * feat(pipeline): add fail-closed tool call authorization gate Gate tool execution against the server-side AllowedTools allowlist built from the RBAC/tenant-aware filtered tool set. Resolve the tool-call prefix before the allowlist lookup so prefixed agents are not wrongly blocked, re-check deny on lazy MCP activation, and expand IsDenied to cover aliased tool names. Co-authored-by: Huy Doan <tui@pm.me> * fix(security): expand file-serve deny-list defense-in-depth Add absolute-path deny prefixes (/home, /Users, /srv, /var/lib, /var/www, /opt) and an explicit fail-closed log when no file-serving boundary is configured. Co-authored-by: Linh Vo Van <linh.vo@e-cq.net> * fix(providers): allow claude cli executable paths Refs: #1185 --------- Co-authored-by: evgyur <evgyur@gmail.com> Co-authored-by: Srini <srinis.k@gmail.com> Co-authored-by: Linh Vo Van <linh.vo@e-cq.net> Co-authored-by: Huy Doan <tui@pm.me> |
||
|
Duy /zuey/
|
94cbaa9f0f |
fix(pipeline): recover truncated codex runs (#126)
* fix(pipeline): recover empty truncated codex responses * fix(providers): continue fallback after content policy errors |
||
|
Goon
|
a8afed2a81 | fix(channels): announce tools before execution | ||
|
Duy /zuey/
|
2a523e3f97 |
feat: add group chat context prompt
Squash merge PR #114 after resolving changelog and pipeline input conflicts with current dev. Local checks passed: go test ./internal/agent ./cmd ./internal/channels/..., go build ./..., go build -tags sqliteonly ./..., and go vet ./.... PR CI run 26705332779 passed release-versioning, go, and web. |
||
|
Duy /zuey/
|
484bcc7b71 |
fix: harden parallel tool scheduling
Squash merge PR #103 after validation. |
||
|
Duy /zuey/
|
ef829c1ab4 |
fix(bitrix24): resolve PR 8-10 stack findings
Resolve the PR #8/#9/#10 stack on current dev, including Bitrix24 install callback hardening, migration renumbering, duplicate-domain fail-closed routing, UI textarea/mobile cleanup, and review hardening. |
||
|
Goon
|
f55ea3dc26 | fix(pipeline): remove BOM from deps source | ||
|
Duy Nguyen
|
d48ae43e4f | feat(tools): add built-in wait tool | ||
|
Duy Nguyen
|
5a189330f9 | fix(test): support windows test execution | ||
|
Kai (Tam Nhu) Tranandviettranx
|
abb10976f7 |
feat(codex-pool,create_image): collapse primary_first + route pools through create_image chain (#1006)
* refactor(codex-pool): remove redundant primary_first strategy * test(tools): update tool schema fixtures to pointer form * feat(create_image): route Codex pools through chain with member failover Codex pool chain entries now iterate pool members per the pool's own strategy (round_robin or priority_order) and fail over internally before the outer chain advances to the next entry. - ChatGPTOAuthRouter.GenerateImage implements NativeImageProvider: iterates orderedProviders, tries each member, advances round-robin state only on success, aggregates errors on pool exhaustion. - media_provider_chain.wrapPoolProvider wraps a *CodexProvider in a router when RoutingDefaults has extras or a non-primary-first strategy. Solo Codex (no extras) stays unwrapped. - Router exposes ProviderType() so chain telemetry records "chatgpt_oauth". Closes #1008 * feat(ui/builtin-tools): pool badge on create_image chain entries Chain entry card shows a read-only "Pool · <strategy>" badge when the entry's provider is a Codex pool base. Strategy label translates in en/vi/zh. No new toggle or selector — pool config lives on the provider itself. #1008 * refactor(providers): move Codex test helpers to providertest subpackage Addresses code-review High finding: NewTestCodexProviderFast and its staticTestTokenSource were exported from a non-_test.go file, compiling into the production binary. Relocating to internal/providers/providertest/ keeps the helper importable from other packages' tests without leaking test-only symbols into production. Also tightens wrapPoolProvider — pools with zero extra members no longer get wrapped in a router (KISS: nothing to rotate between). - Added CodexProvider.WithRetryConfig as a legitimate fluent option (the test helper now uses the public API). - Dropped the internal-package test helper file. #1008 * docs(pr-1006): add pool badge UI evidence Force-UI style capture of the read-only "Pool · Round-robin" badge on the create_image chain entry card when the selected provider carries settings.codex_pool with extras. Captured against staging gateway. * refactor(ui/builtin-tools): unify pool UX with Create Agent pattern The create_image chain Provider dropdown was listing pool members alongside pool owners, letting users accidentally bypass pool semantics by picking a member directly. Matches the pattern already used by Create Agent: hide pool members, show an inline "Pool" chip on owners. - Filter pool members from the chain Provider dropdown via getChatGPTOAuthPoolOwnership.ownerByMember. - Inline "Pool" chip on owner options, reusing the existing providers:list.poolBadge i18n key. - Drop the separate card-level "Pool · <strategy>" badge — the dropdown chip alone conveys the information without duplication. - Remove the now-unused isPoolProvider / poolStrategyOf helpers and the builtin.mediaChain.poolBadge* i18n keys we briefly introduced. #1008 * docs(pr-1006): add pool-filtered dropdown screenshot * fix(ui/builtin-tools): migrate stale pool-member chain entries on load Red-team blocker: a chain saved before the pool-aware UI landed may reference a pool member by name (e.g. openai-codex-2). After the dropdown started hiding members, such an entry produced: - empty Select trigger (no SelectItem matches the stored value) - conflicting Row 1 label still showing the member name - silent runtime misroute (bare solo call, no pool wrap) parseInitialEntries now consults getChatGPTOAuthPoolOwnership and rewrites any chain entry whose provider is a pool member to the owner's name + id. The next save persists the migrated value. Safe no-op for non-pool entries and for entries already pointing at an owner. Also memoize enabledProviders in the parent form so the card's useMemo boundaries actually hold (minor perf ding flagged by same review). * docs(pr-1006): refresh HTML evidence to match filter-based UX * fix(ui/agent-codex-pool): traffic policy reflects effective strategy under inherit On the agent's OpenAI Account Pool page, when Agent routing mode is Use Provider Defaults, the Traffic Policy buttons painted the draft's placeholder strategy ("priority_order") as selected — contradicting the top-of-page chip which already correctly shows the provider's effective strategy. The removal of primary_first in this PR unmasked the latent bug: the placeholder used to be a deprecated value that didn't match any live button, so nothing appeared selected. Derive selectedStrategy from defaultRouting when mode === "inherit": the button highlight now mirrors what actually runs. Buttons remain disabled in inherit mode (unchanged), but the displayed selection no longer misleads the user. * docs(pr-1006): add screenshot of inherit-mode Traffic Policy fix * fix(permissions): remove duplicate MethodSessionsCompact entry The writeExact slice listed MethodSessionsCompact twice. slices.Contains still returned correct results so runtime behavior is unchanged, but the duplicate entry was dead code. Spotted during review of PR #1006. --------- Co-authored-by: viettranx <edu@200lab.io> |
||
|
viettranx
|
eb6723d674 |
fix(pipeline): include tool-schema tokens in overhead + dynamic compact max_tokens
- Add TokenCounter.CountToolSchemas() to measure JSON schema size for all tools - Include tool schemas in OverheadTokens calculation for accurate context usage - Implement dynamic max_tokens: in/25 clamp [1024, 8192] for compaction - Add characterization tests: count_tool_schemas_test.go - Add overhead verification tests: context_stage_overhead_test.go, context_stage_tool_overhead_test.go - Add integration tests: context_stage_integration_test.go - Add compact tests: loop_compact_dynamic_max_test.go, loop_compact_max_tokens_test.go - Add sanitize tests: loop_history_sanitize_max_tokens_test.go - Add integration test: loop_compact_integration_test.go |
||
|
Kai (Tam Nhu) Tranandviettranx
|
4b02c27307 |
feat: native image_generation for Codex + OpenAI-compat providers (#1002)
* refactor(providers): migrate ToolDefinition.Function to pointer + add image response fields
ToolDefinition.Function becomes *ToolFunctionSchema with omitempty so native tool types (image_generation, web_search, etc.) can be declared without a function body. All 9 internal construction sites updated. CleanToolSchemas refactored — function-shape cleaning extracted into cleanFunctionSchema helper, outer pass-through handles native tools.
Added image response fields needed by the next commits: ChatResponse.Images, StreamChunk.Images, ImageContent.Partial (distinguishes partial frames from final images).
* feat(providers): native image_generation for Codex + OpenAI-compat tracks
Codex native (POST /codex/responses): emit image_generation tool object in request tools[] (type, action, model, output_format, partial_images). Handle SSE events response.image_generation_call.partial_image + response.output_item.done (type image_generation_call) + response.completed output[] walk for non-stream. Dedup per item_id. Extend codexSSEEvent/codexItem with output_format, result, partial_image_b64, partial_image_index.
OpenAI-compat (/v1/chat/completions): serialize ToolDefinition{Type:'image_generation'} as {type:'image_generation'} pass-through. Parse choices[0].message.images[] + delta.images[] (data URLs) via new parseDataURL helper; append to ChatResponse.Images.
ProviderCapabilities.ImageGeneration flag; Codex provider + adapter set true. Other providers default false.
* feat(agent,http,store): persist assistant images + tri-level image_generation gate
Agent loop tri-level gate: (provider capability) AND (AgentConfig.AllowImageGeneration, default true, stored in other_config.allow_image_generation) AND (request lacks x-goclaw-no-image-gen header). Gate in loop_tool_filter.go appends ToolDefinition{Type:'image_generation'} only when all three pass. Per-request opt-out parsed in chat_completions.go and propagated via RunRequest.NoImageGen.
Media persistence: persistAssistantImages writes final images (Partial:false) to {workspace}/media/{sha256}.{ext}, returns MediaRef entries, clears inline Images[] from the message. Idempotent on hash, traversal-safe, symlink-guarded. Invoked from pipeline.FinalizeStage via new Deps.PersistAssistantImages callback — covers both stream-final and non-stream paths.
Agent store reads AllowImageGeneration from other_config JSONB with absent/nil = true default (matches V3Flags pattern). No DB migration — code-only default.
* feat(ui/web): image_generation toggle + streaming placeholder + download filename
Composer chip 'Images' visible only when active agent's provider has ImageGeneration capability. Per-agent localStorage persistence via useImageGenToggle hook. When off, sends noImageGen:true to WS chat method (maps to x-goclaw-no-image-gen on upstream HTTP call path).
ActiveRunZone renders a skeleton placeholder while streaming partial_image frames arrive. MediaGallery assigns generated-YYYYMMDD-HHmmss.png as the download filename for hex/UUID PNGs.
i18n keys added to en/vi/zh chat.json: imageGenToggle, imageGenGenerating, imageGenDownloadName. 8 vitest tests for the toggle hook.
* docs: add Image Generation section to codebase-summary + changelog entry
Documents the new native image_generation pipeline across providers layer (Codex + OpenAI-compat), agent gate, media persistence, and web UI surface.
* fix(ui/web): match Codex-routed providers for image_generation toggle
Image-gen toggle visibility was hard-coded to provider id 'chatgpt_oauth' but real Codex-routed agents in production use provider ids like 'cliproxy-codex'. The toggle never rendered.
Replace the Set-has check with a small helper that accepts the literal ids plus any provider string containing 'codex' (case-insensitive). Same logic applied in both chat-input.tsx (composer chip) and chat-page.tsx (streaming placeholder gate).
Verified against a live Codex-routed agent: toggle now renders, noImageGen:true propagates on toggle-off.
* docs(pr-1002): targeted-mode UX evidence report
Captures the UI integration surface for native image_generation against a live Codex-routed agent on the remote dev backend.
Includes: composer toggle chip (rendered), streaming skeleton placeholder, and honest failure-path capture showing the legacy create_image builtin fallback. Self-contained HTML report in .github/pr-assets/1002/index.html.
* fix(permissions): classify sessions.compact as write method
CI RBAC-drift test (TestMethodRole_DriftCoverage_AllProtocolMethodsClassified) was failing because the new sessions.compact method added upstream was not classified in any of isPublicMethod / isAdminMethod / isWriteMethod / isReadMethod.
Sessions compaction mutates session history (compacts messages into summaries), so it belongs with the other sessions.* write methods.
* fix(tests): remove duplicate contains() in integration package
Both tts_gemini_live_test.go and mcp_grant_revoke_test.go declared a file-local func contains(s, substr string) bool with identical bodies, causing 'contains redeclared in this block' at compile time in the integration job.
Replace all call sites with strings.Contains (same semantics, stdlib) and drop the duplicates. No behavior change.
* feat(providers): NativeImageProvider interface + Codex implementation
Defines a provider-level contract (NativeImageProvider.GenerateImage) that OAuth-backed providers can implement to serve image generation without exposing static API credentials. Re-uses the PR's Track A native wire format (POST /codex/responses with image_generation tool, item.result decoding, SSE fallback).
CodexProvider + CodexAdapter implement it. Also adds MediaRef.Prompt field so downstream layers can propagate the generating prompt alongside the asset.
* feat(tools): route create_image through NativeImageProvider for OAuth providers
create_image.callProvider now checks for a NativeImageProvider implementation before the credentialProvider interface. When the provider chain points at a Codex-family provider (no static API key), the tool delegates to the provider's GenerateImage which executes the native ChatGPT Responses API image_generation flow.
Resolves 'provider X does not expose API credentials required for image generation' errors for openai-codex / cliproxy-codex chains. On success the tool embeds the user's prompt as a PNG tEXt 'Description' chunk (file-local pngEmbedPrompt helper to avoid tools→agent import cycle), writes the image to /tmp, and threads the prompt through result.MediaPrompts for downstream MediaRef propagation.
* feat(agent,pipeline): propagate image prompt through MediaRef + PNG tEXt embed helper
Adds EmbedPNGPrompt public helper in internal/agent/png_metadata.go that inserts a tEXt 'Description' chunk (plus 'Software: goclaw') into PNG byte streams before the IEND chunk. Non-PNG inputs are passed through unchanged — the helper never errors on unknown formats.
FinalizeStage wires MediaResult.Prompt (from create_image tool output) onto MediaRef.Prompt so the UI can render the generating prompt alongside the image. Per-image prompt list threaded via pipeline RunState.
* feat(ui/web): show generating prompt as caption under each image in MediaGallery
When a MediaRef carries a prompt, MediaGallery renders it as a muted italic caption (line-clamp-2) beneath the image with the full text in the title tooltip. Caption is hidden when the prompt is absent so non-assistant images (user uploads, legacy data) look unchanged.
MediaItem + session media_refs types extended with an optional prompt field; the chat-message adapter threads ref.prompt through when converting WS payloads to UI state.
* fix(providers/codex): stream:true + instructions for native image_generation
The ChatGPT Responses API on /codex/responses rejects two things hard:
- stream:false → HTTP 400 "Stream must be set to true"
- missing instructions → HTTP 400 "Instructions are required"
buildNativeImageRequestBody now sets stream:true and a purpose-specific instructions string ("Generate an image matching the user's description using the image_generation tool. Return only the image; do not describe it in text."). The existing parseNativeImageSSE path was already in place for stream parsing; routing changed from the non-stream branch to the SSE branch.
Regression assertions added to TestCodexGenerateImage_BuildsNativeRequest so these two fields can't silently regress.
* feat(providers,tools,ui): image_model whitelist (gpt-image-2 default, gpt-image-1.5 legacy)
Replaces the hardcoded "gpt-image-2" literal in buildNativeImageRequestBody with a user-configurable field threaded through NativeImageRequest.ImageModel. The whitelist is enforced by ValidateImageModel which rejects anything outside {gpt-image-2, gpt-image-1.5} with a clear error — prevents silent upstream 400s from model names the Responses API would reject.
create_image.callProvider reads params.image_model from the chain entry and threads it through. Empty / absent falls back to DefaultImageModel (gpt-image-2).
UI: added an 'Image model' select inside the existing openai-codex Settings panel on the Create Image Provider Chain dialog. Options: Default · gpt-image-2 (recommended) and Legacy · gpt-image-1.5. i18n keys in en/vi/zh tools.json under builtin.mediaChain.
Tests: TestCodexGenerateImage covers default/legacy/rejected model cases; TestCreateImageTool_ThreadsImageModel covers params→request threading with empty/legacy/explicit sub-cases.
* fix(tools): raise media chain default timeout to 600s/1 retry for image gen
gpt-image-2 on complex prompts (dense Vietnamese text, infographic layouts) legitimately takes 4–8 minutes to complete. The previous default of 120s × 2 retries routinely died mid-generation with 'context deadline exceeded' — the upstream run was still producing bytes when our ctx cancelled.
Default is now Timeout: 600 / MaxRetries: 1. Retries dropped to 1 because image generation is stateful per upstream run: a mid-flight timeout leaves orphan server work, and retrying a fresh generation doubles cost for no gain. Surface the failure fast so operators can widen the timeout instead.
Operators can still set a tighter value explicitly via the Chain dialog.
* refactor: remove user-facing Images toggle, keep admin-level AllowImageGeneration
The per-request opt-out toggle (composer chip + streaming placeholder + noImageGen header plumbing) was a support footgun — users toggle OFF, forget, then can't generate images and think it's broken. Removed in full.
Kept: AgentConfig.AllowImageGeneration (admin kill-switch, stored in other_config.allow_image_generation, default true). Tri-level gate simplifies to two tiers: provider capability AND agent config allows.
Removed: useImageGenToggle hook, IMAGE_GEN_PROVIDER_IDS set in chat-input, supportsImageGenProvider helper, agentProvider/agentKey props on ChatInput, showImageGenPlaceholder prop on MessageBubble/ActiveRunZone/ChatThread, noImageGen param on use-chat-send, parseNoImageGen in chat_completions.go, NoImageGen on RunRequest, no_image_gen_header_test.go, imageGenToggle/imageGenGenerating i18n keys. Kept imageGenDownloadName — used by MediaGallery for generated-\*.png filename resolution.
* docs(pr-1002): refreshed UX trace + updated codebase notes
Replaces the earlier stealth-state evidence with a clean three-capture trace from a real successful run: inline image + prompt caption, MediaGallery lightbox expansion, Chain dialog with the new Image model dropdown open. Skill-routing rows scrubbed from the capture — they reflect per-agent skill setup, not anything this PR introduces.
codebase-summary + changelog: reflect final state (toggle removed, image_model selector, 600s default chain timeout, gpt-image-2 as quality baseline).
* fix(pipeline): preserve mid-loop image_generation output across iterations
FinalizeStage previously read state.Think.LastResponse.Images, which holds
only the last iteration's response. If the LLM emitted image_generation_call
in iteration N alongside a function_call, then responded text-only in N+1,
the image from N was silently dropped on finalize.
Accumulate final (non-partial) images into state.Observe.AssistantImages
across every iteration via ObserveStage, and source FinalizeStage from the
accumulator instead of LastResponse. Partial streaming frames are filtered
defensively; response.Images is cleared on drain to prevent double-counting
on re-exec.
* test(pipeline): regression coverage for mid-loop image accumulation
Six ObserveStage cases covering image accumulation semantics:
- single-iter image-only, image+tool_call same iter, mid-loop image
surviving text-only final iter, multiple images across iters, partial
frame filtering, nil-response safety.
Two FinalizeStage cases verifying accumulator is the source of truth:
- PersistsFromObserveAccumulator: image in Observe + empty LastResponse
must still be persisted via PersistAssistantImages.
- NoPersistWhenAccumulatorEmpty: no call when no images were emitted.
---------
Co-authored-by: viettranx <viettranx@gmail.com>
|
||
|
viettranx
|
a7962d182a |
feat(pipeline): session compaction overflow recovery (#958)
- Add ZAI/GLM context overflow patterns to error_classify.go - ThinkStage detects overflow, triggers emergency compaction + 1 retry - Wire ReserveTokensFloor config to pipeline budget calculation - Send user-friendly error on channel RunFailed events - Add sessions.compact WebSocket method for manual truncation |
||
|
viettranx
|
b7592bcacf |
feat(hooks/script): integrate script handler — FireResult + source-tier gate + migration
Wave 1 Phase 03. Wires the Phase 02 Goja handler into the dispatcher, widens
the DB schema for script + builtin sources, and refactors Dispatcher.Fire to
return a FireResult so builtin-source hooks can mutate event input.
Dispatcher:
- FireResult{Decision, UpdatedToolInput, UpdatedRawInput} replaces the plain
Decision return. stdDispatcher.runSync keeps a local evMut copy so a
builtin-source hook's updatedInput flows to downstream hooks in the chain
and to the caller; non-builtin (source=ui) script mutations are dropped +
WARN-logged (defense-in-depth; source tier enforced at the dispatcher, not
only at the handler).
- applyBuiltinMutation + placeholder builtinAllowlistFor (rawInput, toolInput)
— Phase 04 overrides the allowlist from builtins.yaml.
- noopDispatcher returns FireResult{DecisionAllow}.
- Script mutation tests: TestDispatcher_ScriptMutation_BuiltinSourceApplies,
TestDispatcher_ScriptMutation_UISourceDenied.
Pipeline + callers (14 Fire sites refactored):
- FireHook wrapper returns FireResult; context_stage.go:52 applies
UpdatedRawInput → state.Input.Message; tool_stage.go:52 applies
UpdatedToolInput → tc.Arguments before ExecuteToolCall.
- delegate_bridge + delegate_tool keep the Decision branch; Updated* ignored.
- dispatcher_test / delegate_bridge_test / delegate_tool_hooks_test /
integration test helpers updated for the new shape.
Validation:
- edition_gate allows HandlerScript on every edition (sandboxed, no shell
escape surface).
- config.validateHandler rejects empty source, source > 32 KiB, goja compile
error; validateTimeout rejects on_timeout=ask|defer (reserved).
- Six new config tests cover the script path.
Migrations:
- PG migration 000053 relaxes handler_type + source CHECKs and drops
uq_hooks_{global,tenant,agent} — scripts routinely want many small hooks
per event. RequiredSchemaVersion → 53.
- SQLite SchemaVersion → 21; patch 20 is a SELECT 1 placeholder because
SQLite can't ALTER a CHECK — rebuildAgentHooksV21 runs outside the
migration tx (parallel to backfillV16) to rename/recreate the table with
widened CHECKs. schema.sql fresh-DB path matches.
- H9 fix: PGHookStore.Create + SqliteHookStore.Create honor caller-provided
cfg.ID (uuid.Nil falls back to UUIDv7), unblocking Phase 04 idempotent
UUIDv5 seed. TestCreateHonorsFixedID on both stores.
Config:
- config.HooksConfig{ScriptConcurrency, ScriptPerTenantConcurrency,
ScriptCacheSize, BuiltinDisable}; buildHookHandlers wires the script
handler with the caps from appCfg.Hooks.
Gates green: go build ./... + sqliteonly, go test -race -count=1 across
hooks/pipeline/sqlitestore.
|
||
|
viettranx
|
553340b3a9 |
feat(hooks/prompt): LLM-driven prompt handler with per-turn cap
Adds PromptHandler that runs an LLM over matched hook events (user prompts, tool calls) to produce decisions (allow/block/modify). Features: - Budget integration: per-tenant token spend enforced via budget.Store - Per-turn counter seeded in ContextStage and propagated through state.Ctx so PreToolUse fires under the same cap as UserPromptSubmit - prompt_template required (non-empty) at validate time to prevent misconfigured hooks from silently no-oping - Injection-hardened: tool_input/raw_input are quoted into system prompt rather than concatenated - Resolver abstraction (RegistryResolver) picks model from provider registry with SystemConfigs fallback |
||
|
viettranx
|
97f9784443 |
feat(hooks): phase 2 — handlers, pipeline wiring, ssrf-safe http client
- add command + http handlers (internal/hooks/handlers) with edition gating and Authorization-header decryption via crypto.Decrypt - add SSRF-safe dialer (internal/security/ssrf.go): DNS-once + pinned IP, blocks loopback / link-local / private ranges - wire dispatcher into pipeline stages: ContextStage fires SessionStart (async) + UserPromptSubmit (sync); ToolStage fires PreToolUse (sync, COW staging for updatedInput) + PostToolUse (async); FinalizeStage fires Stop (async) - bridge delegate events to dispatcher (SubagentStart / SubagentStop) - construct dispatcher in cmd/gateway_managed.go with both handlers and thread it through agent.ResolverDeps + delegateTool.SetHookDispatcher - unit tests for both handlers + integration tests covering HTTP allow with audit row, HTTP block, command Lite-only gate, delegate bridge subscribe - defer worker pool optimization (Phase 2 Step 3) to Phase 4 Refs: GitHub Issue #875 |
||
|
viettranx
|
6544999779 |
fix(pipeline): treat gemini tool_calls+empty-args as truncation retry
Gemini returns finish_reason="tool_calls" (not "length") even when the thinking budget exhausted max_tokens before tool arguments could be emitted. Extend the existing truncation retry condition to detect this case via an allowlist of mutating tools (write_file, edit, exec, create_image, read_file) that virtually never legitimately call with empty args. Nullary tools (datetime, heartbeat) pass through unchanged. Trace: 019d8f33-2de1-7ab2-9a32-9df92cd610dd |
||
|
viettranx
|
5c229bed59 | test: failing tests for scoped truncation retry on mutating tools | ||
|
viettranx
|
48335d7797 |
feat(pruning)!: faithful port of TS context pruning + backfill migration
Port goclaw context pruning to match upstream TS design in
openclaw/src/agents/pi-hooks/context-pruning/:
- Opt-in default: prune only when mode="cache-ttl" (was opt-out)
- Remove Pass 0 per-result 30% guard (duplicated Pass 1 with different
suffix, caused wobble)
- Dedupe double prune call per iteration: PruneStage owns the single
entry point; loop_history only runs limitHistoryTurns + sanitizeHistory
- Add cache-TTL gate for Anthropic prompt cache: skip prune while cache
is live, scoped per-session via sync.Map
- Add context.pruned event emission for observability
- Configurable TTL as Go duration string ("5m", "30s")
BREAKING CHANGE: context pruning now opt-in. Add
contextPruning.mode: "cache-ttl" to config.agents.defaults to restore.
Migration 51 / SQLite v19 backfills mode="cache-ttl" for agents with
existing custom context_pruning config missing the mode field, so
previously-configured agents keep pruning after the opt-in flip.
NULL configs stay NULL (new opt-in default applies).
Web UI adds Cache TTL input + toggle wiring mode to cache-ttl/off.
|
||
|
viettranx
|
221cd78fcf |
fix(agent): emit tool.call event in parallel tool execution path
v3 pipeline's parallel path (makeExecuteToolRaw) skipped the tool.call WebSocket event, so web UI and desktop UI silently dropped tool cards during real-time streaming. Only page refresh (which reloads history) revealed the tool calls. Both UIs rely on tool.call to create entries that later tool.result events can update. Fix: mirror the sequential path's emission in makeExecuteToolRaw. Bus.Broadcast is RWMutex-guarded, safe to call from parallel goroutines. Add tests at two layers to prevent regression: - Pipeline layer (stages_test.go): guards the dispatch contract — multiple tool calls route through ExecuteToolRaw + ProcessToolResult rather than ExecuteToolCall. Previously the parallel path had zero test coverage, which is why this bug escaped. - Agent layer (loop_pipeline_tool_callbacks_test.go): guards the emission contract — both sequential and parallel wrappers emit tool.call with correct payload and routing context. Mutation-verified. |
||
|
viettranx
|
034187ac9c |
fix(pipeline): block_reply dedup falsely suppresses final message (#838)
Only count BlockReplies when tool calls are present (matching when EmitBlockReply actually fires in think_stage). Final answers without tool calls must not increment the counter, otherwise gateway dedup falsely suppresses delivery on non-streaming channels. - Update ObserveStage to check len(resp.ToolCalls) > 0 - Rename test to reflect new behavior (tool calls required) - Add regression test for #838 scenario Closes #838 |
||
|
viettranx
|
8d37dc45ea |
feat(pipeline): per-provider context window, cache/tenant/pipeline hardening
Ship Group A (bug fixes) + Group B (pipeline enhancements) from plans/260410-1009-openclaw-ts-feature-port — five changes that share cmd/ wiring and pipeline plumbing so they commit as a unit. cache: InMemoryCache now supports periodic sweep + max-size cap via variadic options. PermissionCache wires 60s sweep + 10k entry cap + Close() hook in gateway shutdown so long-running gateways don't leak per-user permission entries. Backward compatible for zero-arg callers. store: ContactCollector.seen cache key now includes tenant_id + channel_ instance so the same sender in different tenants (or different bot instances in the same tenant) no longer silently skip upserts against each other. Zero-tenant (Desktop) behaviour preserved. pipeline: EffectiveContextWindow is resolved once per run in ContextStage via a ResolveContextWindow callback (backed by providers.ModelRegistry) so PruneStage bills history against the actual model window instead of a stale static config. Nil resolver / unknown model fall back to Config.ContextWindow for backward compatibility. Locked to the model observed at context build time to prevent mid-run budget drift. pipeline: PipelineConfig.ReserveTokens carves out an optional safety buffer subtracted from the history budget so compaction fires slightly before the hard limit — protects against provider over-delivery and token counter drift on streaming responses. Zero (default) preserves legacy budget math. agent: ModelRegistry flows gateway → ResolverDeps → LoopConfig → Loop → pipeline adapter so resolver can look up per-model capabilities at run time without re-touching gateway internals. 20 regression tests across cache, contact collector, and pipeline cover the critical paths: cross-tenant isolation, cache sweep + eviction + Close idempotency, per-model window override + fallback, and reserve token buffer behaviour. All passing with -race on both go build ./... and go build -tags sqliteonly ./.... |
||
|
|
8f56ddaa64 |
feat(v3): core architecture redesign — pipeline, memory, vault, evolution, providers, orchestration (#790)
* feat(v3): add core interface contracts and migration for v3 redesign
Foundation interfaces: TokenCounter, WorkspaceContext, DomainEventBus,
ProviderAdapter/Capabilities. Pipeline: Stage, RunState, MessageBuffer,
substates, Pipeline orchestrator. Memory: EpisodicStore, AutoInjector,
KG temporal extensions, consolidation workers. System integration:
PromptConfig, ToolCapability, Retriever. Orchestration: OrchestrationMode,
EvolutionMetrics/SuggestionStore. Migration 000037: episodic_summaries,
evolution tables, KG temporal columns. Schema version 36→37.
* refactor(plans): mark all v3 design phases complete with file references
* fix(v3): address code review findings on design contracts
- C1: add missing l0_abstract column to episodic_summaries migration
- C2: align EpisodicSummary ID/TenantID/AgentID to uuid.UUID
- H1: document tenant_id scoping requirement on EpisodicStore
- H2: add UNIQUE constraint on (agent_id, user_id, source_id) for dedup
- H4: clarify ProviderAdapter vs Provider relationship in doc
- M3: set state.ExitCode on BreakLoop/AbortRun in pipeline
- M6: store full PipelineConfig in Pipeline struct
- Edge: add WHERE embedding IS NOT NULL on HNSW index
* fix(v3): second-pass review fixes
- H1: use context.WithoutCancel for finalize + set ExitCode on ctx cancel
- H2: use utf8.RuneCountInString consistently in FallbackCounter
- H3: longest-prefix-match in ModelContextWindow (prevents wrong tokenizer)
- H4: return unsubscribe cleanup func from consolidation.Register
* feat(v3): implement DomainEventBus with worker pool, dedup, and retry
Worker pool processes events from buffered channel. SourceID-based dedup
prevents duplicate processing. Exponential backoff retry on handler error.
Panic recovery per handler. Graceful shutdown via Drain(). 8/8 tests pass
with race detector.
* feat(v3): implement ProviderAdapter for Anthropic, OpenAI, DashScope, Codex
Add CapabilitiesAware to all 6 providers. Create ProviderAdapter
implementations that delegate to existing buildRequestBody/parseResponse
for DRY. ClaudeCLI and ACP get capabilities only (subprocess transport).
DashScope wraps OpenAI adapter with StreamWithTools=false override.
* feat(v3): implement WorkspaceContext Resolver for 6 scenarios
Stateless resolver produces immutable WorkspaceContext at run start.
Handles personal/group/predefined/team-shared/team-isolated/delegation.
Wired into loop_context.go behind v3PipelineEnabled flag (additive,
v2 path unchanged). Includes delegation path boundary check,
master tenant bypass, and tenant slug path composition.
* feat(v3): implement tiktoken TokenCounter with BPE encoding + cache
Adds tiktoken-go for accurate cl100k_base/o200k_base token counting.
Per-message FNV-1a hash cache avoids re-encoding unchanged history.
Falls back to rune/3 heuristic for unknown models. NewTokenCounter
factory selects implementation at build time.
* feat(v3): promote 12 other_config JSONB fields to dedicated agent columns
Extract emoji, agent_description, thinking_level, max_tokens,
self_evolve, skill_evolve, skill_nudge_interval, reasoning_config,
workspace_sharing, chatgpt_oauth_routing, shell_deny_groups, and
kg_dedup_config from the catch-all other_config JSONB into proper
columns with DB-level types and defaults.
- Migration: PG (000037) + SQLite (schema v6→7) with backfill
- Go: AgentData struct + simplified Parse* methods
- Store: SELECT/INSERT/scan updated for both PG and SQLite
- Gateway: create/update handlers accept promoted fields
- HTTP: export/import with legacy backward compat
- Web UI: all 15 frontend files read/write from top level
* feat(v3): implement Knowledge Vault with unified search, wikilinks, and FS sync
Migration 000038 adds vault_documents (FTS+pgvector), vault_links, vault_versions
tables. VaultStore interface with PG implementation for document CRUD, hybrid
FTS+vector search, and bidirectional link management. All queries enforce
tenant_id isolation including JOIN-based scoping on link operations.
FS sync layer: SHA-256 content hashing, VaultInterceptor hooks into write_file/
read_file for auto-registration and lazy sync, fsnotify watcher with 500ms
debounce. Wikilink engine parses [[target]] syntax, resolves targets via
3-step strategy, and maintains vault_links on write.
VaultSearchService fans out queries across vault, episodic, and KG stores in
parallel with per-source score normalization and weighted merge. AutoInjector
and Retriever implementations for pipeline integration.
Three agent tools: vault_search (unified discovery), vault_link (explicit
linking), vault_backlinks (dependency tracing). Feature-flagged via
v3_vault_enabled agent setting.
* feat(v3): wire vault into gateway startup + add unit tests
Wire VaultStore embedding provider, VaultSearchService, VaultInterceptor
on read/write tools, and register vault_search/vault_link/vault_backlinks
tools in gateway_vault_wiring.go. All wiring gated by stores.Vault != nil.
Add 28 unit tests for ContentHash, ContentHashFile, and ExtractWikilinks
covering edge cases, unicode, display text, context windows, and offsets.
* feat(v3): implement stage-based pipeline loop with 8 pluggable stages
Decompose monolithic agent loop into internal/pipeline/ package:
- 6 stages: Context, Think, Prune+MemoryFlush, Tool, Observe+Checkpoint, Finalize
- Foundation types: Stage interface, RunState with 7 typed substates, MessageBuffer
- Pipeline orchestrator with setup/iteration/finalize 3-phase execution
- Callback-based PipelineDeps avoids circular import with agent package
- Feature-flagged via v3PipelineEnabled in Loop.Run()
- All 7 exit conditions preserved (no tools, max iter, truncation, loop kill,
read-only streak, tool budget, ctx cancel)
* feat(v3): wire pipeline callbacks to Loop methods + add 71 unit tests
Wire 15 of 17 PipelineDeps callbacks from Loop methods via closures:
- Context: LoadContextFiles, BuildMessages, EnrichMedia, InjectReminders
- Think: BuildFilteredTools, CallLLM (stream/sync)
- Prune: PruneMessages, CompactMessages
- Memory: RunMemoryFlush
- Finalize: SanitizeContent, FlushMessages, UpdateMetadata, BootstrapCleanup, MaybeSummarize
- Remaining: ExecuteToolCall, CheckReadOnly (deep loop.go integration)
Add comprehensive test suite (71 tests, all passing with -race):
- MessageBuffer: 10 tests (append, flush, replace, counts)
- Pipeline.Run: 14 tests (3-phase flow, exit conditions, ctx cancel)
- Stage tests: 47 tests (ThinkStage nudges/truncation, PruneStage budget,
ToolStage parallel/exit, ObserveStage content, CheckpointStage interval,
FinalizeStage cleanup)
* feat(v3): wire remaining 2 callbacks (ExecuteToolCall, CheckReadOnly)
Complete callback wiring — 17/17 PipelineDeps callbacks now active:
- ExecuteToolCall: resolves tool name, executes via registry, processes
result via existing processToolResult with loop detection bridge
- CheckReadOnly: delegates to checkReadOnlyStreak via bridge runState
- Bridge runState shares loop detection state between pipeline and agent
* fix(v3): eliminate data race in tool execution + capture injected messages
- Remove parallel tool execution path — serialize all tool calls to avoid
data races on shared bridgeRS (loop detector, media results, deliverables)
- Loop kill checked after each tool (mid-batch early exit)
- BuildFilteredTools: capture and append injected tool-awareness messages
- Rename test to reflect sequential execution
* feat(v3): wire ResolveWorkspace, safe parallel tools, ContextStage tests
- Wire ResolveWorkspace callback via workspace.NewResolver() with
ResolveParams from Loop fields (no longer a nil stub)
- Re-add safe parallel tool execution: split into ExecuteToolRaw
(parallel I/O) + ProcessToolResult (sequential state mutation)
with opaque rawData pass-through (no double execution)
- Add 12 unit tests for ContextStage (8) + MemoryFlushStage (3)
- Split tool callbacks to loop_pipeline_tool_callbacks.go (under 200 lines)
- Capture buildFilteredTools injected messages
* feat(v3): add episodic memory store + temporal KG columns
Phase 1 — Episodic Store:
- Migration 000039: episodic_summaries table with pgvector, FTS, L0 abstracts
- EpisodicStore PG impl: CRUD, hybrid FTS+vector search, ExistsBySourceID,
PruneExpired. Idempotent via source_id UNIQUE constraint.
Phase 2 — Temporal KG:
- Migration 000040: valid_from/valid_until on kg_entities + kg_relations,
partial indexes for current-facts queries, epoch→timestamptz backfill
- ListEntitiesTemporal: current-only, point-in-time, or include-expired modes
- SupersedeEntity: atomic expire-old + insert-new in single transaction
Schema version bumped to 40.
* fix(v3): review fixes for episodic store + temporal KG
- C1: Fix column name mismatch turn_count vs message_count in Go SQL
- C2: Remove redundant migration 000040 (000037 already adds temporal KG columns)
- H1: Use time.Time not int64 for TIMESTAMPTZ columns in SupersedeEntity
- H2: Add tenant_id scoping to Get/Delete for tenant isolation
- M2: Fix scanEntityTemporal to convert TIMESTAMPTZ→UnixMilli correctly
- L1: Remove unused uuid import from episodic_search.go
- Schema version corrected to 39 (only 000039 is new)
* feat(v3): implement consolidation pipeline with 3 event-driven workers
Event chain: session.completed → EpisodicWorker → episodic.created →
SemanticWorker → entity.upserted → DedupWorker
- EpisodicWorker: reuses compaction summary or calls LLM, generates L0
abstract (extractive), idempotent via source_id check
- SemanticWorker: extracts KG facts from episodic summary via existing
Extractor, sets temporal valid_from, publishes entity.upserted
- DedupWorker: runs DedupAfterExtraction on new entity IDs (terminal)
- L0 abstract: sentence-based extraction (~50 tokens), no LLM needed
- All workers registered via DomainEventBus.Subscribe()
* feat(v3): implement progressive loading with L0 auto-inject + unified search
- AutoInjector: searches episodic store, builds L0 prompt section (~200 tokens),
skips trivial messages via stopword filter
- L1Cache: in-memory LRU (500 entries, 1h TTL) for structured overviews
- UnifiedSearch: cross-tier search merging episodic + document results by score
- ContextStage integration: AutoInject callback appends memory section to system prompt
- MemorySection field added to ContextState for observability
* feat(v3): add memory_expand tool for L2 episodic retrieval
New tool: memory_expand(id) returns full episodic summary with metadata.
Complements memory_search L0/L1 results with deep L2 access.
Nil-safe: returns error message when episodic store not available.
Gateway wiring + memory_search depth param + kg_search temporal param
deferred to runtime integration phase.
* feat(v3): complete Phase 5 — tool extensions + gateway wiring
- memory_search: add depth param + episodic tier search merged with docs
- kg_search: add as_of temporal param, use ListEntitiesTemporal
- memory_expand: registered in gateway startup
- Gateway: Episodic field in Stores, PGEpisodicStore in factory,
embedding provider wired, tools connected to episodic store
* fix(v3): Phase 3 review fixes — tenant isolation + AutoInject args
- C1: Add tenant_id filter to ftsSearch, vectorSearch, List queries
(prevents cross-tenant episodic memory leaks)
- C2: Fix AutoInject callback signature — agent/tenant captured by
closure, only userMessage + userID passed explicitly
- H1: Add tenant_id to List query
* feat(v3): wire per-agent v3 flags from DB into dual-mode gate
Parse v3_pipeline_enabled, v3_memory_enabled, v3_retrieval_enabled from
agent other_config JSONB via ParseV3Flags(). Resolver now sets all flags
on LoopConfig so the existing gate in loop_run.go reads from DB.
- V3Flags struct + ParseV3Flags() + ValidateV3Flags() in store layer
- v3MemoryEnabled/v3RetrievalEnabled added to Loop, LoopConfig, PipelineConfig
- Auto-inject gated on V3RetrievalEnabled (was unconditional)
- Structured perf logging for v3 pipeline runs
- v3 flag validation on both WS agent.update and HTTP PUT endpoints
* feat(v3): wire AutoInjector into pipeline for L0 memory auto-inject
Create AutoInjector at gateway startup from episodic store, pass through
ResolverDeps → LoopConfig → Loop. Pipeline adapter builds AutoInject
callback capturing agent/tenant context via closure.
ContextStage already gates on V3RetrievalEnabled + AutoInject != nil.
* feat(v3): add tool metadata map + capability-based deny rules
Registry gains per-tool ToolMetadata map with RegisterWithMetadata()
and GetMetadata() (infers defaults from tool name when not explicit).
PolicyEngine gains DenyCapability() for RBAC integration — tools with
denied capabilities filtered at step 8 after existing 7-step pipeline.
* fix(v3): add RWMutex to PolicyEngine capability deny fields
DenyCapability() and SetRegistry() now guarded by sync.RWMutex.
FilterTools reads snapshot under RLock. Prevents data race when
capability rules are modified concurrently with tool filtering.
* feat(v3): implement delegate tool for inter-agent task delegation
New `delegate` tool wraps existing agent_links infrastructure
(CanDelegate, DelegateTargets). Supports async (fire-and-forget)
and sync (block with timeout) modes. Permission checked via
AgentLinkStore. Events emitted: delegate.sent/completed/failed.
DelegateRunFunc injected by gateway to avoid circular dependency.
* feat(v3): complete 3 deferred implementations
1. OrchestrationMode resolution: ResolveOrchestrationMode() checks
team membership → delegate links → spawn (priority order).
2. PG EvolutionMetricsStore: RecordMetric, QueryMetrics, aggregate
tool/retrieval metrics, TTL cleanup. All queries tenant-scoped.
3. BridgePromptBuilder: implements PromptBuilder interface by
delegating to existing BuildSystemPrompt(). Appends v3 memory
L0 section when enabled. Ready for template engine swap later.
* fix(v3): address code review findings on commits 5-6
- C1: CanDelegate now tenant-scoped (fail-closed on missing tenant)
- H1: Sync delegate timeout capped at 600s
- H2: Async goroutine gets 10min deadline (prevents leaks)
- H3: JSONB casts use COALESCE/NULLIF guards (handles missing fields)
- M1/M2: Remove dead code (formatVaultSection, memoryL0ToStrings)
* fix(teams): stop auto-creating agent_links for team members
Teams use agent_team_members table directly — agent_links caused
context confusion between team dispatch and delegation systems.
- Remove autoCreateTeamLinks() calls from team create + member add
- Remove link cleanup from member remove
- Remove dead autoCreateTeamLinks() function
- Append DELETE to migration 000039: clear team-created agent_links
* fix(v3): tenant isolation for all agent_links queries + PromptBuilder Instructions
- DelegateTargets, GetLinkBetween, SearchDelegateTargets,
SearchDelegateTargetsByEmbedding, DeleteTeamLinksForAgent all now
scoped by tenant_id (fail-closed on missing tenant)
- BridgePromptBuilder now maps Instructions/InstructionContent to
AGENTS.md context file (was silently dropped)
* feat(v3): wire orchestration mode + evolution metrics into agent loop
- Orchestration mode: resolver resolves mode from team/links, tool filter
hides delegate/team_tasks based on mode, prompt builder injects delegation
targets section
- Evolution metrics: non-blocking goroutine records tool execution metrics
(name, success, duration) via EvolutionMetricsStore in both v2 loop and
v3 pipeline paths (sequential + parallel)
- Fix review findings: tenant ID propagated via store.WithTenantID in
background goroutine, 5s timeout prevents goroutine leak
* feat(v3): implement suggestion engine with pluggable analysis rules
- PG EvolutionSuggestionStore: CRUD for agent_evolution_suggestions table
- SuggestionEngine: aggregates 7-day metrics, runs rules, deduplicates
pending suggestions per type before creating new ones
- 3 initial rules: LowRetrievalUsage (usage_rate<0.2), ToolFailure
(success_rate<0.1), RepeatedTool (>100 calls/week → suggest skill)
- EventSuggestionCreated event type added to eventbus
- Cron wiring deferred to gateway startup integration pass
* feat(v3): implement auto-adapt guardrails with apply/rollback
- AdaptationGuardrails: max delta per cycle, min data points, locked
params, rollback-on-drop percentage
- ApplySuggestion: applies threshold suggestions to agent other_config
JSONB, stores baseline for rollback
- RollbackSuggestion: restores baseline values from suggestion params
- EvaluateApplied: compares post-apply metrics to baseline, auto-rolls
back when quality drops beyond threshold
- Scope limited to retrieval params only (never security settings)
* feat(v3): wire evolution stores + daily/weekly cron for suggestions
- Add EvolutionMetrics + EvolutionSuggestions to Stores struct + PG factory
- Wire EvolutionMetricsStore into ResolverDeps (cmd/gateway_managed.go)
- Add gateway_evolution_cron.go: daily suggestion analysis + weekly
evaluation/rollback for applied suggestions
- Cron runs as background goroutine with 5-min timeout per cycle
* fix(v3): address code review findings on evolution engine
- C1: persist baseline parameters before marking suggestion as applied
(was building map but never saving — rollback would always fail)
- H1: add tenant_id isolation to UpdateSuggestionStatus, GetSuggestion,
and new UpdateSuggestionParameters method
* test(v3): add unit tests for orchestration, suggestions, guardrails, prompt
- orchestration_mode_test: orchModeDenyTools (4 modes) + ResolveOrchestrationMode
(4 scenarios with mock stores)
- suggestion_rules_test: LowRetrievalUsage, ToolFailure, RepeatedTool with
threshold boundary tests (at/below/above min data points)
- evolution_guardrails_test: DefaultGuardrails values + CheckGuardrails
(insufficient data, locked params, zero-min fallback)
- prompt_builder_orchestration_test: BridgePromptBuilder orchestration section
presence/absence across 4 scenarios + target content verification
* test(v3): add integration tests for evolution metrics + suggestions
- Test helper: shared PG connection with sync.Once migration, per-test
tenant+agent seed with cleanup
- Evolution metrics: RecordMetric, AggregateToolMetrics (success rate),
Cleanup (TTL deletion)
- Evolution suggestions: full CRUD, UpdateSuggestionParameters (baseline
persist), tenant isolation (cross-tenant read blocked)
- Pipeline E2E: seed 25 failed tools + 55 low-usage retrievals, verify
SuggestionEngine creates suggestions, verify dedup on second run
- Fix: migration 039 de-duped (episodic_summaries already in 037)
- Fix: NULL reviewed_by scan via sql.NullString
* feat(v3): add HTTP API handlers for evolution, vault, episodic, orchestration, v3-flags
5 new handler files exposing v3 backend stores as REST endpoints:
- evolution_handlers.go: metrics query/aggregate + suggestions CRUD
- vault_handlers.go: cross-agent document listing + search + links
- episodic_handlers.go: episodic summaries list + hybrid search
- orchestration_handlers.go: computed mode + delegate targets (read-only)
- v3_flags_handlers.go: per-agent v3 feature flag get/toggle
Store fixes from code review:
- episodic FTS: use inline to_tsvector (no stored tsv column)
- episodic: conditional user_id filter in List + Search (admin view)
- episodic: add tenant_id to ExistsBySourceID + PruneExpired
- evolution: require tenant_id in context (no struct fallback)
- evolution: check RowsAffected on suggestion updates
- vault: optional agent_id filter in ListDocuments (cross-agent)
* feat(v3): add web UI for evolution tab, v3 settings, vault page, episodic memory
Agent Detail enhancements:
- V3 Settings section: pipeline/memory/retrieval flag toggles
- Orchestration section: mode badge + delegate targets display
- Evolution section: added metrics + suggestions v3 flag toggles
- Evolution tab: Recharts metrics charts + suggestion review table
with approve/reject/rollback actions + guardrails card
New pages:
- /vault: Knowledge Vault document registry with cross-agent listing,
hybrid search dialog, document detail with wikilinks
- Memory page: added Episodic Memory tab with summary cards,
expandable details, key topic badges, and hybrid search
Infrastructure:
- HttpClient: added patch() method
- Query keys: v3Flags, orchestration, evolution namespaces
- 4 new hooks: use-v3-flags, use-orchestration, use-evolution-metrics,
use-evolution-suggestions, use-vault, use-episodic
- i18n: vault namespace (en/vi/zh), agents + memory keys updated
- Reused formatRelativeTime from lib/format.ts (eliminated 3 duplicates)
* refactor(http): add bindJSON helper and migrate all decode call sites
Replace 36 json.NewDecoder(r.Body).Decode + error blocks with bindJSON
across 20 HTTP handler files. Standardizes decode error responses to
structured writeError format. Fixes unchecked decode in handleIndexAll.
* refactor(store): adopt sqlx for PG scan operations (Phase 1+2)
Add jmoiron/sqlx v1.4.0 with camelToSnake json tag mapper.
Migrate scan-heavy PG store methods to sqlx Get/Select:
- tracing.go: GetTrace, ListTraces, ListChildTraces, GetTraceSpans, GetCostSummary
- heartbeat.go: Get, ListDue, ListLogs
- providers.go: GetProvider, GetProviderByName, ListProviders, ListAllProviders
- mcp_servers.go: GetServer, GetServerByName, ListServers
- pairing.go: ListPending, ListPaired
- agents_export_queries.go: 5 export functions
- agents_export_team_queries.go: exportTeamMembers, ExportAgentLinks
All writes (INSERT/UPDATE/DELETE), execMapUpdate, and dynamic WHERE
builders remain raw SQL. Zero behavior change.
* refactor(store): adopt sqlx for SQLite scan operations (Phase 3)
Migrate SQLite store scan methods to sqlx Get/Select:
- providers.go: GetProvider, GetProviderByName, ListProviders, ListAllProviders
- tenants.go: GetTenant, GetTenantBySlug, ListTenants, GetTenantUser, ListUsers, ListUserTenants
- mcp_servers.go: GetServer, GetServerByName, ListServers
Create sqlx_scan_structs.go with sqliteTime-aware scan structs
(providerRow, tenantRow, tenantUserRow, mcpServerRow) to handle
SQLite TEXT timestamp parsing via StructScan.
* refactor(store): migrate PG bulk scan operations to sqlx (Phase 4)
Migrate scan-heavy methods across 6 PG store files:
- tenant_store.go: GetTenant, GetTenantBySlug, ListTenants, GetTenantUser,
ListUsers, ListUserTenants — removed 3 scan helpers
- teams.go: ListTeams, GetTeam, ListMembers, ListMembersByTenant
- teams_tasks_activity.go: ListComments, ListEvents, ListFollowUps
- pending_message_store.go: ListPending, ListByHistoryKey
- skills_grants.go: ListAgentGrants
- config_permissions.go: CheckPermission
~20 scan ops converted. Files with encryption post-processing,
pq.Array, pgvector, or dynamic SQL kept raw.
* refactor(store): extract shared CamelToSnake mapper, add UUIDArray usage note
- Move camelToSnake to internal/store/column_mapper.go (DRY)
- Both pg and sqlitestore packages now import shared CamelToSnake
- Add planned-use comment on UUIDArray type
* refactor(cli): migrate commands from config.json to HTTP API, add providers/setup/TUI
- Add unified HTTP client (gateway_http_client.go) with auth, error parsing, typed generics
- Rewrite agent list/add/delete to use gateway HTTP API instead of config.json
- Rewrite channels list to HTTP API, add channels add/delete subcommands
- Replace models command with full providers CRUD (list/add/update/delete/verify)
- Add setup wizard command (provider → agent → channel post-onboard flow)
- Add Bubble Tea TUI behind build tag (tui/!tui with noop fallback)
- Update onboard next-steps to mention goclaw setup
- Add build-tui Makefile target
- Fix URL path injection (url.PathEscape on all user-supplied path segments)
- Fix UTF-8 truncation in skills description display
* refactor(store): add explicit db struct tags, fix sqlx mapper for heartbeat scan error
Switch sqlx mapper from NewMapperFunc (which only applies CamelToSnake to
field names, not tag values) to NewMapperFunc("db", CamelToSnake) with
explicit db:"column_name" tags on all store structs.
Root cause: NewMapperFunc("json", fn) sets mapFunc but not tagMapFunc,
so camelCase json tags like "agentId" were used as-is instead of being
converted to "agent_id", causing "missing destination name" scan errors.
Fix: use db struct tags as the source of truth for column mapping.
Every DB entity field gets db:"column_name", nested JSON configs and
runtime-only structs get db:"-".
* test(store): add integration tests for 13 store interfaces (70 tests)
Cover Tier 1 (critical) + Tier 2 (security) stores with integration tests
running against pgvector pg18. Coverage from 2.4% to ~54%.
Stores tested: Session, Agent, Team/Task, Memory, KnowledgeGraph, Vault,
MCP Server, API Key, ConfigPermission, Contact.
Infrastructure: fixture builders (seedTeam, seedMCPServer, etc.),
mock EmbeddingProvider, multi-tenant helpers, expanded cleanup.
* fix(store): resolve NULL scan bugs in MCP server and task metadata
- mcp_servers: COALESCE nullable TEXT columns (display_name, command,
url, api_key, tool_prefix) to prevent sqlx scan failures
- mcp_servers_access: COALESCE nullable JSONB columns in ListAgentGrants
(tool_allow, tool_deny, config_overrides) to prevent silent row drops
- teams_tasks: default task metadata to '{}' instead of nil to satisfy
NOT NULL constraint on CreateTask
- sqlx_helpers: export InitSqlx for integration test setup
* feat(pipeline): fix v3 pipeline context injection, tracing, KG temporal filters
- Pipeline context: add InjectContext + LoadSessionHistory callbacks to
ContextStage, propagate enriched ctx via state.Ctx for iteration stages
- Pipeline tracing: wrap makeCallLLM with emitLLMSpanStart/End, wrap
makeExecuteToolCall/Raw with emitToolSpanStart/End
- Token counter: switch pipeline from FallbackCounter to TiktokenCounter
- KG temporal: add valid_until IS NULL filter to all entity/relation
queries (list, search, vector, FTS, traversal CTE, stats)
- Skills: add SkillEmbedder interface for future hybrid BM25+vector search
- Cache: remove unused tenantResolve dead code from PermissionCache
- Store: fix NULL scan bugs in tracing metadata and agent skill_nudge
- Test: add TestStoreKG_TemporalFilter integration test
- UI: add v3 version badge, evolution section, memory/traces improvements
* refactor(store): migrate KG store from raw sql.Rows to sqlx StructScan
Migrate 6 knowledge graph store files from manual rows.Scan() to
pkgSqlxDB.GetContext/SelectContext with intermediate scan row structs.
- Add entityRow, relationRow, traversalRow, dedupCandidateRow structs
with json.RawMessage for jsonb and time.Time for timestamptz columns
- Add toEntity()/toRelation() converters (UnixMilli + json.Unmarshal)
- Add sqlxTx() helper for wrapping *sql.Tx with sqlx mapper
- Fix ScanDuplicates passing time.Now().Unix() to TIMESTAMPTZ column
- Fix ListEntitiesTemporal missing tenant scope (scopeClause)
- Fix SupersedeEntity missing tenant scope and tenant_id on INSERT
- Fix DedupCandidate.CreatedAt using Unix() instead of UnixMilli()
- Update agents_export_queries.go to reuse new scan row structs
- Net -160 lines of manual scan boilerplate removed
* refactor(store): migrate memory, skills, agents, sessions, mcp, cron, vault stores to sqlx
Batch migration of 19 store files from raw rows.Scan() to
pkgSqlxDB.GetContext/SelectContext with intermediate scan row structs.
Groups migrated:
- Memory: memory_docs, memory_admin, memory_search, memory_embedding_cache
- Episodic: episodic_search, episodic_summaries
- Skills: skills, skills_admin, skills_embedding, skills_export_queries
- Agents: agents (backfill+shares), agents_context, agents_export_team_standalone
- Sessions: sessions_list (List, ListPaged, ListPagedRich)
- MCP: mcp_servers_access, mcp_export_queries
- Cron: cron_exec (GetRunLog)
- Vault: vault_documents (ListDocuments, ftsSearch, vectorSearch)
- Tenant: tenant_configs (ListDisabled, ListAll)
7 new scan row files created. Net -510 lines of manual scan boilerplate.
INSERT/UPDATE/DELETE and scalar COUNT queries kept as raw SQL.
* fix(store): fix 3 sqlx scan struct db tag issues found by audit
- Fix vault FTS alias mismatch: `AS rank` → `AS score` (critical: runtime scan error)
- Fix episodic key_topics type: json.RawMessage → pq.StringArray (TEXT[] column)
- Fix agentShareRow.CreatedAt: string → time.Time, wire to output struct
* feat(providers): implement Wave 2 provider resilience and intelligence
9-phase implementation covering:
- Request middleware chain with composable body transformers
- OpenAI prompt caching, service tier, and fast mode middlewares
- Error classification (9 categories) with two-tier failover
- Model registry with forward-compat resolvers (Anthropic + OpenAI)
- Embedding providers (OpenAI + Voyage) with 1536-dim validation
- Cooldown/probe system with per-provider:model state tracking
- Markdown-aware chunking shared across 5 channels
- Session recall via FTS + pgvector on episodic summaries
- Dreaming/promotion pipeline for long-term memory consolidation
Migrations: 000040 (episodic search index), 000041 (promoted_at column)
Schema version: 39 → 41
* feat(providers): wire model registry into gateway provider construction
Create InMemoryRegistry with Anthropic + OpenAI forward-compat resolvers
at gateway startup. Pass to all Anthropic and OpenAI providers created
from both config and DB sources.
* feat(consolidation): wire DomainEventBus and consolidation pipeline
Create DomainEventBus at gateway startup, thread through resolver →
LoopConfig → Loop → PipelineDeps. Emit session.completed event after
each run finalization. Register consolidation pipeline (episodic →
semantic → KG dedup → dreaming) with event bus subscriptions.
* fix(store): fix episodic key_topics pq.Array, ON CONFLICT, and migration 040 immutability
- episodic_summaries.go Create: json.Marshal(KeyTopics) → pq.Array (text[] column)
- episodic_search.go scanEpisodic/scanEpisodicRow: json.RawMessage → pq.StringArray
- episodic_summaries.go Create: ON CONFLICT add WHERE source_id IS NOT NULL for partial index
- migration 040: add immutable_array_to_string wrapper (array_to_string is STABLE in PG)
* test(store): add 17 integration tests for skills, cron, episodic, tenant configs
- Skills store: 6 tests (CRUD, grants, tenant isolation)
- Cron store: 4 tests (job CRUD, run log sqlx scan, pagination, tenant isolation)
- Episodic store: 4 tests (summary CRUD, list, FTS search, tenant isolation)
- Tenant configs: 3 tests (tool/skill disable, list, tenant isolation)
- Test helper: add cleanup for skills, cron, episodic tables
* fix(permissions): use cron-specific permission check for cron tool (#725)
* fix(security): harden exec path exemption matching (#721)
- Add absolute path exemption for dataDir/skills-store/ (fixes skill
scripts using absolute paths like /app/data/skills-store/ being denied)
- Strip surrounding quotes before prefix matching (LLMs often quote paths)
- Reject path traversal ("..") in exempt fields to prevent escape
- Switch from "any field exempt → skip" to per-field matching: only exempt
if ALL fields that match the deny pattern are individually exempt
- Closes pipe/comment bypass vectors where an exempt path in one argument
would exempt the entire command including non-exempt paths
Includes 27 test cases covering: legitimate access, quoted paths,
path traversal, unicode bypass, pipe/comment bypass, mixed args.
* fix(permissions): use cron-specific permission check for cron tool
Cron tool was hardcoded to check `file_writer` configType via
CheckFileWriterPermission(), ignoring the `cron` configType that
the UI actually saves when granting cron permissions. This caused
agents in group chats to be denied cron access even with correct
permission configured.
Add ConfigTypeCron constant and CheckCronPermission() that checks
`cron` configType first, falling back to `file_writer`.
---------
Co-authored-by: Viet Tran <viettranx@gmail.com>
* fix(chat): load message history on first conversation click (#730)
* fix(chat): load message history when selecting existing conversation from clean state
The skipNextHistoryRef was unconditionally set when sessionKey transitioned
from empty to non-empty. This prevented loadHistory() from running when
clicking an existing conversation from the initial /chat page. The skip
was only intended for the new-chat send flow where the optimistic message
is already displayed.
Guard the skip with expectingRunRef so it only activates when a message
send is in flight.
Closes #729
* docs: add UI diff evidence for PR #730
Before/after screenshots and HTML comparison report showing
first conversation click behavior fix.
* feat(whatsapp): port native WhatsApp channel with whatsmeow from dev
Cherry-pick
|