mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-13 18:15:34 +00:00
Kai (Tam Nhu) Tranandviettranx
4b02c27307
* refactor(providers): migrate ToolDefinition.Function to pointer + add image response fields
ToolDefinition.Function becomes *ToolFunctionSchema with omitempty so native tool types (image_generation, web_search, etc.) can be declared without a function body. All 9 internal construction sites updated. CleanToolSchemas refactored — function-shape cleaning extracted into cleanFunctionSchema helper, outer pass-through handles native tools.
Added image response fields needed by the next commits: ChatResponse.Images, StreamChunk.Images, ImageContent.Partial (distinguishes partial frames from final images).
* feat(providers): native image_generation for Codex + OpenAI-compat tracks
Codex native (POST /codex/responses): emit image_generation tool object in request tools[] (type, action, model, output_format, partial_images). Handle SSE events response.image_generation_call.partial_image + response.output_item.done (type image_generation_call) + response.completed output[] walk for non-stream. Dedup per item_id. Extend codexSSEEvent/codexItem with output_format, result, partial_image_b64, partial_image_index.
OpenAI-compat (/v1/chat/completions): serialize ToolDefinition{Type:'image_generation'} as {type:'image_generation'} pass-through. Parse choices[0].message.images[] + delta.images[] (data URLs) via new parseDataURL helper; append to ChatResponse.Images.
ProviderCapabilities.ImageGeneration flag; Codex provider + adapter set true. Other providers default false.
* feat(agent,http,store): persist assistant images + tri-level image_generation gate
Agent loop tri-level gate: (provider capability) AND (AgentConfig.AllowImageGeneration, default true, stored in other_config.allow_image_generation) AND (request lacks x-goclaw-no-image-gen header). Gate in loop_tool_filter.go appends ToolDefinition{Type:'image_generation'} only when all three pass. Per-request opt-out parsed in chat_completions.go and propagated via RunRequest.NoImageGen.
Media persistence: persistAssistantImages writes final images (Partial:false) to {workspace}/media/{sha256}.{ext}, returns MediaRef entries, clears inline Images[] from the message. Idempotent on hash, traversal-safe, symlink-guarded. Invoked from pipeline.FinalizeStage via new Deps.PersistAssistantImages callback — covers both stream-final and non-stream paths.
Agent store reads AllowImageGeneration from other_config JSONB with absent/nil = true default (matches V3Flags pattern). No DB migration — code-only default.
* feat(ui/web): image_generation toggle + streaming placeholder + download filename
Composer chip 'Images' visible only when active agent's provider has ImageGeneration capability. Per-agent localStorage persistence via useImageGenToggle hook. When off, sends noImageGen:true to WS chat method (maps to x-goclaw-no-image-gen on upstream HTTP call path).
ActiveRunZone renders a skeleton placeholder while streaming partial_image frames arrive. MediaGallery assigns generated-YYYYMMDD-HHmmss.png as the download filename for hex/UUID PNGs.
i18n keys added to en/vi/zh chat.json: imageGenToggle, imageGenGenerating, imageGenDownloadName. 8 vitest tests for the toggle hook.
* docs: add Image Generation section to codebase-summary + changelog entry
Documents the new native image_generation pipeline across providers layer (Codex + OpenAI-compat), agent gate, media persistence, and web UI surface.
* fix(ui/web): match Codex-routed providers for image_generation toggle
Image-gen toggle visibility was hard-coded to provider id 'chatgpt_oauth' but real Codex-routed agents in production use provider ids like 'cliproxy-codex'. The toggle never rendered.
Replace the Set-has check with a small helper that accepts the literal ids plus any provider string containing 'codex' (case-insensitive). Same logic applied in both chat-input.tsx (composer chip) and chat-page.tsx (streaming placeholder gate).
Verified against a live Codex-routed agent: toggle now renders, noImageGen:true propagates on toggle-off.
* docs(pr-1002): targeted-mode UX evidence report
Captures the UI integration surface for native image_generation against a live Codex-routed agent on the remote dev backend.
Includes: composer toggle chip (rendered), streaming skeleton placeholder, and honest failure-path capture showing the legacy create_image builtin fallback. Self-contained HTML report in .github/pr-assets/1002/index.html.
* fix(permissions): classify sessions.compact as write method
CI RBAC-drift test (TestMethodRole_DriftCoverage_AllProtocolMethodsClassified) was failing because the new sessions.compact method added upstream was not classified in any of isPublicMethod / isAdminMethod / isWriteMethod / isReadMethod.
Sessions compaction mutates session history (compacts messages into summaries), so it belongs with the other sessions.* write methods.
* fix(tests): remove duplicate contains() in integration package
Both tts_gemini_live_test.go and mcp_grant_revoke_test.go declared a file-local func contains(s, substr string) bool with identical bodies, causing 'contains redeclared in this block' at compile time in the integration job.
Replace all call sites with strings.Contains (same semantics, stdlib) and drop the duplicates. No behavior change.
* feat(providers): NativeImageProvider interface + Codex implementation
Defines a provider-level contract (NativeImageProvider.GenerateImage) that OAuth-backed providers can implement to serve image generation without exposing static API credentials. Re-uses the PR's Track A native wire format (POST /codex/responses with image_generation tool, item.result decoding, SSE fallback).
CodexProvider + CodexAdapter implement it. Also adds MediaRef.Prompt field so downstream layers can propagate the generating prompt alongside the asset.
* feat(tools): route create_image through NativeImageProvider for OAuth providers
create_image.callProvider now checks for a NativeImageProvider implementation before the credentialProvider interface. When the provider chain points at a Codex-family provider (no static API key), the tool delegates to the provider's GenerateImage which executes the native ChatGPT Responses API image_generation flow.
Resolves 'provider X does not expose API credentials required for image generation' errors for openai-codex / cliproxy-codex chains. On success the tool embeds the user's prompt as a PNG tEXt 'Description' chunk (file-local pngEmbedPrompt helper to avoid tools→agent import cycle), writes the image to /tmp, and threads the prompt through result.MediaPrompts for downstream MediaRef propagation.
* feat(agent,pipeline): propagate image prompt through MediaRef + PNG tEXt embed helper
Adds EmbedPNGPrompt public helper in internal/agent/png_metadata.go that inserts a tEXt 'Description' chunk (plus 'Software: goclaw') into PNG byte streams before the IEND chunk. Non-PNG inputs are passed through unchanged — the helper never errors on unknown formats.
FinalizeStage wires MediaResult.Prompt (from create_image tool output) onto MediaRef.Prompt so the UI can render the generating prompt alongside the image. Per-image prompt list threaded via pipeline RunState.
* feat(ui/web): show generating prompt as caption under each image in MediaGallery
When a MediaRef carries a prompt, MediaGallery renders it as a muted italic caption (line-clamp-2) beneath the image with the full text in the title tooltip. Caption is hidden when the prompt is absent so non-assistant images (user uploads, legacy data) look unchanged.
MediaItem + session media_refs types extended with an optional prompt field; the chat-message adapter threads ref.prompt through when converting WS payloads to UI state.
* fix(providers/codex): stream:true + instructions for native image_generation
The ChatGPT Responses API on /codex/responses rejects two things hard:
- stream:false → HTTP 400 "Stream must be set to true"
- missing instructions → HTTP 400 "Instructions are required"
buildNativeImageRequestBody now sets stream:true and a purpose-specific instructions string ("Generate an image matching the user's description using the image_generation tool. Return only the image; do not describe it in text."). The existing parseNativeImageSSE path was already in place for stream parsing; routing changed from the non-stream branch to the SSE branch.
Regression assertions added to TestCodexGenerateImage_BuildsNativeRequest so these two fields can't silently regress.
* feat(providers,tools,ui): image_model whitelist (gpt-image-2 default, gpt-image-1.5 legacy)
Replaces the hardcoded "gpt-image-2" literal in buildNativeImageRequestBody with a user-configurable field threaded through NativeImageRequest.ImageModel. The whitelist is enforced by ValidateImageModel which rejects anything outside {gpt-image-2, gpt-image-1.5} with a clear error — prevents silent upstream 400s from model names the Responses API would reject.
create_image.callProvider reads params.image_model from the chain entry and threads it through. Empty / absent falls back to DefaultImageModel (gpt-image-2).
UI: added an 'Image model' select inside the existing openai-codex Settings panel on the Create Image Provider Chain dialog. Options: Default · gpt-image-2 (recommended) and Legacy · gpt-image-1.5. i18n keys in en/vi/zh tools.json under builtin.mediaChain.
Tests: TestCodexGenerateImage covers default/legacy/rejected model cases; TestCreateImageTool_ThreadsImageModel covers params→request threading with empty/legacy/explicit sub-cases.
* fix(tools): raise media chain default timeout to 600s/1 retry for image gen
gpt-image-2 on complex prompts (dense Vietnamese text, infographic layouts) legitimately takes 4–8 minutes to complete. The previous default of 120s × 2 retries routinely died mid-generation with 'context deadline exceeded' — the upstream run was still producing bytes when our ctx cancelled.
Default is now Timeout: 600 / MaxRetries: 1. Retries dropped to 1 because image generation is stateful per upstream run: a mid-flight timeout leaves orphan server work, and retrying a fresh generation doubles cost for no gain. Surface the failure fast so operators can widen the timeout instead.
Operators can still set a tighter value explicitly via the Chain dialog.
* refactor: remove user-facing Images toggle, keep admin-level AllowImageGeneration
The per-request opt-out toggle (composer chip + streaming placeholder + noImageGen header plumbing) was a support footgun — users toggle OFF, forget, then can't generate images and think it's broken. Removed in full.
Kept: AgentConfig.AllowImageGeneration (admin kill-switch, stored in other_config.allow_image_generation, default true). Tri-level gate simplifies to two tiers: provider capability AND agent config allows.
Removed: useImageGenToggle hook, IMAGE_GEN_PROVIDER_IDS set in chat-input, supportsImageGenProvider helper, agentProvider/agentKey props on ChatInput, showImageGenPlaceholder prop on MessageBubble/ActiveRunZone/ChatThread, noImageGen param on use-chat-send, parseNoImageGen in chat_completions.go, NoImageGen on RunRequest, no_image_gen_header_test.go, imageGenToggle/imageGenGenerating i18n keys. Kept imageGenDownloadName — used by MediaGallery for generated-\*.png filename resolution.
* docs(pr-1002): refreshed UX trace + updated codebase notes
Replaces the earlier stealth-state evidence with a clean three-capture trace from a real successful run: inline image + prompt caption, MediaGallery lightbox expansion, Chain dialog with the new Image model dropdown open. Skill-routing rows scrubbed from the capture — they reflect per-agent skill setup, not anything this PR introduces.
codebase-summary + changelog: reflect final state (toggle removed, image_model selector, 600s default chain timeout, gpt-image-2 as quality baseline).
* fix(pipeline): preserve mid-loop image_generation output across iterations
FinalizeStage previously read state.Think.LastResponse.Images, which holds
only the last iteration's response. If the LLM emitted image_generation_call
in iteration N alongside a function_call, then responded text-only in N+1,
the image from N was silently dropped on finalize.
Accumulate final (non-partial) images into state.Observe.AssistantImages
across every iteration via ObserveStage, and source FinalizeStage from the
accumulator instead of LastResponse. Partial streaming frames are filtered
defensively; response.Images is cleared on drain to prevent double-counting
on re-exec.
* test(pipeline): regression coverage for mid-loop image accumulation
Six ObserveStage cases covering image accumulation semantics:
- single-iter image-only, image+tool_call same iter, mid-loop image
surviving text-only final iter, multiple images across iters, partial
frame filtering, nil-response safety.
Two FinalizeStage cases verifying accumulator is the source of truth:
- PersistsFromObserveAccumulator: image in Observe + empty LastResponse
must still be persisted via PersistAssistantImages.
- NoPersistWhenAccumulatorEmpty: no call when no images were emitted.
---------
Co-authored-by: viettranx <viettranx@gmail.com>
Multi-Tenant AI Agent Platform
Multi-agent AI gateway built in Go. 20+ LLM providers. 7 channels. Multi-tenant PostgreSQL.
Single binary. Production-tested. Agents that orchestrate for you.
Documentation • Quick Start • Twitter / X
🌐 Languages: 🇨🇳 简体中文 · 🇯🇵 日本語 · 🇰🇷 한국어 · 🇻🇳 Tiếng Việt · 🇵🇭 Tagalog · 🇪🇸 Español · 🇧🇷 Português · 🇮🇹 Italiano · 🇩🇪 Deutsch · 🇫🇷 Français · 🇸🇦 العربية · 🇮🇳 हिन्दी · 🇷🇺 Русский · 🇧🇩 বাংলা · 🇮🇱 עברית · 🇵🇱 Polski · 🇨🇿 Čeština · 🇳🇱 Nederlands · 🇹🇷 Türkçe · 🇺🇦 Українська · 🇮🇩 Bahasa Indonesia · 🇹🇭 ไทย · 🇵🇰 اردو · 🇷🇴 Română · 🇸🇪 Svenska · 🇬🇷 Ελληνικά · 🇭🇺 Magyar · 🇫🇮 Suomi · 🇩🇰 Dansk · 🇳🇴 Norsk
Core Features
- 8-Stage Agent Pipeline — context → history → prompt → think → act → observe → memory → summarize. Pluggable stages, always-on execution
- 4-Mode Prompt System — Full / Task / Minimal / None with section gating, cache boundary optimization, and per-session mode resolution
- 3-Tier Memory — Working (conversation) → Episodic (session summaries) → Semantic (knowledge graph). Progressive loading L0/L1/L2
- Knowledge Vault — Document registry with wikilinks, hybrid search (FTS + pgvector), filesystem sync
- Agent Teams & Orchestration — Shared task boards, inter-agent delegation (sync/async), 3 orchestration modes (auto/explicit/manual)
- Self-Evolution — Metrics → suggestions → auto-adapt with guardrails. Agents refine their own communication style
- Multi-Tenant PostgreSQL — Per-user workspaces, per-user context files, encrypted API keys (AES-256-GCM), RBAC, isolated sessions
- 20+ LLM Providers — Anthropic (native HTTP+SSE with prompt caching), OpenAI, OpenRouter, Groq, DeepSeek, Gemini, Mistral, xAI, MiniMax, DashScope, Claude CLI, Codex, ACP, and any OpenAI-compatible endpoint
- 7 Messaging Channels — Telegram, Discord, Slack, Zalo OA, Zalo Personal, Feishu/Lark, WhatsApp
- Production Security — 5-layer permission system, rate limiting, prompt injection detection, SSRF protection, AES-256-GCM encryption
- Single Binary — ~25 MB static Go binary, no Node.js runtime, <1s startup, runs on a $5 VPS
- Observability — Built-in LLM call tracing with spans and prompt cache metrics, optional OpenTelemetry OTLP export
Desktop Edition (GoClaw Lite)
A native desktop app for local AI agents — no Docker, no PostgreSQL, no infrastructure.
macOS:
curl -fsSL https://raw.githubusercontent.com/nextlevelbuilder/goclaw/main/scripts/install-lite.sh | bash
Windows (PowerShell):
irm https://raw.githubusercontent.com/nextlevelbuilder/goclaw/main/scripts/install-lite.ps1 | iex
What's Included
- Single native app (Wails v2 + React), ~30 MB
- SQLite database (zero setup)
- Chat with agents (streaming, tools, media, file attachments)
- Agent management (max 5), provider config, MCP servers, skills, cron
- Team tasks with Kanban board and real-time updates
- Auto-update from GitHub Releases
Lite vs Standard
| Feature | Lite (Desktop) | Standard (Server) |
|---|---|---|
| Agents | Max 5 | Unlimited |
| Teams | Max 1 (5 members) | Unlimited |
| Database | SQLite (local) | PostgreSQL |
| Memory | FTS5 text search | pgvector semantic |
| Channels | — | Telegram, Discord, Slack, Zalo, Feishu, WhatsApp |
| Knowledge Graph | — | Full |
| RBAC / Multi-tenant | — | Full |
| Auto-update | GitHub Releases | Docker / binary |
Building from Source
# Prerequisites: Go 1.26+, pnpm, Wails CLI (go install github.com/wailsapp/wails/v2/cmd/wails@latest)
make desktop-build # Build .app (macOS) or .exe (Windows)
make desktop-dmg VERSION=0.1.0 # Create .dmg installer (macOS only)
make desktop-dev # Dev mode with hot reload
Desktop Releases
Desktop uses independent versioning with lite-v* tags:
git tag lite-v0.1.0 && git push origin lite-v0.1.0
# → GitHub Actions builds macOS (.dmg + .tar.gz) + Windows (.zip)
# → Creates GitHub Release with all assets
Architecture
Quick Start
Prerequisites: Go 1.26+, PostgreSQL 18 with pgvector, Docker (optional)
From Source
git clone -b main https://github.com/nextlevelbuilder/goclaw.git && cd goclaw
make build
./goclaw onboard # Interactive setup wizard
source .env.local && ./goclaw
Note: The default branch is
dev(active development). Use-b mainto clone the stable release branch.
With Docker
# Generate .env with auto-generated secrets
chmod +x prepare-env.sh && ./prepare-env.sh
# Add at least one GOCLAW_*_API_KEY to .env, then:
make up
# Web Dashboard at http://localhost:18790 (built-in)
# Health check: curl http://localhost:18790/health
# Optional: separate nginx for custom SSL/reverse proxy
# make up WITH_WEB_NGINX=1 → Dashboard at http://localhost:3000
make up creates a Docker network, embeds the correct version from git tags, builds and starts all services, and runs database migrations automatically.
Common commands:
make up # Start all services (build + migrate)
make down # Stop all services
make logs # Tail logs (goclaw service)
make reset # Wipe volumes and rebuild from scratch
Optional services — enable with WITH_* flags:
| Flag | Service | What it does |
|---|---|---|
WITH_BROWSER=1 |
Headless Chrome | Enables browser tool for web scraping, screenshots, automation |
WITH_OTEL=1 |
Jaeger | OpenTelemetry tracing UI for debugging LLM calls and latency |
WITH_SANDBOX=1 |
Docker sandbox | Isolated container for running untrusted code from agents |
WITH_TAILSCALE=1 |
Tailscale | Expose gateway over Tailscale private network |
WITH_REDIS=1 |
Redis | Redis-backed caching layer |
Flags can be combined and work with all commands:
# Start with browser automation and tracing
make up WITH_BROWSER=1 WITH_OTEL=1
# Stop everything including optional services
make down WITH_BROWSER=1 WITH_OTEL=1
When GOCLAW_*_API_KEY environment variables are set, the gateway auto-onboards without interactive prompts — detects provider, runs migrations, and seeds default data.
Docker image variants:
Image Description latestBackend + embedded web UI + Python (recommended) latest-baseBackend API-only, no web UI, no runtimes latest-fullAll runtimes + skill dependencies pre-installed latest-otelLatest + OpenTelemetry tracing goclaw-webStandalone nginx + React SPA (for custom reverse proxy) For custom builds (Tailscale, Redis):
docker build --build-arg ENABLE_TSNET=true ...See the Deployment Guide for details.
Updating
Docker
docker compose pull && docker compose up -d
Binary (with embedded web UI)
goclaw update --apply # Downloads, verifies SHA256, swaps binary, restarts
Web Dashboard
Open About dialog → click Update Now (admin only). The update includes both backend and web dashboard when using the default latest image.
Multi-Agent Orchestration
Each agent runs with its own identity, tools, LLM provider, and context files. Three delegation modes — sync (wait), async (fire-and-forget), bidirectional — connected through explicit permission links with concurrency limits.
Details: Agent Teams docs
Knowledge Vault
Document registry with [[wikilinks]] for bidirectional linking. Hybrid search combines full-text (BM25) and semantic (pgvector) for precise retrieval. Filesystem sync keeps vault in sync with on-disk files.
Self-Evolution
Agents improve themselves through a 3-stage guardrailed pipeline: metrics collection → suggestion analysis → auto-adaptation. Can refine communication style and domain expertise (CAPABILITIES.md) — but never change identity, name, or core purpose.
Provider Adapters
20+ LLM providers unified through a single adapter interface. Capability-based routing, encrypted API keys (AES-256-GCM), extended thinking support per-provider, and prompt caching for Anthropic + OpenAI.
Event-Driven Architecture
Typed domain events power the consolidation pipeline — session summaries, knowledge graph extraction, and dreaming promotion all run asynchronously via worker pools with dedup and retry.
Built-in Tools
30+ tools across 8 categories:
| Category | Tools | Description |
|---|---|---|
| Filesystem | read_file, write_file, edit_file, list_files, search, glob |
File operations with virtual FS routing |
| Runtime | exec, browser |
Shell commands (approval workflow) + browser automation |
| Web | web_search, web_fetch |
Search (Brave, DuckDuckGo) + content extraction |
| Memory | memory_search, memory_get, knowledge_graph_search |
3-tier memory + KG traversal |
| Media | create_image, create_audio, create_video, read_*, tts |
Generation + analysis (multi-provider) |
| Skills | skill_search, use_skill, skill_manage |
BM25 + semantic hybrid search |
| Teams | team_tasks, spawn, delegate, message |
Task board + orchestration + messaging |
| Automation | cron, heartbeat, sessions_* |
Scheduling + session management |
Full tool reference at docs.goclaw.sh
Documentation
Full documentation at docs.goclaw.sh — or browse the source in goclaw-docs/
| Section | Topics |
|---|---|
| Getting Started | Installation, Quick Start, Configuration, Web Dashboard Tour |
| Core Concepts | Agent Loop, Sessions, Tools, Memory, Multi-Tenancy |
| Agents | Creating Agents, Context Files, Personality, Sharing & Access |
| Providers | Anthropic, OpenAI, OpenRouter, Gemini, DeepSeek, +15 more |
| Channels | Telegram, Discord, Slack, Feishu, Zalo, WhatsApp, WebSocket |
| Agent Teams | Teams, Task Board, Messaging, Delegation & Handoff |
| Advanced | Custom Tools, MCP, Skills, Cron, Sandbox, Hooks, RBAC |
| Deployment | Docker Compose, Database, Security, Observability, Tailscale |
| Reference | CLI Commands, REST API, WebSocket Protocol, Environment Variables |
Testing
go test ./... # Unit tests
go test -v ./tests/integration/ -timeout 120s # Integration tests (requires running gateway)
Project Status
See CHANGELOG.md for detailed feature status including what's been tested in production and what's still in progress.
Acknowledgments
GoClaw was originally inspired by the OpenClaw project architecture.
License
CC BY-NC 4.0 — Creative Commons Attribution-NonCommercial 4.0 International