read_video called with a url parameter always failed under an agent budget:
tool:read_video: cannot verify streamed native media against the agent
context budget (no in-memory payload to count); refusing to send
read_audio and read_document appeared to work but only for very small files.
Both symptoms share one root cause.
5ca433b8 introduced the complete-input invariant and, to keep out-of-band
media honest, appended the standard-base64 encoding of the payload as a
synthetic guard-only message. That message is counted as text. Measured against
the bundled BudgetCounter, base64 costs 0.956 tokens per raw byte, so an 11.6 MB
video counted 11,161,058 tokens and exceeded a 200k, a 1M and a 2M window alike.
The practical ceiling was roughly 200 KB at a 200k window while videoMaxBytes is
100 MB. The read_video URL transport streams to the File API without buffering,
so it had no bytes to count at all and was routed to a helper that refused to
send whenever an agent budget was present.
Media is now priced by what a provider actually bills for it. Every number below
is either a published per-unit rate or a published provider limit; none is
derived from a byte size, because a static-image video compresses arbitrarily
small and no bitrate floor exists. An earlier revision of this branch tried one
and a 120-second, 20,627-byte clip priced at 526 tokens against a true 31,560.
internal/mediabudget prices one payload:
- Video and audio: ffprobe measures duration. 263 tokens/second for video,
32 for audio, both published for static processing at 1 FPS.
- PDF: pdfinfo counts pages at 258 tokens/page. When the page count cannot be
read, the charge is the proven ceiling of 1000 pages, which is the most a
provider will accept and therefore the most it can bill.
- Video and audio that cannot be measured are refused. Upstream already
refused unverifiable native media for every URL on the Gemini streamed path;
this narrows that refusal from every URL to only what genuinely cannot be
measured, rather than removing it. PDF differs because its page count is
cheaply measurable and its ceiling is small enough to stay usable.
A remote video is measured without downloading it: two ranged GETs, 512 KB from
the head and 512 KB from the tail, written into a sparse temp file sized to the
declared total and handed to ffprobe. The tail matters because every container
that puts its index at the end keeps it there: head-only probing under-reports
mpeg by 98% and ogg by 79%, and a plain prefix makes ffprobe under-report a
30-second WAV as 0.74 seconds because it clamps to the bytes it can see. Sizing
the temp file to the real total fixes that. Verified end to end on an 11,673,105
byte MP4 served by nginx: 1 MB of ranged reads yielded duration 40.000000,
identical to ffprobe reading the whole URL, for a charge of 10,520 tokens. Those
requests reuse the existing SSRF-safe path, security.WithPinnedIP plus
security.NewSafeClient(0), and no URL is ever handed to an external binary.
Beyond the reported bug, two pre-existing gaps let large media reach a provider
almost unpriced. ExecuteWithChain treated every callProvider error as a provider
failure and advanced to the next entry, and the non-Gemini branches of read_video
and read_document reserved without pricing their payload at all. Measured under a
20,000-token window before this change, a 40 MB video and a 1000-page PDF each
reached a provider charged about 1,600 tokens. Budget refusals are now terminal
in the chain and every media branch prices its payload, so both reach no provider
at all. Genuine provider failures still fail over.
Known limits, stated rather than discovered:
- The /Type /Page scan that guards against a forged /Count is a floor, not a
bound. Pages inside a compressed object stream are invisible to it, and a
9,484-byte PDF built that way is charged 258 tokens for 1000 pages. A real
pdfinfo reads such files correctly; the scan only ever raises a probed count.
- A video URL whose origin does not serve byte ranges is now refused on the
non-Gemini path too, and the refusal is terminal. Upstream forwarded such URLs
unpriced. A HEAD giving only a size is not enough to price one.
- read_video and read_audio require ffprobe. Docker images install it by default
except the base variant; bare binaries and the desktop build do not ship it.
- A hostile origin can craft a container ffprobe reads as about one second.
Reservation.Reconcile overwrites the estimate with the provider's reported
usage, so this weakens the gate rather than defeating it.
Byte ceilings videoMaxBytes, audioMaxBytes and documentMaxBytes are unchanged.
No new module dependency; ffprobe and pdfinfo are optional runtime probes.
Squash merge PR #111 after resolving docs/changelog conflicts. Local checks covered tools/config and both Go builds; PR CI run 26704622503 passed release-versioning, go, and web.
read_image and read_document failed when only claude-cli was configured
because the fallback chains only listed 'anthropic'. Adding claude-cli
as fallback (PR #802) surfaced a deeper bug: buildStreamJSONInput
hardcoded 'type: image' for every content block, so PDFs routed
through claude-cli were rejected by the Anthropic API with a MIME
mismatch.
- providers/claude_cli_session.go: buildStreamJSONInput now picks the
Anthropic block type from MIME — application/pdf -> document,
image/* -> image.
- tools/read_image.go, read_document.go: add claude-cli to the
fallback priority and model defaults (empty string lets the provider
pick its own default model).
- tools/read_image.go, read_document_resolve.go: scope disable_tools to
claude-cli only instead of leaking a CLI-specific flag into the
shared Options map every provider in the chain receives.
- providers/claude_cli_session_test.go: table test covering png/pdf/
mixed/unknown MIME routing plus the empty-text edge case.
Closes#801.
* feat(ui): multi-skill upload with client-side validation
Allow uploading multiple skill ZIP files at once with pre-upload
validation. JSZip parses each ZIP client-side to verify SKILL.md
presence, frontmatter format, and slug validity before upload.
- Add JSZip dependency (lazy-loaded, code-split ~30KB gzip)
- Create validate-skill-zip.ts mirroring server-side checks
- Rewrite skill-upload-dialog for multi-file with status badges
- Add concurrent validation, sequential upload with per-file progress
- Add empty SKILL.md check to backend upload handler
- Add i18n keys for all new UI strings (en/vi/zh)
* fix(ui): duplicate entries and validation hang in multi-skill upload
- Move pending list construction to assignment inside updater return
to prevent StrictMode double-invoke from pushing duplicates
- Wrap per-file validateSkillZip in try/catch so one failure doesn't
block Promise.all and leave entries stuck in "validating" state
* fix(ui): use static import for JSZip instead of dynamic import
Dynamic import("jszip") fails in browser - bare module specifiers
don't resolve at runtime. Use static import which Vite handles
via its module graph and code-splits automatically.
* feat(ui): add inline visibility toggle on skills table
Click the visibility badge on managed skills to cycle through
private → internal → public. File-based skills stay read-only.
* fix(ui): move dedup logic outside state updater in upload dialog
Avoids reading stale entries inside functional updater. Builds
pending list from current entries state before calling setEntries.
* fix(ui): auto-select first active agent when current agent unavailable
When agents load from API, if the current selected agent is not in the active agents list, automatically select the first available active agent instead of remaining unset. Prevents chat page from being unable to send messages when default agent selection is invalid.
* feat(ui): make agent display name editable in setup wizard
Allow users to customize the agent display name during onboarding instead of keeping it hardcoded to "GoClaw". Removed read-only state from the display name input and added a placeholder for guidance.
* feat: add document path enrichment and media filename support
Backend changes:
- enrichDocumentPaths() in agent/media.go: injects persisted file paths into <media:document> tags
- Document paths allow skills (e.g. pdf skill via exec) to access files directly
- chat.go: support new media format {path, filename} alongside legacy string paths
- Updated read_document tool description to guide agent on using path attribute
- Docker: add pypdf to Python dependencies for PDF processing
- Softened MUST language in read_* tool descriptions (changed to Call this)
Frontend changes:
- chat-input.tsx: attach filename with each uploaded file in media payload
- use-chat-send.ts: send media as {path, filename} objects instead of just paths
- i18n: add "uploaded_files" text in en, vi, zh locales
- chat-page.tsx: minor adjustment for media handling
Enables skills to process uploaded documents directly without intermediate copying.
- Update go.mod and Dockerfile to Go 1.26
- Apply `go fix ./...` stdlib modernizations across 170+ files
- Add `go fix` to post-implementation checklist in CLAUDE.md
- Fix go fix misapplied rewrite in loop_history.go
Refactor read_image, read_document, read_video, read_audio to use
ResolveMediaProviderChain + ExecuteWithChain for consistent fallback behavior.
Add hardcoded model lists for MiniMax, DashScope, and Suno providers.
- Add persistent media storage (internal/media/) replacing temp file deletion
- Add MediaRef type for lightweight media references in session messages
- Refactor media pipeline to use bus.MediaFile{Path, MimeType} across all channels
- Add read_document builtin tool for PDF/DOCX/XLSX analysis via Gemini native API
- Move image sanitization from Telegram to shared agent/media layer
- Add media reload for multi-turn conversations (images from last 5 messages)
- Add reply-to-message media resolution for Telegram (re-download on reply)
- Add media inventory to compaction summary to preserve awareness after truncation
- Fix coreToolSummaries for read_image, read_document, create_image tools
- Add real-time trace update events via WebSocket broadcast
- Improve trace detail UI with media refs and tool result display
thotam