Files
goclaw/internal/tools/read_document_resolve.go
thotam ce9e7af3fb fix(tools): price out-of-band media by measured duration, not by its bytes (#1564)
read_video called with a url parameter always failed under an agent budget:

    tool:read_video: cannot verify streamed native media against the agent
    context budget (no in-memory payload to count); refusing to send

read_audio and read_document appeared to work but only for very small files.
Both symptoms share one root cause.

5ca433b8 introduced the complete-input invariant and, to keep out-of-band
media honest, appended the standard-base64 encoding of the payload as a
synthetic guard-only message. That message is counted as text. Measured against
the bundled BudgetCounter, base64 costs 0.956 tokens per raw byte, so an 11.6 MB
video counted 11,161,058 tokens and exceeded a 200k, a 1M and a 2M window alike.
The practical ceiling was roughly 200 KB at a 200k window while videoMaxBytes is
100 MB. The read_video URL transport streams to the File API without buffering,
so it had no bytes to count at all and was routed to a helper that refused to
send whenever an agent budget was present.

Media is now priced by what a provider actually bills for it. Every number below
is either a published per-unit rate or a published provider limit; none is
derived from a byte size, because a static-image video compresses arbitrarily
small and no bitrate floor exists. An earlier revision of this branch tried one
and a 120-second, 20,627-byte clip priced at 526 tokens against a true 31,560.

  internal/mediabudget prices one payload:
  - Video and audio: ffprobe measures duration. 263 tokens/second for video,
    32 for audio, both published for static processing at 1 FPS.
  - PDF: pdfinfo counts pages at 258 tokens/page. When the page count cannot be
    read, the charge is the proven ceiling of 1000 pages, which is the most a
    provider will accept and therefore the most it can bill.
  - Video and audio that cannot be measured are refused. Upstream already
    refused unverifiable native media for every URL on the Gemini streamed path;
    this narrows that refusal from every URL to only what genuinely cannot be
    measured, rather than removing it. PDF differs because its page count is
    cheaply measurable and its ceiling is small enough to stay usable.

A remote video is measured without downloading it: two ranged GETs, 512 KB from
the head and 512 KB from the tail, written into a sparse temp file sized to the
declared total and handed to ffprobe. The tail matters because every container
that puts its index at the end keeps it there: head-only probing under-reports
mpeg by 98% and ogg by 79%, and a plain prefix makes ffprobe under-report a
30-second WAV as 0.74 seconds because it clamps to the bytes it can see. Sizing
the temp file to the real total fixes that. Verified end to end on an 11,673,105
byte MP4 served by nginx: 1 MB of ranged reads yielded duration 40.000000,
identical to ffprobe reading the whole URL, for a charge of 10,520 tokens. Those
requests reuse the existing SSRF-safe path, security.WithPinnedIP plus
security.NewSafeClient(0), and no URL is ever handed to an external binary.

Beyond the reported bug, two pre-existing gaps let large media reach a provider
almost unpriced. ExecuteWithChain treated every callProvider error as a provider
failure and advanced to the next entry, and the non-Gemini branches of read_video
and read_document reserved without pricing their payload at all. Measured under a
20,000-token window before this change, a 40 MB video and a 1000-page PDF each
reached a provider charged about 1,600 tokens. Budget refusals are now terminal
in the chain and every media branch prices its payload, so both reach no provider
at all. Genuine provider failures still fail over.

Known limits, stated rather than discovered:
- The /Type /Page scan that guards against a forged /Count is a floor, not a
  bound. Pages inside a compressed object stream are invisible to it, and a
  9,484-byte PDF built that way is charged 258 tokens for 1000 pages. A real
  pdfinfo reads such files correctly; the scan only ever raises a probed count.
- A video URL whose origin does not serve byte ranges is now refused on the
  non-Gemini path too, and the refusal is terminal. Upstream forwarded such URLs
  unpriced. A HEAD giving only a size is not enough to price one.
- read_video and read_audio require ffprobe. Docker images install it by default
  except the base variant; bare binaries and the desktop build do not ship it.
- A hostile origin can craft a container ffprobe reads as about one second.
  Reservation.Reconcile overwrites the estimate with the provider's reported
  usage, so this weakens the gate rather than defeating it.

Byte ceilings videoMaxBytes, audioMaxBytes and documentMaxBytes are unchanged.
No new module dependency; ffprobe and pdfinfo are optional runtime probes.
2026-09-12 10:31:12 +07:00

216 lines
7.0 KiB
Go

package tools
import (
"context"
"encoding/base64"
"fmt"
"log/slog"
"path/filepath"
"strings"
"github.com/nextlevelbuilder/goclaw/internal/mediabudget"
"github.com/nextlevelbuilder/goclaw/internal/providers"
)
// resolveDocumentFile finds the document file path from an explicit workspace
// path or from context MediaRefs.
func (t *ReadDocumentTool) resolveDocumentFile(ctx context.Context, mediaID, docPath string) (path, mime string, err error) {
if strings.TrimSpace(mediaID) != "" && strings.TrimSpace(docPath) != "" {
return "", "", fmt.Errorf("specify either media_id or path, not both")
}
if docPath != "" {
p, err := resolveDocumentPathArg(ctx, docPath)
if err != nil {
return "", "", err
}
return p, mimeFromDocExt(filepath.Ext(p)), nil
}
refs := MediaDocRefsFromCtx(ctx)
if len(refs) == 0 {
return "", "", fmt.Errorf("no documents available in this conversation. The user may not have sent a document.")
}
// Find specific media_id or use most recent document.
var ref *providers.MediaRef
if mediaID != "" {
for i := range refs {
if refs[i].ID == mediaID {
ref = &refs[i]
break
}
}
if ref == nil {
return "", "", fmt.Errorf("document media_id %q not found in this conversation", mediaID)
}
} else {
// Use the last (most recent) document ref.
ref = &refs[len(refs)-1]
}
// Prefer persisted workspace path; fall back to legacy .media/ lookup.
p := ref.Path
loadedLegacy := false
if p == "" {
var err error
if t.mediaLoader == nil {
return "", "", fmt.Errorf("no media storage configured")
}
p, err = t.mediaLoader.LoadPath(ref.ID)
if err != nil {
return "", "", fmt.Errorf("document file not found: %v", err)
}
loadedLegacy = true
}
if loadedLegacy {
p, err = resolveLoadedMediaRefPath(ctx, t.mediaLoader, p, "document")
} else {
p, err = resolveStructuredMediaRefPath(ctx, p, "document")
}
if err != nil {
return "", "", err
}
// Determine MIME type: prefer ref's stored MIME, fall back to extension.
mime = ref.MimeType
if mime == "" || mime == "application/octet-stream" {
mime = mimeFromDocExt(filepath.Ext(p))
}
return p, mime, nil
}
func resolveDocumentPathArg(ctx context.Context, path string) (string, error) {
return resolveStructuredMediaPath(ctx, path, "document")
}
func isArchiveDocumentPath(path string) bool {
lower := strings.ToLower(path)
switch {
case strings.HasSuffix(lower, ".zip"),
strings.HasSuffix(lower, ".tar"),
strings.HasSuffix(lower, ".tar.gz"),
strings.HasSuffix(lower, ".tgz"),
strings.HasSuffix(lower, ".tar.bz2"),
strings.HasSuffix(lower, ".tbz2"),
strings.HasSuffix(lower, ".tar.xz"),
strings.HasSuffix(lower, ".txz"),
strings.HasSuffix(lower, ".gz"):
return true
default:
return false
}
}
// callProvider dispatches document analysis to the appropriate provider API.
// For Gemini: uses native generateContent API (supports PDF natively).
// For others: uses standard Chat API with base64 document.
func (t *ReadDocumentTool) callProvider(ctx context.Context, cp credentialProvider, providerName, model string, params map[string]any) ([]byte, *providers.Usage, error) {
prompt := GetParamString(params, "prompt", "Analyze this document and describe its contents.")
data, _ := params["data"].([]byte)
mime := GetParamString(params, "mime", "application/octet-stream")
docPath, _ := params[documentLocalPathParam].(string)
// Gemini: use native API (requires credentials; OpenAI-compat endpoint doesn't support non-image MIME types).
ptype := GetParamString(params, "_provider_type", providerTypeFromName(providerName))
if cp != nil && ptype == "gemini" {
slog.Info("read_document: using gemini native API",
"provider", providerName, "model", model,
"doc_size", len(data), "mime", mime)
chatReq := providers.ChatRequest{
Messages: []providers.Message{{
Role: "user",
Content: prompt,
Images: []providers.ImageContent{{MimeType: mime}},
}},
Model: model,
Options: map[string]any{"max_tokens": 16384},
}
reservation, reserveErr := reserveToolLLMUsageWithMedia(ctx, t.usageCaps, t.Name(), providerName, model, chatReq, mediabudget.Payload{Kind: mediabudget.KindDocument, MIME: mime, Size: int64(len(data)), Path: docPath, Head: data})
if reserveErr != nil {
return nil, nil, reserveErr
}
resp, err := geminiNativeDocumentCall(ctx, cp.APIKey(), model, prompt, data, mime)
if reservation != nil {
reservation.Reconcile(ctx, resp, err)
}
if err != nil {
return nil, nil, fmt.Errorf("gemini native call: %w", err)
}
return []byte(resp.Content), resp.Usage, nil
}
// Other providers: use standard Chat API with document as base64 image_url.
p, err := t.registry.Get(ctx, providerName)
if err != nil {
return nil, nil, fmt.Errorf("provider %q not available: %w", providerName, err)
}
slog.Info("read_document: using chat API", "provider", providerName, "model", model, "doc_size", len(data))
opts := map[string]any{
"max_tokens": 16384,
"temperature": 0.2,
}
// Scope disable_tools to claude-cli only — it's a CLI-bridge-specific
// option that skips loading the built-in MCP toolset for one-shot calls.
// Other providers silently ignore unknown keys today, but leaking
// provider-specific flags into the shared Options map couples layers.
if providerName == "claude-cli" {
opts["disable_tools"] = true
}
chatReq := providers.ChatRequest{
Messages: []providers.Message{
{
Role: "user",
Content: prompt,
Images: []providers.ImageContent{{MimeType: mime, Data: base64.StdEncoding.EncodeToString(data)}},
},
},
Model: model,
Options: opts,
}
// Charged on the same terms as the Gemini branch: the base64 document
// rides in a content part the token counter does not price, so an
// unpriced branch here is a way round the gate the Gemini branch applies.
payload := mediabudget.Payload{Kind: mediabudget.KindDocument, MIME: mime, Size: int64(len(data)), Path: docPath, Head: data}
reservation, reserveErr := reserveToolLLMUsageWithMedia(ctx, t.usageCaps, t.Name(), providerName, model, chatReq, payload)
if reserveErr != nil {
return nil, nil, reserveErr
}
resp, err := p.Chat(ctx, chatReq)
if reservation != nil {
reservation.Reconcile(ctx, resp, err)
}
if err != nil {
return nil, nil, fmt.Errorf("chat call: %w", err)
}
return []byte(resp.Content), resp.Usage, nil
}
// mimeFromDocExt returns MIME type for document file extensions.
func mimeFromDocExt(ext string) string {
switch strings.ToLower(ext) {
case ".pdf":
return "application/pdf"
case ".doc", ".docx":
return "application/vnd.openxmlformats-officedocument.wordprocessingml.document"
case ".xls", ".xlsx":
return "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
case ".ppt", ".pptx":
return "application/vnd.openxmlformats-officedocument.presentationml.presentation"
case ".csv":
return "text/csv"
case ".zip":
return "application/zip"
case ".tar":
return "application/x-tar"
case ".gz", ".tgz":
return "application/gzip"
default:
return "application/octet-stream"
}
}