feat(tools): local-first document extraction for read_document

Squash merge PR #111 after resolving docs/changelog conflicts. Local checks covered tools/config and both Go builds; PR CI run 26704622503 passed release-versioning, go, and web.

docs/03-tools-system.md

@@ -167,7 +167,7 @@ Memory layers: L1 (`memory_search`) returns ranked abstracts; L2 (`memory_expand
 |---|---|
 | `read_image` | Analyze/describe images using vision AI (Gemini, Anthropic, OpenRouter, DashScope) |
 | `read_audio` | Transcribe audio to text using Gemini File API, native OpenAI audio input, or OpenAI-compatible transcription models; unsupported provider/model routes fail closed instead of sending audio as image data |
-| `read_document` | Extract and analyze documents (PDF, images) via Gemini or Resolve service |
+| `read_document` | Extract and analyze documents (PDF, DOCX, images). Supports local-first extraction via pdftotext/pandoc before falling back to cloud vision (opt-in via config) |
 | `read_video` | Analyze/transcribe video content |
 
 ### Skills & Content
@@ -217,6 +217,23 @@ User-facing parameter schemas for the most commonly configured tools.
 - `primary`: provider (`elevenlabs`, `openai`, `edge`, `minimax`).
 - Per-agent overrides: `agent.other_config.tts_voice_id`, `agent.other_config.tts_model_id`.
 
+### `document_parser` config shape
+```json
+{
+  "local_first": false,
+  "max_pages": 200,
+  "timeout_sec": 30,
+  "min_text_len": 16
+}
+```
+Controls local-first document text extraction in the `read_document` tool.
+- `local_first`: Enable local extraction via `pdftotext` (PDF) and `pandoc` (DOCX) before cloud vision fallback (default `false` — opt-in). Requires binaries on PATH; present in `full` Docker variant or builds with `ENABLE_FULL_SKILLS=true`.
+- `max_pages`: Page limit for PDF extraction (default 200). Passed to `pdftotext -l`.
+- `timeout_sec`: Per-extraction timeout in seconds (default 30). Process group killed on timeout.
+- `min_text_len`: Minimum characters (after trim) to consider extraction successful; shorter output triggers cloud fallback (default 16).
+
+**Note:** Config values are captured at tool construction (startup) and not picked up by hot-reload. Binary availability is re-checked per call, so runtime binary installations are detected without restart. Any extraction miss (disabled, unsupported mime, missing binary, timeout, empty output) transparently falls back to the cloud vision chain with no caller-visible difference.
+
 ### Custom tool definition
 ```json
 {

docs/journals/260529-local-first-document-extraction.md

@@ -0,0 +1,86 @@
+# Local-First Document Text Extraction (PDF/DOCX)
+
+**Date**: 2026-05-29 17:26
+**Severity**: Medium
+**Component**: read_document tool / DocumentParser
+**Status**: Shipped
+
+## What Happened
+
+Delivered local-first text extraction adapter for `read_document` tool. PDF/DOCX files now optionally extract text via `pdftotext` (poppler) / `pandoc` (DOCX sandbox mode) *before* falling back to cloud vision LLM chain. Trades off extraction quality (plaintext loses layout, tables, multi-column semantics) for privacy + token cost savings on text-rich documents.
+
+Opt-in by default OFF — users must explicitly set `document_parser.local_first: true` in config. This deliberate choice avoids shipping a regression: scanned PDFs + image-only documents yield empty local output, forcing cloud fallback anyway, wasting latency on a guaranteed miss.
+
+## The Brutal Truth
+
+This feature feels overdue and also fragile simultaneously. On one hand, it's a no-brainer privacy win for local deployments — why burn Gemini's vision context on a 100-page plaintext PDF when you can `pdftotext` it in 50ms locally? On the other, plaintext extraction fundamentally cannot preserve document structure. Users who expect formatted table output from local parsing *will* be disappointed; they need the vision chain.
+
+The default-OFF toggle was the pragmatic choice: let power users opt in, don't punish document quality for the majority. But that also means the feature ships with zero real-world usage data. We'll learn on production deployments whether the latency win justifies the extraction quality gap.
+
+## Technical Details
+
+**New interface:** `DocumentParser` in `internal/tools/document_parser.go` — two methods:
+- `Extract(ctx, path, mime) (string, error)` — returns plaintext or sentinel error (`ErrParserUnsupported`, `ErrParserUnavailable`, `ErrParserEmpty`)
+- `Supports(mime) bool` — runtime check: config enabled + binary resolvable via LookPath
+
+**LocalExtractParser implementation:**
+
+1. **No-shell subprocess** (`exec.Command` not `sh -c`) — direct argument array, never string concat.
+2. **Process-group kill on timeout** — `SIGTERM` → 3s grace → `SIGKILL` via `setProcessGroup/killProcessGroup`. Pandoc forks helper processes; killing the group reaps grandchildren. NOT `exec.CommandContext` (simpler, but would only kill direct child, orphaning helpers).
+3. **Minimal subprocess env** — `buildCredentialedEnv(nil)` captures PATH/HOME/LANG/USER only. Never inherit gateway env (holds DB DSNs, API keys, encryption keys). Same posture as exec tool.
+4. **Bounded stdout** — `cappedWriter` buffers up to 500KB (`documentMaxTextBytes`), silently discards overflow, always reports full write so child never blocks on pipe. Appends truncation marker matching the existing direct-text truncation behavior.
+5. **Safe binary args:**
+   - `pdftotext` omits `--` terminator (poppler's parser doesn't treat it as option-end; would consume it as positional, clobbering the real path).
+   - `pandoc --sandbox` for untrusted DOCX (prevents fetch/read-arbitrary-files during conversion).
+6. **Extracted text = untrusted** — never changes trust model. Inline with existing direct-text fast-path; no new injection vector.
+
+**Integration into read_document:**
+
+Local-first block sits *after* existing guards (archive check, text-readable fast-path, 20MB read guard) *before* vision chain. Path validated at exec boundary (`validateExecPath` reuses `resolvePathWithAllowed` + `checkDeniedPath` so MediaRef-derived paths get same workspace confinement check).
+
+Any miss (disabled / unsupported mime / missing binary / empty output / exec error / timeout) falls through silently to vision chain — never errors. Clean hit returns plaintext with **zero Provider/Model/Usage** (no LLM spend).
+
+## What We Tried
+
+N/A — new feature, no prior attempts.
+
+## Root Cause Analysis
+
+Motivation: issue #84 (token cost + privacy on local deployments with text-heavy PDFs). Vision LLM chains waste 250+ tokens per document page for purely textual content; local extraction avoids that spend + keeps data off-cloud for privacy-sensitive deployments.
+
+Decision to default OFF stems from observing plaintext extraction limitations: scanned PDFs (image-only) + multi-column layouts + formatted tables all lose semantic information when extracted as flat text. Forcing this on all users regresses experience. Opt-in preserves quality for majority, enables cost savings for explicit users.
+
+## Lessons Learned
+
+1. **Subprocess timeout strategy matters.** Process-group kill is cleanest for extractors that fork helpers. CommandContext would orphan grandchildren. In production, timeouts are less about hung processes (modern tools timeout internally) and more about defense-in-depth; document a timeout will never be hit in practice, but it saves from catastrophic hangs.
+
+2. **Untrusted input + subprocess = attack surface.** Spent care on: minimal env (no secrets), no-shell exec (no globbing), `pandoc --sandbox` (no file I/O), bounded output (no OOM), path validation at exec boundary (no traversal). These are defensive layers; a single layer's failure shouldn't cascade. Validation + env isolation are the hardest wins.
+
+3. **Opt-in vs. opt-out cuts both ways.** Default-OFF means the feature ships with zero usage data. Production will tell us if local extraction is enough for the use cases pushing for it (and if not, whether vision fallback makes it moot). Default-ON would have been faster to adoption but would have regressed document quality perception immediately. Chose sustainable over frictionless.
+
+4. **Truncation markers must match.** Both direct-text and local extraction append the same `[... truncated at 500KB ...]` marker. Downstream (agent, logs) sees identical output shape. Small detail, huge payoff: no code changes needed to handle truncation; it just works.
+
+5. **Testing on stubbed binaries is essential.** Tests mock `LookPath` so `pdftotext`/`pandoc` don't need to be installed to verify logic (timeout kill, capped write, path validation). Leaves open the question of real binary output quality, but that's expected to vary by tool version.
+
+## Next Steps
+
+1. **Monitoring:** Instrument extraction hits vs. misses in production (`slog.Info` already in place). Monitor fallback rate to vision chain — if >90% miss, feature is unused; if <10% miss, local extraction is carrying load.
+
+2. **Real-world feedback:** Opt-in means early power users will report quality issues or token savings. Collect feedback on whether plaintext extraction is good enough for their PDFs. May need advanced config (e.g., OCR pipeline for scanned docs, pandoc output format tuning).
+
+3. **Binary availability docs:** Desktop Lite (`sqliteonly` build) doesn't include `pdftotext`/`pandoc`. Clarify that local-first requires full server variant (or binaries installed at runtime). Update install guides.
+
+4. **Potential enhancements (deferred):**
+   - OCR pipeline for scanned PDFs (tesseract) — adds latency, changes computation model
+   - Fallback to LLM-only if local extraction is disabled (not yet exposed as easy toggle)
+   - Per-document extraction strategy (hint: "scanned" vs. "text" → choose pipeline)
+
+**Owned by:** Tools team.
+**Deadline:** Monitor week 1 post-ship for adoption metrics.
+
+---
+
+**Status:** SHIPPED
+**Test result:** 13 unit + integration tests pass (9 parser + 4 tool), 581 tools + 48 config tests pass, race-clean
+**Code review:** 5/5 acceptance criteria + 10/10 security constraints verified
+**Files changed:** 8 files, 686 insertions; new `document_parser.go` (222 LOC), test files, config update

docs/project-changelog.md

@@ -6,6 +6,56 @@ Significant changes, features, and fixes in reverse chronological order.
 
 ## 2026-05-29
 
+### Local-first document text extraction adapter (read_document privacy optimization)
+
+Adds optional local extraction pipeline to the `read_document` tool, allowing
+PDF and DOCX parsing via `pdftotext` (poppler-utils) and `pandoc` before
+falling back to cloud vision. Reduces token costs and improves privacy for
+document analysis when local binaries are available.
+
+**New**
+
+- `DocumentParser` interface + `LocalExtractParser` implementation in
+  `internal/tools/document_parser.go`. Handles PDF (`pdftotext`) and DOCX
+  (`pandoc`) with no-shell subprocess exec, binary detection, process-group
+  timeout kill (SIGTERM → 3s grace → SIGKILL), minimal subprocess environment,
+  and 500KB output limit.
+- `DocumentParserConfig` in `internal/config/config_channels.go` under `tools` section:
+  `local_first` (opt-in, default false), `max_pages` (default 200), `timeout_sec`
+  (default 30), `min_text_len` (default 16). Config is read at tool construction;
+  binary availability re-checked per call for runtime installs.
+- Integration in `read_document` tool: `SetLocalParser()` wired at gateway
+  startup (`cmd/gateway_managed.go`). On a clean extraction hit, returns text
+  with no LLM usage (zero tokens, no Provider/Model/Usage fields). Any miss
+  (disabled, unsupported mime, missing binary, timeout, empty output, exec
+  error) transparently falls back to the cloud vision chain unchanged.
+- Security: subprocess exec uses `exec.Command` (no shell), docPath validated
+  at the exec boundary, extracted text treated as untrusted, minimal env
+  (no gateway secrets), `pandoc --sandbox` for DOCX parsing.
+
+**Requirements**
+
+- Requires `pdftotext` and `pandoc` on PATH for local extraction. Present in:
+  - Docker `full` variant (`ghcr.io/nextlevelbuilder/goclaw:vX.Y.Z-full`)
+  - Builds with `-tags "" -X main.enableFullSkills=true` (when available)
+- Desktop Lite edition: local extraction wired but not user-configurable
+  (remains in code for future mobile/CLI usage; server/Docker only now).
+
+**Docs**
+
+- `docs/03-tools-system.md` § 5 — new `document_parser` config block with all
+  four fields, defaults, and fallback semantics.
+- `docs/03-tools-system.md` § 4 (Media Reading) — updated `read_document`
+  description to mention local-first extraction option.
+
+**Testing**
+
+- Behavior unchanged when `local_first: false` (default).
+- Verify local extraction when enabled: create test PDF/DOCX, enable config,
+  inspect tool output for zero-token response vs cloud vision (Gemini, etc.).
+
+---
+
 ### Skill selected download export (issue #80)
 
 **Features**