Merge remote-tracking branch 'origin/dev' into codex/issue-48-google-workspace-cli

# Conflicts: # docs/project-changelog.md
2026-10-13 16:14:57 +00:00 · 2026-05-24 18:57:47 +07:00
commit 3a749ff91c
@@ -54,6 +54,29 @@ Streaming fallback is conservative: backup models are tried only if the stream f

 ---

+## Usage Cap Pricing Enforcement
+
+Standard edition can enforce AI budget caps before billable provider dispatch. API-key providers use OpenRouter `/models` pricing as the catalog source, with optional tenant/provider/model overrides in the dashboard.
+
+Excluded provider classes:
+- `chatgpt_oauth`, `claude_cli`, and `bailian` are treated as subscription/non-API pricing in round one.
+- local/no-key subprocess providers such as `acp` and `ollama` are skipped unless a future feature explicitly enables pricing for them.
+
+Runtime flow:
+1. Resolve the stored provider by name and skip non-billable provider classes.
+2. Load matching policies for tenant, agent, provider, provider type, and model.
+3. Resolve custom pricing override first, then OpenRouter catalog pricing when a matching policy has a cost ceiling. Native provider model IDs are mapped to OpenRouter prefixes for common providers such as OpenAI, Anthropic, and Gemini.
+4. Reserve estimated tokens and cost atomically before each dispatch attempt.
+5. Reconcile reserved counters after the provider returns usage or after a failed call.
+
+Token-only policies do not require catalog pricing. Model fallback routes reserve against the actual candidate provider/model before each attempt. Cached input is separated from uncached input for OpenAI-compatible usage accounting. Partial stream failures keep the estimate, or actual provider usage when available, instead of clearing billed output to zero. Internal LLM calls for memory flush, compaction, media reading tools (`read_image`, `read_document`, `read_audio`, `read_video`), and subagents use the same preflight/reconcile path.
+
+The legacy agent-level `budget_monthly_cents` field is treated as a generated monthly agent USD cap. Existing values are backfilled during migration, and later agent budget edits update or remove the generated cap policy.
+
+Supported price units: input, output, cache read, cache write, reasoning, request, image, and web search.
+
+---
+
 ## 2. Supported Providers

 ### Six Core Provider Types
@@ -246,6 +246,32 @@ User-facing parameter schemas for the most commonly configured tools.
 ```
 Available presets: `gh`, `gcloud`, `aws`, `kubectl`, `terraform`.

+### Credentialed CLI keyword allowlist
+
+`config.tools.commandKeywordAllowlist` lets operators allow specific product or security vocabulary inside selected credentialed CLI content arguments without disabling `deny_args`.
+
+Example:
+
+```json
+{
+  "tools": {
+    "commandKeywordAllowlist": [
+      {
+        "id": "github-content",
+        "command": "gh",
+        "subcommands": ["issue create", "issue edit", "pr create", "pr comment"],
+        "args": ["--body", "--title"],
+        "argPositions": [],
+        "keywords": ["secret", "secrets", "token", "credential"],
+        "reason": "Allow security vocabulary in GitHub issue and PR prose"
+      }
+    ]
+  }
+}
+```
+
+The rule above allows `gh issue create --body "secret rotation notes"` but still blocks command paths like `gh secret set TOKEN`. `argPositions` are 0-based after the matched subcommand. The scanner evaluates command arguments only; it does not read the contents of files passed through arguments such as `--body-file`.
+
 ---

 ## 6. Interception Layer
@@ -346,6 +372,13 @@ Custom tools are shell-based tools defined at runtime via the HTTP API — no re
 | `env` | no | Encrypted environment variables injected at runtime |
 | `enabled` | no | Toggle without deleting (default true) |

+Credentialed CLI env entries support two API/UI kinds:
+
+- `sensitive` (default): encrypted at rest, masked in normal API responses, replace-only in UI, and flattened only at credential injection time.
+- `value`: encrypted at rest but visible to authorized admins in API/UI for non-secret settings such as public URLs, domains, limits, regions, and feature flags.
+
+Legacy env JSON like `{"TOKEN":"..."}` is still accepted and treated as `sensitive`.
+
 **Execution:** Template placeholders are rendered with shell-escaped argument values, then run via `sh -c`. The same deny-pattern check as the `exec` tool applies — no reverse shells, no `curl | sh`, etc.

 **Scope:**
@@ -406,7 +439,7 @@ Current adopters: `web_search`, `web_fetch`, `tts`, `create_image`, `read_image`
 - Per-key: agent value takes precedence over global value
 - Multi-tenant invariant: each tenant's config is isolated

-**Live reload:** Changes to `config.tools.shellDenyGroups` propagate via `bus.TopicConfigChanged` pub/sub. Next agent turn automatically applies new toggles.
+**Live reload:** Changes to `config.tools.shellDenyGroups` and `config.tools.commandKeywordAllowlist` propagate via `bus.TopicConfigChanged` pub/sub. Next agent turn automatically applies new toggles.

 **Deny-group classes** (from `internal/tools/shell_deny_groups.go` — all denied by default):

@@ -442,7 +442,7 @@ All CRUD endpoints require `Authorization: Bearer <token>` and `X-GoClaw-User-Id
 | Method | Path | Description |
 |--------|------|-------------|
 | GET | `/v1/skills` | List skills |
-| POST | `/v1/skills/upload` | Upload skill ZIP (max 20 MB) |
+| POST | `/v1/skills/upload` | Upload skill ZIP (configurable, default 20 MB, max 500 MB) |
 | DELETE | `/v1/skills/{id}` | Delete a skill |

 **Traces** (`/v1/traces`):
@@ -45,6 +45,7 @@ The `Stores` struct is the top-level container holding all PostgreSQL-backed sto
 | ContactStore | `PGContactStore` | Channel contacts (auto-collected), cross-channel deduplication, merge |
 | ActivityStore | `PGActivityStore` | Audit logs, action tracking, compliance |
 | SnapshotStore | `PGSnapshotStore` | Hourly usage snapshots, cost aggregation, time series queries |
+| UsageCapStore | `PGUsageCapStore` | OpenRouter pricing catalog, pricing overrides, cap policies, reservations, counters, events |
 | SecureCLIStore | `PGSecureCLIStore` | CLI binary configs with encrypted credential injection |
 | APIKeyStore | `PGAPIKeyStore` | Gateway API keys, scopes, expiration, revocation |
 | HookStore | `PGHookStore` | Lifecycle hook definitions (event, handler type, matcher, config), execution audit log |
@@ -85,6 +86,33 @@ Migration versions:

 ---

+## Usage Cap Storage
+
+Usage cap enforcement is Standard/PostgreSQL-only in round one. The `UsageCapStore` is wired on the PostgreSQL store factory and left nil in SQLite/Lite builds.
+
+Tables:
+- `usage_pricing_catalog`: OpenRouter model catalog prices, raw upstream model payload, sync time.
+- `usage_pricing_overrides`: tenant/provider/model override prices for custom billing assumptions.
+- `usage_cap_policies`: cap definitions scoped by tenant, agent, provider, provider type, model, `window_key`, and `source`.
+- `usage_cap_counters`: current window used and reserved token/cost counters.
+- `usage_cap_reservations`: preflight reservations keyed by LLM call attempt.
+- `usage_cap_events`: allow/block/reconcile/skip audit events.
+
+Reservation updates are atomic: counters are updated only when `used + reserved + estimate` remains below configured token and cost ceilings.
+Reservation keys are idempotent per policy, so a retry using the same key does not double-increment reserved counters.
+Policy `agent_id` references must belong to the same tenant as the policy. Policy and pricing override `provider_id` references may belong to the same tenant or the master tenant for default provider fallback, but not another non-master tenant.
+Catalog and override price fields are nullable decimal strings with non-negative validation in the store layer and database checks.
+Pricing resolution checks exact override/catalog model IDs first, then provider-derived OpenRouter aliases for native unprefixed model IDs.
+
+Agent `budget_monthly_cents` values are bridged into `usage_cap_policies` with `source = 'agent_budget_monthly_cents'`, an agent scope, `window_key = 'month'`, and `max_cost_micros = budget_monthly_cents * 10000`. Updating or clearing the agent budget keeps that generated policy in sync; manual cap policies continue to use `source = 'manual'`.
+
+Migration versions:
+
+- PostgreSQL: `000070_usage_caps_pricing`, `000071_usage_cap_policies`, `000072_agent_budget_usage_cap_bridge`.
+- SQLite: no schema change; feature is not active in Lite.
+
+---
+
 ## 3. Session Caching

 The session store uses an in-memory write-behind cache to minimize database I/O during the agent tool loop. All reads and writes happen in memory; data is flushed to the persistent backend only when `Save()` is called at the end of a run.
@@ -279,6 +279,30 @@ This decision is re-evaluated each time the system prompt is built, so newly hot

 ---

+## 9.5. Explicit Slash Skill Commands
+
+Users can bypass implicit skill matching by starting a prompt with a slash command:
+
+| Pattern | Behavior |
+|---------|----------|
+| `/<slug> prompt` | Activates the skill by slug and treats `prompt` as the skill input |
+| `/use <slug-or-name> prompt` | Activates the skill by slug or display name |
+| `/list-skills` | Shows available skills for the current agent context |
+| `/help <slug-or-name>` | Shows description and usage guidance for one skill |
+
+Slash detection runs during prompt construction after request context is scoped and before the skills section is built. A matched skill narrows the per-request `SkillFilter` to that skill and injects the full `SKILL.md` instructions into the system prompt for the current turn only. Normal matching remains unchanged for messages that do not start with the configured prefix, path-like strings such as `/home/user/file`, or unresolved commands without suggestions.
+
+Tenant settings live in `system_configs`:
+
+| Key | Default | Behavior |
+|-----|---------|----------|
+| `skills.slash_commands.enabled` | `true` | Enable slash command detection |
+| `skills.slash_commands.suggest_not_found` | `true` | Suggest similar skills for unknown commands |
+| `skills.slash_commands.partial_matching` | `false` | Allow unique prefixes such as `/frontend` |
+| `skills.slash_commands.prefix` | `/` | Single-character command prefix |
+
+---
+
 ## 10. Skills -- BM25 Search

 An in-memory BM25 index provides keyword-based skill search. The index is lazily rebuilt whenever the skill version changes.
@@ -221,11 +221,14 @@ AES-256-GCM encryption for secrets stored in PostgreSQL. Key provided via `GOCLA
 | LLM provider API keys | `llm_providers` | `api_key` |
 | MCP server API keys | `mcp_servers` | `api_key` |
 | Custom tool env vars | `custom_tools` | `env` |
+| Credentialed CLI env vars | `secure_cli_binaries`, `secure_cli_agent_grants`, `secure_cli_user_credentials` | `encrypted_env` |

 **Format**: `"aes-gcm:" + base64(12-byte nonce + ciphertext + GCM tag)`

 Backward compatible: values without the `aes-gcm:` prefix are returned as plaintext (for migration from unencrypted data).

+Credentialed CLI env entries have a separate visibility kind inside the encrypted JSON blob when `GOCLAW_ENCRYPTION_KEY` is configured. `sensitive` entries are masked in normal API/UI responses and never returned raw except through the explicit audited grant reveal flow. `value` entries use the same at-rest storage path but are returned to authorized admins for operational review.
+
 ---

 ## 4. Rate Limiting -- Gateway + Tool
@@ -29,6 +29,11 @@ How skills access Python, Node.js, and system tools inside Docker containers and
 └─────────────────────────────────────────────────────────┘
 ```

+Explicit skill activation is handled before runtime execution. When a user starts
+their prompt with `/<skill-slug>` or `/use <skill name>`, the gateway resolves
+the skill, injects its `SKILL.md` into the current turn, and then normal runtime
+rules apply to any scripts or package dependencies that skill uses.
+
 ---

 ## 2. Pre-installed Packages (Option A)
@@ -139,7 +144,7 @@ To install additional packages: pip3 install <pkg> or npm install -g <pkg>

 - Run Python/Node scripts via exec tool
 - Install packages via `pip3 install` / `npm install -g`
- Access files in `/app/workspace/` including `.media/` subdirectory
+- Access files in `/app/workspace/`, including `.uploads/` for current user uploads and `.media/` for legacy media refs
 - Read skill files from `.goclaw/skills-store/`

 ### What Agents CANNOT Do
@@ -156,16 +161,18 @@ To install additional packages: pip3 install <pkg> or npm install -g <pkg>
 Uploaded files (from web chat, Telegram, Discord, etc.) are persisted to:

 ```
-/app/workspace/.media/{sessionHash}/{uuid}.{ext}
+/app/workspace/.uploads/{safe-original-name}-{8hex}.{ext}
 ```

+Uploads without a usable original filename fall back to `{uuid}.{ext}`. Legacy media refs may still resolve from `.media/{sessionHash}/{uuid}.{ext}`.
+
 The `enrichDocumentPaths()` function injects the full path into `<media:document>` tags:

 ```
-<media:document name="report.pdf" path="/app/workspace/.media/abc123/uuid.pdf">
+<media:document name="report.pdf" path="/app/workspace/.uploads/report-a1b2c3d4.pdf">
 ```

-Agents can read these files directly via exec — no copy to `/tmp` needed.
+Agents can read these files directly via exec — no copy to `/tmp` needed. For archive uploads such as `.zip`, inspect or extract with commands like `unzip -l "<path>"` or `unzip -q "<path>" -d <output-dir>`.

 ---

@@ -183,7 +183,7 @@ Unlike the HTTP upload handler, the tool does **not** archive the skill on missi
 | `..` in relative path | Skip (prevent traversal) |
 | Symlinks | Skip (prevent escape) |
 | System artifacts | Skip (`.DS_Store`, `__MACOSX`, `Thumbs.db`, etc.) |
-| Total dir size > 20 MB | Reject with error |
+| Total upload size exceeds configured limit | Reject with error. Default is 20 MB; configurable via `skills.max_upload_size_mb`, `GOCLAW_SKILLS_MAX_UPLOAD_SIZE_MB`, tenant `skills.max_upload_size_mb`, or SKILL.md frontmatter `max_upload_size_mb`; clamped to 1-500 MB. |

 ---

@@ -321,7 +321,7 @@ Use `direct_selection_count` plus the `selected_provider` sequence to verify rea
 | Method | Path | Description |
 |--------|------|-------------|
 | `GET` | `/v1/skills` | List all skills |
-| `POST` | `/v1/skills/upload` | Upload ZIP with SKILL.md (20 MB limit) |
+| `POST` | `/v1/skills/upload` | Upload ZIP with SKILL.md (configurable 20 MB default, 1-500 MB range) |
 | `GET` | `/v1/skills/{id}` | Get skill details |
 | `PUT` | `/v1/skills/{id}` | Update skill metadata |
 | `DELETE` | `/v1/skills/{id}` | Delete skill (not system skills) |
@@ -331,6 +331,18 @@ Use `direct_selection_count` plus the `selected_provider` sequence to verify rea

 ### Skill Grants

+Skill upload size is enforced per ZIP file. The effective limit resolves in this order:
+tenant `system_configs["skills.max_upload_size_mb"]`, then `SKILL.md` frontmatter
+`max_upload_size_mb`, then config/env `skills.max_upload_size_mb` /
+`GOCLAW_SKILLS_MAX_UPLOAD_SIZE_MB`, then the default 20 MB. Values are clamped
+to 1-500 MB.
+
+Skill slash-command behavior is configured through tenant `system_configs`:
+`skills.slash_commands.enabled`, `skills.slash_commands.suggest_not_found`,
+`skills.slash_commands.partial_matching`, and `skills.slash_commands.prefix`.
+The default prefix is `/`; supported prompt forms are `/<slug> prompt`,
+`/use <slug-or-name> prompt`, `/list-skills`, and `/help <slug-or-name>`.
+
 | Method | Path | Description |
 |--------|------|-------------|
 | `POST` | `/v1/skills/{id}/grants/agent` | Grant skill to agent |
@@ -390,6 +402,36 @@ LLM provider management. API keys are encrypted with AES-256-GCM in the database
 | `GET` | `/v1/embedding/status` | Check global embedding availability |
 | `GET` | `/v1/providers/claude-cli/auth-status` | Check Claude CLI login status |

+### Model Pricing
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `POST` | `/v1/model-pricing/sync-openrouter` | Sync OpenRouter `/models` pricing catalog (master scope) |
+| `GET` | `/v1/model-pricing` | Search catalog models by `model` query |
+| `GET` | `/v1/model-pricing/overrides` | List tenant pricing overrides, optionally by `provider_id` |
+| `PUT` | `/v1/model-pricing/overrides` | Upsert provider/model custom pricing |
+| `DELETE` | `/v1/model-pricing/overrides/{id}` | Delete pricing override |
+
+Override body:
+
+```json
+{
+  "provider_id": "0193a5b0-7000-7000-8000-000000000123",
+  "provider_type": "openrouter",
+  "model_id": "anthropic/claude-sonnet-4-5",
+  "pricing": {
+    "input": "0.000003",
+    "output": "0.000015",
+    "cache_read": "0.0000003",
+    "cache_write": "0.00000375",
+    "reasoning": "0.000015",
+    "request": "0",
+    "image": "0",
+    "web_search": "0"
+  }
+}
+```
+
 **Supported types:** `anthropic_native`, `openai_compat`, `chatgpt_oauth`, `gemini_native`, `dashscope`, `bailian`, `minimax`, `claude_cli`, `acp`

 Reconnect response:
@@ -1271,11 +1313,34 @@ Follow response:
 | `GET` | `/v1/usage/timeseries` | Time-series usage points |
 | `GET` | `/v1/usage/breakdown` | Breakdown by provider/model/channel |
 | `GET` | `/v1/usage/summary` | Summary with period comparison |
+| `GET` | `/v1/usage-caps/policies` | List usage cap policies |
+| `POST` | `/v1/usage-caps/policies` | Create token/cost cap policy |
+| `PATCH` | `/v1/usage-caps/policies/{id}` | Update cap policy |
+| `DELETE` | `/v1/usage-caps/policies/{id}` | Delete cap policy |
+| `GET` | `/v1/usage-caps/utilization` | Current-window used and reserved counters |
+| `GET` | `/v1/usage-caps/events` | Recent allow/block/reconcile/skip events |

 **Query params:** `from`, `to` (RFC 3339), `agent_id`, `provider`, `model`, `channel`, `group_by`

 **Periods:** `24h`, `today`, `7d`, `30d`

+Usage cap policy body:
+
+```json
+{
+  "agent_id": "optional-agent-uuid",
+  "provider_id": "optional-provider-uuid",
+  "provider_type": "openrouter",
+  "model_id": "anthropic/claude-sonnet-4-5",
+  "window": "day",
+  "max_tokens": 500000,
+  "max_cost_usd": 25,
+  "enabled": true
+}
+```
+
+Policy responses include read-only `source`. `source="agent_budget_monthly_cents"` means the policy is generated from the agent monthly budget field and must be changed there; direct policy updates/deletes return `409 Conflict`.
+
 ---

 ## 24. Activity & Audit
@@ -450,7 +450,7 @@ System skills (`is_system=true`) cannot be modified through any path.
 | Symlink detection | `filepath.WalkDir` + `d.Type()&os.ModeSymlink` check |
 | Path traversal | `strings.Contains(rel, "..")` rejection |
 | Content size limit | 100KB max for SKILL.md content |
-| Companion size limit | 20MB max total for companion files (scripts, assets) |
+| Companion size limit | Configurable per ZIP upload; default 20MB, clamped to 1-500MB |
 | Soft-delete | Files moved to `.trash/`, never hard-deleted |

 ---
@@ -18,6 +18,60 @@ Significant changes, features, and fixes in reverse chronological order.

 - Added runtime binary discovery, SecureCLI preset, deny-pattern, and Dockerfile contract coverage for Google Workspace CLI.

+### Slash skill commands
+
+**Features**
+
+- Added explicit slash skill activation for `/<slug>`, `/use <slug-or-name>`, `/list-skills`, and `/help <slug-or-name>`.
+- Added tenant settings for slash command enablement, similar-skill suggestions, partial matching, and custom prefix.
+
+**Tests**
+
+- Added backend coverage for parser false positives, exact/partial skill resolution, suggestions, help/list commands, and config overlays.
+
+### Configurable skill upload limits
+
+**Features**
+
+- Added configurable skill ZIP upload limits with config/env, SKILL.md frontmatter, and tenant system setting support.
+- Added dashboard settings and dynamic upload validation so the Web UI follows the tenant limit instead of hardcoding 20MB.
+
+**Tests**
+
+- Added backend coverage for limit precedence, clamping, oversized rejection, and frontend coverage for parameterized upload validation.
+
+### CLI environment variable visibility
+
+**Features**
+
+- Added `sensitive` and `value` kinds for secure CLI environment variables across binary defaults, agent grant overrides, and user overrides.
+- Plain value entries are visible to authorized admins for operational config review, while sensitive entries remain masked and replace-only.
+
+**Fixes**
+
+- Stopped per-user credential reads from returning legacy sensitive env values raw.
+- Kept legacy `{"KEY":"value"}` env blobs backward-compatible by treating them as sensitive.
+
+**Tests**
+
+- Added backend regression coverage for env kind parsing, sanitized API responses, runtime flattening, and invalid kind rejection.
+- Verified Web UI build after adding env-kind controls and warnings.
+
+### Command keyword allowlist
+
+**Features**
+
+- Added scoped credentialed CLI keyword allowlist config for content arguments and positional arguments, with runtime reload and web config editing.
+
+**Fixes**
+
+- Kept credentialed CLI `deny_args` active for command paths such as `gh secret set` while allowing approved GitHub issue/PR prose to mention security vocabulary.
+- Added security audit logging for real allowlisted pass-throughs without logging full argument values.
+
+**Tests**
+
+- Added regression coverage for scoped keyword masking, disabled rules, unsafe positional rules, config reload, and race-safe policy snapshots.
+
 ### Browser cookie sync and config UI

 **Features**
@@ -39,6 +93,26 @@ Significant changes, features, and fixes in reverse chronological order.

 ## 2026-05-22

+### Usage Cap budget controls
+
+**Features**
+
+- Added Standard/PostgreSQL usage caps for AI budget control by hour, day, week, or month.
+- Caps support token and USD cost ceilings at tenant, agent, provider, provider type, and model scopes.
+- Added OpenRouter catalog sync plus tenant/provider/model pricing overrides for input, output, cache read/write, reasoning, request, image, and web search units.
+- Enforced caps in agent, fallback model, subagent, memory flush, compaction, and media reading tools (`read_image`, `read_document`, `read_audio`, `read_video`) with preflight reservation and post-call reconciliation.
+- Added non-negative validation for catalog and override pricing fields.
+- Added OpenRouter alias resolution for native model IDs, cached-input accounting normalization, and partial-stream failure reconciliation.
+- Bridged legacy `budget_monthly_cents` into generated monthly agent USD cap policies, including migration backfill and save-time sync.
+- Added web dashboard controls on Usage and Provider detail pages.
+- Added Usage page editing for manual cap policies, including enable/disable, scope clearing, and token/USD limit clearing while keeping generated agent-budget caps read-only.
+- Added usage-cap decision metadata to LLM spans, including allow/skip/block reason, policy IDs, estimates, actuals, and reconcile status.
+
+**Tests**
+
+- Added pricing and cap service coverage.
+- Verified Go builds, SQLite build compatibility, full Go test suite, integration race suite, and Web UI production build.
+
 ### Messaging debounce hardening

 **Features**