* fix(build): embed commit SHA for release provenance (#1571 part 2)
Docker builds exclude .git via .dockerignore, so buildvcs cannot read
VCS metadata and published images carry no commit information. A running
image cannot be lined up with the source commit it was built from.
Embed cmd.CommitSHA at link time (maintainer-endorsed option 2) and
surface it in goclaw version output:
- cmd: add CommitSHA var, print commit in version cmd when injected
- Makefile: pass git rev-parse HEAD via LDFLAGS
- Dockerfile: accept COMMIT_SHA build arg (default unknown)
- docker-compose.yml: pass GOCLAW_COMMIT_SHA through as build arg
- release workflows: inject github.sha into release and dev-beta builds
Backward compatible: binaries built without the flag keep the existing
version output format.
* fix(build): pass COMMIT_SHA to docker image builds, surface it in doctor/upgrade
Review follow-up for PR #1590:
- Add COMMIT_SHA=${{ github.sha }} to the build-args of every
docker/build-push-action step (release.yaml, dev-beta-release.yaml,
release-beta.yaml, fork-image.yaml) so published images — the
artifact issue #1571 is about — carry the commit, not just release
tarball binaries.
- Surface the commit in doctor and upgrade output (App version line)
via a shared commitSuffix() helper, so operators can read provenance
from logs without exec'ing goclaw version (review suggestion #2).
- version cmd refactored onto the same helper; output unchanged.
read_video called with a url parameter always failed under an agent budget:
tool:read_video: cannot verify streamed native media against the agent
context budget (no in-memory payload to count); refusing to send
read_audio and read_document appeared to work but only for very small files.
Both symptoms share one root cause.
5ca433b8 introduced the complete-input invariant and, to keep out-of-band
media honest, appended the standard-base64 encoding of the payload as a
synthetic guard-only message. That message is counted as text. Measured against
the bundled BudgetCounter, base64 costs 0.956 tokens per raw byte, so an 11.6 MB
video counted 11,161,058 tokens and exceeded a 200k, a 1M and a 2M window alike.
The practical ceiling was roughly 200 KB at a 200k window while videoMaxBytes is
100 MB. The read_video URL transport streams to the File API without buffering,
so it had no bytes to count at all and was routed to a helper that refused to
send whenever an agent budget was present.
Media is now priced by what a provider actually bills for it. Every number below
is either a published per-unit rate or a published provider limit; none is
derived from a byte size, because a static-image video compresses arbitrarily
small and no bitrate floor exists. An earlier revision of this branch tried one
and a 120-second, 20,627-byte clip priced at 526 tokens against a true 31,560.
internal/mediabudget prices one payload:
- Video and audio: ffprobe measures duration. 263 tokens/second for video,
32 for audio, both published for static processing at 1 FPS.
- PDF: pdfinfo counts pages at 258 tokens/page. When the page count cannot be
read, the charge is the proven ceiling of 1000 pages, which is the most a
provider will accept and therefore the most it can bill.
- Video and audio that cannot be measured are refused. Upstream already
refused unverifiable native media for every URL on the Gemini streamed path;
this narrows that refusal from every URL to only what genuinely cannot be
measured, rather than removing it. PDF differs because its page count is
cheaply measurable and its ceiling is small enough to stay usable.
A remote video is measured without downloading it: two ranged GETs, 512 KB from
the head and 512 KB from the tail, written into a sparse temp file sized to the
declared total and handed to ffprobe. The tail matters because every container
that puts its index at the end keeps it there: head-only probing under-reports
mpeg by 98% and ogg by 79%, and a plain prefix makes ffprobe under-report a
30-second WAV as 0.74 seconds because it clamps to the bytes it can see. Sizing
the temp file to the real total fixes that. Verified end to end on an 11,673,105
byte MP4 served by nginx: 1 MB of ranged reads yielded duration 40.000000,
identical to ffprobe reading the whole URL, for a charge of 10,520 tokens. Those
requests reuse the existing SSRF-safe path, security.WithPinnedIP plus
security.NewSafeClient(0), and no URL is ever handed to an external binary.
Beyond the reported bug, two pre-existing gaps let large media reach a provider
almost unpriced. ExecuteWithChain treated every callProvider error as a provider
failure and advanced to the next entry, and the non-Gemini branches of read_video
and read_document reserved without pricing their payload at all. Measured under a
20,000-token window before this change, a 40 MB video and a 1000-page PDF each
reached a provider charged about 1,600 tokens. Budget refusals are now terminal
in the chain and every media branch prices its payload, so both reach no provider
at all. Genuine provider failures still fail over.
Known limits, stated rather than discovered:
- The /Type /Page scan that guards against a forged /Count is a floor, not a
bound. Pages inside a compressed object stream are invisible to it, and a
9,484-byte PDF built that way is charged 258 tokens for 1000 pages. A real
pdfinfo reads such files correctly; the scan only ever raises a probed count.
- A video URL whose origin does not serve byte ranges is now refused on the
non-Gemini path too, and the refusal is terminal. Upstream forwarded such URLs
unpriced. A HEAD giving only a size is not enough to price one.
- read_video and read_audio require ffprobe. Docker images install it by default
except the base variant; bare binaries and the desktop build do not ship it.
- A hostile origin can craft a container ffprobe reads as about one second.
Reservation.Reconcile overwrites the estimate with the provider's reported
usage, so this weakens the gate rather than defeating it.
Byte ceilings videoMaxBytes, audioMaxBytes and documentMaxBytes are unchanged.
No new module dependency; ffprobe and pdfinfo are optional runtime probes.
Root cause of v3.10.0/v3.11.0 release hangs (cancelled at 6h timeout):
the docker-images (full) job built linux/arm64 under QEMU emulation while
ENABLE_FULL_SKILLS=true triggered pip install of pandas + lxml, whose C
extensions have to compile from source on musl arm64 (no prebuilt wheels).
QEMU compile time for pandas alone exceeded the 6h job ceiling.
Changes:
- docker-images split into docker-build (per variant × platform) +
docker-merge (manifest list per variant). amd64 runs on ubuntu-latest;
arm64 runs on ubuntu-24.04-arm — native, no QEMU. Pandas/lxml compile
natively in ~10-15 min instead of >6h.
- docker-web likewise split into docker-web-build + docker-web-merge.
- Drop the otel variant — niche, users build with --build-arg
ENABLE_OTEL=true. Tailscale was never pre-built (build tag only).
- Add timeout-minutes per job (5–90 depending on workload) so failures
surface in minutes rather than hours.
- provenance: false to skip buildx attestation overhead (~5-10% faster).
- cache scope per (variant, platform) for higher hit rate on rerun.
CLAUDE.md docker variants table updated to reflect dropped OTel.
Replace go-semantic-release (auto on push main) with tag-based trigger:
- Push tag v1.2.3 → auto release (stable only, not beta/rc)
- Manual dispatch with optional tag input as fallback
- gh release create with --generate-notes replaces semantic-release changelog
- Beta/RC tags still handled by release-beta.yaml (unchanged)
go-semantic-release auto-detects the default branch from GitHub API
(which is dev), but releases are cut from main. The CI condition
rejects runs on non-default branches. Use --no-ci to bypass this
check since the workflow already gates on push to main.
go-semantic-release auto-detects the default branch from GitHub API
(which is dev), but releases are cut from main. The CI condition
rejects runs on non-default branches. Use --no-ci to bypass this
check since the workflow already gates on push to main.
- Add internal/webui/ package with //go:build embedui tag for optional
SPA embedding (handler.go serves static files with SPA fallback)
- Add internal/version/ shared semver comparison (DRY: extracted from
gateway/update_check.go and updater/updater.go)
- Enhance UpdateChecker: release notes, ETag caching, filter lite-v* tags
- Add web UI build stage to Dockerfile with ENABLE_EMBEDUI build arg
- Simplify CI: 7 Docker variants → 4 (base, latest, full, otel)
- Add SHA256 checksums job to release workflow
- Add Makefile build-full target (embeds web UI in Go binary)
- Default make up now embeds web UI (no separate nginx needed)
- Add WITH_WEB_NGINX=1 flag for optional nginx reverse proxy
- Update README + 30 translated READMEs: make up, port 18790
- Update docker-compose comments and prepare-env.sh
- About dialog: show release notes with markdown rendering
- Health card: amber badge for available updates
BREAKING: Default Docker setup no longer requires selfservice overlay.
Web dashboard served at :18790 (same port as API).
go-semantic-release creates tags via GITHUB_TOKEN, which doesn't
trigger other workflows (GitHub anti-loop policy). This caused
docker-publish.yaml to stop firing since v1.2.0.
Move Docker image builds (7 variants + web) directly into release.yaml
as parallel jobs alongside build-binaries.
The install script only copied the binary, so `goclaw migrate up` and
`goclaw onboard` failed when run from the installed location because the
migrations directory was missing.
- Include migrations/ in the release tarball
- Install migrations to /usr/local/share/goclaw/migrations
- Guide users to export GOCLAW_MIGRATIONS_DIR before onboard
- Align next-steps output with onboard's own instructions
* feat(infra): add runtime package support for skills
Install nodejs, npm, pandoc, github-cli + pre-install Python packages
(openpyxl, pandas, python-pptx, markitdown) and Node packages
(docx, pptxgenjs). Configure runtime dirs for agent pip/npm installs
with PIP_TARGET, NPM_CONFIG_PREFIX, NODE_PATH to enable dynamic
package installation in read-only container environment.
* feat(infra): add bundled skills with runtime package support
- Add 5 bundled skills: docx, pdf, pptx, xlsx, skill-creator from container skills-store
- Wire GOCLAW_BUILTIN_SKILLS_DIR env var in gateway and CLI
- Support optional runtime packages alongside dynamic skill loading
- Update Dockerfile to COPY bundled-skills at /app/bundled-skills/
- Add PIP_CACHE_DIR in docker-entrypoint.sh for clean pip installs
- Document bundled skills in 14-skills-runtime.md section 6
* feat(infra): remove ai-multimodal skill directory from bundled skills
Remove the ai-multimodal skill package as part of consolidating runtime
package support for bundled skills. This directory is no longer needed
in the bundled skills structure.
* feat(ci): add semantic release and Docker Hub publishing
Add go-semantic-release workflow to auto-create semver tags on merge to
main. Extend docker-publish to push all variants to both GHCR and
Docker Hub (digitop/goclaw).
* feat(skills): add system skills infrastructure with is_system column, dep scanning, and seeder
- Migration 000017: add is_system boolean column with partial index
- Store layer: UpsertSystemSkill, delete protection, IsSystemSkill
- ListAccessible auto-includes system skills (no grants needed)
- ListWithGrantStatus returns is_system field
- Dependency scanner: auto-detect deps from scripts/ or skill-manifest.json
- Dependency checker: verify system binaries, Python/Node packages
- Seeder: seed bundled skills into DB on startup (idempotent via hash)
- Gateway wiring: GOCLAW_BUNDLED_SKILLS_DIR env for bundled skills
- HTTP: delete guard (403), slug conflict check (409), rescan-deps endpoint
- UI: System badge, hide delete for system skills, rescan deps button
- Agent skills tab: "Always available" for system skills
- i18n: en/vi/zh keys for system skills, deps scanning
* feat(skills): conditional system prompt, skill manifests, and Zip Slip fix
- System prompt: only show package list when python3/node are available
- Add skill-manifest.json for pdf, docx, xlsx, pptx bundled skills
- Fix Zip Slip vulnerability in office/unpack.py (all 3 copies)
* refactor(skills): extract shared office code to _shared/ and deduplicate
Move office scripts (pack, unpack, validate, schemas, validators) from
duplicated copies in docx/xlsx/pptx to skills/_shared/office/ with
symlinks. Remove soffice.py (non-functional in containers) and update
SKILL.md references to use soffice binary directly. Update seeder
copyDir to follow symlinks.
Removes ~45K lines of duplicate code across 3 skills.
* fix(skills): address code review findings for system skills integration
- H1: Remove dead symlink branch in copyDir (filepath.Walk follows symlinks)
- H3: Fix rescan-deps to query ALL skills (including archived) and re-activate
when deps become available; add ListAllSkills() + Status field to SkillInfo
- H4: Add Status field to SkillCreateParams, stop overloading Visibility
- M1: Batch Python/Node dep checks into single subprocess per runtime
- M4: Add rows.Err() check in ListSkills to prevent caching partial results
* feat(skills): async dep checking with realtime WS events
Split Seed() into sync DB upsert + async CheckDepsAsync() goroutine.
Gateway startup no longer blocks on Python/Node subprocess dep checks.
- Seed() returns seeded skills list, all initially status="active"
- CheckDepsAsync() runs in background, emits skill.deps.checked per-skill
- skill.deps.complete event emitted when all checks finish
- Each failed dep check: archives skill + BumpVersion() for immediate
cache invalidation so next agent turn picks up the change
- UI: use-query-invalidation listens to skill.deps.* events → auto-refresh
skills list in realtime
* feat(skills): system skills integration with toggle, dep checking, and per-item install
- Add is_system, deps, enabled columns to skills table (migration 017)
- Seed bundled core skills (pdf, docx, pptx, xlsx, skill-creator) on startup
- PYTHONPATH-based dep detection — eliminates false positives from local modules
- Per-item dep install UI with individual status (installing/success/error)
- Enable/disable toggle for core and custom skills (independent of dep status)
- Re-run dep check when skill is toggled back on
- Inline skill thresholds: 40 skills / 5000 tokens before switching to search mode
- Fix UpsertSystemSkill: backfill null file_hash without bumping DB version
- Remove redundant skill-manifest.json files (replaced by deps JSONB column)
- Show author from frontmatter in custom skills tab
- Runtime checker for python3/pip3/node/npm availability
- WS events for dep checking/installing progress
- docs: add 15-core-skills-system.md, 16-skill-publishing.md
---------
Co-authored-by: Goon <duy@wearetopgroup.com>
fchengyan