mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-20 02:17:24 +00:00
feat(packages): GitHub Releases binary installer (#898)
* feat(packages): add GitHub Releases binary installer New runtime source `github:owner/repo[@tag]` for installing Linux CLI binaries from GitHub Releases. Admin-only, SHA256-verified, ELF-validated. Backend: - GitHub API client with 10-min cache + rate-limit mapping - SSRF-guarded streaming downloader (HTTPS + host allowlist, re-validated on every redirect hop, literal-IP rejection) - Checksums.txt / SHA256SUMS lookup with constant-time verify - Archive extract (tar.gz / zip / raw) with path-traversal + zip-bomb guards, symlink skip - ELF magic + 64-bit class + runtime-arch validation - Atomic manifest persistence (temp + rename) HTTP: - POST /v1/packages/install accepts github: spec - GET /v1/packages/github-releases for picker UI (viewer+, arch-filtered) - Extended InstalledPackages response with github field - github-bin runtime probe Infra: - Dockerfile creates /app/data/.runtime/bin (goclaw:goclaw 0755) - docker-entrypoint.sh prepends bin dir to PATH - Env-only config (never config.json): token, max size, org allowlist, bin dir, manifest path UI: - GitHub Binaries section + release picker modal - Dismissable musl/glibc compatibility warning (localStorage) - i18n keys across en/vi/zh Docs: docs/packages-github.md user guide + 14-skills-runtime.md cross-ref. Closes #741 * refactor(packages): revert validPkgName broadening + drop unused sentinel Code review cleanup: - validPkgName regex had `:` added defensively, but github: specs are validated separately via skills.ParseGitHubSpec before reaching this check — the broadening was dead attack surface. - Drop unused ErrUnknownArchive sentinel + the `_ = ErrUnknownArchive` stub in extractRaw. * feat(packages): per-user rate limit on /v1/packages/github-releases Cap picker endpoint at 30 req/min/user (burst 10) to protect the shared GitHub API quota. Key is userID (header X-GoClaw-User-Id) or remote IP for anonymous callers. Returns 429 + Retry-After: 60 when tripped. Standalone token-bucket limiter (stale-entry cleanup every 5 min) lives in internal/http rather than importing internal/gateway, which would create a package cycle. * fix(ui): guard split()[0] for noUncheckedIndexedAccess strict TS CI pnpm build failed on TS2345: `.split('@')[0]` returns `string | undefined` under strict index access. Default to empty string to satisfy the type checker; runtime behaviour unchanged because the downstream regex rejects empty strings. * fix(packages): address Claude review — medium + low findings Medium - rate limiter: atomic.Int64 lastSeen + amortized sweep replaces goroutine-based cleanup → fixes data race on lastSeen and the goroutine leak when tests swap the package-level limiter. - checksum pipeline: slog.Warn on ReadFile and ParseChecksums failures (previously silent). "asset not listed" stays warn+proceed but is now documented as the publisher's choice — ELF validation remains the final gate. - downloader: drop http.Client.Timeout (30s capped the whole request including body read, aborting large downloads on slow links). Context deadline from install timeout (5 min) is the correct bound. Low / style / UI - extractRaw honors maxUncompressed (ErrFileTooLarge on overflow) so the helper is safe outside the hot path. - cmd/gateway_github_installer.go: drop the explicit cfg.Defaults() call — NewGitHubInstaller already invokes it. - GitHubPackageEntry: remove unpopulated InstalledBy field + document why. - owner regex tightened to 39-char GitHub limit (was 40). - mu lock comment corrected: serializes only the disk-write phase. - UI: shared stripPrefixAndTag helper + owner regex mirrors the backend 39-char cap; destructure-with-default kills the split()[0] ?? "" awkwardness while still satisfying noUncheckedIndexedAccess. Verified: go build (pg + sqliteonly) · go vet · go test -race ./internal/skills ./internal/http · pnpm build. * fix(packages): address Claude review round 2 Medium - validRepoPath now rejects trailing hyphens in the owner segment and caps at 39 chars, matching gitHubSpecRE exactly. Previously a subtle drift between the two validators could let `foo-/repo` slip to the GitHub API and surface as a 502 instead of a clean 400. - handleGitHubReleases no longer forwards raw err.Error() from the upstream call. Maps sentinel errors: ErrGitHubRateLimited → 429 + Retry-After ErrGitHubNotFound → 404 ErrGitHubUnauthorized → 502 "github authentication failed" default → 502 "failed to fetch releases" Avoids leaking rate-limit reset timestamps / server internals to viewer-tier callers. Low / UX - Install response now returns the manifest entry for github: specs (new lookupGitHubEntry helper; nil-safe fallback to {ok:true}). Lets the UI display "installed: lazygit v0.42.0" without a list refresh. - gitHubSpecRE tag segment capped at 1..255 chars (git ref-name bound). UI isValidFullSpec mirrors the same cap. * fix(packages): address Claude review round 3 Medium - github_api: URL-encode owner, repo, and tag via url.PathEscape when building API paths. Previously a tag containing '#' would be stripped as a URL fragment and '?' would inject a query parameter, silently hitting the wrong release. Low / polish - Uninstall via full "github:owner/repo[@tag]" spec now falls back to manifest lookup by owner/repo, handling packages whose binary name differs from the repo name (cli/cli → gh). - GitHubClient.cache sweeps expired entries opportunistically when the map grows past 256 entries (prevents theoretical unbounded growth over long uptime). - handleInstall for github: specs now calls GitHubInstaller.Install directly and returns the freshly-created manifest entry, eliminating the double manifest read via List() from the lookupGitHubEntry helper. - pickBinaries comment corrected — actual behavior excludes paths matched by nonBinaryPathRE rather than enforcing a single-depth limit. * fix(packages): address Claude review round 4 (style + ordering) All 4 findings are Low severity: - github_api.go: replace interface{} with any across the cache type, cacheGet return, cacheSet param, and doJSON out param. - doJSON: rename local `url` to `apiURL` to avoid shadowing the "net/url" package import used by GetRelease/ListReleases. - Uninstall: save the updated manifest BEFORE removing binaries on disk. If saveManifest fails we now bail out without leaving a manifest entry that still claims binaries which have been deleted (a retried Uninstall would otherwise hit ErrPackageNotInstalled after the first attempt wiped the files). Disk removal stays best-effort and warn-on-error, which matches the idempotent intent. - pickBinaries: inline comment corrected to reflect actual behavior — depth is not enforced; nonBinaryPathRE filter + downstream ELF validation are the real gates. * fix(packages): address Claude review round 5 All 3 findings are Low severity: - handleInstall github fast-path now wraps the context with skills.InstallTimeout (5 min) before calling gh.Install and emits the same "skills: installing dep" / "dep installed" / "github install failed" log lines as the generic InstallSingleDep path, so operator-observability is identical between github: and pip:/npm: install flows. - installTimeout promoted to exported InstallTimeout so the http layer shares the single source of truth rather than duplicating the 5-minute constant. - cacheMaxEntries comment clarifies it is a soft sweep trigger, not a hard cap — when every entry is still within TTL the map can briefly exceed the threshold by one insert. * fix(packages): address Claude review round 6 (final Lows) Both findings are Low severity (reviewer marked the PR "ready to merge" already): - github_installer: "no checksum asset available" downgraded from slog.Warn to slog.Info. Many popular upstream releases (jq, fzf, older ripgrep, etc.) ship no checksum file at all — that is publisher policy, not a problem with the install. The suspicious cases (checksum file unreadable, unparseable, or missing this asset) stay at Warn so they stand out. - handleGitHubReleases response now uses a narrow assetPreview DTO (name + size_bytes) instead of embedding the full GitHubAsset type which also carried browser_download_url. The picker UI never rendered the URL; trimming the response keeps the viewer-tier surface minimal. UI AssetPreview interface realigned to match. * fix(packages): address Claude review round 7 Narrow the GET /v1/packages GitHub entry to a viewer-safe projection (repo/tag/binaries/name/installed_at), mirroring the assetPreview fix from round 6. Strips asset_url, sha256, and asset_name from the list response — viewer-level callers no longer see CDN download URLs or checksum metadata for installed packages. UI types realigned; the removed fields were never rendered. Finding #2 (install writes binary before manifest save) left as noted — reviewer confirmed informational only, self-heals on retry, no security impact since binaries pass ELF validation before being written. * fix(packages): address Claude review round 8 Map HTTP 429 (GitHub secondary rate limits — abuse detection, unauthenticated bursts, search) to ErrGitHubRateLimited in the API client so the picker endpoint renders 429 "rate limit reached" with Retry-After: 60 instead of falling through to 502 "failed to fetch releases". Primary rate limits (403 + X-RateLimit-Remaining: 0) were already handled; this covers the secondary class documented at https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api#about-secondary-rate-limits * fix(packages): address Claude review round 9 Two defensive hardenings flagged as Very Low: - ParseChecksums: strip leading `./` from checksum filenames. `sha256sum ./file` emits `./file` in the name column; the caller looks up by bare asset basename so `./`-prefixed entries would silently miss. Real release checksums almost never use this form, but the guard is essentially free. - doJSON: cap response body at 8 MiB via io.LimitReader before JSON decode. Current GitHub list/release payloads are well under this (~1 MiB at per_page=100). Guards against future call sites or a misbehaving upstream returning an oversized document. * fix(cron): eliminate cross-test race on runLoopTickInterval `Service.Stop()` closes stopChan but does not wait for the runLoop goroutine to exit. In the test suite, test A's `defer cs.Stop()` can return before the spawned runLoop has reached `ticker := time.NewTicker(runLoopTickInterval)`. If test B then calls `setFastTick()` to mutate the package-level var, the race detector correctly flags it: Read at runLoopTickInterval by goroutine A (runLoop ticker init) Previous write by goroutine B (setFastTick in test B) Fix: snapshot `runLoopTickInterval` inside `Start()` under the mutex before spawning the goroutine, and pass the value as a parameter to `runLoop`. The spawned goroutine no longer reads the package-level var, so the cross-test window is closed. Production behavior unchanged. Verified: `go test -race -count=3 ./internal/cron/...` passes three times in a row; the CI failure on PR #898 reproduced before the fix and is gone after. * fix(packages): address review P0/P1/P2 + new DoS vector P0.1 — UI uninstall 400: parseAndValidatePackage now accepts github:<bare-name> (manifest Name form, no owner/repo) in addition to the full spec. UI sends github:${pkg.name} from the manifest; dispatcher already tolerated bare names — the HTTP validator was the only gate rejecting them. Install path re-validates strictly via ParseGitHubSpec and bare-name install returns 400 now (was 500). P1.1 — ExtractArchive raw-ELF fallback name: add ExtractArchiveAs( path, fallbackName, max). Installer passes parsed.Repo so raw (non-archive) ELF assets no longer end up recorded as /tmp/goclaw-gh-asset-XXXX.bin — that basename would leak into the manifest Binaries entry and break PATH lookup. P1.3 — Archive entry count cap: maxArchiveEntries = 10_000 + ErrTooManyEntries sentinel. Tar: count ALL headers seen (incl. symlinks/dirs we skip) to block the gzip-bomb-of-headers DoS — header bytes don't count against maxUncompressed for zero-size entries. Zip: pre-check via peekZipEntryCount reads the EOCD record manually and rejects oversized archives BEFORE zip.OpenReader allocates []*zip.File of declared capacity (this was a fresh red-team finding; stdlib would otherwise alloc ~1GB for a crafted 200MB zip claiming 4M entries). P1.4 — Rate-limit install/uninstall: packagesWriteLimiter (10/min/user, burst 3). Admin-only mitigates but a compromised token could otherwise flood upstream (GitHub/pip/npm) or spam manifest mutations. P1.6 — Non-Linux early reject: ErrUnsupportedOS guard at the top of Install(). Windows/macOS hosts no longer waste bandwidth fetching a Linux asset just to fail at the ELF machine check. P1.7 — Manifest fsync: OpenFile → Write → Sync → Close → Rename → dir Sync, with tmp cleanup on every error path. POSIX doesn't guarantee durability via rename alone; XFS / ext4 with async journal can reorder. P2.1 — Belt-and-suspenders zip runtime break when cumulative bytes reach the cap (pre-declared check already covers it but the streaming loop now bails immediately). P2.6 — Binary-name collision warn: slog.Warn when a different repo already owns the basename we're about to overwrite. Last-writer- wins unchanged; operator now gets a signal instead of silence. Hardening — rate-limit key: rateLimitKeyFromRequest prefers store.UserIDFromContext over the raw X-GoClaw-User-Id header so an admin can't rotate the header mid-session to dodge the bucket. Header/IP fallback retained for pre-auth / test callers. Tests: 9 new cases on parseAndValidatePackage (github full/bare/ empty/traversal/injection/space/leading-hyphen); TestExtractArchiveAs_RawELFUsesFallbackName; TestExtractTarGz_EntryCountCap + TestExtractZip_EntryCountCap; TestPeekZipEntryCount (DoS pre-check path). Verified: go build ./... && go build -tags sqliteonly ./... && go vet ./... && go test -race ./internal/skills/... ./internal/http/... --------- Co-authored-by: viettranx <viettranx@gmail.com>
This commit is contained in:
Duy /zuey/andviettranx authored and GitHub committed
1 parent
0a4b8df001
commit
2cbf838158
32 files changed
+3384
-19
No files matched your search
@@ -1,8 +1,14 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/permissions"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/skills"
|
||||
@@ -10,9 +16,20 @@ import (
|
||||
)
|
||||
|
||||
// validPkgName allows alphanumeric, hyphens, underscores, dots, @, / (for scoped npm).
|
||||
// `github:` specs are validated separately (via skills.ParseGitHubSpec) and bypass this regex.
|
||||
// Rejects names starting with - to prevent argument injection.
|
||||
var validPkgName = regexp.MustCompile(`^[a-zA-Z0-9@][a-zA-Z0-9._+\-/@]*$`)
|
||||
|
||||
// validGitHubBareName matches bare manifest names used on the uninstall path
|
||||
// (e.g. "gh", "lazygit", "ripgrep-13"). Must not contain `/` or `@` — those
|
||||
// forms are handled by the full-spec branch via skills.ParseGitHubSpec.
|
||||
var validGitHubBareName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`)
|
||||
|
||||
// validRepoPath matches "owner/repo" used by the releases endpoint.
|
||||
// Owner rules mirror skills.gitHubSpecRE — GitHub caps usernames/orgs at 39 chars,
|
||||
// no leading/trailing hyphen.
|
||||
var validRepoPath = regexp.MustCompile(`^([A-Za-z0-9](?:[A-Za-z0-9-]{0,37})?[A-Za-z0-9]|[A-Za-z0-9])/[A-Za-z0-9][A-Za-z0-9._-]*$`)
|
||||
|
||||
// PackagesHandler handles runtime package management HTTP endpoints.
|
||||
type PackagesHandler struct{}
|
||||
|
||||
@@ -27,6 +44,7 @@ func (h *PackagesHandler) RegisterRoutes(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /v1/packages/install", h.adminAuth(h.handleInstall))
|
||||
mux.HandleFunc("POST /v1/packages/uninstall", h.adminAuth(h.handleUninstall))
|
||||
mux.HandleFunc("GET /v1/packages/runtimes", h.readAuth(h.handleRuntimes))
|
||||
mux.HandleFunc("GET /v1/packages/github-releases", h.readAuth(h.handleGitHubReleases))
|
||||
mux.HandleFunc("GET /v1/shell-deny-groups", h.readAuth(h.handleDenyGroups))
|
||||
}
|
||||
|
||||
@@ -63,6 +81,25 @@ func parseAndValidatePackage(w http.ResponseWriter, r *http.Request) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// github: packages carry the scheme prefix through the whole pipeline.
|
||||
// Accept two forms:
|
||||
// 1. Full spec "github:owner/repo[@tag]" — install + uninstall path.
|
||||
// 2. Bare manifest name "github:<name>" — uninstall path only (the UI
|
||||
// table surfaces canonical Name which may differ from repo — e.g.
|
||||
// cli/cli → gh). Install will re-validate via ParseGitHubSpec and
|
||||
// return a clear ErrInvalidGitHubSpec for bare-name form.
|
||||
if strings.HasPrefix(body.Package, "github:") {
|
||||
if _, err := skills.ParseGitHubSpec(body.Package); err == nil {
|
||||
return body.Package
|
||||
}
|
||||
bare := strings.TrimPrefix(body.Package, "github:")
|
||||
if validGitHubBareName.MatchString(bare) {
|
||||
return body.Package
|
||||
}
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid github spec"})
|
||||
return ""
|
||||
}
|
||||
|
||||
// Strip prefix for validation, then validate the bare package name.
|
||||
name := body.Package
|
||||
for _, prefix := range []string{"pip:", "npm:"} {
|
||||
@@ -90,10 +127,56 @@ func (h *PackagesHandler) handleInstall(w http.ResponseWriter, r *http.Request)
|
||||
if !requireMasterScope(w, r) {
|
||||
return
|
||||
}
|
||||
if !enforcePackagesWriteLimit(w, r, "/v1/packages/install") {
|
||||
return
|
||||
}
|
||||
pkg := parseAndValidatePackage(w, r)
|
||||
if pkg == "" {
|
||||
return
|
||||
}
|
||||
|
||||
// Fast path for github: specs — call the installer directly so we can
|
||||
// return the freshly-created manifest entry without a second disk read
|
||||
// via List(). Other prefixes fall through to the generic dispatcher.
|
||||
// Uses the same InstallTimeout + top-level log line as InstallSingleDep
|
||||
// for operator-observability parity across install paths.
|
||||
if strings.HasPrefix(pkg, "github:") {
|
||||
gh := skills.DefaultGitHubInstaller()
|
||||
if gh == nil {
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]any{
|
||||
"ok": false, "error": "github installer not configured",
|
||||
})
|
||||
return
|
||||
}
|
||||
slog.Info("skills: installing dep", "dep", pkg)
|
||||
ctx, cancel := context.WithTimeout(r.Context(), skills.InstallTimeout)
|
||||
defer cancel()
|
||||
entry, err := gh.Install(ctx, pkg)
|
||||
if err != nil {
|
||||
slog.Error("skills: github install failed", "dep", pkg, "error", err)
|
||||
// Classify client-side errors as 400 so the UI can show a clear
|
||||
// validation message instead of a generic 500. The bare-name
|
||||
// form (github:<name>) is accepted by parseAndValidatePackage to
|
||||
// keep the uninstall path alive; Install re-validates strictly
|
||||
// via ParseGitHubSpec and this branch surfaces that to the user.
|
||||
status := http.StatusInternalServerError
|
||||
switch {
|
||||
case errors.Is(err, skills.ErrInvalidGitHubSpec),
|
||||
errors.Is(err, skills.ErrGitHubOrgNotAllowed),
|
||||
errors.Is(err, skills.ErrUnsupportedOS),
|
||||
errors.Is(err, skills.ErrNoMatchingAsset):
|
||||
status = http.StatusBadRequest
|
||||
}
|
||||
writeJSON(w, status, map[string]any{
|
||||
"ok": false, "error": err.Error(),
|
||||
})
|
||||
return
|
||||
}
|
||||
slog.Info("skills: dep installed", "dep", pkg)
|
||||
writeJSON(w, http.StatusOK, map[string]any{"ok": true, "entry": entry})
|
||||
return
|
||||
}
|
||||
|
||||
ok, errMsg := skills.InstallSingleDep(r.Context(), pkg)
|
||||
if !ok {
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]any{"ok": false, "error": errMsg})
|
||||
@@ -111,6 +194,9 @@ func (h *PackagesHandler) handleUninstall(w http.ResponseWriter, r *http.Request
|
||||
if !requireMasterScope(w, r) {
|
||||
return
|
||||
}
|
||||
if !enforcePackagesWriteLimit(w, r, "/v1/packages/uninstall") {
|
||||
return
|
||||
}
|
||||
pkg := parseAndValidatePackage(w, r)
|
||||
if pkg == "" {
|
||||
return
|
||||
@@ -128,6 +214,95 @@ func (h *PackagesHandler) handleRuntimes(w http.ResponseWriter, _ *http.Request)
|
||||
writeJSON(w, http.StatusOK, skills.CheckRuntimes())
|
||||
}
|
||||
|
||||
// handleGitHubReleases proxies the GitHub Releases API for the picker UI.
|
||||
// GET /v1/packages/github-releases?repo=owner/repo&limit=10
|
||||
// Auth: viewer+ (read-only, no secrets exposed).
|
||||
// Throttled via per-user rate limiter to protect the shared GitHub API quota.
|
||||
func (h *PackagesHandler) handleGitHubReleases(w http.ResponseWriter, r *http.Request) {
|
||||
if !enforceGitHubReleasesLimit(w, r) {
|
||||
return
|
||||
}
|
||||
gh := skills.DefaultGitHubInstaller()
|
||||
if gh == nil || gh.Client == nil {
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "github installer not configured"})
|
||||
return
|
||||
}
|
||||
repo := r.URL.Query().Get("repo")
|
||||
if !validRepoPath.MatchString(repo) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid repo; expected owner/repo"})
|
||||
return
|
||||
}
|
||||
parts := strings.SplitN(repo, "/", 2)
|
||||
owner, repoName := parts[0], parts[1]
|
||||
|
||||
if !gh.AllowedOrg(owner) {
|
||||
// Return 404 rather than 403 so allowlist membership is not enumerable.
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "not found"})
|
||||
return
|
||||
}
|
||||
|
||||
limit := 10
|
||||
if s := r.URL.Query().Get("limit"); s != "" {
|
||||
if n, err := strconv.Atoi(s); err == nil && n >= 1 && n <= 50 {
|
||||
limit = n
|
||||
}
|
||||
}
|
||||
|
||||
releases, err := gh.Client.ListReleases(r.Context(), owner, repoName, limit)
|
||||
if err != nil {
|
||||
// Map sentinel errors to generic client-safe messages. Avoid surfacing
|
||||
// raw GitHub API error bodies (may include rate-limit timestamps,
|
||||
// server-side internals) to viewer-level callers.
|
||||
switch {
|
||||
case errors.Is(err, skills.ErrGitHubRateLimited):
|
||||
w.Header().Set("Retry-After", "60")
|
||||
writeJSON(w, http.StatusTooManyRequests, map[string]string{"error": "github rate limit reached"})
|
||||
case errors.Is(err, skills.ErrGitHubNotFound):
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "repository not found"})
|
||||
case errors.Is(err, skills.ErrGitHubUnauthorized):
|
||||
writeJSON(w, http.StatusBadGateway, map[string]string{"error": "github authentication failed"})
|
||||
default:
|
||||
writeJSON(w, http.StatusBadGateway, map[string]string{"error": "failed to fetch releases"})
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// assetPreview is a deliberately-narrow projection of GitHubAsset exposed
|
||||
// to viewer-level callers of the picker. The full GitHubAsset type carries
|
||||
// the CDN download URL which the UI never renders — keep the response
|
||||
// surface minimal (name + size_bytes are all the picker needs).
|
||||
type assetPreview struct {
|
||||
Name string `json:"name"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
}
|
||||
type releaseDTO struct {
|
||||
Tag string `json:"tag"`
|
||||
Name string `json:"name"`
|
||||
PublishedAt string `json:"published_at"`
|
||||
Prerelease bool `json:"prerelease"`
|
||||
MatchingAssets []assetPreview `json:"matching_assets"`
|
||||
AllAssetsCount int `json:"all_assets_count"`
|
||||
}
|
||||
out := make([]releaseDTO, 0, len(releases))
|
||||
for _, rel := range releases {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
dto := releaseDTO{
|
||||
Tag: rel.TagName,
|
||||
Name: rel.Name,
|
||||
PublishedAt: rel.PublishedAt.UTC().Format("2006-01-02T15:04:05Z"),
|
||||
Prerelease: rel.Prerelease,
|
||||
AllAssetsCount: len(rel.Assets),
|
||||
}
|
||||
if pick, perr := skills.SelectAsset(rel.Assets, "linux", runtime.GOARCH); perr == nil && pick != nil {
|
||||
dto.MatchingAssets = []assetPreview{{Name: pick.Name, SizeBytes: pick.SizeBytes}}
|
||||
}
|
||||
out = append(out, dto)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"releases": out})
|
||||
}
|
||||
|
||||
// handleDenyGroups returns all registered shell deny groups with name, description, and default state.
|
||||
func (h *PackagesHandler) handleDenyGroups(w http.ResponseWriter, _ *http.Request) {
|
||||
type groupInfo struct {
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/store"
|
||||
"golang.org/x/time/rate"
|
||||
)
|
||||
|
||||
// perKeyRateLimiter is a minimal per-key token-bucket limiter used to cap
|
||||
// external GitHub API usage initiated through /v1/packages/github-releases.
|
||||
// Key is userID (header X-GoClaw-User-Id) or RemoteAddr when anonymous.
|
||||
//
|
||||
// Stale-entry eviction is amortized: every perKeyRateLimiterSweepInterval
|
||||
// accepted requests trigger an inline scan. This avoids a background
|
||||
// goroutine (which leaked in tests when the package-level instance was
|
||||
// swapped out) and sidesteps data races on per-entry state.
|
||||
type perKeyRateLimiter struct {
|
||||
limiters sync.Map // key → *perKeyEntry
|
||||
rps rate.Limit
|
||||
burst int
|
||||
callCounter atomic.Int64
|
||||
}
|
||||
|
||||
type perKeyEntry struct {
|
||||
limiter *rate.Limiter
|
||||
lastSeen atomic.Int64 // unix nanoseconds; updated on every Allow=true
|
||||
}
|
||||
|
||||
// perKeyRateLimiterSweepInterval controls how often (per accepted call) the
|
||||
// stale-entry sweep runs. Power-of-two for cheap modulo.
|
||||
const perKeyRateLimiterSweepInterval = 1024
|
||||
|
||||
// perKeyRateLimiterStaleAfter is the idle window before an entry is evicted.
|
||||
const perKeyRateLimiterStaleAfter = 10 * time.Minute
|
||||
|
||||
// newPerKeyRateLimiter: rpm is requests per minute, burst is max burst size.
|
||||
// rpm <= 0 disables (always allows).
|
||||
func newPerKeyRateLimiter(rpm, burst int) *perKeyRateLimiter {
|
||||
if burst <= 0 {
|
||||
burst = 5
|
||||
}
|
||||
r := rate.Limit(0)
|
||||
if rpm > 0 {
|
||||
r = rate.Limit(float64(rpm) / 60.0)
|
||||
}
|
||||
return &perKeyRateLimiter{rps: r, burst: burst}
|
||||
}
|
||||
|
||||
// Allow reports whether the request is within budget.
|
||||
func (rl *perKeyRateLimiter) Allow(key string) bool {
|
||||
if rl.rps == 0 {
|
||||
return true // disabled
|
||||
}
|
||||
nowNs := time.Now().UnixNano()
|
||||
|
||||
// Prepare a fresh entry up front; LoadOrStore discards it on existing keys.
|
||||
fresh := &perKeyEntry{limiter: rate.NewLimiter(rl.rps, rl.burst)}
|
||||
fresh.lastSeen.Store(nowNs)
|
||||
|
||||
v, _ := rl.limiters.LoadOrStore(key, fresh)
|
||||
entry := v.(*perKeyEntry)
|
||||
if !entry.limiter.Allow() {
|
||||
return false
|
||||
}
|
||||
entry.lastSeen.Store(nowNs)
|
||||
|
||||
if rl.callCounter.Add(1)%perKeyRateLimiterSweepInterval == 0 {
|
||||
rl.sweepStale()
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// sweepStale evicts entries older than perKeyRateLimiterStaleAfter.
|
||||
// Safe for concurrent invocation — sync.Map.Range + atomic lastSeen guarantee
|
||||
// data-race freedom.
|
||||
func (rl *perKeyRateLimiter) sweepStale() {
|
||||
cutoffNs := time.Now().Add(-perKeyRateLimiterStaleAfter).UnixNano()
|
||||
rl.limiters.Range(func(k, v any) bool {
|
||||
if v.(*perKeyEntry).lastSeen.Load() < cutoffNs {
|
||||
rl.limiters.Delete(k)
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
|
||||
// githubReleasesLimiter caps calls to the picker endpoint to protect the
|
||||
// shared upstream GitHub API quota. 30 req/min/user with burst 10 leaves
|
||||
// plenty of headroom for UX while preventing quota exhaustion.
|
||||
var githubReleasesLimiter = newPerKeyRateLimiter(30, 10)
|
||||
|
||||
// packagesWriteLimiter throttles POST /install and /uninstall. Admin-only
|
||||
// endpoints, but a compromised admin token could otherwise flood the upstream
|
||||
// GitHub API / pip / npm or spam manifest mutations. 10 req/min/user with
|
||||
// burst 3 comfortably covers real admin workflows while breaking automated
|
||||
// abuse.
|
||||
var packagesWriteLimiter = newPerKeyRateLimiter(10, 3)
|
||||
|
||||
// rateLimitKeyFromRequest returns the authenticated user ID if present (from
|
||||
// the request context populated by enrichContext/requireAuth), else falls
|
||||
// back to the raw X-GoClaw-User-Id header, else the remote IP.
|
||||
//
|
||||
// Preferring context over header means an admin with a leaked token can't
|
||||
// rotate the header mid-session to dodge the per-user bucket — the context
|
||||
// is bound to the API-key owner / session principal that survived auth.
|
||||
func rateLimitKeyFromRequest(r *http.Request) string {
|
||||
if uid := store.UserIDFromContext(r.Context()); uid != "" {
|
||||
return "uid:" + uid
|
||||
}
|
||||
if uid := extractUserID(r); uid != "" {
|
||||
return "uid:" + uid
|
||||
}
|
||||
host, _, err := net.SplitHostPort(r.RemoteAddr)
|
||||
if err != nil || host == "" {
|
||||
host = r.RemoteAddr
|
||||
}
|
||||
return "ip:" + host
|
||||
}
|
||||
|
||||
// enforceGitHubReleasesLimit returns true if the request is allowed; false (after
|
||||
// writing 429) if throttled.
|
||||
func enforceGitHubReleasesLimit(w http.ResponseWriter, r *http.Request) bool {
|
||||
key := rateLimitKeyFromRequest(r)
|
||||
if githubReleasesLimiter.Allow(key) {
|
||||
return true
|
||||
}
|
||||
slog.Warn("security.rate_limited", "endpoint", "/v1/packages/github-releases", "key", key)
|
||||
w.Header().Set("Retry-After", "60")
|
||||
writeJSON(w, http.StatusTooManyRequests, map[string]string{
|
||||
"error": "rate limit exceeded; try again in 60 seconds",
|
||||
})
|
||||
return false
|
||||
}
|
||||
|
||||
// enforcePackagesWriteLimit caps POST /install + /uninstall per user.
|
||||
// Returns true when the request is within budget; false (after writing 429)
|
||||
// when throttled.
|
||||
func enforcePackagesWriteLimit(w http.ResponseWriter, r *http.Request, endpoint string) bool {
|
||||
key := rateLimitKeyFromRequest(r)
|
||||
if packagesWriteLimiter.Allow(key) {
|
||||
return true
|
||||
}
|
||||
slog.Warn("security.rate_limited", "endpoint", endpoint, "key", key)
|
||||
w.Header().Set("Retry-After", "60")
|
||||
writeJSON(w, http.StatusTooManyRequests, map[string]string{
|
||||
"error": "rate limit exceeded; try again in 60 seconds",
|
||||
})
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPerKeyRateLimiter_AllowThenBlock(t *testing.T) {
|
||||
rl := newPerKeyRateLimiter(60, 2) // 1 rps, burst 2
|
||||
|
||||
// First two requests for key A succeed (burst).
|
||||
for i := 0; i < 2; i++ {
|
||||
if !rl.Allow("A") {
|
||||
t.Fatalf("request %d should be allowed", i)
|
||||
}
|
||||
}
|
||||
// Third immediate request should be blocked.
|
||||
if rl.Allow("A") {
|
||||
t.Error("third immediate request should be rate-limited")
|
||||
}
|
||||
// Independent key is unaffected.
|
||||
if !rl.Allow("B") {
|
||||
t.Error("different key should not be rate-limited")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPerKeyRateLimiter_Disabled(t *testing.T) {
|
||||
rl := newPerKeyRateLimiter(0, 5)
|
||||
for i := 0; i < 100; i++ {
|
||||
if !rl.Allow("x") {
|
||||
t.Fatalf("disabled limiter should always allow (i=%d)", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimitKeyFromRequest(t *testing.T) {
|
||||
// Authenticated user: prefix with uid:.
|
||||
r := httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||
r.Header.Set("X-GoClaw-User-Id", "alice")
|
||||
if got := rateLimitKeyFromRequest(r); got != "uid:alice" {
|
||||
t.Errorf("want uid:alice, got %s", got)
|
||||
}
|
||||
|
||||
// Anonymous: fall back to IP.
|
||||
r2 := httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||
r2.RemoteAddr = "203.0.113.5:54321"
|
||||
if got := rateLimitKeyFromRequest(r2); got != "ip:203.0.113.5" {
|
||||
t.Errorf("want ip:203.0.113.5, got %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnforceGitHubReleasesLimit_Writes429(t *testing.T) {
|
||||
// Swap the package-level limiter for a tight one, restore after.
|
||||
prev := githubReleasesLimiter
|
||||
githubReleasesLimiter = newPerKeyRateLimiter(60, 1) // burst 1
|
||||
defer func() { githubReleasesLimiter = prev }()
|
||||
|
||||
r := httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||
r.Header.Set("X-GoClaw-User-Id", "bob")
|
||||
|
||||
w1 := httptest.NewRecorder()
|
||||
if !enforceGitHubReleasesLimit(w1, r) {
|
||||
t.Fatal("first call should pass")
|
||||
}
|
||||
w2 := httptest.NewRecorder()
|
||||
if enforceGitHubReleasesLimit(w2, r) {
|
||||
t.Fatal("second call should be throttled")
|
||||
}
|
||||
if w2.Code != http.StatusTooManyRequests {
|
||||
t.Errorf("want 429, got %d", w2.Code)
|
||||
}
|
||||
if w2.Header().Get("Retry-After") == "" {
|
||||
t.Error("Retry-After header missing")
|
||||
}
|
||||
}
|
||||
@@ -118,9 +118,62 @@ func TestParseAndValidatePackage(t *testing.T) {
|
||||
wantStatusErr: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "starts with @",
|
||||
body: `{"package":"@scope/pkg"}`,
|
||||
wantEmpty: false,
|
||||
name: "starts with @",
|
||||
body: `{"package":"@scope/pkg"}`,
|
||||
wantEmpty: false,
|
||||
},
|
||||
// github: scheme — full-spec form (install + uninstall)
|
||||
{
|
||||
name: "github full spec",
|
||||
body: `{"package":"github:cli/cli"}`,
|
||||
wantEmpty: false,
|
||||
},
|
||||
{
|
||||
name: "github full spec with tag",
|
||||
body: `{"package":"github:cli/cli@v2.45.0"}`,
|
||||
wantEmpty: false,
|
||||
},
|
||||
// github: scheme — bare-name form (uninstall path: UI sends github:${pkg.name})
|
||||
{
|
||||
name: "github bare name (uninstall path)",
|
||||
body: `{"package":"github:gh"}`,
|
||||
wantEmpty: false,
|
||||
},
|
||||
{
|
||||
name: "github bare name with dots/hyphens",
|
||||
body: `{"package":"github:ripgrep-13.0"}`,
|
||||
wantEmpty: false,
|
||||
},
|
||||
// github: scheme — rejected forms (injection vectors)
|
||||
{
|
||||
name: "github empty after prefix",
|
||||
body: `{"package":"github:"}`,
|
||||
wantEmpty: true,
|
||||
wantStatusErr: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "github traversal",
|
||||
body: `{"package":"github:../evil"}`,
|
||||
wantEmpty: true,
|
||||
wantStatusErr: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "github with shell injection",
|
||||
body: `{"package":"github:evil;rm -rf"}`,
|
||||
wantEmpty: true,
|
||||
wantStatusErr: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "github with space",
|
||||
body: `{"package":"github:evil tool"}`,
|
||||
wantEmpty: true,
|
||||
wantStatusErr: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "github starts with hyphen",
|
||||
body: `{"package":"github:-flag"}`,
|
||||
wantEmpty: true,
|
||||
wantStatusErr: http.StatusBadRequest,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user