mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-11 03:13:24 +00:00
* feat(webhooks): HTTP webhooks to trigger agents with HMAC auth and durable callbacks
Add multi-tenant HTTP webhook endpoints for agent triggering:
- /v1/webhooks/message: send messages to channels
- /v1/webhooks/llm: sync/async LLM prompts with HMAC-signed callbacks
- HMAC-256 + bearer token authentication
- Rate limiting and tenant isolation
- Durable callback worker with exponential backoff
- PG 000056 + SQLite schema v25 migrations
- Unit + integration tests, P0 tenant isolation invariants
- Channel media capability helpers for attachment routing
- Comprehensive webhook documentation and i18n strings
* fix(webhooks): address post-review findings (K1-K10)
Comprehensive post-merge fixes addressing 10 blocking code review issues
and 2 adversarial re-audit findings in webhook-agent-triggering feature:
K1: Fix auth middleware tenant context lookup sequencing — move
tenant context injection before authenticate() call to prevent
unscoped secret lookups.
K2: Canonicalize JSON payload format for jsonb compatibility across
PostgreSQL and SQLite — ensure consistent serialization without
whitespace variance to prevent hash mismatches.
K3: Add fail-closed JSON parsing in body hash extraction with explicit
error handling for malformed payloads before HMAC verification.
K4: Fix worker queue wedge by properly draining slot reservations
when delivery succeeds, preventing permanent slot occupancy.
K5: Implement lease-token optimistic concurrency control to prevent
duplicate webhook delivery under high concurrency or retry storms.
K6: Add AES-256-GCM encrypted secret storage at rest with fail-fast
skip-mount when GOCLAW_ENCRYPTION_KEY environment variable unset.
K7: Implement IP allowlist enforcement supporting both CIDR ranges
and exact IP matching with proper X-Forwarded-For parsing.
K8: Add HMAC replay nonce cache (5min expiry, non-blocking async flush)
to prevent request replay attacks on webhook handler.
K9: Fix invariant test schema selection — replace hardcoded assumption
with explicit schema name from config to support multi-schema testing.
K10: Consolidate rate limiters into single shared instance to prevent
per-endpoint limiter starvation and ensure fair rate limiting.
New database migrations:
- 000057: webhook_calls.lease_token for optimistic concurrency
- 000058: webhooks.encrypted_secret_key for AES-256-GCM encryption
New i18n keys: MsgWebhookIPDenied, MsgWebhookEncryptionUnavailable
(with English, Vietnamese, Chinese translations).
New modules:
- internal/http/webhooks_payload.go: JSON canonicalization + body hash
- internal/http/webhooks_nonce.go: Replay nonce cache implementation
- internal/http/webhooks_idempotency_test.go: Integration tests
Documentation updates:
- docs/webhooks.md: §13-14 security sections, encryption flow
- docs/00-architecture-overview.md: webhook subsystem security overview
- docs/codebase-summary.md: webhook security patterns
- docs/project-changelog.md: webhook fixes changelog
Test coverage: 53 webhook tests + 4 P0 invariant tests all passing.
No tenant isolation violations. All security gates enforced.
* docs(journals): webhook feature ship + fix cycle entries
* fix(webhooks): address Claude review findings
- webhooks_llm.go: remove misleading ptr() helper; use &completedAt
pattern for error-path audit rows (matches success path)
- webhooks_auth.go: wrap TouchLastUsed context in WithoutCancel so
background DB update isn't cancelled when HTTP response completes
- store GetByIDUnscoped (PG+SQLite): add NOT revoked / revoked = 0
filter for defense-in-depth parity with GetByHashUnscoped
- webhooks/sign.go: fix package doc — HMAC key is raw plaintext
secret bytes, not hex-decoded SHA-256
- webhooks_admin.go: check auth before encKey guard to avoid leaking
config state to unauthenticated callers
- webhooks_ratelimit.go: two-phase Load→LoadOrStore to avoid per-call
entry allocation on the hot path
* docs(webhooks): fix Sign() function doc to match actual key input
Function-level comment still referenced hex-decoded SecretHash after
the package-level doc was corrected. Align with actual caller usage
([]byte(rawSecret)).
* fix(webhooks): use WithoutCancel for worker execute DB updates
Terminal status writes in execute() ran through the worker main-loop
ctx, which is cancelled on graceful shutdown. If the outbound send
completed but the status update raced with shutdown, the row stayed
in 'running' and got re-delivered via reclaimStale. WithoutCancel
lets the DB write survive worker cancellation while preserving
propagated values (tenant ID, etc.).
* fix(webhooks): move tctx init before panic defer in worker execute
Panic recovery called updateRetry with raw ctx (no tenant ID), making
requireTenantID fail and the reset-to-retry DB write silently drop.
Row stayed 'running' until reclaimStale (~90s delay). Init tctx first
so defer closure captures tenant-scoped non-cancellable context.
* fix(webhooks): pass tenant-scoped tctx to invokeAgent in worker
execute() was passing the raw worker-loop ctx (no tenant ID) to
invokeAgent → router.Get → PGAgentStore.GetByID. GetByID reads
TenantIDFromContext which returned uuid.Nil, making every lookup
return 'agent not found'. Async LLM webhook calls silently failed
all retries. Pass tctx (already tenant-scoped + WithoutCancel) so
the router resolves the agent correctly.
* fix(tests): resolve integration test compile errors
- Remove duplicate contains() in mcp_grant_revoke_test.go (already
defined in tts_gemini_live_test.go)
- Update webhooks_admin_test.go RotateSecret call to match current
5-arg signature (newSecretHash, newPrefix, newEncryptedSecret)
* fix(webhooks): default nil scopes/ip_allowlist to empty slice in Create
PG columns are NOT NULL DEFAULT '{}'. Explicit NULL from pqStringArray(nil)
violated the constraint, breaking TestWebhookAdminCRUD/TenantIsolation.
Coerce nil slices to empty []string{} so the default applies at the DB layer.
* chore: trigger CI on digitopvn/goclaw fork
* ci: retrigger workflows
* fix(webhooks): renumber migrations to 000059-000061 for merge train
219 lines
6.4 KiB
Go
219 lines
6.4 KiB
Go
//go:build integration
|
|
|
|
package invariants
|
|
|
|
import (
|
|
"crypto/sha256"
|
|
"database/sql"
|
|
"encoding/hex"
|
|
"testing"
|
|
|
|
"github.com/google/uuid"
|
|
|
|
"github.com/nextlevelbuilder/goclaw/internal/store"
|
|
"github.com/nextlevelbuilder/goclaw/internal/store/pg"
|
|
)
|
|
|
|
// webhookListFilter returns a zero-value filter (list all webhooks for the tenant in context).
|
|
func webhookListFilter() store.WebhookListFilter {
|
|
return store.WebhookListFilter{}
|
|
}
|
|
|
|
// seedWebhook creates a webhook for a tenant.
|
|
func seedWebhook(t *testing.T, db *sql.DB, tenantID uuid.UUID, kind string) uuid.UUID {
|
|
t.Helper()
|
|
|
|
webhookID := uuid.New()
|
|
rawSecret := "wh_secret_" + webhookID.String()[:8]
|
|
h := sha256.Sum256([]byte(rawSecret))
|
|
hashHex := hex.EncodeToString(h[:])
|
|
|
|
_, err := db.Exec(`
|
|
INSERT INTO webhooks (id, tenant_id, name, kind, secret_prefix, secret_hash)
|
|
VALUES ($1, $2, $3, $4, $5, $6)
|
|
`, webhookID, tenantID, "test-webhook-"+webhookID.String()[:8], kind, "wh_test", hashHex)
|
|
if err != nil {
|
|
t.Fatalf("seed webhook: %v", err)
|
|
}
|
|
|
|
return webhookID
|
|
}
|
|
|
|
// P0: TestWebhookTenantIsolationListGet ensures no tenant can list/get another tenant's webhook.
|
|
func TestWebhookTenantIsolationListGet(t *testing.T) {
|
|
db := testDB(t)
|
|
|
|
// Seed 2 independent tenants with their webhooks.
|
|
tenantA, _ := seedTenantAgent(t, db)
|
|
tenantB, _ := seedTenantAgent(t, db)
|
|
|
|
webhookAID := seedWebhook(t, db, tenantA, "llm")
|
|
webhookBID := seedWebhook(t, db, tenantB, "message")
|
|
|
|
store := pg.NewPGWebhookStore(db)
|
|
|
|
ctxA := tenantCtx(tenantA)
|
|
ctxB := tenantCtx(tenantB)
|
|
|
|
// Tenant A lists webhooks — should only see their own.
|
|
listA, err := store.List(ctxA, webhookListFilter())
|
|
if err != nil {
|
|
t.Fatalf("Tenant A list failed: %v", err)
|
|
}
|
|
|
|
for _, w := range listA {
|
|
if w.TenantID != tenantA {
|
|
t.Errorf("P0 VIOLATION: Tenant A listed webhook with tenant_id=%v (not %v)", w.TenantID, tenantA)
|
|
}
|
|
if w.ID == webhookBID {
|
|
t.Errorf("P0 VIOLATION: Tenant A listed Tenant B's webhook")
|
|
}
|
|
}
|
|
|
|
// Tenant B lists webhooks — should only see their own.
|
|
listB, err := store.List(ctxB, webhookListFilter())
|
|
if err != nil {
|
|
t.Fatalf("Tenant B list failed: %v", err)
|
|
}
|
|
|
|
for _, w := range listB {
|
|
if w.TenantID != tenantB {
|
|
t.Errorf("P0 VIOLATION: Tenant B listed webhook with tenant_id=%v (not %v)", w.TenantID, tenantB)
|
|
}
|
|
if w.ID == webhookAID {
|
|
t.Errorf("P0 VIOLATION: Tenant B listed Tenant A's webhook")
|
|
}
|
|
}
|
|
|
|
// Tenant B tries to GET Tenant A's webhook.
|
|
_, err = store.GetByID(ctxB, webhookAID)
|
|
if err != sql.ErrNoRows {
|
|
t.Errorf("P0 VIOLATION: Tenant B was able to GetByID Tenant A's webhook (expected ErrNoRows, got %v)", err)
|
|
}
|
|
}
|
|
|
|
// P0: TestWebhookTenantIsolationRotateRevoke ensures no tenant can rotate/revoke another's webhook.
|
|
func TestWebhookTenantIsolationRotateRevoke(t *testing.T) {
|
|
db := testDB(t)
|
|
|
|
tenantA, _ := seedTenantAgent(t, db)
|
|
tenantB, _ := seedTenantAgent(t, db)
|
|
|
|
webhookAID := seedWebhook(t, db, tenantA, "llm")
|
|
|
|
whs := pg.NewPGWebhookStore(db)
|
|
|
|
ctxA := tenantCtx(tenantA)
|
|
ctxB := tenantCtx(tenantB)
|
|
|
|
// Get the original webhook.
|
|
origWH, err := whs.GetByID(ctxA, webhookAID)
|
|
if err != nil {
|
|
t.Fatalf("Tenant A get their webhook: %v", err)
|
|
}
|
|
origHash := origWH.SecretHash
|
|
|
|
// Tenant B tries to rotate Tenant A's webhook secret.
|
|
newHash := "newsecret_hash_" + uuid.New().String()[:8]
|
|
newPrefix := "wh_newprefix"
|
|
newEncrypted := "encrypted_secret_b64_payload"
|
|
err = whs.RotateSecret(ctxB, webhookAID, newHash, newPrefix, newEncrypted)
|
|
if err == nil {
|
|
// This is a P0 violation — the rotate should have failed (ErrNoRows or equivalent).
|
|
t.Errorf("P0 VIOLATION: Tenant B was able to rotate Tenant A's webhook secret")
|
|
|
|
// Verify it actually changed (worse violation).
|
|
updated, _ := whs.GetByID(ctxA, webhookAID)
|
|
if updated.SecretHash != origHash {
|
|
t.Errorf("P0 VIOLATION: Secret hash actually changed when Tenant B called RotateSecret")
|
|
}
|
|
}
|
|
|
|
// Tenant B tries to revoke Tenant A's webhook.
|
|
err = whs.Revoke(ctxB, webhookAID)
|
|
if err == nil {
|
|
// Check if it actually revoked.
|
|
updated, _ := whs.GetByID(ctxA, webhookAID)
|
|
if updated.Revoked {
|
|
t.Errorf("P0 VIOLATION: Tenant B was able to revoke Tenant A's webhook")
|
|
}
|
|
}
|
|
}
|
|
|
|
// P0: TestWebhookTenantIsolationUpdate ensures no tenant can update another's webhook.
|
|
func TestWebhookTenantIsolationUpdate(t *testing.T) {
|
|
db := testDB(t)
|
|
|
|
tenantA, _ := seedTenantAgent(t, db)
|
|
tenantB, _ := seedTenantAgent(t, db)
|
|
|
|
webhookAID := seedWebhook(t, db, tenantA, "llm")
|
|
|
|
whs := pg.NewPGWebhookStore(db)
|
|
|
|
ctxA := tenantCtx(tenantA)
|
|
ctxB := tenantCtx(tenantB)
|
|
|
|
// Get original rate limit.
|
|
origWH, err := whs.GetByID(ctxA, webhookAID)
|
|
if err != nil {
|
|
t.Fatalf("get original webhook: %v", err)
|
|
}
|
|
origRPM := origWH.RateLimitPerMin
|
|
|
|
// Tenant B tries to update Tenant A's rate limit.
|
|
err = whs.Update(ctxB, webhookAID, map[string]any{
|
|
"rate_limit_per_min": 999,
|
|
})
|
|
if err == nil {
|
|
// Check if it actually updated.
|
|
updated, _ := whs.GetByID(ctxA, webhookAID)
|
|
if updated.RateLimitPerMin != origRPM {
|
|
t.Errorf("P0 VIOLATION: Tenant B was able to update Tenant A's rate_limit_per_min from %d to %d",
|
|
origRPM, updated.RateLimitPerMin)
|
|
}
|
|
}
|
|
}
|
|
|
|
// P0: TestWebhookTenantIsolationGetByHash ensures GetByHash never returns cross-tenant webhook.
|
|
func TestWebhookTenantIsolationGetByHash(t *testing.T) {
|
|
db := testDB(t)
|
|
|
|
tenantA, _ := seedTenantAgent(t, db)
|
|
tenantB, _ := seedTenantAgent(t, db)
|
|
|
|
// Create webhooks with known secrets.
|
|
webhookAID := uuid.New()
|
|
secretA := "wh_secret_a_" + webhookAID.String()[:8]
|
|
hA := sha256.Sum256([]byte(secretA))
|
|
hashA := hex.EncodeToString(hA[:])
|
|
|
|
_, err := db.Exec(`
|
|
INSERT INTO webhooks (id, tenant_id, name, kind, secret_prefix, secret_hash)
|
|
VALUES ($1, $2, $3, 'llm', 'wh_test', $4)
|
|
`, webhookAID, tenantA, "test-webhook-"+webhookAID.String()[:8], hashA)
|
|
if err != nil {
|
|
t.Fatalf("seed webhook A: %v", err)
|
|
}
|
|
|
|
whs := pg.NewPGWebhookStore(db)
|
|
|
|
ctxA := tenantCtx(tenantA)
|
|
ctxB := tenantCtx(tenantB)
|
|
|
|
// Tenant A gets webhook by hash — should succeed.
|
|
whA, err := whs.GetByHash(ctxA, hashA)
|
|
if err != nil {
|
|
t.Fatalf("Tenant A GetByHash failed: %v", err)
|
|
}
|
|
if whA.TenantID != tenantA {
|
|
t.Errorf("Tenant A retrieved webhook with wrong tenant_id: %v", whA.TenantID)
|
|
}
|
|
|
|
// Tenant B gets same hash — should fail (tenant_id check in query).
|
|
whB, err := whs.GetByHash(ctxB, hashA)
|
|
if err != sql.ErrNoRows {
|
|
t.Errorf("P0 VIOLATION: Tenant B GetByHash succeeded (expected ErrNoRows, got %v, webhook=%v)", err, whB)
|
|
}
|
|
}
|