diff --git a/plans/260528-1844-passive-channel-memory-extraction/phase-01-research-and-characterization-tests.md b/plans/260528-1844-passive-channel-memory-extraction/phase-01-research-and-characterization-tests.md new file mode 100644 index 00000000..2c3883b0 --- /dev/null +++ b/plans/260528-1844-passive-channel-memory-extraction/phase-01-research-and-characterization-tests.md @@ -0,0 +1,87 @@ +--- +phase: 1 +title: "Research and Characterization Tests" +status: pending +priority: P1 +effort: "1d" +dependencies: [] +--- + +# Phase 1: Research and Characterization Tests + +## Context Links + +- Issue: `digitopvn/goclaw#64` +- Existing docs: `docs/06-store-data-model.md`, `docs/07-bootstrap-skills-memory.md`, `docs/09-security.md`, `docs/18-http-api.md` +- Channel buffer: `internal/channels/history.go`, `internal/store/pending_message_store.go` +- Memory/KG: `internal/consolidation/episodic_worker.go`, `internal/consolidation/semantic_worker.go` + +## Overview + +Lock current behavior with tests before changing schemas or workers. This phase proves the feature can reuse existing channel pending-message buffers and Memory/KG pipeline without breaking session memory. + +## Requirements + +- Functional: document exact current data flow for group pending messages, channel instance config update, episodic creation, KG extraction, and activity logging. +- Non-functional: tests first, no implementation until expected failures describe the desired behavior. + +## Architecture + +No runtime architecture added yet. The output is a verified map and failing tests for v1 contracts: + +1. channel instance config can carry `passive_memory` +2. pending messages are tenant/channel/history-key scoped +3. extraction candidates exclude DMs/private history in v1 +4. session-created episodic flow remains unchanged + +## Related Code Files + +- Read: `internal/channels/history.go` +- Read: `internal/channels/history_compaction.go` +- Read: `internal/store/pending_message_store.go` +- Read: `internal/store/channel_instance_store.go` +- Read: `internal/consolidation/episodic_worker.go` +- Read: `internal/consolidation/semantic_worker.go` +- Read: `internal/http/channel_instances.go` +- Create: `internal/consolidation/channel_memory_config_test.go` +- Create: `internal/consolidation/channel_memory_worker_test.go` +- Create: `internal/http/channel_memory_handlers_test.go` +- Create or extend: store tests under `internal/store/pg/` and `internal/store/sqlitestore/` + +## Implementation Steps + +1. Re-scout channel adapters to list which currently call `PendingHistory.Record`. +2. Re-scout `channel_pending_messages` PG/SQLite implementations and confirm tenant predicates. +3. Write failing config parser tests for default-disabled `passive_memory`. +4. Write failing candidate-selection tests: group-only, enabled-only, interval-or-cap trigger, minimum useful content gate. +5. Write failing redaction tests using representative secret/PII strings. +6. Write failing API tests for unauthorized settings/run/review actions. +7. Write a short `plans/.../reports/scout-report.md` with verified file/line anchors. + +## Todo List + +- [ ] Enumerate all pending-history recording call sites. +- [ ] Enumerate HTTP + WS channel instance update surfaces. +- [ ] Add failing config parser tests. +- [ ] Add failing worker trigger tests. +- [ ] Add failing redaction tests. +- [ ] Add failing admin API permission tests. +- [ ] Save scout report under this plan. + +## Success Criteria + +- [ ] Tests fail for missing passive-memory config, worker, queue, and handlers. +- [ ] Scout report lists exact files to modify and no fabricated APIs. +- [ ] Plan assumptions match live code after re-grep. + +## Risk Assessment + +Main risk is planning from stale scout notes. Mitigation: every claim in later phases must cite paths found in this phase, especially schema versions and route wiring. + +## Security Considerations + +Tests must include cross-tenant pending-message isolation and reject non-admin write access before any handler implementation exists. + +## Next Steps + +Proceed to schema/store only after tests describe the contracts. diff --git a/plans/260528-1844-passive-channel-memory-extraction/phase-02-schema-and-store-contracts.md b/plans/260528-1844-passive-channel-memory-extraction/phase-02-schema-and-store-contracts.md new file mode 100644 index 00000000..71af39b5 --- /dev/null +++ b/plans/260528-1844-passive-channel-memory-extraction/phase-02-schema-and-store-contracts.md @@ -0,0 +1,100 @@ +--- +phase: 2 +title: "Schema and Store Contracts" +status: pending +priority: P1 +effort: "2d" +dependencies: [1] +--- + +# Phase 2: Schema and Store Contracts + +## Context Links + +- Phase 1 tests and scout report +- PG migration latest during planning: `000073_secure_cli_credential_type.up.sql` +- PG schema version: `internal/upgrade/version.go` +- SQLite schema: `internal/store/sqlitestore/schema.sql`, `internal/store/sqlitestore/schema.go` +- Store registry: `internal/store/stores.go`, `internal/store/pg/factory.go`, `internal/store/sqlitestore/factory.go` + +## Overview + +Add durable run and review-item stores for passive extraction. Keep channel settings inside `channel_instances.config` to avoid another settings subsystem. + +## Requirements + +- Functional: persist extraction runs, pending extracted items, approval/rejection/deletion state, source ranges, redaction stats, and write metadata. +- Non-functional: tenant-scoped SQL, PG + SQLite parity, deterministic source IDs, no raw message body in extraction item after redaction. + +## Architecture + +Use two new store concepts: + +1. `ChannelMemoryExtractionRun`: one batch attempt per channel/history key. +2. `ChannelMemoryExtractionItem`: one extracted candidate fact or KG relation bundle pending review. + +`channel_instances.config.passive_memory` carries settings. Store layer exposes typed config helpers so HTTP/WS/UI do not mutate arbitrary JSON blindly. + +## Related Code Files + +- Create: `migrations/000074_channel_memory_extraction.up.sql` +- Create: `migrations/000074_channel_memory_extraction.down.sql` +- Modify: `internal/upgrade/version.go` +- Modify: `internal/store/sqlitestore/schema.sql` +- Modify: `internal/store/sqlitestore/schema.go` +- Create: `internal/store/channel_memory_extraction_store.go` +- Create: `internal/store/pg/channel_memory_extraction.go` +- Create: `internal/store/sqlitestore/channel_memory_extraction.go` +- Modify: `internal/store/stores.go` +- Modify: `internal/store/pg/factory.go` +- Modify: `internal/store/sqlitestore/factory.go` + +## Implementation Steps + +1. Add PG migration `000074` with tenant-scoped tables and indexes: + - `channel_memory_extraction_runs` + - `channel_memory_extraction_items` +2. Add unique constraints for idempotency: + - run: `(tenant_id, channel_instance_id, history_key, source_start_id, source_end_id)` + - item: `(tenant_id, run_id, item_hash)` +3. Add statuses: + - run: `pending`, `running`, `completed`, `failed`, `cancelled` + - item: `pending_review`, `approved`, `rejected`, `written`, `deleted` +4. Add source metadata fields: channel instance ID/name, history key, platform message ID range, time range, extractor run ID. +5. Add redaction summary fields: count, categories, skipped reason. +6. Add SQLite schema and migration map patch; bump `SchemaVersion` from 42 to 43. +7. Bump PG `RequiredSchemaVersion` from 73 to 74. +8. Implement PG and SQLite stores using existing scope patterns. +9. Wire stores into `store.Stores` and factories. +10. Make Phase 1 store tests pass. + +## Todo List + +- [ ] Add PG migration + version bump. +- [ ] Add SQLite schema + migration map + version bump. +- [ ] Add store interface types. +- [ ] Add PG store implementation. +- [ ] Add SQLite store implementation. +- [ ] Wire factories. +- [ ] Add idempotency tests. +- [ ] Add tenant isolation tests. + +## Success Criteria + +- [ ] Fresh PG migration applies and rolls back. +- [ ] Fresh SQLite DB contains tables. +- [ ] Existing SQLite DB migrates 42 to 43. +- [ ] Store tests pass for create/list/update/claim/status transitions. +- [ ] No SQL string concatenation with user input. + +## Risk Assessment + +Migration conflict risk is high on active `dev`. Before implementation, re-run `ls migrations/*.up.sql | sort | tail` and renumber if another migration claims `000074`. + +## Security Considerations + +All read/write methods require tenant context unless explicitly cross-tenant. Review item payload must store redacted text only; raw body stays in existing pending-message table subject to retention. + +## Next Steps + +Worker can claim runs only after store contracts exist. diff --git a/plans/260528-1844-passive-channel-memory-extraction/phase-03-extractor-worker-and-redaction.md b/plans/260528-1844-passive-channel-memory-extraction/phase-03-extractor-worker-and-redaction.md new file mode 100644 index 00000000..f15c5ca8 --- /dev/null +++ b/plans/260528-1844-passive-channel-memory-extraction/phase-03-extractor-worker-and-redaction.md @@ -0,0 +1,114 @@ +--- +phase: 3 +title: "Extractor Worker and Redaction" +status: pending +priority: P1 +effort: "2d" +dependencies: [1, 2] +--- + +# Phase 3: Extractor Worker and Redaction + +## Context Links + +- Existing compaction: `internal/channels/history_compaction.go` +- Existing KG extractor: `internal/knowledgegraph/extractor.go` +- Background provider resolver: `internal/providerresolve` +- Usage caps: `internal/usage/caps` +- Runtime wiring: `cmd/gateway.go` + +## Overview + +Build the passive extraction worker. It scans enabled channel configs, selects eligible message batches, redacts sensitive content, calls LLM extraction/summarization once per batch, and stores review items. + +## Requirements + +- Functional: interval/message-cap/manual trigger, group-only v1, minimum useful content gate, redaction before durable writes, retry-safe run status. +- Non-functional: bounded worker concurrency, cost controlled by batching, no panic on provider failure, graceful shutdown. + +## Architecture + +Add `internal/consolidation/channel_memory_worker.go` or a focused `internal/channelmemory/` package if the file would exceed 200 LOC. + +Data flow: + +1. scan `ChannelInstanceStore` for enabled channels with `config.passive_memory.enabled=true` +2. inspect `PendingMessageStore.ListGroups` +3. choose groups where interval elapsed or message cap reached +4. load `PendingMessageStore.ListByKey` +5. redact +6. create/claim extraction run +7. call extractor prompt for durable facts only +8. persist `pending_review` items +9. log activity + `slog.Info/Warn` + +## Related Code Files + +- Create: `internal/consolidation/channel_memory_config.go` +- Create: `internal/consolidation/channel_memory_redactor.go` +- Create: `internal/consolidation/channel_memory_worker.go` +- Create: `internal/consolidation/channel_memory_prompt.go` +- Modify: `internal/consolidation/workers.go` or gateway wiring if worker stays separate from existing worker bundle +- Modify: `cmd/gateway.go` +- Modify: `cmd/gateway_managed.go` if managed gateway has separate wiring + +## Implementation Steps + +1. Implement `PassiveMemoryConfig` parser with defaults: + - disabled + - review mode true + - interval 360 minutes + - cap 100 messages + - retention 168 hours + - allowed types default set +2. Implement redactor with tests for: + - API keys/tokens/passwords + - env-like secrets + - connection strings + - payment/banking numbers + - phone/address-like PII + - configured exclude user IDs and regex patterns +3. Implement candidate selector with tests: + - skip disabled + - skip DM/private keys by v1 policy + - skip too few useful messages + - trigger on interval or message cap +4. Implement LLM prompt and parser for durable facts: + - output JSON array of items + - allowed types only + - confidence score + - concise memory text + - optional entity/relation hints +5. Use `usageCaps.Chat` with purpose `channel-memory-extraction`. +6. Store run/item status transitions. +7. Emit audit activity for run started/completed/failed. +8. Wire worker start/stop in gateway with conservative ticker and concurrency. + +## Todo List + +- [ ] Config parser tests pass. +- [ ] Redactor tests pass. +- [ ] Candidate selector tests pass. +- [ ] Worker run lifecycle tests pass. +- [ ] Provider failure leaves run `failed` and does not block gateway. +- [ ] Usage cap purpose covered. +- [ ] Gateway wiring starts worker only when channels + stores are available. + +## Success Criteria + +- [ ] Worker creates review items from eligible group messages. +- [ ] No raw unredacted message text stored in new tables. +- [ ] Duplicate scans do not create duplicate runs/items. +- [ ] Provider failures are non-fatal and auditable. + +## Risk Assessment + +The redactor can false-positive and reduce extraction quality. Accept for v1. Better to drop sensitive content than store it. + +## Security Considerations + +Prompt must explicitly reject secrets and low-confidence guesses. Logs must include counts and IDs, not raw message body. + +## Next Steps + +Approved review items can write to Memory/KG after queue semantics are stable. diff --git a/plans/260528-1844-passive-channel-memory-extraction/phase-04-memory-kg-review-queue-integration.md b/plans/260528-1844-passive-channel-memory-extraction/phase-04-memory-kg-review-queue-integration.md new file mode 100644 index 00000000..f14be8f8 --- /dev/null +++ b/plans/260528-1844-passive-channel-memory-extraction/phase-04-memory-kg-review-queue-integration.md @@ -0,0 +1,99 @@ +--- +phase: 4 +title: "Memory KG Review Queue Integration" +status: pending +priority: P1 +effort: "2d" +dependencies: [2, 3] +--- + +# Phase 4: Memory KG Review Queue Integration + +## Context Links + +- Episodic store: `internal/store/episodic_store.go`, `internal/store/pg/episodic_summaries.go`, `internal/store/sqlitestore/episodic.go` +- Semantic worker: `internal/consolidation/semantic_worker.go` +- KG store: `internal/store/knowledge_graph_store.go` +- Activity store: `internal/store/activity_store.go` + +## Overview + +Add approve/reject/delete semantics and connect approved channel extraction items to existing episodic Memory and KG ingestion. Keep review queue default-on. + +## Requirements + +- Functional: list pending items, approve, reject, delete, manual write, idempotent re-approve, source metadata preserved. +- Non-functional: approved summaries become searchable like episodic memory; KG receives only approved/redacted text; all writes are tenant/agent scoped. + +## Architecture + +Approval writes one episodic summary with: + +- `SourceType`: `channel` +- `SourceID`: deterministic `channel::` +- `SessionKey`: synthetic, e.g. `channel::` +- `Summary`: approved redacted memory text +- `KeyTopics`: extracted normalized topics/entities +- `UserID`: approving admin or configured channel agent owner user, decided in implementation after re-checking current auth model + +Then publish `EventEpisodicCreated` so `semanticWorker` extracts KG facts through the existing path. + +## Related Code Files + +- Modify: `internal/consolidation/episodic_worker.go` only if helpers can be reused without changing session behavior. +- Create: `internal/consolidation/channel_memory_writer.go` +- Modify: `internal/eventbus/event_types.go` only if channel-specific payload is genuinely needed; prefer existing `EventEpisodicCreated`. +- Modify: new `ChannelMemoryExtractionStore` from Phase 2. +- Create: tests in `internal/consolidation/channel_memory_writer_test.go`. + +## Implementation Steps + +1. Write failing tests for approve idempotency: + - first approve creates episodic and marks item written/approved + - second approve no-ops + - rejected/deleted item cannot be approved unless restored is explicitly implemented, which v1 does not need +2. Implement writer service that depends on: + - `ChannelMemoryExtractionStore` + - `EpisodicStore` + - `DomainEventBus` + - `ActivityStore` +3. Create episodic summary from approved item with deterministic source ID. +4. Publish existing `EventEpisodicCreated` with redacted summary. +5. Record activity: + - `channel_memory.item.approved` + - `channel_memory.item.rejected` + - `channel_memory.item.deleted` + - `channel_memory.item.written` +6. Add delete/forget behavior: + - v1 deletes queued item if not written + - if written, mark deleted and delete associated episodic summary when source ID matches + - KG deletion for already-ingested facts may be best-effort by source metadata if existing KG store supports it; otherwise flag as v1 limitation in UI/docs + +## Todo List + +- [ ] Add writer service tests. +- [ ] Implement approve/reject/delete transitions. +- [ ] Implement episodic write. +- [ ] Publish existing semantic event. +- [ ] Add activity logs. +- [ ] Verify session episodic tests remain unchanged. + +## Success Criteria + +- [ ] Approved item appears in episodic search. +- [ ] KG extraction path is triggered through existing event bus. +- [ ] Re-approve cannot duplicate episodic/KG data. +- [ ] Reject/delete are auditable. +- [ ] Tenant mismatch tests fail closed. + +## Risk Assessment + +Deleting KG facts after approval may require source-level relation tracking not present today. If exact deletion is not supported, v1 must be honest: delete queued item + episodic source, and mark KG cleanup as best-effort/manual dedup. + +## Security Considerations + +Approval endpoint must require tenant admin for tenant-scoped channel rows. Master-scope owner is not required because rows are tenant-scoped. + +## Next Steps + +Expose the review queue and settings through API/UI. diff --git a/plans/260528-1844-passive-channel-memory-extraction/phase-05-channel-settings-ui-and-admin-controls.md b/plans/260528-1844-passive-channel-memory-extraction/phase-05-channel-settings-ui-and-admin-controls.md new file mode 100644 index 00000000..a251b7eb --- /dev/null +++ b/plans/260528-1844-passive-channel-memory-extraction/phase-05-channel-settings-ui-and-admin-controls.md @@ -0,0 +1,111 @@ +--- +phase: 5 +title: "Channel Settings UI and Admin Controls" +status: pending +priority: P2 +effort: "2d" +dependencies: [2, 3, 4] +--- + +# Phase 5: Channel Settings UI and Admin Controls + +## Context Links + +- HTTP channel handler: `internal/http/channel_instances.go` +- WS channel methods: `internal/gateway/methods/channel_instances.go` +- Channel UI: `ui/web/src/pages/channels/channels-page.tsx`, `ui/web/src/pages/channels/channel-detail/` +- UI types: `ui/web/src/types/channel.ts` +- i18n: `ui/web/src/i18n/locales/{en,vi,zh}/channels.json` + +## Overview + +Expose passive memory settings and review queue in channel detail. Keep the UI operator-focused: no global dashboard in v1. + +## Requirements + +- Functional: configure, manually run extraction, show last run, show counts, approve/reject/delete pending items. +- Non-functional: mobile-safe controls, no raw secret/PII display beyond already-redacted item text, i18n in EN/VI/ZH. + +## Architecture + +HTTP endpoints under channel instance scope: + +- `GET /v1/channels/instances/{id}/memory-extraction` +- `PUT /v1/channels/instances/{id}/memory-extraction/settings` +- `POST /v1/channels/instances/{id}/memory-extraction/run` +- `GET /v1/channels/instances/{id}/memory-extraction/items` +- `POST /v1/channels/instances/{id}/memory-extraction/items/{itemID}/approve` +- `POST /v1/channels/instances/{id}/memory-extraction/items/{itemID}/reject` +- `DELETE /v1/channels/instances/{id}/memory-extraction/items/{itemID}` + +WS methods are optional in v1. Prefer HTTP because current web hooks already use HTTP for channel instances. + +## Related Code Files + +- Modify: `internal/http/channel_instances.go` +- Create: `internal/http/channel_memory_extraction_handlers.go` +- Create: `internal/http/channel_memory_extraction_handlers_test.go` +- Modify: `ui/web/src/types/channel.ts` +- Create: `ui/web/src/pages/channels/channel-detail/passive-memory-section.tsx` +- Create: `ui/web/src/pages/channels/hooks/use-channel-memory-extraction.ts` +- Modify: `ui/web/src/pages/channels/channel-detail/channel-detail-page.tsx` +- Modify: locale files in `ui/web/src/i18n/locales/en/`, `vi/`, `zh/` + +## Implementation Steps + +1. Add HTTP handler tests first: + - viewer can read status/items + - admin required for settings/run/approve/reject/delete + - tenant mismatch returns not found/forbidden without leaking row existence +2. Add settings validation: + - interval min/max + - message cap min/max + - retention min/max + - allowed types from enum + - exclude patterns bounded length/count +3. Add routes to `ChannelInstancesHandler.RegisterRoutes`. +4. Implement UI hook with TanStack Query and invalidation on mutation. +5. Build `PassiveMemorySection`: + - toggle + - interval and cap numeric inputs + - review mode toggle + - allowed type checkboxes + - last run summary + - manual run button + - pending extracted items list + - approve/reject/delete buttons +6. Add i18n keys to EN/VI/ZH channel namespace. +7. Ensure mobile rules: + - inputs use `text-base md:text-sm` + - tables/lists horizontally safe + - buttons have sufficient hit area + +## Todo List + +- [ ] Add HTTP route tests. +- [ ] Implement handler validation. +- [ ] Add UI hook. +- [ ] Add channel detail section. +- [ ] Add i18n strings. +- [ ] Add basic UI tests if existing setup supports it; otherwise rely on build + manual browser verification. + +## Success Criteria + +- [ ] Admin can enable passive memory for a channel. +- [ ] Admin can force a run. +- [ ] Pending items appear after extraction. +- [ ] Admin can approve/reject/delete items. +- [ ] Viewer cannot mutate settings/items. +- [ ] Web build passes. + +## Risk Assessment + +UI can balloon. Keep it as one compact channel-detail section. Do not add global review inbox until v1 proves useful. + +## Security Considerations + +UI must show redaction status and avoid displaying raw source messages. If source preview is needed later, it must be separate permissioned work. + +## Next Steps + +Run full verification, docs, and issue handoff. diff --git a/plans/260528-1844-passive-channel-memory-extraction/phase-06-verification-documentation-and-ship-readiness.md b/plans/260528-1844-passive-channel-memory-extraction/phase-06-verification-documentation-and-ship-readiness.md new file mode 100644 index 00000000..ea932228 --- /dev/null +++ b/plans/260528-1844-passive-channel-memory-extraction/phase-06-verification-documentation-and-ship-readiness.md @@ -0,0 +1,99 @@ +--- +phase: 6 +title: "Verification Documentation and Ship Readiness" +status: pending +priority: P1 +effort: "1d" +dependencies: [1, 2, 3, 4, 5] +--- + +# Phase 6: Verification Documentation and Ship Readiness + +## Context Links + +- Post-implementation checklist in `CLAUDE.md` +- Docs to update: `docs/05-channels-messaging.md`, `docs/06-store-data-model.md`, `docs/07-bootstrap-skills-memory.md`, `docs/09-security.md`, `docs/18-http-api.md`, `docs/project-changelog.md` +- Local verification memory: `go build ./...`, `go build -tags sqliteonly ./...`, `go vet ./...` + +## Overview + +Prove the feature is safe, documented, and ready for implementation PR review. This phase owns the final consistency sweep and release notes. + +## Requirements + +- Functional: all new tests pass; user-facing docs describe opt-in, review queue, redaction, retention, deletion limits. +- Non-functional: compile in PG and SQLite builds, no known tenant isolation gaps, no stale plan references in code comments. + +## Architecture + +No new architecture. This phase verifies all layers together: + +1. schema migration +2. store contracts +3. worker batching/redaction +4. review queue writer +5. HTTP/UI controls +6. docs + +## Related Code Files + +- Modify: `docs/05-channels-messaging.md` +- Modify: `docs/06-store-data-model.md` +- Modify: `docs/07-bootstrap-skills-memory.md` +- Modify: `docs/09-security.md` +- Modify: `docs/18-http-api.md` +- Modify: `docs/project-changelog.md` +- Possibly modify: `docs/codebase-summary.md` + +## Implementation Steps + +1. Run focused Go tests: + - `go test ./internal/consolidation` + - `go test ./internal/http` + - `go test ./internal/store/pg` + - `go test -tags sqliteonly ./internal/store/sqlitestore` +2. Run compile/static checks: + - `go build ./...` + - `go build -tags sqliteonly ./...` + - `go vet ./...` +3. Run web checks: + - `cd ui/web && pnpm build` +4. If UI changed, run local dev server and browser-check channel detail at desktop/mobile widths. +5. Update docs listed above. +6. Update `docs/project-changelog.md`. +7. Whole-plan consistency sweep: + - search code and docs for stale names such as `passiveExtraction`, `channel_memory`, `memory-extraction` + - verify route names match docs and UI hooks + - verify schema version numbers match migrations +8. Prepare PR summary with test results and privacy caveats. + +## Todo List + +- [ ] Focused tests pass. +- [ ] PG build passes. +- [ ] SQLite build passes. +- [ ] Vet passes. +- [ ] Web build passes. +- [ ] Browser verification done if UI implemented. +- [ ] Docs updated. +- [ ] Changelog updated. +- [ ] Whole-plan consistency sweep clean. + +## Success Criteria + +- [ ] Feature satisfies all issue #64 acceptance criteria selected for v1. +- [ ] No failing tests ignored. +- [ ] Any unsupported v1 behavior documented explicitly. +- [ ] PR is ready for code review. + +## Risk Assessment + +Integration tests with PG/pgvector may require local service. If unavailable, document exactly which integration tests were skipped and why; do not claim full verification. + +## Security Considerations + +Final review must include a targeted scan for raw message body persistence, secrets in logs, missing tenant predicates, and unauthorized mutation paths. + +## Next Steps + +After plan approval, execute with `/ck:cook ` or equivalent implementation workflow. diff --git a/plans/260528-1844-passive-channel-memory-extraction/plan.md b/plans/260528-1844-passive-channel-memory-extraction/plan.md new file mode 100644 index 00000000..68330428 --- /dev/null +++ b/plans/260528-1844-passive-channel-memory-extraction/plan.md @@ -0,0 +1,92 @@ +--- +title: "Passive Channel Memory Extraction" +description: "Add disabled-by-default, reviewable passive extraction from group channel buffers into episodic Memory and Knowledge Graph." +status: pending +priority: P1 +issue: 64 +branch: "codex/issue-64-passive-channel-memory-extraction" +tags: [channels, memory, knowledge-graph, privacy, tdd, issue-64] +blockedBy: [] +blocks: [] +created: "2026-05-28T11:45:20.128Z" +createdBy: "ck:plan" +source: skill +--- + +# Passive Channel Memory Extraction + +## Overview + +Implement issue #64 as a privacy-first v1. Agents may learn durable context from group/channel conversations they already receive, but only after a channel admin enables the feature. Extraction is batched, redacted, reviewable, tenant-scoped, and writes to Memory/KG only after approval or explicit high-confidence auto-approve. + +Hard boundary: no realtime per-message extraction, no default enablement, no permanent raw full-history store, and no cross-channel memory sharing. + +## Current Codebase Anchors + +| Area | Existing anchor | +|------|-----------------| +| Channel instances | `channel_instances.config`, `internal/store/channel_instance_store.go`, `internal/http/channel_instances.go`, `internal/gateway/methods/channel_instances.go` | +| Group buffer | `channel_pending_messages`, `internal/store/pending_message_store.go`, `internal/channels/history.go`, `internal/channels/history_compaction.go` | +| Memory/KG pipeline | `internal/consolidation/episodic_worker.go`, `internal/consolidation/semantic_worker.go`, `internal/knowledgegraph/extractor.go` | +| Activity audit | `internal/store/activity_store.go`, `internal/http/activity.go`, `emitAudit(...)` call sites | +| Runtime wiring | `cmd/gateway.go`, `cmd/gateway_http_handlers.go`, `cmd/gateway_http_wiring.go`, `cmd/gateway_channels_setup.go` | +| Web UI | `ui/web/src/pages/channels/`, `ui/web/src/pages/memory/`, `ui/web/src/types/channel.ts` | +| Schema versions | PG next migration `000074`; bump `internal/upgrade/version.go` from 73. SQLite bump `internal/store/sqlitestore/schema.go` from 42 to 43 | + +## Phases + +| Phase | Name | Status | +|-------|------|--------| +| 1 | [Research and Characterization Tests](./phase-01-research-and-characterization-tests.md) | Pending | +| 2 | [Schema and Store Contracts](./phase-02-schema-and-store-contracts.md) | Pending | +| 3 | [Extractor Worker and Redaction](./phase-03-extractor-worker-and-redaction.md) | Pending | +| 4 | [Memory KG Review Queue Integration](./phase-04-memory-kg-review-queue-integration.md) | Pending | +| 5 | [Channel Settings UI and Admin Controls](./phase-05-channel-settings-ui-and-admin-controls.md) | Pending | +| 6 | [Verification Documentation and Ship Readiness](./phase-06-verification-documentation-and-ship-readiness.md) | Pending | + +## Dependencies + +- GitHub issue: `digitopvn/goclaw#64` +- Existing group pending-message persistence must stay intact for channel context. +- Existing Memory/KG workers must keep session behavior unchanged. +- Standard edition only for channels. Lite has no channels, so SQLite migrations still must compile/apply, but UI should naturally hide channel surfaces through existing edition gates. + +## Recommended Defaults + +| Setting | Default | Reason | +|---------|---------|--------| +| `enabled` | `false` | Sensitive feature; explicit opt-in only | +| `review_mode` | `true` | Prevent accidental durable PII/secret storage | +| `interval_minutes` | `360` | Cheap daily-ish cadence without long wait | +| `message_cap` | `100` | Batching per issue request | +| `retention_hours` | `168` | 7-day raw buffer cap, same spirit as pending-message TTL | +| `allowed_types` | people, projects, decisions, todos, preferences, events | Durable work context only | +| `group_only` | `true` for v1 | Avoid DM privacy ambiguity | + +## Success Criteria + +- [ ] Passive extraction is disabled by default and opt-in per channel instance. +- [ ] Extraction runs only by interval, message cap, or manual trigger. +- [ ] Raw channel messages are read from tenant-scoped pending buffers and are not stored permanently outside existing retention. +- [ ] Redaction blocks common secrets, tokens, credentials, payment/banking data, phone/address patterns, and configurable excluded users/patterns before durable writes. +- [ ] Extracted items go to a review queue by default. +- [ ] Approved items write to `episodic_summaries` with `source_type='channel'` and deterministic `source_id`. +- [ ] KG ingestion receives only approved/redacted summaries and records source metadata. +- [ ] Review reject/delete removes queued items and prevents later writes. +- [ ] Tenant/channel/agent boundaries covered by tests. +- [ ] PG and SQLite schemas both migrate and compile. +- [ ] UI exposes settings, last run status, pending items, approve/reject/delete, and manual run. + +## Key Risks + +| Risk | Mitigation | +|------|------------| +| Durable sensitive-data leak | Redaction before queue, review default, no raw full-history durable copy | +| Duplicate facts | Deterministic run/item source IDs, store-level uniqueness, reuse KG dedup worker | +| Cross-tenant leak | Tenant-scoped queries and tests before implementation | +| Cost spike | Batched triggers, minimum content gate, usage caps purpose string | +| UX overbuild | Channel detail section only in v1; no separate global console | + +## Open Questions + +- None for v1. Assumptions: review queue required, group channels first, UI in channel detail.