feat(channel-memory): document passive extraction

2026-10-15 06:16:10 +00:00 · 2026-05-29 20:48:45 +07:00
1 parent 97b9955a1a
commit 34bb6121b8
@@ -632,7 +632,32 @@ Channel instances are loaded from the database with their assigned agent ID. The

 ---

-## 13. Local Key Propagation
+## 13. Passive Memory Extraction
+
+Passive channel memory is an opt-in per-channel feature. When enabled in
+`channel_instances.config.passive_memory`, the gateway periodically reads the
+existing tenant-scoped `channel_pending_messages` group buffers, redacts sensitive
+content, asks the background LLM provider for durable fact candidates, and stores
+only candidates in the review queue.
+
+Default behavior is privacy-first:
+
+- Disabled by default.
+- Group-only in v1.
+- Review mode enabled by default.
+- Runs by manual trigger, message cap, or interval.
+- New extraction tables store metadata, summaries, topics/entities, confidence,
+  status, and redaction counts, but not raw message bodies.
+
+Approved items write an `episodic_summaries` row with `source_type='channel'`
+and a deterministic `source_id`; existing consolidation workers then handle KG
+promotion. Reject/delete prevents later writes. Delete also removes the linked
+episodic row when one exists; already-promoted KG nodes are not synchronously
+deleted in v1.
+
+---
+
+## 14. Local Key Propagation

 Thread/topic context is preserved through the entire message pipeline using a `local_key` in message metadata. This ensures subagent, delegation, and team message results land in the correct thread — not the root chat.

@@ -648,7 +673,7 @@ All channel state — placeholders, streams, reactions, typing controllers, thre

 ---

-## 14. Per-User Isolation
+## 15. Per-User Isolation

 Channels provide per-user isolation through compound sender IDs and context propagation:

@@ -659,7 +684,7 @@ Channels provide per-user isolation through compound sender IDs and context prop

 ---

-## 15. Pairing System
+## 16. Pairing System

 The pairing system provides a DM authentication flow for channels using the `pairing` DM policy.

@@ -728,6 +728,13 @@ L0 (Working Memory)           L1 (Episodic Memory)        L2 (Semantic Memory)
 | `vault_versions` | Document version history (prepared for v3.1) | `doc_id`, `version`, `content`, `changed_by`, `created_at` |
 | `kg_entities` | Extended with temporal columns | `valid_from` (TIMESTAMPTZ), `valid_until` (TIMESTAMPTZ) for temporal facts |
 | `kg_relations` | Extended with temporal columns | `valid_from` (TIMESTAMPTZ), `valid_until` (TIMESTAMPTZ) for temporal edges |
+| `channel_memory_extraction_runs` | Passive channel extraction run log | `tenant_id`, `channel_instance_id`, `history_key`, `trigger`, `status`, source range, counts, redaction metadata |
+| `channel_memory_extraction_items` | Review queue for passive channel memory candidates | `tenant_id`, `run_id`, `channel_instance_id`, `item_hash`, `item_type`, `summary`, `topics`, `entities`, `status`, approval/write timestamps |
+
+`ChannelMemoryExtractionStore` is implemented for PostgreSQL and SQLite. It is
+tenant-scoped, stores no raw message bodies, and uses deterministic hashes to
+deduplicate the same channel/history/type/summary candidate across repeated
+runs.

 ### 12 Promoted Agent Columns

@@ -596,6 +596,13 @@ flowchart LR
 7. Stores in `episodic_summaries`
 8. Publishes `episodic.created` for downstream workers

+**Passive channel memory** (`internal/channelmemory`):
+1. Reads existing channel pending-message groups only when a channel admin enables `passive_memory.enabled`
+2. Redacts secrets, tokens, connection strings, payment-like numbers, emails, phones, and configured excluded users/patterns
+3. Writes extracted candidates to `channel_memory_extraction_items` for review by default
+4. On approval, creates an `episodic_summaries` row with `source_type='channel'`
+5. Publishes `episodic.created` so SemanticWorker/DedupWorker use the same KG path as session memory
+
 **SemanticWorker** (`internal/consolidation/semantic_worker.go`):
 1. Listens to `episodic.created` events
 2. Parses summary for entity mentions + relationships
@@ -129,6 +129,16 @@ All four filesystem tools (`read_file`, `write_file`, `list_files`, `edit`) impl
 | Dynamic value scrubbing (SSRF) | Server IPs and other runtime-discovered values registered via `AddDynamicScrubValues()` and replaced with `[SERVER_IP]` |
 | Web content wrapping | Fetched content wrapped in `<<<EXTERNAL_UNTRUSTED_CONTENT>>>` tags with security warning |

+### Passive Channel Memory Redaction
+
+Passive channel memory extraction runs an additional pre-LLM redaction pass over
+pending group messages before any durable candidate is created. The pass removes
+common secret/key assignments, bearer/provider tokens, connection strings,
+payment-like digit sequences, phone numbers, email addresses, configured
+excluded sender IDs, and configured regex exclusion patterns. The new extraction
+tables store redaction counts/types and extracted summaries only, not raw message
+bodies.
+
 ### Layer 5: Isolation

 **Per-user workspace isolation** -- Two levels prevent cross-user file access:
@@ -1095,6 +1095,28 @@ Response includes `allowed`, `reason`, `instance_id`, `agent_id`, `group_id`, `u

 Credentials are masked in HTTP responses.

+### Passive Memory Extraction
+
+Tenant admins can configure and operate passive memory extraction per channel
+instance. Read endpoints are viewer-accessible; settings and item transitions
+require tenant admin.
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/v1/channels/instances/{id}/memory-extraction` | Get config, latest run, pending count, and recent items |
+| `PUT` | `/v1/channels/instances/{id}/memory-extraction/settings` | Replace normalized `passive_memory` config in the channel instance config |
+| `POST` | `/v1/channels/instances/{id}/memory-extraction/run` | Trigger a manual extraction run |
+| `GET` | `/v1/channels/instances/{id}/memory-extraction/items` | List review queue items; optional `status` filter |
+| `POST` | `/v1/channels/instances/{id}/memory-extraction/items/{itemID}/approve` | Write candidate to episodic memory and publish KG event |
+| `POST` | `/v1/channels/instances/{id}/memory-extraction/items/{itemID}/reject` | Reject candidate |
+| `DELETE` | `/v1/channels/instances/{id}/memory-extraction/items/{itemID}` | Delete candidate and linked episodic summary when present |
+
+Config fields: `enabled`, `review_mode`, `interval_minutes`, `message_cap`,
+`retention_hours`, `allowed_types`, `exclude_users`, `exclude_patterns`,
+`min_messages`, `group_only`. Defaults are disabled, review mode on, group-only,
+360 minute interval, 100 message cap, 168 hour retention, and durable types:
+people, projects, decisions, todos, preferences, events.
+
 ---

 ## 18. Pending Messages
@@ -6,6 +6,20 @@ Significant changes, features, and fixes in reverse chronological order.

 ## 2026-05-29

+### Passive channel memory extraction (issue #64)
+
+**New**
+
+- Added opt-in per-channel passive memory extraction from pending group buffers, with scheduled/manual runs, redaction before LLM extraction, review queue items, and deterministic channel memory source IDs.
+- Added PostgreSQL migration `000075` and SQLite schema version `44` for `channel_memory_extraction_runs` and `channel_memory_extraction_items`.
+- Added tenant-admin HTTP APIs and channel detail UI controls for settings, manual run, last run status, and approve/reject/delete review actions.
+
+**Security**
+
+- Feature is disabled by default and group-only in v1.
+- Review mode is on by default; approved items write to `episodic_summaries` with `source_type='channel'` and then enter the existing KG consolidation event path.
+- Delete removes the review item and associated episodic summary when present. Existing KG nodes created before deletion may require later dedup/cleanup; raw channel messages are not copied into the new extraction tables.
+
 ### Channel context admin surface (issue #66)

 **New**
@@ -1,7 +1,7 @@
 ---
 phase: 1
 title: "Research and Characterization Tests"
-status: pending
+status: complete
 priority: P1
 effort: "1d"
 dependencies: []
@@ -1,7 +1,7 @@
 ---
 phase: 2
 title: "Schema and Store Contracts"
-status: pending
+status: complete
 priority: P1
 effort: "2d"
 dependencies: [1]
@@ -1,7 +1,7 @@
 ---
 phase: 3
 title: "Extractor Worker and Redaction"
-status: pending
+status: complete
 priority: P1
 effort: "2d"
 dependencies: [1, 2]
@@ -1,7 +1,7 @@
 ---
 phase: 4
 title: "Memory KG Review Queue Integration"
-status: pending
+status: complete
 priority: P1
 effort: "2d"
 dependencies: [2, 3]
@@ -1,7 +1,7 @@
 ---
 phase: 5
 title: "Channel Settings UI and Admin Controls"
-status: pending
+status: complete
 priority: P2
 effort: "2d"
 dependencies: [2, 3, 4]
@@ -1,7 +1,7 @@
 ---
 phase: 6
 title: "Verification Documentation and Ship Readiness"
-status: pending
+status: complete
 priority: P1
 effort: "1d"
 dependencies: [1, 2, 3, 4, 5]
@@ -1,7 +1,7 @@
 ---
 title: "Passive Channel Memory Extraction"
 description: "Add disabled-by-default, reviewable passive extraction from group channel buffers into episodic Memory and Knowledge Graph."
-status: pending
+status: complete
 priority: P1
 issue: 64
 branch: "codex/issue-64-passive-channel-memory-extraction"
@@ -31,18 +31,18 @@ Hard boundary: no realtime per-message extraction, no default enablement, no per
 | Activity audit | `internal/store/activity_store.go`, `internal/http/activity.go`, `emitAudit(...)` call sites |
 | Runtime wiring | `cmd/gateway.go`, `cmd/gateway_http_handlers.go`, `cmd/gateway_http_wiring.go`, `cmd/gateway_channels_setup.go` |
 | Web UI | `ui/web/src/pages/channels/`, `ui/web/src/pages/memory/`, `ui/web/src/types/channel.ts` |
-| Schema versions | PG next migration `000074`; bump `internal/upgrade/version.go` from 73. SQLite bump `internal/store/sqlitestore/schema.go` from 42 to 43 |
+| Schema versions | PG migration `000075`; `internal/upgrade/version.go` is 75. SQLite schema version is 44. |

 ## Phases

 | Phase | Name | Status |
 |-------|------|--------|
-| 1 | [Research and Characterization Tests](./phase-01-research-and-characterization-tests.md) | Pending |
-| 2 | [Schema and Store Contracts](./phase-02-schema-and-store-contracts.md) | Pending |
-| 3 | [Extractor Worker and Redaction](./phase-03-extractor-worker-and-redaction.md) | Pending |
-| 4 | [Memory KG Review Queue Integration](./phase-04-memory-kg-review-queue-integration.md) | Pending |
-| 5 | [Channel Settings UI and Admin Controls](./phase-05-channel-settings-ui-and-admin-controls.md) | Pending |
-| 6 | [Verification Documentation and Ship Readiness](./phase-06-verification-documentation-and-ship-readiness.md) | Pending |
+| 1 | [Research and Characterization Tests](./phase-01-research-and-characterization-tests.md) | Complete |
+| 2 | [Schema and Store Contracts](./phase-02-schema-and-store-contracts.md) | Complete |
+| 3 | [Extractor Worker and Redaction](./phase-03-extractor-worker-and-redaction.md) | Complete |
+| 4 | [Memory KG Review Queue Integration](./phase-04-memory-kg-review-queue-integration.md) | Complete |
+| 5 | [Channel Settings UI and Admin Controls](./phase-05-channel-settings-ui-and-admin-controls.md) | Complete |
+| 6 | [Verification Documentation and Ship Readiness](./phase-06-verification-documentation-and-ship-readiness.md) | Complete |

 ## Dependencies

@@ -65,17 +65,17 @@ Hard boundary: no realtime per-message extraction, no default enablement, no per

 ## Success Criteria

- [ ] Passive extraction is disabled by default and opt-in per channel instance.
- [ ] Extraction runs only by interval, message cap, or manual trigger.
- [ ] Raw channel messages are read from tenant-scoped pending buffers and are not stored permanently outside existing retention.
- [ ] Redaction blocks common secrets, tokens, credentials, payment/banking data, phone/address patterns, and configurable excluded users/patterns before durable writes.
- [ ] Extracted items go to a review queue by default.
- [ ] Approved items write to `episodic_summaries` with `source_type='channel'` and deterministic `source_id`.
- [ ] KG ingestion receives only approved/redacted summaries and records source metadata.
- [ ] Review reject/delete removes queued items and prevents later writes.
- [ ] Tenant/channel/agent boundaries covered by tests.
- [ ] PG and SQLite schemas both migrate and compile.
- [ ] UI exposes settings, last run status, pending items, approve/reject/delete, and manual run.
+- [x] Passive extraction is disabled by default and opt-in per channel instance.
+- [x] Extraction runs only by interval, message cap, or manual trigger.
+- [x] Raw channel messages are read from tenant-scoped pending buffers and are not stored permanently outside existing retention.
+- [x] Redaction blocks common secrets, tokens, credentials, payment/banking data, phone/address patterns, and configurable excluded users/patterns before durable writes.
+- [x] Extracted items go to a review queue by default.
+- [x] Approved items write to `episodic_summaries` with `source_type='channel'` and deterministic `source_id`.
+- [x] KG ingestion receives only approved/redacted summaries and records source metadata.
+- [x] Review reject/delete removes queued items and prevents later writes.
+- [x] Tenant/channel/agent boundaries covered by focused tests and store scoping.
+- [x] PG and SQLite schemas both migrate and compile.
+- [x] UI exposes settings, last run status, pending items, approve/reject/delete, and manual run.

 ## Key Risks