tiennm99bot/plans/260425-1945-mongodb-atlas-migration/phase-04-dual-write-wrappers.md

# Phase 04 — Dual-Write Wrappers + Storage Flag + e2e

## Context Links
- [Schema report](../reports/researcher-260425-1924-mongodb-schema-and-migration.md) §5 (dual-write mechanics)
- [Code-reviewer Findings #2, #8, #18, #20, #27](../reports/code-reviewer-260425-2034-atlas-plan-correctness.md)
- [Brainstormer Finding #10](../reports/brainstormer-260425-2034-atlas-plan-critique.md) (e2e moves earlier)
- [Debugger #8](../reports/debugger-260425-2034-atlas-plan-failure-modes.md)
- `src/db/create-store.js` — KV factory to extend (≤80 LOC currently)
- `src/db/create-sql-store.js` — SQL factory to extend (≤80 LOC currently)
- `scripts/stub-kv.js` — needs sibling `stubMongo` (duck-typed, NOT a string)
- `scripts/register.js:75` — `buildRegistry({MODULES, KV: stubKv, AI: stubAi})` call site

## Overview
- **Priority:** P0
- **Status:** pending
- **Description:** Wrap KV and SQL stores so writes hit BOTH backends and reads hit the configured primary. Single env flag `STORAGE_PRIMARY` toggles read source. **Also lands the e2e storage-roundtrip test** (moved here from phase-08 per brainstormer #10) so dual-write code is end-to-end-tested before any prod soak. This is the ONLY phase that touches the request path's data layer pre-cutover.

## Key Insights
- Single source of truth: `env.STORAGE_PRIMARY` ∈ {`kv`, `mongo`}. Default `kv`.
- Dual-write is **always on** when MongoDB credentials are present, regardless of `STORAGE_PRIMARY`. Goal: keep secondary warm + ready for read-flip.
- Writes to BOTH must be parallel via `Promise.allSettled`. If secondary write fails → log error AND push the failed key onto a small KV retry queue (`__retry:mongo-failed`) for later drain. Throw only on primary failure. (code-reviewer #8 / debugger #8)
- A separate **drift-verifier cron** (1/hr or 1/6h, sample N keys) drains the retry queue + spot-checks parity. Replaces the original "one-shot reconciliation" approach.
- During cutover (Phase 07), the flag flips to `mongo`. After soak, secondary writes can be turned off via a second flag `DUAL_WRITE` (default `1`, set to `0` post-cutover).
- `register.js` runs at deploy time — does NOT need a Mongo connection. Pass **duck-typed `stubMongo`** + `STUB_SENTINEL` so factories short-circuit even if flags drift (code-reviewer #2).

## Requirements

### Functional
- `DualKVStore` class implementing `KVStore` interface:
  - `get`/`getJSON`/`list` → read from primary only.
  - `put`/`putJSON`/`delete` → write to both via `Promise.allSettled`. Log secondary failures with key + error AND enqueue to `__retry:mongo-failed` (a KV list bound to the same `env.KV` namespace, prefix `__retry:mongo-failed:`). Throw only on primary failure.
  - Expose `_kind === "dual"` sentinel for test-side identification (replaces the `_implementations` array seam — code-reviewer #27).
- `DualSqlStore` class implementing `SqlStore` interface:
  - `all`/`first` → read primary only.
  - `run` → write both. Same fault tolerance as KV (retry-queue same namespace, prefix `__retry:mongo-sql-failed:`).
  - `prepare`/`batch` → primary only (D1 stays authoritative for legacy paths until cutover).
- `create-store.js` factory updated:
  - Reads `env.STORAGE_PRIMARY` (default `"kv"`) and `env.DUAL_WRITE` (default `"1"`).
  - **If `env.MONGODB_URI === STUB_SENTINEL`** → unconditionally return CFKVStore-only (deploy-time path).
  - If `MONGODB_URI` absent → return CFKVStore-only (legacy path).
  - If `DUAL_WRITE=0` AND `STORAGE_PRIMARY=mongo` → return MongoKVStore-only.
  - Otherwise → return `DualKVStore(primary, secondary)`.
  - **Pre-design post-cutover shape** (code-reviewer #20): when both flags are absent / unset, factory returns MongoKVStore-only directly (no flag inspection). Phase-07 simplification just deletes the KV branches.
- `create-sql-store.js` factory mirrors logic for SQL (uses `MongoSqlStore` shim from phase-03).
- `register.js` + `scripts/stub-kv.js`: add **duck-typed** `stubMongo` (NOT a string) + `STUB_SENTINEL` so deploy-time registry build doesn't crash.
- **Drift-verifier cron** (replaces one-shot phase-05 reconciliation):
  - New module `src/cron/drift-verifier.js` (or co-located in misc) registered in `wrangler.toml [triggers] crons` at 1/hr (configurable).
  - Drains `__retry:mongo-failed` queue (each entry: re-attempt the secondary write).
  - Spot-checks parity: sample N keys per module, hash-compare values, log mismatches.
- **e2e test** lands HERE (not phase-08 — brainstormer #10): `tests/e2e/storage-roundtrip.test.js` boots a fake env with `MongoKVStore` + `MongoSqlStore` against `fake-mongo.js`, dispatches a representative wordle command + a trading insert, asserts state persisted.

### Non-functional
- Each new file ≤200 LOC.
- New files: `src/db/dual-kv-store.js`, `src/db/dual-sql-store.js`, `src/cron/drift-verifier.js`, `tests/e2e/storage-roundtrip.test.js`.
- All new exports JSDoc'd.
- Behavior must be deterministic given env flags (testable via fake env).
- Logging via `console.warn`/`console.error` with structured fields (`{ phase, op, key, err }`).

## Architecture

### Read/write flow (during dual-write window)
```mermaid
sequenceDiagram
    participant H as Handler
    participant F as createStore (factory)
    participant D as DualKVStore
    participant K as CFKVStore (primary)
    participant M as MongoKVStore (secondary)
    participant Q as KV retry queue

    H->>F: createStore("wordle", env)
    F->>F: read env.STORAGE_PRIMARY (=kv) + env.DUAL_WRITE (=1)
    F-->>H: DualKVStore(K, M)
    H->>D: getJSON("games:42")
    D->>K: getJSON
    K-->>D: doc
    D-->>H: doc

    H->>D: putJSON("games:42", v, {ttl: 86400})
    par parallel
        D->>K: putJSON
    and
        D->>M: putJSON
    end
    K-->>D: ok
    M--xD: error (caught, logged)
    D->>Q: enqueue(__retry:mongo-failed:games:42)
    D-->>H: ok (primary succeeded)
```

### Drift-verifier cron flow
```mermaid
flowchart LR
    Cron[1/hr cron] --> V[drift-verifier]
    V -->|drain| Q[(KV: __retry:mongo-failed)]
    V -->|sample N keys| K[(CF KV)]
    V -->|sample N keys| M[(Atlas)]
    V -->|hash compare| Log[CF Observability]
    V -->|retry secondary| M
```

### Flag matrix

| `STORAGE_PRIMARY` | `DUAL_WRITE` | `MONGODB_URI` | Result |
|--------------------|--------------|----------------|--------|
| (unset) or `kv` | `1` (default) | set | DualKV: read KV, write both |
| (unset) or `kv` | `0` | any | CFKVStore only (legacy / rollback) |
| `mongo` | `1` | set | DualKV: read Mongo, write both (cutover phase) |
| `mongo` | `0` | set | MongoKVStore only (post-cutover) |
| any | any | unset | CFKVStore only (Phase 02/03 not deployed yet) |
| any | any | === STUB_SENTINEL | CFKVStore only (deploy-time register path) |

### Stub for register (duck-typed; code-reviewer #2)
```js
// scripts/stub-kv.js — sketch
export const STUB_SENTINEL = "__stub_mongo__";
export const stubMongo = {
  // duck-typed MongoClient surface that no-ops without network IO
  db() { return { collection: () => { throw new Error("stubMongo: no IO"); } }; },
  connect: async () => undefined,
  close: async () => undefined,
};
// register.js passes env.MONGODB_URI = STUB_SENTINEL; factories short-circuit on sentinel
```
Test: assert zero `MongoClient.connect()` calls when stub is used (`vi.spyOn(MongoClient.prototype, 'connect')`).

### Rollback semantics (code-reviewer #18)
**Rollback to KV-primary AFTER any Mongo-primary period requires reverse-backfill.** During the Mongo-primary window, secondary writes to KV may have failed silently and KV may be missing rows that exist only in Mongo. Cross-link to Phase 07 reverse-backfill scripts (which become Stage-2 prerequisites — see phase-07 step 11 prereq).

## Related Code Files

### CREATE
- `/config/workspace/tiennm99/miti99bot/src/db/dual-kv-store.js`
- `/config/workspace/tiennm99/miti99bot/src/db/dual-sql-store.js`
- `/config/workspace/tiennm99/miti99bot/src/cron/drift-verifier.js`
- `/config/workspace/tiennm99/miti99bot/tests/db/dual-kv-store.test.js`
- `/config/workspace/tiennm99/miti99bot/tests/db/dual-sql-store.test.js`
- `/config/workspace/tiennm99/miti99bot/tests/e2e/storage-roundtrip.test.js`

### MODIFY
- `/config/workspace/tiennm99/miti99bot/src/db/create-store.js` — read flags, branch, honor STUB_SENTINEL.
- `/config/workspace/tiennm99/miti99bot/src/db/create-sql-store.js` — same.
- `/config/workspace/tiennm99/miti99bot/scripts/stub-kv.js` — add `stubMongo` (duck-typed) + `STUB_SENTINEL`.
- `/config/workspace/tiennm99/miti99bot/scripts/register.js:75` — pass `MONGODB_URI: STUB_SENTINEL, STORAGE_PRIMARY: "kv", DUAL_WRITE: "0"` in env stub.
- `/config/workspace/tiennm99/miti99bot/wrangler.toml` — add `[vars] STORAGE_PRIMARY = "kv"`, `DUAL_WRITE = "1"`; add 1/hr cron for drift-verifier.

### DELETE
- (none)

## Implementation Steps
1. Grep `env.KV` and `env.DB` usage outside `src/db/` — should be zero (encapsulated). Confirm.
2. Create `src/db/dual-kv-store.js`:
   - Constructor `(primary, secondary, retryQueue, logger=console)`.
   - 6 methods. Reads → primary. Writes → `Promise.allSettled` with secondary errors logged AND enqueued to `retryQueue`.
   - Expose `_kind = "dual"` (code-reviewer #27).
   - JSDoc.
3. Create `src/db/dual-sql-store.js` — same shape, SqlStore interface.
4. Create `src/cron/drift-verifier.js`:
   - Cron handler signature `(event, ctx)`. `ctx.db / ctx.sql` provide the dual store.
   - Drain `__retry:mongo-failed` (and SQL queue) — re-attempt secondary writes; remove on success.
   - Sample N keys per module via dual store; hash-compare; log mismatches.
   - Tunable N via `env.DRIFT_SAMPLE_N` (default 50).
5. Modify `src/db/create-store.js`:
   - **Short-circuit on STUB_SENTINEL** (code-reviewer #2).
   - Branch on `env.STORAGE_PRIMARY`, `env.DUAL_WRITE`, presence of `env.MONGODB_URI`.
   - When constructing MongoKVStore, pass collection name (sanitized: replace `-` with `_`).
   - Keep the existing prefixed-wrapper closure intact.
   - **Comment the post-cutover shape** (code-reviewer #20): `// post-Phase-07: this entire function returns MongoKVStore-only; KV branches removed`.
6. Modify `src/db/create-sql-store.js` mirrored. Use `MongoSqlStore` shim from phase-03; trading module's init wires `MongoTradesStore` directly via `tradesStore` param.
7. Add `stubMongo` + `STUB_SENTINEL` to `scripts/stub-kv.js`. Update `scripts/register.js:75` to pass `MONGODB_URI: STUB_SENTINEL, STORAGE_PRIMARY: "kv", DUAL_WRITE: "0"`.
8. Update `wrangler.toml` `[vars]` block to declare `STORAGE_PRIMARY = "kv"`, `DUAL_WRITE = "1"`, `DRIFT_SAMPLE_N = "50"`. Add cron `"0 * * * *"` (or `"0 */6 * * *"` if log volume is a concern).
9. Tests:
   - `dual-kv-store.test.js`: write succeeds when both succeed; write succeeds when secondary fails (logs + enqueues retry); write fails when primary fails; reads always from primary; `_kind === "dual"`.
   - `dual-sql-store.test.js`: same pattern.
   - **stubMongo regression** (code-reviewer #2): construct factory with `STUB_SENTINEL` + every flag combo; `vi.spyOn(MongoClient.prototype, 'connect')` asserts ZERO calls.
   - Integration test: factory with all flag combos returns correct concrete type (verify via `_kind` sentinel).
   - **e2e test** `tests/e2e/storage-roundtrip.test.js`: boot fake env (`fake-mongo`), build registry, dispatch a wordle command (KV path) + trading insert (SQL path), assert persistence.
   - Update register dry-run test to confirm command list still derives.
10. Run `npm test`, `npm run lint`, `npm run register:dry`. All pass.

## Todo List
- [ ] `dual-kv-store.js` created (with retry-queue + `_kind` sentinel)
- [ ] `dual-sql-store.js` created
- [ ] `src/cron/drift-verifier.js` created + cron registered in wrangler.toml
- [ ] `create-store.js` reads env flags + honors STUB_SENTINEL + post-cutover shape commented
- [ ] `create-sql-store.js` mirrors logic
- [ ] Duck-typed `stubMongo` + `STUB_SENTINEL` added to `stub-kv.js`
- [ ] `register.js` passes Mongo stub + flags
- [ ] `wrangler.toml` declares `STORAGE_PRIMARY` + `DUAL_WRITE` + `DRIFT_SAMPLE_N` + drift cron
- [ ] All test files written, passing
- [ ] **stubMongo never reaches MongoClient.connect** asserted in test
- [ ] **e2e storage-roundtrip test** passes (wordle KV + trading SQL)
- [ ] `npm run register:dry` succeeds
- [ ] `npm test` passes
- [ ] `npm run lint` passes
- [ ] Manual smoke: `wrangler dev` with `MONGODB_URI` set + `DUAL_WRITE=1` → write to wordle module → verify both KV and Mongo received it

## Success Criteria
- All 6 flag/sentinel combos behave per matrix.
- Secondary failure does NOT fail user-facing request; failure IS logged AND enqueued to retry queue.
- `register:dry` continues to work without Mongo creds; zero MongoClient.connect calls.
- Drift-verifier cron drains retry queue + flags any divergence ≥ threshold.
- e2e roundtrip passes for both KV (wordle) and SQL (trading) paths against `fake-mongo`.

## Risk Assessment

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Secondary write silently masks data divergence | M | H | Retry queue + drift-verifier cron (1/hr). Phase 06 monitors error rate. |
| Write amplification doubles latency on every put | H | M | `Promise.allSettled` parallel; latency = max(KV, Mongo). Cold p99 = 1500ms — flagged in Phase 06 abort gate. |
| `register:dry` connects to Atlas and fails | L | H | `STUB_SENTINEL` short-circuit; regression-tested in step 9. |
| Flag drift between wrangler.toml and .env.deploy | M | M | Phase 08 docs include checklist; lint script checks both files declare `STORAGE_PRIMARY`. |
| `MONGODB_URI` missing in prod → factory falls back silently | L | H | Boot-time assertion: if `STORAGE_PRIMARY=mongo` but URI absent (and not STUB_SENTINEL), throw on first request. |
| Order-of-operations bug: read after write reads stale primary | L | M | Same-isolate writes to primary block reads (single-threaded JS); no interleaving issue. |
| Mongo write fails because collection didn't exist + index not created | L | M | MongoKVStore `_ensureIndex` creates lazily; first write triggers creation. Tested in Phase 02. |
| Drift-verifier cron itself burns M0 ops | L | L | Default 1/hr × N=50 = ~1200 reads/day, well under M0 envelope. Tunable via `DRIFT_SAMPLE_N`. |
| **Rollback to KV-primary after Mongo-primary loses data** | M | H | **Reverse-backfill scripts (phase-07) are Stage-2 prerequisites** (code-reviewer #18). Cross-linked. |

## Security Considerations
- Logged errors must NOT include the document `value` (PII risk for trading).
- Retry-queue values may contain document blobs — same redaction rules for any cron-driven log emission.
- `STORAGE_PRIMARY` and `DUAL_WRITE` are public-readable env vars (in `wrangler.toml`). Acceptable — they're routing flags, not secrets.
- `STUB_SENTINEL` is a public string; safe — not a credential.

## Rollback (this phase only)
1. Set `DUAL_WRITE=0` in wrangler.toml `[vars]`. Redeploy.
2. CFKVStore + CFSqlStore become sole path. Identical to pre-Phase-04 behavior.
3. Disable drift-verifier cron (remove from wrangler.toml triggers).
4. (Optional) Revert phase commits if rollback is permanent.

## Next Steps
- **Blocks:** Phase 05 (backfill must run with dual-write live so concurrent writes don't bypass Mongo).
- **Unblocks:** Phase 05.