From aff4bf42116d0fbd842c02221d0909eac4523d37 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Sat, 16 May 2026 10:33:46 +0700 Subject: [PATCH] =?UTF-8?q?docs(plans):=20close=20cf=E2=86=92aws=20phases?= =?UTF-8?q?=2001-02=20and=20amend=20locked=20decisions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - phase-01: mark complete; record live D1+KV inventory outcome (9 durable KV keys, trading source is KV not D1, retired modules skipped) - phase-02: mark complete; toolchain verified end-to-end against prod CF - phase-03: rewrite to flat KV copy (D1-transform branch dropped) - plan.md: amend staging-only decision to permit pre-cutover bulk import against the live table while it has no AWS-served writes - runbook: lock matrix and trading source, document Phase 02 commands --- docs/cf-to-aws-migration-runbook.md | 165 ++++++++++++------ ...1-source-inventory-and-migration-policy.md | 19 +- ...-02-backfill-toolchain-and-safety-rails.md | 19 +- .../phase-03-trading-and-durable-kv-import.md | 72 ++++---- .../plan.md | 26 +-- 5 files changed, 188 insertions(+), 113 deletions(-) diff --git a/docs/cf-to-aws-migration-runbook.md b/docs/cf-to-aws-migration-runbook.md index cbaabaa..e0efdcc 100644 --- a/docs/cf-to-aws-migration-runbook.md +++ b/docs/cf-to-aws-migration-runbook.md @@ -18,40 +18,52 @@ Examples: - `misc` + `last_ping` - `trading` + `user:` -## Migration matrix (locked from live code) +## Migration matrix (locked from live code AND live CF inventory) -| Source dataset / prefix | Current consumer | Action | AWS target | Notes | -|---|---|---|---|---| -| `wordle:stats:*` | `internal/modules/wordle/state.go` | migrate | `pk=wordle`, `sk=stats:` | durable player stats | -| `wordle:game:*` | `internal/modules/wordle/state.go` | skip | none | ephemeral in-flight game state | -| `loldle:stats:*` | `internal/modules/loldle/state.go` | migrate | `pk=loldle`, `sk=stats:` | durable player stats | -| `loldle:config:*` | `internal/modules/loldle/state.go` | migrate | `pk=loldle`, `sk=config:` | durable per-subject config | -| `loldle:game:*` | `internal/modules/loldle/state.go` | skip | none | ephemeral in-flight game state | -| `twentyq:stats:*` | `internal/modules/twentyq/state.go` | migrate | `pk=twentyq`, `sk=stats:` | durable player stats | -| `twentyq:game:*` | `internal/modules/twentyq/state.go` | skip | none | ephemeral in-flight game state | -| `lolschedule:subscribers` | `internal/modules/lolschedule/subscribers.go` | migrate | `pk=lolschedule`, `sk=subscribers` | durable subscriber list | -| `lolschedule:matches:*` | `internal/modules/lolschedule/api_client.go` | skip | none | cache only | -| `misc:last_ping` | `internal/modules/misc/misc.go` | migrate | `pk=misc`, `sk=last_ping` | `/mstats` still reads it | -| `trading:user:*` or equivalent legacy portfolio state | `internal/modules/trading/portfolio.go` | migrate (transform) | `pk=trading`, `sk=user:` | must become current `Portfolio` JSON | -| `trading:sym:*` | `internal/modules/trading/symbols.go` | skip | none | cache only | -| retired module namespaces (`loldle_emoji`, `loldle_quote`, `loldle_ability`, `loldle_splash`, `semantle`, `doantu`, etc.) | no live Go consumer | archive | none | export only if operator wants cold backup | +Live CF inventory taken from production via wrangler on 2026-05-16: +- D1 `miti99bot-db`: only `trading_trades` exists (plus internal `_cf_KV`, `_migrations`, `sqlite_sequence`). No `users` or `holdings` tables. 11 rows, 1 distinct user. +- KV namespace `f7f190fcb2fa42eb84a05542911334b0`: 21 keys total (see breakdown below). -## Trading source inventory +| Source dataset / prefix | Live keys | Current consumer | Action | AWS target | Notes | +|---|---|---|---|---|---| +| `wordle:stats:*` | 1 | `internal/modules/wordle/state.go` | migrate | `pk=wordle`, `sk=stats:` | durable player stats | +| `wordle:game:*` | 0 | `internal/modules/wordle/state.go` | skip | none | ephemeral; not present | +| `loldle:stats:*` | 4 | `internal/modules/loldle/state.go` | migrate | `pk=loldle`, `sk=stats:` | durable player stats | +| `loldle:config:*` | 1 | `internal/modules/loldle/state.go` | migrate | `pk=loldle`, `sk=config:` | durable per-subject config | +| `loldle:game:*` | 0 | `internal/modules/loldle/state.go` | skip | none | ephemeral; not present | +| `twentyq:stats:*` | 1 | `internal/modules/twentyq/state.go` | migrate | `pk=twentyq`, `sk=stats:` | durable player stats | +| `twentyq:game:*` | 0 | `internal/modules/twentyq/state.go` | skip | none | ephemeral; not present | +| `lolschedule:subscribers` | 1 | `internal/modules/lolschedule/subscribers.go` | migrate | `pk=lolschedule`, `sk=subscribers` | durable subscriber list | +| `lolschedule:matches:*` | 0 | `internal/modules/lolschedule/api_client.go` | skip | none | cache only; not present | +| `misc:last_ping` | 0 | `internal/modules/misc/misc.go` | skip | none | JS Worker never wrote it (KV 404). `/mstats` will start fresh on AWS. | +| `trading:user:*` | 1 | `internal/modules/trading/portfolio.go` | **migrate (flat KV copy)** | `pk=trading`, `sk=user:` | CF KV already holds the exact `Portfolio` JSON shape (`currency`/`assets`/`meta`). No D1 transform required. | +| `trading:sym:*` | 7 | `internal/modules/trading/symbols.go` | skip | none | symbol price cache | +| `trading_trades` (D1) | 11 rows | n/a (no AWS consumer) | archive-only | none | optional cold JSONL export for audit; not import input | +| `doantu:stats:*` | 2 | retired Go module | skip | none | retired; operator chose not to archive | +| `loldle-ability:stats:*` | 1 | retired Go module | skip | none | retired; operator chose not to archive | +| `loldle-emoji:stats:*` | 1 | retired Go module | skip | none | retired; operator chose not to archive | +| `semantle:stats:*` | 1 | retired Go module | skip | none | retired; operator chose not to archive | -What is confirmed in-repo: -- legacy D1 table `trading_trades` exists -- known columns from prior work: `id`, `user_id`, `symbol`, `side`, `qty`, `price_vnd`, `ts` -- current AWS runtime target is `internal/modules/trading/portfolio.go`: - - `currency map[string]float64` - - `assets map[string]int64` - - `meta.invested float64` - - `meta.createdAt int64` +## Trading source — LOCKED 2026-05-16 -What is **not** confirmed in-repo: -- authoritative legacy source table/column for `meta.invested` -- authoritative legacy source table/column for `meta.createdAt` -- whether old Cloudflare runtime stored a user portfolio snapshot separately from `trading_trades` -- whether legacy `users` / `holdings` tables referenced in older plans still exist in production D1 +The legacy CF JS Worker already snapshots the user portfolio into KV at `trading:user:`. The stored value is byte-for-byte the JSON shape the Go AWS runtime expects: + +```json +{ + "currency": {"VND": 170850000}, + "assets": {"TCB": 1000, "TCX": 1000, "FPT": 10000, "VCB": 1000}, + "meta": {"invested": 1000000000, "createdAt": 1776743792792} +} +``` + +Decision record: +- **Source** for trading migration = CF KV key `trading:user:`. Not D1. +- **Mapping rule** = identity copy. Read KV value, write to DynamoDB at `(pk=trading, sk=user:)` with the same `value` attribute. +- **`meta.invested`** = read directly from KV; no derivation. +- **`meta.createdAt`** = read directly from KV; no derivation. +- **D1 `trading_trades`** = archive-only audit export (JSONL). Not used by the import path. Only 11 rows / 1 user as of 2026-05-16 — operator may export or skip without runtime impact. + +This invalidates the earlier "Phase 03 D1 transform" framing. Phase 03 is now a plain KV copy plus optional D1 audit dump. ## Phase 01 operator procedure @@ -83,17 +95,9 @@ wrangler d1 execute --remote \ Do not stop at historical names like `trading_trades`, `users`, or `holdings`; the goal is to inspect whatever production D1 actually contains today. -### 2) Lock the trading transform inputs +### 2) Lock the trading source -Before any import code is written, prove one of these is true: -- a portfolio snapshot table already exists and contains the exact source for `currency`, `assets`, `meta.invested`, and `meta.createdAt`, or -- those fields must be derived, and the derivation is written down explicitly and accepted. - -Minimum decision record to capture: -- source table(s) -- source column(s) -- mapping rule into `Portfolio` -- whether historical `trading_trades` rows are import inputs only or audit-only exports +Resolved 2026-05-16. CF KV `trading:user:` already holds the exact `Portfolio` JSON shape; no transform path is needed. See "Trading source — LOCKED 2026-05-16" above for the decision record. The historical D1 `trading_trades` table is archive-only audit data. ### 3) Inventory Cloudflare KV prefixes @@ -121,23 +125,74 @@ Do not proceed to import tooling until each discovered dataset is tagged as one Anything not in the matrix is out of scope by default. -## Phase 01 done checklist +## Phase 01 done checklist — CLOSED 2026-05-16 -- [ ] Every live Cloudflare KV prefix is classified as `migrate`, `skip`, or `archive` -- [ ] Every migrated KV dataset has an exact DynamoDB `(pk, sk)` target -- [ ] Trading source table(s) and column(s) are locked -- [ ] `meta.invested` source is explicit -- [ ] `meta.createdAt` source is explicit -- [ ] Retired module namespaces are explicitly excluded from runtime import -- [ ] AWS cutover remains gated on a green parity report from the migration plan +- [x] Every live Cloudflare KV prefix is classified as `migrate`, `skip`, or `archive` +- [x] Every migrated KV dataset has an exact DynamoDB `(pk, sk)` target +- [x] Trading source table(s) and column(s) are locked (resolved as KV-only — see "Trading source — LOCKED") +- [x] `meta.invested` source is explicit (KV `trading:user:*`) +- [x] `meta.createdAt` source is explicit (KV `trading:user:*`) +- [x] Retired module namespaces are explicitly excluded from runtime import +- [x] AWS cutover remains gated on a green parity report from the migration plan (`plans/260510-0114-aws-port/phase-07-cutover.md` lines 13, 20, 42, 72) -## Current blocker +## Phase 03 impact -The repo proves the live AWS target shape and most KV policy, but it does **not** prove the authoritative legacy source for trading `meta.invested` and `meta.createdAt`. Phase 03 should stay blocked until the operator finishes the D1 inventory above. +The D1-transform branch of Phase 03 is dropped. Phase 03 is now: +- flat KV → DynamoDB copy across the 9 durable keys above +- optional one-shot `trading_trades` JSONL audit dump (operator-elective) -## Unresolved questions +Effort revised from 4-6h to ~2h. -1. Does production D1 still have `users` / `holdings` tables, or only `trading_trades`? -2. If only `trading_trades` exists, what exact derivation should define `meta.createdAt`? -3. If only `trading_trades` exists, should `meta.invested` be reconstructed from surviving positions only or from full historical cost basis? -4. Are any retired Cloudflare KV namespaces still carrying data the operator wants archived before teardown? +## Phase 02 toolchain (built 2026-05-16) + +Single Go binary `cmd/migrate_cf_data` with 3 subcommands. No admin HTTP routes. No dependency on the running AWS bot process. Shared helpers live under `internal/migration/`. + +### Required environment + +| Var | Purpose | Used by | +|---|---|---| +| `CLOUDFLARE_API_TOKEN` | read-scope token | inventory, kv-import, trading-audit-dump | +| `CLOUDFLARE_ACCOUNT_ID` | production CF account | all | +| `CF_KV_NAMESPACE_ID` | production KV namespace | inventory, kv-import | +| `CF_D1_DATABASE_ID` | production D1 database | trading-audit-dump | +| `AWS_REGION` (+ standard AWS SDK creds) | DynamoDB writes | kv-import (non-dry-run) | + +### Commands + +Inventory (read-only classification, prints counts; no writes anywhere): + +```sh +go run ./cmd/migrate_cf_data inventory +``` + +KV import dry-run against staging — proves wiring and key-shape mapping without any DynamoDB writes: + +```sh +go run ./cmd/migrate_cf_data kv-import --table=miti99bot-staging --dry-run +``` + +KV import (default idempotent — rejects writes where `(pk, sk)` already exists; rerun is safe): + +```sh +go run ./cmd/migrate_cf_data kv-import --table=miti99bot-staging +``` + +KV import with explicit overwrite (drops the `attribute_not_exists` guard; only use when intentionally re-importing a known-good source after rollback): + +```sh +go run ./cmd/migrate_cf_data kv-import --table=miti99bot-staging --overwrite +``` + +Optional `trading_trades` audit dump (D1 → JSONL, audit-only — not an import input): + +```sh +go run ./cmd/migrate_cf_data trading-audit-dump --out=plans/reports/migration-260515-2250-trading-audit.jsonl +``` + +### Report layout + +Every `kv-import` run prints a Migration report block with four buckets: `Imported`, `Skipped (already present)`, `Skipped (policy)`, `Failed`. Each bucket is keyed by source prefix (or by skip-reason for the policy bucket). Phase 04 parity verification compares Phase 01 inventory counts against this report. + +### Verified end-to-end against production CF on 2026-05-16 + +`inventory` and `kv-import --dry-run` both run cleanly against prod CF. 9 durable keys map to the runtime `(pk, sk)` shape exactly as the policy table predicts. No DynamoDB writes were issued during verification (`--dry-run`). diff --git a/plans/260515-2250-cf-data-to-aws-migration/phase-01-source-inventory-and-migration-policy.md b/plans/260515-2250-cf-data-to-aws-migration/phase-01-source-inventory-and-migration-policy.md index 3977769..55bc6dc 100644 --- a/plans/260515-2250-cf-data-to-aws-migration/phase-01-source-inventory-and-migration-policy.md +++ b/plans/260515-2250-cf-data-to-aws-migration/phase-01-source-inventory-and-migration-policy.md @@ -1,10 +1,11 @@ --- phase: 1 title: "Source inventory and migration policy" -status: pending +status: completed priority: P1 effort: "2-3h" dependencies: [] +completed: 2026-05-16 --- # Phase 01: Source inventory and migration policy @@ -38,11 +39,17 @@ Identify the exact Cloudflare KV namespaces and D1 tables still carrying product 6. Update the AWS cutover phase to say final webhook flip is gated on this migration matrix. ## Success Criteria -- [ ] Every live CF dataset is classified as migrate, skip, or archive. -- [ ] Every migrated dataset has an explicit AWS target key shape. -- [ ] Trading `meta.createdAt` and `meta.invested` have authoritative source fields. -- [ ] Retired-module data is excluded by policy. -- [ ] The cutover plan references this migration gate. +- [x] Every live CF dataset is classified as migrate, skip, or archive. (matrix in `docs/cf-to-aws-migration-runbook.md`) +- [x] Every migrated dataset has an explicit AWS target key shape. +- [x] Trading `meta.createdAt` and `meta.invested` have authoritative source fields. (KV `trading:user:` — flat JSON snapshot, no D1 derivation) +- [x] Retired-module data is excluded by policy. (`doantu`, `loldle-ability`, `loldle-emoji`, `semantle` stats keys skipped; no archive) +- [x] The cutover plan references this migration gate. (`plans/260510-0114-aws-port/phase-07-cutover.md` lines 13, 20, 42, 72) + +## Outcome notes (2026-05-16) +- Live inventory taken via wrangler against prod CF account `miti99` (D1 `miti99bot-db`, KV `f7f190fcb2fa42eb84a05542911334b0`). +- D1: only `trading_trades` exists (11 rows, 1 user). No `users` / `holdings` tables. +- KV: 21 keys total. 9 durable, 7 cache, 5 retired-module stats. `misc:last_ping` does not exist upstream. +- Trading transform branch invalidated — JS Worker already stores final `Portfolio` JSON in KV. Phase 03 rewritten to flat KV copy (effort 4-6h → ~2h). ## Risk Assessment Main risk is misclassifying a dataset as disposable when users still care about it. Mitigation: classify by current runtime consumers first, then validate Cloudflare inventory against those exact consumers before any tooling is written. diff --git a/plans/260515-2250-cf-data-to-aws-migration/phase-02-backfill-toolchain-and-safety-rails.md b/plans/260515-2250-cf-data-to-aws-migration/phase-02-backfill-toolchain-and-safety-rails.md index 619da3a..f59d600 100644 --- a/plans/260515-2250-cf-data-to-aws-migration/phase-02-backfill-toolchain-and-safety-rails.md +++ b/plans/260515-2250-cf-data-to-aws-migration/phase-02-backfill-toolchain-and-safety-rails.md @@ -1,10 +1,11 @@ --- phase: 2 title: "Backfill toolchain and safety rails" -status: pending +status: completed priority: P1 effort: "3-4h" dependencies: [1] +completed: 2026-05-16 --- # Phase 02: Backfill toolchain and safety rails @@ -44,11 +45,17 @@ Build operator-run migration tooling in Go that can read legacy Cloudflare data, 6. Document the exact operator workflow in the runbook. ## Success Criteria -- [ ] Tooling reads CF KV metadata and values without touching app code paths. -- [ ] Trading import mode accepts local D1 JSON exports. -- [ ] Every command supports `--dry-run`. -- [ ] Import path is idempotent or safely merge-based. -- [ ] Checkpoint/resume behavior is either justified and documented or intentionally omitted. +- [x] Tooling reads CF KV metadata and values without touching app code paths. (`internal/migration/cloudflare_kv_client.go`) +- [x] ~~Trading import mode accepts local D1 JSON exports.~~ → invalidated by Phase 01. Trading is a flat KV copy; D1 is audit-only via `trading-audit-dump --out=`. +- [x] Every command supports `--dry-run`. (kv-import has --dry-run; inventory and trading-audit-dump are read-only by construction so a dry-run flag is redundant) +- [x] Import path is idempotent or safely merge-based. (`attribute_not_exists(pk)` guard; `--overwrite` is explicit opt-in) +- [x] Checkpoint/resume behavior is intentionally omitted: Phase 01 inventory shows 21 keys total (well below the threshold where resume earns its complexity cost). + +## Outcome notes (2026-05-16) +- Files created: `cmd/migrate_cf_data/main.go`, `internal/migration/policy.go`, `cloudflare_kv_client.go`, `cloudflare_d1_client.go`, `dynamodb_writer.go`, `report.go` + four `*_test.go` files. +- Verify command (`cmd/verify_cf_aws_parity/`) intentionally moved to Phase 04 to remove the cross-phase ownership collision. +- Toolchain smoke-tested against prod CF: `inventory` → 22 keys observed (one cache key drift since Phase 01 inventory); `kv-import --dry-run` → 9 durable keys map to runtime `(pk, sk)` exactly. +- All `go test ./...` pass; `go vet ./...` clean. ## Risk Assessment The main risk is embedding too much migration logic into one giant binary. Mitigation: split command entrypoints and keep shared logic in small `internal/migration/` helpers so each command stays reviewable and under the repo's file-size guidance. diff --git a/plans/260515-2250-cf-data-to-aws-migration/phase-03-trading-and-durable-kv-import.md b/plans/260515-2250-cf-data-to-aws-migration/phase-03-trading-and-durable-kv-import.md index 41cd7eb..30c4e72 100644 --- a/plans/260515-2250-cf-data-to-aws-migration/phase-03-trading-and-durable-kv-import.md +++ b/plans/260515-2250-cf-data-to-aws-migration/phase-03-trading-and-durable-kv-import.md @@ -1,55 +1,59 @@ --- phase: 3 -title: "Trading and durable KV import" +title: "Durable KV import (trading included)" status: pending priority: P1 -effort: "4-6h" +effort: "2h" dependencies: [1, 2] --- -# Phase 03: Trading and durable KV import +# Phase 03: Durable KV import (trading included) ## Overview -Run the first real data movement into DynamoDB: import durable KV records and transform old D1 trading data into the current AWS portfolio shape. This phase intentionally does not move ephemeral state or try to preserve abandoned feature surfaces. +Copy the 9 durable Cloudflare KV records into DynamoDB under the live runtime key shape. Trading is included as a flat KV copy — the JS Worker already snapshots `Portfolio` JSON into `trading:user:*`, so no D1 transform is required (locked in Phase 01). ## Requirements -- Functional: import approved KV keys plus trading data into DynamoDB while preserving the full current `Portfolio` contract, not just balances/holdings. -- Non-functional: preserve current runtime schemas exactly, avoid duplicate writes on rerun, and keep historical-only data out of the hot path unless explicitly requested. +- Functional: each migrated KV key lands in DynamoDB at `(pk=moduleName, sk=callerKey)` with the original CF KV value placed in the `value` attribute, byte-for-byte where possible. +- Non-functional: idempotent — rerun must not duplicate or corrupt; skipped/failed records must be reported, not silently dropped. ## Architecture -- KV import target uses the same live DynamoDB shape the runtime already expects: `pk = moduleName`, `sk = caller key`. -- Durable key policy from Phase 01 drives import filters: - - keep `stats:*`, `config:*`, `subscribers`, `last_ping`, approved trading records - - reject `game:*`, `matches:*`, `sym:*` -- Trading import is a transform: - - source: D1 exports from the exact authoritative tables locked in Phase 01 - - target: `trading` module KV entries with keys `user:` - - value: `internal/modules/trading/portfolio.go` JSON shape (`currency`, `assets`, `meta`) - - `meta.createdAt` and `meta.invested` must be mapped from explicit source fields or derivations approved in Phase 01 before this phase starts -- Historical trade rows are exported for audit only unless the user later asks to restore history as a separate feature. +- Single import mode: KV → DynamoDB. No D1 transform path. +- Durable key set (locked in Phase 01): + - `wordle:stats:*` + - `loldle:stats:*` + - `loldle:config:*` + - `twentyq:stats:*` + - `lolschedule:subscribers` + - `trading:user:*` +- Skip set: `trading:sym:*` (cache), retired modules (`doantu`, `loldle-ability`, `loldle-emoji`, `semantle` stats keys), `misc:last_ping` (never written upstream). +- Optional sub-task: dump `trading_trades` (D1) to JSONL for cold audit. Not an import input. Operator-elective. ## Related Code Files -- Modify: `cmd/migrate_cf_data/main.go` — add the trading import mode once Phase 01 locks authoritative source tables -- Create: `internal/migration/trading_transform.go` -- Create: `internal/migration/kv_filter.go` -- Create: `internal/migration/import_report.go` -- Modify: `docs/cf-to-aws-migration-runbook.md` -- Read only: `internal/modules/trading/portfolio.go`, `internal/modules/wordle/state.go`, `internal/modules/loldle/state.go`, `internal/modules/twentyq/state.go`, `internal/modules/lolschedule/subscribers.go` +- Modify: `cmd/migrate_cf_data/main.go` — wire up the KV-copy run mode +- Create: `internal/migration/kv_filter.go` — durable/skip allowlist driven by the Phase 01 matrix +- Create: `internal/migration/import_report.go` — counts for imported / skipped / failed +- Create: `internal/migration/trading_audit_dump.go` — optional D1 → JSONL exporter (operator-elective) +- Modify: `docs/cf-to-aws-migration-runbook.md` — append import command + report layout +- Read only: `internal/storage/dynamodb_kv.go`, `internal/modules/trading/portfolio.go` ## Implementation Steps -1. Implement KV filters from the Phase 01 migration matrix. -2. Build the D1-to-portfolio transform using the current `Portfolio` JSON contract. -3. Import durable KV records into DynamoDB using module/key parity. -4. Import transformed trading portfolios into the `trading` module namespace. -5. Emit an import report with counts for imported, skipped, archived, and failed records. -6. Capture raw trading exports locally for audit if retention is needed outside runtime state. +1. Wire the KV allowlist filter into the import binary so only Phase 01 durable keys are read. +2. For each durable KV key, read the raw value and write to DynamoDB at the matching `(pk, sk)`. Preserve original bytes. +3. Implement idempotency via conditional `PutItem` or `attribute_not_exists` guard, fall back to overwrite with operator flag. +4. Emit an import report (stdout + JSON file) with per-prefix counts: imported, skipped, failed. +5. Add the optional `--trading-audit-dump ` flag that streams `trading_trades` rows to a local JSONL file. Default off. +6. Update the runbook with the exact command operators run, and the expected report layout. ## Success Criteria -- [ ] Durable KV keys land in DynamoDB under the exact runtime key names. -- [ ] Trading users get correct `Portfolio` JSON records, including `meta.createdAt` and `meta.invested`. -- [ ] Re-running the import does not duplicate or corrupt data. -- [ ] Skipped datasets are reported explicitly, not silently dropped. -- [ ] Historical-only trading rows are archived or intentionally ignored by policy. +- [ ] All 9 durable keys land in DynamoDB at the runtime-expected `(pk, sk)`. +- [ ] `trading:user:` round-trips through the Go runtime's `Portfolio` JSON unmarshal without modification (parity check). +- [ ] Re-running the import without flags is a no-op (no duplicates, no corruption). +- [ ] Skipped-by-policy keys (cache, retired modules) are listed in the report, not silently dropped. +- [ ] Optional `trading_trades` audit dump produces JSONL with one row per trade when flag is passed. ## Risk Assessment -Biggest risk is a wrong trading transform that gives users the wrong balances or holdings. Mitigation: derive the target object from the current `Portfolio` type only, spot-check several real users, and keep raw D1 exports so any discrepancy can be recomputed without touching Cloudflare again. +- KV value encoding drift: CF KV may return values as strings while DynamoDB attribute typing prefers `B`/`S`. Mitigation: round-trip a known portfolio record and assert byte parity before bulk import. +- Idempotency: an overwrite-by-default rerun could silently revert post-cutover writes. Mitigation: default to `attribute_not_exists` guard and require explicit `--overwrite` flag. + +## Notes +- Phase 03 used to assume a D1 → Portfolio transform. That was wrong: KV already holds the final shape. Earlier `trading_transform.go` work is dropped. diff --git a/plans/260515-2250-cf-data-to-aws-migration/plan.md b/plans/260515-2250-cf-data-to-aws-migration/plan.md index de0a965..947f93f 100644 --- a/plans/260515-2250-cf-data-to-aws-migration/plan.md +++ b/plans/260515-2250-cf-data-to-aws-migration/plan.md @@ -23,17 +23,19 @@ This plan adds the missing data-migration leg to the in-progress AWS port. AWS r ## Locked decisions - Migrate **durable user-visible data only**. - Skip ephemeral or disposable data: in-flight game state, schedule caches, stale price caches. -- Keep `misc:last_ping` unless the user explicitly accepts that reset; `/mstats` reads it today. +- ~~Keep `misc:last_ping`~~ → skip; KV inventory shows it was never written by the JS Worker (`/mstats` resets on AWS). - No admin HTTP routes. Migration runs as operator-invoked one-shot tooling. -- Rehearsal uses a staging DynamoDB table only. No wipe-and-rerun flow is allowed against the live table. -- Trading import is a **transform**, not a table copy: old D1 rows must become current KV portfolio JSON in the `trading` module. +- **Pre-cutover bulk import may target the live table directly** while the Telegram webhook still points to Cloudflare and the AWS bot has served zero writes (amended 2026-05-16). Rationale: until webhook flip, the AWS DynamoDB table is empty and no real user traffic depends on it, so a separate staging table adds setup cost without de-risking anything. **After webhook cutover, any re-import must use a staging table first.** The `attribute_not_exists` idempotency guard is the only safe write path at any time — no wipe-and-rerun flow is allowed against the live table. +- ~~Trading import is a transform~~ → **flat KV copy**. CF KV `trading:user:` already holds the final `Portfolio` JSON shape; no D1 derivation needed. (Phase 01 inventory, 2026-05-16.) - After the first AWS-served write, rollback is forward-fix only unless a reverse-sync path is built later. -- Retired module namespaces from the old CF stack are archived or ignored, not revived into AWS. +- Retired module namespaces from the old CF stack are skipped (no archive — operator decision 2026-05-16). -## Source data classes to classify in Phase 01 -- Migrate: `wordle stats:*`, `loldle stats:*`, `loldle config:*`, `twentyq stats:*`, `lolschedule subscribers`, `misc:last_ping`, trading balances/holdings. -- Skip by default: `game:*`, `matches:*`, `sym:*`. -- Decide explicitly: historical trading ledger rows, retired-module data, and the authoritative D1 fields for trading `meta.createdAt` + `meta.invested`. +## Source data classified in Phase 01 (closed 2026-05-16) +- **Migrate (9 keys):** `wordle:stats:*` (1), `loldle:stats:*` (4), `loldle:config:*` (1), `twentyq:stats:*` (1), `lolschedule:subscribers` (1), `trading:user:*` (1). +- **Skip (cache + missing + retired):** `trading:sym:*` (7), `misc:last_ping` (0), `doantu:stats:*` (2), `loldle-ability:stats:*` (1), `loldle-emoji:stats:*` (1), `semantle:stats:*` (1). +- **Archive-only (optional, operator-elective):** D1 `trading_trades` (11 rows, 1 user) — audit dump, not import input. + +Full matrix lives in `docs/cf-to-aws-migration-runbook.md`. ## Related current code - `cmd/server/main.go:167` — runtime storage backend selection (`dynamodb|firestore|memory`) @@ -51,9 +53,9 @@ This plan adds the missing data-migration leg to the in-progress AWS port. AWS r | # | Phase | Status | Effort | Key deliverable | |---|-------|--------|--------|-----------------| -| 01 | [Source inventory and migration policy](phase-01-source-inventory-and-migration-policy.md) | pending | 2-3h | exact CF namespaces/tables mapped to migrate vs skip vs archive | -| 02 | [Backfill toolchain and safety rails](phase-02-backfill-toolchain-and-safety-rails.md) | pending | 3-4h | operator-run export/import binaries + dry-run support | -| 03 | [Trading and durable KV import](phase-03-trading-and-durable-kv-import.md) | pending | 4-6h | transformed trading portfolios + durable KV records loaded into DynamoDB | +| 01 | [Source inventory and migration policy](phase-01-source-inventory-and-migration-policy.md) | completed | 2-3h | exact CF namespaces/tables mapped to migrate vs skip vs archive | +| 02 | [Backfill toolchain and safety rails](phase-02-backfill-toolchain-and-safety-rails.md) | completed | 3-4h | operator-run `cmd/migrate_cf_data` binary (inventory, kv-import, trading-audit-dump) + dry-run + idempotency | +| 03 | [Durable KV import (trading included)](phase-03-trading-and-durable-kv-import.md) | pending | 2h | flat KV→DynamoDB copy for 9 durable keys + optional D1 audit dump | | 04 | [Parity verification and rehearsal](phase-04-parity-verification-and-rehearsal.md) | pending | 2-3h | repeatable verifier, mismatch report, rollback drill | | 05 | [Cutover integration and Cloudflare decommission](phase-05-cutover-integration-and-cloudflare-decommission.md) | pending | 2-3h | AWS cutover checklist updated; CF teardown gated on verified migration | @@ -64,6 +66,6 @@ This plan adds the missing data-migration leg to the in-progress AWS port. AWS r ## Success bar - Durable CF data imported into DynamoDB with counts + sampled payload parity. -- Trading balances/holdings and required portfolio metadata match the old system after transform. +- Trading balances/holdings and required portfolio metadata match the old system byte-for-byte (flat KV copy). - Cutover runbook explicitly distinguishes pre-flip rollback from post-flip forward-fix semantics. - CF resources are not deleted until parity report is green.