diff --git a/plans/260426-2327-cloudflare-deploy-and-smoke/plan.md b/plans/260426-2327-cloudflare-deploy-and-smoke/plan.md index 0228711..62d2f1e 100644 --- a/plans/260426-2327-cloudflare-deploy-and-smoke/plan.md +++ b/plans/260426-2327-cloudflare-deploy-and-smoke/plan.md @@ -7,10 +7,13 @@ effort: 4h branch: main tags: [cloudflare-workers, deploy, smoke, atlas] created: 2026-04-26 -blockedBy: [260426-2015-cloudflare-worker-code-port] +blockedBy: [260426-2015-cloudflare-worker-code-port, 260505-1425-cloudflare-kv-migration-and-deploy] blocks: [] +supersededBy: 260505-1425-cloudflare-kv-migration-and-deploy --- +> **Superseded** — User chose Cloudflare KV over Atlas pre-emptively, before the hard-gate trip. Deploy work is now tracked in [`260505-1425-cloudflare-kv-migration-and-deploy`](../260505-1425-cloudflare-kv-migration-and-deploy/plan.md). This file is kept for history. + # Plan B — Deploy + Smoke Run only after [the code-port plan](../260426-2015-cloudflare-worker-code-port/plan.md) is complete and committed. This plan touches **real accounts**: Atlas (free M0) and Cloudflare (free Workers). diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-01-kv-repositories.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-01-kv-repositories.md new file mode 100644 index 0000000..a34e7d6 --- /dev/null +++ b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-01-kv-repositories.md @@ -0,0 +1,159 @@ +--- +phase: 1 +title: "Replace Mongo with KV in repositories" +status: completed +priority: P1 +effort: "1.5h" +dependencies: [] +--- + +# Phase 01: Replace Mongo with KV in repositories + +## Overview + +Swap the four `*-repository.js` modules from `mongodb` collection ops to `env.STORE_KV.get/put/delete`. Drop the `mongodb` driver dependency. Keep document shape and Java parity intact. + +## Requirements + +**Functional** +- All 13 Telegram commands continue to work without behavior change. +- Daily cron continues to enumerate groups and report stale apps. +- Cached app entries expire after `APP_CACHE_SECONDS`. + +**Non-functional** +- Worker bundle ≤ 500 KiB. +- No `nodejs_compat_v2` flag required. +- `npm install` no longer pulls `mongodb`. + +## Architecture + +KV access pattern (single namespace `STORE_KV`, prefixed keys): + +``` +admin → JSON of {class:"Admin", groups:[...]} +group:{chatId} → JSON of {class:"Group", _id, appleApps[], googleApps[]} +apple:{appId} → JSON of cached Apple response (KV TTL = APP_CACHE_SECONDS) +google:{appId} → JSON of cached Google response (KV TTL = APP_CACHE_SECONDS) +``` + +Read pattern: `await env.STORE_KV.get(key, 'json')` returns parsed JSON or `null`. +Write pattern: `await env.STORE_KV.put(key, JSON.stringify(value), opts?)`. +Delete pattern: `await env.STORE_KV.delete(key)`. + +Cache TTL: pass `{ expirationTtl: appCacheSeconds }` to `put` for `apple:` and `google:` keys. **KV minimum `expirationTtl` is 60s** — Java/Mongo had no such floor. Guard in `kv.js`: clamp `expirationTtl` to `Math.max(60, value)` so an aggressive `APP_CACHE_SECONDS` override doesn't make `put` reject. Document the floor in `.dev.vars.example`. + +Drop the `isAppleAppExpired` / `isGoogleAppExpired` helpers from the read path — KV deletes expired keys, so a `get` returning `null` is the cache miss. Keep the helpers if other code references them; otherwise delete. + +## Related Code Files + +**Replace (rewrite contents)** +- `src/repository/admin-repository.js` — KV ops on key `admin` +- `src/repository/group-repository.js` — KV ops on key `group:{chatId}` +- `src/repository/apple-app-repository.js` — KV ops on key `apple:{appId}` with TTL +- `src/repository/google-app-repository.js` — KV ops on key `google:{appId}` with TTL + +**Delete** +- `src/repository/mongodb.js` — no longer needed + +**Modify** +- `src/repository/store.js` — remove memoization comment about `MongoClient`; factory signature unchanged +- `src/index.js` — drop the "memoized MongoClient" comment block; nothing else changes (the `env` already flows through) +- `src/config.js` — remove `MONGODB_URI` from required env list +- `package.json` — `npm uninstall mongodb`; bump version to 0.3.0 + +**Untouched (verify)** +- `src/models/*.js` — document shape preserved verbatim (Java parity at the doc level: `_id`, `class`, all fields) +- `src/bot/commands/*.js` — repository interfaces unchanged +- `src/scheduler/scheduler.js` — calls `store.admin.getAllGroups()` and `store.group.getGroup()`; both still resolve + +## Java parity preserved (per existing parity reports) + +Cross-checked against `plans/reports/xia-260429-1601-store-scraper-bot-port-status.md` and `researcher-260429-1555-java-vs-js-parity.md`. KV swap preserves every PARITY item: + +- Schema (`_id`, `class`, field names) — written verbatim as JSON values. +- Cache TTL semantics — KV `expirationTtl` is behaviorally equivalent to Java's `(now - millis) > cacheMillis` check on read. Both produce a "cache miss → re-fetch" outcome at the same boundary. +- `getAdmin()` returning `newAdmin()` if doc missing — `getJson('admin') ?? newAdmin()` matches. +- Existence checks — `(await get) !== null` replaces `countDocuments`. +- Concurrent `addGroup` race — Java/Mongo had the same non-atomic R-M-W; not a regression. + +**Two intentional divergences from Java:** + +1. **Eventual consistency (~60s globally)**: Java/Mongo gave strong read-after-write. For a single-admin daily-cron bot this is acceptable; documented in `plan.md`. +2. **TTL floor of 60s**: Java had no minimum. Mitigated by the `kv.js` clamp above. + +## Implementation Steps + +1. **Write `kv.js` helper** at `src/repository/kv.js` — thin wrapper exposing `getJson(env, key)`, `putJson(env, key, value, opts?)`, `del(env, key)`. Single-binding-name constant `STORE_KV`. Throws a typed error if `env.STORE_KV` is missing. + +2. **Rewrite `admin-repository.js`**: + - `init()` → `getJson('admin')`; if null, `putJson('admin', newAdmin())`. + - `getAdmin()` → `getJson('admin') ?? newAdmin()`. + - `save(admin)` → `putJson('admin', admin)`. + - `addGroup`/`removeGroup`/`hasGroup`/`getAllGroups` — unchanged (delegate to `getAdmin` + `save`). + +3. **Rewrite `group-repository.js`**: + - `key(groupId)` → `` `group:${groupIdToKey(groupId)}` ``. + - `exists(groupId)` → `getJson(key(groupId)) !== null`. + - `getGroup(groupId)` → `getJson(key(groupId)) ?? newGroup(groupId)`. + - `saveGroup(group)` → `putJson(\`group:${group._id}\`, group)`. + - `deleteGroup(groupId)` → `del(key(groupId))`. + - `mutateAndSave` unchanged. + +4. **Rewrite `apple-app-repository.js`**: + - `key(appId)` → `` `apple:${appId}` ``. + - `get(appId)` → `getJson(key(appId))`. + - `save(entry)` → `putJson(key(entry._id), entry, { expirationTtl: appCacheSeconds })`. + - `getCached(appId)` → just `get(appId)` — TTL already enforced by KV. Remove the `isAppleAppExpired` call. + +5. **Rewrite `google-app-repository.js`** — same pattern as apple. + +6. **Delete `src/repository/mongodb.js`**. Remove its imports from the four repositories. + +7. **Update `src/config.js`** — drop `'MONGODB_URI'` from the `required` list. + +8. **Update `src/index.js`** — strip the `MongoClient`-related comment in `build()`. Add a sanity check: throw if `env.STORE_KV` is undefined (early failure beats null-dereference at first request). + +9. **Update `package.json`**: + - `npm uninstall mongodb` + - bump `version` to `0.3.0` + +10. **Compile/lint check**: `node --check` each modified file. Run `npm run lint` (the existing secret-leak check). + +11. **Local sanity (no live KV needed)**: `node -e "import('./src/repository/store.js').then(...)"` — confirm no import errors. Cannot fully test without `wrangler dev` (Phase 03). + +## Todo List + +- [ ] Create `src/repository/kv.js` helper +- [ ] Rewrite `admin-repository.js` +- [ ] Rewrite `group-repository.js` +- [ ] Rewrite `apple-app-repository.js` (with KV TTL) +- [ ] Rewrite `google-app-repository.js` (with KV TTL) +- [ ] Delete `src/repository/mongodb.js` +- [ ] Update `src/config.js` required env list +- [ ] Update `src/index.js` env validation +- [ ] `npm uninstall mongodb`; bump `package.json` version +- [ ] `node --check` all modified files +- [ ] `npm run lint` +- [ ] **Java parity check**: spot-diff a written `group:{id}` JSON value against an equivalent Java/Mongo doc structure (sample from upstream `tiennm99/store-scraper-bot` model classes). Field names, ordering-insensitive, must match. + +## Success Criteria + +- [ ] `grep -rn "mongodb\|MongoClient" src/` returns nothing. +- [ ] `package.json` `dependencies` is empty (only `wrangler` in devDeps). +- [ ] All 4 repository factories return the same interface they did before (consumers untouched). +- [ ] `node --check src/index.js` exits 0. +- [ ] `npm run lint` passes. + +## Risk Assessment + +| Risk | Likelihood | Mitigation | +|---|---|---| +| Repository interface drift breaks command handlers | Low | Keep the exported function names + signatures byte-identical | +| KV's eventual consistency surprises a user mid-conversation | Low | Single-admin bot, low concurrency; document in README | +| Concurrent `/addgroup` race clobbers `groups[]` | Low | Same race existed in Mongo; not a regression | +| TTL `expirationTtl` < 60 silently fails | Low | `APP_CACHE_SECONDS` defaults to 600; doc the floor in `.dev.vars.example` | + +## Security Considerations + +- KV namespace ID is not a secret (binding name is what code references). Safe to commit in `wrangler.toml`. +- Removing `MONGODB_URI` removes one secret from the rotation surface. Update `wrangler secret list` documentation in Phase 05. diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-02-wrangler-and-kv-namespace.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-02-wrangler-and-kv-namespace.md new file mode 100644 index 0000000..48105d3 --- /dev/null +++ b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-02-wrangler-and-kv-namespace.md @@ -0,0 +1,96 @@ +--- +phase: 2 +title: "Wrangler config + KV namespace provisioning" +status: pending +priority: P1 +effort: "0.5h" +dependencies: [1] +--- + +# Phase 02: Wrangler config + KV namespace provisioning + +## Overview + +Provision a Cloudflare KV namespace, bind it to the Worker as `STORE_KV`, and remove now-unused MongoDB-related config from `wrangler.toml`. + +## Requirements + +**Functional** +- Worker has access to `env.STORE_KV` at runtime. +- Local `wrangler dev` reads from a preview namespace, not production. + +**Non-functional** +- `wrangler.toml` is the single source of truth (no `wrangler.jsonc`). +- Production and preview namespace IDs are committed (these are not secrets). + +## Architecture + +KV bindings are declared in `wrangler.toml`: + +```toml +[[kv_namespaces]] +binding = "STORE_KV" +id = "" +preview_id = "" +``` + +`binding` is the name surfaced as `env.STORE_KV` in the Worker. `id` is the production namespace UUID. `preview_id` is used by `wrangler dev`. + +Operator-driven steps (run locally; require `wrangler login`): + +```sh +npx wrangler kv namespace create STORE_KV +npx wrangler kv namespace create STORE_KV --preview +``` + +Each command outputs a TOML snippet to copy into `wrangler.toml`. + +## Related Code Files + +**Modify** +- `wrangler.toml` — add `[[kv_namespaces]]` block; remove `nodejs_compat_v2` flag; trim the secrets-comment block to drop `MONGODB_URI`. + +**No code changes** — Phase 01 already wrote everything that consumes `env.STORE_KV`. + +## Implementation Steps + +1. **Run** `npx wrangler kv namespace create STORE_KV` — capture the production `id`. +2. **Run** `npx wrangler kv namespace create STORE_KV --preview` — capture the `preview_id`. +3. **Edit `wrangler.toml`**: + - Add the `[[kv_namespaces]]` block with both IDs. + - Change `compatibility_flags` from `["nodejs_compat_v2"]` to `[]` (or delete the line entirely). + - In the `# Secrets` comment block, remove the `MONGODB_URI` line. +4. **Verify** with `npx wrangler types` (if available) or `npx wrangler deploy --dry-run` — should report the new binding without errors. +5. **Local seed (optional)**: For `wrangler dev` to be useful, the admin singleton must exist. The repository's `init()` is currently called nowhere automatically. Either: + - Seed manually: `npx wrangler kv key put --binding STORE_KV --preview admin '{"_id":"admin","class":"Admin","groups":[]}'` + - Or update `src/index.js` to call `store.admin.init()` on first request (cheap, idempotent). **Recommended**: add the lazy init. + +## Todo List + +- [ ] `wrangler kv namespace create STORE_KV` +- [ ] `wrangler kv namespace create STORE_KV --preview` +- [ ] Add `[[kv_namespaces]]` block to `wrangler.toml` +- [ ] Drop `nodejs_compat_v2` from `compatibility_flags` +- [ ] Remove `MONGODB_URI` mention from the secrets comment +- [ ] Add lazy `store.admin.init()` on first request in `src/index.js` +- [ ] `wrangler deploy --dry-run` succeeds + +## Success Criteria + +- [ ] `wrangler.toml` has `[[kv_namespaces]]` with `binding = "STORE_KV"` and both IDs. +- [ ] `compatibility_flags` no longer contains `nodejs_compat_v2`. +- [ ] `wrangler deploy --dry-run` exits 0 and lists the binding. +- [ ] `wrangler kv namespace list` shows two namespaces: `STORE_KV` and `STORE_KV_preview` (or named per CF convention). + +## Risk Assessment + +| Risk | Likelihood | Mitigation | +|---|---|---| +| Wrong `binding` name → `env.STORE_KV` is undefined at runtime | Low | Phase 01 added the early validation throw | +| Forgot `preview_id`, `wrangler dev` errors | Low | Step 2 explicitly creates it | +| Operator commits secret accidentally | Low | KV IDs are not secrets; existing `npm run lint` scans for the secret patterns we care about | + +## Security Considerations + +- KV namespace IDs are not secrets and are safe to commit. +- Production `STORE_KV` will hold Telegram chat IDs and admin IDs — same sensitivity as Mongo previously held. Cloudflare KV is encrypted at rest. diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-03-deploy-and-bundle-gate.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-03-deploy-and-bundle-gate.md new file mode 100644 index 0000000..0e0db84 --- /dev/null +++ b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-03-deploy-and-bundle-gate.md @@ -0,0 +1,109 @@ +--- +phase: 3 +title: "Deploy + bundle-size sanity gate" +status: pending +priority: P1 +effort: "0.5h" +dependencies: [2] +--- + +# Phase 03: Deploy + bundle-size sanity gate + +## Overview + +Set Worker secrets, run `wrangler deploy`, confirm the bundle is well under the Free 3 MiB cap, and verify a cold-start request reaches the Worker. + +## Requirements + +**Functional** +- Worker is reachable at `https://js-store-scraper-bot..workers.dev`. +- A request without the secret header returns 401. + +**Non-functional** +- Bundle ≤ 500 KiB (informational; we expect ~30 KiB without `mongodb`). +- Cold start CPU < 10ms. + +## Architecture + +`wrangler deploy` reads secrets from the Cloudflare account (set via `wrangler secret put`), KV bindings from `wrangler.toml`, and uploads the bundled Worker. No Atlas, no TCP — just KV (HTTP edge-side), `fetch` to Telegram + iTunes + Google Play. + +Required secrets (set once via `wrangler secret put `): + +| Secret | Purpose | +|---|---| +| `TELEGRAM_BOT_TOKEN` | Posting to Telegram Bot API | +| `TELEGRAM_BOT_USERNAME` | Bot mention parsing | +| `TELEGRAM_WEBHOOK_SECRET` | Validates `X-Telegram-Bot-Api-Secret-Token` | +| `ADMIN_IDS` | Comma-separated Telegram user IDs | + +`MONGODB_URI` is **no longer required**. If a previous deploy set it, leave it (no-op) or run `wrangler secret delete MONGODB_URI` for hygiene. + +## Related Code Files + +**No code changes.** Phase deals with deployment only. + +## Implementation Steps + +1. **Set secrets** (interactive prompts; values supplied by operator): + ```sh + npx wrangler secret put TELEGRAM_BOT_TOKEN + npx wrangler secret put TELEGRAM_BOT_USERNAME + npx wrangler secret put TELEGRAM_WEBHOOK_SECRET + npx wrangler secret put ADMIN_IDS + ``` + `TELEGRAM_WEBHOOK_SECRET`: any random ≥32 chars, e.g. `openssl rand -hex 24`. + +2. **Optional cleanup**: `npx wrangler secret delete MONGODB_URI` if it was set previously. + +3. **Bundle gate**: + ```sh + npx wrangler deploy --dry-run --outdir=.bundle 2>&1 | tee .bundle/dry-run.log + du -h .bundle/*.js + ``` + Expected: total .js output < 500 KiB. If > 500 KiB, abort and investigate (`wrangler deploy --dry-run --metafile` then inspect imports). + +4. **Real deploy**: + ```sh + npx wrangler deploy + ``` + Capture the deployed URL. + +5. **Cold-start probe**: send a probe with a bad secret to confirm the 401 path works (cheap, doesn't touch KV): + ```sh + curl -i -X POST https://js-store-scraper-bot..workers.dev \ + -H "X-Telegram-Bot-Api-Secret-Token: wrong" \ + -d '{}' + ``` + Expected: `HTTP/1.1 401 Unauthorized`. + +6. **CPU check**: open the Cloudflare dashboard → Workers → js-store-scraper-bot → Logs/Metrics. After a few requests, confirm cold-start CPU is well under 50ms (target <10ms without Mongo). + +## Todo List + +- [ ] Set 4 required secrets via `wrangler secret put` +- [ ] (Optional) Delete obsolete `MONGODB_URI` secret +- [ ] Run `wrangler deploy --dry-run`; verify bundle < 500 KiB +- [ ] `wrangler deploy`; capture URL +- [ ] curl probe → expect 401 +- [ ] Confirm cold-start CPU < 10ms in CF dashboard + +## Success Criteria + +- [ ] Bundle size logged and < 500 KiB. +- [ ] `wrangler deploy` succeeds with `Uploaded` line. +- [ ] 401 probe returns 401 status. +- [ ] CPU per request < 10ms in dashboard logs. + +## Risk Assessment + +| Risk | Likelihood | Mitigation | +|---|---|---| +| Bundle balloons due to a transitive dep | Very Low | `mongodb` was the only heavy dep; with it gone, only stdlib + small `wrangler` runtime hooks remain | +| CF account quota exceeded | Very Low | New account = 100k requests/day budget; bot traffic is single-digit per minute | +| Secret typo (e.g. token) → 401 from Telegram on first webhook hit | Medium | Caught in Phase 04 smoke; rerun `wrangler secret put` to fix | + +## Security Considerations + +- All secrets are set via `wrangler secret put` (never in `wrangler.toml`). +- The 401 probe uses a deliberately wrong secret; no sensitive value is logged. +- Run `npm run lint` (existing secret-leak check) before deploy. diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-04-register-webhook-and-smoke.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-04-register-webhook-and-smoke.md new file mode 100644 index 0000000..b92200b --- /dev/null +++ b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-04-register-webhook-and-smoke.md @@ -0,0 +1,135 @@ +--- +phase: 4 +title: "Register webhook + smoke-test" +status: pending +priority: P1 +effort: "1.5h" +dependencies: [3] +--- + +# Phase 04: Register webhook + smoke-test + +## Overview + +Point the Telegram Bot API at the deployed Worker, then exercise every command and the daily cron path against the live deployment. + +## Requirements + +**Functional** +- Telegram delivers updates to `https://` with the secret header. +- All 13 commands return correct responses. +- KV reads/writes show up in `wrangler kv key list --binding STORE_KV`. + +**Non-functional** +- Each command response < 1s (Telegram's webhook timeout is 30s, but slow webhooks lead to retries). + +## Architecture + +Telegram webhook flow: + +``` +Telegram → POST https://js-store-scraper-bot..workers.dev + with header X-Telegram-Bot-Api-Secret-Token = + → Worker validates header (401 on mismatch) + → Worker parses update, ctx.waitUntil(dispatch(...)) + → Worker returns 200 immediately + → dispatch reads/writes STORE_KV, calls Telegram sendMessage +``` + +The cron schedule (`0 0 * * *` UTC = 7am `Asia/Ho_Chi_Minh`) triggers `scheduled()` daily; we manually trigger it once during smoke via `wrangler triggers cron`. + +## Related Code Files + +**No code changes.** Smoke uses the existing `scripts/register-webhook.js` and live commands. + +## Implementation Steps + +1. **Populate `.env.deploy`** from `.env.deploy.example`: + ``` + TELEGRAM_BOT_TOKEN=... + TELEGRAM_WEBHOOK_SECRET=... # must match the secret set in Phase 03 + WORKER_URL=https://js-store-scraper-bot..workers.dev + ``` + +2. **Dry-run register**: + ```sh + npm run register:dry + ``` + Confirms the request body is sane. + +3. **Real register**: + ```sh + npm run register + ``` + Calls `setWebhook` + `setMyCommands`. Expect `ok: true` from Telegram. + +4. **Verify webhook info**: + ```sh + curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getWebhookInfo" | jq + ``` + `url` should match, `pending_update_count` should be 0 once cleared, `has_custom_certificate: false`. + +5. **Smoke test (operator runs in Telegram)**: + + | Command | Expected | KV mutation | + |---|---|---| + | `/info` | bot info text (admin id, build time) | none | + | `/addgroup` (in target chat as admin) | "Group added" | `admin` ← `groups` includes chat id | + | `/listgroup` | shows new group | none | + | `/addapple ` | "Added" | `group:{chatId}` updated | + | `/addgoogle ` | "Added" | `group:{chatId}` updated | + | `/listapp` | renders both lists | none | + | `/checkapp` | table with ✅/❌ marks | populates `apple:{appId}` / `google:{appId}` | + | `/checkappscore` | scores table | uses cache from previous step | + | `/rawappleapp ` | raw JSON dump | reads `apple:{appId}` | + | `/rawgoogleapp ` | raw JSON dump | reads `google:{appId}` | + | `/delapple ` | "Removed" | `group:{chatId}` updated | + | `/delgoogle ` | "Removed" | `group:{chatId}` updated | + | `/delgroup` | "Group removed" | `admin.groups` shrinks | + +6. **Inspect KV** between steps: + ```sh + npx wrangler kv key list --binding STORE_KV --remote | jq + npx wrangler kv key get --binding STORE_KV --remote admin | jq + ``` + +7. **Trigger cron manually**: + ```sh + npx wrangler triggers cron "0 0 * * *" + ``` + (Or use the dashboard's "Trigger" button.) Re-add a test app with a known-stale `updated` date (or a real lagging app) and confirm the daily report is posted to the test group. + +8. **Cache TTL probe**: read `apple:` from KV — `expiration` field should be `now + APP_CACHE_SECONDS`. After waiting past TTL, the next `/rawappleapp` should re-fetch from iTunes. + +## Todo List + +- [ ] Populate `.env.deploy` +- [ ] `npm run register:dry` +- [ ] `npm run register` +- [ ] `getWebhookInfo` reports correct URL +- [ ] Walk through all 13 commands; record responses +- [ ] `wrangler kv key list` shows expected keys after mutations +- [ ] Manually trigger cron; verify daily report message +- [ ] Confirm cache TTL behavior + +## Success Criteria + +- [ ] All 13 commands return success responses (no "Internal server error"). +- [ ] KV inspection matches expected mutations after each command. +- [ ] Manual cron trigger posts a report (or "no stale apps" if none). +- [ ] Worker logs show no unhandled exceptions across the smoke run. + +## Risk Assessment + +| Risk | Likelihood | Mitigation | +|---|---|---| +| Webhook secret mismatch → 401 loop | Low | Step 4 verifies `getWebhookInfo`; rotate via `setWebhook` if needed | +| Telegram rate-limited during smoke | Low | Free tier limits are generous for ad-hoc testing; pause if 429 | +| KV eventual-consistency surprises smoke (write then immediate read on different POP) | Low | Same-region writes are visible immediately on modern KV; watch for the rare read-your-write miss and retry | +| `/checkapp` 30s timeout if upstream stores slow | Medium | Both scrapers have own timeouts; if hit, rerun once | + +## Security Considerations + +- `TELEGRAM_WEBHOOK_SECRET` must be ≥32 chars and unique to this Worker. +- After smoke, verify only authorized admin IDs can issue mutating commands by attempting from a non-admin account → expect "Unauthorized". +- Capture Worker logs from the dashboard and grep for accidental token leaks (none expected; existing structured logger redacts). diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-05-docs-and-cleanup.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-05-docs-and-cleanup.md new file mode 100644 index 0000000..5060b42 --- /dev/null +++ b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-05-docs-and-cleanup.md @@ -0,0 +1,136 @@ +--- +phase: 5 +title: "Docs + cleanup" +status: pending +priority: P2 +effort: "1h" +dependencies: [4] +--- + +# Phase 05: Docs + cleanup + +## Overview + +Update README + project docs to reflect KV-backed storage, archive the obsolete Atlas plan, and remove now-dead Mongo references from `.env*` examples and Docker files. + +## Requirements + +**Functional** +- A new operator can deploy from scratch by following only `README.md` + `docs/deployment-guide.md`. +- No file references MongoDB Atlas as the production storage. + +**Non-functional** +- Keep `docs/*.md` under 800 LOC each (per `docs.maxLoc`). + +## Architecture + +Documentation surface: + +| File | Action | +|---|---| +| `README.md` | Replace MongoDB section with KV; update env table; bump status note | +| `.env.example` | Remove `MONGODB_*`; add note that secrets live in `wrangler secret put` | +| `.dev.vars.example` | Replace `MONGODB_URI=` with KV preview-namespace seeding hint | +| `docs/deployment-guide.md` | Create or update with the Phase 02–04 operator runbook | +| `docs/system-architecture.md` | Diagram: Worker → KV (drop Atlas) | +| `plans/todo.md` | Remove the Atlas pre-flight items; update active-plan link | +| `plans/260426-2327-cloudflare-deploy-and-smoke/` | Move under `plans/archive/` with a "superseded by 260505-1425" note | + +Docker files (`Dockerfile`, `docker-compose.yml`, `docker-compose.dev.yml`) reference the old polling path. **Decision**: keep them as a fallback rollback option but add a banner comment that they target the legacy Mongo runtime, not Workers. + +## Related Code Files + +**Modify** +- `README.md` +- `.env.example` +- `.dev.vars.example` +- `plans/todo.md` +- `Dockerfile` (banner comment only) +- `docker-compose.yml` (banner comment only) +- `docker-compose.dev.yml` (banner comment only) + +**Create** +- `docs/deployment-guide.md` (if missing) +- `docs/system-architecture.md` (if missing) — short, ASCII or mermaid diagram + +**Move (archive)** +- `plans/260426-2327-cloudflare-deploy-and-smoke/` → `plans/archive/260426-2327-cloudflare-deploy-and-smoke/` + +## Implementation Steps + +1. **`README.md`**: + - Remove "Requirements: MongoDB 4.4+". + - Update env table: drop `MONGODB_CONNECTION_STRING`, `MONGO_DATABASE`; note that secrets go via `wrangler secret put`, KV binding is `STORE_KV`. + - Replace "Run via Docker Compose" with "Deploy: `npm run deploy`". + - Update "Differences vs Go / Java" to note storage backend differs (KV instead of Mongo) but document shape preserved. + +2. **`.env.example`**: + - Drop `MONGODB_*` rows; add a comment that production secrets live in `wrangler secret`. + - Keep `ADMIN_IDS`, `TELEGRAM_*` for local-dev reference only. + +3. **`.dev.vars.example`** — drop `MONGODB_URI`. + +4. **`docs/deployment-guide.md`** — short runbook condensing Phases 02–04 (KV namespace creation, secrets, deploy, register webhook, smoke). One file, < 200 lines. + +5. **`docs/system-architecture.md`** — mermaid diagram: + ``` + Telegram --webhook--> CF Worker --KV ops--> STORE_KV + --HTTPS--> iTunes / Play + cron --7am VN--> daily check + ``` + +6. **`plans/todo.md`**: + - Replace "Active plan" link with `260505-1425-cloudflare-kv-migration-and-deploy`. + - Remove the Atlas-specific hard-gate table (no longer applicable). + - Update pre-flight: drop Atlas account requirement; KV needs only CF account. + +7. **Archive old plan**: + ```sh + mv plans/260426-2327-cloudflare-deploy-and-smoke plans/archive/ + ``` + Append a single line at top of its `plan.md`: `> **Superseded by [`260505-1425-cloudflare-kv-migration-and-deploy`](../../260505-1425-cloudflare-kv-migration-and-deploy/plan.md)** — KV pivot taken pre-emptively.` + +8. **Docker file banners** — one-line comment at top: + ``` + # NOTE: Legacy Mongo polling path. Production deploys to Cloudflare Workers via `npm run deploy` and uses Cloudflare KV. + ``` + +9. **Final commit**: single squashed commit `feat: migrate to Cloudflare KV storage + deploy` with the full Phase 01–04 diff. Use conventional-commits. + +## Todo List + +- [ ] Update `README.md` +- [ ] Update `.env.example` +- [ ] Update `.dev.vars.example` +- [ ] Create/update `docs/deployment-guide.md` +- [ ] Create/update `docs/system-architecture.md` +- [ ] Update `plans/todo.md` +- [ ] Move old Atlas plan under `plans/archive/` +- [ ] Add legacy banners to Docker files +- [ ] Commit + push + +## Success Criteria + +- [ ] `grep -rn "MONGODB\|mongodb\|Atlas" README.md docs/ .env.example .dev.vars.example` returns nothing. +- [ ] `docs/deployment-guide.md` exists and is < 200 lines. +- [ ] Old plan lives under `plans/archive/` with the supersession banner. +- [ ] `git log --oneline -5` shows a clean conventional commit. + +## Risk Assessment + +| Risk | Likelihood | Mitigation | +|---|---|---| +| Stale Atlas reference left in some doc | Low | grep step in success criteria | +| Doc exceeds `docs.maxLoc=800` | Low | Each doc is short by design | +| Operator follows old plan by accident | Low | Archive move + supersession banner make it obvious | + +## Security Considerations + +- Confirm no secrets are pasted into README or examples. +- `npm run lint` (the existing `scripts/check-secret-leaks.js`) runs at commit time per existing workflow. + +## Next Steps (out of this plan, into backlog) + +- CI workflow that runs `wrangler deploy --dry-run` as a PR bundle-size gate. +- Observability dashboard pointed at the `STORE_KV` namespace (request count + key count). +- Tests — original project has none; would be additive work. diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md new file mode 100644 index 0000000..7764e75 --- /dev/null +++ b/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md @@ -0,0 +1,100 @@ +--- +title: "Cloudflare KV migration + deploy" +description: "Replace MongoDB Atlas with Cloudflare KV as the storage layer, deploy to Workers, register Telegram webhook, smoke-test, and document. Supersedes the Atlas-based deploy plan." +status: pending +priority: P1 +effort: 5h +branch: main +tags: [cloudflare-workers, cloudflare-kv, deploy, migration] +created: 2026-05-05 +blockedBy: [] +blocks: [260426-2327-cloudflare-deploy-and-smoke] +--- + +# Cloudflare Workers + KV Storage Migration + +User chose KV over Atlas to pre-empt the bundle-size and cold-start risks called out in the Atlas plan. Drops the `mongodb` driver entirely (~4–5 MiB compressed) and the `nodejs_compat_v2` flag along with it. + +## Why KV fits this codebase + +Every existing repository operation is a primary-key lookup — no queries, no aggregations, no cursors: + +| Logical collection | Operations | KV key shape | +|---|---|---| +| `common` (admin singleton) | get/save the single admin doc with `groups[]` | `admin` | +| `group` (per-chat) | exists/get/save/delete by groupId | `group:{groupId}` | +| `apple_app` (cache) | get/save by appId, TTL expiry | `apple:{appId}` (with `expirationTtl`) | +| `google_app` (cache) | get/save by appId, TTL expiry | `google:{appId}` (with `expirationTtl`) | + +KV's native `expirationTtl` replaces the manual `isExpired(entry, now, cacheMs)` check. + +The daily cron reads `admin.groups[]` (single key), then iterates groups (one read each). No list-keys needed. + +## Phases + +| # | Phase | Status | Effort | +|---|---|---|---| +| 01 | [Replace Mongo with KV](phase-01-kv-repositories.md) | completed | 1.5h | +| 02 | [Wrangler + KV namespace](phase-02-wrangler-and-kv-namespace.md) | pending | 0.5h | +| 03 | [Deploy + bundle gate](phase-03-deploy-and-bundle-gate.md) | pending | 0.5h | +| 04 | [Register webhook + smoke](phase-04-register-webhook-and-smoke.md) | pending | 1.5h | +| 05 | [Docs + cleanup](phase-05-docs-and-cleanup.md) | pending | 1h | + +## Critical dependencies + +- 01 → 02 (wrangler binding name must match what code reads from `env.STORE_KV`) +- 02 → 03 (deploy needs the namespace ID provisioned) +- 03 → 04 (webhook registration needs live Worker URL) + +## Hard gates (now sanity checks, not pivots) + +| Gate | Threshold | Notes | +|---|---|---| +| Bundle size | ≤ 500 KiB (well under 3 MiB Free cap) | Without `mongodb`, bundle should drop to ~30 KiB | +| Cold start | < 10ms CPU | No TCP handshake; KV access is HTTP-fetched edge-side | + +If bundle exceeds 500 KiB, investigate — something pulled in unintended deps. + +## Key design decisions + +1. **One KV namespace, prefixed keys** — cleaner than four namespaces; KV billing is by op count not namespace. +2. **Drop `nodejs_compat_v2`** — only needed for the Mongo TCP socket. Removing reduces cold-start and runtime memory. +3. **Drop `mongodb` dependency** — removes ~4–5 MiB compressed. `npm uninstall mongodb`. +4. **Cache TTL via `expirationTtl`** — let KV handle expiry; remove `isAppleAppExpired`/`isGoogleAppExpired` helpers from the read path. +5. **Document model unchanged** — same `_id`, `class`, fields. Stored as JSON values. Java/Go parity preserved at the document level even though the storage engine differs. + +## Java parity verification + +Cross-checked against existing parity reports (commit `e3e375d`): +- `plans/reports/xia-260429-1601-store-scraper-bot-port-status.md` — 11 PARITY · 2 MINOR · **0 GAP** · 3 EXTRA +- `plans/reports/researcher-260429-1555-java-vs-js-parity.md` — schema, scheduler timing, cache TTL, HTML formatting all confirmed identical + +**KV swap preserves every PARITY item.** Document shape (`_id`, `class`, fields) is written verbatim as JSON values. Cache TTL via `expirationTtl` is behaviorally equivalent to Java's `(now - millis) > cacheMillis` read-side check (same cache-miss→refetch boundary). `countDocuments` and `findOne` map cleanly to `get !== null` and `get`. + +Two intentional divergences from Java/Mongo, both accepted: +1. KV ~60s eventual consistency (Java/Mongo was strongly consistent) — fine for single-admin daily-cron use. +2. KV `expirationTtl` 60s floor (Java had no minimum) — mitigated by a clamp in `kv.js`. + +## Known KV trade-offs (accepted) + +- **Eventual consistency (~60s global propagation)**: `/addgroup` followed by an immediate cron run on a different POP could miss the new group. For this single-admin bot with a once-daily cron, acceptable. +- **No atomic read-modify-write**: Two concurrent `/addgroup` calls could clobber each other on the singleton's `groups[]` array. Mongo's `replaceOne` had the same race; not a regression. +- **Free tier**: 100k reads/day, 1k writes/day, 1 GB storage. Bot usage projects <100 reads/day. + +## Out of scope + +- Data migration — greenfield per `plans/todo.md` Q3. +- Tests — none exist; not adding here. +- Custom domain — `*.workers.dev` URL is fine for Telegram webhooks. +- Durable Objects / D1 — KV is sufficient. + +## Rollback path + +- After Phase 01 (code-only): `git revert` reverts the repo refactor; `mongodb` driver back, plan reopens against Atlas-based deploy plan. +- After Phase 03 (deployed but no webhook): `wrangler delete` removes Worker. +- After Phase 04 (webhook live): `setWebhook` with empty URL clears it; rotate `TELEGRAM_WEBHOOK_SECRET` so any in-flight POSTs 401. +- KV namespace can be retained — empty namespaces cost nothing on Free tier. + +## Supersedes + +This plan replaces the storage backend chosen in [`260426-2327-cloudflare-deploy-and-smoke`](../260426-2327-cloudflare-deploy-and-smoke/plan.md). On completion, that plan should be archived.