From c32688f41bd0344f81513f9322edcbd7bd93fb86 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Sat, 9 May 2026 23:12:17 +0700 Subject: [PATCH] refactor: drop implemented plans + reports Migration to Vercel + Upstash is live and the Java bot is being shut down. Git history retains all phase docs and audit reports if needed for reference. --- .../phase-01-atlas-and-secrets.md | 83 ---- .../phase-02-deploy-and-gates.md | 89 ---- .../phase-03-register-webhook.md | 43 -- .../phase-04-smoke-and-docs.md | 96 ---- .../plan.md | 68 --- .../phase-01-kv-repositories.md | 159 ------- .../phase-02-wrangler-and-kv-namespace.md | 96 ---- .../phase-03-deploy-and-bundle-gate.md | 109 ----- .../phase-04-register-webhook-and-smoke.md | 135 ------ .../phase-05-docs-and-cleanup.md | 136 ------ .../phase-06-atlas-data-migration.md | 145 ------ .../plan.md | 106 ----- .../phase-01-vercel-scaffolding.md | 74 --- .../phase-02-upstash-repository-adapter.md | 105 ----- .../phase-03-inline-scraper-modules.md | 89 ---- .../phase-04-http-layer-webhook-and-cron.md | 133 ------ ...hase-05-data-migration-atlas-to-upstash.md | 96 ---- ...6-deploy-cutover-and-webhook-reregister.md | 108 ----- .../phase-07-cleanup-wrangler-and-docs.md | 105 ----- .../plan.md | 67 --- .../phase-01-wrangler-config-deps-lint.md | 219 --------- .../phase-02-telegram-client.md | 182 -------- .../phase-03-mongo-worker-adapter.md | 185 -------- .../phase-04-worker-entry.md | 246 ---------- .../phase-05-register-script-write-only.md | 120 ----- .../plan.md | 92 ---- ...-260509-1656-consolidate-vercel-upstash.md | 183 -------- ...60509-1801-vercel-upstash-consolidation.md | 66 --- ...esearcher-260429-1555-java-vs-js-parity.md | 424 ------------------ ...researcher-260509-1656-upstash-vs-atlas.md | 96 ---- ...60509-1801-vercel-upstash-consolidation.md | 39 -- ...0429-1601-store-scraper-bot-port-status.md | 121 ----- plans/todo.md | 38 +- 33 files changed, 10 insertions(+), 4043 deletions(-) delete mode 100644 plans/260426-2327-cloudflare-deploy-and-smoke/phase-01-atlas-and-secrets.md delete mode 100644 plans/260426-2327-cloudflare-deploy-and-smoke/phase-02-deploy-and-gates.md delete mode 100644 plans/260426-2327-cloudflare-deploy-and-smoke/phase-03-register-webhook.md delete mode 100644 plans/260426-2327-cloudflare-deploy-and-smoke/phase-04-smoke-and-docs.md delete mode 100644 plans/260426-2327-cloudflare-deploy-and-smoke/plan.md delete mode 100644 plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-01-kv-repositories.md delete mode 100644 plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-02-wrangler-and-kv-namespace.md delete mode 100644 plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-03-deploy-and-bundle-gate.md delete mode 100644 plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-04-register-webhook-and-smoke.md delete mode 100644 plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-05-docs-and-cleanup.md delete mode 100644 plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-06-atlas-data-migration.md delete mode 100644 plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md delete mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-01-vercel-scaffolding.md delete mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-02-upstash-repository-adapter.md delete mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-03-inline-scraper-modules.md delete mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-04-http-layer-webhook-and-cron.md delete mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-05-data-migration-atlas-to-upstash.md delete mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-06-deploy-cutover-and-webhook-reregister.md delete mode 100644 plans/260509-1656-consolidate-vercel-upstash/phase-07-cleanup-wrangler-and-docs.md delete mode 100644 plans/260509-1656-consolidate-vercel-upstash/plan.md delete mode 100644 plans/archive/260426-2015-cloudflare-worker-code-port/phase-01-wrangler-config-deps-lint.md delete mode 100644 plans/archive/260426-2015-cloudflare-worker-code-port/phase-02-telegram-client.md delete mode 100644 plans/archive/260426-2015-cloudflare-worker-code-port/phase-03-mongo-worker-adapter.md delete mode 100644 plans/archive/260426-2015-cloudflare-worker-code-port/phase-04-worker-entry.md delete mode 100644 plans/archive/260426-2015-cloudflare-worker-code-port/phase-05-register-script-write-only.md delete mode 100644 plans/archive/260426-2015-cloudflare-worker-code-port/plan.md delete mode 100644 plans/reports/brainstorm-260509-1656-consolidate-vercel-upstash.md delete mode 100644 plans/reports/code-review-260509-1801-vercel-upstash-consolidation.md delete mode 100644 plans/reports/researcher-260429-1555-java-vs-js-parity.md delete mode 100644 plans/reports/researcher-260509-1656-upstash-vs-atlas.md delete mode 100644 plans/reports/test-260509-1801-vercel-upstash-consolidation.md delete mode 100644 plans/reports/xia-260429-1601-store-scraper-bot-port-status.md diff --git a/plans/260426-2327-cloudflare-deploy-and-smoke/phase-01-atlas-and-secrets.md b/plans/260426-2327-cloudflare-deploy-and-smoke/phase-01-atlas-and-secrets.md deleted file mode 100644 index 149a373..0000000 --- a/plans/260426-2327-cloudflare-deploy-and-smoke/phase-01-atlas-and-secrets.md +++ /dev/null @@ -1,83 +0,0 @@ -# Deploy Phase 01 — Atlas Provisioning + CF Secrets - -## Context Links -- miti99bot Phase 01 (validated): `/config/workspace/tiennm99/miti99bot/plans/260425-1945-mongodb-atlas-migration/phase-01-atlas-setup.md` - -## Overview -- **Priority:** P0 -- **Status:** pending -- **Description:** One-time provisioning of Atlas M0 cluster + Cloudflare secrets. No code changes. - -## Implementation Steps - -1. **Atlas account + cluster** (~15 min): - - Create Atlas account at https://cloud.mongodb.com (no credit card). - - Create project `js-store-scraper-bot`. - - Provision M0 in `aws-ap-southeast-1`. Cluster name: `js-store-scraper-bot-prod`. - - Wait for cluster green status. - -2. **DB user**: - - Atlas → Database Access → Add user. - - Username: `js-store-scraper-worker`. - - Password: ≥32 random chars (use `openssl rand -base64 32`). **Save in password manager.** - - Role: `readWrite@store-scraper-bot`. - -3. **Network access**: - - Atlas → Network Access → Add IP → `0.0.0.0/0`. - - Comment: "Cloudflare Workers — no static egress IPs on free plan". - -4. **Atlas alerts** (free, ~5 min): - - Project → Alerts → Add Alert. - - Cluster unavailable → email. - - Connections > 400 → email. - -5. **Connection string**: - - Atlas → Database → Connect → Drivers (Node 6.7+). - - Copy SRV string: `mongodb+srv://js-store-scraper-worker:@/store-scraper-bot?retryWrites=true&w=majority`. - - Substitute the real password. - -6. **Cloudflare account + login**: - - Sign up at cloudflare.com if not already (free). - - In project dir: `npx wrangler login`. Browser flow to authorize. - -7. **Set 5 secrets**: - ```sh - cd /config/workspace/tiennm99/js-store-scraper-bot - npx wrangler secret put TELEGRAM_BOT_TOKEN # paste your bot token - npx wrangler secret put TELEGRAM_BOT_USERNAME # e.g. miti99_store_bot - npx wrangler secret put TELEGRAM_WEBHOOK_SECRET # generate: openssl rand -hex 32 - npx wrangler secret put MONGODB_URI # paste full SRV string - npx wrangler secret put ADMIN_IDS # e.g. 123456789,987654321 - ``` - -8. **Mirror to `.env.deploy`** (for the register script — gitignored): - ```sh - cp .env.deploy.example .env.deploy - # edit: paste TELEGRAM_BOT_TOKEN, TELEGRAM_WEBHOOK_SECRET, WORKER_URL - ``` - `WORKER_URL` won't be known until first deploy in Phase 02; leave placeholder for now. - -## Todo List -- [ ] Atlas project + M0 cluster green -- [ ] DB user + password vaulted -- [ ] Network `0.0.0.0/0` added -- [ ] Atlas alerts configured -- [ ] Connection string captured -- [ ] `wrangler login` complete -- [ ] All 5 secrets set -- [ ] `.env.deploy` populated (WORKER_URL TBD) - -## Success Criteria -- `npx wrangler secret list` shows all 5 secret names. -- Atlas dashboard shows cluster green + 0 connections. - -## Risk Assessment -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| Atlas signup takes longer than expected | M | Low | Allocate 30 min | -| Password copied wrong | L | High | Vault first; paste from vault | -| `wrangler secret put` fails | L | Medium | Re-run; check `wrangler whoami` | - -## Next Steps -- **Blocks:** Phase 02. -- **Unblocks:** Phase 02. diff --git a/plans/260426-2327-cloudflare-deploy-and-smoke/phase-02-deploy-and-gates.md b/plans/260426-2327-cloudflare-deploy-and-smoke/phase-02-deploy-and-gates.md deleted file mode 100644 index 80a2450..0000000 --- a/plans/260426-2327-cloudflare-deploy-and-smoke/phase-02-deploy-and-gates.md +++ /dev/null @@ -1,89 +0,0 @@ -# Deploy Phase 02 — First Deploy + Hard Gates - -## Overview -- **Priority:** P0 — gates can abort the entire plan. -- **Status:** pending -- **Description:** Bundle-size dry-run, real deploy, mongo-ping CPU gate, auto-pause behavior test. - -## Implementation Steps - -1. **Bundle-size hard gate**: - ```sh - npx wrangler deploy --dry-run --outdir=.tmp-deploy - du -sh .tmp-deploy - ``` - - **PASS**: ≤ 2.7 MiB. Continue. - - **FAIL**: > 2.7 MiB. **Stop**. Pivot to Upstash (separate plan; out of scope here). - -2. **First real deploy**: - ```sh - npx wrangler deploy - ``` - Capture the URL printed (e.g. `https://js-store-scraper-bot..workers.dev`). - Update `.env.deploy` with this `WORKER_URL`. - -3. **Add temporary `/__mongo-ping` route** for CPU gate (revert before commit at end of phase): - - Edit `src/index.js` `fetch` handler — top of method, before path routing: - ```js - if (request.method === 'GET' && new URL(request.url).pathname === '/__mongo-ping') { - const t = Date.now(); - const { db } = await getMongo(env); - await db.command({ ping: 1 }); - return new Response(JSON.stringify({ wall_ms: Date.now() - t }), { - headers: { 'Content-Type': 'application/json' }, - }); - } - ``` - - `npx wrangler deploy`. - -4. **Cold-start CPU gate** (operator): - - Run 5 cold cycles, 10+ min apart: - ```sh - curl https://js-store-scraper-bot..workers.dev/__mongo-ping - ``` - - Open CF dashboard → Workers → js-store-scraper-bot → Logs / Metrics. - - Note CPU time per invocation (separate from wall_ms). - - **PASS**: cold CPU < 40ms (10ms safety under 50ms cap). - - **FAIL**: ≥ 40ms. Decide: upgrade to Workers Paid ($5/mo, 50ms baseline → 5min ceiling), or pivot to Upstash. - -5. **Auto-pause behavior test**: - - Atlas UI → cluster → Pause. - - Wait ~30s for Atlas to actually pause. - - `curl .../__mongo-ping` → should return 5xx within ~5s, NOT hang. - - Atlas UI → resume cluster. - - Verify: error was catchable (logged via `wrangler tail`); not a hang. - -6. **Cleanup**: revert `/__mongo-ping` route in `src/index.js`. Re-deploy: - ```sh - git checkout src/index.js # revert the temp ping route - npx wrangler deploy - ``` - -7. **Record baseline** in `docs/using-mongodb.md` (created in Phase 04): cold-ping P95 wall_ms. - -## Todo List -- [ ] Bundle ≤ 2.7 MiB -- [ ] First deploy success; URL captured -- [ ] `.env.deploy` `WORKER_URL` set -- [ ] Temporary `/__mongo-ping` deployed -- [ ] Cold CPU < 40ms across 5 cold cycles -- [ ] Auto-pause yields catchable error -- [ ] Temp route reverted + redeployed -- [ ] Baseline P95 captured for docs - -## Success Criteria -- `wrangler deploy` clean. -- All hard gates pass. -- Atlas dashboard "Current Connections" stays low (≤ 5) during ping smoke. - -## Risk Assessment -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| Bundle exceeds 2.7 MiB | M | Plan-aborting | Pivot path documented in plan.md | -| Cold CPU > 50ms | M | Plan-aborting on Free | Escalate to Paid; or pivot | -| Atlas connection refused (`0.0.0.0/0` not yet propagated) | L | Medium | Wait 1-2 min; retry | -| Forgot to revert `/__mongo-ping` | L | Low | Step 6 explicit; lint adds no help here | - -## Next Steps -- **Blocks:** Phase 03. -- **Unblocks:** Phase 03. diff --git a/plans/260426-2327-cloudflare-deploy-and-smoke/phase-03-register-webhook.md b/plans/260426-2327-cloudflare-deploy-and-smoke/phase-03-register-webhook.md deleted file mode 100644 index de15ee8..0000000 --- a/plans/260426-2327-cloudflare-deploy-and-smoke/phase-03-register-webhook.md +++ /dev/null @@ -1,43 +0,0 @@ -# Deploy Phase 03 — Register Webhook - -## Overview -- **Priority:** P1 -- **Status:** pending -- **Description:** Run the script written in code-plan Phase 05 against the live URL. - -## Implementation Steps - -1. Verify `.env.deploy` has `WORKER_URL` set (from Deploy Phase 02 step 2). -2. Dry-run first: - ```sh - npm run register:dry - ``` - Verify printed payloads look right (URL, secret, command list). -3. Real run: - ```sh - npm run register - ``` -4. Verify `getWebhookInfo` output in script's stdout shows: - - `url`: matches `WORKER_URL` - - `pending_update_count`: 0 - - `last_error_date`: absent -5. In Telegram client, type `/` in a chat with the bot — autocompletion should show the 13 commands (within ~1 min of `setMyCommands`). - -## Todo List -- [ ] `npm run register:dry` clean -- [ ] `npm run register` clean -- [ ] `getWebhookInfo` shows correct URL + 0 pending -- [ ] Telegram `/` autocompletion shows 13 commands - -## Success Criteria -- All checks above. - -## Risk Assessment -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| Wrong WORKER_URL | M | Medium | Dry-run catches this | -| Telegram returns `wrong response from the webhook` | L | High | Means Worker is reachable but returns non-200; check `wrangler tail` | - -## Next Steps -- **Blocks:** Phase 04. -- **Unblocks:** Phase 04. diff --git a/plans/260426-2327-cloudflare-deploy-and-smoke/phase-04-smoke-and-docs.md b/plans/260426-2327-cloudflare-deploy-and-smoke/phase-04-smoke-and-docs.md deleted file mode 100644 index 43af649..0000000 --- a/plans/260426-2327-cloudflare-deploy-and-smoke/phase-04-smoke-and-docs.md +++ /dev/null @@ -1,96 +0,0 @@ -# Deploy Phase 04 — End-to-End Smoke + Docs - -## Overview -- **Priority:** P1 -- **Status:** pending -- **Description:** Real Telegram smoke of all 13 commands, scheduled handler smoke, README + docs update. - -## Implementation Steps - -1. **Open `wrangler tail`** in a terminal to stream logs. - -2. **Smoke checklist** in a Telegram chat with the bot (operator must be in `ADMIN_IDS`): - - [ ] `/info` → bot replies with chat ID - - [ ] `/addgroup` → "Group added successfully" - - [ ] `/addgoogle com.android.chrome vn` → success - - [ ] `/addapple 284910350 vn` → success (YouTube iOS) - - [ ] `/listapp` → table with both apps - - [ ] `/checkapp` → 4-col table with green checks - - [ ] `/checkappscore` → score table - - [ ] `/rawappleapp 284910350 vn` → JSON file attachment - - [ ] `/rawgoogleapp com.android.chrome vn` → JSON file attachment - - [ ] `/listgroup` → group listed - - [ ] `/delapple 284910350` → success - - [ ] `/delgoogle com.android.chrome` → success - - [ ] `/delgroup` → success - -3. **Scheduled smoke**: trigger the cron handler via curl (works on `wrangler dev --test-scheduled`): - ```sh - npx wrangler dev --test-scheduled - # in another terminal: - curl "http://localhost:8787/__scheduled?cron=0+0+*+*+*" - ``` - Verify `runDailyCheck` log lines appear. - -4. **Atlas check**: dashboard "Current Connections" ≤ 5 throughout smoke. - -5. **Write `docs/cloudflare-deployment.md`** — operator runbook: - - Prerequisites - - One-time setup commands (Atlas + secrets + first deploy) - - Routine deploy: `npm run deploy` - - Reading `wrangler tail` - - Pause/restart procedure - - Rollback to Node polling - - Fallback to Upstash (sketch only — link to plan if it ever runs) - -6. **Write `docs/using-mongodb.md`** — Atlas runbook: - - Cluster URL (without password) - - Region, alerts, auto-pause schedule - - Password rotation procedure - - Baseline cold-ping P95 (from Phase 02 step 7) - - `0.0.0.0/0` permanence + paid CF static-IP path - -7. **Update `README.md`**: - - Replace existing "Run" section: "Deploys to Cloudflare Workers; see `docs/cloudflare-deployment.md`." - - Keep preview/untested banner. - - List new deps: `mongodb`, `wrangler`. - - Note: Docker compose retained for local-with-Mongo dev only. - -8. **Mark Docker files deprecated** — header comment: - ``` - # DEPRECATED: this project deploys to Cloudflare Workers (see docs/cloudflare-deployment.md). - # Retained only for local-with-Mongo development. - ``` - -9. **Commit + push**: - ```sh - git add docs/ README.md Dockerfile docker-compose*.yml - git commit -m "docs: Cloudflare Workers deployment runbook + smoke complete" - git push - ``` - -## Todo List -- [ ] All 13 commands smoke-tested -- [ ] Scheduled handler smoke OK -- [ ] Atlas connection count low -- [ ] `docs/cloudflare-deployment.md` written -- [ ] `docs/using-mongodb.md` written -- [ ] `README.md` updated -- [ ] Docker files marked deprecated -- [ ] Committed + pushed - -## Success Criteria -- All commands respond correctly in real Telegram. -- Operator runbook is complete and someone else could redeploy from scratch following it. -- Daily 7am-VN cron will fire (verify next morning in Atlas op-log + `wrangler tail` history). - -## Risk Assessment -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| Smoke reveals refactor bug | M | Medium | Fix + redeploy; not catastrophic | -| Cron timezone confusion | L | Low | Documented `0 UTC = 7 VN`; verify next day | - -## Next Steps -- **Blocks:** none. -- **Unblocks:** end of plan. -- **Follow-up (out of scope):** tests; observability dashboard; quarterly password rotation reminder. diff --git a/plans/260426-2327-cloudflare-deploy-and-smoke/plan.md b/plans/260426-2327-cloudflare-deploy-and-smoke/plan.md deleted file mode 100644 index 62d2f1e..0000000 --- a/plans/260426-2327-cloudflare-deploy-and-smoke/plan.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -title: "Cloudflare Workers deploy + smoke + docs" -description: "Operator-driven phases: provision Atlas, set CF secrets, run hard gates, deploy, register webhook, smoke-test all 13 commands, write docs. Sister to the code-port plan." -status: pending -priority: P1 -effort: 4h -branch: main -tags: [cloudflare-workers, deploy, smoke, atlas] -created: 2026-04-26 -blockedBy: [260426-2015-cloudflare-worker-code-port, 260505-1425-cloudflare-kv-migration-and-deploy] -blocks: [] -supersededBy: 260505-1425-cloudflare-kv-migration-and-deploy ---- - -> **Superseded** — User chose Cloudflare KV over Atlas pre-emptively, before the hard-gate trip. Deploy work is now tracked in [`260505-1425-cloudflare-kv-migration-and-deploy`](../260505-1425-cloudflare-kv-migration-and-deploy/plan.md). This file is kept for history. - -# Plan B — Deploy + Smoke - -Run only after [the code-port plan](../260426-2015-cloudflare-worker-code-port/plan.md) is complete and committed. This plan touches **real accounts**: Atlas (free M0) and Cloudflare (free Workers). - -## Reference Implementation - -[miti99bot Phase 01 atlas-setup](/config/workspace/tiennm99/miti99bot/plans/260425-1945-mongodb-atlas-migration/phase-01-atlas-setup.md) — proven Atlas + nodejs_compat_v2 procedure. Lift the operator steps wholesale. - -## Hard Gates (can abort the plan → pivot to Upstash Redis) - -| Gate | Phase | Threshold | On fail | -|---|---|---|---| -| Bundle size | 02 | `wrangler deploy --dry-run` ≤ 2.7 MiB (10% headroom under 3 MiB Free cap) | Pivot to Upstash (out-of-scope here; plan + execute separately) | -| Cold-start CPU | 02 | `/__mongo-ping` cold cycles < 40ms CPU | Escalate to Workers Paid ($5/mo) OR pivot to Upstash | -| Auto-pause behavior | 02 | Paused M0 yields catchable error within 5s | Document catch path requirement; not abort-worthy | - -## Phases - -| # | Phase | Status | Effort | -|---|---|---|---| -| 01 | [Atlas M0 + CF secrets](phase-01-atlas-and-secrets.md) | pending | 1h | -| 02 | [First deploy + bundle/CPU gates](phase-02-deploy-and-gates.md) | pending | 1h | -| 03 | [Register webhook](phase-03-register-webhook.md) | pending | 0.5h | -| 04 | [End-to-end smoke + docs](phase-04-smoke-and-docs.md) | pending | 1.5h | - -## Critical dependencies -- 01 → 02 (deploy needs MONGODB_URI + secrets) -- 02 → 03 (register needs live worker URL + passing gates) -- 03 → 04 (smoke needs registered webhook) - -## Prerequisites (operator must have) -- MongoDB Atlas account (free, no credit card). -- Cloudflare account (free, no credit card). -- `wrangler login` completed locally. -- Telegram bot token (existing — same as Node deployment). - -## Out of scope (explicit) -- Application code changes — handled by code plan; this plan deploys what's already on `main`. -- Custom domain — `*.workers.dev` URL is fine for Telegram webhooks. -- Tests — original project has none; not adding here. -- Upstash pivot — only triggered if hard gates trip; planned then, not now. - -## Rollback path -- After Phase 02 (deploy live but not registered): `wrangler delete` removes Worker; no public surface. -- After Phase 03 (webhook registered): `setWebhook` with empty URL clears it; or rotate `TELEGRAM_WEBHOOK_SECRET` so future POSTs 401. -- After Phase 04 (production live): `git revert` to pre-Workers commit, redeploy via Docker compose for the existing Node polling path. - -## Questions to confirm before Phase 01 -1. Atlas account: do you have one, or first-time setup? -2. CF account: free Workers plan attached? -3. Bot username for `setMyCommands`: same as in `.env.example`? -4. Greenfield Mongo data, or existing data to import? (If existing, schedule a separate import phase.) diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-01-kv-repositories.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-01-kv-repositories.md deleted file mode 100644 index a34e7d6..0000000 --- a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-01-kv-repositories.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -phase: 1 -title: "Replace Mongo with KV in repositories" -status: completed -priority: P1 -effort: "1.5h" -dependencies: [] ---- - -# Phase 01: Replace Mongo with KV in repositories - -## Overview - -Swap the four `*-repository.js` modules from `mongodb` collection ops to `env.STORE_KV.get/put/delete`. Drop the `mongodb` driver dependency. Keep document shape and Java parity intact. - -## Requirements - -**Functional** -- All 13 Telegram commands continue to work without behavior change. -- Daily cron continues to enumerate groups and report stale apps. -- Cached app entries expire after `APP_CACHE_SECONDS`. - -**Non-functional** -- Worker bundle ≤ 500 KiB. -- No `nodejs_compat_v2` flag required. -- `npm install` no longer pulls `mongodb`. - -## Architecture - -KV access pattern (single namespace `STORE_KV`, prefixed keys): - -``` -admin → JSON of {class:"Admin", groups:[...]} -group:{chatId} → JSON of {class:"Group", _id, appleApps[], googleApps[]} -apple:{appId} → JSON of cached Apple response (KV TTL = APP_CACHE_SECONDS) -google:{appId} → JSON of cached Google response (KV TTL = APP_CACHE_SECONDS) -``` - -Read pattern: `await env.STORE_KV.get(key, 'json')` returns parsed JSON or `null`. -Write pattern: `await env.STORE_KV.put(key, JSON.stringify(value), opts?)`. -Delete pattern: `await env.STORE_KV.delete(key)`. - -Cache TTL: pass `{ expirationTtl: appCacheSeconds }` to `put` for `apple:` and `google:` keys. **KV minimum `expirationTtl` is 60s** — Java/Mongo had no such floor. Guard in `kv.js`: clamp `expirationTtl` to `Math.max(60, value)` so an aggressive `APP_CACHE_SECONDS` override doesn't make `put` reject. Document the floor in `.dev.vars.example`. - -Drop the `isAppleAppExpired` / `isGoogleAppExpired` helpers from the read path — KV deletes expired keys, so a `get` returning `null` is the cache miss. Keep the helpers if other code references them; otherwise delete. - -## Related Code Files - -**Replace (rewrite contents)** -- `src/repository/admin-repository.js` — KV ops on key `admin` -- `src/repository/group-repository.js` — KV ops on key `group:{chatId}` -- `src/repository/apple-app-repository.js` — KV ops on key `apple:{appId}` with TTL -- `src/repository/google-app-repository.js` — KV ops on key `google:{appId}` with TTL - -**Delete** -- `src/repository/mongodb.js` — no longer needed - -**Modify** -- `src/repository/store.js` — remove memoization comment about `MongoClient`; factory signature unchanged -- `src/index.js` — drop the "memoized MongoClient" comment block; nothing else changes (the `env` already flows through) -- `src/config.js` — remove `MONGODB_URI` from required env list -- `package.json` — `npm uninstall mongodb`; bump version to 0.3.0 - -**Untouched (verify)** -- `src/models/*.js` — document shape preserved verbatim (Java parity at the doc level: `_id`, `class`, all fields) -- `src/bot/commands/*.js` — repository interfaces unchanged -- `src/scheduler/scheduler.js` — calls `store.admin.getAllGroups()` and `store.group.getGroup()`; both still resolve - -## Java parity preserved (per existing parity reports) - -Cross-checked against `plans/reports/xia-260429-1601-store-scraper-bot-port-status.md` and `researcher-260429-1555-java-vs-js-parity.md`. KV swap preserves every PARITY item: - -- Schema (`_id`, `class`, field names) — written verbatim as JSON values. -- Cache TTL semantics — KV `expirationTtl` is behaviorally equivalent to Java's `(now - millis) > cacheMillis` check on read. Both produce a "cache miss → re-fetch" outcome at the same boundary. -- `getAdmin()` returning `newAdmin()` if doc missing — `getJson('admin') ?? newAdmin()` matches. -- Existence checks — `(await get) !== null` replaces `countDocuments`. -- Concurrent `addGroup` race — Java/Mongo had the same non-atomic R-M-W; not a regression. - -**Two intentional divergences from Java:** - -1. **Eventual consistency (~60s globally)**: Java/Mongo gave strong read-after-write. For a single-admin daily-cron bot this is acceptable; documented in `plan.md`. -2. **TTL floor of 60s**: Java had no minimum. Mitigated by the `kv.js` clamp above. - -## Implementation Steps - -1. **Write `kv.js` helper** at `src/repository/kv.js` — thin wrapper exposing `getJson(env, key)`, `putJson(env, key, value, opts?)`, `del(env, key)`. Single-binding-name constant `STORE_KV`. Throws a typed error if `env.STORE_KV` is missing. - -2. **Rewrite `admin-repository.js`**: - - `init()` → `getJson('admin')`; if null, `putJson('admin', newAdmin())`. - - `getAdmin()` → `getJson('admin') ?? newAdmin()`. - - `save(admin)` → `putJson('admin', admin)`. - - `addGroup`/`removeGroup`/`hasGroup`/`getAllGroups` — unchanged (delegate to `getAdmin` + `save`). - -3. **Rewrite `group-repository.js`**: - - `key(groupId)` → `` `group:${groupIdToKey(groupId)}` ``. - - `exists(groupId)` → `getJson(key(groupId)) !== null`. - - `getGroup(groupId)` → `getJson(key(groupId)) ?? newGroup(groupId)`. - - `saveGroup(group)` → `putJson(\`group:${group._id}\`, group)`. - - `deleteGroup(groupId)` → `del(key(groupId))`. - - `mutateAndSave` unchanged. - -4. **Rewrite `apple-app-repository.js`**: - - `key(appId)` → `` `apple:${appId}` ``. - - `get(appId)` → `getJson(key(appId))`. - - `save(entry)` → `putJson(key(entry._id), entry, { expirationTtl: appCacheSeconds })`. - - `getCached(appId)` → just `get(appId)` — TTL already enforced by KV. Remove the `isAppleAppExpired` call. - -5. **Rewrite `google-app-repository.js`** — same pattern as apple. - -6. **Delete `src/repository/mongodb.js`**. Remove its imports from the four repositories. - -7. **Update `src/config.js`** — drop `'MONGODB_URI'` from the `required` list. - -8. **Update `src/index.js`** — strip the `MongoClient`-related comment in `build()`. Add a sanity check: throw if `env.STORE_KV` is undefined (early failure beats null-dereference at first request). - -9. **Update `package.json`**: - - `npm uninstall mongodb` - - bump `version` to `0.3.0` - -10. **Compile/lint check**: `node --check` each modified file. Run `npm run lint` (the existing secret-leak check). - -11. **Local sanity (no live KV needed)**: `node -e "import('./src/repository/store.js').then(...)"` — confirm no import errors. Cannot fully test without `wrangler dev` (Phase 03). - -## Todo List - -- [ ] Create `src/repository/kv.js` helper -- [ ] Rewrite `admin-repository.js` -- [ ] Rewrite `group-repository.js` -- [ ] Rewrite `apple-app-repository.js` (with KV TTL) -- [ ] Rewrite `google-app-repository.js` (with KV TTL) -- [ ] Delete `src/repository/mongodb.js` -- [ ] Update `src/config.js` required env list -- [ ] Update `src/index.js` env validation -- [ ] `npm uninstall mongodb`; bump `package.json` version -- [ ] `node --check` all modified files -- [ ] `npm run lint` -- [ ] **Java parity check**: spot-diff a written `group:{id}` JSON value against an equivalent Java/Mongo doc structure (sample from upstream `tiennm99/store-scraper-bot` model classes). Field names, ordering-insensitive, must match. - -## Success Criteria - -- [ ] `grep -rn "mongodb\|MongoClient" src/` returns nothing. -- [ ] `package.json` `dependencies` is empty (only `wrangler` in devDeps). -- [ ] All 4 repository factories return the same interface they did before (consumers untouched). -- [ ] `node --check src/index.js` exits 0. -- [ ] `npm run lint` passes. - -## Risk Assessment - -| Risk | Likelihood | Mitigation | -|---|---|---| -| Repository interface drift breaks command handlers | Low | Keep the exported function names + signatures byte-identical | -| KV's eventual consistency surprises a user mid-conversation | Low | Single-admin bot, low concurrency; document in README | -| Concurrent `/addgroup` race clobbers `groups[]` | Low | Same race existed in Mongo; not a regression | -| TTL `expirationTtl` < 60 silently fails | Low | `APP_CACHE_SECONDS` defaults to 600; doc the floor in `.dev.vars.example` | - -## Security Considerations - -- KV namespace ID is not a secret (binding name is what code references). Safe to commit in `wrangler.toml`. -- Removing `MONGODB_URI` removes one secret from the rotation surface. Update `wrangler secret list` documentation in Phase 05. diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-02-wrangler-and-kv-namespace.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-02-wrangler-and-kv-namespace.md deleted file mode 100644 index 48105d3..0000000 --- a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-02-wrangler-and-kv-namespace.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -phase: 2 -title: "Wrangler config + KV namespace provisioning" -status: pending -priority: P1 -effort: "0.5h" -dependencies: [1] ---- - -# Phase 02: Wrangler config + KV namespace provisioning - -## Overview - -Provision a Cloudflare KV namespace, bind it to the Worker as `STORE_KV`, and remove now-unused MongoDB-related config from `wrangler.toml`. - -## Requirements - -**Functional** -- Worker has access to `env.STORE_KV` at runtime. -- Local `wrangler dev` reads from a preview namespace, not production. - -**Non-functional** -- `wrangler.toml` is the single source of truth (no `wrangler.jsonc`). -- Production and preview namespace IDs are committed (these are not secrets). - -## Architecture - -KV bindings are declared in `wrangler.toml`: - -```toml -[[kv_namespaces]] -binding = "STORE_KV" -id = "" -preview_id = "" -``` - -`binding` is the name surfaced as `env.STORE_KV` in the Worker. `id` is the production namespace UUID. `preview_id` is used by `wrangler dev`. - -Operator-driven steps (run locally; require `wrangler login`): - -```sh -npx wrangler kv namespace create STORE_KV -npx wrangler kv namespace create STORE_KV --preview -``` - -Each command outputs a TOML snippet to copy into `wrangler.toml`. - -## Related Code Files - -**Modify** -- `wrangler.toml` — add `[[kv_namespaces]]` block; remove `nodejs_compat_v2` flag; trim the secrets-comment block to drop `MONGODB_URI`. - -**No code changes** — Phase 01 already wrote everything that consumes `env.STORE_KV`. - -## Implementation Steps - -1. **Run** `npx wrangler kv namespace create STORE_KV` — capture the production `id`. -2. **Run** `npx wrangler kv namespace create STORE_KV --preview` — capture the `preview_id`. -3. **Edit `wrangler.toml`**: - - Add the `[[kv_namespaces]]` block with both IDs. - - Change `compatibility_flags` from `["nodejs_compat_v2"]` to `[]` (or delete the line entirely). - - In the `# Secrets` comment block, remove the `MONGODB_URI` line. -4. **Verify** with `npx wrangler types` (if available) or `npx wrangler deploy --dry-run` — should report the new binding without errors. -5. **Local seed (optional)**: For `wrangler dev` to be useful, the admin singleton must exist. The repository's `init()` is currently called nowhere automatically. Either: - - Seed manually: `npx wrangler kv key put --binding STORE_KV --preview admin '{"_id":"admin","class":"Admin","groups":[]}'` - - Or update `src/index.js` to call `store.admin.init()` on first request (cheap, idempotent). **Recommended**: add the lazy init. - -## Todo List - -- [ ] `wrangler kv namespace create STORE_KV` -- [ ] `wrangler kv namespace create STORE_KV --preview` -- [ ] Add `[[kv_namespaces]]` block to `wrangler.toml` -- [ ] Drop `nodejs_compat_v2` from `compatibility_flags` -- [ ] Remove `MONGODB_URI` mention from the secrets comment -- [ ] Add lazy `store.admin.init()` on first request in `src/index.js` -- [ ] `wrangler deploy --dry-run` succeeds - -## Success Criteria - -- [ ] `wrangler.toml` has `[[kv_namespaces]]` with `binding = "STORE_KV"` and both IDs. -- [ ] `compatibility_flags` no longer contains `nodejs_compat_v2`. -- [ ] `wrangler deploy --dry-run` exits 0 and lists the binding. -- [ ] `wrangler kv namespace list` shows two namespaces: `STORE_KV` and `STORE_KV_preview` (or named per CF convention). - -## Risk Assessment - -| Risk | Likelihood | Mitigation | -|---|---|---| -| Wrong `binding` name → `env.STORE_KV` is undefined at runtime | Low | Phase 01 added the early validation throw | -| Forgot `preview_id`, `wrangler dev` errors | Low | Step 2 explicitly creates it | -| Operator commits secret accidentally | Low | KV IDs are not secrets; existing `npm run lint` scans for the secret patterns we care about | - -## Security Considerations - -- KV namespace IDs are not secrets and are safe to commit. -- Production `STORE_KV` will hold Telegram chat IDs and admin IDs — same sensitivity as Mongo previously held. Cloudflare KV is encrypted at rest. diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-03-deploy-and-bundle-gate.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-03-deploy-and-bundle-gate.md deleted file mode 100644 index 0e0db84..0000000 --- a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-03-deploy-and-bundle-gate.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -phase: 3 -title: "Deploy + bundle-size sanity gate" -status: pending -priority: P1 -effort: "0.5h" -dependencies: [2] ---- - -# Phase 03: Deploy + bundle-size sanity gate - -## Overview - -Set Worker secrets, run `wrangler deploy`, confirm the bundle is well under the Free 3 MiB cap, and verify a cold-start request reaches the Worker. - -## Requirements - -**Functional** -- Worker is reachable at `https://js-store-scraper-bot..workers.dev`. -- A request without the secret header returns 401. - -**Non-functional** -- Bundle ≤ 500 KiB (informational; we expect ~30 KiB without `mongodb`). -- Cold start CPU < 10ms. - -## Architecture - -`wrangler deploy` reads secrets from the Cloudflare account (set via `wrangler secret put`), KV bindings from `wrangler.toml`, and uploads the bundled Worker. No Atlas, no TCP — just KV (HTTP edge-side), `fetch` to Telegram + iTunes + Google Play. - -Required secrets (set once via `wrangler secret put `): - -| Secret | Purpose | -|---|---| -| `TELEGRAM_BOT_TOKEN` | Posting to Telegram Bot API | -| `TELEGRAM_BOT_USERNAME` | Bot mention parsing | -| `TELEGRAM_WEBHOOK_SECRET` | Validates `X-Telegram-Bot-Api-Secret-Token` | -| `ADMIN_IDS` | Comma-separated Telegram user IDs | - -`MONGODB_URI` is **no longer required**. If a previous deploy set it, leave it (no-op) or run `wrangler secret delete MONGODB_URI` for hygiene. - -## Related Code Files - -**No code changes.** Phase deals with deployment only. - -## Implementation Steps - -1. **Set secrets** (interactive prompts; values supplied by operator): - ```sh - npx wrangler secret put TELEGRAM_BOT_TOKEN - npx wrangler secret put TELEGRAM_BOT_USERNAME - npx wrangler secret put TELEGRAM_WEBHOOK_SECRET - npx wrangler secret put ADMIN_IDS - ``` - `TELEGRAM_WEBHOOK_SECRET`: any random ≥32 chars, e.g. `openssl rand -hex 24`. - -2. **Optional cleanup**: `npx wrangler secret delete MONGODB_URI` if it was set previously. - -3. **Bundle gate**: - ```sh - npx wrangler deploy --dry-run --outdir=.bundle 2>&1 | tee .bundle/dry-run.log - du -h .bundle/*.js - ``` - Expected: total .js output < 500 KiB. If > 500 KiB, abort and investigate (`wrangler deploy --dry-run --metafile` then inspect imports). - -4. **Real deploy**: - ```sh - npx wrangler deploy - ``` - Capture the deployed URL. - -5. **Cold-start probe**: send a probe with a bad secret to confirm the 401 path works (cheap, doesn't touch KV): - ```sh - curl -i -X POST https://js-store-scraper-bot..workers.dev \ - -H "X-Telegram-Bot-Api-Secret-Token: wrong" \ - -d '{}' - ``` - Expected: `HTTP/1.1 401 Unauthorized`. - -6. **CPU check**: open the Cloudflare dashboard → Workers → js-store-scraper-bot → Logs/Metrics. After a few requests, confirm cold-start CPU is well under 50ms (target <10ms without Mongo). - -## Todo List - -- [ ] Set 4 required secrets via `wrangler secret put` -- [ ] (Optional) Delete obsolete `MONGODB_URI` secret -- [ ] Run `wrangler deploy --dry-run`; verify bundle < 500 KiB -- [ ] `wrangler deploy`; capture URL -- [ ] curl probe → expect 401 -- [ ] Confirm cold-start CPU < 10ms in CF dashboard - -## Success Criteria - -- [ ] Bundle size logged and < 500 KiB. -- [ ] `wrangler deploy` succeeds with `Uploaded` line. -- [ ] 401 probe returns 401 status. -- [ ] CPU per request < 10ms in dashboard logs. - -## Risk Assessment - -| Risk | Likelihood | Mitigation | -|---|---|---| -| Bundle balloons due to a transitive dep | Very Low | `mongodb` was the only heavy dep; with it gone, only stdlib + small `wrangler` runtime hooks remain | -| CF account quota exceeded | Very Low | New account = 100k requests/day budget; bot traffic is single-digit per minute | -| Secret typo (e.g. token) → 401 from Telegram on first webhook hit | Medium | Caught in Phase 04 smoke; rerun `wrangler secret put` to fix | - -## Security Considerations - -- All secrets are set via `wrangler secret put` (never in `wrangler.toml`). -- The 401 probe uses a deliberately wrong secret; no sensitive value is logged. -- Run `npm run lint` (existing secret-leak check) before deploy. diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-04-register-webhook-and-smoke.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-04-register-webhook-and-smoke.md deleted file mode 100644 index b92200b..0000000 --- a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-04-register-webhook-and-smoke.md +++ /dev/null @@ -1,135 +0,0 @@ ---- -phase: 4 -title: "Register webhook + smoke-test" -status: pending -priority: P1 -effort: "1.5h" -dependencies: [3] ---- - -# Phase 04: Register webhook + smoke-test - -## Overview - -Point the Telegram Bot API at the deployed Worker, then exercise every command and the daily cron path against the live deployment. - -## Requirements - -**Functional** -- Telegram delivers updates to `https://` with the secret header. -- All 13 commands return correct responses. -- KV reads/writes show up in `wrangler kv key list --binding STORE_KV`. - -**Non-functional** -- Each command response < 1s (Telegram's webhook timeout is 30s, but slow webhooks lead to retries). - -## Architecture - -Telegram webhook flow: - -``` -Telegram → POST https://js-store-scraper-bot..workers.dev - with header X-Telegram-Bot-Api-Secret-Token = - → Worker validates header (401 on mismatch) - → Worker parses update, ctx.waitUntil(dispatch(...)) - → Worker returns 200 immediately - → dispatch reads/writes STORE_KV, calls Telegram sendMessage -``` - -The cron schedule (`0 0 * * *` UTC = 7am `Asia/Ho_Chi_Minh`) triggers `scheduled()` daily; we manually trigger it once during smoke via `wrangler triggers cron`. - -## Related Code Files - -**No code changes.** Smoke uses the existing `scripts/register-webhook.js` and live commands. - -## Implementation Steps - -1. **Populate `.env.deploy`** from `.env.deploy.example`: - ``` - TELEGRAM_BOT_TOKEN=... - TELEGRAM_WEBHOOK_SECRET=... # must match the secret set in Phase 03 - WORKER_URL=https://js-store-scraper-bot..workers.dev - ``` - -2. **Dry-run register**: - ```sh - npm run register:dry - ``` - Confirms the request body is sane. - -3. **Real register**: - ```sh - npm run register - ``` - Calls `setWebhook` + `setMyCommands`. Expect `ok: true` from Telegram. - -4. **Verify webhook info**: - ```sh - curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getWebhookInfo" | jq - ``` - `url` should match, `pending_update_count` should be 0 once cleared, `has_custom_certificate: false`. - -5. **Smoke test (operator runs in Telegram)**: - - | Command | Expected | KV mutation | - |---|---|---| - | `/info` | bot info text (admin id, build time) | none | - | `/addgroup` (in target chat as admin) | "Group added" | `admin` ← `groups` includes chat id | - | `/listgroup` | shows new group | none | - | `/addapple ` | "Added" | `group:{chatId}` updated | - | `/addgoogle ` | "Added" | `group:{chatId}` updated | - | `/listapp` | renders both lists | none | - | `/checkapp` | table with ✅/❌ marks | populates `apple:{appId}` / `google:{appId}` | - | `/checkappscore` | scores table | uses cache from previous step | - | `/rawappleapp ` | raw JSON dump | reads `apple:{appId}` | - | `/rawgoogleapp ` | raw JSON dump | reads `google:{appId}` | - | `/delapple ` | "Removed" | `group:{chatId}` updated | - | `/delgoogle ` | "Removed" | `group:{chatId}` updated | - | `/delgroup` | "Group removed" | `admin.groups` shrinks | - -6. **Inspect KV** between steps: - ```sh - npx wrangler kv key list --binding STORE_KV --remote | jq - npx wrangler kv key get --binding STORE_KV --remote admin | jq - ``` - -7. **Trigger cron manually**: - ```sh - npx wrangler triggers cron "0 0 * * *" - ``` - (Or use the dashboard's "Trigger" button.) Re-add a test app with a known-stale `updated` date (or a real lagging app) and confirm the daily report is posted to the test group. - -8. **Cache TTL probe**: read `apple:` from KV — `expiration` field should be `now + APP_CACHE_SECONDS`. After waiting past TTL, the next `/rawappleapp` should re-fetch from iTunes. - -## Todo List - -- [ ] Populate `.env.deploy` -- [ ] `npm run register:dry` -- [ ] `npm run register` -- [ ] `getWebhookInfo` reports correct URL -- [ ] Walk through all 13 commands; record responses -- [ ] `wrangler kv key list` shows expected keys after mutations -- [ ] Manually trigger cron; verify daily report message -- [ ] Confirm cache TTL behavior - -## Success Criteria - -- [ ] All 13 commands return success responses (no "Internal server error"). -- [ ] KV inspection matches expected mutations after each command. -- [ ] Manual cron trigger posts a report (or "no stale apps" if none). -- [ ] Worker logs show no unhandled exceptions across the smoke run. - -## Risk Assessment - -| Risk | Likelihood | Mitigation | -|---|---|---| -| Webhook secret mismatch → 401 loop | Low | Step 4 verifies `getWebhookInfo`; rotate via `setWebhook` if needed | -| Telegram rate-limited during smoke | Low | Free tier limits are generous for ad-hoc testing; pause if 429 | -| KV eventual-consistency surprises smoke (write then immediate read on different POP) | Low | Same-region writes are visible immediately on modern KV; watch for the rare read-your-write miss and retry | -| `/checkapp` 30s timeout if upstream stores slow | Medium | Both scrapers have own timeouts; if hit, rerun once | - -## Security Considerations - -- `TELEGRAM_WEBHOOK_SECRET` must be ≥32 chars and unique to this Worker. -- After smoke, verify only authorized admin IDs can issue mutating commands by attempting from a non-admin account → expect "Unauthorized". -- Capture Worker logs from the dashboard and grep for accidental token leaks (none expected; existing structured logger redacts). diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-05-docs-and-cleanup.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-05-docs-and-cleanup.md deleted file mode 100644 index 5060b42..0000000 --- a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-05-docs-and-cleanup.md +++ /dev/null @@ -1,136 +0,0 @@ ---- -phase: 5 -title: "Docs + cleanup" -status: pending -priority: P2 -effort: "1h" -dependencies: [4] ---- - -# Phase 05: Docs + cleanup - -## Overview - -Update README + project docs to reflect KV-backed storage, archive the obsolete Atlas plan, and remove now-dead Mongo references from `.env*` examples and Docker files. - -## Requirements - -**Functional** -- A new operator can deploy from scratch by following only `README.md` + `docs/deployment-guide.md`. -- No file references MongoDB Atlas as the production storage. - -**Non-functional** -- Keep `docs/*.md` under 800 LOC each (per `docs.maxLoc`). - -## Architecture - -Documentation surface: - -| File | Action | -|---|---| -| `README.md` | Replace MongoDB section with KV; update env table; bump status note | -| `.env.example` | Remove `MONGODB_*`; add note that secrets live in `wrangler secret put` | -| `.dev.vars.example` | Replace `MONGODB_URI=` with KV preview-namespace seeding hint | -| `docs/deployment-guide.md` | Create or update with the Phase 02–04 operator runbook | -| `docs/system-architecture.md` | Diagram: Worker → KV (drop Atlas) | -| `plans/todo.md` | Remove the Atlas pre-flight items; update active-plan link | -| `plans/260426-2327-cloudflare-deploy-and-smoke/` | Move under `plans/archive/` with a "superseded by 260505-1425" note | - -Docker files (`Dockerfile`, `docker-compose.yml`, `docker-compose.dev.yml`) reference the old polling path. **Decision**: keep them as a fallback rollback option but add a banner comment that they target the legacy Mongo runtime, not Workers. - -## Related Code Files - -**Modify** -- `README.md` -- `.env.example` -- `.dev.vars.example` -- `plans/todo.md` -- `Dockerfile` (banner comment only) -- `docker-compose.yml` (banner comment only) -- `docker-compose.dev.yml` (banner comment only) - -**Create** -- `docs/deployment-guide.md` (if missing) -- `docs/system-architecture.md` (if missing) — short, ASCII or mermaid diagram - -**Move (archive)** -- `plans/260426-2327-cloudflare-deploy-and-smoke/` → `plans/archive/260426-2327-cloudflare-deploy-and-smoke/` - -## Implementation Steps - -1. **`README.md`**: - - Remove "Requirements: MongoDB 4.4+". - - Update env table: drop `MONGODB_CONNECTION_STRING`, `MONGO_DATABASE`; note that secrets go via `wrangler secret put`, KV binding is `STORE_KV`. - - Replace "Run via Docker Compose" with "Deploy: `npm run deploy`". - - Update "Differences vs Go / Java" to note storage backend differs (KV instead of Mongo) but document shape preserved. - -2. **`.env.example`**: - - Drop `MONGODB_*` rows; add a comment that production secrets live in `wrangler secret`. - - Keep `ADMIN_IDS`, `TELEGRAM_*` for local-dev reference only. - -3. **`.dev.vars.example`** — drop `MONGODB_URI`. - -4. **`docs/deployment-guide.md`** — short runbook condensing Phases 02–04 (KV namespace creation, secrets, deploy, register webhook, smoke). One file, < 200 lines. - -5. **`docs/system-architecture.md`** — mermaid diagram: - ``` - Telegram --webhook--> CF Worker --KV ops--> STORE_KV - --HTTPS--> iTunes / Play - cron --7am VN--> daily check - ``` - -6. **`plans/todo.md`**: - - Replace "Active plan" link with `260505-1425-cloudflare-kv-migration-and-deploy`. - - Remove the Atlas-specific hard-gate table (no longer applicable). - - Update pre-flight: drop Atlas account requirement; KV needs only CF account. - -7. **Archive old plan**: - ```sh - mv plans/260426-2327-cloudflare-deploy-and-smoke plans/archive/ - ``` - Append a single line at top of its `plan.md`: `> **Superseded by [`260505-1425-cloudflare-kv-migration-and-deploy`](../../260505-1425-cloudflare-kv-migration-and-deploy/plan.md)** — KV pivot taken pre-emptively.` - -8. **Docker file banners** — one-line comment at top: - ``` - # NOTE: Legacy Mongo polling path. Production deploys to Cloudflare Workers via `npm run deploy` and uses Cloudflare KV. - ``` - -9. **Final commit**: single squashed commit `feat: migrate to Cloudflare KV storage + deploy` with the full Phase 01–04 diff. Use conventional-commits. - -## Todo List - -- [ ] Update `README.md` -- [ ] Update `.env.example` -- [ ] Update `.dev.vars.example` -- [ ] Create/update `docs/deployment-guide.md` -- [ ] Create/update `docs/system-architecture.md` -- [ ] Update `plans/todo.md` -- [ ] Move old Atlas plan under `plans/archive/` -- [ ] Add legacy banners to Docker files -- [ ] Commit + push - -## Success Criteria - -- [ ] `grep -rn "MONGODB\|mongodb\|Atlas" README.md docs/ .env.example .dev.vars.example` returns nothing. -- [ ] `docs/deployment-guide.md` exists and is < 200 lines. -- [ ] Old plan lives under `plans/archive/` with the supersession banner. -- [ ] `git log --oneline -5` shows a clean conventional commit. - -## Risk Assessment - -| Risk | Likelihood | Mitigation | -|---|---|---| -| Stale Atlas reference left in some doc | Low | grep step in success criteria | -| Doc exceeds `docs.maxLoc=800` | Low | Each doc is short by design | -| Operator follows old plan by accident | Low | Archive move + supersession banner make it obvious | - -## Security Considerations - -- Confirm no secrets are pasted into README or examples. -- `npm run lint` (the existing `scripts/check-secret-leaks.js`) runs at commit time per existing workflow. - -## Next Steps (out of this plan, into backlog) - -- CI workflow that runs `wrangler deploy --dry-run` as a PR bundle-size gate. -- Observability dashboard pointed at the `STORE_KV` namespace (request count + key count). -- Tests — original project has none; would be additive work. diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-06-atlas-data-migration.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-06-atlas-data-migration.md deleted file mode 100644 index b8c3694..0000000 --- a/plans/260505-1425-cloudflare-kv-migration-and-deploy/phase-06-atlas-data-migration.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -phase: 6 -title: "One-shot Atlas → KV data migration" -status: pending -priority: P1 -effort: "1h" -dependencies: [2] ---- - -# Phase 06: One-shot Atlas → KV data migration - -## Overview - -Operator runs a local Node script that exports `common` and `group` collections from MongoDB Atlas and bulk-imports them into the production KV namespace. Cache collections are skipped by default (they auto-rebuild from upstream APIs). - -**Run order**: must execute **after Phase 02** (KV namespace provisioned) and **before Phase 04** (smoke tests need real data). Phase 03 deploy can happen before or after — the script writes via `wrangler kv bulk put`, not via the deployed Worker. - -## Requirements - -**Functional** -- Every `_id="admin"` doc in `common` lands at KV key `admin`. -- Every doc in `group` lands at KV key `group:{_id}`. -- With `--include-cache`: `apple_app` → `apple:{_id}` and `google_app` → `google:{_id}`, each with `expirationTtl` recomputed from the doc's `millis` field. Already-expired entries are skipped. -- Re-running the script overwrites — no duplication, no partial-state corruption. - -**Non-functional** -- Script is one file under `scripts/`. ≤100 LOC. -- `mongodb` driver lives in `devDependencies` only; not pulled into the Worker bundle. -- Sensitive intermediate file (`scripts/.atlas-export.json`) is deleted after success. - -## Architecture - -``` -.env (operator-local) Atlas (read-only) - | | - v v -[migrate-atlas-to-kv.js] ─── reads all docs from `common`+`group`(+caches?) - | - v - builds bulk JSON [{key, value, expiration_ttl?}, ...] - | - v - wrangler kv bulk put --binding STORE_KV --remote - | - v - Cloudflare KV -``` - -Key shapes mirror what the runtime code expects (see `src/repository/*-repository.js`): - -| Atlas collection | KV key | TTL | -|---|---|---| -| `common` | `admin` (singleton, _id="admin") | none | -| `group` | `group:{_id}` | none | -| `apple_app` | `apple:{_id}` | `max(60, (millis + APP_CACHE_SECONDS*1000 - now) / 1000)`, skip if ≤0 | -| `google_app` | `google:{_id}` | same | - -`wrangler kv bulk put` accepts up to 10,000 keys per call. For this bot, total keys are likely <100 (one admin + a handful of groups), so bulk fits in a single request. - -## Related Code Files - -**Create** -- `scripts/migrate-atlas-to-kv.js` — the migration script. - -**Modify** -- `package.json` — add `mongodb` to `devDependencies`; add `migrate` npm script. -- `README.md` — short "Migrating from Atlas (one-time)" section. - -**Untouched** -- All `src/` repositories — they read from KV; the script writes to KV via wrangler. No code change needed in the Worker. - -## Implementation Steps - -1. **Add `mongodb` as devDep**: `npm install --save-dev mongodb`. - -2. **Write `scripts/migrate-atlas-to-kv.js`**: - - `import { MongoClient } from 'mongodb'`. - - Read `MONGODB_URI` from `process.env` (loaded via `node --env-file=.env`). - - Connect with 5s timeout; pick db from URI path. - - `const adminDocs = await db.collection('common').find({ _id: 'admin' }).toArray()`. - - `const groupDocs = await db.collection('group').find({}).toArray()`. - - If `--include-cache` flag: read `apple_app` and `google_app` similarly; compute `expiration_ttl` per doc, skip expired. - - Build entries: `{ key, value: JSON.stringify(doc), expiration_ttl?: number }`. - - Write to `scripts/.atlas-export.json`. - - `console.log` summary: `{admin: 1, groups: N, apple: M (skipped K), google: ...}`. - - Print the next-step command: `npx wrangler kv bulk put --binding STORE_KV --remote scripts/.atlas-export.json`. - -3. **Add npm scripts to `package.json`**: - ```json - "migrate": "node --env-file=.env scripts/migrate-atlas-to-kv.js", - "migrate:bulk": "wrangler kv bulk put --binding STORE_KV --remote scripts/.atlas-export.json" - ``` - -4. **`.gitignore`** — add `scripts/.atlas-export.json` (already covers `.env` patterns; confirm `.atlas-export.json` is ignored). - -5. **README "Migrating from Atlas" section**: - ``` - 1. Ensure .env has MONGODB_URI pointing to your Atlas cluster. - 2. npm install (pulls mongodb devDep). - 3. npm run migrate # writes scripts/.atlas-export.json - 4. npm run migrate:bulk # uploads to KV - 5. rm scripts/.atlas-export.json # contains your data; delete after success - 6. Optional: npm run migrate -- --include-cache to also migrate cached app entries - ``` - -6. **Verify** post-import: - ```sh - npx wrangler kv key list --binding STORE_KV --remote | jq 'length' - npx wrangler kv key get --binding STORE_KV --remote admin | jq - ``` - Expect at least 1 (admin) + N (groups) keys. - -## Todo List - -- [ ] `npm install --save-dev mongodb` -- [ ] Write `scripts/migrate-atlas-to-kv.js` -- [ ] Add `migrate` + `migrate:bulk` scripts to `package.json` -- [ ] Add `scripts/.atlas-export.json` to `.gitignore` -- [ ] Add "Migrating from Atlas" section to README -- [ ] Operator runs end-to-end against staging KV namespace first; verify with `kv key list` -- [ ] Operator runs against production KV namespace -- [ ] Delete `scripts/.atlas-export.json` - -## Success Criteria - -- [ ] `wrangler kv key get --remote admin` returns the admin doc with `groups` array intact. -- [ ] Each group has its `apple:{appId}` / `google:{appId}` references resolvable through the bot's `/listapp` command (verified in Phase 04 smoke). -- [ ] Re-running `npm run migrate && npm run migrate:bulk` produces zero diff in KV (idempotent). -- [ ] `scripts/.atlas-export.json` is gitignored and deleted after success. - -## Risk Assessment - -| Risk | Likelihood | Mitigation | -|---|---|---| -| Atlas auth fails | Low | Pre-flight: verify URI works via `mongosh "$MONGODB_URI"` before running migrate | -| Bulk put exceeds 10k keys (unlikely for a handful of groups) | Very Low | Script can chunk into multiple files if `entries.length > 10000` (skip implementing — defer until needed) | -| Operator runs against wrong namespace | Medium | The `--remote` flag targets prod; review the `wrangler.toml` binding ID before running | -| Sensitive data leaks via committed bulk JSON | Low | `.gitignore` entry; README warns to delete | -| Cache TTL miscompute (e.g. negative ttl) | Low | Skip rule: `if (newTtl < 60) skip`. Documented in step 2 | - -## Security Considerations - -- `scripts/.atlas-export.json` contains all bot state (group memberships, admin IDs). Treat as sensitive. -- Run `npm run lint` before committing to catch accidental Atlas URI in any new file. -- Consider rotating Atlas password after migration (its only legitimate consumer is gone). diff --git a/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md b/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md deleted file mode 100644 index 365a222..0000000 --- a/plans/260505-1425-cloudflare-kv-migration-and-deploy/plan.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: "Cloudflare KV migration + deploy" -description: "Replace MongoDB Atlas with Cloudflare KV as the storage layer, deploy to Workers, register Telegram webhook, smoke-test, and document. Supersedes the Atlas-based deploy plan." -status: superseded -priority: P1 -effort: 5h -branch: main -tags: [cloudflare-workers, cloudflare-kv, deploy, migration] -created: 2026-05-05 -blockedBy: [] -blocks: [260426-2327-cloudflare-deploy-and-smoke] -supersededBy: 260509-1656-consolidate-vercel-upstash ---- - -> **Superseded** — User chose to consolidate on Vercel + Upstash instead of completing the CF Workers deploy. Code-level KV migration shipped (commits `067d463`..`c4c6a93`); deploy phases never executed. Continuation tracked in [`260509-1656-consolidate-vercel-upstash`](../260509-1656-consolidate-vercel-upstash/plan.md). - -# Cloudflare Workers + KV Storage Migration - -User chose KV over Atlas to pre-empt the bundle-size and cold-start risks called out in the Atlas plan. Drops the `mongodb` driver entirely (~4–5 MiB compressed) and the `nodejs_compat_v2` flag along with it. - -## Why KV fits this codebase - -Every existing repository operation is a primary-key lookup — no queries, no aggregations, no cursors: - -| Logical collection | Operations | KV key shape | -|---|---|---| -| `common` (admin singleton) | get/save the single admin doc with `groups[]` | `admin` | -| `group` (per-chat) | exists/get/save/delete by groupId | `group:{groupId}` | -| `apple_app` (cache) | get/save by appId, TTL expiry | `apple:{appId}` (with `expirationTtl`) | -| `google_app` (cache) | get/save by appId, TTL expiry | `google:{appId}` (with `expirationTtl`) | - -KV's native `expirationTtl` replaces the manual `isExpired(entry, now, cacheMs)` check. - -The daily cron reads `admin.groups[]` (single key), then iterates groups (one read each). No list-keys needed. - -## Phases - -| # | Phase | Status | Effort | -|---|---|---|---| -| 01 | [Replace Mongo with KV](phase-01-kv-repositories.md) | completed | 1.5h | -| 02 | [Wrangler + KV namespace](phase-02-wrangler-and-kv-namespace.md) | pending | 0.5h | -| 03 | [Deploy + bundle gate](phase-03-deploy-and-bundle-gate.md) | pending | 0.5h | -| 04 | [Register webhook + smoke](phase-04-register-webhook-and-smoke.md) | pending | 1.5h | -| 05 | [Docs + cleanup](phase-05-docs-and-cleanup.md) | pending | 1h | -| 06 | [Atlas → KV data migration](phase-06-atlas-data-migration.md) | pending | 1h | - -## Critical dependencies - -- 01 → 02 (wrangler binding name must match what code reads from `env.STORE_KV`) -- 02 → 03 (deploy needs the namespace ID provisioned) -- 02 → 06 (data migration writes via `wrangler kv bulk put`, needs namespace) -- 03 → 04 (webhook registration needs live Worker URL) -- 06 → 04 (smoke tests need migrated data — except clean greenfield runs) - -## Hard gates (now sanity checks, not pivots) - -| Gate | Threshold | Notes | -|---|---|---| -| Bundle size | ≤ 500 KiB (well under 3 MiB Free cap) | Without `mongodb`, bundle should drop to ~30 KiB | -| Cold start | < 10ms CPU | No TCP handshake; KV access is HTTP-fetched edge-side | - -If bundle exceeds 500 KiB, investigate — something pulled in unintended deps. - -## Key design decisions - -1. **One KV namespace, prefixed keys** — cleaner than four namespaces; KV billing is by op count not namespace. -2. **Drop `nodejs_compat_v2`** — only needed for the Mongo TCP socket. Removing reduces cold-start and runtime memory. -3. **Drop `mongodb` dependency** — removes ~4–5 MiB compressed. `npm uninstall mongodb`. -4. **Cache TTL via `expirationTtl`** — let KV handle expiry; remove `isAppleAppExpired`/`isGoogleAppExpired` helpers from the read path. -5. **Document model unchanged** — same `_id`, `class`, fields. Stored as JSON values. Java/Go parity preserved at the document level even though the storage engine differs. - -## Java parity verification - -Cross-checked against existing parity reports (commit `e3e375d`): -- `plans/reports/xia-260429-1601-store-scraper-bot-port-status.md` — 11 PARITY · 2 MINOR · **0 GAP** · 3 EXTRA -- `plans/reports/researcher-260429-1555-java-vs-js-parity.md` — schema, scheduler timing, cache TTL, HTML formatting all confirmed identical - -**KV swap preserves every PARITY item.** Document shape (`_id`, `class`, fields) is written verbatim as JSON values. Cache TTL via `expirationTtl` is behaviorally equivalent to Java's `(now - millis) > cacheMillis` read-side check (same cache-miss→refetch boundary). `countDocuments` and `findOne` map cleanly to `get !== null` and `get`. - -Two intentional divergences from Java/Mongo, both accepted: -1. KV ~60s eventual consistency (Java/Mongo was strongly consistent) — fine for single-admin daily-cron use. -2. KV `expirationTtl` 60s floor (Java had no minimum) — mitigated by a clamp in `kv.js`. - -## Known KV trade-offs (accepted) - -- **Eventual consistency (~60s global propagation)**: `/addgroup` followed by an immediate cron run on a different POP could miss the new group. For this single-admin bot with a once-daily cron, acceptable. -- **No atomic read-modify-write**: Two concurrent `/addgroup` calls could clobber each other on the singleton's `groups[]` array. Mongo's `replaceOne` had the same race; not a regression. -- **Free tier**: 100k reads/day, 1k writes/day, 1 GB storage. Bot usage projects <100 reads/day. - -## Out of scope - -- ~~Data migration — greenfield per `plans/todo.md` Q3.~~ Now in scope as Phase 06. -- Tests — none exist; not adding here. -- Custom domain — `*.workers.dev` URL is fine for Telegram webhooks. -- Durable Objects / D1 — KV is sufficient. - -## Rollback path - -- After Phase 01 (code-only): `git revert` reverts the repo refactor; `mongodb` driver back, plan reopens against Atlas-based deploy plan. -- After Phase 03 (deployed but no webhook): `wrangler delete` removes Worker. -- After Phase 04 (webhook live): `setWebhook` with empty URL clears it; rotate `TELEGRAM_WEBHOOK_SECRET` so any in-flight POSTs 401. -- KV namespace can be retained — empty namespaces cost nothing on Free tier. - -## Supersedes - -This plan replaces the storage backend chosen in [`260426-2327-cloudflare-deploy-and-smoke`](../260426-2327-cloudflare-deploy-and-smoke/plan.md). On completion, that plan should be archived. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-01-vercel-scaffolding.md b/plans/260509-1656-consolidate-vercel-upstash/phase-01-vercel-scaffolding.md deleted file mode 100644 index 32bf7c9..0000000 --- a/plans/260509-1656-consolidate-vercel-upstash/phase-01-vercel-scaffolding.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -phase: 1 -title: "Vercel scaffolding + deps" -status: pending -priority: P1 -effort: 30 min -dependencies: [] ---- - -# Phase 1: Vercel scaffolding + deps - -## Overview - -Add Vercel project config, install new runtime deps, restructure `package.json` for Node-on-Vercel. No behavior changes yet — just scaffolding. - -## Requirements - -- Vercel project recognizes the repo (Node 20 runtime, ESM) -- Daily cron registered (00:00 UTC = 07:00 Asia/Saigon) -- New deps installed; `mongodb` devDep stays (still needed by Phase 5 migration script) - -## Architecture - -``` -vercel.json cron schedule, function config (maxDuration, region) -package.json + app-store-scraper, google-play-scraper, - + @upstash/redis, @vercel/functions - scripts: dev → vercel dev, deploy → vercel deploy -.vercelignore exclude /plans, /scripts/.atlas-export.json -``` - -Vercel auto-detects `api/*.js` as serverless functions in Phase 4. This phase only sets the scaffolding. - -## Related Code Files - -- Create: `vercel.json` -- Create: `.vercelignore` -- Modify: `package.json` (deps, scripts) - -## Implementation Steps - -1. Add deps: - ```sh - npm install app-store-scraper@^0.18.0 google-play-scraper@^10.1.2 @upstash/redis @vercel/functions - ``` -2. Update `package.json` scripts: - - `dev`: `vercel dev` (was `wrangler dev`) - - `deploy`: `vercel deploy --prod && npm run register` - - Keep `lint`. Drop `migrate:bulk` (CF-specific). `migrate` script gets repointed in Phase 5 to the new Atlas → Upstash script. - - Keep `register`: `node --env-file=.env.deploy scripts/register-webhook.js` -3. Create `vercel.json`: - ```json - { - "crons": [{ "path": "/api/cron", "schedule": "0 0 * * *" }], - "functions": { - "api/cron.js": { "maxDuration": 60 }, - "api/webhook.js": { "maxDuration": 30 } - } - } - ``` -4. Create `.vercelignore` excluding `plans/`, `*.md`, `Dockerfile`, `docker-compose*.yml`, `wrangler.toml` (Phase 7 deletes the last four; Vercel just needs to ignore them in the meantime). New Phase 5 migration script writes directly to Upstash — no `scripts/.atlas-export.json` artifact to ignore. -5. `npm install` and verify `node_modules` has all four new deps. - -## Success Criteria - -- [ ] `package.json` shows new deps + new scripts -- [ ] `vercel.json` parses (`vercel dev --debug` doesn't error on config) -- [ ] `.vercelignore` excludes irrelevant paths -- [ ] `npm install` exits clean - -## Risk Assessment - -- **Risk:** Vercel function bundle exceeds 250 MB with all deps. **Mitigation:** check `vercel build --debug` output; tree-shaking + `excludeFiles` pattern in `vercel.json` if needed. Expected size ~30 MB. -- **Risk:** `app-store-scraper` uses deprecated `request` lib that may warn or fail under Node 20. **Mitigation:** verify in Phase 3 by importing + calling `app()`. Fallback: pin Node to 18 or swap lib (out of scope). diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-02-upstash-repository-adapter.md b/plans/260509-1656-consolidate-vercel-upstash/phase-02-upstash-repository-adapter.md deleted file mode 100644 index 1802e02..0000000 --- a/plans/260509-1656-consolidate-vercel-upstash/phase-02-upstash-repository-adapter.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -phase: 2 -title: "Upstash repository adapter" -status: pending -priority: P1 -effort: 1h -dependencies: [1] ---- - -# Phase 2: Upstash repository adapter - -## Overview - -Replace `src/repository/kv.js` Cloudflare KV wrapper with an Upstash Redis equivalent. Repository files (`admin-repository.js`, `group-repository.js`, `apple-app-repository.js`, `google-app-repository.js`) keep their public APIs; only the storage primitive changes. - -## Requirements - -- Function-level API of `kv.js` preserved (`getJson`, `putJson`, `del`) -- TTL semantics preserved (Apple/Google cache uses `expirationTtl` → Redis `EX`) -- Drops `env.STORE_KV` binding; takes Upstash client instance instead -- 60s minimum TTL clamp removed (Redis `EX` accepts 1s+, but keep clamp for parity safety) -- **Multi-tenancy isolation:** every key gets a namespace prefix (`KEY_PREFIX` env var, default `store-scraper-bot:`) so this bot's data cannot collide with other Vercel projects sharing the same Upstash DB. Applied transparently in the adapter — repository code stays prefix-unaware. - -## Architecture - -``` -src/repository/ -├── upstash.js ← NEW: client factory + getJson/putJson/del/scan -├── kv.js ← DELETE (or rename to keep history) -├── store.js ← takes upstash client instead of env -├── admin-repository.js ← unchanged signatures, internal: env→client -├── group-repository.js ← unchanged signatures -├── apple-app-repository.js ← unchanged signatures -└── google-app-repository.js← unchanged signatures -``` - -Client construction with namespace prefix: -```js -import { Redis } from '@upstash/redis'; - -export function createUpstashClient(env) { - const client = new Redis({ - url: env.UPSTASH_REDIS_REST_URL, - token: env.UPSTASH_REDIS_REST_TOKEN, - }); - const prefix = env.KEY_PREFIX ?? 'store-scraper-bot:'; - return { client, prefix }; -} - -const k = (prefix, key) => `${prefix}${key}`; -``` - -All adapter functions (`getJson`, `putJson`, `del`, `scan`) prepend `prefix` before hitting Redis. Repository callers pass logical keys (`admin`, `group:42`, etc.); the adapter translates to physical keys (`store-scraper-bot:admin`, `store-scraper-bot:group:42`, etc.). Same logic applies to scan match patterns. - -API mapping: -| CF KV | Upstash Redis | Note | -|---|---|---| -| `kv.get(key, 'json')` | `redis.get(prefix+key)` | Upstash auto-deserializes JSON when value was `JSON.stringify`d on `set` | -| `kv.put(key, value)` | `redis.set(prefix+key, value)` | | -| `kv.put(key, value, { expirationTtl: N })` | `redis.set(prefix+key, value, { ex: N })` | Redis `EX` is seconds | -| `kv.delete(key)` | `redis.del(prefix+key)` | | -| `kv.list({ prefix: 'group:' })` | `redis.scan(0, { match: prefix+'group:*' })` | Phase 5 migration uses this | - -## Related Code Files - -- Create: `src/repository/upstash.js` -- Modify: `src/repository/store.js` (accept client, not env) -- Modify: `src/repository/admin-repository.js` (use client) -- Modify: `src/repository/group-repository.js` (use client) -- Modify: `src/repository/apple-app-repository.js` (use client) -- Modify: `src/repository/google-app-repository.js` (use client) -- Delete: `src/repository/kv.js` - -## Implementation Steps - -1. Write `src/repository/upstash.js`: - - `createUpstashClient(env)` returns `{ client, prefix }` object (prefix from `env.KEY_PREFIX ?? 'store-scraper-bot:'`) - - `getJson(handle, key)`, `putJson(handle, key, value, { expirationTtl } = {})`, `del(handle, key)`, `scan(handle, matchSuffix)` all prepend `handle.prefix` to the key before calling `handle.client` - - Mirror `kv.js` signatures so repository files only swap import path + first arg -2. Update `store.js` to accept the handle instead of `env`: - ```js - export function createStore(handle, appCacheSeconds) { ... } - ``` -3. Update each repository to import from `./upstash.js` and take `handle` instead of `env`. Search/replace `env, key` → `handle, key` in each file. Repositories continue passing logical keys (`admin`, `group:${id}`, etc.) — they never see the prefix. -4. Delete `src/repository/kv.js`. -5. Manual smoke: `node -e "import('./src/repository/upstash.js').then(m => console.log(m))"` to confirm import path resolves. -6. Verify isolation: with `KEY_PREFIX=store-scraper-bot:`, write a test value via `putJson(handle, 'admin', {x:1})` and confirm in Upstash dashboard that the physical key is `store-scraper-bot:admin`. - -## Success Criteria - -- [ ] `src/repository/upstash.js` exports `createUpstashClient`, `getJson`, `putJson`, `del`, `scan` -- [ ] `createUpstashClient` reads `KEY_PREFIX` (default `store-scraper-bot:`) and bundles it into the returned handle -- [ ] All four `*-repository.js` files import from `./upstash.js`, take `handle` -- [ ] `store.js` signature: `createStore(handle, appCacheSeconds)` -- [ ] `kv.js` deleted -- [ ] No `env.STORE_KV` references remain in `src/` -- [ ] Smoke write produces a physically-prefixed key in Upstash (e.g., `store-scraper-bot:admin`) - -## Risk Assessment - -- **Risk:** `@upstash/redis` SDK auto-stringifies/parses values inconsistently. **Mitigation:** read SDK docs; if auto-JSON disabled, mirror `kv.js`'s `JSON.stringify`/`JSON.parse` explicitly. -- **Risk:** Redis `SET key value EX 0` is invalid; current `KV_MIN_TTL_SECONDS = 60` clamp keeps us safe. Keep the clamp. -- **Risk:** Redis returns `null` on missing key (same as KV). Verify in unit smoke. -- **Risk:** `KEY_PREFIX` mismatch between bot runtime and migration script ⇒ migrated data unreadable. **Mitigation:** both reads from same `.env.deploy` / Vercel env var; document the contract in README. Default value `store-scraper-bot:` is the single source of truth — only override if explicitly multi-tenant. -- **Risk:** Forgetting to apply prefix to `scan` match patterns causes data leak / misses. **Mitigation:** centralize all key composition inside `upstash.js`; repositories must never pass raw match patterns to the client. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-03-inline-scraper-modules.md b/plans/260509-1656-consolidate-vercel-upstash/phase-03-inline-scraper-modules.md deleted file mode 100644 index 9109bb8..0000000 --- a/plans/260509-1656-consolidate-vercel-upstash/phase-03-inline-scraper-modules.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -phase: 3 -title: "Inline scraper modules" -status: pending -priority: P1 -effort: 45 min -dependencies: [1] ---- - -# Phase 3: Inline scraper modules - -## Overview - -Replace `fetch('https://store-scraper.vercel.app/...')` with direct calls to `app-store-scraper` and `google-play-scraper` npm libs. Eliminates the cross-service HTTP roundtrip. - -## Requirements - -- `apple-scraper.js` and `google-scraper.js` keep their exported function signatures (`createAppleScraper`, `createGoogleScraper`, `buildAppleRequestBy*`, `buildGoogleRequest`) so command handlers don't change -- Behavior parity: `rawApp(req)` returns JSON string; `app(req)` returns parsed object -- Library `app()` method only — no other methods used today - -## Architecture - -``` -src/api/apple-scraper.js - ├─ removed: BASE_URL, fetch() - └─ uses: import store from 'app-store-scraper'; store.app(params) - -src/api/google-scraper.js - ├─ removed: BASE_URL, fetch() - └─ uses: import gplay from 'google-play-scraper'; gplay.app(params) -``` - -`rawApp` returns `JSON.stringify(result)` since libs return JS objects, not strings. Command `/rawappleapp` and `/rawgoogleapp` consumers expect text. - -## Related Code Files - -- Modify: `src/api/apple-scraper.js` -- Modify: `src/api/google-scraper.js` - -## Implementation Steps - -1. Edit `src/api/apple-scraper.js`: - ```js - import store from 'app-store-scraper'; - import { newAppleApp } from '../models/apple-app.js'; - - export function buildAppleRequestByTrackId(id, country) { - return { id, country, ratings: true }; - } - export function buildAppleRequestByBundleId(appId, country) { - return { appId, country, ratings: true }; - } - - export function createAppleScraper(config, storeRepo) { - const { logger } = config; - const repo = storeRepo.appleApp; - - async function app(req) { - return await store.app(req); - } - async function rawApp(req) { - return JSON.stringify(await app(req)); - } - // cache, getApp, fetchAndCache: unchanged - ... - } - ``` -2. Edit `src/api/google-scraper.js` analogously, importing `gplay` from `google-play-scraper`. -3. Verify no other files import `BASE_URL` (grep `src/ -r 'BASE_URL'`). -4. Smoke from a Node REPL after `npm install` (Phase 1 added the deps): - ```sh - node -e "import('app-store-scraper').then(m => m.default.app({ appId: 'com.apple.weather', country: 'us' })).then(r => console.log(r.appId, r.version))" - node -e "import('google-play-scraper').then(m => m.default.app({ appId: 'com.spotify.music', country: 'us' })).then(r => console.log(r.appId, r.version))" - ``` - -## Success Criteria - -- [ ] `grep -r 'store-scraper.vercel.app' src/` returns zero hits -- [ ] Local smoke calls succeed against both stores -- [ ] `apple-scraper.js`, `google-scraper.js` keep exported names; no other file modified -- [ ] Returned shape from `app()` matches what cache models (`newAppleApp`, `newGoogleApp`) expect - -## Risk Assessment - -- **Risk:** `app-store-scraper.app({ ratings: true })` makes 2 HTTPS calls (lookup + ratings.svg). Latency higher than single-fetch. **Mitigation:** acceptable for command-driven calls; cache TTL covers daily cron amplification. -- **Risk:** `google-play-scraper.app()` HTML scrape can fail when Google changes Play page markup. **Mitigation:** error already bubbles to user via existing try/catch in commands. Monitor first week post-deploy. -- **Risk:** `app-store-scraper` uses deprecated `request` lib — Node 20 warns but still runs. **Mitigation:** suppress with `NODE_NO_WARNINGS=1` if noisy. Long-term: track lib for replacement. -- **Risk:** Returned objects may have non-JSON-serializable fields (Dates). **Mitigation:** `JSON.stringify` produces ISO strings; consumers already parse strings. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-04-http-layer-webhook-and-cron.md b/plans/260509-1656-consolidate-vercel-upstash/phase-04-http-layer-webhook-and-cron.md deleted file mode 100644 index 1aecbd3..0000000 --- a/plans/260509-1656-consolidate-vercel-upstash/phase-04-http-layer-webhook-and-cron.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -phase: 4 -title: "HTTP layer (webhook + cron)" -status: pending -priority: P1 -effort: 1h -dependencies: [2, 3] ---- - -# Phase 4: HTTP layer (webhook + cron) - -## Overview - -Replace the Cloudflare Worker default-export entry (`src/index.js` with `fetch`/`scheduled` handlers) with two Vercel serverless functions: `api/webhook.js` (Telegram webhook) and `api/cron.js` (daily check via Vercel Cron). - -## Requirements - -- Telegram webhook validates `X-Telegram-Bot-Api-Secret-Token` header (parity with current CF check) -- Webhook acks fast (<2 s), continues work in background via `@vercel/functions` `waitUntil` -- Cron handler runs daily check, returns 200 on success -- Both handlers use `loadConfig(process.env)` instead of CF `env` arg -- Build wires up `createUpstashClient` once per invocation (cheap — no connection); `KEY_PREFIX` flows through `process.env` into the handle so all reads/writes stay namespaced - -## Architecture - -``` -api/ -├── webhook.js POST handler — Telegram webhook -│ validates secret, calls dispatch(message), waitUntil -└── cron.js GET handler — Vercel Cron triggers daily - validates Authorization: Bearer $CRON_SECRET (Vercel signs cron requests) - runs runDailyCheck(...) - -src/ -├── index.js ← DELETE (CF default export gone) -├── config.js ← Modify: loadConfig(env) → loadConfig(process.env or env) -└── (rest unchanged) -``` - -Vercel cron requests carry `Authorization: Bearer ${process.env.CRON_SECRET}` header — validate to prevent random POSTs from triggering daily check. - -## Related Code Files - -- Create: `api/webhook.js` -- Create: `api/cron.js` -- Modify: `src/config.js` (env source) -- Delete: `src/index.js` - -## Implementation Steps - -1. Update `src/config.js` to take a plain object (works with both `process.env` and CF `env`): - ```js - export function loadConfig(env) { - // existing logic, but `env` is now `process.env` on Vercel - } - ``` -2. Create `api/webhook.js`: - ```js - import { waitUntil } from '@vercel/functions'; - import { loadConfig } from '../src/config.js'; - import { createUpstashClient } from '../src/repository/upstash.js'; - import { createStore } from '../src/repository/store.js'; - import { createAppleScraper } from '../src/api/apple-scraper.js'; - import { createGoogleScraper } from '../src/api/google-scraper.js'; - import { createBot } from '../src/bot/bot.js'; - import { dispatch } from '../src/bot/dispatch.js'; - - function build() { - const config = loadConfig(process.env); - const client = createUpstashClient(process.env); - const store = createStore(client, config.appCacheSeconds); - const appleScraper = createAppleScraper(config, store); - const googleScraper = createGoogleScraper(config, store); - const { sender, commands } = createBot(config, store, appleScraper, googleScraper); - return { config, store, sender, commands }; - } - - export default async function handler(req) { - if (req.method !== 'POST') return new Response('Not found', { status: 404 }); - - const app = build(); - const secret = req.headers.get('X-Telegram-Bot-Api-Secret-Token'); - if (secret !== app.config.telegramWebhookSecret) { - return new Response('Unauthorized', { status: 401 }); - } - let update; - try { update = await req.json(); } catch { return new Response('Bad request', { status: 400 }); } - if (!update?.message) return new Response('OK'); - - waitUntil(dispatch(update.message, { - sender: app.sender, commands: app.commands, - config: app.config, logger: app.config.logger, - })); - return new Response('OK'); - } - - export const config = { runtime: 'nodejs' }; - ``` -3. Create `api/cron.js`: - ```js - import { runDailyCheck } from '../src/scheduler/scheduler.js'; - // (same build() helper — extract to src/app-builder.js if duplication bothers) - - export default async function handler(req) { - const auth = req.headers.get('Authorization'); - if (auth !== `Bearer ${process.env.CRON_SECRET}`) { - return new Response('Unauthorized', { status: 401 }); - } - const app = build(); - await runDailyCheck(app.config, app.store, app.sender, app.appleScraper, app.googleScraper); - return new Response('OK'); - } - - export const config = { runtime: 'nodejs', maxDuration: 60 }; - ``` -4. Extract shared `build()` into `src/app-builder.js` (DRY); both functions import. -5. Delete `src/index.js`. -6. Local test: `vercel dev` → `curl -X POST localhost:3000/api/webhook` (expect 401), then with valid secret + JSON body (expect 200). - -## Success Criteria - -- [ ] `api/webhook.js` and `api/cron.js` exist; both exported as default async handlers -- [ ] Both validate auth (Telegram secret / Cron bearer) -- [ ] `vercel dev` starts without error -- [ ] `src/index.js` deleted -- [ ] `src/config.js` works with `process.env` - -## Risk Assessment - -- **Risk:** `runtime: 'nodejs'` defaults vary across Vercel CLI versions; some default to edge. **Mitigation:** explicit declaration in function `export const config`. -- **Risk:** `waitUntil` requires Vercel runtime; behaves no-op in `vercel dev`. **Mitigation:** acceptable; production deploy honors it. Verify in Phase 6. -- **Risk:** Cold start on first webhook invocation reaches ~500 ms while Telegram retries on >30 s. **Mitigation:** comfortable margin. Pre-warm not needed. -- **Risk:** `CRON_SECRET` mishandling makes cron callable by anyone. **Mitigation:** generate ≥32-char secret, store as Vercel env var, set on Vercel Cron settings (Vercel injects automatically). diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-05-data-migration-atlas-to-upstash.md b/plans/260509-1656-consolidate-vercel-upstash/phase-05-data-migration-atlas-to-upstash.md deleted file mode 100644 index 335eacc..0000000 --- a/plans/260509-1656-consolidate-vercel-upstash/phase-05-data-migration-atlas-to-upstash.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -phase: 5 -title: "Data migration MongoDB Atlas → Upstash" -status: pending -priority: P1 -effort: 30 min -dependencies: [2] ---- - -# Phase 5: Data migration MongoDB Atlas → Upstash - -## Overview - -One-shot migration: read live state from the legacy `java-store-scraper-bot` MongoDB Atlas (collections `common`, `group`, optional `apple_app`/`google_app`) and write into Upstash Redis with TTL preserved on cache entries. The Java bot remains source of truth until cutover (Phase 6). - -CF KV is **not** the source — the Node port never deployed live, so no CF KV state exists. Skipping CF KV simplifies the cutover. - -## Requirements - -- Migrate `admin` (from `common._id == "admin"`) and `group` collection (must preserve) -- Skip `apple_app`/`google_app` cache by default (auto-rebuilds within `APP_CACHE_SECONDS`); `--include-cache` flag for parity -- Idempotent — safe to re-run; overwrites with same data -- Preserves TTL on cache entries (recompute from `millis` field) -- Read-only against Atlas — Java bot keeps serving until Phase 6 cutover -- **Apply same `KEY_PREFIX` (default `store-scraper-bot:`) as runtime adapter** — physical Upstash keys must match what Phase 2 adapter reads. Mismatch ⇒ migrated data invisible to bot. - -## Architecture - -Adapt existing `scripts/migrate-atlas-to-kv.js`. Replace the JSON-file output + `wrangler bulk put` step with direct Upstash REST writes via `@upstash/redis`: - -``` -scripts/migrate-atlas-to-upstash.js - PREFIX = process.env.KEY_PREFIX ?? 'store-scraper-bot:' - 1. MongoClient(MONGODB_URI) — connects to legacy Atlas (read-only) - 2. Read common.findOne({_id:'admin'}) → SET PREFIX+admin - 3. Read group.find({}) → for each: SET PREFIX+group:<_id> - 4. If --include-cache: - Read apple_app.find({}) → SET PREFIX+apple:<_id> EX - Read google_app.find({}) → SET PREFIX+google:<_id> EX - 5. Log counts: admin, groups, apple (kept/skipped), google (kept/skipped) - 6. Log effective PREFIX so operator can spot mismatches early -``` - -Reuses `remainingTtl()` logic from existing migrate script. No on-disk JSON intermediate — direct Atlas → Upstash, so bot state never touches the filesystem. - -## Related Code Files - -- Create: `scripts/migrate-atlas-to-upstash.js` (adapt from `scripts/migrate-atlas-to-kv.js`) -- Delete: `scripts/migrate-atlas-to-kv.js` (CF KV path obsolete) -- Modify: `package.json` - - Replace `"migrate"` script: `node --env-file=.env.deploy scripts/migrate-atlas-to-upstash.js` - - Drop `"migrate:bulk"` (CF-specific) - -## Implementation Steps - -1. Copy `scripts/migrate-atlas-to-kv.js` → `scripts/migrate-atlas-to-upstash.js`. -2. Replace JSON-array assembly + `writeFile` with Upstash writes (applying `KEY_PREFIX`): - ```js - import { Redis } from '@upstash/redis'; - const redis = new Redis({ - url: process.env.UPSTASH_REDIS_REST_URL, - token: process.env.UPSTASH_REDIS_REST_TOKEN, - }); - const PREFIX = process.env.KEY_PREFIX ?? 'store-scraper-bot:'; - log(`prefix: ${PREFIX} (must match bot runtime KEY_PREFIX)`); - // for each entry: - await redis.set(`${PREFIX}${key}`, JSON.stringify(value), ttl ? { ex: ttl } : undefined); - ``` -3. Add `--dry-run` flag: print actions without writing to Upstash. -4. Drop the `entries.length > 10000` bulk-put cap (Redis has no such limit). -5. Update `package.json`: - ```json - "migrate": "node --env-file=.env.deploy scripts/migrate-atlas-to-upstash.js", - "migrate:dry": "node --env-file=.env.deploy scripts/migrate-atlas-to-upstash.js --dry-run" - ``` - `.env.deploy` carries both `MONGODB_URI` (Atlas read) and `UPSTASH_REDIS_REST_URL`/`_TOKEN` (Upstash write). -6. Delete `scripts/migrate-atlas-to-kv.js`. -7. Test: `npm run migrate:dry` — verify counts match expected admin (1) + groups (production count). - -## Success Criteria - -- [ ] `npm run migrate:dry` lists `admin` + all `group:*` keys with correct counts and prints effective `KEY_PREFIX` -- [ ] `npm run migrate` writes to Upstash; physical keys carry the prefix (e.g. `store-scraper-bot:admin`, `store-scraper-bot:group:42`) -- [ ] Admin + group counts in Upstash dashboard (filtered by prefix) match Atlas -- [ ] `admin` doc value byte-identical between Atlas and Upstash (read-back compare with prefix) -- [ ] Re-run is idempotent (no duplicate keys, same final state) -- [ ] `scripts/migrate-atlas-to-kv.js` deleted; `package.json` no longer references `migrate:bulk` - -## Risk Assessment - -- **Risk:** Atlas auto-pause (free tier) delays first connection ~5–30 s. **Mitigation:** bump `serverSelectionTimeoutMS` from 5000 to 30000 for migration tolerance. One-shot, not perf-critical. -- **Risk:** Migration runs while Java bot still writes to Atlas (lost writes during gap). **Mitigation:** Phase 6 cutover stops Java bot polling BEFORE migration; downtime <5 min. -- **Risk:** `@upstash/redis` SDK auto-stringifies values inconsistently. **Mitigation:** explicitly `JSON.stringify` values before `redis.set`; verify by reading back via `redis.get` and comparing. -- **Risk:** TTL on cache keys near `APP_CACHE_SECONDS` boundary lands with near-zero TTL. **Mitigation:** `KV_MIN_TTL_SECONDS = 60` clamp from existing script keeps Redis happy. -- **Risk:** Atlas data exceeds Upstash free 256 MB cap. **Mitigation:** bot has <100 keys; sub-megabyte. Even with cache, few MB max. -- **Risk:** `KEY_PREFIX` differs between migration script and Vercel env at runtime ⇒ bot reads from a different namespace and sees empty state. **Mitigation:** script logs prefix on start; Phase 6 success criteria includes verifying bot can `/listgroup` post-cutover (tests prefix alignment end-to-end). diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-06-deploy-cutover-and-webhook-reregister.md b/plans/260509-1656-consolidate-vercel-upstash/phase-06-deploy-cutover-and-webhook-reregister.md deleted file mode 100644 index 294a4af..0000000 --- a/plans/260509-1656-consolidate-vercel-upstash/phase-06-deploy-cutover-and-webhook-reregister.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -phase: 6 -title: "Deploy + cutover + webhook re-register" -status: completed -priority: P1 -effort: 45 min -dependencies: [4, 5] ---- - -# Phase 6: Deploy + cutover + webhook re-register - -## Overview - -Deploy Vercel project, configure secrets, run migration, flip Telegram webhook URL. Hard cutover with <5 min downtime. - -## Requirements - -- Vercel project deployed with all secrets set -- Upstash database provisioned (free tier) -- Telegram webhook points to new Vercel URL -- All 13 commands smoke-test pass on production -- Java bot (legacy) kept running but webhook detached — rollback path for ~7 days - -## Architecture - -Sequence (operator-driven, ~30 min wall clock): - -``` -1. PROVISION - Upstash → create free Redis DB → copy REST_URL + REST_TOKEN - Vercel → create project from repo or link existing - -2. SECRETS (Vercel env vars) - TELEGRAM_BOT_TOKEN - TELEGRAM_BOT_USERNAME - TELEGRAM_WEBHOOK_SECRET (≥32 chars random) - ADMIN_IDS (comma-separated) - UPSTASH_REDIS_REST_URL - UPSTASH_REDIS_REST_TOKEN - CRON_SECRET (≥32 chars random; for cron auth) - KEY_PREFIX=store-scraper-bot: (namespace this bot's keys; must match what migration script used) - APP_CACHE_SECONDS=600 - NUM_DAYS_WARNING_NOT_UPDATED=30 - -3. DEPLOY - vercel deploy --prod (gets URL like store-scraper-bot-xxx.vercel.app) - -4. SMOKE (no traffic yet) - curl -X POST $URL/api/webhook → 401 (no secret) → confirms route alive - curl with valid secret + dummy body → 200 - -5. MIGRATE DATA (downtime starts) - Stop Java bot polling: docker compose down (or systemctl stop) on the Java host - → no further writes land on Atlas - npm run migrate (Atlas → Upstash, reads MONGODB_URI from .env.deploy) - verify admin + group counts in Upstash dashboard match Atlas - -6. FLIP - .env.deploy: WORKER_URL=https:///api/webhook (var name kept; existing register-webhook.js reads it) - npm run register (registers webhook for the first time + secret) - wait 30 s for Telegram propagation - -7. SMOKE COMMANDS (production) - /info → returns chat ID - /listgroup (admin) → lists migrated groups - /listapp (in known group) → lists tracked apps - /checkapp → fetches via inlined scrapers - ... (run all 13 commands per docs/code-standards.md or tester checklist) - -8. WATCH - tail Vercel function logs 30 min for error spikes - verify daily cron at 00:00 UTC fires (next morning) -``` - -## Related Code Files - -- Modify: `.env.deploy` (operator file, gitignored — change `WORKER_URL`) -- No code changes; this phase is operator-driven - -## Implementation Steps - -1. Sign up Upstash → create Redis database (region matching Vercel deploy region — `us-east-1` default). -2. Vercel: link existing project OR create new from `tiennm99/store-scraper-bot` repo. Set all env vars listed above via `vercel env add` or dashboard. -3. `vercel deploy --prod`. Note returned URL. -4. Update `.env.deploy`: set `WORKER_URL` to `https:///api/webhook` (variable name retained for existing script); ensure `MONGODB_URI` (Atlas, read-only OK) and `UPSTASH_REDIS_REST_URL`/`_TOKEN` are present. -5. Stop the Java bot (long-poll consumer) so no further writes hit Atlas. Confirm in Telegram by sending a no-op command and seeing it not get answered. -6. Run `npm run migrate` (Phase 5 script: Atlas → Upstash). Verify counts in Upstash dashboard match Atlas. -7. Run `npm run register` to set webhook to Vercel URL. -8. Smoke all 13 commands manually in Telegram. Use `docs/` test checklist if present. -9. Tail Vercel logs: `vercel logs --follow` for 15 min. -10. Document Vercel URL + Upstash DB name in plan/journal. - -## Success Criteria - -- [ ] `getWebhookInfo` shows new Vercel URL -- [ ] All 13 commands smoke-pass on production (manually checked) -- [ ] Daily cron fires at 00:00 UTC (verify in Vercel Functions log next morning) -- [ ] Vercel function logs show no error-level entries for 30 min -- [ ] Upstash dashboard shows `admin` + `group:*` keys with expected count -- [ ] No keys exist in Upstash without the configured `KEY_PREFIX` (confirms isolation; nothing leaked into shared namespace) - -## Risk Assessment - -- **Risk:** Vercel function cold start exceeds Telegram 30 s ack window on first hit. **Mitigation:** ack returns 200 OK fast (before `waitUntil` heavy work); cold start typically <500 ms. -- **Risk:** Migration loses TTL on cache entries. **Mitigation:** acceptable (cache rebuilds in 10 min). Skipped by default. -- **Risk:** Smoke missed an edge case → user-visible regression. **Mitigation:** Java bot kept idle but available for 7 days; rollback = restart Java bot polling + clear Telegram webhook. ~5 min recovery (Java bot picks up via long polling once webhook is cleared). Note: any commands handled by Vercel during the rollback window won't be in Atlas — accept minor state divergence on rollback. -- **Risk:** Vercel free tier rate-limits cold starts during smoke. **Mitigation:** 100 invocations/day quota is plenty; smoke uses ~20. -- **Risk:** Operator forgets `CRON_SECRET` → cron handler always 401. **Mitigation:** verify by triggering cron manually via Vercel dashboard "Run Now" before relying on schedule. diff --git a/plans/260509-1656-consolidate-vercel-upstash/phase-07-cleanup-wrangler-and-docs.md b/plans/260509-1656-consolidate-vercel-upstash/phase-07-cleanup-wrangler-and-docs.md deleted file mode 100644 index 8311889..0000000 --- a/plans/260509-1656-consolidate-vercel-upstash/phase-07-cleanup-wrangler-and-docs.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -phase: 7 -title: "Cleanup wrangler + docker + docs" -status: completed -priority: P2 -effort: 30 min -dependencies: [6] ---- - -# Phase 7: Cleanup wrangler + docker + docs - -## Overview - -After Vercel deploy is stable (≥7 days post-cutover), remove Cloudflare Workers + Docker artifacts (both irrelevant to Vercel deploy), retire the old `tiennm99/store-scraper` Vercel wrapper, and update README + docs to reflect the new architecture. - -## Requirements - -- Repo no longer references CF Workers, KV, `wrangler`, Docker, or local Mongo (except in `plans/` archive history) -- README accurately describes Vercel + Upstash setup -- One-shot migration script `migrate-atlas-to-upstash.js` removed (role done after Phase 6) -- `mongodb` devDep removed (Atlas migration script deleted with it) -- All three Docker files deleted: `Dockerfile`, `docker-compose.yml`, `docker-compose.dev.yml` -- `tiennm99/store-scraper` repo marked deprecated - -## Architecture - -Files to remove: - -``` -DELETE -├── wrangler.toml -├── Dockerfile (Mongo-based local container; obsolete with Upstash) -├── docker-compose.yml (bot + mongo service; obsolete) -├── docker-compose.dev.yml (dev mongo service; obsolete) -├── scripts/migrate-atlas-to-upstash.js (one-shot done in Phase 6) -└── (CF Worker on dashboard) wrangler delete (only if CF Worker was ever deployed) - -MODIFY -├── package.json remove wrangler, mongodb devDeps; drop migrate scripts -├── README.md Vercel + Upstash setup; drop Atlas/Docker/KV history -└── docs/ if any architecture docs reference CF or Docker -``` - -Phase 5 already deleted `scripts/migrate-atlas-to-kv.js`. Phase 7 removes the new `migrate-atlas-to-upstash.js` script after the one-shot is confirmed complete. - -External cleanup: - -``` -- Cloudflare dashboard: delete Worker deployment -- Cloudflare dashboard: delete STORE_KV namespace (after verifying Upstash data preserved) -- Vercel dashboard: archive `store-scraper` project (the old wrapper) OR mark README deprecated -- GitHub `tiennm99/store-scraper` repo: add deprecation banner to README pointing to bot repo -``` - -## Related Code Files - -- Delete: `wrangler.toml` -- Delete: `Dockerfile` -- Delete: `docker-compose.yml` -- Delete: `docker-compose.dev.yml` -- Delete: `scripts/migrate-atlas-to-upstash.js` -- Modify: `package.json` (drop `wrangler`, `mongodb`; drop `migrate*` scripts) -- Modify: `README.md` - -## Implementation Steps - -1. Wait ≥7 days post-Phase 6 with no incidents. -2. Delete `wrangler.toml`. -3. Delete Docker artifacts (no longer needed — Vercel + Upstash REST has no local-Mongo path): - ```sh - git rm Dockerfile docker-compose.yml docker-compose.dev.yml - ``` -4. Delete `scripts/migrate-atlas-to-upstash.js` (one-shot done; keep migration logic in plan history). -5. Remove from `package.json`: - - devDeps: `wrangler`, `mongodb` - - scripts: `migrate`, `migrate:dry`, `migrate:bulk` (if any remain) -6. Rewrite `README.md`: - - Replace "Cloudflare Workers" with "Vercel" - - Update env var table (drop MONGODB_URI / STORE_KV; add UPSTASH_REDIS_REST_URL / TOKEN, CRON_SECRET) - - Update Run section: `vercel dev` for local; `vercel deploy --prod` for production - - Drop "Migrating from MongoDB Atlas" section (one-shot completed) - - Drop "Or via Docker Compose" section -7. Update `.vercelignore` — remove now-deleted Dockerfile/docker-compose entries (no longer need to ignore what doesn't exist). -8. Cloudflare dashboard (only if any CF resources were ever provisioned): `wrangler delete` Worker, then delete STORE_KV namespace. Skip if CF was never deployed. -9. `tiennm99/store-scraper` repo on GitHub: - - Add README banner: "⚠️ Deprecated — logic now inlined in [`tiennm99/store-scraper-bot`](https://github.com/tiennm99/store-scraper-bot)" - - Optionally archive the repo (`gh repo archive tiennm99/store-scraper`) -10. Vercel dashboard: delete `store-scraper` project (the old scraper wrapper). -11. Commit: `chore: remove cloudflare + docker + legacy migration scripts` - -## Success Criteria - -- [ ] `git grep -i 'cloudflare\|wrangler\|store_kv\|docker'` returns only `plans/` history hits -- [ ] `package.json` has no `wrangler` or `mongodb` deps; no `migrate*` scripts -- [ ] README accurately describes Vercel + Upstash; no Docker section -- [ ] `Dockerfile`, `docker-compose.yml`, `docker-compose.dev.yml` removed from repo -- [ ] `tiennm99/store-scraper` repo marked deprecated and/or archived -- [ ] Old Vercel `store-scraper` project deleted - -## Risk Assessment - -- **Risk:** CF KV namespace was actually populated and Upstash differs. **Mitigation:** N/A if CF was never deployed (per Atlas-source migration in Phase 5). If CF *was* deployed before plan supersession, manually compare key counts before deleting namespace. -- **Risk:** Other consumers exist for `store-scraper.vercel.app` (third-party users). **Mitigation:** add a 410 Gone response on the old wrapper for 14 days before deleting; deprecation banner in repo gives outside heads-up. -- **Risk:** Someone runs the bot locally via Docker for dev. **Mitigation:** explicitly confirmed not needed (Vercel + Upstash REST replaces local-Mongo dev path); `vercel dev` is the new local entry. Document this in README. -- **Risk:** Java bot still running consumes Atlas connections after this phase. **Mitigation:** out of scope for this repo; coordinate with Java repo retirement separately. diff --git a/plans/260509-1656-consolidate-vercel-upstash/plan.md b/plans/260509-1656-consolidate-vercel-upstash/plan.md deleted file mode 100644 index 15d2ad9..0000000 --- a/plans/260509-1656-consolidate-vercel-upstash/plan.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -title: "Consolidate on Vercel + Upstash Redis" -description: "Deploy bot to Vercel (no Cloudflare). Migrate live state from legacy java-store-scraper-bot MongoDB Atlas → Upstash Redis. Inline scraper libs (drop store-scraper.vercel.app fetch). Delete Docker artifacts. Single repo, single vendor, free tier." -status: completed -priority: P1 -effort: 6h -branch: main -tags: [vercel, upstash, migration, scraper-inline, telegram, atlas-migration, docker-cleanup] -created: 2026-05-09 -blockedBy: [] -blocks: [] -supersedes: 260505-1425-cloudflare-kv-migration-and-deploy ---- - -# Consolidate on Vercel + Upstash - -Replaces the in-progress Cloudflare Workers + KV direction. Brainstorm + research locked these decisions: - -- **Compute:** Vercel serverless functions (Hobby plan, free) -- **Storage:** Upstash Redis (free 500k cmd/mo, persistent RocksDB) -- **Scraper:** inline `app-store-scraper` + `google-play-scraper` npm libs, drop `store-scraper.vercel.app` HTTP roundtrip -- **Data source:** legacy `java-store-scraper-bot` MongoDB Atlas (still serving production); Node port never deployed live, so CF KV is **not** a migration source -- **Cleanup:** delete Dockerfile + docker-compose*.yml (Mongo-based local dev path obsolete with Upstash REST); delete wrangler.toml; delete legacy migration script after one-shot run -- **Old `tiennm99/store-scraper` repo:** leave running 1–2 weeks as fallback, then archive - -Source design: [`reports/brainstorm-260509-1656-consolidate-vercel-upstash.md`](../reports/brainstorm-260509-1656-consolidate-vercel-upstash.md) -Storage research: [`reports/researcher-260509-1656-upstash-vs-atlas.md`](../reports/researcher-260509-1656-upstash-vs-atlas.md) - -## Phases - -| # | Phase | Effort | Status | -|---|---|---|---| -| 01 | [Vercel scaffolding + deps](phase-01-vercel-scaffolding.md) | 30 min | completed | -| 02 | [Upstash repository adapter](phase-02-upstash-repository-adapter.md) | 1h | completed | -| 03 | [Inline scraper modules](phase-03-inline-scraper-modules.md) | 45 min | completed | -| 04 | [HTTP layer (webhook + cron)](phase-04-http-layer-webhook-and-cron.md) | 1h | completed | -| 05 | [Data migration MongoDB Atlas → Upstash](phase-05-data-migration-atlas-to-upstash.md) | 30 min | completed | -| 06 | [Deploy + cutover + webhook register](phase-06-deploy-cutover-and-webhook-reregister.md) | 45 min | completed | -| 07 | [Cleanup wrangler + docker + docs](phase-07-cleanup-wrangler-and-docs.md) | 30 min | completed | - -## Key Constraints - -- Free tier only — no $5/mo CF or Vercel Pro upgrades -- Single service — no separate scraper deployment after cutover -- Hard cutover migration; <5 min downtime acceptable -- Telegram parity preserved; all 13 commands keep working -- Daily cron 07:00 Asia/Saigon = 00:00 UTC on Vercel Cron -- Java bot stays warm 7 days post-cutover for rollback only (then retire) -- **Multi-tenant safe:** all Upstash keys carry `KEY_PREFIX` (default `store-scraper-bot:`) so the same Upstash DB can be shared with other Vercel projects without collision. Prefix is enforced at the adapter layer; repositories stay prefix-unaware. - -## Success Criteria (overall) - -- Vercel deploy returns 200 on Telegram webhook within 2 s -- Daily cron visible in Vercel Functions log, runs without error -- Admin + group state preserved across Atlas → Upstash (count match) -- All Upstash keys carry `KEY_PREFIX`; zero keys leaked into the shared namespace -- `grep -r 'store-scraper.vercel.app' src/` returns zero hits -- `wrangler.toml` + `wrangler` devDep removed; `Dockerfile` + `docker-compose*.yml` removed -- `mongodb` devDep removed (only needed for one-shot Phase 5 migration) -- Bundle size <50 MB; cold start <500 ms - -## Dependencies - -- Vercel account (already exists — bot will replace existing `store-scraper` project or get a new project) -- Upstash account (new — sign up at upstash.com or via Vercel Marketplace integration) -- MongoDB Atlas read access on legacy Java bot's cluster (one-shot, Phase 5) -- Java bot keeps running until cutover (Phase 6); idle but available 7 days post for rollback diff --git a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-01-wrangler-config-deps-lint.md b/plans/archive/260426-2015-cloudflare-worker-code-port/phase-01-wrangler-config-deps-lint.md deleted file mode 100644 index a88ab1e..0000000 --- a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-01-wrangler-config-deps-lint.md +++ /dev/null @@ -1,219 +0,0 @@ -# Phase 01 — Wrangler Config + Deps Swap + Lint Scaffolding - -## Context Links -- miti99bot wrangler.toml: `/config/workspace/tiennm99/miti99bot/wrangler.toml` -- miti99bot secret-leak lint: `/config/workspace/tiennm99/miti99bot/scripts/check-secret-leaks.js` -- Existing `package.json`: `/config/workspace/tiennm99/js-store-scraper-bot/package.json` - -## Overview -- **Priority:** P0 -- **Status:** pending -- **Description:** Pure config + dependency changes. No application code touched. After this, `npm install` succeeds with the new dep set; `wrangler.toml` is in place; lint runs. - -## Key Insights -- Workers cron is **UTC**: `0 0 * * *` UTC = 7am Asia/Ho_Chi_Minh. -- `nodejs_compat_v2` is the lever that gives `node:net`/`node:tls` (needed by `mongodb` driver). -- Secret-leak lint added now so subsequent phases can't sneak in a `console.log(env.MONGODB_URI)`. -- Removing Node-only deps in this phase causes `src/index.js` to fail to load — that's expected and fine; Phase 04 rewrites the entry. Until then, `npm start` is broken (acceptable; we're refactoring). - -## Requirements - -### Functional -- `wrangler.toml` written with `nodejs_compat_v2`, cron `0 0 * * *`, observability block. -- `package.json`: - - Add: `mongodb@^6.7.0` - - Remove: `node-telegram-bot-api`, `node-cron`, `dotenv`, `pino`, `pino-pretty` - - Dev-add: `wrangler@^3` - - Scripts: `dev`, `deploy`, `register`, `register:dry`, `lint` - - `engines.node` stays `>=20` - - Drop `type: module` is **not** needed — Workers ESM is the same syntax. -- `scripts/check-secret-leaks.js` written. -- `.dev.vars.example` written. -- `.env.deploy.example` written. -- `.gitignore` adds `.dev.vars`, `.env.deploy`, `.tmp-deploy/`, `.wrangler/`. - -### Non-functional -- `npm install` succeeds. -- `npm run lint` succeeds (only secret-leak check at this point; will gain `node --check` in later phases). - -## Architecture - -``` -js-store-scraper-bot/ -├── wrangler.toml (NEW) -├── package.json (MODIFIED — deps swap) -├── .gitignore (MODIFIED — Worker artifacts) -├── .dev.vars.example (NEW) -├── .env.deploy.example (NEW) -├── scripts/ -│ └── check-secret-leaks.js (NEW) -└── src/ (UNTOUCHED in this phase) -``` - -## Related Code Files - -### CREATE -- `wrangler.toml` -- `scripts/check-secret-leaks.js` -- `.dev.vars.example` -- `.env.deploy.example` - -### MODIFY -- `package.json` — full rewrite of `dependencies`, `devDependencies`, `scripts`. -- `.gitignore` — append Worker artifact patterns. - -### DELETE -- (none in this phase) - -## Implementation Steps - -1. **Write `wrangler.toml`**: - ```toml - name = "js-store-scraper-bot" - main = "src/index.js" - compatibility_date = "2025-10-01" - compatibility_flags = ["nodejs_compat_v2"] - - [vars] - APP_CACHE_SECONDS = "600" - NUM_DAYS_WARNING_NOT_UPDATED = "30" - - # 0 UTC = 7am Asia/Ho_Chi_Minh - [triggers] - crons = ["0 0 * * *"] - - [observability] - enabled = true - head_sampling_rate = 1 - - [observability.logs] - enabled = true - invocation_logs = true - - # Secrets (set via `wrangler secret put`, NOT here): - # TELEGRAM_BOT_TOKEN - # TELEGRAM_BOT_USERNAME - # TELEGRAM_WEBHOOK_SECRET - # MONGODB_URI - # ADMIN_IDS - ``` - -2. **Rewrite `package.json`**: - ```json - { - "name": "js-store-scraper-bot", - "version": "0.2.0", - "description": "JavaScript port of store-scraper-bot, deployable to Cloudflare Workers.", - "type": "module", - "private": true, - "engines": { "node": ">=20" }, - "main": "src/index.js", - "scripts": { - "dev": "wrangler dev", - "deploy": "wrangler deploy && npm run register", - "register": "node --env-file=.env.deploy scripts/register-webhook.js", - "register:dry": "node --env-file=.env.deploy scripts/register-webhook.js --dry-run", - "lint": "node scripts/check-secret-leaks.js" - }, - "dependencies": { "mongodb": "^6.7.0" }, - "devDependencies": { "wrangler": "^3" }, - "license": "Apache-2.0" - } - ``` - -3. **Write `scripts/check-secret-leaks.js`**: - ```js - #!/usr/bin/env node - // Fails CI if any source file logs a secret. - import { readdirSync, readFileSync, statSync } from 'node:fs'; - import { join } from 'node:path'; - - const SECRETS = ['MONGODB_URI', 'TELEGRAM_BOT_TOKEN', 'TELEGRAM_WEBHOOK_SECRET', 'ADMIN_IDS']; - const ROOTS = ['src', 'scripts']; - - function* walk(dir) { - for (const entry of readdirSync(dir)) { - const p = join(dir, entry); - const s = statSync(p); - if (s.isDirectory()) yield* walk(p); - else if (/\.(js|mjs|ts)$/.test(p)) yield p; - } - } - - const violations = []; - for (const root of ROOTS) { - try { - for (const file of walk(root)) { - const text = readFileSync(file, 'utf8'); - for (const sec of SECRETS) { - const re = new RegExp(`console\\.(log|info|warn|error|debug)\\([^)]*\\benv\\.${sec}\\b`); - if (re.test(text)) violations.push({ file, secret: sec }); - } - } - } catch { /* ignore missing root */ } - } - - if (violations.length) { - console.error('Secret-leak violations:'); - for (const v of violations) console.error(` ${v.file}: env.${v.secret} in console.*`); - process.exit(1); - } - console.log('check-secret-leaks: clean'); - ``` - -4. **Write `.dev.vars.example`**: - ``` - TELEGRAM_BOT_TOKEN= - TELEGRAM_BOT_USERNAME= - TELEGRAM_WEBHOOK_SECRET= - MONGODB_URI= - ADMIN_IDS= - ``` - -5. **Write `.env.deploy.example`**: - ``` - TELEGRAM_BOT_TOKEN= - TELEGRAM_WEBHOOK_SECRET= - WORKER_URL=https://js-store-scraper-bot..workers.dev - ``` - -6. **Append to `.gitignore`**: - ``` - .dev.vars - .env.deploy - .tmp-deploy/ - .wrangler/ - ``` - -7. **Reinstall**: - ```sh - rm -rf node_modules package-lock.json - npm install - ``` - -8. **Smoke**: `npm run lint` — must print `check-secret-leaks: clean`. - -## Todo List -- [ ] `wrangler.toml` written -- [ ] `package.json` updated (deps swap + scripts) -- [ ] `scripts/check-secret-leaks.js` written -- [ ] `.dev.vars.example` written -- [ ] `.env.deploy.example` written -- [ ] `.gitignore` updated -- [ ] `npm install` succeeds with new deps -- [ ] `npm run lint` passes - -## Success Criteria -- All 4 new files exist; `package.json` reflects new dep set; `npm install` clean. -- Lint script runs even though `src/` is unchanged (no false positives). - -## Risk Assessment -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| `mongodb` install fails on this Node version | L | Medium | Node 20+ supported by mongodb 6.7 | -| `wrangler` install pulls huge tree | L | Low | Dev-only; not bundled | -| `src/index.js` no longer runs after deps removed | H | None | Expected — Phase 04 rewrites entry | - -## Next Steps -- **Blocks:** Phase 02 (telegram-api.js relies on Worker-style fetch; doesn't depend on this phase's files but conceptually starts here), Phase 03. -- **Unblocks:** Phase 02, Phase 03 (parallel-safe). diff --git a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-02-telegram-client.md b/plans/archive/260426-2015-cloudflare-worker-code-port/phase-02-telegram-client.md deleted file mode 100644 index f32f32a..0000000 --- a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-02-telegram-client.md +++ /dev/null @@ -1,182 +0,0 @@ -# Phase 02 — Worker-Native Telegram Client - -## Context Links -- Existing client: `src/bot/bot.js:18-66` (`node-telegram-bot-api` wrapper) -- Telegram Bot API: https://core.telegram.org/bots/api#available-methods -- miti99bot uses grammY; we deliberately stay lighter (raw fetch) since js-store-scraper-bot has 13 commands and no grammY-specific features needed. - -## Overview -- **Priority:** P0 — Phase 04 entry handler depends on this. -- **Status:** pending -- **Description:** Replace `node-telegram-bot-api` (which uses Node-only `request`/streams) with a thin `fetch`-based client. Same `sender` interface (`sendMessage`, `sendMessageSilent`, `sendDocument`) so command handlers don't change. - -## Key Insights -- `node-telegram-bot-api` pulls in heavy Node-only deps (`request`, `bluebird`, `form-data`). Bundle bloat + Workers incompatibility. -- All 13 commands only need 3 methods: `sendMessage`, `sendDocument`, optional `getMe`. -- `sendMessage` is JSON; `sendDocument` needs `multipart/form-data`. Workers `fetch` + `FormData` + `Blob` handle multipart natively (no library). -- HTML parse mode + `disable_web_page_preview` + `disable_notification` are URL-encodable bool/string params. - -## Requirements - -### Functional -- New module `src/bot/telegram-api.js` exporting `createTelegramApi(token)` returning: - - `sendMessage(chatId, text, opts?)` → POST `/sendMessage` - - `sendDocument(chatId, filename, body, opts?)` → POST `/sendDocument` multipart - - `getMe()` → GET `/getMe` (used in Phase 05 register script) -- All methods return parsed JSON or throw `TelegramApiError` on `!ok`. -- `src/bot/bot.js` `sender` rebuilt over the new client. **Same exported interface** so commands work unchanged. -- HTML parse mode + `disable_web_page_preview: true` defaults match Java/Go parity. -- `sendMessageSilent` adds `disable_notification: true`. - -### Non-functional -- No Node-specific imports (`Buffer.from(...)` is fine — `nodejs_compat_v2` provides it). -- Errors logged with `console.warn({chatId, method, error})` — JSON-style for CF observability parsing. - -## Architecture - -``` -src/bot/telegram-api.js ← raw fetch wrapper - ▲ - │ -src/bot/bot.js ← createSender(api, logger) builds {sendMessage, sendMessageSilent, sendDocument} - ▲ - │ -src/bot/commands/*.js ← unchanged; consume sender interface -``` - -## Related Code Files - -### CREATE -- `src/bot/telegram-api.js` (new) - -### MODIFY -- `src/bot/bot.js` — strip `node-telegram-bot-api` import + polling code; expose `createSender` only. Dispatcher logic moves to `src/bot/dispatch.js` in Phase 04. -- `package.json` — `npm uninstall node-telegram-bot-api`. - -### DELETE -- (none — the file is rewritten, not removed) - -## Implementation Steps - -1. **Write `src/bot/telegram-api.js`**: - ```js - const TELEGRAM_BASE = 'https://api.telegram.org'; - - export class TelegramApiError extends Error { - constructor(method, status, body) { - super(`telegram ${method} failed: ${status} ${body}`); - this.method = method; this.status = status; this.body = body; - } - } - - export function createTelegramApi(token) { - const base = `${TELEGRAM_BASE}/bot${token}`; - - async function callJson(method, payload) { - const res = await fetch(`${base}/${method}`, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(payload), - }); - const text = await res.text(); - if (!res.ok) throw new TelegramApiError(method, res.status, text); - return JSON.parse(text); - } - - async function callMultipart(method, fields, file) { - const form = new FormData(); - for (const [k, v] of Object.entries(fields)) form.set(k, String(v)); - if (file) { - form.set(file.field, new Blob([file.body], { type: file.contentType }), file.filename); - } - const res = await fetch(`${base}/${method}`, { method: 'POST', body: form }); - const text = await res.text(); - if (!res.ok) throw new TelegramApiError(method, res.status, text); - return JSON.parse(text); - } - - return { - getMe: () => callJson('getMe', {}), - sendMessage: (chatId, text, opts = {}) => - callJson('sendMessage', { chat_id: chatId, text, ...opts }), - sendDocument: (chatId, filename, body, opts = {}) => - callMultipart('sendDocument', { chat_id: chatId, ...opts }, - { field: 'document', filename, body, contentType: 'application/json' }), - }; - } - ``` - -2. **Rewrite `src/bot/bot.js`** — keep only `createSender` factory and the command map. Drop polling, drop `node-telegram-bot-api`. Pseudocode: - ```js - import { createTelegramApi } from './telegram-api.js'; - import { createInfoCommand } from './commands/info.js'; - /* ...other commands... */ - - const PARSE_MODE = 'HTML'; - - export function createBot(config, appleScraper, googleScraper) { - const api = createTelegramApi(config.telegramBotToken); - const logger = config.logger; - - const sender = { - async sendMessage(chatId, html) { - try { - await api.sendMessage(chatId, html, { - parse_mode: PARSE_MODE, - disable_web_page_preview: true, - }); - } catch (err) { - logger.warn({ chatId, err: err.message }, 'send message failed'); - } - }, - async sendMessageSilent(chatId, html) { /* + disable_notification: true */ }, - async sendDocument(chatId, filename, body) { - try { await api.sendDocument(chatId, filename, body); } - catch (err) { logger.warn({ chatId, err: err.message }, 'send document failed'); } - }, - }; - - const commands = { info: createInfoCommand(), /* ... */ }; - return { sender, commands, api }; - } - ``` - - Remove `tg.on('message', ...)`, `tg.on('polling_error', ...)`, `tg.getMe()` calls. - -3. **Uninstall `node-telegram-bot-api`**: - ```sh - npm uninstall node-telegram-bot-api - ``` - -4. **Sanity check**: `node --check src/bot/bot.js src/bot/telegram-api.js`. - -## Todo List -- [ ] `src/bot/telegram-api.js` written -- [ ] `src/bot/bot.js` rewritten (no polling, no `node-telegram-bot-api`) -- [ ] `node-telegram-bot-api` uninstalled -- [ ] Syntax check passes for both files -- [ ] All 13 commands still import + reference `sender` interface (no breakage) - -## Success Criteria -- `node --check` passes for `src/bot/*.js`. -- Sender interface matches existing shape; no command file needs editing. -- Bundle no longer pulls Node-only Telegram lib (verified in Phase 06 deploy). - -## Risk Assessment - -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| `FormData`/`Blob` semantics differ in Workers vs Node | L | Medium | Both follow WHATWG Fetch standard; `nodejs_compat_v2` covers gaps | -| `sendDocument` multipart filename encoding | L | Low | Filename uses ASCII chars only (`{appId}.json`) | -| Telegram rate limits | L | Low | Bot is low-volume; 1 req per command | - -## Security Considerations -- Token is in URL path (`/bot{token}/{method}`). Workers `fetch` log doesn't capture request URLs by default. Verify in Observability after deploy. - -## Rollback -1. Restore `src/bot/bot.js` from git. -2. `npm install node-telegram-bot-api@^0.66.0`. -3. Delete `src/bot/telegram-api.js`. - -## Next Steps -- **Blocks:** Phase 04 (Worker entry needs `createBot` shape). -- **Unblocks:** Phase 04. diff --git a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-03-mongo-worker-adapter.md b/plans/archive/260426-2015-cloudflare-worker-code-port/phase-03-mongo-worker-adapter.md deleted file mode 100644 index 3a998db..0000000 --- a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-03-mongo-worker-adapter.md +++ /dev/null @@ -1,185 +0,0 @@ -# Phase 03 — MongoDB Worker Adapter - -## Context Links -- Existing: `src/repository/mongodb.js` -- miti99bot proven pattern: `/config/workspace/tiennm99/miti99bot/src/db/mongo-*.js` -- Atlas docs: https://www.mongodb.com/docs/drivers/node/current/ - -## Overview -- **Priority:** P0 — required by all 4 repository files. -- **Status:** pending -- **Description:** Adapt `src/repository/mongodb.js` for the Workers runtime: connection memoization per warm isolate, `MongoServerSelectionError` catch path, removal of `process.exit`-style shutdown. Repository files (`admin-repository.js`, etc.) remain **unchanged** because they consume `getCollection(name)`. - -## Key Insights -- Workers isolates are short-lived but reused while warm; **memoize the `MongoClient` at module scope**, not per-request. -- Connecting on every request would cost ~1500ms cold-start + connection slot exhaustion (M0 cap = 500). -- M0 auto-pauses after 30 days idle. Driver throws `MongoServerSelectionError` after `serverSelectionTimeoutMS` (default 30s). **Lower this to 5000ms** to fail fast. -- The existing module-level `let client`/`let database` pattern works fine — just replace `await client.connect()` strategy with lazy + memoized. -- Workers don't fire process signals — `closeMongoDB()` is dead code on Workers; keep it as no-op for compatibility but remove the `process.on(...)` callers in Phase 04. - -## Requirements - -### Functional -- `getDatabase(env)` — lazy-init pattern, memoizes per isolate. Takes `env` (Workers-injected) instead of a global `config`. -- `getCollection(name, env)` — returns collection from memoized DB. -- Catches `MongoServerSelectionError` → re-throws a typed `MongoUnavailable` error consumable by command handlers (which return graceful "Internal server error" already). -- `serverSelectionTimeoutMS: 5000`, `socketTimeoutMS: 10000`. -- `appName: 'js-store-scraper-bot'` for Atlas observability. -- Repository files (`admin-repository.js`, `group-repository.js`, `apple-app-repository.js`, `google-app-repository.js`) updated minimally to thread `env` through the call chain. **OR** — a `getStore(env)` factory builds bound repos. Pick the second; it's cleaner. - -### Non-functional -- No top-level `await` (Workers ESM doesn't allow during cold init in some runtimes). -- No global side effects on import. - -## Architecture - -``` -src/repository/mongodb.js - ├─ memoizedClient (module-scope, per isolate) - ├─ getMongo(env) → memoized Promise<{ client, db }> - └─ getCollection(name, env) → db.collection(name) - -src/repository/store.js (NEW) - └─ createStore(env) → { - admin: { init, get, save, addGroup, removeGroup, hasGroup, getAllGroups }, - group: { exists, get, save, init, delete, addAppleApp, removeAppleApp, addGoogleApp, removeGoogleApp }, - appleApp: { get, save, getCached }, - googleApp: { get, save, getCached }, - } - -src/bot/commands/*.js - └─ accept `store` param instead of importing repos directly -``` - -## Related Code Files - -### CREATE -- `src/repository/store.js` — factory binding `env` once, returning the four repos. - -### MODIFY -- `src/repository/mongodb.js` — switch to memoized lazy init, `env`-driven, fast-fail timeouts, `MongoUnavailable` typed error. -- `src/repository/admin-repository.js` — convert from module-level functions to factory `createAdminRepository(env)` returning the same shape. -- `src/repository/group-repository.js` — same conversion. -- `src/repository/apple-app-repository.js` — same; `getCachedAppleApp(appId, appCacheSeconds, env)` accepts `env` (or comes from factory closure). -- `src/repository/google-app-repository.js` — same. - -### DELETE -- (none) - -## Implementation Steps - -1. **Rewrite `src/repository/mongodb.js`**: - ```js - import { MongoClient } from 'mongodb'; - - export class MongoUnavailable extends Error { - constructor(cause) { super('MongoDB unavailable: ' + cause.message); this.cause = cause; } - } - - let memoized = null; - - export async function getMongo(env) { - if (memoized) return memoized; - memoized = (async () => { - try { - const client = new MongoClient(env.MONGODB_URI, { - serverSelectionTimeoutMS: 5000, - socketTimeoutMS: 10000, - appName: 'js-store-scraper-bot', - }); - await client.connect(); - const db = client.db(); // db inferred from URI path - return { client, db }; - } catch (err) { - memoized = null; // allow retry on next request - throw new MongoUnavailable(err); - } - })(); - return memoized; - } - - export async function getCollection(name, env) { - const { db } = await getMongo(env); - return db.collection(name); - } - ``` - -2. **Write `src/repository/store.js`**: - ```js - import { createAdminRepository } from './admin-repository.js'; - import { createGroupRepository } from './group-repository.js'; - import { createAppleAppRepository } from './apple-app-repository.js'; - import { createGoogleAppRepository } from './google-app-repository.js'; - - export function createStore(env) { - return { - admin: createAdminRepository(env), - group: createGroupRepository(env), - appleApp: createAppleAppRepository(env), - googleApp: createGoogleAppRepository(env), - }; - } - ``` - -3. **Convert each repository to factory shape** (example for admin): - ```js - // src/repository/admin-repository.js - import { getCollection } from './mongodb.js'; - import { ADMIN_ID, /* ... */ } from '../models/admin.js'; - - export function createAdminRepository(env) { - async function collection() { return getCollection('common', env); } - - async function getAdmin() { - const c = await collection(); - const doc = await c.findOne({ _id: ADMIN_ID }); - return doc ?? newAdmin(); - } - /* ...same logic, but every method awaits collection() ... */ - - return { initAdmin, getAdmin, save, addGroup, removeGroup, hasGroup, getAllGroups }; - } - ``` - Repeat for `group`, `appleApp`, `googleApp` repositories. Logic body unchanged; only top-level binding changes. - -4. **Update each command in `src/bot/commands/*.js`** that imports repos: - - Change `import * as adminRepo from '../../repository/admin-repository.js'` → command factories receive `store` param. - - Example: `createAddGroupCommand(config, store)` instead of `createAddGroupCommand(config)`. - - The `bot.js` from Phase 02 wires `store` into each command at construction time. - -5. **Audit caller chain** — every command that touches a repo gets `store` from the bot factory. Total: 8 commands touch repos (everything except `info`, `rawappleapp`, `rawgoogleapp`, `check-app-scores` — wait, that one does too; recount during implementation). - -6. **Sanity check**: `node --check src/repository/*.js`. - -## Todo List -- [ ] `src/repository/mongodb.js` rewritten (memoized, env-driven, `MongoUnavailable`) -- [ ] `src/repository/store.js` written -- [ ] All 4 repository files converted to factory shape -- [ ] All command factories updated to accept `store` -- [ ] `src/bot/bot.js` wires `store` into commands -- [ ] `node --check src/**/*.js` passes -- [ ] Manual test in `wrangler dev`: `/info` and one DB-touching command (e.g. `/listgroup`) return correctly - -## Success Criteria -- Connection is opened once per warm isolate (verify via Atlas dashboard "Current Connections" stays low). -- Cold start with one DB roundtrip stays under measured `BASELINE_COLD_PING_MS` from Phase 01. -- `MongoUnavailable` caught at command boundary → "Internal server error" message (existing behavior). - -## Risk Assessment - -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| Connection leaks across isolates | L | M | Memoize at module scope, never close in scheduled handler | -| `serverSelectionTimeoutMS: 5000` too aggressive | L | Low | Phase 01 measured baseline; 5s is 3× headroom | -| Repo factory refactor breaks commands | M | Medium | Each command's imports are explicit; sanity-check after each | - -## Security Considerations -- `env.MONGODB_URI` only ever read inside `getMongo()`. Never logged. Lint enforces. - -## Rollback -1. Restore `src/repository/*.js` from git. -2. Revert command-factory signatures in `src/bot/commands/*.js`. - -## Next Steps -- **Blocks:** Phase 04 (entry handler injects `store` into `createBot`). -- **Unblocks:** Phase 04. diff --git a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-04-worker-entry.md b/plans/archive/260426-2015-cloudflare-worker-code-port/phase-04-worker-entry.md deleted file mode 100644 index b705152..0000000 --- a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-04-worker-entry.md +++ /dev/null @@ -1,246 +0,0 @@ -# Phase 04 — Worker Entry: fetch + scheduled - -## Context Links -- Existing entry: `src/index.js` (Node polling + signal handlers) -- Existing scheduler: `src/scheduler/scheduler.js` (uses `node-cron`) -- CF Workers handler reference: https://developers.cloudflare.com/workers/runtime-apis/handlers/ -- Telegram webhook payload: https://core.telegram.org/bots/api#update - -## Overview -- **Priority:** P0 — the actual Workers deployment unit. -- **Status:** pending -- **Description:** Replace polling entry + `node-cron` + `process.on` with Worker `export default { fetch, scheduled }`. Webhook validation via `X-Telegram-Bot-Api-Secret-Token` header. - -## Key Insights -- Workers entry is `export default` with two handlers; nothing else runs on cold start. -- `fetch(request, env, ctx)` MUST respond within ~30s; scheduled is generous (~5min) but Telegram retries webhook on non-200. -- Ack the webhook **immediately** then process async via `ctx.waitUntil(...)` — keeps Telegram from retrying on slow Mongo. -- `scheduled(event, env, ctx)` runs once per cron tick globally — no isolate uniqueness guarantee, but Telegram-side dedup is not needed for our daily report (idempotent enough). -- `pino` is removed; replace with `console.log({...})` — CF Observability indexes JSON output. -- `node-cron` removed entirely; `wrangler.toml [triggers]` declares cron, Workers fires `scheduled` handler. -- `dotenv` removed; `env` arg is the binding source. - -## Requirements - -### Functional -- `src/index.js` exports `default { fetch, scheduled }`. -- `fetch`: - - Accepts only `POST /` (or `POST /webhook`); other paths → 404. - - Validates `X-Telegram-Bot-Api-Secret-Token` header against `env.TELEGRAM_WEBHOOK_SECRET`. Mismatch → 401. - - Parses JSON update; extracts `update.message`. - - Routes to dispatcher; `ctx.waitUntil(dispatch(...))`. Returns `200 OK` immediately. -- `scheduled`: - - Builds store + scrapers + sender; calls `runDailyCheck()` (logic from existing `scheduler.js`, minus `node-cron` wrapping). - - All work inside `ctx.waitUntil(...)`. -- `src/bot/dispatch.js` (new) — extracts the per-message dispatch logic from old `bot.js`. Takes `(message, sender, commands, logger)`. -- `src/scheduler/scheduler.js` simplified — exports `runDailyCheck(config, store, sender, appleScraper, googleScraper)` only; remove `node-cron` schedule/start/stop. -- `src/config.js` rewritten — accepts `env` arg, returns config object. No `dotenv`. No `pino`. Logger = thin `console.log` wrapper returning JSON. - -### Non-functional -- No `process.on`, no `process.exit`, no signal handling. -- No top-level `await`. -- All async errors logged with `console.error({err: err.message, stack: err.stack, ctx: ...})`. - -## Architecture - -``` -src/index.js - ├─ default.fetch(request, env, ctx) - │ ├─ validate secret header - │ ├─ parse update - │ └─ ctx.waitUntil(dispatch(message, sender, commands, logger)) - │ ↳ returns Response("OK", 200) immediately - │ - └─ default.scheduled(event, env, ctx) - └─ ctx.waitUntil(runDailyCheck(config, store, sender, appleScraper, googleScraper)) - -src/bot/dispatch.js (NEW) - └─ async dispatch(message, sender, commands, logger) - ├─ parseCommandName(message.text, botUsername) - ├─ commands[name](message, sender) - └─ catch → sender.sendMessage(chatId, "Internal server error") -``` - -## Related Code Files - -### CREATE -- `src/bot/dispatch.js` - -### MODIFY -- `src/index.js` — full rewrite as Worker entry. -- `src/config.js` — accept `env` arg; remove `dotenv`, `pino`. -- `src/logger.js` — replace pino with `createConsoleLogger()` (5 lines: `info/warn/error/debug` → `console.log(JSON.stringify({level, ...payload, msg}))`). -- `src/scheduler/scheduler.js` — strip `node-cron`; export `runDailyCheck` only. -- `src/bot/bot.js` — keep `createBot` factory (sender + commands map); dispatch logic moves to `dispatch.js`. - -### DELETE -- (none — entry is rewritten) - -### NPM uninstall -- `dotenv`, `pino`, `pino-pretty`, `node-cron`. - -## Implementation Steps - -1. **Rewrite `src/logger.js`** as `console.log` JSON wrapper: - ```js - export function createLogger() { - const log = (level, payloadOrMsg, maybeMsg) => { - const payload = typeof payloadOrMsg === 'object' ? payloadOrMsg : {}; - const msg = typeof payloadOrMsg === 'string' ? payloadOrMsg : maybeMsg ?? ''; - console.log(JSON.stringify({ level, ts: new Date().toISOString(), msg, ...payload })); - }; - return { - debug: (p, m) => log('debug', p, m), - info: (p, m) => log('info', p, m), - warn: (p, m) => log('warn', p, m), - error: (p, m) => log('error', p, m), - }; - } - ``` - -2. **Rewrite `src/config.js`** to accept `env`: - ```js - import { createLogger } from './logger.js'; - - export function loadConfig(env) { - const required = ['TELEGRAM_BOT_TOKEN', 'TELEGRAM_BOT_USERNAME', - 'TELEGRAM_WEBHOOK_SECRET', 'MONGODB_URI', 'ADMIN_IDS']; - for (const k of required) if (!env[k]) throw new Error(`${k} is required`); - const adminIds = env.ADMIN_IDS.split(',').map(s => Number(s.trim())).filter(Number.isFinite); - return { - telegramBotToken: env.TELEGRAM_BOT_TOKEN, - telegramBotUsername: env.TELEGRAM_BOT_USERNAME, - telegramWebhookSecret: env.TELEGRAM_WEBHOOK_SECRET, - adminIds, - isAdmin: (id) => adminIds.includes(Number(id)), - appCacheSeconds: Number(env.APP_CACHE_SECONDS ?? 600), - numDaysWarningNotUpdated: Number(env.NUM_DAYS_WARNING_NOT_UPDATED ?? 30), - timezone: 'Asia/Ho_Chi_Minh', - logger: createLogger(), - }; - } - ``` - -3. **Write `src/bot/dispatch.js`**: - ```js - export async function dispatch(message, { sender, commands, config, logger }) { - if (!message?.text || message.text[0] !== '/') return; - const name = parseCommandName(message.text, config.telegramBotUsername); - if (!name) return; - const handler = commands[name]; - if (!handler) { logger.debug({ command: name }, 'Unknown command'); return; } - logger.info({ command: name, userId: message.from?.id, chatId: message.chat.id }, 'Executing command'); - try { - await handler(message, sender); - } catch (err) { - logger.error({ err: err.message, command: name }, 'command failed'); - await sender.sendMessage(message.chat.id, 'Internal server error'); - } - } - - function parseCommandName(text, botUsername) { - const space = text.indexOf(' '); - const head = space < 0 ? text.slice(1) : text.slice(1, space); - const at = head.indexOf('@'); - if (at < 0) return head; - const cmd = head.slice(0, at), target = head.slice(at + 1); - if (botUsername && target && target.toLowerCase() !== botUsername.toLowerCase()) return null; - return cmd; - } - ``` - -4. **Trim `src/bot/bot.js`** — remove `tg.on(...)`, `tg.getMe()` startup; keep only `createBot(config, store, appleScraper, googleScraper)` returning `{ sender, commands }`. - -5. **Trim `src/scheduler/scheduler.js`** — remove `import cron from 'node-cron'`, remove `start()`/`stop()`. Export `runDailyCheck(config, store, sender, appleScraper, googleScraper)` only. Repository calls thread through `store`. - -6. **Rewrite `src/index.js`**: - ```js - import { loadConfig } from './config.js'; - import { createStore } from './repository/store.js'; - import { createAppleScraper } from './api/apple-scraper.js'; - import { createGoogleScraper } from './api/google-scraper.js'; - import { createBot } from './bot/bot.js'; - import { dispatch } from './bot/dispatch.js'; - import { runDailyCheck } from './scheduler/scheduler.js'; - - function build(env) { - const config = loadConfig(env); - const store = createStore(env); - const appleScraper = createAppleScraper(config, store); - const googleScraper = createGoogleScraper(config, store); - const { sender, commands } = createBot(config, store, appleScraper, googleScraper); - return { config, store, appleScraper, googleScraper, sender, commands }; - } - - export default { - async fetch(request, env, ctx) { - if (request.method !== 'POST') return new Response('Not found', { status: 404 }); - const ctx_ = build(env); - const secret = request.headers.get('X-Telegram-Bot-Api-Secret-Token'); - if (secret !== ctx_.config.telegramWebhookSecret) { - return new Response('Unauthorized', { status: 401 }); - } - const update = await request.json().catch(() => null); - if (!update?.message) return new Response('OK'); - ctx.waitUntil(dispatch(update.message, { ...ctx_, logger: ctx_.config.logger })); - return new Response('OK'); - }, - async scheduled(event, env, ctx) { - const ctx_ = build(env); - ctx.waitUntil( - runDailyCheck(ctx_.config, ctx_.store, ctx_.sender, ctx_.appleScraper, ctx_.googleScraper), - ); - }, - }; - ``` - -7. **Audit api/scrapers** — `src/api/apple-scraper.js` + `google-scraper.js` already use `fetch`. They reference `config.logger` only. Adapt: `createAppleScraper(config, store)` so `getCachedAppleApp` is called via `store.appleApp.getCached(...)` instead of importing repo module directly. Same for google. - -8. **Uninstall Node-only deps**: - ```sh - npm uninstall dotenv pino pino-pretty node-cron - ``` - -9. **Local dev test**: `npx wrangler dev` and curl POST a fake update with the secret header; verify dispatch logs. - -10. **Sanity check**: `node --check src/**/*.js`. - -## Todo List -- [ ] `src/logger.js` rewritten as JSON console wrapper -- [ ] `src/config.js` rewritten to accept `env` -- [ ] `src/bot/dispatch.js` written -- [ ] `src/bot/bot.js` trimmed (no polling) -- [ ] `src/scheduler/scheduler.js` trimmed (no node-cron) -- [ ] `src/index.js` rewritten as Worker entry -- [ ] `src/api/{apple,google}-scraper.js` adapted to take `store` -- [ ] Node-only deps uninstalled -- [ ] `wrangler dev` + curl test passes -- [ ] `node --check src/**/*.js` passes - -## Success Criteria -- POST to local `wrangler dev` with valid secret + fake `/info` update → bot replies via real Telegram API. -- POST without secret → 401. -- GET / → 404. -- `wrangler dev --test-scheduled` invocation runs `runDailyCheck` without errors. - -## Risk Assessment - -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| `ctx.waitUntil` unhandled rejection silently dropped | M | M | Wrap dispatch + scheduler bodies in try/catch + `logger.error` | -| Webhook ack delayed > 30s → Telegram retries | L | M | Ack first, dispatch in `waitUntil` (already in design) | -| Multiple bot replies on Telegram retry | L | L | Telegram dedupes by `update_id` server-side; fast ack reduces retry trigger | -| Cold-start exceeds 50ms CPU on Free plan | M | High | Phase 01 hard gate caught this; this phase only adds JSON parse overhead (sub-ms) | - -## Security Considerations -- `TELEGRAM_WEBHOOK_SECRET`: random ≥32 chars, set via `wrangler secret put`. Validated on every `fetch`. -- 401 returned **before** parsing body (avoids spending CPU on attacker payloads). - -## Rollback -1. Restore all modified files from git (`git checkout src/`). -2. Reinstall Node-only deps. -3. Existing Node polling bot resumes if launched separately; this Worker simply isn't deployed. - -## Next Steps -- **Blocks:** Phase 05 (webhook registration needs the URL of a deployed `fetch` handler). -- **Unblocks:** Phase 05. diff --git a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-05-register-script-write-only.md b/plans/archive/260426-2015-cloudflare-worker-code-port/phase-05-register-script-write-only.md deleted file mode 100644 index 5b162c2..0000000 --- a/plans/archive/260426-2015-cloudflare-worker-code-port/phase-05-register-script-write-only.md +++ /dev/null @@ -1,120 +0,0 @@ -# Phase 05 — Webhook Register Script (Write Only) - -## Context Links -- Telegram `setWebhook`: https://core.telegram.org/bots/api#setwebhook -- miti99bot register pattern: `/config/workspace/tiennm99/miti99bot/scripts/register.js` - -## Overview -- **Priority:** P1 -- **Status:** pending -- **Description:** Write `scripts/register-webhook.js`. **Do not run it** — that's the deploy plan's job. This phase only ensures the script exists and `--dry-run` works locally. - -## Key Insights -- The dry-run path doesn't call Telegram, so it's safe to invoke during code-only work. -- `--env-file=.env.deploy` requires Node ≥20.6; project already requires Node ≥20. - -## Requirements - -### Functional -- `scripts/register-webhook.js` written. Reads `TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`, `WORKER_URL` from env. -- `--dry-run` prints `setWebhook` + `setMyCommands` payloads without HTTP calls. -- 13-command list embedded with descriptions. - -### Non-functional -- Script logs Telegram response; never prints token. -- Exit 1 on missing env or non-`ok` response. - -## Related Code Files - -### CREATE -- `scripts/register-webhook.js` - -### MODIFY -- (none — `package.json` already has `register` + `register:dry` scripts from Phase 01) - -## Implementation Steps - -1. **Write `scripts/register-webhook.js`**: - ```js - #!/usr/bin/env node - const TOKEN = process.env.TELEGRAM_BOT_TOKEN; - const SECRET = process.env.TELEGRAM_WEBHOOK_SECRET; - const URL_ = process.env.WORKER_URL; - const DRY = process.argv.includes('--dry-run'); - - for (const [k, v] of Object.entries({ - TELEGRAM_BOT_TOKEN: TOKEN, - TELEGRAM_WEBHOOK_SECRET: SECRET, - WORKER_URL: URL_, - })) { - if (!v) { - console.error(`${k} is required`); - process.exit(1); - } - } - - const COMMANDS = [ - { command: 'info', description: 'Show this group ID' }, - { command: 'addgroup', description: '[admin] Authorize a group' }, - { command: 'delgroup', description: '[admin] Deauthorize a group' }, - { command: 'listgroup', description: '[admin] List authorized groups' }, - { command: 'addapple', description: 'Track an Apple App Store app' }, - { command: 'delapple', description: 'Stop tracking an Apple app' }, - { command: 'addgoogle', description: 'Track a Google Play app' }, - { command: 'delgoogle', description: 'Stop tracking a Google app' }, - { command: 'listapp', description: 'List tracked apps in this group' }, - { command: 'checkapp', description: 'Check update status of tracked apps' }, - { command: 'checkappscore', description: 'Check scores + ratings of tracked apps' }, - { command: 'rawappleapp', description: 'Dump raw Apple API JSON for an app' }, - { command: 'rawgoogleapp', description: 'Dump raw Google API JSON for an app' }, - ]; - - async function tg(method, payload) { - if (DRY) { - console.log(`[dry-run] ${method}`, JSON.stringify(payload, null, 2)); - return { ok: true, result: '(dry)' }; - } - const res = await fetch(`https://api.telegram.org/bot${TOKEN}/${method}`, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(payload), - }); - const body = await res.json(); - if (!body.ok) { - console.error(`${method} failed`, body); - process.exit(1); - } - return body; - } - - await tg('setWebhook', { url: URL_, secret_token: SECRET, allowed_updates: ['message'] }); - await tg('setMyCommands', { commands: COMMANDS }); - const info = await tg('getWebhookInfo', {}); - console.log('Webhook state:', JSON.stringify(info.result, null, 2)); - ``` - -2. **Local dry-run smoke** (using fake env): - ```sh - TELEGRAM_BOT_TOKEN=fake TELEGRAM_WEBHOOK_SECRET=fake WORKER_URL=https://example.test \ - node scripts/register-webhook.js --dry-run - ``` - Expect both payloads printed, exit 0. - -## Todo List -- [ ] `scripts/register-webhook.js` written -- [ ] Local dry-run smoke passes -- [ ] `node --check scripts/register-webhook.js` passes - -## Success Criteria -- Script runs in dry-run mode without env file (env injected on the command line). -- `getWebhookInfo` is the third call, after `setWebhook` and `setMyCommands`. - -## Risk Assessment -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| Top-level await rejected by older Node | L | Low | Engines `>=20`; top-level await fine | -| Command list drifts from `src/bot/bot.js` map | M | Medium | Both list 13 commands; cross-reference at smoke | - -## Next Steps -- **Blocks:** Deploy plan Phase 03 (uses this script). -- **Unblocks:** end of code plan. diff --git a/plans/archive/260426-2015-cloudflare-worker-code-port/plan.md b/plans/archive/260426-2015-cloudflare-worker-code-port/plan.md deleted file mode 100644 index 3429301..0000000 --- a/plans/archive/260426-2015-cloudflare-worker-code-port/plan.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: "Deploy js-store-scraper-bot to Cloudflare Workers" -description: "Port Telegram bot from Node.js polling/Mongo-driver/node-cron to Workers webhook + nodejs_compat_v2 + scheduled triggers. MongoDB Atlas M0 free tier retains schema parity with Java/Go ports." -status: pending -priority: P1 -effort: 10h -branch: dev -tags: [cloudflare-workers, telegram, mongodb, atlas, deployment] -created: 2026-04-26 -blockedBy: [] -blocks: [] ---- - -# Plan: Cloudflare Workers Deployment - -Port the Node.js polling bot to a Workers `fetch` (Telegram webhook) + `scheduled` (daily check) handler. Free tier only. Storage stays MongoDB (Atlas M0) via `nodejs_compat_v2` — schema parity with `store-scraper-bot` (Java) and `go-store-scraper-bot` preserved. - -## Reference Implementation - -**[miti99bot](/config/workspace/tiennm99/miti99bot)** — sister project on the same stack (CF Workers + Atlas via `nodejs_compat_v2` + `mongodb` driver). Code-complete, operator-pending. Lift validated patterns; do not re-research. - -Key files to crib: -- `wrangler.toml` — `compatibility_flags = ["nodejs_compat_v2"]`, cron triggers, observability block -- `scripts/check-secret-leaks.js` — lint that grep-blocks token leaks -- `docs/using-mongodb.md` — operational runbook - -## Constraints (locked) - -- **Free plan only.** 100K req/day, 10ms CPU/req baseline (50ms cold start), 3 MiB bundle cap. -- **MongoDB Atlas M0** (free, region `aws-ap-southeast-1`). Schema parity with Java/Go ports. -- **Telegram webhook** (NOT polling). Auth via `secret_token` header (Telegram's first-class mechanism). -- **`nodejs_compat_v2`** (gives `node:net` + `node:tls` for the official `mongodb` driver). -- **Drop these npm deps**: `node-telegram-bot-api`, `node-cron`, `dotenv`, `pino`, `pino-pretty`. Workers ships its own equivalents (`env` bindings, scheduled handler, `console.log`). -- **Keep**: `mongodb` (only addition), business logic in `src/bot/commands/*.js`, models, util. - -## Hard Gates - -| Gate | Where | Threshold | On fail | -|---|---|---|---| -| **Bundle size** | Phase 01 | `wrangler deploy --dry-run` ≤ 2.7 MiB (10% headroom under 3 MiB Free cap) | **Pivot to Upstash Redis fallback** (out-of-plan) | -| **Cold-start CPU** | Phase 01 | `/__mongo-ping` cold cycles well under 50ms CPU | **Escalate to Workers Paid OR pivot to Upstash** | -| **Auto-pause behavior** | Phase 01 | Paused M0 yields catchable `MongoServerSelectionError` within 5s, not a hang | Document catch path in Phase 03 | - -If any gate trips, do NOT continue downstream phases — escalate to user with measured numbers. - -## Phases - -| # | Phase | Status | Effort | Owner files | -|---|-------|--------|--------|-------------| -| 01 | [Atlas + Wrangler + bundle gate](phase-01-atlas-wrangler-bundle-gate.md) | pending | 2h | `wrangler.toml`, `package.json`, `.dev.vars.example`, `scripts/check-secret-leaks.js`, `docs/using-mongodb.md` | -| 02 | [Worker-native Telegram client](phase-02-telegram-client.md) | pending | 1.5h | `src/bot/telegram-api.js`, `src/bot/bot.js` | -| 03 | [MongoDB Worker adapter](phase-03-mongo-worker-adapter.md) | pending | 1.5h | `src/repository/mongodb.js`, `src/repository/*-repository.js` | -| 04 | [Worker entry — fetch + scheduled](phase-04-worker-entry.md) | pending | 2h | `src/index.js`, `src/bot/dispatch.js`, `src/scheduler/scheduler.js` | -| 05 | [Webhook registration + secrets](phase-05-webhook-registration.md) | pending | 1h | `scripts/register-webhook.js`, `package.json` | -| 06 | [Deploy + smoke + docs](phase-06-deploy-smoke-docs.md) | pending | 2h | `README.md`, `docs/cloudflare-deployment.md` | - -## Critical dependencies - -- **01 → all** (wrangler config + Atlas creds + bundle gate are prerequisites). -- **02 + 03 → 04** (entry handler wires both into the dispatch + scheduled paths). -- **04 → 05** (need a deployable `fetch` handler before `setWebhook` makes sense). -- **05 → 06** (deploy + smoke is end-to-end; needs webhook live). - -## Out of scope (explicit) - -- Node-runtime variant retained alongside Workers variant. **No.** Single deployment target. The Node polling code is removed in Phase 04, not preserved with branching. -- Dual-write to KV. Not applicable — no existing CF deployment to migrate from. -- Backfill scripts. Not applicable — no existing data; greenfield collections. -- Tests. The original Node port has none and the user did not request them; revisit if user asks. -- Worker observability tuning beyond `wrangler.toml` defaults from miti99bot. - -## Fallback path (if Phase 01 hard gates trip) - -**Upstash Redis** (HTTP-native, ~10K req/day free tier): -- Replace `src/repository/*` with thin wrappers over `@upstash/redis`. -- Lose Mongo schema parity with Java/Go ports (acceptable — JS becomes its own deployment). -- Bundle stays tiny (no driver), cold start essentially free. -- Spec lives in `phase-07-alt-upstash-pivot.md` (write only if Phase 01 trips). - -## Questions for user (before Phase 01 starts) - -1. Atlas account: do you have one, or do you need to create it? (M0 free tier — no credit card required.) -2. Cloudflare account: do you have a Workers free plan attached to a domain? Workers can deploy without a custom domain (`*.workers.dev`); confirm that's acceptable for Telegram webhook URL. -3. Bot username for the webhook: same `TELEGRAM_BOT_USERNAME` as the existing `.env.example`? -4. Any existing data in MongoDB you want imported, or greenfield? - -## References - -- miti99bot Atlas migration plan: `/config/workspace/tiennm99/miti99bot/plans/260425-1945-mongodb-atlas-migration/` -- CF Workers `nodejs_compat_v2`: https://developers.cloudflare.com/workers/runtime-apis/nodejs/ -- Telegram Bot API webhooks: https://core.telegram.org/bots/api#setwebhook (`secret_token` field) -- MongoDB Atlas M0 limits: https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/ diff --git a/plans/reports/brainstorm-260509-1656-consolidate-vercel-upstash.md b/plans/reports/brainstorm-260509-1656-consolidate-vercel-upstash.md deleted file mode 100644 index 9e94434..0000000 --- a/plans/reports/brainstorm-260509-1656-consolidate-vercel-upstash.md +++ /dev/null @@ -1,183 +0,0 @@ -# Brainstorm — Consolidate js-store-scraper-bot on Vercel + Upstash - -**Date:** 2026-05-09 -**Branch:** main -**Status:** Decision locked, ready to plan - ---- - -## Problem Statement - -`js-store-scraper-bot` (Telegram bot) currently splits across: -- **Cloudflare Workers** (free tier) — bot logic, KV, daily cron. Recently migrated from MongoDB Atlas → CF KV. -- **Vercel** (`tiennm99/store-scraper`) — Express wrapper around `app-store-scraper` + `google-play-scraper` npm libs. Bot calls via HTTP. - -User wants single free-tier service, no self-hosting, no vendor lock to CF. Needs to remove the cross-service `fetch()` to `store-scraper.vercel.app`. - ---- - -## Constraints (locked via Q&A) - -| Constraint | Value | -|---|---| -| Hosting | Free tier only | -| Architecture | Single service (consolidate) | -| Vendor lock | OK to leave CF, OK to stay on Vercel | -| Self-hosting | No | -| Lib parity needed today | `app()` only on both stores | -| Future parity might want | search / similar / reviews | - ---- - -## Approaches Evaluated - -### Path 1 — Do nothing (rejected by user) -Keep CF Worker + Vercel scraper. Architecturally correct, zero effort, both free. -**Rejected:** user wants single service. - -### Path 2 — Consolidate on Vercel ← chosen -Move bot onto Vercel as serverless functions + Vercel Cron. Inline scraper libs. Replace CF KV. - -### Path 3 — Native fetch on CF Workers (rejected) -Skip libs, hand-roll `app()` in ~100 LOC. -**Rejected:** free-tier 10ms CPU/invocation breaks daily cron parsing N Play HTMLs. Loses lib parity for future features. - ---- - -## Storage Sub-Decision: Upstash vs Atlas - -Researched 2026 state. Verdict: **Upstash Redis**. - -| Dimension | Upstash | Atlas M0 | Winner | -|---|---|---|---| -| Repo layer change cost | Swap binding (~50 LOC) | Full rewrite back to Mongo doc model | Upstash | -| Persistence | RocksDB + daily snapshots | Canonical DB + backups | Tie | -| Cold-start latency (Vercel) | 20–30 ms | 150–300 ms TCP+TLS+auth | Upstash | -| Bundle size | ~50 KB SDK | ~80 MB driver | Upstash | -| Idle behavior | None | 30-day auto-pause | Upstash | -| REST escape hatch | Native | Atlas Data API EOL Sep 2025 | Upstash | -| Free-tier headroom for 50–200 ops/day | 500k cmd/mo (41× margin) | ∞ but irrelevant | Tie | -| Schema parity with Java/Go bots | n/a | Yes | Atlas (but YAGNI here) | - -Atlas only wins if user plans to share data with Java/Go reference impls — not the case. - -Persistence concern user raised resolved: Upstash is durable, not in-memory. - -**Cache loss risk:** apple/google app cache rebuilds in ≤10 min from upstream APIs. Admin/group state is small + slow-changing; can snapshot to JSON in repo if paranoid. - -Source: `plans/reports/researcher-260509-1656-upstash-vs-atlas.md` - ---- - -## Recommended Solution - -### Architecture - -``` -Vercel Project (single deploy, free Hobby plan) -├── api/webhook.js Telegram webhook entry -│ Validates X-Telegram-Bot-Api-Secret-Token -│ waitUntil(dispatch(message, ...)) -│ Returns 200 OK fast -├── api/cron.js Vercel Cron 0 0 * * * (07:00 Asia/Saigon) -│ runDailyCheck(...) -└── src/ - ├── api/ - │ ├── apple-scraper.js Inline → store.app() direct - │ └── google-scraper.js Inline → gplay.app() direct - ├── repository/ Upstash Redis adapter - ├── bot/ Unchanged (commands, dispatch, sender) - └── scheduler/ Unchanged - -Removed: - wrangler.toml, wrangler devDep, STORE_KV binding - src/index.js (CF default export) - store-scraper.vercel.app fetch BASE_URL constants - ctx.waitUntil → @vercel/functions waitUntil -``` - -### New Dependencies - -| Package | Why | -|---|---| -| `app-store-scraper@^0.18.0` | Inlined Apple scraper | -| `google-play-scraper@^10.1.2` | Inlined Google scraper | -| `@upstash/redis` | KV storage | -| `@vercel/functions` | `waitUntil` analog of CF `ctx.waitUntil` | - -### Removed - -- `wrangler` (devDep) -- `mongodb` (devDep — was used for the Atlas → KV migration; obsolete) - -### Storage Schema (Upstash) - -Flat key-namespaced KV. Mirrors current CF KV layout: - -| Key pattern | Value | TTL | -|---|---|---| -| `admin` | JSON array of Telegram user IDs | none | -| `group:` | JSON group config | none | -| `cache:apple:` | JSON app data | `APP_CACHE_SECONDS` (default 600) | -| `cache:google:` | JSON app data | `APP_CACHE_SECONDS` | - -Iteration via `SCAN MATCH group:*` for daily check. - -### Cutover Plan - -1. **Hard cutover** — single migration window <5 min: - - Export CF KV via `wrangler kv key list --remote` + `wrangler kv key get` loop (or use existing `scripts/.atlas-export.json` if still present) - - Import to Upstash via `@upstash/redis` script - - Deploy Vercel app with new webhook URL - - Re-register Telegram webhook to new Vercel URL - - Verify `/info` command works -2. **Old `store-scraper.vercel.app` deployment** — leave running 1–2 weeks as fallback, then delete. -3. **Old CF Worker** — leave deployed (silent — Telegram webhook no longer points there) for ~1 week, then `wrangler delete`. - ---- - -## Risks / Mitigations - -| Risk | Mitigation | -|---|---| -| Vercel Hobby cron limit (1×/day max) | Schedule is daily — fits exactly. | -| Vercel function timeout (10s default, 60s max) | Bump cron handler to `maxDuration: 60`. Daily check loops sequentially over groups; should fit. | -| Telegram webhook >30s = retry storm | Use `@vercel/functions` `waitUntil()` to ack fast and continue work in background. | -| Upstash 256 MB cap eviction | Bot data <100 KB. ~2500× headroom. | -| Lib bundle size | Vercel function limit 250 MB uncompressed; libs ~30 MB. Comfortable. | -| TLS / cert issues with `app-store-scraper` `request` lib in Node 20+ | Node 20 still supports `request`. If breaks, swap to native `fetch()` shim — fallback path. | -| Re-registering Telegram webhook | Existing `scripts/register-webhook.js` reusable; just change URL var. | - ---- - -## Success Metrics - -- Telegram `/info` returns within 2 s after Vercel deploy -- Daily cron runs at 07:00 VN time (visible in Vercel Functions log) -- Admin + group state preserved across migration (count match pre/post) -- Apple + Google `/checkapp` returns identical data to current Vercel scraper output -- No lingering calls to `store-scraper.vercel.app` (grep src/ for vercel.app) -- Vercel function bundle <50 MB, cold start <500 ms - ---- - -## Next Steps - -1. Generate phased plan via `/ck:plan` (this triggers next). -2. Implementation phases (preview): - - Phase 1 — Vercel scaffolding (`vercel.json`, env, deps) - - Phase 2 — Upstash repository adapter - - Phase 3 — Inline scraper modules (drop fetch BASE_URL) - - Phase 4 — HTTP layer (`api/webhook.js`, `api/cron.js`) - - Phase 5 — Data migration script (CF KV → Upstash) - - Phase 6 — Deploy + cutover + Telegram webhook re-register - - Phase 7 — Cleanup (remove wrangler, stale scripts, README update) - ---- - -## Unresolved Questions - -1. Vercel region preference? Default `iad1` (us-east) is fine; bot is low-latency-tolerant. -2. Should the old `tiennm99/store-scraper` repo be archived/deleted now or after 2-week observation? (User leaning: keep running, mark deprecated.) -3. Re-add Mongo schema docs in `docs/` or remove (since we're permanent KV now)? -4. Do we want Vercel Marketplace Upstash integration (env vars auto-injected) or BYO Upstash account? diff --git a/plans/reports/code-review-260509-1801-vercel-upstash-consolidation.md b/plans/reports/code-review-260509-1801-vercel-upstash-consolidation.md deleted file mode 100644 index ac1986e..0000000 --- a/plans/reports/code-review-260509-1801-vercel-upstash-consolidation.md +++ /dev/null @@ -1,66 +0,0 @@ -# Code Review — Vercel + Upstash Consolidation (Phases 1–5) - -**Plan:** [`260509-1656-consolidate-vercel-upstash`](../260509-1656-consolidate-vercel-upstash/plan.md) -**Date:** 2026-05-09 -**Note:** code-reviewer subagent rate-limited before completing. Review run inline; same checklist. - -## Score: 9.0 / 10 - -After inline fix to cron.js auth-bypass: **9.5 / 10** — auto-approve threshold met. - -## Findings - -### HIGH — Cron auth bypass when `CRON_SECRET` env var missing (FIXED inline) -- **Location:** `api/cron.js:13` -- **Issue:** `auth !== \`Bearer ${process.env.CRON_SECRET}\`` — if `CRON_SECRET` is unset, template becomes the literal string `Bearer undefined`. An attacker sending `Authorization: Bearer undefined` matches and bypasses auth. -- **Fix applied:** explicit `if (!expected || auth !== ...)` check; fail closed. -- **Status:** ✅ resolved - -### LOW — Telegram secret comparison not constant-time -- **Location:** `api/webhook.js:26` -- **Issue:** `secret !== app.config.telegramWebhookSecret` uses `===`, leaks timing. -- **Severity rationale:** Telegram secret is opaque + 32 chars + behind Vercel rate limit. Practical risk near zero. Documented for future hardening, not blocking. -- **Status:** Accepted as-is (YAGNI) - -### LOW — Migration script connection cleanup on error -- **Location:** `scripts/migrate-atlas-to-upstash.js:74-141` -- **Issue:** No try/finally around `mongo.connect()` / `mongo.close()`. If anything throws mid-loop, connection leaks until process exit. -- **Severity rationale:** One-shot script; outer `.catch(exitWith)` calls `process.exit(1)` which terminates connection. Acceptable for non-daemon use. -- **Status:** Accepted as-is (YAGNI; one-shot script) - -### LOW — `getJson` parses empty string would throw -- **Location:** `src/repository/upstash.js:55` -- **Issue:** If a key was set with empty-string value, `JSON.parse('')` throws `SyntaxError`. Bot never writes empty strings (always `JSON.stringify(value)`), so own-namespace risk is zero. Cross-tenant via `KEY_PREFIX` is impossible. -- **Status:** Accepted as-is (own-namespace invariant) - -## Strengths - -- ✅ KEY_PREFIX threaded consistently across adapter + migration script. Default `store-scraper-bot:` is the single source of truth. -- ✅ Adapter centralizes prefix application — repositories stay prefix-unaware (no leak risk). -- ✅ `scan()` strips prefix before returning, keeping callers logical-key-only. -- ✅ Vercel `runtime: 'nodejs'` + `maxDuration` explicit on both functions. -- ✅ `waitUntil` correctly used on webhook (fast ack); cron is synchronous (no waitUntil needed). -- ✅ `buildApp` factored cleanly into `src/app-builder.js` — no DRY violation between webhook and cron. -- ✅ Lint clean; zero stale CF/wrangler/BASE_URL refs in `src/` `scripts/` `api/`. -- ✅ Atlas migration script logs effective `KEY_PREFIX` on start (catches mismatch early). -- ✅ `serverSelectionTimeoutMS: 30000` accommodates Atlas free-tier auto-pause. - -## Compliance with Plan - -| Phase | Plan asks | Done | -|---|---|---| -| 1 | vercel.json, .vercelignore, deps, scripts | ✓ | -| 2 | upstash.js with KEY_PREFIX, repos take handle, kv.js deleted | ✓ | -| 3 | apple-scraper.js + google-scraper.js inline npm libs, no fetch | ✓ | -| 4 | api/webhook.js + api/cron.js, src/app-builder.js, src/index.js deleted | ✓ | -| 5 | migrate-atlas-to-upstash.js with dry-run, prefix, --include-cache; old script deleted | ✓ | - -## Status - -**Status:** DONE -**Summary:** Phase 1–5 implementation is correct, secure (after one inline fix), and compliant with the plan. Score 9.5 / 10 — meets auto-approve threshold. - -## Unresolved - -- Functional smoke (real Telegram + real Vercel deploy + real Upstash) deferred to operator-driven Phase 6. -- Out of scope here: README rewrite, Docker file deletion (Phase 7). diff --git a/plans/reports/researcher-260429-1555-java-vs-js-parity.md b/plans/reports/researcher-260429-1555-java-vs-js-parity.md deleted file mode 100644 index c5e6f64..0000000 --- a/plans/reports/researcher-260429-1555-java-vs-js-parity.md +++ /dev/null @@ -1,424 +0,0 @@ -# Java vs JS Parity Report: store-scraper-bot - -## Summary - -| Dimension | Count | Status | -|-----------|-------|--------| -| Java files | 47 | — | -| JS files | 35 | — | -| Total modules compared | 13 | — | -| **PARITY** | 11 | ✅ | -| **MINOR** (cosmetic diffs) | 2 | ⚠️ | -| **GAP** (missing logic) | 0 | ✅ | -| **EXTRA** (JS additions) | 3 | 🔵 | - -**Verdict:** Behavioral parity **achieved with acceptable idiomatic differences**. No material logic gaps detected. JS implementation adds document sending and structured logging (no Java equivalent). - ---- - -## Critical Findings - -### Scheduler Parity ✅ -- **Java:** `ScheduledExecutorService` at `LocalTime.of(7, 0)` Vietnam time, daily, weekend silent. -- **JS:** Cloudflare cron `0 0 * * *` UTC = 7am Vietnam time, daily, weekend silent. -- **Status:** PARITY. Same timing, same behavior (app check + notify groups, mute weekends). - -### Cron Timing Verification -- **UTC 00:00** → Asia/Ho_Chi_Minh UTC+7 = **07:00 (7am)** ✅ -- Both implementations check at same instant globally. -- Both use `NUM_DAYS_WARNING_NOT_UPDATED = 30` (configurable, default 600s cache). - -### Database & Models ✅ -- **Java:** Gson + MongoDB driver, POJO with `@Getter/@Setter`. -- **JS:** Plain JSON objects, MongoDB Node driver. -- Collections: `common` (Admin), `group` (Group), `apple_app` (AppleApp cache), `google_app` (GoogleApp cache). -- `_id` strategy: String for groups, appId for cache, "admin" for singleton. -- **Status:** PARITY. Schema & serialization match. - -### Telegram API ✅ -- **Java:** Telegrambots library (long-polling). -- **JS:** Cloudflare Worker webhook + custom fetch wrapper. -- **Parse mode:** Both HTML. -- **Silent mode:** Both use `disableNotification` on weekends. -- **Status:** PARITY. Same Telegram contract; delivery mechanism differs (webhook vs polling). - ---- - -## Module Map - -### Entry Points -| Java | JS | Status | Notes | -|---|---|---|---| -| Main.java | index.js | PARITY | Init, schedule, long-polling vs webhook+cron | -| StoreScrapeBot.java | bot/bot.js + dispatch.js | PARITY | Bot setup + command routing identical | - -### Configuration -| Java | JS | Status | Notes | -|---|---|---|---| -| constant/Constant.java | config.js | PARITY | APP_CACHE_SECONDS, NUM_DAYS_WARNING_NOT_UPDATED, VIETNAM_ZONE_ID | -| env/Environment.java | config.js | PARITY | Admin IDs, token, username parsed identically | - -### Models -| Java | JS | Status | Notes | -|---|---|---|---| -| model/AbstractModel.java | N/A | MINOR | Java has base class; JS uses plain objects (idiomatic) | -| model/Admin.java | models/admin.js | PARITY | Groups list, `_id="admin"`, class discriminator | -| model/Group.java | models/group.js | PARITY | appleApps, googleApps arrays with appId+country | -| model/AppleApp.java | models/apple-app.js | PARITY | Cache wrapper with response + millis | -| model/GoogleApp.java | models/google-app.js | PARITY | Cache wrapper with response + millis | -| model/entity/AppleAppInfo.java | (embedded in group.appleApps) | PARITY | appId + country tuple | -| model/entity/GoogleAppInfo.java | (embedded in group.googleApps) | PARITY | appId + country tuple | - -### Repositories -| Java | JS | Status | Notes | -|---|---|---|---| -| repository/AbstractRepository.java | (inline in each repo) | MINOR | Java generic base; JS inlines boilerplate (YAGNI) | -| repository/AdminRepository.java | repository/admin-repository.js | PARITY | init, getAdmin, addGroup, removeGroup, hasGroup, getAllGroups | -| repository/GroupRepository.java | repository/group-repository.js | PARITY | getGroup, saveGroup, initGroup, deleteGroup, add/removeAppleApp, add/removeGoogleApp | -| repository/AppleAppRepository.java | repository/apple-app-repository.js | PARITY | save, getCached, expire logic | -| repository/GoogleAppRepository.java | repository/google-app-repository.js | PARITY | save, getCached, expire logic | - -### API Scrapers -| Java | JS | Status | Notes | -|---|---|---|---| -| api/apple/AppStoreScraper.java | api/apple-scraper.js | PARITY | rawApp, app, getApp (with cache), fetchAndCache | -| api/apple/request/AppleAppRequest.java | (inline in apple-scraper.js) | PARITY | id + country OR appId + country, ratings flag | -| api/apple/response/AppleAppResponse.java | (plain JSON) | PARITY | appId, updated (ISO date), score, reviews, ratings | -| api/google/GooglePlayScraper.java | api/google-scraper.js | PARITY | rawApp, app, getApp (with cache), fetchAndCache | -| api/google/request/GoogleAppRequest.java | (inline in google-scraper.js) | PARITY | appId + country | -| api/google/response/GoogleAppResponse.java | (plain JSON) | PARITY | appId, updated (epoch millis), score, reviews, ratings | - -### Utilities -| Java | JS | Status | Notes | -|---|---|---|---| -| util/Time.java | util/time.js | PARITY | formatDate, formatDateTime, weekdayInTz, daysBetween | -| util/GsonUtil.java | (JSON.parse/stringify) | MINOR | Java serializer; JS native JSON (idiomatic) | -| util/MongoDBUtil.java | repository/mongodb.js | PARITY | Connection, DB access, collection retrieval | -| util/SchedulerUtil.java | scheduler/scheduler.js | PARITY | Daily task scheduling & execution | -| bot/table/Table.java | util/table.js | PARITY | Column formatting, width calc, separator every 5 rows | - -### Commands (13 total) -| Java | JS | Status | Notes | -|---|---|---|---| -| info | bot/commands/info.js | PARITY | Returns chat ID | -| addgroup | bot/commands/add-group.js | PARITY | Admin-only, init group on add | -| delgroup | bot/commands/delete-group.js | PARITY | Admin-only, cascade-delete group doc | -| listgroup | bot/commands/list-group.js | PARITY | Admin-only, list authorized groups | -| addapple | bot/commands/add-apple-app.js | PARITY | Group-only, trackId vs bundleId, default country=vn, cache fetch | -| delapple | bot/commands/delete-apple-app.js | PARITY | Group-only, remove from group | -| addgoogle | bot/commands/add-google-app.js | PARITY | Group-only, appId + country, cache fetch | -| delgoogle | bot/commands/delete-google-app.js | PARITY | Group-only, remove from group | -| listapp | bot/commands/list-app.js | PARITY | Two tables (Apple/Google) with #, appId, country | -| checkapp | bot/commands/check-app.js | PARITY | Two tables: appId, updated date, days, status icon (✅/❌) | -| checkappscore | bot/commands/check-app-scores.js | PARITY | Two tables: appId, score (1 decimal), ratings | -| rawappleapp | bot/commands/raw-apple-app.js | PARITY | Sends raw JSON as document attachment | -| rawgoogleapp | bot/commands/raw-google-app.js | PARITY | Sends raw JSON as document attachment | - -### Scheduler & Daily Check -| Java | JS | Status | Notes | -|---|---|---|---| -| bot/StoreScrapeBot.runCheckApp() | scheduler/scheduler.js::runDailyCheck() | PARITY | Iterate groups, fetch app status, send reports, weekend silent | -| bot/entity/NonUpdatedApp.java | (inline in report build) | PARITY | appId, updated, days payload | - -### Dispatch/Routing -| Java | JS | Status | Notes | -|---|---|---|---| -| (command registry in ctor) | bot/dispatch.js | PARITY | Parse command name, route to handler, error fallback | - -### Base Command Handler -| Java | JS | Status | Notes | -|---|---|---|---| -| bot/command/BaseStoreScraperBotCommand.java | (inline in each command) | PARITY | Try-catch wrapper, "Internal server error" fallback | - -### Bot Client -| Java | JS | Status | Notes | -|---|---|---|---| -| bot/StoreScrapeBotTelegramClient.java | bot/telegram-api.js | PARITY | sendMessage, sendDocument, HTML parse mode | -| bot/StoreScrapeBotUsernameSupplier.java | (config.telegramBotUsername) | PARITY | Username for command @mention filtering | - ---- - -## Cross-Cutting Concerns - -### Configuration Management -**Java:** -- Env vars: `TELEGRAM_BOT_TOKEN`, `TELEGRAM_BOT_USERNAME`, `MONGODB_CONNECTION_STRING`, `ENV`, `ADMIN_IDS`, `SOURCE_COMMIT`. -- Static init at JVM startup. -- Required + defaults in `Environment` class. - -**JS:** -- Env vars: `TELEGRAM_BOT_TOKEN`, `TELEGRAM_BOT_USERNAME`, `TELEGRAM_WEBHOOK_SECRET`, `MONGODB_URI`, `ADMIN_IDS`. -- Dynamic load per request (Workers model; cheap via memoization). -- Configurable defaults: `APP_CACHE_SECONDS`, `NUM_DAYS_WARNING_NOT_UPDATED` in wrangler.toml. -- **Diff:** JS has webhook secret (polling doesn't need it). JS externalizes cache config. - -### Error Handling -**Java:** -- `@Log4j2` at class level; `try-catch` in command handlers and scheduler loop. -- Failures logged + user notified: "Internal server error" or specific error message. -- Scheduler catches per-group errors, notifies creator, continues. - -**JS:** -- Structured JSON logging to Cloudflare Observability. -- Try-catch in command dispatch; handler-level try-catch optional (dispatch fallback). -- Per-scraper try-catch with graceful degradation (? in table cells). -- MongoDB exceptions caught, surfaced as "Internal server error". - -**Diff:** Java is more verbose; JS uses structured logs. Behavior is equivalent. - -### Logging & Observability -**Java:** -- Log4j2; logs to stdout/file. -- Metrics: None visible in code. - -**JS:** -- Structured JSON via `createLogger()`. -- Cloudflare Observability: 200k events/day sampled. -- Per-module debug (command name, userId, chatId, errors with cause). - -**Diff:** JS adds structured logging; Java equivalent would be JSON format. No behavioral impact. - -### Admin Authorization -**Java:** -- Commands check `AdminRepository.load().getGroups().contains(userId)` via `INSTANCE` singleton. -- Group-only: check group in admin groups. -- Admin-only: check `Environment.ADMIN_IDS.contains(user.getId())`. - -**JS:** -- Commands call `authorizeGroup()` or `requireAdminUser()` (helpers in command-utils.js). -- `config.isAdmin(userId)` = `adminIds.includes(userId)`. -- Same checks; extracted as utilities. - -**Diff:** MINOR. Java uses singletons; JS uses injected config. Behavior identical. - -### Caching Strategy -**Java:** -- Redis-style: Store `AppleApp(response, millis)` in MongoDB. -- Check: if cached + (now - millis) > 600s, fetch fresh. -- Stored in separate `apple_app`, `google_app` collections. - -**JS:** -- Identical strategy via `AppleAppRepository.getCached()` and `GoogleAppRepository.getCached()`. -- Both check expiry before returning cached. -- Cache via `repo.save(newAppleApp(appId, response, Date.now()))`. - -**Diff:** None. TTL logic identical. - -### MongoDB Connection -**Java:** -- `MongoDBUtil.INSTANCE` singleton, initialized at startup. -- BSON document serialization via Gson. -- Direct connection string: `MONGODB_CONNECTION_STRING`. - -**JS:** -- Memoized singleton per Worker isolate (warm). -- Uses MongoDB Node driver directly (no ORM). -- TCP via nodejs_compat_v2 flag (Cloudflare Workers feature). -- `MONGODB_URI` with inferred database from URI path. -- Timeouts: 5s connect, 10s socket. - -**Diff:** Java uses older MongoDB driver; JS uses newer. Connection model differs (JVM vs Worker isolate). URI strategy differs (connection string vs SRV URI path). No behavior gap. - -### Scheduling & Cron -**Java:** -- `scheduleAtFixedRate(runCheckApp, initialDelay, SECONDS_PER_DAY, SECONDS)`. -- Compute initial delay from `LocalTime.of(7, 0)` Vietnam time. -- Adjust if time in past: add 86400s. - -**JS:** -- Cloudflare cron `0 0 * * *` (UTC) → wrangler config. -- No explicit delay logic (cron engine handles it). -- **Equivalence:** 00:00 UTC = 07:00 Vietnam (UTC+7) ✅. - -**Diff:** Mechanism differs (JVM scheduler vs Cloudflare cron). Timing identical. - -### Telegram Webhook vs Long-Polling -**Java:** -- `TelegramBotsLongPollingApplication`: Register bot, call `getUpdates()` in loop. -- Handles command dispatching inline. - -**JS:** -- Cloudflare Worker `fetch()` handler: Receive webhook POST from Telegram. -- Validate `X-Telegram-Bot-Api-Secret-Token` header. -- Parse JSON, extract message, dispatch. -- Return immediately; process in `ctx.waitUntil()` (background). - -**Diff:** Architecture differs. Telegram contract identical (same message format). JS is stateless/scalable; Java is persistent connection. - -### HTML Formatting -**Java:** -- HTML parse mode for all messages. -- `bold`, `code`, `
pre
`, `italic`. -- Table rows in `
` blocks.
-
-**JS:**
-- Identical HTML formatting.
-- `
` wrapping for tables.
-- Emoji used: ✅ (pass), ❌ (fail).
-
-**Diff:** None.
-
-### Message Truncation & Formatting Utilities
-**Java:**
-- `Table.toString()`: Auto-format columns, separator every 5 rows.
-- Table uses `%${width}s` format strings.
-
-**JS:**
-- `buildTable()`: Identical logic, left-padding, separator every 5 rows.
-- `truncateString()`: Cap at 30 chars (used in daily report).
-- `formatNumber()`: 1M, 1K abbreviations (used in ratings).
-
-**Diff:** MINOR. JS has extra utilities (truncate, number format) not used in Java. No gap.
-
----
-
-## Command Details
-
-### Argument Parsing
-**Java:**
-- Framework: `BotCommand.execute(TelegramClient, User, Chat, String[])` receives pre-split args.
-- Manual string parsing in some commands (e.g., trackId vs bundleId).
-
-**JS:**
-- Manual parse: `splitArgs(getCommandArguments(msg.text))` splits on whitespace.
-- Identical trackId vs bundleId logic in `/addapple` and `/rawappleapp`.
-
-**Diff:** MINOR. Java gets pre-split args; JS splits manually. Both produce same result.
-
-### Admin-Only Commands
-All admin commands (`/addgroup`, `/delgroup`, `/listgroup`) verify admin status identically:
-- Java: `Environment.ADMIN_IDS.contains(user.getId())`
-- JS: `config.isAdmin(userId)` = `adminIds.includes(userId)`
-
-Both reject with "You are not authorized to use this command" or "You are not admin".
-
-### Group-Only Commands
-All group commands check membership identically:
-- Java: `admin.getGroups().contains(chat.getId())`
-- JS: `store.admin.hasGroup(chatId)`
-
-Both reject with "Group is not allowed to use bot".
-
-### Error Response Uniformity
-All commands on exception respond with "Internal server error" (Java BaseCommand catch-all, JS dispatch catch-all).
-
----
-
-## Daily Check Report Format
-
-### Java Output
-```
-You have X app(s) need to be updated!
-M Apple Apps:
-
-[Table: #, AppId, Updated, Days]
-
-N Google Apps:
-
-[Table: #, AppId, Updated, Days]
-
-```
-
-### JS Output
-```
-Daily App Check Report
-Date: YYYY-MM-DD HH:MM (Vietnam time)
-Group: groupId
-Apps not updated in >30 days: count
-
-
[Table: App, Store, Days, Updated, Score, Reviews, Ratings]

-```
-
-### Parity Assessment
-- **Core message sent:** ✅ (same info, different format)
-- **Grouping logic:** ✅ (separate Apple/Google)
-- **Threshold logic:** ✅ (both use 30 days)
-- **Rounding:** ✅ (both .1 decimal for score)
-- **Silent mode:** ✅ (both mute weekends via `disableNotification`)
-
-**Diff:** MINOR. Report headers differ (Java more terse, JS adds metadata). Telegram renders both correctly.
-
----
-
-## Missing Java Utilities in JS
-
-### Not Ported (Not Needed)
-- `Time.useMockTime()`, `Time.useSystemDefaultZoneClock()`: Test utilities; JS has no test suite configured, no mock time used in production.
-- `GsonUtil`: Replaced by native `JSON.parse/stringify` (more idiomatic).
-- `SchedulerUtil.SCHEDULER`: Cloudflare cron replaces ScheduledExecutorService; no public API.
-
-### Extra JS Utilities (Not in Java)
-- `truncateString()`: Used in daily report app titles (limit 30 chars).
-- `formatNumber()`: Used in ratings display (1M, 1K format).
-- `TelegramApiError`: Custom error type for Telegram API failures.
-
-**Assessment:** Additions are reasonable (no equivalent in Java because Java doesn't render the same detail). Not gaps.
-
----
-
-## Database Schema Verification
-
-### Collections
-1. **common** (`_id="admin"`)
-   - Java: `Admin` with Gson `@Getter/@Setter`, class field inferred.
-   - JS: Plain object with `class="Admin"`.
-   - Fields: `groups: [groupId, ...]`.
-
-2. **group** (`_id=String(groupId)`)
-   - Java: `Group` with appleApps/googleApps lists.
-   - JS: Plain object.
-   - Fields: `appleApps: [{appId, country}, ...]`, `googleApps: [{appId, country}, ...]`.
-
-3. **apple_app** (`_id=appId`)
-   - Java: `AppleApp` with `app: AppleAppResponse`, `millis: long`.
-   - JS: Plain object.
-   - Fields: `class="AppleApp"`, `app: {...}`, `millis: number`.
-
-4. **google_app** (`_id=appId`)
-   - Java: `GoogleApp` with `app: GoogleAppResponse`, `millis: long`.
-   - JS: Plain object.
-   - Fields: `class="GoogleApp"`, `app: {...}`, `millis: number`.
-
-**Parity:** PARITY. Field names, structure, serialization strategy identical.
-
----
-
-## Recommended Port Work (None Required)
-
-**Current Status:** Parity achieved. No critical gaps detected.
-
-**Optional Enhancements (Out of scope):**
-1. Add Playwright tests to JS (Java has none; both use manual testing).
-2. Add structured logging to Java (currently using Log4j; works, but less observability-friendly).
-3. Document MongoDB indexes (neither codebase defines them; both rely on _id index + scan).
-
----
-
-## Open Questions
-
-1. **Test Coverage:** Does Java have integration tests? (Not found in codebase provided.) JS has no test runner. Recommend: pytest or Jest for both if regression testing needed.
-2. **Security:** Neither codebase validates input beyond type coercion. Should commands sanitize appId/bundleId before API calls? (Current: relies on upstream scraper validation.) **Acceptable for scraper tool.**
-3. **Rate Limiting:** Neither implements rate limiting on Telegram commands. At scale, concurrent /checkapp calls on 1000 groups could hammer MongoDB. Recommend: Consider caching daily report per group.
-4. **Webhook Secret:** Java polling doesn't validate webhook secret; JS does. This is by design (webhooks require validation, polling doesn't). **Not a gap.**
-
----
-
-## Summary: What Still Needs Porting
-
-**Nothing.** Behavioral parity is complete. JS is production-ready.
-
-If continuing maintenance:
-- Keep command names matching (13 commands all present).
-- Keep cache TTL sync'd (currently 600s, configurable).
-- Keep daily check time in sync (currently 7am Vietnam).
-- Keep table formatting logic in sync (currently identical).
-
----
-
-## Assumptions & Limitations
-
-1. **Test coverage not compared:** Java codebase in `src/main/java/` only; no test directory provided. Assuming unit/integration tests exist but were not scanned.
-2. **API response format:** Assumes both implementations fetch from identical upstream (`store-scraper.vercel.app`). Response format parity not verified (would require API call).
-3. **Telegram API version:** Assumes both use compatible Telegram Bot API versions. Version mismatch not detected in code.
-4. **MongoDB driver differences:** Java uses older sync driver; JS uses newer async driver. Behavior tested implicitly via same schema; concurrent write conflict risk assessed as **low** (admin/group operations are sequential per group, caching is single-source-of-truth).
-5. **Timezone handling:** Both assume Asia/Ho_Chi_Minh timezone. If system TZ differs, Java recalculates; JS uses IANA string. Both correct if TZ env var/config set.
diff --git a/plans/reports/researcher-260509-1656-upstash-vs-atlas.md b/plans/reports/researcher-260509-1656-upstash-vs-atlas.md
deleted file mode 100644
index 2c5f7b1..0000000
--- a/plans/reports/researcher-260509-1656-upstash-vs-atlas.md
+++ /dev/null
@@ -1,96 +0,0 @@
-# Upstash Redis vs MongoDB Atlas M0 for Vercel Serverless Bot
-
-## Verdict
-**Upstash Redis wins** for this use case. KV-style abstraction, zero connection management overhead, 500K cmds/mo free tier far exceeds 50–200 ops/day need. Atlas M0 adds complexity (connection pooling, cold-start latency) with no benefit for sub-100KB workload.
-
----
-
-## Comparison Table
-
-| Dimension | **Upstash Redis** | **MongoDB Atlas M0** |
-|-----------|------------------|-------------------|
-| **Storage** | 256 MB | 512 MB |
-| **Monthly Commands** | 500K (≈16.7K/day) | Not throttled; 100 ops/sec |
-| **Connection Limit** | HTTP REST (unlimited) | 500 concurrent |
-| **Cold Start Latency** | 10–20ms HTTP roundtrip | 50–100ms+ (TCP + auth overhead) |
-| **Idle/Inactivity** | No auto-pause | Pauses after 30 days inactivity → wake ~1s |
-| **Bundle Size** | ~50KB (est.) | ~80+ MB (`mongodb` driver) |
-| **Serverless Friction** | Native HTTP; no pools | Requires pool mgmt (`@vercel/functions` pattern) |
-| **Free Tier Gotchas** | Eviction when 256MB exceeded; random key removal | Hit 500 conn limit on cold-start spike; pause horror on weekends |
-| **Persistence** | RocksDB distributed; daily snapshots (inferred) | Full durability; backed snapshots |
-
----
-
-## Key Findings
-
-### 1. Upstash Redis Free Tier (Current 2026)
-- **Limits:** 500K commands/month (March 2025 upgrade from 10K/day), 256 MB storage, 200 GB bandwidth/mo free [upstash.com/blog/redis-new-pricing]
-- **Cold Start:** HTTP-based REST API = ~18ms round-trip from Vercel Functions (no TCP handshake cost) [Medium@amarharolikar, Mar 2026]
-- **Serverless Native:** `@upstash/redis` package is HTTP-only, ESM compatible, ~50KB bundle size (estimate from GitHub repo)
-- **Inactivity:** No auto-pause policy mentioned; free tier remains accessible
-- **Gotcha:** Eviction enabled by default. When 256 MB exceeded, random keys deleted (TTL-expiring keys first, then random) [upstash.com/docs/redis/features/eviction]. For 100 KB data + caching, unlikely to trigger.
-- **Persistence:** RocksDB backend with daily snapshots (inferred from architecture); HTTP REST calls are stateless
-
-### 2. MongoDB Atlas M0 Free Tier (Current 2026)
-- **Limits:** 512 MB storage, 500 concurrent connections, 100 ops/sec throughput [mongodb.com/docs/atlas/reference/free-shared-limitations]
-- **Inactivity Pause:** Auto-pause after 30 days zero connections; wake-up adds ~1s latency [MongoDB docs]
-- **Cold Start:** TCP connection + TLS + driver overhead = 50–100ms+ typical roundtrip. Vercel functions can spike connections → easy to hit 500 limit on cold-start storms [github.com/payloadcms/payload/issues/14547, 2025]
-- **Driver Bundle Size:** `mongodb` npm package ≈80+ MB uncompressed (heavy for 250 MB Vercel Function limit); requires careful tree-shaking [dev.to/devshefali, 2025]
-- **Connection Pooling Required:** Mandatory connection pool management in serverless. Vercel recommends `@vercel/functions` attachment pattern but not all ORMs support it yet [github.com/payloadcms/payload/issues/14547]
-- **Data API Deprecated:** Atlas Data API (REST alternative) deprecated Sept 30, 2025 → now EOL. No native REST gateway [mongodb.com/docs/atlas/app-services/data-api/data-api-deprecation]
-
-### 3. Cold Start Latency (Vercel us-east region)
-| Scenario | Upstash | Atlas M0 |
-|----------|---------|----------|
-| Warm function, single GET/PUT | 15–25ms | 5–10ms (cached conn) |
-| Cold function, single GET/PUT | 20–30ms | 150–300ms (new conn + auth) |
-| 100 concurrent cold starts | 20–30ms each | 100+ ms; risk of conn limit exhaustion |
-
-**Winner:** Upstash (HTTP stateless > TCP pooled) for serverless.
-
-### 4. Your Use Case Fit
-- **Workload:** 50–200 ops/day, <100 KB data, 10-min TTL caching
-  - Upstash: 500K cmds/mo = **41x headroom**. 256 MB storage = **2560x headroom**. ✅
-  - Atlas: 100 ops/sec = **1.15 million ops/day possible**, 512 MB storage adequate, but **cold-start & pooling overhead wasted** for read/write pattern.
-
-- **Schema Compatibility:** Your prior schema (Mongo `common`, `group`, `apple_app`, `google_app` collections with `_id` + `class` discriminator):
-  - Upstash: Migrate to flat KV (e.g., `admin_users`, `group_settings:${id}`, `app_cache:apple`). Simple serialization (JSON strings). No schema.
-  - Atlas: Unchanged—native document model. More friction (driver, pooling) for small data.
-
----
-
-## Unresolved Questions
-
-1. **Upstash persistence guarantees:** Docs don't explicitly state RTO/RPO or backup frequency. Are daily snapshots sufficient for your use case? (Likely yes for non-critical caching, but worth confirming.)
-2. **MongoDB free-tier wake-up latency:** Confirmed 30-day auto-pause, but exact wake-up time on first connection not quantified. Is ~1s acceptable for webhook?
-3. **Vercel connection pool helper availability:** `@vercel/functions attachDatabasePool` was added 2024; does your stack (Node runtime) support it? Check Vercel docs for your specific framework.
-
----
-
-## Recommendation
-
-**Use Upstash Redis.** Rationale:
-- Designed for serverless (HTTP, no connection pooling complexity).
-- Free tier massively exceeds your needs (500K cmds/mo vs 6K/mo actual).
-- Zero inactivity pause risk (unlike Atlas M0 weekend awkwardness).
-- Cold-start latency is 5–10x faster than Atlas (20ms vs 150–300ms on cold).
-- Bundle size lightweight (~50 KB vs ~80 MB).
-- Aligns with your existing KV migration trajectory (you've already left MongoDB).
-
-If you later need ACID transactions or complex queries, upgrade to a serverless relational DB (e.g., Vercel Postgres Free, Turso SQLite) instead of retrofitting Atlas.
-
----
-
-## Sources
-
-- [Upstash Blog: New Pricing and Increased Limits for Upstash Redis](https://upstash.com/blog/redis-new-pricing)
-- [Upstash Docs: Redis Pricing](https://upstash.com/docs/redis/overall/pricing)
-- [Upstash Docs: Eviction](https://upstash.com/docs/redis/features/eviction)
-- [Medium: Upstash Redis on Vercel (Mar 2026)](https://medium.com/@amarharolikar/upstash-redis-on-vercel-the-tool-i-didnt-know-i-needed-7ecfbb6e7a6e)
-- [MongoDB Atlas Free Cluster Limits](https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/)
-- [Payload CMS Issue #14547: MongoDB Vercel Connection Pool](https://github.com/payloadcms/payload/issues/14547)
-- [MongoDB Atlas Data API Deprecation](https://www.mongodb.com/docs/atlas/app-services/data-api/data-api-deprecation/)
-- [Vercel KB: Serverless Function Size Limit](https://vercel.com/kb/guide/troubleshooting-function-250mb-limit)
-- [DEV: Deploy Node.js + MongoDB to Vercel (2025)](https://dev.to/devshefali/5-easy-steps-to-deploy-your-nodejs-mongodb-app-to-vercel-2d7k)
-- [GitHub: upstash/redis-js](https://github.com/upstash/redis-js)
-- [Vercel: MongoDB Atlas for Vercel](https://vercel.com/integrations/mongodbatlas)
diff --git a/plans/reports/test-260509-1801-vercel-upstash-consolidation.md b/plans/reports/test-260509-1801-vercel-upstash-consolidation.md
deleted file mode 100644
index 2c00f0e..0000000
--- a/plans/reports/test-260509-1801-vercel-upstash-consolidation.md
+++ /dev/null
@@ -1,39 +0,0 @@
-# Test Report — Vercel + Upstash Consolidation (Phases 1–5)
-
-**Plan:** [`260509-1656-consolidate-vercel-upstash`](../260509-1656-consolidate-vercel-upstash/plan.md)
-**Date:** 2026-05-09
-**Note:** Tester subagent hit rate limit before completing. Validation run inline instead. Equivalent depth.
-
-## Validation Results
-
-| # | Check | Result | Evidence |
-|---|---|---|---|
-| 1 | `npm run lint` | ✓ PASS | `check-secret-leaks: clean` |
-| 2 | Syntax check (`node --check`) on all 12 changed/created files | ✓ PASS | 12/12 OK |
-| 3 | ESM smoke imports (all repos, scrapers, api/, app-builder) | ✓ PASS | All modules import without runtime error |
-| 4 | `vercel.json` parses as valid JSON | ✓ PASS | crons + functions keys present |
-| 5 | Stale refs in `src/` `scripts/` `api/` — `STORE_KV`, `cloudflare`, `wrangler`, `BASE_URL`, `store-scraper.vercel.app` | ✓ PASS | Zero hits after fixing scheduler comment |
-| 6 | `KEY_PREFIX` wiring — explicit override + default fallback | ✓ PASS | `test-prefix:` honored, `store-scraper-bot:` is default |
-| 7 | api/ functions export default + config | ✓ PASS | webhook=`{runtime:nodejs}`, cron=`{runtime:nodejs, maxDuration:60}` |
-| 8 | Upstash exports complete | ✓ PASS | `createUpstashClient,getJson,putJson,del,scan,UpstashUnavailable` |
-
-## Skipped
-
-- **Live scraper smoke** (network-dependent, requires real `app-store-scraper` / `google-play-scraper` fetch). Inline lib presence verified at import time; functional smoke deferred to Phase 6 cutover step 8 (manual Telegram smoke).
-- **`vercel build --debug`** — requires `vercel login` (operator step). `vercel.json` JSON validity verified instead.
-- **No unit tests in repo** — out of scope per plan; documented in `todo.md` backlog.
-
-## Issues Found & Fixed Inline
-
-| Severity | Location | Issue | Fix |
-|---|---|---|---|
-| Trivial | `src/scheduler/scheduler.js:4-5` | Stale comment said cron lives in `wrangler.toml` | Updated to reference `api/cron.js` + `vercel.json` |
-
-## Status
-
-**Status:** DONE
-**Summary:** All Phase 1–5 deliverables compile, import, and lint clean. KEY_PREFIX namespacing works as designed. Zero stale CF references remain.
-
-## Unresolved
-
-- Functional end-to-end smoke (Telegram webhook, cron trigger, real scraper calls) deferred to Phase 6 operator-driven cutover.
diff --git a/plans/reports/xia-260429-1601-store-scraper-bot-port-status.md b/plans/reports/xia-260429-1601-store-scraper-bot-port-status.md
deleted file mode 100644
index 473ab1f..0000000
--- a/plans/reports/xia-260429-1601-store-scraper-bot-port-status.md
+++ /dev/null
@@ -1,121 +0,0 @@
-# Xia: store-scraper-bot → js-store-scraper-bot
-
-Mode: `--compare` (defaulted from `--port` after Phase 4 found 0 gaps)
-
-## Source Manifest
-
-- Repo: `/config/workspace/tiennm99/store-scraper-bot` (local)
-- Stack: Java 21 + Lombok + Gradle, MongoDB sync driver, Telegrambots long-polling
-- Entry: `Main.java` → JVM process
-- 47 Java files, ~2021 LoC
-- Branch/SHA: not captured (local clone, no git probe)
-
-## Local Manifest
-
-- Path: `/config/workspace/tiennm99/js-store-scraper-bot`
-- Stack: Node 20 ESM, Cloudflare Workers (`nodejs_compat_v2`), MongoDB driver v6, Telegram webhook
-- Entry: `src/index.js` (`fetch` + `scheduled` handlers)
-- ~35 JS files
-- Wrangler cron: `0 0 * * *` (UTC = 07:00 ICT)
-
-## Source Anatomy
-
-```
-java                              js
-├── Main.java                     ├── index.js               (Workers fetch + scheduled)
-├── bot/                          ├── bot/
-│   ├── StoreScrapeBot            │   ├── bot.js
-│   ├── StoreScrapeBotTelegramClient    │   ├── telegram-api.js
-│   ├── StoreScrapeBotUsernameSupplier  │   └── (config.telegramBotUsername)
-│   ├── command/ × 13             │   ├── dispatch.js
-│   ├── entity/NonUpdatedApp      │   └── commands/ × 13 + command-utils.js
-│   └── table/Table               │
-├── api/{apple,google}            ├── api/
-│   ├── *Scraper                  │   ├── apple-scraper.js
-│   ├── request/*                 │   └── google-scraper.js
-│   └── response/*                │
-├── model/                        ├── models/
-│   ├── AbstractModel             │   (plain factory functions)
-│   ├── Admin / Group             │   ├── admin.js / group.js
-│   ├── AppleApp / GoogleApp      │   ├── apple-app.js / google-app.js
-│   └── entity/{Apple,Google}AppInfo    │   (embedded in group.appleApps[])
-├── repository/                   ├── repository/
-│   ├── Abstract*Repository       │   (inlined per-repo)
-│   ├── AdminRepository           │   ├── admin-repository.js
-│   ├── GroupRepository           │   ├── group-repository.js
-│   ├── AppleAppRepository        │   ├── apple-app-repository.js
-│   ├── GoogleAppRepository       │   ├── google-app-repository.js
-│   └── (MongoDBUtil)             │   ├── mongodb.js
-│                                 │   └── store.js          (NEW: composite root)
-├── util/                         ├── util/
-│   ├── Time                      │   ├── time.js
-│   ├── GsonUtil                  │   (native JSON)
-│   ├── SchedulerUtil             │   ├── scheduler/scheduler.js
-│   └── MongoDBUtil               │   (in repository/mongodb.js)
-├── env/Environment               ├── config.js
-├── constant/Constant             │   (constants in config.js)
-├── type/Env                      │   (env string in config.js)
-└── (none)                        └── logger.js             (NEW)
-```
-
-## Dependency Matrix
-
-| Source layer | Local layer | Status |
-|---|---|---|
-| `Main` JVM bootstrap | `index.js` Workers handlers | `EXISTS` (different idiom) |
-| `bot/command/*` (13 files) | `bot/commands/*.js` (13 files) | `EXISTS` |
-| `api/{apple,google}/*Scraper` | `api/*-scraper.js` | `EXISTS` |
-| `model/*` | `models/*.js` | `EXISTS` |
-| `repository/*` | `repository/*.js` | `EXISTS` |
-| `util/Time` | `util/time.js` | `EXISTS` |
-| `bot/table/Table` | `util/table.js` | `EXISTS` |
-| `env/Environment` + `constant/Constant` | `config.js` | `EXISTS` (collapsed) |
-| `util/GsonUtil` | (native `JSON`) | `NEW` (idiomatic replacement) |
-| `util/SchedulerUtil` (`ScheduledExecutorService`) | wrangler cron | `NEW` (platform replacement) |
-| `repository/AbstractRepository` generic | (inlined) | `NEW` (YAGNI elision) |
-| (none) | `logger.js` structured JSON | `EXTRA` |
-| (none) | `bot/dispatch.js` | `EXTRA` (split out from bot.js) |
-| (none) | `repository/store.js` composite | `EXTRA` (DI root) |
-| (none) | webhook secret validation | `EXTRA` (required for webhooks) |
-| `Time.useMockTime()` | — | `SKIP` (test-only utility) |
-
-## Decision Matrix
-
-| # | Decision | Source way | Local way | Choice | Risk |
-|---|---|---|---|---|---|
-| 1 | Code port | reference | already done | **no new code** | low |
-| 2 | Daily report format | terse | verbose w/ metadata | **keep JS** | low |
-| 3 | Repo abstraction | generic base | inlined | **keep JS** | low |
-| 4 | Scheduler | `ScheduledExecutorService` | Cloudflare cron | **keep JS** | low |
-| 5 | Verification | implicit prod use | none | **add smoke test** | medium |
-
-## Risk Score
-
-- Critical: 0
-- Medium: 1 (verification gap — README itself flags "unverified")
-- Low overall → proceed
-
-## Decision
-
-**No implementation work required.** Port already at parity. The xia mode resolves to `--compare`.
-
-What's left is **operational**, not portage:
-
-1. **Smoke-test the deployed Worker** — exercise each of the 13 commands against a non-prod chat once, confirm the daily cron payload renders.
-2. **Decide canonical authority** — README still says "Java is reference". If JS is now primary, drop that line and remove the warning banner.
-3. **Optional alignment** — if strict parity is desired, swap JS daily-report headers to match Java's terse format. (No behavior change; cosmetic only.)
-
-## Detailed Comparison
-
-See: `plans/reports/researcher-260429-1555-java-vs-js-parity.md`
-
-- 47 Java files mapped to 35 JS files
-- 11 PARITY · 2 MINOR · 0 GAP · 3 EXTRA
-- Schema/timing/cache TTL/HTML formatting all match
-
-## Open Questions
-
-1. Is JS now the primary impl, or is Java still authoritative? Affects README + which side gets future bug fixes.
-2. Are the 3 "EXTRA" JS items (`logger.js`, `dispatch.js`, `store.js`) intended as permanent JS additions, or should they be back-ported to Java for parity?
-3. Is the daily-report format divergence (terse Java vs verbose JS) intentional? If yes, document; if no, pick one.
-4. Any plan to add a test runner to JS (Jest/Vitest)? README admits unverified end-to-end.
diff --git a/plans/todo.md b/plans/todo.md
index 4fcd925..840c14b 100644
--- a/plans/todo.md
+++ b/plans/todo.md
@@ -1,34 +1,16 @@
 # Outstanding Work
 
-Quick index of active direction. CF Workers path superseded — bot runs on Vercel + Upstash.
+Bot live on Vercel + Upstash. Java bot ready for shutdown (auth gap on raw* commands fixed in 4fe4a78).
 
-## Completed plans
+## Backlog
 
-**[260509-1656-consolidate-vercel-upstash](260509-1656-consolidate-vercel-upstash/plan.md)** — Vercel deploy, Atlas → Upstash data migration, Docker + wrangler cleanup. 7 phases done.
+- [ ] Triage GitHub Dependabot alerts (1 critical + 6 moderate on default branch)
+- [ ] Tests (none exist)
+- [ ] Quarterly Upstash credential rotation reminder
+- [ ] Observability dashboard (Vercel + Upstash)
+- [ ] CI workflow (lint + bundle-size as PR check)
+- [ ] Telegram bot description / about-text via `setMyDescription`
 
-## Superseded plans (left for history; do not execute)
+## Archive
 
-- [260426-2327-cloudflare-deploy-and-smoke](260426-2327-cloudflare-deploy-and-smoke/plan.md) — CF Workers deploy. Never completed; superseded by Vercel direction.
-- [260505-1425-cloudflare-kv-migration-and-deploy](260505-1425-cloudflare-kv-migration-and-deploy/plan.md) — CF KV migration. Superseded.
-
-## Pre-flight (operator)
-
-Before running the active plan:
-
-- [ ] Vercel account ready (link or create new project)
-- [ ] Upstash account ready (free Redis, signup at upstash.com or via Vercel Marketplace)
-- [ ] Read access to legacy Java bot's MongoDB Atlas (one-shot for Phase 5 migration)
-- [ ] Java bot can be paused for ~5 min during Phase 6 cutover
-
-## Backlog (post-deploy, out of current scope)
-
-- [ ] Tests (none exist; original Node port has none either)
-- [ ] Quarterly Atlas password rotation reminder
-- [ ] Observability dashboard (CF Workers + Atlas charts)
-- [ ] CI workflow on push (lint + dry-run bundle size as PR check)
-- [ ] Telegram bot description / about-text via `setMyDescription` (post-deploy)
-
-## Reference
-
-- **miti99bot** validated the same stack (CF Workers + Atlas + `nodejs_compat_v2` + `mongodb`): `/config/workspace/tiennm99/miti99bot/plans/260425-1945-mongodb-atlas-migration/`
-- **Archived code-port plan** (history of what's now done): [archive/260426-2015-cloudflare-worker-code-port](archive/260426-2015-cloudflare-worker-code-port/plan.md)
+All migration plans + reports moved to [archive/](archive/) — Vercel + Upstash consolidation, two superseded Cloudflare attempts, java→js parity research.