mirror of
https://github.com/tiennm99/store-scraper-bot.git
synced 2026-10-14 08:18:36 +00:00
docs: add KV migration plan, supersede Atlas deploy plan
Adds the 260505-1425 plan that drove the KV swap (Phase 01 already applied in the preceding commit; Phases 02-05 are operator-driven). Marks 260426-2327-cloudflare-deploy-and-smoke as superseded since the KV pivot was taken pre-emptively rather than after a hard-gate trip.
This commit is contained in:
tiennm99 committed
1 parent
9898f408a2
commit
971c9c311f
7 files changed
+739
-1
No files matched your search
@@ -7,10 +7,13 @@ effort: 4h
|
||||
branch: main
|
||||
tags: [cloudflare-workers, deploy, smoke, atlas]
|
||||
created: 2026-04-26
|
||||
blockedBy: [260426-2015-cloudflare-worker-code-port]
|
||||
blockedBy: [260426-2015-cloudflare-worker-code-port, 260505-1425-cloudflare-kv-migration-and-deploy]
|
||||
blocks: []
|
||||
supersededBy: 260505-1425-cloudflare-kv-migration-and-deploy
|
||||
---
|
||||
|
||||
> **Superseded** — User chose Cloudflare KV over Atlas pre-emptively, before the hard-gate trip. Deploy work is now tracked in [`260505-1425-cloudflare-kv-migration-and-deploy`](../260505-1425-cloudflare-kv-migration-and-deploy/plan.md). This file is kept for history.
|
||||
|
||||
# Plan B — Deploy + Smoke
|
||||
|
||||
Run only after [the code-port plan](../260426-2015-cloudflare-worker-code-port/plan.md) is complete and committed. This plan touches **real accounts**: Atlas (free M0) and Cloudflare (free Workers).
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Replace Mongo with KV in repositories"
|
||||
status: completed
|
||||
priority: P1
|
||||
effort: "1.5h"
|
||||
dependencies: []
|
||||
---
|
||||
|
||||
# Phase 01: Replace Mongo with KV in repositories
|
||||
|
||||
## Overview
|
||||
|
||||
Swap the four `*-repository.js` modules from `mongodb` collection ops to `env.STORE_KV.get/put/delete`. Drop the `mongodb` driver dependency. Keep document shape and Java parity intact.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- All 13 Telegram commands continue to work without behavior change.
|
||||
- Daily cron continues to enumerate groups and report stale apps.
|
||||
- Cached app entries expire after `APP_CACHE_SECONDS`.
|
||||
|
||||
**Non-functional**
|
||||
- Worker bundle ≤ 500 KiB.
|
||||
- No `nodejs_compat_v2` flag required.
|
||||
- `npm install` no longer pulls `mongodb`.
|
||||
|
||||
## Architecture
|
||||
|
||||
KV access pattern (single namespace `STORE_KV`, prefixed keys):
|
||||
|
||||
```
|
||||
admin → JSON of {class:"Admin", groups:[...]}
|
||||
group:{chatId} → JSON of {class:"Group", _id, appleApps[], googleApps[]}
|
||||
apple:{appId} → JSON of cached Apple response (KV TTL = APP_CACHE_SECONDS)
|
||||
google:{appId} → JSON of cached Google response (KV TTL = APP_CACHE_SECONDS)
|
||||
```
|
||||
|
||||
Read pattern: `await env.STORE_KV.get(key, 'json')` returns parsed JSON or `null`.
|
||||
Write pattern: `await env.STORE_KV.put(key, JSON.stringify(value), opts?)`.
|
||||
Delete pattern: `await env.STORE_KV.delete(key)`.
|
||||
|
||||
Cache TTL: pass `{ expirationTtl: appCacheSeconds }` to `put` for `apple:` and `google:` keys. **KV minimum `expirationTtl` is 60s** — Java/Mongo had no such floor. Guard in `kv.js`: clamp `expirationTtl` to `Math.max(60, value)` so an aggressive `APP_CACHE_SECONDS` override doesn't make `put` reject. Document the floor in `.dev.vars.example`.
|
||||
|
||||
Drop the `isAppleAppExpired` / `isGoogleAppExpired` helpers from the read path — KV deletes expired keys, so a `get` returning `null` is the cache miss. Keep the helpers if other code references them; otherwise delete.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
**Replace (rewrite contents)**
|
||||
- `src/repository/admin-repository.js` — KV ops on key `admin`
|
||||
- `src/repository/group-repository.js` — KV ops on key `group:{chatId}`
|
||||
- `src/repository/apple-app-repository.js` — KV ops on key `apple:{appId}` with TTL
|
||||
- `src/repository/google-app-repository.js` — KV ops on key `google:{appId}` with TTL
|
||||
|
||||
**Delete**
|
||||
- `src/repository/mongodb.js` — no longer needed
|
||||
|
||||
**Modify**
|
||||
- `src/repository/store.js` — remove memoization comment about `MongoClient`; factory signature unchanged
|
||||
- `src/index.js` — drop the "memoized MongoClient" comment block; nothing else changes (the `env` already flows through)
|
||||
- `src/config.js` — remove `MONGODB_URI` from required env list
|
||||
- `package.json` — `npm uninstall mongodb`; bump version to 0.3.0
|
||||
|
||||
**Untouched (verify)**
|
||||
- `src/models/*.js` — document shape preserved verbatim (Java parity at the doc level: `_id`, `class`, all fields)
|
||||
- `src/bot/commands/*.js` — repository interfaces unchanged
|
||||
- `src/scheduler/scheduler.js` — calls `store.admin.getAllGroups()` and `store.group.getGroup()`; both still resolve
|
||||
|
||||
## Java parity preserved (per existing parity reports)
|
||||
|
||||
Cross-checked against `plans/reports/xia-260429-1601-store-scraper-bot-port-status.md` and `researcher-260429-1555-java-vs-js-parity.md`. KV swap preserves every PARITY item:
|
||||
|
||||
- Schema (`_id`, `class`, field names) — written verbatim as JSON values.
|
||||
- Cache TTL semantics — KV `expirationTtl` is behaviorally equivalent to Java's `(now - millis) > cacheMillis` check on read. Both produce a "cache miss → re-fetch" outcome at the same boundary.
|
||||
- `getAdmin()` returning `newAdmin()` if doc missing — `getJson('admin') ?? newAdmin()` matches.
|
||||
- Existence checks — `(await get) !== null` replaces `countDocuments`.
|
||||
- Concurrent `addGroup` race — Java/Mongo had the same non-atomic R-M-W; not a regression.
|
||||
|
||||
**Two intentional divergences from Java:**
|
||||
|
||||
1. **Eventual consistency (~60s globally)**: Java/Mongo gave strong read-after-write. For a single-admin daily-cron bot this is acceptable; documented in `plan.md`.
|
||||
2. **TTL floor of 60s**: Java had no minimum. Mitigated by the `kv.js` clamp above.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Write `kv.js` helper** at `src/repository/kv.js` — thin wrapper exposing `getJson(env, key)`, `putJson(env, key, value, opts?)`, `del(env, key)`. Single-binding-name constant `STORE_KV`. Throws a typed error if `env.STORE_KV` is missing.
|
||||
|
||||
2. **Rewrite `admin-repository.js`**:
|
||||
- `init()` → `getJson('admin')`; if null, `putJson('admin', newAdmin())`.
|
||||
- `getAdmin()` → `getJson('admin') ?? newAdmin()`.
|
||||
- `save(admin)` → `putJson('admin', admin)`.
|
||||
- `addGroup`/`removeGroup`/`hasGroup`/`getAllGroups` — unchanged (delegate to `getAdmin` + `save`).
|
||||
|
||||
3. **Rewrite `group-repository.js`**:
|
||||
- `key(groupId)` → `` `group:${groupIdToKey(groupId)}` ``.
|
||||
- `exists(groupId)` → `getJson(key(groupId)) !== null`.
|
||||
- `getGroup(groupId)` → `getJson(key(groupId)) ?? newGroup(groupId)`.
|
||||
- `saveGroup(group)` → `putJson(\`group:${group._id}\`, group)`.
|
||||
- `deleteGroup(groupId)` → `del(key(groupId))`.
|
||||
- `mutateAndSave` unchanged.
|
||||
|
||||
4. **Rewrite `apple-app-repository.js`**:
|
||||
- `key(appId)` → `` `apple:${appId}` ``.
|
||||
- `get(appId)` → `getJson(key(appId))`.
|
||||
- `save(entry)` → `putJson(key(entry._id), entry, { expirationTtl: appCacheSeconds })`.
|
||||
- `getCached(appId)` → just `get(appId)` — TTL already enforced by KV. Remove the `isAppleAppExpired` call.
|
||||
|
||||
5. **Rewrite `google-app-repository.js`** — same pattern as apple.
|
||||
|
||||
6. **Delete `src/repository/mongodb.js`**. Remove its imports from the four repositories.
|
||||
|
||||
7. **Update `src/config.js`** — drop `'MONGODB_URI'` from the `required` list.
|
||||
|
||||
8. **Update `src/index.js`** — strip the `MongoClient`-related comment in `build()`. Add a sanity check: throw if `env.STORE_KV` is undefined (early failure beats null-dereference at first request).
|
||||
|
||||
9. **Update `package.json`**:
|
||||
- `npm uninstall mongodb`
|
||||
- bump `version` to `0.3.0`
|
||||
|
||||
10. **Compile/lint check**: `node --check` each modified file. Run `npm run lint` (the existing secret-leak check).
|
||||
|
||||
11. **Local sanity (no live KV needed)**: `node -e "import('./src/repository/store.js').then(...)"` — confirm no import errors. Cannot fully test without `wrangler dev` (Phase 03).
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] Create `src/repository/kv.js` helper
|
||||
- [ ] Rewrite `admin-repository.js`
|
||||
- [ ] Rewrite `group-repository.js`
|
||||
- [ ] Rewrite `apple-app-repository.js` (with KV TTL)
|
||||
- [ ] Rewrite `google-app-repository.js` (with KV TTL)
|
||||
- [ ] Delete `src/repository/mongodb.js`
|
||||
- [ ] Update `src/config.js` required env list
|
||||
- [ ] Update `src/index.js` env validation
|
||||
- [ ] `npm uninstall mongodb`; bump `package.json` version
|
||||
- [ ] `node --check` all modified files
|
||||
- [ ] `npm run lint`
|
||||
- [ ] **Java parity check**: spot-diff a written `group:{id}` JSON value against an equivalent Java/Mongo doc structure (sample from upstream `tiennm99/store-scraper-bot` model classes). Field names, ordering-insensitive, must match.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `grep -rn "mongodb\|MongoClient" src/` returns nothing.
|
||||
- [ ] `package.json` `dependencies` is empty (only `wrangler` in devDeps).
|
||||
- [ ] All 4 repository factories return the same interface they did before (consumers untouched).
|
||||
- [ ] `node --check src/index.js` exits 0.
|
||||
- [ ] `npm run lint` passes.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Repository interface drift breaks command handlers | Low | Keep the exported function names + signatures byte-identical |
|
||||
| KV's eventual consistency surprises a user mid-conversation | Low | Single-admin bot, low concurrency; document in README |
|
||||
| Concurrent `/addgroup` race clobbers `groups[]` | Low | Same race existed in Mongo; not a regression |
|
||||
| TTL `expirationTtl` < 60 silently fails | Low | `APP_CACHE_SECONDS` defaults to 600; doc the floor in `.dev.vars.example` |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- KV namespace ID is not a secret (binding name is what code references). Safe to commit in `wrangler.toml`.
|
||||
- Removing `MONGODB_URI` removes one secret from the rotation surface. Update `wrangler secret list` documentation in Phase 05.
|
||||
+96
@@ -0,0 +1,96 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Wrangler config + KV namespace provisioning"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "0.5h"
|
||||
dependencies: [1]
|
||||
---
|
||||
|
||||
# Phase 02: Wrangler config + KV namespace provisioning
|
||||
|
||||
## Overview
|
||||
|
||||
Provision a Cloudflare KV namespace, bind it to the Worker as `STORE_KV`, and remove now-unused MongoDB-related config from `wrangler.toml`.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- Worker has access to `env.STORE_KV` at runtime.
|
||||
- Local `wrangler dev` reads from a preview namespace, not production.
|
||||
|
||||
**Non-functional**
|
||||
- `wrangler.toml` is the single source of truth (no `wrangler.jsonc`).
|
||||
- Production and preview namespace IDs are committed (these are not secrets).
|
||||
|
||||
## Architecture
|
||||
|
||||
KV bindings are declared in `wrangler.toml`:
|
||||
|
||||
```toml
|
||||
[[kv_namespaces]]
|
||||
binding = "STORE_KV"
|
||||
id = "<production namespace id>"
|
||||
preview_id = "<preview namespace id>"
|
||||
```
|
||||
|
||||
`binding` is the name surfaced as `env.STORE_KV` in the Worker. `id` is the production namespace UUID. `preview_id` is used by `wrangler dev`.
|
||||
|
||||
Operator-driven steps (run locally; require `wrangler login`):
|
||||
|
||||
```sh
|
||||
npx wrangler kv namespace create STORE_KV
|
||||
npx wrangler kv namespace create STORE_KV --preview
|
||||
```
|
||||
|
||||
Each command outputs a TOML snippet to copy into `wrangler.toml`.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
**Modify**
|
||||
- `wrangler.toml` — add `[[kv_namespaces]]` block; remove `nodejs_compat_v2` flag; trim the secrets-comment block to drop `MONGODB_URI`.
|
||||
|
||||
**No code changes** — Phase 01 already wrote everything that consumes `env.STORE_KV`.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Run** `npx wrangler kv namespace create STORE_KV` — capture the production `id`.
|
||||
2. **Run** `npx wrangler kv namespace create STORE_KV --preview` — capture the `preview_id`.
|
||||
3. **Edit `wrangler.toml`**:
|
||||
- Add the `[[kv_namespaces]]` block with both IDs.
|
||||
- Change `compatibility_flags` from `["nodejs_compat_v2"]` to `[]` (or delete the line entirely).
|
||||
- In the `# Secrets` comment block, remove the `MONGODB_URI` line.
|
||||
4. **Verify** with `npx wrangler types` (if available) or `npx wrangler deploy --dry-run` — should report the new binding without errors.
|
||||
5. **Local seed (optional)**: For `wrangler dev` to be useful, the admin singleton must exist. The repository's `init()` is currently called nowhere automatically. Either:
|
||||
- Seed manually: `npx wrangler kv key put --binding STORE_KV --preview admin '{"_id":"admin","class":"Admin","groups":[]}'`
|
||||
- Or update `src/index.js` to call `store.admin.init()` on first request (cheap, idempotent). **Recommended**: add the lazy init.
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] `wrangler kv namespace create STORE_KV`
|
||||
- [ ] `wrangler kv namespace create STORE_KV --preview`
|
||||
- [ ] Add `[[kv_namespaces]]` block to `wrangler.toml`
|
||||
- [ ] Drop `nodejs_compat_v2` from `compatibility_flags`
|
||||
- [ ] Remove `MONGODB_URI` mention from the secrets comment
|
||||
- [ ] Add lazy `store.admin.init()` on first request in `src/index.js`
|
||||
- [ ] `wrangler deploy --dry-run` succeeds
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `wrangler.toml` has `[[kv_namespaces]]` with `binding = "STORE_KV"` and both IDs.
|
||||
- [ ] `compatibility_flags` no longer contains `nodejs_compat_v2`.
|
||||
- [ ] `wrangler deploy --dry-run` exits 0 and lists the binding.
|
||||
- [ ] `wrangler kv namespace list` shows two namespaces: `STORE_KV` and `STORE_KV_preview` (or named per CF convention).
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Wrong `binding` name → `env.STORE_KV` is undefined at runtime | Low | Phase 01 added the early validation throw |
|
||||
| Forgot `preview_id`, `wrangler dev` errors | Low | Step 2 explicitly creates it |
|
||||
| Operator commits secret accidentally | Low | KV IDs are not secrets; existing `npm run lint` scans for the secret patterns we care about |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- KV namespace IDs are not secrets and are safe to commit.
|
||||
- Production `STORE_KV` will hold Telegram chat IDs and admin IDs — same sensitivity as Mongo previously held. Cloudflare KV is encrypted at rest.
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Deploy + bundle-size sanity gate"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "0.5h"
|
||||
dependencies: [2]
|
||||
---
|
||||
|
||||
# Phase 03: Deploy + bundle-size sanity gate
|
||||
|
||||
## Overview
|
||||
|
||||
Set Worker secrets, run `wrangler deploy`, confirm the bundle is well under the Free 3 MiB cap, and verify a cold-start request reaches the Worker.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- Worker is reachable at `https://js-store-scraper-bot.<account>.workers.dev`.
|
||||
- A request without the secret header returns 401.
|
||||
|
||||
**Non-functional**
|
||||
- Bundle ≤ 500 KiB (informational; we expect ~30 KiB without `mongodb`).
|
||||
- Cold start CPU < 10ms.
|
||||
|
||||
## Architecture
|
||||
|
||||
`wrangler deploy` reads secrets from the Cloudflare account (set via `wrangler secret put`), KV bindings from `wrangler.toml`, and uploads the bundled Worker. No Atlas, no TCP — just KV (HTTP edge-side), `fetch` to Telegram + iTunes + Google Play.
|
||||
|
||||
Required secrets (set once via `wrangler secret put <NAME>`):
|
||||
|
||||
| Secret | Purpose |
|
||||
|---|---|
|
||||
| `TELEGRAM_BOT_TOKEN` | Posting to Telegram Bot API |
|
||||
| `TELEGRAM_BOT_USERNAME` | Bot mention parsing |
|
||||
| `TELEGRAM_WEBHOOK_SECRET` | Validates `X-Telegram-Bot-Api-Secret-Token` |
|
||||
| `ADMIN_IDS` | Comma-separated Telegram user IDs |
|
||||
|
||||
`MONGODB_URI` is **no longer required**. If a previous deploy set it, leave it (no-op) or run `wrangler secret delete MONGODB_URI` for hygiene.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
**No code changes.** Phase deals with deployment only.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Set secrets** (interactive prompts; values supplied by operator):
|
||||
```sh
|
||||
npx wrangler secret put TELEGRAM_BOT_TOKEN
|
||||
npx wrangler secret put TELEGRAM_BOT_USERNAME
|
||||
npx wrangler secret put TELEGRAM_WEBHOOK_SECRET
|
||||
npx wrangler secret put ADMIN_IDS
|
||||
```
|
||||
`TELEGRAM_WEBHOOK_SECRET`: any random ≥32 chars, e.g. `openssl rand -hex 24`.
|
||||
|
||||
2. **Optional cleanup**: `npx wrangler secret delete MONGODB_URI` if it was set previously.
|
||||
|
||||
3. **Bundle gate**:
|
||||
```sh
|
||||
npx wrangler deploy --dry-run --outdir=.bundle 2>&1 | tee .bundle/dry-run.log
|
||||
du -h .bundle/*.js
|
||||
```
|
||||
Expected: total .js output < 500 KiB. If > 500 KiB, abort and investigate (`wrangler deploy --dry-run --metafile` then inspect imports).
|
||||
|
||||
4. **Real deploy**:
|
||||
```sh
|
||||
npx wrangler deploy
|
||||
```
|
||||
Capture the deployed URL.
|
||||
|
||||
5. **Cold-start probe**: send a probe with a bad secret to confirm the 401 path works (cheap, doesn't touch KV):
|
||||
```sh
|
||||
curl -i -X POST https://js-store-scraper-bot.<account>.workers.dev \
|
||||
-H "X-Telegram-Bot-Api-Secret-Token: wrong" \
|
||||
-d '{}'
|
||||
```
|
||||
Expected: `HTTP/1.1 401 Unauthorized`.
|
||||
|
||||
6. **CPU check**: open the Cloudflare dashboard → Workers → js-store-scraper-bot → Logs/Metrics. After a few requests, confirm cold-start CPU is well under 50ms (target <10ms without Mongo).
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] Set 4 required secrets via `wrangler secret put`
|
||||
- [ ] (Optional) Delete obsolete `MONGODB_URI` secret
|
||||
- [ ] Run `wrangler deploy --dry-run`; verify bundle < 500 KiB
|
||||
- [ ] `wrangler deploy`; capture URL
|
||||
- [ ] curl probe → expect 401
|
||||
- [ ] Confirm cold-start CPU < 10ms in CF dashboard
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Bundle size logged and < 500 KiB.
|
||||
- [ ] `wrangler deploy` succeeds with `Uploaded` line.
|
||||
- [ ] 401 probe returns 401 status.
|
||||
- [ ] CPU per request < 10ms in dashboard logs.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Bundle balloons due to a transitive dep | Very Low | `mongodb` was the only heavy dep; with it gone, only stdlib + small `wrangler` runtime hooks remain |
|
||||
| CF account quota exceeded | Very Low | New account = 100k requests/day budget; bot traffic is single-digit per minute |
|
||||
| Secret typo (e.g. token) → 401 from Telegram on first webhook hit | Medium | Caught in Phase 04 smoke; rerun `wrangler secret put` to fix |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- All secrets are set via `wrangler secret put` (never in `wrangler.toml`).
|
||||
- The 401 probe uses a deliberately wrong secret; no sensitive value is logged.
|
||||
- Run `npm run lint` (existing secret-leak check) before deploy.
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Register webhook + smoke-test"
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: "1.5h"
|
||||
dependencies: [3]
|
||||
---
|
||||
|
||||
# Phase 04: Register webhook + smoke-test
|
||||
|
||||
## Overview
|
||||
|
||||
Point the Telegram Bot API at the deployed Worker, then exercise every command and the daily cron path against the live deployment.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- Telegram delivers updates to `https://<worker-url>` with the secret header.
|
||||
- All 13 commands return correct responses.
|
||||
- KV reads/writes show up in `wrangler kv key list --binding STORE_KV`.
|
||||
|
||||
**Non-functional**
|
||||
- Each command response < 1s (Telegram's webhook timeout is 30s, but slow webhooks lead to retries).
|
||||
|
||||
## Architecture
|
||||
|
||||
Telegram webhook flow:
|
||||
|
||||
```
|
||||
Telegram → POST https://js-store-scraper-bot.<acct>.workers.dev
|
||||
with header X-Telegram-Bot-Api-Secret-Token = <secret>
|
||||
→ Worker validates header (401 on mismatch)
|
||||
→ Worker parses update, ctx.waitUntil(dispatch(...))
|
||||
→ Worker returns 200 immediately
|
||||
→ dispatch reads/writes STORE_KV, calls Telegram sendMessage
|
||||
```
|
||||
|
||||
The cron schedule (`0 0 * * *` UTC = 7am `Asia/Ho_Chi_Minh`) triggers `scheduled()` daily; we manually trigger it once during smoke via `wrangler triggers cron`.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
**No code changes.** Smoke uses the existing `scripts/register-webhook.js` and live commands.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Populate `.env.deploy`** from `.env.deploy.example`:
|
||||
```
|
||||
TELEGRAM_BOT_TOKEN=...
|
||||
TELEGRAM_WEBHOOK_SECRET=... # must match the secret set in Phase 03
|
||||
WORKER_URL=https://js-store-scraper-bot.<acct>.workers.dev
|
||||
```
|
||||
|
||||
2. **Dry-run register**:
|
||||
```sh
|
||||
npm run register:dry
|
||||
```
|
||||
Confirms the request body is sane.
|
||||
|
||||
3. **Real register**:
|
||||
```sh
|
||||
npm run register
|
||||
```
|
||||
Calls `setWebhook` + `setMyCommands`. Expect `ok: true` from Telegram.
|
||||
|
||||
4. **Verify webhook info**:
|
||||
```sh
|
||||
curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getWebhookInfo" | jq
|
||||
```
|
||||
`url` should match, `pending_update_count` should be 0 once cleared, `has_custom_certificate: false`.
|
||||
|
||||
5. **Smoke test (operator runs in Telegram)**:
|
||||
|
||||
| Command | Expected | KV mutation |
|
||||
|---|---|---|
|
||||
| `/info` | bot info text (admin id, build time) | none |
|
||||
| `/addgroup` (in target chat as admin) | "Group added" | `admin` ← `groups` includes chat id |
|
||||
| `/listgroup` | shows new group | none |
|
||||
| `/addapple <appId> <country>` | "Added" | `group:{chatId}` updated |
|
||||
| `/addgoogle <appId> <country>` | "Added" | `group:{chatId}` updated |
|
||||
| `/listapp` | renders both lists | none |
|
||||
| `/checkapp` | table with ✅/❌ marks | populates `apple:{appId}` / `google:{appId}` |
|
||||
| `/checkappscore` | scores table | uses cache from previous step |
|
||||
| `/rawappleapp <appId>` | raw JSON dump | reads `apple:{appId}` |
|
||||
| `/rawgoogleapp <appId>` | raw JSON dump | reads `google:{appId}` |
|
||||
| `/delapple <appId>` | "Removed" | `group:{chatId}` updated |
|
||||
| `/delgoogle <appId>` | "Removed" | `group:{chatId}` updated |
|
||||
| `/delgroup` | "Group removed" | `admin.groups` shrinks |
|
||||
|
||||
6. **Inspect KV** between steps:
|
||||
```sh
|
||||
npx wrangler kv key list --binding STORE_KV --remote | jq
|
||||
npx wrangler kv key get --binding STORE_KV --remote admin | jq
|
||||
```
|
||||
|
||||
7. **Trigger cron manually**:
|
||||
```sh
|
||||
npx wrangler triggers cron "0 0 * * *"
|
||||
```
|
||||
(Or use the dashboard's "Trigger" button.) Re-add a test app with a known-stale `updated` date (or a real lagging app) and confirm the daily report is posted to the test group.
|
||||
|
||||
8. **Cache TTL probe**: read `apple:<appId>` from KV — `expiration` field should be `now + APP_CACHE_SECONDS`. After waiting past TTL, the next `/rawappleapp` should re-fetch from iTunes.
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] Populate `.env.deploy`
|
||||
- [ ] `npm run register:dry`
|
||||
- [ ] `npm run register`
|
||||
- [ ] `getWebhookInfo` reports correct URL
|
||||
- [ ] Walk through all 13 commands; record responses
|
||||
- [ ] `wrangler kv key list` shows expected keys after mutations
|
||||
- [ ] Manually trigger cron; verify daily report message
|
||||
- [ ] Confirm cache TTL behavior
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] All 13 commands return success responses (no "Internal server error").
|
||||
- [ ] KV inspection matches expected mutations after each command.
|
||||
- [ ] Manual cron trigger posts a report (or "no stale apps" if none).
|
||||
- [ ] Worker logs show no unhandled exceptions across the smoke run.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Webhook secret mismatch → 401 loop | Low | Step 4 verifies `getWebhookInfo`; rotate via `setWebhook` if needed |
|
||||
| Telegram rate-limited during smoke | Low | Free tier limits are generous for ad-hoc testing; pause if 429 |
|
||||
| KV eventual-consistency surprises smoke (write then immediate read on different POP) | Low | Same-region writes are visible immediately on modern KV; watch for the rare read-your-write miss and retry |
|
||||
| `/checkapp` 30s timeout if upstream stores slow | Medium | Both scrapers have own timeouts; if hit, rerun once |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- `TELEGRAM_WEBHOOK_SECRET` must be ≥32 chars and unique to this Worker.
|
||||
- After smoke, verify only authorized admin IDs can issue mutating commands by attempting from a non-admin account → expect "Unauthorized".
|
||||
- Capture Worker logs from the dashboard and grep for accidental token leaks (none expected; existing structured logger redacts).
|
||||
@@ -0,0 +1,136 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Docs + cleanup"
|
||||
status: pending
|
||||
priority: P2
|
||||
effort: "1h"
|
||||
dependencies: [4]
|
||||
---
|
||||
|
||||
# Phase 05: Docs + cleanup
|
||||
|
||||
## Overview
|
||||
|
||||
Update README + project docs to reflect KV-backed storage, archive the obsolete Atlas plan, and remove now-dead Mongo references from `.env*` examples and Docker files.
|
||||
|
||||
## Requirements
|
||||
|
||||
**Functional**
|
||||
- A new operator can deploy from scratch by following only `README.md` + `docs/deployment-guide.md`.
|
||||
- No file references MongoDB Atlas as the production storage.
|
||||
|
||||
**Non-functional**
|
||||
- Keep `docs/*.md` under 800 LOC each (per `docs.maxLoc`).
|
||||
|
||||
## Architecture
|
||||
|
||||
Documentation surface:
|
||||
|
||||
| File | Action |
|
||||
|---|---|
|
||||
| `README.md` | Replace MongoDB section with KV; update env table; bump status note |
|
||||
| `.env.example` | Remove `MONGODB_*`; add note that secrets live in `wrangler secret put` |
|
||||
| `.dev.vars.example` | Replace `MONGODB_URI=` with KV preview-namespace seeding hint |
|
||||
| `docs/deployment-guide.md` | Create or update with the Phase 02–04 operator runbook |
|
||||
| `docs/system-architecture.md` | Diagram: Worker → KV (drop Atlas) |
|
||||
| `plans/todo.md` | Remove the Atlas pre-flight items; update active-plan link |
|
||||
| `plans/260426-2327-cloudflare-deploy-and-smoke/` | Move under `plans/archive/` with a "superseded by 260505-1425" note |
|
||||
|
||||
Docker files (`Dockerfile`, `docker-compose.yml`, `docker-compose.dev.yml`) reference the old polling path. **Decision**: keep them as a fallback rollback option but add a banner comment that they target the legacy Mongo runtime, not Workers.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
**Modify**
|
||||
- `README.md`
|
||||
- `.env.example`
|
||||
- `.dev.vars.example`
|
||||
- `plans/todo.md`
|
||||
- `Dockerfile` (banner comment only)
|
||||
- `docker-compose.yml` (banner comment only)
|
||||
- `docker-compose.dev.yml` (banner comment only)
|
||||
|
||||
**Create**
|
||||
- `docs/deployment-guide.md` (if missing)
|
||||
- `docs/system-architecture.md` (if missing) — short, ASCII or mermaid diagram
|
||||
|
||||
**Move (archive)**
|
||||
- `plans/260426-2327-cloudflare-deploy-and-smoke/` → `plans/archive/260426-2327-cloudflare-deploy-and-smoke/`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **`README.md`**:
|
||||
- Remove "Requirements: MongoDB 4.4+".
|
||||
- Update env table: drop `MONGODB_CONNECTION_STRING`, `MONGO_DATABASE`; note that secrets go via `wrangler secret put`, KV binding is `STORE_KV`.
|
||||
- Replace "Run via Docker Compose" with "Deploy: `npm run deploy`".
|
||||
- Update "Differences vs Go / Java" to note storage backend differs (KV instead of Mongo) but document shape preserved.
|
||||
|
||||
2. **`.env.example`**:
|
||||
- Drop `MONGODB_*` rows; add a comment that production secrets live in `wrangler secret`.
|
||||
- Keep `ADMIN_IDS`, `TELEGRAM_*` for local-dev reference only.
|
||||
|
||||
3. **`.dev.vars.example`** — drop `MONGODB_URI`.
|
||||
|
||||
4. **`docs/deployment-guide.md`** — short runbook condensing Phases 02–04 (KV namespace creation, secrets, deploy, register webhook, smoke). One file, < 200 lines.
|
||||
|
||||
5. **`docs/system-architecture.md`** — mermaid diagram:
|
||||
```
|
||||
Telegram --webhook--> CF Worker --KV ops--> STORE_KV
|
||||
--HTTPS--> iTunes / Play
|
||||
cron --7am VN--> daily check
|
||||
```
|
||||
|
||||
6. **`plans/todo.md`**:
|
||||
- Replace "Active plan" link with `260505-1425-cloudflare-kv-migration-and-deploy`.
|
||||
- Remove the Atlas-specific hard-gate table (no longer applicable).
|
||||
- Update pre-flight: drop Atlas account requirement; KV needs only CF account.
|
||||
|
||||
7. **Archive old plan**:
|
||||
```sh
|
||||
mv plans/260426-2327-cloudflare-deploy-and-smoke plans/archive/
|
||||
```
|
||||
Append a single line at top of its `plan.md`: `> **Superseded by [`260505-1425-cloudflare-kv-migration-and-deploy`](../../260505-1425-cloudflare-kv-migration-and-deploy/plan.md)** — KV pivot taken pre-emptively.`
|
||||
|
||||
8. **Docker file banners** — one-line comment at top:
|
||||
```
|
||||
# NOTE: Legacy Mongo polling path. Production deploys to Cloudflare Workers via `npm run deploy` and uses Cloudflare KV.
|
||||
```
|
||||
|
||||
9. **Final commit**: single squashed commit `feat: migrate to Cloudflare KV storage + deploy` with the full Phase 01–04 diff. Use conventional-commits.
|
||||
|
||||
## Todo List
|
||||
|
||||
- [ ] Update `README.md`
|
||||
- [ ] Update `.env.example`
|
||||
- [ ] Update `.dev.vars.example`
|
||||
- [ ] Create/update `docs/deployment-guide.md`
|
||||
- [ ] Create/update `docs/system-architecture.md`
|
||||
- [ ] Update `plans/todo.md`
|
||||
- [ ] Move old Atlas plan under `plans/archive/`
|
||||
- [ ] Add legacy banners to Docker files
|
||||
- [ ] Commit + push
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `grep -rn "MONGODB\|mongodb\|Atlas" README.md docs/ .env.example .dev.vars.example` returns nothing.
|
||||
- [ ] `docs/deployment-guide.md` exists and is < 200 lines.
|
||||
- [ ] Old plan lives under `plans/archive/` with the supersession banner.
|
||||
- [ ] `git log --oneline -5` shows a clean conventional commit.
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Stale Atlas reference left in some doc | Low | grep step in success criteria |
|
||||
| Doc exceeds `docs.maxLoc=800` | Low | Each doc is short by design |
|
||||
| Operator follows old plan by accident | Low | Archive move + supersession banner make it obvious |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Confirm no secrets are pasted into README or examples.
|
||||
- `npm run lint` (the existing `scripts/check-secret-leaks.js`) runs at commit time per existing workflow.
|
||||
|
||||
## Next Steps (out of this plan, into backlog)
|
||||
|
||||
- CI workflow that runs `wrangler deploy --dry-run` as a PR bundle-size gate.
|
||||
- Observability dashboard pointed at the `STORE_KV` namespace (request count + key count).
|
||||
- Tests — original project has none; would be additive work.
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: "Cloudflare KV migration + deploy"
|
||||
description: "Replace MongoDB Atlas with Cloudflare KV as the storage layer, deploy to Workers, register Telegram webhook, smoke-test, and document. Supersedes the Atlas-based deploy plan."
|
||||
status: pending
|
||||
priority: P1
|
||||
effort: 5h
|
||||
branch: main
|
||||
tags: [cloudflare-workers, cloudflare-kv, deploy, migration]
|
||||
created: 2026-05-05
|
||||
blockedBy: []
|
||||
blocks: [260426-2327-cloudflare-deploy-and-smoke]
|
||||
---
|
||||
|
||||
# Cloudflare Workers + KV Storage Migration
|
||||
|
||||
User chose KV over Atlas to pre-empt the bundle-size and cold-start risks called out in the Atlas plan. Drops the `mongodb` driver entirely (~4–5 MiB compressed) and the `nodejs_compat_v2` flag along with it.
|
||||
|
||||
## Why KV fits this codebase
|
||||
|
||||
Every existing repository operation is a primary-key lookup — no queries, no aggregations, no cursors:
|
||||
|
||||
| Logical collection | Operations | KV key shape |
|
||||
|---|---|---|
|
||||
| `common` (admin singleton) | get/save the single admin doc with `groups[]` | `admin` |
|
||||
| `group` (per-chat) | exists/get/save/delete by groupId | `group:{groupId}` |
|
||||
| `apple_app` (cache) | get/save by appId, TTL expiry | `apple:{appId}` (with `expirationTtl`) |
|
||||
| `google_app` (cache) | get/save by appId, TTL expiry | `google:{appId}` (with `expirationTtl`) |
|
||||
|
||||
KV's native `expirationTtl` replaces the manual `isExpired(entry, now, cacheMs)` check.
|
||||
|
||||
The daily cron reads `admin.groups[]` (single key), then iterates groups (one read each). No list-keys needed.
|
||||
|
||||
## Phases
|
||||
|
||||
| # | Phase | Status | Effort |
|
||||
|---|---|---|---|
|
||||
| 01 | [Replace Mongo with KV](phase-01-kv-repositories.md) | completed | 1.5h |
|
||||
| 02 | [Wrangler + KV namespace](phase-02-wrangler-and-kv-namespace.md) | pending | 0.5h |
|
||||
| 03 | [Deploy + bundle gate](phase-03-deploy-and-bundle-gate.md) | pending | 0.5h |
|
||||
| 04 | [Register webhook + smoke](phase-04-register-webhook-and-smoke.md) | pending | 1.5h |
|
||||
| 05 | [Docs + cleanup](phase-05-docs-and-cleanup.md) | pending | 1h |
|
||||
|
||||
## Critical dependencies
|
||||
|
||||
- 01 → 02 (wrangler binding name must match what code reads from `env.STORE_KV`)
|
||||
- 02 → 03 (deploy needs the namespace ID provisioned)
|
||||
- 03 → 04 (webhook registration needs live Worker URL)
|
||||
|
||||
## Hard gates (now sanity checks, not pivots)
|
||||
|
||||
| Gate | Threshold | Notes |
|
||||
|---|---|---|
|
||||
| Bundle size | ≤ 500 KiB (well under 3 MiB Free cap) | Without `mongodb`, bundle should drop to ~30 KiB |
|
||||
| Cold start | < 10ms CPU | No TCP handshake; KV access is HTTP-fetched edge-side |
|
||||
|
||||
If bundle exceeds 500 KiB, investigate — something pulled in unintended deps.
|
||||
|
||||
## Key design decisions
|
||||
|
||||
1. **One KV namespace, prefixed keys** — cleaner than four namespaces; KV billing is by op count not namespace.
|
||||
2. **Drop `nodejs_compat_v2`** — only needed for the Mongo TCP socket. Removing reduces cold-start and runtime memory.
|
||||
3. **Drop `mongodb` dependency** — removes ~4–5 MiB compressed. `npm uninstall mongodb`.
|
||||
4. **Cache TTL via `expirationTtl`** — let KV handle expiry; remove `isAppleAppExpired`/`isGoogleAppExpired` helpers from the read path.
|
||||
5. **Document model unchanged** — same `_id`, `class`, fields. Stored as JSON values. Java/Go parity preserved at the document level even though the storage engine differs.
|
||||
|
||||
## Java parity verification
|
||||
|
||||
Cross-checked against existing parity reports (commit `e3e375d`):
|
||||
- `plans/reports/xia-260429-1601-store-scraper-bot-port-status.md` — 11 PARITY · 2 MINOR · **0 GAP** · 3 EXTRA
|
||||
- `plans/reports/researcher-260429-1555-java-vs-js-parity.md` — schema, scheduler timing, cache TTL, HTML formatting all confirmed identical
|
||||
|
||||
**KV swap preserves every PARITY item.** Document shape (`_id`, `class`, fields) is written verbatim as JSON values. Cache TTL via `expirationTtl` is behaviorally equivalent to Java's `(now - millis) > cacheMillis` read-side check (same cache-miss→refetch boundary). `countDocuments` and `findOne` map cleanly to `get !== null` and `get`.
|
||||
|
||||
Two intentional divergences from Java/Mongo, both accepted:
|
||||
1. KV ~60s eventual consistency (Java/Mongo was strongly consistent) — fine for single-admin daily-cron use.
|
||||
2. KV `expirationTtl` 60s floor (Java had no minimum) — mitigated by a clamp in `kv.js`.
|
||||
|
||||
## Known KV trade-offs (accepted)
|
||||
|
||||
- **Eventual consistency (~60s global propagation)**: `/addgroup` followed by an immediate cron run on a different POP could miss the new group. For this single-admin bot with a once-daily cron, acceptable.
|
||||
- **No atomic read-modify-write**: Two concurrent `/addgroup` calls could clobber each other on the singleton's `groups[]` array. Mongo's `replaceOne` had the same race; not a regression.
|
||||
- **Free tier**: 100k reads/day, 1k writes/day, 1 GB storage. Bot usage projects <100 reads/day.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Data migration — greenfield per `plans/todo.md` Q3.
|
||||
- Tests — none exist; not adding here.
|
||||
- Custom domain — `*.workers.dev` URL is fine for Telegram webhooks.
|
||||
- Durable Objects / D1 — KV is sufficient.
|
||||
|
||||
## Rollback path
|
||||
|
||||
- After Phase 01 (code-only): `git revert` reverts the repo refactor; `mongodb` driver back, plan reopens against Atlas-based deploy plan.
|
||||
- After Phase 03 (deployed but no webhook): `wrangler delete` removes Worker.
|
||||
- After Phase 04 (webhook live): `setWebhook` with empty URL clears it; rotate `TELEGRAM_WEBHOOK_SECRET` so any in-flight POSTs 401.
|
||||
- KV namespace can be retained — empty namespaces cost nothing on Free tier.
|
||||
|
||||
## Supersedes
|
||||
|
||||
This plan replaces the storage backend chosen in [`260426-2327-cloudflare-deploy-and-smoke`](../260426-2327-cloudflare-deploy-and-smoke/plan.md). On completion, that plan should be archived.
|
||||
Reference in new issue
Block a user