From 10cb776eccf29fa05cb3ded5b4530b8fbc37936c Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Sun, 10 May 2026 02:33:01 +0700 Subject: [PATCH] chore(plans): remove completed migration & mapillary plans --- .../phase-01-adapter-refactor-with-prefix.md | 134 ------------- .../phase-02-data-migration.md | 104 ----------- .../phase-03-cutover-and-cleanup.md | 95 ---------- .../plan.md | 73 -------- ...se-01-per-city-delta-random-point-fetch.md | 176 ------------------ .../plan.md | 94 ---------- .../phase-01-cleanup.md | 63 ------- .../plan.md | 60 ------ 8 files changed, 799 deletions(-) delete mode 100644 plans/260509-2234-upstash-db-migration-with-prefix/phase-01-adapter-refactor-with-prefix.md delete mode 100644 plans/260509-2234-upstash-db-migration-with-prefix/phase-02-data-migration.md delete mode 100644 plans/260509-2234-upstash-db-migration-with-prefix/phase-03-cutover-and-cleanup.md delete mode 100644 plans/260509-2234-upstash-db-migration-with-prefix/plan.md delete mode 100644 plans/260510-0140-port-viguessr-mapillary/phase-01-per-city-delta-random-point-fetch.md delete mode 100644 plans/260510-0140-port-viguessr-mapillary/plan.md delete mode 100644 plans/260510-0204-cleanup-upstash-migration/phase-01-cleanup.md delete mode 100644 plans/260510-0204-cleanup-upstash-migration/plan.md diff --git a/plans/260509-2234-upstash-db-migration-with-prefix/phase-01-adapter-refactor-with-prefix.md b/plans/260509-2234-upstash-db-migration-with-prefix/phase-01-adapter-refactor-with-prefix.md deleted file mode 100644 index 86a4702..0000000 --- a/plans/260509-2234-upstash-db-migration-with-prefix/phase-01-adapter-refactor-with-prefix.md +++ /dev/null @@ -1,134 +0,0 @@ ---- -phase: 1 -title: "Adapter Refactor with Prefix" -status: pending -priority: P1 -effort: "1h" -dependencies: [] ---- - -# Phase 1: Adapter Refactor with Prefix - -## Overview - -Replace the node-redis client (`src/lib/redis.js`) with an `@upstash/redis` REST adapter that bundles client + `KEY_PREFIX` into a single handle. Refactor `session.js` and `leaderboard.js` to call helpers that prepend the prefix, mirroring the store-scraper-bot adapter pattern. After this phase, the runtime targets the new Upstash DB and is multi-tenant safe. - -## Requirements - -- Public API of `session.js` and `leaderboard.js` (function names + arg shapes) **must not change**. API routes must not be modified beyond import paths. -- All physical Redis keys carry `KEY_PREFIX` (default `vngeoguessr:`). Logical keys at call sites stay raw (`session:abc`, `leaderboard:vietnam`, etc.). -- Prefix applied centrally in the adapter — no caller may concatenate the prefix manually. -- Env var naming: accept both `UPSTASH_REDIS_REST_URL`/`UPSTASH_REDIS_REST_TOKEN` and the Vercel Marketplace aliases `KV_REST_API_URL`/`KV_REST_API_TOKEN`. Throw on first call if neither set. -- Sorted-set ordering semantics preserved: scores leaderboard returns highest-first; distance leaderboard returns lowest-first. -- TTL semantics preserved on session keys (30 min via `ex`). -- Followed: JavaScript only, individual parameters (no destructuring in public function signatures). - -## Architecture - -``` -src/lib/ -├── upstash.js ← NEW: createUpstashClient + getJson/putJson/del -│ + zAdd/zRange/zScore/zRank/zRevRank/zRemRangeByRank -├── redis.js ← DELETE -├── session.js ← imports getUpstash() handle; uses putJson/getJson/del -└── leaderboard.js ← imports getUpstash() handle; uses sorted-set helpers -``` - -### Adapter shape - -```js -// src/lib/upstash.js -import { Redis } from '@upstash/redis'; - -const DEFAULT_KEY_PREFIX = 'vngeoguessr:'; - -let handle = null; - -export function getUpstash() { - if (handle) return handle; - const url = process.env.UPSTASH_REDIS_REST_URL ?? process.env.KV_REST_API_URL; - const token = process.env.UPSTASH_REDIS_REST_TOKEN ?? process.env.KV_REST_API_TOKEN; - if (!url) throw new Error('UPSTASH_REDIS_REST_URL or KV_REST_API_URL is required'); - if (!token) throw new Error('UPSTASH_REDIS_REST_TOKEN or KV_REST_API_TOKEN is required'); - const client = new Redis({ url, token }); - const prefix = process.env.KEY_PREFIX ?? DEFAULT_KEY_PREFIX; - handle = { client, prefix }; - return handle; -} - -function pkey(h, key) { return `${h.prefix}${key}`; } -``` - -### Helper inventory (all accept handle as first arg) - -| Helper | Wraps | Caller | -|---|---|---| -| `getJson(h, key)` | `client.get` (auto-parses) | session read | -| `putJson(h, key, value, ttlSeconds)` | `client.set(..., { ex })` | session write | -| `del(h, key)` | `client.del` | session delete | -| `zAdd(h, key, score, member)` | `client.zadd(key, { score, member })` | leaderboard write | -| `zScore(h, key, member)` | `client.zscore` | leaderboard read | -| `zRangeWithScores(h, key, start, stop, rev)` | `client.zrange(..., { withScores, rev })` | leaderboard read | -| `zRank(h, key, member)` | `client.zrank` | distance rank | -| `zRevRank(h, key, member)` | `client.zrevrank` | score rank | -| `zRemRangeByRank(h, key, start, stop)` | `client.zremrangebyrank` | leaderboard trim | - -`@upstash/redis` returns `zrange` with scores as a flat `[member, score, member, score, ...]` array — helper must reshape to `[{ value, score }, ...]` to match what `leaderboard.js` consumes today. - -### Caller migration (mechanical) - -Each `await getRedis()` becomes `getUpstash()`. Each `redis.method(physicalKey, ...)` becomes `helper(handle, logicalKey, ...)`. Examples: - -| Before | After | -|---|---| -| `redis.setEx('session:' + id, 1800, JSON.stringify(data))` | `await putJson(h, 'session:' + id, data, 1800)` | -| `redis.get('session:' + id)` then `JSON.parse` | `await getJson(h, 'session:' + id)` | -| `redis.zAdd(key, { score, value })` | `await zAdd(h, key, score, value)` | -| `redis.zRangeWithScores(key, 0, n-1, { REV: true })` | `await zRangeWithScores(h, key, 0, n-1, true)` | -| `redis.zRemRangeByRank(key, 0, -201)` | `await zRemRangeByRank(h, key, 0, -201)` | - -Note: `session.js` currently `JSON.stringify`s the value before `setEx` and `JSON.parse`s after `get`. The new helpers do this internally; remove the manual stringify/parse at call sites. - -## Related Code Files - -- Create: `src/lib/upstash.js` -- Modify: `src/lib/session.js` (swap to handle + helpers, drop manual JSON marshal) -- Modify: `src/lib/leaderboard.js` (swap to handle + sorted-set helpers; reshape `zrange` results) -- Delete: `src/lib/redis.js` -- Modify: `package.json` — add `@upstash/redis`, remove `redis`. **Highlight to user for manual install** per project's `File Modification Policy` (configuration changes). - -## Implementation Steps - -1. Add `@upstash/redis` dep, drop `redis` dep. - - User runs: `npm install @upstash/redis && npm uninstall redis` - - Agent must NOT auto-modify `package.json` per project rules; flag the install command in the final report. -2. Write `src/lib/upstash.js` with the singleton `getUpstash()` and all helpers listed above. Verify the `zRangeWithScores` reshape matches what `leaderboard.js` expects (`[{ value, score }]`). -3. Rewrite `src/lib/session.js` to use `getUpstash()` + `putJson`/`getJson`/`del`. Keep function signatures identical (`storeGameSession(sessionId, sessionData)`, etc.). -4. Rewrite `src/lib/leaderboard.js` to use `getUpstash()` + sorted-set helpers. Keep `getLeaderboard(cityCode, limit, type)`, `submitScore(username, score, cityCode)`, `submitDistanceRecord(username, distance, cityCode)` signatures intact. -5. Delete `src/lib/redis.js`. -6. Run `npm run lint` and `npm run build` — no compile errors, no `from 'redis'` references remain. -7. Local sanity (user-driven, optional): `npm run dev`, hit `/api/leaderboard` (GET) — should return empty leaderboard from new DB without throwing. - -## Success Criteria - -- [ ] `src/lib/upstash.js` exports `getUpstash` + the helper inventory above -- [ ] `getUpstash()` reads `KEY_PREFIX` (default `vngeoguessr:`) and accepts both env-var naming conventions -- [ ] `src/lib/session.js` and `src/lib/leaderboard.js` import only from `./upstash.js` -- [ ] `src/lib/redis.js` deleted; `grep -r "from './redis" src/` empty -- [ ] `grep -r "REDIS_URL\|node:redis\|from 'redis'" src/` empty -- [ ] `npm run build` passes -- [ ] API route files (`api/guess/route.js`, `api/cron/route.js`, `api/leaderboard/route.js`) untouched - -## Risk Assessment - -- **Risk:** `@upstash/redis` `zrange` shape with `withScores: true` differs from `node-redis` `zRangeWithScores`. **Mitigation:** Helper normalizes to `[{ value, score }]`; verify with one log line during dev smoke before proceeding to Phase 2. -- **Risk:** `@upstash/redis` auto-parses JSON only when value was set as an object via the SDK. Mixing manual `JSON.stringify` with later `client.get` may double-encode. **Mitigation:** centralize encode/decode in `putJson`/`getJson` — pass raw objects in, receive raw objects out. No caller does its own marshal. -- **Risk:** Caller forgets to `await getUpstash()` — but it's synchronous (no connection step in REST SDK). **Mitigation:** helper is sync; no chance of unawaited promise here. -- **Risk:** Hardcoded prefix drift between adapter and migration script (Phase 2). **Mitigation:** both read `process.env.KEY_PREFIX ?? 'vngeoguessr:'`. Document the contract in `plan.md`. -- **Risk:** `next build` triggers env-var lookup on import — `getUpstash()` must be lazy. **Mitigation:** call from request handlers, not module top-level. Pattern matches existing `getRedis()`. - -## Out of Scope - -- Migrating data from old DB → new DB (Phase 2). -- Removing old DB env vars from Vercel (Phase 3, user-driven). -- Updating `docs/tech-stack.md` (Phase 3). diff --git a/plans/260509-2234-upstash-db-migration-with-prefix/phase-02-data-migration.md b/plans/260509-2234-upstash-db-migration-with-prefix/phase-02-data-migration.md deleted file mode 100644 index f227127..0000000 --- a/plans/260509-2234-upstash-db-migration-with-prefix/phase-02-data-migration.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -phase: 2 -title: "Data Migration old → new Upstash" -status: pending -priority: P1 -effort: "30 min" -dependencies: [1] ---- - -# Phase 2: Data Migration old → new Upstash - -## Overview - -One-shot script that copies leaderboard sorted-set data from the OLD Upstash DB to the NEW Upstash DB, prepending `KEY_PREFIX` (default `vngeoguessr:`) on write. Sessions are not migrated — they are 30-min ephemeral and any in-flight games will resubmit cleanly post-cutover. The OLD DB is read-only during this phase. - -## Requirements - -- Migrate exactly these logical keys (sorted sets): - - `leaderboard:vietnam` - - `leaderboard:city:hanoi`, `leaderboard:city:danang`, `leaderboard:city:hcm`, `leaderboard:city:dalat`, `leaderboard:city:duchoa` (and any other city codes present) - - `distance:vietnam` - - `distance:city:*` (same set of city codes) -- Discover keys via `SCAN MATCH 'leaderboard:*'` and `SCAN MATCH 'distance:*'` against the OLD DB rather than hardcoding city codes — script stays correct if a city is added later. -- Write to NEW DB with prefix prepended: `vngeoguessr:leaderboard:vietnam` etc. -- Idempotent: re-runs produce the same final state. Use `DEL +key` on the destination before re-populating, or rely on `ZADD` overwriting score-by-member (safe). -- `--dry-run` flag: log counts and sample keys, no writes. -- Log effective `KEY_PREFIX` and per-key cardinality so operator can verify alignment with runtime. - -## Architecture - -``` -scripts/migrate-upstash.js ← NEW - Reads: OLD_UPSTASH_REDIS_REST_URL / OLD_UPSTASH_REDIS_REST_TOKEN - Writes: UPSTASH_REDIS_REST_URL / UPSTASH_REDIS_REST_TOKEN (NEW DB, same as runtime) - PREFIX = process.env.KEY_PREFIX ?? 'vngeoguessr:' - - steps: - 1. SCAN old DB for keys matching 'leaderboard:*' and 'distance:*' - 2. For each key: - members = oldClient.zrange(key, 0, -1, { withScores: true }) - newClient.zadd(PREFIX + key, ...members) // multi-add - 3. Log: { source_keys: N, total_members: M, prefix: 'vngeoguessr:' } - 4. Print sample physical keys written so operator can spot-check the dashboard -``` - -Single Node.js file, no test framework needed. Run with `node --env-file=.env.migrate scripts/migrate-upstash.js`. - -### Source/destination protocols (confirmed via `vercel env ls`) - -- OLD DB → TCP only, env var `OLD_STORAGE_REDIS_URL` → script uses `redis` (node-redis) -- NEW DB → REST + TCP; env vars `KV_REST_API_URL` + `KV_REST_API_TOKEN` → script uses `@upstash/redis` - -### `.env.migrate` shape (gitignored; populated via `vercel env pull .env.migrate`) - -``` -OLD_STORAGE_REDIS_URL=redis://... # OLD DB (source, TCP) -KV_REST_API_URL=https://... # NEW DB (dest, REST) -KV_REST_API_TOKEN=... -KEY_PREFIX=vngeoguessr: # optional, defaults to vngeoguessr: -``` - -## Related Code Files - -- Create: `scripts/migrate-upstash.js` -- Modify: `.gitignore` — ensure `.env.migrate` is ignored. **Highlight to user** if `.gitignore` already covers `.env*`. -- No `package.json` script entry needed (one-shot; user invokes directly). - -## Implementation Steps - -1. Write `scripts/migrate-upstash.js`: - - Two `Redis` instances (`oldClient`, `newClient`) using `@upstash/redis`. - - `parseArgs` for `--dry-run`. - - `scanAll(client, match)` helper that paginates `client.scan(cursor, { match, count: 200 })` until cursor returns `0`. - - For each key: `members = await oldClient.zrange(key, 0, -1, { withScores: true })`. The SDK returns `[member, score, member, score, ...]`; reshape into the form `zadd` accepts. - - Multi-add: `newClient.zadd(physicalKey, ...members.map(m => ({ score: m.score, member: m.value })))`. - - Skip empty sets. - - In `--dry-run` mode, do not call `zadd`; print what would be written. -2. Verify `.env.migrate` is gitignored (`grep -F '.env' .gitignore`). If not covered, append `.env.migrate` to `.gitignore` and flag for user. -3. Local run: `node --env-file=.env.migrate scripts/migrate-upstash.js --dry-run`. Verify counts match expected (compare to old DB Upstash dashboard). -4. Real run: `node --env-file=.env.migrate scripts/migrate-upstash.js`. Verify NEW DB dashboard shows prefixed keys with matching cardinality. -5. Read-back compare: pick one user from `leaderboard:vietnam`, confirm same score in NEW DB (`vngeoguessr:leaderboard:vietnam`). - -## Success Criteria - -- [ ] `scripts/migrate-upstash.js` exists; runs cleanly with `--dry-run` and without -- [ ] Effective `KEY_PREFIX` printed at start of run (operator-visible alignment check) -- [ ] All `leaderboard:*` and `distance:*` source keys discovered via SCAN (no hardcoded city list) -- [ ] Per-key member count in NEW DB equals OLD DB -- [ ] Physical keys in NEW DB visibly carry the `vngeoguessr:` prefix -- [ ] Re-run is idempotent: `ZCARD` values unchanged on second run -- [ ] `.env.migrate` is gitignored; not committed - -## Risk Assessment - -- **Risk:** Old DB is the node-redis-protocol flavor and has no REST creds. **Mitigation:** if OLD DB lacks REST endpoint, use `redis` npm package (still installed pre-Phase 1 cleanup) for the read side, REST SDK for write side. Confirm during user data-collection step before writing the script. -- **Risk:** `zrange` cursor / `withScores` SDK shape mismatch. **Mitigation:** log first key's payload before the loop; abort on shape mismatch with a clear error. -- **Risk:** Race — players keep submitting to OLD DB while migration runs (writes get lost in NEW DB). **Mitigation:** Phase 3 cutover swaps env vars in one Vercel redeploy; migration runs at low-traffic time. Acceptable: at most a handful of new entries lost; users can re-play. -- **Risk:** Prefix drift between script and runtime (script uses `vngeoguessr:`, runtime uses `vngeoguessr-prod:` etc.). **Mitigation:** script logs prefix; Phase 3 success criteria includes `/api/leaderboard` returning a non-empty list post-cutover (end-to-end alignment check). -- **Risk:** Sessions in flight at cutover see stale state. **Mitigation:** sessions are 30-min TTL and not migrated; affected players just resubmit. Documented expectation, not a defect. - -## Out of Scope - -- Migrating `session:*` keys (intentionally skipped — ephemeral). -- Updating Vercel env vars or removing the OLD DB integration (Phase 3, user-driven). diff --git a/plans/260509-2234-upstash-db-migration-with-prefix/phase-03-cutover-and-cleanup.md b/plans/260509-2234-upstash-db-migration-with-prefix/phase-03-cutover-and-cleanup.md deleted file mode 100644 index 8554991..0000000 --- a/plans/260509-2234-upstash-db-migration-with-prefix/phase-03-cutover-and-cleanup.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -phase: 3 -title: "Cutover and Cleanup" -status: pending -priority: P1 -effort: "30 min" -dependencies: [1, 2] ---- - -# Phase 3: Cutover and Cleanup - -## Overview - -Deploy the refactored app pointed at the NEW Upstash DB, validate end-to-end, then remove the OLD DB integration and stale dependencies. After this phase, the project depends only on the new shared Upstash DB and the `redis` npm package is gone. - -## Requirements - -- Production reads/writes hit the NEW DB (with prefix) — verified via real user-flow. -- OLD DB integration disconnected from the Vercel project (user-driven). -- `redis` npm dep removed from `package.json` and lockfile. -- Documentation in `docs/tech-stack.md` reflects the new SDK and prefix model. - -## Architecture - -No code architecture changes here — this phase is operations + documentation. Sequence: - -``` -1. Deploy preview with refactor (Phase 1) → smoke test against NEW DB -2. Promote preview → production -3. Validate production: - POST /api/guess (full game flow) → leaderboard updates in NEW DB - GET /api/leaderboard → returns migrated entries -4. Disconnect OLD Upstash integration in Vercel dashboard (user) -5. Remove `redis` dep, run `npm install`, commit lockfile -6. Update docs/tech-stack.md -``` - -## Related Code Files - -- Modify: `package.json` — remove `"redis"` dependency. **Highlight to user** per File Modification Policy. -- Modify: `package-lock.json` — regenerated by `npm install` (user runs). -- Modify: `docs/tech-stack.md` — replace "Redis: Complete session and leaderboard storage" + node-redis line with `@upstash/redis` REST SDK + `KEY_PREFIX` multi-tenant note. -- No source-code changes in this phase. - -## Implementation Steps - -1. **Deploy preview** (user runs `vercel` or pushes to a preview branch). Confirm preview reads from NEW DB. -2. **Smoke test on preview:** - - GET `/api/leaderboard?type=score` → should list migrated users. - - Play one full game → POST `/api/guess` → leaderboard rank updates. - - Optional: check Upstash dashboard for new key like `vngeoguessr:leaderboard:vietnam` with the new score reflected. -3. **Promote to production** (`vercel --prod` or merge to `main`). -4. **Production smoke test** (same as step 2, on the production URL). -5. **Disconnect OLD Upstash integration** — user action in Vercel dashboard: - - Project → Storage → remove the OLD Upstash integration. - - Remove `REDIS_URL` env var if it lingers. -6. **Remove `redis` dep:** - - User runs: `npm uninstall redis` - - Verify `npm ls redis` is empty and `package-lock.json` no longer references `node-redis`. -7. **Update `docs/tech-stack.md`:** replace lines about node-redis with `@upstash/redis` REST SDK; add a sentence noting the `KEY_PREFIX` multi-tenant convention. -8. **Final grep sweep:** - - `grep -r "REDIS_URL\|from 'redis'\|node:redis" src/ scripts/` → empty - - `grep -r "vngeoguessr:" src/` → only in `src/lib/upstash.js` (the default constant) - -## Success Criteria - -- [ ] Production GET `/api/leaderboard` returns migrated entries from NEW DB -- [ ] Production POST `/api/guess` increments score in NEW DB (verified in Upstash dashboard) -- [ ] OLD Upstash integration removed from Vercel project (Storage tab clean) -- [ ] `redis` dep removed from `package.json`; `npm ls redis` empty -- [ ] `docs/tech-stack.md` updated to reflect `@upstash/redis` + `KEY_PREFIX` -- [ ] Final grep sweep clean (no `REDIS_URL` / `from 'redis'` references) -- [ ] `npm run build` passes on a fresh checkout - -## Risk Assessment - -- **Risk:** Production deploy reads from NEW DB before migration script ran ⇒ empty leaderboard for users. **Mitigation:** Phase 2 must complete and be verified before promoting Phase 3 to production. The plan ordering enforces this. -- **Risk:** OLD DB integration removed before production cutover settles ⇒ rollback impossible. **Mitigation:** keep OLD DB connected for at least one full traffic cycle (e.g., 24h) post-cutover before disconnecting. -- **Risk:** `KEY_PREFIX` env var missing in Vercel production ⇒ runtime falls back to default `vngeoguessr:`. This is fine (matches migration default), but if a different prefix was set at runtime, leaderboards appear empty. **Mitigation:** explicitly set `KEY_PREFIX=vngeoguessr:` in Vercel env vars during cutover, even though it's the default. -- **Risk:** `docs/tech-stack.md` drift (forgotten update). **Mitigation:** included in success criteria checklist; reviewer will catch. - -## Out of Scope - -- Adding more projects to the shared Upstash DB (the prefix system enables it; actual onboarding of other projects is separate work). -- Adopting `@upstash/redis` for any feature beyond leaderboard + session. -- Refactoring API route files — they remain untouched throughout. - -## Rollback Plan - -If production smoke fails after promotion: -1. In Vercel dashboard, redeploy the previous commit (one click). -2. Old commit still uses node-redis + `REDIS_URL`; OLD DB still connected → instant rollback. -3. Investigate failure offline, redo Phase 1 / 2 steps as needed before retrying. - -This is why steps 5–7 (cleanup) only happen AFTER production has been stable for a full traffic cycle. diff --git a/plans/260509-2234-upstash-db-migration-with-prefix/plan.md b/plans/260509-2234-upstash-db-migration-with-prefix/plan.md deleted file mode 100644 index b127683..0000000 --- a/plans/260509-2234-upstash-db-migration-with-prefix/plan.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: "Migrate Upstash DB with Project Key Prefix" -description: "Switch vngeoguessr from node-redis pointed at the old Upstash DB to @upstash/redis REST SDK pointed at the new Vercel-Marketplace Upstash DB. Introduce KEY_PREFIX adapter pattern (default vngeoguessr:) so the new DB can be safely shared with other Vercel projects (mirrors store-scraper-bot)." -status: pending -priority: P1 -effort: 2h -branch: main -tags: [vercel, upstash, migration, redis, multi-tenant] -created: 2026-05-09 -blockedBy: [] -blocks: [] ---- - -# Migrate Upstash DB with Project Key Prefix - -## Goal - -Cut vngeoguessr over from old Upstash DB → new Upstash DB while introducing a project-scoped key prefix. After this plan, the same Upstash DB can host data for multiple Vercel projects without key collisions. - -## Current State - -- Connection: `src/lib/redis.js` uses `redis@5.8.0` (TCP) via `process.env.REDIS_URL` — a single global client. -- Both old and new Upstash DBs are already linked to the Vercel project (per user); env vars exist for both. New DB connection must come via the Vercel Marketplace Upstash integration (`KV_REST_API_URL` / `KV_REST_API_TOKEN` or `UPSTASH_REDIS_REST_URL` / `UPSTASH_REDIS_REST_TOKEN`). -- Logical key namespaces in use: - - `session:{sessionId}` — string, TTL 30 min (`src/lib/session.js`) - - `leaderboard:vietnam`, `leaderboard:city:{cityCode}` — sorted sets (`src/lib/leaderboard.js`) - - `distance:vietnam`, `distance:city:{cityCode}` — sorted sets (`src/lib/leaderboard.js`) -- Files touching Redis: `src/lib/redis.js`, `src/lib/session.js`, `src/lib/leaderboard.js`, `src/app/api/guess/route.js`, `src/app/api/cron/route.js`, `src/app/api/leaderboard/route.js`. - -## Target State - -- Connection: `@upstash/redis` REST SDK; client + prefix bundled in a single handle returned by `createUpstashClient(env)`. -- Default `KEY_PREFIX = 'vngeoguessr:'`. All physical keys carry it; logical keys at call sites stay unchanged. -- No `process.env.REDIS_URL` references remain. `redis` npm dep removed. -- One-shot migration script copies sorted-set leaderboard data from old DB → new DB, prepending the prefix on write. Sessions are not migrated (30-min TTL, ephemeral). - -## Key Decisions - -- **REST SDK over node-redis.** Vercel Marketplace Upstash exposes REST creds; `@upstash/redis` is fluid-compute friendly (no socket pooling), matches the store-scraper-bot reference. Sorted-set commands (`zadd`, `zrange`, `zscore`, `zrank`, `zrevrank`, `zremrangebyrank`) are all supported. -- **Adapter shape mirrors store-scraper-bot.** `createUpstashClient(env) → { client, prefix }`. Helper functions accept the handle and prepend the prefix. Repositories stay prefix-unaware. Difference vs store-scraper-bot: we expose sorted-set helpers (`zAdd`, `zRange`, `zScore`, `zRank`, `zRevRank`, `zRemRangeByRank`) in addition to JSON `get/set/del`. -- **Env var naming.** Adapter accepts both `UPSTASH_REDIS_REST_URL` / `UPSTASH_REDIS_REST_TOKEN` and `KV_REST_API_URL` / `KV_REST_API_TOKEN` (Vercel Marketplace alias) — same as store-scraper-bot. -- **Migration scope.** Copy leaderboard sorted sets only. Sessions are not migrated. Old DB stays online briefly as fallback during cutover, then gets disconnected from Vercel by the user. -- **Prefix value.** `vngeoguessr:`. Single source of truth; only override if explicitly multi-tenant beyond this project. - -## Phases - -| # | Phase | Effort | Status | -|---|---|---|---| -| 01 | [Adapter Refactor with Prefix](./phase-01-adapter-refactor-with-prefix.md) | 1h | pending | -| 02 | [Data Migration old → new Upstash](./phase-02-data-migration.md) | 30 min | pending | -| 03 | [Cutover and Cleanup](./phase-03-cutover-and-cleanup.md) | 30 min | pending | - -## Success Criteria (overall) - -- App reads/writes against the new Upstash DB via REST SDK. -- All physical keys in new DB carry the `vngeoguessr:` prefix. -- Leaderboard counts (per sorted set, per city) match between old and new DB after migration. -- `npm ls redis` returns nothing; `package.json` lists `@upstash/redis` instead. -- `grep -r "REDIS_URL\|node:redis\|from 'redis'" src/` returns zero hits. -- Production smoke: submit a guess, view leaderboard → both succeed against the new DB. - -## Manual Steps Required (User) - -These cannot be performed by the agent and must be done by the user: -- Confirm the new Upstash DB env vars present in Vercel project (preview + production). -- Set `KEY_PREFIX=vngeoguessr:` in Vercel env (preview + production). Optional but recommended to make the prefix explicit; default also works. -- Provide the OLD DB REST URL+token to a local `.env.migrate` (read-only, for one-shot migration script). -- After cutover validated, disconnect the old Upstash DB from the Vercel project. - -## Dependencies - -- `@upstash/redis` npm package (added in Phase 1). -- New Upstash DB already linked via Vercel Marketplace (per user — confirmed pre-plan). diff --git a/plans/260510-0140-port-viguessr-mapillary/phase-01-per-city-delta-random-point-fetch.md b/plans/260510-0140-port-viguessr-mapillary/phase-01-per-city-delta-random-point-fetch.md deleted file mode 100644 index 5fc275f..0000000 --- a/plans/260510-0140-port-viguessr-mapillary/phase-01-per-city-delta-random-point-fetch.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -phase: 1 -title: "Per-city Delta + Random-Point Fetch" -status: pending -priority: P1 -effort: "30m" -dependencies: [] ---- - -# Phase 1: Per-city Delta + Random-Point Fetch - -## Overview - -Replace the adaptive bbox loop in `src/lib/mapillary.js` with viguessr's random-point dart-throw using a per-city fixed `delta`. Add `mapillaryDelta` to each `CITIES` entry. Update `new-game` route to pass delta and use `thumb_original_url`. - -## Related Code Files - -- Modify: `src/lib/game.js` — add `mapillaryDelta` field on each `CITIES` entry; export `cityDeltas` -- Rewrite: `src/lib/mapillary.js` — random-point + fixed-delta + retry-position -- Modify: `src/app/api/new-game/route.js` — import `cityDeltas`, pass to fetch, switch back to `thumb_original_url` - -## Implementation Steps - -### 1. `src/lib/game.js` - -Inside each `CITIES.` object, add a numeric field `mapillaryDelta`: - -```js -HN: { ..., mapillaryDelta: 0.003 }, -TPHCM: { ..., mapillaryDelta: 0.005 }, -DL: { ..., mapillaryDelta: 0.005 }, -DH: { ..., mapillaryDelta: 0.005 }, -DN: { ..., mapillaryDelta: 0.005 }, // disabled but populated -``` - -Below the existing `cityBboxes` export, add: - -```js -export const cityDeltas = Object.fromEntries( - Object.values(CITIES).map(city => [city.code, city.mapillaryDelta]) -); -``` - -### 2. `src/lib/mapillary.js` (full rewrite) - -```js -// Mapillary API utilities -// -// Strategy ported from viguessr (https://github.com/luuvanduc1999/viguessr): -// pick a random point inside the city bbox, query a small fixed-size sub-bbox -// (side = 2*delta) centered on that point, and re-roll the point on empty -// results or transient 5xx. The fixed small size keeps Mapillary query cost -// below their cap; the dart-throw eventually lands on a pano-rich window. - -const MAX_RETRIES = 10; - -export async function fetchMapillaryImages(bbox, delta) { - const accessToken = process.env.MAPILLARY_ACCESS_TOKEN; - if (!accessToken) { - throw new Error('MAPILLARY_ACCESS_TOKEN environment variable is not set'); - } - - const [minLng, minLat, maxLng, maxLat] = bbox; - let lastError = null; - - for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) { - const lat = Math.random() * (maxLat - minLat) + minLat; - const lng = Math.random() * (maxLng - minLng) + minLng; - const queryBbox = [ - (lng - delta).toFixed(4), - (lat - delta).toFixed(4), - (lng + delta).toFixed(4), - (lat + delta).toFixed(4), - ].join(','); - const apiUrl = `https://graph.mapillary.com/images?access_token=${accessToken}&fields=id,thumb_original_url,geometry,is_pano&limit=3&bbox=${queryBbox}&is_pano=true`; - - try { - const response = await fetch(apiUrl, { - headers: { Accept: 'application/json' }, - next: { revalidate: 0 }, - }); - - if (response.status === 401) { - throw new Error('Mapillary authentication failed'); - } - - if (!response.ok) { - const body = await response.text().catch(() => ''); - lastError = `${response.status}: ${body.slice(0, 200)}`; - console.error(`Attempt ${attempt}/${MAX_RETRIES}: Mapillary ${response.status}; re-rolling`); - continue; - } - - const data = await response.json(); - const images = data.data || []; - const panos = images.filter(img => img.is_pano); - - if (panos.length > 0) { - console.log(`Found ${panos.length} panos on attempt ${attempt}`); - return { success: true, data: panos }; - } - - console.log(`Attempt ${attempt}/${MAX_RETRIES}: no panos; re-rolling`); - } catch (error) { - if (error.message === 'Mapillary authentication failed') { - throw error; - } - lastError = error.message; - console.error(`Attempt ${attempt}/${MAX_RETRIES} threw: ${error.message}; re-rolling`); - } - } - - return { - success: false, - error: lastError - ? `No panos after ${MAX_RETRIES} attempts (last: ${lastError})` - : 'No panos found', - }; -} -``` - -Removed: -- `MAX_BBOX_AREA` / `MIN_BBOX_AREA` / `GROW_FACTOR` / `SHRINK_FACTOR` -- `getRandomSubBbox` helper (replaced inline) -- Adaptive shrink/grow loop - -### 3. `src/app/api/new-game/route.js` - -Update imports + call: - -```js -// Before: -import { cityNames, cityBboxes } from '../../../lib/game.js'; -const bbox = cityBboxes[cityCode]; -const imageResult = await fetchMapillaryImages(bbox); -const imageUrl = selectedImage.thumb_2048_url; - -// After: -import { cityNames, cityBboxes, cityDeltas } from '../../../lib/game.js'; -const bbox = cityBboxes[cityCode]; -const delta = cityDeltas[cityCode]; -const imageResult = await fetchMapillaryImages(bbox, delta); -const imageUrl = selectedImage.thumb_original_url; -``` - -Add a guard for missing delta: - -```js -if (!delta) { - return NextResponse.json({ - success: false, - error: `No Mapillary delta configured for city: ${cityCode}`, - }, { status: 400 }); -} -``` - -## Success Criteria - -- [ ] `npm run build` passes -- [ ] HN: 3 sample requests → ≥2 success -- [ ] TPHCM: 3 sample requests → ≥2 success -- [ ] DL: 3 sample requests → ≥2 success -- [ ] DH: 3 sample requests → ≥2 success -- [ ] No 'reduce data' 500s in production logs after deploy -- [ ] `grep "MAX_BBOX_AREA\|GROW_FACTOR\|SHRINK_FACTOR" src/` empty - -## Risk Assessment - -- **Risk**: viguessr's HN delta=0.003 (666m) might still occasionally hit 'reduce data' if Mapillary tightens further. **Mitigation**: 10 retries; if persistent, drop HN delta to 0.002. -- **Risk**: Per-city delta requires a value for each enabled city; missing entry crashes. **Mitigation**: explicit guard in route; default fallback to 0.005 if undefined. -- **Risk**: `thumb_original_url` may be deprecated. **Mitigation**: viguessr is shipping it in production today; if Mapillary deprecates, swap to `thumb_2048_url`. -- **Risk**: Rare cities with sparse pano coverage (DH 666m windows) might exhaust retries. **Mitigation**: keep delta=0.005 (1.1km) for non-HN cities. - -## Rollback - -Single commit; revert HEAD to restore current adaptive code. Mapillary deploys with no schema/state changes, so rollback is instant. diff --git a/plans/260510-0140-port-viguessr-mapillary/plan.md b/plans/260510-0140-port-viguessr-mapillary/plan.md deleted file mode 100644 index e05f8f7..0000000 --- a/plans/260510-0140-port-viguessr-mapillary/plan.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: "Port viguessr Mapillary strategy" -description: "Replace adaptive shrink/grow bbox loop with viguessr's proven approach: per-city fixed `delta` (half-bbox-side) + random-point dart-throw + simple retry-with-new-point. Fixes 'reduce data' 500s in HN/HCM dense areas." -status: pending -priority: P1 -effort: 30m -created: 2026-05-10 -source: https://github.com/luuvanduc1999/viguessr (assets/js/game.js, obfuscated) -blockedBy: [] -blocks: [] ---- - -# Port viguessr Mapillary strategy - -## Why - -Current `src/lib/mapillary.js` adaptive bbox sizing fails consistently in dense HN/HCM — Mapillary returns 500 'reduce data' when query bbox costs too much to scan. Adaptive grow on empty results worsens this. - -viguessr (a sibling vngeoguessr project) ships a working implementation for the same cities. Their approach: **per-city fixed `delta`**, query a small bbox centered on a random point, retry only by re-rolling the random point (never resizing the bbox). The small fixed bbox keeps Mapillary query cost under their cap; the dart-throw eventually finds pano-rich windows. - -## Source decoded (from obfuscated game.js) - -```js -boundingBoxVN = { - 'HN': [105.77, 20.96, 105.88, 21.05, delta=0.003], - 'TPHCM': [106.62, 10.71, 106.75, 10.83, delta=0.005], - 'HP': [106.65, 20.80, 106.75, 20.90, delta=0.05], - 'ND': [106.00, 20.35, 106.25, 20.50, delta=0.005], - 'DN': [108.17, 16.00, 108.25, 16.10, delta=0.005], -}; - -async function getRandomMapillaryImage() { - const lat = Math.random() * (maxLat - minLat) + minLat; - const lng = Math.random() * (maxLong - minLong) + minLong; - const bbox = [lng-delta, lat-delta, lng+delta, lat+delta].map(n => n.toFixed(4)).join(','); - const url = `https://graph.mapillary.com/images?access_token=${TOKEN}&fields=id,thumb_original_url,geometry,is_pano&limit=3&bbox=${bbox}&is_pano=true`; - const json = await (await fetch(url)).json(); - if (json.data?.length) { - const panos = json.data.filter(d => d.is_pano); - return panos.length ? panos[rand(panos.length)] : json.data[0]; - } - return getRandomMapillaryImage(); // re-roll, same delta -} -``` - -## What changes locally - -### `src/lib/game.js` (CITIES schema) - -Add `mapillaryDelta` field to each enabled `CITIES` entry: - -| Code | New `mapillaryDelta` | Source rationale | -|---|---|---| -| HN | `0.003` | viguessr value | -| TPHCM | `0.005` | viguessr value | -| DL | `0.005` | not in viguessr; use TPHCM-class default | -| DH | `0.005` | not in viguessr; use TPHCM-class default | -| DN | `0.005` (still disabled) | viguessr value, future-proof | - -Export a `cityDeltas` map alongside existing `cityBboxes`/`cityNames`/`cityCenters`. - -### `src/lib/mapillary.js` (rewrite) - -Replace adaptive loop with viguessr-style: random point inside city bbox, fixed-size sub-bbox of side `2*delta`, re-roll position on empty/error. - -Signature change: `fetchMapillaryImages(bbox, delta)`. - -### `src/app/api/new-game/route.js` - -- Import `cityDeltas` from `lib/game.js`. -- Pass `cityDeltas[cityCode]` to `fetchMapillaryImages`. -- Use `selectedImage.thumb_original_url` (revert from `thumb_2048_url`). - -## Phases - -| # | Phase | Effort | Status | -|---|---|---|---| -| 01 | [Per-city Delta + Random-Point Fetch](./phase-01-per-city-delta-random-point-fetch.md) | 30m | pending | - -## Success Criteria - -- HN, TPHCM, DL, DH all return panoramic image on first request (sample ≥ 3 attempts each). -- No `MAX_BBOX_AREA` / `MIN_BBOX_AREA` / shrink/grow logic remains in `src/lib/mapillary.js`. -- `npm run build` passes. - -## Out of scope - -- Token rotation or rate-limit handling -- Caching successful responses -- Adding more cities (DN remains disabled, future work) - -## Manual cleanup after success - -- Remove the diagnostic `console.error` of Mapillary URL/body once confirmed stable (or keep as observability). diff --git a/plans/260510-0204-cleanup-upstash-migration/phase-01-cleanup.md b/plans/260510-0204-cleanup-upstash-migration/phase-01-cleanup.md deleted file mode 100644 index be0fa40..0000000 --- a/plans/260510-0204-cleanup-upstash-migration/phase-01-cleanup.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -phase: 1 -title: "Cleanup" -status: pending -priority: P2 -effort: "15m" -dependencies: [] ---- - -# Phase 1: Cleanup - -## Overview - -Single-pass cleanup of Upstash migration scaffolding. No code logic changes, only file/dependency removal and plan-status update. - -## Related Code Files - -- Modify: `package.json` (remove `redis` dep) — **highlight to user** per File Modification Policy -- Modify: `package-lock.json` (regenerated by `npm uninstall`) — **highlight to user** -- Delete: `scripts/migrate-upstash.js` -- Delete: `.env.migrate` (local only, gitignored) -- Update: `plans/260509-2234-upstash-db-migration-with-prefix/phase-03-cutover-and-cleanup.md` (mark complete via `ck plan check 3`) - -## Implementation Steps - -1. **User runs**: `npm uninstall redis` - - Verify: `npm ls redis` returns empty. - - This regenerates `package-lock.json`. -2. **Agent deletes**: `scripts/migrate-upstash.js`. - - Recoverable from git history (commit `53ee795`) if ever needed. - - The directory `scripts/` will be left empty; remove the directory too. -3. **Agent deletes**: local `.env.migrate`. - - Contains old DB credentials no longer needed; sensitive even if gitignored. -4. **Agent runs**: `cd plans/260509-2234-upstash-db-migration-with-prefix && ck plan check 3` to mark Phase 3 of the migration plan complete. -5. **Agent runs**: `npm run build` to confirm nothing else depends on `redis`. -6. **Agent commits + pushes** the deletion changes (single commit). - -## What stays untouched - -- Vercel env vars `REDIS_URL`, `KV_URL`, `KV_REST_API_READ_ONLY_TOKEN` — auto-provisioned by the Vercel Marketplace Upstash integration. Removing causes drift. -- `src/lib/mapillary.js` console.error lines — kept-on-purpose error observability, not temporary instrumentation. -- The migration plan files themselves (kept as record). - -## Success Criteria - -- [ ] `npm ls redis` empty -- [ ] `scripts/migrate-upstash.js` not present -- [ ] `.env.migrate` not present locally -- [ ] `npm run build` passes -- [ ] Original migration plan Phase 3 marked completed -- [ ] Single commit pushed to main - -## Risk Assessment - -- **Risk**: deleting the migration script loses the bridge if a re-migration becomes necessary. **Mitigation**: git history preserves it (commit `53ee795`). Restoration is `git checkout 53ee795 -- scripts/migrate-upstash.js`. -- **Risk**: removing `redis` dep breaks something we missed. **Mitigation**: `npm run build` step after uninstall surfaces any stray imports. Grep already confirmed no `from 'redis'` or `require('redis')` references in `src/`. -- **Risk**: `.env.migrate` deletion is irreversible. **Mitigation**: file just contains creds copy-pastable from Vercel dashboard if ever needed; no original-source data. - -## Out of Scope - -- Removing Vercel auto-provisioned env vars. -- Refactoring observability in `mapillary.js`. -- Any change to the new Upstash integration or its credentials. diff --git a/plans/260510-0204-cleanup-upstash-migration/plan.md b/plans/260510-0204-cleanup-upstash-migration/plan.md deleted file mode 100644 index 2ca8936..0000000 --- a/plans/260510-0204-cleanup-upstash-migration/plan.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: "Cleanup Upstash migration artifacts" -description: "Remove leftover artifacts from the completed Upstash migration: legacy redis npm dep, one-shot migration script, local .env.migrate, and close out the original migration plan's Phase 3." -status: pending -priority: P2 -effort: 15m -created: 2026-05-10 -related: 260509-2234-upstash-db-migration-with-prefix (closes Phase 3) -blockedBy: [] -blocks: [] ---- - -# Cleanup Upstash migration artifacts - -## Goal - -The Upstash migration is complete and verified in production (15 leaderboard users, 425 sorted-set members migrated, all 4 cities returning panoramas). Old Upstash integration is disconnected from Vercel. This plan retires the migration scaffolding. - -## Current state (verified) - -| Artifact | State | -|---|---| -| `redis` npm dep | still in `package.json` ^5.8.0 (last needed by migration script) | -| `scripts/migrate-upstash.js` | committed; one-shot job done | -| `.env.migrate` | exists locally (gitignored) with old + new DB credentials | -| `OLD_STORAGE_REDIS_URL` env in Vercel | already removed by user | -| `REDIS_URL` env in Vercel | auto-provisioned by Marketplace integration; unused by app | -| `KV_URL` env in Vercel | auto-provisioned; unused | -| `KV_REST_API_READ_ONLY_TOKEN` env in Vercel | auto-provisioned; unused | -| `KV_REST_API_URL` / `KV_REST_API_TOKEN` env in Vercel | **in active use** — keep | -| Mapillary diagnostic logging | already removed (viguessr rewrite replaced it; current logs are kept-on-purpose error observability) | -| Plan `260509-2234-upstash-db-migration-with-prefix` Phase 3 | marked in-progress in tracker; needs close | - -## What this plan does - -| Action | Reason | -|---|---| -| `npm uninstall redis` | App uses `@upstash/redis` exclusively; node-redis no longer needed | -| Delete `scripts/migrate-upstash.js` | One-shot done; recoverable from git history if ever needed | -| Delete local `.env.migrate` | Contains old DB credentials no longer in use; sensitive even if gitignored | -| Mark migration plan Phase 3 complete | Operational status accuracy | - -## What this plan does NOT do - -- **Don't touch `REDIS_URL`, `KV_URL`, `KV_REST_API_READ_ONLY_TOKEN`** — they're auto-provisioned by the Vercel Marketplace Upstash integration. Removing them via CLI causes drift; integration may re-sync them. They're harmless when unused. -- Don't change the diagnostic-but-still-useful console.error lines in `src/lib/mapillary.js` — those provide error observability for the Mapillary 500s and aren't temporary instrumentation. - -## Phases - -| # | Phase | Effort | Status | -|---|---|---|---| -| 01 | [Cleanup](./phase-01-cleanup.md) | 15m | pending | - -## Success Criteria - -- `npm ls redis` returns nothing -- `scripts/` directory empty (or removed entirely) -- `.env.migrate` not present locally -- `npm run build` passes -- Original migration plan reports Phase 3 completed