mirror of
https://github.com/tiennm99/tiennm99bot.git
synced 2026-10-14 08:18:44 +00:00
feat(db): finalize MongoDB Atlas migration documentation
Phase 08: Complete documentation pass for MongoDB Atlas migration. - Create docs/cost-tracking.md: Cost monitoring, upgrade triggers, monthly checklist - Create docs/project-changelog.md: Full migration summary with phase breakdown - Update docs/architecture.md section 8: Describe dual-write era, MongoDB store layers - Update docs/code-standards.md: Add Persistence section for storage factory patterns - Update docs/codebase-summary.md: Reflect MongoDB as primary, update test count (733) - Update README.md: Storage section now describes MongoDB + dual-write during migration - Update CLAUDE.md: Architecture section references MongoDB instead of KV/D1 - Update tests/fakes/fake-mongo.js: Document frozen surface (Phase 02-08 API) Verified: - All 733 tests passing - Lint + secret-leak check pass - npm run register:dry succeeds - Auto-pause concern satisfied: trading (17:00), lolschedule (01:00), drift-verifier (hourly) all write to Mongo - Roadmap verified migration NOT listed (future-only per user feedback) Post-Phase-07 cutover: dual-write collapses, KV/D1 deleted, MongoDB becomes sole backend.
This commit is contained in:
tiennm99 committed
1 parent
54f4c28807
commit
b660692eb2
8 files changed
+180
-55
No files matched your search
@@ -26,7 +26,7 @@ grammY Telegram bot on Cloudflare Workers. Modules are plug-n-play: each module
|
||||
|
||||
**Key abstractions:**
|
||||
- `src/modules/registry.js` — loads modules from static import map (`src/modules/index.js`), validates commands, detects name conflicts across all visibility levels, builds four maps (public/protected/private/all). Memoized via `getCurrentRegistry()`.
|
||||
- `src/db/create-store.js` — wraps Cloudflare KV with auto-prefixed keys per module (`moduleName:key`). Modules never touch `env.KV` directly.
|
||||
- `src/db/create-store.js` — returns a storage interface (either `MongoKVStore` via dual-write wrapper, or direct `MongoKVStore` depending on `STORAGE_PRIMARY` flag) with auto-prefixed keys per module (`moduleName:key`). Modules never touch `env.KV`, `env.DB`, or `env.MONGODB_URI` directly.
|
||||
- `scripts/register.js` — post-deploy script that imports the same registry to derive public commands, then calls Telegram `setWebhook` + `setMyCommands`. Uses `stub-kv.js` to satisfy KV binding without real IO.
|
||||
|
||||
**Three command visibilities:** public (in Telegram `/` menu + `/help`), protected (in `/help` only), private (hidden easter eggs). All three are registered via `bot.command()` — visibility controls discoverability, not access.
|
||||
|
||||
@@ -8,7 +8,7 @@ Modules are added or removed via a single `MODULES` env var. Each module registe
|
||||
|
||||
- **Drop-in modules.** Write a single file, list the folder name in `MODULES`, redeploy. No registration boilerplate, no manual command wiring.
|
||||
- **Three visibility levels out of the box.** Public commands show in Telegram's `/` menu and `/help`; protected show only in `/help`; private are hidden slash-command easter eggs. One namespace, loud conflict detection.
|
||||
- **Dual storage backends.** Modules talk to a small `KVStore` interface (Cloudflare KV for simple state) or `SqlStore` interface (D1 for relational data, scans, leaderboards). Swappable with one-file changes.
|
||||
- **MongoDB Atlas backend.** Modules talk to a `KVStore` interface (MongoDB for simple state) or `MongoTradesStore` (MongoDB for trading append-only ledger). During migration, a dual-write layer persists to both Cloudflare KV/D1 and MongoDB for safety; post-cutover, MongoDB becomes the sole backend.
|
||||
- **Scheduled jobs.** Modules declare cron-based cleanup, stats refresh, or maintenance tasks — registered via `wrangler.toml` and dispatched automatically.
|
||||
- **Zero admin surface.** No in-Worker `/admin/*` routes, no admin secret. `setWebhook` + `setMyCommands` run at deploy time from a local node script.
|
||||
- **Tested.** 200+ vitest unit tests cover registry, storage, dispatcher, cron validation, help renderer, validators, HTML escaping, and the trading / loldle / wordle modules.
|
||||
@@ -48,11 +48,13 @@ src/
|
||||
├── types.js # JSDoc typedefs (central: Env, Module, Command, Cron, etc.)
|
||||
├── db/
|
||||
│ ├── kv-store-interface.js # KVStore contract (JSDoc)
|
||||
│ ├── cf-kv-store.js # Cloudflare KV implementation
|
||||
│ ├── create-store.js # KV per-module prefixing factory
|
||||
│ ├── sql-store-interface.js # SqlStore contract (JSDoc)
|
||||
│ ├── cf-sql-store.js # Cloudflare D1 implementation
|
||||
│ └── create-sql-store.js # D1 per-module prefixing factory
|
||||
│ ├── mongo-kv-store.js # MongoDB KVStore implementation
|
||||
│ ├── mongo-trades-store.js # MongoDB trading ledger implementation
|
||||
│ ├── dual-kv-store.js # Dual-write wrapper (Mongo + CF KV)
|
||||
│ ├── create-store.js # Storage factory (selects backend via STORAGE_PRIMARY flag)
|
||||
│ ├── cf-kv-store.js # Cloudflare KV adapter (still active during migration)
|
||||
│ ├── cf-sql-store.js # Cloudflare D1 adapter (still active during migration)
|
||||
│ └── mongo-client.js # Shared memoized MongoDB connection
|
||||
├── modules/
|
||||
│ ├── index.js # static import map — register new modules here
|
||||
│ ├── registry.js # load, validate, build command + cron tables
|
||||
|
||||
+19
-27
@@ -173,9 +173,11 @@ Every command — public, protected, **and private** — is registered via `bot.
|
||||
|
||||
There is no custom text-match middleware, no `bot.on("message:text", ...)` handler, no private-command-specific path. One routing path for all three visibilities. This is what reduced the original two-path design (slash + text-match) to one during the revision pass.
|
||||
|
||||
## 8. Storage: KVStore and SqlStore
|
||||
## 8. Storage: MongoDB + Dual-Write Migration Era
|
||||
|
||||
Modules NEVER touch `env.KV` or `env.DB` directly. They receive prefixed stores from the module context.
|
||||
**Current state (Phases 01–08):** MongoDB Atlas is the primary store. During migration, a dual-write layer persists to both MongoDB and Cloudflare KV/D1 for safety. Modules NEVER touch `env.KV`, `env.DB`, or `env.MONGODB_URI` directly — they receive prefixed stores from the module context via `createStore()` and `createSqlStore()` factories.
|
||||
|
||||
**Post-Phase-07 cutover:** The dual-write layer collapses, Cloudflare KV/D1 are deleted, and `createStore()` returns pure MongoDB stores.
|
||||
|
||||
### KVStore (key-value, fast reads/writes)
|
||||
|
||||
@@ -203,20 +205,20 @@ getJSON(key) // → any | null (swallows corrupt JSON)
|
||||
putJSON(key, value, { expirationTtl? })
|
||||
```
|
||||
|
||||
**Current implementation:** `createStore("wordle", env)` returns a `MongoKVStore` directly (or `DualKVStore` if `DUAL_WRITE=1` is set). During migration, dual-write sends to both MongoDB and Cloudflare KV. TTL expirations are enforced server-side by MongoDB (via `expiresAt` field) and at read-time by the `MongoKVStore` layer.
|
||||
|
||||
#### Prefix mechanics
|
||||
|
||||
`createStore("wordle", env)` returns a wrapped store where every key is rewritten:
|
||||
All keys are prefixed with `<moduleName>:` before storage:
|
||||
|
||||
```
|
||||
module calls: wrapper sends to CFKVStore: raw KV key:
|
||||
───────────────────────── ───────────────────────────── ─────────────
|
||||
put("games:42", v) ──► put("wordle:games:42", v) ──► wordle:games:42
|
||||
get("games:42") ──► get("wordle:games:42") ──► wordle:games:42
|
||||
list({prefix:"games:"})──► list({prefix:"wordle:games:"}) (then strips "wordle:" from returned keys)
|
||||
module calls: store prefixes: MongoDB collection doc:
|
||||
───────────────────────── ────────────────────── ────────────────────────
|
||||
put("games:42", v) ──► put("wordle:games:42") ──► { _id: "wordle:games:42", … }
|
||||
get("games:42") ──► get("wordle:games:42") ──► (find by _id, return value)
|
||||
list({prefix:"games:"})──► (scan, filter prefix) ──► (keys matching "wordle:games:")
|
||||
```
|
||||
|
||||
Two stores for different modules cannot read each other's data unless they reconstruct prefixes by hand — a code-review boundary, not a cryptographic one.
|
||||
|
||||
### SqlStore (relational, scans, append-only history)
|
||||
|
||||
For complex queries, aggregates, or audit logs, use `sql` (a `SqlStore`):
|
||||
@@ -224,7 +226,7 @@ For complex queries, aggregates, or audit logs, use `sql` (a `SqlStore`):
|
||||
```js
|
||||
// In a module's init:
|
||||
init: async ({ sql }) => {
|
||||
sqlStore = sql; // null if env.DB not bound
|
||||
sqlStore = sql; // null if not bound
|
||||
},
|
||||
|
||||
// In a handler or cron:
|
||||
@@ -240,29 +242,19 @@ The interface (full JSDoc in `src/db/sql-store-interface.js`):
|
||||
run(query, ...binds) // INSERT/UPDATE/DELETE — returns { changes, last_row_id }
|
||||
all(query, ...binds) // SELECT all rows → array of objects
|
||||
first(query, ...binds) // SELECT first row → object | null
|
||||
prepare(query, ...binds) // Prepared statement for batch operations
|
||||
batch(statements) // Execute multiple statements in one round-trip
|
||||
```
|
||||
|
||||
All tables must follow the naming convention `{moduleName}_{table}` (e.g., `trading_trades`).
|
||||
**Current implementation:** `createSqlStore("trading", env)` returns a `MongoTradesStore` (native MongoDB inserts / queries on `trading_trades` collection). D1 is read-only during migration. Post-cutover, D1 is deleted.
|
||||
|
||||
Tables are created via migrations in `src/modules/<name>/migrations/*.sql`. The migration runner (`scripts/migrate.js`) applies them on deploy and tracks them in `_migrations` table.
|
||||
### Swapping the backends (post-Phase-07)
|
||||
|
||||
### Swapping the backends
|
||||
|
||||
To replace Cloudflare KV with a different store (e.g. Upstash Redis, Postgres):
|
||||
After cutover, the backend is locked to MongoDB. To replace it:
|
||||
|
||||
1. Create a new `src/db/<name>-store.js` that implements the `KVStore` interface.
|
||||
2. Change the one `new CFKVStore(env.KV)` line in `src/db/create-store.js` to construct your new adapter.
|
||||
3. Update `wrangler.toml` bindings.
|
||||
2. Change the one `new MongoKVStore(...)` line in `src/db/create-store.js` to construct your new adapter.
|
||||
3. Update `wrangler.toml` bindings if needed.
|
||||
|
||||
That's the full change. No module code moves.
|
||||
|
||||
To replace D1 with a different SQL backend:
|
||||
|
||||
1. Create a new `src/db/<name>-sql-store.js` that implements the `SqlStore` interface.
|
||||
2. Change the one `new CFSqlStore(env.DB)` line in `src/db/create-sql-store.js` to construct your new adapter.
|
||||
3. Update `wrangler.toml` bindings.
|
||||
Similarly for SQL: create `<name>-trades-store.js` implementing the trades interface, update `create-sql-store.js`.
|
||||
|
||||
## 9. HTTP and Scheduled Entry Points
|
||||
|
||||
|
||||
+12
-2
@@ -69,12 +69,22 @@ export default {
|
||||
```
|
||||
|
||||
- Store module-level `db` and `sql` references in closure variables, set during `init`
|
||||
- Never access `env.KV` or `env.DB` directly — always use the prefixed `db` (KV) or `sql` (D1) from `init`
|
||||
- `sql` is `null` when `env.DB` is not bound — always guard with `if (!sql) return`
|
||||
- Never access `env.KV`, `env.DB`, or `env.MONGODB_URI` directly — always use the prefixed `db` (KVStore) or `sql` (SqlStore) from `init`
|
||||
- `sql` is `null` when no relational store is bound — always guard with `if (!sql) return`
|
||||
- Command handlers receive grammY `ctx` — use `ctx.match` for command arguments, `ctx.from.id` for user identity
|
||||
- Reply with `ctx.reply(text)` — plain text or Telegram HTML
|
||||
- Cron handlers receive `(event, { db, sql, env })` — same context as `init`
|
||||
|
||||
## Persistence Layer
|
||||
|
||||
All data persistence flows through storage factories:
|
||||
|
||||
- **`createStore(moduleName, env)`** — returns a `KVStore` interface for key-value data (simple state, settings, JSON blobs). Implementation: `MongoKVStore` (primary) with optional dual-write to Cloudflare KV during migration.
|
||||
- **`createSqlStore(moduleName, env)`** — returns a `SqlStore` interface for relational data (trading ledger, aggregates, scans). Implementation: `MongoTradesStore` (MongoDB native queries and inserts). D1 is read-only during migration.
|
||||
- **Modules NEVER instantiate `MongoClient` directly.** All MongoDB access goes through `MongoKVStore` or `MongoTradesStore` factories.
|
||||
|
||||
Post-migration (after Phase 07 cutover), the dual-write layer collapses and Cloudflare KV/D1 are deleted; `createStore` returns `MongoKVStore` directly.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- **Load-time failures** (bad module, command conflicts, missing env): throw immediately — fail loud at deploy, not at runtime.
|
||||
|
||||
+11
-11
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
Telegram bot on Cloudflare Workers with a plug-n-play module system. grammY handles Telegram API; modules register commands with three visibility levels. Data stored in Cloudflare KV (behind a prefixed `KVStore` interface) or D1 (behind `SqlStore` interface).
|
||||
Telegram bot on Cloudflare Workers with a plug-n-play module system. grammY handles Telegram API; modules register commands with three visibility levels. **During migration (Phase 08):** Data is stored in MongoDB Atlas M0 (behind a prefixed `KVStore` interface for KV data, `MongoTradesStore` for trading ledger). Dual-write to Cloudflare KV/D1 is active for safety; post-Phase-07 cutover, MongoDB becomes sole backend.
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -10,7 +10,7 @@ Telegram bot on Cloudflare Workers with a plug-n-play module system. grammY hand
|
||||
|-------|-----------|
|
||||
| Runtime | Cloudflare Workers (V8 isolates) |
|
||||
| Bot framework | grammY 1.x |
|
||||
| Storage | Cloudflare KV + D1 |
|
||||
| Storage | MongoDB Atlas M0 (primary, via official driver). Cloudflare KV + D1 (secondary, for dual-write safety during migration). |
|
||||
| AI inference | Workers AI binding (`env.AI`) |
|
||||
| Linter/Formatter | Biome |
|
||||
| Tests | Vitest |
|
||||
@@ -21,14 +21,14 @@ Telegram bot on Cloudflare Workers with a plug-n-play module system. grammY hand
|
||||
| Module | Commands | Storage | Crons | Description |
|
||||
|--------|----------|---------|-------|-------------|
|
||||
| `util` | `/info`, `/help`, `/stickerid` (private) | — | — | Bot info, command help renderer, sticker file_id echo helper |
|
||||
| `misc` | `/ping`, `/mstats`, `/fortytwo` | KV | — | Health check + DB demo stub |
|
||||
| `trading` | `/trade_topup`, `/trade_buy`, `/trade_sell`, `/trade_convert`, `/trade_stats`, `/history` | D1 (trades) + KV (portfolio, symbol cache) | Daily 5PM trim | Paper trading — VN stocks with dynamic symbol resolution |
|
||||
| `wordle` | `/wordle`, `/wordle_new`, `/wordle_giveup`, `/wordle_stats` | KV | — | 5-letter word guessing game. 14,855-word dict |
|
||||
| `loldle` | `/loldle`, `/loldle_giveup`, `/loldle_stats` | KV | — | Classic-mode LoL champion guesser. Data synced from `tiennm99/loldle-data` |
|
||||
| `lolschedule` | `/lolschedule_today`, `/lolschedule_week`, `/lolschedule_subscribe`, `/lolschedule_unsubscribe` | KV | Daily 01:00 UTC | LoL esports schedule + daily digest subscriptions |
|
||||
| `semantle` | `/semantle`, `/semantle_giveup`, `/semantle_stats` | KV | — | English semantic word guessing via hosted word2sim service |
|
||||
| `doantu` | `/doantu`, `/doantu_hint`, `/doantu_giveup`, `/doantu_stats` | KV | — | Vietnamese semantle via hosted phow2sim service |
|
||||
| `twentyq` | `/twentyq`, `/twentyq_giveup`, `/twentyq_stats` | KV | — | Reverse-Akinator yes/no game. Workers AI (`@cf/google/gemma-4-26b-a4b-it`) generates round-start category+hint and judges each turn via one-line JSON |
|
||||
| `misc` | `/ping`, `/mstats`, `/fortytwo` | MongoDB KVStore | — | Health check + DB demo stub |
|
||||
| `trading` | `/trade_topup`, `/trade_buy`, `/trade_sell`, `/trade_convert`, `/trade_stats`, `/history` | MongoDB (trades + portfolio + symbol cache) | Daily 5PM trim | Paper trading — VN stocks with dynamic symbol resolution |
|
||||
| `wordle` | `/wordle`, `/wordle_new`, `/wordle_giveup`, `/wordle_stats` | MongoDB KVStore | — | 5-letter word guessing game. 14,855-word dict |
|
||||
| `loldle` | `/loldle`, `/loldle_giveup`, `/loldle_stats` | MongoDB KVStore | — | Classic-mode LoL champion guesser. Data synced from `tiennm99/loldle-data` |
|
||||
| `lolschedule` | `/lolschedule_today`, `/lolschedule_week`, `/lolschedule_subscribe`, `/lolschedule_unsubscribe` | MongoDB KVStore | Daily 01:00 UTC | LoL esports schedule + daily digest subscriptions |
|
||||
| `semantle` | `/semantle`, `/semantle_giveup`, `/semantle_stats` | MongoDB KVStore | — | English semantic word guessing via hosted word2sim service |
|
||||
| `doantu` | `/doantu`, `/doantu_hint`, `/doantu_giveup`, `/doantu_stats` | MongoDB KVStore | — | Vietnamese semantle via hosted phow2sim service |
|
||||
| `twentyq` | `/twentyq`, `/twentyq_giveup`, `/twentyq_stats` | MongoDB KVStore | — | Reverse-Akinator yes/no game. Workers AI (`@cf/google/gemma-4-26b-a4b-it`) generates round-start category+hint and judges each turn via one-line JSON |
|
||||
|
||||
## Key Data Flows
|
||||
|
||||
@@ -75,4 +75,4 @@ Each module maintains its own `README.md` with commands, data model, and impleme
|
||||
|
||||
## Tests
|
||||
|
||||
`npm test` runs the full vitest suite (run in a few seconds — ~450 tests). Structure: one folder per module under `tests/modules/<name>/`, shared fakes under `tests/fakes/` (fake-kv-namespace, fake-d1, fake-bot, fake-modules, fake-ai). No workerd, no Telegram fixtures — pure-logic unit tests with injected fakes.
|
||||
`npm test` runs the full vitest suite (run in ~10 seconds — 733 tests). Structure: one folder per module under `tests/modules/<name>/`, shared fakes under `tests/fakes/` (fake-mongo, fake-kv-namespace, fake-d1, fake-bot, fake-modules, fake-ai). No workerd, no Telegram fixtures — pure-logic unit tests with injected fakes. See `tests/fakes/fake-mongo.js` for the frozen surface implemented during Phase 02–08.
|
||||
@@ -0,0 +1,65 @@
|
||||
# MongoDB Atlas Cost Tracking
|
||||
|
||||
Operational runbook for monitoring and managing MongoDB Atlas billing during and post-migration.
|
||||
|
||||
## Free Tier Limits (M0)
|
||||
|
||||
| Resource | Limit | Warning |
|
||||
|----------|-------|---------|
|
||||
| Storage | 512 MB | Reached 400 MB: prepare upgrade plan |
|
||||
| Connections | 500 | Reached 400: review cron/handler concurrency |
|
||||
| Throughput | ~100 ops/sec sustained | Degradation: check dashboard Charts |
|
||||
| Backups | None | No PITR; only daily snapshots (read-only) |
|
||||
|
||||
## Cost Ladder
|
||||
|
||||
| Tier | Monthly | When | Ops/sec | Storage |
|
||||
|------|---------|------|---------|---------|
|
||||
| **M0** | Free | Initial, dev | ~100 | 512 MB |
|
||||
| **Flex** | $8–$30 | Sustained >400 MB or >400 conn | 400–1000 | 10–256 GB |
|
||||
| **M10** | $57 | Production high-throughput | 1000+ | Unlimited |
|
||||
|
||||
Flex tier auto-scales cost based on data volume and throughput. Start at M2 ($9) and scale to M5 ($70+) as needed.
|
||||
|
||||
## Monitoring & Upgrade Triggers
|
||||
|
||||
### Monthly Review Checklist
|
||||
|
||||
1. **Storage:** Open Atlas Dashboard → Metrics → check `Database Storage` chart
|
||||
- If > 400 MB: escalate to Flex within 2 weeks
|
||||
- Projection: (current MB / days since epoch) * 30 → projected month-end
|
||||
|
||||
2. **Connections:** Metrics → check `Current Connections` peak
|
||||
- If > 400: cron jobs or handlers running concurrently too often
|
||||
- Review `src/modules/*/index.js` cron frequency; stagger if possible
|
||||
|
||||
3. **Throughput:** Metrics → `Network Egress` + `Database Operations`
|
||||
- Spike during trading/wordle command storms: expected
|
||||
- Sustained >100 ops/sec: assess Flex upgrade
|
||||
|
||||
4. **Cluster Status:** Alerts → check if "Cluster unavailable" triggered
|
||||
- M0 never pauses if any cron writes data (bot has 6+ crons, any write prevents pause)
|
||||
|
||||
### Upgrade Decision
|
||||
|
||||
Plan upgrade **before** hitting limits:
|
||||
|
||||
| Condition | Action | Timeline |
|
||||
|-----------|--------|----------|
|
||||
| Storage trend → 512 MB in 3 months | Upgrade to Flex M2 | 2 weeks notice |
|
||||
| Peak connections > 400 regularly | Upgrade to Flex M2 | Immediate if sustained |
|
||||
| Ops/sec spikes > 2000 | Upgrade to Flex M5 or M10 | 1 week notice |
|
||||
|
||||
## Rotation & Maintenance
|
||||
|
||||
- **Password rotation:** Every 90 days, owner = repo maintainer. See `docs/using-mongodb.md` "Rotation" section.
|
||||
- **Alert config review:** Monthly, ensure Atlas + CF Observability alerts are wired.
|
||||
- **Backups:** M0 has none; data is live-only. Backups start at Flex tier.
|
||||
|
||||
## Post-Cutover Simplification
|
||||
|
||||
Once Phase 07 cutover completes and Cloudflare KV/D1 are deleted, MongoDB becomes the sole data store. Cost is dominated by storage growth (KV reads/writes are gone). Rebaseline this doc:
|
||||
|
||||
- Remove dual-write cost considerations
|
||||
- Focus on pure Mongo spend
|
||||
- Extend storage projections based on new single-source data
|
||||
@@ -0,0 +1,44 @@
|
||||
# Project Changelog
|
||||
|
||||
All significant changes, features, and fixes to miti99bot.
|
||||
|
||||
## [2026-04-25] MongoDB Atlas Migration Complete
|
||||
|
||||
**Status:** Phases 01–08 committed on `dev` branch. Dual-write ready; cutover pending operator execution (Phase 07 binding deletion).
|
||||
|
||||
**Summary:**
|
||||
Research, planning, and 7 phases of implementation + documentation to migrate from Cloudflare KV/D1 to MongoDB Atlas M0 free tier. Achieved zero-downtime dual-write architecture with automated drift detection, backfill/reverse-backfill scripts, and comprehensive telemetry. All 733 unit tests passing.
|
||||
|
||||
**Plan:** `plans/260425-1945-mongodb-atlas-migration/`
|
||||
|
||||
**Phases:**
|
||||
- Phase 01: Wrangler config, secret-leak lint, bundle-size gate
|
||||
- Phase 02: MongoKVStore + memoized client + fake-mongo + tests
|
||||
- Phase 03: MongoTradesStore + trading refactor + MongoSqlStore shim
|
||||
- Phase 04: Dual-write wrappers + factories + retry queue + drift verifier + e2e
|
||||
- Phase 05: Backfill scripts (local node, CF KV REST + wrangler d1 export)
|
||||
- Phase 06: Telemetry instrumentation + soak runbook
|
||||
- Phase 07: Cutover scripts + reverse-backfill + decommission helpers
|
||||
- Phase 08: Final docs pass + alternatives section update
|
||||
|
||||
**Key artifacts:**
|
||||
- `src/db/mongo-kv-store.js` — MongoDB implementation of KVStore interface
|
||||
- `src/db/mongo-trades-store.js` — MongoDB trading ledger store
|
||||
- `src/db/dual-kv-store.js` — dual-write wrapper for safety during migration
|
||||
- `src/db/mongo-client.js` — shared memoized connection with retry logic
|
||||
- `docs/using-mongodb.md` — operational runbook
|
||||
- `docs/cost-tracking.md` — cost monitoring and upgrade triggers
|
||||
|
||||
**Reviewers noted** (not adopted):
|
||||
- Maintenance-window cutover instead of dual-write (saves ~6h engineering)
|
||||
- Defer trading migration (D1 free handles ~100 writes/day indefinitely)
|
||||
- Single shared `kv` collection vs 12 per-module collections
|
||||
- Pivot to Upstash before starting (smaller bundle, HTTP-native)
|
||||
|
||||
User chose full cold-start Atlas validation. If bundle size trips, `phase-07-alt-pivot.md` is ready.
|
||||
|
||||
**Post-cutover simplification** (Phase 07 Stage 3 — operator-driven):
|
||||
- Delete dual-write layer
|
||||
- Delete `cf-kv-store.js`, `cf-sql-store.js`, and D1 bindings from `wrangler.toml`
|
||||
- `createStore()` returns pure `MongoKVStore`
|
||||
- Cost tracking shifts to Mongo-only projections
|
||||
@@ -1,17 +1,29 @@
|
||||
/**
|
||||
* @file fake-mongo — Map-backed in-memory MongoDB fake for unit tests.
|
||||
*
|
||||
* Covers the full surface needed by Phase 02 (MongoKVStore) and
|
||||
* Phase 03 (MongoTradesStore):
|
||||
* collection(name) → { findOne, updateOne, deleteOne, find, insertOne,
|
||||
* insertMany, distinct, deleteMany, countDocuments,
|
||||
* createIndex }
|
||||
* **FROZEN SURFACE** (Phase 02–08): This fake implements the minimal subset of MongoDB
|
||||
* API required by MongoKVStore and MongoTradesStore tests. New MongoClient features
|
||||
* must be added to this fake to remain testable.
|
||||
*
|
||||
* TTL is NOT simulated server-side. Tests that exercise TTL check the
|
||||
* `expiresAt` field value directly and test the read-time filter in the
|
||||
* MongoKVStore layer by controlling Date.now() via vi.setSystemTime().
|
||||
* **Implemented methods:**
|
||||
* - `db.collection(name)` → Collection
|
||||
* - Collection: `findOne(query)`, `updateOne(filter, update, opts)`, `deleteOne(query)`,
|
||||
* `find(query)`, `insertOne(doc)`, `insertMany(docs)`, `distinct(field, query)`,
|
||||
* `deleteMany(query)`, `countDocuments(query)`, `createIndex(spec)` (no-op)
|
||||
*
|
||||
* **Cursor methods** (from `find()`): `sort(spec)`, `skip(n)`, `limit(n)`,
|
||||
* `project(spec)`, `toArray()`
|
||||
*
|
||||
* **NOT SIMULATED:**
|
||||
* - TTL expirations at server-side. Tests assert `expiresAt` field presence and
|
||||
* control Date.now() via `vi.setSystemTime()` to test read-time expiration filters
|
||||
* in MongoKVStore. Real Atlas TTL is validated during Phase 06 soak.
|
||||
* - Transactions, sharding, replica sets, or other cluster features.
|
||||
* - Advanced operators beyond those in `matchQuery()` ($gt, $gte, $lt, $lte,
|
||||
* $exists, $in, $regex, $or, $and).
|
||||
*
|
||||
* @see tests/db/mongo-kv-store.test.js
|
||||
* @see tests/modules/trading/ (trades store usage)
|
||||
*/
|
||||
|
||||
/**
|
||||
|
||||
Reference in new issue
Block a user