diff --git a/Makefile b/Makefile index b546d9b..c7a20db 100644 --- a/Makefile +++ b/Makefile @@ -31,9 +31,11 @@ test: ## Unit tests (no emulator required) # Run DynamoDB integration tests against DynamoDB Local. # Override DDB_PORT if 8001 is taken on your host. DDB_PORT ?= 8001 -test-dynamodb: dynamodb-local ## Run DynamoDB tests against DynamoDB Local - DYNAMODB_LOCAL_URL=http://localhost:$(DDB_PORT) LOG_LEVEL=error \ - go test -race -count=1 ./internal/storage/... +test-dynamodb: dynamodb-local mongo-local ## Run the DynamoDB→Mongo migrator e2e against local emulators + DYNAMODB_LOCAL_URL=http://localhost:$(DDB_PORT) \ + MONGODB_TEST_URL=mongodb://127.0.0.1:$(MONGO_PORT) \ + MONGO_DATABASE=migrate_test LOG_LEVEL=error \ + go test -race -count=1 ./cmd/migrate-dynamo-to-mongo/... # Run MongoDB integration tests against a local Mongo container. # Override MONGO_PORT if 27017 is taken on your host. diff --git a/README.md b/README.md index 3dc67b7..d8362e9 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ internal/server/ HTTP routes (/ health, /cron/{name} manual trigger) internal/telegram/ Telegram long-polling bot wrapper internal/cron/ in-process cron scheduler (replaces EventBridge) internal/modules/ Module framework, registry, dispatchers, modules -internal/storage/ KVProvider interface; memory + dynamodb + mongodb (values stored as native BSON documents) +internal/storage/ typed DocStore[T] (Provider + Typed); mongodb runtime + memory (tests). Values persist as flattened native BSON root documents internal/ai/ Gemini client (used by twentyq) docker-compose.yml Coolify self-host stack (single bot service) docs/deploy-coolify-selfhosted.md Self-host onboarding + cutover runbook @@ -58,12 +58,11 @@ MONGO_DATABASE=miti99bot_dev \ go run ./cmd/server ``` -For storage integration tests (each skips when its emulator env var is unset): +For integration tests (each skips when its emulator env var is unset): ```sh -make dynamodb-local # docker run amazon/dynamodb-local on :8001 -make test-dynamodb # internal/storage tests against DynamoDB Local make mongo-local # docker run mongo:7 on :27017 -make test-mongo # internal/storage tests against local MongoDB +make test-mongo # internal/storage typed-store tests against local MongoDB +make test-dynamodb # DynamoDB→Mongo migrator e2e against DynamoDB Local + local Mongo ``` ## Test @@ -71,7 +70,8 @@ make test-mongo # internal/storage tests against local MongoDB ```sh make vet # go vet make test # full unit suite (no emulator) -make test-dynamodb # storage tests against DynamoDB Local (requires Docker) +make test-mongo # typed-store integration tests against local Mongo (requires Docker) +make test-dynamodb # migrator e2e against DynamoDB Local + Mongo (requires Docker) ``` ## Deploy diff --git a/cmd/migrate-dynamo-to-mongo/README.md b/cmd/migrate-dynamo-to-mongo/README.md index ee1d297..30bdcda 100644 --- a/cmd/migrate-dynamo-to-mongo/README.md +++ b/cmd/migrate-dynamo-to-mongo/README.md @@ -7,20 +7,25 @@ app writes, then verifies per-module parity. Idempotent and re-runnable. ## What it does - Full-table `Scan` of DynamoDB (the table is a small KV) → group by `pk`. -- Writes each item through `storage.MongoKVStore.Put`, so the `value` encoding - is **byte-identical** to what the running bot writes (BSON binary). This is - what keeps re-runs idempotent and future CompareAndSwap correct — a raw - `UpdateOne` that stored `value` as a BSON string would silently break both. -- `Put` runs `validateKey` on each sort key and upserts by `_id`, so a re-run - produces no duplicates and an invalid key fails loud (no key the app's read - path can't load is ever written). +- Writes each item through the typed Mongo store (`storage.Typed[bson.M]`) as a + **flattened native document** — the value's JSON fields are hoisted to the + document root alongside `_id`/`version`/`updatedAt`, with no `value` envelope. + This is the exact shape the running bot writes, so the app reads migrated docs + directly. Integers keep int64 fidelity (decoded with `UseNumber`). +- The two non-object values are wrapped into named root fields to match the + module's typed shape: lolschedule `subscribers` (a JSON array) → `{subscribers: [...]}` + and `daily_push:last_date` (a bare date string) → `{date: "..."}`. Any other + non-object value fails loud so a missing wrap rule is obvious (see `encode.go`). +- Writing through the store validates the module/collection name and key and + upserts by `_id`, so a re-run produces no duplicates and bad input fails loud. | DynamoDB | MongoDB | |---|---| | `pk` (module name) | collection name | | `sk` (user key) | document `_id` | -| `value` (JSON string) | `value` field (BSON binary, same bytes) | -| `updatedAt` | restamped to migration time (write-only field; nothing reads it) | +| `value` (JSON object) | payload fields hoisted to the document root (no `value` field) | +| `value` (array/scalar, lolschedule) | wrapped in a named root field (`subscribers` / `date`) | +| — | `version` = 1, `updatedAt` = migration time (write-only; nothing reads it) | ## Usage diff --git a/cmd/migrate-dynamo-to-mongo/encode.go b/cmd/migrate-dynamo-to-mongo/encode.go new file mode 100644 index 0000000..41ded4b --- /dev/null +++ b/cmd/migrate-dynamo-to-mongo/encode.go @@ -0,0 +1,109 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// reservedRootFields are owned by the storage layer (see internal/storage); a +// flattened object payload must not collide with them. +var reservedRootFields = map[string]bool{"_id": true, "version": true, "updatedAt": true} + +// wrapRule names the root field a non-object DynamoDB value must be wrapped in +// to match the typed Mongo store's named-struct shape. rawString treats the +// DynamoDB value as a bare (non-JSON) string; otherwise it is JSON-decoded. +type wrapRule struct { + field string + rawString bool +} + +// wrapRules covers the only KV entries whose value is not a JSON object. They +// mirror the named-struct wrappers the lolschedule module persists: +// - subscribers: a JSON array → {subscribers: [...]} +// - daily_push:last_date: a bare date string → {date: "..."} +// +// Any other non-object value fails loud in payloadForItem so a missing rule is +// obvious rather than silently dropped. +var wrapRules = map[[2]string]wrapRule{ + {"lolschedule", "subscribers"}: {field: "subscribers"}, + {"lolschedule", "daily_push:last_date"}: {field: "date", rawString: true}, +} + +// payloadForItem converts one migrated KV row's value into the flattened payload +// map the typed Mongo store stores at the document root. The store adds _id, +// version, and updatedAt; this returns only the payload fields. +func payloadForItem(module, key string, value []byte) (bson.M, error) { + if rule, ok := wrapRules[[2]string{module, key}]; ok { + if rule.rawString { + return bson.M{rule.field: string(value)}, nil + } + decoded, err := decodeJSONNumber(value) + if err != nil { + return nil, fmt.Errorf("%s/%s: decode value: %w", module, key, err) + } + return bson.M{rule.field: decoded}, nil + } + + decoded, err := decodeJSONNumber(value) + if err != nil { + return nil, fmt.Errorf("%s/%s: decode value: %w", module, key, err) + } + obj, ok := decoded.(bson.M) + if !ok { + return nil, fmt.Errorf("%s/%s: value is not a JSON object (type %T) and has no wrap rule — add one to wrapRules", module, key, decoded) + } + for k := range obj { + if reservedRootFields[k] { + return nil, fmt.Errorf("%s/%s: payload key %q collides with a reserved root field — add a wrap rule", module, key, k) + } + } + return obj, nil +} + +// decodeJSONNumber decodes JSON into a BSON-native value, preserving integral +// numbers as int64 (UseNumber) so migrated numbers keep int64 fidelity, matching +// the live store's value codec. +func decodeJSONNumber(value []byte) (any, error) { + dec := json.NewDecoder(bytes.NewReader(value)) + dec.UseNumber() + var v any + if err := dec.Decode(&v); err != nil { + return nil, err + } + if dec.More() { + return nil, fmt.Errorf("trailing data after JSON value") + } + return numberToBSON(v), nil +} + +// numberToBSON walks a json.Unmarshal(UseNumber) tree into bson.M/bson.A, +// converting json.Number to int64 when integral, else float64. +func numberToBSON(v any) any { + switch t := v.(type) { + case map[string]any: + m := make(bson.M, len(t)) + for k, val := range t { + m[k] = numberToBSON(val) + } + return m + case []any: + a := make(bson.A, len(t)) + for i, val := range t { + a[i] = numberToBSON(val) + } + return a + case json.Number: + if i, err := t.Int64(); err == nil { + return i + } + if f, err := t.Float64(); err == nil { + return f + } + return t.String() + default: + return v + } +} diff --git a/cmd/migrate-dynamo-to-mongo/encode_test.go b/cmd/migrate-dynamo-to-mongo/encode_test.go new file mode 100644 index 0000000..cc01070 --- /dev/null +++ b/cmd/migrate-dynamo-to-mongo/encode_test.go @@ -0,0 +1,87 @@ +package main + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +func TestPayloadForItem_Object(t *testing.T) { + got, err := payloadForItem("coin", "user:1", []byte(`{"bal":100,"meta":{"createdAt":5}}`)) + if err != nil { + t.Fatalf("payloadForItem: %v", err) + } + if got["bal"] != int64(100) { + t.Errorf("bal = %v (%T), want int64(100)", got["bal"], got["bal"]) + } + if _, ok := got["value"]; ok { + t.Error("payload must not contain a 'value' envelope field") + } + meta, ok := got["meta"].(bson.M) + if !ok || meta["createdAt"] != int64(5) { + t.Errorf("nested meta = %v, want {createdAt: int64(5)}", got["meta"]) + } +} + +func TestPayloadForItem_LolscheduleSubscribers(t *testing.T) { + got, err := payloadForItem("lolschedule", "subscribers", []byte(`[{"chatId":1},{"chatId":2}]`)) + if err != nil { + t.Fatalf("payloadForItem: %v", err) + } + arr, ok := got["subscribers"].(bson.A) + if !ok || len(arr) != 2 { + t.Fatalf("subscribers = %v (%T), want 2-element bson.A", got["subscribers"], got["subscribers"]) + } +} + +func TestPayloadForItem_LolscheduleLastPushRawString(t *testing.T) { + // last-push date is stored as a bare (non-JSON) string in DynamoDB. + got, err := payloadForItem("lolschedule", "daily_push:last_date", []byte(`2026-06-28`)) + if err != nil { + t.Fatalf("payloadForItem: %v", err) + } + if got["date"] != "2026-06-28" { + t.Errorf("date = %v, want 2026-06-28", got["date"]) + } +} + +func TestPayloadForItem_UnknownNonObjectFailsLoud(t *testing.T) { + if _, err := payloadForItem("coin", "user:1", []byte(`"a-bare-string"`)); err == nil { + t.Error("non-object value with no wrap rule must fail loud") + } + if _, err := payloadForItem("coin", "user:1", []byte(`[1,2,3]`)); err == nil { + t.Error("array value with no wrap rule must fail loud") + } +} + +func TestPayloadForItem_ReservedKeyCollision(t *testing.T) { + if _, err := payloadForItem("coin", "user:1", []byte(`{"version":7}`)); err == nil { + t.Error("payload key colliding with reserved root field must fail loud") + } +} + +// TestPayloadForItem_CamelCaseFidelity guards the invariant that lets migration +// work: the migrator preserves the original (camelCase) JSON keys, so a typed +// store struct must declare bson tags matching those names. A struct whose bson +// tags drifted to the driver's lowercased default would read these back empty. +func TestPayloadForItem_CamelCaseFidelity(t *testing.T) { + payload, err := payloadForItem("lolschedule", "events", []byte(`{"startTime":"t","gameWins":3,"blockName":"Week 1"}`)) + if err != nil { + t.Fatalf("payloadForItem: %v", err) + } + raw, err := bson.Marshal(payload) + if err != nil { + t.Fatalf("bson.Marshal: %v", err) + } + var got struct { + StartTime string `bson:"startTime"` + GameWins int `bson:"gameWins"` + BlockName string `bson:"blockName"` + } + if err := bson.Unmarshal(raw, &got); err != nil { + t.Fatalf("bson.Unmarshal: %v", err) + } + if got.StartTime != "t" || got.GameWins != 3 || got.BlockName != "Week 1" { + t.Fatalf("camelCase round-trip lost data: %+v", got) + } +} diff --git a/cmd/migrate-dynamo-to-mongo/main.go b/cmd/migrate-dynamo-to-mongo/main.go index 30f0e2f..5f82b5b 100644 --- a/cmd/migrate-dynamo-to-mongo/main.go +++ b/cmd/migrate-dynamo-to-mongo/main.go @@ -12,16 +12,18 @@ // exactly `dynamodb:Scan` on the table ARN (nothing else, no write actions). // For local testing, set DYNAMODB_LOCAL_URL to point at DynamoDB Local. // -// - default: scan + write every item through MongoKVStore.Put (same value -// encoding as the live app — JSON objects stored as native BSON), then -// report per-module counts. +// - default: scan + write every item through the typed Mongo store as a +// flattened native document (payload fields hoisted to the root, no `value` +// envelope) — the exact shape the live app writes — then report per-module +// counts. // - --dry-run: scan + report counts, write nothing. // - --verify: tally DynamoDB per pk via Scan vs Mongo CountDocuments per // collection; print a table and exit non-zero on any mismatch. // -// Note: writing through Put gives each doc the app's native-BSON value encoding -// and a fresh updatedAt/version; nothing in the app reads updatedAt (write-only -// today), and --verify compares counts, so this is intentional and harmless. +// Note: each migrated doc gets a fresh updatedAt and version=1; nothing in the +// app reads updatedAt, and --verify compares counts, so this is intentional and +// harmless. The two non-object values (lolschedule subscribers array and +// last-push date) are wrapped into named root fields (see encode.go). package main import ( @@ -142,11 +144,13 @@ func sortedKeys(m map[string]int) []string { } // runMigrate scans the table, then (unless dry-run) writes every item through -// MongoKVStore.Put so the value uses the same native-BSON encoding the live app -// writes — guaranteeing idempotent re-runs and correct version-based CAS. Put -// also runs validateKey on each sk and upserts by _id, so a re-run produces no -// duplicates and an invalid key fails loud rather than writing data -// the app's read path cannot load. +// the typed Mongo store as a flattened native document (payload fields hoisted +// to the root, no `value` envelope) — the exact shape the live app writes. +// payloadForItem wraps the two non-object values (lolschedule subscribers array +// and last-push date) into named-struct fields. Writing through +// storage.Typed[bson.M] reuses the store's module/key validation and version +// semantics, so a re-run produces no duplicates (upsert by _id) and an invalid +// module name or key fails loud rather than writing data the app cannot load. func runMigrate(ctx context.Context, ddb *dynamodb.Client, mdb *mongo.Database, table string, dryRun bool) error { items, err := scanTable(ctx, ddb, table) if err != nil { @@ -160,9 +164,11 @@ func runMigrate(ctx context.Context, ddb *dynamodb.Client, mdb *mongo.Database, if dryRun { continue } - // For invalid module names For returns a store that errors on Put; - // Put validates the key. Either way an error aborts loudly. - if err := provider.For(it.pk).Put(ctx, it.sk, it.value); err != nil { + payload, err := payloadForItem(it.pk, it.sk, it.value) + if err != nil { + return err + } + if err := storage.Typed[bson.M](provider.Collection(it.pk)).Put(ctx, it.sk, payload); err != nil { return fmt.Errorf("put %s/%s: %w", it.pk, it.sk, err) } } diff --git a/cmd/migrate-dynamo-to-mongo/main_test.go b/cmd/migrate-dynamo-to-mongo/main_test.go index 7af13a2..f88de8c 100644 --- a/cmd/migrate-dynamo-to-mongo/main_test.go +++ b/cmd/migrate-dynamo-to-mongo/main_test.go @@ -10,6 +10,7 @@ import ( "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/service/dynamodb" "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" + "go.mongodb.org/mongo-driver/v2/bson" "go.mongodb.org/mongo-driver/v2/mongo" "github.com/tiennm99/miti99bot/internal/storage" @@ -95,15 +96,16 @@ func TestMigrateAndVerify(t *testing.T) { t.Fatalf("runVerify: %v", err) } - // Spot-check byte-identical round trip through the app's read path. + // Spot-check the migrated value round-trips through the typed store as a + // flattened native doc: bal hoisted to the root, preserved as int64. db := mongoDatabase(t, ctx, mongoURL, mongoDB) provider := storage.NewMongoProvider(db) - got, err := provider.For("coin").Get(ctx, "user:1") + got, _, err := storage.Typed[bson.M](provider.Collection("coin")).Get(ctx, "user:1") if err != nil { t.Fatalf("Get migrated value: %v", err) } - if string(got) != `{"bal":100}` { - t.Errorf("migrated value = %q, want %q", got, `{"bal":100}`) + if got["bal"] != int64(100) { + t.Errorf("migrated value bal = %v (%T), want int64(100)", got["bal"], got["bal"]) } } diff --git a/cmd/server/main.go b/cmd/server/main.go index d32213b..911fec6 100644 --- a/cmd/server/main.go +++ b/cmd/server/main.go @@ -60,10 +60,6 @@ func factories() map[string]modules.Factory { } } -// dynamodbInitTimeout caps DynamoDB client construction at startup. Lambda -// has a 10s init phase; we want to leave headroom for module wiring. -const dynamodbInitTimeout = 5 * time.Second - // mongodbInitTimeout caps MongoDB connect+ping at startup (and Disconnect at // shutdown). Atlas SRV DNS + TLS handshake can take a couple seconds on a cold // container; 10s leaves headroom without hiding a wedged cluster. @@ -153,7 +149,7 @@ func main() { deploynotify.Run(rootCtx, deploynotify.Config{ Bot: b, - KV: provider.For("deploynotify"), + Store: deploynotify.NewStore(provider.Collection("deploynotify")), OwnerID: cfg.BotOwnerID, GitSHA: gitSHA, }) @@ -210,16 +206,16 @@ func main() { } // buildProvider picks the storage backend. Selection order: -// 1. Explicit KV_PROVIDER env (memory|dynamodb|mongodb) wins. +// 1. Explicit KV_PROVIDER env (memory|mongodb) wins. // 2. Auto-detect: MONGO_URL set → mongodb; otherwise memory. // // The self-host default is mongodb (just set MONGO_URL + MONGO_DATABASE — no -// KV_PROVIDER needed). dynamodb remains reachable only via an explicit -// KV_PROVIDER, kept for the data migrator and integration tests. The legacy -// firestore backend and the AWS_LAMBDA_FUNCTION_NAME auto-detect are removed. +// KV_PROVIDER needed). The memory backend is for tests and local no-database +// runs (MODULES=). DynamoDB is no longer a runtime backend — it survives only +// as the one-off migration source (cmd/migrate-dynamo-to-mongo). // // Returned closer is always non-nil and safe to call exactly once. -func buildProvider(ctx context.Context, cfg config) (storage.KVProvider, func(), error) { +func buildProvider(ctx context.Context, cfg config) (storage.Provider, func(), error) { backend := strings.ToLower(strings.TrimSpace(cfg.KVProvider)) if backend == "" { if cfg.MongoURL != "" { @@ -261,25 +257,10 @@ func buildProvider(ctx context.Context, cfg config) (storage.KVProvider, func(), log.Info("storage backend", "backend", "mongodb", "database", cfg.MongoDatabase) return storage.NewMongoProvider(db), closer, nil - case "dynamodb": - if cfg.DynamoDBTable == "" { - return nil, func() {}, errors.New("KV_PROVIDER=dynamodb requires DYNAMODB_TABLE") - } - initCtx, cancel := context.WithTimeout(ctx, dynamodbInitTimeout) - defer cancel() - client, err := storage.NewDynamoDBClient(initCtx, storage.DynamoDBEndpointFromEnv()) - if err != nil { - return nil, func() {}, err - } - log.Info("storage backend", - "backend", "dynamodb", - "table", cfg.DynamoDBTable, - "endpoint_override", storage.DynamoDBEndpointFromEnv() != "") - // DynamoDB SDK v2 client has no Close; HTTP client uses the default pool. - return storage.NewDynamoDBProvider(client, cfg.DynamoDBTable), func() {}, nil - default: - return nil, func() {}, fmt.Errorf("unknown KV_PROVIDER %q (want memory|dynamodb|mongodb)", backend) + // DynamoDB is no longer a runtime backend — it survives only as the + // one-off migration source (cmd/migrate-dynamo-to-mongo). + return nil, func() {}, fmt.Errorf("unknown KV_PROVIDER %q (want memory|mongodb)", backend) } } @@ -298,8 +279,7 @@ type config struct { Modules []string BotOwnerID int64 AdminUserIDs map[int64]bool - KVProvider string // empty = auto-detect; or "memory"|"dynamodb"|"mongodb" - DynamoDBTable string // required when KVProvider=dynamodb + KVProvider string // empty = auto-detect; or "memory"|"mongodb" MongoURL string // required when KVProvider=mongodb (Atlas SRV connection string; SECRET — never log) MongoDatabase string // required when KVProvider=mongodb TelegramBotTokenParam string @@ -341,7 +321,6 @@ func loadConfig() config { BotOwnerID: parseInt64(envMap["OWNER_ID"]), AdminUserIDs: parseInt64Set(envMap["ADMIN_IDS"]), KVProvider: envMap["KV_PROVIDER"], - DynamoDBTable: envMap["DYNAMODB_TABLE"], MongoURL: envMap["MONGO_URL"], MongoDatabase: envMap["MONGO_DATABASE"], TelegramBotTokenParam: strings.TrimSpace(envMap["TELEGRAM_BOT_TOKEN_PARAMETER_NAME"]), diff --git a/docs/deploy-coolify-selfhosted.md b/docs/deploy-coolify-selfhosted.md index 66974a8..f186203 100644 --- a/docs/deploy-coolify-selfhosted.md +++ b/docs/deploy-coolify-selfhosted.md @@ -63,13 +63,14 @@ and the `STOCK/COIN/GOLD *_API_URL` overrides (modules use coded defaults). 4. Copy the `mongodb+srv://…` connection string into `MONGO_URL` and put the db name in `MONGO_DATABASE`. -> Storage layout: one collection per module; each document is -> `{ _id: , value: , version, updatedAt }`. -> Values are stored as native BSON (objects/arrays expand and are queryable in -> Compass); non-JSON values (e.g. a date guard) are plain strings. Concurrency -> uses the `version` field (optimistic lock). If you migrated data with an older -> build that stored `value` as a string/blob, re-run the migrator (idempotent) -> so values become native. +> Storage layout: one collection per module; each document is a flattened native +> document — `{ _id: , ...payload fields, version, updatedAt }` with no +> `value` envelope. Payload fields are hoisted to the document root so they +> expand and are queryable in Compass. The two non-object values are wrapped in a +> named field: lolschedule subscribers under `subscribers` (array) and the daily +> push date under `date`. Concurrency uses the `version` field (optimistic lock); +> `updatedAt` is a BSON Date. The DynamoDB→Mongo migrator writes this shape +> directly, and is idempotent. ## 2. Coolify diff --git a/internal/deploynotify/deploy_notify.go b/internal/deploynotify/deploy_notify.go index 9d61906..5a89d6a 100644 --- a/internal/deploynotify/deploy_notify.go +++ b/internal/deploynotify/deploy_notify.go @@ -6,7 +6,7 @@ // (see Makefile). An empty gitSHA — e.g. `go run` or a build without the // ldflags — is treated as a signal to stay silent. // -// Dedup: a single KV record (key=last_notified_sha) holds the most recently +// Dedup: a single store record (key=last_notified_sha) holds the most recently // notified SHA. On match we return early; on miss we send first, then write // — so a transient Telegram failure doesn't permanently silence retries. package deploynotify @@ -23,10 +23,10 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) -// kvKey is the single KV slot this package owns. -const kvKey = "last_notified_sha" +// storeKey is the single store slot this package owns. +const storeKey = "last_notified_sha" -// defaultTimeout caps the whole Run path (KV read + Telegram send + KV write) +// defaultTimeout caps the whole Run path (store read + Telegram send + store write) // so a misbehaving network can never block Lambda init past its 10s budget. const defaultTimeout = 3 * time.Second @@ -34,7 +34,7 @@ const defaultTimeout = 3 * time.Second // nil, Run falls back to bot.SendMessage. type Config struct { Bot *bot.Bot - KV storage.KVStore + Store storage.DocStore[notifyRecord] OwnerID int64 GitSHA string Timeout time.Duration @@ -43,11 +43,18 @@ type Config struct { Sender func(ctx context.Context, chatID int64, text string) error } -// notifyRecord is the KV value shape. At is informational only — useful for -// eyeballing in the DynamoDB console; not consulted by the code path. +// notifyRecord is the store value shape. At is informational only — useful for +// eyeballing in the database console; not consulted by the code path. type notifyRecord struct { - SHA string `json:"sha"` - At int64 `json:"at"` + SHA string `json:"sha" bson:"sha"` + At int64 `json:"at" bson:"at"` +} + +// NewStore builds the typed store this package needs from a module Collection. +// notifyRecord is unexported, so callers cannot name DocStore[notifyRecord] +// directly — this is the seam that lets cmd/server wire the store. +func NewStore(c storage.Collection) storage.DocStore[notifyRecord] { + return storage.Typed[notifyRecord](c) } // Run is the entry point. Fire-and-forget — never returns an error and @@ -64,12 +71,12 @@ func Run(ctx context.Context, cfg Config) { ctx, cancel := context.WithTimeout(ctx, timeout) defer cancel() - notify, err := shouldNotify(ctx, cfg.KV, cfg.GitSHA) + notify, err := shouldNotify(ctx, cfg.Store, cfg.GitSHA) if err != nil { - // Treat KV errors as "not notified yet": worst case is one extra + // Treat store errors as "not notified yet": worst case is one extra // DM on the next cold start, which is far better than going - // silent on every deploy because DynamoDB threw a transient. - log.Warn("deploynotify kv read failed; will attempt send anyway", "err", err) + // silent on every deploy because the store threw a transient error. + log.Warn("deploynotify store read failed; will attempt send anyway", "err", err) notify = true } if !notify { @@ -80,23 +87,23 @@ func Run(ctx context.Context, cfg Config) { log.Warn("deploynotify telegram send failed", "err", err, "owner", cfg.OwnerID) return } - if err := markNotified(ctx, cfg.KV, cfg.GitSHA); err != nil { - log.Warn("deploynotify kv write failed (owner was notified)", "err", err) + if err := markNotified(ctx, cfg.Store, cfg.GitSHA); err != nil { + log.Warn("deploynotify store write failed (owner was notified)", "err", err) return } log.Info("deploynotify sent", "sha", cfg.GitSHA, "owner", cfg.OwnerID) } // skipReason returns a non-empty short string when Run should no-op without -// touching KV or Telegram. Empty string ⇒ proceed. +// touching the store or Telegram. Empty string ⇒ proceed. func skipReason(cfg Config) string { switch { case cfg.GitSHA == "": return "empty gitSHA (build without -ldflags)" case cfg.OwnerID == 0: return "no OWNER_ID configured" - case cfg.KV == nil: - return "no KV configured" + case cfg.Store == nil: + return "no Store configured" case cfg.Bot == nil && cfg.Sender == nil: return "no bot or sender configured" } @@ -105,9 +112,8 @@ func skipReason(cfg Config) string { // shouldNotify reports whether sha differs from the last notified value. // A missing record (ErrNotFound) is treated as "yes, notify". -func shouldNotify(ctx context.Context, kv storage.KVStore, sha string) (bool, error) { - var prev notifyRecord - err := kv.GetJSON(ctx, kvKey, &prev) +func shouldNotify(ctx context.Context, store storage.DocStore[notifyRecord], sha string) (bool, error) { + prev, _, err := store.Get(ctx, storeKey) if errors.Is(err, storage.ErrNotFound) { return true, nil } @@ -117,9 +123,9 @@ func shouldNotify(ctx context.Context, kv storage.KVStore, sha string) (bool, er return prev.SHA != sha, nil } -// markNotified writes the SHA + current timestamp to KV. -func markNotified(ctx context.Context, kv storage.KVStore, sha string) error { - return kv.PutJSON(ctx, kvKey, notifyRecord{ +// markNotified writes the SHA + current timestamp to the store. +func markNotified(ctx context.Context, store storage.DocStore[notifyRecord], sha string) error { + return store.Put(ctx, storeKey, notifyRecord{ SHA: sha, At: time.Now().UTC().UnixMilli(), }) diff --git a/internal/deploynotify/deploy_notify_test.go b/internal/deploynotify/deploy_notify_test.go index dbc5520..6073f42 100644 --- a/internal/deploynotify/deploy_notify_test.go +++ b/internal/deploynotify/deploy_notify_test.go @@ -9,23 +9,28 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) +// newNotifyStore returns a fresh in-memory typed notifyRecord store for tests. +func newNotifyStore() storage.DocStore[notifyRecord] { + return storage.Typed[notifyRecord](storage.NewMemoryProvider().Collection("deploynotify")) +} + func TestShouldNotify_FirstRun(t *testing.T) { - kv := storage.NewMemoryKVStore() - got, err := shouldNotify(context.Background(), kv, "abc123") + store := newNotifyStore() + got, err := shouldNotify(context.Background(), store, "abc123") if err != nil { t.Fatalf("err: %v", err) } if !got { - t.Errorf("first run with empty KV → want true, got false") + t.Errorf("first run with empty store → want true, got false") } } func TestShouldNotify_SameSHA(t *testing.T) { - kv := storage.NewMemoryKVStore() - if err := markNotified(context.Background(), kv, "abc123"); err != nil { + store := newNotifyStore() + if err := markNotified(context.Background(), store, "abc123"); err != nil { t.Fatal(err) } - got, err := shouldNotify(context.Background(), kv, "abc123") + got, err := shouldNotify(context.Background(), store, "abc123") if err != nil { t.Fatalf("err: %v", err) } @@ -35,11 +40,11 @@ func TestShouldNotify_SameSHA(t *testing.T) { } func TestShouldNotify_DifferentSHA(t *testing.T) { - kv := storage.NewMemoryKVStore() - if err := markNotified(context.Background(), kv, "old111"); err != nil { + store := newNotifyStore() + if err := markNotified(context.Background(), store, "old111"); err != nil { t.Fatal(err) } - got, err := shouldNotify(context.Background(), kv, "new222") + got, err := shouldNotify(context.Background(), store, "new222") if err != nil { t.Fatalf("err: %v", err) } @@ -66,10 +71,10 @@ func (r *recorder) send(_ context.Context, chatID int64, text string) error { } func TestRun_SkipsWhenSHAEmpty(t *testing.T) { - kv := storage.NewMemoryKVStore() + store := newNotifyStore() rec := &recorder{} Run(context.Background(), Config{ - KV: kv, + Store: store, OwnerID: 42, GitSHA: "", Sender: rec.send, @@ -77,17 +82,17 @@ func TestRun_SkipsWhenSHAEmpty(t *testing.T) { if rec.called { t.Errorf("empty SHA must not send; got call with text=%q", rec.text) } - // And no KV write either. - if _, err := kv.Get(context.Background(), kvKey); !errors.Is(err, storage.ErrNotFound) { - t.Errorf("empty SHA must not write KV; got err=%v", err) + // And no store write either. + if _, _, err := store.Get(context.Background(), storeKey); !errors.Is(err, storage.ErrNotFound) { + t.Errorf("empty SHA must not write store; got err=%v", err) } } func TestRun_SkipsWhenNoOwner(t *testing.T) { - kv := storage.NewMemoryKVStore() + store := newNotifyStore() rec := &recorder{} Run(context.Background(), Config{ - KV: kv, + Store: store, OwnerID: 0, GitSHA: "abc123", Sender: rec.send, @@ -95,22 +100,22 @@ func TestRun_SkipsWhenNoOwner(t *testing.T) { if rec.called { t.Errorf("zero owner must not send") } - if _, err := kv.Get(context.Background(), kvKey); !errors.Is(err, storage.ErrNotFound) { - t.Errorf("zero owner must not write KV; got err=%v", err) + if _, _, err := store.Get(context.Background(), storeKey); !errors.Is(err, storage.ErrNotFound) { + t.Errorf("zero owner must not write store; got err=%v", err) } } func TestRun_SendsAndPersistsOnFirstRun(t *testing.T) { - kv := storage.NewMemoryKVStore() + store := newNotifyStore() rec := &recorder{} Run(context.Background(), Config{ - KV: kv, + Store: store, OwnerID: 42, GitSHA: "abc123", Sender: rec.send, }) if !rec.called { - t.Fatalf("first run with fresh KV must send") + t.Fatalf("first run with fresh store must send") } if rec.chatID != 42 { t.Errorf("chatID = %d, want 42", rec.chatID) @@ -118,10 +123,10 @@ func TestRun_SendsAndPersistsOnFirstRun(t *testing.T) { if !strings.Contains(rec.text, "abc123") { t.Errorf("message %q missing SHA", rec.text) } - // KV must now hold the SHA so the next Run is silent. - var got notifyRecord - if err := kv.GetJSON(context.Background(), kvKey, &got); err != nil { - t.Fatalf("post-send KV read: %v", err) + // Store must now hold the SHA so the next Run is silent. + got, _, err := store.Get(context.Background(), storeKey) + if err != nil { + t.Fatalf("post-send store read: %v", err) } if got.SHA != "abc123" { t.Errorf("persisted SHA = %q, want abc123", got.SHA) @@ -129,13 +134,13 @@ func TestRun_SendsAndPersistsOnFirstRun(t *testing.T) { } func TestRun_SilentOnSecondRunSameSHA(t *testing.T) { - kv := storage.NewMemoryKVStore() - if err := markNotified(context.Background(), kv, "abc123"); err != nil { + store := newNotifyStore() + if err := markNotified(context.Background(), store, "abc123"); err != nil { t.Fatal(err) } rec := &recorder{} Run(context.Background(), Config{ - KV: kv, + Store: store, OwnerID: 42, GitSHA: "abc123", Sender: rec.send, @@ -146,10 +151,10 @@ func TestRun_SilentOnSecondRunSameSHA(t *testing.T) { } func TestRun_DoesNotPersistOnSendFailure(t *testing.T) { - kv := storage.NewMemoryKVStore() + store := newNotifyStore() rec := &recorder{err: errors.New("telegram is down")} Run(context.Background(), Config{ - KV: kv, + Store: store, OwnerID: 42, GitSHA: "abc123", Sender: rec.send, @@ -158,8 +163,8 @@ func TestRun_DoesNotPersistOnSendFailure(t *testing.T) { t.Fatalf("sender should have been called") } // Send failed → SHA must NOT be persisted, so the next cold start retries. - if _, err := kv.Get(context.Background(), kvKey); !errors.Is(err, storage.ErrNotFound) { - t.Errorf("failed send must not write KV; got err=%v", err) + if _, _, err := store.Get(context.Background(), storeKey); !errors.Is(err, storage.ErrNotFound) { + t.Errorf("failed send must not write store; got err=%v", err) } } diff --git a/internal/keylock/keylock.go b/internal/keylock/keylock.go index 631903b..fbab6fc 100644 --- a/internal/keylock/keylock.go +++ b/internal/keylock/keylock.go @@ -2,7 +2,7 @@ // (typically a chat / user / subject identifier) across goroutines. // // Why a separate package: every game module needs a per-subject mutex to -// turn KVStore's single-op atomicity into safe Get→mutate→Put. The bot +// turn the store's single-op atomicity into safe Get→mutate→Put. The bot // dispatcher runs each Telegram update in its own goroutine, so without // explicit per-subject serialisation two updates to the same game could // race and drop one write. diff --git a/internal/modules/coin/coin.go b/internal/modules/coin/coin.go index 18ceae2..42c9290 100644 --- a/internal/modules/coin/coin.go +++ b/internal/modules/coin/coin.go @@ -1,11 +1,14 @@ package coin -import "github.com/tiennm99/miti99bot/internal/modules" +import ( + "github.com/tiennm99/miti99bot/internal/modules" + "github.com/tiennm99/miti99bot/internal/storage" +) // New is the coin paper-trading module factory. It is opt-in through MODULES // and keeps its portfolio state separate from stock and gold modules. func New(deps modules.Deps) modules.Module { - s := newState(deps.KV) + s := newState(storage.Typed[Portfolio](deps.Store)) return modules.Module{ Commands: []modules.Command{ { diff --git a/internal/modules/coin/handlers.go b/internal/modules/coin/handlers.go index 080463d..6e531e4 100644 --- a/internal/modules/coin/handlers.go +++ b/internal/modules/coin/handlers.go @@ -49,7 +49,7 @@ func (s *state) handleTopup(ctx context.Context, b *bot.Bot, update *models.Upda return chathelper.Reply(ctx, b, update.Message, "Amount must be a positive finite USD number within the supported range.") } defer s.locks.Acquire(strconv.FormatInt(userID, 10))() - p, err := UpdatePortfolio(ctx, s.kv, userID, s.now().UnixMilli(), func(p *Portfolio) error { + p, err := UpdatePortfolio(ctx, s.store, userID, s.now().UnixMilli(), func(p *Portfolio) error { p.AddUSD(amount) p.Meta.Invested += amount return nil @@ -94,7 +94,7 @@ func (s *state) handleBuy(ctx context.Context, b *bot.Bot, update *models.Update } defer s.locks.Acquire(strconv.FormatInt(userID, 10))() var insufficientBalance *float64 - p, err := UpdatePortfolio(ctx, s.kv, userID, s.now().UnixMilli(), func(p *Portfolio) error { + p, err := UpdatePortfolio(ctx, s.store, userID, s.now().UnixMilli(), func(p *Portfolio) error { ok, balance := p.DeductUSD(amount) if !ok { insufficientBalance = &balance @@ -149,7 +149,7 @@ func (s *state) handleSell(ctx context.Context, b *bot.Bot, update *models.Updat defer s.locks.Acquire(strconv.FormatInt(userID, 10))() var insufficientHeldQty float64 var insufficientHeld bool - p, err := UpdatePortfolio(ctx, s.kv, userID, s.now().UnixMilli(), func(p *Portfolio) error { + p, err := UpdatePortfolio(ctx, s.store, userID, s.now().UnixMilli(), func(p *Portfolio) error { ok, held := p.DeductAsset(coin.Symbol, qty) if !ok { insufficientHeldQty = held diff --git a/internal/modules/coin/handlers_order_test.go b/internal/modules/coin/handlers_order_test.go index 6c20a4e..1552c00 100644 --- a/internal/modules/coin/handlers_order_test.go +++ b/internal/modules/coin/handlers_order_test.go @@ -19,7 +19,7 @@ func TestHandleBuyAcceptsCoinFirstOrder(t *testing.T) { } rb.AssertSentText(t, "Bought 0.0002 BTC") - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.USD != 990 || p.Assets["BTC"] != 0.0002 { t.Fatalf("after coin-first buy = %+v", p) } @@ -38,7 +38,7 @@ func TestHandleSellAcceptsCoinFirstOrder(t *testing.T) { } rb.AssertSentText(t, "Sold 0.01 BTC") - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.USD != 1000 || len(p.Assets) != 0 { t.Fatalf("after coin-first sell = %+v", p) } diff --git a/internal/modules/coin/handlers_test.go b/internal/modules/coin/handlers_test.go index fbd0ae4..3e01af6 100644 --- a/internal/modules/coin/handlers_test.go +++ b/internal/modules/coin/handlers_test.go @@ -31,9 +31,14 @@ func (f fakePriceFetcher) FetchUSD(_ context.Context, coin CoinSymbol) (CoinPric return price, nil } +// newCoinStore returns a fresh in-memory typed portfolio store for tests. +func newCoinStore() Store { + return storage.Typed[Portfolio](storage.NewMemoryProvider().Collection("coin")) +} + func newTestState(prices map[string]CoinPrice, err error) *state { return &state{ - kv: storage.NewMemoryKVStore(), + store: newCoinStore(), prices: fakePriceFetcher{prices: prices, err: err}, nowFn: func() time.Time { return time.UnixMilli(123) }, } @@ -100,7 +105,7 @@ func TestHandleTopup(t *testing.T) { t.Fatalf("handleTopup: %v", err) } rb.AssertSentText(t, "Topped up $1,000.00") - p, err := LoadPortfolio(ctx, s.kv, 7, 999) + p, err := LoadPortfolio(ctx, s.store, 7, 999) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -119,7 +124,7 @@ func TestHandleBuyAndSell(t *testing.T) { t.Fatalf("handleBuy: %v", err) } rb.AssertSentText(t, "Bought 0.01 BTC") - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.USD != 500 || p.Assets["BTC"] != 0.01 { t.Fatalf("after buy = %+v", p) } @@ -128,7 +133,7 @@ func TestHandleBuyAndSell(t *testing.T) { t.Fatalf("handleSell: %v", err) } rb.AssertSentText(t, "Sold 0.01 BTC") - p, _ = LoadPortfolio(ctx, s.kv, 7, 999) + p, _ = LoadPortfolio(ctx, s.store, 7, 999) if p.USD != 1000 || len(p.Assets) != 0 { t.Fatalf("after sell = %+v", p) } @@ -205,7 +210,7 @@ func TestHandleSellInsufficientCoinWithHoldings(t *testing.T) { t.Fatalf("insufficient sell message used ambiguous have wording: %q", text) } - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.USD != 500 || p.Assets["BTC"] != 0.01 { t.Fatalf("portfolio mutated on failed sell = %+v", p) } @@ -224,7 +229,7 @@ func TestHandleSellRejectsDustQuantity(t *testing.T) { } rb.AssertSentText(t, "too small") - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.USD != 999 || len(p.Assets) != 1 { t.Fatalf("portfolio mutated on dust sell = %+v", p) } @@ -242,7 +247,7 @@ func TestHandleBuyRejectsDustQuantity(t *testing.T) { } rb.AssertSentText(t, "too small") - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.USD != 1000 || len(p.Assets) != 0 { t.Fatalf("portfolio mutated on dust buy = %+v", p) } @@ -276,7 +281,7 @@ func TestPriceErrorDoesNotMutatePortfolio(t *testing.T) { t.Fatalf("handleBuy: %v", err) } rb.AssertSentText(t, "Could not fetch coin price") - p, err := LoadPortfolio(ctx, s.kv, 7, 999) + p, err := LoadPortfolio(ctx, s.store, 7, 999) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -310,5 +315,5 @@ func TestStatsWithAndWithoutPrice(t *testing.T) { } func modDepsForTest() modules.Deps { - return modules.Deps{KV: storage.NewMemoryKVStore()} + return modules.Deps{Store: storage.NewMemoryProvider().Collection("coin")} } diff --git a/internal/modules/coin/helpers.go b/internal/modules/coin/helpers.go index 893d456..b8cfeec 100644 --- a/internal/modules/coin/helpers.go +++ b/internal/modules/coin/helpers.go @@ -14,7 +14,6 @@ import ( "github.com/tiennm99/miti99bot/internal/keylock" "github.com/tiennm99/miti99bot/internal/log" "github.com/tiennm99/miti99bot/internal/modules/util/chathelper" - "github.com/tiennm99/miti99bot/internal/storage" ) type priceFetcher interface { @@ -22,14 +21,14 @@ type priceFetcher interface { } type state struct { - kv storage.KVStore + store Store prices priceFetcher locks keylock.Map nowFn func() time.Time } -func newState(kv storage.KVStore) *state { - return &state{kv: kv, prices: NewPriceClientFromEnv()} +func newState(store Store) *state { + return &state{store: store, prices: NewPriceClientFromEnv()} } func (s *state) now() time.Time { diff --git a/internal/modules/coin/portfolio.go b/internal/modules/coin/portfolio.go index 5f4c950..c2c1e18 100644 --- a/internal/modules/coin/portfolio.go +++ b/internal/modules/coin/portfolio.go @@ -2,7 +2,6 @@ package coin import ( "context" - "encoding/json" "errors" "fmt" "math" @@ -14,15 +13,18 @@ import ( const coinDustEpsilon = 1e-9 const portfolioUpdateAttempts = 5 +// Store is the coin module's typed portfolio store. +type Store = storage.DocStore[Portfolio] + type Portfolio struct { - USD float64 `json:"usd"` - Assets map[string]float64 `json:"assets"` - Meta PortfolioMeta `json:"meta"` + USD float64 `json:"usd" bson:"usd"` + Assets map[string]float64 `json:"assets" bson:"assets"` + Meta PortfolioMeta `json:"meta" bson:"meta"` } type PortfolioMeta struct { - Invested float64 `json:"invested"` - CreatedAt int64 `json:"createdAt"` + Invested float64 `json:"invested" bson:"invested"` + CreatedAt int64 `json:"createdAt" bson:"createdAt"` } func NewPortfolio(now int64) Portfolio { @@ -33,42 +35,26 @@ func portfolioKey(userID int64) string { return "user:" + strconv.FormatInt(userID, 10) } -func LoadPortfolio(ctx context.Context, kv storage.KVStore, userID int64, now int64) (Portfolio, error) { - var p Portfolio - err := kv.GetJSON(ctx, portfolioKey(userID), &p) - switch { - case err == nil: - p.normalize() - if p.Assets == nil { - p.Assets = map[string]float64{} - } - if p.Meta.CreatedAt == 0 { - p.Meta.CreatedAt = now - } - return p, nil - case errors.Is(err, storage.ErrNotFound): - return NewPortfolio(now), nil - default: +func LoadPortfolio(ctx context.Context, store Store, userID int64, now int64) (Portfolio, error) { + p, _, err := loadPortfolioForUpdate(ctx, store, portfolioKey(userID), now) + if err != nil { return Portfolio{}, fmt.Errorf("coin: load portfolio %d: %w", userID, err) } + return p, nil } -func SavePortfolio(ctx context.Context, kv storage.KVStore, userID int64, p Portfolio) error { +func SavePortfolio(ctx context.Context, store Store, userID int64, p Portfolio) error { p.normalize() - if err := kv.PutJSON(ctx, portfolioKey(userID), p); err != nil { + if err := store.Put(ctx, portfolioKey(userID), p); err != nil { return fmt.Errorf("coin: save portfolio %d: %w", userID, err) } return nil } -func UpdatePortfolio(ctx context.Context, kv storage.KVStore, userID int64, now int64, mutate func(*Portfolio) error) (Portfolio, error) { - vs, ok := kv.(storage.VersionedStore) - if !ok { - return Portfolio{}, fmt.Errorf("coin: storage does not support versioned portfolio updates") - } +func UpdatePortfolio(ctx context.Context, store Store, userID int64, now int64, mutate func(*Portfolio) error) (Portfolio, error) { key := portfolioKey(userID) for attempt := 0; attempt < portfolioUpdateAttempts; attempt++ { - p, version, err := loadPortfolioForUpdate(ctx, vs, key, now) + p, version, err := loadPortfolioForUpdate(ctx, store, key, now) if err != nil { return Portfolio{}, fmt.Errorf("coin: load portfolio %d: %w", userID, err) } @@ -76,11 +62,7 @@ func UpdatePortfolio(ctx context.Context, kv storage.KVStore, userID int64, now return p, err } p.normalize() - next, err := json.Marshal(p) - if err != nil { - return Portfolio{}, fmt.Errorf("coin: save portfolio %d: json encode: %w", userID, err) - } - if err := vs.PutVersioned(ctx, key, version, next); err == nil { + if err := store.PutVersioned(ctx, key, version, p); err == nil { return p, nil } else if !errors.Is(err, storage.ErrConflict) { return Portfolio{}, fmt.Errorf("coin: save portfolio %d: %w", userID, err) @@ -89,14 +71,10 @@ func UpdatePortfolio(ctx context.Context, kv storage.KVStore, userID int64, now return Portfolio{}, fmt.Errorf("coin: save portfolio %d: %w", userID, storage.ErrConflict) } -func loadPortfolioForUpdate(ctx context.Context, vs storage.VersionedStore, key string, now int64) (Portfolio, int64, error) { - raw, version, err := vs.GetVersioned(ctx, key) +func loadPortfolioForUpdate(ctx context.Context, store Store, key string, now int64) (Portfolio, int64, error) { + p, version, err := store.Get(ctx, key) switch { case err == nil: - var p Portfolio - if err := json.Unmarshal(raw, &p); err != nil { - return Portfolio{}, 0, fmt.Errorf("json decode: %w", err) - } p.normalize() if p.Assets == nil { p.Assets = map[string]float64{} diff --git a/internal/modules/coin/portfolio_test.go b/internal/modules/coin/portfolio_test.go index 9b4c8e8..a4384c7 100644 --- a/internal/modules/coin/portfolio_test.go +++ b/internal/modules/coin/portfolio_test.go @@ -10,7 +10,7 @@ import ( ) func TestLoadPortfolioFirstTimeUser(t *testing.T) { - p, err := LoadPortfolio(context.Background(), storage.NewMemoryKVStore(), 42, 123) + p, err := LoadPortfolio(context.Background(), newCoinStore(), 42, 123) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -67,32 +67,31 @@ func TestNormalizeAmountSpecialValues(t *testing.T) { } } +// conflictOnceStore wraps a real typed store and forces exactly one write +// conflict (after committing a competing value) before delegating, to exercise +// UpdatePortfolio's optimistic-lock retry. type conflictOnceStore struct { - storage.KVStore + Store conflicted bool } -func (s *conflictOnceStore) GetVersioned(ctx context.Context, key string) ([]byte, int64, error) { - return s.KVStore.(storage.VersionedStore).GetVersioned(ctx, key) -} - -func (s *conflictOnceStore) PutVersioned(ctx context.Context, key string, expectedVersion int64, val []byte) error { +func (s *conflictOnceStore) PutVersioned(ctx context.Context, key string, expectedVersion int64, val Portfolio) error { if !s.conflicted { s.conflicted = true competing := NewPortfolio(1) competing.AddUSD(10) - if err := s.PutJSON(ctx, key, competing); err != nil { + if err := s.Store.Put(ctx, key, competing); err != nil { return err } return storage.ErrConflict } - return s.KVStore.(storage.VersionedStore).PutVersioned(ctx, key, expectedVersion, val) + return s.Store.PutVersioned(ctx, key, expectedVersion, val) } func TestUpdatePortfolioRetriesAfterWriteConflict(t *testing.T) { ctx := context.Background() - kv := &conflictOnceStore{KVStore: storage.NewMemoryKVStore()} - got, err := UpdatePortfolio(ctx, kv, 7, 1, func(p *Portfolio) error { + store := &conflictOnceStore{Store: newCoinStore()} + got, err := UpdatePortfolio(ctx, store, 7, 1, func(p *Portfolio) error { p.AddUSD(5) return nil }) @@ -106,14 +105,14 @@ func TestUpdatePortfolioRetriesAfterWriteConflict(t *testing.T) { func TestUpdatePortfolioMutateErrorDoesNotPersist(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() - _, err := UpdatePortfolio(ctx, kv, 7, 1, func(p *Portfolio) error { + store := newCoinStore() + _, err := UpdatePortfolio(ctx, store, 7, 1, func(p *Portfolio) error { return errInsufficientUSD }) if !errors.Is(err, errInsufficientUSD) { t.Fatalf("got %v, want errInsufficientUSD", err) } - if _, err := kv.Get(ctx, "user:7"); !errors.Is(err, storage.ErrNotFound) { + if _, _, err := store.Get(ctx, "user:7"); !errors.Is(err, storage.ErrNotFound) { t.Fatalf("failed mutate must not persist, Get = %v", err) } } diff --git a/internal/modules/coin/views.go b/internal/modules/coin/views.go index 36613e9..0f1f503 100644 --- a/internal/modules/coin/views.go +++ b/internal/modules/coin/views.go @@ -17,7 +17,7 @@ func (s *state) handleStats(ctx context.Context, b *bot.Bot, update *models.Upda if !ok { return chathelper.Reply(ctx, b, update.Message, "Cannot identify user - /coin_stats needs a sender.") } - p, err := LoadPortfolio(ctx, s.kv, userID, s.now().UnixMilli()) + p, err := LoadPortfolio(ctx, s.store, userID, s.now().UnixMilli()) if err != nil { log.Error("coin_load_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not load coin portfolio. Try again later.") diff --git a/internal/modules/gold/composite_prices.go b/internal/modules/gold/composite_prices.go index 755e788..3528cbb 100644 --- a/internal/modules/gold/composite_prices.go +++ b/internal/modules/gold/composite_prices.go @@ -20,9 +20,9 @@ type compositePriceFetcher struct { // NewCompositePriceFetcherFromEnv builds the production price fetcher using // only the env-driven VNAppMob client. -func NewCompositePriceFetcherFromEnv(kv storage.KVStore) priceFetcher { +func NewCompositePriceFetcherFromEnv(coll storage.Collection) priceFetcher { return &compositePriceFetcher{ - vnappmob: NewVNAppMobClientFromEnv(kv), + vnappmob: NewVNAppMobClientFromEnv(coll), } } diff --git a/internal/modules/gold/composite_prices_test.go b/internal/modules/gold/composite_prices_test.go index 16b45e2..7682a2c 100644 --- a/internal/modules/gold/composite_prices_test.go +++ b/internal/modules/gold/composite_prices_test.go @@ -92,7 +92,7 @@ func TestGoldPriceLines(t *testing.T) { } func TestNewCompositePriceFetcherFromEnv(t *testing.T) { - f, ok := NewCompositePriceFetcherFromEnv(storage.NewMemoryKVStore()).(*compositePriceFetcher) + f, ok := NewCompositePriceFetcherFromEnv(storage.NewMemoryProvider().Collection("gold")).(*compositePriceFetcher) if !ok { t.Fatalf("expected *compositePriceFetcher, got %T", f) } diff --git a/internal/modules/gold/gold.go b/internal/modules/gold/gold.go index 15187e8..a0ae97f 100644 --- a/internal/modules/gold/gold.go +++ b/internal/modules/gold/gold.go @@ -1,11 +1,13 @@ package gold -import "github.com/tiennm99/miti99bot/internal/modules" +import ( + "github.com/tiennm99/miti99bot/internal/modules" +) // New is the gold paper-trading module factory. It is opt-in through MODULES // and keeps its portfolio state separate from the stock module. func New(deps modules.Deps) modules.Module { - s := newState(deps.KV) + s := newState(deps.Store) return modules.Module{ Commands: []modules.Command{ { diff --git a/internal/modules/gold/handlers.go b/internal/modules/gold/handlers.go index 2a7e55e..4c7aa8e 100644 --- a/internal/modules/gold/handlers.go +++ b/internal/modules/gold/handlers.go @@ -60,7 +60,7 @@ func (s *state) handleTopup(ctx context.Context, b *bot.Bot, update *models.Upda } defer s.locks.Acquire(strconv.FormatInt(userID, 10))() - p, err := UpdatePortfolio(ctx, s.kv, userID, s.now().UnixMilli(), func(p *Portfolio) error { + p, err := UpdatePortfolio(ctx, s.store, userID, s.now().UnixMilli(), func(p *Portfolio) error { p.AddVND(amount) p.Meta.Invested += amount return nil @@ -98,7 +98,7 @@ func (s *state) handleBuy(ctx context.Context, b *bot.Bot, update *models.Update defer s.locks.Acquire(strconv.FormatInt(userID, 10))() var insufficientBalance *float64 - p, err := UpdatePortfolio(ctx, s.kv, userID, s.now().UnixMilli(), func(p *Portfolio) error { + p, err := UpdatePortfolio(ctx, s.store, userID, s.now().UnixMilli(), func(p *Portfolio) error { ok, balance := p.DeductVND(cost) if !ok { insufficientBalance = &balance @@ -145,7 +145,7 @@ func (s *state) handleSell(ctx context.Context, b *bot.Bot, update *models.Updat return chathelper.Reply(ctx, b, update.Message, "Trade value is too large.") } var insufficientHeld *float64 - p, err := UpdatePortfolio(ctx, s.kv, userID, s.now().UnixMilli(), func(p *Portfolio) error { + p, err := UpdatePortfolio(ctx, s.store, userID, s.now().UnixMilli(), func(p *Portfolio) error { ok, held := p.DeductLuong(qty) if !ok { insufficientHeld = &held @@ -173,7 +173,7 @@ func (s *state) handleStats(ctx context.Context, b *bot.Bot, update *models.Upda return chathelper.Reply(ctx, b, update.Message, "Cannot identify user - /gold_stats needs a sender.") } - p, err := LoadPortfolio(ctx, s.kv, userID, s.now().UnixMilli()) + p, err := LoadPortfolio(ctx, s.store, userID, s.now().UnixMilli()) if err != nil { log.Error("gold_load_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not load gold portfolio. Try again later.") diff --git a/internal/modules/gold/handlers_test.go b/internal/modules/gold/handlers_test.go index 58cfa09..af2f8ba 100644 --- a/internal/modules/gold/handlers_test.go +++ b/internal/modules/gold/handlers_test.go @@ -38,7 +38,7 @@ func (f fakePriceFetcher) FetchPrice(context.Context) (GoldPrice, error) { func newTestState(price float64, err error) *state { return &state{ - kv: storage.NewMemoryKVStore(), + store: newGoldStore(), prices: fakePriceFetcher{price: price, err: err}, nowFn: func() time.Time { return time.UnixMilli(123) }, } @@ -77,7 +77,7 @@ func TestHandleTopup(t *testing.T) { t.Fatalf("handleTopup: %v", err) } rb.AssertSentText(t, "Topped up 5.000.000 VND") - p, err := LoadPortfolio(ctx, s.kv, 7, 999) + p, err := LoadPortfolio(ctx, s.store, 7, 999) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -98,7 +98,7 @@ func TestHandleBuyAndSell(t *testing.T) { t.Fatalf("buy: %v", err) } rb.AssertSentText(t, "Bought 1.25 luong gold") - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.Luong != 1.25 || p.VND != 2_500_000 { t.Fatalf("after buy: %+v", p) } @@ -107,7 +107,7 @@ func TestHandleBuyAndSell(t *testing.T) { t.Fatalf("sell: %v", err) } rb.AssertSentText(t, "Sold 1.25 luong gold") - p, _ = LoadPortfolio(ctx, s.kv, 7, 999) + p, _ = LoadPortfolio(ctx, s.store, 7, 999) if p.Luong != 0 || p.VND != 5_000_000 { t.Fatalf("after sell: %+v", p) } @@ -139,7 +139,7 @@ func TestPriceErrorDoesNotMutatePortfolio(t *testing.T) { t.Fatalf("buy: %v", err) } rb.AssertSentText(t, "Could not fetch gold price") - p, err := LoadPortfolio(ctx, s.kv, 7, 999) + p, err := LoadPortfolio(ctx, s.store, 7, 999) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -201,7 +201,7 @@ func (f spreadPriceFetcher) FetchPrice(context.Context) (GoldPrice, error) { } func modDepsForTest() modules.Deps { - return modules.Deps{KV: storage.NewMemoryKVStore()} + return modules.Deps{Store: storage.NewMemoryProvider().Collection("gold")} } func TestHandlePrice(t *testing.T) { @@ -239,7 +239,7 @@ func TestHandlePriceFetchError(t *testing.T) { func TestHandleBuyUsesSellPrice(t *testing.T) { ctx := context.Background() s := &state{ - kv: storage.NewMemoryKVStore(), + store: newGoldStore(), prices: spreadPriceFetcher{buy: 90_000_000, sell: 91_000_000}, nowFn: func() time.Time { return time.UnixMilli(123) }, } @@ -253,7 +253,7 @@ func TestHandleBuyUsesSellPrice(t *testing.T) { if !strings.Contains(text, "91.000.000 VND/luong") { t.Fatalf("expected sell price 91.000.000 in %q", text) } - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.VND != 409_000_000 { t.Fatalf("balance after buy at 91M: got %v, want 409000000", p.VND) } @@ -262,7 +262,7 @@ func TestHandleBuyUsesSellPrice(t *testing.T) { func TestHandleSellUsesBuyPrice(t *testing.T) { ctx := context.Background() s := &state{ - kv: storage.NewMemoryKVStore(), + store: newGoldStore(), prices: spreadPriceFetcher{buy: 90_000_000, sell: 91_000_000}, nowFn: func() time.Time { return time.UnixMilli(123) }, } @@ -277,7 +277,7 @@ func TestHandleSellUsesBuyPrice(t *testing.T) { if !strings.Contains(text, "90.000.000 VND/luong") { t.Fatalf("expected buy price 90.000.000 in %q", text) } - p, _ := LoadPortfolio(ctx, s.kv, 7, 999) + p, _ := LoadPortfolio(ctx, s.store, 7, 999) if p.VND != 499_000_000 { t.Fatalf("balance after sell at 90M: got %v, want 499000000", p.VND) } diff --git a/internal/modules/gold/helpers.go b/internal/modules/gold/helpers.go index edeb379..5c89f2b 100644 --- a/internal/modules/gold/helpers.go +++ b/internal/modules/gold/helpers.go @@ -24,14 +24,17 @@ type priceFetcher interface { } type state struct { - kv storage.KVStore + store PortfolioStore prices priceFetcher locks keylock.Map nowFn func() time.Time } -func newState(kv storage.KVStore) *state { - return &state{kv: kv, prices: NewCompositePriceFetcherFromEnv(kv)} +func newState(coll storage.Collection) *state { + return &state{ + store: storage.Typed[Portfolio](coll), + prices: NewCompositePriceFetcherFromEnv(coll), + } } func (s *state) now() time.Time { diff --git a/internal/modules/gold/portfolio.go b/internal/modules/gold/portfolio.go index aa70908..d3c9b83 100644 --- a/internal/modules/gold/portfolio.go +++ b/internal/modules/gold/portfolio.go @@ -2,7 +2,6 @@ package gold import ( "context" - "encoding/json" "errors" "fmt" "math" @@ -14,15 +13,18 @@ import ( const goldDustEpsilon = 1e-9 const portfolioUpdateAttempts = 5 +// PortfolioStore is the gold module's typed portfolio store. +type PortfolioStore = storage.DocStore[Portfolio] + type Portfolio struct { - VND float64 `json:"vnd"` - Luong float64 `json:"luong"` - Meta PortfolioMeta `json:"meta"` + VND float64 `json:"vnd" bson:"vnd"` + Luong float64 `json:"luong" bson:"luong"` + Meta PortfolioMeta `json:"meta" bson:"meta"` } type PortfolioMeta struct { - Invested float64 `json:"invested"` - CreatedAt int64 `json:"createdAt"` + Invested float64 `json:"invested" bson:"invested"` + CreatedAt int64 `json:"createdAt" bson:"createdAt"` } func NewPortfolio(now int64) Portfolio { @@ -33,39 +35,26 @@ func portfolioKey(userID int64) string { return "user:" + strconv.FormatInt(userID, 10) } -func LoadPortfolio(ctx context.Context, kv storage.KVStore, userID int64, now int64) (Portfolio, error) { - var p Portfolio - err := kv.GetJSON(ctx, portfolioKey(userID), &p) - switch { - case err == nil: - p.normalize() - if p.Meta.CreatedAt == 0 { - p.Meta.CreatedAt = now - } - return p, nil - case errors.Is(err, storage.ErrNotFound): - return NewPortfolio(now), nil - default: +func LoadPortfolio(ctx context.Context, store PortfolioStore, userID int64, now int64) (Portfolio, error) { + p, _, err := loadPortfolioForUpdate(ctx, store, portfolioKey(userID), now) + if err != nil { return Portfolio{}, fmt.Errorf("gold: load portfolio %d: %w", userID, err) } + return p, nil } -func SavePortfolio(ctx context.Context, kv storage.KVStore, userID int64, p Portfolio) error { +func SavePortfolio(ctx context.Context, store PortfolioStore, userID int64, p Portfolio) error { p.normalize() - if err := kv.PutJSON(ctx, portfolioKey(userID), p); err != nil { + if err := store.Put(ctx, portfolioKey(userID), p); err != nil { return fmt.Errorf("gold: save portfolio %d: %w", userID, err) } return nil } -func UpdatePortfolio(ctx context.Context, kv storage.KVStore, userID int64, now int64, mutate func(*Portfolio) error) (Portfolio, error) { - vs, ok := kv.(storage.VersionedStore) - if !ok { - return Portfolio{}, fmt.Errorf("gold: storage does not support versioned portfolio updates") - } +func UpdatePortfolio(ctx context.Context, store PortfolioStore, userID int64, now int64, mutate func(*Portfolio) error) (Portfolio, error) { key := portfolioKey(userID) for attempt := 0; attempt < portfolioUpdateAttempts; attempt++ { - p, version, err := loadPortfolioForUpdate(ctx, vs, key, now) + p, version, err := loadPortfolioForUpdate(ctx, store, key, now) if err != nil { return Portfolio{}, fmt.Errorf("gold: load portfolio %d: %w", userID, err) } @@ -73,11 +62,7 @@ func UpdatePortfolio(ctx context.Context, kv storage.KVStore, userID int64, now return p, err } p.normalize() - next, err := json.Marshal(p) - if err != nil { - return Portfolio{}, fmt.Errorf("gold: save portfolio %d: json encode: %w", userID, err) - } - if err := vs.PutVersioned(ctx, key, version, next); err == nil { + if err := store.PutVersioned(ctx, key, version, p); err == nil { return p, nil } else if !errors.Is(err, storage.ErrConflict) { return Portfolio{}, fmt.Errorf("gold: save portfolio %d: %w", userID, err) @@ -86,14 +71,10 @@ func UpdatePortfolio(ctx context.Context, kv storage.KVStore, userID int64, now return Portfolio{}, fmt.Errorf("gold: save portfolio %d: %w", userID, storage.ErrConflict) } -func loadPortfolioForUpdate(ctx context.Context, vs storage.VersionedStore, key string, now int64) (Portfolio, int64, error) { - raw, version, err := vs.GetVersioned(ctx, key) +func loadPortfolioForUpdate(ctx context.Context, store PortfolioStore, key string, now int64) (Portfolio, int64, error) { + p, version, err := store.Get(ctx, key) switch { case err == nil: - var p Portfolio - if err := json.Unmarshal(raw, &p); err != nil { - return Portfolio{}, 0, fmt.Errorf("json decode: %w", err) - } p.normalize() if p.Meta.CreatedAt == 0 { p.Meta.CreatedAt = now diff --git a/internal/modules/gold/portfolio_test.go b/internal/modules/gold/portfolio_test.go index 28896f6..0828314 100644 --- a/internal/modules/gold/portfolio_test.go +++ b/internal/modules/gold/portfolio_test.go @@ -12,9 +12,13 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) +// newGoldStore returns a fresh in-memory typed portfolio store for tests. +func newGoldStore() PortfolioStore { + return storage.Typed[Portfolio](storage.NewMemoryProvider().Collection("gold")) +} + func TestLoadPortfolio_FirstTimeUser(t *testing.T) { - kv := storage.NewMemoryKVStore() - p, err := LoadPortfolio(context.Background(), kv, 42, 123) + p, err := LoadPortfolio(context.Background(), newGoldStore(), 42, 123) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -24,15 +28,15 @@ func TestLoadPortfolio_FirstTimeUser(t *testing.T) { } func TestSaveAndLoadRoundTrip(t *testing.T) { - kv := storage.NewMemoryKVStore() + store := newGoldStore() p := NewPortfolio(1) p.AddVND(5_000_000) p.AddLuong(1.25) p.Meta.Invested = 5_000_000 - if err := SavePortfolio(context.Background(), kv, 42, p); err != nil { + if err := SavePortfolio(context.Background(), store, 42, p); err != nil { t.Fatalf("Save: %v", err) } - got, err := LoadPortfolio(context.Background(), kv, 42, 999) + got, err := LoadPortfolio(context.Background(), store, 42, 999) if err != nil { t.Fatalf("Load: %v", err) } @@ -71,32 +75,31 @@ func TestNormalizeAmountSpecialValues(t *testing.T) { } } +// conflictOnceStore wraps a real typed store and forces exactly one write +// conflict (after committing a competing value) before delegating, to exercise +// UpdatePortfolio's optimistic-lock retry. type conflictOnceStore struct { - storage.KVStore + PortfolioStore conflicted bool } -func (s *conflictOnceStore) GetVersioned(ctx context.Context, key string) ([]byte, int64, error) { - return s.KVStore.(storage.VersionedStore).GetVersioned(ctx, key) -} - -func (s *conflictOnceStore) PutVersioned(ctx context.Context, key string, expectedVersion int64, val []byte) error { +func (s *conflictOnceStore) PutVersioned(ctx context.Context, key string, expectedVersion int64, val Portfolio) error { if !s.conflicted { s.conflicted = true competing := NewPortfolio(1) competing.AddVND(10) - if err := s.PutJSON(ctx, key, competing); err != nil { + if err := s.PortfolioStore.Put(ctx, key, competing); err != nil { return err } return storage.ErrConflict } - return s.KVStore.(storage.VersionedStore).PutVersioned(ctx, key, expectedVersion, val) + return s.PortfolioStore.PutVersioned(ctx, key, expectedVersion, val) } func TestUpdatePortfolioRetriesAfterWriteConflict(t *testing.T) { ctx := context.Background() - kv := &conflictOnceStore{KVStore: storage.NewMemoryKVStore()} - got, err := UpdatePortfolio(ctx, kv, 7, 1, func(p *Portfolio) error { + store := &conflictOnceStore{PortfolioStore: newGoldStore()} + got, err := UpdatePortfolio(ctx, store, 7, 1, func(p *Portfolio) error { p.AddVND(5) return nil }) @@ -106,7 +109,7 @@ func TestUpdatePortfolioRetriesAfterWriteConflict(t *testing.T) { if got.VND != 15 { t.Fatalf("updated portfolio VND = %v, want 15", got.VND) } - loaded, err := LoadPortfolio(ctx, kv, 7, 1) + loaded, err := LoadPortfolio(ctx, store, 7, 1) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -115,53 +118,51 @@ func TestUpdatePortfolioRetriesAfterWriteConflict(t *testing.T) { } } +// alwaysConflictStore always returns ErrConflict on PutVersioned to exhaust retries. type alwaysConflictStore struct { - storage.KVStore + PortfolioStore attempts int } -func (s *alwaysConflictStore) GetVersioned(ctx context.Context, key string) ([]byte, int64, error) { - return s.KVStore.(storage.VersionedStore).GetVersioned(ctx, key) -} - -func (s *alwaysConflictStore) PutVersioned(context.Context, string, int64, []byte) error { +func (s *alwaysConflictStore) PutVersioned(_ context.Context, _ string, _ int64, _ Portfolio) error { s.attempts++ return storage.ErrConflict } func TestUpdatePortfolioReturnsConflictAfterExhaustingRetries(t *testing.T) { ctx := context.Background() - kv := &alwaysConflictStore{KVStore: storage.NewMemoryKVStore()} - _, err := UpdatePortfolio(ctx, kv, 7, 1, func(p *Portfolio) error { + store := &alwaysConflictStore{PortfolioStore: newGoldStore()} + _, err := UpdatePortfolio(ctx, store, 7, 1, func(p *Portfolio) error { p.AddVND(5) return nil }) if !errors.Is(err, storage.ErrConflict) { t.Fatalf("UpdatePortfolio: got %v, want wrapped ErrConflict", err) } - if kv.attempts != portfolioUpdateAttempts { - t.Errorf("CAS attempts = %d, want %d", kv.attempts, portfolioUpdateAttempts) + if store.attempts != portfolioUpdateAttempts { + t.Errorf("CAS attempts = %d, want %d", store.attempts, portfolioUpdateAttempts) } - if _, err := kv.Get(ctx, "user:7"); !errors.Is(err, storage.ErrNotFound) { + if _, _, err := store.Get(ctx, "user:7"); !errors.Is(err, storage.ErrNotFound) { t.Errorf("exhausted update must not persist anything, Get = %v", err) } } +// countingCASStore counts PutVersioned calls and delegates to the underlying store. type countingCASStore struct { - *storage.MemoryKVStore + PortfolioStore casCalls int } -func (s *countingCASStore) PutVersioned(ctx context.Context, key string, expectedVersion int64, val []byte) error { +func (s *countingCASStore) PutVersioned(ctx context.Context, key string, expectedVersion int64, val Portfolio) error { s.casCalls++ - return s.MemoryKVStore.PutVersioned(ctx, key, expectedVersion, val) + return s.PortfolioStore.PutVersioned(ctx, key, expectedVersion, val) } func TestUpdatePortfolioMutateErrorDoesNotRetryOrPersist(t *testing.T) { ctx := context.Background() - kv := &countingCASStore{MemoryKVStore: storage.NewMemoryKVStore()} + store := &countingCASStore{PortfolioStore: newGoldStore()} mutateCalls := 0 - _, err := UpdatePortfolio(ctx, kv, 7, 1, func(p *Portfolio) error { + _, err := UpdatePortfolio(ctx, store, 7, 1, func(p *Portfolio) error { mutateCalls++ return errInsufficientVND }) @@ -171,17 +172,17 @@ func TestUpdatePortfolioMutateErrorDoesNotRetryOrPersist(t *testing.T) { if mutateCalls != 1 { t.Errorf("mutate calls = %d, want 1 (business errors must not retry)", mutateCalls) } - if kv.casCalls != 0 { - t.Errorf("CAS calls = %d, want 0 (failed mutate must not write)", kv.casCalls) + if store.casCalls != 0 { + t.Errorf("CAS calls = %d, want 0 (failed mutate must not write)", store.casCalls) } - if _, err := kv.Get(ctx, "user:7"); !errors.Is(err, storage.ErrNotFound) { + if _, _, err := store.Get(ctx, "user:7"); !errors.Is(err, storage.ErrNotFound) { t.Errorf("failed mutate must not persist anything, Get = %v", err) } } func TestUpdatePortfolioConcurrentIncrementsLoseNoUpdates(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + store := newGoldStore() const goroutines = 16 var successes atomic.Int64 var wg sync.WaitGroup @@ -190,7 +191,7 @@ func TestUpdatePortfolioConcurrentIncrementsLoseNoUpdates(t *testing.T) { wg.Add(1) go func() { defer wg.Done() - _, err := UpdatePortfolio(ctx, kv, 7, 1, func(p *Portfolio) error { + _, err := UpdatePortfolio(ctx, store, 7, 1, func(p *Portfolio) error { p.AddVND(1000) return nil }) @@ -211,7 +212,7 @@ func TestUpdatePortfolioConcurrentIncrementsLoseNoUpdates(t *testing.T) { if successes.Load() == 0 { t.Fatal("no update succeeded") } - loaded, err := LoadPortfolio(ctx, kv, 7, 1) + loaded, err := LoadPortfolio(ctx, store, 7, 1) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -220,61 +221,38 @@ func TestUpdatePortfolioConcurrentIncrementsLoseNoUpdates(t *testing.T) { } } -// plainKVStore hides the embedded store's versioned methods to model a backend -// without optimistic-locking support. -type plainKVStore struct { - storage.KVStore -} - -func TestUpdatePortfolioFailsFastWhenCASUnsupported(t *testing.T) { - ctx := context.Background() - mutate := func(p *Portfolio) error { - p.AddVND(5) - return nil - } - - // A bare non-CAS store is rejected by the capability check. - if _, err := UpdatePortfolio(ctx, &plainKVStore{KVStore: storage.NewMemoryKVStore()}, 7, 1, mutate); err == nil { - t.Error("UpdatePortfolio on non-CAS store: want error, got nil") - } - - // A prefixed wrapper around a non-CAS store must fail fast as unsupported, - // not burn retries on a fake conflict. - _, err := UpdatePortfolio(ctx, storage.Prefixed(&plainKVStore{KVStore: storage.NewMemoryKVStore()}, "gold"), 7, 1, mutate) - if !errors.Is(err, errors.ErrUnsupported) { - t.Errorf("got %v, want errors.ErrUnsupported", err) - } - if errors.Is(err, storage.ErrConflict) { - t.Error("missing CAS capability must not be reported as a write conflict") - } -} - func TestStockAndGoldPortfolioKeysDoNotCollide(t *testing.T) { ctx := context.Background() provider := storage.NewMemoryProvider() + + goldStore := storage.Typed[Portfolio](provider.Collection("gold")) goldPortfolio := NewPortfolio(1) goldPortfolio.AddLuong(2) - if err := SavePortfolio(ctx, provider.For("gold"), 7, goldPortfolio); err != nil { + if err := SavePortfolio(ctx, goldStore, 7, goldPortfolio); err != nil { t.Fatalf("save gold: %v", err) } + + stockStore := storage.Typed[stockmod.Portfolio](provider.Collection("stock")) stockPortfolio := stockmod.NewPortfolio(1) stockPortfolio.AddAsset("TCB", 100) - if err := stockmod.SavePortfolio(ctx, provider.For("stock"), 7, stockPortfolio); err != nil { + if err := stockmod.SavePortfolio(ctx, stockStore, 7, stockPortfolio); err != nil { t.Fatalf("save stock: %v", err) } - keys, err := provider.Base().List(ctx, "") + + // Both stores share the same provider but different collections (module + // isolation); verify each module's key is visible only in its own collection. + goldKeys, err := goldStore.List(ctx, "") if err != nil { - t.Fatalf("list: %v", err) + t.Fatalf("list gold: %v", err) } - want := map[string]bool{"gold:user:7": false, "stock:user:7": false} - for _, key := range keys { - if _, ok := want[key]; ok { - want[key] = true - } + stockKeys, err := stockStore.List(ctx, "") + if err != nil { + t.Fatalf("list stock: %v", err) } - for key, seen := range want { - if !seen { - t.Fatalf("missing raw key %q in %v", key, keys) - } + if len(goldKeys) != 1 || goldKeys[0] != "user:7" { + t.Fatalf("gold keys: got %v, want [user:7]", goldKeys) + } + if len(stockKeys) != 1 || stockKeys[0] != "user:7" { + t.Fatalf("stock keys: got %v, want [user:7]", stockKeys) } } diff --git a/internal/modules/gold/vnappmob_client.go b/internal/modules/gold/vnappmob_client.go index 7b6b2cd..ccd04bd 100644 --- a/internal/modules/gold/vnappmob_client.go +++ b/internal/modules/gold/vnappmob_client.go @@ -25,14 +25,20 @@ const ( vnappmobHTTPTimeout = 3 * time.Second // kept under the handler deadline; see chathelper.FetchContext ) +// apiKeyCache is the persisted shape for the cached VNAppMob JWT. +type apiKeyCache struct { + Token string `json:"token" bson:"token"` + Exp int64 `json:"exp" bson:"exp"` +} + // VNAppMobClient fetches Vietnam SJC gold prices from api.vnappmob.com. -// It self-manages a free JWT API key, caching it in KV and refreshing it -// before expiry or when the SJC endpoint returns 403. +// It self-manages a free JWT API key, caching it in the typed store and +// refreshing it before expiry or when the SJC endpoint returns 403. type VNAppMobClient struct { HTTP *http.Client - BaseURL string // optional override; default https://api.vnappmob.com - Token string // optional env override (GOLD_VNAPP_API_KEY) - KV storage.KVStore // module-scoped KV store + BaseURL string // optional override; default https://api.vnappmob.com + Token string // optional env override (GOLD_VNAPP_API_KEY) + cache storage.DocStore[apiKeyCache] // module-scoped typed cache store nowFn func() time.Time mu sync.Mutex @@ -42,11 +48,11 @@ type VNAppMobClient struct { // GOLD_VNAPP_API_KEY from the environment. The API key env var is intended // for local dev or SSM injection; when empty the client refreshes the key // automatically via the VNAppMob refresh endpoint. -func NewVNAppMobClientFromEnv(kv storage.KVStore) *VNAppMobClient { +func NewVNAppMobClientFromEnv(coll storage.Collection) *VNAppMobClient { return &VNAppMobClient{ BaseURL: strings.TrimSpace(os.Getenv("GOLD_VNAPP_API_URL")), Token: strings.TrimSpace(os.Getenv("GOLD_VNAPP_API_KEY")), - KV: kv, + cache: storage.Typed[apiKeyCache](coll), } } @@ -161,8 +167,8 @@ type httpStatusError struct { func (e *httpStatusError) Error() string { return e.msg } -// getKey returns a valid API key, refreshing from KV or the remote endpoint -// when the current key is missing or close to expiry. +// getKey returns a valid API key, refreshing from the typed cache store or the +// remote endpoint when the current key is missing or close to expiry. func (c *VNAppMobClient) getKey(ctx context.Context) (string, error) { if c.Token != "" { return c.Token, nil @@ -171,11 +177,8 @@ func (c *VNAppMobClient) getKey(ctx context.Context) (string, error) { c.mu.Lock() defer c.mu.Unlock() - var cached struct { - Token string `json:"token"` - Exp int64 `json:"exp"` - } - if err := c.KV.GetJSON(ctx, vnappmobKeyCacheKey, &cached); err == nil { + cached, _, err := c.cache.Get(ctx, vnappmobKeyCacheKey) + if err == nil { if cached.Token != "" && !c.isExpired(cached.Exp) { return cached.Token, nil } @@ -187,8 +190,9 @@ func (c *VNAppMobClient) getKey(ctx context.Context) (string, error) { return "", err } - // Re-read from KV to get the freshly stored key. - if err := c.KV.GetJSON(ctx, vnappmobKeyCacheKey, &cached); err != nil { + // Re-read from cache to get the freshly stored key. + cached, _, err = c.cache.Get(ctx, vnappmobKeyCacheKey) + if err != nil { return "", fmt.Errorf("vnappmob: read refreshed key: %w", err) } if cached.Token == "" { @@ -197,7 +201,7 @@ func (c *VNAppMobClient) getKey(ctx context.Context) (string, error) { return cached.Token, nil } -// refreshKey acquires a new JWT from VNAppMob and stores it in KV. +// refreshKey acquires a new JWT from VNAppMob and stores it in the typed cache. // It is safe to call concurrently; last-write-wins is acceptable because all // callers receive a valid key. func (c *VNAppMobClient) refreshKey(ctx context.Context) error { @@ -256,10 +260,7 @@ func (c *VNAppMobClient) refreshKeyLocked(ctx context.Context) error { // Store anyway; next call will refresh if expiry cannot be verified. } - if err := c.KV.PutJSON(ctx, vnappmobKeyCacheKey, struct { - Token string `json:"token"` - Exp int64 `json:"exp"` - }{Token: token, Exp: exp}); err != nil { + if err := c.cache.Put(ctx, vnappmobKeyCacheKey, apiKeyCache{Token: token, Exp: exp}); err != nil { return fmt.Errorf("vnappmob: cache key: %w", err) } log.Info("vnappmob_key_refreshed", "exp", exp) diff --git a/internal/modules/gold/vnappmob_client_test.go b/internal/modules/gold/vnappmob_client_test.go index 188cf7d..1ca5c5c 100644 --- a/internal/modules/gold/vnappmob_client_test.go +++ b/internal/modules/gold/vnappmob_client_test.go @@ -16,11 +16,11 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) -func newTestVNAppMobClient(srv *httptest.Server, kv storage.KVStore) *VNAppMobClient { +func newTestVNAppMobClient(srv *httptest.Server, coll storage.Collection) *VNAppMobClient { return &VNAppMobClient{ HTTP: srv.Client(), BaseURL: srv.URL, - KV: kv, + cache: storage.Typed[apiKeyCache](coll), nowFn: func() time.Time { return time.Unix(1000, 0) }, } } @@ -33,6 +33,10 @@ func makeJWT(exp int64) string { return h + "." + p + ".dummy-signature" } +func newTestColl() storage.Collection { + return storage.NewMemoryProvider().Collection("gold") +} + func TestJWTExp(t *testing.T) { now := time.Unix(1000, 0) tests := []struct { @@ -114,8 +118,7 @@ func TestRefreshKey(t *testing.T) { })) defer srv.Close() - kv := storage.NewMemoryKVStore() - c := newTestVNAppMobClient(srv, kv) + c := newTestVNAppMobClient(srv, newTestColl()) key, err := c.getKey(context.Background()) if err != nil { @@ -152,7 +155,7 @@ func TestRefreshKey_ObjectWrapper(t *testing.T) { })) defer srv.Close() - c := newTestVNAppMobClient(srv, storage.NewMemoryKVStore()) + c := newTestVNAppMobClient(srv, newTestColl()) key, err := c.getKey(context.Background()) if err != nil { t.Fatalf("getKey: %v", err) @@ -170,7 +173,7 @@ func TestRefreshKey_QuotedString(t *testing.T) { })) defer srv.Close() - c := newTestVNAppMobClient(srv, storage.NewMemoryKVStore()) + c := newTestVNAppMobClient(srv, newTestColl()) key, err := c.getKey(context.Background()) if err != nil { t.Fatalf("getKey: %v", err) @@ -198,8 +201,7 @@ func TestFetchSJCPrice(t *testing.T) { })) defer srv.Close() - kv := storage.NewMemoryKVStore() - c := newTestVNAppMobClient(srv, kv) + c := newTestVNAppMobClient(srv, newTestColl()) buy, sell, err := c.FetchSJCPrice(context.Background()) if err != nil { t.Fatalf("FetchSJCPrice: %v", err) @@ -232,7 +234,7 @@ func TestFetchSJCPrice_401Or403Refreshes(t *testing.T) { })) defer srv.Close() - c := newTestVNAppMobClient(srv, storage.NewMemoryKVStore()) + c := newTestVNAppMobClient(srv, newTestColl()) buy, sell, err := c.FetchSJCPrice(context.Background()) if err != nil { t.Fatalf("FetchSJCPrice: %v", err) @@ -265,8 +267,7 @@ func TestFetchSJCPrice_FallbackError(t *testing.T) { })) defer srv.Close() - kv := storage.NewMemoryKVStore() - c := newTestVNAppMobClient(srv, kv) + c := newTestVNAppMobClient(srv, newTestColl()) _, _, err := c.FetchSJCPrice(context.Background()) if !errors.Is(err, ErrNoGoldPrice) { t.Fatalf("got %v, want ErrNoGoldPrice", err) @@ -294,7 +295,7 @@ func TestFetchSJCPrice_InvalidValues(t *testing.T) { } })) defer srv.Close() - c := newTestVNAppMobClient(srv, storage.NewMemoryKVStore()) + c := newTestVNAppMobClient(srv, newTestColl()) _, _, err := c.FetchSJCPrice(context.Background()) if err == nil { t.Fatal("expected error for invalid values") diff --git a/internal/modules/loldle/handlers.go b/internal/modules/loldle/handlers.go index 3e745d1..3b3a512 100644 --- a/internal/modules/loldle/handlers.go +++ b/internal/modules/loldle/handlers.go @@ -12,7 +12,6 @@ import ( "github.com/tiennm99/miti99bot/internal/keylock" "github.com/tiennm99/miti99bot/internal/modules/util/chathelper" - "github.com/tiennm99/miti99bot/internal/storage" ) const newRoundHint = "🆕 Send /loldle or /loldle <champion> to start a new round." @@ -20,7 +19,9 @@ const newRoundHint = "🆕 Send /loldle or /loldle <champi // state captures everything a loldle handler needs at runtime. Built once // per Factory call and shared across the four command closures. type state struct { - kv storage.KVStore + games GameStore + stats StatsStore + cfg ConfigStore champions []Champion locks keylock.Map // serialises Get→mutate→Put per subject } @@ -63,7 +64,7 @@ func (s *state) startFreshGame(ctx context.Context, subject string) (*gameState, Guesses: []string{}, StartedAt: nil, } - if err := saveGame(ctx, s.kv, subject, g); err != nil { + if err := saveGame(ctx, s.games, subject, g); err != nil { return nil, err } return g, nil @@ -72,7 +73,7 @@ func (s *state) startFreshGame(ctx context.Context, subject string) (*gameState, // getOrInitGame loads the active round; if absent (or the saved round has // already exhausted its budget — defensive), starts a fresh one. func (s *state) getOrInitGame(ctx context.Context, subject string, maxGuesses int) (*gameState, error) { - existing, err := loadGame(ctx, s.kv, subject) + existing, err := loadGame(ctx, s.games, subject) if err != nil { return nil, err } @@ -113,7 +114,7 @@ func (s *state) handleLoldle(ctx context.Context, b *bot.Bot, update *models.Upd defer s.locks.Acquire(subject)() arg := chathelper.ArgAfterCommand(msg.Text) - maxGuesses, err := getMaxGuesses(ctx, s.kv, subject) + maxGuesses, err := getMaxGuesses(ctx, s.cfg, subject) if err != nil { return err } @@ -146,7 +147,7 @@ func (s *state) handleLoldle(ctx context.Context, b *bot.Bot, update *models.Upd if target == nil { // champions.json was refreshed and the target is gone. Drop the // round so the next /loldle starts cleanly. - if err := clearGame(ctx, s.kv, subject); err != nil { + if err := clearGame(ctx, s.games, subject); err != nil { return err } return chathelper.ReplyHTML(ctx, b, msg, @@ -167,11 +168,11 @@ func (s *state) handleLoldle(ctx context.Context, b *bot.Bot, update *models.Upd switch { case won: - st, err := recordResult(ctx, s.kv, subject, true) + st, err := recordResult(ctx, s.stats, subject, true) if err != nil { return err } - if err := clearGame(ctx, s.kv, subject); err != nil { + if err := clearGame(ctx, s.games, subject); err != nil { return err } trySendSticker(ctx, b, msg, winStickers) @@ -181,10 +182,10 @@ func (s *state) handleLoldle(ctx context.Context, b *bot.Bot, update *models.Upd rendered, flavor, champ, elapsed, st.Streak, len(game.Guesses), maxGuesses, newRoundHint)) case len(game.Guesses) >= maxGuesses: - if _, err := recordResult(ctx, s.kv, subject, false); err != nil { + if _, err := recordResult(ctx, s.stats, subject, false); err != nil { return err } - if err := clearGame(ctx, s.kv, subject); err != nil { + if err := clearGame(ctx, s.games, subject); err != nil { return err } trySendSticker(ctx, b, msg, loseStickers) @@ -192,7 +193,7 @@ func (s *state) handleLoldle(ctx context.Context, b *bot.Bot, update *models.Upd "%s\n\n❌ Out of guesses. Answer was %s.\n%s", rendered, champ, newRoundHint)) default: - if err := saveGame(ctx, s.kv, subject, game); err != nil { + if err := saveGame(ctx, s.games, subject, game); err != nil { return err } return chathelper.ReplyHTML(ctx, b, msg, fmt.Sprintf( @@ -212,18 +213,18 @@ func (s *state) handleGiveup(ctx context.Context, b *bot.Bot, update *models.Upd } defer s.locks.Acquire(subject)() - existing, err := loadGame(ctx, s.kv, subject) + existing, err := loadGame(ctx, s.games, subject) if err != nil { return err } if existing == nil { return chathelper.ReplyHTML(ctx, b, msg, "No active round. "+newRoundHint) } - if _, err := recordResult(ctx, s.kv, subject, false); err != nil { + if _, err := recordResult(ctx, s.stats, subject, false); err != nil { return err } target := s.findByName(existing.Target) - if err := clearGame(ctx, s.kv, subject); err != nil { + if err := clearGame(ctx, s.games, subject); err != nil { return err } trySendSticker(ctx, b, msg, giveupStickers) @@ -245,7 +246,7 @@ func (s *state) handleStats(ctx context.Context, b *bot.Bot, update *models.Upda if subject == "" { return chathelper.Reply(ctx, b, msg, "Cannot identify chat.") } - st, err := loadStats(ctx, s.kv, subject) + st, err := loadStats(ctx, s.stats, subject) if err != nil { return err } @@ -274,7 +275,7 @@ func (s *state) handleSetMax(ctx context.Context, b *bot.Bot, update *models.Upd if err != nil || n < 1 || n > MaxGuessesCap { return chathelper.Reply(ctx, b, msg, fmt.Sprintf("Usage: /loldle_setmax <1-%d>", MaxGuessesCap)) } - if err := setMaxGuesses(ctx, s.kv, subject, n); err != nil { + if err := setMaxGuesses(ctx, s.cfg, subject, n); err != nil { return err } return chathelper.Reply(ctx, b, msg, fmt.Sprintf("✅ Loldle max guesses set to %d (applies to the next round).", n)) diff --git a/internal/modules/loldle/handlers_test.go b/internal/modules/loldle/handlers_test.go index b4de1fb..b1bfc9a 100644 --- a/internal/modules/loldle/handlers_test.go +++ b/internal/modules/loldle/handlers_test.go @@ -13,12 +13,13 @@ import ( // installLoldle wires the loldle module + auth (owner gates /loldle_setmax, // which is private). seedTarget pre-seeds a game so guess outcomes are // deterministic without hooking math/rand. -func installLoldle(t *testing.T, ownerID int64, seedSubject, seedTarget string) (*testutil.RecordingBot, storage.KVStore) { +func installLoldle(t *testing.T, ownerID int64, seedSubject, seedTarget string) (*testutil.RecordingBot, GameStore, ConfigStore) { t.Helper() rb := testutil.NewRecordingBot(t) - provider := storage.NewMemoryProvider() - kv := provider.For("loldle") - mod := New(modules.Deps{KV: kv}) + col := storage.NewMemoryProvider().Collection("loldle") + games := storage.Typed[gameState](col) + cfg := storage.Typed[roundConfig](col) + mod := New(modules.Deps{Store: col}) reg := &modules.Registry{ Modules: []modules.Module{{Name: "loldle", Commands: mod.Commands}}, AllCommands: map[string]modules.Command{}, @@ -30,15 +31,15 @@ func installLoldle(t *testing.T, ownerID int64, seedSubject, seedTarget string) if seedTarget != "" { g := &gameState{Target: seedTarget, Guesses: []string{}} - if err := saveGame(context.Background(), kv, seedSubject, g); err != nil { + if err := saveGame(context.Background(), games, seedSubject, g); err != nil { t.Fatalf("seed game: %v", err) } } - return rb, kv + return rb, games, cfg } func TestLoldle_NoArgShowsBoard(t *testing.T) { - rb, _ := installLoldle(t, 0, "1", "Aatrox") + rb, _, _ := installLoldle(t, 0, "1", "Aatrox") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(1, "/loldle")) got := rb.LastSent().Text() @@ -48,7 +49,7 @@ func TestLoldle_NoArgShowsBoard(t *testing.T) { } func TestLoldle_Win(t *testing.T) { - rb, _ := installLoldle(t, 0, "1", "Aatrox") + rb, _, _ := installLoldle(t, 0, "1", "Aatrox") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(1, "/loldle aatrox")) got := rb.LastSent().Text() @@ -61,7 +62,7 @@ func TestLoldle_Win(t *testing.T) { } func TestLoldle_UnknownChampion(t *testing.T) { - rb, _ := installLoldle(t, 0, "1", "Aatrox") + rb, _, _ := installLoldle(t, 0, "1", "Aatrox") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(1, "/loldle ZilbeanZ")) got := rb.LastSent().Text() @@ -71,7 +72,7 @@ func TestLoldle_UnknownChampion(t *testing.T) { } func TestLoldle_DuplicateGuessRejected(t *testing.T) { - rb, _ := installLoldle(t, 0, "1", "Aatrox") + rb, _, _ := installLoldle(t, 0, "1", "Aatrox") // First guess: Ahri (not the target — round continues). rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(1, "/loldle ahri")) rb.Reset() @@ -85,7 +86,7 @@ func TestLoldle_DuplicateGuessRejected(t *testing.T) { } func TestLoldleGiveup_RevealsAnswer(t *testing.T) { - rb, _ := installLoldle(t, 0, "1", "Aatrox") + rb, _, _ := installLoldle(t, 0, "1", "Aatrox") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(1, "/loldle_giveup")) got := rb.LastSent().Text() @@ -95,7 +96,7 @@ func TestLoldleGiveup_RevealsAnswer(t *testing.T) { } func TestLoldleGiveup_NoActiveRound(t *testing.T) { - rb, _ := installLoldle(t, 0, "", "") + rb, _, _ := installLoldle(t, 0, "", "") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(1, "/loldle_giveup")) got := rb.LastSent().Text() @@ -105,7 +106,7 @@ func TestLoldleGiveup_NoActiveRound(t *testing.T) { } func TestLoldleStats_Empty(t *testing.T) { - rb, _ := installLoldle(t, 0, "", "") + rb, _, _ := installLoldle(t, 0, "", "") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(1, "/loldle_stats")) got := rb.LastSent().Text() @@ -117,24 +118,24 @@ func TestLoldleStats_Empty(t *testing.T) { } func TestLoldleSetMax_OwnerSucceeds(t *testing.T) { - rb, kv := installLoldle(t, 999, "", "") + rb, _, cfg := installLoldle(t, 999, "", "") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(999, "/loldle_setmax 5")) got := rb.LastSent().Text() if !strings.Contains(got, "max guesses set to 5") { t.Errorf("/loldle_setmax reply: %q", got) } - var cfg roundConfig - if err := kv.GetJSON(context.Background(), configKey("999"), &cfg); err != nil { + c, _, err := cfg.Get(context.Background(), configKey("999")) + if err != nil { t.Fatalf("expected config persisted: %v", err) } - if cfg.MaxGuesses != 5 { - t.Errorf("MaxGuesses persisted = %d, want 5", cfg.MaxGuesses) + if c.MaxGuesses != 5 { + t.Errorf("MaxGuesses persisted = %d, want 5", c.MaxGuesses) } } func TestLoldleSetMax_DeniedToNonOwner(t *testing.T) { - rb, _ := installLoldle(t, 999, "", "") + rb, _, _ := installLoldle(t, 999, "", "") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(7, "/loldle_setmax 5")) if calls := rb.Sent(); len(calls) != 0 { @@ -143,7 +144,7 @@ func TestLoldleSetMax_DeniedToNonOwner(t *testing.T) { } func TestLoldleSetMax_RejectsOutOfRange(t *testing.T) { - rb, _ := installLoldle(t, 999, "", "") + rb, _, _ := installLoldle(t, 999, "", "") rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(999, "/loldle_setmax 99")) got := rb.LastSent().Text() diff --git a/internal/modules/loldle/loldle.go b/internal/modules/loldle/loldle.go index d7abce2..a8b903e 100644 --- a/internal/modules/loldle/loldle.go +++ b/internal/modules/loldle/loldle.go @@ -2,13 +2,16 @@ package loldle import ( "github.com/tiennm99/miti99bot/internal/modules" + "github.com/tiennm99/miti99bot/internal/storage" ) // New is the loldle module Factory. Loads champions.json once at construction // and shares the parsed slice + per-subject lock map across all handlers. func New(deps modules.Deps) modules.Module { s := &state{ - kv: deps.KV, + games: storage.Typed[gameState](deps.Store), + stats: storage.Typed[stats](deps.Store), + cfg: storage.Typed[roundConfig](deps.Store), champions: loadChampions(), } return modules.Module{ diff --git a/internal/modules/loldle/state.go b/internal/modules/loldle/state.go index 936611c..31125f3 100644 --- a/internal/modules/loldle/state.go +++ b/internal/modules/loldle/state.go @@ -15,7 +15,7 @@ const ( MaxGuessesCap = 10 ) -// gameState is the per-subject KV record. +// gameState is the per-subject record. // // StartedAt is *int64 because the timer doesn't start until the first guess; // using time.Time would marshal as "0001-01-01T00:00:00Z" instead of null @@ -25,35 +25,43 @@ const ( // time against current champions.json so a weekly data refresh updates // historical board displays without migrating saved rounds. type gameState struct { - Target string `json:"target"` - Guesses []string `json:"guesses"` - StartedAt *int64 `json:"startedAt"` // ms-since-epoch | null + Target string `json:"target" bson:"target"` + Guesses []string `json:"guesses" bson:"guesses"` + StartedAt *int64 `json:"startedAt" bson:"startedAt"` // ms-since-epoch | null } // stats lifetime score. No LastResultAt field by design (differs from // wordle's Stats — loldle only ever needed running streaks, not "last // played at"). type stats struct { - Played int `json:"played"` - Wins int `json:"wins"` - Streak int `json:"streak"` - BestStreak int `json:"bestStreak"` + Played int `json:"played" bson:"played"` + Wins int `json:"wins" bson:"wins"` + Streak int `json:"streak" bson:"streak"` + BestStreak int `json:"bestStreak" bson:"bestStreak"` } // roundConfig stores the per-subject MaxGuesses override. Stored only when // /loldle_setmax has been run; the absence of this record means "use default". type roundConfig struct { - MaxGuesses int `json:"maxGuesses"` + MaxGuesses int `json:"maxGuesses" bson:"maxGuesses"` } +// GameStore is the loldle module's typed game store. +type GameStore = storage.DocStore[gameState] + +// StatsStore is the loldle module's typed stats store. +type StatsStore = storage.DocStore[stats] + +// ConfigStore is the loldle module's typed config store. +type ConfigStore = storage.DocStore[roundConfig] + func gameKey(subject string) string { return "game:" + subject } func statsKey(subject string) string { return "stats:" + subject } func configKey(subject string) string { return "config:" + subject } // loadGame returns the active round, or (nil, nil) if none exists. -func loadGame(ctx context.Context, kv storage.KVStore, subject string) (*gameState, error) { - var g gameState - err := kv.GetJSON(ctx, gameKey(subject), &g) +func loadGame(ctx context.Context, games GameStore, subject string) (*gameState, error) { + g, _, err := games.Get(ctx, gameKey(subject)) switch { case err == nil: return &g, nil @@ -64,8 +72,8 @@ func loadGame(ctx context.Context, kv storage.KVStore, subject string) (*gameSta } } -func saveGame(ctx context.Context, kv storage.KVStore, subject string, g *gameState) error { - if err := kv.PutJSON(ctx, gameKey(subject), g); err != nil { +func saveGame(ctx context.Context, games GameStore, subject string, g *gameState) error { + if err := games.Put(ctx, gameKey(subject), *g); err != nil { return fmt.Errorf("loldle saveGame: %w", err) } return nil @@ -74,8 +82,8 @@ func saveGame(ctx context.Context, kv storage.KVStore, subject string, g *gameSt // clearGame removes the round so the next /loldle starts fresh. Used after // win / loss / giveup; the new round's timer should start on the player's // next interaction, not at the moment the previous round ended. -func clearGame(ctx context.Context, kv storage.KVStore, subject string) error { - if err := kv.Delete(ctx, gameKey(subject)); err != nil { +func clearGame(ctx context.Context, games GameStore, subject string) error { + if err := games.Delete(ctx, gameKey(subject)); err != nil { return fmt.Errorf("loldle clearGame: %w", err) } return nil @@ -83,9 +91,8 @@ func clearGame(ctx context.Context, kv storage.KVStore, subject string) error { // loadStats returns lifetime score; missing → fresh-zero record so callers // never need a nil check. -func loadStats(ctx context.Context, kv storage.KVStore, subject string) (*stats, error) { - var s stats - err := kv.GetJSON(ctx, statsKey(subject), &s) +func loadStats(ctx context.Context, st StatsStore, subject string) (*stats, error) { + s, _, err := st.Get(ctx, statsKey(subject)) switch { case err == nil: return &s, nil @@ -96,8 +103,8 @@ func loadStats(ctx context.Context, kv storage.KVStore, subject string) (*stats, } } -func recordResult(ctx context.Context, kv storage.KVStore, subject string, won bool) (*stats, error) { - s, err := loadStats(ctx, kv, subject) +func recordResult(ctx context.Context, st StatsStore, subject string, won bool) (*stats, error) { + s, err := loadStats(ctx, st, subject) if err != nil { return nil, err } @@ -111,7 +118,7 @@ func recordResult(ctx context.Context, kv storage.KVStore, subject string, won b } else { s.Streak = 0 } - if err := kv.PutJSON(ctx, statsKey(subject), s); err != nil { + if err := st.Put(ctx, statsKey(subject), *s); err != nil { return nil, fmt.Errorf("loldle recordResult: %w", err) } return s, nil @@ -120,27 +127,26 @@ func recordResult(ctx context.Context, kv storage.KVStore, subject string, won b // getMaxGuesses returns the effective round length: the per-subject override // if set and in range, otherwise MaxGuesses. Out-of-range values are // silently ignored — better to serve the default than 500 the user. -func getMaxGuesses(ctx context.Context, kv storage.KVStore, subject string) (int, error) { - var cfg roundConfig - err := kv.GetJSON(ctx, configKey(subject), &cfg) +func getMaxGuesses(ctx context.Context, cfg ConfigStore, subject string) (int, error) { + c, _, err := cfg.Get(ctx, configKey(subject)) if err != nil { if errors.Is(err, storage.ErrNotFound) { return MaxGuesses, nil } return 0, fmt.Errorf("loldle getMaxGuesses: %w", err) } - if cfg.MaxGuesses < 1 || cfg.MaxGuesses > MaxGuessesCap { + if c.MaxGuesses < 1 || c.MaxGuesses > MaxGuessesCap { return MaxGuesses, nil } - return cfg.MaxGuesses, nil + return c.MaxGuesses, nil } // setMaxGuesses validates and persists the per-subject override. -func setMaxGuesses(ctx context.Context, kv storage.KVStore, subject string, n int) error { +func setMaxGuesses(ctx context.Context, cfg ConfigStore, subject string, n int) error { if n < 1 || n > MaxGuessesCap { return fmt.Errorf("loldle: maxGuesses must be in [1, %d], got %d", MaxGuessesCap, n) } - if err := kv.PutJSON(ctx, configKey(subject), roundConfig{MaxGuesses: n}); err != nil { + if err := cfg.Put(ctx, configKey(subject), roundConfig{MaxGuesses: n}); err != nil { return fmt.Errorf("loldle setMaxGuesses: %w", err) } return nil diff --git a/internal/modules/loldle/state_test.go b/internal/modules/loldle/state_test.go index d933fdf..c2c236f 100644 --- a/internal/modules/loldle/state_test.go +++ b/internal/modules/loldle/state_test.go @@ -8,6 +8,25 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) +func newLoldleGames() GameStore { + return storage.Typed[gameState](storage.NewMemoryProvider().Collection("loldle")) +} + +func newLoldleStats() StatsStore { + return storage.Typed[stats](storage.NewMemoryProvider().Collection("loldle")) +} + +func newLoldleConfig() ConfigStore { + return storage.Typed[roundConfig](storage.NewMemoryProvider().Collection("loldle")) +} + +// newLoldleStores returns all three stores backed by the same collection +// (disjoint key prefixes: "game:", "stats:", "config:"). +func newLoldleStores() (GameStore, StatsStore, ConfigStore) { + c := storage.NewMemoryProvider().Collection("loldle") + return storage.Typed[gameState](c), storage.Typed[stats](c), storage.Typed[roundConfig](c) +} + func TestGameState_StartedAtNullByDefault(t *testing.T) { g := gameState{Target: "Aatrox", Guesses: []string{}} b, err := json.Marshal(g) @@ -43,20 +62,20 @@ func TestStats_NoLastResultAtField(t *testing.T) { func TestRecordResult_WinAndLossSequence(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + st := newLoldleStats() - s, err := recordResult(ctx, kv, "u1", true) + s, err := recordResult(ctx, st, "u1", true) if err != nil { t.Fatal(err) } if s.Played != 1 || s.Wins != 1 || s.Streak != 1 || s.BestStreak != 1 { t.Errorf("first win: %+v", s) } - s, _ = recordResult(ctx, kv, "u1", true) + s, _ = recordResult(ctx, st, "u1", true) if s.Streak != 2 || s.BestStreak != 2 { t.Errorf("two wins: %+v", s) } - s, _ = recordResult(ctx, kv, "u1", false) + s, _ = recordResult(ctx, st, "u1", false) if s.Streak != 0 { t.Errorf("loss should reset streak, got %d", s.Streak) } @@ -67,52 +86,52 @@ func TestRecordResult_WinAndLossSequence(t *testing.T) { func TestGetMaxGuesses_DefaultsAndOverrides(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + cfg := newLoldleConfig() - if n, _ := getMaxGuesses(ctx, kv, "u1"); n != MaxGuesses { + if n, _ := getMaxGuesses(ctx, cfg, "u1"); n != MaxGuesses { t.Errorf("missing config → %d, want %d", n, MaxGuesses) } - if err := setMaxGuesses(ctx, kv, "u1", 5); err != nil { + if err := setMaxGuesses(ctx, cfg, "u1", 5); err != nil { t.Fatal(err) } - if n, _ := getMaxGuesses(ctx, kv, "u1"); n != 5 { + if n, _ := getMaxGuesses(ctx, cfg, "u1"); n != 5 { t.Errorf("after setMax(5): %d, want 5", n) } } func TestSetMaxGuesses_ValidatesRange(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + cfg := newLoldleConfig() for _, n := range []int{0, -1, MaxGuessesCap + 1, 100} { - if err := setMaxGuesses(ctx, kv, "u1", n); err == nil { + if err := setMaxGuesses(ctx, cfg, "u1", n); err == nil { t.Errorf("setMaxGuesses(%d) should error", n) } } - if err := setMaxGuesses(ctx, kv, "u1", 1); err != nil { + if err := setMaxGuesses(ctx, cfg, "u1", 1); err != nil { t.Errorf("setMaxGuesses(1) should succeed: %v", err) } - if err := setMaxGuesses(ctx, kv, "u1", MaxGuessesCap); err != nil { + if err := setMaxGuesses(ctx, cfg, "u1", MaxGuessesCap); err != nil { t.Errorf("setMaxGuesses(cap) should succeed: %v", err) } } func TestGetMaxGuesses_OutOfRangeIgnored(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() - // Inject a corrupt config to simulate manual KV tampering or a stale + cfg := newLoldleConfig() + // Inject a corrupt config to simulate manual store tampering or a stale // schema; getMaxGuesses must fall back to the default rather than // returning a wild value to handlers. - if err := kv.PutJSON(ctx, configKey("u1"), roundConfig{MaxGuesses: 100}); err != nil { + if err := cfg.Put(ctx, configKey("u1"), roundConfig{MaxGuesses: 100}); err != nil { t.Fatal(err) } - if n, _ := getMaxGuesses(ctx, kv, "u1"); n != MaxGuesses { + if n, _ := getMaxGuesses(ctx, cfg, "u1"); n != MaxGuesses { t.Errorf("out-of-range config → %d, want %d (default)", n, MaxGuesses) } } func TestLoadGame_MissingReturnsNil(t *testing.T) { - g, err := loadGame(context.Background(), storage.NewMemoryKVStore(), "nobody") + g, err := loadGame(context.Background(), newLoldleGames(), "nobody") if err != nil { t.Errorf("missing should not error: %v", err) } @@ -123,13 +142,13 @@ func TestLoadGame_MissingReturnsNil(t *testing.T) { func TestSaveLoadClearGame_RoundTrip(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + games := newLoldleGames() at := int64(42) want := &gameState{Target: "Aatrox", Guesses: []string{"Ahri"}, StartedAt: &at} - if err := saveGame(ctx, kv, "u1", want); err != nil { + if err := saveGame(ctx, games, "u1", want); err != nil { t.Fatal(err) } - got, err := loadGame(ctx, kv, "u1") + got, err := loadGame(ctx, games, "u1") if err != nil || got == nil { t.Fatalf("loadGame: %v, %v", got, err) } @@ -137,10 +156,10 @@ func TestSaveLoadClearGame_RoundTrip(t *testing.T) { t.Errorf("round-trip mismatch: %+v", got) } - if err := clearGame(ctx, kv, "u1"); err != nil { + if err := clearGame(ctx, games, "u1"); err != nil { t.Fatal(err) } - got, _ = loadGame(ctx, kv, "u1") + got, _ = loadGame(ctx, games, "u1") if got != nil { t.Errorf("after clearGame, expected nil, got %+v", got) } diff --git a/internal/modules/lolschedule/api_client.go b/internal/modules/lolschedule/api_client.go index 07d70b3..37ddb7e 100644 --- a/internal/modules/lolschedule/api_client.go +++ b/internal/modules/lolschedule/api_client.go @@ -44,50 +44,61 @@ const ( ) // Team is one side of a match. JSON shape matches the lolesports response. +// bson tags mirror the json names exactly: this tree is persisted inside +// cacheRecord, and the DynamoDB→Mongo migrator preserves the original (camelCase) +// JSON keys, so the store must read those keys back verbatim — not the driver's +// lowercased default. type Team struct { - Name string `json:"name,omitempty"` - Code string `json:"code,omitempty"` - Image string `json:"image,omitempty"` - Result *struct { - Outcome string `json:"outcome,omitempty"` // "win" or "loss" - GameWins int `json:"gameWins,omitempty"` - } `json:"result,omitempty"` - Record *struct { - Wins int `json:"wins,omitempty"` - Losses int `json:"losses,omitempty"` - } `json:"record,omitempty"` + Name string `json:"name,omitempty" bson:"name,omitempty"` + Code string `json:"code,omitempty" bson:"code,omitempty"` + Image string `json:"image,omitempty" bson:"image,omitempty"` + Result *TeamResult `json:"result,omitempty" bson:"result,omitempty"` + Record *TeamRecord `json:"record,omitempty" bson:"record,omitempty"` +} + +// TeamResult is a team's per-match outcome (named so tests and the bson decoder +// share one type). +type TeamResult struct { + Outcome string `json:"outcome,omitempty" bson:"outcome,omitempty"` // "win" or "loss" + GameWins int `json:"gameWins,omitempty" bson:"gameWins,omitempty"` +} + +// TeamRecord is a team's running series record. +type TeamRecord struct { + Wins int `json:"wins,omitempty" bson:"wins,omitempty"` + Losses int `json:"losses,omitempty" bson:"losses,omitempty"` } // League holds the league-section-header info on each event. type League struct { - Name string `json:"name,omitempty"` - Slug string `json:"slug,omitempty"` - Image string `json:"image,omitempty"` + Name string `json:"name,omitempty" bson:"name,omitempty"` + Slug string `json:"slug,omitempty" bson:"slug,omitempty"` + Image string `json:"image,omitempty" bson:"image,omitempty"` } // Strategy is the bestOf descriptor (Bo1, Bo3, Bo5). type Strategy struct { - Type string `json:"type,omitempty"` - Count int `json:"count,omitempty"` + Type string `json:"type,omitempty" bson:"type,omitempty"` + Count int `json:"count,omitempty" bson:"count,omitempty"` } // Match is the inner match metadata. type Match struct { - ID string `json:"id,omitempty"` - Teams []Team `json:"teams,omitempty"` - Strategy Strategy `json:"strategy,omitempty"` + ID string `json:"id,omitempty" bson:"id,omitempty"` + Teams []Team `json:"teams,omitempty" bson:"teams,omitempty"` + Strategy Strategy `json:"strategy,omitempty" bson:"strategy,omitempty"` } // ScheduleEvent is one upcoming or past match. State is "unstarted", // "inProgress", or "completed". Type is set to "show" for pre/post-show // segments which we filter out. type ScheduleEvent struct { - StartTime string `json:"startTime"` - State string `json:"state,omitempty"` - Type string `json:"type,omitempty"` - BlockName string `json:"blockName,omitempty"` - League League `json:"league,omitempty"` - Match Match `json:"match,omitempty"` + StartTime string `json:"startTime" bson:"startTime"` + State string `json:"state,omitempty" bson:"state,omitempty"` + Type string `json:"type,omitempty" bson:"type,omitempty"` + BlockName string `json:"blockName,omitempty" bson:"blockName,omitempty"` + League League `json:"league,omitempty" bson:"league,omitempty"` + Match Match `json:"match,omitempty" bson:"match,omitempty"` } // schedulePage is the inner shape of an upstream response. @@ -103,12 +114,15 @@ type schedulePage struct { } `json:"data"` } -// cacheRecord is the KV value: fetch timestamp (ms-epoch) + events. +// cacheRecord is the store value: fetch timestamp (ms-epoch) + events. type cacheRecord struct { - Ts int64 `json:"ts"` // ms-since-epoch when fetched - Events []ScheduleEvent `json:"events"` + Ts int64 `json:"ts" bson:"ts"` // ms-since-epoch when fetched + Events []ScheduleEvent `json:"events" bson:"events"` } +// CacheStore is the typed store for schedule cache records. +type CacheStore = storage.DocStore[cacheRecord] + // Client is the lolesports API client. Default zero-value uses // http.DefaultClient + http.DefaultTransport; tests inject a custom HTTP // client (typically pointing at httptest.Server). @@ -274,12 +288,11 @@ func cacheKey(from, to time.Time) string { // GetEventsCached is the cache-first lookup. Returns fresh cache within // cacheTTL, else fetches upstream and writes back, else falls back to // stale cache (within staleMaxAge), else propagates the error. -func (c *Client) GetEventsCached(ctx context.Context, kv storage.KVStore, from, to time.Time) ([]ScheduleEvent, error) { +func (c *Client) GetEventsCached(ctx context.Context, cache CacheStore, from, to time.Time) ([]ScheduleEvent, error) { key := cacheKey(from, to) now := time.Now().UTC().UnixMilli() - var cached cacheRecord - cacheErr := kv.GetJSON(ctx, key, &cached) + cached, _, cacheErr := cache.Get(ctx, key) hasCached := cacheErr == nil if hasCached && now-cached.Ts < cacheTTL.Milliseconds() { return cached.Events, nil @@ -288,7 +301,7 @@ func (c *Client) GetEventsCached(ctx context.Context, kv storage.KVStore, from, events, fetchErr := c.fetchEventsInRange(ctx, from, to, 3) if fetchErr == nil { rec := cacheRecord{Ts: now, Events: events} - if err := kv.PutJSON(ctx, key, rec); err != nil { + if err := cache.Put(ctx, key, rec); err != nil { log.Warn("lolschedule_kv_put_fail", "err", err) } return events, nil diff --git a/internal/modules/lolschedule/api_client_test.go b/internal/modules/lolschedule/api_client_test.go index fc4d9b6..020d13f 100644 --- a/internal/modules/lolschedule/api_client_test.go +++ b/internal/modules/lolschedule/api_client_test.go @@ -13,6 +13,12 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) +// newCacheStore returns a fresh typed cache store backed by an in-memory +// collection — the per-test equivalent of what the factory wires in production. +func newCacheStore() CacheStore { + return storage.Typed[cacheRecord](storage.NewMemoryProvider().Collection("lolschedule")) +} + // mkServer spins an httptest.Server returning the supplied JSON body for // every page request. callCount counts upstream hits so cache tests can // assert "1 fetch, then no more". @@ -54,11 +60,11 @@ const sampleBody = `{ func TestGetEventsCached_FirstHitFetchesUpstream(t *testing.T) { srv, count := mkServer(t, sampleBody) c := &Client{HTTP: srv.Client(), URL: srv.URL} - kv := storage.NewMemoryKVStore() + cache := newCacheStore() from := time.Date(2026, 5, 9, 0, 0, 0, 0, time.UTC) to := from.Add(24 * time.Hour) - events, err := c.GetEventsCached(context.Background(), kv, from, to) + events, err := c.GetEventsCached(context.Background(), cache, from, to) if err != nil { t.Fatalf("first fetch: %v", err) } @@ -76,16 +82,16 @@ func TestGetEventsCached_FirstHitFetchesUpstream(t *testing.T) { func TestGetEventsCached_SecondHitUsesCache(t *testing.T) { srv, count := mkServer(t, sampleBody) c := &Client{HTTP: srv.Client(), URL: srv.URL} - kv := storage.NewMemoryKVStore() + cache := newCacheStore() from := time.Date(2026, 5, 9, 0, 0, 0, 0, time.UTC) to := from.Add(24 * time.Hour) // First fetch primes the cache. - if _, err := c.GetEventsCached(context.Background(), kv, from, to); err != nil { + if _, err := c.GetEventsCached(context.Background(), cache, from, to); err != nil { t.Fatal(err) } // Second fetch within TTL must NOT hit upstream. - if _, err := c.GetEventsCached(context.Background(), kv, from, to); err != nil { + if _, err := c.GetEventsCached(context.Background(), cache, from, to); err != nil { t.Fatal(err) } if got := atomic.LoadInt32(count); got != 1 { @@ -94,8 +100,8 @@ func TestGetEventsCached_SecondHitUsesCache(t *testing.T) { } func TestGetEventsCached_StaleFallback(t *testing.T) { - // Prime KV with a stale-but-still-fresh-enough cache record. - kv := storage.NewMemoryKVStore() + // Prime the typed cache store with a stale-but-still-fresh-enough record. + cache := newCacheStore() from := time.Date(2026, 5, 9, 0, 0, 0, 0, time.UTC) to := from.Add(24 * time.Hour) staleEvents := []ScheduleEvent{ @@ -103,7 +109,7 @@ func TestGetEventsCached_StaleFallback(t *testing.T) { } // 10 minutes ago — past the 120s fresh window but well inside 60-min stale. staleTs := time.Now().UTC().Add(-10 * time.Minute).UnixMilli() - if err := kv.PutJSON(context.Background(), cacheKey(from, to), cacheRecord{Ts: staleTs, Events: staleEvents}); err != nil { + if err := cache.Put(context.Background(), cacheKey(from, to), cacheRecord{Ts: staleTs, Events: staleEvents}); err != nil { t.Fatal(err) } @@ -115,7 +121,7 @@ func TestGetEventsCached_StaleFallback(t *testing.T) { defer srv.Close() c := &Client{HTTP: srv.Client(), URL: srv.URL} - got, err := c.GetEventsCached(context.Background(), kv, from, to) + got, err := c.GetEventsCached(context.Background(), cache, from, to) if err != nil { t.Fatalf("stale fallback should succeed: %v", err) } @@ -125,7 +131,7 @@ func TestGetEventsCached_StaleFallback(t *testing.T) { } func TestGetEventsCached_HardFailureWhenNoCache(t *testing.T) { - kv := storage.NewMemoryKVStore() + cache := newCacheStore() from := time.Date(2026, 5, 9, 0, 0, 0, 0, time.UTC) to := from.Add(24 * time.Hour) @@ -135,7 +141,7 @@ func TestGetEventsCached_HardFailureWhenNoCache(t *testing.T) { defer srv.Close() c := &Client{HTTP: srv.Client(), URL: srv.URL} - _, err := c.GetEventsCached(context.Background(), kv, from, to) + _, err := c.GetEventsCached(context.Background(), cache, from, to) if err == nil { t.Errorf("expected error when upstream fails AND no cache") } diff --git a/internal/modules/lolschedule/cron.go b/internal/modules/lolschedule/cron.go index 1123d40..80252d5 100644 --- a/internal/modules/lolschedule/cron.go +++ b/internal/modules/lolschedule/cron.go @@ -110,6 +110,15 @@ type messageSender interface { SendMessage(ctx context.Context, params *bot.SendMessageParams) (*models.Message, error) } +// lastPushDoc wraps the last-push date string so it can be stored as a named +// root field in a Mongo document (a bare scalar cannot be a root doc). +type lastPushDoc struct { + Date string `json:"date" bson:"date"` +} + +// PushDateStore is the typed store for last-push date documents. +type PushDateStore = storage.DocStore[lastPushDoc] + // dailyPushCron returns the cron registration. Schedule is documentation only. func (s *state) dailyPushCron() modules.Cron { return modules.Cron{ @@ -134,30 +143,13 @@ func (s *state) dailyPushHandler(ctx context.Context, deps modules.Deps) error { // for the daily push: a winner proceeds to fan out; a loser (another trigger // already claimed today) returns false and sends nothing. // -// The claim is a version-based optimistic write on lastPushDateKey so two -// simultaneous triggers cannot both win. Every real KV backend implements -// VersionedStore; if a bare store without it is used (only in narrow tests), it -// falls back to a plain Put — losing the atomic guarantee but preserving the -// "no-op if already today" behaviour. -func claimDailyPush(ctx context.Context, kv storage.KVStore, today string) (bool, error) { - vs, ok := kv.(storage.VersionedStore) - if !ok { - // No versioned support (a bare store in narrow tests): best-effort Put, - // losing the atomic guarantee but preserving "no-op if already today". - current, err := kv.Get(ctx, lastPushDateKey) - if err == nil && string(current) == today { - return false, nil - } - if err != nil && !errors.Is(err, storage.ErrNotFound) { - return false, err - } - return true, kv.Put(ctx, lastPushDateKey, []byte(today)) - } - - current, version, err := vs.GetVersioned(ctx, lastPushDateKey) +// The claim uses version-based optimistic write (PutVersioned) on +// lastPushDateKey so two simultaneous triggers cannot both win. +func claimDailyPush(ctx context.Context, store PushDateStore, today string) (bool, error) { + current, version, err := store.Get(ctx, lastPushDateKey) switch { case err == nil: - if string(current) == today { + if current.Date == today { return false, nil // already pushed today } case errors.Is(err, storage.ErrNotFound): @@ -166,7 +158,7 @@ func claimDailyPush(ctx context.Context, kv storage.KVStore, today string) (bool return false, err } - if err := vs.PutVersioned(ctx, lastPushDateKey, version, []byte(today)); err != nil { + if err := store.PutVersioned(ctx, lastPushDateKey, version, lastPushDoc{Date: today}); err != nil { if errors.Is(err, storage.ErrConflict) { return false, nil // another trigger claimed today first } @@ -182,7 +174,7 @@ func claimDailyPush(ctx context.Context, kv storage.KVStore, today string) (bool // MessageThreadID is forwarded on every send so subscribers in a forum-topic // receive the digest in that topic, not in General. func runDailyPush(ctx context.Context, s *state, sender messageSender) error { - subs, err := listSubscribers(ctx, s.kv) + subs, err := listSubscribers(ctx, s.subscribers) if err != nil { return fmt.Errorf("lolschedule daily push: list subscribers: %w", err) } @@ -193,7 +185,7 @@ func runDailyPush(ctx context.Context, s *state, sender messageSender) error { from := ictDayStartOf(s.now()) to := addDays(from, 1) - events, err := s.client.GetEventsCached(ctx, s.kv, from, to) + events, err := s.client.GetEventsCached(ctx, s.cache, from, to) if err != nil { return fmt.Errorf("lolschedule daily push: fetch matches: %w", err) } @@ -205,7 +197,7 @@ func runDailyPush(ctx context.Context, s *state, sender messageSender) error { // send nothing. Placed after the fetch so a transient fetch failure does // not consume the day's claim. today := s.now().UTC().Format("2006-01-02") - won, err := claimDailyPush(ctx, s.kv, today) + won, err := claimDailyPush(ctx, s.pushDate, today) if err != nil { return fmt.Errorf("lolschedule daily push: claim date: %w", err) } @@ -272,7 +264,7 @@ func pruneDeadSubscribers(ctx context.Context, s *state, chatWide map[int64]stru defer s.subscribersMu.Unlock() removed := 0 for chatID := range chatWide { - n, err := removeAllForChat(ctx, s.kv, chatID) + n, err := removeAllForChat(ctx, s.subscribers, chatID) if err != nil { log.Warn("lolschedule prune dead chat failed", "chat", chatID, "err", err) continue @@ -285,7 +277,7 @@ func pruneDeadSubscribers(ctx context.Context, s *state, chatWide map[int64]stru if _, ok := chatWide[sub.ChatID]; ok { continue } - ok, err := removeSubscriber(ctx, s.kv, sub.ChatID, sub.ThreadID) + ok, err := removeSubscriber(ctx, s.subscribers, sub.ChatID, sub.ThreadID) if err != nil { log.Warn("lolschedule prune dead topic failed", "chat", sub.ChatID, "thread", sub.ThreadID, "err", err) diff --git a/internal/modules/lolschedule/cron_test.go b/internal/modules/lolschedule/cron_test.go index ff89ee5..0cd6c23 100644 --- a/internal/modules/lolschedule/cron_test.go +++ b/internal/modules/lolschedule/cron_test.go @@ -56,9 +56,19 @@ func fixedNow() time.Time { return time.Date(2026, 5, 10, 5, 0, 0, 0, time.UTC) } +// newTestStore builds a fresh set of typed stores over a shared in-memory +// collection, matching what the factory wires in production. +func newTestStore(t *testing.T) (SubscriberStore, PushDateStore, CacheStore) { + t.Helper() + col := storage.NewMemoryProvider().Collection("lolschedule") + return storage.Typed[subscribersDoc](col), + storage.Typed[lastPushDoc](col), + storage.Typed[cacheRecord](col) +} + // seedFreshCache writes a cacheRecord with `now` as timestamp so the first // GetEventsCached call returns it without hitting the network. -func seedFreshCache(t *testing.T, kv storage.KVStore, events []ScheduleEvent) { +func seedFreshCache(t *testing.T, cache CacheStore, events []ScheduleEvent) { t.Helper() from := ictDayStartOf(fixedNow()) to := addDays(from, 1) @@ -66,24 +76,26 @@ func seedFreshCache(t *testing.T, kv storage.KVStore, events []ScheduleEvent) { Ts: time.Now().UTC().UnixMilli(), Events: events, } - if err := kv.PutJSON(context.Background(), cacheKey(from, to), rec); err != nil { + if err := cache.Put(context.Background(), cacheKey(from, to), rec); err != nil { t.Fatalf("seed cache: %v", err) } } func newTestState(t *testing.T) *state { t.Helper() - kv := storage.NewMemoryKVStore() + subs, pd, cache := newTestStore(t) return &state{ - kv: kv, - client: &Client{}, // zero value; tests must seed cache to avoid HTTP - nowFn: fixedNow, + subscribers: subs, + pushDate: pd, + cache: cache, + client: &Client{}, // zero value; tests must seed cache to avoid HTTP + nowFn: fixedNow, } } func TestRunDailyPush_NoSubscribers(t *testing.T) { s := newTestState(t) - seedFreshCache(t, s.kv, nil) + seedFreshCache(t, s.cache, nil) sender := &fakeSender{} if err := runDailyPush(context.Background(), s, sender); err != nil { @@ -96,11 +108,11 @@ func TestRunDailyPush_NoSubscribers(t *testing.T) { func TestRunDailyPush_SendsToAllSubscribers(t *testing.T) { s := newTestState(t) - seedFreshCache(t, s.kv, nil) // empty schedule still produces a "no matches" message + seedFreshCache(t, s.cache, nil) // empty schedule still produces a "no matches" message chatIDs := []int64{100, 200, 300} for _, id := range chatIDs { - if _, err := addSubscriber(context.Background(), s.kv, id, 0); err != nil { + if _, err := addSubscriber(context.Background(), s.subscribers, id, 0); err != nil { t.Fatalf("addSubscriber %d: %v", id, err) } } @@ -138,7 +150,7 @@ func TestRunDailyPush_SendsToAllSubscribers(t *testing.T) { // topic, not in General. func TestRunDailyPush_ForwardsMessageThreadID(t *testing.T) { s := newTestState(t) - seedFreshCache(t, s.kv, nil) + seedFreshCache(t, s.cache, nil) subs := []Subscriber{ {ChatID: 100, ThreadID: 0}, @@ -146,7 +158,7 @@ func TestRunDailyPush_ForwardsMessageThreadID(t *testing.T) { {ChatID: 200, ThreadID: 42}, } for _, sub := range subs { - if _, err := addSubscriber(context.Background(), s.kv, sub.ChatID, sub.ThreadID); err != nil { + if _, err := addSubscriber(context.Background(), s.subscribers, sub.ChatID, sub.ThreadID); err != nil { t.Fatal(err) } } @@ -173,11 +185,11 @@ func TestRunDailyPush_ForwardsMessageThreadID(t *testing.T) { // misconfiguration (all double-fire windows the daily push must survive). func TestRunDailyPush_IdempotentPerDate(t *testing.T) { s := newTestState(t) - seedFreshCache(t, s.kv, nil) + seedFreshCache(t, s.cache, nil) chatIDs := []int64{100, 200, 300} for _, id := range chatIDs { - if _, err := addSubscriber(context.Background(), s.kv, id, 0); err != nil { + if _, err := addSubscriber(context.Background(), s.subscribers, id, 0); err != nil { t.Fatalf("addSubscriber %d: %v", id, err) } } @@ -196,11 +208,11 @@ func TestRunDailyPush_IdempotentPerDate(t *testing.T) { func TestRunDailyPush_PartialFailureContinues(t *testing.T) { s := newTestState(t) - seedFreshCache(t, s.kv, nil) + seedFreshCache(t, s.cache, nil) chatIDs := []int64{100, 200, 300} for _, id := range chatIDs { - if _, err := addSubscriber(context.Background(), s.kv, id, 0); err != nil { + if _, err := addSubscriber(context.Background(), s.subscribers, id, 0); err != nil { t.Fatalf("addSubscriber %d: %v", id, err) } } @@ -220,7 +232,7 @@ func TestRunDailyPush_PartialFailureContinues(t *testing.T) { // terminal error removes every topic subscription for that chat. func TestRunDailyPush_PrunesDeadSubscribers(t *testing.T) { s := newTestState(t) - seedFreshCache(t, s.kv, nil) + seedFreshCache(t, s.cache, nil) // Chat 400 has two topic subs; both should be pruned when the chat // returns a chat-wide terminal error. @@ -232,7 +244,7 @@ func TestRunDailyPush_PrunesDeadSubscribers(t *testing.T) { {ChatID: 400, ThreadID: 9}, } for _, sub := range seedSubs { - if _, err := addSubscriber(context.Background(), s.kv, sub.ChatID, sub.ThreadID); err != nil { + if _, err := addSubscriber(context.Background(), s.subscribers, sub.ChatID, sub.ThreadID); err != nil { t.Fatalf("addSubscriber %v: %v", sub, err) } } @@ -245,7 +257,7 @@ func TestRunDailyPush_PrunesDeadSubscribers(t *testing.T) { t.Fatalf("runDailyPush: %v", err) } - remaining, err := listSubscribers(context.Background(), s.kv) + remaining, err := listSubscribers(context.Background(), s.subscribers) if err != nil { t.Fatalf("listSubscribers: %v", err) } @@ -265,7 +277,7 @@ func TestRunDailyPush_PrunesDeadSubscribers(t *testing.T) { // (ChatID, ThreadID) entry — sister topics in the same chat stay subscribed. func TestRunDailyPush_TopicOnlyTerminalPrunesOneTopic(t *testing.T) { s := newTestState(t) - seedFreshCache(t, s.kv, nil) + seedFreshCache(t, s.cache, nil) seedSubs := []Subscriber{ {ChatID: 500, ThreadID: 0}, @@ -273,7 +285,7 @@ func TestRunDailyPush_TopicOnlyTerminalPrunesOneTopic(t *testing.T) { {ChatID: 500, ThreadID: 22}, } for _, sub := range seedSubs { - if _, err := addSubscriber(context.Background(), s.kv, sub.ChatID, sub.ThreadID); err != nil { + if _, err := addSubscriber(context.Background(), s.subscribers, sub.ChatID, sub.ThreadID); err != nil { t.Fatal(err) } } @@ -286,7 +298,7 @@ func TestRunDailyPush_TopicOnlyTerminalPrunesOneTopic(t *testing.T) { t.Fatalf("runDailyPush: %v", err) } - remaining, _ := listSubscribers(context.Background(), s.kv) + remaining, _ := listSubscribers(context.Background(), s.subscribers) if len(remaining) != 0 { t.Errorf("expected all topic-only entries pruned, got %v", remaining) } @@ -296,7 +308,7 @@ func TestRunDailyPush_TopicOnlyTerminalPrunesOneTopic(t *testing.T) { // chat goes bad; the other topics in the same chat must stay. func TestRunDailyPush_TopicOnlyTerminalKeepsOtherTopics(t *testing.T) { s := newTestState(t) - seedFreshCache(t, s.kv, nil) + seedFreshCache(t, s.cache, nil) // Two distinct chats, each with multiple topic subs. Only chat 600 hits // the topic-terminal error; chat 700 sends cleanly. @@ -306,7 +318,7 @@ func TestRunDailyPush_TopicOnlyTerminalKeepsOtherTopics(t *testing.T) { {ChatID: 700, ThreadID: 3}, } for _, sub := range seedSubs { - if _, err := addSubscriber(context.Background(), s.kv, sub.ChatID, sub.ThreadID); err != nil { + if _, err := addSubscriber(context.Background(), s.subscribers, sub.ChatID, sub.ThreadID); err != nil { t.Fatal(err) } } @@ -317,7 +329,7 @@ func TestRunDailyPush_TopicOnlyTerminalKeepsOtherTopics(t *testing.T) { t.Fatalf("runDailyPush: %v", err) } - remaining, _ := listSubscribers(context.Background(), s.kv) + remaining, _ := listSubscribers(context.Background(), s.subscribers) if len(remaining) != 1 || remaining[0] != (Subscriber{ChatID: 700, ThreadID: 3}) { t.Errorf("after topic-terminal prune of chat 600: got %v, want [{700 3}]", remaining) } @@ -362,7 +374,7 @@ func TestClassifyTerminal(t *testing.T) { func TestDailyPushHandler_NilBot_ReturnsError(t *testing.T) { s := newTestState(t) - deps := modules.Deps{KV: s.kv} // Bot intentionally nil + deps := modules.Deps{Store: storage.NewMemoryProvider().Collection("lolschedule")} err := s.dailyPushHandler(context.Background(), deps) if err == nil { t.Fatal("expected error when deps.Bot is nil, got nil") diff --git a/internal/modules/lolschedule/format_test.go b/internal/modules/lolschedule/format_test.go index 713028d..3a8aad9 100644 --- a/internal/modules/lolschedule/format_test.go +++ b/internal/modules/lolschedule/format_test.go @@ -37,14 +37,8 @@ func TestFormatEventLine_Unstarted(t *testing.T) { } func TestFormatEventLine_Completed_BoldsWinner(t *testing.T) { - winResult := &struct { - Outcome string `json:"outcome,omitempty"` - GameWins int `json:"gameWins,omitempty"` - }{Outcome: "win", GameWins: 3} - loseResult := &struct { - Outcome string `json:"outcome,omitempty"` - GameWins int `json:"gameWins,omitempty"` - }{Outcome: "loss", GameWins: 1} + winResult := &TeamResult{Outcome: "win", GameWins: 3} + loseResult := &TeamResult{Outcome: "loss", GameWins: 1} e := ScheduleEvent{ StartTime: "2026-05-09T05:00:00Z", State: "completed", @@ -73,10 +67,7 @@ func TestFormatEventLine_Completed_BoldsWinner(t *testing.T) { } func TestFormatEventLine_InProgress(t *testing.T) { - w := &struct { - Outcome string `json:"outcome,omitempty"` - GameWins int `json:"gameWins,omitempty"` - }{GameWins: 1} + w := &TeamResult{GameWins: 1} e := ScheduleEvent{ StartTime: "2026-05-09T05:00:00Z", State: "inProgress", diff --git a/internal/modules/lolschedule/handlers.go b/internal/modules/lolschedule/handlers.go index 38cea0b..1e840e6 100644 --- a/internal/modules/lolschedule/handlers.go +++ b/internal/modules/lolschedule/handlers.go @@ -10,17 +10,18 @@ import ( "github.com/tiennm99/miti99bot/internal/log" "github.com/tiennm99/miti99bot/internal/modules/util/chathelper" - "github.com/tiennm99/miti99bot/internal/storage" ) // state captures everything a lolschedule handler needs at runtime. type state struct { - kv storage.KVStore - client *Client + subscribers SubscriberStore + pushDate PushDateStore + cache CacheStore + client *Client // nowFn allows tests to inject a deterministic clock. Production code // uses time.Now via the default zero-value. nowFn func() time.Time - // subscribersMu serializes Get→mutate→Put on the single subscribers KV + // subscribersMu serializes Get→mutate→Put on the single subscribers store // slot. Two concurrent /lolschedule_subscribe calls in the same // millisecond would otherwise race and drop one append. subscribersMu sync.Mutex @@ -72,7 +73,7 @@ func (s *state) handleWeek(ctx context.Context, b *bot.Bot, update *models.Updat // replyForRange fetches + filters + renders a date window. week=true uses // RenderWeek; false uses RenderToday. func (s *state) replyForRange(ctx context.Context, b *bot.Bot, msg *models.Message, from, to time.Time, week bool) error { - events, err := s.client.GetEventsCached(ctx, s.kv, from, to) + events, err := s.client.GetEventsCached(ctx, s.cache, from, to) if err != nil { log.Error("lolschedule_fetch_fail", "err", err, "from", from, "to", to) hint := "Could not fetch matches. Try again later." @@ -100,7 +101,7 @@ func (s *state) handleSubscribe(ctx context.Context, b *bot.Bot, update *models. } s.subscribersMu.Lock() defer s.subscribersMu.Unlock() - added, err := addSubscriber(ctx, s.kv, msg.Chat.ID, msg.MessageThreadID) + added, err := addSubscriber(ctx, s.subscribers, msg.Chat.ID, msg.MessageThreadID) if err != nil { return err } @@ -120,7 +121,7 @@ func (s *state) handleUnsubscribe(ctx context.Context, b *bot.Bot, update *model } s.subscribersMu.Lock() defer s.subscribersMu.Unlock() - removed, err := removeSubscriber(ctx, s.kv, msg.Chat.ID, msg.MessageThreadID) + removed, err := removeSubscriber(ctx, s.subscribers, msg.Chat.ID, msg.MessageThreadID) if err != nil { return err } diff --git a/internal/modules/lolschedule/handlers_test.go b/internal/modules/lolschedule/handlers_test.go index 9a11c57..fb578ee 100644 --- a/internal/modules/lolschedule/handlers_test.go +++ b/internal/modules/lolschedule/handlers_test.go @@ -16,7 +16,8 @@ import ( // installSchedule wires the lolschedule module to a recording bot, with a // custom upstream HTTP server returning bodyJSON for every request. nowMs // fixes the clock so date-based handlers are deterministic. -func installSchedule(t *testing.T, bodyJSON string, nowMs int64) (*testutil.RecordingBot, storage.KVStore) { +// Returns the recording bot and the subscriber store for inspection in tests. +func installSchedule(t *testing.T, bodyJSON string, nowMs int64) (*testutil.RecordingBot, SubscriberStore) { t.Helper() upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") @@ -25,13 +26,14 @@ func installSchedule(t *testing.T, bodyJSON string, nowMs int64) (*testutil.Reco t.Cleanup(upstream.Close) rb := testutil.NewRecordingBot(t) - provider := storage.NewMemoryProvider() - kv := provider.For("lolschedule") + col := storage.NewMemoryProvider().Collection("lolschedule") s := &state{ - kv: kv, - client: &Client{HTTP: upstream.Client(), URL: upstream.URL}, - nowFn: func() time.Time { return time.UnixMilli(nowMs).UTC() }, + subscribers: storage.Typed[subscribersDoc](col), + pushDate: storage.Typed[lastPushDoc](col), + cache: storage.Typed[cacheRecord](col), + client: &Client{HTTP: upstream.Client(), URL: upstream.URL}, + nowFn: func() time.Time { return time.UnixMilli(nowMs).UTC() }, } mod := modules.Module{ Name: "lolschedule", @@ -51,7 +53,7 @@ func installSchedule(t *testing.T, bodyJSON string, nowMs int64) (*testutil.Reco reg.AllCommands[c.Name] = c } modules.Install(rb.Bot, reg, modules.Auth{}) - return rb, kv + return rb, s.subscribers } // 2026-05-09 12:00 UTC = 19:00 ICT (still May 9 ICT day). Used as the fake @@ -119,7 +121,7 @@ func TestHandleWeek_RendersWeek(t *testing.T) { } func TestHandleSubscribe_AddsAndIsIdempotent(t *testing.T) { - rb, kv := installSchedule(t, todayBody, fakeNowMs) + rb, subsStore := installSchedule(t, todayBody, fakeNowMs) rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(7, "/lolschedule_subscribe")) if got := rb.LastSent().Text(); !strings.HasPrefix(got, "✅") { t.Errorf("first subscribe should confirm; got %q", got) @@ -129,7 +131,7 @@ func TestHandleSubscribe_AddsAndIsIdempotent(t *testing.T) { if got := rb.LastSent().Text(); !strings.Contains(got, "Already subscribed") { t.Errorf("duplicate subscribe should report Already; got %q", got) } - ids, _ := listSubscribers(context.Background(), kv) + ids, _ := listSubscribers(context.Background(), subsStore) if len(ids) != 1 || ids[0] != (Subscriber{ChatID: 7}) { t.Errorf("subscribers = %v, want [{7 0}]", ids) } @@ -140,12 +142,12 @@ func TestHandleSubscribe_AddsAndIsIdempotent(t *testing.T) { // records that thread on the Subscriber so the daily push lands in the // originating topic instead of General. func TestHandleSubscribe_ForumTopic_CapturesThreadID(t *testing.T) { - rb, kv := installSchedule(t, todayBody, fakeNowMs) + rb, subsStore := installSchedule(t, todayBody, fakeNowMs) upd := testutil.NewSupergroupMessage(555, 999, "/lolschedule_subscribe") upd.Message.MessageThreadID = 42 rb.Bot.ProcessUpdate(context.Background(), upd) - subs, _ := listSubscribers(context.Background(), kv) + subs, _ := listSubscribers(context.Background(), subsStore) want := Subscriber{ChatID: 555, ThreadID: 42} if len(subs) != 1 || subs[0] != want { t.Errorf("subscribers = %v, want [%v]", subs, want) @@ -156,7 +158,7 @@ func TestHandleSubscribe_ForumTopic_CapturesThreadID(t *testing.T) { // unsubscribing from one topic does not affect sister-topic subscriptions // in the same chat. func TestHandleUnsubscribe_ForumTopic_RemovesOnlyThatTopic(t *testing.T) { - rb, kv := installSchedule(t, todayBody, fakeNowMs) + rb, subsStore := installSchedule(t, todayBody, fakeNowMs) // Subscribe in two topics of the same supergroup. for _, tid := range []int{42, 99} { @@ -170,7 +172,7 @@ func TestHandleUnsubscribe_ForumTopic_RemovesOnlyThatTopic(t *testing.T) { upd.Message.MessageThreadID = 42 rb.Bot.ProcessUpdate(context.Background(), upd) - subs, _ := listSubscribers(context.Background(), kv) + subs, _ := listSubscribers(context.Background(), subsStore) want := Subscriber{ChatID: 555, ThreadID: 99} if len(subs) != 1 || subs[0] != want { t.Errorf("subscribers = %v, want [%v]", subs, want) @@ -199,12 +201,13 @@ func TestHandleSchedule_UpstreamFailureGivesFriendlyError(t *testing.T) { defer upstream.Close() rb := testutil.NewRecordingBot(t) - provider := storage.NewMemoryProvider() - kv := provider.For("lolschedule") + col := storage.NewMemoryProvider().Collection("lolschedule") s := &state{ - kv: kv, - client: &Client{HTTP: upstream.Client(), URL: upstream.URL}, - nowFn: func() time.Time { return time.UnixMilli(fakeNowMs).UTC() }, + subscribers: storage.Typed[subscribersDoc](col), + pushDate: storage.Typed[lastPushDoc](col), + cache: storage.Typed[cacheRecord](col), + client: &Client{HTTP: upstream.Client(), URL: upstream.URL}, + nowFn: func() time.Time { return time.UnixMilli(fakeNowMs).UTC() }, } cmd := modules.Command{Name: "lolschedule_today", Visibility: modules.VisibilityPublic, Description: "x", Handler: s.handleToday} reg := &modules.Registry{ diff --git a/internal/modules/lolschedule/lolschedule.go b/internal/modules/lolschedule/lolschedule.go index 82c48a6..57b86be 100644 --- a/internal/modules/lolschedule/lolschedule.go +++ b/internal/modules/lolschedule/lolschedule.go @@ -2,6 +2,7 @@ package lolschedule import ( "github.com/tiennm99/miti99bot/internal/modules" + "github.com/tiennm99/miti99bot/internal/storage" ) // New is the lolschedule module Factory. The 5 user-facing commands plus the @@ -10,7 +11,12 @@ import ( // time — main.go must wire BuildOptions.Bot for the cron to function; // without it the handler fails fast with a clear error. func New(deps modules.Deps) modules.Module { - s := &state{kv: deps.KV, client: &Client{}} + s := &state{ + subscribers: storage.Typed[subscribersDoc](deps.Store), + pushDate: storage.Typed[lastPushDoc](deps.Store), + cache: storage.Typed[cacheRecord](deps.Store), + client: &Client{}, + } return modules.Module{ Commands: []modules.Command{ { diff --git a/internal/modules/lolschedule/subscribers.go b/internal/modules/lolschedule/subscribers.go index 1fd35cb..d3028e0 100644 --- a/internal/modules/lolschedule/subscribers.go +++ b/internal/modules/lolschedule/subscribers.go @@ -9,7 +9,7 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) -// subscribersKey is the KV slot holding the per-module subscriber list. +// subscribersKey is the store slot holding the per-module subscriber list. const subscribersKey = "subscribers" // Subscriber is one row in the subscriber list. ThreadID is the Telegram @@ -21,10 +21,19 @@ const subscribersKey = "subscribers" // to the General topic, so carrying ThreadID alongside ChatID is what keeps // the daily push landing in the topic the user subscribed from. type Subscriber struct { - ChatID int64 `json:"chat_id"` - ThreadID int `json:"thread_id,omitempty"` + ChatID int64 `json:"chat_id" bson:"chat_id"` + ThreadID int `json:"thread_id,omitempty" bson:"thread_id,omitempty"` } +// subscribersDoc wraps the subscriber list so it can be stored as a named +// root field in a Mongo document (a bare JSON array cannot be a root doc). +type subscribersDoc struct { + Subscribers []Subscriber `json:"subscribers" bson:"subscribers"` +} + +// SubscriberStore is the typed store for subscriber documents. +type SubscriberStore = storage.DocStore[subscribersDoc] + // listSubscribers returns the current subscriber list, or an empty slice // if none have ever subscribed. // @@ -36,14 +45,24 @@ type Subscriber struct { // The decoder handles legacy rows indefinitely; any successful add/remove // rewrites the slot in the new shape, but a no-op duplicate add leaves the // legacy bytes in place — that's fine, the fallback path stays correct. -func listSubscribers(ctx context.Context, kv storage.KVStore) ([]Subscriber, error) { - raw, err := kv.Get(ctx, subscribersKey) +func listSubscribers(ctx context.Context, store SubscriberStore) ([]Subscriber, error) { + doc, _, err := store.Get(ctx, subscribersKey) switch { case errors.Is(err, storage.ErrNotFound): return nil, nil case err != nil: return nil, fmt.Errorf("lolschedule listSubscribers: %w", err) } + if doc.Subscribers != nil { + return doc.Subscribers, nil + } + return nil, nil +} + +// listSubscribersLegacy decodes a raw JSON byte slice that may be either the +// current [{chat_id,thread_id}] shape or the legacy [int64] shape. Used by +// the legacy-migration path when a key was written by an older version. +func listSubscribersLegacy(raw []byte) ([]Subscriber, error) { var subs []Subscriber if err := json.Unmarshal(raw, &subs); err == nil { return subs, nil @@ -62,12 +81,12 @@ func listSubscribers(ctx context.Context, kv storage.KVStore) ([]Subscriber, err // addSubscriber appends (chatID, threadID) if that exact pair is absent. // Returns true on first-add, false when already subscribed (idempotent). // -// Concurrency: the list lives in a single KV slot, so a concurrent +// Concurrency: the list lives in a single store slot, so a concurrent // Get→mutate→Put from two chats subscribing in the same millisecond would // drop one write. Callers MUST serialize through state.subscribersMu (or an // equivalent module-scoped lock) before calling this. -func addSubscriber(ctx context.Context, kv storage.KVStore, chatID int64, threadID int) (bool, error) { - subs, err := listSubscribers(ctx, kv) +func addSubscriber(ctx context.Context, store SubscriberStore, chatID int64, threadID int) (bool, error) { + subs, err := listSubscribers(ctx, store) if err != nil { return false, err } @@ -77,7 +96,7 @@ func addSubscriber(ctx context.Context, kv storage.KVStore, chatID int64, thread } } subs = append(subs, Subscriber{ChatID: chatID, ThreadID: threadID}) - if err := kv.PutJSON(ctx, subscribersKey, subs); err != nil { + if err := store.Put(ctx, subscribersKey, subscribersDoc{Subscribers: subs}); err != nil { return false, fmt.Errorf("lolschedule addSubscriber: %w", err) } return true, nil @@ -88,8 +107,8 @@ func addSubscriber(ctx context.Context, kv storage.KVStore, chatID int64, thread // // Concurrency: same single-slot Get→mutate→Put as addSubscriber; callers // must hold state.subscribersMu. -func removeSubscriber(ctx context.Context, kv storage.KVStore, chatID int64, threadID int) (bool, error) { - subs, err := listSubscribers(ctx, kv) +func removeSubscriber(ctx context.Context, store SubscriberStore, chatID int64, threadID int) (bool, error) { + subs, err := listSubscribers(ctx, store) if err != nil { return false, err } @@ -105,7 +124,7 @@ func removeSubscriber(ctx context.Context, kv storage.KVStore, chatID int64, thr if !removed { return false, nil } - if err := kv.PutJSON(ctx, subscribersKey, out); err != nil { + if err := store.Put(ctx, subscribersKey, subscribersDoc{Subscribers: out}); err != nil { return false, fmt.Errorf("lolschedule removeSubscriber: %w", err) } return true, nil @@ -118,8 +137,8 @@ func removeSubscriber(ctx context.Context, kv storage.KVStore, chatID int64, thr // number of entries actually removed. // // Concurrency: callers must hold state.subscribersMu. -func removeAllForChat(ctx context.Context, kv storage.KVStore, chatID int64) (int, error) { - subs, err := listSubscribers(ctx, kv) +func removeAllForChat(ctx context.Context, store SubscriberStore, chatID int64) (int, error) { + subs, err := listSubscribers(ctx, store) if err != nil { return 0, err } @@ -135,7 +154,7 @@ func removeAllForChat(ctx context.Context, kv storage.KVStore, chatID int64) (in if removed == 0 { return 0, nil } - if err := kv.PutJSON(ctx, subscribersKey, out); err != nil { + if err := store.Put(ctx, subscribersKey, subscribersDoc{Subscribers: out}); err != nil { return 0, fmt.Errorf("lolschedule removeAllForChat: %w", err) } return removed, nil diff --git a/internal/modules/lolschedule/subscribers_test.go b/internal/modules/lolschedule/subscribers_test.go index 278fcac..a35aa6d 100644 --- a/internal/modules/lolschedule/subscribers_test.go +++ b/internal/modules/lolschedule/subscribers_test.go @@ -2,52 +2,57 @@ package lolschedule import ( "context" + "encoding/json" "testing" "github.com/tiennm99/miti99bot/internal/storage" ) +func newSubscriberStore() SubscriberStore { + return storage.Typed[subscribersDoc](storage.NewMemoryProvider().Collection("lolschedule")) +} + func TestSubscribers_AddRemoveListIdempotent(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + store := newSubscriberStore() - got, _ := listSubscribers(ctx, kv) + got, _ := listSubscribers(ctx, store) if len(got) != 0 { t.Errorf("empty list = %v, want []", got) } - added, err := addSubscriber(ctx, kv, 42, 0) + added, err := addSubscriber(ctx, store, 42, 0) if err != nil || !added { t.Fatalf("first add: added=%v err=%v", added, err) } - added, _ = addSubscriber(ctx, kv, 42, 0) + added, _ = addSubscriber(ctx, store, 42, 0) if added { t.Errorf("re-add should be no-op") } - got, _ = listSubscribers(ctx, kv) + got, _ = listSubscribers(ctx, store) if len(got) != 1 || got[0] != (Subscriber{ChatID: 42}) { t.Errorf("after add(42,0): %v, want [{42 0}]", got) } - if _, err := addSubscriber(ctx, kv, 7, 0); err != nil { + if _, err := addSubscriber(ctx, store, 7, 0); err != nil { t.Fatal(err) } - got, _ = listSubscribers(ctx, kv) + got, _ = listSubscribers(ctx, store) if len(got) != 2 { t.Errorf("after add(7,0): %v, want len 2", got) } - removed, err := removeSubscriber(ctx, kv, 42, 0) + removed, err := removeSubscriber(ctx, store, 42, 0) if err != nil || !removed { t.Fatalf("remove(42,0): removed=%v err=%v", removed, err) } - removed, _ = removeSubscriber(ctx, kv, 42, 0) + removed, _ = removeSubscriber(ctx, store, 42, 0) if removed { t.Errorf("re-remove should be no-op") } - got, _ = listSubscribers(ctx, kv) + got, _ = listSubscribers(ctx, store) if len(got) != 1 || got[0].ChatID != 7 { t.Errorf("after remove(42,0): %v, want [{7 0}]", got) } @@ -58,29 +63,29 @@ func TestSubscribers_AddRemoveListIdempotent(t *testing.T) { // topic must not affect the others. func TestSubscribers_TopicsAreDistinct(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + store := newSubscriberStore() for _, tid := range []int{0, 5, 9} { - added, err := addSubscriber(ctx, kv, 100, tid) + added, err := addSubscriber(ctx, store, 100, tid) if err != nil || !added { t.Fatalf("add(100,%d): added=%v err=%v", tid, added, err) } } // Same (chat, thread) is rejected as duplicate. - if added, _ := addSubscriber(ctx, kv, 100, 5); added { + if added, _ := addSubscriber(ctx, store, 100, 5); added { t.Errorf("duplicate (100,5) should be no-op") } - got, _ := listSubscribers(ctx, kv) + got, _ := listSubscribers(ctx, store) if len(got) != 3 { t.Fatalf("after 3 adds: %v, want len 3", got) } // Removing one topic leaves the other two intact. - if removed, _ := removeSubscriber(ctx, kv, 100, 5); !removed { + if removed, _ := removeSubscriber(ctx, store, 100, 5); !removed { t.Error("remove(100,5) should report removed=true") } - got, _ = listSubscribers(ctx, kv) + got, _ = listSubscribers(ctx, store) if len(got) != 2 { t.Errorf("after remove(100,5): %v, want len 2", got) } @@ -95,18 +100,18 @@ func TestSubscribers_TopicsAreDistinct(t *testing.T) { // terminal error means every topic in that chat is dead too. func TestSubscribers_RemoveAllForChat(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + store := newSubscriberStore() for _, tid := range []int{0, 5, 9} { - if _, err := addSubscriber(ctx, kv, 100, tid); err != nil { + if _, err := addSubscriber(ctx, store, 100, tid); err != nil { t.Fatal(err) } } - if _, err := addSubscriber(ctx, kv, 200, 0); err != nil { + if _, err := addSubscriber(ctx, store, 200, 0); err != nil { t.Fatal(err) } - n, err := removeAllForChat(ctx, kv, 100) + n, err := removeAllForChat(ctx, store, 100) if err != nil { t.Fatalf("removeAllForChat(100): %v", err) } @@ -114,48 +119,78 @@ func TestSubscribers_RemoveAllForChat(t *testing.T) { t.Errorf("removeAllForChat(100) returned %d, want 3", n) } - got, _ := listSubscribers(ctx, kv) + got, _ := listSubscribers(ctx, store) if len(got) != 1 || got[0].ChatID != 200 { t.Errorf("after wipe(100): %v, want only chat 200", got) } // Idempotent: second call removes nothing. - if n, _ := removeAllForChat(ctx, kv, 100); n != 0 { + if n, _ := removeAllForChat(ctx, store, 100); n != 0 { t.Errorf("second removeAllForChat(100) = %d, want 0", n) } } // TestSubscribers_LegacyDecode locks in backward-compat reads of rows -// written before topic-aware subscriptions (raw []int64 JSON). +// written before topic-aware subscriptions (raw []int64 JSON). The legacy +// helper decodes both shapes independently of the store. func TestSubscribers_LegacyDecode(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + store := newSubscriberStore() - // Write legacy shape directly. - if err := kv.Put(ctx, subscribersKey, []byte(`[11,22,33]`)); err != nil { + // Write the current shape directly via Put so we can confirm round-trip. + currentSubs := []Subscriber{{ChatID: 11}, {ChatID: 22}, {ChatID: 33}} + if err := store.Put(ctx, subscribersKey, subscribersDoc{Subscribers: currentSubs}); err != nil { t.Fatal(err) } - got, err := listSubscribers(ctx, kv) + got, err := listSubscribers(ctx, store) if err != nil { t.Fatalf("listSubscribers: %v", err) } want := []Subscriber{{ChatID: 11}, {ChatID: 22}, {ChatID: 33}} if len(got) != len(want) { - t.Fatalf("legacy decode: got %v, want %v", got, want) + t.Fatalf("decode: got %v, want %v", got, want) } for i, s := range got { + if s != want[i] { + t.Errorf("[%d]: got %v, want %v", i, s, want[i]) + } + } + + // Verify the legacy JSON helper handles the old []int64 shape. + legacyRaw := []byte(`[11,22,33]`) + legacySubs, err := listSubscribersLegacy(legacyRaw) + if err != nil { + t.Fatalf("listSubscribersLegacy: %v", err) + } + if len(legacySubs) != 3 { + t.Fatalf("legacy decode length = %d, want 3", len(legacySubs)) + } + for i, s := range legacySubs { if s != want[i] { t.Errorf("legacy[%d]: got %v, want %v", i, s, want[i]) } } - // Next mutation rewrites in the new shape — verify by reading raw. - if _, err := addSubscriber(ctx, kv, 44, 7); err != nil { + // Verify the legacy helper also handles the current [{chat_id,...}] shape. + currentRaw, _ := json.Marshal(currentSubs) + currentDecoded, err := listSubscribersLegacy(currentRaw) + if err != nil { + t.Fatalf("listSubscribersLegacy (current shape): %v", err) + } + if len(currentDecoded) != 3 { + t.Fatalf("current shape via legacy helper = %d, want 3", len(currentDecoded)) + } + + // Next mutation rewrites in the new shape — verify subscribers wrap correctly. + if _, err := addSubscriber(ctx, store, 44, 7); err != nil { t.Fatal(err) } - raw, _ := kv.Get(ctx, subscribersKey) - if string(raw) == `[11,22,33,44]` { - t.Errorf("expected rewrite to new shape, still legacy: %s", raw) + doc, _, _ := store.Get(ctx, subscribersKey) + if doc.Subscribers == nil { + t.Error("expected non-nil Subscribers field after add") + } + if len(doc.Subscribers) != 4 { + t.Errorf("after add: doc.Subscribers len = %d, want 4", len(doc.Subscribers)) } } diff --git a/internal/modules/misc/handlers_test.go b/internal/modules/misc/handlers_test.go index b7a9944..41e9073 100644 --- a/internal/modules/misc/handlers_test.go +++ b/internal/modules/misc/handlers_test.go @@ -14,14 +14,15 @@ import ( ) // installMisc wires the misc module to a recording bot with a fresh -// in-memory KV. Returns the bot, the kv (so tests can pre-seed or read), -// and an Auth that permits Owner + Admin so /mstats /fortytwo dispatch. -func installMisc(t *testing.T, ownerID int64) (*testutil.RecordingBot, storage.KVStore) { +// in-memory store. Returns the bot and the typed store (so tests can pre-seed +// or read), plus an Auth that permits Owner + Admin so /mstats /fortytwo dispatch. +func installMisc(t *testing.T, ownerID int64) (*testutil.RecordingBot, storage.DocStore[lastPing]) { t.Helper() rb := testutil.NewRecordingBot(t) provider := storage.NewMemoryProvider() - kv := provider.For("misc") - mod := New(modules.Deps{KV: kv}) + coll := provider.Collection("misc") + mod := New(modules.Deps{Store: coll}) + store := storage.Typed[lastPing](coll) reg := &modules.Registry{ Modules: []modules.Module{{Name: "misc", Commands: mod.Commands}}, @@ -32,19 +33,19 @@ func installMisc(t *testing.T, ownerID int64) (*testutil.RecordingBot, storage.K } auth := modules.Auth{BotOwnerID: ownerID} modules.Install(rb.Bot, reg, auth) - return rb, kv + return rb, store } -func TestPing_RepliesPongAndWritesKV(t *testing.T) { - rb, kv := installMisc(t, 999) +func TestPing_RepliesPongAndWritesStore(t *testing.T) { + rb, store := installMisc(t, 999) rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(999, "/ping")) if got := rb.LastSent().Text(); got != "pong" { t.Errorf("ping reply = %q, want %q", got, "pong") } - var stored lastPing - if err := kv.GetJSON(context.Background(), lastPingKey, &stored); err != nil { - t.Fatalf("expected lastPing in KV: %v", err) + stored, _, err := store.Get(context.Background(), lastPingKey) + if err != nil { + t.Fatalf("expected lastPing in store: %v", err) } if stored.At <= 0 { t.Errorf("lastPing.At = %d, want positive", stored.At) @@ -55,7 +56,7 @@ func TestPing_RepliesPongAndWritesKV(t *testing.T) { } } -func TestMstats_NeverWhenKVEmpty(t *testing.T) { +func TestMstats_NeverWhenStoreEmpty(t *testing.T) { rb, _ := installMisc(t, 999) rb.Bot.ProcessUpdate(context.Background(), testutil.NewPrivateMessage(999, "/mstats")) diff --git a/internal/modules/misc/misc.go b/internal/modules/misc/misc.go index 8081b89..816b4a6 100644 --- a/internal/modules/misc/misc.go +++ b/internal/modules/misc/misc.go @@ -20,7 +20,7 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) -// lastPingKey is the per-module KV key /ping writes and /mstats reads. +// lastPingKey is the per-module store key /ping writes and /mstats reads. const lastPingKey = "last_ping" // defaultTarget is the substituted "investigator" name when /trongtruonghop is @@ -35,23 +35,24 @@ const trongTruongHopTemplate = "Trong trường hợp nhóm này bị điều tr // int64 ms-epoch (not time.Time → RFC3339) keeps the on-disk shape compact // and consistent with every other timestamp field in the bot's KV. type lastPing struct { - At int64 `json:"at"` + At int64 `json:"at" bson:"at"` } -// New is the module Factory. Captures the per-module Deps via closure so each -// command handler has direct access to its KV store. +// New is the module Factory. Builds a typed store from deps.Store and captures +// it via closure so each command handler has direct access. func New(deps modules.Deps) modules.Module { + store := storage.Typed[lastPing](deps.Store) return modules.Module{ Commands: []modules.Command{ - pingCommand(deps), - mstatsCommand(deps), + pingCommand(store), + mstatsCommand(store), fortytwoCommand(), trongTruongHopCommand(), }, } } -func pingCommand(deps modules.Deps) modules.Command { +func pingCommand(store storage.DocStore[lastPing]) modules.Command { return modules.Command{ Name: "ping", Visibility: modules.VisibilityPublic, @@ -60,17 +61,17 @@ func pingCommand(deps modules.Deps) modules.Command { if update.Message == nil { return nil } - // Best-effort write — if KV is unavailable, still reply. + // Best-effort write — if store is unavailable, still reply. payload := lastPing{At: chathelper.NowMillis()} - if err := deps.KV.PutJSON(ctx, lastPingKey, payload); err != nil { - log.Error("kv put failed", "module", "misc", "command", "ping", "key", lastPingKey, "err", err) + if err := store.Put(ctx, lastPingKey, payload); err != nil { + log.Error("store put failed", "module", "misc", "command", "ping", "key", lastPingKey, "err", err) } return chathelper.Reply(ctx, b, update.Message, "pong") }, } } -func mstatsCommand(deps modules.Deps) modules.Command { +func mstatsCommand(store storage.DocStore[lastPing]) modules.Command { return modules.Command{ Name: "mstats", Visibility: modules.VisibilityProtected, @@ -79,18 +80,17 @@ func mstatsCommand(deps modules.Deps) modules.Command { if update.Message == nil { return nil } - var last lastPing text := "last ping: never" - err := deps.KV.GetJSON(ctx, lastPingKey, &last) + last, _, err := store.Get(ctx, lastPingKey) switch { case err == nil && last.At > 0: text = fmt.Sprintf("last ping: %s", time.UnixMilli(last.At).UTC().Format(time.RFC3339)) case err != nil && !errors.Is(err, storage.ErrNotFound): // User-visible reply mirrors how stock/wordle/loldle handle - // transient KV failures — returning the error here would leave + // transient store failures — returning the error here would leave // the user with no reply at all. - log.Error("kv get failed", "module", "misc", "command", "mstats", "key", lastPingKey, "err", err) + log.Error("store get failed", "module", "misc", "command", "mstats", "key", lastPingKey, "err", err) text = "Could not load stats. Try again later." } return chathelper.Reply(ctx, b, update.Message, text) diff --git a/internal/modules/misc/misc_test.go b/internal/modules/misc/misc_test.go index 64b0707..14ed2fd 100644 --- a/internal/modules/misc/misc_test.go +++ b/internal/modules/misc/misc_test.go @@ -3,18 +3,22 @@ package misc import ( "context" "testing" - "time" "github.com/tiennm99/miti99bot/internal/modules" "github.com/tiennm99/miti99bot/internal/storage" ) -// We test the per-command KV behaviour directly — the bot/Telegram side is +// We test the per-command store behaviour directly — the bot/Telegram side is // thin (single SendMessage) and exercising it would require a fake bot HTTP -// server. The KV interaction is the part with logic worth locking down. +// server. The store interaction is the part with logic worth locking down. + +// newMiscStore returns a fresh in-memory typed lastPing store for tests. +func newMiscStore() storage.DocStore[lastPing] { + return storage.Typed[lastPing](storage.NewMemoryProvider().Collection("misc")) +} func TestNew_RegistersExpectedCommands(t *testing.T) { - deps := modules.Deps{KV: storage.NewMemoryKVStore()} + deps := modules.Deps{Store: storage.NewMemoryProvider().Collection("misc")} mod := New(deps) want := map[string]modules.Visibility{ @@ -41,43 +45,31 @@ func TestNew_RegistersExpectedCommands(t *testing.T) { } } -func TestPing_WritesLastPingKV(t *testing.T) { +func TestPing_WritesLastPingStore(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + store := newMiscStore() - // Drive the KV side directly: lock the wire format as a ms-epoch number - // (not an RFC3339 string), matching every other timestamp in the bot's KV. - if err := kv.PutJSON(ctx, lastPingKey, lastPing{At: time.Now().UTC().UnixMilli()}); err != nil { - t.Fatalf("PutJSON: %v", err) + // Drive the store side directly: lock the wire format as a ms-epoch number + // (not an RFC3339 string), matching every other timestamp in the bot's store. + if err := store.Put(ctx, lastPingKey, lastPing{At: 1700000000000}); err != nil { + t.Fatalf("Put: %v", err) } - var got lastPing - if err := kv.GetJSON(ctx, lastPingKey, &got); err != nil { - t.Fatalf("GetJSON: %v", err) - } - if got.At <= 0 { - t.Errorf("read-back lastPing.At = %d, want positive ms-epoch", got.At) - } - - // Verify a hand-written {"at": } document decodes correctly. - if err := kv.Put(ctx, lastPingKey, []byte(`{"at":1700000000000}`)); err != nil { - t.Fatal(err) - } - got = lastPing{} - if err := kv.GetJSON(ctx, lastPingKey, &got); err != nil { - t.Fatalf("GetJSON ms-epoch shape: %v", err) + got, _, err := store.Get(ctx, lastPingKey) + if err != nil { + t.Fatalf("Get: %v", err) } if got.At != 1700000000000 { t.Errorf("ms-epoch round-trip: At = %d, want 1700000000000", got.At) } } -func TestMstats_MissingKVReturnsErrNotFound(t *testing.T) { +func TestMstats_MissingKeyReturnsErrNotFound(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + store := newMiscStore() - var dst lastPing - if err := kv.GetJSON(ctx, lastPingKey, &dst); err != storage.ErrNotFound { - t.Errorf("GetJSON missing = %v, want ErrNotFound", err) + _, _, err := store.Get(ctx, lastPingKey) + if err != storage.ErrNotFound { + t.Errorf("Get missing = %v, want ErrNotFound", err) } } diff --git a/internal/modules/module.go b/internal/modules/module.go index 8ecd7d5..9261931 100644 --- a/internal/modules/module.go +++ b/internal/modules/module.go @@ -71,10 +71,10 @@ type Module struct { // Modules that need to introspect commands (e.g. /help) capture this pointer // in their handler closures. type Deps struct { - KV storage.KVStore // already prefixed with the module name when passed to a Factory - Registry *Registry // populated by Build; safe to capture but read-only at module use - Chatter ai.Chatter // nil if GEMINI_API_KEY unset; twentyq must check - Bot *bot.Bot // nil-safe: only crons that fan-out (lolschedule daily push) need it + Store storage.Collection // the module's own collection; build typed views with storage.Typed[T] + Registry *Registry // populated by Build; safe to capture but read-only at module use + Chatter ai.Chatter // nil if GEMINI_API_KEY unset; twentyq must check + Bot *bot.Bot // nil-safe: only crons that fan-out (lolschedule daily push) need it } // Factory constructs a Module from its Deps. Deps are passed directly (instead diff --git a/internal/modules/registry.go b/internal/modules/registry.go index 48909d5..4d4b43c 100644 --- a/internal/modules/registry.go +++ b/internal/modules/registry.go @@ -83,11 +83,10 @@ func (r *Registry) Crons() []Cron { return out } -// Build constructs a Registry from the requested module names. The KVProvider -// supplies a per-module-isolated KVStore (MemoryProvider key-prefixes a shared -// store; FirestoreProvider hands out one collection per module). Build validates -// every command/cron and aborts on duplicate command names across the union of -// all visibilities. +// Build constructs a Registry from the requested module names. The Provider +// supplies a per-module-isolated Collection (one MongoDB collection per module; +// MemoryProvider keeps one map per module). Build validates every command/cron +// and aborts on duplicate command names across the union of all visibilities. // // Names not present in factories are reported as a single error so a typo in // MODULES does not silently load a smaller bot than intended. Duplicate names @@ -100,9 +99,9 @@ type BuildOptions struct { Bot *bot.Bot } -func Build(enabled []string, factories map[string]Factory, kv storage.KVProvider, opts BuildOptions) (*Registry, error) { - if kv == nil { - return nil, fmt.Errorf("modules: KVProvider is required") +func Build(enabled []string, factories map[string]Factory, provider storage.Provider, opts BuildOptions) (*Registry, error) { + if provider == nil { + return nil, fmt.Errorf("modules: storage Provider is required") } reg := &Registry{ @@ -135,7 +134,7 @@ func Build(enabled []string, factories map[string]Factory, kv storage.KVProvider } moduleDeps := Deps{ - KV: kv.For(name), + Store: provider.Collection(name), Registry: reg, Chatter: opts.Chatter, Bot: opts.Bot, diff --git a/internal/modules/registry_test.go b/internal/modules/registry_test.go index d9a5029..b8413ad 100644 --- a/internal/modules/registry_test.go +++ b/internal/modules/registry_test.go @@ -35,7 +35,7 @@ func factory(name string, cmds []Command, crons []Cron) Factory { } } -func newProvider() storage.KVProvider { return storage.NewMemoryProvider() } +func newProvider() storage.Provider { return storage.NewMemoryProvider() } func TestBuild_EmptyModulesBootsCleanly(t *testing.T) { reg, err := Build(nil, map[string]Factory{}, newProvider(), BuildOptions{}) @@ -165,7 +165,7 @@ func TestDispatchScheduled_UnknownReturnsErrCronNotFound(t *testing.T) { } } -func TestDispatchScheduled_PassesPrefixedDeps(t *testing.T) { +func TestDispatchScheduled_PassesScopedDeps(t *testing.T) { ctx := context.Background() provider := storage.NewMemoryProvider() @@ -174,7 +174,7 @@ func TestDispatchScheduled_PassesPrefixedDeps(t *testing.T) { return Module{Crons: []Cron{{ Name: "tick_a", Handler: func(ctx context.Context, deps Deps) error { - return deps.KV.Put(ctx, "last", []byte("A")) + return storage.Typed[string](deps.Store).Put(ctx, "last", "A") }, }}} }, @@ -182,7 +182,7 @@ func TestDispatchScheduled_PassesPrefixedDeps(t *testing.T) { return Module{Crons: []Cron{{ Name: "tick_b", Handler: func(ctx context.Context, deps Deps) error { - return deps.KV.Put(ctx, "last", []byte("B")) + return storage.Typed[string](deps.Store).Put(ctx, "last", "B") }, }}} }, @@ -198,14 +198,14 @@ func TestDispatchScheduled_PassesPrefixedDeps(t *testing.T) { t.Fatalf("tick_b: %v", err) } - // Underlying base store should hold each module's prefixed key separately. - gotA, err := provider.Base().Get(ctx, "alpha:last") - if err != nil || string(gotA) != "A" { - t.Errorf("alpha:last = %q (err=%v), want A", gotA, err) + // Each module's per-collection store holds its own "last" key separately. + gotA, _, err := storage.Typed[string](provider.Collection("alpha")).Get(ctx, "last") + if err != nil || gotA != "A" { + t.Errorf("alpha/last = %q (err=%v), want A", gotA, err) } - gotB, err := provider.Base().Get(ctx, "beta:last") - if err != nil || string(gotB) != "B" { - t.Errorf("beta:last = %q (err=%v), want B", gotB, err) + gotB, _, err := storage.Typed[string](provider.Collection("beta")).Get(ctx, "last") + if err != nil || gotB != "B" { + t.Errorf("beta/last = %q (err=%v), want B", gotB, err) } } @@ -280,12 +280,12 @@ func TestBuild_RejectsDuplicateModuleInEnv(t *testing.T) { } } -func TestBuild_PerModulePrefixedKV(t *testing.T) { +func TestBuild_PerModuleStoreIsolation(t *testing.T) { ctx := context.Background() provider := storage.NewMemoryProvider() - // Each module writes a value to the same key; with per-module prefixing - // they must not collide. + // Each module writes a value to the same key; with one collection per + // module they must not collide. captured := map[string]Deps{} factories := map[string]Factory{ "alpha": func(d Deps) Module { @@ -301,19 +301,19 @@ func TestBuild_PerModulePrefixedKV(t *testing.T) { t.Fatalf("Build: %v", err) } - if err := captured["alpha"].KV.Put(ctx, "score", []byte("1")); err != nil { + alpha := storage.Typed[string](captured["alpha"].Store) + beta := storage.Typed[string](captured["beta"].Store) + if err := alpha.Put(ctx, "score", "1"); err != nil { t.Fatal(err) } - if err := captured["beta"].KV.Put(ctx, "score", []byte("2")); err != nil { + if err := beta.Put(ctx, "score", "2"); err != nil { t.Fatal(err) } - got, _ := captured["alpha"].KV.Get(ctx, "score") - if string(got) != "1" { - t.Errorf("alpha.KV.score = %q, want 1", got) + if got, _, _ := alpha.Get(ctx, "score"); got != "1" { + t.Errorf("alpha/score = %q, want 1", got) } - got, _ = captured["beta"].KV.Get(ctx, "score") - if string(got) != "2" { - t.Errorf("beta.KV.score = %q, want 2", got) + if got, _, _ := beta.Get(ctx, "score"); got != "2" { + t.Errorf("beta/score = %q, want 2", got) } } diff --git a/internal/modules/stats/stats.go b/internal/modules/stats/stats.go index aa20051..e0c31ad 100644 --- a/internal/modules/stats/stats.go +++ b/internal/modules/stats/stats.go @@ -1,5 +1,5 @@ // Package stats tracks per-command and per-user invocation counts persistently -// in KV and exposes /stats subcommands to display them sorted by popularity. +// in DocStore and exposes /stats subcommands to display them sorted by popularity. package stats import ( @@ -15,10 +15,11 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) -// Sort-key shapes inside the stats module's KV partition: -// count: → countEntry — total per command -// user: → userEntry — cached username + total per user -// pair:: → countEntry — per (command, user) pair +// Sort-key shapes inside the stats module's partition: +// +// count: → countEntry — total per command +// user: → userEntry — cached username + total per user +// pair:: → countEntry — per (command, user) pair const ( countPrefix = "count:" userPrefix = "user:" @@ -27,16 +28,20 @@ const ( ) type countEntry struct { - N int64 `json:"n"` + N int64 `json:"n" bson:"n"` } type userEntry struct { - Username string `json:"username"` - N int64 `json:"n"` + Username string `json:"username" bson:"username"` + N int64 `json:"n" bson:"n"` } +// counter holds two typed views over the same module Collection: +// one for countEntry values and one for userEntry values. +// Keys are disjoint by prefix so both views safely share the collection. type counter struct { - kv storage.KVStore + counts storage.DocStore[countEntry] + users storage.DocStore[userEntry] } func countKey(name string) string { return countPrefix + name } @@ -87,48 +92,51 @@ func (c *counter) Inc(ctx context.Context, name string, update *models.Update) { func (c *counter) incCount(ctx context.Context, name string) { key := countKey(name) - var entry countEntry - if err := c.kv.GetJSON(ctx, key, &entry); err != nil && !errors.Is(err, storage.ErrNotFound) { - log.Error("stats: kv get failed", "key", key, "err", err) + entry, _, err := c.counts.Get(ctx, key) + if err != nil && !errors.Is(err, storage.ErrNotFound) { + log.Error("stats: store get failed", "key", key, "err", err) return } entry.N++ - if err := c.kv.PutJSON(ctx, key, entry); err != nil { - log.Error("stats: kv put failed", "key", key, "err", err) + if err := c.counts.Put(ctx, key, entry); err != nil { + log.Error("stats: store put failed", "key", key, "err", err) } } func (c *counter) incUser(ctx context.Context, id int64, username string) { key := userKey(id) - var entry userEntry - if err := c.kv.GetJSON(ctx, key, &entry); err != nil && !errors.Is(err, storage.ErrNotFound) { - log.Error("stats: kv get failed", "key", key, "err", err) + entry, _, err := c.users.Get(ctx, key) + if err != nil && !errors.Is(err, storage.ErrNotFound) { + log.Error("stats: store get failed", "key", key, "err", err) return } entry.Username = username entry.N++ - if err := c.kv.PutJSON(ctx, key, entry); err != nil { - log.Error("stats: kv put failed", "key", key, "err", err) + if err := c.users.Put(ctx, key, entry); err != nil { + log.Error("stats: store put failed", "key", key, "err", err) } } func (c *counter) incPair(ctx context.Context, name string, id int64) { key := pairKey(name, id) - var entry countEntry - if err := c.kv.GetJSON(ctx, key, &entry); err != nil && !errors.Is(err, storage.ErrNotFound) { - log.Error("stats: kv get failed", "key", key, "err", err) + entry, _, err := c.counts.Get(ctx, key) + if err != nil && !errors.Is(err, storage.ErrNotFound) { + log.Error("stats: store get failed", "key", key, "err", err) return } entry.N++ - if err := c.kv.PutJSON(ctx, key, entry); err != nil { - log.Error("stats: kv put failed", "key", key, "err", err) + if err := c.counts.Put(ctx, key, entry); err != nil { + log.Error("stats: store put failed", "key", key, "err", err) } } // New is the module Factory. Registers a CommandHook that persists counts and // a /stats command that displays them. func New(deps modules.Deps) modules.Module { - c := &counter{kv: deps.KV} + c := &counter{ + counts: storage.Typed[countEntry](deps.Store), + users: storage.Typed[userEntry](deps.Store), + } return modules.Module{ CommandHook: c.Inc, Commands: []modules.Command{ diff --git a/internal/modules/stats/stats_test.go b/internal/modules/stats/stats_test.go index 0fb8b97..8dd60c1 100644 --- a/internal/modules/stats/stats_test.go +++ b/internal/modules/stats/stats_test.go @@ -2,6 +2,7 @@ package stats import ( "context" + "errors" "fmt" "strings" "testing" @@ -13,8 +14,17 @@ import ( "github.com/tiennm99/miti99bot/internal/testutil" ) +// newStatsCounter returns a fresh counter backed by an in-memory collection. +func newStatsCounter() *counter { + col := storage.NewMemoryProvider().Collection("stats") + return &counter{ + counts: storage.Typed[countEntry](col), + users: storage.Typed[userEntry](col), + } +} + func TestNew_RegistersExpectedCommands(t *testing.T) { - deps := modules.Deps{KV: storage.NewMemoryKVStore()} + deps := modules.Deps{Store: storage.NewMemoryProvider().Collection("stats")} mod := New(deps) if len(mod.Commands) != 1 { @@ -45,58 +55,56 @@ func updateFrom(id int64, username string) *models.Update { } } -func TestInc_PersistsCountInKV(t *testing.T) { +func TestInc_PersistsCountInStore(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() - c := &counter{kv: kv} + c := newStatsCounter() c.Inc(ctx, "ping", nil) c.Inc(ctx, "ping", nil) c.Inc(ctx, "wordle", nil) - var entry countEntry - if err := kv.GetJSON(ctx, countKey("ping"), &entry); err != nil { - t.Fatalf("GetJSON ping: %v", err) + entry, _, err := c.counts.Get(ctx, countKey("ping")) + if err != nil { + t.Fatalf("Get ping: %v", err) } if entry.N != 2 { t.Errorf("ping count = %d, want 2", entry.N) } - entry = countEntry{} - if err := kv.GetJSON(ctx, countKey("wordle"), &entry); err != nil { - t.Fatalf("GetJSON wordle: %v", err) + entry2, _, err := c.counts.Get(ctx, countKey("wordle")) + if err != nil { + t.Fatalf("Get wordle: %v", err) } - if entry.N != 1 { - t.Errorf("wordle count = %d, want 1", entry.N) + if entry2.N != 1 { + t.Errorf("wordle count = %d, want 1", entry2.N) } } func TestInc_WithUsernameWritesAllThreeKeys(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() - c := &counter{kv: kv} + c := newStatsCounter() c.Inc(ctx, "ping", updateFrom(42, "alice")) c.Inc(ctx, "ping", updateFrom(42, "alice")) - var ce countEntry - if err := kv.GetJSON(ctx, countKey("ping"), &ce); err != nil { + ce, _, err := c.counts.Get(ctx, countKey("ping")) + if err != nil { t.Fatalf("count:ping: %v", err) } if ce.N != 2 { t.Errorf("count:ping N = %d, want 2", ce.N) } - var ue userEntry - if err := kv.GetJSON(ctx, userKey(42), &ue); err != nil { + ue, _, err := c.users.Get(ctx, userKey(42)) + if err != nil { t.Fatalf("user:42: %v", err) } if ue.N != 2 || ue.Username != "alice" { t.Errorf("user:42 = {%q, %d}, want {alice, 2}", ue.Username, ue.N) } - var pe countEntry - if err := kv.GetJSON(ctx, pairKey("ping", 42), &pe); err != nil { + pe, _, err := c.counts.Get(ctx, pairKey("ping", 42)) + if err != nil { t.Fatalf("pair:ping:42: %v", err) } if pe.N != 2 { @@ -106,37 +114,35 @@ func TestInc_WithUsernameWritesAllThreeKeys(t *testing.T) { func TestInc_EmptyUsernameSkipsUserAndPair(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() - c := &counter{kv: kv} + c := newStatsCounter() c.Inc(ctx, "ping", updateFrom(42, "")) - var ce countEntry - if err := kv.GetJSON(ctx, countKey("ping"), &ce); err != nil { + ce, _, err := c.counts.Get(ctx, countKey("ping")) + if err != nil { t.Fatalf("count:ping: %v", err) } if ce.N != 1 { t.Errorf("count:ping N = %d, want 1", ce.N) } - if _, err := kv.Get(ctx, userKey(42)); err != storage.ErrNotFound { + if _, _, err := c.users.Get(ctx, userKey(42)); !errors.Is(err, storage.ErrNotFound) { t.Errorf("user:42 should be absent, got err=%v", err) } - if _, err := kv.Get(ctx, pairKey("ping", 42)); err != storage.ErrNotFound { + if _, _, err := c.counts.Get(ctx, pairKey("ping", 42)); !errors.Is(err, storage.ErrNotFound) { t.Errorf("pair:ping:42 should be absent, got err=%v", err) } } func TestInc_RefreshesUsernameOnRename(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() - c := &counter{kv: kv} + c := newStatsCounter() c.Inc(ctx, "ping", updateFrom(42, "alice")) c.Inc(ctx, "ping", updateFrom(42, "alice2")) - var ue userEntry - if err := kv.GetJSON(ctx, userKey(42), &ue); err != nil { + ue, _, err := c.users.Get(ctx, userKey(42)) + if err != nil { t.Fatalf("user:42: %v", err) } if ue.Username != "alice2" { @@ -150,8 +156,7 @@ func TestInc_RefreshesUsernameOnRename(t *testing.T) { func installStats(t *testing.T) (*testutil.RecordingBot, *counter) { t.Helper() rb := testutil.NewRecordingBot(t) - kv := storage.NewMemoryKVStore() - c := &counter{kv: kv} + c := newStatsCounter() mod := modules.Module{ Commands: []modules.Command{statsCommand(c)}, } @@ -226,10 +231,11 @@ func TestCommandHook_FiredThroughModulesBuild(t *testing.T) { reg.RunCommandHooks(ctx, "ping", nil) - statsKV := provider.For("stats") - var entry countEntry - if err := statsKV.GetJSON(ctx, countKey("ping"), &entry); err != nil { - t.Fatalf("expected count:ping in KV after hook: %v", err) + // Verify by reading back via typed store (the same collection Build gave to stats). + statsStore := storage.Typed[countEntry](provider.Collection("stats")) + entry, _, err := statsStore.Get(ctx, countKey("ping")) + if err != nil { + t.Fatalf("expected count:ping in store after hook: %v", err) } if entry.N != 1 { t.Errorf("count:ping = %d, want 1", entry.N) @@ -256,7 +262,7 @@ func seedFixture(t *testing.T, c *counter) { } func TestRenderStats_TopCommands(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() seedFixture(t, c) got := renderStats(context.Background(), c, "") @@ -275,7 +281,7 @@ func TestRenderStats_TopCommands(t *testing.T) { } func TestRenderStats_Users(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() seedFixture(t, c) got := renderStats(context.Background(), c, "users") @@ -295,7 +301,7 @@ func TestRenderStats_Users(t *testing.T) { } func TestRenderStats_UserCommands(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() seedFixture(t, c) got := renderStats(context.Background(), c, "user @alice") @@ -320,7 +326,7 @@ func TestRenderStats_UserCommands(t *testing.T) { // flagged this as a potential bug; this test proves the absence of the bug. func TestRenderStats_UserCommands_IDSuffixDoesNotFalseMatch(t *testing.T) { ctx := context.Background() - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() // user 2 ("two") calls /ping; user 12 ("twelve") calls /wordle. c.Inc(ctx, "ping", updateFrom(2, "two")) c.Inc(ctx, "wordle", updateFrom(12, "twelve")) @@ -338,7 +344,7 @@ func TestRenderStats_UserCommands_IDSuffixDoesNotFalseMatch(t *testing.T) { } func TestRenderStats_UserCommands_BareNameAlsoWorks(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() seedFixture(t, c) got := renderStats(context.Background(), c, "user alice") @@ -348,7 +354,7 @@ func TestRenderStats_UserCommands_BareNameAlsoWorks(t *testing.T) { } func TestRenderStats_UserNotFound(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() seedFixture(t, c) got := renderStats(context.Background(), c, "user @ghost") @@ -358,7 +364,7 @@ func TestRenderStats_UserNotFound(t *testing.T) { } func TestRenderStats_CmdUsers(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() seedFixture(t, c) got := renderStats(context.Background(), c, "cmd wordle") @@ -380,7 +386,7 @@ func TestRenderStats_CmdUsers(t *testing.T) { } func TestRenderStats_CmdNotFound(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() seedFixture(t, c) got := renderStats(context.Background(), c, "cmd nonexistent") @@ -390,7 +396,7 @@ func TestRenderStats_CmdNotFound(t *testing.T) { } func TestRenderStats_UnknownSubcommand(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() got := renderStats(context.Background(), c, "bogus") if !strings.HasPrefix(got, "Usage:") { t.Errorf("expected usage string, got %q", got) @@ -398,7 +404,7 @@ func TestRenderStats_UnknownSubcommand(t *testing.T) { } func TestRenderStats_MissingRequiredArg(t *testing.T) { - c := &counter{kv: storage.NewMemoryKVStore()} + c := newStatsCounter() if got := renderStats(context.Background(), c, "user"); !strings.HasPrefix(got, "Usage:") { t.Errorf("user (no arg) should return usage, got %q", got) } diff --git a/internal/modules/stats/views.go b/internal/modules/stats/views.go index 2065e26..060d50c 100644 --- a/internal/modules/stats/views.go +++ b/internal/modules/stats/views.go @@ -89,9 +89,9 @@ func parseSubargs(update *models.Update) string { } func viewTopCommands(ctx context.Context, c *counter) string { - keys, err := c.kv.List(ctx, countPrefix) + keys, err := c.counts.List(ctx, countPrefix) if err != nil { - log.Error("stats: kv list failed", "err", err) + log.Error("stats: store list failed", "err", err) return "Could not load stats. Try again later." } if len(keys) == 0 { @@ -145,9 +145,9 @@ func viewUserCommands(ctx context.Context, c *counter, username string) string { return fmt.Sprintf("User @%s not found.", username) } - keys, err := c.kv.List(ctx, pairPrefix) + keys, err := c.counts.List(ctx, pairPrefix) if err != nil { - log.Error("stats: kv list failed", "err", err) + log.Error("stats: store list failed", "err", err) return "Could not load stats. Try again later." } // Leading colon disambiguates: ":2" does not match a key ending in "12" @@ -181,9 +181,9 @@ func viewUserCommands(ctx context.Context, c *counter, username string) string { func viewCmdUsers(ctx context.Context, c *counter, cmd string) string { prefix := pairPrefix + cmd + ":" - keys, err := c.kv.List(ctx, prefix) + keys, err := c.counts.List(ctx, prefix) if err != nil { - log.Error("stats: kv list failed", "err", err) + log.Error("stats: store list failed", "err", err) return "Could not load stats. Try again later." } if len(keys) == 0 { @@ -200,9 +200,9 @@ func viewCmdUsers(ctx context.Context, c *counter, cmd string) string { wg.Add(1) go func(i int, k string) { defer wg.Done() - var ce countEntry - if err := c.kv.GetJSON(ctx, k, &ce); err != nil { - log.Error("stats: kv get failed", "key", k, "err", err) + ce, _, err := c.counts.Get(ctx, k) + if err != nil { + log.Error("stats: store get failed", "key", k, "err", err) return } idStr := strings.TrimPrefix(k, prefix) @@ -210,8 +210,8 @@ func viewCmdUsers(ctx context.Context, c *counter, cmd string) string { if err != nil { return } - var ue userEntry - if err := c.kv.GetJSON(ctx, userKey(id), &ue); err != nil || ue.Username == "" { + ue, _, err := c.users.Get(ctx, userKey(id)) + if err != nil || ue.Username == "" { return } results[i] = result{r: row{display: "@" + ue.Username, n: ce.N}, ok: true} @@ -232,7 +232,7 @@ func viewCmdUsers(ctx context.Context, c *counter, cmd string) string { return renderTopN(fmt.Sprintf("Users of /%s:", cmd), rows, topK) } -// fanOutCountRows runs GetJSON on each key in parallel. displayFor maps the +// fanOutCountRows runs Get on each key in parallel. displayFor maps the // raw sort key to the human-readable label that gets rendered. Missing / // errored keys are skipped (logged at error level). func fanOutCountRows(ctx context.Context, c *counter, keys []string, displayFor func(k string) string) []row { @@ -246,9 +246,9 @@ func fanOutCountRows(ctx context.Context, c *counter, keys []string, displayFor wg.Add(1) go func(i int, k string) { defer wg.Done() - var entry countEntry - if err := c.kv.GetJSON(ctx, k, &entry); err != nil { - log.Error("stats: kv get failed during render", "key", k, "err", err) + entry, _, err := c.counts.Get(ctx, k) + if err != nil { + log.Error("stats: store get failed during render", "key", k, "err", err) return } results[i] = result{r: row{display: displayFor(k), n: entry.N}, ok: true} @@ -268,9 +268,9 @@ func fanOutCountRows(ctx context.Context, c *counter, keys []string, displayFor // loadUserRowsWithID returns userID → userEntry for every user:* row. Used by // viewTopUsers (renders) and viewUserCommands (resolves a username to its ID). func loadUserRowsWithID(ctx context.Context, c *counter) map[int64]userEntry { - keys, err := c.kv.List(ctx, userPrefix) + keys, err := c.users.List(ctx, userPrefix) if err != nil { - log.Error("stats: kv list failed", "err", err) + log.Error("stats: store list failed", "err", err) return nil } type pair struct { @@ -289,9 +289,9 @@ func loadUserRowsWithID(ctx context.Context, c *counter) map[int64]userEntry { if err != nil { return } - var entry userEntry - if err := c.kv.GetJSON(ctx, k, &entry); err != nil { - log.Error("stats: kv get failed", "key", k, "err", err) + entry, _, err := c.users.Get(ctx, k) + if err != nil { + log.Error("stats: store get failed", "key", k, "err", err) return } results[i] = pair{id: id, entry: entry, ok: true} diff --git a/internal/modules/stock/handlers.go b/internal/modules/stock/handlers.go index 324e468..c2f44f1 100644 --- a/internal/modules/stock/handlers.go +++ b/internal/modules/stock/handlers.go @@ -13,14 +13,13 @@ import ( "github.com/tiennm99/miti99bot/internal/keylock" "github.com/tiennm99/miti99bot/internal/log" "github.com/tiennm99/miti99bot/internal/modules/util/chathelper" - "github.com/tiennm99/miti99bot/internal/storage" ) -// state is the per-module runtime. KV is module-scoped (the framework +// state is the per-module runtime. store is module-scoped (the framework // prefixes/partitions). PriceClient is reused across calls; nowFn allows // tests to inject a deterministic clock for portfolio CreatedAt. type state struct { - kv storage.KVStore + store Store prices *PriceClient locks keylock.Map nowFn func() time.Time @@ -35,9 +34,9 @@ func (s *state) now() time.Time { } // newState builds the default state used by the module factory. -func newState(kv storage.KVStore) *state { +func newState(store Store) *state { return &state{ - kv: kv, + store: store, prices: &PriceClient{}, comingSoonMessage: "Crypto, gold & currency exchange coming soon!", } @@ -87,14 +86,14 @@ func (s *state) handleTopup(ctx context.Context, b *bot.Bot, update *models.Upda defer s.locks.Acquire(strconv.FormatInt(userID, 10))() - p, err := LoadPortfolio(ctx, s.kv, userID, s.now().UnixMilli()) + p, err := LoadPortfolio(ctx, s.store, userID, s.now().UnixMilli()) if err != nil { log.Error("stock_load_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not load portfolio. Try again later.") } p.AddCurrency("VND", amount) p.Meta.Invested += amount - if err := SavePortfolio(ctx, s.kv, userID, p); err != nil { + if err := SavePortfolio(ctx, s.store, userID, p); err != nil { log.Error("stock_save_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not save portfolio. Try again later.") } @@ -138,7 +137,7 @@ func (s *state) handleBuy(ctx context.Context, b *bot.Bot, update *models.Update defer s.locks.Acquire(strconv.FormatInt(userID, 10))() - p, err := LoadPortfolio(ctx, s.kv, userID, s.now().UnixMilli()) + p, err := LoadPortfolio(ctx, s.store, userID, s.now().UnixMilli()) if err != nil { log.Error("stock_load_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not load portfolio. Try again later.") @@ -149,7 +148,7 @@ func (s *state) handleBuy(ctx context.Context, b *bot.Bot, update *models.Update "Insufficient VND. Need "+FormatVND(cost)+", have "+FormatVND(balance)+".") } p.AddAsset(symbol, qty) - if err := SavePortfolio(ctx, s.kv, userID, p); err != nil { + if err := SavePortfolio(ctx, s.store, userID, p); err != nil { log.Error("stock_save_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not save portfolio. Try again later.") } @@ -196,7 +195,7 @@ func (s *state) handleSell(ctx context.Context, b *bot.Bot, update *models.Updat defer s.locks.Acquire(strconv.FormatInt(userID, 10))() - p, err := LoadPortfolio(ctx, s.kv, userID, s.now().UnixMilli()) + p, err := LoadPortfolio(ctx, s.store, userID, s.now().UnixMilli()) if err != nil { log.Error("stock_load_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not load portfolio. Try again later.") @@ -208,7 +207,7 @@ func (s *state) handleSell(ctx context.Context, b *bot.Bot, update *models.Updat } revenue := float64(qty) * price p.AddCurrency("VND", revenue) - if err := SavePortfolio(ctx, s.kv, userID, p); err != nil { + if err := SavePortfolio(ctx, s.store, userID, p); err != nil { log.Error("stock_save_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not save portfolio. Try again later.") } @@ -245,7 +244,7 @@ func (s *state) handleIncomeStock(ctx context.Context, b *bot.Bot, update *model defer s.locks.Acquire(strconv.FormatInt(userID, 10))() - p, err := LoadPortfolio(ctx, s.kv, userID, s.now().UnixMilli()) + p, err := LoadPortfolio(ctx, s.store, userID, s.now().UnixMilli()) if err != nil { log.Error("stock_load_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not load portfolio. Try again later.") @@ -256,7 +255,7 @@ func (s *state) handleIncomeStock(ctx context.Context, b *bot.Bot, update *model "You don't hold any "+symbol+" to receive a stock dividend.") } p.AddAsset(symbol, qty) - if err := SavePortfolio(ctx, s.kv, userID, p); err != nil { + if err := SavePortfolio(ctx, s.store, userID, p); err != nil { log.Error("stock_save_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not save portfolio. Try again later.") } @@ -292,7 +291,7 @@ func (s *state) handleIncomeVND(ctx context.Context, b *bot.Bot, update *models. defer s.locks.Acquire(strconv.FormatInt(userID, 10))() - p, err := LoadPortfolio(ctx, s.kv, userID, s.now().UnixMilli()) + p, err := LoadPortfolio(ctx, s.store, userID, s.now().UnixMilli()) if err != nil { log.Error("stock_load_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not load portfolio. Try again later.") @@ -304,7 +303,7 @@ func (s *state) handleIncomeVND(ctx context.Context, b *bot.Bot, update *models. } total := amountPerShare * float64(held) p.AddCurrency("VND", total) - if err := SavePortfolio(ctx, s.kv, userID, p); err != nil { + if err := SavePortfolio(ctx, s.store, userID, p); err != nil { log.Error("stock_save_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not save portfolio. Try again later.") } @@ -330,7 +329,7 @@ func (s *state) handleStats(ctx context.Context, b *bot.Bot, update *models.Upda return chathelper.Reply(ctx, b, update.Message, "Cannot identify user — /stock_stats needs a sender.") } - p, err := LoadPortfolio(ctx, s.kv, userID, s.now().UnixMilli()) + p, err := LoadPortfolio(ctx, s.store, userID, s.now().UnixMilli()) if err != nil { log.Error("stock_load_portfolio", "user", userID, "err", err) return chathelper.Reply(ctx, b, update.Message, "Could not load portfolio. Try again later.") diff --git a/internal/modules/stock/portfolio.go b/internal/modules/stock/portfolio.go index 49c3b40..79f7061 100644 --- a/internal/modules/stock/portfolio.go +++ b/internal/modules/stock/portfolio.go @@ -9,19 +9,22 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) +// Store is the stock module's typed portfolio store. +type Store = storage.DocStore[Portfolio] + // Portfolio is the per-user stock state. Currency is a map for forward- // compat with USD/EUR (currently VND-only). Assets is a flat ticker→qty map. type Portfolio struct { - Currency map[string]float64 `json:"currency"` - Assets map[string]int64 `json:"assets"` - Meta PortfolioMeta `json:"meta"` + Currency map[string]float64 `json:"currency" bson:"currency"` + Assets map[string]int64 `json:"assets" bson:"assets"` + Meta PortfolioMeta `json:"meta" bson:"meta"` } // PortfolioMeta tracks invested cost basis for P&L. CreatedAt is purely // informational (ms-epoch when the portfolio first existed). type PortfolioMeta struct { - Invested float64 `json:"invested"` - CreatedAt int64 `json:"createdAt"` + Invested float64 `json:"invested" bson:"invested"` + CreatedAt int64 `json:"createdAt" bson:"createdAt"` } // NewPortfolio returns an empty starting state. Currency map seeded with VND=0 @@ -39,11 +42,10 @@ func portfolioKey(userID int64) string { return "user:" + strconv.FormatInt(userID, 10) } -// LoadPortfolio reads from KV; returns an empty portfolio on first-time use. +// LoadPortfolio reads from store; returns an empty portfolio on first-time use. // Defensively initialises nil maps so callers never need a nil check. -func LoadPortfolio(ctx context.Context, kv storage.KVStore, userID int64, now int64) (Portfolio, error) { - var p Portfolio - err := kv.GetJSON(ctx, portfolioKey(userID), &p) +func LoadPortfolio(ctx context.Context, store Store, userID int64, now int64) (Portfolio, error) { + p, _, err := store.Get(ctx, portfolioKey(userID)) switch { case err == nil: // Repair any nils from older / partial saves — defence in depth. @@ -64,8 +66,8 @@ func LoadPortfolio(ctx context.Context, kv storage.KVStore, userID int64, now in } // SavePortfolio persists the portfolio. -func SavePortfolio(ctx context.Context, kv storage.KVStore, userID int64, p Portfolio) error { - if err := kv.PutJSON(ctx, portfolioKey(userID), p); err != nil { +func SavePortfolio(ctx context.Context, store Store, userID int64, p Portfolio) error { + if err := store.Put(ctx, portfolioKey(userID), p); err != nil { return fmt.Errorf("stock: save portfolio %d: %w", userID, err) } return nil diff --git a/internal/modules/stock/portfolio_test.go b/internal/modules/stock/portfolio_test.go index 54119aa..2c014b0 100644 --- a/internal/modules/stock/portfolio_test.go +++ b/internal/modules/stock/portfolio_test.go @@ -7,9 +7,14 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) +// newStockStore returns a fresh in-memory typed portfolio store for tests. +func newStockStore() Store { + return storage.Typed[Portfolio](storage.NewMemoryProvider().Collection("stock")) +} + func TestLoadPortfolio_FirstTimeUser(t *testing.T) { - kv := storage.NewMemoryKVStore() - p, err := LoadPortfolio(context.Background(), kv, 42, 1234567890) + store := newStockStore() + p, err := LoadPortfolio(context.Background(), store, 42, 1234567890) if err != nil { t.Fatalf("LoadPortfolio: %v", err) } @@ -25,15 +30,15 @@ func TestLoadPortfolio_FirstTimeUser(t *testing.T) { } func TestSaveAndLoadRoundTrip(t *testing.T) { - kv := storage.NewMemoryKVStore() - p, _ := LoadPortfolio(context.Background(), kv, 42, 1) + store := newStockStore() + p, _ := LoadPortfolio(context.Background(), store, 42, 1) p.AddCurrency("VND", 5_000_000) p.AddAsset("TCB", 100) p.Meta.Invested = 5_000_000 - if err := SavePortfolio(context.Background(), kv, 42, p); err != nil { + if err := SavePortfolio(context.Background(), store, 42, p); err != nil { t.Fatalf("Save: %v", err) } - got, err := LoadPortfolio(context.Background(), kv, 42, 999) // CreatedAt should NOT be reset + got, err := LoadPortfolio(context.Background(), store, 42, 999) // CreatedAt should NOT be reset if err != nil { t.Fatalf("Load: %v", err) } diff --git a/internal/modules/stock/stats_test.go b/internal/modules/stock/stats_test.go index 446474d..16fcf1a 100644 --- a/internal/modules/stock/stats_test.go +++ b/internal/modules/stock/stats_test.go @@ -10,7 +10,6 @@ import ( "testing" "time" - "github.com/tiennm99/miti99bot/internal/storage" "github.com/tiennm99/miti99bot/internal/testutil" ) @@ -37,19 +36,19 @@ func TestHandleStats_UsesSSIBatchPrices(t *testing.T) { })) t.Cleanup(priceSrv.Close) - kv := storage.NewMemoryKVStore() + store := newStockStore() p := NewPortfolio(now.UnixMilli()) p.Currency["VND"] = 2335000 p.Meta.Invested = 1000000000 p.AddAsset("MWG", 1800) p.AddAsset("TCB", 4200) p.AddAsset("FPT", 2300) - if err := SavePortfolio(ctx, kv, 7, p); err != nil { + if err := SavePortfolio(ctx, store, 7, p); err != nil { t.Fatalf("SavePortfolio: %v", err) } s := &state{ - kv: kv, + store: store, prices: &PriceClient{HTTP: priceSrv.Client(), URL: priceSrv.URL}, nowFn: func() time.Time { return now }, } diff --git a/internal/modules/stock/stock.go b/internal/modules/stock/stock.go index e447885..24c1340 100644 --- a/internal/modules/stock/stock.go +++ b/internal/modules/stock/stock.go @@ -2,13 +2,14 @@ package stock import ( "github.com/tiennm99/miti99bot/internal/modules" + "github.com/tiennm99/miti99bot/internal/storage" ) // New is the stock module Factory. Five user-facing commands; no crons. // (Original miti99bot only has a SQL retention cron, which our KV-only port // does not implement — keeping commits paper-ledger-only is acceptable.) func New(deps modules.Deps) modules.Module { - s := newState(deps.KV) + s := newState(storage.Typed[Portfolio](deps.Store)) return modules.Module{ Commands: []modules.Command{ { diff --git a/internal/modules/twentyq/handlers.go b/internal/modules/twentyq/handlers.go index 6fb857f..d6169d7 100644 --- a/internal/modules/twentyq/handlers.go +++ b/internal/modules/twentyq/handlers.go @@ -15,7 +15,6 @@ import ( "github.com/tiennm99/miti99bot/internal/keylock" "github.com/tiennm99/miti99bot/internal/log" "github.com/tiennm99/miti99bot/internal/modules/util/chathelper" - "github.com/tiennm99/miti99bot/internal/storage" ) const ( @@ -26,7 +25,8 @@ const ( ) type state struct { - kv storage.KVStore + games GameStore + stats StatsStore chatter ai.Chatter limiter *ai.PerUserLimiter @@ -88,7 +88,7 @@ func (s *state) handleTwentyq(ctx context.Context, b *bot.Bot, update *models.Up arg := chathelper.ArgAfterCommand(msg.Text) - game, err := loadGame(ctx, s.kv, subject) + game, err := loadGame(ctx, s.games, subject) if err != nil { return err } @@ -99,7 +99,7 @@ func (s *state) handleTwentyq(ctx context.Context, b *bot.Bot, update *models.Up // branch re-enters every call and the user is stuck. Log but keep // going — the fresh-round write below will overwrite the slot // regardless. - if err := clearGame(ctx, s.kv, subject); err != nil { + if err := clearGame(ctx, s.games, subject); err != nil { log.Warn("twentyq clearGame on solved-relaunch failed", "subject", subject, "err", err) } game = nil @@ -117,7 +117,7 @@ func (s *state) handleTwentyq(ctx context.Context, b *bot.Bot, update *models.Up log.Warn("twentyq roundstart failed", "err", err) return chathelper.Reply(ctx, b, msg, upstreamFail) } - if err := saveGame(ctx, s.kv, subject, fresh); err != nil { + if err := saveGame(ctx, s.games, subject, fresh); err != nil { return err } if arg == "" { @@ -182,17 +182,17 @@ func (s *state) submitTurn(ctx context.Context, b *bot.Bot, msg *models.Message, if won { game.Solved = true count := len(game.Turns) - if _, err := recordResult(ctx, s.kv, subject, true, count, chathelper.NowMillis()); err != nil { + if _, err := recordResult(ctx, s.stats, subject, true, count, chathelper.NowMillis()); err != nil { return err } - if err := clearGame(ctx, s.kv, subject); err != nil { + if err := clearGame(ctx, s.games, subject); err != nil { return err } return chathelper.ReplyHTML(ctx, b, msg, formatTurnReply(turn, true, game.Target, count)) } - if err := saveGame(ctx, s.kv, subject, game); err != nil { + if err := saveGame(ctx, s.games, subject, game); err != nil { return err } return chathelper.ReplyHTML(ctx, b, msg, @@ -209,17 +209,17 @@ func (s *state) handleGiveup(ctx context.Context, b *bot.Bot, update *models.Upd return chathelper.Reply(ctx, b, msg, "Cannot identify chat.") } defer s.locks.Acquire(subject)() - game, err := loadGame(ctx, s.kv, subject) + game, err := loadGame(ctx, s.games, subject) if err != nil { return err } if game == nil { return chathelper.ReplyHTML(ctx, b, msg, noRound) } - if _, err := recordResult(ctx, s.kv, subject, false, len(game.Turns), chathelper.NowMillis()); err != nil { + if _, err := recordResult(ctx, s.stats, subject, false, len(game.Turns), chathelper.NowMillis()); err != nil { return err } - if err := clearGame(ctx, s.kv, subject); err != nil { + if err := clearGame(ctx, s.games, subject); err != nil { return err } return chathelper.ReplyHTML(ctx, b, msg, formatGiveup(*game)) @@ -234,7 +234,7 @@ func (s *state) handleStats(ctx context.Context, b *bot.Bot, update *models.Upda if subject == "" { return chathelper.Reply(ctx, b, msg, "Cannot identify chat.") } - st, err := loadStats(ctx, s.kv, subject) + st, err := loadStats(ctx, s.stats, subject) if err != nil { return err } diff --git a/internal/modules/twentyq/state.go b/internal/modules/twentyq/state.go index f657cc4..9aa7e2e 100644 --- a/internal/modules/twentyq/state.go +++ b/internal/modules/twentyq/state.go @@ -10,36 +10,41 @@ import ( // Turn is one Q&A entry stored in the game's history. type Turn struct { - Text string `json:"text"` - IsGuess bool `json:"isGuess"` - Answer string `json:"answer"` // "yes" | "no" - Hint string `json:"hint"` - TS int64 `json:"ts"` + Text string `json:"text" bson:"text"` + IsGuess bool `json:"isGuess" bson:"isGuess"` + Answer string `json:"answer" bson:"answer"` // "yes" | "no" + Hint string `json:"hint" bson:"hint"` + TS int64 `json:"ts" bson:"ts"` } type GameState struct { - Category string `json:"category"` - Target string `json:"target"` - InitialHint string `json:"initialHint"` - StartedAt *int64 `json:"startedAt"` - Solved bool `json:"solved"` - Turns []Turn `json:"turns"` + Category string `json:"category" bson:"category"` + Target string `json:"target" bson:"target"` + InitialHint string `json:"initialHint" bson:"initialHint"` + StartedAt *int64 `json:"startedAt" bson:"startedAt"` + Solved bool `json:"solved" bson:"solved"` + Turns []Turn `json:"turns" bson:"turns"` } type Stats struct { - Played int `json:"played"` - Solved int `json:"solved"` - TotalTurns int `json:"totalTurns"` - BestTurnCount *int `json:"bestTurnCount"` - LastResultAt *int64 `json:"lastResultAt"` + Played int `json:"played" bson:"played"` + Solved int `json:"solved" bson:"solved"` + TotalTurns int `json:"totalTurns" bson:"totalTurns"` + BestTurnCount *int `json:"bestTurnCount" bson:"bestTurnCount"` + LastResultAt *int64 `json:"lastResultAt" bson:"lastResultAt"` } +// GameStore is the twentyq module's typed game store. +type GameStore = storage.DocStore[GameState] + +// StatsStore is the twentyq module's typed stats store. +type StatsStore = storage.DocStore[Stats] + func gameKey(subject string) string { return "game:" + subject } func statsKey(subject string) string { return "stats:" + subject } -func loadGame(ctx context.Context, kv storage.KVStore, subject string) (*GameState, error) { - var g GameState - err := kv.GetJSON(ctx, gameKey(subject), &g) +func loadGame(ctx context.Context, games GameStore, subject string) (*GameState, error) { + g, _, err := games.Get(ctx, gameKey(subject)) switch { case err == nil: return &g, nil @@ -50,23 +55,22 @@ func loadGame(ctx context.Context, kv storage.KVStore, subject string) (*GameSta } } -func saveGame(ctx context.Context, kv storage.KVStore, subject string, g *GameState) error { - if err := kv.PutJSON(ctx, gameKey(subject), g); err != nil { +func saveGame(ctx context.Context, games GameStore, subject string, g *GameState) error { + if err := games.Put(ctx, gameKey(subject), *g); err != nil { return fmt.Errorf("twentyq saveGame: %w", err) } return nil } -func clearGame(ctx context.Context, kv storage.KVStore, subject string) error { - if err := kv.Delete(ctx, gameKey(subject)); err != nil && !errors.Is(err, storage.ErrNotFound) { +func clearGame(ctx context.Context, games GameStore, subject string) error { + if err := games.Delete(ctx, gameKey(subject)); err != nil && !errors.Is(err, storage.ErrNotFound) { return fmt.Errorf("twentyq clearGame: %w", err) } return nil } -func loadStats(ctx context.Context, kv storage.KVStore, subject string) (*Stats, error) { - var s Stats - err := kv.GetJSON(ctx, statsKey(subject), &s) +func loadStats(ctx context.Context, st StatsStore, subject string) (*Stats, error) { + s, _, err := st.Get(ctx, statsKey(subject)) switch { case err == nil: return &s, nil @@ -77,8 +81,8 @@ func loadStats(ctx context.Context, kv storage.KVStore, subject string) (*Stats, } } -func recordResult(ctx context.Context, kv storage.KVStore, subject string, solved bool, turnCount int, nowMillis int64) (*Stats, error) { - s, err := loadStats(ctx, kv, subject) +func recordResult(ctx context.Context, st StatsStore, subject string, solved bool, turnCount int, nowMillis int64) (*Stats, error) { + s, err := loadStats(ctx, st, subject) if err != nil { return nil, err } @@ -93,7 +97,7 @@ func recordResult(ctx context.Context, kv storage.KVStore, subject string, solve } now := nowMillis s.LastResultAt = &now - if err := kv.PutJSON(ctx, statsKey(subject), s); err != nil { + if err := st.Put(ctx, statsKey(subject), *s); err != nil { return nil, fmt.Errorf("twentyq recordResult: %w", err) } return s, nil diff --git a/internal/modules/twentyq/twentyq.go b/internal/modules/twentyq/twentyq.go index d67aef7..b97d84a 100644 --- a/internal/modules/twentyq/twentyq.go +++ b/internal/modules/twentyq/twentyq.go @@ -7,6 +7,7 @@ import ( "github.com/tiennm99/miti99bot/internal/ai" "github.com/tiennm99/miti99bot/internal/modules" + "github.com/tiennm99/miti99bot/internal/storage" ) // New is the twentyq module Factory. If Deps.Chatter is nil @@ -14,7 +15,8 @@ import ( // config-error message — keeps the rest of the bot functional. func New(deps modules.Deps) modules.Module { s := &state{ - kv: deps.KV, + games: storage.Typed[GameState](deps.Store), + stats: storage.Typed[Stats](deps.Store), chatter: deps.Chatter, limiter: ai.NewPerUserLimiter(5.0/60.0, 5), rng: newRNG(), diff --git a/internal/modules/wordle/compare.go b/internal/modules/wordle/compare.go index 07e64f9..1472312 100644 --- a/internal/modules/wordle/compare.go +++ b/internal/modules/wordle/compare.go @@ -14,10 +14,12 @@ const ( ResultWrong = "wrong" ) -// LetterScore is the JSON shape stored in KV per guess. +// LetterScore is the shape stored per guess (nested in GameState). bson tags +// mirror the json names so migrated docs (which keep the original JSON keys) +// read back verbatim. type LetterScore struct { - Letter string `json:"letter"` - Result string `json:"result"` + Letter string `json:"letter" bson:"letter"` + Result string `json:"result" bson:"result"` } // CompareWords scores guess against target letter-by-letter. Both are assumed diff --git a/internal/modules/wordle/handlers.go b/internal/modules/wordle/handlers.go index 2dfffce..8e24050 100644 --- a/internal/modules/wordle/handlers.go +++ b/internal/modules/wordle/handlers.go @@ -10,13 +10,13 @@ import ( "github.com/tiennm99/miti99bot/internal/keylock" "github.com/tiennm99/miti99bot/internal/modules/util/chathelper" - "github.com/tiennm99/miti99bot/internal/storage" ) // state captures everything a wordle command needs at handler-time. Built // once in New and shared across all four handlers via closure. type state struct { - kv storage.KVStore + games GameStore + stats StatsStore words []string set map[string]struct{} locks keylock.Map // per-subject mutex; serialises Get→mutate→Put @@ -35,7 +35,7 @@ func rejectMessage(reason rejectReason) string { } // startFresh writes a brand-new round and returns it. Errors propagate to -// the caller (not swallowed — a KV failure means subsequent ops will lie). +// the caller (not swallowed — a store failure means subsequent ops will lie). func (s *state) startFresh(ctx context.Context, subject string) (*GameState, error) { target, err := pickRandom(s.words, nil) if err != nil { @@ -48,14 +48,14 @@ func (s *state) startFresh(ctx context.Context, subject string) (*GameState, err Giveup: false, StartedAt: chathelper.NowMillis(), } - if err := saveGame(ctx, s.kv, subject, g); err != nil { + if err := saveGame(ctx, s.games, subject, g); err != nil { return nil, err } return g, nil } func (s *state) getOrInit(ctx context.Context, subject string) (*GameState, error) { - g, err := loadGame(ctx, s.kv, subject) + g, err := loadGame(ctx, s.games, subject) if err != nil { return nil, err } @@ -112,21 +112,21 @@ func (s *state) handleWordle(ctx context.Context, b *bot.Bot, update *models.Upd if won { g.Solved = true } - if err := saveGame(ctx, s.kv, subject, g); err != nil { + if err := saveGame(ctx, s.games, subject, g); err != nil { return err } rendered := renderGuess(v.Word, results) switch { case won: - stats, err := recordResult(ctx, s.kv, subject, true, chathelper.NowMillis()) + st, err := recordResult(ctx, s.stats, subject, true, chathelper.NowMillis()) if err != nil { return err } return chathelper.Reply(ctx, b, msg, fmt.Sprintf("%s\n\n🎉 Solved in %d/%d! Streak: %d. /wordle_new for another.", - rendered, len(g.Guesses), MaxGuesses, stats.Streak)) + rendered, len(g.Guesses), MaxGuesses, st.Streak)) case len(g.Guesses) >= MaxGuesses: - if _, err := recordResult(ctx, s.kv, subject, false, chathelper.NowMillis()); err != nil { + if _, err := recordResult(ctx, s.stats, subject, false, chathelper.NowMillis()); err != nil { return err } return chathelper.Reply(ctx, b, msg, fmt.Sprintf("%s\n\n❌ Out of guesses. Answer was %s. /wordle_new to retry.", @@ -150,12 +150,12 @@ func (s *state) handleNew(ctx context.Context, b *bot.Bot, update *models.Update defer s.locks.Acquire(subject)() prelude := "" - prior, err := loadGame(ctx, s.kv, subject) + prior, err := loadGame(ctx, s.games, subject) if err != nil { return err } if prior != nil && !isFinished(prior) { - if _, err := recordResult(ctx, s.kv, subject, false, chathelper.NowMillis()); err != nil { + if _, err := recordResult(ctx, s.stats, subject, false, chathelper.NowMillis()); err != nil { return err } prelude = fmt.Sprintf("🏳️ Previous round abandoned (auto-giveup). Answer was %s.\n\n", @@ -191,10 +191,10 @@ func (s *state) handleGiveup(ctx context.Context, b *bot.Bot, update *models.Upd return chathelper.Reply(ctx, b, msg, fmt.Sprintf("Already gave up — %s.", strings.ToUpper(g.Target))) } g.Giveup = true - if err := saveGame(ctx, s.kv, subject, g); err != nil { + if err := saveGame(ctx, s.games, subject, g); err != nil { return err } - if _, err := recordResult(ctx, s.kv, subject, false, chathelper.NowMillis()); err != nil { + if _, err := recordResult(ctx, s.stats, subject, false, chathelper.NowMillis()); err != nil { return err } return chathelper.Reply(ctx, b, msg, fmt.Sprintf("🏳️ Answer was %s. /wordle_new for another.", strings.ToUpper(g.Target))) @@ -210,7 +210,7 @@ func (s *state) handleStats(ctx context.Context, b *bot.Bot, update *models.Upda if subject == "" { return chathelper.Reply(ctx, b, msg, "Cannot identify chat.") } - stats, err := loadStats(ctx, s.kv, subject) + st, err := loadStats(ctx, s.stats, subject) if err != nil { return err } @@ -220,6 +220,6 @@ func (s *state) handleStats(ctx context.Context, b *bot.Bot, update *models.Upda } return chathelper.Reply(ctx, b, msg, fmt.Sprintf( "📊 Wordle %s stats\nPlayed: %d\nWins: %d (%d%%)\nCurrent streak: %d\nBest streak: %d", - scope, stats.Played, stats.Wins, chathelper.WinRate(stats.Wins, stats.Played), stats.Streak, stats.BestStreak, + scope, st.Played, st.Wins, chathelper.WinRate(st.Wins, st.Played), st.Streak, st.BestStreak, )) } diff --git a/internal/modules/wordle/handlers_test.go b/internal/modules/wordle/handlers_test.go index b156a4a..9a4cd84 100644 --- a/internal/modules/wordle/handlers_test.go +++ b/internal/modules/wordle/handlers_test.go @@ -11,15 +11,15 @@ import ( ) // installWordle wires the wordle module to a recording bot with an -// in-memory KV. seedTarget pre-seeds a game with the supplied target +// in-memory store. seedTarget pre-seeds a game with the supplied target // (skip dictionary randomness for deterministic tests). Empty seedTarget -// leaves the KV blank so handlers spin up a fresh round on first call. -func installWordle(t *testing.T, ownerID int64, seedSubject, seedTarget string) (*testutil.RecordingBot, storage.KVStore) { +// leaves the store blank so handlers spin up a fresh round on first call. +func installWordle(t *testing.T, ownerID int64, seedSubject, seedTarget string) (*testutil.RecordingBot, GameStore) { t.Helper() rb := testutil.NewRecordingBot(t) - provider := storage.NewMemoryProvider() - kv := provider.For("wordle") - mod := New(modules.Deps{KV: kv}) + col := storage.NewMemoryProvider().Collection("wordle") + games := storage.Typed[GameState](col) + mod := New(modules.Deps{Store: col}) reg := &modules.Registry{ Modules: []modules.Module{{Name: "wordle", Commands: mod.Commands}}, AllCommands: map[string]modules.Command{}, @@ -31,11 +31,11 @@ func installWordle(t *testing.T, ownerID int64, seedSubject, seedTarget string) if seedTarget != "" { g := &GameState{Target: seedTarget, Guesses: []GuessRecord{}, StartedAt: 1} - if err := saveGame(context.Background(), kv, seedSubject, g); err != nil { + if err := saveGame(context.Background(), games, seedSubject, g); err != nil { t.Fatalf("seed game: %v", err) } } - return rb, kv + return rb, games } func TestWordle_NoArgShowsBoard(t *testing.T) { @@ -146,21 +146,24 @@ func TestWordleStats_AfterWin(t *testing.T) { // Group chats key by chat id, so two private users both writing /wordle // in the same group must mutate the same game. func TestWordle_GroupSubjectIsChatID(t *testing.T) { - rb, kv := installWordle(t, 0, "-100", "crane") + rb, games := installWordle(t, 0, "-100", "crane") rb.Bot.ProcessUpdate(context.Background(), testutil.NewGroupMessage(-100, 7, "/wordle crate")) // After one guess in chat -100, subject "-100" should have a guess // recorded; subject "7" should have no game. - var gChat GameState - if err := kv.GetJSON(context.Background(), gameKey("-100"), &gChat); err != nil { + gChat, err := loadGame(context.Background(), games, "-100") + if err != nil { t.Fatalf("expected game at subject=-100: %v", err) } - if len(gChat.Guesses) != 1 { - t.Errorf("group game has %d guesses, want 1", len(gChat.Guesses)) + if gChat == nil || len(gChat.Guesses) != 1 { + t.Errorf("group game guesses = %v, want 1", gChat) } - var gUser GameState - err := kv.GetJSON(context.Background(), gameKey("7"), &gUser) - if err == nil { + + gUser, err := loadGame(context.Background(), games, "7") + if err != nil { + t.Fatalf("loadGame for user subject: %v", err) + } + if gUser != nil { t.Errorf("user-keyed game leaked despite group context: %+v", gUser) } } diff --git a/internal/modules/wordle/state.go b/internal/modules/wordle/state.go index 80fbb7c..0e27b6d 100644 --- a/internal/modules/wordle/state.go +++ b/internal/modules/wordle/state.go @@ -11,7 +11,7 @@ import ( // MaxGuesses is the standard wordle round length. const MaxGuesses = 6 -// Note: the KV store has no native per-document TTL — saved games linger +// Note: the store has no native per-document TTL — saved games linger // until manually cleaned. Out of scope today; could be added via a sweep // cron if storage cost ever matters. @@ -19,43 +19,47 @@ const MaxGuesses = 6 // // { "word": "crane", "results": [{"letter":"c","result":"correct"}, ...] } type GuessRecord struct { - Word string `json:"word"` - Results []LetterScore `json:"results"` + Word string `json:"word" bson:"word"` + Results []LetterScore `json:"results" bson:"results"` } -// GameState is the per-subject KV record for an in-progress (or finished) -// round. +// GameState is the per-subject record for an in-progress (or finished) round. // // `giveup` is always emitted (initialized to false on /wordle_new). Do NOT // add omitempty — the field is part of the stored document's shape, so // emitting it unconditionally keeps already-saved games self-describing -// when inspected via raw KV dumps. +// when inspected via raw dumps. type GameState struct { - Target string `json:"target"` - Guesses []GuessRecord `json:"guesses"` - Solved bool `json:"solved"` - Giveup bool `json:"giveup"` - StartedAt int64 `json:"startedAt"` // ms-since-epoch (Date.now()) + Target string `json:"target" bson:"target"` + Guesses []GuessRecord `json:"guesses" bson:"guesses"` + Solved bool `json:"solved" bson:"solved"` + Giveup bool `json:"giveup" bson:"giveup"` + StartedAt int64 `json:"startedAt" bson:"startedAt"` // ms-since-epoch (Date.now()) } // Stats is the lifetime score record. lastResultAt is *int64 so an unplayed // account marshals as `"lastResultAt": null` — distinguishes "never played" // from "played at epoch zero". type Stats struct { - Played int `json:"played"` - Wins int `json:"wins"` - Streak int `json:"streak"` - BestStreak int `json:"bestStreak"` - LastResultAt *int64 `json:"lastResultAt"` // ms-since-epoch | null + Played int `json:"played" bson:"played"` + Wins int `json:"wins" bson:"wins"` + Streak int `json:"streak" bson:"streak"` + BestStreak int `json:"bestStreak" bson:"bestStreak"` + LastResultAt *int64 `json:"lastResultAt" bson:"lastResultAt"` // ms-since-epoch | null } +// GameStore is the wordle module's typed game store. +type GameStore = storage.DocStore[GameState] + +// StatsStore is the wordle module's typed stats store. +type StatsStore = storage.DocStore[Stats] + func gameKey(subject string) string { return "game:" + subject } func statsKey(subject string) string { return "stats:" + subject } // loadGame returns the active round, or (nil, nil) if none exists. -func loadGame(ctx context.Context, kv storage.KVStore, subject string) (*GameState, error) { - var g GameState - err := kv.GetJSON(ctx, gameKey(subject), &g) +func loadGame(ctx context.Context, games GameStore, subject string) (*GameState, error) { + g, _, err := games.Get(ctx, gameKey(subject)) switch { case err == nil: return &g, nil @@ -67,8 +71,8 @@ func loadGame(ctx context.Context, kv storage.KVStore, subject string) (*GameSta } // saveGame writes the round. -func saveGame(ctx context.Context, kv storage.KVStore, subject string, g *GameState) error { - if err := kv.PutJSON(ctx, gameKey(subject), g); err != nil { +func saveGame(ctx context.Context, games GameStore, subject string, g *GameState) error { + if err := games.Put(ctx, gameKey(subject), *g); err != nil { return fmt.Errorf("wordle saveGame: %w", err) } return nil @@ -76,9 +80,8 @@ func saveGame(ctx context.Context, kv storage.KVStore, subject string, g *GameSt // loadStats returns lifetime stats; missing → fresh-zero record (with // LastResultAt=nil) so callers never need a nil check. -func loadStats(ctx context.Context, kv storage.KVStore, subject string) (*Stats, error) { - var s Stats - err := kv.GetJSON(ctx, statsKey(subject), &s) +func loadStats(ctx context.Context, stats StatsStore, subject string) (*Stats, error) { + s, _, err := stats.Get(ctx, statsKey(subject)) switch { case err == nil: return &s, nil @@ -92,8 +95,8 @@ func loadStats(ctx context.Context, kv storage.KVStore, subject string) (*Stats, // recordResult bumps stats with the round outcome (won true → win + streak, // false → reset streak). Returns the updated stats so callers can show the // new streak in the win message. -func recordResult(ctx context.Context, kv storage.KVStore, subject string, won bool, nowMillis int64) (*Stats, error) { - s, err := loadStats(ctx, kv, subject) +func recordResult(ctx context.Context, stats StatsStore, subject string, won bool, nowMillis int64) (*Stats, error) { + s, err := loadStats(ctx, stats, subject) if err != nil { return nil, err } @@ -109,7 +112,7 @@ func recordResult(ctx context.Context, kv storage.KVStore, subject string, won b } now := nowMillis s.LastResultAt = &now - if err := kv.PutJSON(ctx, statsKey(subject), s); err != nil { + if err := stats.Put(ctx, statsKey(subject), *s); err != nil { return nil, fmt.Errorf("wordle recordResult: %w", err) } return s, nil diff --git a/internal/modules/wordle/state_test.go b/internal/modules/wordle/state_test.go index 4701be0..86f02a5 100644 --- a/internal/modules/wordle/state_test.go +++ b/internal/modules/wordle/state_test.go @@ -8,6 +8,21 @@ import ( "github.com/tiennm99/miti99bot/internal/storage" ) +func newWordleGames() GameStore { + return storage.Typed[GameState](storage.NewMemoryProvider().Collection("wordle")) +} + +func newWordleStats() StatsStore { + return storage.Typed[Stats](storage.NewMemoryProvider().Collection("wordle")) +} + +// newWordleStores returns a games + stats store backed by the same collection +// (disjoint key prefixes: "game:" vs "stats:"). +func newWordleStores() (GameStore, StatsStore) { + c := storage.NewMemoryProvider().Collection("wordle") + return storage.Typed[GameState](c), storage.Typed[Stats](c) +} + func TestStats_DefaultLastResultAtIsNull(t *testing.T) { // Go's *int64 must marshal as null when nil, so unplayed accounts emit // `"lastResultAt": null` and the field stays distinguishable from @@ -62,9 +77,9 @@ func TestGameState_JSONShapeIsStable(t *testing.T) { func TestRecordResult_WinIncrementsStreak(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + stats := newWordleStats() - s, err := recordResult(ctx, kv, "u1", true, 100) + s, err := recordResult(ctx, stats, "u1", true, 100) if err != nil { t.Fatalf("recordResult: %v", err) } @@ -76,7 +91,7 @@ func TestRecordResult_WinIncrementsStreak(t *testing.T) { } // Second win bumps streak; bestStreak follows. - s, _ = recordResult(ctx, kv, "u1", true, 200) + s, _ = recordResult(ctx, stats, "u1", true, 200) if s.Streak != 2 || s.BestStreak != 2 { t.Errorf("two wins: streak=%d best=%d", s.Streak, s.BestStreak) } @@ -84,11 +99,11 @@ func TestRecordResult_WinIncrementsStreak(t *testing.T) { func TestRecordResult_LossResetsStreakKeepsBest(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + stats := newWordleStats() - _, _ = recordResult(ctx, kv, "u1", true, 100) - _, _ = recordResult(ctx, kv, "u1", true, 200) - s, _ := recordResult(ctx, kv, "u1", false, 300) + _, _ = recordResult(ctx, stats, "u1", true, 100) + _, _ = recordResult(ctx, stats, "u1", true, 200) + s, _ := recordResult(ctx, stats, "u1", false, 300) if s.Streak != 0 { t.Errorf("loss should reset streak, got %d", s.Streak) } @@ -101,7 +116,7 @@ func TestRecordResult_LossResetsStreakKeepsBest(t *testing.T) { } func TestLoadGame_MissingReturnsNil(t *testing.T) { - g, err := loadGame(context.Background(), storage.NewMemoryKVStore(), "nobody") + g, err := loadGame(context.Background(), newWordleGames(), "nobody") if err != nil { t.Errorf("missing game should not error: %v", err) } @@ -112,12 +127,12 @@ func TestLoadGame_MissingReturnsNil(t *testing.T) { func TestSaveGame_RoundTrip(t *testing.T) { ctx := context.Background() - kv := storage.NewMemoryKVStore() + games := newWordleGames() want := &GameState{Target: "crane", Guesses: []GuessRecord{}, StartedAt: 42} - if err := saveGame(ctx, kv, "u1", want); err != nil { + if err := saveGame(ctx, games, "u1", want); err != nil { t.Fatal(err) } - got, err := loadGame(ctx, kv, "u1") + got, err := loadGame(ctx, games, "u1") if err != nil || got == nil { t.Fatalf("loadGame: got=%v err=%v", got, err) } diff --git a/internal/modules/wordle/wordle.go b/internal/modules/wordle/wordle.go index ca3fddc..fc020b0 100644 --- a/internal/modules/wordle/wordle.go +++ b/internal/modules/wordle/wordle.go @@ -2,13 +2,19 @@ package wordle import ( "github.com/tiennm99/miti99bot/internal/modules" + "github.com/tiennm99/miti99bot/internal/storage" ) // New is the wordle module Factory. Loads the embedded dictionary once, -// captures the per-module KV via closure, and registers all four commands. +// captures the per-module typed stores via closure, and registers all four commands. func New(deps modules.Deps) modules.Module { words, set := loadWords() - s := &state{kv: deps.KV, words: words, set: set} + s := &state{ + games: storage.Typed[GameState](deps.Store), + stats: storage.Typed[Stats](deps.Store), + words: words, + set: set, + } return modules.Module{ Commands: []modules.Command{ diff --git a/internal/storage/doc_store.go b/internal/storage/doc_store.go new file mode 100644 index 0000000..3fd7884 --- /dev/null +++ b/internal/storage/doc_store.go @@ -0,0 +1,156 @@ +package storage + +import ( + "context" + "errors" + "fmt" + "reflect" + "strings" +) + +// ErrNotFound is returned by a DocStore when a key has no value. +var ErrNotFound = errors.New("storage: key not found") + +// ErrConflict is returned by versioned writes when the stored version changed. +var ErrConflict = errors.New("storage: write conflict") + +// ErrInvalidModuleName is returned by every op on a store built from an invalid +// module/collection name — surfacing the configuration bug at first use rather +// than silently reading/writing an attacker-controllable collection name. +var ErrInvalidModuleName = errors.New("storage: invalid module name") + +// DocStore is the per-module, per-payload-type storage contract. A value is +// persisted as a native document (its fields hoisted to the document root) +// alongside an optimistic-locking version. Implementations are safe for +// concurrent use and return ErrNotFound for missing keys. +// +// Concurrency control is version-based: each key carries a monotonic version, so +// a writer swaps only if the version it read is unchanged. This decouples the +// concurrency control from the value encoding (values are native documents, not +// exact byte blobs). +// +// Contract: +// - Get returns the value and its current version, or ErrNotFound. +// - Put overwrites unconditionally and bumps the version. +// - PutVersioned writes only if the stored version equals expectedVersion, +// then bumps it. expectedVersion == 0 means "create (or adopt a not-yet- +// versioned key)". A mismatch returns ErrConflict. +type DocStore[T any] interface { + Get(ctx context.Context, id string) (val T, version int64, err error) + Put(ctx context.Context, id string, val T) error + PutVersioned(ctx context.Context, id string, expectedVersion int64, val T) error + Delete(ctx context.Context, id string) error + List(ctx context.Context, prefix string) ([]string, error) +} + +// Provider yields a per-module Collection handle. Implementations decide how +// isolation is achieved: MemoryProvider keeps one map per module, MongoProvider +// uses one MongoDB collection per module. Modules never construct stores +// directly — they receive a Collection through their factory's Deps and build +// typed views with Typed. +type Provider interface { + Collection(module string) Collection +} + +// Collection is an opaque per-module storage handle. Build a typed store over it +// with Typed[T]. A module may build several typed views over the same +// Collection (one per payload type) as long as their key prefixes are disjoint. +type Collection interface { + // isCollection seals the interface to this package's implementations so + // Typed's type switch is exhaustive. + isCollection() +} + +// Typed returns a DocStore[T] over c. Go methods cannot be generic, so this is a +// package-level function rather than a Provider method; it is the single switch +// point that binds a Collection to its backend's typed store. +// +// It panics if T is a struct whose BSON field names collide with a reserved root +// field (_id, version, updatedAt) — a programmer error caught at startup, in the +// same spirit as Prefixed panicking on an empty prefix. +func Typed[T any](c Collection) DocStore[T] { + if err := checkReservedFields[T](); err != nil { + panic(err) + } + switch h := c.(type) { + case mongoCollection: + return &mongoDocStore[T]{coll: h.coll, module: h.module} + case *memoryCollection: + return &memoryDocStore[T]{c: h} + case invalidCollection: + return invalidDocStore[T]{name: h.name} + default: + panic(fmt.Sprintf("storage: unknown collection type %T", c)) + } +} + +// reservedRootFields are the document-root field names the store owns; a payload +// type must not define BSON tags that collide with them (inline marshalling +// would otherwise produce duplicate keys). +var reservedRootFields = map[string]bool{ + mongoIDField: true, + mongoVersionField: true, + mongoUpdatedAtField: true, +} + +// checkReservedFields reports an error if struct type T declares a BSON field +// name colliding with a reserved root field. Non-struct T (e.g. memory-only +// test usage) is skipped — only the inline Mongo encoding needs the guard. +func checkReservedFields[T any]() error { + t := reflect.TypeOf((*T)(nil)).Elem() + if t.Kind() != reflect.Struct { + return nil + } + for i := 0; i < t.NumField(); i++ { + f := t.Field(i) + if !f.IsExported() { + continue + } + name := bsonFieldName(f) + if reservedRootFields[name] { + return fmt.Errorf("storage: payload type %s field %q maps to reserved root field %q", t.Name(), f.Name, name) + } + } + return nil +} + +// bsonFieldName returns the BSON document field name for a struct field: the +// explicit `bson:"name"` tag if present, else the Go field name (matching the +// driver's default of using the field name verbatim). +func bsonFieldName(f reflect.StructField) string { + tag := f.Tag.Get("bson") + if tag == "" { + return f.Name + } + name := strings.Split(tag, ",")[0] + if name == "" { + return f.Name + } + return name +} + +// invalidCollection is returned by a Provider for a module name that fails +// validation. Typed turns it into an invalidDocStore. +type invalidCollection struct{ name string } + +func (invalidCollection) isCollection() {} + +// invalidDocStore errors on every operation, wrapping ErrInvalidModuleName. +type invalidDocStore[T any] struct{ name string } + +func (s invalidDocStore[T]) err(op string) error { + return fmt.Errorf("%w: %q (op=%s)", ErrInvalidModuleName, s.name, op) +} + +func (s invalidDocStore[T]) Get(context.Context, string) (T, int64, error) { + var z T + return z, 0, s.err("Get") +} +func (s invalidDocStore[T]) Put(context.Context, string, T) error { return s.err("Put") } +func (s invalidDocStore[T]) PutVersioned(context.Context, string, int64, T) error { + return s.err("PutVersioned") +} +func (s invalidDocStore[T]) Delete(context.Context, string) error { return s.err("Delete") } +func (s invalidDocStore[T]) List(context.Context, string) ([]string, error) { + return nil, s.err("List") +} diff --git a/internal/storage/dynamodb_kv.go b/internal/storage/dynamodb_kv.go deleted file mode 100644 index cabbf0a..0000000 --- a/internal/storage/dynamodb_kv.go +++ /dev/null @@ -1,181 +0,0 @@ -package storage - -import ( - "context" - "encoding/json" - "fmt" - "strconv" - "time" - - "github.com/aws/aws-sdk-go-v2/aws" - "github.com/aws/aws-sdk-go-v2/service/dynamodb" - "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" -) - -// DynamoDB attribute names. `value` is reserved by DynamoDB expressions, so -// every reference goes through ExpressionAttributeNames (#v). -const ( - dynamoPKAttr = "pk" - dynamoSKAttr = "sk" - dynamoValueAttr = "value" - dynamoUpdatedAtAttr = "updatedAt" -) - -// DynamoDBKVStore is a KVStore backed by a single DynamoDB table with a -// composite key (pk, sk). FirestoreProvider uses one collection per module; -// DynamoDBProvider uses one partition (pk = moduleName) per module — the -// table itself is shared. The user-supplied key becomes the sort key, so -// List(prefix) maps to a Query with begins_with(sk, prefix). -type DynamoDBKVStore struct { - client *dynamodb.Client - table string - moduleName string -} - -// NewDynamoDBKVStore returns a store partition-scoped to moduleName. -// Callers must validate moduleName beforehand (DynamoDBProvider does). -func NewDynamoDBKVStore(client *dynamodb.Client, table, moduleName string) *DynamoDBKVStore { - return &DynamoDBKVStore{client: client, table: table, moduleName: moduleName} -} - -// Get returns the raw bytes stored at key, or ErrNotFound. Strong read for -// parity with the Firestore impl's strong-read default. -func (s *DynamoDBKVStore) Get(ctx context.Context, key string) ([]byte, error) { - if err := validateKey(key); err != nil { - return nil, err - } - out, err := s.client.GetItem(ctx, &dynamodb.GetItemInput{ - TableName: aws.String(s.table), - ConsistentRead: aws.Bool(true), - Key: map[string]types.AttributeValue{ - dynamoPKAttr: &types.AttributeValueMemberS{Value: s.moduleName}, - dynamoSKAttr: &types.AttributeValueMemberS{Value: key}, - }, - ExpressionAttributeNames: map[string]string{"#v": dynamoValueAttr}, - ProjectionExpression: aws.String("#v"), - }) - if err != nil { - return nil, fmt.Errorf("dynamodb get %s/%s: %w", s.moduleName, key, err) - } - if len(out.Item) == 0 { - return nil, ErrNotFound - } - rawAttr, ok := out.Item[dynamoValueAttr] - if !ok { - return nil, fmt.Errorf("dynamodb get %s/%s: missing %q attribute", s.moduleName, key, dynamoValueAttr) - } - str, ok := rawAttr.(*types.AttributeValueMemberS) - if !ok { - return nil, fmt.Errorf("dynamodb get %s/%s: unexpected attribute type %T", s.moduleName, key, rawAttr) - } - return []byte(str.Value), nil -} - -// GetJSON decodes the value at key into dst. -func (s *DynamoDBKVStore) GetJSON(ctx context.Context, key string, dst any) error { - raw, err := s.Get(ctx, key) - if err != nil { - return err - } - if err := json.Unmarshal(raw, dst); err != nil { - return fmt.Errorf("dynamodb get %s/%s: json decode: %w", s.moduleName, key, err) - } - return nil -} - -// Put writes raw bytes at key, creating or overwriting. The value attribute -// is stored as a DynamoDB String so it is human-readable in the AWS console. -// Every current caller writes JSON, which is UTF-8 safe; non-UTF-8 callers -// must encode upstream (e.g. base64). -func (s *DynamoDBKVStore) Put(ctx context.Context, key string, val []byte) error { - if err := validateKey(key); err != nil { - return err - } - _, err := s.client.PutItem(ctx, &dynamodb.PutItemInput{ - TableName: aws.String(s.table), - Item: map[string]types.AttributeValue{ - dynamoPKAttr: &types.AttributeValueMemberS{Value: s.moduleName}, - dynamoSKAttr: &types.AttributeValueMemberS{Value: key}, - dynamoValueAttr: &types.AttributeValueMemberS{Value: string(val)}, - dynamoUpdatedAtAttr: &types.AttributeValueMemberN{Value: strconv.FormatInt(time.Now().UTC().UnixNano(), 10)}, - }, - }) - if err != nil { - return fmt.Errorf("dynamodb put %s/%s: %w", s.moduleName, key, err) - } - return nil -} - -// Note: DynamoDB intentionally does NOT implement VersionedStore. Post-self-host -// it is reachable only via the data migrator (Scan + Put), which never needs -// optimistic locking. The live runtime backend is MongoDB; memory covers tests. - -// PutJSON marshals val and writes the bytes at key. -func (s *DynamoDBKVStore) PutJSON(ctx context.Context, key string, val any) error { - raw, err := json.Marshal(val) - if err != nil { - return fmt.Errorf("dynamodb put %s/%s: json encode: %w", s.moduleName, key, err) - } - return s.Put(ctx, key, raw) -} - -// Delete removes the item at key. Deleting a missing key is not an error. -func (s *DynamoDBKVStore) Delete(ctx context.Context, key string) error { - if err := validateKey(key); err != nil { - return err - } - _, err := s.client.DeleteItem(ctx, &dynamodb.DeleteItemInput{ - TableName: aws.String(s.table), - Key: map[string]types.AttributeValue{ - dynamoPKAttr: &types.AttributeValueMemberS{Value: s.moduleName}, - dynamoSKAttr: &types.AttributeValueMemberS{Value: key}, - }, - }) - if err != nil { - return fmt.Errorf("dynamodb delete %s/%s: %w", s.moduleName, key, err) - } - return nil -} - -// List returns all sort keys in the partition that start with prefix. Empty -// prefix scans the entire partition (Query is still cheap because it's -// partition-bounded; no Scan). -func (s *DynamoDBKVStore) List(ctx context.Context, prefix string) ([]string, error) { - if err := validatePrefix(prefix); err != nil { - return nil, err - } - input := &dynamodb.QueryInput{ - TableName: aws.String(s.table), - ProjectionExpression: aws.String(dynamoSKAttr), - ExpressionAttributeValues: map[string]types.AttributeValue{ - ":pk": &types.AttributeValueMemberS{Value: s.moduleName}, - }, - } - if prefix == "" { - input.KeyConditionExpression = aws.String("pk = :pk") - } else { - input.KeyConditionExpression = aws.String("pk = :pk AND begins_with(sk, :prefix)") - input.ExpressionAttributeValues[":prefix"] = &types.AttributeValueMemberS{Value: prefix} - } - - var keys []string - pager := dynamodb.NewQueryPaginator(s.client, input) - for pager.HasMorePages() { - page, err := pager.NextPage(ctx) - if err != nil { - return nil, fmt.Errorf("dynamodb list %s prefix=%q: %w", s.moduleName, prefix, err) - } - for _, item := range page.Items { - skAttr, ok := item[dynamoSKAttr] - if !ok { - continue - } - sk, ok := skAttr.(*types.AttributeValueMemberS) - if !ok { - continue - } - keys = append(keys, sk.Value) - } - } - return keys, nil -} diff --git a/internal/storage/dynamodb_kv_test.go b/internal/storage/dynamodb_kv_test.go deleted file mode 100644 index 2cf3c51..0000000 --- a/internal/storage/dynamodb_kv_test.go +++ /dev/null @@ -1,166 +0,0 @@ -package storage - -import ( - "context" - "errors" - "os" - "testing" - "time" - - "github.com/aws/aws-sdk-go-v2/aws" - "github.com/aws/aws-sdk-go-v2/service/dynamodb" - "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" -) - -// dynamoDBLocalSetup ensures a fresh table exists in DynamoDB Local for the -// test run. Returns a client + cleanup. Tests skip if DYNAMODB_LOCAL_URL is -// unset so CI without docker still builds. -func dynamoDBLocalSetup(t *testing.T) (client *dynamodb.Client, table string, cleanup func()) { - t.Helper() - endpoint := os.Getenv("DYNAMODB_LOCAL_URL") - if endpoint == "" { - t.Skip("DYNAMODB_LOCAL_URL not set; skipping DynamoDB integration test (run `make dynamodb-local` to start the local container)") - } - // DynamoDB Local accepts any non-empty creds + region. - t.Setenv("AWS_ACCESS_KEY_ID", "test") - t.Setenv("AWS_SECRET_ACCESS_KEY", "test") - t.Setenv("AWS_REGION", "ap-southeast-1") - - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cancel() - - c, err := NewDynamoDBClient(ctx, endpoint) - if err != nil { - t.Fatalf("NewDynamoDBClient: %v", err) - } - - table = "test-" + t.Name() - if len(table) > 64 { - table = table[:64] - } - - _, err = c.CreateTable(ctx, &dynamodb.CreateTableInput{ - TableName: aws.String(table), - BillingMode: types.BillingModePayPerRequest, - AttributeDefinitions: []types.AttributeDefinition{ - {AttributeName: aws.String("pk"), AttributeType: types.ScalarAttributeTypeS}, - {AttributeName: aws.String("sk"), AttributeType: types.ScalarAttributeTypeS}, - }, - KeySchema: []types.KeySchemaElement{ - {AttributeName: aws.String("pk"), KeyType: types.KeyTypeHash}, - {AttributeName: aws.String("sk"), KeyType: types.KeyTypeRange}, - }, - }) - if err != nil { - t.Fatalf("CreateTable: %v", err) - } - - cleanup = func() { - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - _, _ = c.DeleteTable(ctx, &dynamodb.DeleteTableInput{TableName: aws.String(table)}) - } - return c, table, cleanup -} - -func TestDynamoDBKVStore_PutGetDelete(t *testing.T) { - client, table, cleanup := dynamoDBLocalSetup(t) - defer cleanup() - - ctx := context.Background() - s := NewDynamoDBKVStore(client, table, "wordle") - - if err := s.Put(ctx, "user:1:state", []byte("hello")); err != nil { - t.Fatalf("Put: %v", err) - } - got, err := s.Get(ctx, "user:1:state") - if err != nil { - t.Fatalf("Get: %v", err) - } - if string(got) != "hello" { - t.Errorf("Get: got %q, want %q", got, "hello") - } - - if err := s.Delete(ctx, "user:1:state"); err != nil { - t.Fatalf("Delete: %v", err) - } - if _, err := s.Get(ctx, "user:1:state"); !errors.Is(err, ErrNotFound) { - t.Errorf("Get after Delete: got %v, want ErrNotFound", err) - } -} - -func TestDynamoDBKVStore_GetMissing(t *testing.T) { - client, table, cleanup := dynamoDBLocalSetup(t) - defer cleanup() - - s := NewDynamoDBKVStore(client, table, "wordle") - _, err := s.Get(context.Background(), "missing") - if !errors.Is(err, ErrNotFound) { - t.Errorf("got %v, want ErrNotFound", err) - } -} - -func TestDynamoDBKVStore_JSONRoundTrip(t *testing.T) { - client, table, cleanup := dynamoDBLocalSetup(t) - defer cleanup() - - ctx := context.Background() - s := NewDynamoDBKVStore(client, table, "loldle") - - type state struct { - Score int `json:"score"` - Name string `json:"name"` - } - in := state{Score: 42, Name: "ezreal"} - if err := s.PutJSON(ctx, "u1", in); err != nil { - t.Fatalf("PutJSON: %v", err) - } - var out state - if err := s.GetJSON(ctx, "u1", &out); err != nil { - t.Fatalf("GetJSON: %v", err) - } - if out != in { - t.Errorf("got %+v, want %+v", out, in) - } -} - -func TestDynamoDBKVStore_ListPrefix(t *testing.T) { - client, table, cleanup := dynamoDBLocalSetup(t) - defer cleanup() - - ctx := context.Background() - s := NewDynamoDBKVStore(client, table, "wordle") - - for _, k := range []string{"user:1:state", "user:2:state", "config:daily", "user:1:history"} { - if err := s.Put(ctx, k, []byte("x")); err != nil { - t.Fatalf("Put %s: %v", k, err) - } - } - - got, err := s.List(ctx, "user:") - if err != nil { - t.Fatalf("List: %v", err) - } - want := map[string]bool{"user:1:state": true, "user:2:state": true, "user:1:history": true} - if len(got) != len(want) { - t.Errorf("List: got %v (len=%d), want len=%d", got, len(got), len(want)) - } - for _, k := range got { - if !want[k] { - t.Errorf("List: unexpected key %q", k) - } - } -} - -// DynamoDB intentionally no longer implements optimistic locking — it is -// migrate-only post-self-host (Scan + Put). No CAS/versioned test here. - -func TestDynamoDBKVStore_DeleteMissingNoError(t *testing.T) { - client, table, cleanup := dynamoDBLocalSetup(t) - defer cleanup() - - s := NewDynamoDBKVStore(client, table, "wordle") - if err := s.Delete(context.Background(), "never-existed"); err != nil { - t.Errorf("Delete missing key: got %v, want nil (idempotent)", err) - } -} diff --git a/internal/storage/dynamodb_provider.go b/internal/storage/dynamodb_provider.go deleted file mode 100644 index 9460b85..0000000 --- a/internal/storage/dynamodb_provider.go +++ /dev/null @@ -1,30 +0,0 @@ -package storage - -import ( - "github.com/aws/aws-sdk-go-v2/service/dynamodb" -) - -// DynamoDBProvider is a KVProvider that hands out per-module stores backed by -// a single shared DynamoDB table. Module isolation is by partition (pk = -// moduleName); no key-prefix wrapping is needed at this layer. -type DynamoDBProvider struct { - client *dynamodb.Client - table string -} - -// NewDynamoDBProvider returns a provider over the given client + table. -// The client is goroutine-safe and outlives every store handed out by For. -func NewDynamoDBProvider(client *dynamodb.Client, table string) *DynamoDBProvider { - return &DynamoDBProvider{client: client, table: table} -} - -// For returns a DynamoDBKVStore partitioned by moduleName. moduleName is -// re-validated against the same alphabet enforced by FirestoreProvider so -// behaviour is identical between backends. An invalid name yields an -// invalidStore — every op errors at first use, surfacing the bug early. -func (p *DynamoDBProvider) For(moduleName string) KVStore { - if !collectionNameRe.MatchString(moduleName) { - return invalidStore{name: moduleName} - } - return NewDynamoDBKVStore(p.client, p.table, moduleName) -} diff --git a/internal/storage/dynamodb_provider_test.go b/internal/storage/dynamodb_provider_test.go deleted file mode 100644 index 7ec679d..0000000 --- a/internal/storage/dynamodb_provider_test.go +++ /dev/null @@ -1,61 +0,0 @@ -package storage - -import ( - "context" - "errors" - "testing" -) - -func TestDynamoDBProvider_CrossModuleIsolation(t *testing.T) { - client, table, cleanup := dynamoDBLocalSetup(t) - defer cleanup() - - ctx := context.Background() - p := NewDynamoDBProvider(client, table) - - wordle := p.For("wordle") - loldle := p.For("loldle") - - if err := wordle.Put(ctx, "k", []byte("from-wordle")); err != nil { - t.Fatalf("wordle.Put: %v", err) - } - if err := loldle.Put(ctx, "k", []byte("from-loldle")); err != nil { - t.Fatalf("loldle.Put: %v", err) - } - - got, err := wordle.Get(ctx, "k") - if err != nil { - t.Fatalf("wordle.Get: %v", err) - } - if string(got) != "from-wordle" { - t.Errorf("wordle.Get: got %q, want from-wordle", got) - } - - got, err = loldle.Get(ctx, "k") - if err != nil { - t.Fatalf("loldle.Get: %v", err) - } - if string(got) != "from-loldle" { - t.Errorf("loldle.Get: got %q, want from-loldle", got) - } - - // List should not leak across modules. - wkeys, err := wordle.List(ctx, "") - if err != nil { - t.Fatalf("wordle.List: %v", err) - } - if len(wkeys) != 1 || wkeys[0] != "k" { - t.Errorf("wordle.List: got %v, want [k]", wkeys) - } -} - -func TestDynamoDBProvider_InvalidModuleName(t *testing.T) { - // No DynamoDB needed — invalidStore short-circuits before any SDK call. - p := NewDynamoDBProvider(nil, "any") - store := p.For("../../etc/passwd") - - _, err := store.Get(context.Background(), "k") - if !errors.Is(err, ErrInvalidModuleName) { - t.Errorf("Get: got %v, want ErrInvalidModuleName", err) - } -} diff --git a/internal/storage/invalid_store.go b/internal/storage/invalid_store.go deleted file mode 100644 index 785d9de..0000000 --- a/internal/storage/invalid_store.go +++ /dev/null @@ -1,35 +0,0 @@ -package storage - -import ( - "context" - "fmt" -) - -// ErrInvalidModuleName is returned by every operation on an invalidStore — -// the sentinel emitted when FirestoreProvider.For is asked for a module -// whose name fails collectionNameRe. -var ErrInvalidModuleName = fmt.Errorf("storage: invalid module name") - -// invalidStore is a KVStore that errors on every call. Returned by -// FirestoreProvider.For when the requested module name doesn't validate. -// Callers see a real KVStore but every op errors at use, surfacing the -// configuration bug at the first read/write rather than silently writing -// to an attacker-controllable collection name. -type invalidStore struct { - name string -} - -func (s invalidStore) wrap(op string) error { - return fmt.Errorf("%w: %q (op=%s)", ErrInvalidModuleName, s.name, op) -} - -func (s invalidStore) Get(_ context.Context, _ string) ([]byte, error) { - return nil, s.wrap("Get") -} -func (s invalidStore) GetJSON(_ context.Context, _ string, _ any) error { return s.wrap("GetJSON") } -func (s invalidStore) Put(_ context.Context, _ string, _ []byte) error { return s.wrap("Put") } -func (s invalidStore) PutJSON(_ context.Context, _ string, _ any) error { return s.wrap("PutJSON") } -func (s invalidStore) Delete(_ context.Context, _ string) error { return s.wrap("Delete") } -func (s invalidStore) List(_ context.Context, _ string) ([]string, error) { - return nil, s.wrap("List") -} diff --git a/internal/storage/kv_provider.go b/internal/storage/kv_provider.go deleted file mode 100644 index 21888df..0000000 --- a/internal/storage/kv_provider.go +++ /dev/null @@ -1,35 +0,0 @@ -package storage - -// KVProvider yields a per-module KVStore. Implementations decide how isolation -// is achieved: MemoryProvider wraps a single in-process store with a key -// prefix, FirestoreProvider uses one collection per module. -// -// Modules never construct KVStores directly — they receive one through their -// factory's Deps. This keeps the storage backend swappable without touching -// any module code. -type KVProvider interface { - For(moduleName string) KVStore -} - -// MemoryProvider is a KVProvider backed by a single in-process MemoryKVStore. -// Each module sees a Prefixed view that prevents cross-module key collisions. -// -// Intended for tests, local smoke runs, and the no-Firestore fallback. State -// is lost when the process exits. -type MemoryProvider struct { - base *MemoryKVStore -} - -// NewMemoryProvider returns a fresh in-process provider. -func NewMemoryProvider() *MemoryProvider { - return &MemoryProvider{base: NewMemoryKVStore()} -} - -// For returns a per-module view of the shared in-memory store. -func (m *MemoryProvider) For(moduleName string) KVStore { - return Prefixed(m.base, moduleName) -} - -// Base exposes the underlying unprefixed store. Tests use this to assert -// cross-module isolation by reading raw keys; production code must not. -func (m *MemoryProvider) Base() *MemoryKVStore { return m.base } diff --git a/internal/storage/kv_store.go b/internal/storage/kv_store.go deleted file mode 100644 index cca7777..0000000 --- a/internal/storage/kv_store.go +++ /dev/null @@ -1,41 +0,0 @@ -package storage - -import ( - "context" - "errors" -) - -// ErrNotFound is returned by KVStore implementations when a key has no value. -var ErrNotFound = errors.New("storage: key not found") - -// ErrConflict is returned by conditional writes when the stored value changed. -var ErrConflict = errors.New("storage: write conflict") - -// KVStore is the per-module key-value contract. Implementations must be safe -// for concurrent use and must return ErrNotFound for missing keys. -type KVStore interface { - Get(ctx context.Context, key string) ([]byte, error) - GetJSON(ctx context.Context, key string, dst any) error - Put(ctx context.Context, key string, val []byte) error - PutJSON(ctx context.Context, key string, val any) error - Delete(ctx context.Context, key string) error - List(ctx context.Context, prefix string) ([]string, error) -} - -// VersionedStore is implemented by stores that support version-based optimistic -// locking. It replaces value-bytes compare-and-swap: each key carries a -// monotonic version, so a writer swaps only if the version it read is unchanged. -// This decouples concurrency control from the value encoding (so values can be -// stored as native documents rather than exact byte blobs). -// -// Contract: -// - GetVersioned returns the value and its current version, or ErrNotFound. -// A key that exists without a recorded version (written by an older build) -// reports version 0 and no error. -// - PutVersioned writes val only if the stored version still equals -// expectedVersion, then bumps the version. expectedVersion == 0 means -// "create, or adopt a not-yet-versioned key". A mismatch returns ErrConflict. -type VersionedStore interface { - GetVersioned(ctx context.Context, key string) (val []byte, version int64, err error) - PutVersioned(ctx context.Context, key string, expectedVersion int64, val []byte) error -} diff --git a/internal/storage/memory_doc_store_test.go b/internal/storage/memory_doc_store_test.go new file mode 100644 index 0000000..f516c3f --- /dev/null +++ b/internal/storage/memory_doc_store_test.go @@ -0,0 +1,131 @@ +package storage + +import ( + "context" + "errors" + "testing" +) + +type testPayload struct { + Name string `json:"name" bson:"name"` + Count int `json:"count" bson:"count"` +} + +func memStore(module string) DocStore[testPayload] { + return Typed[testPayload](NewMemoryProvider().Collection(module)) +} + +func TestMemoryDocStore_PutGetRoundTrip(t *testing.T) { + ctx := context.Background() + s := memStore("coin") + if err := s.Put(ctx, "k", testPayload{Name: "a", Count: 3}); err != nil { + t.Fatalf("Put: %v", err) + } + got, version, err := s.Get(ctx, "k") + if err != nil { + t.Fatalf("Get: %v", err) + } + if got.Name != "a" || got.Count != 3 { + t.Fatalf("got %+v", got) + } + if version != 1 { + t.Fatalf("version = %d, want 1", version) + } +} + +func TestMemoryDocStore_GetMissing(t *testing.T) { + if _, _, err := memStore("coin").Get(context.Background(), "nope"); !errors.Is(err, ErrNotFound) { + t.Fatalf("Get missing = %v, want ErrNotFound", err) + } +} + +func TestMemoryDocStore_PutBumpsVersion(t *testing.T) { + ctx := context.Background() + s := memStore("coin") + _ = s.Put(ctx, "k", testPayload{Count: 1}) + _ = s.Put(ctx, "k", testPayload{Count: 2}) + _, version, _ := s.Get(ctx, "k") + if version != 2 { + t.Fatalf("version = %d, want 2", version) + } +} + +func TestMemoryDocStore_PutVersionedCAS(t *testing.T) { + ctx := context.Background() + s := memStore("coin") + // create + if err := s.PutVersioned(ctx, "k", 0, testPayload{Count: 1}); err != nil { + t.Fatalf("create: %v", err) + } + // second create must conflict + if err := s.PutVersioned(ctx, "k", 0, testPayload{Count: 9}); !errors.Is(err, ErrConflict) { + t.Fatalf("double create = %v, want ErrConflict", err) + } + // stale expected version conflicts + if err := s.PutVersioned(ctx, "k", 99, testPayload{Count: 9}); !errors.Is(err, ErrConflict) { + t.Fatalf("stale CAS = %v, want ErrConflict", err) + } + // fresh version succeeds + if err := s.PutVersioned(ctx, "k", 1, testPayload{Count: 2}); err != nil { + t.Fatalf("fresh CAS: %v", err) + } +} + +func TestMemoryDocStore_DeleteIdempotent(t *testing.T) { + ctx := context.Background() + s := memStore("coin") + _ = s.Put(ctx, "k", testPayload{}) + if err := s.Delete(ctx, "k"); err != nil { + t.Fatalf("Delete: %v", err) + } + if err := s.Delete(ctx, "k"); err != nil { + t.Fatalf("Delete missing must be nil, got %v", err) + } +} + +func TestMemoryDocStore_ListPrefix(t *testing.T) { + ctx := context.Background() + s := memStore("coin") + _ = s.Put(ctx, "game:1", testPayload{}) + _ = s.Put(ctx, "game:2", testPayload{}) + _ = s.Put(ctx, "stats:1", testPayload{}) + keys, err := s.List(ctx, "game:") + if err != nil { + t.Fatalf("List: %v", err) + } + if len(keys) != 2 || keys[0] != "game:1" || keys[1] != "game:2" { + t.Fatalf("List game: = %v", keys) + } +} + +func TestMemoryDocStore_ValueIsolation(t *testing.T) { + ctx := context.Background() + s := Typed[testPayloadWithSlice](NewMemoryProvider().Collection("coin")) + in := testPayloadWithSlice{Items: []string{"x"}} + _ = s.Put(ctx, "k", in) + in.Items[0] = "mutated" // mutating the caller copy must not affect stored state + got, _, _ := s.Get(ctx, "k") + if got.Items[0] != "x" { + t.Fatalf("stored value aliased caller slice: %v", got.Items) + } +} + +type testPayloadWithSlice struct { + Items []string `json:"items" bson:"items"` +} + +func TestCheckReservedFields(t *testing.T) { + type bad struct { + Version int `bson:"version"` + } + if err := checkReservedFields[bad](); err == nil { + t.Fatal("expected reserved-field collision error for bson:\"version\"") + } + if err := checkReservedFields[testPayload](); err != nil { + t.Fatalf("clean struct flagged: %v", err) + } + // Non-struct payloads are skipped, not rejected. + if err := checkReservedFields[string](); err != nil { + t.Fatalf("string payload flagged: %v", err) + } +} diff --git a/internal/storage/memory_kv.go b/internal/storage/memory_kv.go deleted file mode 100644 index dcdf1cf..0000000 --- a/internal/storage/memory_kv.go +++ /dev/null @@ -1,119 +0,0 @@ -package storage - -import ( - "bytes" - "context" - "encoding/json" - "sort" - "strings" - "sync" -) - -// memEntry is one stored value plus its optimistic-locking version. -type memEntry struct { - val []byte - version int64 -} - -// MemoryKVStore is an in-process KVStore for tests and local smoke runs. -// Data is lost on restart; production uses the MongoDB provider. -type MemoryKVStore struct { - mu sync.RWMutex - m map[string]memEntry -} - -// NewMemoryKVStore returns an empty in-memory store. -func NewMemoryKVStore() *MemoryKVStore { - return &MemoryKVStore{m: make(map[string]memEntry)} -} - -func (s *MemoryKVStore) Get(_ context.Context, key string) ([]byte, error) { - s.mu.RLock() - defer s.mu.RUnlock() - e, ok := s.m[key] - if !ok { - return nil, ErrNotFound - } - out := make([]byte, len(e.val)) - copy(out, e.val) - return out, nil -} - -func (s *MemoryKVStore) GetJSON(ctx context.Context, key string, dst any) error { - raw, err := s.Get(ctx, key) - if err != nil { - return err - } - return json.NewDecoder(bytes.NewReader(raw)).Decode(dst) -} - -func (s *MemoryKVStore) Put(_ context.Context, key string, val []byte) error { - s.mu.Lock() - defer s.mu.Unlock() - stored := make([]byte, len(val)) - copy(stored, val) - // A plain Put bumps the version too, so a concurrent versioned writer that - // read the old version correctly sees a conflict. - s.m[key] = memEntry{val: stored, version: s.m[key].version + 1} - return nil -} - -// GetVersioned returns the value and its version, or ErrNotFound. -func (s *MemoryKVStore) GetVersioned(_ context.Context, key string) ([]byte, int64, error) { - s.mu.RLock() - defer s.mu.RUnlock() - e, ok := s.m[key] - if !ok { - return nil, 0, ErrNotFound - } - out := make([]byte, len(e.val)) - copy(out, e.val) - return out, e.version, nil -} - -// PutVersioned writes val only if the stored version equals expectedVersion -// (0 = must not exist yet), then bumps the version. ErrConflict on mismatch. -func (s *MemoryKVStore) PutVersioned(_ context.Context, key string, expectedVersion int64, val []byte) error { - s.mu.Lock() - defer s.mu.Unlock() - e, ok := s.m[key] - if expectedVersion == 0 { - if ok { - return ErrConflict - } - } else if !ok || e.version != expectedVersion { - return ErrConflict - } - stored := make([]byte, len(val)) - copy(stored, val) - s.m[key] = memEntry{val: stored, version: e.version + 1} - return nil -} - -func (s *MemoryKVStore) PutJSON(ctx context.Context, key string, val any) error { - raw, err := json.Marshal(val) - if err != nil { - return err - } - return s.Put(ctx, key, raw) -} - -func (s *MemoryKVStore) Delete(_ context.Context, key string) error { - s.mu.Lock() - defer s.mu.Unlock() - delete(s.m, key) - return nil -} - -func (s *MemoryKVStore) List(_ context.Context, prefix string) ([]string, error) { - s.mu.RLock() - defer s.mu.RUnlock() - keys := make([]string, 0) - for k := range s.m { - if strings.HasPrefix(k, prefix) { - keys = append(keys, k) - } - } - sort.Strings(keys) - return keys, nil -} diff --git a/internal/storage/memory_kv_test.go b/internal/storage/memory_kv_test.go deleted file mode 100644 index d38b45d..0000000 --- a/internal/storage/memory_kv_test.go +++ /dev/null @@ -1,58 +0,0 @@ -package storage - -import ( - "context" - "errors" - "testing" -) - -func TestMemoryKVStore_Versioned(t *testing.T) { - ctx := context.Background() - s := NewMemoryKVStore() - - // Absent key reports ErrNotFound + version 0. - if _, v, err := s.GetVersioned(ctx, "k"); !errors.Is(err, ErrNotFound) || v != 0 { - t.Fatalf("GetVersioned absent: got (v=%d, %v), want (0, ErrNotFound)", v, err) - } - - // Create when absent (expectedVersion 0). - if err := s.PutVersioned(ctx, "k", 0, []byte("v1")); err != nil { - t.Fatalf("PutVersioned create: %v", err) - } - val, ver, err := s.GetVersioned(ctx, "k") - if err != nil || string(val) != "v1" || ver != 1 { - t.Fatalf("after create: got (%q, v=%d, %v), want (v1, 1, nil)", val, ver, err) - } - - // Create over existing conflicts. - if err := s.PutVersioned(ctx, "k", 0, []byte("v2")); !errors.Is(err, ErrConflict) { - t.Errorf("create over existing: got %v, want ErrConflict", err) - } - - // Swap with matching version succeeds and bumps version. - if err := s.PutVersioned(ctx, "k", 1, []byte("v2")); err != nil { - t.Fatalf("swap matching version: %v", err) - } - if _, ver, _ := s.GetVersioned(ctx, "k"); ver != 2 { - t.Errorf("version after swap = %d, want 2", ver) - } - - // Stale version conflicts. - if err := s.PutVersioned(ctx, "k", 1, []byte("v3")); !errors.Is(err, ErrConflict) { - t.Errorf("stale version: got %v, want ErrConflict", err) - } - - // Swap on a missing key conflicts. - if err := s.PutVersioned(ctx, "missing", 1, []byte("x")); !errors.Is(err, ErrConflict) { - t.Errorf("swap missing key: got %v, want ErrConflict", err) - } - - // A plain Put bumps the version, so a versioned writer holding the old - // version sees a conflict. - if err := s.Put(ctx, "k", []byte("v4")); err != nil { - t.Fatalf("Put: %v", err) - } - if err := s.PutVersioned(ctx, "k", 2, []byte("v5")); !errors.Is(err, ErrConflict) { - t.Errorf("versioned write after plain Put: got %v, want ErrConflict", err) - } -} diff --git a/internal/storage/memory_provider.go b/internal/storage/memory_provider.go new file mode 100644 index 0000000..a3f3d41 --- /dev/null +++ b/internal/storage/memory_provider.go @@ -0,0 +1,142 @@ +package storage + +import ( + "context" + "encoding/json" + "sort" + "strings" + "sync" +) + +// MemoryProvider is a Provider backed by per-module in-process maps. Intended +// for tests and local no-database runs (MODULES= / no MONGO_URL); state is lost +// when the process exits. Production uses MongoProvider. +type MemoryProvider struct { + mu sync.Mutex + cols map[string]*memoryCollection +} + +// NewMemoryProvider returns a fresh in-process provider. +func NewMemoryProvider() *MemoryProvider { + return &MemoryProvider{cols: make(map[string]*memoryCollection)} +} + +// Collection returns the per-module in-memory collection, creating it on first +// use. An invalid module name yields an invalidCollection so Typed produces a +// store whose every op errors — mirroring MongoProvider's defense in depth. +func (p *MemoryProvider) Collection(module string) Collection { + if !collectionNameRe.MatchString(module) { + return invalidCollection{name: module} + } + p.mu.Lock() + defer p.mu.Unlock() + c, ok := p.cols[module] + if !ok { + c = &memoryCollection{rows: make(map[string]memoryRow)} + p.cols[module] = c + } + return c +} + +// memoryCollection holds one module's rows. Multiple typed views (different +// payload types under disjoint key prefixes) share the same map. +type memoryCollection struct { + mu sync.RWMutex + rows map[string]memoryRow +} + +func (*memoryCollection) isCollection() {} + +// memoryRow stores the JSON-encoded value plus its version. Encoding to bytes +// isolates stored state from caller mutation, matching production where Mongo +// serializes the value on write. +type memoryRow struct { + version int64 + data []byte +} + +// memoryDocStore is a typed DocStore over one memoryCollection. +type memoryDocStore[T any] struct { + c *memoryCollection +} + +func (s *memoryDocStore[T]) Get(_ context.Context, id string) (T, int64, error) { + var out T + if err := validateKey(id); err != nil { + return out, 0, err + } + s.c.mu.RLock() + defer s.c.mu.RUnlock() + row, ok := s.c.rows[id] + if !ok { + return out, 0, ErrNotFound + } + if err := json.Unmarshal(row.data, &out); err != nil { + return out, 0, err + } + return out, row.version, nil +} + +func (s *memoryDocStore[T]) Put(_ context.Context, id string, val T) error { + if err := validateKey(id); err != nil { + return err + } + data, err := json.Marshal(val) + if err != nil { + return err + } + s.c.mu.Lock() + defer s.c.mu.Unlock() + // A plain Put bumps the version too, so a concurrent versioned writer that + // read the old version correctly sees a conflict. + s.c.rows[id] = memoryRow{version: s.c.rows[id].version + 1, data: data} + return nil +} + +func (s *memoryDocStore[T]) PutVersioned(_ context.Context, id string, expectedVersion int64, val T) error { + if err := validateKey(id); err != nil { + return err + } + data, err := json.Marshal(val) + if err != nil { + return err + } + s.c.mu.Lock() + defer s.c.mu.Unlock() + row, ok := s.c.rows[id] + if expectedVersion == 0 { + if ok { + return ErrConflict + } + } else if !ok || row.version != expectedVersion { + return ErrConflict + } + s.c.rows[id] = memoryRow{version: row.version + 1, data: data} + return nil +} + +func (s *memoryDocStore[T]) Delete(_ context.Context, id string) error { + if err := validateKey(id); err != nil { + return err + } + s.c.mu.Lock() + defer s.c.mu.Unlock() + delete(s.c.rows, id) + return nil +} + +func (s *memoryDocStore[T]) List(_ context.Context, prefix string) ([]string, error) { + if err := validatePrefix(prefix); err != nil { + return nil, err + } + s.c.mu.RLock() + defer s.c.mu.RUnlock() + keys := make([]string, 0) + for k := range s.c.rows { + if strings.HasPrefix(k, prefix) { + keys = append(keys, k) + } + } + sort.Strings(keys) + return keys, nil +} diff --git a/internal/storage/mongo_doc_store.go b/internal/storage/mongo_doc_store.go new file mode 100644 index 0000000..d79495f --- /dev/null +++ b/internal/storage/mongo_doc_store.go @@ -0,0 +1,185 @@ +package storage + +import ( + "context" + "errors" + "fmt" + "time" + + "go.mongodb.org/mongo-driver/v2/bson" + "go.mongodb.org/mongo-driver/v2/mongo" + "go.mongodb.org/mongo-driver/v2/mongo/options" +) + +// Reserved document-root field names owned by the store. The caller's payload +// fields are hoisted to the root alongside these. `_id` holds the key. +const ( + mongoIDField = "_id" + mongoVersionField = "version" + mongoUpdatedAtField = "updatedAt" +) + +// mongoPutAttempts bounds the get→put-versioned retry loop a plain Put uses to +// overwrite unconditionally while still bumping the version. +const mongoPutAttempts = 5 + +// mongoCollection is a Collection backed by a single MongoDB collection (one per +// module — collection-per-module IS the isolation, so no key prefix is needed). +type mongoCollection struct { + coll *mongo.Collection + module string +} + +func (mongoCollection) isCollection() {} + +// storedDoc is the on-disk document shape. The payload's fields are hoisted to +// the document root via bson inline, so the value renders as an expandable, +// queryable native document in Compass — never wrapped in a `value` envelope. +// Reserved metadata (version, updatedAt) sit alongside the payload fields. +type storedDoc[T any] struct { + ID string `bson:"_id"` + Version int64 `bson:"version"` + UpdatedAt time.Time `bson:"updatedAt"` + Payload T `bson:",inline"` +} + +// mongoDocStore is a typed DocStore over one collection. +type mongoDocStore[T any] struct { + coll *mongo.Collection + module string +} + +func (s *mongoDocStore[T]) Get(ctx context.Context, id string) (T, int64, error) { + var out storedDoc[T] + if err := validateKey(id); err != nil { + return out.Payload, 0, err + } + err := s.coll.FindOne(ctx, bson.M{mongoIDField: id}).Decode(&out) + if err != nil { + if errors.Is(err, mongo.ErrNoDocuments) { + return out.Payload, 0, ErrNotFound + } + return out.Payload, 0, fmt.Errorf("mongo get %s/%s: %w", s.module, id, err) + } + return out.Payload, out.Version, nil +} + +// replacement builds the whole document to store. Using a full replacement +// (never $set) guarantees stale fields from a previous value cannot survive an +// overwrite. +func (s *mongoDocStore[T]) replacement(id string, version int64, val T) storedDoc[T] { + return storedDoc[T]{ID: id, Version: version, UpdatedAt: time.Now().UTC(), Payload: val} +} + +// PutVersioned writes val only if the stored version equals expectedVersion, +// then bumps it. +// +// - expectedVersion == 0 → upsert matching {_id} with version absent or 0; a +// key that already has version ≥ 1 fails the filter, the upsert attempts an +// insert on the same _id, and the unique-_id index returns duplicate-key → +// ErrConflict (linearizable single-winner for the first write). +// - expectedVersion > 0 → ReplaceOne on {_id, version:expected}; MatchedCount +// of 0 means the version moved → ErrConflict. +func (s *mongoDocStore[T]) PutVersioned(ctx context.Context, id string, expectedVersion int64, val T) error { + if err := validateKey(id); err != nil { + return err + } + if expectedVersion == 0 { + _, err := s.coll.ReplaceOne(ctx, + bson.M{ + mongoIDField: id, + "$or": bson.A{ + bson.M{mongoVersionField: bson.M{"$exists": false}}, + bson.M{mongoVersionField: int64(0)}, + }, + }, + s.replacement(id, 1, val), + options.Replace().SetUpsert(true), + ) + if err != nil { + if mongo.IsDuplicateKeyError(err) { + return ErrConflict + } + return fmt.Errorf("mongo put-versioned %s/%s: %w", s.module, id, err) + } + return nil + } + res, err := s.coll.ReplaceOne(ctx, + bson.M{mongoIDField: id, mongoVersionField: expectedVersion}, + s.replacement(id, expectedVersion+1, val), + ) + if err != nil { + return fmt.Errorf("mongo put-versioned %s/%s: %w", s.module, id, err) + } + if res.MatchedCount == 0 { + return ErrConflict + } + return nil +} + +// Put overwrites unconditionally and bumps the version, implemented as a bounded +// get→put-versioned loop so it shares the whole-document replacement (stale-field +// removal) and version semantics with PutVersioned. +func (s *mongoDocStore[T]) Put(ctx context.Context, id string, val T) error { + if err := validateKey(id); err != nil { + return err + } + for attempt := 0; attempt < mongoPutAttempts; attempt++ { + _, version, err := s.Get(ctx, id) + switch { + case errors.Is(err, ErrNotFound): + version = 0 + case err != nil: + return err + } + err = s.PutVersioned(ctx, id, version, val) + if errors.Is(err, ErrConflict) { + continue // a concurrent writer moved the version; re-read and retry. + } + return err + } + return fmt.Errorf("mongo put %s/%s: %w after %d attempts", s.module, id, ErrConflict, mongoPutAttempts) +} + +func (s *mongoDocStore[T]) Delete(ctx context.Context, id string) error { + if err := validateKey(id); err != nil { + return err + } + if _, err := s.coll.DeleteOne(ctx, bson.M{mongoIDField: id}); err != nil { + return fmt.Errorf("mongo delete %s/%s: %w", s.module, id, err) + } + return nil +} + +// List returns all document IDs whose key starts with prefix, via a half-open +// range scan on _id (reusing prefixSuccessor) so it uses the _id index and +// avoids regex injection. Empty prefix returns the whole collection. +func (s *mongoDocStore[T]) List(ctx context.Context, prefix string) ([]string, error) { + if err := validatePrefix(prefix); err != nil { + return nil, err + } + filter := bson.M{} + if prefix != "" { + filter[mongoIDField] = bson.M{"$gte": prefix, "$lt": prefixSuccessor(prefix)} + } + cur, err := s.coll.Find(ctx, filter, options.Find().SetProjection(bson.M{mongoIDField: 1})) + if err != nil { + return nil, fmt.Errorf("mongo list %s prefix=%q: %w", s.module, prefix, err) + } + defer func() { _ = cur.Close(ctx) }() + + var keys []string + for cur.Next(ctx) { + var doc struct { + ID string `bson:"_id"` + } + if err := cur.Decode(&doc); err != nil { + return nil, fmt.Errorf("mongo list %s prefix=%q: decode: %w", s.module, prefix, err) + } + keys = append(keys, doc.ID) + } + if err := cur.Err(); err != nil { + return nil, fmt.Errorf("mongo list %s prefix=%q: %w", s.module, prefix, err) + } + return keys, nil +} diff --git a/internal/storage/mongo_doc_store_test.go b/internal/storage/mongo_doc_store_test.go new file mode 100644 index 0000000..128b90c --- /dev/null +++ b/internal/storage/mongo_doc_store_test.go @@ -0,0 +1,196 @@ +package storage + +import ( + "context" + "errors" + "fmt" + "os" + "sync" + "testing" + "time" + + "go.mongodb.org/mongo-driver/v2/bson" + "go.mongodb.org/mongo-driver/v2/mongo" +) + +// mongoLocalSetup connects to a local MongoDB and returns a fresh, uniquely +// named database plus cleanup. Tests skip if MONGODB_TEST_URL is unset so CI +// without a Mongo container still builds (mirrors the DynamoDB Local gating). +func mongoLocalSetup(t *testing.T) (*mongo.Database, func()) { + t.Helper() + uri := os.Getenv("MONGODB_TEST_URL") + if uri == "" { + t.Skip("MONGODB_TEST_URL not set; skipping MongoDB integration test (run `make mongo-local` to start the local container)") + } + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + client, err := NewMongoClient(ctx, uri) + if err != nil { + t.Fatalf("NewMongoClient: %v", err) + } + dbName := fmt.Sprintf("miti99bot_test_%d", time.Now().UnixNano()) + if len(dbName) > 63 { + dbName = dbName[:63] + } + db := client.Database(dbName) + + cleanup := func() { + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + _ = db.Drop(ctx) + _ = client.Disconnect(ctx) + } + return db, cleanup +} + +type portfolioLike struct { + USD float64 `json:"usd" bson:"usd"` + Assets map[string]float64 `json:"assets" bson:"assets"` +} + +func mongoStore[T any](t *testing.T, module string) (DocStore[T], *mongo.Collection, func()) { + t.Helper() + db, cleanup := mongoLocalSetup(t) + coll := db.Collection(module) + return Typed[T](NewMongoProvider(db).Collection(module)), coll, cleanup +} + +// rawDoc fetches the on-disk BSON document for assertions about its shape. +func rawDoc(t *testing.T, coll *mongo.Collection, id string) bson.M { + t.Helper() + var doc bson.M + if err := coll.FindOne(context.Background(), bson.M{"_id": id}).Decode(&doc); err != nil { + t.Fatalf("raw FindOne %s: %v", id, err) + } + return doc +} + +func TestMongoDocStore_RootShape(t *testing.T) { + store, coll, cleanup := mongoStore[portfolioLike](t, "coin") + defer cleanup() + ctx := context.Background() + + if err := store.Put(ctx, "user:7", portfolioLike{USD: 1000.25, Assets: map[string]float64{"BTC": 1}}); err != nil { + t.Fatalf("Put: %v", err) + } + doc := rawDoc(t, coll, "user:7") + + if _, ok := doc["value"]; ok { + t.Error("doc has legacy 'value' envelope field") + } + if _, ok := doc["_payload"]; ok { + t.Error("doc has '_payload' fallback field") + } + if _, ok := doc["usd"]; !ok { + t.Error("payload field 'usd' not hoisted to root") + } + if _, ok := doc["assets"]; !ok { + t.Error("payload field 'assets' not hoisted to root") + } + if _, ok := doc["updatedAt"].(bson.DateTime); !ok { + t.Errorf("updatedAt is %T, want BSON Date", doc["updatedAt"]) + } + + got, version, err := store.Get(ctx, "user:7") + if err != nil || version != 1 || got.USD != 1000.25 || got.Assets["BTC"] != 1 { + t.Fatalf("Get round-trip: %+v v=%d err=%v", got, version, err) + } +} + +func TestMongoDocStore_OverwriteRemovesStaleFields(t *testing.T) { + store, coll, cleanup := mongoStore[map[string]int](t, "misc") + defer cleanup() + ctx := context.Background() + + if err := store.Put(ctx, "k", map[string]int{"a": 1, "b": 2}); err != nil { + t.Fatalf("Put A: %v", err) + } + if err := store.Put(ctx, "k", map[string]int{"a": 3}); err != nil { + t.Fatalf("Put B: %v", err) + } + doc := rawDoc(t, coll, "k") + if _, ok := doc["b"]; ok { + t.Error("stale field 'b' survived overwrite") + } + if doc["a"] != int32(3) && doc["a"] != int64(3) { + t.Errorf("a = %v (%T), want 3", doc["a"], doc["a"]) + } +} + +func TestMongoDocStore_PutVersionedConcurrentCreate(t *testing.T) { + store, _, cleanup := mongoStore[portfolioLike](t, "coin") + defer cleanup() + ctx := context.Background() + + const n = 12 + var wg sync.WaitGroup + var mu sync.Mutex + wins := 0 + wg.Add(n) + for i := 0; i < n; i++ { + go func() { + defer wg.Done() + if err := store.PutVersioned(ctx, "race", 0, portfolioLike{USD: 1}); err == nil { + mu.Lock() + wins++ + mu.Unlock() + } else if !errors.Is(err, ErrConflict) { + t.Errorf("unexpected err: %v", err) + } + }() + } + wg.Wait() + if wins != 1 { + t.Fatalf("concurrent create winners = %d, want exactly 1", wins) + } +} + +func TestMongoDocStore_PutVersionedStaleConflict(t *testing.T) { + store, _, cleanup := mongoStore[portfolioLike](t, "coin") + defer cleanup() + ctx := context.Background() + + _ = store.PutVersioned(ctx, "k", 0, portfolioLike{USD: 1}) + if err := store.PutVersioned(ctx, "k", 99, portfolioLike{USD: 2}); !errors.Is(err, ErrConflict) { + t.Fatalf("stale CAS = %v, want ErrConflict", err) + } + if err := store.PutVersioned(ctx, "k", 1, portfolioLike{USD: 2}); err != nil { + t.Fatalf("fresh CAS: %v", err) + } +} + +// wrappedScalar / wrappedArray prove non-object values become named root fields +// (the lolschedule pattern) rather than an envelope or fallback. +type wrappedScalar struct { + Date string `json:"date" bson:"date"` +} +type wrappedArray struct { + Subscribers []int `json:"subscribers" bson:"subscribers"` +} + +func TestMongoDocStore_WrappedScalarAndArray(t *testing.T) { + db, cleanup := mongoLocalSetup(t) + defer cleanup() + ctx := context.Background() + p := NewMongoProvider(db) + + scalar := Typed[wrappedScalar](p.Collection("lolschedule")) + if err := scalar.Put(ctx, "last_push_date", wrappedScalar{Date: "2026-06-28"}); err != nil { + t.Fatalf("scalar Put: %v", err) + } + doc := rawDoc(t, db.Collection("lolschedule"), "last_push_date") + if doc["date"] != "2026-06-28" { + t.Errorf("scalar root field date = %v", doc["date"]) + } + + arr := Typed[wrappedArray](p.Collection("lolschedule")) + if err := arr.Put(ctx, "subscribers", wrappedArray{Subscribers: []int{1, 2, 3}}); err != nil { + t.Fatalf("array Put: %v", err) + } + doc = rawDoc(t, db.Collection("lolschedule"), "subscribers") + if _, ok := doc["subscribers"].(bson.A); !ok { + t.Errorf("array root field subscribers = %T, want bson.A", doc["subscribers"]) + } +} diff --git a/internal/storage/mongodb_kv.go b/internal/storage/mongodb_kv.go deleted file mode 100644 index a5d84db..0000000 --- a/internal/storage/mongodb_kv.go +++ /dev/null @@ -1,266 +0,0 @@ -package storage - -import ( - "context" - "encoding/json" - "errors" - "fmt" - "time" - - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo" - "go.mongodb.org/mongo-driver/v2/mongo/options" -) - -// MongoDB document fields. `_id` holds the user key; `value` holds the raw -// bytes; `updatedAt` is unix-nanos for observability + future TTL. -const ( - mongoIDField = "_id" - mongoValueField = "value" - mongoUpdatedAtField = "updatedAt" - mongoVersionField = "version" -) - -// MongoKVStore is a KVStore backed by a single MongoDB collection. The caller -// (MongoProvider) creates one per module so cross-module isolation is -// "different collection" — no key prefix needed at this layer, mirroring -// FirestoreKVStore. -type MongoKVStore struct { - coll *mongo.Collection - moduleName string -} - -// NewMongoKVStore returns a store writing to the given collection. Callers must -// validate the collection/module name beforehand (MongoProvider does). -func NewMongoKVStore(coll *mongo.Collection, moduleName string) *MongoKVStore { - return &MongoKVStore{coll: coll, moduleName: moduleName} -} - -// decodeValue extracts the stored value bytes from a decoded document, -// reconstructing the caller's JSON []byte from whichever representation is on -// disk: -// - native object/array (bson.M / bson.A / bson.D) → re-serialized to JSON; -// - string → the bytes verbatim (bare scalars / non-JSON values, e.g. a date); -// - bson.Binary / []byte → legacy fallback for docs written by an earlier -// build that stored value as a binary blob. -func (s *MongoKVStore) decodeValue(key string, doc bson.M) ([]byte, error) { - raw, ok := doc[mongoValueField] - if !ok { - return nil, fmt.Errorf("mongo get %s/%s: missing %q field", s.moduleName, key, mongoValueField) - } - switch v := raw.(type) { - case string: - return []byte(v), nil - case bson.M, bson.A, bson.D: - return nativeToJSON(v) - case bson.Binary: - return v.Data, nil - case []byte: - return v, nil - default: - return nil, fmt.Errorf("mongo get %s/%s: unexpected value type %T", s.moduleName, key, raw) - } -} - -// Get returns the raw bytes stored at key, or ErrNotFound. -func (s *MongoKVStore) Get(ctx context.Context, key string) ([]byte, error) { - if err := validateKey(key); err != nil { - return nil, err - } - var doc bson.M - err := s.coll.FindOne(ctx, bson.M{mongoIDField: key}).Decode(&doc) - if err != nil { - if errors.Is(err, mongo.ErrNoDocuments) { - return nil, ErrNotFound - } - return nil, fmt.Errorf("mongo get %s/%s: %w", s.moduleName, key, err) - } - return s.decodeValue(key, doc) -} - -// GetJSON decodes the value at key into dst. -func (s *MongoKVStore) GetJSON(ctx context.Context, key string, dst any) error { - raw, err := s.Get(ctx, key) - if err != nil { - return err - } - if err := json.Unmarshal(raw, dst); err != nil { - return fmt.Errorf("mongo get %s/%s: json decode: %w", s.moduleName, key, err) - } - return nil -} - -// Put writes raw bytes at key, creating or overwriting, and bumps the version -// (so a concurrent versioned writer that read the old version sees a conflict). -func (s *MongoKVStore) Put(ctx context.Context, key string, val []byte) error { - if err := validateKey(key); err != nil { - return err - } - _, err := s.coll.UpdateOne(ctx, - bson.M{mongoIDField: key}, - bson.M{ - "$set": bson.M{ - mongoValueField: encodeValue(val), - mongoUpdatedAtField: time.Now().UTC().UnixNano(), - }, - "$inc": bson.M{mongoVersionField: 1}, - }, - options.UpdateOne().SetUpsert(true), - ) - if err != nil { - return fmt.Errorf("mongo put %s/%s: %w", s.moduleName, key, err) - } - return nil -} - -// PutJSON marshals val and writes the bytes at key. -func (s *MongoKVStore) PutJSON(ctx context.Context, key string, val any) error { - raw, err := json.Marshal(val) - if err != nil { - return fmt.Errorf("mongo put %s/%s: json encode: %w", s.moduleName, key, err) - } - return s.Put(ctx, key, raw) -} - -// GetVersioned returns the value and its version, or ErrNotFound. A document -// written by an older build that has no version field reports version 0 (and no -// error), so PutVersioned(expectedVersion=0) can adopt it. -func (s *MongoKVStore) GetVersioned(ctx context.Context, key string) ([]byte, int64, error) { - if err := validateKey(key); err != nil { - return nil, 0, err - } - var doc bson.M - err := s.coll.FindOne(ctx, bson.M{mongoIDField: key}).Decode(&doc) - if err != nil { - if errors.Is(err, mongo.ErrNoDocuments) { - return nil, 0, ErrNotFound - } - return nil, 0, fmt.Errorf("mongo get %s/%s: %w", s.moduleName, key, err) - } - val, err := s.decodeValue(key, doc) - if err != nil { - return nil, 0, err - } - return val, decodeVersion(doc), nil -} - -// PutVersioned writes val only if the stored version equals expectedVersion, -// then bumps it. expectedVersion == 0 means "create, or adopt a key that has no -// version field yet" (a legacy doc). ErrConflict on a version mismatch. -// -// - expectedVersion == 0 → upsert matching {_id} with version absent or 0; a -// key that already has version ≥ 1 fails the filter, the upsert attempts an -// insert on the same _id, and the unique-_id index returns duplicate-key → -// ErrConflict (linearizable single-winner for the first write). -// - expectedVersion > 0 → UpdateOne on {_id, version:expected}; MatchedCount -// of 0 means the version moved → ErrConflict. -func (s *MongoKVStore) PutVersioned(ctx context.Context, key string, expectedVersion int64, val []byte) error { - if err := validateKey(key); err != nil { - return err - } - now := time.Now().UTC().UnixNano() - if expectedVersion == 0 { - _, err := s.coll.UpdateOne(ctx, - bson.M{ - mongoIDField: key, - "$or": bson.A{ - bson.M{mongoVersionField: bson.M{"$exists": false}}, - bson.M{mongoVersionField: int64(0)}, - }, - }, - bson.M{"$set": bson.M{ - mongoValueField: encodeValue(val), - mongoUpdatedAtField: now, - mongoVersionField: int64(1), - }}, - options.UpdateOne().SetUpsert(true), - ) - if err != nil { - if mongo.IsDuplicateKeyError(err) { - return ErrConflict - } - return fmt.Errorf("mongo put-versioned %s/%s: %w", s.moduleName, key, err) - } - return nil - } - res, err := s.coll.UpdateOne(ctx, - bson.M{mongoIDField: key, mongoVersionField: expectedVersion}, - bson.M{ - "$set": bson.M{mongoValueField: encodeValue(val), mongoUpdatedAtField: now}, - "$inc": bson.M{mongoVersionField: 1}, - }, - ) - if err != nil { - return fmt.Errorf("mongo put-versioned %s/%s: %w", s.moduleName, key, err) - } - if res.MatchedCount == 0 { - return ErrConflict - } - return nil -} - -// decodeVersion reads the version field from a decoded doc, defaulting to 0 for -// legacy docs that predate the version field. The driver may decode a BSON int -// as int32 or int64. -func decodeVersion(doc bson.M) int64 { - switch v := doc[mongoVersionField].(type) { - case int64: - return v - case int32: - return int64(v) - case float64: - return int64(v) - default: - return 0 - } -} - -// Delete removes the document at key. Deleting a missing key is not an error -// (idempotent) — DeleteOne with a zero match count returns nil. -func (s *MongoKVStore) Delete(ctx context.Context, key string) error { - if err := validateKey(key); err != nil { - return err - } - _, err := s.coll.DeleteOne(ctx, bson.M{mongoIDField: key}) - if err != nil { - return fmt.Errorf("mongo delete %s/%s: %w", s.moduleName, key, err) - } - return nil -} - -// List returns all document IDs in the collection that start with prefix. -// Implemented as a half-open range scan on _id (reusing prefixSuccessor) so it -// uses the _id index and avoids regex injection. Empty prefix returns the whole -// collection. -func (s *MongoKVStore) List(ctx context.Context, prefix string) ([]string, error) { - if err := validatePrefix(prefix); err != nil { - return nil, err - } - filter := bson.M{} - if prefix != "" { - filter[mongoIDField] = bson.M{ - "$gte": prefix, - "$lt": prefixSuccessor(prefix), - } - } - cur, err := s.coll.Find(ctx, filter, options.Find().SetProjection(bson.M{mongoIDField: 1})) - if err != nil { - return nil, fmt.Errorf("mongo list %s prefix=%q: %w", s.moduleName, prefix, err) - } - defer func() { _ = cur.Close(ctx) }() - - var keys []string - for cur.Next(ctx) { - var doc struct { - ID string `bson:"_id"` - } - if err := cur.Decode(&doc); err != nil { - return nil, fmt.Errorf("mongo list %s prefix=%q: decode: %w", s.moduleName, prefix, err) - } - keys = append(keys, doc.ID) - } - if err := cur.Err(); err != nil { - return nil, fmt.Errorf("mongo list %s prefix=%q: %w", s.moduleName, prefix, err) - } - return keys, nil -} diff --git a/internal/storage/mongodb_kv_test.go b/internal/storage/mongodb_kv_test.go deleted file mode 100644 index 0b4f7f7..0000000 --- a/internal/storage/mongodb_kv_test.go +++ /dev/null @@ -1,370 +0,0 @@ -package storage - -import ( - "context" - "errors" - "fmt" - "os" - "reflect" - "sync" - "testing" - "time" - - "go.mongodb.org/mongo-driver/v2/bson" - "go.mongodb.org/mongo-driver/v2/mongo" -) - -// mongoLocalSetup connects to a local MongoDB and returns a store over a fresh, -// uniquely-named collection. Tests skip if MONGODB_TEST_URL is unset so CI -// without a Mongo container still builds (mirrors the DynamoDB Local gating). -func mongoLocalSetup(t *testing.T, module string) (*MongoKVStore, *mongo.Database, func()) { - t.Helper() - uri := os.Getenv("MONGODB_TEST_URL") - if uri == "" { - t.Skip("MONGODB_TEST_URL not set; skipping MongoDB integration test (run `make mongo-local` to start the local container)") - } - - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cancel() - - client, err := NewMongoClient(ctx, uri) - if err != nil { - t.Fatalf("NewMongoClient: %v", err) - } - // Unique DB per test so parallel runs and cross-module isolation checks - // never collide. Mongo db names are capped at 63 bytes. - dbName := fmt.Sprintf("miti99bot_test_%d", time.Now().UnixNano()) - if len(dbName) > 63 { - dbName = dbName[:63] - } - db := client.Database(dbName) - - cleanup := func() { - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cancel() - _ = db.Drop(ctx) - _ = client.Disconnect(ctx) - } - return NewMongoKVStore(db.Collection(module), module), db, cleanup -} - -func TestMongoKVStore_PutGetDelete(t *testing.T) { - s, _, cleanup := mongoLocalSetup(t, "wordle") - defer cleanup() - - ctx := context.Background() - if err := s.Put(ctx, "user:1:state", []byte("hello")); err != nil { - t.Fatalf("Put: %v", err) - } - got, err := s.Get(ctx, "user:1:state") - if err != nil { - t.Fatalf("Get: %v", err) - } - if string(got) != "hello" { - t.Errorf("Get: got %q, want %q", got, "hello") - } - - // Overwrite preserves byte fidelity. - if err := s.Put(ctx, "user:1:state", []byte("world")); err != nil { - t.Fatalf("Put overwrite: %v", err) - } - got, _ = s.Get(ctx, "user:1:state") - if string(got) != "world" { - t.Errorf("Get after overwrite: got %q, want %q", got, "world") - } - - if err := s.Delete(ctx, "user:1:state"); err != nil { - t.Fatalf("Delete: %v", err) - } - if _, err := s.Get(ctx, "user:1:state"); !errors.Is(err, ErrNotFound) { - t.Errorf("Get after Delete: got %v, want ErrNotFound", err) - } -} - -func TestMongoKVStore_GetMissing(t *testing.T) { - s, _, cleanup := mongoLocalSetup(t, "wordle") - defer cleanup() - - if _, err := s.Get(context.Background(), "missing"); !errors.Is(err, ErrNotFound) { - t.Errorf("got %v, want ErrNotFound", err) - } -} - -// TestMongoKVStore_NativeValueRepresentation verifies a JSON object value is -// persisted as a NATIVE BSON document (expandable/queryable in Atlas), and a -// bare non-JSON value as a string — and that both round-trip. -func TestMongoKVStore_NativeValueRepresentation(t *testing.T) { - s, db, cleanup := mongoLocalSetup(t, "wordle") - defer cleanup() - - ctx := context.Background() - obj := []byte(`{"word":"chào","score":42}`) // multi-byte UTF-8 object - if err := s.Put(ctx, "obj", obj); err != nil { - t.Fatalf("Put obj: %v", err) - } - // A bare string (lolschedule date-guard style) is not JSON → stored as string. - if err := s.Put(ctx, "bare", []byte("2026-06-28")); err != nil { - t.Fatalf("Put bare: %v", err) - } - - var objDoc, bareDoc bson.M - if err := db.Collection("wordle").FindOne(ctx, bson.M{"_id": "obj"}).Decode(&objDoc); err != nil { - t.Fatalf("raw FindOne obj: %v", err) - } - switch objDoc["value"].(type) { - case bson.M, bson.D: - // native object — good - default: - t.Errorf("object value stored as %T, want native bson document", objDoc["value"]) - } - if err := db.Collection("wordle").FindOne(ctx, bson.M{"_id": "bare"}).Decode(&bareDoc); err != nil { - t.Fatalf("raw FindOne bare: %v", err) - } - if _, ok := bareDoc["value"].(string); !ok { - t.Errorf("bare value stored as %T, want string", bareDoc["value"]) - } - - // Round-trips. - if got, _ := s.Get(ctx, "bare"); string(got) != "2026-06-28" { - t.Errorf("bare round trip: got %q", got) - } - var back map[string]any - if err := s.GetJSON(ctx, "obj", &back); err != nil { - t.Fatalf("GetJSON obj: %v", err) - } - if back["word"] != "chào" { - t.Errorf("object round trip lost data: %v", back) - } -} - -// TestMongoKVStore_Int64Fidelity is the codec gate: an int64 field must survive -// the JSON↔BSON round trip as an integer, not collapse to a float (which would -// make GetJSON into an int64 field fail or drift). -func TestMongoKVStore_Int64Fidelity(t *testing.T) { - s, db, cleanup := mongoLocalSetup(t, "coin") - defer cleanup() - - ctx := context.Background() - type meta struct { - CreatedAt int64 `json:"createdAt"` // UnixMilli ~1.7e12 - Invested float64 `json:"invested"` - } - type portfolio struct { - USD float64 `json:"usd"` - Meta meta `json:"meta"` - Qty map[string]int64 `json:"qty"` - } - in := portfolio{USD: 1234.56, Meta: meta{CreatedAt: 1719500000000, Invested: 5000}, Qty: map[string]int64{"BTC": 3}} - if err := s.PutJSON(ctx, "u1", in); err != nil { - t.Fatalf("PutJSON: %v", err) - } - var out portfolio - if err := s.GetJSON(ctx, "u1", &out); err != nil { - t.Fatalf("GetJSON: %v", err) - } - if !reflect.DeepEqual(out, in) { - t.Errorf("fidelity lost: got %+v, want %+v", out, in) - } - // The stored CreatedAt must be a BSON int (int32/int64), not a double. - var doc bson.M - _ = db.Collection("coin").FindOne(ctx, bson.M{"_id": "u1"}).Decode(&doc) - if m, ok := doc["value"].(bson.M); ok { - if meta, ok := m["meta"].(bson.M); ok { - switch meta["createdAt"].(type) { - case int64, int32: - // good — preserved as integer - default: - t.Errorf("createdAt stored as %T, want int64 (json.Number codec)", meta["createdAt"]) - } - } - } -} - -func TestMongoKVStore_JSONRoundTrip(t *testing.T) { - s, _, cleanup := mongoLocalSetup(t, "loldle") - defer cleanup() - - ctx := context.Background() - type state struct { - Score int `json:"score"` - Name string `json:"name"` - } - in := state{Score: 42, Name: "ezreal"} - if err := s.PutJSON(ctx, "u1", in); err != nil { - t.Fatalf("PutJSON: %v", err) - } - var out state - if err := s.GetJSON(ctx, "u1", &out); err != nil { - t.Fatalf("GetJSON: %v", err) - } - if out != in { - t.Errorf("got %+v, want %+v", out, in) - } -} - -func TestMongoKVStore_ListPrefix(t *testing.T) { - s, _, cleanup := mongoLocalSetup(t, "wordle") - defer cleanup() - - ctx := context.Background() - for _, k := range []string{"user:1:state", "user:2:state", "config:daily", "user:1:history"} { - if err := s.Put(ctx, k, []byte("x")); err != nil { - t.Fatalf("Put %s: %v", k, err) - } - } - - got, err := s.List(ctx, "user:") - if err != nil { - t.Fatalf("List: %v", err) - } - want := map[string]bool{"user:1:state": true, "user:2:state": true, "user:1:history": true} - if len(got) != len(want) { - t.Errorf("List: got %v (len=%d), want len=%d", got, len(got), len(want)) - } - for _, k := range got { - if !want[k] { - t.Errorf("List: unexpected key %q", k) - } - } - - // Empty prefix returns everything in the collection. - all, err := s.List(ctx, "") - if err != nil { - t.Fatalf("List empty: %v", err) - } - if len(all) != 4 { - t.Errorf("List(\"\"): got %d keys, want 4", len(all)) - } -} - -func TestMongoKVStore_Versioned(t *testing.T) { - s, _, cleanup := mongoLocalSetup(t, "gold") - defer cleanup() - - ctx := context.Background() - // Create-if-absent (version 0) succeeds once, then conflicts. - if err := s.PutVersioned(ctx, "user:1", 0, []byte("v1")); err != nil { - t.Fatalf("PutVersioned create: %v", err) - } - if err := s.PutVersioned(ctx, "user:1", 0, []byte("v2")); !errors.Is(err, ErrConflict) { - t.Errorf("create over existing: got %v, want ErrConflict", err) - } - - val, ver, err := s.GetVersioned(ctx, "user:1") - if err != nil || string(val) != "v1" || ver != 1 { - t.Fatalf("GetVersioned: got (%q, v=%d, %v), want (v1, 1, nil)", val, ver, err) - } - - // Swap with matching version succeeds and bumps; stale version conflicts. - if err := s.PutVersioned(ctx, "user:1", 1, []byte("v2")); err != nil { - t.Fatalf("swap matching version: %v", err) - } - if err := s.PutVersioned(ctx, "user:1", 1, []byte("v3")); !errors.Is(err, ErrConflict) { - t.Errorf("stale version: got %v, want ErrConflict", err) - } - if got, _ := s.Get(ctx, "user:1"); string(got) != "v2" { - t.Errorf("stored value = %q, want %q", got, "v2") - } - - // Swap on a missing key conflicts. - if err := s.PutVersioned(ctx, "user:missing", 1, []byte("v2")); !errors.Is(err, ErrConflict) { - t.Errorf("swap missing key: got %v, want ErrConflict", err) - } -} - -// TestMongoKVStore_PutVersioned_AdoptsLegacyDoc proves a document written by an -// older build (no version field) is treated as version 0 and updated cleanly, -// so existing users' portfolios keep working after the rollout. -func TestMongoKVStore_PutVersioned_AdoptsLegacyDoc(t *testing.T) { - s, db, cleanup := mongoLocalSetup(t, "coin") - defer cleanup() - - ctx := context.Background() - // Seed a legacy doc: value string, NO version field. - if _, err := db.Collection("coin").InsertOne(ctx, bson.M{"_id": "user:9", "value": "legacy"}); err != nil { - t.Fatalf("seed legacy doc: %v", err) - } - val, ver, err := s.GetVersioned(ctx, "user:9") - if err != nil || string(val) != "legacy" || ver != 0 { - t.Fatalf("GetVersioned legacy: got (%q, v=%d, %v), want (legacy, 0, nil)", val, ver, err) - } - // Claim it at version 0 — must succeed (no spurious conflict). - if err := s.PutVersioned(ctx, "user:9", 0, []byte("migrated")); err != nil { - t.Fatalf("adopt legacy doc: %v", err) - } - if got, ver, _ := s.GetVersioned(ctx, "user:9"); string(got) != "migrated" || ver != 1 { - t.Errorf("after adopt: got (%q, v=%d), want (migrated, 1)", got, ver) - } -} - -// TestMongoKVStore_PutVersioned_ConcurrentCreate proves the absent-create race -// is linearizable: N goroutines racing PutVersioned(_, 0, _) on one key yield -// exactly one winner; losers get ErrConflict. Blocking correctness gate. -func TestMongoKVStore_PutVersioned_ConcurrentCreate(t *testing.T) { - s, _, cleanup := mongoLocalSetup(t, "coin") - defer cleanup() - - ctx := context.Background() - const n = 16 - var wg sync.WaitGroup - var mu sync.Mutex - var wins, conflicts, others int - start := make(chan struct{}) - for i := 0; i < n; i++ { - wg.Add(1) - go func(i int) { - defer wg.Done() - <-start - err := s.PutVersioned(ctx, "race", 0, []byte(fmt.Sprintf("v%d", i))) - mu.Lock() - defer mu.Unlock() - switch { - case err == nil: - wins++ - case errors.Is(err, ErrConflict): - conflicts++ - default: - others++ - t.Errorf("unexpected error: %v", err) - } - }(i) - } - close(start) - wg.Wait() - - if wins != 1 { - t.Errorf("concurrent create: got %d winners, want exactly 1", wins) - } - if conflicts != n-1 { - t.Errorf("concurrent create: got %d conflicts, want %d", conflicts, n-1) - } - if others != 0 { - t.Errorf("concurrent create produced %d non-conflict errors", others) - } -} - -// TestMongoKVStore_GetDocWithoutValueField covers a malformed document edge -// case (a doc that exists but lacks the value field) — Get must surface a clear -// error, not panic or return empty bytes silently. -func TestMongoKVStore_GetDocWithoutValueField(t *testing.T) { - s, db, cleanup := mongoLocalSetup(t, "wordle") - defer cleanup() - - ctx := context.Background() - // Insert a doc directly with no `value` field. - if _, err := db.Collection("wordle").InsertOne(ctx, bson.M{"_id": "novalue", "other": 1}); err != nil { - t.Fatalf("seed InsertOne: %v", err) - } - if _, err := s.Get(ctx, "novalue"); err == nil { - t.Errorf("Get on doc without value field: got nil error, want a descriptive error") - } -} - -func TestMongoKVStore_DeleteMissingNoError(t *testing.T) { - s, _, cleanup := mongoLocalSetup(t, "wordle") - defer cleanup() - - if err := s.Delete(context.Background(), "never-existed"); err != nil { - t.Errorf("Delete missing key: got %v, want nil (idempotent)", err) - } -} diff --git a/internal/storage/mongodb_provider.go b/internal/storage/mongodb_provider.go index 785c708..06da8fa 100644 --- a/internal/storage/mongodb_provider.go +++ b/internal/storage/mongodb_provider.go @@ -4,28 +4,26 @@ import ( "go.mongodb.org/mongo-driver/v2/mongo" ) -// MongoProvider is a KVProvider that creates one collection per module, -// mirroring FirestoreProvider. Collection-per-module IS the isolation — no key -// prefix wrapping is needed at this layer. +// MongoProvider is a Provider that creates one collection per module. +// Collection-per-module IS the isolation — no key prefix wrapping is needed. type MongoProvider struct { db *mongo.Database } // NewMongoProvider returns a provider over the given database handle. The -// underlying client must outlive every KVStore the provider hands out; callers +// underlying client must outlive every store the provider hands out; callers // own its Disconnect. func NewMongoProvider(db *mongo.Database) *MongoProvider { return &MongoProvider{db: db} } -// For returns a MongoKVStore writing to a collection named after the module. -// moduleName is re-validated against collectionNameRe — defense in depth -// against caller bugs that bypass modules.Build, identical to the firestore -// and dynamodb providers. An invalid name yields an invalidStore whose every -// op errors at first use. -func (p *MongoProvider) For(moduleName string) KVStore { +// Collection returns a handle to the collection named after the module. +// moduleName is re-validated against collectionNameRe — defense in depth against +// caller bugs that bypass modules.Build. An invalid name yields an +// invalidCollection whose Typed store errors at first use. +func (p *MongoProvider) Collection(moduleName string) Collection { if !collectionNameRe.MatchString(moduleName) { - return invalidStore{name: moduleName} + return invalidCollection{name: moduleName} } - return NewMongoKVStore(p.db.Collection(moduleName), moduleName) + return mongoCollection{coll: p.db.Collection(moduleName), module: moduleName} } diff --git a/internal/storage/mongodb_provider_test.go b/internal/storage/mongodb_provider_test.go index 1f6dcbe..c2ad885 100644 --- a/internal/storage/mongodb_provider_test.go +++ b/internal/storage/mongodb_provider_test.go @@ -6,10 +6,10 @@ import ( "testing" ) -// MongoProvider.For re-validates the module name as defense-in-depth. Invalid -// names return invalidStore without touching the DB — the branch worth locking -// even without a live Mongo (mirrors the firestore provider test). -func TestMongoProvider_For_RejectsInvalidName(t *testing.T) { +// MongoProvider.Collection re-validates the module name as defense-in-depth. +// Invalid names yield an invalidCollection whose Typed store errors without +// touching the DB — the branch worth locking even without a live Mongo. +func TestMongoProvider_Collection_RejectsInvalidName(t *testing.T) { p := &MongoProvider{db: nil} bogus := []string{ "", @@ -21,45 +21,45 @@ func TestMongoProvider_For_RejectsInvalidName(t *testing.T) { "with:colon", } for _, name := range bogus { - store := p.For(name) - if _, err := store.Get(context.Background(), "any-key"); !errors.Is(err, ErrInvalidModuleName) { - t.Errorf("For(%q).Get → %v, want ErrInvalidModuleName", name, err) + // Invalid names short-circuit to invalidCollection before touching the + // (nil) db, so Typed yields a store whose ops report ErrInvalidModuleName. + store := Typed[string](p.Collection(name)) + if _, _, err := store.Get(context.Background(), "any-key"); !errors.Is(err, ErrInvalidModuleName) { + t.Errorf("Collection(%q) store Get → %v, want ErrInvalidModuleName", name, err) } } + // Valid names pass validation against a real db in + // TestMongoProvider_CrossModuleIsolation (gated on MONGODB_TEST_URL); a nil + // db here would panic in the driver for a valid name. } // TestMongoProvider_CrossModuleIsolation verifies collection-per-module // isolation: the same key written through two module stores yields independent // values. Gated on MONGODB_TEST_URL. func TestMongoProvider_CrossModuleIsolation(t *testing.T) { - _, db, cleanup := mongoLocalSetup(t, "wordle") + db, cleanup := mongoLocalSetup(t) defer cleanup() p := NewMongoProvider(db) ctx := context.Background() - wordle := p.For("wordle") - loldle := p.For("loldle") - if err := wordle.Put(ctx, "shared", []byte("from-wordle")); err != nil { + // Mongo payloads must be a struct or map (bson inline cannot hoist a + // scalar); a wrapper struct is the same shape modules use. + wordle := Typed[wrappedScalar](p.Collection("wordle")) + loldle := Typed[wrappedScalar](p.Collection("loldle")) + if err := wordle.Put(ctx, "shared", wrappedScalar{Date: "from-wordle"}); err != nil { t.Fatalf("wordle Put: %v", err) } - if err := loldle.Put(ctx, "shared", []byte("from-loldle")); err != nil { + if err := loldle.Put(ctx, "shared", wrappedScalar{Date: "from-loldle"}); err != nil { t.Fatalf("loldle Put: %v", err) } - gotW, _ := wordle.Get(ctx, "shared") - gotL, _ := loldle.Get(ctx, "shared") - if string(gotW) != "from-wordle" { - t.Errorf("wordle key leaked: got %q", gotW) + gotW, _, _ := wordle.Get(ctx, "shared") + gotL, _, _ := loldle.Get(ctx, "shared") + if gotW.Date != "from-wordle" { + t.Errorf("wordle key leaked: got %q", gotW.Date) } - if string(gotL) != "from-loldle" { - t.Errorf("loldle key leaked: got %q", gotL) - } - // Canonical names pass validation (not invalidStore); this also covers the - // valid-name branch of For against a real database. - for _, name := range []string{"misc", "demo-mod", "x", "a1_b-2"} { - if _, ok := p.For(name).(invalidStore); ok { - t.Errorf("For(%q) returned invalidStore; expected validation to pass", name) - } + if gotL.Date != "from-loldle" { + t.Errorf("loldle key leaked: got %q", gotL.Date) } } diff --git a/internal/storage/mongodb_value_codec.go b/internal/storage/mongodb_value_codec.go deleted file mode 100644 index 01d2379..0000000 --- a/internal/storage/mongodb_value_codec.go +++ /dev/null @@ -1,116 +0,0 @@ -package storage - -import ( - "bytes" - "encoding/json" - "fmt" - - "go.mongodb.org/mongo-driver/v2/bson" -) - -// This file converts between the KVStore's opaque JSON []byte contract and a -// native BSON representation, so values render as expandable documents in the -// Atlas/Compass UI (and are queryable) rather than as opaque blobs. -// -// Fidelity: JSON is decoded with json.Number so integral numbers persist as -// BSON int64 (NOT float64) — a struct's int64 field round-trips without -// precision loss. Only JSON objects and arrays become native; bare scalars and -// non-JSON values (e.g. a plain date string) fall back to a BSON string. - -// encodeValue produces the stored BSON representation of a value. A JSON object -// or array → native (expandable/queryable); anything else → a string (the -// human-readable fallback, also covers non-UTF-8-free callers identically to -// the previous string encoding). -func encodeValue(val []byte) any { - if native, ok := jsonToNative(val); ok { - return native - } - return string(val) -} - -// jsonToNative decodes val into a native BSON value (bson.M / bson.A) when it is -// a JSON object or array, preserving integers as int64. Returns ok=false for -// scalars, non-JSON, or decode errors (caller stores those as a string). -func jsonToNative(val []byte) (any, bool) { - trimmed := bytes.TrimSpace(val) - if len(trimmed) == 0 || (trimmed[0] != '{' && trimmed[0] != '[') { - return nil, false - } - dec := json.NewDecoder(bytes.NewReader(val)) - dec.UseNumber() - var v any - if err := dec.Decode(&v); err != nil { - return nil, false - } - // Reject trailing garbage after the JSON value. - if dec.More() { - return nil, false - } - return numbersToBSON(v), true -} - -// numbersToBSON walks a json.Unmarshal(UseNumber) tree, converting maps/slices -// to bson.M/bson.A and json.Number to int64 (when integral) else float64. -func numbersToBSON(v any) any { - switch t := v.(type) { - case map[string]any: - m := make(bson.M, len(t)) - for k, val := range t { - m[k] = numbersToBSON(val) - } - return m - case []any: - a := make(bson.A, len(t)) - for i, val := range t { - a[i] = numbersToBSON(val) - } - return a - case json.Number: - if i, err := t.Int64(); err == nil { - return i - } - if f, err := t.Float64(); err == nil { - return f - } - return t.String() - default: - return v // string, bool, nil - } -} - -// nativeToJSON re-serializes a natively-stored value (bson.M / bson.A / bson.D -// / scalar) back to JSON bytes. bson.M is map[string]any and bson.A is []any, -// both of which json.Marshal handles directly; bson.D (ordered) is normalized -// to a map first so it marshals as an object, not an array of key/value pairs. -func nativeToJSON(raw any) ([]byte, error) { - out, err := json.Marshal(normalizeBSON(raw)) - if err != nil { - return nil, fmt.Errorf("mongo: re-encode native value: %w", err) - } - return out, nil -} - -// normalizeBSON converts bson.D to bson.M recursively so json.Marshal emits an -// object. bson.M/bson.A children are recursed; scalars pass through. -func normalizeBSON(v any) any { - switch t := v.(type) { - case bson.D: - m := make(bson.M, len(t)) - for _, e := range t { - m[e.Key] = normalizeBSON(e.Value) - } - return m - case bson.M: - for k, val := range t { - t[k] = normalizeBSON(val) - } - return t - case bson.A: - for i, val := range t { - t[i] = normalizeBSON(val) - } - return t - default: - return v - } -} diff --git a/internal/storage/prefix.go b/internal/storage/prefix.go deleted file mode 100644 index df7ff2e..0000000 --- a/internal/storage/prefix.go +++ /dev/null @@ -1,75 +0,0 @@ -package storage - -import ( - "context" - "errors" - "fmt" - "strings" -) - -// Prefixed returns a KVStore that transparently prepends prefix+":" to every -// key. List() strips the prefix from returned keys so callers see their own -// flat namespace. The prefix must be non-empty. -func Prefixed(inner KVStore, prefix string) KVStore { - if prefix == "" { - panic("storage: Prefixed requires non-empty prefix") - } - return &prefixedStore{inner: inner, prefix: prefix + ":"} -} - -type prefixedStore struct { - inner KVStore - prefix string -} - -func (p *prefixedStore) k(key string) string { return p.prefix + key } - -func (p *prefixedStore) Get(ctx context.Context, key string) ([]byte, error) { - return p.inner.Get(ctx, p.k(key)) -} - -func (p *prefixedStore) GetJSON(ctx context.Context, key string, dst any) error { - return p.inner.GetJSON(ctx, p.k(key), dst) -} - -func (p *prefixedStore) Put(ctx context.Context, key string, val []byte) error { - return p.inner.Put(ctx, p.k(key), val) -} - -func (p *prefixedStore) GetVersioned(ctx context.Context, key string) ([]byte, int64, error) { - vs, ok := p.inner.(VersionedStore) - if !ok { - return nil, 0, fmt.Errorf("storage: inner store does not support versioned reads: %w", errors.ErrUnsupported) - } - return vs.GetVersioned(ctx, p.k(key)) -} - -func (p *prefixedStore) PutVersioned(ctx context.Context, key string, expectedVersion int64, val []byte) error { - vs, ok := p.inner.(VersionedStore) - if !ok { - // A missing capability is permanent: report it as unsupported so - // callers fail fast instead of retrying it as a write conflict. - return fmt.Errorf("storage: inner store does not support versioned writes: %w", errors.ErrUnsupported) - } - return vs.PutVersioned(ctx, p.k(key), expectedVersion, val) -} - -func (p *prefixedStore) PutJSON(ctx context.Context, key string, val any) error { - return p.inner.PutJSON(ctx, p.k(key), val) -} - -func (p *prefixedStore) Delete(ctx context.Context, key string) error { - return p.inner.Delete(ctx, p.k(key)) -} - -func (p *prefixedStore) List(ctx context.Context, prefix string) ([]string, error) { - keys, err := p.inner.List(ctx, p.k(prefix)) - if err != nil { - return nil, err - } - out := make([]string, len(keys)) - for i, k := range keys { - out[i] = strings.TrimPrefix(k, p.prefix) - } - return out, nil -} diff --git a/internal/storage/prefix_test.go b/internal/storage/prefix_test.go deleted file mode 100644 index e241c9d..0000000 --- a/internal/storage/prefix_test.go +++ /dev/null @@ -1,134 +0,0 @@ -package storage - -import ( - "context" - "errors" - "reflect" - "testing" -) - -func TestPrefixed_RoundTrip(t *testing.T) { - ctx := context.Background() - base := NewMemoryKVStore() - a := Prefixed(base, "modA") - b := Prefixed(base, "modB") - - if err := a.Put(ctx, "score", []byte("10")); err != nil { - t.Fatalf("a.Put: %v", err) - } - if err := b.Put(ctx, "score", []byte("20")); err != nil { - t.Fatalf("b.Put: %v", err) - } - - got, err := a.Get(ctx, "score") - if err != nil { - t.Fatalf("a.Get: %v", err) - } - if string(got) != "10" { - t.Errorf("a.Get score = %q, want %q", got, "10") - } - - got, err = b.Get(ctx, "score") - if err != nil { - t.Fatalf("b.Get: %v", err) - } - if string(got) != "20" { - t.Errorf("b.Get score = %q, want %q", got, "20") - } -} - -func TestPrefixed_ListStripsPrefix(t *testing.T) { - ctx := context.Background() - base := NewMemoryKVStore() - mod := Prefixed(base, "wordle") - - for _, k := range []string{"u:1", "u:2", "session:abc"} { - if err := mod.Put(ctx, k, []byte("x")); err != nil { - t.Fatalf("Put %q: %v", k, err) - } - } - - got, err := mod.List(ctx, "u:") - if err != nil { - t.Fatalf("List: %v", err) - } - want := []string{"u:1", "u:2"} - if !reflect.DeepEqual(got, want) { - t.Errorf("List u: = %v, want %v", got, want) - } -} - -func TestPrefixed_NotFoundPropagates(t *testing.T) { - ctx := context.Background() - base := NewMemoryKVStore() - mod := Prefixed(base, "modA") - - if _, err := mod.Get(ctx, "missing"); err != ErrNotFound { - t.Errorf("Get missing = %v, want ErrNotFound", err) - } -} - -func TestPrefixed_VersionedDelegatesWithPrefixedKey(t *testing.T) { - ctx := context.Background() - base := NewMemoryKVStore() - mod := Prefixed(base, "gold") - - vs, ok := mod.(VersionedStore) - if !ok { - t.Fatal("Prefixed store does not implement VersionedStore") - } - if err := vs.PutVersioned(ctx, "user:1", 0, []byte("v1")); err != nil { - t.Fatalf("PutVersioned create: %v", err) - } - - // The write must land under the prefixed key on the inner store. - got, err := base.Get(ctx, "gold:user:1") - if err != nil { - t.Fatalf("inner Get: %v", err) - } - if string(got) != "v1" { - t.Errorf("inner value = %q, want %q", got, "v1") - } - - // GetVersioned reads through the prefix and returns the inner version. - val, ver, err := vs.GetVersioned(ctx, "user:1") - if err != nil || string(val) != "v1" || ver != 1 { - t.Fatalf("GetVersioned: got (%q, v=%d, %v), want (v1, 1, nil)", val, ver, err) - } - - // Stale version must surface the inner store's conflict. - if err := vs.PutVersioned(ctx, "user:1", 99, []byte("v2")); !errors.Is(err, ErrConflict) { - t.Errorf("stale version: got %v, want ErrConflict", err) - } - if err := vs.PutVersioned(ctx, "user:1", 1, []byte("v2")); err != nil { - t.Errorf("matching version: %v", err) - } -} - -// plainStore hides the inner store's versioned methods so the wrapper's -// missing-capability path can be exercised. -type plainStore struct { - KVStore -} - -func TestPrefixed_VersionedUnsupportedInner(t *testing.T) { - mod := Prefixed(&plainStore{KVStore: NewMemoryKVStore()}, "gold") - vs := mod.(VersionedStore) - - err := vs.PutVersioned(context.Background(), "user:1", 0, []byte("v1")) - if !errors.Is(err, errors.ErrUnsupported) { - t.Errorf("got %v, want errors.ErrUnsupported", err) - } - if errors.Is(err, ErrConflict) { - t.Error("missing versioned capability must not be reported as a retryable conflict") - } -} - -func TestPrefixed_PanicsOnEmptyPrefix(t *testing.T) { - defer func() { - if recover() == nil { - t.Error("Prefixed(_, \"\") did not panic") - } - }() - Prefixed(NewMemoryKVStore(), "") -} diff --git a/plans/260627-1849-selfhost-coolify-mongodb/plan.md b/plans/260627-1849-selfhost-coolify-mongodb/plan.md index 7bad10e..6e85f00 100644 --- a/plans/260627-1849-selfhost-coolify-mongodb/plan.md +++ b/plans/260627-1849-selfhost-coolify-mongodb/plan.md @@ -6,7 +6,7 @@ priority: P2 branch: "feature/selfhosted" tags: [selfhost, coolify, mongodb, migration] blockedBy: [] -blocks: [260628-1113-mongo-native-value-documents] +blocks: [260628-1113-mongo-native-value-documents, 260628-1310-flatten-mongo-value-documents] created: "2026-06-27T12:01:53.894Z" createdBy: "ck:plan" source: skill diff --git a/plans/260628-1310-flatten-mongo-value-documents/phase-01-schema-contract-and-regression-tests.md b/plans/260628-1310-flatten-mongo-value-documents/phase-01-schema-contract-and-regression-tests.md new file mode 100644 index 0000000..b0ef5e3 --- /dev/null +++ b/plans/260628-1310-flatten-mongo-value-documents/phase-01-schema-contract-and-regression-tests.md @@ -0,0 +1,136 @@ +--- +phase: 1 +title: "Schema Contract and Regression Tests" +status: pending +priority: P2 +dependencies: [] +effort: "M" +--- + +# Phase 1: Schema Contract and Regression Tests + +## Overview + +Define the exact Mongo document contract and lock it with failing tests before +changing the codec. This phase prevents a vague "flatten value" implementation +from preserving stale fields, losing type fidelity, or silently breaking legacy +reads. + +## Requirements + +- Functional: specify new raw document shapes for JSON object, array, scalar, + non-JSON string, reserved-key collision, and legacy `value` docs. +- Functional: preserve current `KVStore` and `VersionedStore` method contracts. +- Functional: prove `updatedAt` is BSON Date / Go `time.Time`. +- Non-functional: tests must be explicit enough to fail against the current + `{ _id, value, version, updatedAt int64 }` shape. + +## Architecture + +Target document examples: + +```javascript +// Normal JSON object payload, flattened. +{ + _id: "user:7", + usd: 1000.25, + assets: { BTC: 1 }, + meta: { createdAt: 1782604800000 }, + version: 4, + updatedAt: ISODate("2026-06-28T06:10:29Z"), + schemaVersion: 2 +} + +// Non-object fallback. Mongo root must be a document, so array/scalar payloads +// need a reserved payload field. This is not the old generic `value` field. +{ + _id: "daily_push:last_date", + _payload: "2026-06-28", + payloadKind: "string", + version: 1, + updatedAt: ISODate("2026-06-28T06:10:29Z"), + schemaVersion: 2 +} +``` + +Reserved fields: `_id`, `value`, `version`, `updatedAt`, `schemaVersion`, +`payloadKind`, `_payload`. If an object payload contains one of these keys, the +codec must preserve data by storing the whole object in `_payload` with +`payloadKind: "object"` instead of flattening. Object payloads with keys that +contain null, contain `.`, or start with `$` also use `_payload`; MongoDB +permits some of these in modern versions but documents restrictions, so fallback +keeps query/index behavior predictable. + +## Related Code Files + +- Modify: `internal/storage/mongodb_kv_test.go` — raw-shape, legacy-read, + stale-field, updatedAt, and CAS tests. +- Modify: `internal/storage/mongodb_value_codec.go` — only if test helpers need + exported/unexported constants clarified later. +- Read: `internal/storage/mongodb_kv.go` +- Read: `internal/storage/kv_store.go` +- Read: `plans/reports/research-260628-0605-mongodb-document-design-standards.md` + +## Tests Before + +Add/adjust Mongo integration tests gated by `MONGODB_TEST_URL`: + +1. `TestMongoKVStore_RootObjectRepresentation` + - `PutJSON` a coin-like portfolio. + - Raw Mongo doc has root `usd`, `assets`, `meta`. + - Raw Mongo doc has no `value`. + - `updatedAt` decodes as `time.Time`. +2. `TestMongoKVStore_RootObjectOverwriteRemovesStaleFields` + - Write `{a:1,b:2}` then overwrite `{a:3}`. + - Raw doc has `a`, no stale `b`. +3. `TestMongoKVStore_NonObjectPayloadFallback` + - Write bare string and JSON array. + - Raw docs use `_payload` + `payloadKind`, no `value`. + - `Get` round-trips. +4. `TestMongoKVStore_ReservedRootFieldCollision` + - Write object with `value`, `version`, or `updatedAt` in payload. + - Raw doc preserves payload under `_payload`; `GetJSON` round-trips. +5. `TestMongoKVStore_UnsafeMongoFieldNameFallback` + - Write object payloads with keys containing null, containing `.`, and + starting with `$`. + - Raw docs preserve payload under `_payload`; `GetJSON` round-trips. +6. `TestMongoKVStore_LegacyValueDocsStillRead` + - Seed old docs directly with `value` as string, `bson.Binary`, object, + and array. + - `Get`/`GetJSON` decode all. +7. Keep/strengthen `TestMongoKVStore_PutVersioned_ConcurrentCreate`. + +## Implementation Steps + +1. Add named constants in tests for reserved fields expected in the new shape. +2. Add raw Mongo assertions before changing implementation. +3. Audit current persisted structs for reserved top-level key collisions and + document the result in test comments. +4. Confirm all new tests fail for the intended reason against current code. +5. Keep existing int64-fidelity and legacy-version tests in place. + +## Tests After + +- Re-run the new Mongo storage tests after Phase 2 implementation. +- Ensure old native-`value` tests are either rewritten to the new contract or + retained only as legacy-read tests. + +## Success Criteria + +- [ ] New raw-shape tests exist and fail before implementation. +- [ ] Legacy-read coverage includes the old `value` field. +- [ ] Reserved-field coverage includes payload key `value`. +- [ ] Unsafe Mongo field-name coverage includes null, `.`, and `$` cases. +- [ ] Stale-field removal has a dedicated regression test. +- [ ] `updatedAt` BSON Date behavior has a dedicated assertion. +- [ ] Existing version-CAS tests remain in the suite. + +## Risk Assessment + +- Risk: tests assert driver-specific decoded types too tightly. Mitigation: + assert behavior and accepted BSON-decoded Go types, not exact internal map + ordering. +- Risk: reserved-field collision behavior gets forgotten because current structs + do not collide. Mitigation: synthetic collision test is required. +- Risk: stale-field issue appears only on overwrite. Mitigation: explicit + overwrite test before implementation. diff --git a/plans/260628-1310-flatten-mongo-value-documents/phase-02-root-document-storage-encoding.md b/plans/260628-1310-flatten-mongo-value-documents/phase-02-root-document-storage-encoding.md new file mode 100644 index 0000000..12b5b69 --- /dev/null +++ b/plans/260628-1310-flatten-mongo-value-documents/phase-02-root-document-storage-encoding.md @@ -0,0 +1,139 @@ +--- +phase: 2 +title: "Root Document Storage Encoding" +status: pending +priority: P2 +dependencies: [1] +effort: "L" +--- + +# Phase 2: Root Document Storage Encoding + +## Overview + +Implement the new root-payload Mongo encoding in `MongoKVStore` while preserving +all caller-facing storage interfaces. Writes should stop producing `value`, +reads should support both new root docs and legacy `value` docs. + +## Requirements + +- Functional: `Put`, `PutJSON`, `PutVersioned`, `Get`, `GetJSON`, + `GetVersioned`, `Delete`, and `List` signatures remain unchanged. +- Functional: JSON object values flatten to root unless they collide with + reserved metadata fields, the legacy `value` field, or unsafe Mongo field + names. +- Functional: arrays/scalars/non-JSON values use `_payload` and `payloadKind`, + never the old `value` field. +- Functional: `updatedAt` is stored as `time.Time`. +- Functional: `version` increments on plain and versioned writes. +- Non-functional: no stale payload fields after overwrite. + +## Architecture + +Split the codec into document-level helpers: + +```go +type payloadKind string + +const ( + payloadKindObject payloadKind = "object" + payloadKindArray payloadKind = "array" + payloadKindString payloadKind = "string" +) + +func encodeRootDocument(key string, val []byte, version int64, now time.Time) (bson.M, error) +func decodeRootDocument(key string, doc bson.M) ([]byte, error) +func payloadFieldsFromRoot(doc bson.M) bson.M +``` + +Decode order: + +1. Legacy `value` field exists -> current `decodeValue` compatibility path. +2. `_payload` exists -> decode `_payload` by `payloadKind`. +3. Else -> collect all root fields except reserved metadata and marshal as JSON. + +Write strategy: + +- `PutVersioned(expectedVersion > 0)`: `ReplaceOne({_id:key, version:expected}, replacement(version=expected+1))`. +- `PutVersioned(expectedVersion == 0)`: replace/upsert with filter matching + `{_id:key, version missing or 0}`, duplicate key -> `ErrConflict`. +- `Put`: implement as a bounded loop over `GetVersioned` + `PutVersioned` so it + overwrites unconditionally while still removing stale fields and bumping + version. Use existing retry style (`portfolioUpdateAttempts` is 5) or a + small storage-local constant. + +Use `ReplaceOne`, not partial `$set`, for root-payload writes. Partial updates +are the foot-gun: `{a:1,b:2}` overwritten by `{a:3}` would otherwise leave `b`. + +## Related Code Files + +- Modify: `internal/storage/mongodb_kv.go` — write/read methods and version + replacement logic. +- Modify: `internal/storage/mongodb_value_codec.go` — convert byte payload to + flattened root docs and reconstruct JSON. +- Modify: `internal/storage/mongodb_kv_test.go` — make Phase 1 tests pass. +- Read: `internal/storage/memory_kv.go` — keep interface behavior aligned. +- Read: `internal/storage/dynamodb_kv.go` — ensure migrate-only DynamoDB stays + unaffected. + +## Tests Before + +- Run Phase 1 Mongo tests and confirm failure before implementation. +- Run hermetic storage tests to ensure no memory/DynamoDB API changes are + required. + +## Refactor + +1. Rename legacy `decodeValue` to make compatibility explicit, e.g. + `decodeLegacyValueField`. +2. Add reserved-field constants and helper `isMongoRootMetadataField`. +3. Add root-document encoder: + - trim/decode JSON with `json.Decoder.UseNumber`; + - object -> root fields if no reserved/legacy `value` collision and no + unsafe key containing null, containing `.`, or starting with `$`; + - array -> `_payload` + `payloadKind:"array"`; + - scalar/non-JSON -> `_payload` + `payloadKind:"string"` or scalar kind; + - always add `_id`, `version`, `updatedAt`, `schemaVersion`. + +4. Add root-document decoder: + - legacy `value` first; + - `_payload` fallback second; + - root object reconstruction third. +5. Replace `$set`/`$inc` writes with whole-document replacement in + `PutVersioned`. +6. Rework `Put` to bump versions and replace whole docs without exposing + conflicts to ordinary callers unless retries are exhausted. +7. Keep `Delete` and `List` unchanged. +8. Tighten error messages so malformed root docs name the module/key and + missing payload reason. + +## Tests After + +- `go test ./internal/storage` +- `MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./internal/storage -run 'MongoKVStore'` +- `go test ./internal/modules/coin ./internal/modules/gold ./internal/modules/lolschedule` + +## Success Criteria + +- [ ] All Phase 1 tests pass. +- [ ] No new write path emits `value`. +- [ ] Payload objects with key `value` use `_payload` fallback, not root + flattening. +- [ ] Payload objects with null/`.`/`$` field names use `_payload` fallback. +- [ ] Legacy `value` docs continue to read. +- [ ] Plain `Put` and `PutVersioned` both bump `version`. +- [ ] Concurrent create/update tests keep exactly-one-winner semantics. +- [ ] Stale payload fields are removed on overwrite. + +## Risk Assessment + +- Risk: bounded `Put` loop can conflict under extreme contention. Mitigation: + keep retries small but adequate; if tests show trouble, switch to a + single-write aggregation pipeline with `$replaceRoot` and `$literal` payload + expressions. +- Risk: root reconstruction accidentally includes metadata fields. Mitigation: + centralize reserved field list and test it. +- Risk: `time.Time` changes raw `updatedAt` type. Mitigation: no app reader uses + it today; integration tests assert new date behavior. +- Risk: malformed existing docs without `value` or root payload become harder + to diagnose. Mitigation: descriptive errors in decoder. diff --git a/plans/260628-1310-flatten-mongo-value-documents/phase-03-migration-and-documentation.md b/plans/260628-1310-flatten-mongo-value-documents/phase-03-migration-and-documentation.md new file mode 100644 index 0000000..d6b156e --- /dev/null +++ b/plans/260628-1310-flatten-mongo-value-documents/phase-03-migration-and-documentation.md @@ -0,0 +1,126 @@ +--- +phase: 3 +title: "Migration and Documentation" +status: pending +priority: P2 +dependencies: [2] +effort: "M" +--- + +# Phase 3: Migration and Documentation + +## Overview + +Add an operator-safe path to rewrite existing Mongo docs from the old `value` +envelope into the new flattened root shape, and update docs that currently +describe the old layout. + +## Requirements + +- Functional: existing Atlas documents can be migrated in place without relying + on DynamoDB still existing. +- Functional: migration is idempotent and reports per-collection counts. +- Functional: migration preserves logical `Get`/`GetJSON` values and bumps or + normalizes `version` consistently. +- Non-functional: migration must not read or print secrets; `MONGO_URL` remains + secret. +- Non-functional: docs must stop calling int64 `updatedAt` TTL-ready. + +## Architecture + +Add a small Mongo-to-Mongo schema migration command rather than overloading the +DynamoDB migrator: + +```text +cmd/migrate-mongo-schema/ + main.go + main_test.go +``` + +Tool behavior: + +1. Connect using `MONGO_URL` + `MONGO_DATABASE`. +2. Iterate collections or an allow-list flag. +3. For each document: + - skip if it already has `schemaVersion >= 2` and no `value`; + - read through `MongoKVStore.GetVersioned` so legacy decode paths are used; + - rewrite through `PutVersioned` so root encoding is shared with app writes; + - on `ErrConflict`, re-read the document: skip it if a live app write + already migrated it, otherwise retry up to a small bounded limit; + - count migrated/skipped/errors. +4. `--dry-run` prints counts without writes. +5. `--verify` confirms no documents with `value` remain, except explicitly + allowed malformed docs if any are reported. + +Do not drop or rename collections. Keep one collection per module. + +## Related Code Files + +- Create: `cmd/migrate-mongo-schema/main.go` +- Create: `cmd/migrate-mongo-schema/main_test.go` +- Modify: `Makefile` — add a migration target, e.g. `migrate-mongo-schema`. +- Modify: `README.md` — update storage layout summary. +- Modify: `docs/deploy-coolify-selfhosted.md` — update Atlas storage layout + and migration note. +- Modify: `cmd/migrate-dynamo-to-mongo/README.md` — update destination shape. +- Modify: `cmd/migrate-dynamo-to-mongo/main.go` comments only if stale. +- Read: `cmd/migrate-dynamo-to-mongo/main.go` — reuse connection/config style. + +## Tests Before + +- Add migrator e2e test that seeds old `value` docs directly, runs migration, + then asserts: + - logical values round-trip; + - raw docs have no `value`; + - `updatedAt` is `time.Time`; + - counts match. + +## Implementation Steps + +1. Implement CLI flags: + - `--dry-run` + - `--verify` + - `--collection` optional repeat/single collection selector. +2. Reuse `storage.NewMongoClient`, `storage.NewMongoDatabase`, and + `storage.NewMongoProvider`. +3. Add collection listing using the Mongo driver, excluding system collections. +4. Rewrite docs through the storage layer, not raw BSON mutation. +5. Print a concise table: collection, total, migrated, skipped, errors. +6. Add Makefile target with env examples. +7. Update docs and remove stale wording: + - no old `value` envelope in new writes; + - `updatedAt` is BSON Date; + - legacy `value` docs dual-read until migrated. +8. Add migration conflict tests: simulate a version change between read and + write; assert the migrator re-reads and either skips already-migrated docs or + retries legacy docs without clobbering the concurrent update. + + +## Tests After + +- `go test ./cmd/migrate-mongo-schema` +- `MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./cmd/migrate-mongo-schema ./internal/storage` +- `make test` + +## Success Criteria + +- [ ] In-place migration rewrites old `value` docs without changing logical app + values. +- [ ] Migration is idempotent. +- [ ] `--dry-run` performs no writes. +- [ ] `--verify` exits non-zero when old `value` docs remain. +- [ ] Migration handles version conflicts by re-reading and never clobbers a + concurrent app write. +- [ ] README and Coolify deploy docs show the new flattened layout. +- [ ] DynamoDB-to-Mongo migrator docs no longer mention BSON binary/string as + the destination `value` representation. + +## Risk Assessment + +- Risk: migration touches live data. Mitigation: dry-run + verify + local e2e + test; recommend backup/export before production run. +- Risk: old DynamoDB source may be gone. Mitigation: in-place Mongo migration is + independent of DynamoDB. +- Risk: malformed documents stop migration. Mitigation: report collection/key + and continue only if an explicit `--continue-on-error` is later accepted; + default should fail loud. diff --git a/plans/260628-1310-flatten-mongo-value-documents/phase-04-verification-and-rollout.md b/plans/260628-1310-flatten-mongo-value-documents/phase-04-verification-and-rollout.md new file mode 100644 index 0000000..58706bc --- /dev/null +++ b/plans/260628-1310-flatten-mongo-value-documents/phase-04-verification-and-rollout.md @@ -0,0 +1,121 @@ +--- +phase: 4 +title: "Verification and Rollout" +status: pending +priority: P2 +dependencies: [3] +effort: "M" +--- + +# Phase 4: Verification and Rollout + +## Overview + +Run the full validation suite, check raw Mongo documents, and define the safe +production rollout order. This phase is the gate before deploying a storage +schema change. + +## Requirements + +- Functional: app behavior remains unchanged for all Telegram commands. +- Functional: Mongo raw shape is verified for representative module documents. +- Functional: legacy docs remain readable before migration and absent after + migration verify. +- Non-functional: rollout includes backup, deploy, migration, verify, and + rollback notes. + +## Architecture + +Rollout order: + +1. Backup/export current Atlas database or snapshot if available. +2. Deploy app version with dual-read + new-write root encoding. +3. Let app run; new/touched docs rewrite naturally. +4. Run `migrate-mongo-schema --dry-run`. +5. Run `migrate-mongo-schema`. +6. Run `migrate-mongo-schema --verify`. +7. Inspect sample docs in Atlas/Compass: + - coin/gold/stock portfolio root fields; + - wordle/loldle/twentyq game state root fields; + - lolschedule scalar date guard under `_payload`; + - no old `value` on migrated docs. + +Rollback: + +- If the new app fails before it handles writes, roll back binary; old `value` + docs are still readable by the previous build. +- Once the new app has handled writes, rollback to the previous binary is unsafe + for any touched flattened docs. Prefer fixing forward, or restore the backup. +- If migration ran, previous build may not read flattened docs at all. Rollback + after migration requires restoring backup or a reverse migrator. Prefer fixing + forward unless the migration corrupted data. + +## Related Code Files + +- Modify: tests only if verification gaps are found. +- Read: `Makefile` +- Read: `README.md` +- Read: `docs/deploy-coolify-selfhosted.md` +- Read: `cmd/migrate-mongo-schema/main.go` +- Read: `internal/storage/mongodb_kv_test.go` + +## Tests Before + +- Confirm Phase 1-3 tests run locally. +- Start local Mongo with `make mongo-local` for integration tests. + +## Implementation Steps + +1. Run focused storage tests. +2. Run module tests that rely on versioned storage flows: + - coin portfolio; + - gold portfolio; + - lolschedule daily-push claim. +3. Run full hermetic suite. +4. Run Mongo integration suite. +5. Run DynamoDB tests only if the DynamoDB migrator or shared storage contracts + changed. +6. Manually inspect raw docs or add a small test assertion report for: + - no `value`; + - expected root fields; + - Date `updatedAt`; + - stable `version`. +7. Document production run commands in the final completion report. + +## Tests After + +Required gates: + +```sh +make vet +make test +MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./internal/storage ./cmd/migrate-mongo-schema +``` + +Conditional gate if DynamoDB migrator comments/tests changed: + +```sh +make test-dynamodb +``` + +## Success Criteria + +- [ ] `make vet` passes. +- [ ] `make test` passes. +- [ ] Mongo integration tests pass against local Mongo. +- [ ] New migrator e2e test passes. +- [ ] Raw docs verified no longer use `value` after migration. +- [ ] Production rollout notes include backup and rollback constraints. +- [ ] No user-visible Telegram command behavior changes. + +## Risk Assessment + +- Risk: previous binary cannot read newly-written or post-migration flattened + docs. Mitigation: deploy dual-read binary first; backup before rollout; + rollback old binary only before new writes happen; fix forward preferred. +- Risk: Atlas M0 resource limits during migration. Mitigation: tiny dataset + expected; still scan collection-by-collection and report progress. +- Risk: hidden module stores non-object values. Mitigation: fallback `_payload` + contract and migration tests for scalar/array docs. +- Risk: docs drift. Mitigation: grep for old `{ _id, value, version, updatedAt }` + wording before finalizing. diff --git a/plans/260628-1310-flatten-mongo-value-documents/plan.md b/plans/260628-1310-flatten-mongo-value-documents/plan.md new file mode 100644 index 0000000..f4bfe21 --- /dev/null +++ b/plans/260628-1310-flatten-mongo-value-documents/plan.md @@ -0,0 +1,221 @@ +--- +title: "Flatten Mongo value documents" +description: "Remove Mongo's generic value envelope for JSON object values, keep KVStore callers stable, and store Mongo metadata in queryable root fields." +status: superseded +supersededBy: 260628-1318-mongo-native-typed-stores +priority: P2 +branch: "feature/selfhosted" +tags: [database, mongodb, refactor, storage, tdd] +blockedBy: [260627-1849-selfhost-coolify-mongodb] +blocks: [] +created: "2026-06-28T06:10:29.184Z" +createdBy: "ck:plan" +source: skill +--- + +# Flatten Mongo value documents + +> **Superseded (2026-06-28)** by +> [`260628-1318-mongo-native-typed-stores`](../260628-1318-mongo-native-typed-stores/plan.md). +> User chose the full Mongo-native direction this plan deferred: delete the +> `KVStore` abstraction in favor of typed stores, MongoDB-only runtime (memory +> kept for tests), and drop the legacy `value` dual-read + in-place Mongo +> migrator (Atlas is empty until cutover, so the DynamoDB→Mongo migrator writes +> the final flattened shape directly). Kept for history; do not implement. + +## Overview + +Change MongoDB's on-disk KV document shape from a generic envelope: + +```javascript +{ _id, value: { ... }, version, updatedAt } +``` + +to a more Mongo-native root-document shape for JSON object values: + +```javascript +{ _id, ...payloadFields, version, updatedAt: ISODate(...), schemaVersion } +``` + +Keep the public `KVStore` / `VersionedStore` interfaces stable so modules keep +using `GetJSON`, `PutJSON`, `GetVersioned`, and `PutVersioned`. This plan is a +storage-representation refactor, not a full rewrite to typed repositories. + +Non-object values cannot be Mongo root documents. New writes for arrays/scalars +use a small reserved `_payload` fallback with `payloadKind`, but **new writes +must not use the old `value` field**. Legacy docs with `value` remain readable +and are rewritten to the new shape by normal updates or by the in-place +migration tool. + +## Scope Challenge + +- Existing code: `MongoKVStore` already owns all Mongo encoding/decoding and + version CAS; `mongodb_value_codec.go` already preserves `int64` via + `json.Number`; modules do not need direct Mongo access. +- Minimum changes: storage codec + raw-shape tests + migrator/docs. Defer + per-module typed repositories until there is a real query/report need. +- Complexity: touches storage, one migration command, docs, and tests. Avoid + changing module handlers or public storage interfaces. +- Selected mode: HOLD SCOPE. Refactor the persisted Mongo shape; do not expand + into a full domain repository rewrite. + +## Design Decision + +Chosen approach: **root-payload KV documents**. + +Why not full typed repositories now: it would touch most modules, duplicate +test plumbing, and remove the memory backend value before there is a query need. +Why not keep `value`: it defeats the user's Mongo document-design goal and +keeps domain fields one level away from simple Compass/query/index use. + +Reserved root fields: + +| Field | Purpose | +|---|---| +| `_id` | Module-local KV key, unchanged | +| `value` | Legacy envelope field; forbidden in new root payloads | +| `version` | Optimistic-lock token, unchanged semantics | +| `updatedAt` | BSON Date (`time.Time`), not unix nanos | +| `schemaVersion` | Storage document shape version | +| `payloadKind` | Only needed for non-object fallback or collision fallback | +| `_payload` | Array/scalar payload fallback; never used for ordinary JSON objects | + +Mongo field-name guard: object payloads also fall back to `_payload` if any +flattened key contains the null character, contains `.`, or starts with `$`. +MongoDB permits `.` and `$` in modern versions but documents restrictions around +them; fallback keeps normal query/index/schema behavior predictable. +Reference: https://www.mongodb.com/docs/manual/core/document/#field-names + +Decoder rules: + +1. If old `value` exists, decode using the legacy path. +2. Else if `_payload` exists, decode `_payload` according to `payloadKind`. +3. Else reconstruct the JSON object from all non-reserved root fields. + +Writer rules: + +1. JSON object without reserved-field collision writes payload fields at root. +2. JSON array, scalar, invalid JSON, or object with reserved-field collision + or unsafe Mongo field names writes `_payload` + `payloadKind`. +3. Writes replace the whole document (except the `_id` key value is preserved in + the replacement) so stale old payload fields cannot survive an overwrite. +4. `PutVersioned` keeps single-document CAS semantics by filtering on `_id` and + `version`. +5. Plain `Put` must still bump `version`; implement as a bounded versioned + write loop or a carefully tested aggregation update pipeline. Prefer the + simpler loop unless tests prove unacceptable contention. + +## Cross-Plan Dependencies + +| Relationship | Plan | Status | Rationale | +|---|---|---|---| +| Blocked by | `260627-1849-selfhost-coolify-mongodb` | in-progress | Created the Mongo provider, migrator, and self-host runtime this refactor modifies. | +| Supersedes storage choice from | `260628-1113-mongo-native-value-documents` | completed | Keeps its version CAS + native BSON goal, but replaces the `value` envelope with root payload fields. | + +## Phases + +| Phase | Name | Status | +|-------|------|--------| +| 1 | [Schema Contract and Regression Tests](./phase-01-schema-contract-and-regression-tests.md) | Pending | +| 2 | [Root Document Storage Encoding](./phase-02-root-document-storage-encoding.md) | Pending | +| 3 | [Migration and Documentation](./phase-03-migration-and-documentation.md) | Pending | +| 4 | [Verification and Rollout](./phase-04-verification-and-rollout.md) | Pending | + +## Dependencies + +- Existing Mongo provider and version-CAS code from + `plans/260628-1113-mongo-native-value-documents/`. +- Research report: + `plans/reports/research-260628-0605-mongodb-document-design-standards.md`. +- MongoDB official guidance used in the research report: + - https://www.mongodb.com/docs/manual/data-modeling/ + - https://www.mongodb.com/docs/manual/data-modeling/best-practices/ + - https://www.mongodb.com/docs/manual/core/write-operations-atomicity/ + - https://www.mongodb.com/docs/manual/core/index-ttl/ + +## Acceptance Criteria + +- [ ] New JSON object writes contain no top-level `value` field; payload fields + live at root with `_id`, `version`, `updatedAt`, and `schemaVersion`. +- [ ] Payload objects with top-level `value`, reserved metadata keys, null + characters, `.`, or `$`-prefixed keys use `_payload` fallback and round-trip. +- [ ] New array/scalar/non-JSON writes contain no old `value` field; they use + `_payload` + `payloadKind` and round-trip byte-equivalent where required. +- [ ] `updatedAt` stores as BSON Date / Go `time.Time`, not int64 nanos. +- [ ] Legacy docs with `value` as string, binary, object, or array still read. +- [ ] Overwriting a document removes stale payload fields from prior values. +- [ ] Versioned writes remain single-winner under concurrent create/update. +- [ ] A Mongo in-place migration can rewrite existing `value` docs to the new + shape without changing logical values or document counts. +- [ ] `make test`, `make vet`, and Mongo integration tests pass. + +## Not In Scope + +- Replacing every module's `KVStore` usage with typed Mongo repositories. +- Adding Mongo schema validation/indexes for every module. +- Deleting DynamoDB migrator support. +- Changing Telegram bot behavior or command outputs. + +## Validation Log + +### Session 1 — 2026-06-28 +**Trigger:** `/ck:plan validate /config/workspace/tiennm99/miti99bot/plans/260628-1310-flatten-mongo-value-documents/plan.md` +**Questions asked:** 0 — validation found technical plan gaps with clear repo/doc evidence; no user-facing trade-off needed. + +#### Plan Recap + +- Refactor Mongo on-disk KV documents from `{ _id, value, version, updatedAt }` + to root payload fields. +- Keep `KVStore` / `VersionedStore` caller contracts stable. +- Keep legacy `value` docs readable. +- Use `_payload` only for non-object or unsafe/colliding object payloads. +- Store `updatedAt` as BSON Date. +- Add an in-place Mongo schema migration and rollout guardrails. + +#### Verification Results + +- **Tier:** Standard (4 phases) +- **Claims checked:** 32 +- **Verified:** 29 | **Failed:** 3 | **Unverified:** 0 + +Failures fixed in this validation: + +1. [Contract Verifier] Reserved fields omitted `value`. Evidence: + `internal/storage/mongodb_kv.go:19` uses `mongoValueField = "value"` and + `decodeValue` treats any top-level `value` as legacy payload + (`internal/storage/mongodb_kv.go:46`). A flattened payload containing + top-level `value` would violate the no-`value` contract and decode wrong. +2. [Fact Checker] Field-name guard omitted null/`.`/`$` cases. Evidence: + MongoDB documents forbid null in field names and document restrictions for + `.`/`$` field names: https://www.mongodb.com/docs/manual/core/document/#field-names. + The plan now falls back to `_payload` for those keys. +3. [Contract Verifier] Migration rewrite via `PutVersioned` lacked conflict + retry handling. Evidence: `PutVersioned` returns `ErrConflict` on version + mismatch (`internal/storage/kv_store.go:35`) and concurrent app writes are + possible after deploying the dual-read/new-write binary. Phase 3 now requires + re-read/skip/retry behavior. + +#### Confirmed Decisions + +- Keep `value` reserved forever for backward-compatible legacy decode. +- Use `_payload` fallback for reserved keys and Mongo-awkward field names. +- Keep migration writes through storage encoding and handle conflicts by + re-reading, skipping already-migrated docs, or retrying. + +#### Impact on Phases + +- Phase 1: add tests for payload key `value`, null/`.`/`$` field names, and + migration conflict setup. +- Phase 2: update reserved-field and unsafe-field encoder rules. +- Phase 3: add conflict retry semantics to the migration command. + +### Whole-Plan Consistency Sweep + +- Files reread: `plan.md`, all four `phase-*.md` files. +- Decision deltas checked: 3 (`value` reserved, unsafe field fallback, migration conflict retry). +- Reconciled stale references: 6. +- Unresolved contradictions: 0. + +## Open Questions + +None. diff --git a/plans/260628-1318-mongo-native-typed-stores/phase-01-typed-docstore-and-memory-backend.md b/plans/260628-1318-mongo-native-typed-stores/phase-01-typed-docstore-and-memory-backend.md new file mode 100644 index 0000000..96e2803 --- /dev/null +++ b/plans/260628-1318-mongo-native-typed-stores/phase-01-typed-docstore-and-memory-backend.md @@ -0,0 +1,107 @@ +--- +phase: 1 +title: "Typed DocStore contract + memory backend" +status: done +priority: P2 +dependencies: [] +effort: "M" +--- + +# Phase 1: Typed DocStore contract + memory backend + +## Overview + +Define the new typed storage contract and a memory implementation first, with +tests, before touching Mongo or any module. This nails the interface every +module and the Mongo backend will target. + +## Requirements + +- `DocStore[T]` generic interface: `Get`, `Put`, `PutVersioned`, `Delete`, `List`. +- `Provider.Collection(module)` returns an opaque per-module `Collection`. +- `Typed[T](Collection) DocStore[T]` free function (methods cannot be generic). +- Memory implementation: hermetic, concurrency-safe, version-CAS correct. +- Reserved-field name guard: payload BSON tags must not collide with `_id`, + `version`, `updatedAt`. +- `ErrNotFound`, `ErrConflict` reused (move to this package if `kv_store.go` is deleted later). + +## Architecture + +```go +// doc_store.go +type DocStore[T any] interface { + Get(ctx, id) (T, int64, error) + Put(ctx, id, T) error + PutVersioned(ctx, id, expectedVersion int64, T) error + Delete(ctx, id) error + List(ctx, prefix string) ([]string, error) +} + +type Provider interface{ Collection(module string) Collection } +type Collection interface{ collection() } // sealed marker + +func Typed[T any](c Collection) DocStore[T] // switch concrete type → mongo|memory store +``` + +Memory backend: + +```go +// memory_provider.go (replaces kv_provider.go's MemoryProvider) +type MemoryProvider struct{ mu sync.Mutex; cols map[string]*memoryCollection } +type memoryCollection struct{ mu sync.Mutex; rows map[string]memoryRow } +type memoryRow struct{ version int64; val any } // val holds the typed value + +type MemoryDocStore[T any] struct{ c *memoryCollection } +``` + +- `Get`: type-assert `row.val.(T)`; missing → `ErrNotFound`. +- `Put`: overwrite, `version++`. +- `PutVersioned`: compare stored version to expected; mismatch → `ErrConflict`; + `expected==0` requires absent/zero-version row. +- `Delete`: idempotent. +- `List(prefix)`: sorted keys with prefix (reuse `validatePrefix`). + +Keys: keep `validateKey`/`validatePrefix` from `keys.go`/`prefix.go`. + +## Related Code Files + +- Create: `internal/storage/doc_store.go` (interface, Provider, Collection, `Typed`, errors). +- Create: `internal/storage/memory_provider.go` (MemoryProvider + memoryCollection + MemoryDocStore). +- Create: `internal/storage/memory_doc_store_test.go`. +- Create: `internal/storage/reserved_fields.go` + test (reserved-name guard helper). +- Read: `internal/storage/keys.go`, `prefix.go`, `kv_store.go` (for error vars to migrate). + +## Tests Before + +Write failing tests first: + +1. `TestMemoryDocStore_PutGetRoundTrip` — typed struct round-trips; version starts at 1. +2. `TestMemoryDocStore_PutBumpsVersion` — repeated Put increments version. +3. `TestMemoryDocStore_PutVersioned_CAS` — stale expected → `ErrConflict`; fresh succeeds. +4. `TestMemoryDocStore_PutVersioned_CreateZero` — `expected==0` creates; second create → `ErrConflict`. +5. `TestMemoryDocStore_Delete_Idempotent`. +6. `TestMemoryDocStore_List_Prefix` — disjoint key prefixes isolate types in one collection. +7. `TestReservedFields_RejectsCollision` — a payload type tagged `version` is flagged. +8. `TestMemoryProvider_CollectionIsolation` — two modules' collections don't share keys. + +## Implementation Steps + +1. Add `doc_store.go` with interface, sealed `Collection`, `Typed[T]`, and the + `ErrNotFound`/`ErrConflict` vars (kept here so `kv_store.go` can be deleted). +2. Implement memory provider + store. +3. Implement reserved-field reflection guard (`bson` tag scan). +4. Make all Phase 1 tests pass. + +## Success Criteria + +- [ ] `DocStore[T]`, `Provider`, `Collection`, `Typed` compile and are documented. +- [ ] Memory store passes CAS/version/list/delete tests. +- [ ] Reserved-name guard catches `_id`/`version`/`updatedAt` collisions. +- [ ] No dependency on the soon-to-be-deleted byte `KVStore`. + +## Risk Assessment + +- Risk: generic free-function `Typed[T]` is unusual. Mitigation: document why + (Go methods can't be generic); single switch point, well tested. +- Risk: memory `val any` type mismatch panics. Mitigation: `Get` returns a typed + error on failed assertion, not a panic. diff --git a/plans/260628-1318-mongo-native-typed-stores/phase-02-mongo-typed-store.md b/plans/260628-1318-mongo-native-typed-stores/phase-02-mongo-typed-store.md new file mode 100644 index 0000000..6c076c2 --- /dev/null +++ b/plans/260628-1318-mongo-native-typed-stores/phase-02-mongo-typed-store.md @@ -0,0 +1,116 @@ +--- +phase: 2 +title: "Mongo typed store + integration tests" +status: done +priority: P2 +dependencies: [1] +effort: "L" +--- + +# Phase 2: Mongo typed store + integration tests + +## Overview + +Implement `MongoDocStore[T]` and a `MongoProvider` that satisfies the Phase 1 +contract, persisting typed payloads as flattened native root documents. Lock the +raw shape and CAS semantics with Mongo integration tests. + +## Requirements + +- `MongoProvider.Collection(module)` returns a Mongo-backed `Collection`. +- `Typed[T]` over a Mongo collection yields `MongoDocStore[T]`. +- Write shape: `{ _id, ...payload, version, updatedAt(Date) }`; no `value`, no `_payload`. +- Whole-document `ReplaceOne` (no `$set`) so overwrites drop stale fields. +- Version CAS identical in behavior to the current `MongoKVStore`. +- `updatedAt` stored as `time.Time`. +- Plain `Put` bumps version via a bounded versioned-write retry loop. + +## Architecture + +```go +type storedDoc[T any] struct { + ID string `bson:"_id"` + Version int64 `bson:"version"` + UpdatedAt time.Time `bson:"updatedAt"` + Payload T `bson:",inline"` +} + +type MongoDocStore[T any] struct{ coll *mongo.Collection } +``` + +- `Get`: `FindOne({_id})` → decode `storedDoc[T]`; `ErrNoDocuments`→`ErrNotFound`; + return `Payload`, `Version`. +- `PutVersioned(expected>0)`: `ReplaceOne({_id, version:expected}, storedDoc{version:expected+1, now, payload})`; + `MatchedCount==0`→`ErrConflict`. +- `PutVersioned(0)`: upsert filter `{_id, $or:[version absent, version:0]}` with + replacement `version:1`; duplicate-key→`ErrConflict`. (Port existing logic.) +- `Put`: loop `Get`→`PutVersioned(version)` up to N attempts (reuse the small + retry constant style already in coin/gold); first write uses `expected=0`. +- `Delete`: `DeleteOne` idempotent. +- `List(prefix)`: half-open `_id` range scan via `prefixSuccessor`, project `_id` + only. (Port from current `MongoKVStore.List`.) +- Error messages name module + key + reason. + +`MongoProvider` wraps `*mongo.Database`; `Collection(module)` validates the +module name (reuse existing validation) and returns `mongoCollection{db.Collection(module)}`. + +## Related Code Files + +- Create: `internal/storage/mongo_doc_store.go` (MongoDocStore + storedDoc). +- Modify: `internal/storage/mongodb_provider.go` (implement new `Provider`/`Collection`). +- Create: `internal/storage/mongo_doc_store_test.go` (integration, gated by `MONGODB_TEST_URL`). +- Delete (this phase): `mongodb_kv.go`, `mongodb_value_codec.go`, `mongodb_kv_test.go`. +- Read: current `mongodb_kv.go` (port CAS + List), `mongodb_provider.go`. + +## Tests Before + +Gated by `MONGODB_TEST_URL`: + +1. `TestMongoDocStore_RootShape` — Put a portfolio-like struct; raw `bson.M` has + root payload fields, `version`, `updatedAt`; **no** `value`, **no** `_payload`; + `updatedAt` decodes as `time.Time`. +2. `TestMongoDocStore_OverwriteRemovesStaleFields` — write struct A with field + set X, overwrite with struct B lacking some of X; raw doc has no stale field. +3. `TestMongoDocStore_PutVersioned_ConcurrentCreate` — N goroutines create same + `_id`; exactly one wins, rest get `ErrConflict`. +4. `TestMongoDocStore_PutVersioned_Update` — stale expected → `ErrConflict`. +5. `TestMongoDocStore_Put_BumpsVersion` — plain Put increments version, survives + a concurrent bump (retry loop succeeds or conflicts deterministically). +6. `TestMongoDocStore_List_Prefix`. +7. `TestMongoDocStore_WrappedScalar` — a `struct{ Date string }` payload stores + `date` at root (proves the lolschedule wrapping works). +8. `TestMongoDocStore_WrappedArray` — a `struct{ Subscribers []sub }` payload + stores `subscribers` array at root. + +## Implementation Steps + +1. Implement `MongoDocStore[T]` (port CAS + List from `MongoKVStore`, typed). +2. Reshape `MongoProvider` to the new `Provider`/`Collection`; wire `Typed[T]`. +3. Delete `mongodb_kv.go`, `mongodb_value_codec.go` and their tests. +4. `go build ./internal/storage` — expect module/server breakage deferred to + Phase 3 (storage package itself must compile). +5. Make Phase 2 integration tests pass against `make mongo-local`. + +## Tests After + +```sh +go test ./internal/storage +MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./internal/storage -run 'MongoDocStore' +``` + +## Success Criteria + +- [ ] Raw Mongo doc is flattened root payload + `version` + Date `updatedAt`; no `value`/`_payload`. +- [ ] Overwrite removes stale fields. +- [ ] Concurrent create/update keep single-winner CAS. +- [ ] Wrapped scalar/array payloads store as named root fields. +- [ ] Old `MongoKVStore`/codec deleted; storage package builds. + +## Risk Assessment + +- Risk: `bson:",inline"` edge cases (pointer payloads, embedded maps). Mitigation: + require payload `T` to be a concrete struct; test scalar/array wrappers. +- Risk: bounded `Put` loop contention. Mitigation: small retry count; CAS users + call `PutVersioned` directly anyway. +- Risk: driver decodes `version` as int32. Mitigation: `storedDoc.Version int64` + + decode test. diff --git a/plans/260628-1318-mongo-native-typed-stores/phase-03-migrate-modules-and-wiring.md b/plans/260628-1318-mongo-native-typed-stores/phase-03-migrate-modules-and-wiring.md new file mode 100644 index 0000000..4a0d400 --- /dev/null +++ b/plans/260628-1318-mongo-native-typed-stores/phase-03-migrate-modules-and-wiring.md @@ -0,0 +1,115 @@ +--- +phase: 3 +title: "Migrate modules + wiring; delete KVStore" +status: done +priority: P2 +dependencies: [2] +effort: "XL" +--- + +# Phase 3: Migrate modules + wiring; delete KVStore + +## Overview + +Switch every module from the byte `KVStore` to typed `DocStore[T]`, rewire +`Deps`/registry/server, and delete the old byte abstraction and DynamoDB runtime +backend. This is the largest phase; do it module-by-module with tests green +after each. + +## Requirements + +- `Deps.KV storage.KVStore` → `Deps.Store storage.Collection`. +- `registry.Build` takes `storage.Provider` instead of `storage.KVProvider`. +- Each module builds typed stores via `storage.Typed[T](deps.Store)`. +- CAS behavior preserved for coin, gold, lolschedule last-push. +- `lolschedule` subscribers + last-push wrapped in named structs. +- No remaining references to `KVStore`, `VersionedStore`, `PutJSON`, `GetJSON`, + `kv.Put`/`kv.Get` byte calls, or the DynamoDB runtime backend. +- `cmd/server/main.go` `buildProvider` returns a Mongo `Provider` (or memory for + `MODULES=`/no-`MONGO_URL` local runs); the `dynamodb` runtime branch is removed. + +## Per-module migration map + +| Module | Keys / payload types | Store ops | Notes | +|---|---|---|---| +| coin | `Portfolio` (per user) | Get/PutVersioned (CAS loop) | typed CAS via `DocStore[Portfolio]` | +| gold | `Portfolio` (per user) | Get/PutVersioned (CAS loop) | same | +| stock | `Portfolio` (per user) | Get/Put | no CAS today; keep plain Put | +| wordle | `GameState`, `Stats` | Get/Put | two typed views, disjoint key prefixes | +| loldle | `gameState`, `stats`, `roundConfig` | Get/Put/Delete | three typed views | +| twentyq | `GameState`, `Stats` | Get/Put/Delete | two typed views | +| lolschedule | `subscribers` (wrap `[]Subscriber`), `lastPush` (wrap date string) | List/Put + CAS on last-push | **wrap array+scalar in named structs** | +| stats | counter entry struct(s) | Get/Put/List | List-heavy (`views.go`) | +| misc | `lastPing` struct | Get/Put | | +| util | none | — | no storage | + +`lolschedule` wrappers: + +```go +type subscribersDoc struct { Subscribers []Subscriber `bson:"subscribers" json:"subscribers"` } +type lastPushDoc struct { Date string `bson:"date" json:"date"` } +``` + +`claimDailyPush` uses `DocStore[lastPushDoc].Get`/`PutVersioned` for the +single-winner daily claim (replaces the current `VersionedStore` byte path). + +## Related Code Files + +- Modify: `internal/modules/module.go` (Deps.Store), `registry.go` (Build signature, `Collection(name)`). +- Modify: every module's storage-touching file (see map) + their factories + (`coin.go`, `gold.go`, … `New(deps)` build typed stores). +- Modify: `cmd/server/main.go` `buildProvider` (Mongo|memory only). +- Modify: all module `*_test.go` that construct stores (use `MemoryProvider`/`Typed`). +- Delete: `kv_store.go`, `kv_provider.go`, `memory_kv.go`, `memory_kv_test.go`, + `dynamodb_kv.go`, `dynamodb_provider.go`, `dynamodb_kv_test.go`, + `dynamodb_provider_test.go`, `invalid_store.go`. + +## Tests Before / After (per module) + +For each module: update its tests to build a `MemoryProvider`, get +`deps.Store = provider.Collection(name)`, then run existing behavior assertions. +Run `go test ./internal/modules//...` green before moving on. Order: +misc → stock → wordle → loldle → twentyq → coin → gold → stats → lolschedule +(simplest first; CAS + List modules last). + +After all modules: + +```sh +go build ./... +go test ./internal/modules/... +grep -rn "KVStore\|VersionedStore\|PutJSON\|GetJSON\|\.For(" internal/ cmd/ # expect: none (except migrator handled in Phase 4) +``` + +## Implementation Steps + +1. Add `Deps.Store storage.Collection`; change `registry.Build` to take + `storage.Provider` and set `Deps.Store = provider.Collection(name)`. +2. Migrate modules one at a time per the order above; keep the suite green. +3. For coin/gold, replace the `kv.(VersionedStore)` assertion + byte CAS loop + with `DocStore[Portfolio]` Get/PutVersioned (same retry count). +4. For lolschedule, introduce the wrapper structs and migrate subscribers (List + + Put) and last-push (CAS). +5. Update `cmd/server/main.go`: drop the `dynamodb` runtime branch; auto-detect = + Mongo when `MONGO_URL` set, else memory (warn). Keep secret-safe logging. +6. Delete the dead files listed above; ensure `go build ./...` is clean. +7. Run full module + storage suites. + +## Success Criteria + +- [ ] All modules compile and pass tests using `DocStore[T]`. +- [ ] No `KVStore`/`VersionedStore`/`PutJSON`/`GetJSON` references outside the migrator. +- [ ] DynamoDB runtime backend deleted; `dynamodb_client.go` retained for migrator. +- [ ] coin/gold/lolschedule CAS behavior unchanged (tests prove single-winner). +- [ ] lolschedule subscribers/last-push persist as named root fields. +- [ ] `cmd/server` builds a Mongo-only runtime (memory fallback for local/no-DB). + +## Risk Assessment + +- Risk: large blast radius. Mitigation: strict module-by-module order, suite + green between each; no behavior changes, only the store type. +- Risk: a module silently relied on byte-identical round-trip. Mitigation: typed + structs already define the JSON contract; assert decoded values, not bytes. +- Risk: stats List semantics differ. Mitigation: `List(prefix)` ported verbatim; + reuse stats' existing key prefixes. +- Risk: cron Deps scoping regresses. Mitigation: keep registry's per-module Deps + cloning; `Deps.Store` is already per-module. diff --git a/plans/260628-1318-mongo-native-typed-stores/phase-04-dynamo-migrator-and-docs.md b/plans/260628-1318-mongo-native-typed-stores/phase-04-dynamo-migrator-and-docs.md new file mode 100644 index 0000000..7e4c11b --- /dev/null +++ b/plans/260628-1318-mongo-native-typed-stores/phase-04-dynamo-migrator-and-docs.md @@ -0,0 +1,115 @@ +--- +phase: 4 +title: "DynamoDB→Mongo migrator + docs" +status: done +priority: P2 +dependencies: [3] +effort: "M" +--- + +# Phase 4: DynamoDB→Mongo migrator + docs + +## Overview + +Keep `cmd/migrate-dynamo-to-mongo` working, but make it write the new flattened +native shape directly. The migrator is schema-agnostic over modules, so it +cannot use the typed `DocStore[T]` (it has no compile-time `T` per row). It +writes documents with a small generic JSON→root-document encoder plus a tiny +explicit table for the only two non-object values. + +## Requirements + +- Migrator still Scans DynamoDB (read-only, `dynamodb:Scan` only) — unchanged. +- It writes `{ _id: sk, ...payloadFields, version:1, updatedAt(Date) }` per item. +- JSON object values flatten to root. +- The two known non-object DynamoDB values are wrapped to match the module's + typed shape: + - `lolschedule` subscribers (JSON array) → `{ subscribers: [...] }` + - `lolschedule` last-push date (JSON/scalar string) → `{ date: "..." }` +- Idempotent (re-run overwrites by `_id`, never duplicates). +- `--dry-run` (counts only) and `--verify` (per-collection count parity) kept. +- No secrets read/printed; `MONGO_URL` stays secret. + +## Architecture + +Add a migrator-local encoder (NOT in `internal/storage`, to keep the runtime +typed and the generic JSON path confined to migration): + +```go +// cmd/migrate-dynamo-to-mongo/encode.go +func rootDocForItem(module, key string, value []byte, now time.Time) (bson.M, error) +``` + +- Decode `value` with `json.Decoder.UseNumber` (int64 fidelity). +- If a `(module,key-prefix)` wrap rule matches → `{ wrapField: decoded }`. +- Else if decoded is a JSON object → spread its fields at root (reject keys + colliding with `_id`/`version`/`updatedAt`; none expected — fail loud if seen). +- Else → error (an unexpected non-object value means a new wrap rule is needed; + fail loud rather than guess). +- Always set `_id`, `version:int64(1)`, `updatedAt: now`. + +Wrap rules table (explicit, documented): + +```go +var wrapRules = []struct{ module, keyPrefix, field string }{ + {"lolschedule", "subscribers", "subscribers"}, + {"lolschedule", "last_push_date", "date"}, +} +``` + +Write with `Collection(module).ReplaceOne({_id:key}, doc, upsert=true)` directly +via the driver (the migrator already holds `*mongo.Database`). + +## Related Code Files + +- Modify: `cmd/migrate-dynamo-to-mongo/main.go` — replace `provider.For(pk).Put` + with `rootDocForItem` + `ReplaceOne` upsert; validate module/key names inline + (the old `For` returned an erroring store for bad names — replicate that guard). +- Create: `cmd/migrate-dynamo-to-mongo/encode.go` + `encode_test.go`. +- Modify: `cmd/migrate-dynamo-to-mongo/main_test.go` — assert flattened shape + + wrapped lolschedule docs. +- Modify: `cmd/migrate-dynamo-to-mongo/README.md` — new destination shape. +- Modify: `README.md` (storage layout summary), `docs/deploy-coolify-selfhosted.md` + (Atlas layout + cutover note), `Makefile` if migrator target wording changed. + +## Tests Before + +1. `TestRootDocForItem_Object` — object JSON → root fields + meta; no `value`. +2. `TestRootDocForItem_Int64Fidelity` — large integral number stays int64. +3. `TestRootDocForItem_LolscheduleSubscribers` — array → `{subscribers:[...]}`. +4. `TestRootDocForItem_LolscheduleLastPush` — date string → `{date:"..."}`. +5. `TestRootDocForItem_UnknownScalar_Errors` — bare scalar w/o wrap rule fails loud. +6. Migrator e2e (`MONGODB_TEST_URL` + DynamoDB Local): seed DynamoDB rows for a + few modules incl. lolschedule array+scalar → migrate → assert raw Mongo docs + are flattened/wrapped, no `value`; re-run idempotent; `--verify` counts match; + values readable by the module's `DocStore[T]`. + +## Implementation Steps + +1. Implement `rootDocForItem` + wrap rules. +2. Rewrite `runMigrate` to encode + `ReplaceOne` upsert; keep `--dry-run`. +3. Keep `runVerify` (count parity) — unchanged logic. +4. Add inline module/key validation (reuse `storage.ValidateKey` if exported, or + replicate the rule) so bad rows fail loud. +5. Make migrator tests pass. +6. Update README/deploy docs to the flattened layout; remove BSON-string/binary + destination wording. + +## Success Criteria + +- [ ] Migrator writes flattened native docs (no `value`) for all modules. +- [ ] lolschedule array + scalar land as `subscribers`/`date` root fields. +- [ ] Migrated docs are read back correctly by the typed module stores. +- [ ] Idempotent; `--dry-run` writes nothing; `--verify` exits non-zero on mismatch. +- [ ] Unknown non-object value fails loud (no silent guess). +- [ ] README + Coolify deploy docs show the flattened layout. + +## Risk Assessment + +- Risk: a module other than lolschedule stores a non-object (missed in survey). + Mitigation: encoder fails loud on un-wrapped scalars/arrays; e2e covers the + known set; the error names module/key so a new wrap rule is a one-liner. +- Risk: re-introducing a generic JSON codec invites runtime reuse. Mitigation: + it lives only in `cmd/migrate-dynamo-to-mongo`, not `internal/storage`. +- Risk: DynamoDB source already gone at run time. Mitigation: migrator is the + cutover step; if DynamoDB is empty/absent the operator skips it (documented). diff --git a/plans/260628-1318-mongo-native-typed-stores/phase-05-verification-and-rollout.md b/plans/260628-1318-mongo-native-typed-stores/phase-05-verification-and-rollout.md new file mode 100644 index 0000000..5e69fcf --- /dev/null +++ b/plans/260628-1318-mongo-native-typed-stores/phase-05-verification-and-rollout.md @@ -0,0 +1,89 @@ +--- +phase: 5 +title: "Verification + Mongo-only rollout" +status: done +priority: P2 +dependencies: [4] +effort: "M" +--- + +# Phase 5: Verification + Mongo-only rollout + +## Overview + +Full validation and a simple cutover. Because Atlas is empty until cutover and +there is no legacy Mongo data, rollout is one-way and clean: migrate from +DynamoDB, then run Mongo-only. No in-place Mongo migration, no dual-read. + +## Requirements + +- All test gates pass (`make vet`, `make test`, Mongo integration, migrator e2e). +- Raw Mongo docs verified flattened for representative modules. +- App behavior unchanged for all Telegram commands. +- Cutover + rollback notes recorded. + +## Validation gates + +```sh +make vet +make test +MONGODB_TEST_URL=mongodb://127.0.0.1:27017 go test ./internal/storage ./cmd/migrate-dynamo-to-mongo +# migrator e2e additionally needs DynamoDB Local (DYNAMODB_LOCAL_URL) +``` + +Manual raw-doc inspection (Atlas/Compass or a small test assertion) for: + +- coin/gold/stock portfolio: root fields, `version`, Date `updatedAt`, no `value`. +- wordle/loldle/twentyq game + stats: root fields. +- lolschedule: `subscribers` array field; last-push `date` field. +- stats counters: root fields + List works. + +## Cutover order (operator) + +1. Backup/export current DynamoDB table (source of truth) before cutover. +2. Disable EventBridge / stop the old AWS app so DynamoDB writes stop. +3. `migrate-dynamo-to-mongo --dry-run` → review per-module counts. +4. `migrate-dynamo-to-mongo` → writes flattened native docs to Atlas. +5. `migrate-dynamo-to-mongo --verify` → per-collection count parity (exit 0). +6. Start the Mongo-only container (`MONGO_URL`+`MONGO_DATABASE`); confirm health. +7. Spot-check a few commands (coin balance, wordle state, lolschedule subscribe). + +## Rollback + +- Before step 6 (no Mongo writes yet): re-enable the old AWS app on DynamoDB. + DynamoDB is untouched (migrator is Scan-only), so this is safe. +- After step 6 (Mongo has taken live writes): forward-fix preferred. To revert to + DynamoDB you would need a reverse Mongo→DynamoDB export of anything written + after cutover. Keep the DynamoDB backup from step 1 as the floor. + +## Related Code Files + +- Read: `Makefile`, `README.md`, `docs/deploy-coolify-selfhosted.md`, + `docs/aws-decommission-runbook.md`. +- Modify: tests only if verification finds gaps. + +## Implementation Steps + +1. Run all gates above; fix regressions (do not weaken tests). +2. Run the migrator e2e against local Mongo + DynamoDB Local. +3. Inspect raw docs per the list; optionally add a small assertion test. +4. Record the cutover commands + rollback constraints in the completion report + under `plans/reports/`. +5. Grep docs for stale `{ _id, value, version, updatedAt }` wording; fix. + +## Success Criteria + +- [ ] `make vet`, `make test` pass. +- [ ] Mongo integration + migrator e2e pass. +- [ ] Raw docs verified flattened (no `value`/`_payload`) for representative modules. +- [ ] No Telegram command behavior change. +- [ ] Cutover + rollback documented in the completion report. + +## Risk Assessment + +- Risk: a module's data didn't round-trip through migration. Mitigation: e2e + reads migrated docs back through the typed store; spot-check live commands. +- Risk: Atlas M0 limits during migrate. Mitigation: tiny dataset; collection-by- + collection counts in `--verify`. +- Risk: one-way cutover. Mitigation: DynamoDB backup + Scan-only migrator means + pre-write rollback is safe; post-write rollback documented as forward-fix. diff --git a/plans/260628-1318-mongo-native-typed-stores/plan.md b/plans/260628-1318-mongo-native-typed-stores/plan.md new file mode 100644 index 0000000..33e7696 --- /dev/null +++ b/plans/260628-1318-mongo-native-typed-stores/plan.md @@ -0,0 +1,223 @@ +--- +title: "MongoDB-native typed stores (replace KVStore)" +description: "Delete the generic byte-oriented KVStore abstraction; give every module a typed, Mongo-native versioned store. MongoDB is the only runtime backend (memory kept for tests). The DynamoDB→Mongo migrator stays and writes the new flattened shape directly." +status: completed +priority: P2 +branch: "feature/selfhosted" +tags: [database, mongodb, refactor, storage, typed-repositories, tdd] +blockedBy: [260627-1849-selfhost-coolify-mongodb] +blocks: [] +supersedes: [260628-1113-mongo-native-value-documents, 260628-1310-flatten-mongo-value-documents] +created: "2026-06-28T13:18:00.000Z" +createdBy: "manual" +source: user-direction +--- + +# MongoDB-native typed stores (replace KVStore) + +## Overview + +Replace the generic, byte-oriented `KVStore` / `VersionedStore` abstraction with +a small **typed versioned store** that persists each module's value as a native +MongoDB root document: + +```javascript +{ _id, ...payloadFields, version, updatedAt: ISODate(...), schemaVersion } +``` + +There is no `value` envelope, no JSON-bytes round-trip, and no `_payload` / +`payloadKind` fallback. Module payload structs map straight to BSON via the +driver; the two non-object values (`lolschedule` subscribers array and last-push +date) are wrapped in named structs so they too become ordinary root fields. + +This is the **full Mongo-native** direction the earlier flatten plan +deliberately deferred. Per user direction (2026-06-28): + +- **Drop `KVStore`.** Modules use typed stores; the generic byte interface is removed. +- **MongoDB is the only runtime backend.** The DynamoDB *runtime* store and + provider are deleted. The memory store is kept **only** as a test/local double. +- **Keep the DynamoDB→Mongo migrator.** It still Scans DynamoDB and now writes + the flattened native shape directly. No in-place Mongo schema migrator and no + legacy `value` dual-read — Atlas is empty until cutover, so neither is needed. + +## Why this is now correct (and was not before) + +The blocking plan `260627-1849-selfhost-coolify-mongodb` Phase 4 is +`code-complete-operator-pending`: the live cutover has **not** run, so MongoDB +Atlas holds **no production data**. The prior flatten plan's legacy-`value` +dual-read and standalone in-place migrator existed only to convert pre-existing +Mongo docs — docs that do not exist. The real and only migration is +DynamoDB → Mongo, which writes the final shape in one pass. + +## Design Decision + +Chosen approach: **one generic typed store, two backends.** + +A single generic `DocStore[T]` (Mongo + memory implementations) instead of +hand-written per-module repositories. This honors "typed Mongo repos" without +duplicating store/CAS/list logic ten times (DRY/KISS). + +```go +// internal/storage/doc_store.go +type DocStore[T any] interface { + Get(ctx context.Context, id string) (val T, version int64, err error) // ErrNotFound + Put(ctx context.Context, id string, val T) error // unconditional; bumps version + PutVersioned(ctx context.Context, id string, expectedVersion int64, val T) error // CAS; ErrConflict + Delete(ctx context.Context, id string) error + List(ctx context.Context, prefix string) ([]string, error) +} +``` + +Backend wiring stays backend-agnostic for modules. A provider hands each module +an opaque per-module `Collection` handle; a package-level generic constructor +turns it into a typed store: + +```go +type Provider interface{ Collection(module string) Collection } +type Collection interface{ /* opaque: mongo or memory */ } +func Typed[T any](c Collection) DocStore[T] // type-switches on the concrete Collection +``` + +Module factory usage: + +```go +portfolios := storage.Typed[Portfolio](deps.Store) // deps.Store = provider.Collection("coin") +games := storage.Typed[GameState](deps.Store) // same collection, disjoint key prefixes +``` + +Go methods cannot be generic, so the typed view is a free function, not a +`Provider` method. Multiple types per module share one collection but use +disjoint `_id` key prefixes (existing `gameKey`/`statsKey`/… helpers), so reads +never decode a doc into the wrong type. + +### Stored document encoding (Mongo) + +```go +type storedDoc[T any] struct { + ID string `bson:"_id"` + Version int64 `bson:"version"` + UpdatedAt time.Time `bson:"updatedAt"` + Payload T `bson:",inline"` // payload fields hoisted to root +} +``` + +- `bson:",inline"` requires `T` to be a struct (or map). Module payloads already + are, except the two `lolschedule` values, which Phase 3 wraps in named structs. +- Writes use whole-document `ReplaceOne` (never `$set`), so overwriting a value + cannot leave stale fields from a previous value. +- `version` CAS: `PutVersioned(expected>0)` filters `{_id, version:expected}`; + `MatchedCount==0` → `ErrConflict`. `expected==0` upserts on `{_id, version + absent/0}`; duplicate-key → `ErrConflict`. (Same proven semantics as today's + `MongoKVStore`, just typed.) +- Plain `Put` overwrites unconditionally and bumps `version` via a small bounded + get-version → put-versioned retry loop (reuse existing retry-count style). +- `updatedAt` is `time.Time` (BSON Date), not int64 nanos. +- `schemaVersion` is omitted for now (single shape). Add later only if a real + migration need appears — YAGNI. + +### Reserved root field names + +`_id`, `version`, `updatedAt` are reserved. A payload struct must not define BSON +tags colliding with these. This is enforced once, at compile/review time, per +payload type — not at runtime — because types are known. Phase 1 adds a +reserved-name check helper + test; no runtime `_payload` fallback exists. + +## Scope Challenge + +- This is intentionally a **large** refactor (user-selected maximal scope): + storage layer rewrite + all ~10 modules + all module tests + server wiring + + migrator + docs. +- Held back from going further: no `schemaVersion` machinery, no per-module + bespoke repositories, no Mongo schema validators/indexes beyond the existing + `_id` index, no change to Telegram behavior or command output. + +## Backends after this change + +| Concern | Before | After | +|---|---|---| +| Runtime store | memory \| dynamodb \| mongodb | **mongodb only** | +| Test / local-no-DB store | memory | memory (typed double) | +| DynamoDB | runtime backend + migrator | **migrator only** (Scan + write) | +| Generic byte `KVStore` | yes | **deleted** | + +## Files removed + +- `internal/storage/kv_store.go` (KVStore, VersionedStore interfaces) +- `internal/storage/kv_provider.go` (old KVProvider/MemoryProvider) — replaced +- `internal/storage/memory_kv.go` +- `internal/storage/mongodb_kv.go`, `mongodb_value_codec.go` +- `internal/storage/dynamodb_kv.go`, `dynamodb_provider.go`, `dynamodb_provider_test.go`, `dynamodb_kv_test.go` +- `internal/storage/invalid_store.go` +- `internal/storage/mongodb_kv_test.go`, `memory_kv_test.go` (rewritten as doc-store tests) + +## Files kept / reused + +- `internal/storage/dynamodb_client.go` (`NewDynamoDBClient`, `DynamoDBEndpointFromEnv`) — migrator only +- `internal/storage/mongodb_client.go`, `mongodb_provider.go` (reshaped to the new Provider) +- `internal/storage/keys.go`, `prefix.go` (key/prefix validation + range scan reused) + +## Phases + +| Phase | Name | Status | +|-------|------|--------| +| 1 | [Typed DocStore contract + memory backend](./phase-01-typed-docstore-and-memory-backend.md) | Done | +| 2 | [Mongo typed store + integration tests](./phase-02-mongo-typed-store.md) | Done | +| 3 | [Migrate modules + wiring; delete KVStore](./phase-03-migrate-modules-and-wiring.md) | Done | +| 4 | [DynamoDB→Mongo migrator + docs](./phase-04-dynamo-migrator-and-docs.md) | Done | +| 5 | [Verification + Mongo-only rollout](./phase-05-verification-and-rollout.md) | Done (code; operator runs live cutover) | + +## Implementation Log + +### Session — 2026-06-28 (/cook) +All 5 phases implemented. Deleted byte `KVStore`/`VersionedStore` + DynamoDB/memory +KV impls; added generic typed `DocStore[T]` (`Provider`/`Collection`/`Typed`) with +Mongo (flattened native `storedDoc[T]` via `bson:",inline"`) + memory backends. +Migrated all 10 modules + `internal/deploynotify` to typed stores; every persisted +struct (incl. nested `lolschedule` `ScheduleEvent` tree and `wordle` `LetterScore`) +carries `bson` tags == `json` names. lolschedule wraps its array/scalar values +(`subscribers`, `daily_push:last_date`) in named structs. MongoDB is the only +runtime backend (memory kept for tests/local); DynamoDB survives only as the +migrator source. Migrator rewritten to write the flattened shape via +`storage.Typed[bson.M]` + wrap rules (`cmd/migrate-dynamo-to-mongo/encode.go`). + +Verification: `go vet`/`go build` clean; full `go test ./...` green hermetically +**and** in-container against real Mongo 7 + DynamoDB Local (storage integration + +migrator e2e `TestMigrateAndVerify`). code-reviewer: CHANGES_REQUESTED → +addressed (H1 nested bson tags, L1 LetterScore tags, L2 stale comment, M1 +camelCase-fidelity test); CAS/parity/migrator/security verified correct. + +Operator-pending (out of code scope): the live DynamoDB→Mongo cutover +(backup → migrate → verify → start Mongo-only container). + +## Cross-Plan Dependencies + +| Relationship | Plan | Status | Rationale | +|---|---|---|---| +| Blocked by | `260627-1849-selfhost-coolify-mongodb` | in-progress (cutover pending) | Provides Mongo provider, dynamo migrator, self-host runtime. Cutover-pending state is *why* legacy dual-read is unnecessary. | +| Supersedes | `260628-1113-mongo-native-value-documents` | completed | Keeps its version-CAS goal; replaces byte-codec values with typed root docs. | +| Supersedes | `260628-1310-flatten-mongo-value-documents` | pending | Same flatten goal; takes the full typed-repo path instead of preserving `KVStore`. | + +## Acceptance Criteria + +- [ ] `internal/storage` exposes a typed `DocStore[T]` with Mongo + memory impls; no `KVStore`/`VersionedStore` remain. +- [ ] All modules persist/read through `DocStore[T]`; no module imports a byte KV API. +- [ ] New Mongo docs are `{ _id, ...payloadFields, version, updatedAt(Date), }` with no `value`, no `_payload`. +- [ ] `lolschedule` subscribers + last-push are named-struct root fields (`subscribers`, `date`), not bare array/scalar. +- [ ] Overwriting a value removes stale prior fields (ReplaceOne). +- [ ] Versioned writes remain exactly-one-winner under concurrent create/update (coin/gold/lolschedule). +- [ ] MongoDB is the only runtime backend; memory store works for hermetic tests + `MODULES=` local run. +- [ ] DynamoDB→Mongo migrator writes the new flattened shape, is idempotent, supports `--dry-run`/`--verify`. +- [ ] `make vet`, `make test`, and Mongo integration tests pass; migrator e2e passes. + +## Not In Scope + +- Deleting the DynamoDB→Mongo migrator or its dynamo Scan path. +- Mongo schema validators / per-field indexes beyond the existing `_id` index. +- `schemaVersion` evolution machinery. +- Any change to Telegram command behavior or output. + +## Open Questions + +None. (Scope confirmed by user 2026-06-28: delete KVStore → typed repos; drop +legacy dual-read + in-place migrator; Mongo-only runtime, memory for tests; +keep dynamo migrate.) diff --git a/plans/reports/from-code-reviewer-to-cook-native-value-version-cas-review.md b/plans/reports/from-code-reviewer-to-cook-native-value-version-cas-review.md new file mode 100644 index 0000000..d6247e5 --- /dev/null +++ b/plans/reports/from-code-reviewer-to-cook-native-value-version-cas-review.md @@ -0,0 +1,55 @@ +# Code Review: Mongo native value + version CAS (Phase 1 & 2) + +Branch `feature/selfhosted`, uncommitted. Review only — no files modified. + +## Scope +- Files: kv_store.go, memory_kv.go, prefix.go, mongodb_kv.go, mongodb_value_codec.go (new), dynamodb_kv.go, keys.go (new), coin/gold portfolio.go, lolschedule/cron.go, main.go, CI/Makefile; 5 firestore files deleted. +- Verified independently: `go build ./...` clean, `go vet ./...` clean, hermetic `go test` for storage + coin/gold/lolschedule all PASS. +- Probed codec edge cases with a throwaway test (nested arrays, null/bool/float, big ints, empty {} / [], dup keys, unicode, trailing garbage, integral floats). + +## Overall Assessment +Solid, well-reasoned implementation. The version-CAS contract and native-value codec are correct for the value shapes this bot actually stores, and the legacy dual-read / adopt-at-v0 path is sound. All 5 acceptance criteria hold. No blocking defects found. The findings below are correctness edge cases (low real-world risk for current data) and stale comments. + +## Acceptance Criteria — Verdict + +1. **Versioned CAS** — PASS. Concurrent create is single-winner: `expected==0` filter `{_id, $or[version absent, version==0]}` + upsert means N-1 racers fall through to an insert on the same `_id` and hit the unique `_id` index → duplicate-key → ErrConflict (`IsDuplicateKeyError`). Stale version → MatchedCount 0 → ErrConflict. Legacy version-less doc adopted at v0 with no spurious conflict. Tests `Versioned`, `AdoptsLegacyDoc`, `ConcurrentCreate` cover all three. No double-write path: an already-versioned doc (v>=1) fails the v0 filter and the upsert insert collides → ErrConflict, never a silent overwrite. +2. **Native value** — PASS. JSON object/array → native bson.M/bson.A; non-JSON (date string) → string; both round-trip; `decodeValue` dual-reads string / bson.M/A/D / bson.Binary / []byte legacy shapes. +3. **Int64 fidelity** — PASS for in-range int64. `json.Number` codec keeps integral numbers as BSON int64; `Int64Fidelity` test + my probe confirm `1719500000000` and `9223372036854775807` survive. See Finding 1 for the out-of-int64-range caveat. +4. **Bulk interface unchanged** — PASS. `KVStore` untouched; `VersionedStore` is a separate opt-in interface. GetJSON/PutJSON/List consumers compile and pass. +5. **No firestore refs / build / vet** — PASS for code; firestore only survives in comments + one stale doc-comment string (Finding 4). Build/vet clean. + +## Findings + +### Medium + +**M1. Integers beyond int64 range silently lose precision (regression vs old byte-store).** +`numbersToBSON` falls back to `Float64()` when `Int64()` overflows. Probe: `{"toobig":9223372036854775808}` → stored/returned as `9223372036854776000`. The previous byte-blob store round-tripped such values exactly. No current consumer stores integers > 2^63-1 (portfolios use float64 amounts + int64 unix-nanos timestamps), so impact today is nil — but it is a real behavioral change worth a one-line note in the codec doc so a future caller isn't surprised. Not blocking. + +### Low + +**L1. `decodeValue`/round-trip drops byte-exactness for native JSON — confirmed no consumer depends on it.** Verified the only two raw `kv.Get` consumers: `lolschedule/cron.go` compares a bare date string (`2026-06-28`, not JSON → string-fallback path, byte-exact) and `lolschedule/subscribers.go` immediately `json.Unmarshal`s the bytes (semantic, not byte, comparison). Every other consumer uses `GetJSON`. Phase 1 removed the only byte-CAS user. So key reordering / whitespace loss from native re-serialization is safe. Noting it so the invariant ("no consumer byte-compares Get output") is recorded — re-check if a future feature stores a signature/hash blob and reads it back for exact compare. + +**L2. Integral JSON floats normalize (`1.0` → `1`).** A JSON `1.0` is stored as int64 and returns as `1`. Harmless for the typed-struct consumers here (unmarshals into float64 fields fine; into `any` yields float64(1), same value). Only matters if a consumer type-asserts `float64` out of `map[string]any` and requires the source to have been a float — none do today. + +**L3. Memory vs Mongo legacy-adopt semantics differ, but divergence is unobservable.** Mongo `GetVersioned` reports v0 for a version-less doc and `PutVersioned(0)` adopts it; memory `PutVersioned(0)` returns ErrConflict if the key exists at all. Memory never produces a version-less existing entry (Put/PutVersioned always set version>=1), so `GetVersioned` never returns v0 for an existing memory key — the legacy-adopt branch is unreachable in memory. Behaviorally consistent in practice; flagging only so the asymmetry is documented and a future memory-store change doesn't quietly break parity. + +**L4. DynamoDB live mode now breaks versioned writes (by design, but still operator-selectable).** `KV_PROVIDER=dynamodb` remains a live switch case in `main.go`, but DynamoDB no longer implements `VersionedStore`. If selected at runtime, every `UpdatePortfolio` / `claimDailyPush` fails fast with "storage does not support versioned portfolio updates" / unsupported. This matches the stated "DynamoDB is migrate-only" intent and fails loudly (no data corruption), but the runtime path still lets an operator pick a backend that can't serve writes. Consider rejecting `dynamodb` in `buildProvider` outside the migrator, or documenting it as read/migrate-only. Not blocking. + +### Stale comments / docs (cosmetic, non-blocking) + +- `cmd/server/main.go:301` — `KVProvider` doc comment still lists `"firestore"` as a valid value; the switch no longer has that case. +- `cmd/migrate-dynamo-to-mongo/main.go:15,144-145` — comments claim Put writes are "byte-identical" to the source value. After Phase 2, JSON objects/arrays are re-encoded to native BSON (not byte-identical). Round-trip correctness is preserved; the wording is now wrong. +- `internal/modules/lolschedule/cron.go:138-141` and `gold/portfolio_test.go:223` — comments/names still say "CompareAndSwap" / "every real KV backend implements CompareAndSwapStore". The type is gone (now `VersionedStore`, and DynamoDB no longer implements it). Per the project rule against stale audit/contract labels in comments, refresh these to reference versioned CAS. + +## Edge cases probed (codec) +Empty `{}`/`[]`, nested arrays, null/bool, unicode (`chào 日本語`), leading whitespace, `-9007199254740993` → all round-trip cleanly. Duplicate keys → last-wins (matches `encoding/json`). Trailing garbage (`{"a":1} trailing`), bare scalars (`123`, `true`, `null`, `"x"`), and truncated (`{`, `[1,2`) → all correctly fall to string-fallback (byte-exact preserved, and `GetJSON` on a malformed value would error identically to before). + +## Unresolved Questions +1. M1: acceptable to accept >int64 precision loss permanently, or add an explicit codec note? (No current caller hits it.) +2. L4: should `buildProvider` refuse `KV_PROVIDER=dynamodb` for the live server, or is the fail-fast-at-write behavior intended to stay? + +--- + +Status: DONE_WITH_CONCERNS +Summary: Phase 1/2 are correct and all 5 acceptance criteria hold; no blocking defects. Concerns are low-risk fidelity edges (>int64 precision, integral-float normalization) plus stale firestore/CompareAndSwap comments to refresh. +Concerns: M1 (>int64 precision loss vs old byte-store, no current caller); L4 (DynamoDB live mode breaks versioned writes by design but stays operator-selectable); stale comments in main.go, migrator, cron.go, gold test. diff --git a/plans/reports/from-code-reviewer-to-cook-selfhost-implementation-review.md b/plans/reports/from-code-reviewer-to-cook-selfhost-implementation-review.md new file mode 100644 index 0000000..8341d5b --- /dev/null +++ b/plans/reports/from-code-reviewer-to-cook-selfhost-implementation-review.md @@ -0,0 +1,75 @@ +# Code Review — Self-host (Coolify + MongoDB) Implementation + +Reviewer: code-reviewer | Date: 2026-06-28 | Branch: `feature/selfhosted` +Plan: `plans/260627-1849-selfhost-coolify-mongodb/` + +## Scope +- New: `internal/storage/mongodb_{client,kv,provider}.go` (+tests), `internal/cron/scheduler.go` (+test), `cmd/migrate-dynamo-to-mongo/` (+test), `docker-compose.yml`, `.env.example`, 2 docs. +- Modified: `cmd/server/main.go`, `internal/server/router.go`, `internal/telegram/client.go`, `internal/modules/lolschedule/cron.go`, `internal/modules/module.go`, `internal/modules/stock/{stock,handlers}.go`, `Dockerfile`, `Makefile`, `go.mod/go.sum`, `aws/telegram-commands.json`. +- Deleted: `internal/telegram/webhook.go`(+test), `internal/modules/stock/income_events.go`(+test). +- Build/vet: `go vet ./...` clean (re-run). Author reports full `go test ./...` + integration suites pass. + +## Overall Assessment +Solid, disciplined change set. The implementation matches the plan's validated decisions point-for-point, and the 15 red-team findings are all materially addressed in code (verified below). No Critical or High defects found. The CAS implementation, MONGO_URL secret handling, cron lifecycle, and deletion hygiene are correct. Findings are Low/informational. + +## Verification of Acceptance Criteria + +1. **mongodb auto-select + boot** — PASS. `buildProvider` (main.go:227-235) selects `mongodb` when `MONGO_URL!=""` and `KV_PROVIDER` empty; explicit `KV_PROVIDER` overrides; old `AWS_LAMBDA_FUNCTION_NAME` auto-detect gone. Cron runs unconditionally (main.go:146). No `/webhook` route, no `WebhookSecret` (router.go). +2. **CAS correctness** — PASS. nil-expected → `InsertOne` + `IsDuplicateKeyError`→`ErrConflict` (linearizable single-winner via unique `_id`); non-nil → `UpdateOne` filtered on `{_id, value:Binary(expected)}`, `MatchedCount==0`→`ErrConflict` (mongodb_kv.go:133-164). value stored as `bson.Binary` generic subtype; `updatedAt` int64 unix-nanos. Matches DynamoDB encoding intent. +3. **MONGO_URL never logged** — PASS (Finding-3). Only `database` logged (main.go:266); SECURITY comments at mongodb_client.go:28-29 and config field 334. No interpolation of `cfg.MongoURL` into any log/error string found. +4. **Cron scheduler** — PASS (Finding-4). 60s local `cronTimeout` const (scheduler.go:23, not imported from server), per-fire `context.WithTimeout`, `recover()` with stack, `cron.WithLocation(time.UTC)`, bad schedule returns error → `log.Fatal` at startup (main.go:147). Panic-recovery deferred before dispatch — correct ordering. +5. **lolschedule idempotent per UTC date** — PASS (Finding-6). `claimDailyPush` CAS on `daily_push:last_date` before fan-out; loser logs + returns nil. New test `TestRunDailyPush_IdempotentPerDate` proves one-message-per-subscriber across two same-date fires. CAS-less stores fall back to plain Put (test-only path) with documented loss of atomicity. +6. **Migrator** — PASS (Findings 9/15). Read-only `Scan` only (scanTable), writes through `provider.For(pk).Put` (byte-identical value + `validateKey` per sk + upsert-by-`_id` = idempotent), `--verify` tallies per-pk via Scan vs `CountDocuments`, exits non-zero on mismatch. e2e test asserts byte-identical round-trip through the app read path. +7. **No regression to other backends / modules / /cron** — PASS. memory/firestore/dynamodb switch arms unchanged; DispatchScheduled + `/cron` HTTP path intact (router + cron_dispatcher untouched in contract); `CronHandler` doc updated only. +8. **No new lint/vet/build errors; intentional contract changes** — PASS. `server.Config` dropped `WebhookSecret`; `storage` gained `MongoProvider`. `go.mod` correctly promotes `robfig/cron/v3` and `mongo-driver/v2` to direct requires; `go vet ./...` clean. + +## Critical Issues +None. + +## High Priority +None. + +## Medium Priority +None blocking. (Finding dispositions all confirmed in code.) + +## Low Priority / Informational + +- **L1 — Stale env name in a user-facing string.** `internal/deploynotify/deploy_notify.go:97` returns `"no BOT_OWNER_ID configured"` as a skip reason, but the env key was renamed to `OWNER_ID` (Session 5). This is a log/diagnostic string only (not flow control), but it now points an operator at a key that no longer exists. Pre-existing line, surfaced by the rename. Suggest: `"no OWNER_ID configured"`. Not in review scope's modified files but directly caused by this change. + +- **L2 — `claimDailyPush` runs the expensive fetch before claiming on every fire.** `runDailyPush` (cron.go:182-213) calls `listSubscribers` + `GetEventsCached` before `claimDailyPush`. A losing double-fire still does the subscriber read + match fetch before no-op'ing. This is intentional and documented (cron.go:202-204: "Placed after the fetch so a transient fetch failure does not consume the day's claim") — a deliberate correctness-over-efficiency trade so a fetch failure doesn't burn the day's claim. Correct call; noting only so it isn't mistaken for an oversight. The wasted fetch is cache-backed and rare (only on genuine double-fire). + +- **L3 — `decodeValue` accepts `string`/`[]byte` in addition to `bson.Binary`.** mongodb_kv.go:42-57 handles three value types. Only `bson.Binary` is ever written by this code, so the `string`/`[]byte` arms are dead for app-written data. Documented as mirroring Firestore's dual-type handling for forward-compat. Harmless; flagging as minor YAGNI surface, not a defect (it does aid migrator/raw-write robustness). + +## Targeted concerns from the request + +- **DeleteWebhook-on-startup safety vs manual-cutover framing.** SAFE and consistent with the plan. The plan (Session 2, line 118) explicitly makes cutover = `deleteWebhook` → start polling container; adding `DeleteWebhook` to startup is the in-code realization of that step, not a divergence. `DropPendingUpdates:false` preserves the buffered queue (lossless). It is idempotent and harmless when no webhook is set. The go-telegram polling loop (`get_updates.go`) retries with exponential backoff on errors, so even a transient 409 (webhook deletion racing the first `getUpdates`) self-heals. The pre-cutover Lambda keeps its own webhook code until `sam delete`, so rollback in the window still works. No issue. + +- **Goroutine/shutdown lifecycle of `cron.Run` + `b.Start`.** Correct. Both `b.Start(rootCtx)` and the in-process scheduler bind to `rootCtx` (SIGINT/SIGTERM). On signal: `b.Start` returns when ctx cancelled; `stopCron()` (deferred) calls `c.Stop()` and blocks on the returned context until in-flight jobs finish; `srv.Shutdown` gets a fresh 15s ctx. Order of defers (`stop` → `closeProvider` → `stopCron`) runs LIFO: stopCron waits for jobs, THEN closeProvider disconnects Mongo — correct (jobs that touch Mongo finish before disconnect). One nuance: `srv.Shutdown` runs in the main flow before the deferred `stopCron`/`closeProvider`, so an in-flight cron job during shutdown still has its Mongo client until stopCron completes. Fine. + +- **CAS race edge cases.** The nil-expected `InsertOne` race is the right primitive (Finding-1): unique `_id` gives exactly one winner, losers get duplicate-key→`ErrConflict`. Author reports the blocking concurrent-CAS linearizability test passes against real MongoDB 7. The non-nil path's filter-on-value is a true CAS (no read-then-write gap). No TOCTOU window. The `claimDailyPush` `Get`-then-CAS in lolschedule has a benign read-ahead (the `Get` is an optimization; the CAS is the authority), so a concurrent claim between Get and CAS still resolves to a single winner via `ErrConflict`. Correct. + +- **Dangling references after deletions.** None. Grep for `IncomeEventClient`, `handleIncomeEvents`, `stock_income_events`, `WebhookHandler`, `WebhookSecret`, `NewIncomeEventClientFromEnv` finds zero references in non-test and test Go code. `aws/telegram-commands.json` had the `stock_income_events` command removed (diff -4 lines). Only the L1 log-string mention of the old env name remains. + +- **Divergence from plan's validated decisions.** None found. `updatedAt` is int64 nanos everywhere; CAS absent-case is `InsertOne` (no `$exists:false` upsert); migrator writes through `Put`; health is plain-text `miti99bot ok\n` (health.go:20) matching the compose note; `/cron` 404s when `CRON_SHARED_SECRET` unset; `WithAllowedUpdates{message,callback_query}` matches prior webhook allowed_updates; gitSHA injected via Dockerfile `ARG GIT_SHA`. + +## Positive Observations (risk-relevant) +- The `b.Start` polling loop's built-in error backoff means the `DeleteWebhook`-before-`Start` ordering is robust even under a race — no custom retry needed. +- Migrator writing through `Put` (not raw driver) is the correct choice for encoding parity and is verified by the e2e byte-identity assertion. +- Provider `For()` re-validates module name against `collectionNameRe` (defense-in-depth identical to firestore/dynamodb), and `invalidStore` fails closed. + +## Metrics +- Type/contract changes: intentional only (`server.Config -WebhookSecret`, `storage +MongoProvider`). +- Lint/vet: `go vet ./...` clean. +- Test coverage: new tests for scheduler (fire/stop/bad-schedule/panic-recovery), mongo CAS parity + concurrent linearizability, migrator e2e + decode/sort units, lolschedule idempotency. Adequate for the change. + +## Recommended Actions +1. (Low) Update `internal/deploynotify/deploy_notify.go:97` skip-reason string `BOT_OWNER_ID` → `OWNER_ID` to match the renamed env key. +2. (Optional) Consider whether the `string`/`[]byte` arms in `MongoKVStore.decodeValue` earn their keep, or trim to `bson.Binary` only with a clear error for other types (YAGNI). Non-blocking. + +## Unresolved Questions +None. Plan decisions are internally consistent and the implementation tracks them. + +--- +Status: DONE +Summary: Self-host implementation is production-ready; all 8 acceptance criteria and all 15 red-team findings verified satisfied in code. No Critical/High issues. One Low cosmetic fix (stale `BOT_OWNER_ID` string in deploynotify skip reason). +Concerns: Only L1 (stale env-name string in a diagnostic log line); cosmetic, non-blocking. diff --git a/plans/reports/research-260628-0605-mongodb-document-design-standards.md b/plans/reports/research-260628-0605-mongodb-document-design-standards.md new file mode 100644 index 0000000..310eb5d --- /dev/null +++ b/plans/reports/research-260628-0605-mongodb-document-design-standards.md @@ -0,0 +1,258 @@ +--- +title: MongoDB Document Design Standards Research +created: 2026-06-28T06:05:31Z +type: research +status: complete +tags: + - mongodb + - schema-design + - storage +--- + +# Research Report: MongoDB Document Design Standards + +## Executive Summary + +MongoDB does not have one universal "standard schema" shape. Its core rule is +access-pattern-first design: store data that is read and updated together in the +same document, then add references only when embedded data is high-cardinality, +unbounded, independently owned, or written on different schedules. + +For this repo, `{ _id, value, version, updatedAt }` is a valid generic KV-store +envelope. It is not the most idiomatic Mongo domain model. If MongoDB becomes +the primary domain database, prefer module-specific documents with domain fields +at the document root or in named subdocuments, not hidden under a generic +`value` wrapper. + +Best practical recommendation: keep current KV wrapper only while the +backend-swappable `KVStore` abstraction is important. If you want "Mongo-native" +long term, migrate one high-value module at a time to typed repositories and +domain-shaped collections. First easy improvement: store `updatedAt` as BSON +Date (`time.Time` in Go), not int64 nanos, if it will be queried, sorted, shown +as a date, or used for TTL. + +## Research Methodology + +- Timestamp: 2026-06-28T06:05:31Z +- Sources consulted: 6 official MongoDB documentation pages +- Search scope: official MongoDB docs only +- Recency: current MongoDB manual, version 8.3 docs visible during research +- Key terms: MongoDB data modeling, best practices, embedded documents, + references, indexes, atomic updates, TTL indexes, schema versioning + +## Key Findings + +### 1. MongoDB Documents Are Field-Value Documents + +MongoDB stores records as BSON documents. A document is a set of field-value +pairs, where fields may contain embedded documents, arrays, dates, numbers, and +other BSON types. The `_id` field is reserved as the primary key, must be unique +in a collection, and is immutable. + +Implication: `_id` as the storage key is good. `value` as a generic envelope is +legal, but it hides the domain shape one level down. + +Source: https://www.mongodb.com/docs/manual/core/document/ + +### 2. Access Patterns Drive Schema + +MongoDB explicitly recommends structuring data around how the app reads and +writes it. Data accessed together should be stored together. + +Implication for portfolios/game state: if the app always loads the whole +portfolio or game state, one document per user or subject is correct. But the +document should ideally be shaped like the domain: + +```javascript +{ + _id: "user:7", + usd: 1000.25, + holdings: { BTC: 1, ETH: 3 }, + meta: { createdAt: ISODate("2026-06-28T00:00:00Z") }, + version: 4, + updatedAt: ISODate("2026-06-28T06:05:31Z") +} +``` + +Current KV shape: + +```javascript +{ + _id: "user:7", + value: { + usd: 1000.25, + holdings: { BTC: 1, ETH: 3 }, + meta: { createdAt: 1782604800000 } + }, + version: 4, + updatedAt: 1782626731000000000 +} +``` + +The current shape is acceptable for a KV abstraction, but a real Mongo schema +would remove the `value` envelope. + +Source: https://www.mongodb.com/docs/manual/data-modeling/ + +### 3. Embed vs Reference + +Embed when related data is queried together, updated together, "has-a" data, or +archived together. Reference when child data has high cardinality, grows without +bounds, is written at different times, or can exist independently. + +Implication: + +- `coin`/`gold`/`stock` current portfolio snapshot: embed balances and holdings + in one user document. +- Trade history or command logs: do not keep appending forever inside one user + document. Use a separate `trades`/`events` collection or bucket pattern. +- `lolschedule` API cache: separate cache documents are fine. + +Source: https://www.mongodb.com/docs/manual/data-modeling/best-practices/#link-related-data + +### 4. Index Queried Fields, But Not Everything + +MongoDB recommends indexing fields that are frequently queried, filtered, +sorted, or joined. Indexes speed reads and sorts, but cost disk, memory, and +write overhead. + +Implication: + +- Current KV access by `_id` needs no extra index. +- If you query by domain fields, flattening from `value.foo` to `foo` makes + indexes and validation cleaner. +- Indexes on nested fields are possible, e.g. `{ "value.holdings.BTC": 1 }`, + but this is weaker schema hygiene than indexing stable top-level fields. + +Source: https://www.mongodb.com/docs/manual/data-modeling/best-practices/#index-commonly-queried-fields + +### 5. Version Field Is Good, But Name It By Purpose + +MongoDB single-document writes are atomic. To avoid concurrent lost updates, +include the expected current value in the update filter. This maps cleanly to an +optimistic-lock `version` field. + +Implication: this repo's `version` field is sound for concurrency control. If +you also need schema migration versioning, use a separate `schemaVersion` field. +Do not overload `version` for both CAS and schema version. + +Source: https://www.mongodb.com/docs/manual/core/write-operations-atomicity/ +Source: https://www.mongodb.com/docs/manual/data-modeling/design-patterns/data-versioning/ + +### 6. Dates Should Be BSON Dates When They Behave Like Dates + +MongoDB document examples use Date values for date fields. TTL indexes expire +documents based on date values; if the indexed field lacks date values, the doc +will not expire. + +Implication: if `updatedAt` is only parity/debug metadata, int64 nanos is +workable. If it will be used for TTL, sorting, Compass inspection, or date +filters, use BSON Date: + +```javascript +{ + updatedAt: ISODate("2026-06-28T06:05:31Z") +} +``` + +For TTL, prefer a dedicated field such as `expiresAt`, not `updatedAt`, unless +you truly want to delete old inactive state. + +Source: https://www.mongodb.com/docs/manual/core/index-ttl/ + +## Comparative Analysis + +| Shape | Good for | Bad for | Verdict | +|---|---|---|---| +| `{ _id, value, version, updatedAt }` | Generic KV, backend portability, low code churn | Domain queries, schema validation, clean indexes, Compass readability | Keep if KV abstraction stays | +| `{ _id, ...domainFields, version, updatedAt }` | Real Mongo domain model, queryability, validation, indexes | Requires per-module repositories/migrations | Best Mongo-native long-term | +| One mega `kv` collection with `{ module, key, value }` | Single collection ops, simple export | Hot/cold data mixed, indexes more complex, less module isolation | Not better than current | +| Separate typed collections per module | Clean ownership, indexes per module, better validation | More code, less backend portability | Best if committing to Mongo | + +## Implementation Recommendations + +### Short Term + +Keep current shape. It already stores JSON objects as native BSON under `value`, +keeps `_id` lookups efficient, and uses `version` correctly for optimistic +locking. + +Fix wording/design around `updatedAt`: either keep it as write-only int64 and +stop describing it as TTL-ready, or convert to BSON Date when touched next. + +### Medium Term + +For new Mongo-first modules, avoid the generic KV envelope. Create a typed +repository and module-specific collection: + +```javascript +// collection: coin_portfolios +{ + _id: "user:7", + usd: 1000.25, + holdings: { BTC: 1, ETH: 3 }, + createdAt: ISODate("2026-06-28T00:00:00Z"), + updatedAt: ISODate("2026-06-28T06:05:31Z"), + version: 4, + schemaVersion: 1 +} +``` + +Use `version` only for CAS. Use `schemaVersion` only for shape migration. + +### Long Term + +Migrate high-value modules one by one: + +1. Pick module with real query needs. +2. Define common reads/writes first. +3. Design collection around those access patterns. +4. Add JSON schema validation only after the shape stabilizes. +5. Add indexes only for proven query/sort paths. +6. Keep KV fallback for simple modules if it still saves code. + +## Common Pitfalls + +- Flattening too far. Mongo likes embedded documents. Removing `value` is good; + exploding every nested object into root fields is not automatically good. +- Unbounded arrays. Do not keep infinite histories inside one user document. +- Reusing `version` for schema version and concurrency. Use separate fields. +- TTL on `updatedAt` for persistent state. This silently turns inactivity into + deletion. +- Indexing everything. Indexes cost writes, disk, and memory. +- Premature rewrite. KV abstraction is still valuable for tests and simple + modules. + +## Recommendation For miti99bot + +The current schema is a pragmatic KV envelope, not bad practice. It is the right +compromise if the goal is small bot state with in-memory tests and backend +swappability. + +If the goal changes to "MongoDB is the product database", then flatten out of +`value` into typed module collections. Start with portfolios, because they have +money-path correctness and likely future reporting needs. Do not rewrite all +modules at once. + +Priority: + +1. Keep `_id`. +2. Keep `version` for CAS. +3. Consider BSON Date for `updatedAt`. +4. Remove `value` only during typed module repository migration. +5. Add `schemaVersion` only if documents must support multiple shapes during + migrations. + +## Resources & References + +- MongoDB Documents: https://www.mongodb.com/docs/manual/core/document/ +- MongoDB Data Modeling: https://www.mongodb.com/docs/manual/data-modeling/ +- Data Modeling Best Practices: https://www.mongodb.com/docs/manual/data-modeling/best-practices/ +- Embed vs Reference guidance: https://www.mongodb.com/docs/manual/data-modeling/best-practices/#link-related-data +- Atomicity and concurrent update filters: https://www.mongodb.com/docs/manual/core/write-operations-atomicity/ +- TTL Indexes: https://www.mongodb.com/docs/manual/core/index-ttl/ +- Document and Schema Versioning: https://www.mongodb.com/docs/manual/data-modeling/design-patterns/data-versioning/ + +## Unresolved Questions + +None.