tiennm99bot/plans/260627-1849-selfhost-coolify-mongodb/phase-01-mongodb-storage-provider.md

phase, title, status, priority, dependencies, effort

phase	title	status	priority	dependencies	effort
1	MongoDB Storage Provider	pending	P1		M

Phase 1: MongoDB Storage Provider

Overview

Add mongodb as a 4th KVProvider, modeled exactly on the existing firestore provider (collection-per-module isolation). Wire it into buildProvider and config via KV_PROVIDER=mongodb, MONGO_URL, MONGO_DATABASE. No module code changes — the KVStore interface is the only contract.

Requirements

• Functional: implement KVStore (Get/GetJSON/Put/PutJSON/Delete/List) + CompareAndSwapStore (CompareAndSwap) backed by MongoDB.
• Functional: one Mongo collection per module (mirrors FirestoreProvider); document _id = key, field value = raw bytes, field updatedAt = timestamp.
• Functional: List(prefix) = range/regex query on _id with begins_with semantics; empty prefix = whole collection.
• Non-functional: behavior parity with firestore/dynamodb — same ErrNotFound, ErrConflict, key validation (reuse validateKey/validatePrefix), same collectionNameRe module-name guard via invalidStore.
• Non-functional: bounded startup connect timeout (match dynamodbInitTimeout = 5s style), graceful Close().

Architecture

Reuse the shared helpers already in internal/storage:

• validateKey / validatePrefix (in firestore_kv.go) — key constraints. Mongo _id has no / restriction, but reusing keeps cross-backend parity and is harmless.
• collectionNameRe (in firestore_provider.go) — module-name alphabet.
• invalidStore (in invalid_store.go) — returned for bad module names.

Document shape (parity with firestore value/updatedAt):

{ "_id": "<key>", "value": <BinData>, "updatedAt": <int64 nanos> }

Store value as BSON binary (bson.Binary) so non-UTF-8 round-trips; on read accept both binary and string (firestore does the same dual-type handling). Store updatedAt as int64 unix-nanos (NOT BSON datetime) — matches DynamoDB exactly (dynamodb_kv.go:101), keeps migration byte-faithful, and avoids ms-truncation if a future TTL/sort ever reads it. The migrator (Phase 4) and the provider MUST share one encoding — see Phase 4 (migrator writes through MongoKVStore.Put, not raw UpdateOne).

CompareAndSwap mapping — the expected == nil branch is a LIVE path (first write of every new coin/gold portfolio, coin/portfolio.go:81-83, gold/portfolio.go:62-80), not an edge case. Map it to a plain InsertOne and rely SOLELY on the unique _id index for the conflict:

• expected == nil → InsertOne({_id, value, updatedAt}); mongo.IsDuplicateKeyError(err) → ErrConflict. Do NOT use a {value:{$exists:false}} upsert filter (it can false-conflict on a value-less doc and muddies the contract).
• expected != nil → UpdateOne({_id, value: expected}, {$set:{value,updatedAt}}); MatchedCount == 0 → ErrConflict. _id is unique by default, so the absent-insert race is linearizable: exactly one InsertOne wins, losers get duplicate-key → ErrConflict → caller retry loop reloads the winner's state. This must be proven by a blocking concurrent-writer test (see Success Criteria), not just asserted.

List(prefix): Find({_id: {$gte: prefix, $lt: prefixSuccessor(prefix)}}, projection={_id:1}) — reuse the existing prefixSuccessor helper from firestore_kv.go. Empty prefix → Find({}). Avoids regex injection and uses the _id index.

Related Code Files

• Create: internal/storage/mongodb_client.go — NewMongoClient(ctx, uri) (*mongo.Client, error) with connect+ping timeout; NewMongoDatabase. Mirror dynamodb_client.go.
• Create: internal/storage/mongodb_provider.go — MongoProvider{ db *mongo.Database }, For(module) returns invalidStore on bad name else NewMongoKVStore. Mirror firestore_provider.go.
• Create: internal/storage/mongodb_kv.go — MongoKVStore, all methods. Mirror firestore_kv.go.
• Create: internal/storage/mongodb_kv_test.go + mongodb_provider_test.go — parity tests, gated on MONGODB_TEST_URL (skip when unset), mirroring dynamodb_kv_test.go gating on DYNAMODB_LOCAL_URL.
• Modify: cmd/server/main.go — add mongodb case to buildProvider; add MongoURL, MongoDatabase to config + loadConfig (MONGO_URL, MONGO_DATABASE); update buildProvider doc comment + auto-detect note (mongo is explicit-only).
• Modify: go.mod / go.sum — add go.mongodb.org/mongo-driver/v2.
• Modify: Makefile — add mongo-local (docker mongo:7) + test-mongo target gated by MONGODB_TEST_URL, mirroring dynamodb-local/test-dynamodb.
• Modify: README.md — add mongodb to the storage backend list + local-run snippet.

Implementation Steps

1. go get go.mongodb.org/mongo-driver/v2/mongo (and /bson).
2. Write mongodb_client.go: connect with options.Client().ApplyURI(uri), client.Ping under a 5s context, return client; helper to get *mongo.Database from MONGO_DATABASE.
3. Write mongodb_kv.go: implement methods per Architecture; reuse validateKey, validatePrefix, prefixSuccessor; constants mongoValueField="value", mongoUpdatedAtField="updatedAt".
4. Write mongodb_provider.go: For guards with collectionNameRe, returns db.Collection(module)-backed store.
5. Wire buildProvider: case "mongodb": require MONGO_URL + MONGO_DATABASE (error if missing, mirror dynamodb's DYNAMODB_TABLE check); construct client under timeout; closer calls client.Disconnect. The startup log.Info("storage backend", …) line MUST log only non-secret fields — "backend","mongodb","database",cfg.MongoDatabase. NEVER log MONGO_URL (it is mongodb+srv://user:pass@…; the firestore/dynamodb cases at main.go:234-253 log a benign identifier, but the mongo equivalent is a credential). If a host is wanted for diagnostics, parse and log only the host, never the userinfo.
6. Add config fields + env reads. Mongo is explicit-only (not in the auto-detect switch) to avoid surprising Lambda.
7. Tests: replicate the firestore/dynamodb test bodies against a real Mongo (mongo-local), covering Get/Put/Delete/List/prefix/CAS-absent/CAS-match/CAS-conflict/ErrNotFound + cross-module isolation.
8. make vet && make test && MONGODB_TEST_URL=mongodb://localhost:27017 make test-mongo.

Success Criteria

• internal/storage exposes MongoProvider/MongoKVStore passing the same test matrix as DynamoDBKVStore.
• KV_PROVIDER=mongodb with MONGO_URL/MONGO_DATABASE boots; missing either errors clearly at startup.
• Cross-module isolation verified (collection-per-module).
• CompareAndSwap returns ErrConflict on stale expected + on absent-with-non-nil-expected; succeeds on nil-expected insert.
• (blocking) Concurrent-writer CAS test: N goroutines race a nil-expected insert + a stale-update on the same key against a real Mongo; assert exactly one winner, losers get ErrConflict. Plus a "doc exists without value field" edge case.
• Startup log shows backend=mongodb database=<db> and does NOT contain the connection string / any credential.
• make vet and make test pass; AWS/firestore paths untouched.

Risk Assessment

• CAS semantics drift: Mongo upsert race differs from DynamoDB conditional put. Mitigation: unique _id + IsDuplicateKeyError for the absent case; assert with a concurrent-writer test.
• Value type on read: driver may decode as binary or string. Mitigation: dual-type switch like firestore's.
• TLS to Atlas: mongodb+srv:// URIs need DNS SRV + TLS. Mitigation: driver handles it; document that MONGO_URL is the full Atlas SRV connection string incl. credentials.
• Driver version: v2 API differs from v1 (mongo.Connect signature). Mitigation: pin v2, follow current docs.