Commit Graph
15 Commits
Author SHA1 Message Date
tiennm99 6ed9c2dcb0 chore: drop migration leftovers, refresh env examples + secret-leak scope
- remove MONGODB_URI from .env.example (Atlas migration done; deleted from
  Vercel cloud env too)
- trim .env.deploy.example to vars actually consumed by deploy scripts
  (Upstash creds were only needed by the now-deleted migration script)
- README config table: drop ENV / SOURCE_COMMIT / SCHEDULE_CHECK_APP_TIME
  (never read by code; Java-era leftovers)
- check-secret-leaks: drop MONGODB_URI; add UPSTASH/KV/CRON tokens; widen
  scan roots to include api/
- add scripts/list-upstash-keys.js read-only ops helper
2026-05-10 00:23:16 +07:00
tiennm99 cad84e21a4 feat: accept KV_REST_API_* env vars as Upstash fallback
Vercel Marketplace Upstash integration injects KV_REST_API_URL and
KV_REST_API_TOKEN — different names from vanilla Upstash signup
(UPSTASH_REDIS_REST_URL / UPSTASH_REDIS_REST_TOKEN). Adapter and
migration script now accept either form, so the operator doesn't have
to duplicate values when sharing an Upstash DB provisioned via the
Vercel integration. UPSTASH_* takes precedence when both are set.
2026-05-09 20:17:06 +07:00
tiennm99 73762fa0a0 feat: migrate to Vercel + Upstash with KEY_PREFIX namespacing
Phases 1-5 of consolidate-vercel-upstash plan. Replaces Cloudflare
Workers + KV with Vercel serverless functions + Upstash Redis. Inlines
app-store-scraper / google-play-scraper npm libs (drops the
store-scraper.vercel.app HTTP roundtrip). KEY_PREFIX (default
'store-scraper-bot:') namespaces all Redis keys so the Upstash DB can
be safely shared with other Vercel projects.

- vercel.json + .vercelignore + Vercel-aware package.json scripts
- api/webhook.js + api/cron.js Vercel functions (with shared
  src/app-builder.js); cron auth fails closed when CRON_SECRET unset
- src/repository/upstash.js replaces kv.js; all 4 repos take a handle
  bundling client + prefix
- scripts/migrate-atlas-to-upstash.js writes legacy Java Atlas state
  directly to Upstash with --dry-run + --include-cache flags
- .env.example refreshed for the new env surface

Phases 6 (Vercel deploy + webhook cutover) and 7 (Docker + wrangler
cleanup) remain operator-driven post-deploy.
2026-05-09 20:07:07 +07:00
tiennm99 3c97553c69 feat: initial JavaScript port of store-scraper-bot
Node.js 20+ ESM port mirroring Java/Go implementations.

- 13 Telegram commands matching Java identifiers
- MongoDB schema parity (common, group, apple_app, google_app collections)
- Apple/Google scrapers calling store-scraper.vercel.app with 10-min cache
- Daily 7am Vietnam-time cron with weekend-silent mode
- HTML table renderer matching Java/Go output
- Docker + Compose (prod and dev)

Untested end-to-end against live Telegram or upstream API.
2026-04-26 20:16:00 +07:00
tiennm99 ca1a409610 Clear and rewrite
Retire the Go port; history continues from the JavaScript implementation (store-scraper-bot).
2026-08-04 23:00:01 +07:00
tiennm99 9b03eb007a feat(core): add logic converted from original repo by Claude Code 2025-11-15 18:54:06 +07:00
tiennm99 c5f3628f01 Clear and rewrite
Retire the Java implementation; history continues from the Go port (store-scraper-bot-go).
2026-08-04 22:59:44 +07:00
tiennm99 8394f909ae feat(mongo): cleanup 2025-12-15 22:46:21 +07:00
tiennm99 0eae89ac16 feat(db): use mongodb instead of couchbase (WIP) 2025-11-22 23:05:39 +07:00
tiennm99 0f5fdca3e7 feat(loki): send logs to loki to centralize logs 2025-11-13 18:24:13 +07:00
tiennm99 a4646c91ca Add raw app response commands for Apple and Google
Introduces RawAppleAppCommand and RawGoogleAppCommand to allow admins to fetch and download raw JSON responses from the Apple App Store and Google Play Store scrapers. Refactors AppStoreScraper and GooglePlayScraper to expose rawApp methods and improves method naming for clarity. Updates StoreScrapeBot to register the new commands and adjusts .env.example for better configuration clarity.
2025-11-09 10:12:19 +07:00
tiennm99 dac6a82b71 Migrate from Redis to Couchbase for data storage
Replaces Redis with Couchbase as the primary data store. Updates environment variables, dependencies, and Docker Compose configuration to use Couchbase. Refactors repository and model classes to use Couchbase APIs, removes Redis utility and related code, and adds Couchbase utility for collection management. Updates AdminRepository to use 'admin' as the key instead of an empty string.
2025-11-06 20:30:16 +07:00
tiennm99 def45c7402 Migrate data storage from Couchbase to Redis
Replaces Couchbase with Redis for data persistence, updating environment variables, Docker Compose files, and repository logic. Removes Couchbase and Jackson dependencies, introduces Gson for JSON serialization, and refactors models and repositories to support Redis. Adds utility classes for Redis and Gson, and updates tests and scrapers to use Gson.
2025-11-05 21:47:58 +07:00
tiennm99 bae876b086 Refactor bot client and add group management commands
Replaced ScoreScrapeBotTelegramClient with StoreScrapeBotTelegramClient and updated related references. Added BaseStoreScraperBotCommand for command error handling. Implemented AddGroupCommand and GetGroupCommand for managing allowed groups via admin commands. Improved CouchbaseUtil to ensure scope and collection creation. Updated logging configuration and environment example. Minor model and repository adjustments for initialization and group handling.
2025-11-04 23:06:46 +07:00
tiennm99 4bf94aaad4 Add Docker support and environment configuration
Introduced Dockerfile, .dockerignore, and example environment file for containerization. Added compose.yml for Docker Compose setup and renamed docker-compose.dev.yml to compose.dev.yml. Updated build.gradle.kts for Java 21 and shadow plugin, and improved .gitignore and .gitattributes. Refactored Config.java to reorder ENV initialization.
2025-11-04 21:18:28 +07:00