Commit Graph
71 Commits
Author SHA1 Message Date
tiennm99 3f586d1d9f refactor: dispatcher pre-parses args, drop arg helpers + info.js
The dispatcher now extracts both the command name and its arguments
once and passes the args array to handlers via a 3rd parameter. Each
command file drops its splitArgs(getCommandArguments(msg.text)) call
and the corresponding imports. command-utils.js loses the now-unused
arg helpers (only auth helpers remain). info.js — a 12-line file with
one sendMessage — folds into bot.js's commands map.
2026-05-09 22:35:38 +07:00
tiennm99 ee491f9eb5 refactor: merge apple/google app caches into one parametrized repo
Two near-identical per-store repository files collapse into a single
createAppCacheRepository(handle, prefix, ttl) factory used twice from
app-builder. Scrapers now receive the cache directly (one less layer),
and the cache entry shape drops the obsolete _id field — the Redis key
already encodes the appId.
2026-05-09 22:32:56 +07:00
tiennm99 9bcdaa9130 refactor: drop dead repo + telegram-api exports, inline createStore
Delete repository/store.js (one-line aggregator) — wiring now inline in
app-builder.js. Drop unused exports: scan + UpstashUnavailable from
upstash.js, getMe + TelegramApiError class from telegram-api.js,
init/getAdmin/save from admin-repository.js, exists/saveGroup from
group-repository.js. Generic Error replaces the named error classes.
2026-05-09 22:32:12 +07:00
tiennm99 32b14217cc refactor: drop Mongo class discriminator + delete src/models/
Inline trivial factory bodies into the repos and scrapers that used them.
The class:/_id: fields were Java-Mongo parity artifacts that nothing
in this codebase reads — Redis docs with the old fields still parse
fine; the next write drops them.
2026-05-09 22:30:48 +07:00
tiennm99 f1710a5698 chore: remove cloudflare + docker + legacy migration scripts
Phase 7 cleanup of the Vercel + Upstash consolidation plan:

- delete wrangler.toml, Dockerfile, docker-compose{,.dev}.yml,
  scripts/migrate-atlas-to-upstash.js (one-shot migration done)
- drop wrangler + mongodb devDeps and migrate* npm scripts;
  regenerate package-lock.json (-70 packages)
- prune CF/Wrangler/Atlas-export entries from .gitignore + .vercelignore
- drop MONGODB_URI from .env.deploy.example
- rewrite README for Vercel + Upstash architecture
- refresh stale Cloudflare comments in src/{logger,models,repository}
2026-05-09 21:49:48 +07:00
tiennm99 cad84e21a4 feat: accept KV_REST_API_* env vars as Upstash fallback
Vercel Marketplace Upstash integration injects KV_REST_API_URL and
KV_REST_API_TOKEN — different names from vanilla Upstash signup
(UPSTASH_REDIS_REST_URL / UPSTASH_REDIS_REST_TOKEN). Adapter and
migration script now accept either form, so the operator doesn't have
to duplicate values when sharing an Upstash DB provisioned via the
Vercel integration. UPSTASH_* takes precedence when both are set.
2026-05-09 20:17:06 +07:00
tiennm99 73762fa0a0 feat: migrate to Vercel + Upstash with KEY_PREFIX namespacing
Phases 1-5 of consolidate-vercel-upstash plan. Replaces Cloudflare
Workers + KV with Vercel serverless functions + Upstash Redis. Inlines
app-store-scraper / google-play-scraper npm libs (drops the
store-scraper.vercel.app HTTP roundtrip). KEY_PREFIX (default
'store-scraper-bot:') namespaces all Redis keys so the Upstash DB can
be safely shared with other Vercel projects.

- vercel.json + .vercelignore + Vercel-aware package.json scripts
- api/webhook.js + api/cron.js Vercel functions (with shared
  src/app-builder.js); cron auth fails closed when CRON_SECRET unset
- src/repository/upstash.js replaces kv.js; all 4 repos take a handle
  bundling client + prefix
- scripts/migrate-atlas-to-upstash.js writes legacy Java Atlas state
  directly to Upstash with --dry-run + --include-cache flags
- .env.example refreshed for the new env surface

Phases 6 (Vercel deploy + webhook cutover) and 7 (Docker + wrangler
cleanup) remain operator-driven post-deploy.
2026-05-09 20:07:07 +07:00
tiennm99 9898f408a2 feat: replace MongoDB driver with Cloudflare KV storage
Drops the mongodb dependency entirely; all four logical collections
(admin singleton, group, apple_app, google_app) now live in a single KV
namespace bound as STORE_KV with prefixed keys. Cache TTL is delegated
to KV via expirationTtl (clamped to the 60s minimum). Document shape,
field names, and Java parity at the doc level are preserved.

- Adds src/repository/kv.js helper (getJson/putJson/del with TTL clamp)
- Rewrites all four *-repository.js modules on top of KV
- Removes src/repository/mongodb.js and the MONGODB_URI env requirement
- Adds an early STORE_KV-binding guard in src/index.js
- Bumps to 0.3.0
2026-05-05 20:39:01 +07:00
tiennm99 797f040574 feat: Cloudflare Workers code port (deploy pending)
Refactors source to be Worker-shaped. No live deploy yet — sister deploy plan
runs Atlas provisioning + smoke later.

- wrangler.toml with nodejs_compat_v2, daily UTC 0 cron (= 7am Asia/Ho_Chi_Minh)
- package.json: drop node-telegram-bot-api, node-cron, dotenv, pino,
  pino-pretty; add wrangler devDep; bump to 0.2.0
- src/bot/telegram-api.js: raw fetch wrapper for Telegram Bot API
- src/bot/dispatch.js: per-message dispatcher extracted from polling loop
- src/repository/mongodb.js: memoized MongoClient per warm isolate, typed
  MongoUnavailable error, fast-fail timeouts
- src/repository/store.js: factory binding env once
- All 4 repositories converted to factory shape
- src/api/{apple,google}-scraper.js: take store instead of importing repos
- src/index.js: Worker entry exporting { fetch, scheduled }; webhook validates
  X-Telegram-Bot-Api-Secret-Token; ack-then-waitUntil pattern
- src/scheduler/scheduler.js: trimmed; runDailyCheck only (no node-cron)
- src/config.js, src/logger.js: env-driven, console.log JSON output
- scripts/register-webhook.js: setWebhook + setMyCommands; --dry-run supported
- scripts/check-secret-leaks.js: lint blocks console.log(env.<SECRET>)
- plans/260426-2015-cloudflare-worker-code-port: this code port plan
- plans/260426-2327-cloudflare-deploy-and-smoke: sister deploy plan

Validated via node --check on all 32 source files; lint clean. Real deploy
gates (bundle size, cold-start CPU) run in deploy plan.
2026-04-26 23:36:39 +07:00
tiennm99 3c97553c69 feat: initial JavaScript port of store-scraper-bot
Node.js 20+ ESM port mirroring Java/Go implementations.

- 13 Telegram commands matching Java identifiers
- MongoDB schema parity (common, group, apple_app, google_app collections)
- Apple/Google scrapers calling store-scraper.vercel.app with 10-min cache
- Daily 7am Vietnam-time cron with weekend-silent mode
- HTML table renderer matching Java/Go output
- Docker + Compose (prod and dev)

Untested end-to-end against live Telegram or upstream API.
2026-04-26 20:16:00 +07:00
tiennm99 c5f3628f01 Clear and rewrite
Retire the Java implementation; history continues from the Go port (store-scraper-bot-go).
2026-08-04 22:59:44 +07:00
tiennm99 e794000df3 feat(mongodb): manual handle expire 2025-12-23 21:15:33 +07:00
tiennm99 d99798fae7 feat(repo): cleanup 2025-12-16 22:08:25 +07:00
tiennm99 bb4c12e29e fix(mongo): codec 2025-12-16 21:57:57 +07:00
tiennm99 b048cbc0a5 fix(deserialize): 2025-12-15 22:56:37 +07:00
tiennm99 c060aa891d feat(codec): try to auto handle (de)serialize 2025-12-15 22:49:57 +07:00
tiennm99 8394f909ae feat(mongo): cleanup 2025-12-15 22:46:21 +07:00
tiennm99 0eae89ac16 feat(db): use mongodb instead of couchbase (WIP) 2025-11-22 23:05:39 +07:00
tiennm99 dae676da69 fix(schedule): exception break schedule 2025-11-17 12:07:21 +07:00
tiennm99 23fd181c0f Update log4j2.xml 2025-11-14 18:55:22 +07:00
tiennm99 d5467f16bc Update log4j2.xml 2025-11-14 18:32:53 +07:00
tiennm99 56b6dc5cb3 feat(loki): try to enable loki again 2025-11-14 18:25:46 +07:00
tiennm99 7335c2757e feat(loki): remove loki 2025-11-14 01:29:02 +07:00
tiennm99 d8a945a619 Revert "Update Main.java"
This reverts commit e53b1884c4.
2025-11-14 01:25:35 +07:00
tiennm99 fb4401f345 Update Main.java 2025-11-14 01:22:08 +07:00
tiennm99 3f27a213e4 Update Log4j2 config and shadowJar transformer usage
Removed the custom 'packages' attribute from log4j2.xml configuration and updated the shadowJar task to use the class reference for Log4j2PluginsCacheFileTransformer. This streamlines configuration and aligns with recommended usage.
2025-11-14 01:05:24 +07:00
tiennm99 805db99045 Update log4j2 config and shadowJar task
Added 'packages' attribute to log4j2.xml Configuration for custom Log4j2 plugins. Removed mergeServiceFiles() from shadowJar task in build.gradle.kts, relying only on Log4j2PluginsCacheFileTransformer.
2025-11-14 01:01:32 +07:00
tiennm99 671f1c64d9 feat(loki): add loki again
after fix shadow jar issue
2025-11-14 00:49:04 +07:00
tiennm99 7739590c4a Update log4j2.xml 2025-11-14 00:35:46 +07:00
tiennm99 ce302ee243 Update GooglePlayScraperTest.java 2025-11-13 21:55:38 +07:00
tiennm99 4fd11e99d4 Update Main.java 2025-11-13 21:48:03 +07:00
tiennm99 aabc96db10 Update AddAppleAppCommand.java 2025-11-13 21:40:20 +07:00
tiennm99 617e8073b5 refactor: cleanup code
And test docker build time
2025-11-13 21:29:24 +07:00
tiennm99 fda465925c Update CouchbaseUtil.java 2025-11-13 21:05:33 +07:00
tiennm99 0f5fdca3e7 feat(loki): send logs to loki to centralize logs 2025-11-13 18:24:13 +07:00
tiennm99 ec9ad27f09 feat(bot): add delgoogle 2025-11-12 10:08:26 +07:00
tiennm99 22a3f6bc5c feat(api): update base url 2025-11-11 20:59:56 +07:00
tiennm99 de1f84d278 feat(self scraper): try to crawl without country
But fail
2025-11-12 23:10:00 +07:00
tiennm99 0f64f53d70 refactor: old api 2025-11-12 22:20:15 +07:00
tiennm99 8830041ace Create GooglePlayAppScraper.java 2025-11-11 20:26:50 +07:00
tiennm99 de46765d0c feat(apple): correct api response 2025-11-10 11:32:00 +07:00
tiennm99 f67b175f27 feat(app score): round score 2025-11-10 11:16:06 +07:00
tiennm99 3c1d60b248 feat(score): add score text 2025-11-10 11:04:42 +07:00
tiennm99 a4646c91ca Add raw app response commands for Apple and Google
Introduces RawAppleAppCommand and RawGoogleAppCommand to allow admins to fetch and download raw JSON responses from the Apple App Store and Google Play Store scrapers. Refactors AppStoreScraper and GooglePlayScraper to expose rawApp methods and improves method naming for clarity. Updates StoreScrapeBot to register the new commands and adjusts .env.example for better configuration clarity.
2025-11-09 10:12:19 +07:00
tiennm99 b3c3ca9cf3 Format long sendMessage calls for readability
Refactored sendMessage calls in DeleteAppleAppCommand and InfoCommand to split long lines for improved code readability. No functional changes were made.
2025-11-08 01:28:53 +07:00
tiennm99 c0facd5ba2 Refactor repository structure for Couchbase collections
Introduced AbstractCollectionRepository and AbstractSingletonRepository to clarify repository types and encapsulate logic for single-key and collection-based repositories. Updated AdminRepository, AppleAppRepository, GoogleAppRepository, and GroupRepository to extend the appropriate abstract classes. Removed COMMON_COLLECTION_NAME from Constant.java and enforced collection name restrictions in AbstractRepository.
2025-11-08 00:05:09 +07:00
tiennm99 7349501086 Add scheduled app update checks and notifications
Introduces a scheduled task that checks for apps not updated within a warning threshold and sends notifications to groups. Adds scheduling utilities, constants for scheduling, and logic to format and send update reports for both Apple and Google apps. Also includes a new NonUpdatedApp record and improvements to the Table class for better formatting.
2025-11-07 23:45:30 +07:00
tiennm99 77f4e08946 Refactor config to Environment and add Table utility
Replaces Config with Environment throughout the codebase for environment variable management. Introduces a Table utility class for formatting tabular data in bot commands, refactoring commands to use this for improved output formatting. Removes unused AppType enum. Updates test cases and build configuration for better IDE integration.
2025-11-07 23:11:16 +07:00
tiennm99 4c62fde11f Replace SOURCE_VERSION with SOURCE_COMMIT in config
Renamed the SOURCE_VERSION constant to SOURCE_COMMIT in Config.java and updated its usage in Main.java. The environment variable and related references now consistently use SOURCE_COMMIT to better reflect the tracked value.
2025-11-07 21:08:05 +07:00
tiennm99 cced06fda9 Notify creator on bot startup and enhance config
Added a startup notification to the creator with the current bot version. Refactored ADMIN_IDS to a List, introduced CREATOR_ID and SOURCE_VERSION constants in Config, and improved environment variable handling.
2025-11-07 21:04:02 +07:00