From de46809c962d77737cf36fc33d24c39f0931a97f Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Sat, 3 Oct 2026 11:23:45 +0700 Subject: [PATCH] docs: describe the in-tree monkeyd crawler and record the port plan --- AGENTS.md | 9 +- README.md | 23 ++--- docs/deploy-coolify-selfhosted.md | 19 ++--- .../plan.md | 83 +++++++++++++++++++ 4 files changed, 98 insertions(+), 36 deletions(-) create mode 100644 plans/261003-1118-port-monkeyd-crawler-in-tree/plan.md diff --git a/AGENTS.md b/AGENTS.md index 7906bc7..fc993b3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,11 +7,10 @@ in-memory otherwise, which is what tests and local no-database runs use. Read `README.md` before implementation work. -`third_party/monkeyd-crawler` is a git submodule resolved through a `go.mod` -`replace` directive, not a versioned dependency. Go commands fail until it is -checked out (`git submodule update --init --recursive`). Changes to the crawler -belong in its own repository and must be pushed before the submodule pointer is -advanced here, or fresh clones cannot resolve the pinned commit. +The `monkeyd` module's crawler, PDF renderer, and export flow are an in-tree +port of `tiennm99/mttools/monkeyd-crawler` under +`internal/modules/monkeyd/{crawler,pdf,export}`. Change them here; they no +longer track the mttools copy. ## Development Rules diff --git a/README.md b/README.md index 5a1f849..91ec9f8 100644 --- a/README.md +++ b/README.md @@ -163,9 +163,11 @@ prevents a position opened after Record date from applying an older event. `/monkeyd_crawl [font_size]` downloads every chapter of a monkeydd.com novel and sends it back as a single PDF document, sized for reading on a phone. -The crawling and rendering come from the -[monkeyd-crawler](https://github.com/tiennm99/monkeyd-crawler) submodule; the -module is the Telegram surface around it. +The crawler, the PDF renderer, and the export flow live in +`internal/modules/monkeyd/{crawler,pdf,export}`, ported from +[mttools/monkeyd-crawler](https://github.com/tiennm99/mttools/tree/main/monkeyd-crawler) +and trimmed to what the bot uses: a fixed phone page and the bundled DejaVu +Sans font. `font_size` is the body text size in points and accepts half points. It ranges from 6 to 24 and defaults to the crawler's own default of 10, which fits roughly @@ -238,7 +240,6 @@ internal/modules/ Module framework, registry, dispatchers, modules internal/storage/ typed DocStore[T] (Provider + Typed); mongodb runtime + memory (tests). Values persist as flattened native BSON root documents internal/systemstate/ shared `system` collection helper for startup migration records internal/log/, metrics/ JSON logging (LOG_LEVEL) and periodic metrics flush -third_party/monkeyd-crawler/ git submodule; resolved by a go.mod replace directive compose.yml Coolify self-host stack (single bot service) ``` @@ -254,20 +255,6 @@ compose.yml Coolify self-host stack (single bot service) ## Run locally -Clone with submodules — the `monkeyd` module builds against -`third_party/monkeyd-crawler`, and Go resolves it through a `replace` directive -pointing at that directory: - -```sh -git clone --recurse-submodules https://github.com/tiennm99/miti99bot.git - -# already cloned without them: -git submodule update --init --recursive -``` - -Without the submodule checked out, every Go command fails to resolve -`github.com/tiennm99/monkeyd-crawler`. - In-memory storage requires no database. Set the environment variables for your shell, then run the server with Go: diff --git a/docs/deploy-coolify-selfhosted.md b/docs/deploy-coolify-selfhosted.md index 6f4337e..082d77f 100644 --- a/docs/deploy-coolify-selfhosted.md +++ b/docs/deploy-coolify-selfhosted.md @@ -42,7 +42,7 @@ Copy [`.env.example`](../.env.example) → `.env` (gitignored) and fill in. | `GOLD_VNAPP_API_KEY` | leave unset | VNAppMob key; unset = the gold module fetches one and caches it in MongoDB | | `KV_PROVIDER` | leave unset | `memory` or `mongodb`; unset = `mongodb` when `MONGO_URL` is set, otherwise `memory` | | `PORT` | leave unset | health server port; default `8080` | -| `SOURCE_COMMIT` | never set | provided by Coolify at runtime for the deploy DM (see step 6 below) | +| `SOURCE_COMMIT` | never set | provided by Coolify at runtime for the deploy DM (see step 5 below) | Stock, coin, and gold provider URL overrides are not supported in runtime env; modules use coded defaults. There is no `TELEGRAM_WEBHOOK_SECRET`: long polling @@ -148,22 +148,15 @@ MP4, with the same text fallback. 1. New resource → from this Git repo (Docker Compose), or a prebuilt image. The committed [`compose.yml`](../compose.yml) defines the single `bot` service. -2. **Enable submodule checkout.** The `monkeyd` module builds against - `third_party/monkeyd-crawler`, a git submodule wired in through a `go.mod` - `replace` directive. Coolify must clone submodules, or the Docker build - fails at `go mod download` with an unresolved - `github.com/tiennm99/monkeyd-crawler`. Turn on Coolify's recursive-clone / - submodule option for the resource. There is no build without it: leaving - `monkeyd` out of `MODULES` only disables the commands at runtime. -3. Set the env vars above in Coolify. -4. **No public domain / port** is needed — polling is outbound-only. Do not +2. Set the env vars above in Coolify. +3. **No public domain / port** is needed — polling is outbound-only. Do not publish a port or attach a domain. `expose: 8080` keeps the health endpoint reachable only inside Coolify's network. -5. **Exactly one replica.** Telegram permits only one `getUpdates` consumer per +4. **Exactly one replica.** Telegram permits only one `getUpdates` consumer per bot token; a second poller gets HTTP 409, and a second in-process scheduler double-fires crons. Prefer **stop-first redeploys** so two containers never overlap near a cron time. -6. **deploynotify commit SHA:** `SOURCE_COMMIT` is a Coolify predefined +5. **deploynotify commit SHA:** `SOURCE_COMMIT` is a Coolify predefined variable. The bot reads it at startup and DMs the owner on every boot; outside Coolify (local `docker compose up`) it is unset and the DM shows `unknown`. Keep "Include Source Commit in Build" disabled: that setting @@ -171,7 +164,7 @@ MP4, with the same text fallback. invalidate Docker cache on every commit. Do not add `SOURCE_COMMIT` to `compose.yml`; an interpolated empty value can override Coolify's runtime env-file value. -7. **Health check:** use Coolify's HTTP monitor against `GET /` (returns +6. **Health check:** use Coolify's HTTP monitor against `GET /` (returns `text/plain` `miti99bot ok`). The committed `compose.yml` defines no `healthcheck`, and `cmd/server` has no `-healthcheck` flag. Note: `/` reports healthy even if Mongo is unreachable (the driver auto-reconnects on diff --git a/plans/261003-1118-port-monkeyd-crawler-in-tree/plan.md b/plans/261003-1118-port-monkeyd-crawler-in-tree/plan.md new file mode 100644 index 0000000..6e87a1d --- /dev/null +++ b/plans/261003-1118-port-monkeyd-crawler-in-tree/plan.md @@ -0,0 +1,83 @@ +--- +status: completed +mode: port +--- + +# Port monkeyd-crawler into the monkeyd module + +## Outcome + +The bot builds without the `third_party/monkeyd-crawler` submodule. The crawler, +PDF renderer, and export flow live in-tree under `internal/modules/monkeyd/`, +trimmed to what the bot uses. Coolify deploys succeed again. + +Why now: `tiennm99/monkeyd-crawler` was merged into `tiennm99/mttools` and the +old repo is gone, so every recursive clone fails (Coolify deployment +`6klach0etmr5ymnfrvbkq77a`, "Repository not found"). + +## Source manifest + +- Source: `tiennm99/mttools`, path `monkeyd-crawler/`, commit `03b1400` +- It matches the pinned submodule commit `d88f2a4` except for import paths +- License: Apache-2.0, same author as this repo (also Apache-2.0); bundled + DejaVu Sans keeps its `NOTICE.md` + +## Source anatomy + +| Package | Role | Bot uses | +|---|---|---| +| `monkeyd` | HTTP client (rate limit, retries, size cap), disk cache, novel and chapter parsing, CSS word-class decoding, worker pool | `Crawler.NovelInfo` and `NewClient` for `/monkeyd_tags`, plus the export path | +| `pdfout` | fpdf layout, page presets, font discovery plus bundled DejaVu | the export path, plus `Presets` in a test | +| `export` | orchestration: defaults, validation, crawl, render | `Export`, `Request`, `Result`, `DefaultFontSize`, `DefaultDelay` | +| `cmd/monkeyd-crawler` | CLI | nothing | + +## Dependency matrix + +| Source | Local | Status | +|---|---|---| +| `monkeyd` package | `internal/modules/monkeyd/crawler` | NEW (port as-is) | +| `pdfout` package | `internal/modules/monkeyd/pdf` | NEW (drop system font discovery) | +| `export` package | `internal/modules/monkeyd/export` | NEW (trim the request surface) | +| CLI | none | DROPPED | +| `go-pdf/fpdf`, `x/net`, `x/sync`, `x/image` | `go.mod` | EXISTS (indirect deps become direct) | +| submodule, `replace`, Dockerfile COPY, CI `submodules: true` | none | REMOVED | + +## Decision matrix + +| Decision | Source's way | Our way | Recommendation | +|---|---|---|---| +| Distribution | Go module in mttools | in-tree packages | In-tree, as requested. No cross-repo pin to drift, and Coolify needs no recursive clone | +| Font | system font discovery, then bundled | bundled only | Bundled only. Production has no system fonts, so its output is unchanged, and dev machines render the same PDF | +| Page presets | phone, a5, a4 | phone only | Fixed phone page; the bot exposes no page option | +| Export knobs | page, font file, spacing, margin, workers, retries, delay, limit, out path, no-cache, no-delay | URL, out dir, font size, cache dir, log | Keep only what the bot sets. The rest become constants with the source's defaults | +| CLI | `cmd/monkeyd-crawler` | none | Drop; mttools keeps the CLI | + +Risk score: 3 of 10. The logic is Go to Go and identical to what runs today. +The main risk is a trim that drops behavior, and the ported tests cover that. + +## Phases + +1. Port the packages. Copy `monkeyd`, `pdfout`, and `export` with their tests + into `internal/modules/monkeyd/{crawler,pdf,export}`. Rewrite imports and + apply the trims above. Remove tests that only cover dropped features. +2. Rewire the module. Point `export_job.go`, `monkeyd.go`, `tags_command.go`, + and `handlers_test.go` at the new packages. +3. Remove the submodule. Delete `third_party/monkeyd-crawler`, `.gitmodules`, + and the `replace`/`require` lines; run `go mod tidy`. Drop the Dockerfile + COPY and CI `submodules: true`. +4. Update docs: README (layout, run locally, clone instructions), AGENTS.md, + and `docs/deploy-coolify-selfhosted.md` (recursive clone note). + +## Acceptance criteria + +- `go build ./...`, `go vet ./...`, `go test ./...`, and `golangci-lint run` + pass with no submodule present +- `docker compose build` passes +- A live export of a real novel URL produces a PDF with Vietnamese diacritics +- The Coolify deployment of the pushed commit finishes + +## Rollback + +Revert the commits. The old submodule URL no longer resolves, so a rollback +needs `.gitmodules` pointed at `tiennm99/mttools` with the path adjusted, or a +module dependency on `github.com/tiennm99/mttools/monkeyd-crawler`.