docs: describe the in-tree monkeyd crawler and record the port plan

AGENTS.md

@@ -7,11 +7,10 @@
 in-memory otherwise, which is what tests and local no-database runs use. Read
 `README.md` before implementation work.
 
-`third_party/monkeyd-crawler` is a git submodule resolved through a `go.mod`
-`replace` directive, not a versioned dependency. Go commands fail until it is
-checked out (`git submodule update --init --recursive`). Changes to the crawler
-belong in its own repository and must be pushed before the submodule pointer is
-advanced here, or fresh clones cannot resolve the pinned commit.
+The `monkeyd` module's crawler, PDF renderer, and export flow are an in-tree
+port of `tiennm99/mttools/monkeyd-crawler` under
+`internal/modules/monkeyd/{crawler,pdf,export}`. Change them here; they no
+longer track the mttools copy.
 
 ## Development Rules
 

README.md

@@ -163,9 +163,11 @@ prevents a position opened after Record date from applying an older event.
 
 `/monkeyd_crawl <url> [font_size]` downloads every chapter of a monkeydd.com
 novel and sends it back as a single PDF document, sized for reading on a phone.
-The crawling and rendering come from the
-[monkeyd-crawler](https://github.com/tiennm99/monkeyd-crawler) submodule; the
-module is the Telegram surface around it.
+The crawler, the PDF renderer, and the export flow live in
+`internal/modules/monkeyd/{crawler,pdf,export}`, ported from
+[mttools/monkeyd-crawler](https://github.com/tiennm99/mttools/tree/main/monkeyd-crawler)
+and trimmed to what the bot uses: a fixed phone page and the bundled DejaVu
+Sans font.
 
 `font_size` is the body text size in points and accepts half points. It ranges
 from 6 to 24 and defaults to the crawler's own default of 10, which fits roughly
@@ -238,7 +240,6 @@ internal/modules/            Module framework, registry, dispatchers, modules
 internal/storage/            typed DocStore[T] (Provider + Typed); mongodb runtime + memory (tests). Values persist as flattened native BSON root documents
 internal/systemstate/        shared `system` collection helper for startup migration records
 internal/log/, metrics/      JSON logging (LOG_LEVEL) and periodic metrics flush
-third_party/monkeyd-crawler/ git submodule; resolved by a go.mod replace directive
 compose.yml                  Coolify self-host stack (single bot service)
 ```
 
@@ -254,20 +255,6 @@ compose.yml                  Coolify self-host stack (single bot service)
 
 ## Run locally
 
-Clone with submodules — the `monkeyd` module builds against
-`third_party/monkeyd-crawler`, and Go resolves it through a `replace` directive
-pointing at that directory:
-
-```sh
-git clone --recurse-submodules https://github.com/tiennm99/miti99bot.git
-
-# already cloned without them:
-git submodule update --init --recursive
-```
-
-Without the submodule checked out, every Go command fails to resolve
-`github.com/tiennm99/monkeyd-crawler`.
-
 In-memory storage requires no database. Set the environment variables for your
 shell, then run the server with Go:
 

docs/deploy-coolify-selfhosted.md

@@ -42,7 +42,7 @@ Copy [`.env.example`](../.env.example) → `.env` (gitignored) and fill in.
 | `GOLD_VNAPP_API_KEY` | leave unset | VNAppMob key; unset = the gold module fetches one and caches it in MongoDB |
 | `KV_PROVIDER` | leave unset | `memory` or `mongodb`; unset = `mongodb` when `MONGO_URL` is set, otherwise `memory` |
 | `PORT` | leave unset | health server port; default `8080` |
-| `SOURCE_COMMIT` | never set | provided by Coolify at runtime for the deploy DM (see step 6 below) |
+| `SOURCE_COMMIT` | never set | provided by Coolify at runtime for the deploy DM (see step 5 below) |
 
 Stock, coin, and gold provider URL overrides are not supported in runtime env;
 modules use coded defaults. There is no `TELEGRAM_WEBHOOK_SECRET`: long polling
@@ -148,22 +148,15 @@ MP4, with the same text fallback.
 1. New resource → from this Git repo (Docker Compose), or a prebuilt image.
    The committed [`compose.yml`](../compose.yml) defines the single
    `bot` service.
-2. **Enable submodule checkout.** The `monkeyd` module builds against
-   `third_party/monkeyd-crawler`, a git submodule wired in through a `go.mod`
-   `replace` directive. Coolify must clone submodules, or the Docker build
-   fails at `go mod download` with an unresolved
-   `github.com/tiennm99/monkeyd-crawler`. Turn on Coolify's recursive-clone /
-   submodule option for the resource. There is no build without it: leaving
-   `monkeyd` out of `MODULES` only disables the commands at runtime.
-3. Set the env vars above in Coolify.
-4. **No public domain / port** is needed — polling is outbound-only. Do not
+2. Set the env vars above in Coolify.
+3. **No public domain / port** is needed — polling is outbound-only. Do not
    publish a port or attach a domain. `expose: 8080` keeps the health endpoint
    reachable only inside Coolify's network.
-5. **Exactly one replica.** Telegram permits only one `getUpdates` consumer per
+4. **Exactly one replica.** Telegram permits only one `getUpdates` consumer per
    bot token; a second poller gets HTTP 409, and a second in-process scheduler
    double-fires crons. Prefer **stop-first redeploys** so two containers never
    overlap near a cron time.
-6. **deploynotify commit SHA:** `SOURCE_COMMIT` is a Coolify predefined
+5. **deploynotify commit SHA:** `SOURCE_COMMIT` is a Coolify predefined
    variable. The bot reads it at startup and DMs the owner on every boot;
    outside Coolify (local `docker compose up`) it is unset and the DM shows
    `unknown`. Keep "Include Source Commit in Build" disabled: that setting
@@ -171,7 +164,7 @@ MP4, with the same text fallback.
    invalidate Docker cache on every commit. Do not add `SOURCE_COMMIT` to
    `compose.yml`; an interpolated empty value can override Coolify's runtime
    env-file value.
-7. **Health check:** use Coolify's HTTP monitor against `GET /` (returns
+6. **Health check:** use Coolify's HTTP monitor against `GET /` (returns
    `text/plain` `miti99bot ok`). The committed `compose.yml` defines no
    `healthcheck`, and `cmd/server` has no `-healthcheck` flag. Note: `/`
    reports healthy even if Mongo is unreachable (the driver auto-reconnects on

plans/261003-1118-port-monkeyd-crawler-in-tree/plan.md

@@ -0,0 +1,83 @@
+---
+status: completed
+mode: port
+---
+
+# Port monkeyd-crawler into the monkeyd module
+
+## Outcome
+
+The bot builds without the `third_party/monkeyd-crawler` submodule. The crawler,
+PDF renderer, and export flow live in-tree under `internal/modules/monkeyd/`,
+trimmed to what the bot uses. Coolify deploys succeed again.
+
+Why now: `tiennm99/monkeyd-crawler` was merged into `tiennm99/mttools` and the
+old repo is gone, so every recursive clone fails (Coolify deployment
+`6klach0etmr5ymnfrvbkq77a`, "Repository not found").
+
+## Source manifest
+
+- Source: `tiennm99/mttools`, path `monkeyd-crawler/`, commit `03b1400`
+- It matches the pinned submodule commit `d88f2a4` except for import paths
+- License: Apache-2.0, same author as this repo (also Apache-2.0); bundled
+  DejaVu Sans keeps its `NOTICE.md`
+
+## Source anatomy
+
+| Package | Role | Bot uses |
+|---|---|---|
+| `monkeyd` | HTTP client (rate limit, retries, size cap), disk cache, novel and chapter parsing, CSS word-class decoding, worker pool | `Crawler.NovelInfo` and `NewClient` for `/monkeyd_tags`, plus the export path |
+| `pdfout` | fpdf layout, page presets, font discovery plus bundled DejaVu | the export path, plus `Presets` in a test |
+| `export` | orchestration: defaults, validation, crawl, render | `Export`, `Request`, `Result`, `DefaultFontSize`, `DefaultDelay` |
+| `cmd/monkeyd-crawler` | CLI | nothing |
+
+## Dependency matrix
+
+| Source | Local | Status |
+|---|---|---|
+| `monkeyd` package | `internal/modules/monkeyd/crawler` | NEW (port as-is) |
+| `pdfout` package | `internal/modules/monkeyd/pdf` | NEW (drop system font discovery) |
+| `export` package | `internal/modules/monkeyd/export` | NEW (trim the request surface) |
+| CLI | none | DROPPED |
+| `go-pdf/fpdf`, `x/net`, `x/sync`, `x/image` | `go.mod` | EXISTS (indirect deps become direct) |
+| submodule, `replace`, Dockerfile COPY, CI `submodules: true` | none | REMOVED |
+
+## Decision matrix
+
+| Decision | Source's way | Our way | Recommendation |
+|---|---|---|---|
+| Distribution | Go module in mttools | in-tree packages | In-tree, as requested. No cross-repo pin to drift, and Coolify needs no recursive clone |
+| Font | system font discovery, then bundled | bundled only | Bundled only. Production has no system fonts, so its output is unchanged, and dev machines render the same PDF |
+| Page presets | phone, a5, a4 | phone only | Fixed phone page; the bot exposes no page option |
+| Export knobs | page, font file, spacing, margin, workers, retries, delay, limit, out path, no-cache, no-delay | URL, out dir, font size, cache dir, log | Keep only what the bot sets. The rest become constants with the source's defaults |
+| CLI | `cmd/monkeyd-crawler` | none | Drop; mttools keeps the CLI |
+
+Risk score: 3 of 10. The logic is Go to Go and identical to what runs today.
+The main risk is a trim that drops behavior, and the ported tests cover that.
+
+## Phases
+
+1. Port the packages. Copy `monkeyd`, `pdfout`, and `export` with their tests
+   into `internal/modules/monkeyd/{crawler,pdf,export}`. Rewrite imports and
+   apply the trims above. Remove tests that only cover dropped features.
+2. Rewire the module. Point `export_job.go`, `monkeyd.go`, `tags_command.go`,
+   and `handlers_test.go` at the new packages.
+3. Remove the submodule. Delete `third_party/monkeyd-crawler`, `.gitmodules`,
+   and the `replace`/`require` lines; run `go mod tidy`. Drop the Dockerfile
+   COPY and CI `submodules: true`.
+4. Update docs: README (layout, run locally, clone instructions), AGENTS.md,
+   and `docs/deploy-coolify-selfhosted.md` (recursive clone note).
+
+## Acceptance criteria
+
+- `go build ./...`, `go vet ./...`, `go test ./...`, and `golangci-lint run`
+  pass with no submodule present
+- `docker compose build` passes
+- A live export of a real novel URL produces a PDF with Vietnamese diacritics
+- The Coolify deployment of the pushed commit finishes
+
+## Rollback
+
+Revert the commits. The old submodule URL no longer resolves, so a rollback
+needs `.gitmodules` pointed at `tiennm99/mttools` with the path adjusted, or a
+module dependency on `github.com/tiennm99/mttools/monkeyd-crawler`.