mirror of
https://github.com/tiennm99/MTTools.git
synced 2026-10-18 12:14:32 +00:00
feat: add monkeydd novel crawler with phone-sized PDF export
Fetches every chapter of a monkeydd.com novel and renders it as a single
PDF laid out for reading on a phone.
Two site behaviours drive the extractor design:
- Roughly a fifth of each chapter's words are not in the markup. The page
emits empty spans and supplies the word from the stylesheet via
":before { content: ... }" rules, so reading DOM text alone drops them
with no error. The extractor resolves those rules and substitutes the
words back; a test asserts they disappear when the rule is removed.
- Chapter URLs cannot be generated. Numbering has gaps and slugs are not
uniform across novels, so chapter links are always parsed from the page.
The chapter list is read from the landing page and cross-checked against
the dropdown embedded in each chapter page, so a truncated list cannot
silently shorten the export.
PDF defaults to a 90x160mm page rather than A4 with large type: viewers
scale a whole page to fit the screen, so a phone-shaped page fills it at
100% zoom where 12pt stays comfortable. A5 and A4 remain available.
Requests are spaced globally, so raising worker count does not raise the
request rate. Raw pages cache to disk so re-exporting at different font
or page settings needs no network.
This commit is contained in:
1 parent
9a1b3a43b0
commit
6bcdd742d6
16 files changed
+1667
No files matched your search
@@ -0,0 +1,9 @@
|
||||
module github.com/tiennm99/monkeyd-crawler
|
||||
|
||||
go 1.26.4
|
||||
|
||||
require (
|
||||
github.com/go-pdf/fpdf v0.9.0
|
||||
golang.org/x/net v0.57.0
|
||||
golang.org/x/sync v0.22.0
|
||||
)
|
||||
Reference in new issue
Block a user