Add a pdfout package that renders the fetched chapters to a single PDF
and an export package holding the whole pipeline, so the CLI and an
embedding program share one code path and reject the same inputs.
The default page is 90x160mm rather than A4 with large type: a viewer
scales a whole page to fit the screen, so on a phone the page shape
decides legibility and a wide page just gets shrunk.
Embed DejaVu Sans in the binary. Vietnamese needs the Latin Extended
Additional block, and a headless host often has no fonts installed at
all, so rendering must not depend on finding one. An explicitly named
font is never silently substituted. Font data is passed as bytes because
fpdf resolves a font path against its own font directory.
Keep the original per-chapter text output available behind -txt.
The crawler took one hardcoded novel URL, wrote per-chapter text files
into ./data, and fetched serially until the site answered 429.
Rename the module to hako-crawler and split fetching, page parsing and
crawl orchestration into a hako package that takes any novel URL.
The chapter body is no longer served as markup: #chapter-content now
carries a shuffled, XOR-obfuscated payload that the site's own script
expands in the browser, so the previous p[id=<digits>] extractor
returned nothing at all. Decode it, keeping the plain-markup path as a
fallback. The scheme is read from the page and an unrecognised one is a
hard error, because the alternative is exporting a book of mojibake.
Space requests globally instead of per-goroutine, so concurrency no
longer raises the request rate, and retry only what a retry can fix.
Cache raw pages so re-runs cost no requests.
BREAKING CHANGE: module path is now github.com/tiennm99/hako-crawler,
and the novel URL is a required argument rather than a constant.
Nodejs/node_modules was committed (4,275 files), which is also how three
upstream yarn.lock files ended up in the repository. package-lock.json already
pins everything, and all five dependencies still resolve from the registry, so
the tree is reproducible with npm ci and does not need to be stored here.
Add a .gitignore; the repository had none.
Novel gains Tags, read from the anchors carrying schema.org itemprop="genre".
Matching that microdata rather than the /the-loai/ href shape is what keeps the
site-wide genre navigation out: a sampled page links 69 categories against the
novel's own 6.
NovelInfo fetches just the landing page and parses it, so a caller that only
wants metadata pays one request instead of the two Novel needs for its chapter
list cross-check.
On the default 90x160mm phone page, 12pt fits about 36 characters per line;
10pt fits about 43 across 26 lines, which reads denser without becoming small
on a phone at 100% zoom. -font-size still overrides it.
PDF rendering failed on Linux with "stat usr/share/fonts/...: no such file or
directory": fpdf joins the font path onto its own font directory, which it
defaults to ".", so path.Join turns an absolute path into a
working-directory-relative one. It only resolved when the process happened to
run from the filesystem root, which is why Windows was unaffected. Font data is
now read by pdfout and handed over as bytes.
LoadFont also falls back to a bundled DejaVu Sans, so a host with no fonts
installed still renders. An explicitly requested font remains a hard error when
unreadable rather than being silently substituted. Tests verify the bundled font
parses and covers Vietnamese, which is the reason it exists.
Move monkeyd and pdfout out of internal/ so other modules can import them,
and add export.Export, which holds the crawl-to-PDF sequence the CLI used to
inline. The CLI now parses flags and delegates, so an embedding program gets
the same defaults and validation.
Three blocks the site injects inside the chapter body were being extracted
as prose: the sponsor block, a source watermark planted mid-chapter, and
the prev/next chapter links.
Anchors and images are now skipped, since inside a chapter body they are
always site chrome, and the sponsor wrapper and watermark are skipped by
class. Inline emphasis tags are left alone so italics in the prose survive.
The sponsor wrapper has a sibling that must NOT be skipped: on most
chapters the real text sits in a container hidden with display:none,
because the body is gated behind a click on the sponsor link and revealed
with JavaScript. Skipping hidden elements, or treating the two sibling
classes as one family, discards the entire chapter. A test pins the gated
body as kept while its sibling is dropped.
Removes 662 words of injected text across the sample novel and lets the
repeated chapter-number paragraph be recognised again, which the sponsor
block had been masking by displacing it from the top of the body.
Fetches every chapter of a monkeydd.com novel and renders it as a single
PDF laid out for reading on a phone.
Two site behaviours drive the extractor design:
- Roughly a fifth of each chapter's words are not in the markup. The page
emits empty spans and supplies the word from the stylesheet via
":before { content: ... }" rules, so reading DOM text alone drops them
with no error. The extractor resolves those rules and substitutes the
words back; a test asserts they disappear when the rule is removed.
- Chapter URLs cannot be generated. Numbering has gaps and slugs are not
uniform across novels, so chapter links are always parsed from the page.
The chapter list is read from the landing page and cross-checked against
the dropdown embedded in each chapter page, so a truncated list cannot
silently shorten the export.
PDF defaults to a 90x160mm page rather than A4 with large type: viewers
scale a whole page to fit the screen, so a phone-shaped page fills it at
100% zoom where 12pt stays comfortable. A5 and A4 remain available.
Requests are spaced globally, so raising worker count does not raise the
request rate. Raw pages cache to disk so re-exporting at different font
or page settings needs no network.
Port both scripts into one Go program with a command per behavior:
gallery walks the numbered ghibli.jp film galleries, scrape downloads
every image referenced by a page.
Downloads now run through a bounded worker pool, and non-200 responses
are no longer written to disk as image files. Drop the orphaned
package-lock.json, which pinned the deprecated request dependency and
was the source of the repository's Dependabot alerts.
tiennm99
dependabot[bot]