Nodejs/node_modules was committed (4,275 files), which is also how three
upstream yarn.lock files ended up in the repository. package-lock.json already
pins everything, and all five dependencies still resolve from the registry, so
the tree is reproducible with npm ci and does not need to be stored here.
Add a .gitignore; the repository had none.
Novel gains Tags, read from the anchors carrying schema.org itemprop="genre".
Matching that microdata rather than the /the-loai/ href shape is what keeps the
site-wide genre navigation out: a sampled page links 69 categories against the
novel's own 6.
NovelInfo fetches just the landing page and parses it, so a caller that only
wants metadata pays one request instead of the two Novel needs for its chapter
list cross-check.
On the default 90x160mm phone page, 12pt fits about 36 characters per line;
10pt fits about 43 across 26 lines, which reads denser without becoming small
on a phone at 100% zoom. -font-size still overrides it.
PDF rendering failed on Linux with "stat usr/share/fonts/...: no such file or
directory": fpdf joins the font path onto its own font directory, which it
defaults to ".", so path.Join turns an absolute path into a
working-directory-relative one. It only resolved when the process happened to
run from the filesystem root, which is why Windows was unaffected. Font data is
now read by pdfout and handed over as bytes.
LoadFont also falls back to a bundled DejaVu Sans, so a host with no fonts
installed still renders. An explicitly requested font remains a hard error when
unreadable rather than being silently substituted. Tests verify the bundled font
parses and covers Vietnamese, which is the reason it exists.
Move monkeyd and pdfout out of internal/ so other modules can import them,
and add export.Export, which holds the crawl-to-PDF sequence the CLI used to
inline. The CLI now parses flags and delegates, so an embedding program gets
the same defaults and validation.
Three blocks the site injects inside the chapter body were being extracted
as prose: the sponsor block, a source watermark planted mid-chapter, and
the prev/next chapter links.
Anchors and images are now skipped, since inside a chapter body they are
always site chrome, and the sponsor wrapper and watermark are skipped by
class. Inline emphasis tags are left alone so italics in the prose survive.
The sponsor wrapper has a sibling that must NOT be skipped: on most
chapters the real text sits in a container hidden with display:none,
because the body is gated behind a click on the sponsor link and revealed
with JavaScript. Skipping hidden elements, or treating the two sibling
classes as one family, discards the entire chapter. A test pins the gated
body as kept while its sibling is dropped.
Removes 662 words of injected text across the sample novel and lets the
repeated chapter-number paragraph be recognised again, which the sponsor
block had been masking by displacing it from the top of the body.
Fetches every chapter of a monkeydd.com novel and renders it as a single
PDF laid out for reading on a phone.
Two site behaviours drive the extractor design:
- Roughly a fifth of each chapter's words are not in the markup. The page
emits empty spans and supplies the word from the stylesheet via
":before { content: ... }" rules, so reading DOM text alone drops them
with no error. The extractor resolves those rules and substitutes the
words back; a test asserts they disappear when the rule is removed.
- Chapter URLs cannot be generated. Numbering has gaps and slugs are not
uniform across novels, so chapter links are always parsed from the page.
The chapter list is read from the landing page and cross-checked against
the dropdown embedded in each chapter page, so a truncated list cannot
silently shorten the export.
PDF defaults to a 90x160mm page rather than A4 with large type: viewers
scale a whole page to fit the screen, so a phone-shaped page fills it at
100% zoom where 12pt stays comfortable. A5 and A4 remain available.
Requests are spaced globally, so raising worker count does not raise the
request rate. Raw pages cache to disk so re-exporting at different font
or page settings needs no network.
Port both scripts into one Go program with a command per behavior:
gallery walks the numbered ghibli.jp film galleries, scrape downloads
every image referenced by a page.
Downloads now run through a bounded worker pool, and non-200 responses
are no longer written to disk as image files. Drop the orphaned
package-lock.json, which pinned the deprecated request dependency and
was the source of the repository's Dependabot alerts.
tiennm99
dependabot[bot]