Three blocks the site injects inside the chapter body were being extracted
as prose: the sponsor block, a source watermark planted mid-chapter, and
the prev/next chapter links.
Anchors and images are now skipped, since inside a chapter body they are
always site chrome, and the sponsor wrapper and watermark are skipped by
class. Inline emphasis tags are left alone so italics in the prose survive.
The sponsor wrapper has a sibling that must NOT be skipped: on most
chapters the real text sits in a container hidden with display:none,
because the body is gated behind a click on the sponsor link and revealed
with JavaScript. Skipping hidden elements, or treating the two sibling
classes as one family, discards the entire chapter. A test pins the gated
body as kept while its sibling is dropped.
Removes 662 words of injected text across the sample novel and lets the
repeated chapter-number paragraph be recognised again, which the sponsor
block had been masking by displacing it from the top of the body.
Fetches every chapter of a monkeydd.com novel and renders it as a single
PDF laid out for reading on a phone.
Two site behaviours drive the extractor design:
- Roughly a fifth of each chapter's words are not in the markup. The page
emits empty spans and supplies the word from the stylesheet via
":before { content: ... }" rules, so reading DOM text alone drops them
with no error. The extractor resolves those rules and substitutes the
words back; a test asserts they disappear when the rule is removed.
- Chapter URLs cannot be generated. Numbering has gaps and slugs are not
uniform across novels, so chapter links are always parsed from the page.
The chapter list is read from the landing page and cross-checked against
the dropdown embedded in each chapter page, so a truncated list cannot
silently shorten the export.
PDF defaults to a 90x160mm page rather than A4 with large type: viewers
scale a whole page to fit the screen, so a phone-shaped page fills it at
100% zoom where 12pt stays comfortable. A5 and A4 remain available.
Requests are spaced globally, so raising worker count does not raise the
request rate. Raw pages cache to disk so re-exporting at different font
or page settings needs no network.
Port both scripts into one Go program with a command per behavior:
gallery walks the numbered ghibli.jp film galleries, scrape downloads
every image referenced by a page.
Downloads now run through a bounded worker pool, and non-200 responses
are no longer written to disk as image files. Drop the orphaned
package-lock.json, which pinned the deprecated request dependency and
was the source of the repository's Dependabot alerts.
tiennm99