mirror of
https://github.com/tiennm99/MTTools.git
synced 2026-10-13 22:16:19 +00:00
tiennm99
ad8d2649c1
The crawler took one hardcoded novel URL, wrote per-chapter text files into ./data, and fetched serially until the site answered 429. Rename the module to hako-crawler and split fetching, page parsing and crawl orchestration into a hako package that takes any novel URL. The chapter body is no longer served as markup: #chapter-content now carries a shuffled, XOR-obfuscated payload that the site's own script expands in the browser, so the previous p[id=<digits>] extractor returned nothing at all. Decode it, keeping the plain-markup path as a fallback. The scheme is read from the page and an unrecognised one is a hard error, because the alternative is exporting a book of mojibake. Space requests globally instead of per-goroutine, so concurrency no longer raises the request rate, and retry only what a retry can fix. Cache raw pages so re-runs cost no requests. BREAKING CHANGE: module path is now github.com/tiennm99/hako-crawler, and the novel URL is a required argument rather than a constant.