mirror of
https://github.com/tiennm99/MTTools.git
synced 2026-10-11 03:13:42 +00:00
tiennm99
ad8d2649c1
The crawler took one hardcoded novel URL, wrote per-chapter text files into ./data, and fetched serially until the site answered 429. Rename the module to hako-crawler and split fetching, page parsing and crawl orchestration into a hako package that takes any novel URL. The chapter body is no longer served as markup: #chapter-content now carries a shuffled, XOR-obfuscated payload that the site's own script expands in the browser, so the previous p[id=<digits>] extractor returned nothing at all. Decode it, keeping the plain-markup path as a fallback. The scheme is read from the page and an unrecognised one is a hard error, because the alternative is exporting a book of mojibake. Space requests globally instead of per-goroutine, so concurrency no longer raises the request rate, and retry only what a retry can fix. Cache raw pages so re-runs cost no requests. BREAKING CHANGE: module path is now github.com/tiennm99/hako-crawler, and the novel URL is a required argument rather than a constant.
MTTools
Personal toolbox — small utilities and scripts, each formerly a standalone repository, merged here with full history.
| Directory | Tool | Language |
|---|---|---|
utils/ |
Utility batch and shell scripts to make life easier (or harder) | Batch/Shell |
go-util/ |
Miscellaneous CLI scripts (Gitea org cleanup, etc.), multi-binary cmd/ layout |
Go |
export-chrome-cookies/ |
Export Google Chrome cookies on Windows | Python |
miti-scraper/ |
Configurable single-site web scraper | Go |
circular-dependency-checker/ |
Detects cyclic dependencies in Godot projects | Python |
image-resizer/ |
Batch image resizer CLI — scales to a percentage, keeps aspect ratio | Go |
webp-converter/ |
HTTP microservice converting WebP images to PNG on-the-fly | Go |
Unmerged Go-rewrite drafts live on feature/export-chrome-cookies-go and
feature/circular-dependency-checker-go.