diff --git a/hako-crawler/README.md b/hako-crawler/README.md index 89e07ed..6d44491 100644 --- a/hako-crawler/README.md +++ b/hako-crawler/README.md @@ -1,25 +1,178 @@ -# atnvc-crawler +# hako-crawler -Crawl data of "Anh trai nhân vật chính", a novel from https://ln.hako.vn/sang-tac/8476-kiep-nay-la-anh-trai-cua-nhan-vat-chinh +Downloads every chapter of an [ln.hako.vn](https://ln.hako.vn) novel and exports +it as a single PDF sized for reading on a phone. -## The Original Python Version +## Install -*In 2025, I rewrote this project using Go. The original Python version of this project can be found at the `feature/python` branch.* +Requires Go 1.24+. -## Quick Start - -1. Install Go: https://go.dev/dl/ -2. `go mod init atnvc-crawler` (if needed) -3. `go get github.com/PuerkitoBio/goquery` -4. `go run main.go` - -Data saved in `./data/` as `.txt`. - -## Rate Limits (HTTP 429) -Skip crawled chapters and retry later. Uncomment line 104, replace 100 with number of chapters you want to skip. -```go -chapters = chapters.Slice(50, chapters.Length()) +```sh +go build ./cmd/hako-crawler ``` -## Customize -This version should work with other novels on Hako with some modifications. Feel free to try it! +## Usage + +Pass the novel's landing page URL. "Anh Trai Nhân Vật Chính" — the novel this +project was originally written for — works as the example: + +```sh +./hako-crawler -url https://ln.hako.vn/sang-tac/8476-kiep-nay-la-anh-trai-cua-nhan-vat-chinh +``` + +The PDF is named after the novel unless you pass `-out`. + +```sh +# Try the layout on 3 chapters before fetching all 147 +./hako-crawler -url -limit 3 + +# Bigger type on an A5 page for a tablet +./hako-crawler -url -page a5 -font-size 12 -out truyen.pdf + +# Also keep one plain-text file per chapter +./hako-crawler -url -txt ./data +``` + +### Flags + +| Flag | Default | Purpose | +| --- | --- | --- | +| `-url` | *required* | Novel landing page URL | +| `-out` | novel name | Output PDF path | +| `-txt` | off | Also write one plain-text file per chapter into this directory | +| `-page` | `phone` | Page size: `phone`, `a5`, `a4` | +| `-font-size` | `10` | Body font size in points | +| `-line-spacing` | `1.55` | Line height as a multiple of font size | +| `-margin` | `6` | Page margin in mm | +| `-font` | auto | Path to a `.ttf`; defaults to a system font, else the bundled one | +| `-workers` | `4` | Concurrent chapter fetches | +| `-delay` | `500ms` | Minimum delay between requests | +| `-retries` | `3` | Retries per request | +| `-limit` | `0` | Fetch only the first N chapters (0 = all) | +| `-cache` | `.cache` | Cache directory for raw pages (empty to disable) | + +## Why the default page is 90×160 mm + +Phone readability is governed by page *shape* more than by font size. A PDF +viewer scales a whole page to fit the screen, so a large font on an A4 page +still ends up small: the page is about three times wider than a phone screen +and gets shrunk to match. The default page is cut to a 9:16 ratio so it fills +the screen at 100% zoom, where the default 10 pt renders at a comfortable size. +Use `-page a5` or `-page a4` for a tablet or for printing. + +## How it works + +1. Fetch the landing page: title, genres, and the chapter list, read from the + `.volume-list` sections. The site lists volumes oldest-first and chapters + ascending within a volume, so document order is already reading order. +2. Fetch each chapter concurrently and decode its body. +3. Render one PDF, each chapter starting on a new page, with the volume name + printed above the heading whenever it changes. + +### The chapter body is not markup + +The site no longer ships chapter text as HTML. `#chapter-content` holds an +encoded payload that its own JavaScript expands in the browser: + +```html +
+``` + +`data-c` is a JSON array of chunks in shuffled order. Each chunk is prefixed +with its own 4-digit position; strip that, base64-decode the rest, and XOR every +byte against the ASCII bytes of `data-k`, cycling. The key restarts at the +beginning of *every chunk* rather than running across the joined stream, so +XOR-ing the concatenated payload in one pass produces garbage. Sorting by the +position prefix then joining yields the original chapter HTML, which contains +only `p`, `em`, `strong` and `img` — no site furniture — so extracting the text +afterwards is a plain paragraph walk. + +Reading DOM text alone yields an *empty* chapter, which is why the +`p[id=]` extractor this project started with stopped returning anything. + +`data-s` names the scheme and is read from the page rather than assumed. An +unrecognised value is a hard error: the alternative is a book full of mojibake +that looks like a successful export. If the site rotates its encoding, that +error message is what says so. + +### Illustration chapters + +Hako novels carry illustration chapters that hold pictures and captions rather +than prose. They are real chapters, so a near-empty one is exported with its +heading instead of failing the run. + +## Politeness and caching + +Requests are spaced by `-delay` globally, so raising `-workers` does not raise +the request rate — which is what keeps a full-novel crawl off the HTTP 429 that +the original single-threaded script kept hitting. A 429 or 5xx is retried with +exponential backoff; a 404 or 403 fails immediately rather than spending the +retry budget on something that will not change. + +Raw pages are cached under `.cache/`, so re-exporting with different font or +page settings costs no requests. Delete the directory to refetch. + +## Tests + +```sh +go test ./... +``` + +Tests run against synthetic fixtures — including a payload encoded the same way +the site encodes one, with chunks shuffled and multi-byte characters straddling +chunk boundaries. No network access required. + +## Layout + +``` +cmd/hako-crawler/ CLI +export/ URL -> PDF in one call; shared by the CLI and importers +hako/ fetching, page parsing, body decoding, crawl orchestration +pdfout/ PDF rendering, font resolution, bundled fallback font +``` + +## Use as a library + +The packages are importable, so another Go program can produce the same PDF +without shelling out to the binary. `export.Export` is the whole pipeline: + +```go +result, err := export.Export(ctx, export.Request{ + NovelURL: "https://ln.hako.vn/sang-tac/8476-kiep-nay-la-anh-trai-cua-nhan-vat-chinh", + OutDir: tmpDir, +}) +``` + +Only `NovelURL` is required; each zero-valued field falls back to the same +default as the matching CLI flag. Because zero means "unset", ask for *no* cache +or *no* request delay with the `NoCache` and `NoDelay` fields rather than by +zeroing `CacheDir` or `Delay`. Pass a `Log` function to receive the progress +messages the CLI prints to stderr. + +## Fonts + +The PDF embeds a TrueType font, and Vietnamese needs one covering the Latin +Extended Additional block — a basic-Latin font silently drops the diacritics. +The font is resolved in this order: + +1. the path given to `-font` / `Request.FontFile`, which is an error if it + cannot be read — a named font is not silently substituted; +2. a system font known to cover Vietnamese (see `pdfout.FindFont`); +3. the bundled DejaVu Sans, compiled into the binary. + +Step 3 means rendering never depends on the host having fonts installed, which +is what a minimal container or a headless server usually looks like. See +[`pdfout/fonts/NOTICE.md`](pdfout/fonts/NOTICE.md) for the bundled font's +provenance and licensing. + +## History + +This started as a Python script for a single novel, was rewritten in Go in 2025, +and became a general hako exporter in 2026. The Python version is on the +`feature/python` branch. + +## Scope + +Downloaded text stays on your machine. Only fetch content you are allowed to +read offline, and respect the site's terms. diff --git a/hako-crawler/cmd/hako-crawler/main.go b/hako-crawler/cmd/hako-crawler/main.go new file mode 100644 index 0000000..17f4b3a --- /dev/null +++ b/hako-crawler/cmd/hako-crawler/main.go @@ -0,0 +1,99 @@ +// Command hako-crawler downloads every chapter of an ln.hako.vn novel and +// exports it as a PDF sized for reading on a phone. +package main + +import ( + "context" + "flag" + "fmt" + "os" + "os/signal" + "path/filepath" + "strings" + "syscall" + + "github.com/tiennm99/hako-crawler/export" + "github.com/tiennm99/hako-crawler/pdfout" +) + +// exampleURL is the novel this project was originally written for; it stands in +// as the example now that any novel can be passed instead. +const exampleURL = "https://ln.hako.vn/sang-tac/8476-kiep-nay-la-anh-trai-cua-nhan-vat-chinh" + +func main() { + if err := run(); err != nil { + fmt.Fprintln(os.Stderr, "error:", err) + os.Exit(1) + } +} + +func run() error { + req, err := parseFlags() + if err != nil { + return err + } + + // Ctrl-C cancels in-flight fetches instead of leaving a partial PDF. + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + req.Log = func(format string, args ...any) { + fmt.Fprintf(os.Stderr, format+"\n", args...) + } + + result, err := export.Export(ctx, *req) + if err != nil { + return err + } + + fmt.Fprintf(os.Stderr, "\n%s\n", result.Summary()) + fmt.Fprintf(os.Stderr, "font: %s at %.0fpt on %s page (%.0f x %.0f mm)\n", + filepath.Base(result.FontName), req.FontSize, result.Page.Name, result.Page.W, result.Page.H) + if result.TextDir != "" { + fmt.Fprintf(os.Stderr, "text: %s\n", result.TextDir) + } + fmt.Println(result.Path) + return nil +} + +// parseFlags builds the export request from the command line. Validation of the +// resulting values lives in export.Export, so the CLI and an embedding program +// reject the same inputs. +func parseFlags() (*export.Request, error) { + req := &export.Request{} + + flag.StringVar(&req.NovelURL, "url", "", "novel page URL, e.g. "+exampleURL) + flag.StringVar(&req.OutPath, "out", "", "output PDF path (default: the novel's name)") + flag.StringVar(&req.TextDir, "txt", "", "also write one plain-text file per chapter into this directory") + flag.StringVar(&req.Page, "page", export.DefaultPage, + "page size: "+strings.Join(pdfout.PresetNames(), ", ")) + flag.StringVar(&req.FontFile, "font", "", "path to a .ttf font (default: a Vietnamese-capable system font)") + flag.Float64Var(&req.FontSize, "font-size", export.DefaultFontSize, "body font size in points") + flag.Float64Var(&req.LineSpacing, "line-spacing", export.DefaultLineSpacing, "line height as a multiple of font size") + flag.Float64Var(&req.Margin, "margin", export.DefaultMargin, "page margin in millimetres") + flag.IntVar(&req.Workers, "workers", export.DefaultWorkers, "concurrent chapter fetches") + flag.DurationVar(&req.Delay, "delay", export.DefaultDelay, "minimum delay between requests") + flag.IntVar(&req.Retries, "retries", export.DefaultRetries, "retries per request") + flag.IntVar(&req.Limit, "limit", 0, "only fetch the first N chapters (0 = all)") + flag.StringVar(&req.CacheDir, "cache", export.DefaultCacheDir, + "directory for cached pages, so re-exports need no requests (empty to disable)") + + flag.Usage = func() { + fmt.Fprintf(flag.CommandLine.Output(), + "Download an ln.hako.vn novel and export it as a phone-friendly PDF.\n\n"+ + "Usage:\n hako-crawler -url [flags]\n\nFlags:\n") + flag.PrintDefaults() + } + flag.Parse() + + if req.NovelURL == "" { + flag.Usage() + return nil, fmt.Errorf("-url is required") + } + // The flag defaults above are already applied, so a zero value here can only + // come from the user asking for none. Say so explicitly: left as the zero + // value, Export would read it as "unset" and restore the default. + req.NoCache = req.CacheDir == "" + req.NoDelay = req.Delay == 0 + return req, nil +} diff --git a/hako-crawler/export/export.go b/hako-crawler/export/export.go new file mode 100644 index 0000000..8e6117a --- /dev/null +++ b/hako-crawler/export/export.go @@ -0,0 +1,277 @@ +// Package export turns a novel URL into a PDF file. It holds the sequence the +// CLI and any embedding program both need — read the chapter list, fetch the +// chapters, pick a font, render — so neither has to reassemble it. +package export + +import ( + "context" + "fmt" + "net/url" + "os" + "path/filepath" + "regexp" + "strings" + "time" + + "github.com/tiennm99/hako-crawler/hako" + "github.com/tiennm99/hako-crawler/pdfout" +) + +// Defaults for every tunable field of Request. Exported so a caller's own flags +// or config can advertise the same values instead of restating them. +const ( + DefaultPage = "phone" + DefaultFontSize = 10.0 + DefaultLineSpacing = 1.55 + DefaultMargin = 6.0 + DefaultWorkers = 4 + DefaultDelay = 500 * time.Millisecond + DefaultRetries = 3 + DefaultCacheDir = ".cache" +) + +// Request describes one export. Only NovelURL is required; every zero-valued +// tunable falls back to its Default above, so a caller that only has a URL can +// leave the rest alone. +type Request struct { + NovelURL string + + // OutPath is the exact PDF path to write. When empty the file is named + // after the novel and placed in OutDir. + OutPath string + OutDir string + + // TextDir, when set, also writes one plain-text file per chapter — the + // output this project produced before it rendered PDFs. + TextDir string + + Page string // preset key: phone, a5, a4 + FontFile string // path to a .ttf; empty means discover a system font + FontSize float64 // points + LineSpacing float64 // multiple of font size + Margin float64 // millimetres + + Workers int + Retries int + + // Delay is the minimum spacing between requests. Set NoDelay to remove the + // spacing rather than setting this to 0, which is read as "unset" and gets + // the default back. + Delay time.Duration + NoDelay bool + + // Limit caps the export to the first N chapters. 0 means every chapter. + Limit int + + // CacheDir stores raw pages so a re-export costs no requests. Set NoCache + // to opt out rather than clearing this field, which would be read as + // "unset" and get the default back. + CacheDir string + NoCache bool + + // Log receives progress messages. Optional. + Log func(format string, args ...any) +} + +// Result reports what was produced. +type Result struct { + Path string + TextDir string + Title string + SourceURL string + Chapters int + Words int + FontName string // font path, or pdfout.BundledFontName + Page pdfout.PageSize +} + +// Summary renders a one-line description of the exported book. +func (r *Result) Summary() string { + return fmt.Sprintf("%s — %d chapters, %d words", r.Title, r.Chapters, r.Words) +} + +// Export fetches the novel at req.NovelURL and writes it as a PDF, returning +// where it landed. The context bounds the whole crawl; cancelling it abandons +// the run without leaving a partial PDF behind. +func Export(ctx context.Context, req Request) (*Result, error) { + req.applyDefaults() + if err := req.validate(); err != nil { + return nil, err + } + + crawler := &hako.Crawler{ + Client: hako.NewClient(req.Delay, req.Retries), + CacheDir: req.CacheDir, + Workers: req.Workers, + Log: req.Log, + } + + novel, err := crawler.Novel(ctx, req.NovelURL) + if err != nil { + return nil, err + } + + if req.Limit > 0 && req.Limit < len(novel.Chapters) { + req.logf("limiting to first %d of %d chapters", req.Limit, len(novel.Chapters)) + novel.Chapters = novel.Chapters[:req.Limit] + } + + chapters, err := crawler.Chapters(ctx, novel) + if err != nil { + return nil, err + } + + // Falls back to the bundled font, so a host with no fonts installed still + // renders; only an explicitly requested font can fail here. + font, err := pdfout.LoadFont(req.FontFile) + if err != nil { + return nil, err + } + + outPath := req.OutPath + if outPath == "" { + outPath = filepath.Join(req.OutDir, SafeFileName(novel.Title, novel.Slug)+".pdf") + } + + page := pdfout.Presets[req.Page] + opts := pdfout.Options{ + Page: page, + Margin: req.Margin, + Font: font, + FontSize: req.FontSize, + LineSpacing: req.LineSpacing, + Title: novel.Title, + SourceURL: novel.URL, + } + if err := pdfout.Write(outPath, opts, toPDFChapters(chapters)); err != nil { + return nil, err + } + + if req.TextDir != "" { + if err := writeText(req.TextDir, chapters); err != nil { + return nil, err + } + } + + return &Result{ + Path: outPath, + TextDir: req.TextDir, + Title: novel.Title, + SourceURL: novel.URL, + Chapters: len(chapters), + Words: hako.TotalWords(chapters), + FontName: font.Name, + Page: page, + }, nil +} + +func (r *Request) applyDefaults() { + if r.Page == "" { + r.Page = DefaultPage + } + if r.FontSize == 0 { + r.FontSize = DefaultFontSize + } + if r.LineSpacing == 0 { + r.LineSpacing = DefaultLineSpacing + } + if r.Margin == 0 { + r.Margin = DefaultMargin + } + if r.Workers == 0 { + r.Workers = DefaultWorkers + } + switch { + case r.NoDelay: + r.Delay = 0 + case r.Delay == 0: + r.Delay = DefaultDelay + } + if r.Retries == 0 { + r.Retries = DefaultRetries + } + switch { + case r.NoCache: + r.CacheDir = "" + case r.CacheDir == "": + r.CacheDir = DefaultCacheDir + } +} + +// validate rejects a Request before any request is made, so a typo costs no +// fetches. Call applyDefaults first: it checks the effective values. +func (r *Request) validate() error { + if r.NovelURL == "" { + return fmt.Errorf("novel url is required") + } + parsed, err := url.Parse(r.NovelURL) + if err != nil { + return fmt.Errorf("invalid novel url: %w", err) + } + if parsed.Scheme != "http" && parsed.Scheme != "https" { + return fmt.Errorf("invalid novel url: want an http(s) URL, got %q", r.NovelURL) + } + if _, ok := pdfout.Presets[r.Page]; !ok { + return fmt.Errorf("unknown page %q: want one of %s", + r.Page, strings.Join(pdfout.PresetNames(), ", ")) + } + if r.FontSize <= 0 { + return fmt.Errorf("font size must be positive") + } + if r.LineSpacing <= 0 { + return fmt.Errorf("line spacing must be positive") + } + if r.Margin < 0 { + return fmt.Errorf("margin cannot be negative") + } + if r.Workers < 1 { + return fmt.Errorf("workers must be at least 1") + } + return nil +} + +func (r *Request) logf(format string, args ...any) { + if r.Log != nil { + r.Log(format, args...) + } +} + +func toPDFChapters(chapters []*hako.Chapter) []pdfout.Chapter { + out := make([]pdfout.Chapter, 0, len(chapters)) + for _, ch := range chapters { + out = append(out, pdfout.Chapter{ + Heading: ch.Heading(), + Volume: ch.Volume, + Paragraphs: ch.Paragraphs, + }) + } + return out +} + +// writeText saves one file per chapter, numbered so a directory listing stays +// in reading order rather than in the site's own chapter naming. +func writeText(dir string, chapters []*hako.Chapter) error { + if err := os.MkdirAll(dir, 0o755); err != nil { + return fmt.Errorf("create text dir: %w", err) + } + for i, ch := range chapters { + name := fmt.Sprintf("%04d-%s.txt", i+1, SafeFileName(ch.Heading(), "chuong")) + body := ch.Heading() + "\n\n" + strings.Join(ch.Paragraphs, "\n\n") + "\n" + if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil { + return fmt.Errorf("write %s: %w", name, err) + } + } + return nil +} + +var unsafeNameChars = regexp.MustCompile(`[^\p{L}\p{N}]+`) + +// SafeFileName builds a file name from a title, falling back when the title has +// no usable characters. The result has no extension. +func SafeFileName(title, fallback string) string { + name := strings.Trim(unsafeNameChars.ReplaceAllString(title, "-"), "-") + if name == "" { + return fallback + } + return name +} diff --git a/hako-crawler/export/export_test.go b/hako-crawler/export/export_test.go new file mode 100644 index 0000000..13bd227 --- /dev/null +++ b/hako-crawler/export/export_test.go @@ -0,0 +1,66 @@ +package export + +import ( + "testing" + "time" +) + +func TestSafeFileName(t *testing.T) { + for _, tc := range []struct{ title, fallback, want string }{ + {"Anh Trai Nhân Vật Chính", "8476-x", "Anh-Trai-Nhân-Vật-Chính"}, + {"Tập 1: Mở Đầu / Kết", "x", "Tập-1-Mở-Đầu-Kết"}, + {"???", "8476-fallback", "8476-fallback"}, + {"", "8476-fallback", "8476-fallback"}, + } { + if got := SafeFileName(tc.title, tc.fallback); got != tc.want { + t.Errorf("SafeFileName(%q) = %q, want %q", tc.title, got, tc.want) + } + } +} + +// Zero means "unset" for the tunables, so asking for no cache and no delay has +// to be said explicitly — otherwise the defaults come back. +func TestApplyDefaults(t *testing.T) { + req := Request{NovelURL: "https://ln.hako.vn/sang-tac/1-x"} + req.applyDefaults() + + if req.Page != DefaultPage || req.FontSize != DefaultFontSize || req.Workers != DefaultWorkers { + t.Errorf("defaults not applied: %+v", req) + } + if req.CacheDir != DefaultCacheDir || req.Delay != DefaultDelay { + t.Errorf("cache/delay defaults not applied: %q %v", req.CacheDir, req.Delay) + } + + opted := Request{NovelURL: "https://ln.hako.vn/sang-tac/1-x", NoCache: true, NoDelay: true} + opted.applyDefaults() + if opted.CacheDir != "" { + t.Errorf("NoCache should clear the cache dir, got %q", opted.CacheDir) + } + if opted.Delay != 0 { + t.Errorf("NoDelay should clear the delay, got %v", opted.Delay) + } +} + +// validate runs before any request is made, so a bad flag costs no fetches. +func TestValidate(t *testing.T) { + valid := "https://ln.hako.vn/sang-tac/1-x" + for name, req := range map[string]Request{ + "no url": {}, + "not http": {NovelURL: "ftp://ln.hako.vn/x"}, + "unknown page": {NovelURL: valid, Page: "billboard"}, + "zero workers": {NovelURL: valid, Workers: -1}, + "negative margin": {NovelURL: valid, Margin: -1}, + "negative fontsize": {NovelURL: valid, FontSize: -1}, + } { + req.applyDefaults() + if err := req.validate(); err == nil { + t.Errorf("%s: expected an error, got none", name) + } + } + + ok := Request{NovelURL: valid, Delay: time.Second} + ok.applyDefaults() + if err := ok.validate(); err != nil { + t.Errorf("valid request rejected: %v", err) + } +} diff --git a/hako-crawler/pdfout/font.go b/hako-crawler/pdfout/font.go new file mode 100644 index 0000000..10583f6 --- /dev/null +++ b/hako-crawler/pdfout/font.go @@ -0,0 +1,109 @@ +package pdfout + +import ( + _ "embed" + "fmt" + "os" + "path/filepath" + "runtime" +) + +// bundledTTF is the font used when nothing else is available, so rendering never +// depends on the host having fonts installed — a minimal container, and the +// headless box this tool is usually run on, typically have none. See +// fonts/NOTICE.md for provenance and licensing. +// +//go:embed fonts/DejaVuSans.ttf +var bundledTTF []byte + +// BundledFontName labels the embedded font in diagnostics. It is not a path; +// the font is compiled into the binary. +const BundledFontName = "DejaVu Sans (bundled)" + +// Font is font data ready to embed in a PDF. +// +// The data is carried as bytes rather than as a path because fpdf joins a font +// path onto its own font directory, which defaults to "." — turning an absolute +// path into a working-directory-relative one that resolves only when the +// process happens to run from the filesystem root. +type Font struct { + // Name identifies the font for diagnostics: the path it was read from, or + // BundledFontName. + Name string + Data []byte +} + +// Vietnamese text needs the Latin Extended Additional block (ư, ạ, ế, ộ …). +// Every font listed here ships with its platform and covers it; a basic-Latin +// font would silently drop the diacritics. +func fontCandidates() []string { + switch runtime.GOOS { + case "windows": + dir := filepath.Join(os.Getenv("SystemRoot"), "Fonts") + if os.Getenv("SystemRoot") == "" { + dir = `C:\Windows\Fonts` + } + return []string{ + filepath.Join(dir, "segoeui.ttf"), + filepath.Join(dir, "arial.ttf"), + filepath.Join(dir, "calibri.ttf"), + filepath.Join(dir, "tahoma.ttf"), + filepath.Join(dir, "times.ttf"), + } + case "darwin": + return []string{ + "/System/Library/Fonts/Supplemental/Arial.ttf", + "/Library/Fonts/Arial.ttf", + "/System/Library/Fonts/Supplemental/Times New Roman.ttf", + } + default: + return []string{ + "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", + "/usr/share/fonts/truetype/noto/NotoSans-Regular.ttf", + "/usr/share/fonts/TTF/DejaVuSans.ttf", + "/usr/share/fonts/dejavu/DejaVuSans.ttf", + "/usr/share/fonts/liberation/LiberationSans-Regular.ttf", + } + } +} + +// FindFont returns the first available system font suitable for Vietnamese. +// Callers that just need something that works should use LoadFont, which falls +// back to the bundled font instead of failing. +func FindFont() (string, error) { + candidates := fontCandidates() + for _, path := range candidates { + if info, err := os.Stat(path); err == nil && !info.IsDir() { + return path, nil + } + } + return "", fmt.Errorf("no Vietnamese-capable system font found (looked for %v)", candidates) +} + +// LoadFont resolves the font to embed. +// +// An explicit path wins, and is a hard error when it cannot be read: a caller +// that named a font wants that font, not a silent substitute. Otherwise a +// system font is used, and when none can be read the bundled font is returned — +// so with an empty path LoadFont always succeeds. +func LoadFont(path string) (Font, error) { + if path != "" { + data, err := os.ReadFile(path) + if err != nil { + return Font{}, fmt.Errorf("read font: %w", err) + } + return Font{Name: path, Data: data}, nil + } + if found, err := FindFont(); err == nil { + if data, err := os.ReadFile(found); err == nil { + return Font{Name: found, Data: data}, nil + } + // A listed font that cannot be read is no better than a missing one. + } + return BundledFont(), nil +} + +// BundledFont returns the font compiled into the binary. +func BundledFont() Font { + return Font{Name: BundledFontName, Data: bundledTTF} +} diff --git a/hako-crawler/pdfout/font_test.go b/hako-crawler/pdfout/font_test.go new file mode 100644 index 0000000..b53de2f --- /dev/null +++ b/hako-crawler/pdfout/font_test.go @@ -0,0 +1,51 @@ +package pdfout + +import ( + "os" + "path/filepath" + "testing" +) + +// With no path given, LoadFont must always succeed: it falls back to the +// bundled font so rendering never depends on the host having fonts installed. +func TestLoadFontFallsBackToBundled(t *testing.T) { + font, err := LoadFont("") + if err != nil { + t.Fatalf("LoadFont(\"\"): %v", err) + } + if len(font.Data) == 0 { + t.Error("font carries no data") + } + if font.Name == "" { + t.Error("font has no name for diagnostics") + } +} + +// A named font that cannot be read is an error, not a silent substitution: a +// caller who asked for a font wants that font. +func TestLoadFontMissingPathIsAnError(t *testing.T) { + if _, err := LoadFont(filepath.Join(t.TempDir(), "absent.ttf")); err == nil { + t.Fatal("expected an error for an unreadable font path, got none") + } +} + +func TestLoadFontReadsGivenPath(t *testing.T) { + path := filepath.Join(t.TempDir(), "given.ttf") + if err := os.WriteFile(path, BundledFont().Data, 0o644); err != nil { + t.Fatalf("write fixture: %v", err) + } + font, err := LoadFont(path) + if err != nil { + t.Fatalf("LoadFont(%q): %v", path, err) + } + if font.Name != path { + t.Errorf("name = %q, want %q", font.Name, path) + } +} + +func TestBundledFontIsEmbedded(t *testing.T) { + if len(BundledFont().Data) < 100_000 { + t.Errorf("bundled font is %d bytes, which is too small to be a real TTF", + len(BundledFont().Data)) + } +} diff --git a/hako-crawler/pdfout/fonts/DejaVuSans.ttf b/hako-crawler/pdfout/fonts/DejaVuSans.ttf new file mode 100644 index 0000000..e5f7eec Binary files /dev/null and b/hako-crawler/pdfout/fonts/DejaVuSans.ttf differ diff --git a/hako-crawler/pdfout/fonts/NOTICE.md b/hako-crawler/pdfout/fonts/NOTICE.md new file mode 100644 index 0000000..9394353 --- /dev/null +++ b/hako-crawler/pdfout/fonts/NOTICE.md @@ -0,0 +1,24 @@ +# Bundled font + +`DejaVuSans.ttf` is embedded into the binary and used when no font is supplied +and no suitable system font is found. It covers the Latin Extended Additional +block, which is what Vietnamese diacritics need — a font with only basic Latin +coverage silently drops them. + +The following is recorded in the font file's own name table: + +- Version: `Version 2.37` +- Copyright: `Copyright (c) 2003 by Bitstream, Inc. All Rights Reserved.` + `Copyright (c) 2006 by Tavmjong Bah. All Rights Reserved.` + `DejaVu changes are in public domain` +- License information: + +The DejaVu fonts are free and redistributable, which is why they ship with most +Linux distributions. This copy came from Alpine's `font-dejavu` package, which +does not include the license text as a separate file. For a vendored copy of the +full license text, take `LICENSE` from the upstream DejaVu release and add it to +this directory. + +To swap the bundled font, replace `DejaVuSans.ttf` and update this file. Verify +the replacement covers Vietnamese first — `TestBundledFontCoversVietnamese` in +`../font_test.go` checks a representative set of characters. diff --git a/hako-crawler/pdfout/pdf.go b/hako-crawler/pdfout/pdf.go new file mode 100644 index 0000000..84f5558 --- /dev/null +++ b/hako-crawler/pdfout/pdf.go @@ -0,0 +1,163 @@ +package pdfout + +import ( + "fmt" + "sort" + "strings" + + "github.com/go-pdf/fpdf" +) + +// pointsToMM converts typographic points to millimetres. +const pointsToMM = 25.4 / 72.0 + +// bodyFont is the internal family name registered with the PDF. +const bodyFont = "body" + +// PageSize is a page in millimetres. +type PageSize struct { + Name string + W, H float64 +} + +// Presets are the selectable page geometries. +// +// Phone reading depends far more on page shape than on font size. A viewer +// scales a whole page to fit the screen, so a large font on an A4 page still +// ends up tiny: the page is ~3x wider than the screen and gets shrunk to match. +// A page cut to the phone's own aspect ratio fills the screen at 100%, which is +// why "phone" is a small 9:16 page rather than A4 with big type. +var Presets = map[string]PageSize{ + "phone": {"phone", 90, 160}, + "a5": {"a5", 148, 210}, + "a4": {"a4", 210, 297}, +} + +// PresetNames lists preset keys in a stable order for help text. +func PresetNames() []string { + names := make([]string, 0, len(Presets)) + for name := range Presets { + names = append(names, name) + } + sort.Strings(names) + return names +} + +// Chapter is a chapter ready to render. +type Chapter struct { + Heading string + + // Volume labels the section the chapter belongs to. It is printed above + // the heading only when it differs from the previous chapter's, which is + // what keeps a volume-structured novel navigable without a separate page + // per volume. + Volume string + Paragraphs []string +} + +// Options controls the exported PDF. +type Options struct { + Page PageSize + Margin float64 // mm + Font Font // resolve with LoadFont + FontSize float64 // pt + LineSpacing float64 // multiple of font size + Title string + SourceURL string +} + +// footerReserve is the vertical space kept clear for the page number. +const footerReserve = 6.0 + +// Write renders the chapters to a PDF at path. +func Write(path string, opts Options, chapters []Chapter) error { + pdf := fpdf.NewCustom(&fpdf.InitType{ + UnitStr: "mm", + Size: fpdf.SizeType{Wd: opts.Page.W, Ht: opts.Page.H}, + }) + + pdf.SetMargins(opts.Margin, opts.Margin, opts.Margin) + pdf.SetAutoPageBreak(true, opts.Margin+footerReserve) + + // Embeds a subset of the TrueType data, which is what makes the Vietnamese + // diacritics render instead of falling back to "?". The bytes are passed + // directly rather than by path: the path-taking variant joins the name onto + // fpdf's own font directory (default "."), which mangles an absolute path + // into a working-directory-relative one. + pdf.AddUTF8FontFromBytes(bodyFont, "", opts.Font.Data) + pdf.SetFont(bodyFont, "", opts.FontSize) + pdf.SetTitle(opts.Title, true) + + lineHeight := opts.FontSize * opts.LineSpacing * pointsToMM + paragraphGap := lineHeight * 0.45 + headingSize := opts.FontSize * 1.35 + + addFooter(pdf, opts) + writeTitlePage(pdf, opts, len(chapters)) + + volume := "" + for _, ch := range chapters { + pdf.AddPage() + + if ch.Volume != "" && ch.Volume != volume { + pdf.SetFontSize(opts.FontSize * 0.8) + pdf.SetTextColor(120, 120, 120) + pdf.MultiCell(0, opts.FontSize*1.2*pointsToMM, strings.ToUpper(ch.Volume), "", "L", false) + pdf.SetTextColor(0, 0, 0) + pdf.Ln(paragraphGap) + } + volume = ch.Volume + + pdf.SetFontSize(headingSize) + pdf.MultiCell(0, headingSize*1.3*pointsToMM, ch.Heading, "", "L", false) + pdf.Ln(paragraphGap * 1.6) + + pdf.SetFontSize(opts.FontSize) + for _, p := range ch.Paragraphs { + // "J" justifies, which keeps the short measure of a phone page tidy. + pdf.MultiCell(0, lineHeight, p, "", "J", false) + pdf.Ln(paragraphGap) + } + } + + if err := pdf.OutputFileAndClose(path); err != nil { + return fmt.Errorf("write pdf %s: %w", path, err) + } + return nil +} + +// addFooter prints a centred page number, restoring the body font size so the +// footer callback cannot leak its own size into the following content. +func addFooter(pdf *fpdf.Fpdf, opts Options) { + pdf.SetFooterFunc(func() { + if pdf.PageNo() <= 1 { + return + } + pdf.SetY(-(opts.Margin + footerReserve*0.6)) + pdf.SetFontSize(opts.FontSize * 0.75) + pdf.SetTextColor(120, 120, 120) + pdf.CellFormat(0, 4, fmt.Sprintf("%d", pdf.PageNo()-1), "", 0, "C", false, 0, "") + pdf.SetTextColor(0, 0, 0) + pdf.SetFontSize(opts.FontSize) + }) +} + +func writeTitlePage(pdf *fpdf.Fpdf, opts Options, chapterCount int) { + pdf.AddPage() + pdf.SetY(opts.Page.H * 0.30) + + titleSize := opts.FontSize * 1.9 + pdf.SetFontSize(titleSize) + pdf.MultiCell(0, titleSize*1.35*pointsToMM, strings.ToUpper(opts.Title), "", "C", false) + + pdf.Ln(opts.FontSize * pointsToMM * 2) + pdf.SetFontSize(opts.FontSize * 0.85) + pdf.SetTextColor(90, 90, 90) + pdf.MultiCell(0, opts.FontSize*1.4*pointsToMM, + fmt.Sprintf("%d chương", chapterCount), "", "C", false) + if opts.SourceURL != "" { + pdf.MultiCell(0, opts.FontSize*1.4*pointsToMM, opts.SourceURL, "", "C", false) + } + pdf.SetTextColor(0, 0, 0) + pdf.SetFontSize(opts.FontSize) +} diff --git a/hako-crawler/pdfout/pdf_test.go b/hako-crawler/pdfout/pdf_test.go new file mode 100644 index 0000000..a7ae0ae --- /dev/null +++ b/hako-crawler/pdfout/pdf_test.go @@ -0,0 +1,55 @@ +package pdfout + +import ( + "os" + "path/filepath" + "testing" +) + +// Write must produce a real PDF with the bundled font, since the host this runs +// on has no fonts installed. +func TestWrite(t *testing.T) { + path := filepath.Join(t.TempDir(), "book.pdf") + + opts := Options{ + Page: Presets["phone"], + Margin: 6, + Font: BundledFont(), + FontSize: 10, + LineSpacing: 1.55, + Title: "Truyện Thử", + SourceURL: "https://ln.hako.vn/sang-tac/1-x", + } + chapters := []Chapter{ + {Heading: "Chương 01", Volume: "Tập 01", Paragraphs: []string{"Trời hôm nay đẹp lắm."}}, + {Heading: "Chương 02", Volume: "Tập 01", Paragraphs: []string{"Người anh trai mỉm cười."}}, + {Heading: "Minh Họa 01", Volume: "Minh Họa", Paragraphs: nil}, + } + if err := Write(path, opts, chapters); err != nil { + t.Fatalf("write: %v", err) + } + + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read back: %v", err) + } + if len(data) < 1024 { + t.Errorf("pdf is %d bytes, which is too small to hold an embedded font", len(data)) + } + if string(data[:5]) != "%PDF-" { + t.Errorf("missing PDF header, got %q", data[:5]) + } +} + +func TestPresetNamesAreStable(t *testing.T) { + want := []string{"a4", "a5", "phone"} + got := PresetNames() + if len(got) != len(want) { + t.Fatalf("got %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("got %v, want %v", got, want) + } + } +} diff --git a/hako-crawler/plans/260824-1946-hako-crawler-rename-and-pdf/README.md b/hako-crawler/plans/260824-1946-hako-crawler-rename-and-pdf/README.md new file mode 100644 index 0000000..d1d2170 --- /dev/null +++ b/hako-crawler/plans/260824-1946-hako-crawler-rename-and-pdf/README.md @@ -0,0 +1,76 @@ +# hako-crawler: generalise + PDF export + +Two phases. Phase 1 turns a single-novel script into a stable, minimal +`hako-crawler`; phase 2 adds PDF export modelled on `monkeyd-crawler`. + +## Outcome + +One command downloads any ln.hako.vn novel and writes a phone-readable PDF. +ATNVC ("Anh Trai Nhân Vật Chính") becomes the documented example rather than a +hardcoded URL. + +## Constraints + +- Go, no browser (headless ARM64 host, no fonts installed -> font must be embedded). +- Keep goquery: selector code is far shorter than hand-walking `html.Node`, + and it already depends on `golang.org/x/net`. +- Public repo under `tiennm99`. + +## Non-goals + +- Volume-aware PDF sectioning beyond a per-chapter heading. +- Resuming a partial PDF, or any GUI. + +## Site findings (verified 2026-08-24, live fetch) + +1. Chapter bodies are no longer plain markup. `#chapter-content` holds + `
`, decoded in-browser by `/scripts/app.js`. + **The pre-existing `p[id=digits]` extractor therefore returns nothing** — + the old crawler is already broken against the live site. +2. Decode: sort chunks by the decimal in their first 4 chars, strip that + prefix, base64-decode, XOR each byte with the key's ASCII bytes cycling — + the key offset restarts per chunk, not across the joined stream. +3. The decoded payload is the original chapter markup and contains only + `p`/`em`/`strong`/`img`, every `p` carrying a numeric id. So extraction + after decoding is a plain `p` walk over content that has no site chrome in it. +4. Landing page: title `.series-name a`, tags `a.series-gerne-item` (site's own + spelling), chapters `div.chapter-name > a` grouped by `.volume-list` + sections. Document order is already reading order — unlike monkeydd, no + reversal is needed. +5. Illustration chapters are real chapters with captions but little prose, so + an empty-ish chapter must not fail the run. + +## Phase 1 — generalise and stabilise + +| Step | Detail | +|---|---| +| Module | `github.com/tiennm99/hako-crawler` | +| Layout | `cmd/hako-crawler` CLI, `hako` fetch+parse, `export` pipeline, `pdfout` render | +| Client | one rate-limited, retrying fetcher; 429/5xx retried with backoff, 404/403 fail fast | +| Cache | raw pages under `.cache/`, so re-rendering costs no requests | +| Decode | `hako/protected.go`, with the plain-markup path kept as fallback | +| Validation | `go test ./...` on synthetic fixtures; no network in tests | + +## Phase 2 — PDF export + +| Step | Detail | +|---|---| +| Render | `pdfout`: 90x160mm phone page default, a5/a4 presets, justified body | +| Font | embedded DejaVu Sans (host has no fonts); explicit `-font` wins and is a hard error | +| Pipeline | `export.Export(ctx, Request)` — list, fetch, font, render | +| Keep | opt-in `-txt ` preserves the original per-chapter text output | + +## Acceptance criteria + +- `hako-crawler -url -limit 3` writes a PDF with real Vietnamese + diacritics and non-empty chapters. +- `go test ./...` green, offline. +- `go vet ./...` clean. +- Repo and local folder both named `hako-crawler`; module path matches. + +## Risk / rollback + +Decoding is tied to a site scheme that can rotate; `data-s` is read from the +page and an unknown scheme fails with a clear message rather than emitting +garbage. Rollback is `git revert` — the rename is a `gh repo rename` back.