mirror of
https://github.com/tiennm99/MTTools.git
synced 2026-10-14 14:17:46 +00:00
feat: parse novel tags and add a single-request metadata fetch
Novel gains Tags, read from the anchors carrying schema.org itemprop="genre". Matching that microdata rather than the /the-loai/ href shape is what keeps the site-wide genre navigation out: a sampled page links 69 categories against the novel's own 6. NovelInfo fetches just the landing page and parses it, so a caller that only wants metadata pays one request instead of the two Novel needs for its chapter list cross-check.
This commit is contained in:
1 parent
79cf77a471
commit
c1ea1876bb
3 files changed
+103
-3
No files matched your search
@@ -77,6 +77,21 @@ func (c *Crawler) Novel(ctx context.Context, novelURL string) (*Novel, error) {
|
||||
return novel, nil
|
||||
}
|
||||
|
||||
// NovelInfo fetches only the landing page and returns what it carries: title,
|
||||
// slug, tags, and the chapter list as that page shows it.
|
||||
//
|
||||
// Unlike Novel it does not also fetch a chapter page to cross-check the chapter
|
||||
// list, so it costs a single request. Callers that only want metadata should
|
||||
// prefer it; callers about to export every chapter want Novel, whose
|
||||
// reconciliation guards against a silently truncated list.
|
||||
func (c *Crawler) NovelInfo(ctx context.Context, novelURL string) (*Novel, error) {
|
||||
page, err := c.page(ctx, novelURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return ParseNovelPage(page, novelURL)
|
||||
}
|
||||
|
||||
// Chapters fetches every chapter concurrently and returns them in reading
|
||||
// order. Any chapter that cannot be fetched or parsed fails the whole run
|
||||
// rather than yielding a book with a hole in it.
|
||||
|
||||
Reference in new issue
Block a user