From b04f9844f9c038bed6e082e8d1fa644f2d303c17 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Thu, 13 Aug 2026 22:05:07 +0700 Subject: [PATCH] refactor(crawler): read the file lists from the source articles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both sources carried their download links as a hardcoded array, which is not a crawl: the lists could drift from what the articles actually published, and nothing would say so. A source now names the article and how to name what it finds there, and internal/article reads the links out of that page at run time. 2016 takes its filenames straight from the URL. 2017 cannot — the CDN names are inconsistent (Angiang.xls, 1BaRiaVungTau.xls, 23HaiPhong.xls) — so it derives them from the province in the link text, transliterated to ASCII the same way go-parser builds ho_ten_ascii. Filenames stay load-bearing: go-parser sorts inputs bytewise and inserts last-wins, so they decide which row survives a duplicate exam number. Saved copies of both articles are committed as fixtures, and a test asserts that reading them and applying each naming rule reproduces data/ exactly, in both directions. Resolve also rejects a page that yields the wrong number of links or two links that would write the same file, since either silently costs the dataset files that only the row-count guard would notice afterwards. Verified against the live 2017 article: a from-scratch crawl of all 63 files leaves the committed data unchanged. --- .github/workflows/deploy-pages.yml | 4 +- README.md | 10 +- crawler/cmd/crawl/main.go | 32 ++- crawler/go.mod | 5 + crawler/go.sum | 4 + crawler/internal/article/article.go | 144 ++++++++++++ crawler/internal/article/article_test.go | 152 +++++++++++++ crawler/internal/fetch/fetch.go | 4 + crawler/internal/sources/source_2016.go | 198 +++------------- crawler/internal/sources/source_2017.go | 143 ++++-------- crawler/internal/sources/sources.go | 98 ++++++-- crawler/internal/sources/sources_test.go | 212 ++++++++++-------- .../sources/testdata/article-2016.html.gz | Bin 0 -> 26224 bytes .../sources/testdata/article-2017.html.gz | Bin 0 -> 27970 bytes docs/data-pipeline.md | 62 +++-- 15 files changed, 653 insertions(+), 415 deletions(-) create mode 100644 crawler/go.sum create mode 100644 crawler/internal/article/article.go create mode 100644 crawler/internal/article/article_test.go create mode 100644 crawler/internal/sources/testdata/article-2016.html.gz create mode 100644 crawler/internal/sources/testdata/article-2017.html.gz diff --git a/.github/workflows/deploy-pages.yml b/.github/workflows/deploy-pages.yml index 80bb772..052f987 100644 --- a/.github/workflows/deploy-pages.yml +++ b/.github/workflows/deploy-pages.yml @@ -33,7 +33,9 @@ jobs: - uses: actions/setup-go@v5 with: go-version: '1.26' - cache-dependency-path: go-parser/go.sum + cache-dependency-path: | + go-parser/go.sum + crawler/go.sum - uses: actions/setup-node@v4 with: diff --git a/README.md b/README.md index 192fdf8..7b82751 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,8 @@ web/ the frontend — one Vite app serving both datasets and th src/router.js pathname → dataset scripts/ site assembly crawler/ Go — re-fetches the source spreadsheets - internal/sources/ one file per data source: its links and local filenames + internal/sources/ per dataset: which article to read, how to name its files + internal/article/ pulls the download links out of that article internal/fetch/ concurrent, resumable downloading go-parser/ Go — Excel to SQLite internal/schema/ canonical 22-column table: DDL, INSERT, subject regexes @@ -58,11 +59,12 @@ them: ```bash npm run crawl:2016 # re-fetch data/2016/ -npm run crawl:2017 # re-fetch data/2017/ from the baotintuc.vn CDN +npm run crawl:2017 # re-fetch data/2017/ ``` -Crawling is idempotent — files already present are skipped — and is never part -of the build. +Each reads the download links out of the article that published the dataset, so +no link list is kept in the repository. Crawling is idempotent — files already +present are skipped — and is never part of the build. Pushing to `main` runs the same steps in `.github/workflows/deploy-pages.yml` and publishes to GitHub Pages. diff --git a/crawler/cmd/crawl/main.go b/crawler/cmd/crawl/main.go index ac95d6d..f05dc9b 100644 --- a/crawler/cmd/crawl/main.go +++ b/crawler/cmd/crawl/main.go @@ -3,24 +3,27 @@ // The argument is the dataset id, the same one used by go-parser's configs and // the published site paths. // -// crawl 2017 # from the baotintuc.vn CDN -// crawl 2016 # source not yet configured +// crawl 2016 # 119 exam-cluster files +// crawl 2017 # 63 province files // crawl 2017 --list # show what would be downloaded // -// Runs are idempotent: a file already present and non-empty is skipped, so an -// interrupted crawl can simply be re-run. +// Each run reads the download links out of the article that published them, so +// --list needs network access too. Runs are idempotent: a file already present +// and non-empty is skipped, so an interrupted crawl can simply be re-run. package main import ( "context" "flag" "fmt" + "net/http" "os" "os/signal" "path/filepath" "syscall" "time" + "github.com/tiennm99/thptqg/crawler/internal/article" "github.com/tiennm99/thptqg/crawler/internal/fetch" "github.com/tiennm99/thptqg/crawler/internal/sources" ) @@ -70,12 +73,22 @@ func run(args []string) error { return err } - files, err := src.Files() + // Ctrl-C cancels the article fetch and any in-flight download; each worker + // deletes its partial file on the way out, so an interrupted run leaves + // nothing half-written. + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + client := &http.Client{Timeout: *timeout} + + fmt.Printf("Reading %s\n", src.Article) + links, err := article.Fetch(ctx, client, src.Article, src.Headers, src.Exts...) if err != nil { return err } - if len(files) == 0 { - return fmt.Errorf("source %q produced no files", src.ID) + files, err := src.Resolve(links) + if err != nil { + return err } outDir, err := filepath.Abs(*out) @@ -99,11 +112,6 @@ func run(args []string) error { return nil } - // Ctrl-C cancels in-flight requests; each worker deletes its partial file - // on the way out, so an interrupted run leaves nothing half-written. - ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) - defer stop() - fmt.Printf("Downloading %d files to %s...\n", len(items), outDir) results, runErr := fetch.Run(ctx, items, fetch.Options{ diff --git a/crawler/go.mod b/crawler/go.mod index dd80379..218e045 100644 --- a/crawler/go.mod +++ b/crawler/go.mod @@ -1,3 +1,8 @@ module github.com/tiennm99/thptqg/crawler go 1.26.5 + +require ( + golang.org/x/net v0.58.0 + golang.org/x/text v0.41.0 +) diff --git a/crawler/go.sum b/crawler/go.sum new file mode 100644 index 0000000..cc9727d --- /dev/null +++ b/crawler/go.sum @@ -0,0 +1,4 @@ +golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= +golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= +golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= +golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= diff --git a/crawler/internal/article/article.go b/crawler/internal/article/article.go new file mode 100644 index 0000000..c808d2b --- /dev/null +++ b/crawler/internal/article/article.go @@ -0,0 +1,144 @@ +// Package article reads download links out of a published web page. +// +// This is what makes the crawler a crawler: the file lists are not carried in +// the repository, they are read from the articles that published them, at run +// time. A source declares which page to read and how to name what it finds. +package article + +import ( + "context" + "fmt" + "io" + "net/http" + "net/url" + "path" + "strings" + + "golang.org/x/net/html" +) + +// Link is one downloadable file found on a page. +type Link struct { + // URL is absolute, resolved against the page it was found on. + URL string + // Text is the anchor's visible text, whitespace-collapsed. Some pages use + // it for the thing being linked ("Bạc Liêu"), others for boilerplate + // ("xem TẠI ĐÂY"), so a source decides whether it is worth anything. + Text string + // File is the last segment of the URL path, query string excluded. + File string +} + +// Extract returns every anchor on the page whose target ends in one of exts. +// +// Pure: no network, so the parsing rules can be tested against a saved copy of +// a real page. Order is document order, and duplicates are kept — deciding +// whether two links to the same file is a problem belongs to the caller, which +// knows what it would name them. +func Extract(pageURL string, body io.Reader, exts ...string) ([]Link, error) { + base, err := url.Parse(pageURL) + if err != nil { + return nil, fmt.Errorf("bad page URL %q: %w", pageURL, err) + } + + doc, err := html.Parse(body) + if err != nil { + return nil, fmt.Errorf("parse %s: %w", pageURL, err) + } + + var out []Link + var walk func(*html.Node) + walk = func(n *html.Node) { + if n.Type == html.ElementNode && n.Data == "a" { + if link, ok := linkOf(base, n, exts); ok { + out = append(out, link) + } + } + for c := n.FirstChild; c != nil; c = c.NextSibling { + walk(c) + } + } + walk(doc) + + return out, nil +} + +// linkOf turns one element into a Link, or reports that it is not one of +// the files being looked for. +func linkOf(base *url.URL, n *html.Node, exts []string) (Link, bool) { + var href string + for _, a := range n.Attr { + if strings.EqualFold(a.Key, "href") { + href = strings.TrimSpace(a.Val) + break + } + } + if href == "" { + return Link{}, false + } + + // Relative hrefs are the norm on the pages this reads; resolving against + // the page URL is what lets a source name a mirror and get its files. + ref, err := url.Parse(href) + if err != nil { + return Link{}, false + } + abs := base.ResolveReference(ref) + + file := path.Base(abs.Path) + if !hasExt(file, exts) { + return Link{}, false + } + + return Link{URL: abs.String(), Text: textOf(n), File: file}, true +} + +func hasExt(name string, exts []string) bool { + lower := strings.ToLower(name) + for _, e := range exts { + if strings.HasSuffix(lower, strings.ToLower(e)) { + return true + } + } + return false +} + +// textOf collects an element's visible text, collapsing whitespace. Anchors on +// these pages wrap the label in styling tags, so the text is rarely a single +// child node. +func textOf(n *html.Node) string { + var b strings.Builder + var walk func(*html.Node) + walk = func(n *html.Node) { + if n.Type == html.TextNode { + b.WriteString(n.Data) + } + for c := n.FirstChild; c != nil; c = c.NextSibling { + walk(c) + } + } + walk(n) + return strings.Join(strings.Fields(b.String()), " ") +} + +// Fetch downloads a page and runs Extract on it. +func Fetch(ctx context.Context, client *http.Client, pageURL string, headers map[string]string, exts ...string) ([]Link, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, pageURL, nil) + if err != nil { + return nil, err + } + for k, v := range headers { + req.Header.Set(k, v) + } + + resp, err := client.Do(req) + if err != nil { + return nil, fmt.Errorf("fetch %s: %w", pageURL, err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("fetch %s: HTTP %d", pageURL, resp.StatusCode) + } + + return Extract(pageURL, resp.Body, exts...) +} diff --git a/crawler/internal/article/article_test.go b/crawler/internal/article/article_test.go new file mode 100644 index 0000000..254bae3 --- /dev/null +++ b/crawler/internal/article/article_test.go @@ -0,0 +1,152 @@ +package article + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +const page = "https://example.test/news/scores.html" + +func extract(t *testing.T, body string, exts ...string) []Link { + t.Helper() + links, err := Extract(page, strings.NewReader(body), exts...) + if err != nil { + t.Fatal(err) + } + return links +} + +// TestResolvesRelativeHrefs is the behaviour the 2016 source depends on: its +// article links files as /upload/..., so they only become fetchable once +// resolved against the page they were found on. +func TestResolvesRelativeHrefs(t *testing.T) { + got := extract(t, ` + root-relative + page-relative + already absolute + protocol-relative + `, ".xlsx") + + want := []string{ + "https://example.test/upload/s/a.xlsx", + "https://example.test/news/sub/b.xlsx", + "https://cdn.other/c.xlsx", + "https://cdn.other/d.xlsx", + } + if len(got) != len(want) { + t.Fatalf("got %d links, want %d", len(got), len(want)) + } + for i, w := range want { + if got[i].URL != w { + t.Errorf("link %d = %q, want %q", i, got[i].URL, w) + } + } +} + +func TestFiltersByExtension(t *testing.T) { + got := extract(t, ` + keep + keep + drop + drop + keep, case-insensitive + no href at all + `, ".xls", ".xlsx") + + if len(got) != 3 { + t.Fatalf("got %d links, want 3: %+v", len(got), got) + } + // ".xls" must not swallow ".xlsx" or vice versa when only one is asked for. + only := extract(t, `xy`, ".xlsx") + if len(only) != 1 || only[0].File != "b.xlsx" { + t.Errorf("extension filter is too loose: %+v", only) + } +} + +// TestFileIgnoresQueryString: a query string is not part of the filename, and +// writing one to disk would produce a name go-parser never sees. +func TestFileIgnoresQueryString(t *testing.T) { + got := extract(t, `x`, ".xlsx") + if len(got) != 1 { + t.Fatalf("got %d links", len(got)) + } + if got[0].File != "report.xlsx" { + t.Errorf("File = %q, want report.xlsx", got[0].File) + } + if !strings.Contains(got[0].URL, "v=2") { + t.Errorf("the query string must survive in the URL: %q", got[0].URL) + } +} + +// TestTextCollectsNestedMarkup: both real articles wrap the label in styling +// tags, so the anchor text is never a single child node. 2017 names its files +// from this text. +func TestTextCollectsNestedMarkup(t *testing.T) { + got := extract(t, `Bà Rịa -Vũng Tàu`, ".xls") + if len(got) != 1 { + t.Fatalf("got %d links", len(got)) + } + if got[0].Text != "Bà Rịa -Vũng Tàu" { + t.Errorf("Text = %q, want %q", got[0].Text, "Bà Rịa -Vũng Tàu") + } +} + +func TestEntitiesInHrefAreDecoded(t *testing.T) { + got := extract(t, `x`, ".xlsx") + if len(got) != 1 || !strings.Contains(got[0].URL, "x=1&y=2") { + t.Errorf("href entity not decoded: %+v", got) + } +} + +// TestMalformedHTMLStillParses: these are hand-edited CMS pages, and the parser +// must not give up on unclosed tags. +func TestMalformedHTMLStillParses(t *testing.T) { + got := extract(t, `

listonetwo`, ".xls") + if len(got) != 2 { + t.Errorf("got %d links, want 2: %+v", len(got), got) + } +} + +func TestExtractRejectsBadPageURL(t *testing.T) { + if _, err := Extract("://nonsense", strings.NewReader("x"), ".xls"); err == nil { + t.Error("expected an error for an unparseable page URL") + } +} + +func TestFetchSendsHeadersAndParses(t *testing.T) { + var ua, ref string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + ua, ref = r.Header.Get("User-Agent"), r.Header.Get("Referer") + w.Write([]byte(`An Giang`)) + })) + defer srv.Close() + + links, err := Fetch(t.Context(), srv.Client(), srv.URL, + map[string]string{"User-Agent": "test-agent", "Referer": "https://ref.test/"}, ".xls") + if err != nil { + t.Fatal(err) + } + if ua != "test-agent" || ref != "https://ref.test/" { + t.Errorf("headers not sent: ua=%q referer=%q", ua, ref) + } + if len(links) != 1 || links[0].Text != "An Giang" || links[0].File != "a.xls" { + t.Errorf("unexpected links: %+v", links) + } +} + +// TestFetchFailsLoudly: a CMS that answers 404 or 403 with an HTML error page +// would otherwise yield zero links and read as "nothing to download". +func TestFetchFailsLoudly(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "not found", http.StatusNotFound) + })) + defer srv.Close() + + if _, err := Fetch(t.Context(), srv.Client(), srv.URL, nil, ".xls"); err == nil { + t.Fatal("expected an error for a non-200 response") + } else if !strings.Contains(err.Error(), "404") { + t.Errorf("error should name the status, got: %v", err) + } +} diff --git a/crawler/internal/fetch/fetch.go b/crawler/internal/fetch/fetch.go index 5afff19..5e34d7e 100644 --- a/crawler/internal/fetch/fetch.go +++ b/crawler/internal/fetch/fetch.go @@ -138,6 +138,10 @@ func Run(ctx context.Context, items []Item, opts Options) ([]Result, error) { // complete. Without that, an interrupted run leaves a truncated file at the // final path — and because the skip check only tests for a non-empty file, // every later run would skip it and the corruption would persist silently. +// +// A .part left behind by an abrupt kill needs no cleanup: it does not satisfy +// the skip check, os.Create truncates it, and the next run re-fetches the file. +// go-parser ignores it in the meantime, since it reads only .xls and .xlsx. func download(ctx context.Context, client *http.Client, it Item, headers map[string]string) Result { if st, err := os.Stat(it.Path); err == nil && st.Size() > 0 { return Result{Item: it, Status: StatusSkip, Size: st.Size()} diff --git a/crawler/internal/sources/source_2016.go b/crawler/internal/sources/source_2016.go index 58f1764..680b31e 100644 --- a/crawler/internal/sources/source_2016.go +++ b/crawler/internal/sources/source_2016.go @@ -1,182 +1,40 @@ package sources +import ( + "fmt" + + "github.com/tiennm99/thptqg/crawler/internal/article" +) + // source2016 fetches the 2016 dataset: one spreadsheet per exam cluster // (cụm thi), 4 .xls and 115 .xlsx. // -// # Provenance +// The article is a mirror. The site that first published this list +// (dtntbacgiang.edu.vn) no longer resolves; this copy of the same article is +// still online, and its links are read at crawl time like any other source. // -// The list below was recovered from the aggregator article that originally -// published these files: +// Dest keeps the server's own filename verbatim — a 32-hex content hash, the +// cluster slug, then a millisecond timestamp. Two reasons not to prettify it: // -// cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html +// - go-parser sorts inputs bytewise and inserts last-wins, so filenames decide +// which row survives a duplicate exam number. That is live here, not +// hypothetical: 877,464 source rows collapse to 877,461, so three rows' +// contents depend on this ordering. +// - go-parser/testdata/reader-fidelity-hashes.tsv is keyed by full path, and +// it is frozen — it was produced by the Rust reader, which no longer exists. // -// The site that first carried it (dtntbacgiang.edu.vn) no longer resolves, so -// the list was read from the Internet Archive's copy. It is verified, not -// guessed: all 119 filenames match data/2016/ exactly in both directions, which -// TestSource2016MatchesFilesOnDisk asserts. -// -// # Filenames are load-bearing — do not "tidy" them -// -// Dest is the server-assigned name, a 32-hex content hash followed by the -// cluster slug and a millisecond timestamp. It is kept verbatim because -// go-parser sorts input files bytewise and inserts last-wins, so filenames -// decide which row survives a duplicate exam number. That is live for 2016, not -// hypothetical: 877,464 source rows collapse to 877,461, so three rows' contents -// depend on this sort order. The hashes in -// go-parser/testdata/reader-fidelity-hashes.tsv are keyed by full path as well, -// and they are frozen — they were produced by the Rust reader, which no longer -// exists. -// -// # The host is not verified -// -// Only the paths are known to be right; the host that still serves them is not. -// The original is gone and the Internet Archive captured the article but none of -// the spreadsheets, so this could not be confirmed by fetching. baseURL points -// at a mirror of the same article that is still online. If it serves these files -// from a different directory, uploadPath is the single constant to change. -const ( - baseURL = "https://dtnt.bacninh.edu.vn" - uploadPath = "/upload/s/20180102/" -) - +// Unlike 2017 this needs no transliteration: the name comes from the URL, so it +// is exact by construction rather than derived from a label. var source2016 = Source{ - ID: "2016", - Summary: "119 exam-cluster files (4 .xls + 115 .xlsx)", - Files: func() ([]File, error) { - out := make([]File, 0, len(clusters2016)) - for _, c := range clusters2016 { - out = append(out, File{ - Name: c.name, - URL: baseURL + uploadPath + c.file, - Dest: c.file, - }) + ID: "2016", + Summary: "119 exam-cluster files (4 .xls + 115 .xlsx)", + Article: "https://dtnt.bacninh.edu.vn/tin-tuc/tin-tuc-su-kien/cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html", + Exts: []string{".xls", ".xlsx"}, + WantFiles: 119, + Dest: func(l article.Link) (string, error) { + if l.File == "" { + return "", fmt.Errorf("link %s has no filename", l.URL) } - return out, nil + return l.File, nil }, } - -type cluster struct{ name, file string } - -// clusters2016 is in the article's publication order. That order is incidental — -// go-parser re-sorts by filename — but it is kept as published so the list can -// be diffed against the source article. -var clusters2016 = []cluster{ - {"Trường Đại học Bách khoa Hà Nội", "b34c777942ca8a4de91f23a35cce1c6bdhbachkhoahn-1468901491486.xlsx"}, - {"Trường Đại học Sư phạm Hà Nội", "76f19098881e9c79f66ac4f1b42a8a55dhsuphamhanoi-1468901144203.xlsx"}, - {"Trường Đại học Thuỷ lợi * Cơ sở 1 ở phía Bắc", "4a3fa95a83fe9b3b2fafab309323fd22dhthuyloi-1468901840919.xlsx"}, - {"Học viện Kỹ thuật Quân sự * Cơ sở 1 ở phía Bắc (Quân đội)", "4334ea476a2b8196d54ba40a341831echvkythuatquansu-1468920342994.xlsx"}, - {"Trường Đại học Lâm nghiệp", "c5ecc58a731fd1a54b7e6cc3bc99449ddhlamnghiep-1468939300790.xlsx"}, - {"Trường Đại học Bách khoa - Đại học Quốc gia Thành phố Hồ Chí Minh", "357c6aaf59aeaa6c11c3e83c595d38cfdhbachkhoa-tphcm-1468939678829.xlsx"}, - {"Trường Đại học Khoa học tự nhiên - Đại học Quốc gia Thành phố Hồ Chí Minh", "64e44466ffeff9a587260795644805d0dhkhtunhien-dhqgtphcm-1468921015146.xlsx"}, - {"Trường Đại học Khoa học xã hội và Nhân văn - Đại học Quốc gia Thành phố Hồ Chí Minh", "5fd13ae5287cbf0149bd733b507e0608dhkhxhnv-dhqgtphcm-1468920630193.xlsx"}, - {"Trường Đại học Sư phạm Tp.HCM", "a995de3f7a7055cf10b2f002f0e194f7dhsptphcm-1468920792291.xlsx"}, - {"Trường Đại học Hàng Hải", "c6ef16bc9f75ce0fd16c1229afad4e71dhhanghai-1468906457479.xlsx"}, - {"Trường Đại học Sư phạm - Đại học Thái Nguyên", "ee09c723da4e86cdbe26213413450af2dhsupham-dhthainguyen-1468939061872.xlsx"}, - {"Trường Đại học Kỹ thuật công nghiệp - Đại học Thái Nguyên", "8af8f1b29b6a40e26ccb4a2e19e8eededhkythuatcongnghiep-dhthainguyen-1468901919568.xlsx"}, - {"Trường Đại học Nông lâm - Đại học Thái Nguyên", "53e5e20ebd74d84741513b703fbd13bfdhnonglam-dhthainguyen-1468939023637.xlsx"}, - {"Học viện Ngân hàng", "f2846894bd5533e02b122a1e3b4333f6hvnganhang-1468939472850.xlsx"}, - {"Trường Đại học Luật Hà Nội", "9e0e564ae801e347bd7649da8b89ad90dhluathn-1468920377978.xlsx"}, - {"Trường Đại học Tân Trào", "20db5eaace927596f1806f151649ee6edhtantrao-1468901296040.xlsx"}, - {"Trường Đại học Xây dựng Hà Nội", "7b5d74564ab9f6e0e137f6d161fab6f0dhxaydung-1468940561682.xlsx"}, - {"Trường Đại học Khoa học - Đại học Thái Nguyên", "c2cfaaf287fb1e7120894e545771b027dhkhoahoc-dhthainguyen-1468900365545.xlsx"}, - {"Đại học Thái Nguyên", "3daa6ee33fdafadc3c112c18bff18c2ddhthainguyen-1468940268065.xlsx"}, - {"Học viện Tài chính", "96c95c85c5549a06f2e92ffcacf8ff21hvtaichinh-1468934470924.xlsx"}, - {"Trường Đại học Tây Bắc", "74e6cd0dc78a3fb7dcf56ae57283b9b9dhtaybac-1468940302052.xlsx"}, - {"Trường Đại học Hùng Vương", "778ab074ed6a95d209fa901fd5d68843dhhungvuong-1468901964127.xlsx"}, - {"Trường Đại học Sư phạm Hà Nội 2", "a11e707e6ed5fe4d29a8e32380d7558cdhsphanoi-2-1468902006284.xlsx"}, - {"Trường Đại học Ngoại thương * Cơ sở 1 ở phía Bắc", "ec84067268f47d523b1e80b493248377dhngoaithuong-1468920480753.xlsx"}, - {"Trường Đại học Kinh tế Quốc dân", "8f948c2a5ab49a3c5fd03b6ea04887a7dhkinhtequocdanfix-1468902083782.xlsx"}, - {"Trường Đại học Giao thông Vận tải", "51bb2f9b6e9f565c2d705606fd56958bdhgiaothongvantai-1468939107463.xlsx"}, - {"Học viện Nông Nghiệp Việt Nam", "5aede4951e7af8c1e1056c747a553386hvnongnghiepvn-1468900610577.xlsx"}, - {"Trường Đại học Sư phạm Kỹ thuật Hưng Yên", "f85f04bb2bbd84457b83ac846728a2a6dhspkythuathungyen-1468920667091.xlsx"}, - {"Trường Đại học Hải Phòng", "deca197a916a8f633f6894b4e13c3ca4dhhaiphong-1468901178368.xlsx"}, - {"Trường Đại học Thương mại", "b2552e20c9ff4b237244dfbf2e278a47dhthuongmai-1468902876264.xlsx"}, - {"Trường Đại học Công nghiệp Hà Nội", "82338a4a93ea1e0cde26e0061066ca8adhcongnghiephanoi-1468920154757.xlsx"}, - {"Y Dược Thái Bình", "22e8617886412abd90ae3d33cc6d7db4dhyduocthaibinh-1468941020145.xlsx"}, - {"Trường Đại học Mỏ Địa chất", "c174631d90303276ac63ff38b31157bbdhmodiachat-1468900697349.xlsx"}, - {"Trường Đại học Hồng Đức", "bd857add0cfe92c1c247469ff2104527dh-hong-duc-1468900427119.xlsx"}, - {"Trường Đại học Vinh", "64d2635efd30a0d0db4f7c30b9735261dhvinh-1468940230027.xlsx"}, - {"Trường ĐH Sư phạm - ĐH Huế", "ab2f259ef55cb08b3a436d3f15d42cc7dhsupham-dhhue-1468934431828.xlsx"}, - {"Trường ĐH Khoa học - ĐH Huế", "87ffaf3197ba20f0919f98a0b3f33914dhkhoahoc-dhhue-1468938729472.xlsx"}, - {"Trường ĐH Kinh tế - ĐH Huế", "8916da7b3c4d58cffc1dd29dc7a8aabfdhkinhte-dhhue-1468938560205.xlsx"}, - {"Đại học Huế", "919276a0348634459f8937dfcd6c7129dhhue-1468938792115.xlsx"}, - {"ĐH Đà Nẵng", "b42fd58ead4c54279d7df5f53160a851dhdanang-1468920187203.xlsx"}, - {"Trường ĐH Bách khoa - ĐH Đà Nẵng", "43213f61074068d9177685eb6c3f2d3bdhbachkhoa-dhdanang-1468938494437.xlsx"}, - {"Trường ĐH Sư phạm - ĐH Đà Nẵng", "54bbcf865c137d58cf62af15733bd376dhsupham-dhdanang-1468938526558.xlsx"}, - {"Trường Đại học Quy Nhơn", "70bf8cd2b6e87523219e55149a8524c5dhquynhon-1468938878328.xlsx"}, - {"Trường ĐH Xây dựng miền Trung", "4e9bb192d56c9fa92b3e0ec13d8236c8dhxaydungmientrung-1468940978346.xlsx"}, - {"Trường ĐH Nông Lâm TPHCM", "b2a739584a8861502573d1c6372fd6e9dhnonglamtphcm-1468939555326.xlsx"}, - {"Trường ĐH Ngoại ngữ - ĐH Đà Nẵng", "9ba098a7913ad837a66e173b3fc41fd7dhngoaingu-dhdanang-1468902750424.xlsx"}, - {"Trường ĐH Tây Nguyên", "216ee4e385dbc236767be899b068bf98dhtaynguyen-1468940490240.xlsx"}, - {"Trường ĐH Tài chính Marketing", "af6828f837a80e03d77a65f3d310ec60dh-tai-chinh-marketing-1468924711596.xlsx"}, - {"Trường ĐH Nha Trang", "6375fd13557bee08d20f1f3e46dac130dhnhatrang-coso1-1468901350425.xlsx"}, - {"Trường ĐH Giao thông vận tải TP.HCM", "c5c28959fcd0b4e39dd9fd538db5448adhgtvttphcm-1468939150821.xlsx"}, - {"Trường ĐH Sư phạm Kĩ thuật TP.HCM", "514877c1cfebdc9c2251f651b5f02c7edhspkythuattphcm-1468906568655.xlsx"}, - {"Trường ĐH Đà Lạt", "2b33640b0dd154490f2809d96be81797dhdalat-1468902130616.xlsx"}, - {"Trường ĐH Kinh tế TP.HCM", "f0aa5a19c2c8c1fde8211341f0e98e26dhkinhtetphcm-1468939229043.xlsx"}, - {"Trường ĐH Kinh tế - Luật - ĐHQG TP.HCM", "ff877788e43bd84f0119dae026996892dhkinhteluat-dhqgtphcm-1468939860894.xlsx"}, - {"Trường ĐH Công nghiệp Thực phẩm TP.HCM", "167116af6c8a096c2a9307edfa115f77dhcnthucpham-tphcm-1468910578187.xls"}, - {"Trường ĐH Công nghiệp TP.HCM", "487c7287c1c9735722b31d877942becddhcongnghieptphcm-1468983373303.xlsx"}, - {"Trường ĐH Sài Gòn", "d75282bff0a4d8cd1421c3fb74f789c3dhsaigon-1468900867561.xlsx"}, - {"Trường ĐH Đồng Tháp", "388fcdfdac18ef6eee6125a2a015714ddhdongthap-1468934545359.xlsx"}, - {"Trường Đại học An Giang", "fc511852659150f6305f6f624445300ddhangiang-1468940189321.xlsx"}, - {"Trường ĐH Tôn Đức Thắng", "9d78f6dce7965fb4b7a337279920d4b1dhtonducthang-1468902168623.xlsx"}, - {"Trường ĐH Tiền Giang", "bacd1b2133b37d0ea9fbf46445265036dhtiengiang-1468940391597.xlsx"}, - {"Trường ĐH Cần Thơ", "b35a3f26a162bec3740e2da5639327afdhcantho-1468907131146.xls"}, - {"Trường ĐH Cần Thơ - Hậu Giang", "7f1cbe7f35d1f539ca817a0ab8bd00c2dhcantho-haugiang-1468907167921.xls"}, - {"Trường ĐH Luật TP.HCM", "d652d24e8bd4fe6a48d634b116f37c27dhluattphcm-1468939398928.xlsx"}, - {"Trường ĐH Sư phạm Kỹ thuật Vĩnh Long", "4c2fd846ed2a0670ba6d1200d1d68e1dspkythuatvinhlong-1468902640438.xlsx"}, - {"Trường ĐH Trà Vinh", "7418e83ff9d07b146216a3beafa927bbdhtravinh-1468902716174.xlsx"}, - {"Trường ĐH Ngân hàng TP.HCM", "86a692bebf9b0b7c58adb9876b9befd5dhnganhangtphcm-1468902244566.xlsx"}, - {"Trường ĐH Cần Thơ - Bạc Liêu", "5ca241aa1de8d25a3e6ff6db75234025dhcantho-baclieu-1468907071460.xls"}, - {"Trường ĐH Kiên Giang", "cdc912664d7217c75eb9b67c6ef3797adhkiengiang-1468901222446.xlsx"}, - {"Trường ĐH Y Dược Cần Thơ", "023718c7d3cf7ace3a7116fabb12bd9cdhyduoccantho-1468920829104.xlsx"}, - {"Sở GDĐT Hà Nội", "1c66db0265bf45df237b37a7bcaeaef3hanoi-1468932790746.xlsx"}, - {"Sở GDĐT Hà Giang", "178814c33c8b85a77d0fc295fa6d2be1hagiang-1468932847230.xlsx"}, - {"Sở GDĐT Cao Bằng", "ddb631f833af894200ef10cd408d2ef0caobang-1468932877769.xlsx"}, - {"Sở GDĐT Lai Châu", "a3aa106104f114c6426deb2d5e5da5fdlaichau-1468932954079.xlsx"}, - {"Sở GDĐT Lào Cai", "d035f470c13adbf5785797c6ccdcb204laocai-1468919805337.xlsx"}, - {"xem TẠI ĐÂY ) 07 Sở GDĐT Tuyên Quang", "d5ecd2574b5ab36c62dffad435262cb8tuyenquang-1468902296807.xlsx"}, - {"Sở GDĐT Lạng Sơn", "7afd5d7fdff7c170a6342cd8f9474b22langson-1468932984587.xlsx"}, - {"Sở GDĐT Bắc Kạn", "17843bccde10f362c2b9600bc49cd3f8backan-1468933016076.xlsx"}, - {"Sở GDĐT Thái Nguyên", "5fb3a1b66aff077ee3da482be08cd391thainguyen-1468902797020.xlsx"}, - {"Sở GDĐT Yên Bái", "97aa741a4c2b1276559eb021779daa32yenbai-1468933116055.xlsx"}, - {"Sở GDĐT Sơn La", "4b98fdd7300b86a94636e4d15394fc19sonla-1468899319124.xlsx"}, - {"Sở GDĐT Phú Thọ", "5465cca59495dcd9a48f689e2d38b85ephutho-1468902409682.xlsx"}, - {"xem TẠI ĐÂY ) 14 Sở GDĐT Vĩnh Phúc", "4e0ccee19eb64513d23a6dff96ff7f9cvinhphuc-1468919874814.xlsx"}, - {"Sở GDĐT Quảng Ninh", "a31b3446bae5a17c517348d8c38c5b7equangninh-1468919920197.xlsx"}, - {"Sở GDĐT Bắc Giang", "b7a0c1ecce7c8445b3a1021091a5e654bacgiang-1468983153932.xlsx"}, - {"Sở GDĐT Bắc Ninh", "7e0d0b6fb981d407cffdb79d724e8aecbacninh-1468919984998.xlsx"}, - {"Sở GDĐT Hải Dương", "7cfc91bd6a1f5ae9d7917e920f469b90haiduong-1468899640007.xlsx"}, - {"Sở GDĐT Hưng Yên", "92045214159cb0335b2836aa1c4bb25ahungyen-1468899732002.xlsx"}, - {"Sở GDĐT Hoà Bình", "505a876dfd1ea321291e29192aebe95ehoabinh-1468983220125.xlsx"}, - {"Sở GDĐT Hà Nam", "e693bc9de67acc916032c6a3b2fc9eedhanam-1468933146526.xlsx"}, - {"Sở GDĐT Nam Định", "dfa63c5b68a8f638379ca0e9c5a1c556namdinh-1468933234061.xlsx"}, - {"Sở GDĐT Thái Bình", "6960b6fa493e1365a183af461e945389thaibinh-1468920019273.xlsx"}, - {"xem TẠI ĐÂY ) 24 Sở GDĐT Ninh Bình", "0b583c9875443e65ffa449d5ca76fae8ninhbinh-1468899778557.xlsx"}, - {"xem TẠI ĐÂY ) 25 Sở GDĐT Thanh Hoá", "238adc19e80c0daef089a307af23edcethanhhoa-1468933304999.xlsx"}, - {"Sở GDĐT Nghệ An", "7f12b90225e91f9d00aac3bcdf9974e1nghean-1468920043121.xlsx"}, - {"Sở GDĐT Quảng Bình", "9c730c43bf4309171528c3853ed53b19quangbinh-1468933376032.xlsx"}, - {"Sở GDĐT Quảng Trị", "2208903e1fa12220d100f93e40d5ff3bquangtri-1468933440097.xlsx"}, - {"Sở GDĐT Thừa Thiên -Huế", "6badc7cfa258b59ad67e2b64f713b95fthuathienhue-1468983258438.xlsx"}, - {"Sở GDĐT Quảng Nam", "ec765c32190773f5a18cd8202a5eaf0dquangnam-1468933501467.xlsx"}, - {"Sở GDĐT Quảng Ngãi", "e80dc028d53fa49ee247391a0653dee5quangngai-1468933592400.xlsx"}, - {"Sở GDĐT Kon Tum", "cc57dc986acb525086b86e85df54949dkontum-1468933632113.xlsx"}, - {"Sở GDĐT Bình Định", "412e62265f35ca0078a0d94af5d1ed8abinhdinh-1468899884192.xlsx"}, - {"Sở GDĐT Gia Lai", "698b930ab7697a0672bbc39168024c9cgialai-1468933753098.xlsx"}, - {"Sở GDĐT Đắk Lắk", "2004a8d225fd87524e324d2d519e87a4daclac-1468899960505.xlsx"}, - {"Sở GDĐT Khánh Hoà", "4ad416ccf014bd5d01f415238566b210khanhhoa-1468933789968.xlsx"}, - {"Sở GDĐT Lâm Đồng", "2fa6d9a308bd11bc09f02d787eef2c15lamdong-1468933821082.xlsx"}, - {"Sở GDĐT Ninh Thuận", "15c85d901a73dd49f2bae71aadcdbb5cninhthuan-1468920071763.xlsx"}, - {"Sở GDĐT Đồng Nai", "538c9676b70f2509573245d58fba93bcdongnai-1468938241935.xlsx"}, - {"Sở GDĐT Đồng Tháp", "575df8a4df74e87abb8f55d807324374dongthap-1468933902699.xlsx"}, - {"Sở GDĐT Kiên Giang", "7435bb054e8e4ec34c173af1b6ad8402kiengiang-1468900147395.xlsx"}, - {"Sở GDĐT Cần Thơ", "5b16d7156a03bb7ed0618796bae87bc3cantho-1468933954714.xlsx"}, - {"Sở GDĐT Bến Tre", "74145979ee30186bf4e40ee3e6a3ac74bentre-1468934010040.xlsx"}, - {"Sở GDĐT Vĩnh Long", "7f4362ef567cdf74495368e0fc11072avinhlong-1468934067981.xlsx"}, - {"Sở GDĐT Trà Vinh", "08cdeb3636dcb2adaef829d62968274atravinh-1468902443859.xlsx"}, - {"Sở GDĐT Sóc Trăng", "a8ff80d9478747e36d21ec95cc9ecb8fsoctrang-1468902686409.xlsx"}, - {"Sở GDĐT Bạc Liêu", "18055d17e46854ecc6c8be585bcb6e5cbaclieu-1468934144150.xlsx"}, - {"Sở GDĐT Điện Biên", "3c0e56abd8c5334f4e1ff7df3ac23a15dienbien-1468920113820.xlsx"}, - {"Sở GDĐT Đăk Nông", "bf259a908ff8a8dcc067a761204459f4daknong-1468934173803.xlsx"}, - {"Sở GDĐT Hậu Giang", "964c4368131f6be6058fabb1caefba1chaugiang-1468934247324.xlsx"}} diff --git a/crawler/internal/sources/source_2017.go b/crawler/internal/sources/source_2017.go index c3a8738..4f9d088 100644 --- a/crawler/internal/sources/source_2017.go +++ b/crawler/internal/sources/source_2017.go @@ -1,121 +1,70 @@ package sources import ( + "fmt" "regexp" "strings" + + "golang.org/x/text/unicode/norm" + + "github.com/tiennm99/thptqg/crawler/internal/article" ) -// article is the page the links were taken from. The CDN rejects requests that -// do not carry it as a Referer. -const article = "https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm" - -// browserUA is required too: the CDN 403s an unrecognised User-Agent. +// browserUA and the Referer are both required: the CDN 403s an unrecognised +// User-Agent, and rejects requests that do not carry the article as Referer. const browserUA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36" -// source2017 fetches the 2017 dataset from the baotintuc.vn CDN: one .xls per -// province. +const article2017 = "https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm" + +// source2017 fetches the 2017 dataset: one .xls per province, still served from +// the CDN that originally published it. // -// This is the only dataset that can be re-fetched from its original source. -// The CDN filenames are inconsistent (Angiang.xls, 1BaRiaVungTau.xls, -// 23HaiPhong.xls, gia-lai.xls), so local names are derived from the province -// name instead — which is what produced the data/2017/ files currently on disk, -// and what any re-crawl must keep producing. +// Dest cannot use the CDN's filenames the way 2016 does, because they are +// inconsistent — Angiang.xls, 1BaRiaVungTau.xls, 23HaiPhong.xls, Gia-Lai.xls — +// carrying upload-order prefixes and arbitrary capitalisation. The province name +// in the link text is the stable identifier, so the local name is derived from +// that instead, which is what produced the files in data/2017/. var source2017 = Source{ - ID: "2017", - Summary: "63 province .xls files from the baotintuc.vn CDN", + ID: "2017", + Summary: "63 province .xls files from the baotintuc.vn CDN", + Article: article2017, + Exts: []string{".xls"}, + WantFiles: 63, Headers: map[string]string{ "User-Agent": browserUA, - "Referer": article, + "Referer": article2017, }, - Files: func() ([]File, error) { - out := make([]File, 0, len(provinces2017)) - for _, p := range provinces2017 { - out = append(out, File{ - Name: p.name, - URL: p.url, - Dest: slug(p.name) + ".xls", - }) + Dest: func(l article.Link) (string, error) { + s := slug(l.Text) + if s == "" { + return "", fmt.Errorf("link %s has no usable label to name it by", l.URL) } - return out, nil + return s + ".xls", nil }, } var nonAlphanumeric = regexp.MustCompile(`[^a-z0-9]+`) -// slug converts a province name to its local filename stem: lowercase, with -// every run of non-alphanumeric characters collapsed to a single hyphen. +// slug turns a province name into a filename stem: transliterated to ASCII, +// lowercased, with every run of other characters collapsed to one hyphen. // -// "Ba Ria - Vung Tau" -> "ba-ria-vung-tau" +// "Bà Rịa -Vũng Tàu" -> "ba-ria-vung-tau" +// "Bắc Kạn" -> "bac-kan" // -// The names are already unaccented ASCII, so no transliteration is involved. +// The article writes the names with diacritics, so this has to strip them. +// Combining marks are removed by the literal range U+0300–U+036F rather than by +// the unicode.Mn category, matching what go-parser does to build ho_ten_ascii. func slug(name string) string { - return strings.Trim(nonAlphanumeric.ReplaceAllString(strings.ToLower(name), "-"), "-") -} - -type province struct{ name, url string } - -var provinces2017 = []province{ - {"An Giang", "https://cdnmedia.baotintuc.vn/2017/07/06/17/57/Angiang.xls"}, - {"Bac Lieu", "https://cdnmedia.baotintuc.vn/2017/07/06/17/59/Baclieu.xls"}, - {"Ba Ria - Vung Tau", "https://cdnmedia.baotintuc.vn/2017/07/06/08/17/1BaRiaVungTau.xls"}, - {"Bac Giang", "https://cdnmedia.baotintuc.vn/2017/07/06/09/33/BacGiang.xls"}, - {"Bac Kan", "https://cdnmedia.baotintuc.vn/2017/07/06/08/27/BacKan.xls"}, - {"Bac Ninh", "https://cdnmedia.baotintuc.vn/2017/07/06/09/33/BacNinh.xls"}, - {"Ben Tre", "https://cdnmedia.baotintuc.vn/2017/07/06/09/34/BenTre.xls"}, - {"Binh Duong", "https://cdnmedia.baotintuc.vn/2017/07/06/09/34/BinhDuong.xls"}, - {"Binh Thuan", "https://cdnmedia.baotintuc.vn/2017/07/06/09/35/BinhThuan.xls"}, - {"Binh Phuoc", "https://cdnmedia.baotintuc.vn/2017/07/06/09/09/BinhPhuoc.xls"}, - {"Binh Dinh", "https://cdnmedia.baotintuc.vn/2017/07/06/13/05/Binhdinh.xls"}, - {"Ca Mau", "https://cdnmedia.baotintuc.vn/2017/07/06/13/05/Camau.xls"}, - {"Cao Bang", "https://cdnmedia.baotintuc.vn/2017/07/06/18/02/Caobang.xls"}, - {"Can Tho", "https://cdnmedia.baotintuc.vn/2017/07/06/13/07/Cantho.xls"}, - {"Da Nang", "https://cdnmedia.baotintuc.vn/2017/07/06/18/03/Danang.xls"}, - {"Dak Nong", "https://cdnmedia.baotintuc.vn/2017/07/06/09/35/DakNong.xls"}, - {"Dak Lak", "https://cdnmedia.baotintuc.vn/2017/07/06/18/02/Daklak.xls"}, - {"Dong Nai", "https://cdnmedia.baotintuc.vn/2017/07/06/13/08/dongnai.xls"}, - {"Dong Thap", "https://cdnmedia.baotintuc.vn/2017/07/06/17/59/Dongthap.xls"}, - {"Dien Bien", "https://cdnmedia.baotintuc.vn/2017/07/06/09/08/DienBien.xls"}, - {"Gia Lai", "https://cdnmedia.baotintuc.vn/2017/07/06/13/09/Gia-Lai.xls"}, - {"Ha Giang", "https://cdnmedia.baotintuc.vn/2017/07/06/18/18/Hagiang.xls"}, - {"Ha Noi", "https://cdnmedia.baotintuc.vn/2017/07/07/08/16/HaNoi.xls"}, - {"Ha Nam", "https://cdnmedia.baotintuc.vn/2017/07/06/09/07/Hanam.xls"}, - {"Ha Tinh", "https://cdnmedia.baotintuc.vn/2017/07/06/09/36/HaTinh.xls"}, - {"Hai Phong", "https://cdnmedia.baotintuc.vn/2017/07/06/08/18/23HaiPhong.xls"}, - {"Hai Duong", "https://cdnmedia.baotintuc.vn/2017/07/06/09/36/HaiDuong.xls"}, - {"Hau Giang", "https://cdnmedia.baotintuc.vn/2017/07/06/18/00/Haugiang.xls"}, - {"Ho Chi Minh", "https://cdnmedia.baotintuc.vn/2017/07/06/08/26/HCM.xls"}, - {"Hoa Binh", "https://cdnmedia.baotintuc.vn/2017/07/06/09/06/HoaBinh.xls"}, - {"Hung Yen", "https://cdnmedia.baotintuc.vn/2017/07/06/08/25/HungYen.xls"}, - {"Khanh Hoa", "https://cdnmedia.baotintuc.vn/2017/07/06/18/04/Khanhhoa.xls"}, - {"Kien Giang", "https://cdnmedia.baotintuc.vn/2017/07/06/08/28/KienGiang.xls"}, - {"Kon Tum", "https://cdnmedia.baotintuc.vn/2017/07/06/18/05/KonTum.xls"}, - {"Nam Dinh", "https://cdnmedia.baotintuc.vn/2017/07/06/07/55/13NamDinh.xls"}, - {"Nghe An", "https://cdnmedia.baotintuc.vn/2017/07/06/07/55/14NgheAn.xls"}, - {"Ninh Binh", "https://cdnmedia.baotintuc.vn/2017/07/06/09/37/NinhBinh.xls"}, - {"Ninh Thuan", "https://cdnmedia.baotintuc.vn/2017/07/06/18/00/Ninhthuan.xls"}, - {"Lao Cai", "https://cdnmedia.baotintuc.vn/2017/07/06/08/52/11LaoCai.xls"}, - {"Lai Chau", "https://cdnmedia.baotintuc.vn/2017/07/06/13/33/LaiChau.xls"}, - {"Lang Son", "https://cdnmedia.baotintuc.vn/2017/07/06/18/05/Langson.xls"}, - {"Lam Dong", "https://cdnmedia.baotintuc.vn/2017/07/06/09/00/LamDong.xls"}, - {"Long An", "https://cdnmedia.baotintuc.vn/2017/07/06/08/53/12LongAn.xls"}, - {"Quang Binh", "https://cdnmedia.baotintuc.vn/2017/07/06/09/23/QuangBinh.xls"}, - {"Quang Nam", "https://cdnmedia.baotintuc.vn/2017/07/06/09/24/QuangNam.xls"}, - {"Quang Ninh", "https://cdnmedia.baotintuc.vn/2017/07/06/18/06/Quangninh.xls"}, - {"Quang Ngai", "https://cdnmedia.baotintuc.vn/2017/07/06/09/24/QuangNgai.xls"}, - {"Quang Tri", "https://cdnmedia.baotintuc.vn/2017/07/06/09/25/QuangTri.xls"}, - {"Phu Tho", "https://cdnmedia.baotintuc.vn/2017/07/06/09/23/PhuTho.xls"}, - {"Phu Yen", "https://cdnmedia.baotintuc.vn/2017/07/06/09/38/PhuYen.xls"}, - {"Son La", "https://cdnmedia.baotintuc.vn/2017/07/06/18/01/Sonla.xls"}, - {"Soc Trang", "https://cdnmedia.baotintuc.vn/2017/07/06/18/06/Soctrang.xls"}, - {"Tay Ninh", "https://cdnmedia.baotintuc.vn/2017/07/06/09/26/TayNinh.xls"}, - {"Thai Binh", "https://cdnmedia.baotintuc.vn/2017/07/06/09/09/ThaiBinh.xls"}, - {"Thai Nguyen", "https://cdnmedia.baotintuc.vn/2017/07/06/13/35/ThaiNguyen.xls"}, - {"Thanh Hoa", "https://cdnmedia.baotintuc.vn/2017/07/06/13/10/thanhoa.xls"}, - {"Tra Vinh", "https://cdnmedia.baotintuc.vn/2017/07/06/09/38/TraVinh.xls"}, - {"Thua Thien Hue", "https://cdnmedia.baotintuc.vn/2017/07/06/13/10/thuathienhue.xls"}, - {"Tien Giang", "https://cdnmedia.baotintuc.vn/2017/07/06/13/11/tiengiang.xls"}, - {"Tuyen Quang", "https://cdnmedia.baotintuc.vn/2017/07/06/09/39/TuyenQuang.xls"}, - {"Vinh Phuc", "https://cdnmedia.baotintuc.vn/2017/07/06/09/40/VinhPhuc.xls"}, - {"Vinh Long", "https://cdnmedia.baotintuc.vn/2017/07/06/13/13/vinhlong.xls"}, - {"Yen Bai", "https://cdnmedia.baotintuc.vn/2017/07/06/18/07/Yenbai.xls"}, + var b strings.Builder + for _, r := range norm.NFD.String(name) { + switch { + case r >= 0x0300 && r <= 0x036F: + // combining mark: drop + case r == 'đ' || r == 'Đ': + b.WriteByte('d') + default: + b.WriteRune(r) + } + } + return strings.Trim(nonAlphanumeric.ReplaceAllString(strings.ToLower(b.String()), "-"), "-") } diff --git a/crawler/internal/sources/sources.go b/crawler/internal/sources/sources.go index 0efafc7..c18da33 100644 --- a/crawler/internal/sources/sources.go +++ b/crawler/internal/sources/sources.go @@ -1,25 +1,16 @@ -// Package sources describes where each dataset's spreadsheets come from. +// Package sources says where each dataset's spreadsheets are published and +// what to call them locally. // -// A source is a dataset id plus the list of remote files that fill it. -// Downloading is internal/fetch's job; deciding what to download and what to -// call it locally is this package's. +// It carries no link lists. Each source names the article that published the +// files; internal/article reads the links out of that page at run time, and +// internal/fetch moves the bytes. package sources -import "fmt" +import ( + "fmt" -// File is one remote spreadsheet. -type File struct { - // Name is the human label used in progress output, e.g. "An Giang". - Name string - URL string - // Dest is the filename within the dataset directory. - // - // This is not cosmetic. go-parser sorts its input files and inserts with - // INSERT OR REPLACE, which is last-wins, so filenames decide which row - // survives a duplicate exam number. A re-crawl that names files differently - // can produce a database with the same row count and different content. - Dest string -} + "github.com/tiennm99/thptqg/crawler/internal/article" +) // Source is one crawlable dataset. type Source struct { @@ -28,12 +19,73 @@ type Source struct { // cannot be pointed at the wrong dataset's directory. ID string Summary string - // Headers are sent with every request for this source. + + // Article is the published page listing this dataset's files. + Article string + + // Exts are the file extensions to pick out of that page. + Exts []string + + // Headers are sent with every request for this source, for the article and + // the files alike. Headers map[string]string - // Files lists what to download. It returns an error rather than an empty - // list when a source is known but not yet configured, so "nothing to do" - // can never be mistaken for success. - Files func() ([]File, error) + + // WantFiles is how many links the article is expected to yield. A page that + // suddenly yields fewer has changed shape, and silently crawling a partial + // dataset is the failure this exists to prevent — go-parser would happily + // build a short database and only the row-count guard would catch it, after + // the fact. + WantFiles int + + // Dest names the local file for one discovered link. + // + // This is the load-bearing part. go-parser sorts input files bytewise and + // inserts last-wins, so the names chosen here decide which row survives a + // duplicate exam number. Two sources answer it differently and both have a + // reason: see source_2016.go and source_2017.go. + Dest func(article.Link) (string, error) +} + +// File is one spreadsheet to download. +type File struct { + // Name is the human label used in progress output. + Name string + URL string + // Dest is the filename within the dataset directory. + Dest string +} + +// Resolve turns the links found on a source's article into the files to fetch, +// enforcing the count and rejecting any two links that would write to the same +// name — which would silently cost the dataset a file. +func (s Source) Resolve(links []article.Link) ([]File, error) { + if len(links) != s.WantFiles { + return nil, fmt.Errorf( + "%s: found %d links on %s, expected %d — the page has changed shape; "+ + "check it before crawling, a partial dataset builds without complaint", + s.ID, len(links), s.Article, s.WantFiles) + } + + out := make([]File, 0, len(links)) + seen := make(map[string]string, len(links)) + for _, l := range links { + dest, err := s.Dest(l) + if err != nil { + return nil, fmt.Errorf("%s: %w", s.ID, err) + } + if prev, dup := seen[dest]; dup { + return nil, fmt.Errorf("%s: %q and %q both name the local file %q", + s.ID, prev, l.URL, dest) + } + seen[dest] = l.URL + + name := l.Text + if name == "" { + name = dest + } + out = append(out, File{Name: name, URL: l.URL, Dest: dest}) + } + return out, nil } // registry is ordered: it drives the help output. diff --git a/crawler/internal/sources/sources_test.go b/crawler/internal/sources/sources_test.go index dca1379..b4586f8 100644 --- a/crawler/internal/sources/sources_test.go +++ b/crawler/internal/sources/sources_test.go @@ -1,69 +1,62 @@ package sources import ( + "compress/gzip" "os" "path/filepath" "strings" "testing" + + "github.com/tiennm99/thptqg/crawler/internal/article" ) -func TestSlug(t *testing.T) { - for in, want := range map[string]string{ - "An Giang": "an-giang", - "Ba Ria - Vung Tau": "ba-ria-vung-tau", - "Ho Chi Minh": "ho-chi-minh", - "Thua Thien Hue": "thua-thien-hue", - } { - if got := slug(in); got != want { - t.Errorf("slug(%q) = %q, want %q", in, got, want) - } - } -} - -func TestSource2017IsComplete(t *testing.T) { - files, err := source2017.Files() +// The fixtures are saved copies of the two source articles, kept so the link +// extraction and the naming rules can be exercised without the network. +// +// article-2017 is the live page. article-2016 came from the Internet Archive's +// copy of the site that first published that list (dtntbacgiang.edu.vn, which +// no longer resolves); the mirror the crawler actually reads carries the same +// article. Its hrefs are relative, so they resolve against whichever host the +// source names, and the filenames are identical either way. +func fixture(t *testing.T, id string) *gzip.Reader { + t.Helper() + f, err := os.Open(filepath.Join("testdata", "article-"+id+".html.gz")) if err != nil { t.Fatal(err) } - if len(files) != 63 { - t.Errorf("got %d provinces, want 63", len(files)) - } - - // A duplicate Dest would silently overwrite one province with another and - // leave the dataset one file short, with no error anywhere. - seenDest := map[string]string{} - seenURL := map[string]string{} - for _, f := range files { - if prev, dup := seenDest[f.Dest]; dup { - t.Errorf("%q and %q both write to %s", prev, f.Name, f.Dest) - } - seenDest[f.Dest] = f.Name - if prev, dup := seenURL[f.URL]; dup { - t.Errorf("%q and %q share a URL: %s", prev, f.Name, f.URL) - } - seenURL[f.URL] = f.Name - - if !strings.HasPrefix(f.URL, "https://") { - t.Errorf("%s: URL is not https: %s", f.Name, f.URL) - } - if !strings.HasSuffix(f.Dest, ".xls") { - t.Errorf("%s: Dest %q does not end in .xls", f.Name, f.Dest) - } + t.Cleanup(func() { f.Close() }) + gz, err := gzip.NewReader(f) + if err != nil { + t.Fatal(err) } + t.Cleanup(func() { gz.Close() }) + return gz } -// TestMatchesFilesOnDisk is the guard that matters. +// resolveFixture runs a source's full pipeline over its saved article. +func resolveFixture(t *testing.T, src Source) []File { + t.Helper() + links, err := article.Extract(src.Article, fixture(t, src.ID), src.Exts...) + if err != nil { + t.Fatal(err) + } + files, err := src.Resolve(links) + if err != nil { + t.Fatal(err) + } + return files +} + +// TestReproducesFilesOnDisk is the guard that matters. // // go-parser sorts its inputs and inserts last-wins, so filenames decide which -// row survives a duplicate exam number. If a re-crawl started naming files -// differently, the rebuilt database could differ in content while still passing -// the row-count guard in build-db.js. The committed data// is the oracle: -// whatever a source would write must match it exactly, in both directions. +// row survives a duplicate exam number. If extraction or naming drifted, a +// re-crawl could rebuild a database with the same row count and different +// content, which the row-count guard in build-db.js would not catch. // -// For 2016 this is also the only evidence that the recovered link list is the -// right one — the host that served those files is gone, so the list cannot be -// checked by fetching it. -func TestMatchesFilesOnDisk(t *testing.T) { +// The committed data// is the oracle: reading the source article and +// applying the source's naming rule must reproduce it exactly, both directions. +func TestReproducesFilesOnDisk(t *testing.T) { for _, src := range All() { t.Run(src.ID, func(t *testing.T) { dir := filepath.Join("..", "..", "..", "data", src.ID) @@ -74,7 +67,6 @@ func TestMatchesFilesOnDisk(t *testing.T) { if err != nil { t.Fatal(err) } - onDisk := map[string]bool{} for _, e := range entries { if !e.IsDir() { @@ -82,11 +74,7 @@ func TestMatchesFilesOnDisk(t *testing.T) { } } - files, err := src.Files() - if err != nil { - t.Fatal(err) - } - for _, f := range files { + for _, f := range resolveFixture(t, src) { if !onDisk[f.Dest] { t.Errorf("crawler would write %q, which is not in %s — "+ "a renamed input can change which duplicate row survives", f.Dest, dir) @@ -94,29 +82,106 @@ func TestMatchesFilesOnDisk(t *testing.T) { delete(onDisk, f.Dest) } for name := range onDisk { - t.Errorf("%s/%s exists but no source produces it", dir, name) + t.Errorf("%s/%s exists but the article produces no link for it", dir, name) } }) } } +// TestResolvedURLsAreAbsolute: 2016's article uses relative hrefs, so a +// mis-resolved base would produce URLs that cannot be fetched at all. +func TestResolvedURLsAreAbsolute(t *testing.T) { + for _, src := range All() { + t.Run(src.ID, func(t *testing.T) { + for _, f := range resolveFixture(t, src) { + if !strings.HasPrefix(f.URL, "https://") { + t.Errorf("%s: %q is not an absolute https URL", f.Dest, f.URL) + } + } + }) + } +} + +func TestExtensionsMatchWhatIsExpected(t *testing.T) { + counts := map[string]map[string]int{} + for _, src := range All() { + c := map[string]int{} + for _, f := range resolveFixture(t, src) { + c[filepath.Ext(f.Dest)]++ + } + counts[src.ID] = c + } + if got := counts["2016"]; got[".xls"] != 4 || got[".xlsx"] != 115 { + t.Errorf("2016: %d .xls and %d .xlsx, want 4 and 115", got[".xls"], got[".xlsx"]) + } + if got := counts["2017"]; got[".xls"] != 63 { + t.Errorf("2017: %d .xls, want 63", got[".xls"]) + } +} + +// TestResolveRejectsShortPage: a page that has changed shape must stop the +// crawl, not quietly produce a partial dataset. +func TestResolveRejectsShortPage(t *testing.T) { + src := source2017 + _, err := src.Resolve([]article.Link{{URL: "https://x/a.xls", Text: "An Giang", File: "a.xls"}}) + if err == nil { + t.Fatal("expected an error for a short link list") + } + if !strings.Contains(err.Error(), "expected 63") { + t.Errorf("error should name the expected count, got: %v", err) + } +} + +// TestResolveRejectsCollidingNames: two links naming the same local file would +// cost the dataset a file with no error anywhere downstream. +func TestResolveRejectsCollidingNames(t *testing.T) { + src := Source{ + ID: "t", WantFiles: 2, + Dest: func(article.Link) (string, error) { return "same.xls", nil }, + } + _, err := src.Resolve([]article.Link{ + {URL: "https://x/a.xls", File: "a.xls"}, + {URL: "https://x/b.xls", File: "b.xls"}, + }) + if err == nil || !strings.Contains(err.Error(), "same.xls") { + t.Errorf("expected a collision error naming the file, got: %v", err) + } +} + +func TestSlug(t *testing.T) { + // The article writes province names with diacritics; the committed + // filenames are ASCII. These are the awkward ones. + for in, want := range map[string]string{ + "An Giang": "an-giang", + "Bạc Liêu": "bac-lieu", + "Bắc Kạn": "bac-kan", + "Bà Rịa -Vũng Tàu": "ba-ria-vung-tau", + "Thừa Thiên - Huế": "thua-thien-hue", + "Đắk Lắk": "dak-lak", + } { + if got := slug(in); got != want { + t.Errorf("slug(%q) = %q, want %q", in, got, want) + } + } +} + func TestLookup(t *testing.T) { for _, s := range All() { got, err := Lookup(s.ID) if err != nil { t.Errorf("Lookup(%q): %v", s.ID, err) } - if got.ID != s.ID || got.Summary == "" { - t.Errorf("Lookup(%q) returned %+v", s.ID, got) + if got.ID != s.ID || got.Article == "" || got.WantFiles == 0 { + t.Errorf("Lookup(%q) returned an incomplete source: %+v", s.ID, got) } } if _, err := Lookup("nope"); err == nil { - t.Error("Lookup of an unknown source must fail") + t.Error("Lookup of an unknown dataset must fail") } } // TestIDsAreDatasetIDs: the ID doubles as the directory under data/, so a typo -// would send a crawl into a new directory the parser never reads. +// would send a crawl into a directory the parser never reads. func TestIDsAreDatasetIDs(t *testing.T) { for _, s := range All() { dir := filepath.Join("..", "..", "..", "data", s.ID) @@ -125,32 +190,3 @@ func TestIDsAreDatasetIDs(t *testing.T) { } } } - -// TestSource2016IsComplete: 4 .xls + 115 .xlsx, matching what the source -// article published and what data/2016/ holds. -func TestSource2016IsComplete(t *testing.T) { - files, err := source2016.Files() - if err != nil { - t.Fatal(err) - } - if len(files) != 119 { - t.Errorf("got %d clusters, want 119", len(files)) - } - var xls, xlsx int - for _, f := range files { - switch filepath.Ext(f.Dest) { - case ".xls": - xls++ - case ".xlsx": - xlsx++ - default: - t.Errorf("%s: unexpected extension in %q", f.Name, f.Dest) - } - if !strings.HasPrefix(f.URL, baseURL+uploadPath) { - t.Errorf("%s: URL does not use the configured base: %s", f.Name, f.URL) - } - } - if xls != 4 || xlsx != 115 { - t.Errorf("got %d .xls and %d .xlsx, want 4 and 115", xls, xlsx) - } -} diff --git a/crawler/internal/sources/testdata/article-2016.html.gz b/crawler/internal/sources/testdata/article-2016.html.gz new file mode 100644 index 0000000000000000000000000000000000000000..3c60dc06258a096fc47c279ca032c9e8f2914641 GIT binary patch literal 26224 zcmV)3K+C@$iwFqc)_rOM17UJ>E@*UZYyjL`TaO&ab$;JpQ5wbJ5@yXz&+N|b>}3>k zsgR85H6=|H2SGs8bk|Imwr}doUbNDKqX!2F3`mF##7T^h76e0q70Yp6vio4bL-2ph zKgl^&eW~ubaF^Oy#uTWXnNzn@=lWH3SCN&SH}3v$um87?-X?bJI=vnIOC08nc58>C z2ERG8dMkvi?8G8=cyDj~Ki-H8Cg)#}XJ3fP`DZSPZ9)3?ANAk8M_Nszvqk#;_=$&g zenE^@lbDn9uT-sPUyQ#b=J@xdUa#*oREM%}m&c5F%;h*LKMMRXt`XDsV(!JewIgB0 z_O8Vbg~_S>af=8~#DY0AGMU47je7GIaoMqO6IUlC5f9~WHULty){w*HBieBsQ2tRO z4tHxmqJMpx?)h%OVgZ%4;UB!c%dHWw^=7o%QV1p`*1# z|JZ6YW^@29@6k9hvrif&^gwX00c}Tg;8RO*mx9va{~-QoMA0;8>@yDw!7QkmsB()@ zQ&q2{n<+6k!aE=#cR*##^F0B6Un6$Nhp&dGIl>k$8}W;YD+wH*S&gUxV{SH$W~*WG zO}5eLt~WbI+uGP1@J`1ZbPQv|V1qTY+cDNUo84}!yVebCKL-9NQVgTH(cNryTJ^vi zfvhx1Np2oTp?LvGEsLF!x5=zfBVds(iVYZo3;wkUm<42Ls)rfl` z46k^0xg`=;^ToW`p`ah^>+OX15G0zQ31njw(@NcRD@X>8h-_{>js-fU*4|<*lG|#CVACrDf7yM%se96`$)8;O#-`%U& zdI0gO?uVm>Wj46OaS~C(s2hzb=I-v*`%AGq%%d{~dtkvBVkk1M$K8KkT%-)F^p^Y&y$p ztUq-XV#G}Wy>xTs){Wai(Ae`sSVn+88#2X3jUj{egYVVhPc4lHfE!~oi3!qc1mjtn z5F5u-(G*NmovU&Q?ae64`iM^)4w~n27F4hhqQ_dd-Dr24-BzPzG>wL=SBIjv+O?PI zv>jRjq=%En5Y$B35d>kE)3y3Wy(tIO$#qwFI8;psnLBS#N>t1xi1$d$Mr3saf9oGc zHwl#a{>b6vw#S?kNJ9~!6bw!o2w~BQXA*!3D*Qwce~ylh>Lb|(`f!;CVhii0?>4Z{ z#)r{c!rFx;Rjbu)ZEUvNw7m|?$%g7L@3^;eBs|MMszXj<57`M1$u5~9eewzUyT9LF z877{Iwn}h>)tjd)d5wCIMD{8RM=}FNH@81sk#%l-7~QxSnhVFv}9&Xsa0LE>azk$4ApwKOE|0Gx}#6VuKIF~FM`okU?E$@&ygJnS>%f>H7yrOj{c38_iB@ zZD25?*|i4duw!-FYt827pk=ff#GvlLvbvoCtnj<6yRp_9wE23sYZ`n2ZMaceIHZ*D zUxJn&SOu>On<$^GRBxc6zz28?h$eA*senLe=&HR;T)Esx#R@S%ID9xGc(k+Nu~zF% zg^soPL9p;>75=E>L%gMjF)959oCJs!R}a351!plDY*=K-kJbQBXO#q1nX);`(d{uG*5koe_tkPSlYeAmWGfAGV z3}N9NZxQ^ly&`u&w+KsOe|zQAIc-S2DyyC&PFK>jC97@q0TpEqC~1qB z011G9z4*umkEWmv6Z-Z1Be1pYl>tOS9^!A0Z>B?W=8lSDY?3kwVUta@$0EApt zNR=63=NFfBJxg{J8$g#mMotSspco9qAr*-m^Yke*Xl`Hf*Bk>&GcbjfhVMJ14#{-@ zsVybvzcJ^^iEi2cArDViU=%;xdUH4&zEof336oTYdZ*yF2cytWJZtNag{zbfLgraC z2m#55CpU4_okr?kGlEwZ-L$`|a-~rvT&-USvnfcX46T+y$5bomWJO+cAkRTkuBAN8 z9<+0zK{FpOVMU3sBr-H;(M?JzfFhYfibGVR!%k{>))egBxX^4; z56Enz{8R2hL0ZYBw^9a3D>HgJN{3P5w+gT!bH^%wlthY<7(KGcE2)~QmDGcR1-0gA zy2=cAyiZbQsQjT#vQoDQ5m9kfD|am*;c3&HEtdh4xHWT~sbF0OI7)7E#$A@{bR|5cIQ!t9BNPL$2o}8SZA9Yh9bbX`21g z28!P|QwJe+sX|q+$Qzp~n(Ggb77vt~{H!@>(UT;hvwEYdU1P&&wTwo){#TFg-MC30 zNz0rFS(1i4;4IerNb{PB*x zTBUYs4uI<%7DV_65w~Gevc!9mo{@n!9`DxP97HtjAgh5-Ddg&zAH%ZP;<$Il9B7G* z;7U2)Lkp?zIeYUo0c>595iAGhu)j3oD)uc$oRn<|zJRg^2|L@INvNjsQ1_ukZw ztfBT)t6P}bYDmFzA~!U1G=#Y@+vm)&pIiVZXpkmDHqyrkIlAk^B59-^$V_i;C>*$8 z_bg>XZW2}4A*8e)`$CRJ!pFyvf+sm|_r&D<-vsH~@<9)bDwR;CW_u}l_myy!_5UF| zWNJHqYdhJET;ywcCnL;7` z@cFls^It-Rnc^wRwP`HW+ND{Id};wU^{HtmX3pmO&!5UicK=M`$@$NvfqiXeJkBg! z8iTRTJR9<#+(}QZ-X1^kY|%?5qa5~1LefaDsgai~r(xPc zzM8-^W8brddR^k#=aaK93nQ=6ugAC>08}U}mt9&4Q+FU^+!N!cE-449Dt&ndo=Fhj z?y!9374nnM)1Q5g@15kb(`ZX)3Av%l0X@MB#?Nup8KV zyp?YF!L!di^2i?lcVU86`qD)SpU0SzMVYl9jlU%K!4*V_v7CB+qJz&9j*sHJGK28v zcVyqzRa~WChwNf>4%-i=nEN?+1BmL&pvqldaE;L=wlUdb&NorjP{1JI!dH_; zC$`hnPT~+`X)1-|i{`|goc(Y5u8nM_MWlK=3L>A!FBp%|xpbypUE7T}WA) zN-#>v7@xDTj~1!|dB%rE`YuAc9C}FKjWdJZ#c#Y zjr%2y&+xv^ZWl@=BK=ih?mo_z&uBq!i~ z%BXYpW|l68#q?wfO^>FO%tszBc=w)eK`Ch|yd3ZtBL%9l6zF5+_^2yA)@$TzbTZt; z$33soKu;3V!7G+3w-}jMyEI&pJhDHt?*iTWK}-WIjge5_&6<~GIco{!v%jP zOQ^%l5=I~YDj^O62nopxSX?5%p6V*Z(J-9>q&*G=Obb~+g|FauV7sh{RTc6T2Znj< zMLHNn7NW6+1arbQaf3nL8`-?X(IIYF0(El!6P+)Od`;ZNHpeBGU)Cfr@EMV0U6DEU zvJv%)qbw63P)wT*lhTncyL1mvbuvtKz zgvDji4Wwj(d4iiReyzgY1;o_|Y#-NAKyq@?J+g79dLZLmYE%zP!kb&+ik)WvSJd_VH2pWv`(nh6Vc~&GW?o-nd#S_F2kVN*7lo?M>SsEV}PcLQ& zDATv9C+9zw`v#Oe`y7Mk_{Z9A^^C%$F(^Z)#+nL{8ap{*X)Ji=flZ@>gMx|uy?5@z z?B9Rqo&EeFUO{>lD|NYHTf*mRN45;kb=qsX#LMK+P@9@nso<`dIYTm4`qI`dIe!2U zaNaIHd7!QaVF6~H{n{fgc$o_pg(=O)hv_9CHmAn2*v2;oeEEvE_O;j+k`MV)Vzv{c z(XP!QO17%KlZuy#d|f;k$Ox~I3uM=4L?f%7pTZjXhw}QKkWn*9@@smkd;Hn$ROS8s zlzOEWbrCL8d^MWFd%x3ARY6)lN_G%FXd&dGd|g8i9tGus+?1~fm_gcUcq~6QP`TEl zdBNkFv3lr)QL^4t5Z<?VTtIdK8$059I|{Gqk#z5yyrx zuCQMR&o#}NQb(K8_w^|))Xoby%ee?b;l-&DuHIN@YlCL9X|T;flXW`lEo;5Q&CNB* zS{i*Gc0iRHoYRFK-OD&Qf) zA@X$Y<8>mh?g{)L2`JujRvkW_=bN&Ts;3T0Ny6jQadQ6OMye)(h?Hq{Uo~$9Xf=E&x!ssqwHJfzpUgGu6+CPf!+e%Gy|F8#xq2 z=F`YN^VZnjm0k=B+#6-yB56g)xW~;nM>LpMjP-ffNDw!s>oxrWOBjI_2@qS9m+J9N%YXkn0=WjkWd7PJ6w%*_XX`C$dM%C?O%X=FIGx=3vsKVybQ1$B?koP(B!mBB@7#LiIIaWyEdPpz z07Yc7JF5GIQW%JoZ5omY2_lH-$*Al0gu64t+3wXYf;`xM2m(I@h7kmT5r`BaC{84p zF>DB0LNMTQ_&@faq^f#)Zc9=cYiDN1C_yCW+Fess=c{w-Tx#^7JGP%LxRJ4P!AuM9 z-n!dS`)*iWc{P~455p2#K7Fi$NY2#ON=99B8ULcU_v+3+elMzU*NRu&aM&Nd`n`X; zIyM1kb9~2}0C8@Frrj$JhP!l<^E>zHWN@~{=H7Ywcp*OVd}WaZS=Pg@^Tk9(NEJO2 zA;8PmzoFdfWhk|JF9kT=PXSKh=AGhZ*^dGBzv`x|{%5znR65sTvB*lR+^gDl`#~qa z{ziHfJ8%5&`C>3Lz^CU6-i%j?G4`L)uhCwft_9=8inJe;z(HQ(b2EwrnRX)o`bXfSJnBST6qpM5#$0 zjd?4fB|FbnOU~Us{<=T?B6=|&nPSo@Eh)pL9)&x;n{Jajp$fkz<$R6589FNVL26vP z_Znqd55IAmEKHH6TA43K(5O(nylexDB+AyAsLrOUGO2#f?9>K31q<^ULOx?m-3-gkb0$usOUBHDNnGW^ za2E$*%!=M#KTWURUYHm!hsk<9{JH4srk(6fjGMNMYJd#W!Vt;`_FmgCg2GZNaClu| z0tiALt##a*-I6BF>f@LolAtgR;zBv*f@#!FG6cHg#0>uY!v_or^Q*7HKg!nQbMAv;(ChA(E}L4#_6 z@g$wC*T#l!A9hLS$fOcd(TeA}+Rpe(OR0NIWG0Df_PA5fB zkeY5M#s28)pf+B6#{NP^xf#`HKYv;^Ahe&Cz-0^7&QVgZwXXo5UQhOnw&lPmo!Jkr1M{J(OwLoOG-u18b=RX zp_$Q1PU7vp2%?f~hyCa{AEs9HqGj@vI*i2!y=mEDorIvdX;V&R ze1*O4=yZ@qd!d!j9otrB>S(=%<>|>z+IjfW#5MHEKUYQUoc+;h6rj`Jq-?#@&+6G_ zYmH-n*et@|eF8N54^tcG*d5FeYp&;PB7Pd@Pfo7I*oN4K>svPCuUuT;-I?^7PAldQ zXhOu#0&5bD8?`2JJMD{Cvm~1I-ya{$dcwj)`ndaoIXh-gbPGjoAbhM>s(OA(h8zD+ z+U<&>+wQ!Ln>Jn|=2fboa7}*7ew4A%*E{D%{{PnA#of+(qyIXXOr4#{7H6}*Ux^`+(P+F{ z9M{SD5>&p2Cp$CpeGo@teBq#;4Astf@}?q?WvsLnbG*JdC~j86{o-$mAM6x+qu&=* zn$eTXL{s5QGqZ+Tl@yOS#ipux6^||IrTQ$*mgeX*EuS;%NQ}sx>~3CN^PpeYc)d|y zgq;`rDJ7>C9DF;;3ly%epCfS<57-`?;K^MZcP@VP?gw|`>*MVP2q~ic8#esf9$##G zbLmO_+-sBi!;`n-Asf?Qer`O*PY9j4TzD^x?O}hvYuk-OkUR?E{o^=z>3ZYqhnLzx ztkO|%-@W^rEuw08QI`tWvM;`t=t%a={(WA5HJc*e*!zEGPsS#n6FShLTtIY@VwY}o zfs1g`V1uQzCA zW`S8!V7}2I7c*4U-dSGMI(L=TdgIQ^%HprOK1sq&H3%mf z$c?0=#Ij<=t)SPXa$9lVU0uJ@D2Cz5cj%&L`7tpuoO4N#Q9=>xB=jh{T#RwhFl|e% zOc`5L_L4_M0!Snzf_0Af505Wj&qh1@&e`dMAH`(QpMDto?rNstWuPI-G+ZAIpCRf> zD{O?P$PkCEf*vC6F>_u6N3>OUVs>#fLZfT5qYekhTf@x{o<2@t*bfiZ#ZAsUlt2n0 zpp%A6&zLgALrqZND5bVMtOun!2sV1Z+kz7kf&k9o%-~##12}_oqu`Xzc?i9B#!_Z1 zQJOdltpc1%qbYx@IjL&9ka?%9&z^i)?4Lb()GhwHxcT&9s#8y!N{Lj?g6%FNKr(4~+-GDFvHiX#L%2*Xa0 zp71Zfgl2eC6=Ymt>Fu*8UloBi$)Wgu9fQMCoX3mcSX^m7m?DFkDip`iTc`pnxnss)Z2}F6w6@$= z2eBz`(2x>K3;>rPlQNTXDZyu%lxx>4KlYw^?5IP)Y0HTfgcC?1)*N83 zZFt~q8GilV{_}KBgc;@;ml>DK;WFp=Y+!ftsCS8--gEgqxA&q)J9@15R!cvRLMx z>42k*C>NeOOnA@)8Ie{AEu^v%*wBTlYHJE1fH{{uo5-1fOA;_=-)z7c)0~6Sv8r44 zO+5W(sQO#S!#hzMyfYj<=&zQ92v|*>QpQOX&zAAw_3#&kfrrj97cAhW2ix63BpD!C z8O$hL5`{UJX8kDa)y8dM4&vXBJ~>Gz7sb$=H5E+L)be;zp3>5@pdCWYJ+qKOuQc;P zAv{(mwr*`=0%GF3$e!nn!6h-6vtKslG>MMqHmUHM00L}f8qDUrCI%U=lt+Sl4IPIF z1P(e%DCbZE#BEcDw{M>1WCobg@X^f6%**9?nKN59XF&&GUS7L-Qy9cnDdrr;-p1H% zgAIhpK%spwoei?*;YR>s!Klvg%kW#WBzaC~*^mR}s4`pPHolWi!z|+=VJmi-Fe9lE z)C4_FD!@E)5_xZw@xprqL!YUxDZ9OL>^3HZ1Br5N$4tE?sh2ZY)=#}44k@UxuJks# zN>0{M44f0kj0M(kp|IE78c&&Wlp}$))H1qCPP8(u)`tNCG;a}R{w>MBoSd@R;T31Py#>U72!?7mbhlf5EFwk5p=p>U|0n`-z-m2qc-tP9wqB=YMLtBE@jy_qj zFIsp>gh7g`fJuc7^%h{woC;hNT*Z)_(5TAYulvTTp6?z{)XPBB9KM*jxhywxLdyoi z7t28vC3vr^I`w6COg{&VKkTZ0P%u;Fhr4H|U-Tj+S6y$BLamXP0fy33Mh&tYPzIQ{ zlu0VIEkk$qt1{@4N2*t~br-ROfD@KA%1p*3$(Zw7Hez_gfWlgBOO1!bz}s!%u)Fb} zs<^EVR?I@>tRs>_$CVcbIv7xBIN>T*fDp0b!=ysk(IjG+&FZFa}Z)mXgs^l|v;hbs@~_JVpqg085w6@*Z(1y)87#*t7+vFMr^mr;y9>q$^I z0*rBxPbtWJT$YbHqh(VWfI;Y4$Cl;f#FoE%^KfwdX6UE4T;)u4v){khbsyA&L4R2N zExh^4_u(ryZoKj){i_##ZhIntqS;-%SB`$W>HwhdV-%EJNywD&ES43E6~~B!*kGeb zJnf)6C}-k-AQw0nq0Ka1k_|Yg$ZXb;!-Yp0cu0`+Kq!iXbs3uAArlZX!>seF-mju7 zXL|z5TW9z|d~vm;)QD2#5HQkvq7mkZ5bL!MuW_LWkLiTb;e9YU83qZ?5X+%K8I?<- zGUwH-S@ArKnrF-dKJ50&PO0i{t9Um0=+*%7=KO&N&#uXH(Y|ejaR8u_*eU3Mu?{hp zb|o1zN+?@Sk2PpwC&zX-7;&Zy!`@?|F@%(BZOenc z?J85MYT~z+9Fu%-YDVFbD9o8N8>`G%+ATae`s~Rso7bNFdc~PB+6v0UPX@+2i3~J| z1S-l{bq5d{D=+RQSBkAo3qgW7gc*Vvf=fa$=f!OFOuwUG($&50*1_;@chz<=BNj1C zowpb$U_mg2JvSKWpzTq_pe>K4<}g4QfLu)?<8N8~<@A;d$KSUwhEw^2+HD8*Zq0@) z?xq)R)n})FSaAgjjXa{#c}k52!TePOGmAZ>mLOw-3V2u#%5-9B=*dY1MhpeYK{XkJ zOJXqRvTQ&LoH~l@L$fU`TsWCvEi4q=d*v~dTq^`TkWhLl6wugXEFqaah+qms!BITO zG{Z2%a5)&}e3tdYPz?r$opN$`)ixa0a%-jl#QIPuA+w5WY>1`8cyFM!NLs^{P%55e z-xPqDCn06AGHoVdCgE}<%$X}2IKjSlGaG_2;Y96UPLJ`tcVzYK3u<3<2m?$63#KlW zKCq2jZ!9w$1_;h6e-c5Mk(ZIToB)%YUb5~D!P@pN>w#0ZqI|~C=hNS>I)P3Tou?ueL=u9t8#q&O zNh;=4khN=l&Tb0{&$)lsMnX!oHeN7lU@&y!ejh{1L&|ZWAm(hw2|Uk1iWsHY51cu; zBnNYL$ojRz`_n_0$LFkUPZ;3)^+|ENeEM+3HM=;_UKoosb72Tk&<~}f6cQOMV-)OV zTi&lvdS$=Y@=tJqa~w?O-?IG6DIx#Q-ue7Sa$I-(Ul|B+ZrrWkKMW~`R%?icC_;j0 zBR&zY>QzlomYktDGb`;eryPs`F>=Z&yAlFJi3LSLE?FJ|WCZ;m{7+IfoSB`KY}l=+ zd)UT1>~50)iQby7Pk&#%_r9-h9eBBngQ)}jn>u$CZ*RV?PBD|)o4=cm&aRi8hr`K- zz`a0iVjZg|f=nzGMTkyF6CF06!ZR$laXCrTMZKSGQM4}6l_O9>IpeSQFIe8 zmyn0T!aHZ{lhtQs7k_a4&ih@PW`UJnI|a@eu9+wyF!E@m6&X$P*i=L73%Fcdm1W#I z`)jSGApQ~mTJn#YLJl`#beS)PaxSf(V!2&yaVgG1a#7ZF^O9lWR8kw z+z!v1j+fs#9Av_3rUcmu5*Axxk(xyMmW7LJSa%687So~V!m}D9pN^N$Iydqc+@z$E zbQ(h}k+3y3BWFqy3^sNTBA@X#w9Ly2c4%$ zqK?9O?Q=vHN@(m6eMT)E2CY=J&>t_JE?&!#aLt^cp>~A8mI$OilD-jG^TUfFdS<=% z%@?~VpM6kJbLFg!JmwTI`Y43v*=QbgW+K|OH{o*+h0JK*m`Z@lrQ(?-?~*WEkF3|?U?DtH5WxV6a)F_D<~XETwyN;-&>|Jb46WZ4XrggvlN zIUB8THAhmXDOn+c78Wkx1bOrJiu zYOd~m-niCJzq|U+j+;l&YT-o4nSnwVkt@NY%)zQ`osSZmIZVb|Lm9@LvD#3X41upD z_^5N_AoJx);5HQbSLX@Wxx#17>cv&J(38&Ph`H88yOd2z$&^PP1Td~GS4H+=k$sB> zuD0{!w9-=3RAItiOYBi|$YFBAFdp7q^|z00%D5o_AxCXZIjTR_WD!@}Y8Bh+5RDpAorH#U3bMd=m&kJT(-1l%XF9SU2vxoWBH(yMf(aEgynMzS) zbjlH-9406F3ax^Y++!@3?olKjH*{igOQA|mqBvG;@C8_!2P;RMxHUw-I*HFQ?8yLb3x6}np&}#`j>iXyndVgS> zGv~o`4jF@zfulp7nH7r3067DkUBgjaE{`fGj~44O8S}E~AHe+T)B8VqY4zE3)U2M@ z!s#C3X=lKA5mLxLuwbKhfWbR!f)6R9uG5xaTI*-_9R-C@N*l7&B5<|@Cv}ZbVx>i5 z1F8?8=73K|Iy&M$-OFM z#-C)=(fkVHt3~)@#8Z0j*Zn}Qy}5n_eyleky;Q&VQr+C+kNSpqfA#rjUY_Q^8U*08iTj#inQbre?25XV!D7?;KP=v%ci@2qqQ&vbu^dtJUqMs!C z^-aI|EW#}8@$X*K?j4xb?C{rJ;|?(x$+MFZvhzwXmNI7%Rh`A2GTW@#4tLgCL$-p1 zyLPx!guA|RcL9^wz%Kp!G+m2V?Iu54zE%GB@m>?~1SX?*n$>3=SH^$@;LLEQGB3K% zC}A)KfjVgfopG~{TL5~km?Z!b0NVl35CD4v;2)~_@G3uoeKVhn<*3n-dQA_{F)&=904n8lm{mDIwB7~Hx+<>Td^L#ZgA3?mQ_2-<E9W6gOp3kNa z#og+FW>NBex8UR3f6wUKJV&v01rY=;)Mo zggwGuJM4MF-XTLrgWB-wN0%I!B)n23%JJ4l0O!4nxEDI!Yb-vxefd=!y;tGbc?MO< zP-4KuS(Zfuir@`nAu1nHW)~nnfh`p!v}BssPM7l`9&7MDh{ehKrXD&+NZz-x7H(o z6a|Q3n;(#p7bU1QF_D=?$2JbcU>T%%c1iBT1DtMO> ztt{cLGvJkxCMo1vdX*#h>*YGyG*U|Lxgc{s;$1u5QR#5+yt`_ai)OZsdhfp46x03f z{OVp~?}KLbvJR`f`G<}}U}bcSpuO@MBl;5m=A!D7Kw0}J(%qtM!(YY9!zc)SguYhj zqcY+jTncjM&t7Q|JF5gyiN%-_FF~GBnoATC4Cd@j&Nlm4orwzv;;tafD#u9x5e3>& zKobQHc?hO;9mplLkjRw5Xay_5ja3XvypxSNyN2obT(H=j|D2ycdUXD_z5fSdA1-fS zem)(KPCNH-2g!p}$^=LnxQ&`ysT{f}UDUy*b*0>N=i$!ltZz<}ZHbaCDM=j%2c@JE z+E|uM=EwriPC2cOkX-rzN=G}evXMT9ep_IXr=lQjbV})(Qe}cO&p{`nRe-qHdGBOB zRFf1#5*OQXk=hA*KRzovw+fm;RE^x>u6r416DuzUlY@606|*sC4A!HqvmYVF&6@2* zpi|`!!H!_p3U<^oaR8I02`*;?aTz%-QSYSs4}be&I_@?>PzV=24`_gA4XKdMbDK(V zo0ajwv+cTVv!g~aMonTwyq3hHZi&NKXfCJYp%exmfBPTP@wx=ATjCYADJ{6-+Q-18 z)Im6D0C-d(2o0Mx+XBx@!5GD}1iV)JS5PCwLC^W5Wx>mAVn!)S>{hNVFLw&wh8U6? z%sJojZ1sHfX!ZN917FQFIBPH?Pap*so~QE13rHb(othbhUEu4=nzHf?5EY4vt*A(i z5(fx;r*JtO7j?MomUGE?8KTEzEW{G|8YV@o)4D~8gHyuwa_w3ll+l8kV2N_=C`YXi zy;H7iS{zC)qcE;)+1s1nbgk(ygOO1O}qW`i?*d*9XV?_(T-@> zigwiaaFCFIv##-$3v~*xFbPl4G%16C!;Sbt&~hg=fVT!QAe-^4awCCjvWN*5x;d>Q%;j0 z&<+CB@bJSOaGN?M_jtB?(HR7UbbyijG$Gr{m`K=I=*uzi#dLW0!DcE<*M1*2(kikuiWML&RL9jY|A( zy>=0gDxWHy@(}0RagOTqd*@u)G!N_pA^iibf9jZH4+qh!*~2N1k5;m%WXMV~&n?%Y zHfri3?6eHrubVZ)-qE(SHP1)TBj~k)9@Xg|{8Mf=TrJhoZQHwfJuO;=UR=;2B&DSR0OIS3MG;Nk?AQ`>e zn}3J$&!l?R9vv47{!hoLy#N?(%pkN2+5;M!L@+8_E`!&3e+hSGs@f?@{h36-RvTJU z@&4g--%zSmch*FmSFA>Bbd-X7X06P%q#>dcOn^?18E#i?tNJu67o_@#dF_}-rTGVB z-WY}z?4FKSpLaZ@AUTGS35QzNa;}twveI*4MoUad?-p*yJugbUOWY&wwc;L?a`n!= zC*_wfhoSP`n@vY&%Z?RZjM`%K&Oi)WXy#0CCOM5+E3Z7>t=E=wrbt0fC9A}_cATT8 zr2glz)S=7|Y94akn>2)REO&*uo9?8Jhq7>yj1Wd=sUt9E9We9?avkc54z}yItvB zbpbgKk@?I7T4z(rB67W1wJq*kK_CqkCET^d9ra!ucAQ}H!J=akgh!5o1p&z@7Xv_4 zMm*j#&^@0|;WGV1K0f`$>G9d|FW)}>`Qu-n&nH;ErOvNlobE1-yWh>a)tTh1S>C?< za@s9HxsFbwXB@0VR6dIU$#MXU7=m7pn5pH=ubOr(w}x`Q#L!kqP*lf!P=<;uJx@9c zpb8MsU=aS<2S5SL4wq|5v7L%va#qtmXFC0e@ZjWxltEHv}lN})7rmjhJm z-YPApyKIRQihD_iheXD9WTa~6-h=77aTg6-ZKLT=Qg}EB(HD(72vsE=i z5`xQG7y12v>|O0|+eVuI?!RJI?ZwX3lt_xAB->FpcC$%tU0<5GNw-B&42L6;5k<0m zQS1x)p~#0~u{d1wVR0xHhifhMVrJ7_M8vuOV;H+*ity!mC%`&OoGE}`_n*={q)o>%o+~k{s-|4__z5x1_ zgCC=Q;sloh?lwj*zV*S>`{~&QTY8pe+D)xtQnhYbO>V2E)@rs*i-O%}vASegVdpiB zre@5cZVq*)g1Y&hw%4&VCP*B&YH>e_F2@lr>!TNc2gr-~#UftQ(2PdA+2*_sNv72` zxS_%gH))-+rXe|&A)W!RsK$I3?;Pk8uJ|PP6v1MQ|Dj(^o7d@VBp0) zyP6Es%W+0uwEs1^?_TsqUbV$+Zq?1YX){Zw_^pkW-nLC0giUW6G>HW4W#K#BFzT(w zeC}cnd8dQC`F4-jvcD}3aGA;_vGVTi|CIsi9Xz?{ISbnAc3XH_L9^8~njDZ%*Enl8 z7;AH8wYoXia?CSZrlHO;Z;pAV!@T(#=-0Y_uge>aX2Z~Rv#qsFTkCdp!?K!s3q~`3 zRM{@Af&QgO`kpbe>F)nIz2Jq-Ew$0GwI&DHYZ#m{leH|~Xf-TmnY_U)$~>2+x%aCX zhSrz|`{uBCs!^`agMF{V`}HmO|B@VZ;Ewjb@X0zZVYyhqt2321OlsLJ)~MHwuE9Y| zH0^GkGQCcXMF+V?NZ?8d>D2T%HBYyZcKp5Cy! zdZSI-D!RY4WizIlP1-ayT{VnGyK7`jE9|>wy`|2#tZEBfqi(jey4E!#Dy^uq(| z_4(qxIr^PyFgNC#c2CE!jnUpWjBGjkUy~&I=`W+b|L}W6A?u^Pukd>ndruh&o&Ep8 zDS>0e8!oKC-ofyeCD~V z4-E*y-R)q*b3-ejk-fB7#T4xti^gK#x}cTzic&Oied*c3~P zn_MnSJVrGNBTUF4sbqOT2}K}MWbR~g1Sv9IZ&o_nqrE@k2nBc{2T$G6-je}gqrGn_ zvG%`q$okHYbKm6F;tQN(C2T#R`c0Cx-E(%PQcOu_mi2AGSHZ%JggKN zx=_L{s6VV<_Re%xr8AQu*$@1)Civc2s9IjY`k%xOjX3F$e|)@PyK(5z-4)kI^{R;c z)ddg&;2hLT>g)l)uE$pwKAR#?WoeuQ0|oW{Vm^j%YoP(rtNkkI^fGwse$#7vt&Az4=ip^)P`DMHl_*GMwABhJ>HcCmCcl*0`d;( zfY=V+4t6ET7WSor!%op3FW^Y*f|6Lt^yBf*ld2WDy!&_|vrlFDdSTGM!5AC!MCNKZ z?iMo7>WyZz`LZIQ!Qh+Ggg>5qYtr7^c?lUus|pOsa)FT&4rEVhqNhp}bwg78^?<{G zS}K`-O|u#9Tz!0Om4$(;L_7`yKL&XrIfpVSIKII`%D8=9;z&$VZ#oGqi5S$M3njS& zENCJ(=hGYQcDvjt$)$7h>b{DULcMfe1IfZIKD+mJ+rpv58410@LCum6yBJkp`vNMr6M#2dWPfF!Uy8NU(5eCLsdyz@?C9ZR%;BF8kw{3JmI%_d78X7F!IC))qKTDB1BkOVn$G;*?$(*wl5g#FBE$35ibfUzF z4o6p?2v7!Jnu{yxE974TV-6qCY7q4VzGP*gZR_ZTqhUx#LJ|t2AcQnvDh^#}3eAJl z3Bqg2@?RxR6+9-RW(TffQB2$A_!ma3Kt4II|G58@paBW^Z3o4zT@iKbt zYc)ro_Dl-PylX+v^f?9=mr40fVBL(Q%w+vY5crrpCS_BT{K_2w(kB%4!)FuJC$yHL z<#tDzJe^e>eR&-yR3bBY06M4YPz>;2wqOA&C~|URhlfhyq>4jb;1$T4jelUMC=Li; z-wC)9Lbf;o8ox@etCOO(K6>%rOitI39q?X5k~pKi?+DN%NwEL54}v0oP(WOAYqa;3 zOZxE0zj1QM+5ek3mJ4Gq{=t(0*e99^xhWZr_I_aGHXzJ}(w__LB@tLK=fFY+Np-;( z@g0CakjMV6Vy7Z4al%CTG!2AQD+5pfaXKldcSZDO|N8{wXa9SGL_l(IYXRx+|1ZYV ze}Ej9Y~Q-GozF7>EkL**eR6NK_kE@Fvq{XQ?UG2Fz`(;8Y0f}Ml)k>xPZUN&A%WAH zSb)<}C}tjwVJJzzz4hB|!a(doFs%xC8sk?8a!51c0^fk>zcNI*hJsLs%KUYfKNt8* zBJiK40Huhl0}$yUxRn98cmez{3Af_U5Es7MDEp8M;=xlTo-#b&OLs+V`wsw(gVElf zTojbgQ7rx)y!p}I^GrgJ-m*l{q3|5+{r^Jo+xv>4;End4q+&9r(m8L6bCJ9xA{kTW zApQnugiTI3L74|$PT#@hmxS&A6`}IQ_uw&}8>zVrj^iNG3$JBDjH8$|!DbjjJ%AgKZN`gQWDz|aDwU%ex$p}J?iJ$KR#oF{;vTEM2AzzS?*u@_+$jnq31ac#lT-z24JQG(c8sh z;XB|F{t;tzC>ZaI-9TT)HKP3&nDBe>6w6@|q>OP8G4G+PY8%81;16`?MNyth>Mso~ zLG7}%*5?>YrYNX25JyK5Xjz1wK`L?1O-#uR0YSYWR5Vp_M1lt(>-VSt&`J4x2PolO z8KB%ae2gN4lV>>1_Qc2OwSah|y+4J}W5mJKFMP5oSH$;({7i<~OespuO6Nk!&n5Me z=yi#q7V7m}h=o|n_}y4oQN3eu4?9=FCAtLXc#bZ9W! znuS!J6QQ~Mog!dzZfb~-dxvUNDeIi8J1<7|%SrU>9ld3nUG7{YYOcxDV)sT64aCCcDDXsfAb6KAC9iZ8MIjrA zsQDrp*)B+4tI20Yh->n-8cr|yJsi6=oOu$VcR+X?Q9u0!*NFpi7w<33B9I}3FqnWu zfE>T+!r;TH9W-eE6Sq*EfI>bAQ0d$%BJC%7Q9_TMV#(XtfX97HFDrN4bg>{=VV)aS zI`{GVWaUbm(nFpB43-VJQV&d$s>xmj2@`@aD-Jmoj?DhbVL4V0v!YIqoWSE4(&D9P zWEKMBVmy?BS(BN?lKCm&*7Tnp!a8XLt5j(HM1{)L$TCI;T4umYwFiJp2fn71NM%U` zn@T~?MMUf*D^u*^GI5XYPDK$X2*gS@Y9riobU0ExuoX=|%*T_30*jq)*}`*d73fpu z6im%BP%Yqk^fy?uuGcw{3)5Gp`I@)lFnAv$PgA1N1^61v&0%Pacn`y8a$V}-h9BK- zpJoqoKOH{?scuh1WEITFmO!dR+~i?fuyKM62TL`9S1lS5`BPx&+9oJA&1Lq1=XR#6bJCiOz4W3^pg3^q zbsNLLU!*u`fIE~XQ?qSL=TwOd%*R}Dg|qz~!&%%PNMC5cydpC*&$m97SZL|`<%Mt0 z@oSQ9QHycPIkQ5zr6uFsB^zQvQr7tF{+tUH=bbVqt!0szqWoJJ$Pjz=_P)Y(bw%tW zuh5YXoO2CJc7j;0Gwz%F2T#PBm*=>0^$vdTlX$6&Gs-F+`M^b0aol^jVtNbGzz`{V zu#|RDy(s1%{GP+7CK4bC21{O(a@x7pm_=@wtgM!i4kr`7D*&ZPIA1OZOR&^( z>0gxFJx)2Ut0aak}TeL+%#|h(WoFvemm1h-C8Kc=0{F+l%Osufl7glRr=jG8YBQqY@K5 zGv}GRLR4MF^Z6SswUs?`V?bSR<%(~`;VSON#{m1U51pnEN|wmuCBW}{9+Kn=r3|-! z;<~^5IsssJg=c4hN9j(SNP!e+KIU%N=^YGbc!$oenLMCCRKVp)!aZ?R5m-Qj9-&?` zN#YPu5x%)r(NqvI>EK+e)LUR=Mr?fgkJeK4X06NmwJ>yfIUD8Nc1yN=vB_H;VY0CHD@FOp?Gi5`w{MX4#~Ak#P?|Na9@!y{lHJ-m z5Ci_xu=X&nxx7`{5vp`u@W~$0$akTaAI4aeBr-(}yCH^t2;#UppgYX=t1KAQFg)0Z z9d}r(SDRIRd|w^7{u$`Wnn90L4WO3Y&7$?IkGOBU-H#Pz%n0Cuc{CTmP5Z>LR#1RQlkWn-rA4TG++PM+#2tf2)DQdYC6puRqWAA^l54~Z znDC~mVH6|*pw?T9wc6raMY0pb2_!A9F5odHE|YJGgmLjNEiq$Gm|C72J17^PYv~HG zy`6FU?fPjr<93%GxxO7dst%puZQo6;X&OeRzFm%hgm%|4q92;Gc$@+c$RK-OwJ|57flI z>4cjP?mC+fwubNv^wgBzx|>)VA3D|t*PDx13oc4Dy+ad6v>%L*G;)*O?V!(n;G@=P zs-|vO8r9UcZLzLtn{`7~Tb8cXXKvqUu5EfEQ#ZqYc9-@Vrgl#j!&Euf1p4Z5&DRuCHLRdW&*d zqAAHwx!jAg%J#@8zuztt8k9&$lqiu)QnqD}zhE%uIqYe%mjz}H3-oRFu!jNmKju$1 zA~KUqQk3m_^zIz0-9S&^1K4X{{Ht$y5jS7 z<_eG*qoc#(^GGSLdneURGC71h_;s7Y%8TSzP1Tx1Hd4#YC<=-i%>5d5;G7=VL(r=m zsQnf-E-7>|#3TJHu5Hmx%i6kukB?3WBhN-4B%J^!9#};iONcH5A*7C?M4jn&OfqOR znW2PK5(Zp1rbj5zR8r8Th2x*EHFJE3NV2MAYsX~~MqB}Y0IP6$@y5}MtCy6TVHiZf zv=CsDKydBAiFN zV>^s!qJ3O1DC=<-cD84202X2Rn!|AJ0Xpy%wua9zdTR3y_<$N9Q1MWBvXEohG^;%# zZEawkt#<-r<3S}uY~LK(YMpQ5f!7RQZ#X$dfq*Lh<}42f5Sm;X!HCFU_P=b^)(;qjvdyN*Pf5@ zDU9Mdoi1jSb&_m(9gCK+8BjX%x?NYSYDnXa4eJ`BVRJXz)cpuFh6uX{A9hON5!O~R zck}ia1g%4_g_bkJlt&gQZk`69c*MV0~J`D!ArsBQ>3OZBSyrShr@d4(o0O&o`!>Dob5#66#X zL&M^K{?k-O7;Wt1w$q<62>Dl21#1`0=DKDFffsN#a7G9P3pFv*nxqv7Y-JK7fQqP*Hr3_)d#6^O)} z1)C!pD++RIMv;RtcMQaG7*n1j01OD~79|C}Unwbzsj9;eDlL(UmdIL`#VvCL8r!jr zZrqhr^8t~Dwz^h&x=cXn)1_iHDwPrIf>F;V=KzS`2o!tzMiz*49rSV(P2rU!;}!2E zU#V29m4%wVd$^;n6DTs&E>Qi03UPhX2c-hPBb87t!IVT+V;^8JSPK$eEPLpTdV&a%b9YmP8FiCCUa0(9~q z0&PE?&%BJta~%yt7$zv4W7l0*W)>9pojqT&>*Z2Gybdr}4n-5J1ddt<6Or^DPz)R7 zl{1;RuFcBQbedaJN)^xTm!KZDv5ZNUHisw z;N|qGctJd%$te-=k(+J;nvAjhfl=t1cTDi`8-C=79M6#?BlK+>$CcCM*tcSH{bIhmbKpk95Uggb2xnKRKTzM6nDZdZvcFU3iKU z9}e~!EbJ29G`5w;6D#n1==#yq{CJJp@~WbX(#Ribsnkl5mu;a)hsNa4IP}J$jXUKs zadzBe*|mXsA(cpdk)F?!g~8UjPyUqLdFhrF)wn%m4XD$Q#ClNVMqi@*#WiOeRIw$- z9LdEQW}a|4rD&;loFI%cMpz9M#TY8D)-mIle7=zu&%*kiAH$z{) zH(gI3+6z#<>x{yM&>DQVy{YZ@Ui(o#V`Z^ zB^~Cc7?N<`aT6KUyjYIyJGrz&_e=h41~Fn`9>ca*vShExz-#R6^!qRCpcA! zTbF4-$eNYnJmh`|MfpcFGiIjUdBdR zv|ajwYOCL&mo9%Y(6rbZ@Xa_cV!OsO5lrme6-he)0*O|4j%LTOi{Q-~TAa zJH~~daQO)B-IIg;VzT9KPj>3>e}G?q`BNo z$~gerurD_ek8^anYnT`N;q}SV!S$teAJcQ@4lsZJV#~PdzrWr4u-!Y@Y2IF)URC!` zcL)2Y*Y4GQmvCKGw}BSxraf|ZuN%$fz5A`Dz0lV;c) zy5sAe^F{b=5Z^A3J!)HZZ4BkcR%3S!Cg{=w57$%cDp$jF)O&Cfp+qMiAR{-JM0G64 zs;NUcfhqlo%PsPimH@5e0kNe$B3FBDqsE35i2 z|A1A%*u;>ofIXNHB4nH;*$xVGm&S=2 z3`g{q22R9wAV5+8JQI!~3TD7|1>fte(`Ps!ha0s`p^7%{LXWMkHOBnrK{ zfHrkq(OBSVvY#ihfnG=94PdNbDYhBJ-bx*}c8kyNzzIGa(u*r>HOFV@X3v*N@GOQ^ zg2u?Azn2Z8#ACnV6GwPVl#b@osaLERrHYYm3hMPqODM=jn34D?S&@N=Ysu`FebUUa z<#^g>Im9g|Q?{NllqY4x#lZPAhGz)=*rs>E$S6rRBQgKfJMnEB=I)<#m<5N$w9xZ8 zgFjj!K+d76lOam>Uv=$hi@66jCrvQ@nOXTB3nB$babmq@$^G#6|N1j%;OxFQbsCZG zjXiJE*g{ zEFRaIU4UNHdLo~_!<8FiCcFpJW-Ny2w!JRJIC5EEg#RMtlK@Il} zV7__eP9Fh}ZH`X6UcN8B$0t?nKA(ncQ)t*w_E4muip@K1wjeMXra6w`Z&Qv)7FLOK zPRf(Sd&Nxp#J(+eVXXyDJd8Af&hA{Ryr(2JrEST1N% z?(;@c4y1k_Aa+(k%PwfT%xZGwVhs!xK>sfoN)H9Rcn6d^U`fYuU6t7cK6}UMp#T$F zox)8qJ-KwW&U5&hNMRko_TG5d8iCY=1cn8*grYtlbl=Gzap$*dN8{Fxdv7(!b_UzM za(}jSJ~`Xj9+9u0e?>lmR_(ag+G6=&PuKQNZk<+bcW`~V+pw|RXtQ!P1bEfnnF%&~ zbL)8f%-D0j#dmRZaTk88|DwIKJwf;<7vx(YAIo*d|B_et4XI_a-TyMCG^iNv!7jBx*B#5~T-rlhmOd z9MTaMbs4lCM6~aLW?E9nOlo40y0TKiAaBbKHu3E)e$)pSRprChNkjcV7Roj<{iZ!U zcH4v9C1hs1x;C-4)+WH{$v>*_?SE)a_WFCqZyWp=2w2@1`8`Q zgN2m@5XO>KzEezEWmZgDWa3uxM(b{t}QJa)s^bX65c!Dc>~8*m}j9>xa+irZv2a1T7Wfgs`@H(&jZ-^zE2$j za_MzZH$qLPm^m6@_RYJ7H}qXQO8f4jVDcc{?y$W%rvS{$?ajT9Oz%d%Nd>y5@8HBD z{nRmV&Kp6G%fEo}?{0JXaPE`wne7oI@y- zwp-94__*_eJ^JPBM1=OO7l20YTiXYF1bQ~KgJ{!+@njcy8?dEu2b2_zdFgOVPyjCX zLjb7JiC#$%yJgP7JPG^dmtVT=PM1#l;zbwil8ZD?MSzm;$umNl4T2mRFFLMMs3c}W z6E(^-QKWHY0x4WCKmy~JCin$?9zth(U05UVikE)4;Japonk>5R*g=snf5SoQ*_a~i zU}ZLYeGf#2d~QXFai$*KIuTu*g->vonU5~T!B>*dg(X?!ptm-jlegllTzPw-s#79X zAQ9UD0b}x&1dPV4;q(&A*sWIKi)4$7)?*0WOBv$iB*QJAu;&{BR934iGpQ87%1RC< z`3Mt~STs@__K)d1bnqIT=}VR{Ktzt!1-evFbKMP<6rr$Z+fjVST+Ss-v0uC;lFE*~ fg3bHqn+?2iMqS8Wa76bi0ZIQ0H?bHl18V~S_51_O literal 0 HcmV?d00001 diff --git a/crawler/internal/sources/testdata/article-2017.html.gz b/crawler/internal/sources/testdata/article-2017.html.gz new file mode 100644 index 0000000000000000000000000000000000000000..b8b87d4f666b9a201a6631a660a580d17e2496de GIT binary patch literal 27970 zcmV(@K-Rw>iwFoM+kI*R17dV8Xmo9C0NgxXZ`)XQ&kZpD;f}Ic+XzYVQ=()mGfwQ- z**ag2;>>0+5Uxl{qRp=%Dao2lfC7Ce`qo9CilV^GE{er$TeLv?w0S5{FZw^yU($0f zNr|E;Sx)Rsut`Lb_nv#sx##Pidw7-4ZtZ-&_vC*2*H8D!AaQu4 z3ORQjFKSf=aU3+O)$w>Nj1|ET`&FsYXjIP;y26H8)FigqsvN?NR3SlV_3ZOj#qT#s zI@s?Af@@X1=r-wf7h9{n?rlQD*ft!iq&MxkUAepCS}`Rab*)yV@B4kn;v+k@_>mRb zJ=>tM?Ryns_+D&zF)(!e#H~M4sQ%qOzkPo4F!Uyq7yZzCws(Jc;(dGn#pJ1X@W38v zPxdRMnqo)ty!GwS96HI)25?DFBv2`9-asJ!H7|<}X;?^+k@j62i+p%Naz4z|z zKajqC@uN@7cW?jMAjZXid41A-_x3mBPrtF>z5Q#K!~>he&=3=^KY*sc2}t|!X`2Js zAB_QtLrRQyZ+{w22YgSu)Q@d19vZ^P+o>j`@fanw1pCSyM|qa4SKor4@Zql+KN zpZ@OR2Ol{2HJ@AVd_#U#$ux2xGw~*TnpR|lc2LmQ^NSxr zMzw04BbPBuHV^j5|NYh9LtA?bs9gLmFX1JV!Fl|ilA3~I|66YWlN((8;0*-Q9$fqz ziEM9BK3zuBW_p3<|NW15Z~qBMIktC3OnmxB8(H}6L@3TP^t*nXSA^}E);ULdzT^1g zB6MQXFk;;Sh968qyFbVexSzF=|M`i?qAZf`WX^7iNRD^g4<8@oFdXbZ-apj^fbn4hxoc;P_qYENIIb*$_{7 z+Ka6`;}S9HBx2A^=b`1aDiG!Xz_Sbx{s2N=5-B5yT!!gIF~lN6n4hr``cdSAZfp;v zVZ*( zg5kR#j5q*H)Ra@?(Zn;;Me0)Q3$Ag_gH`dkwpy_<;L$ci-^m%N?-kg}!>?(6~e$RYY4A;a{yP>QzaqnzrS_a>&cVY0xjjjkzF9 zJ0FwVr3)%%2n6vjTa!sJz?D1>FaF>yS-y~@6~+^mYR2;^TADL10Um~V&&h-^H}`lr zu{<7Af-0m0yfGZ|IC~za9}j|i~3>!czg=;YI`8b53Q*0xT!HRXUCe{^A zl52HYfb6kMexDd)b3#k)W`Vm8vz+70k)X0Jq9rc7_V`l2U3+w)GO_l!qBFe@K^XJdEt3ApC4dp^gkxhE6Wc%|UMCr^ zK13UesyIUE+r0|G6esn#>N%fax(dCkV=TuVPaxkNSXP`NVD&H~iq@Ri*y?rwy6RHf z6I|Pq1bF&l)RJXUmBqv@KNb(s*aqDq7XlkRC(mSE%M&=IBbu}>CeX{soJQ5tZ--Vm zVN_0|%H5r6g5ySLBJ)24V(!e@Ezyl%rYYk{b!S#CpV03$BvN@+G#_Q zv9NZ-G?{xaf?`7wdG&_LIgo6W33C3Ld}mZCYS$CQ8VgJ5+IzU;n)~U z4a!4m#mkB;>h+o?>8etb8|u)>@q?4E>W`1by1Hy?Io*$K&-BLvWSsPfPOK2pjD@D}zb9XRvppNr z8+t5}B1m>~i@YXNSRoijgH1?phAyZg+S(>>X5$$6#%Z*{5ziWvJ;-u5x3;#60I-7e z>^=Z(1}2PvpMEWF5(={-g-U$+WDr1m+!v^!Qo9} zJFhqTu?sO7(xi>%t(lz&eJgImK527{vqoTBO$GUIba-J*x*cuzAY$wzhA*+1gC>&zZi#+=kmo z1h>KEIO8@B+K>2`x~3kztUUm4nCXJ4V&5~%)R7A-nC=u4ot$M0n7Vaz@_N>Uf4bB- z>xcf(Gx?;PK!%~SnNI^dx)6B%ydM|FMfP_vOsWOv1sKzg0*6jutK;GJlVIT)5OM!E9R>M{VX z*lz>5pRFjKQ)hl!(>ER|6lPs|bQelhMlkO?LWi1*8 zmhhV&SYj*UOsn-xQWwQyUsjcG5>?Tc*adpYBj2=p_EnrDJu|UF4o|HRUA)K=Rv@wa z=;9CLY%OkYV>5U?|PGOhn9GDI_b8b zwfFSHc0<+f1PA)d$93gdb@%Y-t7olN%1LUMtF`*st-n~0<=hF|0`Iw=Og7r?gFQv6 z%`-B4X|}Rvlw1Y$JuA$i3w-wIs7t5cvvx=oV?Ni(V-S6vmU5E)KUQo$x81-G zW9r4*Z;-WYJHL{p>tTk_d_-r3sjLX4C|NCOTKl*rnpdEBjZ_6Q^aIvqdp*CnkflnA z{h&#BNt!FWx0fMKF-G|=90E?LY5VXmM)U<~YOk}aPIITbHSHr$g9KDj9 zX<1Tf_>Lcf1QbOnDUbHR(XvvzQd&S|5aV<{bx|wXYoa@Q(=QqZAsMz-%V57KiE&-Wx zI+jFLtd#8RcZSZg{k~)0UA`mO0k85hYV7RI3gBeGsLv9EE{!a<(m=WiSA6TS;^%s@ zD2i1WT1hkunOR*B>ry2Z`0ROPTjP8Fd8;B~d}fX5*JYeB?gAPQ7<*6QrlZ&D74~kK zPtqK0BuQhd@<`;w!vyt7@^vV_s*s-T!2Sfw=ko_~)+VW!k<=5CnxNG+P8T#)qLL`o z;UWDJ*GkC5NP3R%#UhMN1`0<+I*hNfyM0#HC-!P=!;{{yaj~#wmw2p@?lt3m6miyD&Mh7?bI3IK3;id>NgF zz-lGQ>df#Js%eeh6*w&ls8%MRS|T77)^}9}Css9uHUv@Axa69wow=3)1RDv$f`nq^ z(yug8U`Sp8GF?z>wGo4)njk9Z?|4nH!M)r_t|>OqiPGSbYle&ASQ;EyRKVaM>+<`_ z=&}H@R*)zU6vt6?d3dSP8w(iu^ISdHpmu%#jGg1chWQ`3u)??JXTPG4@kbosr=6}t zA-a6*fRi!Xkven6nzfTnY3oYiM-ytQo(n(e^L+SWjp^6>hMz`RhTTXrY(OnFxP-tV zxR}0+7yP6iA#x1lq^vKwJ!Am1qz^wQthYEaPxjTd#gRs%Y^IH5rgBXYlv)k=m9^SQ zLV8fgrg_#y#yZbNQ+XdAd zf->R-P1g+(f{&uGGz`N~0}?L)QIh+CAi|6lv4)>)#&FBy<)jXBQ6%iI<0t9or~QXg zO{cIUYN%XtP5N<+qb#{?)UG}IFOn@Q&fAkPqP-;vxQW9&r1Ek}ZUMam5^2diDO-uUEF?tAb4 z_&blde%E%a>-XxNYH2AuUlnKXvH8fNp)puy#d9ya*^#sP(fInTLIMY}-GT3z7VHz- z@813kIU8L3CnfR4zq;hie)q@!c9YlO39&P4g1@?fuOPG#S7^n<(1WIZe0#IWepte7 z+@3;jY%S$43r(|w7+i0&>4Iu@y6b(YluBnmaz;RTt899*Uf#zzwW37y zGKuKd5qDPwi*g4pp7kheTXShX!zN(|jqcNU*NpIJ0~S&0SPjNN58FUlhi6zuV2>p!8YkDmRIo53 zsqigv8kS@r+thSuQ^8pATd8wuqt0pUqjEI~zEutS0Ivpyi2XW@QQ;%P2bLNQ@N2RP z?KSw;fh{w7I{l5zORLMb{eTxGp1(U+uK?$^aV~hoZm|wg$boijx4+K+)AS<1pB+}8 z{bjiONEs@>a(mCuL|r{>f7{#dSHTjksER+V$K&tbfZ874Zhv+1c=}&UFfC<&uiM;` za}-G^cP($Y%IFFQF|J~87_ad-%ko7jlV$$2IF1llbfpgd4;Xla737M)dHBgFcibg5 zJ^X+2p0KxVBg@a(UoorhH3?R(r*UCBZRECb(nfWgelaMD5xJH~RU{R=dj&QP z4n?oGxa4*VT<>r|5?`^G#Rl7=1s1iiw;$xoC;tiklY4K56o(QeD>gS3m;{zY&T!t$ zH>dXz9hL_~#(xpAqu>{u_faJrNQ$o}JfCx2*{r|FvJM4cK zA$tcZ=2p3)={mvKC}P1bCrXf+O95^eL0>n_!Be__WZ3_5l$}C1X&hzO5{iF9yo%mPY-5#s)$)}gBMfzf(~b#j0HrehQMZ6#}cRn%1*N$9FAGv{Nb5 z(u9bpR4E1I&IF$Y{BDIhN+4Ec&|sc@`|J@XahRq=g}RLD6Dp^2shk3KrvrbrBU zMHzx9mB%^b+nZZeQUeaivqh<#TOAcF22aHG>X)!^3zG*3q^6+|1TFmdF2@+@ZoDof zDo?Wnrlc$75|EcvJ{x(R(1k2a-z$y=i;mIJBLWF-nEb-0f%gQgWORh!0)>z$D=A=k zRu{5>;`QO-Hy#zKUjq%f=)4(Wwyt@AP#|5GqEk)*oeAKwfW_M2Ax=EHM#eNITTRqj zFdIoEXrlRVH&G$is1-y_gA+`Lo+%;j!NS5Pjlqu-;@h(fpRF53ISt;L1ad)z_KOqS zdm3Fuic)1l1%uBX6I3S=w ze;qBe+bEF(CZ`tU6tFoCDnkwl=5krDepnUOtG8-N{+_zIhadYun-wtyp_~l)r^4yN ztfo7r))87h=<0Gw8JEmAs)M^X$UE;35C2q)`CJuXq+Ho4pD(j4Wh;_e=eWOVY;r_#Ab;Qhlfu+m5lNOB4&$BfaX<9%t90x?br|vswvGv(HuW5nyQz@i? z-Gpv{jOO~01zaE9fA$CzW%Jp$I2|7`_4&8@2*oD#ZFfLQr!noPK>Mi>d5-;ijlVd@ z(IL%?irN^w_qSP2QxJ*X3c<)*2c7eu2gy@ggTF>pNllQBu!}juT$t~5%@KMuwN6uL z?F*t>6l1tNHv&CLgU{6kzNrvd(G86o@j@A?d(>&F38FG#q~1RIY>QL}zgy#%yGj`a zNKmjC%iW{moiENd=w{fy2T4oXJYxEWK%fs30Ca-~8F*a}E`=60ob?rLG=Aj;qUkpkXgUe5yI(m?e1y0SdZOe@q7QF;aRZ_qdYdaHI!a&I{2u=m|l z33?s&KewqieUA=pJk)iBNTn>LfW!odFS!nHI)dwf0zbjnfqVZOx%up2OnV2m9^PLJ zrXCnb+UQ}38)Dj;2Q!Z<2E|m2Fcd~MnAY&{Z|Eby&0F=sQ!+e!V&^8dE7UCkca5oO zpfC|+CgL>cu7n!dsfP24Ce}gNfBBg-V6X0h7#qatUYNwNqi8hFc<9&>qzXn#a( z4#rmOVM_gdL8j4;Me%Hu76+7#s2+Sqx-|d$Gjz7DVB4qvBJgRetdivO$GBY8rJUeTLxKFndyUijg~wZ35eZd>CD>( z6X9lJ;?iJQG?dgW>M4d<1f$K3@0K1Bl!7rvXGZB1uvx$;EXDqn9^w9|L*DVJwPXXM zy6N5lFBtDcFewe8*EPC(!DHA~DryRto~t`AIk$9u9W-^v5EOO7=5ULWDbF2MIk(F0 zE>yBOx-%tdCyZLC03E4vdi!585lJ%af2o_Kj%pbf6o?UlbpueRiiH%g7K4(F9s4*E z(qrl@?)P2?iXiUMlSm39`ltq7*L1o3l1EnkJL(HnH(Fe@;h8)3f7h|L~C$-vmQ# zR6ULPa9U~7dSflL3vcxoN(HS+Kr`W5!R;NMkJQ34 zghkGjE(jgp5Vp~D02nN%fyW7O=dzJ8A;#Hzm}a9M%VK0lvZZIx9m9ajN*WlQ3Of^O zCMFukq(Z??KE~H#h8|eq@MeD%6J4GitI2?~u|7^$feKUZ_pFBD#PjXLFTsXalaRqb z^qg&8d4o4O%4I3HN^dU6?hLK9OI#DrLOgl2P*=dZlfykdyA0tko|=9dbaGj9=0Iqe zyC+3_avjKo;JM*6bj|hVK_;u2CN5cv=U#53;pEv!qoX!(-U@A}YY6Shqg8{PB^|BN zu;`}v&I%Fd$y_pdfI2|f=^5Qq9LT0FsVaX&Qg9tVB8lD%zD{v^x>OJoPGbx1hQ|WY z#<$s58t11&^QA$RLsYU6OSI2H(@8;yX+|E)*7i<$$7r8RwD&@`=S&e?53XEo(v8IQI zkGHD0!xz3z)gbCW&92rlE!~Z^yt(z}^{el#-KcJE+{~@=SEqtww6&)832>emkZW&T zUAtMk{@$(J>geaGn#Z+K173krp4b+!&Zl8PckDyznC}0}#_3)(3)L6em?ejX>?geN zApU-EK-x6fO?!Cw7cMS8e*d(DAQaVtpo$aFff#I)Kp~|I_VY9_X6OY|Rr6YSyc4Ny zgCfdZ&C!GBm%#~7JkRM_9t>1#BpfA`SZS`4v&J8qxxnC^h}5XZ-eYnbql9H9s2Kw2 zMY^#_X3PY2gIJkOdvRpftbYC>q94gI!%A z)x(p-KDyVY0WSwMcM$r>F)RTM0yOJinyTgkrI$a4RfQhldB#Oe1v#-oDzC2bE|VyT7Z#@1(6PYWOEzMCK$CiHi<+wBHoG? zO#z%PGb=2>XmvU%Xc&Nk%%q7&LK(;~)HuV~Fqwa)!94s5GJ@vd;Xa1i{IQLWgIVPZ zFwzDnK{(_0tRAzY;^_rEhg79;o%`^S#Zt+V4>z|7^Ee$naDaMaOmW_t3xI?>PPf$% zbiX4=1xdw0$0_;)EGHX&hhTYbHGaJS1K8b%m)OBnLwN3lC(vyh_*!Ye?OxBqF^z@8 z5BuNKkl7C1k^BVKGW1-8cYWYLdhpuVS<@eGZ{}8?ul_<%Hse-dhEbbp15}(Ds#2Pu z>Vr0QoH#@al7|9|LQ#%c_z!~BT_w z$R@3f4&ra@z}vA2^)Yeutnx+GUdja=X=Q#@rCc~cBPkaab*!ZdjSq{In1vZBOKd!QQ}*J&`~Rui~Vz?qZpFTkLX8SAJp`oNEIfZ@30@ze)j zTQt7>fLt5=2)vXIhz)K=;UbV|7s+$i)GVGxdHn)L8`qv{T^9C@4~>)JxcY#>1M}BT z2OqOqm{vCRK5vPz+J?Z6j-R?9rw$kD8MSOnZ#nu*Uro$E)6hJ8O((fr;yxX#ebO^i z3!nGw^YC~04ZoNVB`EJ{T@n}}E)(0hLvGTG3%RAZO(IWz9RZ-k>E1B9+m>->g%tBq zk>6bkvV63f-b!O)xh*nr_+6 z6;fbOlEI-tiFo@Rm{Xpy=Pm258&V5xln3;Q0KGBxR!9Z^k)K=bj?)EQ!3$dt-M&kf zE3mGRrLiODx#pPn2#{Fp>?Jk^9mVcNXyH2=`Om`sUIF5*0Wt2>0CLUImynu)4$i+a znh1p6JAY})14zo_58%D2541ot^tC2^XnSjFHE7MdFoyaF4{-bEg7AS+7iyr1U(;Nm zn;JE{32TkErFWebvWCvT4b59?xGO{~7V@H0fxmKIE-86YDd)wal9x(F_%7xZQHF0Z zuRt9|h2N?q=an)P$i=*j-^Eg1u9Wf${gd;m2;Z^-WfC+d=2fWxe(;TVwBI?aGo5uaxyTHY+DMT=WEbbuXNc_2oQ_lCS(JcO8wMXM8->jLU|G z;b~S26?f0Dy-5>wINgI=OJnuNu17OAyx%fzLnX%0gqCIR@^d+GM>AJ6!9VsRX)arK z#}CnMT8)Nbvp^mgg@;oo2W-6MLm~4r#!S+>z?q`?5?$}^B-*M^$ox8C;RnuX>6{w} z4NZ+<{~H~BH=~QV^_+88MhzwTET@JL%$L$|N_JO@J=d@Az}Dc+Q`bucNJ z>Zo%vu9w0LsV}%=C#2tu5GaZ>a%#TdPLJ}fmJ@mkcey00XXqK{KO5JsrZbnFF*i*@ zO=K6jDNeUK&Jn%qx9ig<$haEFm6+lfEIL`oo2qp+q7VB|e0I#*JB|>H+^JA_qu38=RKZ$*Gw_FXWSS$Mmfv1$TaQ*&3GbgD*64^s6}n- z=w3Td3hHv9v`hyl7FP;|K<$3juDhK}|H!_^dSc2W20E=HC$pcVjMS?jYhc1P$JkMD zbCO;Gr}7C4ff4EgP2e^t#h^ZRQHz|}^U#Y83j%kk={U`nv7|fg<&Ix(Suk#RPww)@ z+SU!x9qM$%VlU9tsrNB-nc6PL@Ky-S2? zSS+#(ggB>}6O0xEjI{)$wYlHZ18JCj7fs$VG{Eqp6(@e}AQySS8v`(U1ot+J! zE|>CT9ai;(6WerS%hCP8Lx=5aAs4P*p<4lu?$<7&XXPL;RMwFPz$1SqEi`wE zzV+$pxKz&5Lk^#KVZ^>UJbY@i6CHwKgi9DJrHI&>@9Vku z0=xi<2{pK7+w`eh#uxTknk*AttuzFI5>+px97bczoTyXZBu_Dt2rJ)$s48(M*A~dt zKQsS~PX>P_yEuIT0*xa33gZldq`W;)XT!JYMSHyho8oVE=C7tgd0x`_d%#{KYDo&& zn&kXDGeUtzOZzDd@TRGC4XS!YG1YK~(9gg$b#g@3Y_sWjIJN1tjpvS}mgFdk8h^(k zs3n=;X5bTEN<4tMSdb^qn3wXRR8@NAQ3UlX zw{PE`Bg%IixRL~1i2}AId0Ud7D^Pn@An}Z4&#W-}v50yQi=YPXUe0w5&+podNZ! zQ_xz^4tcR!C>0e^8U;g*y{P#&pq%#Ae9}sOH6H-uit!s{jE1PYM0Odvf=1{8#v?=?a(jJzq)??$}PBWxVqLcF2tS(5}t3}F=dYk z0N-(Zk6!%b%&n}5ll+C$uEu8QStv1H2IqPR9H z&SU+Lu8AuAsFqP|mp^J3g_0(+JwEtf5ZBlhy+)r9JW$RTtLk1DY_AW^ zG<3zSM$RYdn+%OcHLXehpS^2qZrsSu@2=Ya(2+MDyF?K83#FOy>f#+Gqfy2)v=YyY zstG~_Mj*f;07~*xT&~nUq*4`^pIlp8tC=mA%ig4X+kJ>sDL>Ge|H1r{oYUY16hQ$R zOA`u)M<&7UMx*=mcTRt&Z+JjG=q|hDZO@O>yhC=ZkPTvR++CdA2}V_s8?xFkr2S#> z53J)mONJs@x*$k`Zc$6tEGo&mB&iL#rZ){qX47+!#iTNyN7TAn^=oHT-UqHJt0x zPGGH!E>_!t&mHfFU^^RN>->WiQ8W}5I@-tyjAKPG+a&QnI4_p#2Jt*T*rfRP@CNof zejZ*z`5J!n7{0pWnbOfSovwelB23D_z>HzXR8sL|{nmRqeV&}!qWRdVhD<2|G@7P>Pd(4}PYCqt1; zK5V)-5#I2DzP2Y(BzDB&2YPXK@wS-pm@%jnkk*f^6;WbEX94@P?YPG)Z#|3raCNQg zP&dA;#9smR0xqz+&VKxRm0%h6fhE%bZl1{j=l%m6VI`0JHud7brP*Coz)9Oy)1lLp z6#GgX#}_mnE>xXg^I8t|TGb;rI9D5hm)~7!xRJ$PTz?OfB{uzxE-w@S$yxr9ED~aP zH7WQjU=aEEb68Zo44bpildFKEi-0%OH=Ju)!G33zTm;2=NEGD~i8+v18^>!?Uz)b(mD!wx3lNEjYp1&%NFFn{S)$tZT#kT|K}^OMe6;N{Fo(kCAB!1HLCh+fvj3u&@6P;cv(gZ zdM`(34kq*)^XB*GeAt~&NX09moz*DT}TXp1+DkvkCN=AIc5Uer; z*X<~q6f*=vN?hqz(&Ca9FBX#F50U)Pj(js0k{{Y^#j;tFD zmf7K-1@rStid+yRZ_sw;JjO=JmmauwFQ1oWxFo|1XNqNhlX}^f7?pOa_*PK@LN z?B6N5XEEnz1we9?=a?lx@_$AJKbb){P(KD^=6u_YlcCLvb%`*@zW z?b8ex-?#hEJ)59WE-WE7N$+s`TM?CkFqAk zAQ6F7B_A&N@M4)_omW*}ly_)*GuyF)ROML=B_&>9AS*j9+gi)=>>$zL?000Xxa(|d zB_Y0HQ*}CzN15grUBhI>8(k~ua6zW&5T0R<^Y$2w3wYdKNsEgyLZM2$C~i}KBXj4& zHomQVqhLEJOG;dX-8-_(Z&PQ(&OD$DQsLC|i{c(pCJH4bzHwt~+th0XeioJ3#y45C z$m5sg`z%UIyl|@~1VC|nGkXc0$C<-xnCONNu9bYa zkj7Y&`S&B>6gh_3-j4>Se*zre_>s1YS#kN2x0?oGC5G@$=0?mBf^Ya{%#s!t(z(-O z7<>6GarvsZk{Bi?-6FE07t>nffH(s!{Kn7wsdxH|gaN+hRm8`fJ4xY$A_B}UicM?w8 zk!@OX;gSn4-U!8G-weCiJCq*xe=vav|MW?LkDL(8SG?UcCam#wdc2c8zESPQ)Ou0X+_7WzTGQn3{^y*+lD+=Y zR{n|tAL98Cna4xaF%{YB|4`i1A;j`^Y&UNNtlp)&IlBFB|Ag$8+_mJci)n!)8rqKK z*^!lLh6iVVrlfAO(@r?D6%9^*%nfRCZEpc6nQRH`X;_PV2#C&e6l>iXoPAutqkBq< zTnv&?SY@og9Q z{(@7pJ5_e-$ikA?tmLc>I3k%{qEA7`&nva zWM1p;cK>U#+kfHl02ZIr9goxpXJ26t9>-r*>C#FGVryC9MWr0yHPRIxTt%O2MkXKH z4glA)xg)F19atgvETS9+hIR*k_$CKwIY>++&cri>IM1dbhx#Pwf+_(46f_xNZ-?zA z5;`FQt&VR@z-$R*OAZF7KZfKPu1+S9HK}hstOIQyoPBB%$eaw${z!iMwErR^w@f^I z=qv9Q@$A9r7vZ~0)0Nx{?WoqZ>QEHp z-xHDWd72t5RgP3MFxqpxeLuH=uN_arvSFb(hON9s!~d=fg*l~>OXMu&xPxw5(EV%< z&b}iD4wTO$UaNm{Of+7v5VQZ>B#*IJ_Z+k;&B56hCh7H0$WQ-{#~X!%QwR+Pr~eH} z|Khbs1L^`*a0X|ev{@A~Z8#Q!+5Z84K_m&gl$c%?`zLLT`;wA2PTW`7 z@duxz-kf|?dvNx7$ck)()NuMLmK`gD31z@mzzPk6C&luxkL_WFAnNQJs0Ul$MGmb7 z323x}CO!||=bO^JX=Y>6?nn(w=#lFjSb+^4_=IyYUbRAu&JhvLcdMic-&T`|5cy_P z#DC)?%rj4mAN8t46yiLS6pbhlxSFOY8#dq^0CTpIQdO+NRDt8i;}cd%(+N5*JzgP> zhmTsFr-n&E>^C@+#S&@KkaE)`ZOXKUIO4;s2K4uoGoy$@!#Xh=XdEDza+(ZQ0E`Qp zM&&v-7=qVEQMWUiu15FH#{Nkeb=YZY>Y+C73|2=KBYYxm|dq6uDHHQ zv7A*FbE{1qH}qHDef+`t-@ms}zjJH3>6=m8^1>=Joy)i0TK|X54%jyPhcW7v{ z5P>Dgyflu7X(6wxx<_P`DPfO8hyDtw>vvkQTv7yV_bxIOPANEmTB!%C6xW6}4h; z>XSBSaQe1V(IB^R_1;7s(S?g^cpDpKE_J0@NWB7O_U35w6#})>g@^tBBUBOOS?I)p z03ESbF}74HSLC>@Ik@U+TUt-q5{+F^x0PK{TpLGMF#VGUq6QaR5U<$AleWz1RIG>? zi~l%rE&j!R*D9g}Hy9OpOWV~U)5{?xwL(f-g_NJ8a+e3>r8;mdd&jjOP{-Q?4?Ebj zLa<`u9>mpYD;TrU1M4_AgdRGriQ{2`al%8EkPDB%u!BeVXL4Zof1oR5=%lCIAGz}? z=sI^3x&;N|f$)94+(L4=%*f#~(5PkLuqXa9BZpHy**YY?h)LowNbIa;rL2tI+ITz! z3YJznxc8f77bxQIqU9k~Tm~yJRVaMGHzZD!ND>E*u7LU_64Q-6bD!N z__n+ArS{5J+u83z=Q3TCK%5@9+3?$p1Jj^$^saYjhEYKGgPFuG7j)p@wwYZZ7T{lD zpOhGu6#h2R>B|N(2JAAN{ox~+pjc+P5K20sD5QZehEcK^CGJ7XXZTfl<>0O@7Rp{*mv88=_+9q9Tu6r7@bO8t-XfR@+e3 zs_D1+22ZCP#p*B%gjEc}3Ao-kSETp6CKl*C=F(8~GutZ9Bd#UWlvK&b_(!PXbMTc& zXf`<0_n>XPi7C8oOS2OzJ*R2;)4DZ7y+@gbe#bGv{hwyzQ{*U4;+Wp~BF=L3AZoX5_e!qHvb?KQH4O-~0&H5s%Zg-k z#x9FP79NLb_zApxO@wg05`PYK#*4vzbNlEToWAKV=lC~gJX^@J>^UsDTmZ89QtlWzQZ-)swWAOCK#d& zoiF@>U7w;{Rkv@%s6W+qcu%{kg}rfq`LrY(yH zgW}mdR%%Z$_C%RyLZCg$LAPZ{TD9YiW=yWLZhQs{binknddwYn8&{h@v>JP?a>20b zI3mi)9Q40&ZTQQK|0OY+5Pb~Q>l4Ws|L1io={+ZPYf}7K^?+_{%DhMJMC+hJ=FCjp zT+epg9--Avx$h#?dxq=(K;0bfh?-mzN0gd4pv1%O{t4$YK+O2wHG#6R=gFSSv%kay>L7sWf z*HNsjwn~oy-#(3kUcDiJwsgQGVW&&E9(1yfX>*aNO=g|3Ww`))TN4|)4gbRhbjVF=SIwja;$hS(Gy9#a9feQ2yukVh?-Ib)8ON?Wy|I(}uwWgH0$mxP-C{m&fDF z?)WOTIFg`NRaGIfS`+aihm)A%D1rx+GIul#1y@mv{pyNZRTMq;c;`6FIE%_0@}7|r z7yHVh0**4Bjm2vq_xRc4(ryP&J$|+*Rt-bn7KON6;N(|CVJ*%*{v6J^NwM3_0@dMX z?3KsiqBPI-#%2~Ik+4>Vi&qw`nkrZ|%?F=b5X9RAZke-pxB1o;dGLeN7vo*U=^p$! z52R!Jp-+NCr(<=w!!GS~ric4e3OhqVMz;qp^fZ~vGtmAF7qrBL^ch@(Z-+E9qChY` z4*C^gOhUn+VOwx6bV5DoexP4^7FnFl1`o!4QB4vhxdu3!S`?RyAzbMq$PYR|9h&^c zhWUO^+<)41|4!ZV8{$)2Xs_{r18|M8%=l^&pcS6Ui^-Ld(J?NKR91(3wg`xZEk6t6 zwU6zgQ)}b#b7B>wlw_wSBvK@}*BhLE*Cy`Z6gsfp;Ot}kfL`ob6lV^%#~$(pT9ISz z!s~9vtj<2U*=-^?_Ijd=gx3H50)h3v9t-PobLul){B`M3Jcecig|q2+vB)t;eMaCg z8t9u1*`w?+8Dn|yp^t?w=&GowLQPr_-Ra&t=pF8Kp6bdI@jbKi(fgVkp4XiaT3O+t zJ{l*=I9QfYqAnZEfhNOqa!s;@x&x@Mv;8*X#lk2if>G+KAJe?aI^vLJ@q3dEFMZrb z#@8O^h6N!1`_G@@g?<$pbj=NzsWs=N)jt_Z>K2m_E^%Pexs_iki>WH69}ixjtwP79 z8G&sAALxKBuEIg-khwTR5yyg+6mW4y;~O_dGh}{Lx%0}g@$xw~+2_tg5qhzHc&!Qk z(cfuCU=W#)bl;BE1P|PayBJEs*at$#>F4Xf6}Lz?)$5;lty?qo?LTLS3w=LT z&xJ%aCj)ZE(!mvkCYqBTWn+5|C#js*@;TWMaV7;jYZPjWYtatZjY6<7TsP4HOM;=w z(89X1Ac)`m-~;VZ_voWXM;mh2Zb!du%I5(;sVk>K?R!=?*%?gQkFDKd90|crMzFVm z9#DsXuT&wR0}!pQ5|9@=kZf@FFCO3l8AWSw`bCFyA%QBahG-N2NVU+ocW5O-}yoLFtB$1%XR|Hm0~cBU!e=0u^dnqW!;KAkF^ilq46k<7l(}IT0En zGd4#-1k(1o2Gqo*P3!;IySCmot|a{)VE;ppGGoaYQS1ww7klJDmJ~~lZnk7O_KVRZ zTkN*_LiL3r9poVxEEYkqK!PAxEEWrl*9Zoljo}?&7K_LT_JP^`ANnU-Rp<1Lq$JX! zXvQ1NI3k;UJ9Vn+)a9!>z7dli^rToQss(jQ_(1J*^_C)q@isG!O>LcwssMg|oL7YIHR1(-~nz}qn`F25d(i{KaW`eygX0mJH^ z^bKh``LbN9!wNM*S!P6_+E2hyXJdv@c|!>14S_g8o1!0Ugsb^JW==%|h7bC%73fwj$&%0@GKe{38?Feh9Z@v|lql(2%EXO+JP*6{U zd!D>@D3|=4a?vNRg3Gi2gy;1lDIp?)MRXwn(fe*1;7_Ls*L^h%(WXzpVDUxGlMhTA zR$CXBTPw0p1MCuV}apOsCp9FsPrRzE~C|xi~G0!rI4=n>tR39H@AZmRF<(mCQ78ZyfI387iD^{p^yI(b51P(#bP98zjsRqVdB*v61PgyE z!B6{|BisS%2z#75my?r8P0xkybN7ccXo7*K0rcp;7(k2ZTwm%gsS?sUA(GAY;AI0H zRSVLSk=4{$elTEK2`*`Ccr>7ZMNH#@QjrQJQJEIM!VVhS53IF^x3|@u?e(>1>&Kqd z7oe5%IOrT7bFPxZlAfqX3f8 z!eC^L>PAd~YU7!M0tdc#gf10m3(TY@_9Ec{s)4(3F~z!nVnx?Ilp=?N zQ>ptRMWN{vC#1Lm&KoS}O-ivh4MNp`dD{x}NYtl5*E+`fs<@ihP=Aq+NB=oA7*G;^ z?%<3WbfE?&0URb+PJlOVp{}dL2PeJui)E!)P}NdpN<{DD#_pcCTnUO*=k~M5OPgiw z;m*BYpZ;mS5lCtZZMDy6M6vwSH!o^Jm90ovwU04KCkbe+F_FQLW)#{26&=e}d)%5LhIp@ewTY{}wLiaGwCJpeC?nWHurf|JdZk`|(wm6V_>En~Mayf3mPq zLNrmcYh7F{4UD~Ewcp<|lhXZ5NUaZQ#w{*0Ib2CHq*(puYgrHB!fY);#`mqRY zjwcg=HwLp@a6UZL$n{h(M-nATRx9PgFDkPs<*RO8&&z2G5^Fmx=8T~$@hv$HUtr|t zdtw0QRiGh-Q+@K%78)i$J(1%YX96#(b|l^yMHzxKyle{cNdf#TvydPIVUu8Xuo4mUQTqe zBB@HHP*#-EFDO{C48!{bN0(Pb{9IwyWJ*q=Y>UkT5y<|n(~~!ms^d*?0v?F20TV<4 zI=U}Ku3GF6u*hv-osO`@PBw)l-q-kY0`m^$^CBU_oDEV8;eCUs)jdH-}IH|xj7AUMj zK^99T`Ioe;h6aYtwfKj$JSj+Gkh8p9RIu*^OtyU@)$&H#9$Rz0RnR+ywKt8QtO>++XI+ zYS>z1I3N{V|AQGNm&IbG ztn@ZY{EHo2lP=zB}Weda4G3D?bpMZWEoBk@4hSFb_zi%D2^i`wwR9&}? z?aHS$qfddAb7dNBetCO;^?q(O<=?WVh!_t#YpC2I<->go<&>IDMx_h-7*5rc4u$pq zgQ46SpzH9ZGh)Is*h9U>AI!|XiqygYB}=*WcIf}%yPyrM8${_P}alO^A zoWnWCG0@wjL^^t;Jc|(k8)-RTN28lFN=Yih|71z2h}lBcE#7Ul_VXXn`o*y0VVre% z9pDtbV%Puhp?Wp>H1&`8qJu7r!@_;=D)oWy`H+ublB19 zaph6rHF-FnMOUz%qEr^6E*duYj1C+MFeHO8@fKpJYCdx!Ev-rZr4N7GAPuP!P8HnU zVmYfIsbX@6pd2@Zbnj)VmVl>nEcm~Yk9hKD8#l+9$Iv`x@E9ZGsX;se5u{p=yK``IMX;eBl0{WHLy|43Rv^;FNvku+H&HL( z{~bdP?Q=}*{d%;#^s6&+7Js54$V<|EHz!NZQE*I!X>~9G>I7>5bJEA~wafeKqI<%u zOa{BNFmY;O5ssmw9bsUo+etF0cxDq*rpzeC8#`33Ltt#GwZKoyWs3=<1XI2A|{vs~l)ZtbOS>gu&+5VkkX`%}@StCn^BK5u%@B!Y?l2_$X93 zj0N<*jCHUmzUOv!M#LDO2b36G%-wW$N zL5N`afOF`>5mh*(1V&LSaz)BUkZ4i@EYxWTl1OVc-wpy%s)9klBF-orHk4AMShnyT zc}*|*1$Os&2a6J3JG}N2cmkV1V41=^|AH+0=8K`C2JiCyeq~JLLIk{E@};E%*C7@L zYm+(T#X%KLs2Yd5ulu+xxCFAMLIX-`S~|k0Q>Dq0TrP-8IU7q@#wjJtN{vz{R6UHd zhtL*Qfz0$5F7RG1(_S4i(aO<~&aJcLKfyXhPhFmWtB;0e2Wdfax%- zH`zCuE@m6i?Hs0wZYG1_Pp<({7{ZHW$JL`1$AR%2dyB41@Y%K8RYj)uJqN zfDPZU#+HTI;b$;Zed_K;yHb%+t08vYr$1^im6JSB>uCI_viW!Eitccp6ij;I`M$ytoKnwQJ@9sJ!7?3kBPmVuFLX6xWg0G3Eg=0k zyR8ESFN?!_M-$9Ne-iW=S>Uh!rKU{GBds5o=G1l8(~&v~@Dw}Cjv}wH-|!j>X1c9) zeAmW6u8!ecyJZ3^>8W`Kg|DdP;*<#X%I1pmxU;&UmD&$h_UaqHCRL96P=fFuph3Xm z5@NOiN>U|6n*NVb_eVTCLQuyTL1CG-lxBTF_?ax_x)H7CpC%L(c47*SXCa@!lyXrX zk}Ta<=aB>#$Fs;WvK(LW8=VOt@4i|Lt+rU*DXL24D3IjMUES$jB}-*C zij-_^oxb@eQ?OhPSF1&CZ0sgwC{c5Eo~~Ha@RiCJrv7@wY{T*)l`j{ilV)@wv&@2tSeKQbm#dV zf<08qMmI4$*&E$t^&f+Rntk!hROr2bLPiRy)`<7eCBqfQ#^B4l3#cahZX;%Zx^#wm z&eLES*8-m0#0no)8_Ml+=1Dt^kV665n0ul-u}Yt#uoI`1WU)|`ise#iN)`{TL@ zal7h1y}caPS`QvJoOTjbl`2@=E0^;FMw_Z;Kqnj|pSKhZF$eE)Ys`1qx)LOL&o{3q z#>OjJU-G|S#{Rd{Ku|SH#A81UMS-gW?y{%qxqz;hTs|f}-KjuIFS)?Z@?FFwyd9ak zFViHV3N0efTg6i*@@~^+KPJ3Dr7mhg%*BmL!8CB2IY~^wY*+w;{NhupAzcL{iLxXX z3&qORIAFWW_Galp_)vPN6ql^~h0cQqpWN%OwPZs~XG-E4B_M`Si`%HSqF9Ip>qS$L z#6I96*3~c$Li&uMmT3ahlqpu~88F5blEJ3%dc6bfD`pxYb1{|&Pp?#2Fb$Y)ijU+X zol8(6EP$n>h~UQ<9FtErr2Auks56gj|_6SU`-{}C0sn~>vOPDa~J%Ax~_VBTZ z@Vnp%F6SeWwEJoRIlKSKNlCfJh>R3H=_es|8I8#|vT}HbZ#%sjg_&D~^5Tq?m|_th zMZL=+ZYY2k1v<8E{}Ymay9F@+6nd<)_g%@r4?zJYgt8giBIHya^Sic z#!dRYm`*c{hH04fy6&)RjQWPz(r1L0W`*#R$OQTCJ~*E+eEVzm&Cla47M}VawH7I@ z9FOe+>awB7q&%oDTc+*=b2`v8Vb+%B2{mC+fU9)fS)h#-g}fjOIiVzq_~~Z$bi#*7SH>J=IP!-(`bGaF3t$Q7pn8Z43;ocG4l+330&8zYTlyb&I{2a z=}X=8^jhG0rtZ^sKhT1ZeT4T5qibNau7!7CKY6-uKGUkeG5D?KHamt%rPlPF?Mw-z zn~!#;ER@xGA#j`Ya&`Cz?}X*Y^t}e`sx2Gk5PC3an&s>K@aH`fO7Ees zA9$Lr&*?27Xy3$|^dnF*CV4{-RxKSr-tMf|=Vr}=W!J{9v+1Xs2g01`Z)scXz6-d4 z;q~kMrEc~Fda1$yKSdYp&zFmJZY2n2WFay&#E$A03?n4^BNpDT*qxA=g*e7eHab z8HF0*iXR z9?|n=!i)~;L{YeQO`uOXDMjQBxADZ+o%*Mn8*A{=_w}O?%$GT*chJ$=dr-}Jupp(D zBmSZquHHgjKhQzPd#>+;s%w}IEM<=CbZj^DduRg^>VQT}6s)5M`Ih-)e=G0T4}mft zwhyYdwJ2R{>7I{jv{haZuW8NZdVNt+iv^`1RZ3;0tjOv$Eeu>3%2KmQ4DQ3~g=NY$ zpxy3f>HcofXlBTp&bYBP5|`yPwQGKym|MVh4sh2eus{c{4d2sS6)#XOuT~WnqW*{r zi6QwJspuOO^>@E9?WXGmniJe)!i5jK)zdj5^PHIhjo{AMBsuOZk=gi|I?BbCd2bKq zoLW$80W+#^QMYEQr2;dx2`lVwx^HU^>%tMC);fLjJx(RK_`(C$L7Us9xkjE$$Iw{a;)b}JQeD7Uil=$PsVpj$a$$-DrnI@U>a3~{ zHy?(FjmM9}plq#1^1_ChF28y}M+8s9u=DGHl{@?Y`yV0J*}Q5B8+`o0CSjh+s90ig zQj&266EloZj7S#B1imk_;5ugDhljWwZUgR1$=j-%rak!UX3Gq;T#f3rHIq7dFbto8 z)!VjfyUAS0vLZ=}Dov4)3;}|c6l^0br(iYPoG`*35+;>puVNRCUKUwd-CEyY-&M$&qTN}_ZDK?$OE5i#EfQAB(b*#zJR zn7Y@{S-v%-YEpze9U8btqtBYT^QND4GdsQcTtQW(x57tXmH1r*q$B}c*-CF+Hk#{%)u)EJa|2=*j|?W*T*7y%oBaNXxA zqILh&L$LQjpbk6KiUb?LV^KC0WWzbCNP5od?%^E)iI?Vt_K%l6h3rlBg=hU#4vDM+A|2_k)Ul5$+(VgdI+* zdD`8L+Grf{s!zlnJG{7$?Hc=ihj(`~s2|Zc%orxH##hnK6|AzwVEdDX!nF^jigI%z z@sx?P!BCNw^J$LFF0%B-mga+K-Tm7@I(Ppq6uMu$4`Pg1ivR2G+IrhKuJE(}#ez;D z8(30%@rXU=8j%$YOaK}VNiwVRd=ObmC#OBHuq_Y`*MWodR(<5f{(RO(kjP()Y!_LZjT21Z0vy$8Xz?49f^n32814H*G^0Dh z%JyjW+h42 zq{7?eX>BX_THS41+3#FmzH|Hj`t60g3-L)TYwgbzX{!tLgyUix3^X`iGmhZ@OUmVN zTsaXb&{g(ZW~o&8vcEoM#IHJY3b(LT4{C7TQrvB zR+o>3i%ZHS`EnTkeUY_Y_}?!-5!R{wk*+j0A|fd`6a|CZjE{QL(3?r|YaE>!k6yGJ zJpxM_9<9!U&KRa+vxz$aC6lk4Wk@2(eA;BX-*vn+t~D`GawxzGXI{_2^h(N|u~iL) zS{`23SXkaD1&}^G*KZ@VY!PHm@+^)tQ41dK6XLfgFb#A2c;L*!RC80A@J*7N<7ics zhJ-84Rw+WXL->Xo-Q+auJ&Dq*_v}1qmhEHOu{|gBhyja(ae!$7YB;I31B+nSacb#` zxo@_Ox|w#{j%!r+(%kkBuj(R~R8^C+*)yked)Ugb(v^Z?htf-n92VLSSMs;$+8j^| z=qA)oo`2C5;6FPsVi@%V%6AVvJENhH7B(3bgGtJ?R_osk>&H>h5)EI2X^mj;7YHmI}ryQT7do1KZk^VH!EiHl9(hyJw=K zyxIt}6O#VVDg*caW2+Tn6r!r9paIIFuMki1ZiP@9m+G`C8!6MOuiQNvB`?AW#vNl zti_W;d&$)rN)Q|*kcuF(_!kT&6LaSTkti*C@Zz`VXuyHw_*oyQgM^B3GkqUZF(`EH-(#6|%);H7z{rdg7WxJ(9u_SLdSK@1!%_^>9D%sHf`tqhw zgpOR``|O2vT*T(vRcWo82~o>mm25=dH9qx%)ejIy8VZ&IbG*h@D{Yn<3s=`+EMTkc z{E^78O>ymn-jutqMgNbAhw&$$Dti-DY(1`}n6-TKDLu3HjJ&Q{Rhf;R?-|b5}wxtU3z?WNICh z5P+rf%^+Y}^D=^!m*5TzOJp$m5j4JB?*~3)6x4sYK1O5kt7$Ecz8Mi3cXc$%xF7=v zFm0yTd+Phi>O8G-pF!v+!wo3a#tbd(%nV}!I;SHHm^n{<1A)!w6$xfpxFx7V9XDs) zK4RxOXsorEcV&|XU;!t z4l~{t58U7<0CAYf_^8EQJ3KBJ#88yRi(H*l>f9am161VZ<}A0n?J<3_+8~?T5Uw?0#5@rsvc`uJAT( z(p0qS&HDXPD_452v2?f5*tb_pyIk4sYaxeFtoy}D!QWsK@x#~LL389M8GC38@FhNt zq%38PAD-BPCb9kOfznyFF!h0J0<3$9u`5ix-sf?`^}ZTFG}JU8*tyss)^waboI6`V zqAxTW@M&N)c#h2ynXm{G?CgSttXenSwE7EL@JH43LS8#-o1bMy2`%J|xq&w@JJIoH zLHCIZY-GvQ7`jT#_M>qQ9WVmYtM66`qGEz!Exdpx)F{jKMQk@aKk{u+zk zf#C>a;defrXZrNTp>Hs4C*v#$B_fFf|M;p}>3T>-+dK?GFrEIs+cx*3??=u9Sfnu4 z8Y@9~SFp_kVT&#<%|xJ&GP?&eVjO;pI+M<(*@l_(!JIII$V@Y0>>byrnv14)V4C)v zph)77B39Z?*PRnIcz)r+1>h)L5C~fA2ez?q)eVL``9YGvNUh~2)FqRoV!Uej2dOpa zeY_uk|3T>b0SDa*0ZqmE;u>wwjv^in1a9Iukrku5R|hiKwK?J9ZWjMu5voqpX~VUe z#lNZfC6*5uhn0QW{J^||l#zzEH#DCw1@=fFx zLYQCZJq_FR;E+rrGyvc<(P>Az;nkK=hrZVTEJIjBCsF#)2x8;wgvK9S(mOX}yV$ZXR%fGQgIVF(-8T!sgmS&!*JcBr0NbOLHPZ6KDd!Ih_^TLVG`?5FGj76eu^R6rV84t>+gVS#^j;{ z1}KdqFpP{ON9pb&BEI3+TE-C9xuMwgKS>yxAPi|JyIc-uv`lKL>*bmP>1 z;pG~>xDuKr^*=W|5c+(K)(^C%1F+H9TXdbUSH505&v~OeciU$!q+5nvQ@}>80cFp!t45T&M)-c3r*aGEv;z`>S z%;<^{HBA#wcYMfB;Xsx&DPb1|v&B?6MK$i(A1HpE{W{s16{BO~M&LBQnN~Ek8V*}c zyYp@siu*v#IPLl@t_9A*FN0n}&StV%ImI!sgKOwb4i=q5=wk#)U^o54Q1NvQuR%51 zgp(arg8m1QrEvf=*-e4nd^fe8%jmi$%33DBElU|N#EWoH)or{}lvTK0$S7LAp=6}I zx|YurG(}WJMWnk$4e!#s_dvPYhTYk9+AR=9JZ&1Dc_|Hr%8CGof}FdY5_YX7^uHH# z68;^Lr6Jx1eNY>8)&@zGii{-N>`;{M@h{oVuw3w0k^Bi&Oekk;W|0)H0X-JHG*STNriGt%8szNVT%sd$@ss(SKb^&4(_grF>ok>Zat>0m^s$p-?U^Y>?{t=aZ)u z2CXUJ4!;w&et4Xk=Vz0z%rCkaKhsU?vKFt-&D%YYpRJSQuT%5=^C@&pDA^cO z-`}n+t8$``DsC(Y#f=}IE(vL2srUQRicnf9_I~r@Q|PpYKAU{)Sy%)}Trkho)m*&t zi_pgdJuyuGbCABs<>af_>0(@zW2W_Ng6Z!OT2gAB9VcIxnq^?*7IIoocKjHt_>m#F z{_cg;eC)>L>$h*!OefmGu3lT-VBS#7)3T%G73ga5=Ge7~yY+{;;S)xB+Ol_@q@_LB zKF4==h5+iv-%I1nYvhEmZ@|;+G&*TGeZg@EZCqZcS+hK+ou(88gUc+R9RC4P_$W}7 zFl^_9tCjf@Wr$@Vt);qgb+$5pwbCAH3~x4i-wrLRj%%~OvN4YS_+xm1rOXrIM-*3I zCLp0Qy!8Cvo;d0&>`VTSx+llq3PmQcA}Bdl_$G9_b>Z57zeE|X43ru-AvDp&oq=q` zgg|Ya^bF*sJEXcR7NMfKjoz0o+cKa?vn!kF#qDCbd~FR^#J+F^yWV?zB*^)+hKq7~ zW)uN{nn=Viu(Z$o?R`LFXzCUguTz4wa)a+RxHBt^@pd6-{iC4E*csH&u5sKRyQa8B@_LI13)r<4t1^LGlXrtMQTzIzHU z68kmShh;engO4odg}zvk{Lq$;8x6w>H}`HAcWV@#-s3C0CKvF}@Mh&R&>BhAvf3$C zUv3#zll{tbjE-a2%>DB7IzB)otx*O1=SyLcGJkLoAP?pFQwnT-C@dn%UfHt!9iJFq zehnMq(JN_xnQ27vOCVV-p<_5GBOrV=is-WtMCD`4-{InCth7Pvx;(zU3R;X4IE=hS z*~D5se@1&szd=Zd7mw)tsXbImPgk%(-EugKTSN(QIDn(;xLsG0lle<{4!uo$AuY)~ zhh8FYo+ybhffZr0D8ft#{^j9X7s1kpd%6gAEF4u4>@_&bA`F@$4CA7lRWwo0!vjqf zH9V?0Q7hy`o&ITJR)(Xd!!sycm&L3CAHa9=@HSHdGaQj$Cxw}1n) z(lmE>t;1I;g1YIST*!dpm~(N*bKC(D0OR;Ilb%W{>~f~sw5od{-zLxVhHrvN)D(q^ zQ#*ne4cls=hw1}AU0oR$=2H@S?H-82Oa~+!QzS;bm+>|6S#)=iieaI2g5Myv{vUB0 J(Z?#N0sy>*YViO7 literal 0 HcmV?d00001 diff --git a/docs/data-pipeline.md b/docs/data-pipeline.md index b88c711..c842ed8 100644 --- a/docs/data-pipeline.md +++ b/docs/data-pipeline.md @@ -21,7 +21,7 @@ runs are idempotent: files already present are skipped. ```bash npm run crawl:2016 # crawler/internal/sources/source_2016.go npm run crawl:2017 # crawler/internal/sources/source_2017.go -npm run crawl:2017 -- --list # show what would be downloaded, fetch nothing +npm run crawl:2017 -- --list # read the article, list the files, download none ``` **2017** comes from the article @@ -29,19 +29,36 @@ npm run crawl:2017 -- --list # show what would be downloaded, fetch nothing and its CDN is still serving the files. **2016** comes from the aggregator article -`cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html`. The site -that originally published it (`dtntbacgiang.edu.vn`) no longer resolves, so the -link list was read from the Internet Archive's copy of the page. Two caveats -that matter: +`cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html`, served from +a mirror — the site that first published it (`dtntbacgiang.edu.vn`) no longer +resolves. -- **The list is verified; the host is not.** All 119 filenames match `data/2016/` - exactly in both directions, which is asserted by a test. But the Archive - captured only the article, not the spreadsheets, so the URLs themselves could - not be confirmed by fetching. `baseURL` in `source_2016.go` points at a mirror - of the same article that is still online; if it serves the files from a - different directory, `uploadPath` is the one constant to change. -- **`data/2016/` is still the only confirmed copy.** Do not delete it on the - assumption that a crawl can restore it. +That mirror is **not reachable from every network.** It resolves to a Vietnamese +address that times out from at least some hosts abroad, in which case the crawl +stops with a connection error before downloading anything. `data/2016/` is +therefore still the only confirmed copy: do not delete it on the assumption that +a crawl can restore it. + +## How a source is defined + +A source carries no link list. It names the article that published the files and +says how to name what it finds there: + +| field | meaning | +| --- | --- | +| `Article` | the page to read links from | +| `Exts` | which file extensions to pick out of it | +| `WantFiles` | how many links to expect — fewer aborts the crawl | +| `Dest` | the local filename for one link | + +`internal/article` does the fetching and HTML parsing; `internal/fetch` does the +downloading. Because `Article` is read at run time, `--list` needs network access +too. + +`WantFiles` exists because a partial crawl is otherwise silent: go-parser will +build a short database from whatever files are present, and only the row-count +guard would notice, after the fact. A page that changes shape stops the crawl +instead. ### Local filenames are load-bearing @@ -56,13 +73,14 @@ Each source therefore pins its local names, and committed `data//` in both directions — every name it would write exists, and every file present is accounted for. -2017 derives its names from the province (the CDN's own names are inconsistent: +2017 derives its names from the province name in the link text, transliterated +to ASCII (the CDN's own names are inconsistent: `Angiang.xls`, `1BaRiaVungTau.xls`, `23HaiPhong.xls`). 2016 keeps the server-assigned names verbatim, since it did not choose them. ## Source Excel shapes -The 2017 datasets share one layout: +2017 has one layout: | Col | Name | Content | | --- | --- | --- | @@ -188,8 +206,12 @@ repo was unified (a hardcoded Windows path in one, an undeclared The latter two are superseded by `differential-parity.mjs`, which compares more and cannot silently skip a dataset. -`crawl-baotintuc.js` was not dropped but rewritten in Go as -`crawler/internal/sources/source_2017.go`, keeping the same link list and the same -local filenames. The rewrite added a per-file timeout and made downloads land on -a `.part` file first — writing straight to the destination meant an interrupted -run left a truncated file that the skip check would then skip forever. +`crawl-baotintuc.js` was not dropped but rewritten as the Go `crawler/` module, +producing the same local filenames. Two changes beyond the port: + +- It carried its 63 links as a hardcoded array. The Go version reads them from + the article instead, so the list cannot drift from what was published. +- Downloads land on a `.part` file and are renamed on completion. Writing + straight to the destination left a truncated file after an interrupted run, + and since the skip check only tests for a non-empty file, every later run + would skip it — the corruption was permanent and silent.