refactor(crawler): read the file lists from the source articles

Both sources carried their download links as a hardcoded array, which is not a
crawl: the lists could drift from what the articles actually published, and
nothing would say so. A source now names the article and how to name what it
finds there, and internal/article reads the links out of that page at run time.

2016 takes its filenames straight from the URL. 2017 cannot — the CDN names are
inconsistent (Angiang.xls, 1BaRiaVungTau.xls, 23HaiPhong.xls) — so it derives
them from the province in the link text, transliterated to ASCII the same way
go-parser builds ho_ten_ascii.

Filenames stay load-bearing: go-parser sorts inputs bytewise and inserts
last-wins, so they decide which row survives a duplicate exam number. Saved
copies of both articles are committed as fixtures, and a test asserts that
reading them and applying each naming rule reproduces data/<id> exactly, in both
directions. Resolve also rejects a page that yields the wrong number of links or
two links that would write the same file, since either silently costs the
dataset files that only the row-count guard would notice afterwards.

Verified against the live 2017 article: a from-scratch crawl of all 63 files
leaves the committed data unchanged.
This commit is contained in:
tiennm99 committed 2026-08-13 22:05:07 +07:00
1 parent ceb694a747
commit b04f9844f9
15 files changed
+653 -415

No files matched your search

+3 -1
View File
@@ -33,7 +33,9 @@ jobs:
- uses: actions/setup-go@v5
with:
go-version: '1.26'
cache-dependency-path: go-parser/go.sum
cache-dependency-path: |
go-parser/go.sum
crawler/go.sum
- uses: actions/setup-node@v4
with: