mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-13 10:16:05 +00:00
refactor(crawler): read the file lists from the source articles
Both sources carried their download links as a hardcoded array, which is not a crawl: the lists could drift from what the articles actually published, and nothing would say so. A source now names the article and how to name what it finds there, and internal/article reads the links out of that page at run time. 2016 takes its filenames straight from the URL. 2017 cannot — the CDN names are inconsistent (Angiang.xls, 1BaRiaVungTau.xls, 23HaiPhong.xls) — so it derives them from the province in the link text, transliterated to ASCII the same way go-parser builds ho_ten_ascii. Filenames stay load-bearing: go-parser sorts inputs bytewise and inserts last-wins, so they decide which row survives a duplicate exam number. Saved copies of both articles are committed as fixtures, and a test asserts that reading them and applying each naming rule reproduces data/<id> exactly, in both directions. Resolve also rejects a page that yields the wrong number of links or two links that would write the same file, since either silently costs the dataset files that only the row-count guard would notice afterwards. Verified against the live 2017 article: a from-scratch crawl of all 63 files leaves the committed data unchanged.
This commit is contained in:
1 parent
ceb694a747
commit
b04f9844f9
15 files changed
+653
-415
No files matched your search
@@ -33,7 +33,9 @@ jobs:
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: '1.26'
|
||||
cache-dependency-path: go-parser/go.sum
|
||||
cache-dependency-path: |
|
||||
go-parser/go.sum
|
||||
crawler/go.sum
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
|
||||
Reference in new issue
Block a user