The repository now reads as the pipeline it is: crawler fetches, parser
converts, assembler verifies and publishes, with data/ and web/ as the stores
they hand work through. go-parser is renamed parser now that there is no other.
The assembler replaces build-db.js and assemble-site.js. It compiles the
parser, builds and verifies each database, compresses it, runs the Vite build
and assembles _site — one command, and the only place that knows the order.
It also closes a real hole: nothing previously asserted that a database reached
the site. An empty staging directory assembled happily, so every page rendered,
every query 404d and CI stayed green. The row-count and size guards could not
catch that, since they only run when a database was built at all.
Removing Node from the root forced the dataset list out of web/src/datasets.js,
which the assembler cannot import. datasets.json is now the registry both sides
read — JSON because Go and the browser both parse it without a dependency —
while presentation stays in the web app, keyed by id and cross-checked against
the registry so a half-added dataset fails instead of half-working.
Guards verified by making each one fail: a missing database, and an expected
row count one higher than the truth.
Both sources carried their download links as a hardcoded array, which is not a
crawl: the lists could drift from what the articles actually published, and
nothing would say so. A source now names the article and how to name what it
finds there, and internal/article reads the links out of that page at run time.
2016 takes its filenames straight from the URL. 2017 cannot — the CDN names are
inconsistent (Angiang.xls, 1BaRiaVungTau.xls, 23HaiPhong.xls) — so it derives
them from the province in the link text, transliterated to ASCII the same way
go-parser builds ho_ten_ascii.
Filenames stay load-bearing: go-parser sorts inputs bytewise and inserts
last-wins, so they decide which row survives a duplicate exam number. Saved
copies of both articles are committed as fixtures, and a test asserts that
reading them and applying each naming rule reproduces data/<id> exactly, in both
directions. Resolve also rejects a page that yields the wrong number of links or
two links that would write the same file, since either silently costs the
dataset files that only the row-count guard would notice afterwards.
Verified against the live 2017 article: a from-scratch crawl of all 63 files
leaves the committed data unchanged.
tiennm99