The key is hashFiles over data/**, parser/** and datasets.json, so it
invalidates by path. Both the guide and the workflow comment described it by
intent instead — "a web or docs change restores the databases" — which the
previous commit disproved by paying for a full 348 MB rebuild to fix a comment
in datasets.json and a sentence in parser/README.md.
Over-invalidating is the safe direction, and narrowing the globs would mean
remembering to extend them for every future path that can change a database.
Record the trade rather than making it.
Two caches, one on each side.
In the browser, the response is stored in Cache Storage keyed by the
server's ETag, so the transfer is paid once per device rather than once
per visit. A stored copy opens without the gate: consent was given the
first time and reuse costs no network. A redeploy changes the ETag, so
the new version replaces the old instead of answering with last week's
data, and every older version of that database is dropped so one dataset
never occupies the disk twice. When the server cannot be reached at all,
any stored version is used, which incidentally makes the site work
offline. Storing is best effort — a full disk means downloading again
next time, which is no reason to fail the page.
The response is cached from a clone while the original is read for the
progress bar, rather than buffering the file a second time in the one
place where memory is already the binding constraint.
In CI, the built databases are cached on the inputs that determine them:
data/**, parser/** and datasets.json. Parsing 348 MB of spreadsheets is
the slow part of the job, and a web or docs change cannot alter a
database, so those pushes restore instead of rebuilding. Exact matches
only — no restore-keys, since a near-miss would publish databases built
from inputs the commit does not describe.
Reading the file where it lay never worked well enough. Two costs were
structural rather than bugs: reads are serial, because the worker uses
synchronous XHR, so a name search touching 390 pages waited 17 seconds
to move 608 KB — roughly one request per result row, which no page size
removes — and the first visitor after each deploy waited ~26 seconds for
the CDN to fill its cache with a 288 MB object.
The browser now downloads the whole database once and queries it in
memory with sql.js. A dataset page is gated behind that: the gate states
what it will cost, in transfer and in memory, and offers only the
download, because there is nothing to show without it.
Dropping the structures that existed to make range-request queries
index-driven halved the file. name_word carried one row per word of
every name, about 3.5 million of them, and with the partial score
indexes it was more than half of what every visitor would now download.
Measured on rebuilt databases: 2016 went 288.6 -> 142.5 MB (31 MB
gzipped on the wire), 2017 237.7 -> 119.3 MB, both with row counts and
audits unchanged. Queries on the result: an exam number is immediate, a
name scans all 877,460 rows in about 240 ms.
Alternatives were measured before choosing this. sqlite-wasm-http sizes
files from a HEAD Content-Length with no override, so on a host that
gzips it silently uses the compressed size. DuckDB-WASM ships 32-37 MB
of WebAssembly before its Parquet extension, more than this whole
download. Static pre-generated shards are the most robust option but
cannot answer arbitrary SQL, and cannot stop early the way LIMIT does.
The published name loses its chunk index, the byte budgets and the SQL
consent modal go with the range reads that made them necessary, and the
docs no longer describe a design the site does not use.
The previous attempt passed fileLength in the inline config, and the
worker discarded it. sqlite.worker.ts builds the lazy file's config
itself and hardcodes:
fileLength: config.serverMode === "chunked"
? config.databaseLengthBytes
: undefined
So in full mode there is no way to supply a length, and the library
falls back to sizing the file with a HEAD request — which GitHub Pages
answers with the gzipped length, and then refuses to use.
Chunked mode is the only mode that takes a length. One chunk holds the
whole database, so the chunk index is always 0 and every request goes to
urlPrefix + "0"; the assembler therefore publishes <id>.sqlite30. The
length still comes from the range probe, which also checks the bytes are
a SQLite header.
dbPrefixOf and dbOf derive one form from the other and are handed to
RemoteDatabase together, so the prefix the library appends an index to
and the file the assembler writes cannot drift apart. A test pins that;
nothing else would catch it, because the symptom is a 404 per query.
The stray-artifact guard now also rejects a leftover <id>.sqlite3, which
after this change is a stale artifact rather than the published one.
sqlite-wasm-http was checked as an alternative and does not help: its
worker takes the size from a HEAD Content-Length too, and its options
expose no way to override it, so on Pages it would silently use the
compressed size. Its shared-cache backend wants COOP/COEP, but it ships
a fallback that does not, so isolation was never the blocker — the
architecture note claiming otherwise is corrected.
The previous attempt passed fileLength in the inline config, and the
worker discarded it. sqlite.worker.ts builds the lazy file's config
itself and hardcodes:
fileLength: config.serverMode === "chunked"
? config.databaseLengthBytes
: undefined
So in full mode there is no way to supply a length, and the library
falls back to sizing the file with a HEAD request — which GitHub Pages
answers with the gzipped length, and then refuses to use.
Chunked mode is the only mode that takes a length. One chunk holds the
whole database, so the chunk index is always 0 and every request goes to
urlPrefix + "0"; the assembler therefore publishes <id>.sqlite30. The
length still comes from the range probe, which also checks the bytes are
a SQLite header.
dbPrefixOf and dbOf derive one form from the other and are handed to
RemoteDatabase together, so the prefix the library appends an index to
and the file the assembler writes cannot drift apart. A test pins that;
nothing else would catch it, because the symptom is a 404 per query.
The stray-artifact guard now also rejects a leftover <id>.sqlite3, which
after this change is a stale artifact rather than the published one.
sqlite-wasm-http was checked as an alternative and does not help: its
worker takes the size from a HEAD Content-Length too, and its options
expose no way to override it, so on Pages it would silently use the
compressed size. Its shared-cache backend wants COOP/COEP, but it ships
a fallback that does not, so isolation was never the blocker — the
architecture note claiming otherwise is corrected.
GitHub Pages serves .sqlite3 as application/octet-stream, which mime-db
marks compressible, so an un-ranged response comes back gzipped with the
compressed length. sql.js-httpvfs sizes a file with a HEAD request, sees
that length is unusable and refuses to open the database:
Length of the file not known. It must either be supplied in the config
or given by the HTTP server.
Page reads were never affected. The Fetch standard requires browsers to
send Accept-Encoding: identity on any request carrying a Range header,
and the live site returns 206 with raw bytes to one. So the length is
probed the same way and passed as fileLength, which is the escape hatch
the library's own message points at.
The probe reads the first 100 bytes, so it also checks the file starts
with the SQLite magic and that its page size matches the request size —
a host that ever compresses a ranged response now fails with a clear
message rather than feeding the library the wrong bytes.
The post-deploy check the docs prescribed could not have caught this: a
bare `curl -sI` advertises no encoding, so it reports success whatever
the host does. It is replaced, in the docs and in CI, by a ranged read
that verifies the bytes.
The group was the literal string "pages", so every run of this workflow
shared one lane regardless of branch, and cancel-in-progress meant the
newest arrival won. A pull-request run therefore cancels an in-flight
deploy of main.
That is not theoretical: the deploy of #10 was killed 3m22s in by a
pull-request run that started after it, and the site quietly stayed on
the previous build. Nothing reported a failure — the PR checks were
green and the deploy showed "cancelled", which reads like something
someone chose.
Keyed by ref, a push still cancels its own superseded run, which is the
case worth cancelling, and a branch can no longer interrupt a deploy.
Also drops "compress it" from the pipeline comment: the databases have
shipped uncompressed since they started being read by range request.
setup-go v7 exports GOTOOLCHAIN=local, so Go no longer downloads the
toolchain a module asks for — whatever setup-go installed has to satisfy
it. `go-version: '1.26'` resolves to the runner manifest's patch, which
is 1.26.5, against modules requiring 1.26.6 since this morning's CVE
bump. The parser tests stopped on that.
go-version-file installs exactly what the modules declare, and removes
the fourth place a Go version was written down.
The runner warns that the node20 action runtime is deprecated. Every
action in the workflow was on it:
checkout v4 -> v7
setup-go v5 -> v7
setup-node v4 -> v7
upload-pages-artifact v3 -> v5
deploy-pages v4 -> v5
The two Pages actions move together because the artifact format is
shared between them.
None of the inputs this workflow passes changed across those majors —
the releases are ESM migrations and the runtime bump. `node-version: 24`
was already the Node the build runs on; this is about the runtime the
actions themselves execute in.
Every .ts becomes .js, every lang="ts" becomes lang-less, and the type
declarations go with them: types.ts held nothing but types, so it is
deleted outright.
Tooling follows. typescript, svelte-check, typescript-eslint and
@types/sql.js are uninstalled; tsconfig.json becomes jsconfig.json, which
still extends the generated SvelteKit config so $lib and $app resolve in
an editor; `npm run lint` is now ESLint alone, and CI's comment about it
covering the type check goes too.
What this gives up, stated plainly: a mistyped column name like
row.nguvan used to fail the build and now renders blank, and the
datasets.json-to-CONTENT cross-check is back to throwing at module load
rather than at compile time. The runtime guard for the latter is still
there and still throws loudly.
Two mechanical notes. The svelte/no-navigation-without-resolve rule
started flagging the footer's source link, which points at an off-site
article — without type information the rule can no longer tell an
external URL from a route, so that one line carries a disable comment.
And Vitest's include pattern had to follow the tests to .js.
Lint, 25 tests and the build all pass.
The app was a single-entry React SPA: one index.html that the assembler
copied to every dataset path, with a hand-rolled router resolving the
dataset from window.location and the title patched in at runtime because
one file had to serve every route.
SvelteKit prerenders a real page per route instead. The entry generator
in routes/[dataset]/+page.ts reads the same datasets.json the assembler
does, so the set of pages and the set of databases cannot drift apart,
and each dataset page ships its own <title> and description.
Everything framework-free moved across unchanged in behaviour and became
typed: the admission blocks, subject list, query classifier and SQL
presets. `Student` now mirrors the 22-column table, so a mistyped column
name fails the build rather than rendering blank.
Tailwind replaces the stylesheet. Theme tokens are CSS variables, which
keeps dark mode a single block of overrides rather than a `dark:` variant
on every class. The score tiers stay hand-written CSS: the class is
chosen at runtime from a score, and no utility generator can see that.
Adds the tests the frontend never had, over the three modules where a
silent wrong answer is possible — most importantly that toAscii here
folds exactly as ToAscii does in the parser, which is what makes
accent-insensitive search find anything.
The assembler stops copying index.html per dataset and checks that the
build prerendered each one instead. The database presence, raw-artifact
and idempotence guards are untouched.
Verified: 25 tests, ESLint and svelte-check clean, and a full
`assemble site` producing /thptqg/, /thptqg/2016/, /thptqg/2017/ and
404.html with absolute asset URLs. Not verified in a browser — this
machine has none.
Comments across the tree justified the code by pointing at a Rust
implementation that is no longer in the repository, citing files and line
numbers (config.rs:132, schema.rs:26-54, reader.rs:42) that cannot be
opened, plus crates and datasets that are equally gone. A reader could not
check any of it.
Every invariant those comments carried is kept and restated so it stands on
its own: the bytewise sort that decides which row survives a duplicate exam
number, the literal U+0300..U+036F range that must match the site's toAscii,
the trailing space in "SINH ", the BIFF and shared-string corrections, the
VACUUM-after-COMMIT rule, the deploy-from-main guard.
The reader's contract is now anchored to the frozen oracle in
parser/testdata, which still exists and is still checked, rather than to the
tool that originally produced it.
TestDDLMatchesRust becomes TestDDLIsFrozen: it compares against a copy of
the DDL inside the test and never read schema.rs, so both the name and the
failure message were misleading.
ToAscii no longer claims the d-replacement must precede lowercasing. Both
cases map to 'd' and ToLower runs last, so the order has no effect.
The pipeline is now Go outside web/. differential-parity.mjs becomes
assembler/internal/verify, reachable as `assemble verify A B`. The port fixed a
real weakness: the JavaScript hashed each row's fields joined bare, so a value
shifted across a column boundary produced the same digest. A test now pins that.
The hub still rendered "Phiên bản cũ của trang 2017" above a permanently empty
list — it split datasets on id.includes("old"), and both such datasets are gone.
The heading and the filter are removed. index.html titled every page "THPT QG
2017", including 2016 and the hub, because one file is copied to every route;
the static title is now neutral and the app sets the dataset's own.
Dead code removed: the isOld2/containsOld branches in the stats block, which
only 2017-old2 could ever reach; SUBJECT_LABELS, DATASET_IDS and the unread
`short` subject field; an unused vite.svg and a favicon link to a file that
never existed; two unused CSS rules and --shadow-sm; site.Paths.Root.
Corrected comments that were confidently wrong rather than merely stale: the
reader claimed to be row-streaming when both implementations decode the whole
workbook into memory first, and the fidelity oracle still spoke of 299 input
files when it covers 182. Candidate counts in the hub now derive from
datasets.json instead of being written a second time as prose.
plans/ is emptied. The parity report it held was cited by docs/data-pipeline.md,
so the evidence that the recovered foreign-language scores are real — not the
citation, the four arguments themselves — is now inline there.
Verified: 2017 rebuilt after the writer change hashes identically to the build
before it.
The repository now reads as the pipeline it is: crawler fetches, parser
converts, assembler verifies and publishes, with data/ and web/ as the stores
they hand work through. go-parser is renamed parser now that there is no other.
The assembler replaces build-db.js and assemble-site.js. It compiles the
parser, builds and verifies each database, compresses it, runs the Vite build
and assembles _site — one command, and the only place that knows the order.
It also closes a real hole: nothing previously asserted that a database reached
the site. An empty staging directory assembled happily, so every page rendered,
every query 404d and CI stayed green. The row-count and size guards could not
catch that, since they only run when a database was built at all.
Removing Node from the root forced the dataset list out of web/src/datasets.js,
which the assembler cannot import. datasets.json is now the registry both sides
read — JSON because Go and the browser both parse it without a dependency —
while presentation stays in the web app, keyed by id and cross-checked against
the registry so a half-added dataset fails instead of half-working.
Guards verified by making each one fail: a missing database, and an expected
row count one higher than the truth.
Both sources carried their download links as a hardcoded array, which is not a
crawl: the lists could drift from what the articles actually published, and
nothing would say so. A source now names the article and how to name what it
finds there, and internal/article reads the links out of that page at run time.
2016 takes its filenames straight from the URL. 2017 cannot — the CDN names are
inconsistent (Angiang.xls, 1BaRiaVungTau.xls, 23HaiPhong.xls) — so it derives
them from the province in the link text, transliterated to ASCII the same way
go-parser builds ho_ten_ascii.
Filenames stay load-bearing: go-parser sorts inputs bytewise and inserts
last-wins, so they decide which row survives a duplicate exam number. Saved
copies of both articles are committed as fixtures, and a test asserts that
reading them and applying each naming rule reproduces data/<id> exactly, in both
directions. Resolve also rejects a page that yields the wrong number of links or
two links that would write the same file, since either silently costs the
dataset files that only the row-count guard would notice afterwards.
Verified against the live 2017 article: a from-scratch crawl of all 63 files
leaves the committed data unchanged.
Move the frontend into web/, the repo's only npm workspace, and replace the
JS crawler with a Go module covering both remaining datasets. The crawler
writes to a .part file and renames on completion: writing straight to the
destination left truncated files that the skip-if-present check would then
skip forever.
Remove the 2017-old and 2017-old2 datasets. They were successive publications
of the same exam, kept side by side so the disagreement stayed inspectable;
the current 2017 supersedes them and they remain in git history.
Recover the 2016 crawler source from the Internet Archive's copy of the
aggregator article, whose original host no longer resolves. All 119 filenames
are verified against data/2016 in both directions, but no archive captured the
spreadsheets themselves, so the host still serving them is unconfirmed and
data/2016 remains the only confirmed copy.
Filenames are load-bearing throughout: go-parser sorts inputs bytewise and
inserts last-wins, so they decide which row survives a duplicate exam number.
Adds go-parser/, a Go reimplementation of the xlsxread parser, verified
byte-for-byte against the Rust original before any cutover.
Reader fidelity is exact across all 299 input files: the canonical cell dump
of every sheet matches calamine's, locked in as a test against a committed
hash oracle. Reaching that required replacing extrame/xls, which corrupted
69% of cells and dropped a further 28% on the BIFF corpus, with pbnjay/grate;
correcting excelize's number-format application and trailing-cell trimming;
restoring carriage returns that XML line-ending normalisation strips from
2,233 ten_cum_thi values; and gating numeric re-rendering on cell type so
shared strings that merely look numeric keep their leading zeros.
The differential gate compares both parsers over all four datasets:
3,265,641 rows with identical full-table SHA-256, identical per-column
non-NULL counts, identical schema metadata and identical stdout.
Config moves from TOML to YAML for both parsers, so they keep reading the
same files and the gate stays meaningful. Verified by rebuilding 2016 and
2017-old2 with Rust under the new configs and matching the recorded counts.
build-db.js now refuses to publish a database whose row count does not match
the known figure, closing a path where an under-producing parser could ship a
truncated public dataset with green CI. The deploy workflow gains a
pull_request trigger and guards deploy to main, so branch verification can no
longer publish to production.
The four build variants existed only because the app could not resolve its own
dataset. Now that it can, vite.config.js is a single build with an absolute
base and no VARIANT switching, and the workflow compiles Rust once, installs
Node dependencies once, and builds the site once — it previously built the same
Rust crate twice and ran two separate pnpm installs.
Because base is absolute, the emitted index.html references /thptqg/assets/...
regardless of where it is served from, so the same file works as an entry point
at any depth. scripts/assemble-site.js copies it to each dataset path and to
the two legacy nested URLs, giving a real static file at every published route.
That is what removes the need for an SPA 404-fallback redirect, which would
otherwise have rewritten URLs and interfered with the ?q= deep links.
Generated databases move to a gitignored .build/public, which Vite consumes as
its publicDir. build-db.js gzips without -k, and the assemble step then refuses
to finish if any uncompressed database artefact reached the output — .db,
.db-journal, .db-wal or .db-shm. Previously a raw 100+ MB database was written
into the source tree and deleted afterwards by an rm in the workflow, so
shipping one was a missing cleanup step away.
Both build scripts import DATASET_IDS from src/datasets.js rather than
repeating the dataset list in workflow shell, so the four ids are declared in
exactly one place across the frontend, the database build and the assembly.
Verified by running the full pipeline locally and serving the artifact over
HTTP: all nine routes return 200, every entry point is byte-identical, asset
references are absolute, and the databases are fetchable. The guard was tested
by injecting a raw .db and .db-journal into the build, which failed the
assembly as intended.
Each year keeps its full pipeline under its directory; vite bases move to
/thptqg/<year>/ (2017 keeps its old and old2 generations), one deploy
workflow builds both years' databases and bundles behind a root index.
tiennm99