refactor(parser): reimplement the parser in Go alongside the Rust crate

Adds go-parser/, a Go reimplementation of the xlsxread parser, verified
byte-for-byte against the Rust original before any cutover.

Reader fidelity is exact across all 299 input files: the canonical cell dump
of every sheet matches calamine's, locked in as a test against a committed
hash oracle. Reaching that required replacing extrame/xls, which corrupted
69% of cells and dropped a further 28% on the BIFF corpus, with pbnjay/grate;
correcting excelize's number-format application and trailing-cell trimming;
restoring carriage returns that XML line-ending normalisation strips from
2,233 ten_cum_thi values; and gating numeric re-rendering on cell type so
shared strings that merely look numeric keep their leading zeros.

The differential gate compares both parsers over all four datasets:
3,265,641 rows with identical full-table SHA-256, identical per-column
non-NULL counts, identical schema metadata and identical stdout.

Config moves from TOML to YAML for both parsers, so they keep reading the
same files and the gate stays meaningful. Verified by rebuilding 2016 and
2017-old2 with Rust under the new configs and matching the recorded counts.

build-db.js now refuses to publish a database whose row count does not match
the known figure, closing a path where an under-producing parser could ship a
truncated public dataset with green CI. The deploy workflow gains a
pull_request trigger and guards deploy to main, so branch verification can no
longer publish to production.
This commit is contained in:
tiennm99 committed 2026-08-13 20:27:44 +07:00
1 parent 8902232747
commit 0eb174721d
60 files changed
+6050 -189

No files matched your search

+5 -5
View File
@@ -4,8 +4,8 @@ From raw Excel files to a compressed SQLite file the browser can load.
One Rust binary (`parser/`) builds every dataset. What differs per dataset is
parse rules only — sheet strategy, column layout, validation guards — declared
in `parser/configs/<id>.toml`. The table shape, the INSERT and the subject
regexes are canonical and live in `parser/src/schema.rs`.
in `parser/configs/<id>.yml`. The table shape, the INSERT and the subject
regexes are canonical and live in `go-parser/internal/schema/schema.go`.
## Sources
@@ -54,7 +54,7 @@ which is why only 2016 populates those columns.
## Score text parsing
`SCORE_PATTERNS` in `parser/src/schema.rs` defines one regex per subject, and
`SCORE_PATTERNS` in `go-parser/internal/schema/schema.go` defines one regex per subject, and
**all 16 run against every dataset**. A subject a given exam year did not offer
simply never matches and stays NULL.
@@ -110,7 +110,7 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what
| id | Source rows | Skipped | DB rows |
| --- | --- | --- | --- |
| `2016` | 877,464 | 3 duplicate SBDs collapsed | **877,461** |
| `2017` | 861,131 | 63 empty | **861,068** |
| `2017` | 861,068 | 0 | **861,068** |
| `2017-old` | 847,349 | 1 header leak | **847,348** |
| `2017-old2` | 679,764 | 0 | **679,764** |
@@ -135,7 +135,7 @@ regenerated — the two old crates no longer exist. Both scripts use the built-i
```bash
rm data/2017/*.xls
node parser/scripts/crawl-baotintuc.js
node parser/scripts/build-db.js 2017
node go-parser/scripts/build-db.js 2017
```
Then re-run the parity check above and confirm the row count still matches.
+6 -6
View File
@@ -7,8 +7,8 @@ One-time setup: **Settings → Pages → Source: GitHub Actions**.
## What the workflow does
1. Checkout, Rust toolchain, Node 24, `npm ci`
2. `npm run build:rust` — one parser binary
1. Checkout, Go toolchain, Node 24, `npm ci`
2. `npm run build:go` — one parser binary
3. `npm run build:db` — builds and gzips all four databases into
`.build/public/db/`
4. `npm run build:site` — one Vite build, then `scripts/assemble-site.js`
@@ -35,7 +35,7 @@ string intact.
```bash
npm ci
npm run build:rust
npm run build:go
npm run build:db # all four; pass an id to build just one
npm run build:site # vite build + assemble into _site/
npx serve _site
@@ -44,7 +44,7 @@ npx serve _site
To rebuild a single dataset:
```bash
node parser/scripts/build-db.js 2017-old
node go-parser/scripts/build-db.js 2017-old
```
## Base path
@@ -56,9 +56,9 @@ up as a blank page with 404s on `/assets/...`.
## Adding a dataset
1. Put the Excel files in `data/<id>/`
2. Add `parser/configs/<id>.toml` with the parse rules — sheet mode, column
2. Add `parser/configs/<id>.yml` with the parse rules — sheet mode, column
indices, SBD validation, header tokens, blank-row stripping. No SQL: the
schema is canonical and lives in `parser/src/schema.rs`
schema is canonical and lives in `go-parser/internal/schema/schema.go`
3. Add an entry to `DATASETS` in `src/datasets.js`
Nothing else. The build script, the site assembly and the router all read that
+3 -3
View File
@@ -31,11 +31,11 @@ data/<id>/*.xls(x)
One identifier ties the whole pipeline together:
```
data/2017-old/ → parser/configs/2017-old.toml → db/2017-old.db.gz → /thptqg/2017-old/
data/2017-old/ → parser/configs/2017-old.yml → db/2017-old.db.gz → /thptqg/2017-old/
```
`src/datasets.js` declares the four ids once. The frontend, the database build
(`parser/scripts/build-db.js`) and the site assembly all import that list, so
(`go-parser/scripts/build-db.js`) and the site assembly all import that list, so
adding a dataset means adding one entry and one config file.
| id | Exam | Rows | Source |
@@ -47,7 +47,7 @@ adding a dataset means adding one entry and one config file.
## Canonical schema
Defined once in `parser/src/schema.rs` — DDL, INSERT, column order and the 16
Defined once in `go-parser/internal/schema/schema.go` — DDL, INSERT, column order and the 16
subject regexes. The four TOML configs carry no SQL at all, only per-dataset
parse rules. Config parsing uses `deny_unknown_fields`, so a leftover `[schema]`
block fails loudly instead of looking effective while `schema.rs` drives the