refactor(parser): remove the Rust crate now that Go is at parity

The Go parser has matched the Rust one field-by-field across all four datasets,
so the Rust crate is retired and CI builds the Go binary instead.

crawl-baotintuc.js moves to go-parser/scripts/ — it is the only mechanism for
refreshing data/2017 and has a documented runbook. check-duplicates.js and
diff-datasets.js are dropped: both had been broken since before the repo was
unified, and neither had a caller. db-stats.js and verify-parity.js are dropped
as superseded by differential-parity.mjs, which compares more and cannot
silently skip a dataset.

The reader-fidelity oracle is kept and marked frozen. It was produced by the
Rust reader so it can no longer be regenerated, but it still fails if any single
cell of any of the 299 input files reads differently.

Source comments cite the original Rust by file and line; those paths resolve at
tag pre-go-parser-removal, recorded in go-parser/README.md.
This commit is contained in:
tiennm99 committed 2026-08-13 20:38:25 +07:00
1 parent 0eb174721d
commit 00a08d5fab
42 files changed
+112 -4790

No files matched your search

+30 -20
View File
@@ -2,9 +2,9 @@
From raw Excel files to a compressed SQLite file the browser can load.
One Rust binary (`parser/`) builds every dataset. What differs per dataset is
One Rust binary (`go-parser/`) builds every dataset. What differs per dataset is
parse rules only — sheet strategy, column layout, validation guards — declared
in `parser/configs/<id>.yml`. The table shape, the INSERT and the subject
in `go-parser/configs/<id>.yml`. The table shape, the INSERT and the subject
regexes are canonical and live in `go-parser/internal/schema/schema.go`.
## Sources
@@ -19,7 +19,7 @@ regexes are canonical and live in `go-parser/internal/schema/schema.go`.
Only `2017` can be re-fetched:
```bash
node parser/scripts/crawl-baotintuc.js
node go-parser/scripts/crawl-baotintuc.js
```
Idempotent — skips files already present, saves to `data/2017/`. Source article:
@@ -116,34 +116,44 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what
## Verifying a rebuild
`parser/scripts/db-stats.js` dumps row counts, per-column non-NULL counts, file
size and a deterministic student sample as JSON.
`parser/scripts/verify-parity.js` diffs two such files and exits non-zero on any
mismatch.
`npm run build:db` verifies itself: each database's row count must match the
figure in the table above, and each `.db.gz` must be at least 90% of its usual
size, or the build fails rather than publishing. That guard is the reason a
truncated dataset cannot reach the site with a green pipeline.
For a deeper check, `go-parser/scripts/differential-parity.mjs` compares two sets
of databases field-by-field — row counts, per-column non-NULL counts, a
full-table SHA-256 over every row ordered by `so_bao_danh`, schema metadata, and
build stdout:
```bash
node parser/scripts/db-stats.js 2016=<path>.db … > current.json
node parser/scripts/verify-parity.js plans/reports/parser-parity-baseline.json current.json
node go-parser/scripts/differential-parity.mjs \
--rust /path/to/a-{id}.db --go /path/to/b-{id}.db
```
The committed baseline was built with the pre-refactor code and cannot be
regenerated — the two old crates no longer exist. Both scripts use the built-in
`node:sqlite`, so they need no dependencies.
It exits non-zero on any mismatch and fails loudly if a dataset is missing rather
than skipping it. Written for the Rust-to-Go migration, it works for any two
builds. Uses the built-in `node:sqlite`, so it needs no dependencies.
`go-parser/internal/reader` additionally carries a frozen oracle of per-file
cell-dump hashes covering all 299 inputs; `npm run test:go` fails if any single
cell of any input file reads differently.
## Refreshing the 2017 data
```bash
rm data/2017/*.xls
node parser/scripts/crawl-baotintuc.js
node go-parser/scripts/crawl-baotintuc.js
node go-parser/scripts/build-db.js 2017
```
Then re-run the parity check above and confirm the row count still matches.
The row-count guard in `build:db` confirms the rebuild matches the expected total.
## Legacy scripts
## Removed scripts
`parser/scripts/check-duplicates.js` and `diff-datasets.js` are one-off audits
that were already broken before the repo was unified — a hardcoded Windows path
in one, an undeclared `better-sqlite3` dependency and stale paths in the other.
Each carries a comment saying so. For comparing two builds, use `db-stats.js`
plus `verify-parity.js` instead.
`check-duplicates.js`, `diff-datasets.js`, `db-stats.js` and `verify-parity.js`
were dropped with the Rust parser. The first two had been broken since before the
repo was unified (a hardcoded Windows path in one, an undeclared
`better-sqlite3` dependency in the other) and neither had any automated caller.
The latter two are superseded by `differential-parity.mjs`, which compares more
and cannot silently skip a dataset.
+1 -1
View File
@@ -56,7 +56,7 @@ up as a blank page with 404s on `/assets/...`.
## Adding a dataset
1. Put the Excel files in `data/<id>/`
2. Add `parser/configs/<id>.yml` with the parse rules — sheet mode, column
2. Add `go-parser/configs/<id>.yml` with the parse rules — sheet mode, column
indices, SBD validation, header tokens, blank-row stripping. No SQL: the
schema is canonical and lives in `go-parser/internal/schema/schema.go`
3. Add an entry to `DATASETS` in `src/datasets.js`
+1 -1
View File
@@ -31,7 +31,7 @@ data/<id>/*.xls(x)
One identifier ties the whole pipeline together:
```
data/2017-old/ → parser/configs/2017-old.yml → db/2017-old.db.gz → /thptqg/2017-old/
data/2017-old/ → go-parser/configs/2017-old.yml → db/2017-old.db.gz → /thptqg/2017-old/
```
`src/datasets.js` declares the four ids once. The frontend, the database build