tiennm99
6ff2ed99ec
The repo held two near-duplicate projects. 2016/ and 2017/ each carried their
own React frontend, their own copy of the same Rust crate, and their own
package manager setup. 2016/tools/sync-from-thptqg2017.sh existed purely to
copy the parser source between them.
New layout:
index.html + src/ the 2017 frontend, now the only one
data/<id>/ 2016, 2017, 2017-old, 2017-old2
parser/ the single Rust crate, configs renamed to <id>.toml
docs/ both projects' docs, 2016 copies suffixed -2016-legacy
pending the merge pass
<id> is now one identifier end to end: data/<id>/ feeds parser/configs/<id>.toml
and produces db/<id>.db.gz.
pnpm gives way to npm. pnpm-workspace.yaml existed only to whitelist
better-sqlite3's native build, which npm permits by default, so it has no
equivalent and is simply gone. Lockfiles cannot be converted; package-lock.json
is generated fresh. The migration direction is safe — pnpm's strict layout
forbids phantom dependencies, so anything that resolved under pnpm resolves
under npm's flat tree.
Adds parser/scripts/build-db.js and src/datasets.js: the four dataset IDs are
declared once and read by both the build tooling and (from the next phase) the
frontend.
Follow-on fixes the move made necessary:
- eslint's Node-globals override pointed at scripts/, now parser/scripts/
- crawl-baotintuc.js wrote to <root>/data, now data/2017
- golden tests loaded configs by their old thptqg*-data.toml names
Drops the #[ignore]d Rust-vs-Node golden test. It shelled out to pnpm to run
scripts/build-database.js, a file removed when the parser was ported to Rust,
so it could never pass. check-duplicates.js and diff-datasets.js were already
broken before this change and are annotated as such rather than half-fixed.
63 Rust tests pass and clippy is clean from the new location.
4.2 KiB
Data Pipeline
From raw Excel files to a compressed SQLite file the browser can load.
Canonical source (data/)
63 Excel files from baotintuc.vn CDN. Source article:
https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm
Re-download anytime:
node scripts/crawl-baotintuc.js
Idempotent — skips files already present. Saves to data/<ascii-kebab-province>.xls.
Reference sources (data-old/, data-old2/)
Both were collected from Vietnamese news sites around the time of the 2017 exam.
Specific publisher URLs were not recorded and cannot be recovered, so these
datasets are not reproducible — the files committed in git are the only
copy. They are kept for historical comparison and to expose the deploy matrix;
the main deployment always uses the reproducible data/ source.
Source Excel shape
Every file is a single sheet (for data-old/) or up to 3 sheets (for data/ .xls) with columns:
| Col | Name | Content |
|---|---|---|
| 0 | HO_TEN | full name in Vietnamese |
| 1 | NGAY_SINH | dd/mm/yyyy |
| 2 | SOBAODANH | 8-digit string, first 2 digits = province |
| 3 | DIEM_THI | concatenated per-subject score string in the source language, e.g. "Toan: 6.80 Van: 5.25 ... English: 5.80" |
Some files lack a header row. isHeaderRow() in build-lib.js detects and skips.
Score text parsing
SCORE_PATTERNS in build-lib.js defines one regex per subject. The patterns must match the exact subject labels the source files use (Vietnamese).
Subjects supported (DB column / label in source text):
| Column | Source label | English |
|---|---|---|
| toan | Toán | Math |
| ngu_van | Ngữ văn | Literature |
| vat_ly | Vật lí | Physics |
| hoa_hoc | Hóa học | Chemistry |
| sinh_hoc | Sinh học | Biology |
| khtn | KHTN | Natural Sciences combined |
| lich_su | Lịch sử | History |
| dia_ly | Địa lí | Geography |
| gdcd | GDCD | Civic Education |
| khxh | KHXH | Social Sciences combined |
| tieng_anh | Tiếng Anh | English |
| tieng_phap | Tiếng Pháp | French |
| tieng_nga | Tiếng Nga | Russian |
| tieng_trung | Tiếng Trung | Chinese |
Missing-in-string → NULL in DB (student didn't take that subject).
Why three parsers
One shared lib + three thin parsers, not one parser with flags. Each dataset's quirks stay visible in its own file:
| File | Source dir | Sheet strategy | Extra guard |
|---|---|---|---|
build-database.js |
data/ |
iterate ALL sheets (.xls row overflow) | — |
build-database-old.js |
data-old/ |
sheet 0 only | reject rows where so_bao_danh isn't digits-only (rejects an HO_TEN → SOBAODANH header leak present in this export) |
build-database-old2.js |
data-old2/ |
iterate ALL sheets (HCM overflow) | skip fully blank rows before counting |
Overflow-sheet gotcha
The old .xls binary format caps at 65,536 rows per sheet. Hanoi and Ho Chi Minh City exceed that in the baotintuc export, so Excel auto-splits the overflow into Sheet2. Only reading Sheet1 silently drops 13,720 students (Hanoi +7,275, HCM +6,445).
The audit node scripts/audit-row-counts.js catches this by comparing source-row-count (across all sheets) to DB-row-count.
Audits
Run any time after a rebuild:
node scripts/audit-row-counts.js # source rows vs DB rows (zero-loss check)
node scripts/check-duplicates.js # md5 + row-content duplicates in data/
node scripts/diff-datasets.js # compare two DBs (needs backup/ dir)
Expected baseline:
| DB | Source rows | Skipped | DB rows |
|---|---|---|---|
| main | 861,131 | 63 empty rows | 861,068 |
| old | 847,349 | 1 header-leak | 847,348 |
| old2 | 679,764 | 0 | 679,764 |
Refreshing the data
rm data/*.xls # clear current snapshot
node scripts/crawl-baotintuc.js # re-download from CDN
npm run build:db # rebuild
node scripts/audit-row-counts.js # verify
gzip -kf -9 public/thptqg2017.db # compress for shipping
Adding a new source dataset
See docs/deployment-guide.md → "Adding a new variant".