refactor: collapse the two projects into one tree and move to npm

The repo held two near-duplicate projects. 2016/ and 2017/ each carried their
own React frontend, their own copy of the same Rust crate, and their own
package manager setup. 2016/tools/sync-from-thptqg2017.sh existed purely to
copy the parser source between them.

New layout:

  index.html + src/  the 2017 frontend, now the only one
  data/<id>/         2016, 2017, 2017-old, 2017-old2
  parser/            the single Rust crate, configs renamed to <id>.toml
  docs/              both projects' docs, 2016 copies suffixed -2016-legacy
                     pending the merge pass

<id> is now one identifier end to end: data/<id>/ feeds parser/configs/<id>.toml
and produces db/<id>.db.gz.

pnpm gives way to npm. pnpm-workspace.yaml existed only to whitelist
better-sqlite3's native build, which npm permits by default, so it has no
equivalent and is simply gone. Lockfiles cannot be converted; package-lock.json
is generated fresh. The migration direction is safe — pnpm's strict layout
forbids phantom dependencies, so anything that resolved under pnpm resolves
under npm's flat tree.

Adds parser/scripts/build-db.js and src/datasets.js: the four dataset IDs are
declared once and read by both the build tooling and (from the next phase) the
frontend.

Follow-on fixes the move made necessary:
  - eslint's Node-globals override pointed at scripts/, now parser/scripts/
  - crawl-baotintuc.js wrote to <root>/data, now data/2017
  - golden tests loaded configs by their old thptqg*-data.toml names

Drops the #[ignore]d Rust-vs-Node golden test. It shelled out to pnpm to run
scripts/build-database.js, a file removed when the parser was ported to Rust,
so it could never pass. check-duplicates.js and diff-datasets.js were already
broken before this change and are annotated as such rather than half-fixed.

63 Rust tests pass and clippy is clean from the new location.
This commit is contained in:
tiennm99 committed 2026-08-13 11:27:01 +07:00
1 parent 82db8295bb
commit 6ff2ed99ec
396 files changed
+2753 -8720

No files matched your search

+110
View File
@@ -0,0 +1,110 @@
# Data Pipeline
From raw Excel files to a compressed SQLite file the browser can load.
## Canonical source (data/)
63 Excel files from baotintuc.vn CDN. Source article:
`https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm`
Re-download anytime:
```bash
node scripts/crawl-baotintuc.js
```
Idempotent — skips files already present. Saves to `data/<ascii-kebab-province>.xls`.
## Reference sources (data-old/, data-old2/)
Both were collected from Vietnamese news sites around the time of the 2017 exam.
Specific publisher URLs were not recorded and cannot be recovered, so these
datasets are **not reproducible** — the files committed in git are the only
copy. They are kept for historical comparison and to expose the deploy matrix;
the main deployment always uses the reproducible `data/` source.
## Source Excel shape
Every file is a single sheet (for `data-old/`) or up to 3 sheets (for `data/` .xls) with columns:
| Col | Name | Content |
|---|---|---|
| 0 | HO_TEN | full name in Vietnamese |
| 1 | NGAY_SINH | `dd/mm/yyyy` |
| 2 | SOBAODANH | 8-digit string, first 2 digits = province |
| 3 | DIEM_THI | concatenated per-subject score string in the source language, e.g. `"Toan: 6.80 Van: 5.25 ... English: 5.80"` |
Some files lack a header row. `isHeaderRow()` in `build-lib.js` detects and skips.
## Score text parsing
`SCORE_PATTERNS` in `build-lib.js` defines one regex per subject. The patterns must match the exact subject labels the source files use (Vietnamese).
Subjects supported (DB column / label in source text):
| Column | Source label | English |
|---|---|---|
| toan | Toán | Math |
| ngu_van | Ngữ văn | Literature |
| vat_ly | Vật lí | Physics |
| hoa_hoc | Hóa học | Chemistry |
| sinh_hoc | Sinh học | Biology |
| khtn | KHTN | Natural Sciences combined |
| lich_su | Lịch sử | History |
| dia_ly | Địa lí | Geography |
| gdcd | GDCD | Civic Education |
| khxh | KHXH | Social Sciences combined |
| tieng_anh | Tiếng Anh | English |
| tieng_phap | Tiếng Pháp | French |
| tieng_nga | Tiếng Nga | Russian |
| tieng_trung | Tiếng Trung | Chinese |
Missing-in-string → `NULL` in DB (student didn't take that subject).
## Why three parsers
One shared lib + three thin parsers, not one parser with flags. Each dataset's quirks stay visible in its own file:
| File | Source dir | Sheet strategy | Extra guard |
|---|---|---|---|
| `build-database.js` | `data/` | iterate ALL sheets (.xls row overflow) | — |
| `build-database-old.js` | `data-old/` | sheet 0 only | reject rows where `so_bao_danh` isn't digits-only (rejects an `HO_TEN` → `SOBAODANH` header leak present in this export) |
| `build-database-old2.js` | `data-old2/` | iterate ALL sheets (HCM overflow) | skip fully blank rows before counting |
## Overflow-sheet gotcha
The old `.xls` binary format caps at 65,536 rows per sheet. Hanoi and Ho Chi Minh City exceed that in the baotintuc export, so Excel auto-splits the overflow into `Sheet2`. **Only reading `Sheet1` silently drops 13,720 students** (Hanoi +7,275, HCM +6,445).
The audit `node scripts/audit-row-counts.js` catches this by comparing source-row-count (across all sheets) to DB-row-count.
## Audits
Run any time after a rebuild:
```bash
node scripts/audit-row-counts.js # source rows vs DB rows (zero-loss check)
node scripts/check-duplicates.js # md5 + row-content duplicates in data/
node scripts/diff-datasets.js # compare two DBs (needs backup/ dir)
```
Expected baseline:
| DB | Source rows | Skipped | DB rows |
|---|---|---|---|
| main | 861,131 | 63 empty rows | **861,068** |
| old | 847,349 | 1 header-leak | **847,348** |
| old2 | 679,764 | 0 | **679,764** |
## Refreshing the data
```bash
rm data/*.xls # clear current snapshot
node scripts/crawl-baotintuc.js # re-download from CDN
npm run build:db # rebuild
node scripts/audit-row-counts.js # verify
gzip -kf -9 public/thptqg2017.db # compress for shipping
```
## Adding a new source dataset
See `docs/deployment-guide.md` → "Adding a new variant".