mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-15 12:15:01 +00:00
refactor: collapse the two projects into one tree and move to npm
The repo held two near-duplicate projects. 2016/ and 2017/ each carried their
own React frontend, their own copy of the same Rust crate, and their own
package manager setup. 2016/tools/sync-from-thptqg2017.sh existed purely to
copy the parser source between them.
New layout:
index.html + src/ the 2017 frontend, now the only one
data/<id>/ 2016, 2017, 2017-old, 2017-old2
parser/ the single Rust crate, configs renamed to <id>.toml
docs/ both projects' docs, 2016 copies suffixed -2016-legacy
pending the merge pass
<id> is now one identifier end to end: data/<id>/ feeds parser/configs/<id>.toml
and produces db/<id>.db.gz.
pnpm gives way to npm. pnpm-workspace.yaml existed only to whitelist
better-sqlite3's native build, which npm permits by default, so it has no
equivalent and is simply gone. Lockfiles cannot be converted; package-lock.json
is generated fresh. The migration direction is safe — pnpm's strict layout
forbids phantom dependencies, so anything that resolved under pnpm resolves
under npm's flat tree.
Adds parser/scripts/build-db.js and src/datasets.js: the four dataset IDs are
declared once and read by both the build tooling and (from the next phase) the
frontend.
Follow-on fixes the move made necessary:
- eslint's Node-globals override pointed at scripts/, now parser/scripts/
- crawl-baotintuc.js wrote to <root>/data, now data/2017
- golden tests loaded configs by their old thptqg*-data.toml names
Drops the #[ignore]d Rust-vs-Node golden test. It shelled out to pnpm to run
scripts/build-database.js, a file removed when the parser was ported to Rust,
so it could never pass. check-duplicates.js and diff-datasets.js were already
broken before this change and are annotated as such rather than half-fixed.
63 Rust tests pass and clippy is clean from the new location.
This commit is contained in:
1 parent
82db8295bb
commit
6ff2ed99ec
396 files changed
+2753
-8720
No files matched your search
@@ -0,0 +1,110 @@
|
||||
# Data Pipeline
|
||||
|
||||
From raw Excel files to a compressed SQLite file the browser can load.
|
||||
|
||||
## Canonical source (data/)
|
||||
|
||||
63 Excel files from baotintuc.vn CDN. Source article:
|
||||
`https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm`
|
||||
|
||||
Re-download anytime:
|
||||
|
||||
```bash
|
||||
node scripts/crawl-baotintuc.js
|
||||
```
|
||||
|
||||
Idempotent — skips files already present. Saves to `data/<ascii-kebab-province>.xls`.
|
||||
|
||||
## Reference sources (data-old/, data-old2/)
|
||||
|
||||
Both were collected from Vietnamese news sites around the time of the 2017 exam.
|
||||
Specific publisher URLs were not recorded and cannot be recovered, so these
|
||||
datasets are **not reproducible** — the files committed in git are the only
|
||||
copy. They are kept for historical comparison and to expose the deploy matrix;
|
||||
the main deployment always uses the reproducible `data/` source.
|
||||
|
||||
## Source Excel shape
|
||||
|
||||
Every file is a single sheet (for `data-old/`) or up to 3 sheets (for `data/` .xls) with columns:
|
||||
|
||||
| Col | Name | Content |
|
||||
|---|---|---|
|
||||
| 0 | HO_TEN | full name in Vietnamese |
|
||||
| 1 | NGAY_SINH | `dd/mm/yyyy` |
|
||||
| 2 | SOBAODANH | 8-digit string, first 2 digits = province |
|
||||
| 3 | DIEM_THI | concatenated per-subject score string in the source language, e.g. `"Toan: 6.80 Van: 5.25 ... English: 5.80"` |
|
||||
|
||||
Some files lack a header row. `isHeaderRow()` in `build-lib.js` detects and skips.
|
||||
|
||||
## Score text parsing
|
||||
|
||||
`SCORE_PATTERNS` in `build-lib.js` defines one regex per subject. The patterns must match the exact subject labels the source files use (Vietnamese).
|
||||
|
||||
Subjects supported (DB column / label in source text):
|
||||
|
||||
| Column | Source label | English |
|
||||
|---|---|---|
|
||||
| toan | Toán | Math |
|
||||
| ngu_van | Ngữ văn | Literature |
|
||||
| vat_ly | Vật lí | Physics |
|
||||
| hoa_hoc | Hóa học | Chemistry |
|
||||
| sinh_hoc | Sinh học | Biology |
|
||||
| khtn | KHTN | Natural Sciences combined |
|
||||
| lich_su | Lịch sử | History |
|
||||
| dia_ly | Địa lí | Geography |
|
||||
| gdcd | GDCD | Civic Education |
|
||||
| khxh | KHXH | Social Sciences combined |
|
||||
| tieng_anh | Tiếng Anh | English |
|
||||
| tieng_phap | Tiếng Pháp | French |
|
||||
| tieng_nga | Tiếng Nga | Russian |
|
||||
| tieng_trung | Tiếng Trung | Chinese |
|
||||
|
||||
Missing-in-string → `NULL` in DB (student didn't take that subject).
|
||||
|
||||
## Why three parsers
|
||||
|
||||
One shared lib + three thin parsers, not one parser with flags. Each dataset's quirks stay visible in its own file:
|
||||
|
||||
| File | Source dir | Sheet strategy | Extra guard |
|
||||
|---|---|---|---|
|
||||
| `build-database.js` | `data/` | iterate ALL sheets (.xls row overflow) | — |
|
||||
| `build-database-old.js` | `data-old/` | sheet 0 only | reject rows where `so_bao_danh` isn't digits-only (rejects an `HO_TEN` → `SOBAODANH` header leak present in this export) |
|
||||
| `build-database-old2.js` | `data-old2/` | iterate ALL sheets (HCM overflow) | skip fully blank rows before counting |
|
||||
|
||||
## Overflow-sheet gotcha
|
||||
|
||||
The old `.xls` binary format caps at 65,536 rows per sheet. Hanoi and Ho Chi Minh City exceed that in the baotintuc export, so Excel auto-splits the overflow into `Sheet2`. **Only reading `Sheet1` silently drops 13,720 students** (Hanoi +7,275, HCM +6,445).
|
||||
|
||||
The audit `node scripts/audit-row-counts.js` catches this by comparing source-row-count (across all sheets) to DB-row-count.
|
||||
|
||||
## Audits
|
||||
|
||||
Run any time after a rebuild:
|
||||
|
||||
```bash
|
||||
node scripts/audit-row-counts.js # source rows vs DB rows (zero-loss check)
|
||||
node scripts/check-duplicates.js # md5 + row-content duplicates in data/
|
||||
node scripts/diff-datasets.js # compare two DBs (needs backup/ dir)
|
||||
```
|
||||
|
||||
Expected baseline:
|
||||
|
||||
| DB | Source rows | Skipped | DB rows |
|
||||
|---|---|---|---|
|
||||
| main | 861,131 | 63 empty rows | **861,068** |
|
||||
| old | 847,349 | 1 header-leak | **847,348** |
|
||||
| old2 | 679,764 | 0 | **679,764** |
|
||||
|
||||
## Refreshing the data
|
||||
|
||||
```bash
|
||||
rm data/*.xls # clear current snapshot
|
||||
node scripts/crawl-baotintuc.js # re-download from CDN
|
||||
npm run build:db # rebuild
|
||||
node scripts/audit-row-counts.js # verify
|
||||
gzip -kf -9 public/thptqg2017.db # compress for shipping
|
||||
```
|
||||
|
||||
## Adding a new source dataset
|
||||
|
||||
See `docs/deployment-guide.md` → "Adding a new variant".
|
||||
Reference in new issue
Block a user