refactor: split into web/crawler/go-parser and drop the 2017 archives

Move the frontend into web/, the repo's only npm workspace, and replace the JS crawler with a Go module covering both remaining datasets. The crawler writes to a .part file and renames on completion: writing straight to the destination left truncated files that the skip-if-present check would then skip forever. Remove the 2017-old and 2017-old2 datasets. They were successive publications of the same exam, kept side by side so the disagreement stayed inspectable; the current 2017 supersedes them and they remain in git history. Recover the 2016 crawler source from the Internet Archive's copy of the aggregator article, whose original host no longer resolves. All 119 filenames are verified against data/2016 in both directions, but no archive captured the spreadsheets themselves, so the host still serving them is unconfirmed and data/2016 remains the only confirmed copy. Filenames are load-bearing throughout: go-parser sorts inputs bytewise and inserts last-wins, so they decide which row survives a duplicate exam number.
2026-10-18 12:14:37 +00:00 · 2026-08-13 21:45:05 +07:00
1 parent 8530df0154
commit ceb694a747
@@ -2,32 +2,63 @@

 From raw Excel files to a compressed SQLite file the browser can load.

-One Rust binary (`go-parser/`) builds every dataset. What differs per dataset is
+One Go binary (`go-parser/`) builds every dataset. What differs per dataset is
 parse rules only — sheet strategy, column layout, validation guards — declared
 in `go-parser/configs/<id>.yml`. The table shape, the INSERT and the subject
 regexes are canonical and live in `go-parser/internal/schema/schema.go`.

 ## Sources

-| id | Files | Reproducible | Origin |
+| id | Files | Origin | Host live? |
 | --- | --- | --- | --- |
-| `2016` | 4 `.xls` + 115 `.xlsx` | no | Bộ GD&ĐT, collected 2016 |
-| `2017` | 63 `.xls` | **yes** | baotintuc.vn CDN |
-| `2017-old` | 63 `.xlsx` | no | pre-refresh archive |
-| `2017-old2` | 54 `.xlsx` | no | corrected re-export |
+| `2016` | 4 `.xls` + 115 `.xlsx` | aggregator article, 119 exam clusters | unconfirmed |
+| `2017` | 63 `.xls` | baotintuc.vn CDN | **yes** |

-Only `2017` can be re-fetched:
+Crawling lives in `crawler/`, a separate Go module. It is never part of the
+build — the source files are committed, so a crawl only refreshes them. Both
+runs are idempotent: files already present are skipped.

 ```bash
-node go-parser/scripts/crawl-baotintuc.js
+npm run crawl:2016              # crawler/internal/sources/source_2016.go
+npm run crawl:2017              # crawler/internal/sources/source_2017.go
+npm run crawl:2017 -- --list    # show what would be downloaded, fetch nothing
 ```

-Idempotent — skips files already present, saves to `data/2017/`. Source article:
+**2017** comes from the article
 `https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm`
+and its CDN is still serving the files.

-The other three were collected from Vietnamese news sites at the time and the
-publisher URLs were not recorded. **The files committed in git are the only
-copy.**
+**2016** comes from the aggregator article
+`cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html`. The site
+that originally published it (`dtntbacgiang.edu.vn`) no longer resolves, so the
+link list was read from the Internet Archive's copy of the page. Two caveats
+that matter:
+
+- **The list is verified; the host is not.** All 119 filenames match `data/2016/`
+  exactly in both directions, which is asserted by a test. But the Archive
+  captured only the article, not the spreadsheets, so the URLs themselves could
+  not be confirmed by fetching. `baseURL` in `source_2016.go` points at a mirror
+  of the same article that is still online; if it serves the files from a
+  different directory, `uploadPath` is the one constant to change.
+- **`data/2016/` is still the only confirmed copy.** Do not delete it on the
+  assumption that a crawl can restore it.
+
+### Local filenames are load-bearing
+
+`go-parser` sorts its input files and inserts with `INSERT OR REPLACE`, which is
+last-wins, so **filenames decide which row survives a duplicate exam number**. A
+re-crawl that names files differently can produce a database with the same row
+count and different content, which the row-count guard in `build-db.js` would
+not catch.
+
+Each source therefore pins its local names, and
+`crawler/internal/sources/sources_test.go` checks every source against its
+committed `data/<id>/` in both directions — every name it would write exists,
+and every file present is accounted for.
+
+2017 derives its names from the province (the CDN's own names are inconsistent:
+`Angiang.xls`, `1BaRiaVungTau.xls`, `23HaiPhong.xls`). 2016 keeps the
+server-assigned names verbatim, since it did not choose them.

 ## Source Excel shapes

@@ -41,7 +72,7 @@ The 2017 datasets share one layout:
 | 3 | DIEM_THI | concatenated per-subject scores, e.g. `"Toán: 6.80 Ngữ văn: 5.25 …"` |

 2016 has **three** layouts across its 119 files, chosen per file at runtime via
-`format_detection = "thptqg2016"` in its config:
+`format_detection: thptqg2016` in its config:

 | Format | Detected by | Notes |
 | --- | --- | --- |
@@ -95,8 +126,6 @@ scores rather than false matches.
 | --- | --- | --- |
 | `2016` | all sheets | per-file format detection; header-token rows rejected |
 | `2017` | all sheets — Hà Nội and HCM overflow | none |
-| `2017-old` | first sheet only | reject non-numeric SBD (rejects a header leak in this export) |
-| `2017-old2` | all sheets — HCM overflows | reject non-numeric SBD; skip blank rows before counting |

 ### Overflow-sheet gotcha

@@ -111,8 +140,6 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what
 | --- | --- | --- | --- |
 | `2016` | 877,464 | 3 duplicate SBDs collapsed | **877,461** |
 | `2017` | 861,068 | 0 | **861,068** |
-| `2017-old` | 847,349 | 1 header leak | **847,348** |
-| `2017-old2` | 679,764 | 0 | **679,764** |

 ## Verifying a rebuild

@@ -143,11 +170,14 @@ cell of any input file reads differently.

 ```bash
 rm data/2017/*.xls
-node go-parser/scripts/crawl-baotintuc.js
-node go-parser/scripts/build-db.js 2017
+npm run crawl:2017
+npm run build:db 2017
 ```

-The row-count guard in `build:db` confirms the rebuild matches the expected total.
+The row-count guard in `build:db` confirms the rebuild matches the expected
+total. That guard checks the count only, so if the crawl was expected to change
+the data, compare content with `differential-parity.mjs` against a copy of the
+previous database rather than trusting the count.

 ## Removed scripts

@@ -157,3 +187,9 @@ repo was unified (a hardcoded Windows path in one, an undeclared
 `better-sqlite3` dependency in the other) and neither had any automated caller.
 The latter two are superseded by `differential-parity.mjs`, which compares more
 and cannot silently skip a dataset.
+
+`crawl-baotintuc.js` was not dropped but rewritten in Go as
+`crawler/internal/sources/source_2017.go`, keeping the same link list and the same
+local filenames. The rewrite added a per-file timeout and made downloads land on
+a `.part` file first — writing straight to the destination meant an interrupted
+run left a truncated file that the skip check would then skip forever.
@@ -1,7 +1,7 @@
 # Deployment Guide

 Deploys to GitHub Pages via `.github/workflows/deploy-pages.yml`. Every push to
-`main` rebuilds all four datasets and redeploys the whole site.
+`main` rebuilds both datasets and redeploys the whole site.

 One-time setup: **Settings → Pages → Source: GitHub Actions**.

@@ -9,9 +9,9 @@ One-time setup: **Settings → Pages → Source: GitHub Actions**.

 1. Checkout, Go toolchain, Node 24, `npm ci`
 2. `npm run build:go` — one parser binary
-3. `npm run build:db` — builds and gzips all four databases into
+3. `npm run build:db` — builds and gzips both databases into
   `.build/public/db/`
-4. `npm run build:site` — one Vite build, then `scripts/assemble-site.js`
+4. `npm run build:site` — one Vite build, then `web/scripts/assemble-site.js`
 5. `actions/upload-pages-artifact` + `actions/deploy-pages`

 The database build dominates the runtime: roughly 419 MB of Excel is parsed on
@@ -23,20 +23,18 @@ every deploy.
 https://<user>.github.io/thptqg/
 https://<user>.github.io/thptqg/2016/
 https://<user>.github.io/thptqg/2017/
-https://<user>.github.io/thptqg/2017-old/
-https://<user>.github.io/thptqg/2017-old2/
 ```

-`/thptqg/2017/old/` and `/thptqg/2017/old2/` were the pre-flattening URLs. They
-are still served, and the router rewrites them to the flat form with the query
-string intact.
+`/thptqg/2017/old/` and `/thptqg/2017/old2/` were the pre-flattening URLs for
+the two removed 2017 archives. They are no longer served; like any unknown path
+they now render the hub via `404.html`.

 ## Local reproduction

 ```bash
 npm ci
 npm run build:go
-npm run build:db      # all four; pass an id to build just one
+npm run build:db      # both; pass an id to build just one
 npm run build:site    # vite build + assemble into _site/
 npx serve _site
 ```
@@ -44,7 +42,7 @@ npx serve _site
 To rebuild a single dataset:

 ```bash
-node go-parser/scripts/build-db.js 2017-old
+npm run build:db 2017
 ```

 ## Base path
@@ -59,7 +57,7 @@ up as a blank page with 404s on `/assets/...`.
 2. Add `go-parser/configs/<id>.yml` with the parse rules — sheet mode, column
   indices, SBD validation, header tokens, blank-row stripping. No SQL: the
   schema is canonical and lives in `go-parser/internal/schema/schema.go`
-3. Add an entry to `DATASETS` in `src/datasets.js`
+3. Add an entry to `DATASETS` in `web/src/datasets.js`

 Nothing else. The build script, the site assembly and the router all read that
 one list, and the frontend adapts to whichever columns the dataset populates.
@@ -98,7 +96,7 @@ run rebuilds the older state. There is no data to migrate.
 | Symptom | Typical cause |
 | --- | --- |
 | Blank page, 404 on assets | `base` in `vite.config.js` does not match the repo name |
-| `Failed to fetch database: 404` | Dataset id in `src/datasets.js` does not match the file in `db/` |
+| `Failed to fetch database: 404` | Dataset id in `web/src/datasets.js` does not match the file in `db/` |
 | A route 404s | `assemble-site.js` did not run, or the id is missing from `DATASETS` |
 | WASM fails to load | `sql.js.org` unreachable — self-host `sql-wasm.wasm` and update `SQL_WASM_URL` in `use-sqlite.js` |
 | Deploy fails on assembly | An uncompressed database artefact reached the output; the error names the files |
@@ -4,7 +4,7 @@

 A public lookup tool for Vietnam's National High School Graduation Exam scores,
 running entirely in the browser and hosted for free on GitHub Pages. Covers the
-2016 and 2017 exams — 1.7 million candidates across four datasets.
+2016 and 2017 exams — 1.7 million candidates across two datasets.

 ## Scope

@@ -21,7 +21,7 @@ running entirely in the browser and hosted for free on GitHub Pages. Covers the

 ## Constraints

- **Zero backend.** The full database (38–48 MB gzipped per dataset) is
+- **Zero backend.** The full database (44–48 MB gzipped per dataset) is
  downloaded to the browser and queried in-process.
 - **Read-only.** `INSERT`/`UPDATE`/`DELETE` are rejected, so nobody is misled
  into thinking edits persist. `sql.js` is in-memory anyway.
@@ -36,16 +36,17 @@ running entirely in the browser and hosted for free on GitHub Pages. Covers the
 | --- | --- | --- | --- |
 | `2016` | 2016 | 877,461 | 119 files, three column layouts |
 | `2017` | 2017 | 861,068 | current generation, reproducible from source |
-| `2017-old` | 2017 | 847,348 | pre-refresh archive |
-| `2017-old2` | 2017 | 679,764 | corrected re-export |

-The three 2017 datasets are kept side by side because they disagree, and the
-disagreement is itself informative. Only `2017` is re-fetchable; the rest exist
-solely as the copies committed here.
+Two further 2017 datasets (`2017-old`, `2017-old2`) were kept alongside these
+because the three publications disagreed. They have been removed; git history
+still has them.

-Original 2016 aggregator link
-(<https://dtntbacgiang.edu.vn/tin-tuc/tin-tuc-su-kien/cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html>)
-is no longer accessible, which is why the raw files are mirrored in `data/2016/`.
+Both datasets now have a crawler source. 2017 comes from the baotintuc.vn CDN,
+which is still live. 2016 comes from an aggregator article whose original host
+(`dtntbacgiang.edu.vn`) no longer resolves — its link list was recovered from
+the Internet Archive and is pointed at a mirror that is still online. See
+[data-pipeline](./data-pipeline.md#sources) for what that does and does not
+guarantee.

 ## History

@@ -53,7 +54,7 @@ Each year began as a standalone repository (`thptqg2016`, `thptqg2017`), merged
 here with full history. They initially kept separate frontends and separate
 copies of the same Rust parser, synchronised by hand. That duplication was
 removed: there is now one frontend, one parser, and one canonical schema, with
-per-dataset differences confined to four small config files and one registry
+per-dataset differences confined to one small config file and one registry
 entry each.

 The unification also fixed a latent data-loss bug — neither year's parser
@@ -3,23 +3,24 @@
 Static site, no backend. The browser downloads a compressed SQLite file at boot
 and every query runs locally via `sql.js` (SQLite compiled to WebAssembly).

-One frontend, one parser, one schema, four datasets.
+One frontend, one parser, one schema, two datasets.

 ## Data flow

 ```
+        ▲  crawler/  (Go — manual refresh only, never part of the build)
 data/<id>/*.xls(x)
        │
-        ▼  parser/  (Rust, one binary, one config per dataset)
+        ▼  go-parser/  (Go, one binary, one config per dataset)
   .build/public/db/<id>.db
        │
        ▼  gzip -9   (no -k: the raw file does not survive)
   .build/public/db/<id>.db.gz
        │
-        ▼  vite build   (publicDir = .build/public)
-   dist/
+        ▼  vite build   (root = web/, publicDir = .build/public)
+   web/dist/
        │
-        ▼  scripts/assemble-site.js
+        ▼  web/scripts/assemble-site.js
   _site/   →  GitHub Pages
        │
        ▼  browser
@@ -31,26 +32,28 @@ data/<id>/*.xls(x)
 One identifier ties the whole pipeline together:

 ```
-data/2017-old/  →  go-parser/configs/2017-old.yml  →  db/2017-old.db.gz  →  /thptqg/2017-old/
+data/2017/  →  go-parser/configs/2017.yml  →  db/2017.db.gz  →  /thptqg/2017/
 ```

-`src/datasets.js` declares the four ids once. The frontend, the database build
-(`go-parser/scripts/build-db.js`) and the site assembly all import that list, so
-adding a dataset means adding one entry and one config file.
+`web/src/datasets.js` declares the two ids once. The frontend, the database
+build (`go-parser/scripts/build-db.js`) and the site assembly all import that
+list, so adding a dataset means adding one entry and one config file.
+
+That import crosses a package boundary — go-parser reaches into the web
+workspace for it. It stays there because the same entries also carry the UI's
+labels and SQL presets, and splitting them would mean two lists to keep in step.

 | id | Exam | Rows | Source |
 | --- | --- | --- | --- |
 | `2016` | 2016 | 877,461 | Bộ GD&ĐT |
 | `2017` | 2017 | 861,068 | baotintuc.vn |
-| `2017-old` | 2017 | 847,348 | pre-refresh archive |
-| `2017-old2` | 2017 | 679,764 | corrected re-export |

 ## Canonical schema

 Defined once in `go-parser/internal/schema/schema.go` — DDL, INSERT, column order and the 16
-subject regexes. The four TOML configs carry no SQL at all, only per-dataset
-parse rules. Config parsing uses `deny_unknown_fields`, so a leftover `[schema]`
-block fails loudly instead of looking effective while `schema.rs` drives the
+subject regexes. The two YAML configs carry no SQL at all, only per-dataset
+parse rules. Config parsing sets `KnownFields(true)`, so a leftover `schema:`
+block fails loudly instead of looking effective while `schema.go` drives the
 build.

 ```sql
@@ -73,7 +76,7 @@ CREATE INDEX idx_ten_cum_thi  ON student(ten_cum_thi) WHERE ten_cum_thi IS NOT N

 Every dataset gets all 22 columns; ones it has no data for are NULL, costing
 about a byte per row. `khtn`, `khxh` and `gdcd` are empty on 2016;
-`ten_cum_thi` and `gioi_tinh` are empty on the 2017 datasets.
+`ten_cum_thi` and `gioi_tinh` are empty on 2017.

 `idx_ten_cum_thi` is partial, so it holds zero entries where the column is
 always NULL.
@@ -86,15 +89,13 @@ URLs are flat, one segment per dataset, and the segment is the id:
 /thptqg/            hub
 /thptqg/2016/
 /thptqg/2017/
-/thptqg/2017-old/
-/thptqg/2017-old2/
 ```

-`src/router.js` is an exact match on that segment. The nested form used before
-(`/thptqg/2017/old/`) would have needed longest-prefix matching, since it also
-starts with `/thptqg/2017/`. Those two legacy URLs still resolve: the router
-rewrites them to the flat equivalent with `history.replaceState`, preserving the
-query string so `?q=` deep links survive.
+`web/src/router.js` is an exact match on that segment. The nested form used
+before (`/thptqg/2017/old/`) would have needed longest-prefix matching, since it
+also starts with `/thptqg/2017/`. Both of those URLs addressed the two removed
+2017 archives, so the rewrite that kept them working has gone with them; they
+fall through to the hub like any other unknown path.

 A single Vite build emits one `index.html`. Because `base` is absolute
 (`/thptqg/`), that file references `/thptqg/assets/...` regardless of the
@@ -115,11 +116,11 @@ No component contains a per-dataset conditional. Two mechanisms do the work:
  self-exclude wherever those languages were not sat.

 Anything genuinely per-dataset — title, source, database size, search examples,
-SQL presets — lives in `src/datasets.js`.
+SQL presets — lives in `web/src/datasets.js`.

 ## Exam ID formats

-`src/lib/query-mode.js` decides whether a query is an exam ID or a name, and is
+`web/src/lib/query-mode.js` decides whether a query is an exam ID or a name, and is
 shared by `App.jsx` and `search-form.jsx` (they previously held separate copies
 and had drifted apart on exactly this rule).

@@ -135,7 +136,7 @@ candidates (70.3%). Letter prefixes are upper-cased before lookup, so

 ## Score tiers

-Six-level ladder in `scoreTier()` (`src/lib/admission-blocks.js`), paired with a
+Six-level ladder in `scoreTier()` (`web/src/lib/admission-blocks.js`), paired with a
 symbol so meaning is never colour-only.

 | Tier | Range | Vietnamese |
@@ -150,7 +151,7 @@ symbol so meaning is never colour-only.
 ## Admission blocks

 Vietnamese universities admit on three-subject combinations (khối thi).
-`src/lib/admission-blocks.js` lists the blocks computable from this schema
+`web/src/lib/admission-blocks.js` lists the blocks computable from this schema
 (A00–A11, B00–B08, C00–C20, D01–D15, plus D05/D06 for German and Japanese).
 `computeBlocks(student)` returns those where all three scores exist, sorted by
 total descending.
@@ -165,11 +166,11 @@ total descending.
 | Diacritics search | Pre-computed `ho_ten_ascii` | `LOWER(REPLACE(...))` at query time defeats the index |
 | SQL safety | Leading-keyword allowlist | `sql.js` is in-memory so writes cannot persist; the allowlist prevents confusion |
 | Row caps | 100 (lookup), 1000 (SQL) | Keeps DOM render sizes reasonable |
-| Routing | Hand-rolled, ~20 lines | Five static routes do not justify a router dependency |
+| Routing | Hand-rolled, ~15 lines | Three static routes do not justify a router dependency |

 ## Risks and limitations

- **Database size.** 38–48 MB gzipped per dataset; slow links wait, mitigated by
+- **Database size.** 44–48 MB gzipped per dataset; slow links wait, mitigated by
  a progress bar.
 - **Browser memory.** The full database lives in RAM; older mobile devices may
  run out.
@@ -177,4 +178,4 @@ total descending.
  load. Self-hosting `sql-wasm.wasm` and updating `SQL_WASM_URL` in
  `use-sqlite.js` is the fix.
 - **Excel format drift.** A new source file with an unseen header layout needs a
-  new branch in `format_detect_2016.rs` or a new config.
+  new branch in `go-parser/internal/ingest/detect2016.go` or a new config.